Tutorial: a GPU cluster end to end

Act as the platform operator

Point the CLI at the API and give it the token of a root administrator. With BETTER_COMPUTE_URL set, every command is an HTTP call to that server (plain http is allowed only for loopback addresses); BETTER_COMPUTE_TENANT is the default --tenant of every token command.

export BETTER_COMPUTE_URL=https://compute.example.com
export BETTER_COMPUTE_TOKEN='biam_…'      # a root administrator's token
export BETTER_COMPUTE_TENANT=root         # the root tenant's id

better-compute api system.whoami

better-compute help lists every command, and better-compute help <command> shows one; see the CLI for credentials, output formats and --query.

Create a site and a pool

A site is where machines live; a pool is a set of machines of one GPU model with one policy. A platform site is shared (offered to tenants) or dedicated, and only the root tenant creates those (sites).

export SITE=$(better-compute sites create --kind shared --region eu-1 --key site-eu-1 --query id)

The pool's policy decides what a joining machine must pass before it takes work. Write it to a file:

{
  "acceptance": [{ "kind": "diag.nvml" }],
  "expect": { "gpuCount": 8 }
}

export POOL=$(better-compute pools create --site $SITE --class metal-host --gpu-model H100 \
  --policy-file pool-policy.json --key pool-eu-1-h100 --query id)
better-compute pools get $POOL

• --class metal-host makes joining machines host nodes, whose GPUs are reserved as holds (classes). The agent defaults to bcd, which join tokens require.
• --gpu-model is the GPU family inventory derives from the card's name, such as H100 for H100-SXM5-80GB. A pool holds one model (one GPU model).
• diag.nvml with expect.gpuCount adopts a machine only when it shows 8 healthy GPUs (see acceptance checks). Set the count to what your server has.
• --key makes the call retry-safe: the same key returns the same site or pool.

The pool's owner, here the root tenant, gets an effectively unlimited gpu quota on it. To let a customer tenant reserve from it, offer it with a quota. The resource is gpu (singular): a quota row for any other name grants nothing.

better-compute pools offer $POOL --to acme --quota gpu=16

See offerings.

Mint a join token

A join token lets a limited number of machines join one pool for a limited time. Bind it to the machine you mean when you know one of its identifiers, and write it to a file instead of the terminal:

better-compute join-tokens create --pool $POOL --ttl 2h --max-uses 1 \
  --expect-serial SN-7701 --key join-sn-7701 --out ./join-token

--out writes a new file with mode 0600 and never overwrites one. The token is shown once; the control plane keeps only a hash of its secret. --ttl takes 1m to 24h. A machine whose firmware reports no product UUID or serial (WSL 2 and many VMs) has no trusted hardware key, and a token without an expect binding is refused for it; bind such a token with --expect-mac instead, as tests/e2e/bcd-host-real.test.ts does. See binding a token to one machine.

Join the server with bcd

Build bcd for the server. The default build reads GPUs through nvidia-smi and needs no cgo:

cd daemon
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/bcd ./cmd/bcd

There are two ways to install it.

With the install script. Serve the binary from the API (downloads, which the examples set from BETTER_COMPUTE_DOWNLOADS_DIR; see serving the binaries) under goreleaser's file name bcd_<version>_linux_amd64, then generate a script that pins its digest and the PKI's keys:

better-compute daemon install-script --pool $POOL --site $SITE \
  --gateway-url https://gw.eu-1.example.com:8443 --base-url https://compute.example.com \
  --bcd-version 0.1.0 --sha256 amd64=$(sha256sum bin/bcd | cut -d' ' -f1) --out install-bcd.sh

Copy install-bcd.sh and join-token to the server and run the script as root, with the token on standard input:

sudo sh install-bcd.sh < join-token

It checks the host, downloads and verifies bcd, writes /etc/bcd/config.toml with the pins, runs bcd doctor, pipes the token into bcd join and enables bcd.service. See what the script does.

By hand, from the well-known document. When the control plane publishes its gateway (BETTER_COMPUTE_GATEWAY_URL in the examples), a machine with no config.toml can join with the control plane's URL and three pins. Print the pins on the control plane, from the sealed bundle:

better-compute pki show --query servingCaSpkiPin
better-compute pki show --query joinSigningKeyPin
better-compute pki show --query joinEncryptionKeyPin

Then, on the server, as root, with bcd copied to /usr/local/bin/bcd:

bcd join --server https://compute.example.com/api/compute \
  --serving-ca-pin 'sha256/…' --join-signing-pin 'sha256/…' --join-encryption-pin 'sha256/…' \
  --token-file ./join-token
bcd node --config /etc/bcd/config.toml

bcd join reads <server>/.well-known/better-compute, refuses it when a published pin differs from yours, joins, and only then writes /etc/bcd/config.toml and join-encryption.jwk. --token-file takes only a regular file its owner alone can read (0600 or 0400), and deletes it after the join. The token is never an argument: bcd reads it from BCD_JOIN_TOKEN, standard input or --token-file, and refuses positional arguments. On success it prints joined as node_… (site site_…, role node); certificate valid until …, and a second run prints already joined as node_… without spending a token. Run bcd node under systemd in production; the install script's unit is in the systemd unit. See joining from the well-known document.

What happens inside the join (the sealed token, the signed JoinResponse, the node certificate) is in the join flow.

Watch the node get adopted

The node is created in adopting. Once its session is up, the NodeLifecycle controller sends the pool's acceptance checks; when diag.nvml passes, the node becomes adopted, gets desired-state generation 1, and the placer starts to consider it.

better-compute nodes list --pool $POOL
export NODE=node_…   # the id nodes list printed (or bcd join)
better-compute nodes get $NODE --query provisionState
better-compute nodes get $NODE --query conditions
better-compute nodes events $NODE

nodes events shows node.joined, node.connected, node.acceptance.started and node.adopted in that order. On the server, bcd status reads the running agent over its unix socket (/run/bcd/bcd.sock by default):

sudo bcd status            # session, spec generations, sections, GPUs
sudo bcd status --json

The text form shows the node, its session (connected (epoch …)), the applied and observed spec generations, one line per section, and one line per GPU with its utilization, temperature, power and memory, and held by <allocation> once a hold has it. See bcd status and telemetry and status.

When the node stays adopting:

A failed acceptance is sticky. Fix the cause and start a new attempt with nodes.adopt, which has no command of its own; call it through api:

better-compute api nodes.adopt tenantId=$BETTER_COMPUTE_TENANT nodeId=$NODE reason='GPU 3 reseated'

Check capacity

better-compute capacity --pool $POOL
better-compute capacity --preview --pool $POOL --gpus 4 --for 2h

The rollup reports, per resource, total, allocatable, allocated, reserved and available, plus the GPUs that are provisioning, unhealthy, in maintenance or unreachable. One adopted 8-GPU server gives 8 allocatable GPUs. A preview books nothing; it answers whether the request would fit now, with the host CPUs and memory it would book.

Reserve GPUs

better-compute reserve --pool $POOL --gpus 4 --for 2h

reserve calls reservations.create with a new idempotency key, which it prints on standard error; run it again with --key <that key> to retry without booking twice. It waits up to --wait seconds (default 20) for the placement. With the default placer the answer is already placed: the placer picked the node, locked its GPUs and booked four of them, together with the pool's per-GPU share of the host's CPUs and memory (see host resources). Other flags narrow the choice: --model, --product, --selector "vramGiB >= 80", --nvlink for one NVLink domain, --start +2h for a later start. See reservations.

The reservation then binds on its own:

At the start, the node's desired state gains a device assignment for the allocation. bcd writes the hold's CDI spec and env file, then journals an AllocationBound signed with its node key; once the control plane verified it, the reservation is active.

export RSV=rsv_…   # the id reserve printed
better-compute reservations get $RSV --query state
better-compute reservations get $RSV --query binding

{
  "nodeId": "node_…",
  "cdiDevices": ["bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…"],
  "envFile": "/run/bcd/allocations/alloc_….env",
  "cgroupParent": "bcd-alloc-alloc_….slice"
}

tests/e2e/spine.test.ts checks this end to end (3c3a53f): the CDI device names reservations.get returns are exactly the GPU UUIDs the database booked, and the node's CDI spec lists exactly those GPUs. A hold that fails on the node (overlap, device-missing, resources-exhausted) is placed again, at most twice; no report within 5 minutes counts as a failure (bindings).

Use the GPUs on the node

The binding is a hold: the GPUs are set aside for the reservation, and a workload manager on the node uses them by their CDI names. On the server:

ls /etc/cdi/                                  # bcd-alloc_….json beside nvidia.json
cat /etc/cdi/bcd-alloc_….json
cat /run/bcd/allocations/alloc_….env

Since 6ecb860 a hold's spec also carries the device nodes, mounts and hooks of exactly the held GPUs, copied from NVIDIA's own CDI spec, /etc/cdi/nvidia.json. So the name bcd.dev/gpu=<uuid> alone gives a container its GPUs and nothing else. bcd writes that vendor spec with the container-toolkit unit when a profile names it; on a server you adopted, install the NVIDIA container toolkit and generate the same spec once, as the unit does:

nvidia-ctk cdi generate --format=json --device-name-strategy=uuid > /etc/cdi/nvidia.json

Without a vendor spec a hold carries only NVIDIA_VISIBLE_DEVICES. Any CDI-aware runtime then takes the names as devices. For example, with Podman, placing the container in the hold's slice:

set -a; . /run/bcd/allocations/alloc_….env; set +a
podman run --rm --device bcd.dev/gpu=GPU-… --device bcd.dev/gpu=GPU-… \
  --cgroup-parent "$BCD_CGROUP_PARENT" nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -L

bcd does not move processes into the slice and never drops a hold on its own, not even while disconnected; the workload manager (containerd, Slurm, a wrapper) places the workload. For Kubernetes, reserve with --kubernetes on a pool with policy.kubernetes, and bcd's device plugin hands the held GPUs to the kubelet as bcd.dev/gpu (see Kubernetes workers). The node side is described in device assignments.

Follow the events

Every step above published events through the transactional outbox. With a host that mounts the events module's GET /watch (the examples do), follow them live, one JSON line each:

better-compute events watch --types 'node.*,reservation.*'

Run it in a second terminal before step 7 to see the reservation move: reservation.placed, reservation.binding, reservation.active, and after step 10 reservation.released. Each line carries a sequence; --since <sequence> resumes after it. Without a /watch route, page through the same events with better-compute events list or better-compute nodes events $NODE. See watch and the event reference.

Release

better-compute release $RSV
better-compute capacity --pool $POOL

release calls reservations.release: the bookings end at the current database time, the quota is refunded, and the device assignment leaves the node's desired state. bcd removes the CDI spec and the env file, reports AllocationReleased signed with its node key, and stops the hold's slice once it is empty, so a release never kills a running workload: stop it first. available goes back up. A reservation that has not started is withdrawn with release --cancel; one that should run longer is extended with better-compute extend $RSV --by 2h (start, lease, extend and release).