Running bcd on real hardware

Most of bcd is tested against fakes: a fake HGX node (fakehw), a fake host for the config units, fake virsh, qemu-img and nvidia-smi. Round 6 (2026-10-04, bbfa7f4 onwards) ran it on a real machine for the first time and fixed what that found. This page is for the operator bringing bcd up on a new server or workstation: the order to do things in, the two local commands that tell you what bcd sees (bcd hardware, bcd doctor), how the privilege mode changes what bcd can do, what is special about WSL 2, and an exact list of what has and has not run on real hardware. Installing the binary and the systemd units is on installing bcd; the GPU side in depth is on GPUs on a node.

Which build

Real hardware needs a host build: one without the fakehw tag, which reads /proc, /sys and nvidia-smi.

Both host builds read GPUs through nvidia-smi; the nonvml build only makes doctor's nvml check warn (see the NVML backend). bcd version prints the build tags. bcd hardware says which backend it runs (host or fakehw) in its first line, and --capture refuses a fakehw build.

cd daemon
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags nonvml -o bin/bcd ./cmd/bcd

Bring-up checklist

1. Firmware. Turn on VT-x or AMD-V for VMs, and VT-d or AMD-Vi for GPU passthrough. Under Secure Boot the NVIDIA kernel module must be signed by an enrolled key.
2. Operating system. systemd as init, cgroup v2 (unified) mounted at /sys/fs/cgroup, an NTP client, and nouveau blacklisted on a GPU host. Install the NVIDIA driver yourself, or leave it to the nvidia-driver unit.
3. Look before joining. Copy a host build of bcd onto the machine and run sudo bcd hardware. Read the findings: a missing product UUID, placeholder serials or no NICs change how the machine can join (below).
4. Preflight. Write /etc/bcd/config.toml (the install script does) and run sudo bcd doctor --config /etc/bcd/config.toml. Fix every fail line. The install script runs doctor too, but only prints its warning.
5. Pick the privilege mode. Root (the install script's default) or --unprivileged with bcd helper. Running bcd node as an ordinary user without the helper works for trying bcd out, but takes no GPU holds (below).
6. Join. A machine whose DMI gives no product UUID or serials (WSL, many VMs, boards with unprogrammed SMBIOS) has no trusted hardware key, and JoinService refuses it unless the join token is bound to an identifier it does have: create the token with expect: { mac } (--expect-mac on the CLI), the MAC of its default-route NIC (see binding a token to one machine).
7. Check the node. nodes.get lists its conditions. GpuModelUnknown keeps a GPU node from scheduling until the control plane's model table knows the card (see GPU naming); GpuCountMismatch means the pool expected another count or model.
8. Watch it run. bcd status shows the session, the generations, each section and one line per GPU (see bcd status).

bcd hardware

bcd hardware (e82d3a8) prints what this binary sees of the machine, without joining or talking to anything: the inventory bcd would report and a list of bring-up findings. Run it as root: the product UUID, serials, DIMMs and NVMe health are root-only, and an ordinary user reads them empty.

The summary has one line per system, CPU (with its NUMA nodes and isolated CPUs), memory (with each DIMM), disk, NIC, GPU, BMC and OS:

bcd <version> (host backend)

System    <vendor> <product> (board <vendor> <board>), BIOS <vendor> <version>, UEFI (Secure Boot), TPM 2.0
CPU       <model> (<vendor> x86_64): 2 sockets, 96 cores, 192 threads
  node0   cpus 0-47,96-143, 503.8 GiB
Memory    1007.6 GiB
  DIMM    <slot>       64.0 GiB DDR5 4800 MT/s ECC <manufacturer> <part>
Disk      nvme0n1    3.5 TiB   nvme <model> (serial <serial>, wwn <wwn>)
NIC       eth0       <mac> mlx5_core 0000:18:00.0 100000 Mb/s fw <version>
GPU 0     NVIDIA H100 80GB HBM3 00000000:1B:00.0 GPU-<uuid>
BMC       <vendor> <mac> <ip> (IPMI, Redfish) fw <version>
OS      ubuntu 24.04, kernel <release>, cgroup v2, runtimes containerd,libvirt

Findings
  ok    dmi       <vendor> <product>, UUID <uuid>
  ...

The GPUs come from the same collection bcd node makes (procfs, then nvidia-smi; see where GPU facts come from), except that bcd hardware keeps no memory of GPUs, so a GPU passed through to a VM at that moment is not listed. A capture copies procfs and sysfs only; nvidia-smi's answers are not part of it (GPU fixtures are captured separately, under daemon/internal/nvml/testdata).

Findings

inventory.HostFindings reviews the collection. Every finding has a status (ok, warn, fail), and bcd hardware exits 1 when one fails:

Round 6 fixed what the first real machine showed (e82d3a8): disks behind Hyper-V's storvsc, iSCSI or a RAID controller's virtual drives are no longer called SAS; SCSI and SATA serials come from the kernel's vpd_pg80; a NIC on VMBus has no PCI address; SR-IOV virtual functions and a BMC's Redfish USB host interface are not the machine's NICs; and arm64 topology comes from sysfs, with core names from the MIDR part (Grace reads Neoverse-V2). Later commits added DIMMs from SMBIOS (82fef0e), NIC firmware (9b83ece), NVMe health (362435e), SATA and SAS health through smartctl (97089e9) and the BMC (9ddb5ea).

bcd doctor as preflight

bcd doctor --config FILE checks the configuration, nvidia-smi, the inventory, the host and the node's identity, one line per check, and exits 1 when any check fails. The checks it always had are listed on the overview, the GPU checks on GPUs on a node. Since e2a0cf0 a host build ends with a real-host preflight (doctor_host.go):

The GPU checks of f84f1f9 (gpu-bus, nouveau, persistence, fabric-manager, gpu-host) and da60bbd (gpu-binding) run in the same pass. On the round-6 workstation doctor reported nvml ok (616.56), the gpu line ok, gpu-host warning about WSL, systemd ok, the clock synchronized, hugetlb not yet delegated and no DMI.

Root, bcd helper or unprivileged

bcd runs in one of three ways, and they differ in what reaches the control plane:

dmi.read and hw.read (29ed48d, 32beca7, 45bc17f) are read-only and take no arguments: the helper enumerates the entries and controllers itself and never lets the node name a path. What they return is inventory only, never an attestation input.

WSL 2

WSL 2 is a convenient place to try bcd on a GPU workstation, and it is where round 6 ran. It is a Hyper-V VM with a Linux kernel, and differs from a server in ways bcd notices:

testdata/hosts/wsl2-threadripper-9995wx.tar.gz in daemon/internal/inventory is that machine, captured with bcd hardware --capture … --redact: the first real-host fixture, replayed by the inventory tests.

What has run on real hardware

Every item here ran on the round-6 workstation, as an ordinary user unless it says otherwise.

# Linux, or Windows with WSL (BCD_WSL_DISTRO, default Ubuntu); builds bcd with -tags nonvml first
BCD_E2E_REAL=1 pnpm vitest run tests/e2e/bcd-host-real.test.ts
# The real-GPU Go tests (scripts/daemon-test.mjs forwards BCD_* into WSL)
BCD_REAL_GPU=1 node scripts/daemon-test.mjs --run TestSnapshotLiveHost ./internal/nvml/
BCD_REAL_GPU=1 node scripts/daemon-test.mjs --run TestRealGPUUnitsPlan ./internal/hostconfig/

The suite asserts only what holds on any Linux host and logs the rest (GPU count, KVM), so it runs on a laptop, in WSL or on a GPU server; its GPU tests skip without nvidia-smi.

What has not

• A GPU hold on real hardware. bcd ran unprivileged without [helper], so no CDI spec was written to a real /etc/cdi, no hold slice was created on a real systemd, and the reservation test took its unprivileged branch.
• A container from a CDI name. The workstation has no nvidia-ctk, so no vendor spec was generated and no container was started with bcd.dev/gpu=<id>.
• Passthrough. No IOMMU, so no GPU was bound to vfio-pci, and the persistence-mode sequence around it (f25abbe, 5bb9f22) and the passed-through GPU memory (c5c52c4) are tested against fakes only. Neither libvirt-kvm nor cloud-hypervisor has run a real VM.
• MIG, NVLink, NVSwitch, Fabric Manager, MPS. The only GPU is a GeForce card without them.
• Data-center GPUs, ECC and row remapping, XIDs, gpu.reset, DCGM. None on a real board.
• Applying host units. Driver installs, MIG layouts, boot arguments, 1 GiB pages, power limits and Kubernetes joins are planned against the real driver at most, never applied.
• The TPM. No TPM on the workstation; every TPM test uses the software TPM.
• Root-only readers. The NVMe and IPMI ioctl paths, DIMMs from SMBIOS and smartctl have not run as root on a real server.
• A released binary through the install script. No release has been tagged (see installing bcd).