# Delivery (/docs/daemon/delivery) > How NodeSpec generations and one-shot commands reach bcd, the acceptance checks, lanes and credits, cancellation, signatures, and the journal that makes delivery safe. The control plane tells a node what to do in two ways, and the difference matters. **Desired state** is a standing fact: which devices are held for which allocation, which instances should run, which SSH CA to trust. **Commands** are one-shot actions: reboot, run diagnostics, collect logs. The rule from PLAN.md §7.3: > If losing it leaves the wrong steady state, it is desired state. If running it late could do harm, it is a > command with `not_after`, plus `min_generation` when it presupposes a spec. Both travel down the node's [session](/docs/daemon/sessions), and both are delivered at least once. The daemon's journal turns at-least-once delivery into at-most-once effect. The concepts are introduced in [desired state and commands](/docs/guides/concepts/desired-state-and-commands); this page is the wire-level detail. ## Desired state [#desired-state] ### NodeSpec and its sections [#nodespec-and-its-sections] A NodeSpec is split into sections. The daemon converges them in this order, each with an idempotent `plan(desired, observed)` and `apply`: | Section | Holds | Phase | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | | `host` | Config units (kernel modules, sysctl, huge pages, the NVIDIA driver, Fabric Manager, the MIG layout, GPU sharing, the container toolkit, a Kubernetes join), each a unit name and its settings as canonical JSON; see [host configuration](/docs/daemon/host-config) | v0.2 | | `devices` | Device assignments: allocation, reservation, binding, devices, extra env lines | v0.1 | | `instances` | VPS instances with their runtime, size, image, NICs and GPUs | v0.3 | | `access` | The exec-ticket JWKS, SSH CA keys and principals, revoked certificate serials; filled when `tunnels` is configured | v0.3 | | `telemetry` | The OTLP endpoint, export interval, resource attributes and a secret reference for headers | v0.1 | | `daemon` | The self-update target: version, URL, SHA-256, signature | v0.4 | | `site` | The site agent's boot lease state and `cache_images`, the images its cache pre-warms (`url`, `sha256`, `format`; the agent re-verifies each digest before serving it); draft, site agents only | v0.2 | The control plane builds the spec in the NodeSpec reconciler: the base sections plus every module's `nodeSpec()` contribution (a `NodeSpecPart`; the `site` section merges field by field across modules), stored in `nodes.spec` as protobuf JSON, with `cache_images` sorted by URL. It raises `nodes.generation` only when the canonical JSON (sorted keys) differs from the stored spec, so re-running the reconciler never produces a new generation for nothing. Desired state applies once a node is adopted or deployed. ### Generations [#generations] Three numbers describe where a node stands: | Number | Owner | Meaning | | --------------------- | ----------------------------------------- | --------------------------------------------------------------------- | | `generation` | control plane, `nodes.generation` | The newest spec written | | `applied_generation` | daemon, in `Hello`, heartbeats and status | The newest spec the daemon holds, converged or not | | `observed_generation` | control plane, from status reports | The newest spec at which every section converged or failed terminally | `observed_generation` is written with `WHERE observed_generation < $g`, so it never moves backwards, whichever session a late report arrives on. Controllers that wait for the node to act on a spec wait for it. ### How a spec reaches the node [#how-a-spec-reaches-the-node] * **At Hello**, when `generation` is above `applied_generation`, the gateway sends a full `DesiredState` before replaying commands. * **On change**, the transaction that wrote a new generation pokes the node's gateway, which pushes the spec to that node's session at once. * **As a backstop**, every 10 seconds the gateway compares each open session with `nodes.generation` and sends a full spec to any node that is behind both what it holds and what was already sent on this session. * **On request**, the daemon sends `SpecRequest{have_generation, reason}` and gets the full spec back. | SpecRequest reason | When | | --------------------- | ---------------------------------------------------------------------------------- | | `PATCH_BASE_MISMATCH` | A patch's `base_generation` is not the applied generation | | `MIN_GENERATION` | A command's `min_generation` is above the applied generation | | `NO_SPEC` | The journal holds no usable spec (new journal, corruption, failed signature check) | A `DesiredStateBody` carries the node id, the generation, a `spec_hash` and either a `full` NodeSpec or a `patch`. A patch applies against `base_generation`: every section it sets replaces that section wholesale (an empty section clears it), unset sections stay. The daemon ignores a body for another node and any generation not above the one it holds. It never recomputes `spec_hash`; it echoes it back as `applied_spec_hash`. The FakeDaemon applies both forms. The gateway in this release sends full specs only (`base_generation` 0). When the NodeSpec controller stored the encoded body of the current generation (`nodes.spec_body`, and with a signing key `nodes.spec_signature`, migration `core/0021_signed_specs`), the gateway relays those bytes verbatim and never re-encodes them; otherwise it renders `nodes.spec` unsigned with pki's `desiredStateBodyFromJson`, the writer's own encoder, byte for byte. The daemon stores the last full spec, its hash and its signature in the journal, and enforces it while offline. ### Convergence and status [#convergence-and-status] `observed_generation` advances to a spec's generation only when every section has converged or failed terminally; a retryable failure keeps the section `PROGRESSING` and it is retried on the next pass. A section the daemon has no converger for converges trivially when empty and is `UNSUPPORTED` otherwise, which counts as terminal, with condition `SectionUnsupported` (reason: the field number) so the rest of the spec still converges. bcd applies the rule with `converge.Run`. It has convergers for `host` (the config units, 6c2343f; see [host configuration](/docs/daemon/host-config)), `devices`, `access`, `instances` (below), `telemetry` (since 27c7ee2 bcd's OTLP export of its own metrics; see [node metrics](/docs/operations/observability#node-metrics-from-bcd)), `site` (the boot lease the boot server obeys, and the `cache_images` the site agent pulls into its image cache: the section stays `PROGRESSING` while images warm and `FAILED` for an image whose bytes do not match its digest) and, on `bcd node` and `bcd site`, `daemon` ([self-update](/docs/daemon/install#self-update)). A section number bcd does not know is `UNSUPPORTED`. With `[host] units = false` bcd has no `host` converger: a `host` section with units is `UNSUPPORTED` with reason `HostManagedExternally`, which raises no condition and does not hold back `observed_generation` (a193249). A non-empty `daemon` section is `UNSUPPORTED` in `bcd sim` and `bcd ramdisk`. A `bcd node` under [`bcd helper`](/docs/daemon#privilege-separation) converges it too: it downloads the release, and the helper verifies and swaps it (2d67b25). The FakeDaemon, in turn, converges `instances` and `daemon` when its options enable them but has no `access` converger. The daemon reports progress in `StatusReport`: observed and applied generation, the spec hash, one `SectionStatus` per section (`CONVERGED`, `PROGRESSING`, `FAILED`, `UNSUPPORTED`, with the generation, a reason and a message), and its full condition set. Conditions carry `age_ms`, an age on the daemon's monotonic clock, and the control plane derives `since` from `compute_now()`: agents report ages, not timestamps. From v0.3 the report also lists every VM or container the host runtimes know. Status is coalesced latest-wins: the gateway keeps the last report per session and writes it about once a second, running the host's status handlers (XID health, VPS instance status), each under its own savepoint. Every report runs the handlers, but since 318b396 the `node.status` event is published only when the report says something new: `statusKey` hashes it without its gauges (host pressure and volume storage), and a session skips a report whose key it already published (a new session publishes its first report again). See [scaling](/docs/operations/scaling#gateway-sessions). ### Device assignments [#device-assignments] The `devices` section is how a `hold` [reservation](/docs/guides/capacity/reservations) reaches a machine. For each assignment the daemon writes a CDI spec `/etc/cdi/bcd-.json`, an env file under `/run/bcd/allocations/` and device ownership, then journals an attested `AllocationBound` event with the CDI device names (`bcd.dev/gpu=`). Removing an assignment releases it (`AllocationReleased`). An assignment the daemon cannot honor (a device missing, a device already held by another allocation, an unsupported binding) produces an `AllocationFailed` event with a reason such as `overlap` or `device-missing`, and the section fails terminally at that generation. The daemon never drops an assignment on its own, not even while disconnected. bcd and the FakeDaemon both implement this converger: bcd writes the CDI spec and env file in the FakeDaemon's shape, reads its holds back from `/etc/cdi` on every pass, and signs `AllocationBound` and `AllocationReleased` with the node key. `tests/e2e/bcd.test.ts` binds and releases a 2-GPU reservation through it. What bcd holds (46a61db, d02ba4f): | Device class | Atom id | Resolves to | Held when | | ------------ | ----------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `gpu` | the GPU's UUID | itself | the inventory has it | | `gpu.mig` | `/gi/ci` | the MIG instance's `MIG-…` UUID from the inventory | the instance exists; until the [`mig-layout` unit](/docs/daemon/host-config#mig-layout) has made it, the hold fails with `device-missing` | | `gpu.slot` | `/s` | its GPU | the GPU exists | * **Conflicts** are parent-aware: a whole GPU is refused (`overlap`) while part of it is held, a MIG instance or slot while its GPU is held whole, and two holds never share an atom. No hold is bound on a GPU the `instances` converger passes through to a VM, counting the domains still running after the last pass, and since 6b1480b a VM start is refused a GPU that a live hold has, whole, as a MIG instance or as a slot, since passthrough would take it off its host driver. * **CDI names** cannot contain `/`, so each device's CDI name is its atom id with `/` as `_`, and the real id rides in the device annotation `bcd.dev/device-id` (CDI 0.6), which is how bcd reads its holds back. `AllocationBound` reports the names as `bcd.dev/gpu=`. * **Env.** Devices carry no env of their own. A hold's spec has one spec-level `NVIDIA_VISIBLE_DEVICES` listing what the driver knows (a MIG instance's UUID; a GPU once, however many of its slots are held), and so does the env file. Every workload also gets `BCD_ALLOCATION_ID`, and a slot hold gets `CUDA_MPS_PIPE_DIRECTORY=/run/nvidia-mps` when that directory exists (the [`gpu-sharing` unit](/docs/daemon/host-config#gpu-sharing) runs MPS). * **Bindings.** `BINDING_HOLD` and `BINDING_KUBERNETES` bind the same way; a Kubernetes hold's spec also carries the annotation `bcd.dev/binding: kubernetes` and no `NVIDIA_VISIBLE_DEVICES`, since the kubelet device plugin hands a pod only part of it and sets its own (see [Kubernetes workers](/docs/guides/capacity/kubernetes)). Any other binding fails with `unsupported-binding`, and any other device class with `unsupported-device`. **Host resources.** A reservation that booked host CPU, memory or 1 GiB pages (see [host resources](/docs/guides/capacity/reservations#host-resources)) reaches the node as `DeviceAssignment.limits` (c5a419c): `cpu_millis` from `vcpu` (1000 per CPU), `memory_bytes` from `memoryGiB`, `hugepages_1gi`, and a cpuset on the NUMA nodes of the hold's devices, `cpus` only when those nodes have at least `vcpu` CPUs and `mems` only when they hold the booked memory. bcd then creates the hold's cgroup-v2 slice, `bcd.slice/bcd-alloc.slice/bcd-alloc-.slice`, before the CDI spec names it (260e33c): | Slice `bcd-alloc-.slice` | From | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | `AllowedCPUs`, `AllowedMemoryNodes` | `cpus`, `mems` | | `CPUQuota` | `cpu_millis`, as whole percents of one CPU | | `MemoryMax`, and `MemoryHigh` at 90% of it | `memory_bytes` | | `hugetlb.2MB.max`, `hugetlb.1GB.max` in cgroupfs | `hugepages_2mi`, `hugepages_1gi` (systemd has no property; bcd enables the `hugetlb` controller on the slice's ancestors) | The limits are runtime properties (`systemctl set-property --runtime`), which a reboot drops while the hold stays, so a pass that finds a held hold's slice missing creates it again. The CDI spec names the slice in the annotation `bcd.dev/cgroup-parent`, and every workload gets `BCD_CGROUP_PARENT=bcd-alloc-.slice`: containerd, Slurm or a wrapper places the workload there; bcd does not move processes. On release, bcd stops the slice only once `cgroup.events` says it is empty, so a release never kills a workload. A slice still in use is kept, and since e01d7ff every pass, the 10-minute drift pass included, lists the hold slices on the host and tries again to stop any whose hold is neither wanted nor released in that pass, so it goes once its workload exits. A hold without limits gets no slice, and nothing changes for it. Neither does a `BINDING_KUBERNETES` hold: its pods live in the kubelet's own cgroups (`kubepods.slice`), where the kubelet enforces their requests and limits, so bcd creates no slice for it and names no cgroup parent in its CDI spec or env (e01d7ff). **Host admission of holds** (31555bf). A new hold whose limits carry a memory limit or huge pages is admitted against the host the way an instance start is (see [host-side admission](#instances)): from `MemTotal` less the huge page pool and 1 GiB of headroom come the running and paused domains' commitments and those of the holds already bound (their `MemoryMax`, and the huge pages of their limits not yet faulted in, per NUMA node). A hold that does not fit fails with `AllocationFailed{reason: resources-exhausted}`, and the control plane re-places it. VM starts count the bound holds in turn, so VMs and holds never take what the other was promised. A Kubernetes hold's booked limits count here too, although it has no slice. A host whose memory cannot be read binds without the check, with a warning. **Capabilities** (409f783). An older bcd drops `DeviceAssignment.limits` and `BINDING_KUBERNETES` as unknown fields, so a hold would run without its limits, or a Kubernetes reservation's GPUs would fail as `unsupported-binding` only after placement. On a GPU host bcd's `Hello` therefore advertises `devices.limits` and `binding.kubernetes`, which the gateway writes as `bc.dev/cap.` system labels, and capacity places a reservation that books host resources only on `devices.limits` nodes and a `binding: 'kubernetes'` one only on `binding.kubernetes` nodes (see [capabilities](/docs/daemon/sessions#hello)). A plain GPU hold without limits still goes to older daemons. Under [privilege separation](/docs/daemon#privilege-separation) (`[helper] socket` set, 28dc2ae) the converger does the same work through `bcd helper`: the node asks for a `file.write` or `file.remove` of kind `cdi` or `allocation-env` with the allocation id, and the root helper builds the path itself; a hold's slice is `slice.apply` or `slice.remove`, which accept only a `bcd-alloc-.slice` name and cpulists. A refused or failed operation is an error of the action that asked for it (`helper: …`), exactly as a failed direct write would be: retryable, so the section stays `PROGRESSING` until a later pass succeeds. The `access` converger's sshd files and reload, the instances' nftables table and its 5-second check, `gpu.reset`, `node.reboot` and the privileged half of self-update (the `daemon` section and `daemon.upgrade`: verify, swap, confirm, roll back) go through the helper the same way. `tests/e2e/bcd.test.ts` binds and releases a GPU hold, and delivers a signed update, with an unprivileged node and a real helper. ### Break-glass SSH and tunnel keys [#break-glass-ssh-and-tunnel-keys] The `access` section carries what sshd and the tunnel code on the node trust. bcd's converger writes the CA keys, this node's own principals (`@` only) and a KRL of the revoked serials into `/etc/ssh`, plus a drop-in that points sshd at them, and reloads sshd only when that drop-in changes; an empty section removes all of it. It also keeps the exec-ticket JWKS, which every `OpenTunnel` is verified against, and takes a new one on every pass, even when no file changes. Because the CA keys and principals become lines of sshd's files, bcd refuses rather than strips anything that could add a line: a CA key or principal with a control character, a CA key that is not exactly one OpenSSH public key, or a principal with whitespace makes the whole spec invalid, signed or not, and it is not applied (the converger checks again and fails terminally). See [break-glass SSH](/docs/guides/access/ssh#what-the-node-needs) and [tunnels](/docs/guides/access/tunnels#the-node-half). ### Instances [#instances] The `instances` section is the whole set of VMs and containers a VPS host should run. bcd's converger (on node agents, over the runtimes the build provides) follows the FakeDaemon's `FakeHypervisor` rules: * A domain bcd created that no spec names is removed, and so is one whose desired state is `DELETED`. A domain bcd did not create is never touched, only reported, so the control plane can flag it as a leak. * User data is never in the spec: bcd fetches it with a `SecretRequest` bound to the instance's allocation before it defines the domain. A refusal fails that instance; an unavailable broker keeps the section `PROGRESSING`. * Desired states are `RUNNING`, `PAUSED` (a resume is not a boot) and `STOPPED`. After a host reboot the runtime's domains are stopped and the next pass starts the ones that should run. * An unknown runtime, a bad image, refused user data or a start the runtime refuses for lack of capacity fails only that instance (`ERROR` with its message); the section is then `FAILED` with reason `InstanceFailed`. A start that bcd's own host admission refuses is not a failure (below); the `FakeHypervisor`, whose only capacity check is its vCPU and memory totals, reports a start beyond them that way since 6922553. * **Placement** (a184ec2). `InstanceSpec.placement` (from a flavor's `placement`, chosen by vps; see [flavors](/docs/guides/vps/flavors#placement)) names host CPUs (`cpus`, a cpulist of at least as many CPUs as vCPUs), NUMA nodes for the memory (`mems`), a huge page size (2 MiB or 1 GiB, with the memory in whole pages) and a balloon (never with huge pages). bcd checks it before anything is created, and a placement the host cannot apply fails the instance. `libvirt-kvm` maps it to a vCPU `cpuset` (floating) or `vcpupin` (exactly one CPU per vCPU), `numatune` in strict mode, locked `memoryBacking` huge pages and an explicit `memballoon` (none unless asked); `cloud-hypervisor` to `cpus.affinity`, `memory.hugepages` and a balloon with free page reporting (see [memory ballooning](#memory-ballooning)), with `AllowedCPUs` and `AllowedMemoryNodes` on the VMM's unit, whose `MemoryMax` drops to the VMM's overhead when the guest's memory is in huge pages. A restore keeps bcd's pinning, never the snapshot's. * **Host-side admission** (0dd122a). Before a runtime starts a domain that is not running (a new one, a stopped one; a resume from `PAUSED` is not a start), bcd checks that the host can give it its memory, whatever the control plane decided, so a drifted control plane never overcommits a node. It counts what is committed, not what is used, since a VM that has not touched its memory yet will: * normal memory: `MemTotal` less the huge page pool (`Hugetlb`) and 1 GiB of headroom for the host, against the memory plus VMM overhead (1/32 of the memory plus 256 MiB) of every running or paused domain bcd manages on normal pages, of the bound holds' memory limits (since 31555bf), and of the starts admitted earlier in the same pass; * huge pages, for an instance whose `placement` asks for them: free pages of that size on the placement's NUMA nodes, and since aeaa982 also no more than the host-wide count (free less reserved, less what takers without a NUMA node need), or free less reserved pages host-wide without one; its VMM overhead still counts against normal memory. A start that does not fit stays stopped, and the instance's status message reads `ResourcesExhausted: … MiB of memory needed, … MiB left on this host` (or the huge pages needed and free). It is not `ERROR`, so vps does not fail the instance or its sandbox on the spot; the section stays `PROGRESSING` and the start is tried again on the next pass. vps records the refusal as the instance condition `HostResourcesExhausted` and, since 564f543, once it has stood a minute, places an instance with nothing of its tenant's on the host yet (a new one, or a warm one no sandbox has seen run) on another host, dropping it from this host's spec; one that already ran here, or whose volumes live here, keeps waiting for this host (see [hosts that cannot start an instance](/docs/guides/vps/instances#hosts-that-cannot-start-an-instance)). A runtime that checks capacity itself (the fakehw runtime) is left alone, and a host whose `/proc/meminfo` cannot be read starts without the check, with a warning. * On a host with a runtime, every status report lists everything the runtimes hold in `StatusReport.instances`, with `instances_complete`, and since b8e7271 the volume store's space in `StatusReport.volume_storage` (see [persistent volumes](#persistent-volumes)). * `restore_from` boots a new instance from a snapshot instead of an image. bcd downloads it, over https only since a snapshot may hold a tenant's memory, to `data_dir/snapshots/` and verifies it first; an `http://` URL, a missing digest, a mismatch, another runtime, or a memory snapshot from another CPU model fails that instance. The download runs in the background: the converger waits on it as on user data (the instance is reported waiting "for the user data or snapshot"), goes on deleting domains and loading firewall rules meanwhile, and is triggered again when it ends; a failed download is reported for a minute before it is tried again (6827df8). A downloaded snapshot is deleted once no spec restores from it (it goes with its instance), on every pass of the converger. The `instance.snapshot` command writes one: the runtime captures the disk (and with `include_memory` the memory state) into a staging file under `data_dir/snapshots` (the data disk, never a tmpfs), and bcd hashes it and uploads it to a presigned https URL. Uploads a crash left behind are deleted too. Both transfers follow redirects only to https, and end when a 2-minute window moves less than 1 MiB, or after 6 hours in all, so a stalled or trickling peer cannot pin a bulk worker or keep a restore alive. * Before any domain starts, the instances' `network_policy` becomes host nftables rules on their taps (one `bcd_instances` table, replaced atomically): `PRIVATE` keeps a NIC to its IPAM subnets and the host, `NONE` to the host only, `EGRESS` adds nothing. A policy that cannot be expressed (a NIC without IPAM addresses or a valid MAC), or rules that fail to load, fail the instance and stop it if it runs. Taps are matched by the name every runtime gives them (`runtime.TapName`), so the rules exist before the tap does. Every instance tap (`bc*`), guarded or not, drops DHCP server replies (UDP source port 67 or 547), router advertisements and redirects, and may never send from or claim (ARP, neighbour adverts) the host's own addresses on the bridges its instances attach to. Whatever the policy, every tap whose NIC has a MAC and IPAM addresses is also anti-spoofed in the same table: frames must carry the NIC's MAC, IP packets its own addresses (plus `0.0.0.0` and `::` for DHCP, and its own EUI-64 link-local address, derived from the MAC, for neighbour discovery; guests that use stable-privacy link-local addresses lose IPv6 neighbour discovery), ARP its own IPv4 addresses; each guarded address, the link-local one included, is bound to its tap, so no other port may send from it or claim it (ARP, neighbour adverts), and a deleted instance keeps its guard until its domain is gone. The metadata service and instance tunnels, which name an instance by its address, trust only addresses the loaded rules bind, never one two instances claim. The same transaction loads backstop tables (933c51b, completed in b12c217): `inet bcd_taps` with `iifname "bc*" drop` on the input and forward hooks and `oifname "bc*" drop` on forward and output, and `arp bcd_taps` with the same drops on ARP input and output, which the inet hooks never see. A tap on its bridge reaches those hooks as the bridge, so only a tap off its bridge, which the bridge table cannot see, is dropped there. A NIC on a bridge whose name starts with `bc` (it would match every tap rule) fails its instance. Every 5 seconds bcd lists the tables (`nft -j list table bridge bcd_instances`, then `inet` and `arp bcd_taps`) and counts their rules against the ruleset it loaded, so a reload, `flush ruleset`, or a flushed table or chain ends that trust at once; bcd then loads the last ruleset again itself, outside any converge pass, and if that fails the next pass retries (6827df8). A host whose nft cannot load the table keeps egress instances running, logs a warning and retries on the next pass. bcd advertises the `Hello` capability `instance.network-policy` on hosts with a runtime, which the gateway writes as the node label `bc.dev/cap.instance.network-policy=1`; vps places `private` and `none` instances only there. * An instance with `ssh_from_host_only` (vps sets it on every sandbox instance) takes tcp/22 only from the host (474d51f, security review 13 L5), whatever its policy. For each of its taps the same table drops tcp/22 forwarded from any other bridge port (`oifname tcp dport 22 drop`) and tcp/22 the host sends into the tap unless its source is one of the host's own addresses on the instances' bridges, where the tunnel dial comes from; a family with no host address has its tcp/22 dropped outright. Both rules sit ahead of every accept, so an exec certificate is usable only through a tunnel, which the control plane closes on revocation, and no guest-side KRL is needed. * **Capabilities.** On hosts with a runtime, bcd's `Hello` advertises what it enforces, and the gateway writes each as a `bc.dev/cap.=1` label that vps places on (see [capabilities](/docs/daemon/sessions#hello)): `instance.network-policy` (`private` and `none` instances), `instance.ssh-host-only` (sandbox and warm instances), `instance.placement` (flavors with a `placement`) and `instance.volumes` (instances that attach volumes), the last three since 82f4f51. An older bcd would drop the field and run the guest without it. * The rules match taps by name, so a bcx runtime plugin is told each NIC's tap name (`runtime.TapName`) in its spec (b8a1801). When the rules guard an instance (a `PRIVATE` or `NONE` policy, or SSH from the host only) and a running instance of a plugin runtime has no interface under one of those names, bcd stops it and fails it with `runtime did not create taps …: its network rules cannot apply`: it fails closed rather than run a guest outside the rules. #### The runtimes [#the-runtimes] A host build has two built-in runtimes, and runs each one whose tools are on its `PATH`. There is no configuration key: `virsh` turns on `libvirt-kvm` (`runtime.Libvirt`, 362317c; files under `data_dir/libvirt`), and the `cloud-hypervisor` binary turns on `cloud-hypervisor` (`runtime.CloudHypervisor`, 2598f0a; files under `data_dir/cloud-hypervisor`). A host with both runs both. Each instance names its runtime in its spec (`InstanceSpec.runtime`, which the control plane takes from the pool), and an instance whose runtime this host lacks fails with `runtime is not installed on this host`. Since 0fca413 the control plane does not send one there: bcd advertises each runtime it runs in `Hello` (the node label `bc.dev/driver.`), and vps places a hosted instance only on hosts carrying its runtime's label. | Concern | `libvirt-kvm` | `cloud-hypervisor` | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Driven through | `virsh -c qemu:///system` and `qemu-img` | One cloud-hypervisor VMM per instance, started as a confined systemd transient unit (`systemd-run --unit=bcd-ch-.service --collect --slice=bcd-instances.slice`, so the VM outlives bcd; see [confinement](#the-cloud-hypervisor-unit)) and driven over its REST API on `instances//api.sock`; `qemu-img` for disks. It boots the firmware at `/usr/share/cloud-hypervisor/CLOUDHV.fd` | | Disk | A qcow2 overlay (`instances//disk.qcow2`) on a base image downloaded once over https only, verified against the image's SHA-256 and [checked](#tenant-images) before it becomes visible, and kept as `images/.` (`qcow2`, `raw` or `iso`). An `iso` image boots from a cdrom onto a blank qcow2 disk | The same overlay on the same verified and checked base images (it reuses `libvirt-kvm`'s image code over its own directory). Disks carry an explicit `image_type` (`Qcow2` for the overlay, `Raw` for the seed), and `backing_files` is on only for bcd's own overlay, never for a restored disk. An `iso` image is refused ("cloud-hypervisor boots disk images, not an iso") | | Growing a disk | `qemu-img resize -f qcow2` while stopped, `virsh blockresize` while running; a smaller size fails the instance | `qemu-img resize -f qcow2` while stopped, `vm.resize-disk` while running; a smaller size fails the instance | | User data | A NoCloud seed ISO (volume `cidata`, `user-data` and `meta-data` with the instance id as `instance-id` and `local-hostname`), built with `genisoimage`, else `xorriso`, attached as a read-only cdrom | The same seed ISO, attached as a read-only disk | | NICs | virtio NICs on the named bridges, on taps named `runtime.TapName`, so bcd's nftables rules exist before the tap does | Persistent taps named `runtime.TapName` (`ip tuntap add`), put on their bridge and brought up by bcd before the VMM starts (933c51b); since b12c217 bcd opens each tap and hands its fd to the VMM (`vm.add-net`, or `net_fds` on a restore), which never names a tap. A guest reboot reopens the same tap; a failed attach means no VM; a NIC without a network is refused; the taps are deleted when the VM stops or is destroyed | | GPUs | VFIO passthrough (`` PCI, managed: libvirt rebinds the device itself), the PCI address taken from the inventory's GPU bus id; a GPU without one fails the instance | VFIO passthrough of `/sys/bus/pci/devices/
/`, the address found the same way; a GPU without one fails the instance. Since be4de2a bcd binds every endpoint of the GPU's IOMMU group to `vfio-pci` itself before the start (`driver_override`, unbind, `drivers_probe`; the GPU's audio function too, since VFIO refuses a group with a device on a host driver; bridges stay on `pcieport`) and checks that they moved, so a host without `vfio-pci` loaded fails the start. It records the groups and addresses it handed to the VMM: a stop gives the groups back to root, and a destroy hands the devices back to their host driver, even when the spec dropped a GPU meanwhile | | CPU | `host-passthrough`, so a memory snapshot restores only on the same CPU model | `boot_vcpus` and `max_vcpus` both the instance's vCPUs; a memory snapshot restores only on the same CPU model | | Ownership | bcd marks each domain it defines in its metadata (namespace `https://better-compute.dev/libvirt/1`: allocation, image digest, addresses, the snapshot it was restored from). Any other domain of the connection is listed as unmanaged, the leak the control plane reports, and is never defined over, stopped or destroyed | cloud-hypervisor keeps no registry, so bcd's record is `instances//vm.json` (allocation, image digest, addresses, MACs, bridges, GPUs and the VM configuration). `List` reports only directories with that record, with the state from `vm.info`; a directory without one is not bcd's and is never defined over or removed | | States | `start` (or `start --paused`), `suspend` and `resume`; a stop is `virsh shutdown`, then `virsh destroy` after 60 seconds; a start that fails with "Cannot allocate memory" is a capacity failure. Delete destroys and undefines the domain and removes its directory | `vm.create` and `vm.boot` (then `vm.pause` for a paused start), `vm.pause` and `vm.resume`; a stop is the ACPI power button for up to 60 seconds, then `vm.shutdown`, `vmm.shutdown` and `systemctl stop`, and it fails unless `systemctl show` then reports the unit `inactive` or `failed`, so a VMM that survived is never forgotten; a boot that fails for lack of memory is a capacity failure. Delete stops the VM (and so never removes the directory of a VMM still running) and removes its taps and directory | | Snapshots | A tar of `meta.json` (runtime, CPU model, `includeMemory`), the overlay flattened onto its base (`qemu-img convert`), the seed and, with memory, `virsh save`'s state, after which the domain carries on from that state with `virsh restore --xml` of bcd's own `domain.xml`, never the definition embedded in the state file, which the helper cannot validate (81f9c21); a disk-only snapshot of a running domain pauses it for the copy | A tar of `meta.json`, the flattened overlay, the seed and, with memory, cloud-hypervisor's own snapshot of the paused VM under `ch/` (`config.json`, `state.json`, `memory-ranges`); a running VM is paused for it and resumed after | | Restores | Unpacks the tar into the new instance's directory, refusing one taken by another runtime or, with memory, on another CPU model, then [checks](#tenant-images) its disk and seed; a memory restore's first start is `virsh restore` with the new domain's XML | The same refusals and checks. A memory snapshot's `config.json` must have exactly the instance's vCPUs (`boot_vcpus` and `max_vcpus`) and memory size, and is rebuilt before its first start, which is `vm.restore` then `vm.resume`: from the snapshot it keeps only CPU, memory and platform keys (`cpus`, `memory`, `platform` allowlists), bcd supplies the payload, disks, NICs, serial, console, rng and devices, and anything else (`fs`, `pmem`, `vsock`, memory zones, CPU affinity, a debug console, landlock rules) is dropped | | Console | The pty libvirt gave the domain's serial port, for [instance console tunnels](/docs/guides/access/tunnels#the-node-half) | The serial port's unix socket, `instances//serial.sock` | A snapshot is tied to the runtime that took it: `meta.json` names it, and the other runtime refuses the tar. Both runtimes refuse a memory snapshot of an instance with passthrough GPUs up front, since the GPU's state lives in the device and no VMM saves it (`an instance with passthrough GPUs cannot be snapshotted with its memory; snapshot its disk only`, be4de2a), and of an instance with a read-write volume (`ErrMemorySnapshotWithVolumes`, 550da91); a disk-only snapshot of such an instance still works. vps refuses both before a command is sent (see [snapshots](/docs/guides/vps/sandboxes#take-one)). **QEMU reaches its files** (f7b8805). libvirt runs QEMU as its own user (`libvirt-qemu` or `qemu`), and its dynamic ownership relabels a domain's files but never the directories on their way, while bcd's are 0700 root. Before it defines a domain, bcd therefore adds `o+x` (search, never read) to each directory from `data_dir` down to the disks, seed and volumes the domain names and to the overlay's base image directory. Listing stays closed, and bcd's own files there are 0600 (identity, keys, journal) or relabelled by libvirt. **Under bcd helper** (ed3c7d2, d418820). An unprivileged `bcd node` is not in the root-equivalent `libvirt` group. Its `libvirt-kvm` runtime sends `define`, `start`, `shutdown`, `destroy`, `suspend`, `resume`, `undefine`, `blockresize`, the rebind disk's `attach-disk` and `detach-disk`, `save`, `restore` and the console to the helper's `libvirt.*` operations, where root checks every domain definition default-deny (disks only under bcd's directories and re-checked as root, taps named for the instance on `[helper] instance_bridges`, hostdevs only of the host's GPUs) and hands the console back as a file descriptor; the balloon's `setmem` goes there too. Since b6df6f1 (security review 2) `libvirt.restore` refuses a request without a definition before anything runs, since `virsh restore` alone starts the domain the node-written state file embeds: the definition is checked like a `define` (and must be the requested domain's), staged in root's directory and always passed as `--xml`. Since 3fcd1a1 the helper also checks an overlay's base image as root: the node's own plain image, naming no backing or data file, named by its SHA-256. Reads (`list`, `domstate`, `dumpxml`, `domstats`) use libvirt's read-only socket (`virsh -r`). libvirt runs with `dynamic_ownership = 0`, so instead of the relabelling and `o+x` above, QEMU's user is in the node's group: disks are 0660 (read-only ones 0640) and the directories on the way get group search. See [libvirt through the helper](/docs/daemon#libvirt-through-the-helper). **cloud-hypervisor under bcd helper** (d86fcbe, 7fe74a1, 94843ec). Until 7fe74a1 an unprivileged node dropped `cloud-hypervisor` (its VMMs need root: the `systemd-run` units, the taps, files handed to the VMM's user) and did not advertise it. Now it runs there too, with its root-only steps behind the helper's `ch.*` operations (`runtime.CHPrivileged`): | Step | As root | Under bcd helper | | ------------ | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Files | bcd chowns the instance's files to the VMM's user for the run | The node keeps owning every file and grants the VMM's user POSIX ACL entries on the instance directory and what is in it, inherited by what the VMM creates there; the unit's `UMask` is `0007` instead of `0077` (its group is the VMM's own), so those entries stay effective. Volumes get a read-write or read-only entry for the run, removed at the stop | | Taps | `ip tuntap add` by bcd | `ch.taps`: root creates exactly `runtime.TapName(instance, i)`, owned by the node (which opens it and hands the VMM the fd, as above), on a bridge of `[helper] instance_bridges`, and refuses an existing interface of that name that is not a tap. `ch.untaps` deletes only those names | | GPUs | bcd binds each IOMMU group to `vfio-pci` | `ch.vfio`: root takes the GPUs from the instance's record, only GPUs of this host that root does not see holding work and whose `/dev/vfio` group no other running VMM unit's user owns (94843ec), and only when every device of the group is one of this host's GPUs, a function of one, or a PCI bridge; it binds them to `vfio-pci` and gives `/dev/vfio/` (0600) to the VMM's user. `ch.unvfio` gives them back to root and unbinds them, never a group a running VMM unit's user owns; `ch.stop` gives back exactly the groups the stopped unit's user owned. Root decides who holds a group from `/dev/vfio` itself, never from a node-written record | | The VMM unit | `systemd-run` with bcd's arguments | `ch.launch`: root reads and checks the record itself (`runtime.CHRecord`: `vm.json` and the disk and seed the node's regular files, no symlink in the instance directory, the base image `images/<64 hex>.` and the overlay re-checked, volumes in the volume store, GPUs of this host, the VMM user in bcd's range and not another running unit's, vCPU and memory bounds), then builds the same confined unit (`runtime.CHLaunch`) with root's own `cloud-hypervisor` binary, under a lock. No command line of the node's reaches root. `ch.stop` stops the unit, answers its state and takes the VFIO devices back | The node still drives each VMM over its API socket; what it asks there (`vm.create`'s disks and firmware, `vm.add-disk`) the VMM reaches only as its own user, in its mount namespace, through its ACL entries. Every `ch.*` call names the instance by the domain-name pattern and is logged with it. `bcd doctor` checks `[helper] instance_bridges`: without it, it warns that the helper refuses every instance tap. Like the rest of `cloud-hypervisor`, this path is tested against fake tools only (the runtime's `TestCloudHypervisorUnderTheHelper` and `TestCHRecordRefusals`, the helper's `ch.*` tests), not a real VMM under the helper. #### Memory ballooning [#memory-ballooning] A guest whose `InstanceSpec.placement` asks for a balloon lends the host the memory it does not use (12e9ced, PLAN.md §3.8). vps asks for one on every guest of a pool that overcommits memory, except huge-page-backed and GPU guests (see [flavors](/docs/guides/vps/flavors#placement)), and bcd never balloons a guest with huge pages. Every 30 seconds (the health interval) bcd reads the host's `MemTotal` and `MemAvailable` and the memory pressure stall (PSI `some avg10`), and resizes the balloons of its running, managed guests that have one: | Host state | What bcd does | | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Less than 10% of memory available, or a memory stall of 10% or more | Inflates balloons to cover the shortfall below 10% (at least 2% of host memory while stalled), taking first from the guests with the most memory above their floor, at most a quarter of a guest's memory per pass | | More than 20% available and no stall | Deflates every inflated balloon a step (a quarter of the guest's memory) at a time, within what is free above 20% | | In between | Leaves the balloons as they are | A guest never goes below its floor: `[instances] balloon_floor_percent` of its memory (10 to 100, default 50), or what it uses plus 10% of its memory when that is higher. Its use comes from the runtime's usage reading (see [node metrics](/docs/operations/observability#node-metrics-from-bcd)); a guest without one counts as using all its memory, so nothing is taken from it. `libvirt-kvm` sets the size with `virsh setmem --live` (through the helper's `libvirt.setmem` under `bcd helper`) and reads it from `domstats --balloon`; `cloud-hypervisor` with `vm.resize`'s `desired_balloon` and `vm.info`'s balloon size. A guest defined without a balloon device is never touched (the runtime answers `ErrNoBalloon`, e48f40e), and a memory-restored `cloud-hypervisor` VM keeps bcd's balloon, never the one saved in the tenant's snapshot (9dddde4). Host admission still charges each VM its full configured memory, so ballooning never lets bcd start more guests than fit. The OTLP export reports what each running ballooned guest lends the host (its memory less what the balloon leaves it) as `bcd.instance.memory.balloon.reclaimed`, only for guests whose spec asks for a balloon (9bdd43a; it was `bcd.instance.memory.balloon`, reported for every libvirt domain). The policy and the controller are tested against fake runtimes (`internal/agent` `balloon_test.go`, `balloon_agent_test.go`), and `tests/e2e/bcd-placement.test.ts` inflates and deflates one through a fakehw bcd under simulated memory pressure (427787a); no balloon has run on a real host. #### Persistent volumes [#persistent-volumes] A host keeps the volumes its `instances` section lists (`InstancesSection.volumes`, 014bf0d; bcd 550da91), the disks behind [sandbox volumes](/docs/guides/vps/sandboxes#volumes), across the instances that attach them: * **The store.** Each volume is a sparse raw file, `/volumes/.raw` (0600), with its serial beside it in `.json`. Raw rather than qcow2: its format is fixed and never probed, it has no header a VMM could rewrite, and it grows by truncation. An id is 1 to 80 of `[A-Za-z0-9_-]`, a serial 1 to 20 of `[a-z0-9-]`, and a size a positive multiple of 512 bytes. * **Created and grown, never shrunk.** bcd creates missing volumes and grows those whose size grew before any instance is defined; a smaller size fails. A volume that grew is resized live in every running or paused guest that attaches it (`virsh blockresize` on `libvirt-kvm`, `vm.resize-disk` on `cloud-hypervisor`; b5415a2). * **Attached by serial.** `InstanceSpec.volumes` attaches them in order as `vdb` onward after the root disk: raw virtio disks on `libvirt-kvm` (read-only when asked), `Raw` disks with their serial on `cloud-hypervisor` (bound into the VMM's unit; a read-write one handed to the VMM's user while it runs), and path and serial for a bcx runtime. The guest finds each at `/dev/disk/by-id/virtio-`. * **One writer.** An instance that attaches a volume beside a running or paused writer, or as a writer beside readers, is refused; the first in spec order keeps it. Since 448dc5f a pass applies stops and deletes before starts, so a volume handed from one instance to another in the same pass is let go before it is taken. Since 6b1480b bcd tracks which instances have which volumes open, from the running domains and as the pass stops, deletes and starts them, and checks every start (and a bcx runtime's live re-apply) against that, so a stop that failed never lets a second writer in. Spec-level refusals (a shrink, a volume the section does not list, a pending attach) no longer stop a running VM; only a second holder of an open volume makes one let go. * **Destroyed when unlisted.** A volume the section no longer lists and no instance attaches is destroyed after the pass: its blocks are punched out, then the file is unlinked. A volume a running VM still has open is never destroyed (39458f9), and one whose VM is stopped or deleted in the same pass goes in that pass (408e002). * **No memory snapshots beside a writer**, as above. * **Sparse, but never overcommitted** (851016e). A volume's unwritten space is promised, not taken, so a tenant filling its volume could otherwise leave the other volumes, the instances' overlays and bcd's own journal without space. A new volume, or a growth, must fit in the data directory filesystem's available space (`statfs` `Bavail`) less a reserve (5% of the filesystem, at least 2 GiB) less every volume's unwritten space. One that does not is refused, keeps its size, and the instances that attach it report why (`volume : … MiB needed, … MiB can still be promised to volumes on this host (… MiB free, … MiB reserved, … MiB promised to volumes' unwritten space)`). * **Space reported** (37d6d6d, b8e7271). Every status report of a host with a runtime carries `StatusReport.volume_storage`: | Field | Value | | ------------------- | -------------------------------------------------- | | `total_bytes` | The data directory's filesystem | | `available_bytes` | Its `statfs` `Bavail` | | `reserved_bytes` | Kept from volumes: 5% of the total, at least 2 GiB | | `committed_bytes` | The sum of every volume's size | | `outstanding_bytes` | The sum of every volume's unwritten (sparse) space | Since 131f5ee vps keeps each host's last report (`host_volume_storage`) and books a volume, or a growth, only on a host whose store can hold it by the same rule, less what it booked there that bcd has not reported yet; a host that has not reported takes no volume (see [volumes](/docs/guides/vps/sandboxes#volumes)). * **Closed again once detached** (bdeb452). On `cloud-hypervisor` a read-write volume is handed to the running VMM's user (0600) and given back to root when it stops. A read-only volume gets a POSIX ACL entry (`user::r`) for each VMM user that has it open, removed when that VMM stops, so it is readable by exactly those VMMs rather than by a group; the last reader gone, it is root's 0600 without an ACL. On a filesystem without ACLs it falls back to group `kvm` and 0640, reverted once no other instance of the runtime was given it. Under `bcd helper`, a stopped or destroyed `libvirt-kvm` domain's volumes go back to 0600 unless another running domain still has them open, and a `cloud-hypervisor` volume stays the node's: the VMM's user gets an ACL entry, read-write or read-only, for the run, removed at the stop (d86fcbe). A bcd that keeps volumes advertises `instance.volumes`, and only such hosts receive instances with volumes. #### Tenant images [#tenant-images] A tenant's image is untrusted input to a hypervisor. Security review 12 (fixed in 933c51b) found that a qcow2 image could name a backing file, such as the host's disk or another instance's, and the hypervisor would open it. Its verification found that probing the image with `qemu-img info` was itself a risk, since `qemu-img` may follow what an image names, so since b12c217 no tool inspects a tenant image: bcd reads a qcow2 header and its header extensions in Go (`runtime/qcow2.go`), which opens nothing else. A `qcow2` base image is checked when it is downloaded, before it is renamed into `images/`, and again every time an instance uses it, and it is refused, failing the instance, when: * it is not a qcow2 (version 2 or 3) at all, or its header is truncated or malformed; * it names a backing file; * it names an external data file (incompatible feature bit 2, or the data-file header extension), or sets an incompatible feature bit bcd does not know; * it is encrypted. `raw` and `iso` images are never probed: every tool opens them with their format stated, so their content is only data. Base images are mode 0644 in a 0700 `images/` directory, so an unprivileged VMM can read only the one bound into its unit. An instance's own disk is checked too, because a VMM that once ran it can rewrite its header (`checkOverlay`): `disk.qcow2` must name exactly the recorded base image and format as its backing file, or none for a restored snapshot or an iso install. `cloud-hypervisor` checks it before every start, snapshot and resize; `libvirt-kvm` before every start, snapshot and grow, with the base format now recorded in the domain's metadata. A restored snapshot's disk must be such a flat qcow2, and its seed is attached as raw, never probed; a failed check removes the new instance's directory. `qemu-img create -u` builds an overlay without opening the base, and `libvirt-kvm`'s grow reads the virtual size from the header. #### The cloud-hypervisor unit [#the-cloud-hypervisor-unit] Each VMM runs in its own transient unit `bcd-ch-.service` in `bcd-instances.slice`, as its instance's own unprivileged user (b12c217): a uid from `0x7FFD0000` up, kept in `vm.json`, with `User=` and `Group=` set to it. The unit name always ends in `.service`, so an instance id never names another kind of unit. The VMM needs no network privilege, because it never names a tap: bcd creates and bridges the persistent taps, opens each one (`TUNSETIFF`) and hands the file descriptors to the VMM over `SCM_RIGHTS`. `vm.create` carries no NICs, `vm.add-net` adds each NIC with its fd, a restore passes them as `net_fds`, and the saved configuration's NICs carry no tap name. The instance's files and its GPUs' `/dev/vfio` groups are chowned to the VMM's uid at start, and the groups are given back to root at stop; under `bcd helper` the node grants ACL entries instead and root hands over the groups (see [cloud-hypervisor under bcd helper](#the-runtimes)). `systemd-run` gets these properties: | Property | Value | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Identity and privileges | `User=` and `Group=` the instance's uid, `CapabilityBoundingSet=` and `AmbientCapabilities=` empty, `NoNewPrivileges=yes`, `LimitMEMLOCK=infinity`, `UMask=0077` (`0007` under `bcd helper`, so the inherited ACL entries stay effective) | | Devices | `DevicePolicy=closed` with `DeviceAllow=/dev/kvm rw`; with GPUs also `/dev/vfio/vfio rw` and `char-vfio rw` | | Files | `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp=yes`, `ProtectProc=invisible`; read-only `TemporaryFileSystem`s over `/run` and `/var` (hiding systemd's, D-Bus's and every daemon's sockets, bcd's data, libvirt and other instances), and over bcd's `instances/` and `images/` when its directory is outside `/var`; `InaccessiblePaths` for `/etc/bcd`, `/etc/ssh`, `/etc/libvirt`, `/etc/shadow` and `/etc/gshadow`; only the instance's own directory bound read-write (`BindPaths`, `ReadWritePaths`) and its base image read-only (`BindReadOnlyPaths`), so a disk naming another file finds nothing to open | | Kernel and system | `ProtectKernelTunables`, `ProtectKernelModules`, `ProtectKernelLogs`, `ProtectControlGroups`, `ProtectClock`, `ProtectHostname`, `RestrictSUIDSGID`, `RestrictRealtime`, `RestrictNamespaces`, `LockPersonality`, and `SystemCallFilter=~@debug @mount @reboot @swap @module @raw-io @clock @cpu-emulation @obsolete` | | Network | `PrivateNetwork=yes`, `RestrictAddressFamilies=AF_UNIX`, `IPAddressDeny=any` | | Limits from the flavor | `MemoryMax` = memory + memory/32 + 256 MiB, `TasksMax` = 4 per vCPU + 128, `CPUQuota` = (vCPUs + 1) x 100%; a restored memory configuration drops hugepages, which `MemoryMax` would not charge | This needs systemd 247 or later and a cloud-hypervisor whose API takes NIC file descriptors. > **Tested against fakes only.** `libvirt-kvm`'s tests (`daemon/internal/runtime/libvirt_test.go`) run it against a fake `virsh` and `qemu-img`: the domain XML, metadata, states, grow and shrink, snapshot and clone with memory, a CPU mismatch, destroy, a mismatched image digest, and the image checks against real qcow2 headers (the fake `qemu-img` writes them since b12c217): `TestBaseImageChecks` (a backing file, a data file, a `qcow2` that is not one, an image cached before the check) and `TestCheckOverlay` (an overlay naming another backing file or format). `cloud-hypervisor`'s tests (`cloudhypervisor_test.go`) run it against a fake VMM API and fake `systemd-run`, `qemu-img`, `ip` and seed tools: define, the states, a memory snapshot, a restore into a clone with its paths rewritten, a memory snapshot of a larger VM refused, the rebuilt restore configuration (`TestRestoredConfig`), taps bridged before the VMM exists, the unit's uid and confinement properties, NICs added and restored with fds, the chowns and the VFIO group handed back, a rewritten overlay that will not start, a VMM that survives its stop, stop and destroy, a directory that is not bcd's left alone, and the identity rebind. Neither has run on a real KVM host, and no end-to-end test drives either through the control plane. fakehw builds run a file-backed runtime named `libvirt-kvm` instead, whose domains outlive the process and honor vCPU and memory limits; `tests/e2e/bcd-vps.test.ts` drives that one through the control plane. On `bcd node`, a bcx plugin whose driver lists the kind `runtime` is also a runtime (`list`, `apply`, `remove` over `Invoke`; no snapshots). A host with neither tool and no runtime plugin runs no instances: a non-empty section fails with `this host runs no instance runtime`, and `bcd doctor` warns ("neither virsh nor cloud-hypervisor: this host runs no VPS instances"). #### Identity rebind [#identity-rebind] A memory restore resumes its source guest with the source's sshd principals and host key, so an exec certificate for one restore would open the source and its siblings (security review 13 H2). When vps gives a memory restore its own identity, its spec carries `InstanceSpec.identity_rebind_ref`, a sealed bundle `{"v":1,"account","principals","hostKey"?,"hostCertificate"?}`, and bcd installs it into the running guest (`agent/rebind.go`, `runtime/rebind.go`; ca08d5c): 1. Once the instance runs, the agent fetches the bundle with a `SecretRequest` bound to the instance's allocation, as for user data, and checks it (`runtime.ParseRebind`: version 1, the account name, principal lines, and a PEM host key with an OpenSSH certificate line, or neither). It works off the converge pass, since the guest may take a while. 2. The runtime writes a raw 1 MiB disk holding a tar at offset 0 (`nonce`, `account`, `principals` and, with a host key, `host_key` and `host_key-cert.pub`), reserves its last 4 KiB for the guest's answer, and hot-adds it with virtio serial `bc-rebind`: `vm.add-disk` on `cloud-hypervisor` (the file chowned to the VMM's user), `virsh attach-disk … vdz --live` on `libvirt-kvm`. 3. The guest's udev rule starts `bc-rebind@.service`, whose script installs the principals and host key under `/etc/ssh`, reloads sshd and writes `BCREBOUND ` at the start of the last 4 KiB. 4. bcd waits up to 2 minutes for that line, then removes the disk (`vm.remove-device`, `virsh detach-disk vdz --live`) and zeroes and deletes its file either way, so the secret never stays on the host. 5. On success it records the reference in `data_dir/instances-rebound.json` and reports it as `InstanceStatus.identity_rebound_ref`, from which vps re-points exec at the restore alone. Instances no spec names any more are pruned from the file. A guest that does not answer is tried at most 3 times per reference, 5 minutes apart; a bundle the control plane refuses to release, or a malformed one, is not retried. Nothing is reported then, so the control plane keeps refusing exec while the identity is shared: it fails closed. The guest half (the udev rule, the unit and the script) is written in `daemon/README.md`, and vps's sandbox user data installs it verbatim at first boot (3e25c78), so restores of sandboxes booted since then answer. A bcx runtime plugin is not a rebinder: since b8a1801 its spec carries `identityRebindRef`, `sshFromHostOnly`, `placement` and each NIC's tap name, but bcd neither rebinds through it nor reports `identity_rebound_ref` for it (and plugins take no snapshots, so they restore none). The Fake runtime acts as a compliant guest. Tests: `TestParseRebind`, `TestRebindImage`, the `cloud-hypervisor` and `libvirt-kvm` rebinds against an acknowledging and a silent fake guest, and `TestIdentityRebind`, which drives the converger through the secret fetch, the status report and a restart. See [a restore's own SSH identity](/docs/guides/vps/sandboxes#a-restores-own-ssh-identity). ## Commands [#commands] ### The command body [#the-command-body] A `Command` is the serialized `CommandBody`, an optional signature, and the delivery `attempt` (informational, outside the signature). The control plane stores the body and signature bytes when it enqueues the command and the gateway resends exactly those bytes. | Field | Rule | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | `id` | A ULID; the journal's deduplication key | | `node_id` | Must be the session's node, or the command is `UNSUPPORTED` | | `seq` | Per-node, from `nodes.next_cmd_seq` in the enqueue transaction; replay order | | `kind`, `payload` | A registered kind and its request message, packed as `Any` | | `lane` | `control`, `provision` or `bulk` | | `not_after` | Mandatory, in server time: after it the command must not start | | `min_generation` | The spec generation the command presupposes; 0 for none | | `retry_safe` | May run again when a crash interrupted it before it finished | | `idempotency_key` | Unique per node; workflow steps use `wf::` | | `self_terminating` | The side effect ends the daemon process | | `attested` | The result must be node-signed (verified from v0.2) | | `workflow_id`, `step_id` | The workflow step this command executes; secrets bind to it (v0.2) | | `traceparent` | W3C trace context, from the api through the worker and gateway to the daemon and driver | | `batch_id` | The destructive batch the command belongs to (v1.0 quorum), set by `enqueue({ batchId })`; empty for an unbatched command | The gateway also guards the stored body before sending it: a body that does not decode, lacks `not_after`, or names another command or node is cancelled with reason `invalid-body` instead of sent. ### Dispatch [#dispatch] Enqueue inserts the command `pending` in the caller's transaction and, after commit, pokes the node's gateway. Each gateway dispatches on a poke or every second, joining commands to the sessions it holds: ```sql SELECT c.*, s.session_epoch FROM commands c JOIN node_sessions s USING (node_id) WHERE s.gateway_id = $me AND c.state = 'pending' AND c.not_before <= compute_now() AND c.not_after > compute_now() ORDER BY c.node_id, c.seq LIMIT 1000 FOR UPDATE OF c SKIP LOCKED FOR SHARE OF s ``` A row is marked `dispatched` with the session's epoch only when that epoch equals the gateway's in-memory epoch for the node and the lane has credit. Order within a lane is kept: once a lane is out of credit, later commands of that lane wait. Some commands are cancelled at this point instead of sent: a pending command whose cancel was requested, a kind outside the certificate's `allowed_kinds` (`kind-not-allowed`), and a destructive kind for a tenant-controlled node (`tenant-controlled`). ### Acceptance checks [#acceptance-checks] A delivered command first goes through the journal's duplicate rules (below). A new command, or a retry-safe one a crash interrupted, then passes these checks in order: 1. **`UNSUPPORTED`**: the body names another node, or the kind is unknown or not allowed for this agent's role. While the session is upgrade-only, every kind but `daemon.upgrade` is refused this way. 2. **`min_generation`** above the applied generation: the daemon sends `SpecRequest` and defers the command. A newer spec re-runs the checks; still behind at `not_after`, the command is `STALE_GENERATION`. 3. **`EXPIRED`**: past `not_after`, in server time through the `Welcome` offset. A command without `not_after` is `EXPIRED`, so the control plane always sets one. 4. **`BAD_SIGNATURE`**: once the trust bundle holds command-signing roots, a missing, unknown-key or invalid signature on any command, whatever its kind (0d42de8). 5. **`BATCH_UNAPPROVED`**, **`UNBATCHED_LIMIT`** (v1.0): with approver keys in the trust bundle, a destructive command whose batch approval or scope proof does not admit it, or an unbatched one beyond the hourly allowance. See [the quorum](#signed-commands-and-attested-results). 6. **`BUSY`**: no free credit on the lane, or the journal is full. Not terminal: see [BUSY](/docs/daemon/sessions#busy). `UNSUPPORTED`, `EXPIRED`, `STALE_GENERATION`, `BAD_SIGNATURE` and the two quorum reasons are terminal: the command becomes `rejected`, and the rejection is journaled, so a duplicate delivery or a `Hello` replay gets the same answer. `BUSY` is never journaled. bcd (`dispatch.Precheck`, then a lane credit) and the FakeDaemon both follow this order, so a bad signature is refused even when the lane has no credit. ### Lanes, credits and workers [#lanes-credits-and-workers] | Lane | Gateway credits | Daemon workers | For | | ----------- | --------------- | -------------- | --------------------------------------------------------------------------------------------- | | `control` | 8 | 4 | Short, latency-sensitive work: diagnostics, inventory, reboots, resets, BMC operations, kexec | | `provision` | 4 | 2 | Long provisioning work: disk wipes, image writes, DCGM diagnostics (v0.2) | | `bulk` | 2 | 1 | Large transfers and slow jobs: log collection, image pre-warm | Credits bound the commands admitted at once, running or waiting for a worker; workers bound the ones running. Each lane has a fixed worker pool, never a goroutine per command, and a per-resource lock (a GPU UUID, a disk, an allocation) keeps two lanes from acting on the same device at once. A command holds its credit from acceptance until its result is journaled. ### Kinds by lane [#kinds-by-lane] These are the kinds bcd registers (`daemon/internal/agent/kinds.go` and `site.go`), with the lanes, roles and flags of `commandKinds` in `@better-compute/protocol`, `metalCommandKinds` in `@better-compute/metal` and `instanceSnapshotKind` in `@better-compute/vps`, plus `driver.invoke`, which `provisionModule()` registers. A kind outside the agent's role is `UNSUPPORTED`. | Lane | Kind | Roles | Flags | | ----------- | ------------------------------------------ | ------------------- | ---------------------------------------------------------------------------------------------------------- | | `control` | `inventory.refresh` | node, site, ramdisk | retry-safe | | `control` | `diag.nvml` | node, ramdisk | retry-safe, attested | | `control` | `gpu.reset` | node | attested, destructive | | `control` | `node.reboot` | node, site | retry-safe, self-terminating, destructive | | `control` | `bmc.query` | site | retry-safe | | `control` | `bmc.boot` | site | retry-safe, destructive | | `control` | `bmc.power`, `bmc.firmware`, `bmc.storage` | site | destructive | | `control` | `bmc.rotate` | site | attested, destructive | | `control` | `site.httpProxy` | site | destructive (since 0d42de8; never to a BMC) | | `control` | `boot.prepareRamdisk`, `boot.cleanUp` | site | retry-safe | | `control` | `boot.kexec` | ramdisk | self-terminating | | `provision` | `disk.wipe` | ramdisk | attested, destructive | | `provision` | `deploy.image` | ramdisk | destructive | | `provision` | `diag.dcgm` | ramdisk, node | retry-safe, attested | | `provision` | `driver.invoke` | node, site, ramdisk | destructive; runs in a bcx plugin bcd hosts (see [drivers](/docs/extending/drivers#driver-plugins-in-bcd)) | | `bulk` | `logs.collect` | node, site, ramdisk | retry-safe | | `bulk` | `instance.snapshot` | node | none (not retry-safe) | What each metal kind does on the agent is on [BMC drivers](/docs/guides/bare-metal/bmc#redfish-in-bcd-site), [sites](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site) and [deploying](/docs/guides/bare-metal/deploying#what-the-ramdisk-does). ### Results and acknowledgements [#results-and-acknowledgements] The daemon reports `ACKED` once `accepted` is durable, `RUNNING` when the side effect starts, and one terminal outcome: `SUCCEEDED`, `FAILED` (with an error code such as `interrupted`, `invalid-payload`, `internal` or a kind-specific one), `REJECTED` (with a reason) or `CANCELLED`. Each update records the `boot_id` it was produced in. Reports may skip steps; a lost `ACKED` does not stop a result from landing. The gateway applies updates through the command state table, scoped to the reporting node, so a node can never move another node's command, and a stale or repeated update changes nothing. The terminal result and the node's signature are stored exactly as received. Every terminal update is then acknowledged in `Ack.command_ids`, including ones stored earlier or unknown to this node, and only then does the daemon stop replaying it. Settled commands publish `command.settled` and wake the node's controller. ### Timeouts [#timeouts] The CommandSweeper, a singleton, runs every 5 seconds: a `pending` command past `not_after` becomes `expired`, and a `dispatched`, `acked` or `running` one past `not_after` plus its kind's timeout becomes `failed` with reason `timeout`. For self-terminating kinds the timeout covers the rejoin window. The built-in timeouts are 2 minutes for `diag.nvml` and `inventory.refresh`, 5 for `gpu.reset`, and 15 for `logs.collect` and `node.reboot`; an unregistered kind gets 15 minutes. Metal registers its own: 1 minute for `site.httpProxy`, 2 for `bmc.query`, `bmc.boot`, `boot.prepareRamdisk` and `boot.cleanUp`, 5 for `bmc.power` and `bmc.rotate`, 15 for `boot.kexec`, an hour for `bmc.firmware`, `bmc.storage`, `diag.dcgm` and `deploy.image`, and 6 hours for `disk.wipe`. ### Cancel [#cancel] A cancel requested before dispatch wins: the command is cancelled and never sent. For an in-flight command the gateway sends `Cancel{command_id, reason}` once per session, and again after every reconnect, until the daemon settles it. The daemon stops between actions and never hides a completed side effect: a command still queued or deferred is reported `CANCELLED`, while one whose side effect has started reports its real outcome. A real outcome always wins over a late cancel. ### gpu.reset and live holds [#gpureset-and-live-holds] `gpu.reset` (control lane, destructive, attested, not retry-safe, node role only) resets the GPUs listed in `gpu_uuids`, after a release or for remediation such as XID 63. bcd checks the list before touching anything: * an empty list fails `invalid-payload`; * a GPU that a live CDI hold from the `devices` section names, whole or by one of its MIG instances or slots, or that a VM has passed through, fails the whole command `gpu-busy`, naming the hold or the instance (539c4ce), so a reset never pulls a GPU from under a running workload; `diag.dcgm` applies the same check from level 2; * a GPU the inventory does not know comes back `reset: false` with the message `unknown GPU`. Each remaining GPU is reset with `nvidia-smi --gpu-reset -i `; a failure is that GPU's `message`. bcd then collects a fresh inventory and marks each reset GPU `healthy` unless it shows uncorrectable volatile ECC errors or a pending or failed row remap. Since cb987e5 a host build reads ECC counters and row remapping from `nvidia-smi -q -x` when it is installed; without it the inventory carries neither, and every GPU that reset reads `healthy`. A GPU that reset and reads healthy loses its XID conditions at once instead of keeping them for the rest of the boot (662fb55). The fake HGX node of the `fakehw` build carries them too. The FakeDaemon's `fakeGpuReset` (opt in with its `kinds` option) follows the same checks. ### Signed commands and attested results [#signed-commands-and-attested-results] Two signatures keep a compromised gateway, which can delay, drop or read frames, from forging intent or results: * **Signed intent.** `Command.signature` is a worker key's signature under purpose `command`; the key chains to the command-signing roots of the daemon's trust bundle. The server signs every command when `pki.commandSigning` is configured, and sends the matching trust bundle in `JoinResponse` and `Welcome`; without it commands go unsigned, as v0.1 allows. Whether a signature is required never depends on a flag the body could carry. bcd enforces nothing until its trust bundle holds command-signing roots; from then on every command, whatever its kind, must carry a valid signature, or it is `BAD_SIGNATURE` (0d42de8). PLAN.md staged this (destructive kinds from v0.2, all kinds from v0.4), but a kind table that exempts non-destructive kinds lets a compromised gateway inject an unsigned "harmless" kind that still acts: `site.httpProxy` could POST a Redfish reset or `InsertMedia` to any BMC of a site around the quorum. The FakeDaemon verifies every command with `required: true` too (12ccfa1). The destructive flag still matters for the quorum below: `gpu.reset`, `node.reboot`, `bmc.power`, `bmc.firmware`, `bmc.storage`, `disk.wipe`, `deploy.image`, since 3ff8131 `bmc.boot` (it decides what a machine boots) and `bmc.rotate` (it replaces the BMC credentials), and since 0d42de8 `site.httpProxy`. `metalCommandKinds` marks the same metal kinds destructive (9c14146, f66e236). `site.httpProxy` also never reaches a BMC: the site agent remembers every BMC host it drives through a `bmc.*` command (`data_dir/bmc-hosts.json`) and refuses those hosts, by name and by resolved address, any `/redfish` path, and any address in its local `[site] bmc_networks`, with `forbidden-target`; a `site.httpProxy` other than a `GET` needs a batch leaf bound to its payload under the quorum. And the URL's host must belong to the command's `target_node_id` (a2a853d): once the site spec lists proxy targets (`SiteSpec.proxy_targets`, which metal fills with the site's agentless machines and their inspected addresses, never a BMC host; 8391d74), an IP literal must be one of the node's listed addresses and a name must match a listed name or resolve only to listed addresses, else `forbidden-target`. * **Signed desired state.** Specs have the same envelope under purpose `desired-state`. With `pki.commandSigning`, the NodeSpec controller signs each new generation's `DesiredStateBody` with the worker key (`CommandSigner.signDesiredState`) in the transaction that bumps the generation, and every bump rewrites both columns, so a signature never outlives its generation. The daemon checks the signature with `verifyDesiredState`: one is required for a spec that changes the `access` section (v0.2) and, with the v0.4 policy, for every spec; a present one is always checked. A refused spec is not applied. The FakeDaemon implements this (`intent.signedSpecs`, default `'always'`: with command-signing roots every spec must be signed, because an unsigned spec with a high generation would freeze all later ones; `'access'` for a control plane that signs only access changes; refusals in `specRejections`). bcd implements the `'always'` rule: once it holds command-signing roots, a spec without a valid signature is logged and not applied. * **Attested results.** For attested kinds (`diag.nvml`, `diag.dcgm`, `gpu.reset`, `disk.wipe`, and `bmc.rotate` signed by the site agent) the node signs `CommandUpdate.node_sig` over the command id, the state and the SHA-256 of the result, and allocation reports travel as node-signed `AttestedReport`s. Workers verify them against `node_certs` before any state transition; the gateway never judges or strips them. With `attestedBindings` (on by default with `pki.commandSigning`), a reservation goes active only on the node's own signed `AllocationBound`; see [attested bindings](/docs/guides/capacity/reservations#attested-bindings). * **The destructive-batch quorum (v1.0).** Signed intent does not stop a compromised api or worker, which holds a worker key. With `approver_keys` in the trust bundle (`pki.approvers`), a destructive command with a `batch_id` runs only when `Command.batch_approval` carries enough signatures of distinct approver keys over a scope whose `Command.batch_proof` proves the command's `(subject, kind)`; else it is `BATCH_UNAPPROVED`. Destructive commands without a `batch_id` are refused `UNBATCHED_LIMIT` beyond `unbatched_destructive_per_hour` (default 0), counted per rolling hour of the host's own clock (bcd keeps the count in `data_dir`, so a restart does not reset it). Each batch leaf runs at most once. bcd (e7ddb38, 8b0f50c) and the FakeDaemon enforce it after the signature check. On the server, the `quorum` module stores batches and collects approvals, the command queue holds batched commands until their batch is approved, and the gateway sends each one with its batch's `batch_approval` and its own leaf's `batch_proof` (40be696). The built-in workflows create their batches: metal gives each workflow one of its own or runs it under a `metal.bulk` batch (f66e236), and the health reconciler's reboots and GPU resets get a node-scoped one (e594a4f). See [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10). ## Events [#events] Everything the daemon observes on its own (allocation reports, failures) is an event, journaled with a sequence number that increases from 1 per `journal_id`, and sent in `EventBatch`es. The gateway stores the events above the node's cumulative ack for its current journal, publishes each as `node.event` (attested reports decoded), runs the report handlers, advances `node_sessions.acked_event_seq`, and answers `Ack{journal_id, event_seq}`. A batch for another journal id is ignored. A report whose body names another node is published as `security.node-mismatch` and never reaches a handler; an undecodable one is `security.invalid-report`. The allocation handlers also ignore a report about an allocation that is not placed on the reporting node, as they ignore unknown allocations, so one node's session cannot bind, release or fail another node's allocation. Handlers (`allocationBound`, `allocationReleased`, `allocationFailed`) run in the ingest transaction under a savepoint: a failing one rolls back its own writes, is published as `node.report-failed`, and does not stop the stream, since an unacked event would only be replayed into the same failure. ## The journal [#the-journal] The journal is the daemon's durable memory. bcd's store is bbolt (`journal.OpenBolt`, `journal.db` under `data_dir`); the Go package also holds the contract, the command rules and an in-memory store for tests, and the FakeDaemon mirrors them rule for rule. | Part | Holds | | ------------ | --------------------------------------------------------------------------------------------------------------------------- | | `meta` | Node id, `journal_id`, the event sequence, the last epoch, the applied generation, the server time mark | | `commands` | Command id to `accepted`, `running{boot_id}` or `done` with its result; done entries kept 7 days | | `events` | Unacked events | | `spec` | The last full spec, its hash and its signature | | `certs` | The certificate chain and the command-signing trust bundle (bcd keeps these in `identity.json` beside the journal) | | exec tickets | The `jti` of every exec ticket already used, kept until the ticket expires (at most 10,000; a full set refuses new tickets) | The server time mark is the highest server time the daemon has seen (on `Welcome` and every heartbeat). It only moves forward, and a `Welcome` whose `server_time` is more than 5 minutes before it is refused, so a gateway cannot wind the daemon's clock back into an old worker certificate's window and replay expired signed intent. bcd also mirrors the mark into `identity.json`, so a lost journal does not reset it. **Durability.** Every write is fsynced before it returns. `accepted` reaches disk before any side effect, and self-terminating kinds write `running{boot_id}` immediately before theirs, so a restart can tell the reboot they caused from a crash. **Bounds.** 64 MiB and 10,000 unacked events, which `config.toml` may lower but never raise. Events are classed metric, transition or result. When the journal is full, the oldest metric events are dropped; results and transitions never are. If that is not enough, the daemon answers new commands `BUSY` until acks drain the backlog. ### Duplicates and crash recovery [#duplicates-and-crash-recovery] Every daemon process start picks a fresh incarnation id. An entry written by another incarnation belongs to a process that no longer exists, so there is nothing to re-attach to. A delivered command is resolved against the journal first (`journal.Resolve`): | Journal entry | Action | | ------------------------------------------------- | --------------------------------------------------------------------- | | none | Execute: run the acceptance checks, journal `accepted`, run it | | `done` | Resend the stored result | | `accepted` or `running` in this process | Re-attach: the execution in flight reports the result | | `running`, self-terminating, from an earlier boot | Report `SUCCEEDED` with `rebooted`: the reboot it caused completed it | | left by an earlier process, retry-safe | Re-execute, after the acceptance checks run again | | left by an earlier process, not retry-safe | Report `FAILED` with `interrupted`, for the controller to decide | At start, `journal.Recover` settles the entries whose outcome needs no server input (`rebooted` and `interrupted`) so they go out in `Hello.unacked_results`; retry-safe entries wait for the server to redeliver them, so a re-execution passes the `not_after` and `min_generation` checks again. `disk.wipe`, `deploy.image`, `boot.kexec`, `bmc.power`, `bmc.rotate`, `bmc.firmware`, `bmc.storage`, `site.httpProxy`, `instance.snapshot` and `gpu.reset` are not retry-safe. `journal.Sweep` deletes done entries older than 7 days, and accepted or running entries that an earlier process left and nothing redelivered within those 7 days (the server has long expired them); entries of the running process are never swept. The server keeps the matching half at `Hello` (see [resume rules](/docs/daemon/sessions#resume-rules)): a new boot completes self-terminating commands, and a new journal fails non-retry-safe in-flight commands with `journal-lost` instead of replaying them into a journal that cannot deduplicate. bcd and the FakeDaemon agree on these rules, including the `cancelled` journal outcome, the producing `boot_id` on every result, `NodeRebootResult.previous_boot_id` on a `rebooted` result, and journaled terminal rejections. `tests/e2e/bcd.test.ts` checks the reboot path end to end: `node.reboot` completes `SUCCEEDED` from the next boot.