Delivery

The control plane tells a node what to do in two ways, and the difference matters. Desired state is a standing fact: which devices are held for which allocation, which instances should run, which SSH CA to trust. Commands are one-shot actions: reboot, run diagnostics, collect logs. The rule from PLAN.md §7.3:

If losing it leaves the wrong steady state, it is desired state. If running it late could do harm, it is a command with not_after, plus min_generation when it presupposes a spec.

Both travel down the node's session, and both are delivered at least once. The daemon's journal turns at-least-once delivery into at-most-once effect. The concepts are introduced in desired state and commands; this page is the wire-level detail.

Desired state

NodeSpec and its sections

A NodeSpec is split into sections. The daemon converges them in this order, each with an idempotent plan(desired, observed) and apply:

The control plane builds the spec in the NodeSpec reconciler: the base sections plus every module's nodeSpec() contribution (a NodeSpecPart; the site section merges field by field across modules), stored in nodes.spec as protobuf JSON, with cache_images sorted by URL. It raises nodes.generation only when the canonical JSON (sorted keys) differs from the stored spec, so re-running the reconciler never produces a new generation for nothing. Desired state applies once a node is adopted or deployed.

Generations

Three numbers describe where a node stands:

observed_generation is written with WHERE observed_generation < $g, so it never moves backwards, whichever session a late report arrives on. Controllers that wait for the node to act on a spec wait for it.

How a spec reaches the node

• At Hello, when generation is above applied_generation, the gateway sends a full DesiredState before replaying commands.
• On change, the transaction that wrote a new generation pokes the node's gateway, which pushes the spec to that node's session at once.
• As a backstop, every 10 seconds the gateway compares each open session with nodes.generation and sends a full spec to any node that is behind both what it holds and what was already sent on this session.
• On request, the daemon sends SpecRequest{have_generation, reason} and gets the full spec back.

A DesiredStateBody carries the node id, the generation, a spec_hash and either a full NodeSpec or a patch. A patch applies against base_generation: every section it sets replaces that section wholesale (an empty section clears it), unset sections stay. The daemon ignores a body for another node and any generation not above the one it holds. It never recomputes spec_hash; it echoes it back as applied_spec_hash. The FakeDaemon applies both forms. The gateway in this release sends full specs only (base_generation 0). When the NodeSpec controller stored the encoded body of the current generation (nodes.spec_body, and with a signing key nodes.spec_signature, migration core/0021_signed_specs), the gateway relays those bytes verbatim and never re-encodes them; otherwise it renders nodes.spec unsigned with pki's desiredStateBodyFromJson, the writer's own encoder, byte for byte.

The daemon stores the last full spec, its hash and its signature in the journal, and enforces it while offline.

Convergence and status

observed_generation advances to a spec's generation only when every section has converged or failed terminally; a retryable failure keeps the section PROGRESSING and it is retried on the next pass. A section the daemon has no converger for converges trivially when empty and is UNSUPPORTED otherwise, which counts as terminal, with condition SectionUnsupported (reason: the field number) so the rest of the spec still converges. bcd applies the rule with converge.Run. It has convergers for host (the config units, 6c2343f; see host configuration), devices, access, instances (below), telemetry (since 27c7ee2 bcd's OTLP export of its own metrics; see node metrics), site (the boot lease the boot server obeys, and the cache_images the site agent pulls into its image cache: the section stays PROGRESSING while images warm and FAILED for an image whose bytes do not match its digest) and, on bcd node and bcd site, daemon (self-update). A section number bcd does not know is UNSUPPORTED. With [host] units = false bcd has no host converger: a host section with units is UNSUPPORTED with reason HostManagedExternally, which raises no condition and does not hold back observed_generation (a193249). A non-empty daemon section is UNSUPPORTED in bcd sim and bcd ramdisk. A bcd node under bcd helper converges it too: it downloads the release, and the helper verifies and swaps it (2d67b25). The FakeDaemon, in turn, converges instances and daemon when its options enable them but has no access converger.

The daemon reports progress in StatusReport: observed and applied generation, the spec hash, one SectionStatus per section (CONVERGED, PROGRESSING, FAILED, UNSUPPORTED, with the generation, a reason and a message), and its full condition set. Conditions carry age_ms, an age on the daemon's monotonic clock, and the control plane derives since from compute_now(): agents report ages, not timestamps. From v0.3 the report also lists every VM or container the host runtimes know. Status is coalesced latest-wins: the gateway keeps the last report per session and writes it about once a second, running the host's status handlers (XID health, VPS instance status), each under its own savepoint. Every report runs the handlers, but since 318b396 the node.status event is published only when the report says something new: statusKey hashes it without its gauges (host pressure and volume storage), and a session skips a report whose key it already published (a new session publishes its first report again). See scaling.

Device assignments

The devices section is how a hold reservation reaches a machine. For each assignment the daemon writes a CDI spec /etc/cdi/bcd-<allocation_id>.json, an env file under /run/bcd/allocations/ and device ownership, then journals an attested AllocationBound event with the CDI device names (bcd.dev/gpu=<id>). Removing an assignment releases it (AllocationReleased). An assignment the daemon cannot honor (a device missing, a device already held by another allocation, an unsupported binding) produces an AllocationFailed event with a reason such as overlap or device-missing, and the section fails terminally at that generation. The daemon never drops an assignment on its own, not even while disconnected. bcd and the FakeDaemon both implement this converger: bcd writes the CDI spec and env file in the FakeDaemon's shape, reads its holds back from /etc/cdi on every pass, and signs AllocationBound and AllocationReleased with the node key. tests/e2e/bcd.test.ts binds and releases a 2-GPU reservation through it.

What bcd holds (46a61db, d02ba4f):

• Conflicts are parent-aware: a whole GPU is refused (overlap) while part of it is held, a MIG instance or slot while its GPU is held whole, and two holds never share an atom. No hold is bound on a GPU the instances converger passes through to a VM, counting the domains still running after the last pass, and since 6b1480b a VM start is refused a GPU that a live hold has, whole, as a MIG instance or as a slot, since passthrough would take it off its host driver.
• CDI names cannot contain /, so each device's CDI name is its atom id with / as _, and the real id rides in the device annotation bcd.dev/device-id (CDI 0.6), which is how bcd reads its holds back. AllocationBound reports the names as bcd.dev/gpu=<name>.
• Env. Devices carry no env of their own. A hold's spec has one spec-level NVIDIA_VISIBLE_DEVICES listing what the driver knows (a MIG instance's UUID; a GPU once, however many of its slots are held), and so does the env file. Every workload also gets BCD_ALLOCATION_ID, and a slot hold gets CUDA_MPS_PIPE_DIRECTORY=/run/nvidia-mps when that directory exists (the gpu-sharing unit runs MPS).
• Bindings. BINDING_HOLD and BINDING_KUBERNETES bind the same way; a Kubernetes hold's spec also carries the annotation bcd.dev/binding: kubernetes and no NVIDIA_VISIBLE_DEVICES, since the kubelet device plugin hands a pod only part of it and sets its own (see Kubernetes workers). Any other binding fails with unsupported-binding, and any other device class with unsupported-device.

Host resources. A reservation that booked host CPU, memory or 1 GiB pages (see host resources) reaches the node as DeviceAssignment.limits (c5a419c): cpu_millis from vcpu (1000 per CPU), memory_bytes from memoryGiB, hugepages_1gi, and a cpuset on the NUMA nodes of the hold's devices, cpus only when those nodes have at least vcpu CPUs and mems only when they hold the booked memory. bcd then creates the hold's cgroup-v2 slice, bcd.slice/bcd-alloc.slice/bcd-alloc-<allocation>.slice, before the CDI spec names it (260e33c):

The limits are runtime properties (systemctl set-property --runtime), which a reboot drops while the hold stays, so a pass that finds a held hold's slice missing creates it again. The CDI spec names the slice in the annotation bcd.dev/cgroup-parent, and every workload gets BCD_CGROUP_PARENT=bcd-alloc-<allocation>.slice: containerd, Slurm or a wrapper places the workload there; bcd does not move processes. On release, bcd stops the slice only once cgroup.events says it is empty, so a release never kills a workload. A slice still in use is kept, and since e01d7ff every pass, the 10-minute drift pass included, lists the hold slices on the host and tries again to stop any whose hold is neither wanted nor released in that pass, so it goes once its workload exits. A hold without limits gets no slice, and nothing changes for it. Neither does a BINDING_KUBERNETES hold: its pods live in the kubelet's own cgroups (kubepods.slice), where the kubelet enforces their requests and limits, so bcd creates no slice for it and names no cgroup parent in its CDI spec or env (e01d7ff).

Host admission of holds (31555bf). A new hold whose limits carry a memory limit or huge pages is admitted against the host the way an instance start is (see host-side admission): from MemTotal less the huge page pool and 1 GiB of headroom come the running and paused domains' commitments and those of the holds already bound (their MemoryMax, and the huge pages of their limits not yet faulted in, per NUMA node). A hold that does not fit fails with AllocationFailed{reason: resources-exhausted}, and the control plane re-places it. VM starts count the bound holds in turn, so VMs and holds never take what the other was promised. A Kubernetes hold's booked limits count here too, although it has no slice. A host whose memory cannot be read binds without the check, with a warning.

Capabilities (409f783). An older bcd drops DeviceAssignment.limits and BINDING_KUBERNETES as unknown fields, so a hold would run without its limits, or a Kubernetes reservation's GPUs would fail as unsupported-binding only after placement. On a GPU host bcd's Hello therefore advertises devices.limits and binding.kubernetes, which the gateway writes as bc.dev/cap.<name> system labels, and capacity places a reservation that books host resources only on devices.limits nodes and a binding: 'kubernetes' one only on binding.kubernetes nodes (see capabilities). A plain GPU hold without limits still goes to older daemons.

Under privilege separation ([helper] socket set, 28dc2ae) the converger does the same work through bcd helper: the node asks for a file.write or file.remove of kind cdi or allocation-env with the allocation id, and the root helper builds the path itself; a hold's slice is slice.apply or slice.remove, which accept only a bcd-alloc-<id>.slice name and cpulists. A refused or failed operation is an error of the action that asked for it (helper: …), exactly as a failed direct write would be: retryable, so the section stays PROGRESSING until a later pass succeeds. The access converger's sshd files and reload, the instances' nftables table and its 5-second check, gpu.reset, node.reboot and the privileged half of self-update (the daemon section and daemon.upgrade: verify, swap, confirm, roll back) go through the helper the same way. tests/e2e/bcd.test.ts binds and releases a GPU hold, and delivers a signed update, with an unprivileged node and a real helper.

Break-glass SSH and tunnel keys

The access section carries what sshd and the tunnel code on the node trust. bcd's converger writes the CA keys, this node's own principals (<account>@<nodeId> only) and a KRL of the revoked serials into /etc/ssh, plus a drop-in that points sshd at them, and reloads sshd only when that drop-in changes; an empty section removes all of it. It also keeps the exec-ticket JWKS, which every OpenTunnel is verified against, and takes a new one on every pass, even when no file changes. Because the CA keys and principals become lines of sshd's files, bcd refuses rather than strips anything that could add a line: a CA key or principal with a control character, a CA key that is not exactly one OpenSSH public key, or a principal with whitespace makes the whole spec invalid, signed or not, and it is not applied (the converger checks again and fails terminally). See break-glass SSH and tunnels.

Instances

The instances section is the whole set of VMs and containers a VPS host should run. bcd's converger (on node agents, over the runtimes the build provides) follows the FakeDaemon's FakeHypervisor rules:

• A domain bcd created that no spec names is removed, and so is one whose desired state is DELETED. A domain bcd did not create is never touched, only reported, so the control plane can flag it as a leak.

• User data is never in the spec: bcd fetches it with a SecretRequest bound to the instance's allocation before it defines the domain. A refusal fails that instance; an unavailable broker keeps the section PROGRESSING.

• Desired states are RUNNING, PAUSED (a resume is not a boot) and STOPPED. After a host reboot the runtime's domains are stopped and the next pass starts the ones that should run.

• An unknown runtime, a bad image, refused user data or a start the runtime refuses for lack of capacity fails only that instance (ERROR with its message); the section is then FAILED with reason InstanceFailed. A start that bcd's own host admission refuses is not a failure (below); the FakeHypervisor, whose only capacity check is its vCPU and memory totals, reports a start beyond them that way since 6922553.

• Placement (a184ec2). InstanceSpec.placement (from a flavor's placement, chosen by vps; see flavors) names host CPUs (cpus, a cpulist of at least as many CPUs as vCPUs), NUMA nodes for the memory (mems), a huge page size (2 MiB or 1 GiB, with the memory in whole pages) and a balloon (never with huge pages). bcd checks it before anything is created, and a placement the host cannot apply fails the instance. libvirt-kvm maps it to a vCPU cpuset (floating) or vcpupin (exactly one CPU per vCPU), numatune in strict mode, locked memoryBacking huge pages and an explicit memballoon (none unless asked); cloud-hypervisor to cpus.affinity, memory.hugepages and a balloon with free page reporting (see memory ballooning), with AllowedCPUs and AllowedMemoryNodes on the VMM's unit, whose MemoryMax drops to the VMM's overhead when the guest's memory is in huge pages. A restore keeps bcd's pinning, never the snapshot's.

• Host-side admission (0dd122a). Before a runtime starts a domain that is not running (a new one, a stopped one; a resume from PAUSED is not a start), bcd checks that the host can give it its memory, whatever the control plane decided, so a drifted control plane never overcommits a node. It counts what is committed, not what is used, since a VM that has not touched its memory yet will:

  • normal memory: MemTotal less the huge page pool (Hugetlb) and 1 GiB of headroom for the host, against the memory plus VMM overhead (1/32 of the memory plus 256 MiB) of every running or paused domain bcd manages on normal pages, of the bound holds' memory limits (since 31555bf), and of the starts admitted earlier in the same pass;
  • huge pages, for an instance whose placement asks for them: free pages of that size on the placement's NUMA nodes, and since aeaa982 also no more than the host-wide count (free less reserved, less what takers without a NUMA node need), or free less reserved pages host-wide without one; its VMM overhead still counts against normal memory.

  A start that does not fit stays stopped, and the instance's status message reads ResourcesExhausted: … MiB of memory needed, … MiB left on this host (or the huge pages needed and free). It is not ERROR, so vps does not fail the instance or its sandbox on the spot; the section stays PROGRESSING and the start is tried again on the next pass. vps records the refusal as the instance condition HostResourcesExhausted and, since 564f543, once it has stood a minute, places an instance with nothing of its tenant's on the host yet (a new one, or a warm one no sandbox has seen run) on another host, dropping it from this host's spec; one that already ran here, or whose volumes live here, keeps waiting for this host (see hosts that cannot start an instance). A runtime that checks capacity itself (the fakehw runtime) is left alone, and a host whose /proc/meminfo cannot be read starts without the check, with a warning.

• On a host with a runtime, every status report lists everything the runtimes hold in StatusReport.instances, with instances_complete, and since b8e7271 the volume store's space in StatusReport.volume_storage (see persistent volumes).

• restore_from boots a new instance from a snapshot instead of an image. bcd downloads it, over https only since a snapshot may hold a tenant's memory, to data_dir/snapshots/<sha256> and verifies it first; an http:// URL, a missing digest, a mismatch, another runtime, or a memory snapshot from another CPU model fails that instance. The download runs in the background: the converger waits on it as on user data (the instance is reported waiting "for the user data or snapshot"), goes on deleting domains and loading firewall rules meanwhile, and is triggered again when it ends; a failed download is reported for a minute before it is tried again (6827df8). A downloaded snapshot is deleted once no spec restores from it (it goes with its instance), on every pass of the converger. The instance.snapshot command writes one: the runtime captures the disk (and with include_memory the memory state) into a staging file under data_dir/snapshots (the data disk, never a tmpfs), and bcd hashes it and uploads it to a presigned https URL. Uploads a crash left behind are deleted too. Both transfers follow redirects only to https, and end when a 2-minute window moves less than 1 MiB, or after 6 hours in all, so a stalled or trickling peer cannot pin a bulk worker or keep a restore alive.

• Before any domain starts, the instances' network_policy becomes host nftables rules on their taps (one bcd_instances table, replaced atomically): PRIVATE keeps a NIC to its IPAM subnets and the host, NONE to the host only, EGRESS adds nothing. A policy that cannot be expressed (a NIC without IPAM addresses or a valid MAC), or rules that fail to load, fail the instance and stop it if it runs. Taps are matched by the name every runtime gives them (runtime.TapName), so the rules exist before the tap does. Every instance tap (bc*), guarded or not, drops DHCP server replies (UDP source port 67 or 547), router advertisements and redirects, and may never send from or claim (ARP, neighbour adverts) the host's own addresses on the bridges its instances attach to. Whatever the policy, every tap whose NIC has a MAC and IPAM addresses is also anti-spoofed in the same table: frames must carry the NIC's MAC, IP packets its own addresses (plus 0.0.0.0 and :: for DHCP, and its own EUI-64 link-local address, derived from the MAC, for neighbour discovery; guests that use stable-privacy link-local addresses lose IPv6 neighbour discovery), ARP its own IPv4 addresses; each guarded address, the link-local one included, is bound to its tap, so no other port may send from it or claim it (ARP, neighbour adverts), and a deleted instance keeps its guard until its domain is gone. The metadata service and instance tunnels, which name an instance by its address, trust only addresses the loaded rules bind, never one two instances claim. The same transaction loads backstop tables (933c51b, completed in b12c217): inet bcd_taps with iifname "bc*" drop on the input and forward hooks and oifname "bc*" drop on forward and output, and arp bcd_taps with the same drops on ARP input and output, which the inet hooks never see. A tap on its bridge reaches those hooks as the bridge, so only a tap off its bridge, which the bridge table cannot see, is dropped there. A NIC on a bridge whose name starts with bc (it would match every tap rule) fails its instance. Every 5 seconds bcd lists the tables (nft -j list table bridge bcd_instances, then inet and arp bcd_taps) and counts their rules against the ruleset it loaded, so a reload, flush ruleset, or a flushed table or chain ends that trust at once; bcd then loads the last ruleset again itself, outside any converge pass, and if that fails the next pass retries (6827df8). A host whose nft cannot load the table keeps egress instances running, logs a warning and retries on the next pass. bcd advertises the Hello capability instance.network-policy on hosts with a runtime, which the gateway writes as the node label bc.dev/cap.instance.network-policy=1; vps places private and none instances only there.

• An instance with ssh_from_host_only (vps sets it on every sandbox instance) takes tcp/22 only from the host (474d51f, security review 13 L5), whatever its policy. For each of its taps the same table drops tcp/22 forwarded from any other bridge port (oifname <tap> tcp dport 22 drop) and tcp/22 the host sends into the tap unless its source is one of the host's own addresses on the instances' bridges, where the tunnel dial comes from; a family with no host address has its tcp/22 dropped outright. Both rules sit ahead of every accept, so an exec certificate is usable only through a tunnel, which the control plane closes on revocation, and no guest-side KRL is needed.

• Capabilities. On hosts with a runtime, bcd's Hello advertises what it enforces, and the gateway writes each as a bc.dev/cap.<name>=1 label that vps places on (see capabilities): instance.network-policy (private and none instances), instance.ssh-host-only (sandbox and warm instances), instance.placement (flavors with a placement) and instance.volumes (instances that attach volumes), the last three since 82f4f51. An older bcd would drop the field and run the guest without it.

• The rules match taps by name, so a bcx runtime plugin is told each NIC's tap name (runtime.TapName) in its spec (b8a1801). When the rules guard an instance (a PRIVATE or NONE policy, or SSH from the host only) and a running instance of a plugin runtime has no interface under one of those names, bcd stops it and fails it with runtime <id> did not create taps …: its network rules cannot apply: it fails closed rather than run a guest outside the rules.

The runtimes

A host build has two built-in runtimes, and runs each one whose tools are on its PATH. There is no configuration key: virsh turns on libvirt-kvm (runtime.Libvirt, 362317c; files under data_dir/libvirt), and the cloud-hypervisor binary turns on cloud-hypervisor (runtime.CloudHypervisor, 2598f0a; files under data_dir/cloud-hypervisor). A host with both runs both. Each instance names its runtime in its spec (InstanceSpec.runtime, which the control plane takes from the pool), and an instance whose runtime this host lacks fails with runtime <id> is not installed on this host. Since 0fca413 the control plane does not send one there: bcd advertises each runtime it runs in Hello (the node label bc.dev/driver.<runtime>), and vps places a hosted instance only on hosts carrying its runtime's label.

A snapshot is tied to the runtime that took it: meta.json names it, and the other runtime refuses the tar. Both runtimes refuse a memory snapshot of an instance with passthrough GPUs up front, since the GPU's state lives in the device and no VMM saves it (an instance with passthrough GPUs cannot be snapshotted with its memory; snapshot its disk only, be4de2a), and of an instance with a read-write volume (ErrMemorySnapshotWithVolumes, 550da91); a disk-only snapshot of such an instance still works. vps refuses both before a command is sent (see snapshots).

QEMU reaches its files (f7b8805). libvirt runs QEMU as its own user (libvirt-qemu or qemu), and its dynamic ownership relabels a domain's files but never the directories on their way, while bcd's are 0700 root. Before it defines a domain, bcd therefore adds o+x (search, never read) to each directory from data_dir down to the disks, seed and volumes the domain names and to the overlay's base image directory. Listing stays closed, and bcd's own files there are 0600 (identity, keys, journal) or relabelled by libvirt.

Under bcd helper (ed3c7d2, d418820). An unprivileged bcd node is not in the root-equivalent libvirt group. Its libvirt-kvm runtime sends define, start, shutdown, destroy, suspend, resume, undefine, blockresize, the rebind disk's attach-disk and detach-disk, save, restore and the console to the helper's libvirt.* operations, where root checks every domain definition default-deny (disks only under bcd's directories and re-checked as root, taps named for the instance on [helper] instance_bridges, hostdevs only of the host's GPUs) and hands the console back as a file descriptor; the balloon's setmem goes there too. Since b6df6f1 (security review 2) libvirt.restore refuses a request without a definition before anything runs, since virsh restore alone starts the domain the node-written state file embeds: the definition is checked like a define (and must be the requested domain's), staged in root's directory and always passed as --xml. Since 3fcd1a1 the helper also checks an overlay's base image as root: the node's own plain image, naming no backing or data file, named by its SHA-256. Reads (list, domstate, dumpxml, domstats) use libvirt's read-only socket (virsh -r). libvirt runs with dynamic_ownership = 0, so instead of the relabelling and o+x above, QEMU's user is in the node's group: disks are 0660 (read-only ones 0640) and the directories on the way get group search. See libvirt through the helper.

cloud-hypervisor under bcd helper (d86fcbe, 7fe74a1, 94843ec). Until 7fe74a1 an unprivileged node dropped cloud-hypervisor (its VMMs need root: the systemd-run units, the taps, files handed to the VMM's user) and did not advertise it. Now it runs there too, with its root-only steps behind the helper's ch.* operations (runtime.CHPrivileged):

The node still drives each VMM over its API socket; what it asks there (vm.create's disks and firmware, vm.add-disk) the VMM reaches only as its own user, in its mount namespace, through its ACL entries. Every ch.* call names the instance by the domain-name pattern and is logged with it. bcd doctor checks [helper] instance_bridges: without it, it warns that the helper refuses every instance tap. Like the rest of cloud-hypervisor, this path is tested against fake tools only (the runtime's TestCloudHypervisorUnderTheHelper and TestCHRecordRefusals, the helper's ch.* tests), not a real VMM under the helper.

Memory ballooning

A guest whose InstanceSpec.placement asks for a balloon lends the host the memory it does not use (12e9ced, PLAN.md §3.8). vps asks for one on every guest of a pool that overcommits memory, except huge-page-backed and GPU guests (see flavors), and bcd never balloons a guest with huge pages. Every 30 seconds (the health interval) bcd reads the host's MemTotal and MemAvailable and the memory pressure stall (PSI some avg10), and resizes the balloons of its running, managed guests that have one:

A guest never goes below its floor: [instances] balloon_floor_percent of its memory (10 to 100, default 50), or what it uses plus 10% of its memory when that is higher. Its use comes from the runtime's usage reading (see node metrics); a guest without one counts as using all its memory, so nothing is taken from it. libvirt-kvm sets the size with virsh setmem --live (through the helper's libvirt.setmem under bcd helper) and reads it from domstats --balloon; cloud-hypervisor with vm.resize's desired_balloon and vm.info's balloon size. A guest defined without a balloon device is never touched (the runtime answers ErrNoBalloon, e48f40e), and a memory-restored cloud-hypervisor VM keeps bcd's balloon, never the one saved in the tenant's snapshot (9dddde4). Host admission still charges each VM its full configured memory, so ballooning never lets bcd start more guests than fit. The OTLP export reports what each running ballooned guest lends the host (its memory less what the balloon leaves it) as bcd.instance.memory.balloon.reclaimed, only for guests whose spec asks for a balloon (9bdd43a; it was bcd.instance.memory.balloon, reported for every libvirt domain). The policy and the controller are tested against fake runtimes (internal/agent balloon_test.go, balloon_agent_test.go), and tests/e2e/bcd-placement.test.ts inflates and deflates one through a fakehw bcd under simulated memory pressure (427787a); no balloon has run on a real host.

Persistent volumes

A host keeps the volumes its instances section lists (InstancesSection.volumes, 014bf0d; bcd 550da91), the disks behind sandbox volumes, across the instances that attach them:

• The store. Each volume is a sparse raw file, <data_dir>/volumes/<id>.raw (0600), with its serial beside it in <id>.json. Raw rather than qcow2: its format is fixed and never probed, it has no header a VMM could rewrite, and it grows by truncation. An id is 1 to 80 of [A-Za-z0-9_-], a serial 1 to 20 of [a-z0-9-], and a size a positive multiple of 512 bytes.

• Created and grown, never shrunk. bcd creates missing volumes and grows those whose size grew before any instance is defined; a smaller size fails. A volume that grew is resized live in every running or paused guest that attaches it (virsh blockresize on libvirt-kvm, vm.resize-disk on cloud-hypervisor; b5415a2).

• Attached by serial. InstanceSpec.volumes attaches them in order as vdb onward after the root disk: raw virtio disks on libvirt-kvm (read-only when asked), Raw disks with their serial on cloud-hypervisor (bound into the VMM's unit; a read-write one handed to the VMM's user while it runs), and path and serial for a bcx runtime. The guest finds each at /dev/disk/by-id/virtio-<serial>.

• One writer. An instance that attaches a volume beside a running or paused writer, or as a writer beside readers, is refused; the first in spec order keeps it. Since 448dc5f a pass applies stops and deletes before starts, so a volume handed from one instance to another in the same pass is let go before it is taken. Since 6b1480b bcd tracks which instances have which volumes open, from the running domains and as the pass stops, deletes and starts them, and checks every start (and a bcx runtime's live re-apply) against that, so a stop that failed never lets a second writer in. Spec-level refusals (a shrink, a volume the section does not list, a pending attach) no longer stop a running VM; only a second holder of an open volume makes one let go.

• Destroyed when unlisted. A volume the section no longer lists and no instance attaches is destroyed after the pass: its blocks are punched out, then the file is unlinked. A volume a running VM still has open is never destroyed (39458f9), and one whose VM is stopped or deleted in the same pass goes in that pass (408e002).

• No memory snapshots beside a writer, as above.

• Sparse, but never overcommitted (851016e). A volume's unwritten space is promised, not taken, so a tenant filling its volume could otherwise leave the other volumes, the instances' overlays and bcd's own journal without space. A new volume, or a growth, must fit in the data directory filesystem's available space (statfs Bavail) less a reserve (5% of the filesystem, at least 2 GiB) less every volume's unwritten space. One that does not is refused, keeps its size, and the instances that attach it report why (volume <id>: … MiB needed, … MiB can still be promised to volumes on this host (… MiB free, … MiB reserved, … MiB promised to volumes' unwritten space)).

• Space reported (37d6d6d, b8e7271). Every status report of a host with a runtime carries StatusReport.volume_storage:

  Field	Value
  total_bytes	The data directory's filesystem
  available_bytes	Its statfs Bavail
  reserved_bytes	Kept from volumes: 5% of the total, at least 2 GiB
  committed_bytes	The sum of every volume's size
  outstanding_bytes	The sum of every volume's unwritten (sparse) space

  Since 131f5ee vps keeps each host's last report (host_volume_storage) and books a volume, or a growth, only on a host whose store can hold it by the same rule, less what it booked there that bcd has not reported yet; a host that has not reported takes no volume (see volumes).

• Closed again once detached (bdeb452). On cloud-hypervisor a read-write volume is handed to the running VMM's user (0600) and given back to root when it stops. A read-only volume gets a POSIX ACL entry (user:<uid>:r) for each VMM user that has it open, removed when that VMM stops, so it is readable by exactly those VMMs rather than by a group; the last reader gone, it is root's 0600 without an ACL. On a filesystem without ACLs it falls back to group kvm and 0640, reverted once no other instance of the runtime was given it. Under bcd helper, a stopped or destroyed libvirt-kvm domain's volumes go back to 0600 unless another running domain still has them open, and a cloud-hypervisor volume stays the node's: the VMM's user gets an ACL entry, read-write or read-only, for the run, removed at the stop (d86fcbe).

A bcd that keeps volumes advertises instance.volumes, and only such hosts receive instances with volumes.

Tenant images

A tenant's image is untrusted input to a hypervisor. Security review 12 (fixed in 933c51b) found that a qcow2 image could name a backing file, such as the host's disk or another instance's, and the hypervisor would open it. Its verification found that probing the image with qemu-img info was itself a risk, since qemu-img may follow what an image names, so since b12c217 no tool inspects a tenant image: bcd reads a qcow2 header and its header extensions in Go (runtime/qcow2.go), which opens nothing else. A qcow2 base image is checked when it is downloaded, before it is renamed into images/, and again every time an instance uses it, and it is refused, failing the instance, when:

• it is not a qcow2 (version 2 or 3) at all, or its header is truncated or malformed;
• it names a backing file;
• it names an external data file (incompatible feature bit 2, or the data-file header extension), or sets an incompatible feature bit bcd does not know;
• it is encrypted.

raw and iso images are never probed: every tool opens them with their format stated, so their content is only data. Base images are mode 0644 in a 0700 images/ directory, so an unprivileged VMM can read only the one bound into its unit.

An instance's own disk is checked too, because a VMM that once ran it can rewrite its header (checkOverlay): disk.qcow2 must name exactly the recorded base image and format as its backing file, or none for a restored snapshot or an iso install. cloud-hypervisor checks it before every start, snapshot and resize; libvirt-kvm before every start, snapshot and grow, with the base format now recorded in the domain's metadata. A restored snapshot's disk must be such a flat qcow2, and its seed is attached as raw, never probed; a failed check removes the new instance's directory. qemu-img create -u builds an overlay without opening the base, and libvirt-kvm's grow reads the virtual size from the header.

The cloud-hypervisor unit

Each VMM runs in its own transient unit bcd-ch-<id>.service in bcd-instances.slice, as its instance's own unprivileged user (b12c217): a uid from 0x7FFD0000 up, kept in vm.json, with User= and Group= set to it. The unit name always ends in .service, so an instance id never names another kind of unit. The VMM needs no network privilege, because it never names a tap: bcd creates and bridges the persistent taps, opens each one (TUNSETIFF) and hands the file descriptors to the VMM over SCM_RIGHTS. vm.create carries no NICs, vm.add-net adds each NIC with its fd, a restore passes them as net_fds, and the saved configuration's NICs carry no tap name. The instance's files and its GPUs' /dev/vfio groups are chowned to the VMM's uid at start, and the groups are given back to root at stop; under bcd helper the node grants ACL entries instead and root hands over the groups (see cloud-hypervisor under bcd helper). systemd-run gets these properties:

This needs systemd 247 or later and a cloud-hypervisor whose API takes NIC file descriptors.

fakehw builds run a file-backed runtime named libvirt-kvm instead, whose domains outlive the process and honor vCPU and memory limits; tests/e2e/bcd-vps.test.ts drives that one through the control plane. On bcd node, a bcx plugin whose driver lists the kind runtime is also a runtime (list, apply, remove over Invoke; no snapshots). A host with neither tool and no runtime plugin runs no instances: a non-empty section fails with this host runs no instance runtime, and bcd doctor warns ("neither virsh nor cloud-hypervisor: this host runs no VPS instances").

Identity rebind

A memory restore resumes its source guest with the source's sshd principals and host key, so an exec certificate for one restore would open the source and its siblings (security review 13 H2). When vps gives a memory restore its own identity, its spec carries InstanceSpec.identity_rebind_ref, a sealed bundle {"v":1,"account","principals","hostKey"?,"hostCertificate"?}, and bcd installs it into the running guest (agent/rebind.go, runtime/rebind.go; ca08d5c):

1. Once the instance runs, the agent fetches the bundle with a SecretRequest bound to the instance's allocation, as for user data, and checks it (runtime.ParseRebind: version 1, the account name, principal lines, and a PEM host key with an OpenSSH certificate line, or neither). It works off the converge pass, since the guest may take a while.
2. The runtime writes a raw 1 MiB disk holding a tar at offset 0 (nonce, account, principals and, with a host key, host_key and host_key-cert.pub), reserves its last 4 KiB for the guest's answer, and hot-adds it with virtio serial bc-rebind: vm.add-disk on cloud-hypervisor (the file chowned to the VMM's user), virsh attach-disk … vdz --live on libvirt-kvm.
3. The guest's udev rule starts bc-rebind@<dev>.service, whose script installs the principals and host key under /etc/ssh, reloads sshd and writes BCREBOUND <nonce> at the start of the last 4 KiB.
4. bcd waits up to 2 minutes for that line, then removes the disk (vm.remove-device, virsh detach-disk vdz --live) and zeroes and deletes its file either way, so the secret never stays on the host.
5. On success it records the reference in data_dir/instances-rebound.json and reports it as InstanceStatus.identity_rebound_ref, from which vps re-points exec at the restore alone. Instances no spec names any more are pruned from the file.

A guest that does not answer is tried at most 3 times per reference, 5 minutes apart; a bundle the control plane refuses to release, or a malformed one, is not retried. Nothing is reported then, so the control plane keeps refusing exec while the identity is shared: it fails closed. The guest half (the udev rule, the unit and the script) is written in daemon/README.md, and vps's sandbox user data installs it verbatim at first boot (3e25c78), so restores of sandboxes booted since then answer. A bcx runtime plugin is not a rebinder: since b8a1801 its spec carries identityRebindRef, sshFromHostOnly, placement and each NIC's tap name, but bcd neither rebinds through it nor reports identity_rebound_ref for it (and plugins take no snapshots, so they restore none). The Fake runtime acts as a compliant guest. Tests: TestParseRebind, TestRebindImage, the cloud-hypervisor and libvirt-kvm rebinds against an acknowledging and a silent fake guest, and TestIdentityRebind, which drives the converger through the secret fetch, the status report and a restart. See a restore's own SSH identity.

Commands

The command body

A Command is the serialized CommandBody, an optional signature, and the delivery attempt (informational, outside the signature). The control plane stores the body and signature bytes when it enqueues the command and the gateway resends exactly those bytes.

The gateway also guards the stored body before sending it: a body that does not decode, lacks not_after, or names another command or node is cancelled with reason invalid-body instead of sent.

Dispatch

Enqueue inserts the command pending in the caller's transaction and, after commit, pokes the node's gateway. Each gateway dispatches on a poke or every second, joining commands to the sessions it holds:

SELECT c.*, s.session_epoch FROM commands c JOIN node_sessions s USING (node_id)
 WHERE s.gateway_id = $me AND c.state = 'pending'
   AND c.not_before <= compute_now() AND c.not_after > compute_now()
 ORDER BY c.node_id, c.seq LIMIT 1000
 FOR UPDATE OF c SKIP LOCKED FOR SHARE OF s

A row is marked dispatched with the session's epoch only when that epoch equals the gateway's in-memory epoch for the node and the lane has credit. Order within a lane is kept: once a lane is out of credit, later commands of that lane wait. Some commands are cancelled at this point instead of sent: a pending command whose cancel was requested, a kind outside the certificate's allowed_kinds (kind-not-allowed), and a destructive kind for a tenant-controlled node (tenant-controlled).

Acceptance checks

A delivered command first goes through the journal's duplicate rules (below). A new command, or a retry-safe one a crash interrupted, then passes these checks in order:

1. UNSUPPORTED: the body names another node, or the kind is unknown or not allowed for this agent's role. While the session is upgrade-only, every kind but daemon.upgrade is refused this way.
2. min_generation above the applied generation: the daemon sends SpecRequest and defers the command. A newer spec re-runs the checks; still behind at not_after, the command is STALE_GENERATION.
3. EXPIRED: past not_after, in server time through the Welcome offset. A command without not_after is EXPIRED, so the control plane always sets one.
4. BAD_SIGNATURE: once the trust bundle holds command-signing roots, a missing, unknown-key or invalid signature on any command, whatever its kind (0d42de8).
5. BATCH_UNAPPROVED, UNBATCHED_LIMIT (v1.0): with approver keys in the trust bundle, a destructive command whose batch approval or scope proof does not admit it, or an unbatched one beyond the hourly allowance. See the quorum.
6. BUSY: no free credit on the lane, or the journal is full. Not terminal: see BUSY.

UNSUPPORTED, EXPIRED, STALE_GENERATION, BAD_SIGNATURE and the two quorum reasons are terminal: the command becomes rejected, and the rejection is journaled, so a duplicate delivery or a Hello replay gets the same answer. BUSY is never journaled. bcd (dispatch.Precheck, then a lane credit) and the FakeDaemon both follow this order, so a bad signature is refused even when the lane has no credit.

Lanes, credits and workers

Credits bound the commands admitted at once, running or waiting for a worker; workers bound the ones running. Each lane has a fixed worker pool, never a goroutine per command, and a per-resource lock (a GPU UUID, a disk, an allocation) keeps two lanes from acting on the same device at once. A command holds its credit from acceptance until its result is journaled.

Kinds by lane

These are the kinds bcd registers (daemon/internal/agent/kinds.go and site.go), with the lanes, roles and flags of commandKinds in @better-compute/protocol, metalCommandKinds in @better-compute/metal and instanceSnapshotKind in @better-compute/vps, plus driver.invoke, which provisionModule() registers. A kind outside the agent's role is UNSUPPORTED.

What each metal kind does on the agent is on BMC drivers, sites and deploying.

Results and acknowledgements

The daemon reports ACKED once accepted is durable, RUNNING when the side effect starts, and one terminal outcome: SUCCEEDED, FAILED (with an error code such as interrupted, invalid-payload, internal or a kind-specific one), REJECTED (with a reason) or CANCELLED. Each update records the boot_id it was produced in. Reports may skip steps; a lost ACKED does not stop a result from landing.

The gateway applies updates through the command state table, scoped to the reporting node, so a node can never move another node's command, and a stale or repeated update changes nothing. The terminal result and the node's signature are stored exactly as received. Every terminal update is then acknowledged in Ack.command_ids, including ones stored earlier or unknown to this node, and only then does the daemon stop replaying it. Settled commands publish command.settled and wake the node's controller.

Timeouts

The CommandSweeper, a singleton, runs every 5 seconds: a pending command past not_after becomes expired, and a dispatched, acked or running one past not_after plus its kind's timeout becomes failed with reason timeout. For self-terminating kinds the timeout covers the rejoin window. The built-in timeouts are 2 minutes for diag.nvml and inventory.refresh, 5 for gpu.reset, and 15 for logs.collect and node.reboot; an unregistered kind gets 15 minutes. Metal registers its own: 1 minute for site.httpProxy, 2 for bmc.query, bmc.boot, boot.prepareRamdisk and boot.cleanUp, 5 for bmc.power and bmc.rotate, 15 for boot.kexec, an hour for bmc.firmware, bmc.storage, diag.dcgm and deploy.image, and 6 hours for disk.wipe.

Cancel

A cancel requested before dispatch wins: the command is cancelled and never sent. For an in-flight command the gateway sends Cancel{command_id, reason} once per session, and again after every reconnect, until the daemon settles it. The daemon stops between actions and never hides a completed side effect: a command still queued or deferred is reported CANCELLED, while one whose side effect has started reports its real outcome. A real outcome always wins over a late cancel.

gpu.reset and live holds

gpu.reset (control lane, destructive, attested, not retry-safe, node role only) resets the GPUs listed in gpu_uuids, after a release or for remediation such as XID 63. bcd checks the list before touching anything:

• an empty list fails invalid-payload;
• a GPU that a live CDI hold from the devices section names, whole or by one of its MIG instances or slots, or that a VM has passed through, fails the whole command gpu-busy, naming the hold or the instance (539c4ce), so a reset never pulls a GPU from under a running workload; diag.dcgm applies the same check from level 2;
• a GPU the inventory does not know comes back reset: false with the message unknown GPU.

Each remaining GPU is reset with nvidia-smi --gpu-reset -i <uuid>; a failure is that GPU's message. bcd then collects a fresh inventory and marks each reset GPU healthy unless it shows uncorrectable volatile ECC errors or a pending or failed row remap. Since cb987e5 a host build reads ECC counters and row remapping from nvidia-smi -q -x when it is installed; without it the inventory carries neither, and every GPU that reset reads healthy. A GPU that reset and reads healthy loses its XID conditions at once instead of keeping them for the rest of the boot (662fb55). The fake HGX node of the fakehw build carries them too. The FakeDaemon's fakeGpuReset (opt in with its kinds option) follows the same checks.

Signed commands and attested results

Two signatures keep a compromised gateway, which can delay, drop or read frames, from forging intent or results:

• Signed intent. Command.signature is a worker key's signature under purpose command; the key chains to the command-signing roots of the daemon's trust bundle. The server signs every command when pki.commandSigning is configured, and sends the matching trust bundle in JoinResponse and Welcome; without it commands go unsigned, as v0.1 allows. Whether a signature is required never depends on a flag the body could carry. bcd enforces nothing until its trust bundle holds command-signing roots; from then on every command, whatever its kind, must carry a valid signature, or it is BAD_SIGNATURE (0d42de8). PLAN.md staged this (destructive kinds from v0.2, all kinds from v0.4), but a kind table that exempts non-destructive kinds lets a compromised gateway inject an unsigned "harmless" kind that still acts: site.httpProxy could POST a Redfish reset or InsertMedia to any BMC of a site around the quorum. The FakeDaemon verifies every command with required: true too (12ccfa1). The destructive flag still matters for the quorum below: gpu.reset, node.reboot, bmc.power, bmc.firmware, bmc.storage, disk.wipe, deploy.image, since 3ff8131 bmc.boot (it decides what a machine boots) and bmc.rotate (it replaces the BMC credentials), and since 0d42de8 site.httpProxy. metalCommandKinds marks the same metal kinds destructive (9c14146, f66e236). site.httpProxy also never reaches a BMC: the site agent remembers every BMC host it drives through a bmc.* command (data_dir/bmc-hosts.json) and refuses those hosts, by name and by resolved address, any /redfish path, and any address in its local [site] bmc_networks, with forbidden-target; a site.httpProxy other than a GET needs a batch leaf bound to its payload under the quorum. And the URL's host must belong to the command's target_node_id (a2a853d): once the site spec lists proxy targets (SiteSpec.proxy_targets, which metal fills with the site's agentless machines and their inspected addresses, never a BMC host; 8391d74), an IP literal must be one of the node's listed addresses and a name must match a listed name or resolve only to listed addresses, else forbidden-target.
• Signed desired state. Specs have the same envelope under purpose desired-state. With pki.commandSigning, the NodeSpec controller signs each new generation's DesiredStateBody with the worker key (CommandSigner.signDesiredState) in the transaction that bumps the generation, and every bump rewrites both columns, so a signature never outlives its generation. The daemon checks the signature with verifyDesiredState: one is required for a spec that changes the access section (v0.2) and, with the v0.4 policy, for every spec; a present one is always checked. A refused spec is not applied. The FakeDaemon implements this (intent.signedSpecs, default 'always': with command-signing roots every spec must be signed, because an unsigned spec with a high generation would freeze all later ones; 'access' for a control plane that signs only access changes; refusals in specRejections). bcd implements the 'always' rule: once it holds command-signing roots, a spec without a valid signature is logged and not applied.
• Attested results. For attested kinds (diag.nvml, diag.dcgm, gpu.reset, disk.wipe, and bmc.rotate signed by the site agent) the node signs CommandUpdate.node_sig over the command id, the state and the SHA-256 of the result, and allocation reports travel as node-signed AttestedReports. Workers verify them against node_certs before any state transition; the gateway never judges or strips them. With attestedBindings (on by default with pki.commandSigning), a reservation goes active only on the node's own signed AllocationBound; see attested bindings.
• The destructive-batch quorum (v1.0). Signed intent does not stop a compromised api or worker, which holds a worker key. With approver_keys in the trust bundle (pki.approvers), a destructive command with a batch_id runs only when Command.batch_approval carries enough signatures of distinct approver keys over a scope whose Command.batch_proof proves the command's (subject, kind); else it is BATCH_UNAPPROVED. Destructive commands without a batch_id are refused UNBATCHED_LIMIT beyond unbatched_destructive_per_hour (default 0), counted per rolling hour of the host's own clock (bcd keeps the count in data_dir, so a restart does not reset it). Each batch leaf runs at most once. bcd (e7ddb38, 8b0f50c) and the FakeDaemon enforce it after the signature check. On the server, the quorum module stores batches and collects approvals, the command queue holds batched commands until their batch is approved, and the gateway sends each one with its batch's batch_approval and its own leaf's batch_proof (40be696). The built-in workflows create their batches: metal gives each workflow one of its own or runs it under a metal.bulk batch (f66e236), and the health reconciler's reboots and GPU resets get a node-scoped one (e594a4f). See the quorum.

Events

Everything the daemon observes on its own (allocation reports, failures) is an event, journaled with a sequence number that increases from 1 per journal_id, and sent in EventBatches. The gateway stores the events above the node's cumulative ack for its current journal, publishes each as node.event (attested reports decoded), runs the report handlers, advances node_sessions.acked_event_seq, and answers Ack{journal_id, event_seq}. A batch for another journal id is ignored.

A report whose body names another node is published as security.node-mismatch and never reaches a handler; an undecodable one is security.invalid-report. The allocation handlers also ignore a report about an allocation that is not placed on the reporting node, as they ignore unknown allocations, so one node's session cannot bind, release or fail another node's allocation. Handlers (allocationBound, allocationReleased, allocationFailed) run in the ingest transaction under a savepoint: a failing one rolls back its own writes, is published as node.report-failed, and does not stop the stream, since an unacked event would only be replayed into the same failure.

The journal

The journal is the daemon's durable memory. bcd's store is bbolt (journal.OpenBolt, journal.db under data_dir); the Go package also holds the contract, the command rules and an in-memory store for tests, and the FakeDaemon mirrors them rule for rule.

The server time mark is the highest server time the daemon has seen (on Welcome and every heartbeat). It only moves forward, and a Welcome whose server_time is more than 5 minutes before it is refused, so a gateway cannot wind the daemon's clock back into an old worker certificate's window and replay expired signed intent. bcd also mirrors the mark into identity.json, so a lost journal does not reset it.

Durability. Every write is fsynced before it returns. accepted reaches disk before any side effect, and self-terminating kinds write running{boot_id} immediately before theirs, so a restart can tell the reboot they caused from a crash.

Bounds. 64 MiB and 10,000 unacked events, which config.toml may lower but never raise. Events are classed metric, transition or result. When the journal is full, the oldest metric events are dropped; results and transitions never are. If that is not enough, the daemon answers new commands BUSY until acks drain the backlog.

Duplicates and crash recovery

Every daemon process start picks a fresh incarnation id. An entry written by another incarnation belongs to a process that no longer exists, so there is nothing to re-attach to. A delivered command is resolved against the journal first (journal.Resolve):

At start, journal.Recover settles the entries whose outcome needs no server input (rebooted and interrupted) so they go out in Hello.unacked_results; retry-safe entries wait for the server to redeliver them, so a re-execution passes the not_after and min_generation checks again. disk.wipe, deploy.image, boot.kexec, bmc.power, bmc.rotate, bmc.firmware, bmc.storage, site.httpProxy, instance.snapshot and gpu.reset are not retry-safe. journal.Sweep deletes done entries older than 7 days, and accepted or running entries that an earlier process left and nothing redelivered within those 7 days (the server has long expired them); entries of the running process are never swept.

The server keeps the matching half at Hello (see resume rules): a new boot completes self-terminating commands, and a new journal fails non-retry-safe in-flight commands with journal-lost instead of replaying them into a journal that cannot deduplicate.

bcd and the FakeDaemon agree on these rules, including the cancelled journal outcome, the producing boot_id on every result, NodeRebootResult.previous_boot_id on a rebooted result, and journaled terminal rejections. tests/e2e/bcd.test.ts checks the reboot path end to end: node.reboot completes SUCCEEDED from the next boot.