Instance runtimes

A VPS host's bcd node runs the instances its NodeSpec's instances section lists through a runtime: a driver for one hypervisor or container engine behind a small Go interface. The instances converger decides what should exist and in which state; the runtime makes it so. This page covers the runtimes themselves. What the converger does around them (the desired set, user data, admission, network rules, snapshot transfers) is on instances, and the control plane's side is on VPS instances.

Which runtimes a host runs

There is no configuration key. A host build of bcd node runs each built-in runtime whose tool is on its PATH when it starts (newRuntimes in cmd/bcd/hardware_host.go), then adds bcx plugin runtimes:

Only node agents run runtimes; bcd site and bcd ramdisk run none. Every runtime the node runs is advertised in Hello.drivers with kind runtime, which the gateway turns into the node label bc.dev/driver.<runtime>. vps places a hosted instance only on hosts that carry its runtime's label (0fca413). If a spec names a runtime this host lacks anyway, that instance fails with runtime <id> is not installed on this host. A host with no runtime at all fails a non-empty instances section terminally with this host runs no instance runtime, and bcd doctor warns ("neither virsh nor cloud-hypervisor: this host runs no VPS instances").

All volumes share one store, data_dir/volumes, whatever the runtime (see volumes).

The Runtime interface

Every runtime implements runtime.Runtime (daemon/internal/runtime/runtime.go):

Optional interfaces add what only some runtimes do: Console (a serial console), Rebind (the identity rebind disk), VolumeGrower (tell a running guest a volume grew), the balloon's size and usage readers, SelfAdmitting (a runtime that checks capacity itself, so bcd's admission leaves it alone) and NormalPagesRestore (memory restores run on normal pages).

Two rules hold for every runtime:

• Ownership. A domain is Managed only when bcd created it. Every other domain is listed so the control plane can report the leak, and is never defined over, started, stopped or destroyed.
• Tap names. NIC i of instance id lives on the host interface runtime.TapName(id, i): bc, the first 10 hex digits of the SHA-256 of the instance id, n, then the index (it fits IFNAMSIZ). bcd's nftables rules match taps by name, so they exist before the tap does.

libvirt-kvm

runtime.Libvirt drives virsh -c qemu:///system and qemu-img. Each domain it defines carries bcd's metadata (namespace https://better-compute.dev/libvirt/1: allocation, image digest and format, addresses, the snapshot it was restored from), which is how bcd tells its own domains apart.

Running as root, bcd adds o+x (search, never read) to each directory from data_dir down to a domain's disks, seed, volumes and base image, so QEMU's own user (libvirt-qemu or qemu) reaches them while listing stays closed (f7b8805). Under the helper the model is different (below).

fakehw builds replace the runtime with a file-backed fake under the same id. It writes the domain XML bcd would define on a real host to data_dir/fakehw-plans/<id>.xml (or <id>.error when the spec would be refused) through runtime.RenderLibvirt, so tests can assert pinning, NUMA binding, huge pages, the balloon, volumes and GPU addresses that the fake cannot run.

cloud-hypervisor

runtime.CloudHypervisor runs one cloud-hypervisor VMM per instance in a systemd transient unit bcd-ch-<id>.service in bcd-instances.slice (started with systemd-run --collect, so a VM outlives bcd), and drives it over its REST API on instances/<id>/api.sock. cloud-hypervisor keeps no registry, so instances/<id>/vm.json is bcd's record: the allocation, image and base, the VMM's uid, addresses, networks, MACs, GPUs, placement, volumes, the VM configuration, and what was handed to the running VMM (VFIO groups, bound PCI addresses, volumes). A directory without that record, marked better-compute/cloud-hypervisor/1, is not bcd's.

It needs systemd 247 or later, which bcd doctor checks, and a cloud-hypervisor whose API takes NIC file descriptors.

bcx plugin runtimes

A bcx driver plugin whose manifest lists the kind runtime becomes a runtime on bcd node (agent.PluginRuntime, walkthrough 4). It speaks the TypeScript RuntimeDriver contract over Invoke, with positional JSON arguments:

Ensure only records the spec; SetState sends apply with the desired state, so the plugin both defines and starts. A new instance's user data and a restore (as a local file:// URL of the verified snapshot) go with its first apply only. The idempotency key is instance:<id>:apply:<state> or instance:<id>:remove. Everything a runtime plugin lists counts as bcd's.

What bcd passes a plugin, and what it does not:

• Passed (b8a1801): each NIC's tap name, gateway and nameservers, the network policy, sshFromHostOnly, identityRebindRef, the placement, and each volume's path and serial in attach order.
• Enforced by bcd. When the tap rules guard an instance (a PRIVATE or NONE policy, or SSH from the host only) and a running plugin instance has no interface under one of its tap names, bcd stops and fails it (runtime <id> did not create taps …: its network rules cannot apply).
• Not supported. Snapshots (Snapshot answers unsupported, so no plugin snapshot exists to restore from, although a restore's verified file would be passed on); the identity rebind and identity_rebound_ref; consoles.

Plugins need root to start, so an unprivileged node under the helper hosts none (see bcd helper).

The fake runtime

runtime.Fake keeps its domains in a JSON file, so they outlive the daemon process. A new boot id (a host reboot) stops every domain, as it would stop real VMs. It honors optional vCPU and memory limits itself (AdmitsItself, so bcd's host admission skips it), refuses a disk that shrinks, refuses a snapshot from another runtime and a memory snapshot from another CPU model, refuses a memory snapshot beside a read-write volume, echoes on its "console", and acts as a compliant guest for the identity rebind. The TypeScript FakeHypervisor follows the same rules, which is what better-compute dev --sim and the simulator run.

Passthrough GPUs

An instance's gpu_device_ids are GPU UUIDs. Both built-in runtimes find each GPU's PCI address from the inventory's bus id (a GPU already on vfio-pci is found by its remembered address), and an instance whose GPU has none fails.

Persistence mode matters because nvidia-persistenced (which the nvidia-driver host unit keeps running) holds every GPU open, and the driver's unbind waits until the GPU is closed. Both runtimes refuse a memory snapshot of an instance with passthrough GPUs, since the GPU's state lives in the device (an instance with passthrough GPUs cannot be snapshotted with its memory; snapshot its disk only). The host's IOMMU and vfio-pci come from the iommu and kernel-modules units, and bcd doctor warns when vfio-pci is missing on a host with an active IOMMU.

NoCloud seeds

User data never travels in the spec. The converger fetches it with a SecretRequest bound to the allocation, and the runtime writes it into a NoCloud seed, built with genisoimage (-volid cidata -joliet -rock), else xorriso -as mkisofs with the same options:

libvirt-kvm attaches seed.iso as a read-only cdrom; cloud-hypervisor attaches the same file as a read-only raw disk. A restored instance keeps the seed in its snapshot, attached as raw and never probed. bcd doctor fails a VPS host without either genisoimage or xorriso.

Placement

InstanceSpec.placement (from a flavor's placement, see flavors) is checked before anything is created (runtime.PlacementOf), and a placement the host cannot apply fails the instance:

A restore keeps bcd's pinning, never the snapshot's. A cloud-hypervisor memory restore runs on normal pages whatever the placement says, because its saved configuration keeps no huge pages; admission charges it that way (RestoresOnNormalPages). The balloon controller that resizes balloons under host memory pressure is on memory ballooning. bcd advertises instance.placement in Hello on hosts with a runtime, and vps sends placements only there.

Admission

Before a runtime starts a domain that is not running (a resume from PAUSED is not a start), bcd checks that the host can give it its memory (0dd122a), counting what is committed rather than used: running and paused managed domains on normal pages with their VMM overhead (1/32 of the memory plus 256 MiB), bound holds' memory limits, and starts already admitted in the same pass, against MemTotal less the huge page pool and 1 GiB of headroom. An instance with huge pages is checked against free pages of that size on its NUMA nodes and host-wide (aeaa982). A start that does not fit stays stopped with ResourcesExhausted: … in its status, is retried on the next pass, and is not an ERROR. The rules and what vps does with the refusal are on instances.

A runtime that reports ErrNoCapacity, or a libvirt-kvm start that fails for lack of memory, is a capacity failure of that instance. A runtime that admits itself (the fake) is left alone, and a host whose /proc/meminfo cannot be read starts without the check, with a warning.

Volumes

Persistent volumes are sparse raw files in data_dir/volumes/<id>.raw, shared by every runtime on the host and attached by serial (/dev/disk/by-id/virtio-<serial> in the guest), vdb onward after the root disk:

Growth is applied live through VolumeGrower (virsh blockresize, vm.resize-disk; b5415a2). bcd tracks which domains have which volumes open and never lets a second writer in, applies stops and deletes before starts, and never destroys a volume a running VM has open. Both built-in runtimes refuse a memory snapshot beside a read-write volume (ErrMemorySnapshotWithVolumes). The store's rules, space accounting and reports are on persistent volumes.

Snapshots and restores

instance.snapshot asks the runtime for a tar written to a staging file under data_dir/snapshots, which bcd hashes and uploads; restore_from downloads and verifies a snapshot before a new instance boots from it. The transfers are on instances; the runtimes' formats are:

A snapshot names its runtime in meta.json, and the other runtime refuses it. A memory snapshot restores only on the same CPU model. A memory restore's guest gets its own SSH identity through the identity rebind (ca08d5c), which both built-in runtimes implement with a hot-added vdz disk.

Serial consoles

Instance console tunnels (f8273ef) reach a running, managed instance's serial port, never the host's shell. libvirt-kvm opens the pty libvirt gave the domain's serial port; cloud-hypervisor dials instances/<id>/serial.sock; the fake echoes what it is sent, prefixed with console: . Under the helper, root opens the libvirt pty and passes the node its descriptor (libvirt.console). Plugin runtimes have no console.

Under bcd helper

On an unprivileged bcd node ([helper] set), forHelper (cmd/bcd/agent.go) wires each built-in runtime to bcd helper:

libvirt then runs with dynamic_ownership = 0, and QEMU's user is in the node's group, so instance disks are 0660 (read-only ones 0640) and the directories on the way get group search instead of others' search. Without [helper] instance_bridges, root defines no domain with a NIC and creates no tap. What root checks for each operation is on bcd helper and in the overview's libvirt through the helper and cloud-hypervisor through the helper.

What the tests cover

None of these runs a real VMM. Round 6 (2026-10-04) brought bcd onto a real host with an RTX 5090 in WSL, but that host has no hypervisor installed, so no instance has booted on real KVM.