# Roadmap (/docs/reference/roadmap) > The version plan from P0 to v1.0 with each phase's deliverables and exit gates, and the honest state of every item in the repository today. Better Compute is built in phases, each defined in PLAN.md §16 by its deliverables, its acceptance criteria and its exit tests. A phase closes only when its exit tests are green in CI; the benchmark gates are measured on the reference hardware in `bench/REFERENCE.md` (a gateway host with 8 vCPU and 32 GiB, PostgreSQL 17 on 16 vCPU and 64 GiB NVMe). The phases were planned in order, but the work did not stay in order: the TypeScript control plane for every phase up to v0.5 exists in the repository, and the Go daemon covers the v0.1 node agent, while the daemon's later modes and the pieces that need a PostgreSQL server or real hardware do not exist yet. So no phase after P0 is formally closed yet. This page tracks each item as: * **Done**: in the code, with tests, and listed in the changelog. * **In progress**: partly there. The note says which part is missing. * **Not started**: nothing in the repository yet. > **Three things hold every phase open.** - **The Go daemon is not released, and only its node agent runs end to end.** `bcd join`, `bcd node` and `bcd status` work and are tested end to end against a real gateway (`tests/e2e/bcd.test.ts`, the fake-hardware build in WSL), including a node console tunnel, and the node agent also converges break-glass SSH (`access`). `bcd site` and `bcd ramdisk` are written and covered by Go unit tests only. The `instances` converger, instance tunnels and the IMDS run end to end only on fakehw builds (`tests/e2e/bcd-vps.test.ts`); the `libvirt-kvm` and `cloud-hypervisor` runtimes of host builds (362317c, 2598f0a) are tested against fake tools only. The release pipeline exists but no release has been cut. GPUs are read through `nvidia-smi` (cb987e5) with no go-nvml backend. Self-update, the WebSocket fallback and `bcd helper` (self-update included) are done and exercised through the TypeScript `FakeDaemon`. `bcd sim` runs many real node agents in one process (fakehw builds only). One real host has run a host build of bcd (round 6, 2026-10-04): a WSL 2 workstation with a GeForce RTX 5090. `tests/e2e/bcd-host-real.test.ts` (`BCD_E2E_REAL=1`) joins it to a dev stack, adopts it, reads its real inventory and GPU, exports the GPU's live metrics over OTLP and answers `bcd status` and `bcd doctor`. That bcd ran as an ordinary user without `[helper]`, so it binds no GPUs: no real `/etc/cdi` spec, hold slice, passthrough or MIG has run yet. See [running bcd on real hardware](/docs/daemon/real-hardware) and [what bcd does today](/docs/daemon#what-bcd-does-today). - **No PostgreSQL server has run the suites.** Local runs use PGlite (PostgreSQL 18 in WebAssembly). Suites that need real concurrency skip unless `BETTER_COMPUTE_POSTGRES_URL` is set, and every benchmark number below is from PGlite on a development machine, which the gates do not count. - **The hardware gates (PLAN.md D15) have not run.** They need one NVIDIA box, a self-hosted KVM runner and the reference benchmark machines, and rented A100 or H100 capacity before the v0.5 exit. ## At a glance [#at-a-glance] | Phase | Theme | State | | -------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [P0](#p0-scaffold) | Scaffold | Done | | [v0.1](#v01-adopt-and-reserve) | Adopt and reserve | Control plane done; bcd's node agent done and tested end to end, not released (one real GPU host has joined, been adopted and reported its GPU; a real GPU hold, a data-center GPU server and the install-script run are open; GPUs come from `nvidia-smi`, cb987e5); PostgreSQL and the reference gate runs | | [v0.2](#v02-metal) | Bare metal | Control plane done and tested in simulation; `bcd site` and `bcd ramdisk` written (Go unit tests only, not released), never run against the control plane or on hardware | | [v0.3](#v03-vps-and-access) | VPS and access | Control plane done; tunnels and the `access` section done in bcd (not released); `libvirt-kvm` and `cloud-hypervisor` written in bcd and tested against fake tools only | | [v0.4](#v04-scale-and-ha) | Scale and HA | Mostly done in simulation; `bcd helper` done, self-update through it included; the numbers need the reference PostgreSQL | | [v0.5](#v05-gpu-depth-and-ecosystem) | GPU depth and ecosystem | Control plane done ahead of schedule; sandboxes served in full on the control plane; hardware exits not run | | [v1.0](#v10-freeze) | Freeze | Started: the API surface freeze check, bcd's SLSA3 release pipeline, the N-2 skew matrix and the upgrade-in-place scenario exist (FakeDaemons), and the destructive-batch quorum is built end to end, its producers included; the security review and npm provenance do not | | [v1.1](#cpu-memory-and-kubernetes-v11) | CPU, memory and Kubernetes | Mostly in the code (host configuration, host resources, slices, MIG and slot holds, Kubernetes workers, bcd as a DaemonSet), tested against fakes only | Every package is on npm at version 0.1.1 on the `latest` dist-tag (0.1.0 was the first release, on 2026-10-07) and accepts Better IAM `>=0.2.0 <0.4.0` from npm. Provenance attestations come once CI publishes from the GitHub repository. bcd has no released binary yet. ## P0: Scaffold [#p0-scaffold] P0 was committed with `pnpm check` green, and its contracts are frozen: `proto/bettercompute/agent/v1`, the `@better-compute/core` contracts, the v0.1 schema and the v0.1 rows of PLAN.md §9.2 now change only by additive, reviewed edits. | Deliverable | State | Notes | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | git and `git init` | Done | With GitHub Desktop's bundled git; git is not on the machine's `PATH` | | A pnpm workspace mirroring Better IAM's | Done | | | Scripts `check-deps`, `check-sql-now`, `umbrella`, `sync-better-iam`, `proto-gen`, `authz-matrix` | Done | Under `scripts/`; `pnpm lint:repo` runs the repository rules | | Better IAM vendored | Done | 0.1.0 as tarballs in `vendor/better-iam`, pinned through `overrides`, until Better IAM was on npm; the releases depend on Better IAM from npm (`>=0.2.0 <0.4.0` since 0.1.1) | | Packages `core`, `protocol`, `postgres` (with `./pglite` and `compute_now()`), shells of `server`, `client`, `cli`, `testing`, stubs of `inventory`, `provision`, `metal`, `vps`, and the umbrella | Done | The stubs have since been implemented | | `protocol` with TypeScript and Go generated code committed | Done | TypeScript by `scripts/proto-gen.mjs`; Go in `daemon/gen/` by `scripts/daemon-proto-gen.mjs` (protoc-gen-go v1.36.12, protoc-gen-connect-go v1.21.0), with `--check` | | `daemon/` skeleton and goreleaser configuration | Done | The module builds and its tests pass (`node scripts/daemon-test.mjs`; cross-compiled and run in WSL on Windows). goreleaser is configured and checked in CI, but no release has been cut | | CI: Ubuntu and Windows on Node 22 and 24, a PostgreSQL job, a Go job; the IAM contract test; `bench/REFERENCE.md` | Done | `.github/workflows/ci.yml` and `daemon.yml` are defined, and since then `nightly.yml` (the simulation matrix with the 10k cold start, the IAM-guarded suite in both IAM modes, the whole suite on Postgres, a green-night streak) and `release.yml` (bcd with SBOMs, cosign and SLSA3 provenance); see [CI](/docs/operations/testing#ci-and-the-repository-checks). This page does not track CI runs | | Exit | State | | -------------------------------------------------------------------------------- | ---------------------------------------------- | | `pnpm check` green, including `umbrella --check` | Done locally | | `buf breaking` against `proto/baseline.binpb` | Done: the baseline is committed and CI runs it | | Packed smoke importing `better-compute/{inventory,provision,metal,vps}` | Done: `test:packed` | | Freeze of `agent/v1`, the core contracts, the v0.1 schema and the v0.1 §9.2 rows | Done | Of the day-one verifications, PGlite's `btree_gist` is verified. The go-nvml mock (go-nvml is not a dependency yet), the KVM runner and CDI on target hosts are not, for want of hardware. ## v0.1: Adopt and reserve [#v01-adopt-and-reserve] v0.1 adopts servers you already run (a brownfield join with a token) and reserves their GPUs. It was split into five workstreams. | Workstream | State | Notes | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **W1, Go daemon**: join, session, journal, Renew with proof of possession, NVML inventory, `devices` and `access` convergence, CDI holds, `diag.nvml`, `node.reboot`, release artifacts, install script | In progress | Done: `bcd join` (sealed token, verified `JoinResponse`), `bcd node` (session over pinned mTLS, bbolt journal, Renew with proof of possession at 60% of the lifetime, the `devices` converger with CDI holds and node-signed `AllocationBound`, the `access` converger for break-glass SSH (CA, node-bound principals, a native KRL), `inventory.refresh`, attested `diag.nvml`, `logs.collect`, `gpu.reset`, self-terminating `node.reboot`, signed-intent checks, the server time mark, the trust version in `Hello` and `StatusReport`), and beyond W1: tunnels with node-verified exec tickets (instance tickets never reach the host), `SecretRequest` and `WorkloadTokenRequest`, XIDs from the kernel log (kept across a bcd restart in the same boot) and host pressure, and the v0.2 `bcd site` and `bcd ramdisk` and the v0.3 `instances` converger (see below). Also `bcd status` over the unix socket, `bcd doctor`'s host checks, procfs/sysfs inventory, and the install script (`better-compute daemon install-script`); `tests/e2e/bcd.test.ts` runs it against a real gateway. The GPU inventory is read through `nvidia-smi` without cgo (cb987e5: MIG, NVLink, topology, ECC and row remapping, tested against captured output; no go-nvml backend and no NVML events). Open: release artifacts (the release workflow, with SBOMs, cosign signatures and SLSA provenance, exists, but no tag has been released), and a run on a real host | | **W2, gateway and PKI**: mTLS sessions, the serving CA, epoch fencing, liveness batching, dispatch by join, resume rules, `BUSY` re-queue, local CAs, the `join-token` attestor, JoinService, the downloads route | Done | `@better-compute/gateway`, `@better-compute/pki` | | **W3, store, capacity and inventory**: schema, `compute_now()`, work queue, outbox, leases, CommandSweeper, `InlinePlacer`, atoms, rollups, `gpu-models.json`, conformance and torture tests | Done | `@better-compute/postgres`, `capacity`, `inventory` | | **W4, IAM bridge**: the catalog, plugin and inbox, projection and resolver, `embedded()`, IAM jobs, flag cache, usage sink, IamReconcile, contract test, authz matrix | Done | `@better-compute/iam` | | **W5, server, client, CLI, simulator**: `operation()`, the method registry, the v0.1 groups, the typed client, the CLI (`dev`, `iam setup`, `daemon install-script`, `pki issue-gateway`), the deterministic simulator and the real-time harness | Done | Since then the CLI has commands for every API group (PLAN.md §9.3; 44915e5, e2fe6f8, bb89083, d1e83ca, 573b0b0, 171d9be), and a test ties every route they call to the method registry (fb6304f). See [command groups](/docs/guides/clients/cli#command-groups) | | Acceptance | State | Notes | | ------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A real `bcd` installed by the install script, plus 3 FakeDaemons, join and match the golden inventory | In progress | The FakeDaemon half is done, and Go's `fakehw` matches the TypeScript section hashes. A real `bcd` (fakehw, in WSL) joins and is adopted in `tests/e2e/bcd.test.ts`, but not installed by the script (there is no release binary) and not beside FakeDaemons | | An org user reserves 4 GPUs from an offered pool; the CDI spec lists exactly the booked UUIDs, which `reservations.get` returns | In progress | `tests/e2e/spine.test.ts` runs it with embedded Better IAM, the example policies, a real gateway and three FakeDaemons: the node's CDI spec, held devices and env file list exactly the booked UUIDs. `tests/e2e/bcd.test.ts` binds a 2-GPU reservation on the real bcd and checks its CDI spec file under the fake root. A CDI spec on a real GPU host has not been checked | | `ReserveBoundedH100` allows 8 GPUs and denies 9; `BoundedExtend` denies 72 h + 1 h | Done | IAM bridge and contract tests | | `gpu.h100.hours` reaches Better IAM billing without duplicates across worker restarts, even with a deleted identity in a batch | Done | Idempotency keys on every usage window; the sink re-attributes deleted identities | | Place-to-active p99 under 2 s with 1k FakeDaemons in the real-time harness | Done, indicative | 1.02 s at 5 placements/s on PGlite; the inline placer saturates one pool between 5 and 10/s | | No upstream Better IAM change | Done | The supported topology of PLAN.md §12.1 | | Exit test | State | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The spine end to end (`tests/e2e/spine.test.ts`) | Done with FakeDaemons: join with sealed tokens, `diag.nvml`, adopt, reserve 4 GPUs as a policy-bound org user, bind, release; 9 GPUs, 73 h, non-pool-users and other organizations denied. `tests/server-spine-v01.test.ts` covers the same path in-process | | `tests/capacity-concurrency.test.ts`: 1,000 concurrent operations never double-book, and the torture test | Done on PGlite (interleaved); the truly concurrent PostgreSQL half skips without a server | | Faults: gateway kill mid-command runs nothing twice, journal loss fails instead of replaying, a reboot completes on rejoin, invariants I1, I2, I5, I8 | Done in `tests/sim/fleet-gateway.test.ts` and `fleet-stub.test.ts` | | Security negatives: an IamPort spy records zero IAM calls on fleet paths over 10 simulated minutes | Done (`tests/sim/iam-negative.test.ts`) | | Security negatives: authz matrix at 100%, resolver timeout denies, a failed `authorizeMany` fails the list, join tokens honour `maxUses` and TTL, node A cannot get node B's spec, Renew without proof of possession is rejected, cross-tenant requests leak nothing | Done | The v0.1 gates are where the phase is really held open: | Gate | State | | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Gateway**: one Node gateway holds 5,000 mTLS sessions at 10 s heartbeats for 30 minutes, dispatch p99 under 500 ms, RSS under 2 GiB | Sessions, dispatch and RSS criteria met on a workstation; the reference run remains. On a development workstation (PGlite, 60 s, the gateway and its store in their own process): 5,000 mTLS sessions, one TLS 1.3 connection each, dispatch p99 26 ms, gateway process RSS 1,240 MiB, 0 drops, and a reconnect storm with 5,000 of 5,000 back in 39 s and every command settled once. A 30-minute run of the same setup on PGlite met every criterion (dispatch p99 24 ms over 90,000 commands, 0 drops), RSS only just: it grew to 2,016 MiB with the JS heap flat at 235 MiB, which a 5-minute attributed run traced to PGlite's WebAssembly memory in the gateway process. The 30-minute run with PostgreSQL 17 on the reference host, which is to confirm the RSS criterion, has not happened, so D20 (a Go gateway) stays formally undecided. See [scaling](/docs/operations/scaling#gateway-sessions) | | **IAM lane**: embedded Better IAM on adapter-postgres sustains 50 admissions/s with lock wait under 50%, and the ceiling is recorded | Not started: the suite exists (`tests/bench/iam-lane.bench.ts`) but needs PostgreSQL, so D18 (proposing R1 upstream) stays open | | Placements/s recorded | Done on PGlite: 35.7/s for the inline placer at 1k nodes | ## v0.2: Metal [#v02-metal] v0.2 takes servers from a BMC and network boot to a deployed OS and back, with verified wipes. | Deliverable | State | Notes | | ------------------------------------------------------------------------------ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `bcd site` and `bcd ramdisk` | In progress | Written, with Go unit tests. `bcd site`: Redfish BMC operations, BIOS settings, RAID layouts and firmware updates from the verified image cache included (944221c), each only under its own `bmc.*` kind, other BMC drivers as hosted bcx plugins with `bcx-ipmi` for IPMI power, boot device and inspection (51806ab, 7818f19; tested against a fake `ipmitool`; no vendor driver), `site.httpProxy`, boot entries, and the boot server (proxy DHCP, TFTP, iPXE scripts with signed discovery assertions under the boot lease, the image cache). `bcd ramdisk`: the `discovery` join from its kernel command line, `disk.wipe`, `deploy.image`, `boot.kexec`, `diag.dcgm`. Open: a run against the control plane or on hardware, a release, and a ramdisk image and iPXE builds. The FakeDaemon plays both roles in the tests | | `provision` and `metal()` | Done | The workflow engine, profiles, the `metal` group and reconciler | | `siteTokens` and the SiteBoot lease; `joinRequests` | Done | A standby site agent is promoted within 30 s of the active one's loss | | `discovery`, `provisioned` and `x509pop` attestors | Done | | | Signed destructive commands and the trust bundle; node-signed attested results | Done | Every command is worker-signed when `commandSigning` is configured; a gateway-forged wipe result is refused | | `SecretRequest` sealed secrets, the `secrets` API and BMC rotation | Done | Two-phase BMC password rotation. bcd sends them for BMC credentials, `site.httpProxy` secret headers, discovery tokens and ramdisk user data, and signs `bmc.rotate` results | | The image catalog and site caches; FleetConfig plan and apply | Done | | | XID and health classification, and `diag.dcgm` acceptance | In progress | Classification, quarantines and remediation are done, and bcd reports XIDs from the kernel log and runs `gpu.reset` and `diag.dcgm` (`dcgmi diag`, in the ramdisk and on nodes); no DCGM run on real GPUs yet | | The revocation-push SLA | Done | A revoked node's session closes within 5 s (about 1 s in tests) | | Emulation CI | In progress | `examples/emulated-site` describes metal on QEMU/KVM with sushy-tools and `bcd site` on a Linux host with KVM; nobody has run it (it also needs iPXE builds and a ramdisk image; BMC addresses may name each VM's system on one emulator since 4e4c1d7), and there is no KVM runner (D15) | | Walkthrough 1, Ubuntu autoinstall | Done in simulation | `tests/extensions/ubuntu-autoinstall.test.ts` | | Acceptance and exit | State | | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 20 unattended enroll, ready, deployed, released, ready loops on UEFI and BIOS VMs | Done in simulation; not run on VMs | | Killing the site agent or the worker mid-deploy resumes cleanly (`metal-chaos`) | Done (`tests/metal-chaos.test.ts`) | | Killing the active site agent mid-PXE: the standby takes over boot within 30 s | Done in simulation (`tests/metal-site-failover.test.ts`): the standby is promoted when the dead agent's lease runs out, the machine boots from it, and the boot step's BMC commands ran once | | A gateway-forged `AllocationBound` is rejected | Done (`tests/e2e/attested-bindings.test.ts`, through a real gateway with signing FakeDaemons); see [attested bindings](/docs/guides/capacity/reservations#attested-bindings) | | Every `ready` follows a verified wipe; `metal-lifecycle` | Done | | Site agents refuse other sites' steps | Done | | A `SecretResponse` cannot be opened with another key or after its TTL | Done | | Unsigned images are rejected; a ramdisk certificate cannot open a `node` session; unsigned destructive commands are rejected | Done | ## v0.3: VPS and access [#v03-vps-and-access] | Deliverable | State | Notes | | -------------------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `vps()`: instances, flavors, provider accounts, overcommit, IPAM | Done | The control plane and the instance reconciler | | Host runtimes `libvirt-kvm` and `cloud-hypervisor` | In progress | They run inside bcd on each host, under its `instances` converger. `libvirt-kvm` is written (362317c): a host build with `virsh` runs instances through `virsh` and `qemu-img` (qcow2 overlays on sha256-verified base images, NoCloud seeds, VFIO GPUs, host-passthrough CPU, snapshots and restores), but it has been tested only against a fake `virsh` and `qemu-img`, never on a real KVM host. `cloud-hypervisor` is written (2598f0a): a host build with the `cloud-hypervisor` binary runs one VMM per instance as a confined `systemd-run` unit over its REST API, with the same overlays and seeds, VFIO GPUs, memory snapshots and restores; it has been tested only against a fake VMM API and fake tools, never on a real KVM host. Security review 12's two high-severity findings are fixed (933c51b): taps are persistent and bridged before the VMM starts, with an `inet bcd_taps` backstop for a tap off its bridge, and both runtimes refuse a tenant image or restored snapshot that names a backing file or an external data file; the VMM's unit sees only its own instance and base image and is bounded by its flavor. fakehw builds run a file-backed runtime, and the FakeDaemon's `FakeHypervisor` stands in for the control-plane tests; see [delivery](/docs/daemon/delivery#the-runtimes) | | `provider-hetzner` (Cloud, and Robot with `bootstrap`) | Done | Outside the umbrella package | | Single-tenant containers | In progress | Pools declare `policy.runtime` and must set `policy.singleTenant` for container runtimes, and vps refuses a container on a pool that serves other tenants; the host runtime needs bcd | | Tunnels with daemon-verified tickets, `nodes.issueSshCert`, the `access` section | Done | Including node-bound break-glass certificates and an SSH KRL for revocation. bcd (not released) re-verifies tickets against the pinned JWKS and its journaled `jti` set, runs node consoles on a real pseudo-terminal, dials instance `ssh` and `tcp` tunnels to the instance's own address, answers an instance `console` ticket with the instance's serial console (from its runtime: the pty `libvirt-kvm` gave the serial port, or `cloud-hypervisor`'s serial socket; f8273ef), which `tunnels.open` issues as op `console` with `instanceId`, decided as `compute:instances:exec` (798e710; `better-compute console --instance`, the console's `/instances//console` page, 191898b), advertises `tunnel.instance-dial` when it runs a runtime, and converges `access` into sshd; it refuses `exec` into a guest, which needs a guest agent. The FakeDaemon answers tunnels with an echo shell and has no `access` converger | | Webhooks and SSE | Done | `@better-compute/events` | | `mcp` | Done | `@better-compute/mcp` | | Delegated confirmation | Done | | | The `cloud-iid` attestor | Done | | | Walkthrough 2 (a VPS provider plugin) and walkthrough 3 (Talos) | Done in tests | `tests/extensions/acme-cloud.test.ts` (against an emulator, live and from a recording), `tests/extensions/talos.test.ts` | | Acceptance and exit | State | | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 200-instance churn on 3 emulated hosts with no leaks | Done (`tests/vps-churn.test.ts`) against emulated hosts in the test process | | Host loss recreates instances per policy | Done | | An agent reserves over MCP with a person's confirmation, retrying with the same key | Done (`tests/mcp-e2e-confirmation.test.ts`) | | A replayed exec ticket is rejected by the gateway and the daemon | Done: end to end with the FakeDaemon as the daemon, and in bcd's own unit and interop tests, which also check that the `jti` is journaled | | `delegation:revoke` closes the agent's stream within 5 s | Done (`tests/e2e/access.test.ts`) | | Multi-tenant container offers are rejected; no tenant VM runs bcd | Done | | `tests/e2e/vps.test.ts` | Done with FakeDaemon hosts on the dev stack (real server runtime and gateway, fake hypervisors): cloud-config through the secret broker, 200-instance churn with no leaks, host loss and recreation, container offers refused, and a tenant VM cannot join as a node with anything it holds. Real KVM hosts need bcd's `libvirt-kvm` or `cloud-hypervisor`, neither of which has run on one yet | ## v0.4: Scale and HA [#v04-scale-and-ha] | Deliverable | State | Notes | | -------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `PoolLeasePlacer` | Done | Opt-in: `capacityModule({ placer: 'pool-lease' })` (D22) | | Shard rebalancing | Done | A silent worker's shards are taken at lease expiry | | Several gateways | Done in simulation | Three real gateways behind the fleet simulator, with gateway kill and zombie fencing | | WebSocket fallback, signed frames only | Done | Gateway and FakeDaemon sides, and bcd (6cd48be): `[websocket] url`, `proxy_ca_file` and `audience`, the gateway's proof checked against the pinned serving CA, renewal without a client certificate over `POST /v1/renew` on the WebSocket listener (06cb334), no tunnels, command-signing roots required; `daemon install-script --websocket-url` writes the section (0ea4ab4). `tests/e2e/bcd.test.ts` runs bcd over the real WebSocket gateway, renewal included (4b2d5aa). See [the fallback](/docs/daemon/protocol#websocket-fallback-v04) | | A/B self-update and Rollout | Done, not released | Signed releases and wave rollouts, and bcd's node half (de9f380): signed manifests under `[update] release_pins`, a release floor, A/B slots, `upgrade.json` with a 120 s deadline, confirmation on the first `Welcome`, a swap back without one, a crash-loop rollback unit, and `daemon.upgrade`; `tests/e2e/bcd.test.ts` updates a real bcd through a real exec (fc0ca6a). A confirmed release is never undone: bcd refuses every older binary, the other slot's included (c139b9c), `rollouts.abort` with `rollback` returns only unconfirmed upgrades (8347a3d), and a bad release is fixed by rolling forward to one with a higher sequence. No release has been cut. See [self-update](/docs/daemon/install#self-update) | | `bcd helper` | Done | Privilege separation (28dc2ae): with `[helper] socket` and `user`, `bcd node` runs unprivileged and `bcd helper`, as root, serves it typed operations over a unix socket checked with `SO_PEERCRED` and the caller's executable: the files bcd owns by kind and id (CDI specs, allocation env files, the sshd drop-in, CA, KRL and principals), `firewall.apply` and `firewall.check` on `bridge bcd_instances` and `inet bcd_taps`, `gpu.reset` of the host's own GPUs, `reboot` and `ssh.reload`, each logged. `tests/e2e/bcd.test.ts` runs an unprivileged node beside a real helper through a GPU hold's CDI spec. `daemon install-script --unprivileged` sets it up (169c459): the `bcd` account, `bcd-helper.service`, a `User=bcd` drop-in, the socket in `/run/bcd-helper`. Self-update through the helper (2d67b25): the `update.*` operations, root's `[update] slots_dir`, the helper re-verifying the manifest and hashing the staged binary as it copies it, its own deadline watchdog; the e2e delivers a signed build to an unprivileged node. Security review 11 (c139b9c): no downgrade through the other slot, the staged file opened beneath its directory without symlinks or FIFOs, the node's claimed version never trusted, at most 32 requests at once, and root-only directories on the way to the slots and socket; its follow-ups (9e71b1f) keep the entrypoint out of `data_dir`, adopt only a root-owned binary, and answer `busy` over the cap. The instance runtimes go through it too: `libvirt-kvm`'s checked `libvirt.*` operations (ed3c7d2) and, since 7fe74a1, `cloud-hypervisor`'s `ch.*` (taps, VFIO groups and VMM units root builds from the instance record it checks; tested against fakes only). Not helper operations: PLAN.md's `runtime.*` and `mig.apply`; bcx plugins need a root node. See [privilege separation](/docs/daemon#privilege-separation) | | Signing of all commands and specs | Done | With `pki.commandSigning`, every command and every NodeSpec generation is signed by a worker key; the gateway relays stored signed specs verbatim, and the FakeDaemon verifies them (`intent.signedSpecs`). bcd verifies them too once its trust bundle holds command-signing roots: every spec and every command, whatever its kind (0d42de8; the FakeDaemon since 12ccfa1) | | Remote IAM (`remoteIam`, `computeIamHost`, `replicaProjection`, `postgresInbox`) | Done | U0 is an upstream Better IAM item outside this repository. With `remoteIam({ refresh })`, an admission waits (at most 2 s) until the IAM host's replica has replayed compute's commit and re-reads the row, failing closed with `UNAVAILABLE`; `doctor` warns `iam.refresh` when it is missing | | Remote-IAM MCP | Done | `remoteMcpIam()` gives `@better-iam/mcp`'s gate an IAM reached over HTTP; see [MCP](/docs/guides/clients/mcp#remote-iam) | | Remote-IAM event intake over webhooks | Done | `host.iam.webhook`: Better IAM's signed subtree webhooks into `iam_inbox`, for deployments that cannot read the IAM database. See [IAM events](/docs/guides/iam#iam-events-compute-reacts-to) | | Scheduler worker role | Done | `Reconciler.role` in core, the runtime's scheduler role with its own shard membership, and the pool-lease placer `capacity.placer` running in it | | Partitioned history | Done | `audit_events`, `inventory_snapshots` and `usage_windows` by UTC month (`core/0018_partitioned_history`); `retentionModule()` keeps months created ahead, prunes and drops emptied months; audit months are archived by hand. See [retention](/docs/operations/retention#history-partitions) | | Merkle audit checkpoints | Done | `@better-compute/audit`, with `better-compute audit verify` | | `bcd sim` | In progress | Written (aa7501c): `bcd sim --nodes N --seed S` runs N real node agents on fakehw in one process against a gateway, each with its own identity, journal and session; `tests/e2e/bcd.test.ts` joins five with one five-use token. Open: the nightly `bcd sim --nodes 10000` against real gateways and PostgreSQL through Toxiproxy. `better-compute sim` remains the FakeDaemon fleet simulator; see [bcd sim](/docs/daemon#bcd-sim) | | R1, if the IAM-lane benchmark demands it | Not started | Waits for the v0.1 IAM-lane gate | The v0.4 acceptance is set at 10,000 nodes and 100,000 GPUs on the reference PostgreSQL. What exists is simulation: | Criterion | State | | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Gateway-kill recovery under 60 s | Met in simulation: at 10k nodes, the 3,334 re-homed nodes are back at a p99 of 13.25 s simulated time, with no invariant violated | | 0 false unreachable marks | Met in simulation (`tests/sim/v04-scenarios.test.ts`) | | At least 200 placements/s | Indicative: with the pool-lease placer on PGlite, 1,092/s end to end at 1k nodes; at 10k nodes 464/s in the latest recorded run (`bench/placements/results`), and about 1,058/s in the measurement taken when the pool index started loading devices per node on demand (not yet a recorded result); see [scaling](/docs/operations/scaling#placements). The benchmark's sustained mode (`BENCH_PLACE_SUSTAIN_SECONDS=1800` for the acceptance, gate `BENCH_PLACE_GATE`, default 200, judged on the slowest 10 s window, I1 checked) measured 699/s, slowest window 639/s, over 30 s at 1k nodes on PGlite | | Cold start under 3 minutes | Met in simulation: `runColdStart` in `@better-compute/testing` runs the scenario on the real control plane and 3 real gateways in simulated time. The recorded 10k run (`bench/cold-start`) had every node connected, adopted and schedulable at 80 s and back 40 s after a power cycle, with 0 unreachable marks and 0 invariant violations; see [scaling](/docs/operations/scaling#cold-start). The CI-sized run (24 nodes, `tests/sim/cold-start.test.ts`) checks the same in every build. The wall-clock run needs the reference PostgreSQL | | Command p99 under 1.5 s with polling only, PostgreSQL CPU under 50% | Not measured: needs the reference PostgreSQL | | Suites green in both IAM modes | Done: `pnpm check` runs `pnpm test:iam-remote`, which reruns the IAM-dependent suites against IAM over HTTP, the authz matrix's IAM side included (every §9.2 action and resource pair decided through `bridge.remote`) | | Every §15 v0.4 scenario green on 3 consecutive nights | In progress: the scenarios run in the simulator, and the `nightly` workflow runs `sim:nightly`, the IAM-guarded suite and the Postgres suite every night and counts consecutive green nights; this page records no nightly history | ## v0.5: GPU depth and ecosystem [#v05-gpu-depth-and-ecosystem] Most of v0.5 landed early, on the control plane. Sandboxes joined the phase on 2026-09-24 (D24): all 15 methods are served, with templates, TTL expiry, warm pools, snapshots and exec, and the exit test runs on emulated hosts. bcd enforces the template `network`, and a host build runs sandboxes on `libvirt-kvm` (362317c) or `cloud-hypervisor` (2598f0a), both tested against fake tools only: no sandbox has booted on a real host yet. | Deliverable | State | Notes | | ------------------------------------------------------------------------------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | MIG, time-slice and MPS atoms | Done | See [fractional GPUs](/docs/guides/capacity/fractional-gpus); applying a MIG layout on a GPU is bcd's | | Calendar reservations and BookingRepair | Done | See [gang and calendar](/docs/guides/capacity/gang-and-calendar) | | Gang scheduling | Done | | | Spot | Done | See [spot and preemption](/docs/guides/capacity/spot-and-preemption) | | `dedicated-root` metal and `binding: 'node'` | Done | See [dedicated root](/docs/guides/bare-metal/dedicated-root) | | TPM attestor | Done in code, software TPM only | The verifier and a software TPM producer are done. `bcd join --attestor tpm` joins with the host's TPM (`/dev/tpmrm0`) and no token (8545688), and `daemon install-script --attestor tpm` writes a script for it (a006d9f). A fixture contract checks bcd's evidence and answer against `tpmAttestor` in both directions, and `tests/e2e/bcd-tpm.test.ts` joins a fakehw bcd with its software TPM through the real JoinService, reclaiming the same node after a reinstall (db9bd46); since 8a9af54 with an EK certificate from a test manufacturer root the attestor verifies (no `allowUncertifiedEk`) and the node key in that TPM. bcd keeps its node key in the host's TPM when it can open one (965c14d, 50e18f9, 0ba2a79). Pools can require it with `policy.joinAttestors` (1516728). Since cf24876 a `tpm` join also proves the node key: bcd makes it in the attestor's TPM, the AK certifies it over `sha256(csr)`, and `tpmAttestor` refuses a join without that proof by default (`nodeKey: 'certified'`), recording `nodeKey: 'tpm'` in `node.joined` and the join's audit row. Since d94f1f3 the proof holds through renewal: the certificate names the AK (`urn:bc:tpm-ak::` beside the SPIFFE ID), and every renewal must carry that AK's certification of the next key (`RenewRequest.key_attestation`), refused otherwise with no downgrade; bcd makes the next key in the TPM or fails the renewal loudly, and the fakehw e2e renews and keeps the statement. Open: a join, a renewal and a TPM node key on a hardware TPM with a real manufacturer's EK certificate. See [bcd with its TPM](/docs/guides/nodes/attestation#bcd-with-its-tpm) | | Optional workload identity | Done on the control plane | Workload OIDC tokens and the IMDS token exchange in pki, `WorkloadTokenRequest` over the session, the workload broker, and the issuer's discovery document and JWKS in `serve`. bcd sends the request from its own metadata service (`[imds] listen`), which serves the instances of any runtime it runs; end to end it has run only with fakehw builds' file-backed runtime. See [workload identity](/docs/guides/vps/instances#workload-identity) | | bcx and `driver-sdk` | In progress | The TypeScript SDK runs drivers out of process with mutual TLS. bcd hosts sha256-pinned plugins from `[plugins] dir` and runs `driver.invoke` in them (`daemon/internal/bcx`, with `bcx.Serve` as the start of a Go plugin SDK), tested by Go unit tests only. Plugins run sandboxed on Linux (d132fef, 3872c6b): executed from the file that was hashed, each under its own uid (root only when named in `[plugins] privileged`), with `no_new_privs`, a seccomp filter on every thread and optional cgroup limits, and restarted when they exit (9006eee: back-off from 1 s to 1 min, re-hashed and sandboxed again). The first Go plugin in the repository is `bcx-ipmi`, the IPMI BMC driver a site agent hosts (51806ab), shipped as its own release archive (348ff56). Open: an end-to-end run of bcd with a plugin, health checks, and the control plane merging advertised drivers into its registry. See [drivers](/docs/extending/drivers#driver-plugins-in-bcd) | | `provider-proxmox` | Done | | | console-kit and `apps/console` | Done | Including an instance's serial console page (191898b) beside the exec terminal | | Walkthrough 4, a Kata runtime in Go | In progress | bcd serves runtimes from bcx plugins: a hosted driver of kind `runtime` becomes an instances runtime (`list`, `apply`, `remove` over `Invoke`; no snapshots), covered by Go unit tests (494de69). No Kata plugin is written. See [walkthrough 4](/docs/extending/walkthroughs#walkthrough-4-a-kata-runtime-over-bcx) | | Sandboxes (PLAN.md §3.7): templates, TTL, warm pools, snapshots, exec over SSH certificates | Done on the control plane: 15 of 15 methods served | Added to v0.5 by D24. Served: `sandboxTemplates` create, list, get, update and delete (with `env` and `network`), `sandboxes` create (with `fromSnapshot`), list, get, extend, snapshot, exec and delete, and `sandboxSnapshots` list, get and delete, with migrations `vps/0004` to `vps/0012`, the `vps.sandboxes` reconciler (TTL expiry, `sandbox.*` events, warm pools, snapshot settling and purging), the `vps.sandbox.warm` meter, the instance guard and vps's tenant teardown. The CLI's `sandbox` and `sandbox-template` commands, eight MCP tools and the console's Sandboxes page use them. bcd enforces `network` (6db49c8) and runs sandboxes on `libvirt-kvm` (362317c) and `cloud-hypervisor` (2598f0a). Open: a run of either on a real host (both are tested against fake tools only). See [sandboxes](#sandboxes) | | Exit | State | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | The MIG torture test and the real-hardware nightly | The torture test runs on a simulated MIG pool; nothing has run on hardware (D15) | | A `dedicated-root` lease and release yield a new identity generation | Done | | A third-party TypeScript driver runs on a node via bcx | Done with a FakeDaemon as the node (`tests/e2e/bcx.test.ts`): a module profile's `power.*` steps go out as `driver.invoke` commands through the real engine, queue and gateway, and run in a pinned bcx plugin process. bcd now hosts bcx plugins and runs `driver.invoke`, but no test runs a TypeScript plugin under bcd yet | | IMDS cross-instance access fails | Done with FakeDaemon hosts (`tests/e2e/imds.test.ts`, `tests/pki-workload.test.ts`): one instance's session token fails for another and on another host, a host gets nothing for an instance placed elsewhere, forwarded requests and replays are refused | | `tests/e2e/sandboxes.test.ts`: 500 sandboxes expire on emulated hosts with the v0.3 zero-leak diff; a warm claim reaches `ready` without a placement; a snapshot round trip restores memory state; another tenant's snapshot is refused; instance methods on a sandbox's instance are refused | Done with FakeDaemon hosts (a8f3dfc): `tests/e2e/sandboxes.test.ts` runs the real server and gateway with three FakeDaemon VPS hosts (fake `cloud-hypervisor`, which pauses). It proves that a warm claim reaches `ready` without a placement (the paused instance resumes on the same node and allocation, booted once) and the pool refills to `warm.min`; that 500 sandboxes (`E2E_SANDBOXES`) with a 60 s TTL expire on their own, each publishing `sandbox.expired` then `sandbox.deleted` once, with the zero-leak diff clean (domains, overlays, taps, IP leases, bookings, scalar claims) and every readiness and expiry counted in the metrics; that guest memory captured on one sandbox comes back in one created `fromSnapshot`, and the stored object is deleted with its sandbox; and that deleting the warm template drains its pool until the hosts are empty. Since bfb1d45 it also asserts the last two bullets: `instances.stop`, `start` and `delete` on a sandbox's instance are refused `INVALID_STATE` naming the sandbox (`get` still works), and a user of another tenant, with its own template on the same pool, gets `NOT_FOUND` restoring from the snapshot and reading it (`sandboxSnapshots.get`); `tests/vps-sandboxes.test.ts` covers both against emulated host reports too. Nothing runs on a real host | ## Sandboxes [#sandboxes] Sandboxes are part of the product beside GPU and CPU capacity. Decision D24 (PLAN.md section 17) was taken on 2026-09-24: build a dedicated sandbox API, a `sandboxes` group on `vps()`. PLAN.md drops "a sandbox product" from the v1 non-goals, describes the flow in section 3.7, adds the `sandboxes` and `sandboxTemplates` rows to section 9.2, and makes it a v0.5 deliverable with its own exit test ([v0.5](#v05-gpu-depth-and-ecosystem)). What exists and what does not: | Item | State | Notes | | ---------------------------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The sandbox catalog | Done | `compute-sandbox` (parent: the pool; attributes `templateId`, `vcpu`, `memoryGiB`, `gpus`, `ttlSeconds`, `fromSnapshot`, `warm`), `compute-sandbox-template` and `compute-sandbox-snapshot` (attributes `sandboxId`, `templateId`, `includeMemory`; owned by the sandbox's owner, whoever took it), the `compute:sandboxes:*` actions, and 15 method rows at phase v0.5, all registered. See [actions and resources](/docs/guides/iam/actions-and-resources) | | Templates | Done | `sandboxTemplates` create, list, get, update and delete: a hosted pool, a flavor, an image and the pool's own runtime, checked once; user data; `defaultTtlSeconds` (1 h) and `maxTtlSeconds` (24 h); `warm: { min, max }`. Also in FleetConfig (`sandboxTemplates`). See [templates](/docs/guides/vps/sandboxes#templates) | | Sandboxes with a lifetime | Done | `sandboxes` create, list, get, extend and delete. One decision on `compute-sandbox/{draftId}`, the pool's budget check, then one hosted instance admitted, placed and charged like `instances.create`. The TTL starts at admission; `extend` stays within `maxTtlSeconds`; the `vps.sandboxes` reconciler expires sandboxes into an ordinary instance delete | | Warm pools | Done | `warm.min` instances per template, kept `paused` on runtimes that can pause (`libvirt-kvm`, `cloud-hypervisor`, `firecracker`) and `stopped` otherwise, at most 4 created per template per pass; claimed with `FOR UPDATE SKIP LOCKED` only after the sandbox's own allow, without a new placement, and only when made from the template's current shape; drained beyond `warm.max` and when stale; metered as `vps.sandbox.warm` until claimed | | Paused instances | Done | `desiredState: 'paused'` in the lifecycle, `InstanceObservedState.PAUSED` in host reports, `RuntimeTraits.pause` | | Instance guard and teardown | Done | `instances.start`, `stop`, `resize` and `delete` refuse a sandbox's or a warm instance with `INVALID_STATE`, and so does `tunnels.open` for every op (`reason: 'sandbox-managed'`); `instances.list` hides warm instances; `vps().tenantDeleted` deletes every instance and snapshot of a deleted tenant, sandboxes' and warm ones included | | Hosted runtimes on your own servers | In progress | bcd has the `instances` converger (fab94d4), `instance.snapshot` and `restore_from` (be1e7d9), serves runtimes from bcx plugins (494de69), and runs `libvirt-kvm` on a host build with `virsh` (362317c: overlays on verified base images, NoCloud seeds, VFIO GPUs, snapshots with memory, restores checked for runtime and CPU model) and `cloud-hypervisor` on one with the `cloud-hypervisor` binary (2598f0a, the same features). Both are tested only against fake tools and no Kata plugin exists, so no sandbox has booted on a real host yet | | Snapshots and restore | Done on the control plane | `sandboxes.snapshot` with `vps({ snapshots: SnapshotStore })`, the `instance.snapshot` command kind, the `sandbox_snapshots` table, `sandbox.snapshot.ready`, `.failed` and `.deleted`; `create({ fromSnapshot })` of the caller's own snapshot on the same runtime and, for memory, the same `cpu.model`; `sandboxSnapshots` list, get and delete, so a kept snapshot can be removed. bcd runs snapshots and restores on fakehw builds and, as a tar of the flattened disk, the seed and `virsh save`'s memory state, on `libvirt-kvm` (tested against a fake `virsh`), and with cloud-hypervisor's own memory snapshot on `cloud-hypervisor` (tested against a fake VMM API). See [snapshots](/docs/guides/vps/sandboxes#snapshots-and-restore) | | Exec | Done | `sandboxes.exec` (tunnels module): an `ssh` ticket and a certificate of at most 15 minutes, never past the sandbox's expiry, from the deployment's sandbox SSH CA (`pki add-sandbox-ca`, `tunnelsOptions`), which guests trust through vps's user data; the tunnel is re-decided on the sandbox every minute; with `certifyHostKey`, guests present host keys the same CA certified and the CLI pins them. Exec is the CLI's `better-compute sandbox exec`, which runs your OpenSSH client through the tunnel. No agent in the guest. Deferred: the SDK helpers PLAN.md section 3.7 describes (`sandbox.exec(cmd)`, `files.put`, `files.get`), because they need an SSH client in the SDK, a new dependency decision. See [running code inside](/docs/guides/vps/sandboxes#run-code-inside) | | Template `env` and `network`; the `vps.sandbox.warm` meter | Done | `env` from the tenant's secrets, sealed into `/etc/sandbox/env`; `network` carried as `InstanceSpec.network_policy` and enforced by bcd with host nftables rules (6db49c8; `private` and `none` need the pool's IP range and place only on hosts labelled `bc.dev/cap.instance.network-policy`, cfdb918); `vps.sandbox.warm` from placement to claim (`iam setup --meters` creates the meter). Template secrets must be the tenant's own, written with `secrets.put` and not fleet-scoped, and naming them also decides `compute:secrets:manage` | | Clients | Done | The CLI's `sandbox`, `sandbox-template` and `sandbox-volume` commands, eight MCP tools with gate rules (none takes a snapshot or runs code), the console's Sandboxes page, and FleetConfig `sandboxTemplates`. See [MCP](/docs/guides/clients/mcp#the-tools) | ## v1.0: Freeze [#v10-freeze] v1.0 freezes the API and the protocol, guarantees N-2 version skew, adds an external security review and SLSA level 3 builds, and publishes to npm with provenance from CI. Its acceptance: upgrading from v0.5 disturbs no running allocation, with no open critical or high findings. | Item | State | Notes | | -------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | API freeze | In progress | `scripts/api-surface.mjs`, run by `pnpm lint:repo`, fails on a breaking change to any served method or error code against `generated/api-surface.json`; additions are recorded with `pnpm api:surface`. See [the API surface freeze](/docs/operations/testing#the-api-surface-freeze) | | Protocol freeze | In progress | `agent/v1` and `common/v1` have been frozen since P0 (`buf breaking` in CI); the draft `site` and `driver` packages are not yet | | N-2 version skew | In progress | `createGateway({ protocol })` sets the range a gateway speaks. `tests/skew-matrix.test.ts` runs the matrix on real dev stacks: FakeDaemons built like the N, N-1 (v0.4) and N-2 (v0.3) lines (`daemonSkewProfiles` in `@better-compute/testing`) are welcomed, adopted and refuse the kinds they predate `UNSUPPORTED`, and a daemon below the gateway's minimum gets `upgrade_required`. No older real bcd build runs in it, and only protocol version 1 is defined. See [version skew](/docs/daemon/protocol#version-skew) | | SLSA level 3 builds | In progress | `release.yml` builds bcd with SBOMs, keyless cosign signatures and SLSA3 provenance; no release has been cut | | npm publishing with provenance | In progress | 0.1.0 and 0.1.1 are on npm (`latest`: 0.1.1), published from a verified clean checkout, with Better IAM `>=0.2.0 <0.4.0` from npm. Provenance attestations come once `ci.yml`'s publish job publishes from the GitHub repository | | External security review | Not started | A process item | | Destructive-batch quorum (PLAN.md §13) | In progress | Fleet-wide destructive batches need k-of-n operator approval, checked on the machine. Done: the protocol seams (`CommandBody.batch_id`, `BatchApproval`, `BatchProof`, the trust bundle's `approver_keys`, `approver_threshold`, `quorum_tiers` and `unbatched_destructive_per_hour`, `BATCH_UNAPPROVED` and `UNBATCHED_LIMIT`; 525c199, 4f11548, 081928a), the pki primitives (approver keys, the `(subject, kind)` scope tree, approvals, `verifyBatchCommand`; 36387c8, 2d2fe6f), `pki.approvers` in the server's signed trust bundle (b816600), the server's `quorum` module with `commandBatches` get, list, scope, approve and cancel (ce067e0), the command queue's `createBatch` and `enqueue({ batchId })`, which holds batched commands until approval (6067791), metal batch scopes and steps that declare what they send (add7e0e), `better-compute batch keygen` and `batch sign` (0210671), and enforcement in the FakeDaemon (4b48f08) and bcd (e7ddb38). Since then: quorum review #6 (payload-bound leaves, site-qualified BMC subjects, the trust domain in approvals, each leaf run once; 77cdf4f, d566cbe, 8b0f50c), the gateway attaching approvals and proofs to batched commands (40be696), `batch list`, `show`, `approve` and `cancel` with the approver-side checks (b3fd712), approver-set changes signed by the current approvers (`batch sign-approvers`, `pki.approvers.setSignatures`; 6a7f1b7, edbd63f), a trust bundle version that follows its content (9c40cd5), and enqueue refusing an unbatched destructive command up front (`batch-required`, 508728e). Then the review 7 fixes: every command signed once signed intent is in force and `site.httpProxy` destructive (0d42de8, 12ccfa1), approver-set signatures bound to the trust domain (4835f4f), and leaves that carry their payload for approvers (`quorum/0007_leaf_payload`, 2ca0ba4), then approver sets with their own epoch (c6214dc), a first set pinned out of band (`--approver-set-sha256`, 9b7a199), and leaves limited to the tenant's own machines with payloads of their kind's type (fc2df7d); and the producers: metal's per-workflow batches and `metal.bulk` (f66e236), `metal.power` (478025a), the health reconciler's remediations (e594a4f) and the MCP server's `propose_command_batch` (171ad2f). Open: the batches built-in workflows create leave their leaves' payloads unbound, since the bytes are only known later (a BMC credential rotated mid-workflow, user data rendered by an earlier step); PLAN.md §13 (a1bd0aa) names binding a leaf to the operation instead of its bytes, for example `payloadMatch { operation, args }` checked by the agent after decoding, as a post-v1.0 option. Since then, review 10: one epoch-adoption rule for every change, the held set reported in `Hello` and `StatusReport`, recorded by the gateway (fa39201, `core/0026`) and checked by doctor and `pki trust-version`, a node holding no keys counting as lagging (5fe7cbf, e171f0d, 6612496, 08eb62e), the pin enforced by bcd, operations grouped by payload digest in the approver CLI (b118616), and `batch create` and `batch members`. See [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10) | | Upgrading from v0.5 disturbs no running allocation | Done in simulation | `upgradeInPlace(rig, { migrations, rollDaemons })` in `@better-compute/testing` runs expand migrations, workers and api, each gateway killed and revived, then the daemons, and reports every disturbed allocation. `tests/sim/upgrade.test.ts` (6 nodes, 3 gateways, 6 active reservations, a signed rollout to 0.6.0) finds none, bills the hour exactly, and keeps I1, I2, I5 and I8; it also runs two instances and a sandbox on two VPS hosts (`ControlPlaneSim` `extraPools`), and every VM stays running on its host with the same boot count while the sandbox keeps its expiry. A second scenario (80195df) starts from a v0.5-era database without the quorum tables, upgrades, then turns the quorum on: no allocation moves, setting approvers moves no spec generation, the trust bundle's version bumps exactly once, every daemon takes the set, a `node.reboot` queued before the switch is sent but rejected `UNBATCHED_LIMIT`, and a new unbatched one is refused `batch-required` (see [turning the quorum on](/docs/operations/pki#turning-the-quorum-on)); FakeDaemons only | ## CPU, memory and Kubernetes (v1.1) [#cpu-memory-and-kubernetes-v11] PLAN.md §3.8 (a760805) designs the complete node daemon for v1.1: compute books a host's CPUs, memory and huge pages with its GPUs, the node enforces exactly that, bcd applies the host's configuration, and a reservation can hand its GPUs to a Kubernetes cluster. The principle stays P1: compute decides who gets what, and the node enforces exactly that and reports it. Most of it is now in the code, on both sides. Every node-side piece is tested against fakes only (a fake host, a fake root, a fake kubelet, fake VMMs); none has run on a real machine or cluster. | Item | State | Notes | | --------------------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Host configuration applied by bcd | Done in code, mostly fakes | The host converger applies twelve units in order, `kernel-modules`, `sysctl`, `hugepages`, `cpu`, `iommu`, `nvidia-driver`, `fabric-manager`, `gpu-settings`, `mig-layout`, `gpu-sharing`, `container-toolkit` and `k8s-join` (6c2343f, 7579184, 1cad924, 3fc3628, 3fd1c41), a failing one never holding back the rest (c21ef84), drift repaired by a pass every 10 minutes (3c0fe02), and settings that wait for a reboot reported as the informational node condition `RebootRequired` (c21ef84, 0aea5c0); under `bcd helper` each unit is applied as root only from a signed spec (03c023d, 2d18327), and root writes only under root's directories, with its records in `/var/lib/bcd-hostconfig` (f8e13ef, 02485b3, 1bb78d6). Provision renders every unit from profiles and from pool policy (8ce9140, 1f0f683, 7db110a, 09669d3, e4b0c01), and the pools API refuses a host policy nodes cannot take (37dcd02); `[host] units = false` for hosts configured by something else (a193249), which `bcd helper` also honours from root's own root-only `config.toml` (3fa9021). `TestRealKernel` runs sysctl, THP and 2 MiB pages on a real kernel (60143bf); `tests/e2e/bcd-host.test.ts` runs a pool's units through the control plane (91f5e81). See [host configuration](/docs/daemon/host-config). Open: no driver install, MIG repartition, boot argument or cluster join has run on a real machine | | Host resources on every host | Done | `capacity.hosts` writes `vcpu`, `memoryGiB`, `hugepages2Mi` and `hugepages1Gi` for every host, under `policy.hostReserve` and the overcommit ratio (8a7f8eb); vps's `memoryGiB` leaves out the huge page pools (9d55b36). Inventory reports per-NUMA huge pages and isolated CPUs, with the labels `cpu.count`, `cpu.sockets`, `cpu.isolated`, `numa.nodes`, `memory.totalGiB`, `hugepages.2Mi` and `hugepages.1Gi` (badb8c2). Inventory does not report available memory | | Reservations book CPU and memory | Done | `reservations.create` takes `resources { vcpu, memoryGiB, hugepages1Gi }`, booked on the devices' node in the same statement; without it a request books `policy.perGpu` or the pool's smallest per-GPU share, for whole GPUs and, since 004579e, for MIG instances and slots as fractions of a GPU (a21c1c1). The share is the floor, and anything above it is charged to `vcpu` and `memoryGiB` quota (41826a2, 004579e). See [host resources](/docs/guides/capacity/reservations#host-resources) | | Enforcement on the node | Done in code, fakes only | Device assignments carry the booked resources as limits, with a cpuset on the devices' NUMA nodes that leaves out the host's reserved CPUs, the lowest non-isolated, `policy.hostReserve.vcpu` (c5a419c, b6b804e); bcd admits a hold against the host's memory beside its VMs, failing it `resources-exhausted` for re-placement (31555bf), creates `bcd-alloc-.slice` before the CDI spec names it, restores it after a reboot and stops it once empty, retrying every pass (1d03c28, 260e33c, e01d7ff), and reports it in `AllocationBound.cgroup_parent`, which `reservations.get` shows as `binding.cgroupParent` (6aa412b, c81f230, 19dbf60). VMs get `InstanceSpec.placement` on both runtimes (a184ec2), pinned guests take isolated CPUs first (b6b804e), guests of memory-overcommitted pools get a balloon (073324a), and bcd refuses an instance start the host cannot back (`ResourcesExhausted`, 0dd122a), which vps records as `HostResourcesExhausted` and answers by placing a not-yet-started instance on another host after a minute, at most three times (564f543). `bcd doctor` checks the cgroup controllers, huge page pools, `vfio-pci` and systemd (ab26e87). Reservations with host resources, and `kubernetes` bindings, place only on a bcd that advertises `devices.limits` or `binding.kubernetes` (409f783). Open: the workload manager must place workloads in the slice | | Holds of MIG instances and slots | Done in code, fakes only | bcd holds `gpu.mig` and `gpu.slot` atoms with parent-aware conflicts and no hold beside a VM's GPU (46a61db), and `gpu.reset` and `diag.dcgm` refuse GPUs those holds or VMs use (539c4ce) | | Kubernetes | Done in code, fakes only | Pools name the cluster in `policy.kubernetes`; the `k8s-join` unit installs, joins and keeps the Node's labels and taints, with the bootstrap token released by binding rule 3 (88ecb07, 7579184, e93bf9c); `binding: 'kubernetes'` (a21c1c1) binds like a hold (d02ba4f), and bcd's device plugin advertises exactly those devices as `bcd.dev/gpu`, `.mig` and `.slot` and deletes pods that keep released ones. See [Kubernetes workers](/docs/guides/capacity/kubernetes). The DRA driver `gpu.bcd.dev` offers the same devices through one ResourceSlice per node instead, chosen by `gpuApi` (7d24836, fa8611e, e4b0c01), and neither API offers a GPU that went bad (6f5a99f). bcd also runs as a DaemonSet on an existing cluster (8df8c70, 2c9f953), with hosted join tokens that land its nodes `tenant-controlled` unless a platform-owned pool is platform-operated (a27156a, 259d82c), in hosted mode with the device plugins or DRA and its `bcd.dev/*` labels but no join (1e5f96b), and `kubernetes manifest` renders it with operator-given pins (a2eb28f, d2cd813); see [bcd on Kubernetes](/docs/daemon/kubernetes). Kubernetes bindings get no slice: the kubelet enforces pods' limits, and their booked limits count in host admission (e01d7ff). The pools API checks `policy.kubernetes` (37dcd02). Only a pool's owner books `binding: 'kubernetes'` (3825ec7), and `ForeignGpuExposure`, `PodResourcesUnavailable` and `KubernetesReleaseStuck` are NoSchedule node conditions, the first also degrading the node's reservations (ebd9806, 4570dfe). Security review 2 (8bcbe92, f88bcdb): a released GPU stays in use until no pod, prepared DRA claim or pod process holds it, failing closed on any node that ever was a worker, and `ForeignGpuExposure` also follows the toolkit the container runtime runs, other GPU DRA drivers and, hosted, an unreadable runtime configuration. The DaemonSet mounts only the kubelet sockets and credentials bcd uses, never `/var/lib/kubelet` or `/etc/kubernetes` (27f2c1b, a5eee84), and the runtime and toolkit configuration read-only (d11810c, 3775f46). `daemon/Containerfile` bundles a pinned, checksummed `kubectl` v1.34.1, which bcd runs as the kubelet (9bdd43a, e7d0765); CI builds the image and server-side dry-runs the manifests on kind (ac1ca4b). Open: no real cluster has run it (the CI job is a dry run), and pod requests are not derived from the reservation | | Sandboxes and VM hosts | Done in code, fakes only | Flavors take `placement { numaAffine, pinned, hugepages }`, which sandbox templates inherit; vps chooses the NUMA node and pinned CPUs and marks a host that cannot honour them `PlacementDegraded` (9d55b36). Sandboxes, placed flavors and volume-bearing instances land only on a bcd that advertises the enforcing capability, `instance.ssh-host-only`, `instance.placement` or `instance.volumes` (82f4f51). Memory snapshots of GPU sandboxes and of sandboxes with a read-write volume are refused up front (2a09202). See [flavors](/docs/guides/vps/flavors#placement) | | Agent memory: sandbox volumes | Done in code, fakes only | `sandboxVolumes.create`, `list`, `get`, `resize` and `delete`: a persistent disk booked as `diskGiB` on one host (core/0028, vps/0020), attached by serial to the sandboxes of templates that name it, one writer or many readers, pinned to its host, metered as `vps.volume.gib.hours` (2a09202, 8cbece0); bcd keeps each as a sparse raw file, grows it live, keeps one writer through handoffs and destroys it when unlisted (550da91, b5415a2, 448dc5f, 39458f9, 408e002). bcd never promises more than its data directory's filesystem holds and reports the space in `StatusReport.volume_storage` (851016e, 37d6d6d, b8e7271), which vps books volumes against, never overcommitted (131f5ee). The CLI's `sandbox-volume` commands and `sandbox-template --volumes` (f181f43), four MCP tools (e6759df), FleetConfig templates (94b8b70) and the console's Volumes page (50add82) expose them. A template mounts a volume only with a decision on that volume (77bf9f5), lists are per caller by default, and create and resize are decided with the size, so a policy can bound it (6cb30e9). See [volumes](/docs/guides/vps/sandboxes#volumes). The console's Sandboxes page attaches and detaches a template's volumes (dd5555b). At scale (38405a0), `tests/e2e/volumes.test.ts` churns 240 sandboxes over 12 volumes on three FakeDaemon hosts through the real server and gateway, host reboots included, with no leak, no second writer, no overbooked store and no lost data; see [testing](/docs/operations/testing#volumes-at-scale) | The work's owners are in `.claude/coordination.md` ("Round 5"). ## What unblocks what [#what-unblocks-what] | Unblocked by | Moves | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Writing more of bcd (Go is installed now; P0's code generation, W1's node agent, `bcd site`, `bcd ramdisk`, the `instances` converger, `libvirt-kvm`, `cloud-hypervisor`, self-update, the WebSocket fallback, `bcd helper`, bcx plugin supervision and the `bcx-ipmi` BMC driver are done) | Vendor BMC drivers, bcx plugins beside an unprivileged node, NVML events and a go-nvml backend; an end-to-end run of the site agent and ramdisk, and of bcd with a bcx plugin | | A bcd release | Installing real nodes with the install script | | A PostgreSQL 17 server | The concurrent capacity and torture suites, the IAM-lane gate, the 30-minute gateway gate, every v0.4 number | | The D15 hardware | CDI and a bound GPU hold on an NVIDIA host (real inventory and live GPU metrics have run, on one RTX 5090 under WSL 2), emulation CI on a KVM runner, the first `libvirt-kvm` run on real KVM, the MIG and real-hardware nightlies | For what changed recently, see the [changelog](/docs/reference/changelog).