Testing

@better-compute/testing (also better-compute/testing) is the kit the repository's own tests run on, and the one you use to test modules, plugins and policies. It gives you a migrated control plane on PGlite with a clock you can move, a scripted IAM, the daemon side of the agent protocol (FakeDaemon), in-memory links with fault injection, a deterministic fleet simulator with invariant checkers, and real-time harnesses for latency measurements. No database server is needed.

The kit needs the optional peer @electric-sql/pglite as a devDependency; without it computeFixture() fails with MISSING_DEPENDENCY. The embedded Better IAM of @better-compute/iam/testing needs the optional @better-iam/adapter-sqlite peer.

Test a module with computeFixture

computeFixture(options?) builds betterCompute() on an in-memory PGlite, runs compute.initialize() (the registry check, the base schema and every module's migrations) and hands you the pieces. It does not start the workers: your test drives reconcilers one pass at a time with compute.jobs[name].tick().

import { afterEach, expect, it } from 'vitest';
import { closeFixtures, computeFixture } from 'better-compute/testing';
import { myModule } from '../src/module';

afterEach(closeFixtures);

it('asks IAM before listing, and reconciles', async () => {
  const { request, iam, tenants, compute, advance } = await computeFixture({ modules: [myModule()] });
  iam.allow({ token: tenants.acme.token, action: 'compute:nodes:read' });

  const answer = await request('nodes', 'list', { tenantId: tenants.acme.id }, { token: tenants.acme.token });
  expect(answer.ok).toBe(true);
  expect(iam.calls.length).toBeGreaterThan(0); // authenticate, then the tenant decision

  await advance(60_000); // compute's Clock and compute_now() move together
  await compute.jobs['my.reconciler']!.tick(); // one deterministic pass
});

Options: clock ('sim', the default, or 'system'), start (default 2026-01-01T00:00:00Z), iam (extra fakeIam() options), modules, plugins and log. request() answers { ok, status, headers, requestId } plus data or error.

The sim clock. Under clock: 'sim', compute's Clock and SQL's compute_now() read the same time (principle P9). advance(ms) and clock.set(ms) move the compute_sim_clock table first and the TypeScript clock second, so leases, not_before, column defaults and admission contexts all move together. Time never moves backwards. Under 'system', advance rejects with INVALID_STATE.

fakeIam(options?) is a scripted IamPort. Bearer tokens map to callers (an unknown token is UNAUTHENTICATED). A decision is allowed only when an allow rule matches and no deny rule does; a rule matches on any of token, tenantId, action, type and id, and {} matches everything. Every call is recorded in calls; count(method) counts them. billingCheck allows, recordUsage records, featureEnabled is false and requestConfirmation answers 'not-held-back'.

Plugins, drivers and providers have conformance suites: functions that return ConformanceCase[] ({ name, run(subject) }, from @better-compute/core), such as providerConformanceCases() from @better-compute/vps and storeConformanceCases() from @better-compute/postgres. Run them from any runner:

for (const test of providerConformanceCases()) it(test.name, async () => test.run(await createSubject()));

Real Better IAM in tests

computeFixture() still uses the scripted fakeIam(). When a test must exercise real policies, use createEmbeddedTestIam(options?) from better-compute/iam/testing: a real better-iam on in-memory SQLite, carrying compute's plugin, resolver and inbox, bootstrapped with a root administrator (TOTP MFA) and two organizations, "Acme" and "Other". Pass its port to betterCompute({ iam }) and call bind(sql) once the compute store is migrated.

• Each organization (acme, other, or a new one from tenant(name)) offers member(name), signIn(name), policy(document), grant(identityId, ...documents) and share(type, id, relation, subject).
• Options: now, sql, sessionJwts, runIamJobs, inbox ('store' or 'memory'), iamOptions, and the catalog's modules, plugins, extra and typePrefix.
• mode: 'embedded' (the default, or BETTER_COMPUTE_IAM_MODE) or 'remote', which hands compute bridge.remote() over the same better-iam's HTTP handler, so the suite runs as it would against IAM in another process.
• The secret is fixed and public. Never use it outside tests.

pnpm test:iam-remote reruns the IAM-dependent suites in remote mode (vitest.iam-remote.config.ts, 13 files: the example policies, the resolver timeout, confirmation, the testing IAM, streams, MCP confirmation, watch, the e2e access and spine tests, the authz matrix, the budget check, webhook intake and serve's compose file), and pnpm check runs it after pnpm test, so every change stays green in both IAM modes (a v0.4 exit). The server-*, capacity-* and vps-* suites use the scripted fakeIam() port, so an IAM mode does not apply to them. pnpm check:head runs scripts/head-check.mjs, which checks the committed tree in a separate worktree.

scripts/head-check.mjs exists because several sessions edit the shared checkout at once, so a full pnpm check there mostly reports other people's work in progress. It keeps a git worktree under the temporary directory (bc-head-check), moves it to the commit (--ref, default this checkout's HEAD), links every node_modules of the checkout into it without installing anything, builds with tsc -b unless --no-build, and runs vitest run with the filters you give. Each concurrent run takes its own tree (bc-head-check, then -2 up to -8) through a <tree>.lock file holding its pid, taken over when that pid is dead and released on exit, so parallel runs never move each other's tree mid-run. --run <repo-relative .mjs> [args…] runs that script in the held tree after the build instead of vitest, and exits with its status; the docs reference is regenerated from committed code this way:

node scripts/head-check.mjs tests/vps-sandboxes.test.ts
node scripts/head-check.mjs --run apps/docs/scripts/sync-from-head.mjs "$PWD"

The authz matrix has two halves. tests/generated/authz-matrix.test.ts, generated by scripts/authz-matrix.mjs, covers the envelope side, one test per registered method. tests/iam-bridge-matrix.test.ts covers the IAM side: every distinct (action, resource type) pair of the PLAN.md §9.2 table, alternatives included, is decided through compute's own IamPort against a real better-iam carrying the bridge's plugin and resolver. Each pair is denied without a grant, allowed with one, and never allowed on a resource type it was not granted on. Under pnpm test the port is bridge.embedded; under pnpm test:iam-remote it is bridge.remote over the same IAM's HTTP API, and the test asserts that the decisions went over HTTP.

spyIamPort(port) from better-compute/iam wraps any IamPort and records every call. Code paths tag themselves with spy.path(name, fn); each call is attributed to the innermost path across awaits, so a test can assert that fleet paths never reach IAM. That is how the v0.1 security negative is tested: tests/sim/iam-negative.test.ts runs ten simulated minutes of heartbeats, inventory, commands and a reservation, and asserts that every IAM call came from an API admission. See policies.

The FakeDaemon

FakeDaemon is the daemon side of the protocol in TypeScript, rule for rule with the Go packages in daemon/internal/{journal,dispatch,session,converge,agent,fakehw}. The real bcd has its own tests (see the Go daemon); the FakeDaemon is what runs where features bcd lacks, or many daemons on simulated time, are needed. bcd sim runs many real agents too, but in real time against a real gateway. See the daemon.

• Journal. accepted before any side effect; running{boot_id} right before a self-terminating one; done results kept 7 days; unacked events capped at 64 MiB and 10,000. A crash keeps the journal, a reboot also changes boot_id and clears /run, and loseJournal() starts a new journal_id.
• Duplicate delivery. Done: resend the stored result. Running: re-attach. Accepted but not running after a crash: re-execute if retry_safe, else FAILED{interrupted}. Self-terminating across a reboot: SUCCEEDED{rebooted}.
• Acceptance order. Another node's command or a disallowed kind is UNSUPPORTED; a min_generation above the applied spec defers with a SpecRequest; past not_after is EXPIRED; no free lane credit is BUSY.
• Session. Hello from the journal, Welcome, heartbeats, PING liveness, full-jitter reconnect, Drain, RevocationNotice, Cancel; a Welcome more than 5 minutes before the journal's server time mark is refused. The mark (JournalMeta.serverTimeMs) is set on every Welcome and heartbeat and only moves forward, so a gateway cannot rewind the unsigned Welcome.server_time to make a recorded signed command or an old worker certificate valid again; the session drops with server time went backwards on every redial. It is not clamped to the local clock, since sim and manual server clocks sit far from the host's.
• Desired state. The devices converger writes the CDI spec and env file in an in-memory file map, reports AllocationBound, AllocationReleased and AllocationFailed, and never drops a hold on its own.
• Kinds. diag.nvml, inventory.refresh, logs.collect and node.reboot, plus the opt-in fakeGpuReset (not retry-safe). knobs make them fail: diag.nvmlInitFails, diag.failGpus, diag.error, logs.uploadFails, cdiWriteFailures.
• Beyond v0.1. With intent, it verifies signed commands (BAD_SIGNATURE; with command-signing roots every command must carry a valid signature, whatever its kind, 12ccfa1, which tests/testing-daemon-intent.test.ts checks with an unsigned non-destructive command) and signed NodeSpecs: intent.signedSpecs is 'always' (the default: with command-signing roots every spec must be signed, since an unsigned spec with a high generation would freeze all later ones) or 'access' (only a spec that changes access must be signed, for a control plane that signs nothing else), and refused specs are listed in specRejections. intent.quorum (a FakeQuorumTrust: approver keys, threshold, tiers, unbatchedPerHour), or the quorum of a newer signed trust bundle, enforces the v1.0 destructive-batch quorum: destructive commands are rejected BATCH_UNAPPROVED or UNBATCHED_LIMIT, and a newer trust bundle that changes the quorum without the current approvers' signatures for the FakeDaemon's own trust domain is ignored (logged trust.ignored; 6a7f1b7, 4835f4f) (see the quorum). A refused session whose close carries retryAfterMs (the gateway's retry pushback) reconnects after exactly that long. setPressure(pressure) puts a HostPressure into every status report. setDiscovered(machines) (537e672) plays a site agent's PXE discovery list: each { mac, productUuid?, arch?, firstSeen?, lastSeen?, count?, relay? } (times in epoch milliseconds, default now; MAC and UUID lower-cased) goes into StatusReport.discovered of every report from then on, the current report is re-sent, and undefined stops it. With tunnels, it answers OpenTunnel; with nodeKey, attested results and allocation reports carry node_sig. With instances (a FakeHypervisor or its options) it plays a VPS host, advertising instance.network-policy in Hello and converging domains, overlay disks, NoCloud seeds and taps and fetching user data through the secret broker; as such a host, handleImds(request, instanceId) answers a running instance's metadata request (v0.5, a FakeImds), minting its workload token through a WorkloadTokenRequest on the session. With selfUpdate: { badVersions?, swapMs? } it converges NodeSpec.daemon by swapping and restarting, and a bad build falls back to its previous slot and reports the section failed. fakeDriverInvoke({ drivers }) adds the driver.invoke kind, running in-process or bcx drivers. createWsDialer() dials the WebSocket fallback.

Fake hardware. fakeHardware({ seed, variant, missingGpu? }) gives hgx-h100 (8x H100 SXM5 80GB on NVSwitch, field for field the Go fakehw: the same seed gives the same section hashes), hgx-h100-gpu-missing (7 visible, so GpuCountMismatch), l40s (4x L40S 48GB on PCIe) and cpu (no gpus section). encodeInventorySections() reproduces the Go golden bytes.

Links. createAgentLink() builds a daemon end and a gateway end whose frames cross as protobuf bytes, with faults per direction from the link's seeded RNG: latencyMs (default 1), jitterMs, drop with dropMode (silent, or reset, which breaks the stream as gRPC would), duplicate, and reorder within reorderWindowMs (default 50). Controls: stall, partition and heal, halfOpen, reset, dropNext, duplicateNext and intercept. createDaemonHarness() wires one daemon to scripted servers on a sim scheduler.

Host volumes (38405a0). A FakeHypervisor keeps sandbox volumes the way bcd does, so a FakeDaemon VPS host can stand in for bcd in volume tests:

• InstancesSection.volumes are created, or grown in place, only when the store has room (available minus reserved minus outstanding); otherwise the volume keeps its size and the instances naming it fail. A volume is never shrunk, and one no longer listed is destroyed once nothing has it open. The files would be volumePath(id), /var/lib/bcd/volumes/<id>.raw.
• An instance's InstanceSpec.volumes are opened at boot, and what counts is what running and paused domains really have open, not their specs: one writer, and read-only volumes shared only with other readers. A pass applies stops before starts, so a volume handed from one instance to another is closed before it is opened again.
• Volumes survive powerLoss; wipe() clears them. The hooks writeVolume(instanceId, volumeId, content, bytes?) (a content marker, 1 MiB written by default), volumeContent(volumeId) and openBy(volumeId) let a test check what a guest would find and who has a volume open.
• With volumeStore: { totalBytes, reservedBytes? } in its options, volumeStorage() feeds StatusReport.volume_storage, which the FakeDaemon sends in every status report: available is total minus what guests wrote, reserved defaults to bcd's 5% of the total with a 2 GiB floor, committed is the sum of the sizes, and outstanding the sum of size minus written.

Host admission (6922553). A start the FakeHypervisor has no free vCPU or memory for is no longer an instance failure. Like bcd's host admission (agent/admission.go), the domain stays stopped with a message starting ResourcesExhausted: (the prefix is exported as RESOURCES_EXHAUSTED), and the section is PROGRESSING with reason Retrying and the message ResourcesExhausted: this host cannot start <ids> now, unless another instance failed in the same pass (then it is FAILED, InstanceFailed). The next pass tries the start again. That is the report vps turns into HostResourcesExhausted and a re-placement (see hosts that cannot start an instance).

tests/testing-daemon-instances.test.ts has the unit test.

A VPS host under bcd helper (1c9684a). The FakeDaemon plays the control-plane side of a bcd that runs unprivileged under bcd helper, with cloud-hypervisor through root's ch.* operations:

• Runtimes in Hello. A FakeDaemon with instances lists each of its FakeHypervisor's runtimes as a driver manifest of kind runtime (runsOn: NODE, its daemon version), as bcd does for its built-ins, so the gateway labels the node bc.dev/driver.<runtime>. An explicit drivers option replaces the list.
• FakeHypervisorOptions.helper: { instanceBridges } plays [helper] instance_bridges: a NIC on any other bridge (an empty list refuses every NIC) fails its instance in the helper's words, libvirt-kvm before the domain exists (domain: bridge "…" is not an instance network) and cloud-hypervisor when it starts (ch.taps: "…" is not an instance bridge of this host ([helper] instance_bridges)). helperVpsHost(options?) is the preset: { runtimes: ['cloud-hypervisor'], helper: { instanceBridges: ['br0'] } }, merged under options.
• Live usage. FakeHypervisor.usage() returns each domain's usage as bcd's runtime.UsageReaders read it: cloud-hypervisor's managed domains that are running or paused, libvirt-kvm's running ones, none for other runtimes. The default is fakehw's synthetic usage (a CPU second per boot, half the configured memory, zero I/O and network); setUsage(instanceId, usage) merges cpuSeconds, memoryBytes, ioReadBytes, ioWriteBytes, netRxBytes and netTxBytes over it (FakeUsage), and the domain's next boot clears it.
• FakeDaemonOptions.telemetry: { export(batch) } receives the OTLP export the telemetry section runs, never over the Session. While the section names an otlp_endpoint, the FakeDaemon exports once when the section applies, then every export_interval_ms (default 60 s, at least 5 s); an unchanged section keeps its export and interval, a changed one restarts it, and an empty endpoint stops it. Each FakeTelemetryExport has nodeId, endpoint, at, a resource (the section's resource_attributes, then service.name=bcd, service.version and bcd.node.id, which win) and metrics from instanceMetrics(): bcd.instances, bcd.instance.vcpu, bcd.instance.memory and the live bcd.instance.cpu.time, memory.used, io.read, io.write, network.receive and network.transmit (FakeMetric, FakeMetricPoint). headers_ref is not fetched, and the GPU and host series are not modelled. Without the option nothing is exported; the section converges either way.
• DevJoinOptions.daemon gives one dev.join its own FakeDaemon options over the stack's, such as { instances: helperVpsHost() } for helper hosts beside root ones. A FakeHypervisor instance passed there would be shared by every machine of the join, so pass options.

tests/testing-helper-vps-host.test.ts covers these on a harness. tests/e2e/helper-vps.test.ts runs 12 sandboxes (E2E_HELPER_SANDBOXES) on two helper hosts through the whole stack, with nodeTelemetryModule pointing the hosts at a fake collector: both hosts carry the system label bc.dev/driver.cloud-hypervisor, every sandbox reaches ready, each host's next export carries its sandboxes' usage, a guest's setUsage shows up in the export after, and deleting them leaves no leak and no usage in the next export.

Since 0fca413 vps places every hosted instance, built-in runtimes included, only on a host carrying bc.dev/driver.<runtime>, so these labels are what lets the sandboxes place at all. The PGlite vps suites seed the labels a current bcd's Hello writes, and tests/vps-sandboxes.test.ts checks that a cloud-hypervisor sandbox waits, Unschedulable naming the label, until a host advertises the runtime.

Device capabilities. A FakeDaemon with GPUs advertises cdi, nvml, devices.limits and binding.kubernetes (409f783, 2782094), so capacity places host-resource and Kubernetes reservations on it. A skew profile or an explicit capabilities list replaces the defaults, so an older-bcd profile gets neither, as it should (see capabilities).

Volumes at scale

tests/e2e/volumes.test.ts (38405a0) is the v0.5 zero-leak churn applied to volumes, the "agent memory" of PLAN.md §3.8. It runs the whole stack, the real server runtime and the real gateway, with three FakeDaemon VPS hosts on the cloud-hypervisor runtime, each reporting a 150 GiB volume store with 10 GiB reserved. It needs no Go and no WSL.

1. It waits until every host has reported its resources, its volume store (host_volume_storage) and the system-sourced bc.dev/cap.instance.volumes label.
2. It creates 12 volumes concurrently, then an over-asking round, some of which must fail CAPACITY_UNAVAILABLE.
3. It makes a read-write template per volume and shared read-only ones.
4. It churns E2E_VOLUME_SANDBOXES sandboxes (40 by default since df4b371; the nightly guarded job sets 240): created (some refused CONFLICT beside a live writer), deleted, expiring at 60 s, writing markers, volumes growing, two host reboots and two volumes retired.

Mid-churn and at the end it checks that:

• the v0.3 no-leak diff is clean (domains, overlays, taps, IP leases, bookings);
• the hosts' volumes and sandbox_volumes agree, at the booked size;
• a read-write volume never has two writers, on the hosts or in volume_attachments;
• every sandbox runs on its volumes' host, with them open;
• headroom never goes below zero and no host's store is over-promised;
• the markers are on the hosts, and a new sandbox of a volume's template finds the last writer's marker after the reboots.

Deleting everything leaves no host volume and no volume booking. It passed the first time, but at 240 sandboxes it took about 9 minutes, much of it deletions waiting on periodic passes: a deleted sandbox stayed deleting for up to 15 seconds, until the next vps.sandboxes pass, and a deleted volume up to 30 seconds, until the next vps.volumes pass. Since df4b371 both finish when their instance or host does: the instance reconciler wakes vps.sandboxes when it deletes a sandbox's instance, and the volume-store status handler wakes vps.volumes for each deleting volume whose host has applied a later spec, in the same transaction (see sandboxes). The run now takes about 43 seconds at 240 sandboxes and 11 at 40, so the default suite runs 40 and the nightly guarded job (.github/workflows/nightly.yml) sets E2E_VOLUME_SANDBOXES=240. Its hosts are FakeDaemons; bcd's own volume handling is covered by its Go tests (see the Go daemon).

The Go daemon

bcd has its own tests in daemon/ (Go 1.26 or later) and one end-to-end suite against the real control plane.

Unit and interop tests. node scripts/daemon-test.mjs runs the module's tests with the tags nonvml,fakehw (no cgo, no NVIDIA driver, the fake HGX node). On Linux it is go test ./.... On Windows, where bcd cannot run and Application Control may refuse locally built executables, it cross-compiles each package's test binary for linux/amd64 into daemon/bin/tests/ and runs it inside WSL from the package directory, so testdata paths resolve. The pki tests read daemon/internal/pki/testdata/interop.json, which node scripts/daemon-fixtures.mjs writes with @better-compute/pki (a node certificate, a signed JoinResponse, a sealed JWE, signed commands, specs and a trust bundle, an exec ticket, and secret and workload-token exchanges), so Go is checked against the TypeScript implementation byte for byte. The tunnel tests run a shell on a real pseudo-terminal (a resize is seen by stty), and the access tests check the generated KRL with ssh-keygen -Q when ssh-keygen is installed.

node scripts/daemon-test.mjs                          # every package
node scripts/daemon-test.mjs --run TestSweep -v ./internal/journal
node scripts/daemon-test.mjs --build-bcd              # also writes daemon/bin/bcd-linux-amd64 (fakehw)

bcd helper as root. node scripts/helper-e2e.mjs [--distro NAME] [-v] (da8751e) is the one test that runs bcd helper as real root. It builds bcd and internal/helper/roote2e with -tags nonvml,helpere2e (which gives bcd an e2e-only helper-call, so the node's requests come from the installed binary as the bcd user), renders the install script with --unprivileged --instance-bridges bcbr0, and runs TestHelperAsRoot as root under unshare --mount --net, over overlays of /etc, /usr/local and /var and a fresh /run, so the host is never changed (the test fails outside private namespaces). On Windows it runs in WSL as root; on Linux it needs root; elsewhere, or without unshare and useradd, it prints why and exits 0. Its steps: the script's unprivileged mode; the helper started as bcd-helper.service runs it; signed host units and their refusals; libvirt against a fake virsh; cloud-hypervisor with real taps (b9667d5); and since 035fc41 self-update under the helper, with the script's [update] pinning the two release keys of daemon/internal/pki/testdata/release.json (threshold 2, sequence 5). That step checks that update.prepare refuses a tampered manifest, another digest and a missing one; that update.swap refuses nothing staged, a symlink, a root-owned file and other bytes without moving current; that a good swap puts the release in slot b behind the entrypoint (sequence 7 recorded in upgrade.json), after which a second swap and confirming another version are refused; that update.rollback brings the installed binary back and update.settle records the release as failed, so preparing it again is refused; and that the slots and records are root's and not group- or world-writable, with the swap in the helper's log.

Site agent, ramdisk and instances. The site agent and ramdisk have Go tests only; no end-to-end or interop suite runs bcd site or bcd ramdisk against the control plane yet (a VPS host has one, below). The agent tests drive a real Agent over a scripted session (newHarness), answering its SecretRequests with sealed JWEs:

Host configuration, holds and Kubernetes (v1.1). These run against fakes: hostconfig.FakeSystem (an in-memory host whose files and command answers a test scripts), a fake root for a real Agent, and a fake kubelet on real unix sockets. Only TestRealKernel touches a real kernel.

The FakeDaemon holds BINDING_KUBERNETES assignments like bcd since 4e61f59: its CDI spec carries bcd.dev/binding: kubernetes and no NVIDIA_VISIBLE_DEVICES, while a plain hold's spec sets it once for the whole hold (tests/testing-fake-daemon.test.ts).

End to end. tests/e2e/bcd.test.ts runs the real binary (linux/amd64, fakehw, BCD_FAKEHW_SEED=41) against createDevStack({ externalTls }): the dev stack's mTLS listener with a dev serving CA, which the test pins in a generated config.toml together with the join keys, gateway_server_name = "gw.dev.local" and a data directory under /tmp. It checks that:

• bcd join with a sealed token on standard input succeeds and never prints the token;

• bcd node is adopted after its node-signed diag.nvml, and bcd status --json answers on the socket;

• inventory.refresh succeeds;

• a 2-GPU reservation turns active on a node-signed AllocationBound (checked by pkiAllocationBindingCheck), its CDI hold appears under the fake root, and it disappears on release;

• a node console opened with tunnels.open (op console) through the dev stack's tunnel relay and gateway reaches a real shell in bcd: echo hi-$((6*7)) answers hi-42, and exit 5 comes back as exit code 5;

• node.reboot completes SUCCEEDED from the next boot, with the new boot id recorded in node_sessions;

• a provisioned OS bootstraps with nothing but a token file, the control plane URL and the three pins (what provision's user data and the Robot bootstrap run): bcd join --server reads the real server's well-known document, joins through the real gateway and JoinService, removes the token, writes config.toml and join-encryption.jwk, succeeds again on a re-run without a token, and bcd node connects on the written configuration alone (472a424);

• bcd sim --nodes 5 --seed 100, given one five-use join token on standard input, joins five simulated machines that connect as five distinct nodes (five distinct hardware keys), and never prints the token (aa7501c; see bcd sim);

• self-update (fc0ca6a): a second fakehw build reporting 0.9.0 is released under a key pinned in the node's [update] section; the node restarts from an entrypoint it may replace, gets a daemon.upgrade enqueued as the rollout controller does, downloads and verifies the release, swaps to slot b, execs, and is confirmed by its Welcome: node_sessions.daemon_version is 0.9.0, current is bcd-b, and upgrade.json is gone.

• the WebSocket fallback (4b2d5aa): the TypeScript serveGatewayWs runs beside the stack's gateway with a serving certificate from the CA the node pins and audience gw.dev.local; the node restarts with [websocket] url, runs a signed inventory.refresh over the WebSocket session, and renews over POST /v1/renew, which the fakehw knob BCD_RENEW_AFTER=3s brings forward.

• node metrics (efdff1d): a test module sets NodeSpec.telemetry (http to the WSL host, a 5-second interval and an extra resource attribute), the node's spec is rendered again, and the fakehw bcd posts OTLP/JSON to a local receiver in the test: bcd.gpu.utilization for all 8 GPUs, memory, temperature, power and active XIDs, with service.name=bcd, bcd.node.id and the attribute as resource attributes. The receiver is the test's own, not a collector product.

tests/e2e/bcd-host.test.ts (91f5e81, same gating) runs NodeSpec.host end to end: a metal-host pool with policy.hugepages (2 MiB, 64) and policy.kernelModules (tun, vhost_net), provision renders the units, the gateway sends them, and the fakehw bcd applies them under its fake root. tests/e2e/bcd-placement.test.ts (427787a, same gating) runs PLAN.md §3.8 against fakehw bcd nodes, in 6 tests: a pinned, NUMA-affine flavor (vcpupin and a strict numatune in the libvirt domain the fake runtime plans, within one NUMA node and off the host-reserved CPUs), a 2 MiB huge-page flavor on the pages provision's unit reserved (page backing, locked, no balloon), an overcommitted pool's guest ballooned to its floor under simulated memory pressure and deflated again, a sandbox volume attached by its serial that outlives the sandbox, an instance on a host without the firecracker plugin label staying unschedulable, and a GPU hold with booked vCPUs and memory in its bcd-alloc-<id>.slice, reported as binding.cgroupParent and removed on release. It found that a re-collected inventory waited up to ten heartbeats for its hashes, fixed in the same commit. It relies on two fakehw features: the runtime writes the libvirt XML it would define to <data_dir>/fakehw-plans/<id>.xml and keeps its balloons on disk (9a6aaf4), and the inventory reads the huge pages the host units reserved back from the fake root (cd7a311); see local development. tests/e2e/kubernetes-bindings.test.ts (8a084ca) runs binding: 'kubernetes' on the dev stack's FakeDaemons: the hold marked bcd.dev/binding=kubernetes without NVIDIA_VISIBLE_DEVICES, a plain hold on the same pool not, the booked limits in the assignment, release, and the refusal on a pool without policy.kubernetes. tests/vps-sandboxes.test.ts checks that a sandbox lands only on a host labelled bc.dev/cap.instance.ssh-host-only (3b9e81d), and tests/vps-volumes.test.ts covers sandbox volumes: booking, quota, replay, pinning, serial mounts, one writer, read-only sharing, resize, deletion and metering (2a09202, 8cbece0). tests/metal-discovery-e2e.test.ts (452d6bf, not gated: FakeDaemons only) runs PXE discovery through the real control plane and gateway, with a FakeDaemon site agent carrying metal's site kinds: an IPMI machine enrolled with only its product UUID, metal.ensureRamdisk setting PXE and power-cycling it through the agent, the agent reporting it with setDiscovered in DHCP's mixed-endian UUID spelling, metal's module status handler (composed into the dev stack's gateway) storing the report, and metal.discovered showing the MAC as the machine's candidate with discovered_mac set, macs still empty and no boot entry sent for 2 seconds. Only after metal.confirmMac does the agent get boot.prepareRamdisk for exactly that MAC, with a secret:// discovery token reference and no sanboot digest. The dev stack has no ramdisk to join, so it ends at the boot entry; tests/metal-discovery.test.ts covers the rest on the sim.

The whole file runs on createDevStack({ signedIntent: true }) (5071dbc), which puts the stack under a command-signing root: joined daemons receive the roots in JoinResponse.trust and every command and spec bcd takes is signed and verified. With bcd-vps.test.ts that is 9 of 9 bcd end-to-end tests.

tests/e2e/bcd-vps.test.ts runs the same fakehw binary as a VPS host: bcd joins a vps-host pool over the dev stack's mTLS listener, instances.create places an instance on it, the instances converger defines and starts it on the fakehw build's file-backed libvirt-kvm runtime (not the libvirt runtime of host builds), StatusReport.instances carries it and the vps module moves it to running, and instances.delete removes the domain from the host. It has the same gating.

tests/e2e/bcd-tpm.test.ts (db9bd46, 8a9af54, same gating) runs the tpm join end to end: a fakehw bcd whose software TPM lives in <data_dir>/fakehw-tpm.json runs bcd join --attestor tpm with no token against the dev stack's JoinService with tpmAttestor (createDevStack({ attestors }) passes join attestors to betterCompute()). Since 8a9af54 the test makes a manufacturer root, the fakehw TPM carries an EK certificate it issued (BCD_FAKEHW_TPM_EK_CA), and the attestor trusts only that root (ekRoots, no allowUncertifiedEk, and a resolve allow-list for the TPM pool); with BCD_FAKEHW_TPM_KEYS=1 the node key is made in the same TPM. It checks the two rounds, resolve seeing the EK certificate chained to the manufacturer and the SHA-256 PCR bank 0 to 7, the node in the resolved pool and bound to sha256(EK) in node_bindings, node.key holding only a BCD TPM NODE KEY record, the node key certified by the TPM's AK so that both joins' node.joined events say nodeKey: 'tpm' (cf24876; the attestor keeps its default nodeKey: 'certified'), a rejoin as the same node after the identity is removed (a reinstall), already joined on a further run, and bcd node, whose session and statements are signed in the TPM, adopted on that identity. Since d94f1f3 it also checks that the certificate names the TPM's AK (urn:bc:tpm-ak:), and that bcd node (with BCD_RENEW_AFTER=3s) renews with the AK certifying the next key: one more node_certs row, still carrying the statement, and node.key still a TPM record. Its wire format is pinned in both directions without a stack too: bcd's TestTSContract and tests/pki-attestors-tpm-bcd.test.ts share the fixtures in daemon/internal/tpm/testdata, which scripts/daemon-tpm-fixture.mjs regenerates (see bcd with its TPM). Since b973e05 those fixtures include a test manufacturer root (ek-root.der) and the EK certificate it issued (ek-cert.der) in the sim's NV, so that pair runs tpmAttestor with ekRoots and no allowUncertifiedEk, and checks that another root is refused (ek-certificate-chain). Since cf24876 the fixture's answer also carries the AK's certification of a node key in the sim, which the TypeScript side admits as nodeKey: 'tpm' and refuses for another CSR key (node-key-not-csr); tests/pki-attestors-tpm.test.ts covers the same checks with the TypeScript software TPM, and suites whose software TPM holds no node key opt into nodeKey: 'any'. None of these touches a hardware TPM.

On Linux the suites run in pnpm test whenever Go is installed; they build the binary first. On Windows they are opt-in: they need BCD_E2E=1, a WSL distribution, and Go at C:\Program Files\Go\bin (or an existing daemon/bin/bcd-linux-amd64). bcd runs in WSL through wsl -e, never a shell, and reaches the gateway at WSL's default route. BCD_E2E_LOG=1 streams bcd's log to standard error.

$env:BCD_E2E = '1'; pnpm vitest run tests/e2e/bcd.test.ts

The fleet simulator

createFleetSim(options) runs N FakeDaemons, a fresh faulty link per dial, a seeded chaos schedule and a pluggable GatewayPort. The default port is GatewayStub, a minimal in-memory control plane (Hello, Welcome, dispatch with credits, result ingestion, liveness, the command sweeper, bookings under an exclusion check, usage flushes). The stub is a test double for the daemon side, not a reference for the server.

import { assertInvariants, createFleetSim } from 'better-compute/testing';

const sim = createFleetSim({
  seed: 'fleet-100',
  nodes: 100,
  up: { latencyMs: 5, jitterMs: 20, drop: 0.0005, dropMode: 'reset' },
  down: { latencyMs: 5, jitterMs: 20, reorder: 0.002 },
  chaos: { everyMs: 5_000, crash: 0.3, reboot: 0.1, journalLoss: 0.05, partition: 0.3, gatewayKill: 0.02 },
});
await sim.start();
await sim.run(10 * 60_000); // ten simulated minutes
sim.quiesce();
await sim.run(16 * 60_000);
assertInvariants(sim.invariants());
console.log(sim.digest()); // the same seed gives the same event log, byte for byte

The simulator also exposes each fault as a method (crash, reboot, loseJournal, partition, isolate, heal, halfOpen, resetLink, stall, killGateway), every(ms, fn) for workloads, and rng.fork(label) for independent seeded streams: adding a node never shifts another node's draws. better-compute sim is this simulator with a built-in workload; see local development.

Invariants

The checkers are pure functions over collected facts, so the same checks run over the fleet simulator, the real control plane or a database snapshot.

Two more I1 checkers (40b1ce8) cover PLAN.md §3.8's host resources and Kubernetes bindings:

With the knob kubePodReleaseMs, a FakeDaemon's pods keep a released Kubernetes hold's devices that long: it stops offering them at once, keeps them claimed (a new hold on them fails with overlap) and reports AllocationReleased only then, as bcd's fail-closed release does.

assertInvariants(violations, limit?) throws one error listing them. The fleet simulator's invariants() checks I1, I2, I3, I4, I5 and I8 (I4, I5 and I8 read the gateway port's facts(), which the stub gateway and the control-plane simulator provide). I7 needs a database: pass it to runNightly({ invariants }), as the CI-sized night does. PLAN.md section 15 also names I6 (latency SLOs); the kit does not check it, and the real-time benchmarks measure I6-style latencies instead.

I4 is also tested under gateway clock skew, which section 15 calls for: tests/sim/clock-skew.test.ts runs 9 nodes on 3 real gateways whose own clocks are 90 s ahead, 90 s behind and on time (more than the 40 s liveness timeout), with compute_now() and the daemons on simulated time. Every gateway holds sessions, diag.nvml settles through all of them, and when the unskewed gateway dies its nodes re-home onto the skewed ones; connectivity sampled every simulated second never shows unreachable, no node.unreachable is published, and I1 to I5 and I8 hold, because gateways flush ages rather than timestamps.

The control-plane simulator

createControlPlaneSim(options) puts the real control plane and real gateways behind the fleet simulator: betterCompute on the sim clock with compute_now() pinned to simulated time, gateways real gateways on manual jobs (default 1), and nodes that join through JoinService with sealed tokens bound to their CSRs. It sets up a site and an H100 pool with diag.nvml acceptance offered to acme; reconcilers and gateway jobs run on the simulator's ticks.

The rig offers run(ms), api(fn), killGateway(i), reviveGateway(i), stallGateway(i, stalled), pauseWorkers(paused), revokeNode(nodeId), homeOf(nodeId), serialOf(nodeId) and invariants(), which reads the facts from compute's own tables (bookings, billed allocations and usage windows, command outcomes, connectivity) and checks I1, I2, I3, I4, I5 and I8. tests/sim/multi-gateway.test.ts uses it for a gateway killed mid-command and a zombie gateway fenced by its session epoch.

tests/sim/kubernetes-host-resources-1k.test.ts (246c95b) churns Kubernetes-bound and hold reservations with vCPU and memory shares over a GPU pool of SIM_K8S_NODES hosts (default 1,000) under link faults, crashes, partitions and resets, re-inventories hosts with less CPU and memory and then restores them, and has pods keep released Kubernetes GPUs for 45 seconds. After every step it asserts checkHostResources and the fleet invariants; the recorded run at 1,000 nodes is green in about 4.5 minutes. A second, strict 12-node run of checkKubeAdvertisements checks that no kubelet is offered a GPU booked elsewhere. It used to fail (capacity freed a Kubernetes-bound GPU at release and re-booked it before pods let go, so the node rejected the new binding: 55 of 240 reservations at 12 nodes); since e7d922c a Kubernetes booking ends at the node's AllocationReleased, and the run passes with no failed reservations.

The nightly simulation

runNightly({ seed, nodes?, gateways?, minutes?, chaos?, stepMs?, modules?, inspect?, invariants? }) runs one simulated night on the control-plane simulator: three minutes of start-up, then minutes (default 30) of tenants reserving with every placement feature (half and whole nodes, gangs of two, spot, calendar starts, early releases) and operators running commands, under chaos. Chaos adds gateway kills and revivals, worker outages and, at heavy, one certificate revocation per night. invariants(rig) adds checks run after the quiet period, such as I7 with capacity's placeablePending. The report has the invariant violations, kindLeaks (commands queued outside a restricted certificate's kinds, which must be 0), place-to-active and command latencies in simulated time, and two digests: the workload (a function of the seed) and the id-free outcome. Compute mints random ids, so outcomes can differ between runs of one seed.

pnpm sim:nightly
SIM_NIGHTLY_SEEDS=1,2,3,4 SIM_NIGHTLY_CHAOS=heavy SIM_NIGHTLY_NODES=32 SIM_NIGHTLY_MINUTES=120 pnpm sim:nightly

pnpm sim:nightly (vitest.nightly.config.ts, tests/nightly/*.nightly.ts) runs every seed times chaos level one after another, with auditModule checkpointing every 10 simulated minutes and every tenant's audit chain verified at the end. Defaults: seeds 1,2,3, chaos light,heavy, 16 nodes, 3 gateways, 60 simulated minutes; reports land in bench/sim-nightly/results. A CI-sized night (4 nodes, 2 gateways, 6 minutes) runs in tests/sim/nightly.test.ts; both check I7 through invariants. The v0.4 exit asks for every section 15 scenario green on three consecutive nights; the nightly CI workflow counts them (see CI).

Cold start

runColdStart({ seed, nodes?, gateways?, startSpreadMs?, limitMs?, settleMs?, restart?, restartDownMs?, stepMs?, workerPasses? }) is the v0.4 "10k cold start in 3 minutes" scenario on the control-plane simulator. The fleet powers on at random times within startSpreadMs (default 30 s) behind gateways (default 3) gateways; each node's first session (node.connected) and its adoption (node.adopted) are timed from power-on, in simulated time, from the outbox. The run is ready when every node is connected, adopted and schedulable, and it must be ready within limitMs (default 3 minutes). With restart (default true) every gateway then goes down and every node power-cycles, and the way back is timed the same way. The invariants I1 to I5 and I8 are checked at the end, and a node.unreachable mark during boot fails the run. Defaults: 100 nodes and workerPasses 10. The report has per-phase latency summaries, unreachableMarks, schedulable, the violations and wall.loops, the wall time spent in each part of the loop.

The recorded 10k run (bench/cold-start/README.md) was ready at 80 simulated seconds: every node connected by 30 s (p99 29.7 s, bound by the power-on spread), adopted by 79 s and schedulable at 80 s, and every node was back 40 s after the power cycle, with 0 unreachable marks and 0 invariant violations. Run it alone with:

COLD_START_NODES=10000 node node_modules/vitest/vitest.mjs run --config vitest.nightly.config.ts tests/nightly/cold-start.nightly.ts

See scaling for the 1k and 10k results side by side.

Real-time harnesses

The fake clock never measures latency. The real-time harnesses run on the wall clock over cleartext HTTP/2 (createH2cClient), against PGlite by default or a Postgres you name with postgresUrl (it must be disposable: the harness migrates it).

• createRealtimeHarness({ nodes?, postgresUrl?, gateway?, services?, daemon?, sharedConnection?, pollMs? }): a real gateway and database with FakeDaemons on pre-registered nodes (default 50). connectAll(), dispatch({ kind, perNode }) (enqueue to acknowledgement and to terminal), load({ ratePerSecond, durationMs }) (steady state) and storm({ fraction, mode: 'disconnect' | 'crash' }) (reconnect storms).
• createRealtimeFleet({ postgresUrl?, variant?, pollMs?, worker?, gateway?, daemon?, joinConcurrency? }): the whole v0.1 spine, a started control plane with JoinService on real PKI and a gateway wired to its services. join(n) times each node from its Join call to adopted; place(n, { gpus?, ratePerSecond?, concurrency? }) times reservations from placed to active.
• probeLeases(sql) samples the leases table: renewal gaps, the least headroom before expiry, lapses while owned, and takeovers.

Every result is a LatencySummary (count, p50, p90, p99, max). Small runs are in tests/realtime/; the 1k runs belong to the benchmarks.

Postgres-only suites

Everything about locking and fencing needs real PostgreSQL: PGlite is one serialized connection. These suites skip unless BETTER_COMPUTE_POSTGRES_URL is set:

BETTER_COMPUTE_POSTGRES_URL=postgres://postgres:postgres@localhost:5432/compute_test pnpm test:postgres

pnpm test:postgres runs tests/postgres-conformance.test.ts (the full store conformance suite, concurrency cases included, each case in a fresh schema; the database needs btree_gist or the right to create it) and tests/postgres-schema.test.ts. CI runs them against a postgres:17 service. The benchmarks read the same variable.

Benchmarks

pnpm bench:scale runs tests/bench/**/*.bench.ts with vitest.bench.config.ts (the test aliases, a 30-minute timeout), on demand only. gateway-streams sets its own timeout, BENCH_GW_DURATION_MS plus 20 minutes, so the 30-minute gate run with its ramp, storm and settle is not cut off. Each suite writes bench/<suite>/results/<date>-<database>-<n>.json with the commit, the machine, the Node.js version and the configuration. Commit only runs on the reference hardware of bench/REFERENCE.md, or runs labeled referenceHardware: false.

The numbers measured so far, and what they mean, are on scaling.

CI and the repository checks

Four GitHub Actions workflows are defined under .github/workflows/. This page describes what they run, not whether they are green.

pnpm check runs typecheck, test, test:iam-remote, lint:repo and validate:packages. pnpm lint:repo runs the repository rules in order: check-deps (dependency rules), check-sql-now (the one-clock rule), umbrella --check (umbrella drift), proto-gen --check and proto-gen --breaking, then apps/docs/scripts/generate.mjs --check (the generated docs reference must match the code; 267f8f4), then the API surface freeze. The generator check comes first because the freeze reads the committed reference: before it, a narrowed input that nobody regenerated passed the freeze. Under pnpm check it runs after tsc -b, so it sees the built code.

If you change the registry (a method, its input schema, an error code, a command kind, an event or a metric), regenerate the reference in the same change. On a failure the check lists each file out of date or stale and prints the command to run, which regenerates from committed code only, so other sessions' uncommitted builds never reach the reference:

node scripts/head-check.mjs --run apps/docs/scripts/sync-from-head.mjs "$PWD"

The API surface freeze

PLAN.md's v1.0 freezes the API. scripts/api-surface.mjs compares the frozen surface, generated/api-surface.json (every method with its route, kind, IAM action, resource, alternatives, list flag, status, phase and input JSON Schema, and every error code with its HTTP status), with the current one, which it reads from the docs generator's committed apps/docs/generated/api.json and errors.json.

Additions pass. These fail the check:

• a served method removed, or planned again;
• its route, kind, IAM action or resource changed, an IAM alternative removed, or a list method no longer one;
• its input schema rejecting an input the frozen schema accepted: a new required field, a type, enum value or property that goes, a tighter length, range, pattern, format or item count, a closed object that was open, a lost anyOf branch, or a constraint on a value that was unconstrained;
• an error code removed, or its HTTP status changed.

A method that is only planned may still change or go. When you add or change something compatibly, regenerate the docs and record it, so the addition is frozen too:

node apps/docs/scripts/generate.mjs   # api.json and errors.json follow the code
pnpm api:surface                      # node scripts/api-surface.mjs --update: records additions, refuses breaks
node scripts/api-surface.mjs --update --accept-breaking   # only for a deliberate, reviewed break

The plain check prints the number of additions not yet recorded without failing. tests/scripts-api-surface.test.ts covers the rules and checks the committed snapshot against the generated docs.