# Testing (/docs/operations/testing) > The testing kit - a control plane on PGlite, the FakeDaemon, faulty links, the fleet simulator and its invariants, real-time harnesses and benchmarks. `@better-compute/testing` (also `better-compute/testing`) is the kit the repository's own tests run on, and the one you use to test modules, plugins and policies. It gives you a migrated control plane on PGlite with a clock you can move, a scripted IAM, the daemon side of the agent protocol (`FakeDaemon`), in-memory links with fault injection, a deterministic fleet simulator with invariant checkers, and real-time harnesses for latency measurements. No database server is needed. | Layer | Exports | Clock | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------- | | Control plane fixture | `computeFixture`, `closeFixtures`, `fakeIam` | sim or wall | | Daemon side | `FakeDaemon`, `createDaemonHarness`, `fakeHardware`, `MemoryJournal` and the journal rules | sim | | Links | `createAgentLink`, `createFramePair`, `FrameConn` | sim | | Fleet simulator | `createFleetSim`, `GatewayStub`, `checkI1`, `checkI2`, `checkI3`, `checkI4`, `checkI5`, `checkI7`, `checkI8`, `assertInvariants` | sim | | Real control plane under the simulator | `createControlPlaneSim`, `controlPlaneFacts`, `runNightly` | sim | | Wall-clock harnesses | `createRealtimeHarness`, `createRealtimeFleet`, `probeLeases`, `createH2cClient` | wall | | Local stack | `createDevStack`, `attachDevFleet` (see [local development](/docs/operations/local-development#the-dev-stack-in-code)) | wall | | Plumbing | `createSimScheduler`, `createRealtimeScheduler`, `createRng`, `EventLog` | either | The kit needs the optional peer `@electric-sql/pglite` as a devDependency; without it `computeFixture()` fails with `MISSING_DEPENDENCY`. The embedded Better IAM of `@better-compute/iam/testing` needs the optional `@better-iam/adapter-sqlite` peer. ## Test a module with computeFixture [#test-a-module-with-computefixture] `computeFixture(options?)` builds `betterCompute()` on an in-memory PGlite, runs `compute.initialize()` (the registry check, the base schema and every module's migrations) and hands you the pieces. It does not start the workers: your test drives reconcilers one pass at a time with `compute.jobs[name].tick()`. ```ts title="my-module.test.ts" import { afterEach, expect, it } from 'vitest'; import { closeFixtures, computeFixture } from 'better-compute/testing'; import { myModule } from '../src/module'; afterEach(closeFixtures); it('asks IAM before listing, and reconciles', async () => { const { request, iam, tenants, compute, advance } = await computeFixture({ modules: [myModule()] }); iam.allow({ token: tenants.acme.token, action: 'compute:nodes:read' }); const answer = await request('nodes', 'list', { tenantId: tenants.acme.id }, { token: tenants.acme.token }); expect(answer.ok).toBe(true); expect(iam.calls.length).toBeGreaterThan(0); // authenticate, then the tenant decision await advance(60_000); // compute's Clock and compute_now() move together await compute.jobs['my.reconciler']!.tick(); // one deterministic pass }); ``` | Field | What it is | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `compute` | The initialized control plane, typed by `modules` and `plugins` | | `database` | The migrated PGlite `SqlExecutor` | | `clock`, `advance(ms)` | The fixture clock | | `iam` | The `fakeIam()` port compute calls | | `tenants` | `root`, `acme` and `other`, each `{ id, token, caller }`; ids `tenant-root`, `tenant-acme`, `tenant-other` | | `request(group, method, input?, options?)` | `POST {basePath}/{group}/{method}` through `compute.handler`, with the CSRF headers; `options.plugin` routes to `plugins/{plugin}/…`, `options.headers` replaces defaults | | `close()` | Stops compute and closes the store; idempotent | Options: `clock` (`'sim'`, the default, or `'system'`), `start` (default 2026-01-01T00:00:00Z), `iam` (extra `fakeIam()` options), `modules`, `plugins` and `log`. `request()` answers `{ ok, status, headers, requestId }` plus `data` or `error`. **The sim clock.** Under `clock: 'sim'`, compute's `Clock` and SQL's `compute_now()` read the same time (principle P9). `advance(ms)` and `clock.set(ms)` move the `compute_sim_clock` table first and the TypeScript clock second, so leases, `not_before`, column defaults and admission contexts all move together. Time never moves backwards. Under `'system'`, `advance` rejects with `INVALID_STATE`. **`fakeIam(options?)`** is a scripted `IamPort`. Bearer tokens map to callers (an unknown token is `UNAUTHENTICATED`). A decision is allowed only when an `allow` rule matches and no `deny` rule does; a rule matches on any of `token`, `tenantId`, `action`, `type` and `id`, and `{}` matches everything. Every call is recorded in `calls`; `count(method)` counts them. `billingCheck` allows, `recordUsage` records, `featureEnabled` is false and `requestConfirmation` answers `'not-held-back'`. Plugins, drivers and providers have **conformance suites**: functions that return `ConformanceCase[]` (`{ name, run(subject) }`, from `@better-compute/core`), such as `providerConformanceCases()` from `@better-compute/vps` and `storeConformanceCases()` from `@better-compute/postgres`. Run them from any runner: ```ts for (const test of providerConformanceCases()) it(test.name, async () => test.run(await createSubject())); ``` ## Real Better IAM in tests [#real-better-iam-in-tests] `computeFixture()` still uses the scripted `fakeIam()`. When a test must exercise real policies, use `createEmbeddedTestIam(options?)` from `better-compute/iam/testing`: a real better-iam on in-memory SQLite, carrying compute's plugin, resolver and inbox, bootstrapped with a root administrator (TOTP MFA) and two organizations, "Acme" and "Other". Pass its `port` to `betterCompute({ iam })` and call `bind(sql)` once the compute store is migrated. * Each organization (`acme`, `other`, or a new one from `tenant(name)`) offers `member(name)`, `signIn(name)`, `policy(document)`, `grant(identityId, ...documents)` and `share(type, id, relation, subject)`. * Options: `now`, `sql`, `sessionJwts`, `runIamJobs`, `inbox` (`'store'` or `'memory'`), `iamOptions`, and the catalog's `modules`, `plugins`, `extra` and `typePrefix`. * `mode`: `'embedded'` (the default, or `BETTER_COMPUTE_IAM_MODE`) or `'remote'`, which hands compute `bridge.remote()` over the same better-iam's HTTP handler, so the suite runs as it would against IAM in another process. * The secret is fixed and public. Never use it outside tests. `pnpm test:iam-remote` reruns the IAM-dependent suites in remote mode (`vitest.iam-remote.config.ts`, 13 files: the example policies, the resolver timeout, confirmation, the testing IAM, streams, MCP confirmation, watch, the e2e access and spine tests, the authz matrix, the budget check, webhook intake and `serve`'s compose file), and `pnpm check` runs it after `pnpm test`, so every change stays green in both IAM modes (a v0.4 exit). The `server-*`, `capacity-*` and `vps-*` suites use the scripted `fakeIam()` port, so an IAM mode does not apply to them. `pnpm check:head` runs `scripts/head-check.mjs`, which checks the committed tree in a separate worktree. `scripts/head-check.mjs` exists because several sessions edit the shared checkout at once, so a full `pnpm check` there mostly reports other people's work in progress. It keeps a git worktree under the temporary directory (`bc-head-check`), moves it to the commit (`--ref`, default this checkout's `HEAD`), links every `node_modules` of the checkout into it without installing anything, builds with `tsc -b` unless `--no-build`, and runs `vitest run` with the filters you give. Each concurrent run takes its own tree (`bc-head-check`, then `-2` up to `-8`) through a `.lock` file holding its pid, taken over when that pid is dead and released on exit, so parallel runs never move each other's tree mid-run. `--run [args…]` runs that script in the held tree after the build instead of vitest, and exits with its status; the docs reference is regenerated from committed code this way: ```bash node scripts/head-check.mjs tests/vps-sandboxes.test.ts node scripts/head-check.mjs --run apps/docs/scripts/sync-from-head.mjs "$PWD" ``` The authz matrix has two halves. `tests/generated/authz-matrix.test.ts`, generated by `scripts/authz-matrix.mjs`, covers the envelope side, one test per registered method. `tests/iam-bridge-matrix.test.ts` covers the IAM side: every distinct (action, resource type) pair of the PLAN.md §9.2 table, alternatives included, is decided through compute's own `IamPort` against a real better-iam carrying the bridge's plugin and resolver. Each pair is denied without a grant, allowed with one, and never allowed on a resource type it was not granted on. Under `pnpm test` the port is `bridge.embedded`; under `pnpm test:iam-remote` it is `bridge.remote` over the same IAM's HTTP API, and the test asserts that the decisions went over HTTP. `spyIamPort(port)` from `better-compute/iam` wraps any `IamPort` and records every call. Code paths tag themselves with `spy.path(name, fn)`; each call is attributed to the innermost path across awaits, so a test can assert that fleet paths never reach IAM. That is how the v0.1 security negative is tested: `tests/sim/iam-negative.test.ts` runs ten simulated minutes of heartbeats, inventory, commands and a reservation, and asserts that every IAM call came from an API admission. See [policies](/docs/guides/iam/policies). ## The FakeDaemon [#the-fakedaemon] `FakeDaemon` is the daemon side of the protocol in TypeScript, rule for rule with the Go packages in `daemon/internal/{journal,dispatch,session,converge,agent,fakehw}`. The real bcd has its own tests (see [the Go daemon](#the-go-daemon)); the FakeDaemon is what runs where features bcd lacks, or many daemons on simulated time, are needed. `bcd sim` runs many real agents too, but in real time against a real gateway. See [the daemon](/docs/daemon). * **Journal.** `accepted` before any side effect; `running{boot_id}` right before a self-terminating one; done results kept 7 days; unacked events capped at 64 MiB and 10,000. A crash keeps the journal, a reboot also changes `boot_id` and clears `/run`, and `loseJournal()` starts a new `journal_id`. * **Duplicate delivery.** Done: resend the stored result. Running: re-attach. Accepted but not running after a crash: re-execute if `retry_safe`, else `FAILED{interrupted}`. Self-terminating across a reboot: `SUCCEEDED{rebooted}`. * **Acceptance order.** Another node's command or a disallowed kind is `UNSUPPORTED`; a `min_generation` above the applied spec defers with a `SpecRequest`; past `not_after` is `EXPIRED`; no free lane credit is `BUSY`. * **Session.** Hello from the journal, Welcome, heartbeats, PING liveness, full-jitter reconnect, `Drain`, `RevocationNotice`, `Cancel`; a `Welcome` more than 5 minutes before the journal's server time mark is refused. The mark (`JournalMeta.serverTimeMs`) is set on every Welcome and heartbeat and only moves forward, so a gateway cannot rewind the unsigned `Welcome.server_time` to make a recorded signed command or an old worker certificate valid again; the session drops with `server time went backwards` on every redial. It is not clamped to the local clock, since sim and manual server clocks sit far from the host's. * **Desired state.** The devices converger writes the CDI spec and env file in an in-memory file map, reports `AllocationBound`, `AllocationReleased` and `AllocationFailed`, and never drops a hold on its own. * **Kinds.** `diag.nvml`, `inventory.refresh`, `logs.collect` and `node.reboot`, plus the opt-in `fakeGpuReset` (not retry-safe). `knobs` make them fail: `diag.nvmlInitFails`, `diag.failGpus`, `diag.error`, `logs.uploadFails`, `cdiWriteFailures`. * **Beyond v0.1.** With `intent`, it verifies signed commands (`BAD_SIGNATURE`; with command-signing roots every command must carry a valid signature, whatever its kind, 12ccfa1, which `tests/testing-daemon-intent.test.ts` checks with an unsigned non-destructive command) and signed NodeSpecs: `intent.signedSpecs` is `'always'` (the default: with command-signing roots every spec must be signed, since an unsigned spec with a high generation would freeze all later ones) or `'access'` (only a spec that changes `access` must be signed, for a control plane that signs nothing else), and refused specs are listed in `specRejections`. `intent.quorum` (a `FakeQuorumTrust`: approver keys, threshold, tiers, `unbatchedPerHour`), or the quorum of a newer signed trust bundle, enforces the v1.0 destructive-batch quorum: destructive commands are rejected `BATCH_UNAPPROVED` or `UNBATCHED_LIMIT`, and a newer trust bundle that changes the quorum without the current approvers' signatures for the FakeDaemon's own trust domain is ignored (logged `trust.ignored`; 6a7f1b7, 4835f4f) (see [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10)). A refused session whose close carries `retryAfterMs` (the gateway's retry pushback) reconnects after exactly that long. `setPressure(pressure)` puts a `HostPressure` into every status report. `setDiscovered(machines)` (537e672) plays a site agent's PXE discovery list: each `{ mac, productUuid?, arch?, firstSeen?, lastSeen?, count?, relay? }` (times in epoch milliseconds, default now; MAC and UUID lower-cased) goes into `StatusReport.discovered` of every report from then on, the current report is re-sent, and `undefined` stops it. With `tunnels`, it answers `OpenTunnel`; with `nodeKey`, attested results and allocation reports carry `node_sig`. With `instances` (a `FakeHypervisor` or its options) it plays a VPS host, advertising `instance.network-policy` in `Hello` and converging domains, overlay disks, NoCloud seeds and taps and fetching user data through the secret broker; as such a host, `handleImds(request, instanceId)` answers a running instance's metadata request (v0.5, a `FakeImds`), minting its workload token through a `WorkloadTokenRequest` on the session. With `selfUpdate: { badVersions?, swapMs? }` it converges `NodeSpec.daemon` by swapping and restarting, and a bad build falls back to its previous slot and reports the section failed. `fakeDriverInvoke({ drivers })` adds the `driver.invoke` kind, running in-process or bcx drivers. `createWsDialer()` dials the WebSocket fallback. | Option | Default | Meaning | | --------------------------------- | -------------------------------- | ------------------------------------------------------------------- | | `nodeId`, `scheduler` | required | The certificate's node id; the scheduler that runs its timers | | `dial` | none | Opens a Session stream (`FrameConn`); `undefined` means unreachable | | `hardware` | `fakeHardware({ seed: nodeId })` | The board it reports | | `role` | node | `AgentRole` | | `kinds` | the four defaults | Merged over the defaults; `null` removes one | | `workers` | control 4, provision 2, bulk 1 | Worker pool per lane | | `journal` | 64 MiB, 10,000 events | Journal caps | | `backoff` | 1 s to 60 s | Reconnect back-off with full jitter | | `pingIntervalMs`, `pingTimeoutMs` | 20 s each | HTTP/2 PING | | `welcomeTimeoutMs` | 30 s | A session without Welcome is dropped | | `rebootDurationMs` | 60 s | How long a reboot keeps the node down | | `clockSkewMs` | 0 | The daemon's clock minus the server's | **Fake hardware.** `fakeHardware({ seed, variant, missingGpu? })` gives `hgx-h100` (8x H100 SXM5 80GB on NVSwitch, field for field the Go `fakehw`: the same seed gives the same section hashes), `hgx-h100-gpu-missing` (7 visible, so `GpuCountMismatch`), `l40s` (4x L40S 48GB on PCIe) and `cpu` (no `gpus` section). `encodeInventorySections()` reproduces the Go golden bytes. **Links.** `createAgentLink()` builds a daemon end and a gateway end whose frames cross as protobuf bytes, with faults per direction from the link's seeded RNG: `latencyMs` (default 1), `jitterMs`, `drop` with `dropMode` (`silent`, or `reset`, which breaks the stream as gRPC would), `duplicate`, and `reorder` within `reorderWindowMs` (default 50). Controls: `stall`, `partition` and `heal`, `halfOpen`, `reset`, `dropNext`, `duplicateNext` and `intercept`. `createDaemonHarness()` wires one daemon to scripted servers on a sim scheduler. **Host volumes** (38405a0). A `FakeHypervisor` keeps [sandbox volumes](/docs/guides/vps/sandboxes#volumes) the way bcd does, so a FakeDaemon VPS host can stand in for bcd in volume tests: * `InstancesSection.volumes` are created, or grown in place, only when the store has room (available minus reserved minus outstanding); otherwise the volume keeps its size and the instances naming it fail. A volume is never shrunk, and one no longer listed is destroyed once nothing has it open. The files would be `volumePath(id)`, `/var/lib/bcd/volumes/.raw`. * An instance's `InstanceSpec.volumes` are opened at boot, and what counts is what running and paused domains really have open, not their specs: one writer, and read-only volumes shared only with other readers. A pass applies stops before starts, so a volume handed from one instance to another is closed before it is opened again. * Volumes survive `powerLoss`; `wipe()` clears them. The hooks `writeVolume(instanceId, volumeId, content, bytes?)` (a content marker, 1 MiB written by default), `volumeContent(volumeId)` and `openBy(volumeId)` let a test check what a guest would find and who has a volume open. * With `volumeStore: { totalBytes, reservedBytes? }` in its options, `volumeStorage()` feeds `StatusReport.volume_storage`, which the FakeDaemon sends in every status report: available is total minus what guests wrote, reserved defaults to bcd's 5% of the total with a 2 GiB floor, committed is the sum of the sizes, and outstanding the sum of size minus written. **Host admission** (6922553). A start the `FakeHypervisor` has no free vCPU or memory for is no longer an instance failure. Like [bcd's host admission](/docs/daemon/delivery#instances) (`agent/admission.go`), the domain stays `stopped` with a message starting `ResourcesExhausted:` (the prefix is exported as `RESOURCES_EXHAUSTED`), and the section is `PROGRESSING` with reason `Retrying` and the message `ResourcesExhausted: this host cannot start now`, unless another instance failed in the same pass (then it is `FAILED`, `InstanceFailed`). The next pass tries the start again. That is the report vps turns into `HostResourcesExhausted` and a re-placement (see [hosts that cannot start an instance](/docs/guides/vps/instances#hosts-that-cannot-start-an-instance)). `tests/testing-daemon-instances.test.ts` has the unit test. **A VPS host under `bcd helper`** (1c9684a). The FakeDaemon plays the control-plane side of a bcd that runs unprivileged under [`bcd helper`](/docs/daemon#privilege-separation), with `cloud-hypervisor` through root's `ch.*` operations: * **Runtimes in `Hello`.** A FakeDaemon with `instances` lists each of its `FakeHypervisor`'s runtimes as a driver manifest of kind `runtime` (`runsOn: NODE`, its daemon version), as bcd does for its built-ins, so the gateway labels the node `bc.dev/driver.`. An explicit `drivers` option replaces the list. * **`FakeHypervisorOptions.helper: { instanceBridges }`** plays `[helper] instance_bridges`: a NIC on any other bridge (an empty list refuses every NIC) fails its instance in the helper's words, `libvirt-kvm` before the domain exists (`domain: bridge "…" is not an instance network`) and `cloud-hypervisor` when it starts (`ch.taps: "…" is not an instance bridge of this host ([helper] instance_bridges)`). `helperVpsHost(options?)` is the preset: `{ runtimes: ['cloud-hypervisor'], helper: { instanceBridges: ['br0'] } }`, merged under `options`. * **Live usage.** `FakeHypervisor.usage()` returns each domain's usage as bcd's `runtime.UsageReader`s read it: `cloud-hypervisor`'s managed domains that are running or paused, `libvirt-kvm`'s running ones, none for other runtimes. The default is fakehw's synthetic usage (a CPU second per boot, half the configured memory, zero I/O and network); `setUsage(instanceId, usage)` merges `cpuSeconds`, `memoryBytes`, `ioReadBytes`, `ioWriteBytes`, `netRxBytes` and `netTxBytes` over it (`FakeUsage`), and the domain's next boot clears it. * **`FakeDaemonOptions.telemetry: { export(batch) }`** receives the OTLP export the `telemetry` section runs, never over the Session. While the section names an `otlp_endpoint`, the FakeDaemon exports once when the section applies, then every `export_interval_ms` (default 60 s, at least 5 s); an unchanged section keeps its export and interval, a changed one restarts it, and an empty endpoint stops it. Each `FakeTelemetryExport` has `nodeId`, `endpoint`, `at`, a `resource` (the section's `resource_attributes`, then `service.name=bcd`, `service.version` and `bcd.node.id`, which win) and `metrics` from `instanceMetrics()`: `bcd.instances`, `bcd.instance.vcpu`, `bcd.instance.memory` and the live `bcd.instance.cpu.time`, `memory.used`, `io.read`, `io.write`, `network.receive` and `network.transmit` (`FakeMetric`, `FakeMetricPoint`). `headers_ref` is not fetched, and the GPU and host series are not modelled. Without the option nothing is exported; the section converges either way. * **`DevJoinOptions.daemon`** gives one `dev.join` its own FakeDaemon options over the stack's, such as `{ instances: helperVpsHost() }` for helper hosts beside root ones. A `FakeHypervisor` instance passed there would be shared by every machine of the join, so pass options. `tests/testing-helper-vps-host.test.ts` covers these on a harness. `tests/e2e/helper-vps.test.ts` runs 12 sandboxes (`E2E_HELPER_SANDBOXES`) on two helper hosts through the whole stack, with `nodeTelemetryModule` pointing the hosts at a fake collector: both hosts carry the system label `bc.dev/driver.cloud-hypervisor`, every sandbox reaches `ready`, each host's next export carries its sandboxes' usage, a guest's `setUsage` shows up in the export after, and deleting them leaves no leak and no usage in the next export. Since 0fca413 vps places every hosted instance, built-in runtimes included, only on a host carrying `bc.dev/driver.`, so these labels are what lets the sandboxes place at all. The PGlite vps suites seed the labels a current bcd's `Hello` writes, and `tests/vps-sandboxes.test.ts` checks that a `cloud-hypervisor` sandbox waits, `Unschedulable` naming the label, until a host advertises the runtime. **Device capabilities.** A FakeDaemon with GPUs advertises `cdi`, `nvml`, `devices.limits` and `binding.kubernetes` (409f783, 2782094), so capacity places host-resource and Kubernetes reservations on it. A skew profile or an explicit `capabilities` list replaces the defaults, so an older-bcd profile gets neither, as it should (see [capabilities](/docs/daemon/sessions#hello)). ### Volumes at scale [#volumes-at-scale] `tests/e2e/volumes.test.ts` (38405a0) is the v0.5 zero-leak churn applied to volumes, the "agent memory" of PLAN.md §3.8. It runs the whole stack, the real server runtime and the real gateway, with three FakeDaemon VPS hosts on the `cloud-hypervisor` runtime, each reporting a 150 GiB volume store with 10 GiB reserved. It needs no Go and no WSL. 1. It waits until every host has reported its resources, its volume store (`host_volume_storage`) and the system-sourced `bc.dev/cap.instance.volumes` label. 2. It creates 12 volumes concurrently, then an over-asking round, some of which must fail `CAPACITY_UNAVAILABLE`. 3. It makes a read-write template per volume and shared read-only ones. 4. It churns `E2E_VOLUME_SANDBOXES` sandboxes (40 by default since df4b371; the nightly `guarded` job sets 240): created (some refused `CONFLICT` beside a live writer), deleted, expiring at 60 s, writing markers, volumes growing, two host reboots and two volumes retired. Mid-churn and at the end it checks that: * the v0.3 no-leak diff is clean (domains, overlays, taps, IP leases, bookings); * the hosts' volumes and `sandbox_volumes` agree, at the booked size; * a read-write volume never has two writers, on the hosts or in `volume_attachments`; * every sandbox runs on its volumes' host, with them open; * headroom never goes below zero and no host's store is over-promised; * the markers are on the hosts, and a new sandbox of a volume's template finds the last writer's marker after the reboots. Deleting everything leaves no host volume and no volume booking. It passed the first time, but at 240 sandboxes it took about 9 minutes, much of it deletions waiting on periodic passes: a deleted sandbox stayed `deleting` for up to 15 seconds, until the next `vps.sandboxes` pass, and a deleted volume up to 30 seconds, until the next `vps.volumes` pass. Since df4b371 both finish when their instance or host does: the instance reconciler wakes `vps.sandboxes` when it deletes a sandbox's instance, and the volume-store status handler wakes `vps.volumes` for each `deleting` volume whose host has applied a later spec, in the same transaction (see [sandboxes](/docs/guides/vps/sandboxes#read-extend-and-delete)). The run now takes about 43 seconds at 240 sandboxes and 11 at 40, so the default suite runs 40 and the nightly `guarded` job (`.github/workflows/nightly.yml`) sets `E2E_VOLUME_SANDBOXES=240`. Its hosts are FakeDaemons; bcd's own volume handling is covered by its Go tests (see [the Go daemon](#the-go-daemon)). ## The Go daemon [#the-go-daemon] bcd has its own tests in `daemon/` (Go 1.26 or later) and one end-to-end suite against the real control plane. **Unit and interop tests.** `node scripts/daemon-test.mjs` runs the module's tests with the tags `nonvml,fakehw` (no cgo, no NVIDIA driver, the fake HGX node). On Linux it is `go test ./...`. On Windows, where bcd cannot run and Application Control may refuse locally built executables, it cross-compiles each package's test binary for linux/amd64 into `daemon/bin/tests/` and runs it inside WSL from the package directory, so `testdata` paths resolve. The pki tests read `daemon/internal/pki/testdata/interop.json`, which `node scripts/daemon-fixtures.mjs` writes with `@better-compute/pki` (a node certificate, a signed `JoinResponse`, a sealed JWE, signed commands, specs and a trust bundle, an exec ticket, and secret and workload-token exchanges), so Go is checked against the TypeScript implementation byte for byte. The tunnel tests run a shell on a real pseudo-terminal (a resize is seen by `stty`), and the `access` tests check the generated KRL with `ssh-keygen -Q` when `ssh-keygen` is installed. ```bash node scripts/daemon-test.mjs # every package node scripts/daemon-test.mjs --run TestSweep -v ./internal/journal node scripts/daemon-test.mjs --build-bcd # also writes daemon/bin/bcd-linux-amd64 (fakehw) ``` | Flag or variable | Meaning | | --------------------------------- | -------------------------------------------------- | | `--run REGEXP` | Only tests matching the pattern | | `-v` | Verbose output | | `--distro NAME`, `BCD_WSL_DISTRO` | The WSL distribution on Windows (default `Ubuntu`) | | `--build-bcd` | Also build the fakehw `bcd` for linux/amd64 | | `packages...` | Go package patterns (default `./...`) | **bcd helper as root.** `node scripts/helper-e2e.mjs [--distro NAME] [-v]` (da8751e) is the one test that runs `bcd helper` as real root. It builds bcd and `internal/helper/roote2e` with `-tags nonvml,helpere2e` (which gives bcd an e2e-only `helper-call`, so the node's requests come from the installed binary as the `bcd` user), renders the install script with `--unprivileged --instance-bridges bcbr0`, and runs `TestHelperAsRoot` as root under `unshare --mount --net`, over overlays of `/etc`, `/usr/local` and `/var` and a fresh `/run`, so the host is never changed (the test fails outside private namespaces). On Windows it runs in WSL as root; on Linux it needs root; elsewhere, or without `unshare` and `useradd`, it prints why and exits 0. Its steps: the script's unprivileged mode; the helper started as `bcd-helper.service` runs it; signed host units and their refusals; libvirt against a fake `virsh`; `cloud-hypervisor` with real taps (b9667d5); and since 035fc41 self-update under the helper, with the script's `[update]` pinning the two release keys of `daemon/internal/pki/testdata/release.json` (threshold 2, sequence 5). That step checks that `update.prepare` refuses a tampered manifest, another digest and a missing one; that `update.swap` refuses nothing staged, a symlink, a root-owned file and other bytes without moving `current`; that a good swap puts the release in slot b behind the entrypoint (sequence 7 recorded in `upgrade.json`), after which a second swap and confirming another version are refused; that `update.rollback` brings the installed binary back and `update.settle` records the release as failed, so preparing it again is refused; and that the slots and records are root's and not group- or world-writable, with the swap in the helper's log. **Site agent, ramdisk and instances.** The site agent and ramdisk have Go tests only; no end-to-end or interop suite runs `bcd site` or `bcd ramdisk` against the control plane yet (a VPS host has one, below). The agent tests drive a real `Agent` over a scripted session (`newHarness`), answering its `SecretRequest`s with sealed JWEs: | Test | Checks | | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `internal/agent` `TestSiteBMCAndBootEntries` | A BMC that does not answer succeeds with `ok: false` and `unreachable`; the credential is fetched for the workflow step; a driver the agent does not host is `unsupported-driver`; boot entries persist, lapse and are removed by `boot.cleanUp`; the site kinds are refused on a node | | `TestHTTPProxyRefusesTargetsOutsideTheSite` | Loopback, the metadata address, a public address and `ftp://` fail `forbidden-target`; private, CGNAT and ULA addresses are site addresses | | `TestParseCredentialForms` | JSON and bare-password BMC credentials, with the username fallback | | `TestBootServerRendersARamdiskScript` | 503 without the lease, 404 for an unknown MAC, then a script whose kernel line `bootparams.Parse` reads back, with the discovery token fetched once and an assertion signed by the agent's key at the lease epoch; 503 again once the gateway has been silent past the lease TTL | | `TestDiskWipeQuickVerifies`, `TestDeployImageStreamsAndChecksTheDigest`, `TestPickRootDiskAndDCGM`, `TestPartitionsAndBootFiles` | Quick wipes zero and verify both ends of sparse fake disks and are node-signed; a gzip image lands byte for byte with its user data; a wrong digest fails `checksum-mismatch` and leaves the disk's ends zeroed; root-disk choice, DCGM parsing, `lsblk` parsing and kernel lookup | | `TestSiteCacheWarmsAndServes`, `TestDeployImageFromTheSiteCache` | The image cache pulls `cache_images`, verifies them by digest and serves them only under a grant, `/cache/t//` (a plain digest path and another image's grant are 404; b7ed8a2); `deploy.image` takes the image from the cache | | `TestExtWriteInUserspace` (Linux, skipped without `mkfs.ext4` and `debugfs`) | On a real ext4 image with a planted `/etc/bcd/config.toml -> /dev/sdb` symlink: the ramdisk's `debugfs` writes replace the symlink, never follow it, and read back as root-owned 0600 regular files; relative paths, `..` and shell metacharacters are refused (1ae1789) | | `TestInstancesConverger` | The `instances` converger over the fake runtime | | `internal/runtime` `TestLibvirtRuntime` | The `libvirt-kvm` runtime against a fake `virsh` and `qemu-img`: the domain XML and metadata, states, grow and shrink, snapshot and clone with memory, a CPU model mismatch, destroy, a mismatched image digest (362317c). No test runs it on a real KVM host | | `internal/runtime` `TestBaseImageChecks`, `TestCheckOverlay` | Tenant base images against real qcow2 headers read in Go (933c51b, b12c217): a qcow2 naming a backing file or an external data file, and a `qcow2` that is not one, are refused and not kept; good `raw` and `qcow2` images pass; an image cached before the check is checked again when used; an instance disk naming another backing file or format than its recorded base is refused | | `internal/runtime` `TestParseRebind`, `TestRebindImage`, `internal/agent` `TestIdentityRebind` | The identity rebind (ca08d5c): bundle validation, the tar and acknowledgement region of the rebind disk, `cloud-hypervisor` and `libvirt-kvm` rebinds against an acknowledging and a silent fake guest, and the converger end to end (the secret fetch, `identity_rebound_ref` in status, persistence across a restart) | | `internal/runtime` `TestCloudHypervisorRuntime` | The `cloud-hypervisor` runtime against a fake VMM REST API and fake `systemd-run`, `systemctl`, `ip`, `qemu-img` and seed tools: define (explicit disk `image_type`s), the states, a memory snapshot, a restore into a clone with its paths rewritten, stop and destroy, and a directory that is not bcd's left alone (2598f0a); since 933c51b also the tap created and bridged before the VMM's unit starts, the unit's confinement and flavor limits, a memory snapshot of an 8 GiB VM refused in a 1 GiB flavor, a VMM that survives its stop failing `Destroy` with its directory kept, and the taps deleted on stop. No test runs it on a real KVM host | | `internal/runtime` `TestRestoredConfig` | A memory snapshot's saved configuration is rebuilt from the allowlist: affinity, memory zones, `fs`, `vsock`, `pmem`, `debug_console`, landlock rules, unknown platform keys and the snapshot's disks are dropped, and a 2-vCPU snapshot is refused for a 4-vCPU instance (933c51b) | | `internal/agent` `TestBMCThroughAHostedDriver` | A `bmc.*` operation for a driver other than Redfish goes to the hosted bcx driver with the operation's concern and name, the address as config and the credentials as secrets; a driver the agent does not host, or one without the concern, is refused and nothing is invoked (51806ab); the BMC's driver options reach the plugin as `config.options`, and a Redfish request with options answers `invalid-config` (7753df0) | | `cmd/bcx-ipmi` `TestIPMIOperations`, `TestIPMICipherSuiteOption`, `TestIPMIRefusals` | The IPMI plugin against a fake `ipmitool`: the argv with cipher suite 17 (`-C 17`), or 3 from `options.cipherSuite` (7753df0), the password read from `-f /dev/fd/3` and absent from the environment, and refusals that run nothing, among them a BMC with a key pin, a cipher suite outside 17, 12, 8 and 3, an unknown option and a password over 20 bytes (51806ab, 933c51b, 7753df0) | | `cmd/bcx-ipmi` `TestIPMIInspect` | `inspect.inspect` against a fake `ipmitool`: the Redfish driver's shape from `fru print 0`, `mc info`, `lan print` and `sdr list` (7818f19) | | `internal/selfupdate` `TestManagerNeverSwapsBelowTheFloor`, `TestManagerStagedFileChecks` | A lying `Settle` right after a swap and a swap below the floor under any claimed version are refused; a FIFO (no hang), a hard link, a foreign owner and a symlinked staging directory at the staged path are refused; the slots directory is 0755 (c139b9c) | | `cmd/bcd` `TestPCIAddress`, `TestDoctorToolChecks` | NVML bus ids normalized to libvirt's PCI addresses for VFIO; `bcd doctor`'s VPS host tool checks, for a `libvirt-kvm` host (4a0165f) and a `cloud-hypervisor` one (2598f0a) | | `internal/redfish` `TestRedfishOperations` | Every operation against a fake Redfish service over TLS pinned by SPKI; after a rotation the old password is `auth-failed`; an unknown operation is `unsupported`; a wrong pin and a plain-http BMC are refused | | `internal/pxe` `TestProxyDHCPAnswersPreparedMachines`, `TestTFTPServesFilesUnderItsRoot` | On real loopback UDP sockets: PXE firmware of a prepared MAC gets `ipxe.efi` with the client's UUID and architecture read, iPXE gets the chain URL, an unprepared MAC gets no answer; TFTP serves a file from its root | | `internal/bootparams` `TestRenderParseRoundTrip`, `cmd/bcd` `TestRamdiskNeedsBootParameters` | The kernel command line round-trips; `bcd ramdisk` without its parameters exits 2 | | `cmd/bcd` `TestSimNeedsNodesAndAConfiguration` | `bcd sim --nodes 0` exits 2; `bcd sim` with a missing configuration file exits 1 | **Host configuration, holds and Kubernetes (v1.1).** These run against fakes: `hostconfig.FakeSystem` (an in-memory host whose files and command answers a test scripts), a fake root for a real `Agent`, and a fake kubelet on real unix sockets. Only `TestRealKernel` touches a real kernel. | Test | Checks | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `internal/hostconfig` (22 tests) | Every config unit's plan and apply, drift, cleanup, refusals and reboot cases: sysctl, kernel modules before the sysctl they provide, huge pages with the mount unit, the GRUB and grubby fallbacks and per-NUMA pools, `cpu`, `iommu` and `gpu-settings`, the NVIDIA driver installed or asking for a reboot, the toolkit's CDI regeneration, Fabric Manager and `FabricManagerDown`, the MIG layout grammar and busy GPUs, MPS and `MPSDown`, unknown units and a unit waiting for a reboot not holding back the others (6c2343f, c21ef84). See [host configuration](/docs/daemon/host-config#what-the-tests-cover) | | `internal/hostconfig` `TestRealKernel` | As root with `BCD_HOSTCONFIG_REAL=1` on Linux: sysctl, the transparent huge page selector and 2 MiB huge pages against the real kernel, each put back, and root refusing to write under `/tmp`, a world-writable or a symlinked parent (60143bf, 02485b3). Passes on WSL Ubuntu | | `internal/agent` `TestHostSectionConverges`, `TestHostSectionReportsRebootRequired`, `TestHostDriftRepairedWithoutANewSpec`, `TestSignedHostSectionKeptForTheHelper`, `TestExternalHostAppliesNoUnit`, `TestGPUBusy` | The host section through a real `Agent` under a fake root, with `FabricManagerDown` and `RebootRequired` in the status report, drift repaired by the 10-minute pass (3c0fe02), the signed host section kept for the helper, `[host] units = false`; a GPU is busy when a hold's atom id names it (0f68c9a) | | `internal/helper` `TestHelperHostUnit`, `TestHostUnitNeedsTheSignedSpec`, `intent_bundle_test.go` | `host.unit` through a real helper socket: re-planned as root with the unit's own validation, refused keys, names and unknown units, cleans, busy GPUs, an unconfigured helper (e0fe5ec), and a helper whose own `config.toml` says `[host] units = false` refusing the unit with nothing written (3fa9021; `cmd/bcd` `TestRootOnlyFile` checks the root-owned config rule); the signed spec under pinned roots (03c023d) and under the trust bundle the join key signed, with a root rotation, a replayed bundle, another key's bundle and a rootless one (2d18327) | | `internal/agent` `TestHoldsAndVMsShareTheHostsMemory`, `TestAHoldTheHostCannotFitFailsAsResourcesExhausted` | Host admission of holds beside VMs, and `resources-exhausted` (31555bf) | | `internal/agent`, `internal/runtime` volume tests | Persistent volumes: sparse, grown and never shrunk, discarded; attached in order by serial; one writer, handoffs within a pass, what running VMs have open; grown volumes told to running guests (550da91, b5415a2, 448dc5f, 39458f9, 408e002) | | `internal/kube` `TestHostedManager`, `TestDRADriver`, `TestResourceSliceV1beta1`, `TestUnhealthyDevicesAreNeverOffered` | Hosted mode's device plugins and `bcd.dev/*` labels without a join (1e5f96b); the DRA driver's registration, slice, prepare and reaper (7d24836, fa8611e); an unhealthy GPU never offered by either API (6f5a99f) | | `cmd/bcd` `TestResourceChecks`, `TestHelperRuntimeChecks` | `bcd doctor`'s cgroup, huge page, `vfio-pci` and systemd checks (ab26e87), and the `libvirt` and `cloud-hypervisor` checks under the helper: the node out of the `libvirt` group, `dynamic_ownership = 0` and QEMU's user in the node's group (ed3c7d2), and `cloud-hypervisor` warning without `[helper] instance_bridges` (7fe74a1) | | `cmd/bcd` `TestDoctorTSS`, `TestJoinReset` | `bcd doctor`'s `tpm` check: silent without `[helper]`, a failure naming `gpasswd -d bcd tss` for a node user in `tss`, silent outside it or without a `tss` group (e6c3f53); `bcd join --reset`: nothing to move lets the join go on, an unconfirmed reset moves nothing and exits 2, a confirmed one renames `identity.json` and `node.key` to `.reset-` names, after which no identity loads (a1c1a46) | | `cmd/bcd` `TestAnUnprivilegedNodesRuntimes`, `internal/helper` `ch_test.go`, `internal/runtime` `TestCloudHypervisorUnderTheHelper`, `TestCHRecordRefusals` | `cloud-hypervisor` under the helper against fakes (d86fcbe, 7fe74a1, 94843ec): both runtimes kept for an unprivileged node, taps only of bcd's names on allowed bridges, a launch never taking another running unit's uid, VFIO only for whole groups of this host's GPUs and never a group another running VMM's user owns, root's checks of the instance record; since 1c9684a the running domain's live usage (CPU time, memory, tap counters) read from its cgroup and taps as the node, with no `ch.*` operation | | `internal/agent` `TestNodeKeyInTheTPM`, `internal/tpm` `TestNodeKey` | The node key in `tpmsim` (965c14d, 50e18f9, 0ba2a79): a join stores only the record, the key loads again, node statements verify under the certificate's key and `kid`, renewal makes the next key in the TPM (and moves a file-key node there), no TPM means no load; SHA-256 and -384 signatures, a CSR, `TPM2_Certify` by the AK. `cmd/bcd` `TestUserCanOpen` went with the check it tested (8870b8d) | | `internal/helper` `TestHelperTPMKeys` | An unprivileged node's TPM node keys through the helper's `tpm.*` operations against `tpmsim` (8870b8d): the same key and AK as the TPM's own, signing, certifying, at most 8 keys loaded with the least recently used unloaded, a released key derived again, and refusals of bad shapes, unknown verbs and a helper without a TPM. `tests/cli-pki.test.ts` asserts the install script no longer adds `bcd` to `tss` and removes it on upgrade | | `internal/agent` `TestRenewCertifiedByTheAK` | A certificate naming the TPM's AK (`urn:bc:tpm-ak:`) renews twice with the next key made in `tpmsim` and certified by that AK over `sha256(csr)` in `key_attestation`; with no TPM, or another TPM's AK, the renewal fails before any request, never with a file key (d94f1f3). `tests/server-join-tpm-renew.test.ts` is the control-plane side: the statement carried through renewal and every refusal. See [renewing a TPM-held key](/docs/guides/nodes/attestation#renewing-a-tpm-held-key) | | `internal/agent` `TestResolveAtomNamesParentsAndWhatTheDriverSees`, `TestHoldConflictsAcrossWholeGPUsPartsAndVMs`, `TestCDISpecOfPartsAndLimits` | MIG and slot atoms resolved against the inventory, parent-aware conflicts and the VM passthrough guard, CDI names without `/`, the slice annotation and env, one `NVIDIA_VISIBLE_DEVICES` per hold (46a61db, d02ba4f) | | `internal/slice`, `internal/agent` slice tests | The hold slice's systemd properties and hugetlb limits, the name and cpulist checks, removal only once empty (1d03c28, 260e33c) | | `internal/kube` `TestJoinFromBareHost`, `TestJoinRefusesAnotherCluster`, `TestJoinSettingsAreCheckedAgain`, `TestJoinUnderHelper`, `TestContainerdImport`, `TestJoinConfigurationV1beta3` | The `k8s-join` unit against `FakeSystem` (88ecb07) | | `internal/kube` `TestDevicePlugin`, `TestResourceOf`, `TestWireMatchesUpstream` | The device plugin against a fake kubelet: registration, ListAndWatch with health and NUMA, Allocate and its refusal of an unbound device, preferred allocation, the reaper's grace period, re-registration after a kubelet restart; the hand-encoded messages against the upstream descriptors | | `internal/agent` `TestHostAdmissionCountsCommittedMemoryNotUse`, `TestHostAdmissionTakesHugepagesFromThePlacementNodes`, `TestStartsOnlyWhatIsNotAlreadyRunning` | Host-side admission of instance starts (0dd122a) | | `internal/agent` telemetry tests, `internal/telemetry` | The OTLP exporter against a fake collector (27c7ee2) | | `internal/runtime` netconfig tests | The NoCloud `network-config` for leased NICs (0791fc9) | | `cmd/bcd` `TestKubernetesDaemonSetContract` | `@better-compute/provider-kubernetes`'s generated `config.toml` and command lines parse with bcd's own parser and flags (2c9f953) | The FakeDaemon holds `BINDING_KUBERNETES` assignments like bcd since 4e61f59: its CDI spec carries `bcd.dev/binding: kubernetes` and no `NVIDIA_VISIBLE_DEVICES`, while a plain hold's spec sets it once for the whole hold (`tests/testing-fake-daemon.test.ts`). **End to end.** `tests/e2e/bcd.test.ts` runs the real binary (linux/amd64, fakehw, `BCD_FAKEHW_SEED=41`) against `createDevStack({ externalTls })`: the dev stack's mTLS listener with a dev serving CA, which the test pins in a generated `config.toml` together with the join keys, `gateway_server_name = "gw.dev.local"` and a data directory under `/tmp`. It checks that: * `bcd join` with a sealed token on standard input succeeds and never prints the token; * `bcd node` is adopted after its node-signed `diag.nvml`, and `bcd status --json` answers on the socket; * `inventory.refresh` succeeds; * a 2-GPU reservation turns active on a node-signed `AllocationBound` (checked by `pkiAllocationBindingCheck`), its CDI hold appears under the fake root, and it disappears on release; * a node console opened with `tunnels.open` (op `console`) through the dev stack's tunnel relay and gateway reaches a real shell in bcd: `echo hi-$((6*7))` answers `hi-42`, and `exit 5` comes back as exit code 5; * `node.reboot` completes `SUCCEEDED` from the next boot, with the new boot id recorded in `node_sessions`; * a provisioned OS bootstraps with nothing but a token file, the control plane URL and the three pins (what provision's user data and the Robot bootstrap run): `bcd join --server` reads the real server's well-known document, joins through the real gateway and JoinService, removes the token, writes `config.toml` and `join-encryption.jwk`, succeeds again on a re-run without a token, and `bcd node` connects on the written configuration alone (472a424); * `bcd sim --nodes 5 --seed 100`, given one five-use join token on standard input, joins five simulated machines that connect as five distinct nodes (five distinct hardware keys), and never prints the token (aa7501c; see [bcd sim](/docs/daemon#bcd-sim)); * self-update (fc0ca6a): a second fakehw build reporting 0.9.0 is released under a key pinned in the node's `[update]` section; the node restarts from an entrypoint it may replace, gets a `daemon.upgrade` enqueued as the rollout controller does, downloads and verifies the release, swaps to slot `b`, execs, and is confirmed by its `Welcome`: `node_sessions.daemon_version` is 0.9.0, `current` is `bcd-b`, and `upgrade.json` is gone. * the WebSocket fallback (4b2d5aa): the TypeScript `serveGatewayWs` runs beside the stack's gateway with a serving certificate from the CA the node pins and audience `gw.dev.local`; the node restarts with `[websocket] url`, runs a signed `inventory.refresh` over the WebSocket session, and renews over `POST /v1/renew`, which the fakehw knob `BCD_RENEW_AFTER=3s` brings forward. * node metrics (efdff1d): a test module sets `NodeSpec.telemetry` (http to the WSL host, a 5-second interval and an extra resource attribute), the node's spec is rendered again, and the fakehw bcd posts OTLP/JSON to a local receiver in the test: `bcd.gpu.utilization` for all 8 GPUs, memory, temperature, power and active XIDs, with `service.name=bcd`, `bcd.node.id` and the attribute as resource attributes. The receiver is the test's own, not a collector product. `tests/e2e/bcd-host.test.ts` (91f5e81, same gating) runs `NodeSpec.host` end to end: a `metal-host` pool with `policy.hugepages` (2 MiB, 64) and `policy.kernelModules` (`tun`, `vhost_net`), provision renders the units, the gateway sends them, and the fakehw bcd applies them under its fake root. `tests/e2e/bcd-placement.test.ts` (427787a, same gating) runs PLAN.md §3.8 against fakehw bcd nodes, in 6 tests: a pinned, NUMA-affine flavor (`vcpupin` and a strict `numatune` in the libvirt domain the fake runtime plans, within one NUMA node and off the host-reserved CPUs), a 2 MiB huge-page flavor on the pages provision's unit reserved (page backing, locked, no balloon), an overcommitted pool's guest ballooned to its floor under simulated memory pressure and deflated again, a sandbox volume attached by its serial that outlives the sandbox, an instance on a host without the `firecracker` plugin label staying unschedulable, and a GPU hold with booked vCPUs and memory in its `bcd-alloc-.slice`, reported as `binding.cgroupParent` and removed on release. It found that a re-collected inventory waited up to ten heartbeats for its hashes, fixed in the same commit. It relies on two fakehw features: the runtime writes the libvirt XML it would define to `/fakehw-plans/.xml` and keeps its balloons on disk (9a6aaf4), and the inventory reads the huge pages the host units reserved back from the fake root (cd7a311); see [local development](/docs/operations/local-development). `tests/e2e/kubernetes-bindings.test.ts` (8a084ca) runs `binding: 'kubernetes'` on the dev stack's FakeDaemons: the hold marked `bcd.dev/binding=kubernetes` without `NVIDIA_VISIBLE_DEVICES`, a plain hold on the same pool not, the booked limits in the assignment, release, and the refusal on a pool without `policy.kubernetes`. `tests/vps-sandboxes.test.ts` checks that a sandbox lands only on a host labelled `bc.dev/cap.instance.ssh-host-only` (3b9e81d), and `tests/vps-volumes.test.ts` covers sandbox volumes: booking, quota, replay, pinning, serial mounts, one writer, read-only sharing, resize, deletion and metering (2a09202, 8cbece0). `tests/metal-discovery-e2e.test.ts` (452d6bf, not gated: FakeDaemons only) runs [PXE discovery](/docs/guides/bare-metal/sites#discovering-new-machines) through the real control plane and gateway, with a FakeDaemon site agent carrying metal's site kinds: an IPMI machine enrolled with only its product UUID, `metal.ensureRamdisk` setting PXE and power-cycling it through the agent, the agent reporting it with `setDiscovered` in DHCP's mixed-endian UUID spelling, metal's module status handler (composed into the dev stack's gateway) storing the report, and `metal.discovered` showing the MAC as the machine's candidate with `discovered_mac` set, `macs` still empty and no boot entry sent for 2 seconds. Only after `metal.confirmMac` does the agent get `boot.prepareRamdisk` for exactly that MAC, with a `secret://` discovery token reference and no sanboot digest. The dev stack has no ramdisk to join, so it ends at the boot entry; `tests/metal-discovery.test.ts` covers the rest on the sim. The whole file runs on `createDevStack({ signedIntent: true })` (5071dbc), which puts the stack under a command-signing root: joined daemons receive the roots in `JoinResponse.trust` and every command and spec bcd takes is signed and verified. With `bcd-vps.test.ts` that is 9 of 9 bcd end-to-end tests. `tests/e2e/bcd-vps.test.ts` runs the same fakehw binary as a VPS host: bcd joins a `vps-host` pool over the dev stack's mTLS listener, `instances.create` places an instance on it, the `instances` converger defines and starts it on the fakehw build's file-backed `libvirt-kvm` runtime (not the libvirt runtime of host builds), `StatusReport.instances` carries it and the vps module moves it to `running`, and `instances.delete` removes the domain from the host. It has the same gating. `tests/e2e/bcd-tpm.test.ts` (db9bd46, 8a9af54, same gating) runs the `tpm` join end to end: a fakehw bcd whose software TPM lives in `/fakehw-tpm.json` runs `bcd join --attestor tpm` with no token against the dev stack's JoinService with `tpmAttestor` (`createDevStack({ attestors })` passes join attestors to `betterCompute()`). Since 8a9af54 the test makes a manufacturer root, the fakehw TPM carries an EK certificate it issued (`BCD_FAKEHW_TPM_EK_CA`), and the attestor trusts only that root (`ekRoots`, no `allowUncertifiedEk`, and a `resolve` allow-list for the TPM pool); with `BCD_FAKEHW_TPM_KEYS=1` the node key is made in the same TPM. It checks the two rounds, `resolve` seeing the EK certificate chained to the manufacturer and the SHA-256 PCR bank 0 to 7, the node in the resolved pool and bound to `sha256(EK)` in `node_bindings`, `node.key` holding only a `BCD TPM NODE KEY` record, the node key certified by the TPM's AK so that both joins' `node.joined` events say `nodeKey: 'tpm'` (cf24876; the attestor keeps its default `nodeKey: 'certified'`), a rejoin as the same node after the identity is removed (a reinstall), `already joined` on a further run, and `bcd node`, whose session and statements are signed in the TPM, adopted on that identity. Since d94f1f3 it also checks that the certificate names the TPM's AK (`urn:bc:tpm-ak:`), and that `bcd node` (with `BCD_RENEW_AFTER=3s`) renews with the AK certifying the next key: one more `node_certs` row, still carrying the statement, and `node.key` still a TPM record. Its wire format is pinned in both directions without a stack too: bcd's `TestTSContract` and `tests/pki-attestors-tpm-bcd.test.ts` share the fixtures in `daemon/internal/tpm/testdata`, which `scripts/daemon-tpm-fixture.mjs` regenerates (see [bcd with its TPM](/docs/guides/nodes/attestation#bcd-with-its-tpm)). Since b973e05 those fixtures include a test manufacturer root (`ek-root.der`) and the EK certificate it issued (`ek-cert.der`) in the sim's NV, so that pair runs `tpmAttestor` with `ekRoots` and no `allowUncertifiedEk`, and checks that another root is refused (`ek-certificate-chain`). Since cf24876 the fixture's answer also carries the AK's certification of a node key in the sim, which the TypeScript side admits as `nodeKey: 'tpm'` and refuses for another CSR key (`node-key-not-csr`); `tests/pki-attestors-tpm.test.ts` covers the same checks with the TypeScript software TPM, and suites whose software TPM holds no node key opt into `nodeKey: 'any'`. None of these touches a hardware TPM. On Linux the suites run in `pnpm test` whenever Go is installed; they build the binary first. On Windows they are opt-in: they need `BCD_E2E=1`, a WSL distribution, and Go at `C:\Program Files\Go\bin` (or an existing `daemon/bin/bcd-linux-amd64`). bcd runs in WSL through `wsl -e`, never a shell, and reaches the gateway at WSL's default route. `BCD_E2E_LOG=1` streams bcd's log to standard error. ```powershell $env:BCD_E2E = '1'; pnpm vitest run tests/e2e/bcd.test.ts ``` ## The fleet simulator [#the-fleet-simulator] `createFleetSim(options)` runs N FakeDaemons, a fresh faulty link per dial, a seeded chaos schedule and a pluggable `GatewayPort`. The default port is `GatewayStub`, a minimal in-memory control plane (Hello, Welcome, dispatch with credits, result ingestion, liveness, the command sweeper, bookings under an exclusion check, usage flushes). The stub is a test double for the daemon side, not a reference for the server. ```ts import { assertInvariants, createFleetSim } from 'better-compute/testing'; const sim = createFleetSim({ seed: 'fleet-100', nodes: 100, up: { latencyMs: 5, jitterMs: 20, drop: 0.0005, dropMode: 'reset' }, down: { latencyMs: 5, jitterMs: 20, reorder: 0.002 }, chaos: { everyMs: 5_000, crash: 0.3, reboot: 0.1, journalLoss: 0.05, partition: 0.3, gatewayKill: 0.02 }, }); await sim.start(); await sim.run(10 * 60_000); // ten simulated minutes sim.quiesce(); await sim.run(16 * 60_000); assertInvariants(sim.invariants()); console.log(sim.digest()); // the same seed gives the same event log, byte for byte ``` | Option | Default | Meaning | | --------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `seed` | required | Everything random derives from it | | `nodes` | 10 | A count, or one `{ nodeId?, hardware?, role?, daemon? }` per node | | `start` | 2026-01-01T00:00:00Z | Sim start time | | `gateway` | a `GatewayStub` | A factory for another `GatewayPort` | | `up`, `down` | none | Link faults in each direction | | `daemon` | none | FakeDaemon options for every node | | `tickMs` | 1 s | Period of `GatewayPort.tick` | | `startSpreadMs` | 5 s | Daemons start at random times within this window | | `logFrames` | true | Log every frame delivery, drop and duplicate | | `chaos` | none | `everyMs`, `forMs`, and per-draw probabilities `crash`, `reboot`, `journalLoss`, `partition`, `isolate`, `halfOpen`, `reset`, `stall`, `gatewayKill`, with their duration ranges | The simulator also exposes each fault as a method (`crash`, `reboot`, `loseJournal`, `partition`, `isolate`, `heal`, `halfOpen`, `resetLink`, `stall`, `killGateway`), `every(ms, fn)` for workloads, and `rng.fork(label)` for independent seeded streams: adding a node never shifts another node's draws. `better-compute sim` is this simulator with a built-in workload; see [local development](/docs/operations/local-development#the-fleet-simulator). ## Invariants [#invariants] The checkers are pure functions over collected facts, so the same checks run over the fleet simulator, the real control plane or a database snapshot. | Id | Invariant | Checker | | -- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | I1 | No atom is booked by two allocations at one instant, and no node holds one device for two allocations at once (and only binds what its allocation booked) | `checkI1` | | I2 | A non-retry-safe or self-terminating command starts its side effect at most once; a retry-safe one re-runs only after a crash interrupted it; after a journal loss, in-flight non-retry-safe commands fail rather than replay | `checkI2` | | I3 | Every `DesiredState` a daemon received is reported observed (a status report at that generation or later) within `CONVERGENCE_MS` (60 s), judged only while a gateway heard from the node at least every 15 s and only once the window has passed | `checkI3` | | I4 | No node is marked `unreachable` or `lost` within `LIVENESS_WINDOW_MS` (20 s) of a gateway receiving a frame from it; the times come from the transport on the simulator's clock, so gateway clock skew cannot hide a false positive | `checkI4` | | I5 | Every release has an allowed cause (end of booking, release, cancel, lapsed lease, failed bind); none is caused by a disconnect | `checkI5` | | I7 | After a quiet period, no pending reservation would fit if planned alone now; the input is capacity's offline solver, `placeablePending(tx, { now })` | `checkI7` | | I8 | Per allocation, usage windows do not overlap, stay inside the booking and add up to its duration times atoms, with no duplicate keys | `checkI8` | Two more I1 checkers (40b1ce8) cover PLAN.md §3.8's host resources and Kubernetes bindings: | Checker | Checks | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `checkHostResources(samples)` | Over sampled `node_resources` rows (`HostResourceSample`: vCPU, memory, huge pages), `allocated` never exceeds `floor(allocatable x ratio)` and always equals what the node's unreleased allocations claim, so a release that subtracts twice or a re-book without a claim shows as drift | | `checkKubeAdvertisements({ bookings, advertisements })` | No device is offered to a node's kubelet for one allocation while the control plane books it to another, since a kubelet can hand an offered device to a new pod at any moment. The offers (`KubeAdvertisement`) come from `FleetFacts.advertisements`: a FakeDaemon records `kube.advertised` when it offers a Kubernetes-bound hold and `kube.withdrawn` at its release | With the knob `kubePodReleaseMs`, a FakeDaemon's pods keep a released Kubernetes hold's devices that long: it stops offering them at once, keeps them claimed (a new hold on them fails with overlap) and reports `AllocationReleased` only then, as bcd's fail-closed release does. `assertInvariants(violations, limit?)` throws one error listing them. The fleet simulator's `invariants()` checks I1, I2, I3, I4, I5 and I8 (I4, I5 and I8 read the gateway port's `facts()`, which the stub gateway and the control-plane simulator provide). I7 needs a database: pass it to `runNightly({ invariants })`, as the CI-sized night does. PLAN.md section 15 also names I6 (latency SLOs); the kit does not check it, and the real-time benchmarks measure I6-style latencies instead. I4 is also tested under gateway clock skew, which section 15 calls for: `tests/sim/clock-skew.test.ts` runs 9 nodes on 3 real gateways whose own clocks are 90 s ahead, 90 s behind and on time (more than the 40 s liveness timeout), with `compute_now()` and the daemons on simulated time. Every gateway holds sessions, `diag.nvml` settles through all of them, and when the unskewed gateway dies its nodes re-home onto the skewed ones; connectivity sampled every simulated second never shows `unreachable`, no `node.unreachable` is published, and I1 to I5 and I8 hold, because gateways flush ages rather than timestamps. ## The control-plane simulator [#the-control-plane-simulator] `createControlPlaneSim(options)` puts the real control plane and real gateways behind the fleet simulator: `betterCompute` on the sim clock with `compute_now()` pinned to simulated time, `gateways` real gateways on manual jobs (default 1), and nodes that join through JoinService with sealed tokens bound to their CSRs. It sets up a site and an H100 pool with `diag.nvml` acceptance offered to `acme`; reconcilers and gateway jobs run on the simulator's ticks. | Option | Meaning | | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `seed`, `nodes` (default 4), `variant`, `start` | As for the fleet simulator | | `gateways` | Gateway processes behind the simulator (default 1) | | `iam` | Wraps the fake IAM, for example `(fake) => spyIamPort(fake)` | | `trace` | Runs each part of the loop under a name (`join`, `fleet`, `gateway`, `reconcilers`, `api`), for an IAM spy's `path` | | `restricted` | The last `count` nodes join with a certificate that allows only `allowedKinds` | | `modules` | Extra control-plane modules, ticked with the rest | | `gateway` | Extra gateway options, such as lease timings | | `workerPasses` | Batches each control-plane job may run per simulator tick while it keeps finding work, as a real worker loops without sleeping (default 1) | | `gatewayClockSkewMs` | One number per gateway slot: that gateway's own clock offset from simulated time in ms (positive is ahead); the database clock and the daemons stay on simulated time | The rig offers `run(ms)`, `api(fn)`, `killGateway(i)`, `reviveGateway(i)`, `stallGateway(i, stalled)`, `pauseWorkers(paused)`, `revokeNode(nodeId)`, `homeOf(nodeId)`, `serialOf(nodeId)` and `invariants()`, which reads the facts from compute's own tables (bookings, billed allocations and usage windows, command outcomes, connectivity) and checks I1, I2, I3, I4, I5 and I8. `tests/sim/multi-gateway.test.ts` uses it for a gateway killed mid-command and a zombie gateway fenced by its session epoch. `tests/sim/kubernetes-host-resources-1k.test.ts` (246c95b) churns Kubernetes-bound and hold reservations with vCPU and memory shares over a GPU pool of `SIM_K8S_NODES` hosts (default 1,000) under link faults, crashes, partitions and resets, re-inventories hosts with less CPU and memory and then restores them, and has pods keep released Kubernetes GPUs for 45 seconds. After every step it asserts `checkHostResources` and the fleet invariants; the recorded run at 1,000 nodes is green in about 4.5 minutes. A second, strict 12-node run of `checkKubeAdvertisements` checks that no kubelet is offered a GPU booked elsewhere. It used to fail (capacity freed a Kubernetes-bound GPU at release and re-booked it before pods let go, so the node rejected the new binding: 55 of 240 reservations at 12 nodes); since e7d922c a Kubernetes booking ends at the node's `AllocationReleased`, and the run passes with no failed reservations. ### The nightly simulation [#the-nightly-simulation] `runNightly({ seed, nodes?, gateways?, minutes?, chaos?, stepMs?, modules?, inspect?, invariants? })` runs one simulated night on the control-plane simulator: three minutes of start-up, then `minutes` (default 30) of tenants reserving with every placement feature (half and whole nodes, gangs of two, spot, calendar starts, early releases) and operators running commands, under chaos. Chaos adds gateway kills and revivals, worker outages and, at `heavy`, one certificate revocation per night. `invariants(rig)` adds checks run after the quiet period, such as I7 with capacity's `placeablePending`. The report has the invariant violations, `kindLeaks` (commands queued outside a restricted certificate's kinds, which must be 0), place-to-active and command latencies in simulated time, and two digests: the workload (a function of the seed) and the id-free outcome. Compute mints random ids, so outcomes can differ between runs of one seed. ```bash pnpm sim:nightly SIM_NIGHTLY_SEEDS=1,2,3,4 SIM_NIGHTLY_CHAOS=heavy SIM_NIGHTLY_NODES=32 SIM_NIGHTLY_MINUTES=120 pnpm sim:nightly ``` `pnpm sim:nightly` (`vitest.nightly.config.ts`, `tests/nightly/*.nightly.ts`) runs every seed times chaos level one after another, with `auditModule` checkpointing every 10 simulated minutes and every tenant's audit chain verified at the end. Defaults: seeds `1,2,3`, chaos `light,heavy`, 16 nodes, 3 gateways, 60 simulated minutes; reports land in `bench/sim-nightly/results`. A CI-sized night (4 nodes, 2 gateways, 6 minutes) runs in `tests/sim/nightly.test.ts`; both check I7 through `invariants`. The v0.4 exit asks for every section 15 scenario green on three consecutive nights; the `nightly` CI workflow counts them (see [CI](#ci-and-the-repository-checks)). ### Cold start [#cold-start] `runColdStart({ seed, nodes?, gateways?, startSpreadMs?, limitMs?, settleMs?, restart?, restartDownMs?, stepMs?, workerPasses? })` is the v0.4 "10k cold start in 3 minutes" scenario on the control-plane simulator. The fleet powers on at random times within `startSpreadMs` (default 30 s) behind `gateways` (default 3) gateways; each node's first session (`node.connected`) and its adoption (`node.adopted`) are timed from power-on, in simulated time, from the outbox. The run is ready when every node is connected, adopted and schedulable, and it must be ready within `limitMs` (default 3 minutes). With `restart` (default true) every gateway then goes down and every node power-cycles, and the way back is timed the same way. The invariants I1 to I5 and I8 are checked at the end, and a `node.unreachable` mark during boot fails the run. Defaults: 100 nodes and `workerPasses` 10. The report has per-phase latency summaries, `unreachableMarks`, `schedulable`, the violations and `wall.loops`, the wall time spent in each part of the loop. | Where | Size | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `tests/sim/cold-start.test.ts` | 24 nodes, in CI: boot and power cycle each within 3 simulated minutes, nothing marked unreachable | | `tests/nightly/cold-start.nightly.ts` (`pnpm sim:nightly`) | `COLD_START_NODES` (default 10,000), `COLD_START_GATEWAYS` (3), `COLD_START_SPREAD_MS` (30,000), `COLD_START_SEED` (`1`); writes a report to `COLD_START_OUT` (default `bench/cold-start/results`). The recorded 10k run took 641 s of wall time on a 64-vCPU workstation | The recorded 10k run (`bench/cold-start/README.md`) was ready at 80 simulated seconds: every node connected by 30 s (p99 29.7 s, bound by the power-on spread), adopted by 79 s and schedulable at 80 s, and every node was back 40 s after the power cycle, with 0 unreachable marks and 0 invariant violations. Run it alone with: ```bash COLD_START_NODES=10000 node node_modules/vitest/vitest.mjs run --config vitest.nightly.config.ts tests/nightly/cold-start.nightly.ts ``` See [scaling](/docs/operations/scaling#cold-start) for the 1k and 10k results side by side. ## Real-time harnesses [#real-time-harnesses] The fake clock never measures latency. The real-time harnesses run on the wall clock over cleartext HTTP/2 (`createH2cClient`), against PGlite by default or a Postgres you name with `postgresUrl` (it must be disposable: the harness migrates it). * **`createRealtimeHarness({ nodes?, postgresUrl?, gateway?, services?, daemon?, sharedConnection?, pollMs? })`**: a real gateway and database with FakeDaemons on pre-registered nodes (default 50). `connectAll()`, `dispatch({ kind, perNode })` (enqueue to acknowledgement and to terminal), `load({ ratePerSecond, durationMs })` (steady state) and `storm({ fraction, mode: 'disconnect' | 'crash' })` (reconnect storms). * **`createRealtimeFleet({ postgresUrl?, variant?, pollMs?, worker?, gateway?, daemon?, joinConcurrency? })`**: the whole v0.1 spine, a started control plane with JoinService on real PKI and a gateway wired to its services. `join(n)` times each node from its Join call to adopted; `place(n, { gpus?, ratePerSecond?, concurrency? })` times reservations from placed to active. * **`probeLeases(sql)`** samples the `leases` table: renewal gaps, the least headroom before expiry, lapses while owned, and takeovers. Every result is a `LatencySummary` (`count`, `p50`, `p90`, `p99`, `max`). Small runs are in `tests/realtime/`; the 1k runs belong to the benchmarks. ## Postgres-only suites [#postgres-only-suites] Everything about locking and fencing needs real PostgreSQL: PGlite is one serialized connection. These suites skip unless `BETTER_COMPUTE_POSTGRES_URL` is set: ```bash BETTER_COMPUTE_POSTGRES_URL=postgres://postgres:postgres@localhost:5432/compute_test pnpm test:postgres ``` `pnpm test:postgres` runs `tests/postgres-conformance.test.ts` (the full store conformance suite, concurrency cases included, each case in a fresh schema; the database needs `btree_gist` or the right to create it) and `tests/postgres-schema.test.ts`. CI runs them against a `postgres:17` service. The benchmarks read the same variable. ## Benchmarks [#benchmarks] `pnpm bench:scale` runs `tests/bench/**/*.bench.ts` with `vitest.bench.config.ts` (the test aliases, a 30-minute timeout), on demand only. `gateway-streams` sets its own timeout, `BENCH_GW_DURATION_MS` plus 20 minutes, so the 30-minute gate run with its ramp, storm and settle is not cut off. Each suite writes `bench//results/--.json` with the commit, the machine, the Node.js version and the configuration. Commit only runs on the reference hardware of `bench/REFERENCE.md`, or runs labeled `referenceHardware: false`. | Suite | Measures | Knobs | | ----------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `gateway-streams` | Sessions one gateway holds at 10 s heartbeats, dispatch latency, RSS, a reconnect storm | `BENCH_GW_SESSIONS` (5000), `BENCH_GW_DURATION_MS`, `BENCH_GW_COMMANDS_PER_S` (50), `BENCH_GW_STORM`, `BENCH_GW_TLS=1` (mTLS, one connection per daemon), `BENCH_GW_SPLIT=1` (the gateway and its store in their own process), `BENCH_GW_GATE=1` | | `iam-lane` | Embedded better-iam admissions on adapter-postgres, lock wait | `BETTER_COMPUTE_IAM_POSTGRES_URL`, `BENCH_IAM_CONCURRENCY` (`1,4,16,32`), `BENCH_IAM_DENY_SHARE`, `BENCH_IAM_GATE=1` | | `placements` | Placements/s and lock hold, inline and pool-lease | `BENCH_PLACE_NODES` (1000), `BENCH_PLACE_COUNT` (300), `BENCH_PLACE_GPUS` (1), `BENCH_PLACE_CONCURRENCY` | | `realtime-fleet` | Command dispatch to ack and terminal, steady load, reconnect storms | `BENCH_RT_NODES` (1000), `BENCH_RT_PER_NODE`, `BENCH_RT_LOAD_RATE`, `BENCH_RT_STORM`, `BENCH_RT_GATE=1` | | `realtime-spine` | Join to adopted, place to active on the whole spine | `BENCH_SPINE_NODES` (1000), `BENCH_SPINE_RATE` (5), `BENCH_SPINE_RESERVATIONS` (200), `BENCH_SPINE_BURST` (100), `BENCH_SPINE_GATE=1` | | `multi-gateway` | Re-homing after a gateway kill, in simulated time | `BENCH_MG_NODES` (1000), `BENCH_MG_GATEWAYS` (3), `BENCH_MG_GATE=1` | | `postgres-spine` | The spine end to end on Postgres (skipped without a URL) | `BETTER_COMPUTE_POSTGRES_URL` | The numbers measured so far, and what they mean, are on [scaling](/docs/operations/scaling). ## CI and the repository checks [#ci-and-the-repository-checks] Four GitHub Actions workflows are defined under `.github/workflows/`. This page describes what they run, not whether they are green. | Workflow | Runs on | Does | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `ci.yml` | every push to `main` and pull request | On Ubuntu and Windows with Node 22 and 24: `pnpm check`, `pnpm proto:check`, `buf breaking` (Ubuntu, Node 24), `pack:all`, `check:packed-types`, `test:packed`. A job with a `postgres:17` service runs `pnpm test:postgres`. After both, on `main`, the publish job publishes a version that is not on npm yet with `pnpm publish -r --provenance`, then tags the release and dispatches `release.yml` | | `nightly.yml` | 02:17 UTC every day, or by hand | `sim`: `pnpm sim:nightly` with the 10k cold start (`COLD_START_NODES=10000`), results uploaded. `guarded`: `pnpm test` and `pnpm test:iam-remote` with `BETTER_COMPUTE_IAM_GUARD=throw`, so any `IamPort` call inside a compute transaction fails. `postgres`: the whole `pnpm test` against `postgres:17`, so the Postgres-only cases run. `streak`: counts consecutive green scheduled nights and reports when the v0.4 exit's three are reached | | `release.yml` | a `v*` tag, or a dispatch with a tag | bcd's release: goreleaser archives and binaries for linux amd64 and arm64 with SHA-256 files, an SPDX SBOM per archive (syft), keyless cosign signatures, and SLSA level 3 provenance (`bcd.intoto.jsonl`) from the SLSA generic generator; see [installing bcd](/docs/daemon/install) | | `daemon.yml` | pushes to `main` and pull requests that touch `daemon/`, `proto/`, `packages/provider-kubernetes/` or `scripts/k8s-render-manifests.mjs` | On Ubuntu and Windows: `go vet` and `go test` with `nonvml,fakehw`; on Linux also `go vet` of the NVML build and `goreleaser check`. The `container` job (ac1ca4b) builds `daemon/Containerfile` for `linux/amd64` with buildx, runs `bcd --version` and `kubectl version --client` from the image, renders the DaemonSet manifests (`scripts/k8s-render-manifests.mjs`, placeholder pins and digest) and, on a kind cluster, applies the Namespace and server-side dry-runs the rest (`kubectl apply --dry-run=server`), so the API server checks the schema and Pod Security admission. Nothing is scheduled or pulled: it is not a run of bcd as a DaemonSet (see [bcd on Kubernetes](/docs/daemon/kubernetes)) | `pnpm check` runs `typecheck`, `test`, `test:iam-remote`, `lint:repo` and `validate:packages`. `pnpm lint:repo` runs the repository rules in order: `check-deps` (dependency rules), `check-sql-now` (the one-clock rule), `umbrella --check` (umbrella drift), `proto-gen --check` and `proto-gen --breaking`, then `apps/docs/scripts/generate.mjs --check` (the generated docs reference must match the code; 267f8f4), then the API surface freeze. The generator check comes first because the freeze reads the committed reference: before it, a narrowed input that nobody regenerated passed the freeze. Under `pnpm check` it runs after `tsc -b`, so it sees the built code. If you change the registry (a method, its input schema, an error code, a command kind, an event or a metric), regenerate the reference in the same change. On a failure the check lists each file `out of date` or `stale` and prints the command to run, which regenerates from committed code only, so other sessions' uncommitted builds never reach the reference: ```bash node scripts/head-check.mjs --run apps/docs/scripts/sync-from-head.mjs "$PWD" ``` ### The API surface freeze [#the-api-surface-freeze] PLAN.md's v1.0 freezes the API. `scripts/api-surface.mjs` compares the frozen surface, `generated/api-surface.json` (every method with its route, kind, IAM action, resource, alternatives, list flag, status, phase and input JSON Schema, and every error code with its HTTP status), with the current one, which it reads from the docs generator's committed `apps/docs/generated/api.json` and `errors.json`. Additions pass. These fail the check: * a served method removed, or planned again; * its route, kind, IAM action or resource changed, an IAM alternative removed, or a list method no longer one; * its input schema rejecting an input the frozen schema accepted: a new required field, a type, enum value or property that goes, a tighter length, range, pattern, format or item count, a closed object that was open, a lost `anyOf` branch, or a constraint on a value that was unconstrained; * an error code removed, or its HTTP status changed. A method that is only planned may still change or go. When you add or change something compatibly, regenerate the docs and record it, so the addition is frozen too: ```bash node apps/docs/scripts/generate.mjs # api.json and errors.json follow the code pnpm api:surface # node scripts/api-surface.mjs --update: records additions, refuses breaks node scripts/api-surface.mjs --update --accept-breaking # only for a deliberate, reviewed break ``` The plain check prints the number of additions not yet recorded without failing. `tests/scripts-api-surface.test.ts` covers the rules and checks the committed snapshot against the generated docs. - [Local development](/docs/operations/local-development): `dev --sim` and `better-compute sim` . - [Scaling](/docs/operations/scaling): The section 16 gates and the measured numbers. - [Observability](/docs/operations/observability): Metrics, spans and doctor.