Local development

better-compute dev runs the API and every worker role in one process on 127.0.0.1, with no setup: an embedded PostgreSQL (PGlite), a generated deployment secret and migrations on start. --sim N adds a gateway and N simulated GPU machines that join the way the real daemon does, so reservations, commands, liveness and tunnels all run end to end on your laptop. better-compute sim is the other tool: a seeded, deterministic fleet simulation that checks the fleet invariants and prints a digest you can reproduce.

Start a server

better-compute dev                 # PGlite in .better-compute/dev-db, kept between runs
better-compute dev --memory        # a throwaway in-memory database
better-compute dev --memory --sim 8

dev always migrates first, then serves the API under /api/compute and runs the worker roles controllers, scheduler, sinks and iam. It stops on SIGINT or SIGTERM, releasing its leases.

Without a configuration (and without BETTER_COMPUTE_DATABASE_URL), dev builds its own deployment: PGlite in --data, and a generated secret in .better-compute/dev-secret (owner-only), so repeated runs keep their data. BETTER_COMPUTE_BASE_URL defaults to http://<host>:<port>.

With a configuration file, dev uses that deployment. --memory then sets BETTER_COMPUTE_DATABASE_URL to pglite:memory for the run, which the examples and the configuration template read. The host export's HTTP routes and metrics are mounted; its gateway role stays off in dev.

Either way the CLI adds retentionModule() from @better-compute/postgres unless the configuration registers a retention module itself. See retention.

IAM. A deployment without Better IAM configured answers only public methods. better-compute dev --sim N replaces the missing IAM with an embedded one; plain dev does not. See Better IAM.

PGlite

PGlite is PostgreSQL 18 compiled to WebAssembly, with btree_gist. It is one connection, so the executor serializes transactions behind a mutex. PGlite has no autovacuum, so the executor runs ANALYZE after migrate() and then every analyzeEveryMs (default 60,000) to keep query plans close to Postgres's. It is the database of dev, the tests and the simulators, not of production; see database.

Simulated machines

--sim N joins N machines to the dev server's own control plane. Nothing is mocked above the daemon:

Standard error shows the progress (from examples/dev):

better-compute listening on http://127.0.0.1:3000/api/compute
Dev gateway (h2c, header auth, loopback only): http://127.0.0.1:52011
Embedded IAM root tenant ten_…; root token (dev only): biam_ses_…
Joined 8 simulated hgx-h100 node(s) into pool_…
8 simulated node(s) adopted

The embedded IAM lives in memory for this run only, and its root token is a development credential.

Look around

In a second terminal, as the root administrator (see the CLI):

export BETTER_COMPUTE_URL=http://127.0.0.1:3000
export BETTER_COMPUTE_TOKEN=biam_ses_…   # the root token dev printed
export BETTER_COMPUTE_TENANT=ten_…       # the root tenant dev printed
better-compute nodes list
export POOL=pool_…                        # the pool dev created (pools list)
better-compute pools offer $POOL --to $BETTER_COMPUTE_TENANT --quota gpu=64
better-compute reserve --pool $POOL --gpus 4 --for 1h
source <(better-compute completion bash) # optional: tab completion for every command

The reservation binds through the node's desired state and turns active once the simulated daemon reports AllocationBound; its CDI device names are in the reservation's binding. With a configuration that mounts the events module's GET /watch (the examples do), better-compute events watch --types "node.*,reservation.*" prints the tenant's events one JSON line each (curl -N on the same route shows the raw stream); cordon a node in another terminal (better-compute nodes cordon node_… --reason demo) to see node.cordoned arrive. This is the walkthrough of examples/dev/README.md (786cc80).

Tunnels

With --sim, dev issues exec tickets (the configured tunnels.issuer, or a throwaway one) and mounts GET /api/compute/tunnels/{ticket}. The gateway verifies tickets and serves its control API on loopback, and every simulated machine answers OpenTunnel with an echo shell, so tunnels work end to end: better-compute console --node node_… opens a session on a simulated machine, where size prints your terminal size, exit ends it with status 0, false with 1, and Ctrl-] detaches (tests/cli-console.test.ts). Unless the deployment's tunnels options bring their own, it also adds a throwaway break-glass SSH CA (tunnels.ssh, for nodes.issueSshCert and better-compute ssh-cert) and a separate throwaway sandbox SSH CA (tunnels.sandboxSsh), so both can be tried against the simulated fleet. It adds these throwaway CAs only on a PGlite store: against a real database every node's spec would start trusting a CA that dies with the process, so there dev --sim uses only the CAs the deployment configures. With the embedded IAM, an open tunnel is not re-authorized every 60 seconds: the embedded test IAM has no session reviewer. tunnels.open still decides every ticket.

The dev stack in code

The same pieces are exported from @better-compute/testing for your own hosts and tests.

import { afterEach, expect, it } from 'vitest';
import { createDevStack, type DevStack } from 'better-compute/testing';

let dev: DevStack | undefined;
afterEach(() => dev?.close());

it('adopts three simulated machines', async () => {
  dev = await createDevStack();
  const { poolId } = await dev.createPool({ expectGpuCount: 8 });
  const nodes = await dev.join({ poolId, count: 3 }); // sealed-token Join, then a Session each
  await dev.waitFor(nodes); // adopted, NodeSpec generation 1 or later applied
  expect(dev.daemons.size).toBe(3);
});

A DevFleet has createPool({ gpuModel?, key?, expectGpuCount? }), join({ poolId, count?, variant?, seed? }) (1 to 10,000 machines on one token), joinSite({ siteId, count?, kinds?, seed? }) for site agents (1 to 16, through siteTokens.create), waitFor(nodeIds, { provisionState?, timeoutMs? }) (default adopted, 30 s), the running daemons, the gatewayUrl, and close(). createDevStack() takes database, iam, modules, baseURL, secret, trustDomain, worker (default pollMs 100), region, gateway and daemon options, and signedIntent: true (5071dbc), which gives the stack a fresh command-signing root (pki.commandSigning), so every command and spec is signed and joining daemons receive the roots in JoinResponse.trust.

A real bcd

Two options let a daemon outside the process, such as a real bcd, join the dev stack:

For externalTls, write a config.toml with gateway_url set to the listener, serving_ca_spki_pin from servingCaCertificate, the join pins from pins, and gateway_server_name = "gw.dev.local" when you dial an IP address; put pins.joinEncryptionJwk next to it as join-encryption.jwk. Then pipe a join token into bcd join and start bcd node. tests/e2e/bcd.test.ts does exactly this from WSL; see testing. Since the stack publishes its gateway, a machine without any config.toml can also join with bcd join --server <the stack's base URL>, which takes the gateway from the well-known document: a host build also needs --serving-ca-pin, --join-signing-pin and --join-encryption-pin (from servingCaCertificate and pins), while a fakehw build may join with --server alone, taking the pins from the document on trust (and accepts an http:// server); see joining from the well-known document.

For many real daemons at once, a fakehw build runs bcd sim --config <that config.toml> --nodes N with one join token allowing N uses on standard input: N node agents in one process, each a distinct fakehw machine with its own identity under data_dir/sim/<i> (see bcd sim).

Two fakehw behaviours help when you test placement against a real bcd:

• Reserved huge pages show in the inventory (cd7a311). The host units write under the fake root (<data_dir>/fakehw-root), and the fakehw inventory reads nr_hugepages back from there: the machine-wide pools, and each NUMA node's own file or an even share of the machine-wide pool (node 0 takes the remainder). A pool with policy.hugepages therefore books hugepages2Mi and hugepages1Gi on a fakehw node as it would on a real host.
• The libvirt plan is written out (9a6aaf4). The fakehw runtime still runs every domain itself, but also writes the libvirt domain XML bcd would define on a real host to <data_dir>/fakehw-plans/<id>.xml (vcpupin, numatune, huge-page backing, the memory balloon, volumes by serial, GPU hostdevs), or <id>.error when libvirt would refuse the spec, and removes it on destroy. Its balloons live on disk (<runtime>.balloons), so they survive a restart. tests/e2e/bcd-placement.test.ts reads both (see testing).

The example

examples/dev is the whole deployment every example shares (examples/shared/compose.ts: provision and metal, vps, events, fleet, retention, and rollout and audit when their keys are present) with eight simulated machines:

cd examples/dev
better-compute dev --memory --sim 8     # the package's "dev" script
better-compute dev --sim 8              # "dev:persistent": data in .better-compute/example-db

The console locally

apps/console is the Next.js console (Better IAM and Better Compute on one origin). node server.mjs --dev (its dev script) runs Next.js plus the WebSocket upgrade for tunnels on http://localhost:3100. --dev is the only mode in which the console generates its own secrets under .console-data: the session secret, the exec ticket and SSH CA keys, and on first start the root administrator's password, printed once. Its compute store defaults to PGlite in .console-data/compute and its IAM store to a libSQL file. The console does not start simulated machines. See the console.

The fleet simulator

better-compute sim runs --nodes FakeDaemons on fake hardware against the in-memory gateway stub, entirely in simulated time. A steady workload runs for --duration (every 2 simulated seconds: three commands drawn from diag.nvml, inventory.refresh, logs.collect and node.reboot, a GPU booking and sometimes a release), with the faults of --chaos. The simulator then quiesces for --settle, flushes usage and checks the invariants. Nothing reads the wall clock, so the same seed prints the same digest.

better-compute sim
better-compute sim --seed nightly-42 --nodes 500 --duration 1h --chaos heavy --log sim.log

It checks I1 (no overlapping bookings or node-held devices), I2 (at most one execution per command; after a journal loss, in-flight commands that are not retry-safe fail instead of replaying), I3 (a delivered spec is reported observed within 60 s while the node stays reachable), I4 (no node marked unreachable within 20 s of a gateway hearing from it), I5 (no release caused by connectivity) and I8 (usage windows add up to booked time times atoms, with no duplicate keys). The command's own help text still lists only I1, I2, I5 and I8; the checks come from the simulator's invariants(). See testing.

The report is JSON: seed, nodes, chaos, simulatedMs, wallMs, digest (SHA-256 of the event log), commands (total, open), faults (crashes, reboots, journalLosses, gatewayKills), violations and the first 20 violations. On a violation the command prints the report and exits with SIM_INVARIANT_VIOLATED; re-run with the same --seed to reproduce it, and add --log to get the event log. sim needs @better-compute/testing.

Working on the repository

The repository is a pnpm workspace; docs/development.md is the full guide.

• Node.js 22.12 or later, ESM only. CI runs Node 22 and 24 on Ubuntu and Windows.
• pnpm 11.19.0, pinned in packageManager. When your global pnpm is another major, run the pinned copy: node node_modules/pnpm/bin/pnpm.mjs <args>.
• No database server. PGlite backs the tests. Suites that need real Postgres skip unless BETTER_COMPUTE_POSTGRES_URL is set.
• Go 1.26 or later only for daemon/: node scripts/daemon-test.mjs runs go test -tags nonvml,fakehw without cgo or GPUs, natively on Linux and through WSL on Windows. The bcd end-to-end suite is opt-in on Windows (BCD_E2E=1); see testing.

Tests live in the root tests/ folder as tests/<area>-<topic>.test.ts; vitest.config.ts maps every @better-compute/* specifier and better-compute/* subpath to packages/*/src. See testing.