better/compute

Walkthroughs

The acceptance walkthroughs of PLAN.md 5.6 and the emulated-site example, what each one proves about extending Better Compute, and how to run them.

PLAN.md §5.6 sets the bar for the extension points: each one is proven by a walkthrough that adds something real to Better Compute with zero edits to @better-compute/*. A walkthrough is a third-party plugin, written in a test the way an outside package would write it, plus the test that shows the platform picks it up and runs it. They double as the best worked examples of the contracts on the other pages of this section.

WalkthroughPhaseExtension pointsWhereState
1. Ubuntu autoinstall via Redfish virtual mediav0.2a control-plane step, a profile, a plugintests/extensions/ubuntu-autoinstall.test.tsruns in simulation
2. VPS provider acme-cloudv0.3defineProvider, conformance cases, a provider pooltests/extensions/acme-cloud.test.tsruns against an in-memory cloud emulator, live and from a recording
3. Talos on an agentless poolv0.3a relaying step, a profile, site.httpProxytests/extensions/talos.test.ts, tests/metal-fakedaemon.test.tsruns in simulation and on the dev stack
4. Kata runtime in Gov0.5a bcx runtime plugin in bcd's [plugins] dirdaemon/internal/agent/pluginruntime_test.gobcd serves runtimes from bcx plugins (Go unit tests); no Kata plugin written
Emulated sitev0.2a hardware profile, the whole metal lifecycleexamples/emulated-siteconfiguration checked; needs Linux, KVM and a bcd build to run

No real hardware yet

Every metal walkthrough needs bcd site, and all but the agentless Talos one need bcd ramdisk. Both modes of the Go daemon are now written and answer every command these walkthroughs send, BIOS, RAID and firmware updates through Redfish included (944221c), but they are not released and only their Go unit tests run them. No walkthrough runs against them yet: the tests answer the commands with simulated site agents and machines, or with a FakeDaemon site agent over the real gateway. Running one for real also needs what the repository does not provide (iPXE builds, a ramdisk image, a KVM host). PLAN.md's pass criterion for walkthrough 1, "a sushy-tools VM reaches deployed", has not been met. See what bcd does today.

Run them all from the repository root:

pnpm vitest run tests/extensions tests/metal-fakedaemon.test.ts tests/examples-smoke.test.ts

Walkthrough 1: Ubuntu autoinstall

The plugin. acme-autoinstall installs Ubuntu with its own installer from a per-machine ISO mounted through the BMC, instead of streaming an image from a ramdisk. It is three definitions:

const renderAutoinstallIso = defineStep({ id: 'autoinstall.renderIso', runsOn: 'control-plane', /* … */ });
const ubuntuAutoinstall = defineProfile({ id: 'ubuntu-24.04-autoinstall', version: 1, target: 'metal', role: 'host', /* … */ });
const acmeAutoinstall = definePlugin({ id: 'acme-autoinstall', extensions: [renderAutoinstallIso], profiles: [ubuntuAutoinstall] });
  • autoinstall.renderIso mints a single-use provisioned join token through ctx.pki, renders it into an autoinstall late-commands entry with renderTemplate, stores the ISO under its sha256, and returns { url }, an unguessable per-node URL behind the site cache.
  • The profile chains built-in metal steps around it: media.insert on the site agent with fromStep: 'autoinstall.renderIso', management.setBootDevice to cd, power.setPower cycle, wait.agentJoined for role node, and media.eject.

Both are written out in full on steps and profiles and deploying.

The harness. createMetalSim({ modules: [acmeAutoinstall], install }) in tests/extensions/metal-sim.ts builds a simulated site: PGlite, the real provision engine and metal module, a site agent, and per-machine BMC state (boot device, mounted media, power, firmware, BIOS and RAID). Booting from PXE brings up the ramdisk; booting from a CD whose ISO the install hook accepts brings up the installed OS as role node. Results come back as each kind's binary result message, node-signed where the kind is attested, as real agents send them.

What it proves.

  • profiles.validate resolves every step of the plugin's profile: the engine found a third-party step and profile.
  • One machine goes enroll, ready, deployed with role host, undeploy, ready. The ISO is ejected, and the join token appears inside the ISO and nowhere in any command the agents saw.
  • The v0.2 acceptance of PLAN.md §16 in simulation: 20 consecutive unattended deploy and release loops, a fresh single-use token for each install, and every transition into ready preceded by exactly one attested wipe in that workflow (checked in SQL over workflows and workflow_steps).
  • A release whose wipe is not attested never reaches ready: it ends failed from releasing, with maintenance on.
pnpm vitest run tests/extensions/ubuntu-autoinstall.test.ts

What it does not prove. The ISO build and hosting are stubbed (a map from digest to token), the installed OS rejoining as the same node is simulated, and no real BMC mounts anything. A token a plugin mints itself joins through the provisioned attestor only if it carries expect: { productUuid } or the host's resolver can trace it; see attestors.

Walkthrough 3: Talos on an agentless pool

Talos manages itself through its own API and never runs bcd. The plugin is one step and one profile:

  • wait.talosReady is a control-plane step with relays: ['site'] and sends: ['site.httpProxy'], the kind it relays. site.httpProxy is destructive (0d42de8), so the declaration is what puts it in the workflow's quorum batch: without it, a deployment with pki.approvers set refuses the step's commands at enqueue (see relaying under the quorum). It reads the machine's address from metal_nodes.addresses (recorded by Redfish inspection), and calls httpViaSite(ctx, { url, tlsSpkiSha256, poll: true }) on https://<address>:50000/machine/version. Until Talos answers 200, it throws stepNotReady.
  • talos-1.8 mounts the Talos ISO with media.insert, sets boot to cd, power-cycles, and waits in wait.talosReady.

The pool opts in to what an agentless machine cannot do: agent: 'none', policy.allowUnverifiedRelease: true, and policy.provisioningCidrs for plain-HTTP targets.

httpViaSite is the extension point this walkthrough added to metal: a control-plane step reaches a machine only through its site agent's site.httpProxy command, and the control plane refuses anything else before a command exists. The request takes url, method (GET, POST, PUT, DELETE), headers, secretHeaders (secret references only), body, tlsSpkiSha256, timeoutMs, key and poll. With poll, each call returns the newest settled answer and sends the next request, which is what a readiness loop needs.

What it proves (tests/extensions/talos.test.ts, on PGlite with a stub command queue):

  • Enrollment and commissioning run out of band (the agentless variants of metal's profiles); the recorded addresses come from Redfish.
  • The deploy polls the Talos API until it answers, always at the machine's own address, with the pinned SPKI, and the node is deployed with role host. Every command went to the site agent, never to the machine.
  • The node is never connected, so it is never schedulable for hold reservations.
  • Release reaches ready without an attested wipe only because the pool opted in, and says so with node.release.unverified. Rescue fails the node with UNSUPPORTED.
  • site.httpProxy is SSRF-bounded: other hosts, names, credentials in URLs, plain HTTP off the provisioning network, forbidden headers and header line breaks are refused; a request for another node's IP creates no command; a redirect or an oversize response is DRIVER_ERROR; secret headers travel by reference.

tests/metal-fakedaemon.test.ts runs the same Talos plugin on the dev stack: the real control plane with provision and metal, the real gateway, JoinService, and a site agent that is a FakeDaemon of role site, joined with a site token (tests/metal-dev-site.test.ts), in front of simulated BMCs. Every command crosses the gateway and back, and the real server's batch check runs (f66e236). Both tests declare sends on the Talos step.

pnpm vitest run tests/extensions/talos.test.ts tests/metal-fakedaemon.test.ts

Walkthrough 2: a VPS provider

PLAN.md describes it as defineProvider(…), providerConformance(acmeCloud, { recordReplay }), a provider account, and a class: 'provider' pool with agent: 'none' whose instances run through the generic provider steps. tests/extensions/acme-cloud.test.ts runs it as a third party would, with zero edits to @better-compute/*:

  • The plugin. tests/extensions/acme-cloud/provider.ts is a defineProvider over Acme Cloud's HTTP API with an injectable fetch; cloud.ts is an in-memory emulator of that API, and record-replay.ts a transport that records every exchange and replays it.
  • Conformance, live and offline. runConformance(providerConformanceCases(), subject) passes against the emulator through the recording transport, leaving no server behind, and passes again offline from the recording with every exchange consumed. A wrong account token is a configuration error, not a provider outage.
  • On a real server. With vps({ providers: [acmeCloudProvider({ fetch })] }) on a compute fixture: a provider account whose token is sealed in the secret store, a class: 'provider' pool (its agent is none) offered to a tenant with a quota, and a flavor mapped onto the provider's offer. The tenant's instance is created once (a retried reconciler pass creates nothing new), with the SSH key and no join token in its user data, followed to running, stopped (the machine powered off) and deleted; a third instance over the tenant's quota is refused, and the pool never gets a node.

The conformance API is providerConformanceCases() with runConformance, not a single providerConformance function, and recording is left to the transport: adapters take a fetch. Instances of provider pools are driven by the vps.instances reconciler calling the account's client, not by workflow steps. tests/vps-provider.test.ts (manifest-typed accounts and shape refusals) and the Hetzner Cloud, Hetzner Robot and Proxmox suites cover the same contract piecewise. Providers shows how to write and test one.

pnpm vitest run tests/extensions/acme-cloud.test.ts

Walkthrough 4: a Kata runtime over bcx

A runtime driver written in Go, pinned in bcd's [plugins] dir, advertised in Hello.drivers, and selected by pools through vps({ runtimes }). The TypeScript half of bcx is in @better-compute/driver-sdk, with a real-subprocess test suite. bcd hosts pinned plugins and advertises their drivers, and bcx.Serve in daemon/internal/bcx is the start of a Go plugin SDK.

On bcd node, a hosted driver whose manifest lists the kind runtime, and whose id is not a runtime built into bcd, becomes an instances runtime (agent.PluginRuntime, 494de69). bcd speaks the TypeScript RuntimeDriver contract to it over Invoke, with positional JSON arguments:

OperationArgumentsAnswers
listnone[{ id, state, addresses, message? }]
applycore's InstanceSpec as JSON, with desiredState{ id, state, addresses, message? }; state: 'error' fails the instance
removethe instance idnull

apply converges one instance, desired state included, so bcd records each spec and applies it when the state changes. A new instance's first apply carries its user data and, for a restore, the snapshot bcd already downloaded and verified, as a file:// URL; later ones do not. Snapshots answer unsupported, since they are not in the bcx contract yet. A plugin never shadows a built-in runtime of the same name, and Hello lists its manifest once. Like every plugin it runs sandboxed: under its own uid by default, with a seccomp filter that refuses mounts, namespaces, bpf and module loading, so a runtime that needs them (Kata) must be named in [plugins] privileged, which runs it as root without the filter. Go unit tests (pluginruntime_test.go) cover it; no Kata plugin exists yet. The control-plane half is in place: the gateway writes each driver a daemon advertises in Hello.drivers as the node label bc.dev/driver.<id>=<version>, and vps places an instance, whatever its runtime (bcd's built-ins too since 0fca413), only on nodes carrying bc.dev/driver.<runtime> (runtimeNodeLabels, through PlacementRequest.nodeLabels; see placement). See drivers.

The emulated site

examples/emulated-site is the metal walkthrough without a rack: QEMU/KVM virtual machines stand in for servers, sushy-emulator gives each a Redfish BMC over libvirt, and a bcd site agent on the libvirt bridge runs proxy DHCP, iPXE and the image cache. Its better-compute.config.ts uses the shared example deployment (examples/shared/compose.ts, every module) and adds one hardware profile:

examples/emulated-site/better-compute.config.ts
export const hardwareProfiles = {
  'emulated-redfish': {
    description: 'QEMU/KVM machines behind sushy-tools (Redfish), PXE-booted by the site agent',
    drivers: { power: 'redfish', management: 'redfish', boot: 'ipxe', deploy: 'image-stream' },
  },
};

A pool that names it in policy.hardwareProfile drives the emulated BMCs through commission, deploy, release and retire. The run needs Linux with /dev/kvm, QEMU, libvirt, sushy-tools and a bcd build, plus iPXE builds for the site agent's TFTP root and a ramdisk kernel and initramfs with bcd ramdisk in it, so none of it runs in CI. One sushy-emulator serves both VMs, so each enrollment names its system in the BMC address (https://192.168.122.1:8000/redfish/v1/Systems/<libvirt domain uuid>); bcd's Redfish client then drives that system and its own manager (see Redfish in bcd site). What CI checks, in tests/examples-smoke.test.ts, is that the configuration loads, migrates on PGlite and passes better-compute doctor with no error findings (on an unconfigured machine it reports iam.not-configured).

The step-by-step run, from virt-install to metal.undeploy, is on deploying. Two details there supersede the example's README: site agents join with a site token from siteTokens.create, and after metal.enroll the reconciler carries a machine to ready on its own, so metal.manage, metal.inspect and metal.provide are not needed.

Writing your own

A walkthrough for your extension follows the same recipe:

  1. Write the plugin in the test file, importing only public entry points, the way an outside package would.
  2. Register it the way a deployment would (provisionModule({ modules }), vps({ providers }), betterCompute({ attestors })), not by reaching into internals.
  3. Assert the platform's view first: profiles.validate returns no issues, drivers.list shows the driver, the conformance report is empty.
  4. Drive the lifecycle end to end on a simulated site or the dev stack, and assert on states, events and the commands agents saw, including what must not appear (a token in a command body, a destructive command sent twice).
  5. Add the failure cases: an unattested result, a crash mid-step (tests/metal-chaos.test.ts kills the worker at every point of a deploy), a refused request.

Was this page helpful?

Better Compute is created by Sean Filimon

Last updated

On this page