Steps and profiles

The provision engine runs a profile, a versioned document, as a workflow of persisted steps for one node or instance. Metal's whole lifecycle is built this way: enrolling, inspecting, commissioning, deploying, releasing and retiring are each a profile that metal ships. You extend the engine with your own steps and profiles, and around them with user-data templates, config units and image signers, without editing @better-compute/*.

How workflows run, fail, retry and are aborted is on provisioning workflows; this page is about writing the pieces.

Three kinds of step

Relaying steps exist because the command often needs data only the control plane has: a BMC address and credential reference, a resolved catalog image, a freshly minted token. relays: ['site'] lets profiles still write { use: 'power.setPower', on: 'site' }.

defineStep

import { createHash } from 'node:crypto';
import { manifest } from 'better-compute/core';
import { defineStep, renderTemplate } from 'better-compute/provision';

export const renderAutoinstallIso = defineStep({
  id: 'autoinstall.renderIso',
  runsOn: 'control-plane',
  timeout: '5m',
  retrySafe: false,
  input: manifest({}),
  async run(_input, ctx) {
    const node = ctx.node!;
    const { token } = await ctx.sql.transaction((tx) =>
      ctx.pki!.issueJoinToken(tx, { tenantId: 'root', role: 'provisioned', siteId: node.siteId, ttlSeconds: 7200, maxUses: 1 }),
    );
    const userData = renderTemplate(autoinstallTemplate, { hostname: node.id.replace(/_/g, '-'), joinToken: token });
    const digest = createHash('sha256').update(userData).digest('hex');
    await publishIso(digest, userData); // your ISO build and storage behind the site cache
    return { url: `https://cache.site-a/autoinstall/${digest}.iso` };
  },
});

This is the step of walkthrough 1. defineStep checks the shape up front and returns the step frozen with extension: 'step'; every refusal is INVALID_CONFIG with details.path.

An agent step's command must be a command kind some module registers (ComputeModule.commandKinds, see modules and command kinds); the server's command queue refuses others with INVALID_CONFIG. Its payload is the step's validated input as the kind's proto3 JSON. The agent has to implement the kind, which for a new kind means a change to bcd.

The step context

A control-plane step's run receives a StepContext: the whole ModuleContext, plus the workflow's coordinates.

pki and secrets are optional on ModuleContext. A step that needs them should fail with INVALID_CONFIG when they are missing, as userdata.render does, rather than crash on undefined.

• ctx.pki.issueJoinToken mints the credential an agent your step brings up will join with. Roles are node, site, provisioned and discovery. For anything a machine uses to join, mint maxUses: 1, a short ttlSeconds, and expect (mac, productUuid, serial or gpuUuid) when you know the machine, so a join with other hardware claims is refused without spending the token.
• ctx.secrets.put stores a value and returns a SecretRef. Put tokens and rendered documents there and send only the reference. An agent can fetch a reference's value only when a live command of this workflow step for that agent (key wf:<workflow>:<step>, with any #n or :key suffix) carries the reference in its body, so the reference must travel in the step's own command.

Never return a secret from run. The result is stored in workflow_steps.result and readable through workflows.get; userdata.render returns { userDataRef, kind, joinTokenId }, not the document.

Idempotency and wake-ups

A control-plane step is marked running in one transaction, runs outside any transaction, and is settled in another that first checks it is still running with the same attempt. A worker that dies in between leaves the step running, and the next owner of the shard runs it again under the same idempotency key. The engine does not read retrySafe today, so every run must be safe to repeat. The pattern is find before create:

async run(input, ctx) {
  return ctx.sql.transaction(async (tx) => {
    const { rows } = await tx.query('SELECT ip FROM plg_acme_ips WHERE step_key = $1', [ctx.idempotencyKey]);
    if (rows[0]) return rows[0]; // a re-run after a crash finds the first run's work
    const ip = await nextFreeIp(tx); // your allocator, over your plugin's own tables
    await tx.query('INSERT INTO plg_acme_ips (step_key, ip) VALUES ($1, $2)', [ctx.idempotencyKey, ip]);
    return { ip };
  });
}

Key per attempt, not per step, when a retry must start over: stepCommandKey(ctx, key) is wf:<id>:<step>#<attempt>[:<key>].

Waiting

A step that cannot finish yet throws stepNotReady(ms). The engine settles nothing, and runs the step again after ms (5,000 by default), never later than its deadline; at the deadline the step fails TIMEOUT. The built-in waits are written this way:

async run(input, ctx) {
  const { rows } = await ctx.sql.query('SELECT generation, observed_generation FROM nodes WHERE id = $1', [ctx.node!.id]);
  if (Number(rows[0].observed_generation) < Number(rows[0].generation)) throw stepNotReady(5_000);
  return { generation: Number(rows[0].generation) };
}

Sending commands from a step

awaitCommand(ctx, { nodeId, kind, payload, key?, timeoutMs? }) is find-before-create for commands. It looks the command up by wf:<id>:<step>#<attempt>[:<key>] on that node, enqueues it only if absent (with notAfter now plus timeoutMs, 10 minutes by default), and then:

• throws stepNotReady(2000) while the command is pending, dispatched, acked or running;
• returns { commandId, result, attested } once it succeeded, with the agent's raw result bytes;
• throws DRIVER_ERROR with the state and reason once it failed, was rejected, expired or cancelled.

Give each command of one step its own key ('boot-device', 'power-cycle'): metal.ensureRamdisk sends a one-time PXE boot, a boot entry per MAC and a power cycle in sequence, and a re-run skips the ones already done because it finds them by key. To reach a node's site agent, siteAgentFor(ctx.sql, nodeId) returns a connected agent of the node's site, the boot-lease holder first, or undefined, in which case throw stepNotReady and wait for one. Metal's httpViaSite(ctx, request) is a relaying helper built the same way: one HTTP request from the site agent to the machine's own address, with SSRF checks; see walkthroughs.

A command's result is not trusted just because it arrived. A result that decides a transition, like a wipe, should be node-signed and verified: metal's steps check the stored signature against the node's certificates before they accept attested.

Relaying under the quorum

With the v1.0 destructive-batch quorum on, a destructive command runs only in a batch its approvers signed, and the batch's scope is fixed before the workflow sends anything. The engine records every kind a workflow can send when it starts (workflows.command_kinds: each agent step's command, and each control-plane step's sends), metal creates the workflow's batch from those kinds (f66e236), and awaitCommand and the engine's agent steps put ctx.batchId on the destructive commands only. So a relaying step must declare what it sends, or the kind is missing from the scope:

const talosReady = defineStep({
  id: 'wait.talosReady',
  runsOn: 'control-plane',
  relays: ['site'],
  sends: ['site.httpProxy'], // httpViaSite enqueues site.httpProxy, a destructive kind
  timeout: '20m',
  retrySafe: true,
  input: manifest({ port: field.int({ min: 1, max: 65535 }).default(50000) }),
  async run(input, ctx) {
    const response = await httpViaSite(ctx, { url: `https://${address}:${input.port}/machine/version`, poll: true });
    // ...
  },
});

Without sends, the step's site.httpProxy carries the workflow's batch but is not in its scope, and with approver keys set the queue refuses it at enqueue (INVALID_INPUT; or INVALID_STATE, reason batch-required, when the workflow sends nothing else destructive and so got no batch). With sends, enqueueing any undeclared kind is INVALID_CONFIG. defineStep refuses sends on an agent step, whose command already names its kind. See who creates batches and, for batches over many nodes, metal.bulk.

What wakes a workflow

Nothing depends on a wake-up arriving. The workflow reconciler is keyed by the node id (or the workflow id for workflows without a node), and after each pass it queues itself in the work queue with notBefore set to the next moment it cares about:

• an agent step waiting on its command: in 10 seconds (workflowPollMs), or at the step's deadline if sooner;
• a control-plane step that threw stepNotReady(ms): in ms, or at the deadline if sooner (a relaying step's awaitCommand uses 2 seconds);
• a verb (abort, retry, resume) or a new workflow: at once.

When a workflow ends, its owner hears about it. ctx.workflows.start(tx, { …, owner }) names a reconciler that is queued under the workflow's key when it finishes. Metal starts every workflow with owner: 'metal', so the node moves on without polling.

Profiles

import { defineProfile } from 'better-compute/provision';

export const ubuntuAutoinstall = defineProfile({
  id: 'ubuntu-24.04-autoinstall',
  version: 1,
  target: 'metal',
  role: 'host',
  steps: [
    { use: 'autoinstall.renderIso' },
    { use: 'media.insert', on: 'site', with: { fromStep: 'autoinstall.renderIso' } },
    { use: 'management.setBootDevice', on: 'site', with: { device: 'cd' } },
    { use: 'power.setPower', on: 'site', with: { action: 'cycle' } },
    { use: 'wait.agentJoined', with: { role: 'node' } },
    { use: 'media.eject', on: 'site' },
  ],
});

defineProfile rejects unknown fields, malformed steps, ids and durations, non-JSON values, fields of the other target, and an image, user data or steps on an adopt profile; it returns a deeply frozen copy. The fields are listed on profiles and images.

Steps pass data forward through their results. media.insert with fromStep reads the url the latest succeeded step with that action returned in the same workflow; deploy.deploy finds userdata.render's reference the same way (renderedUserData(sql, workflowId)). Read earlier results from workflow_steps by workflow_id and action, never from memory.

When a workflow starts, each use: resolves once and the result is persisted:

1. a registered step with that id (an on: must match its runsOn or one of its relays);
2. otherwise, with on: naming an agent, a driver operation <concern>.<operation> sent as one driver.invoke command (see drivers);
3. otherwise fallback:, resolved the same way.

with is validated against the step's input at the same moment. profiles.validate reports every unresolved use and rejected with with its path, and profiles.render returns the expanded steps; run both before the first deploy. A version becomes immutable the first time a workflow uses it, and a workflow holds at most 512 steps.

Shipping steps and profiles

The engine resolves only what provisionModule knows. There are two ways to tell it:

import { definePlugin } from 'better-compute/core';
import { metal } from 'better-compute/metal';
import { provisionModule } from 'better-compute/provision';

// 1. Directly: provision registers them as its own.
const provision = provisionModule({ steps: [talosReady], profiles: [talos] });

// 2. As a plugin: the engine borrows its steps and profiles, the plugin registers them itself.
const acmeAutoinstall = definePlugin({ id: 'acme-autoinstall', extensions: [renderAutoinstallIso], profiles: [ubuntuAutoinstall] });
const bareMetal = metal();
const provisionWithPlugin = provisionModule({ modules: [bareMetal, acmeAutoinstall] });

With the second form, register the plugin with the server as well (betterCompute({ plugins: [acmeAutoinstall] })), the way metal is listed both in provisionModule({ modules }) and in modules: provision registers only its own steps and profiles, and the registry refuses an extension registered twice. Shipped profiles are read-only, visible to every tenant, and listed under the tenant builtin; tenants can still store their own with profiles.create, and a lookup prefers the tenant's own profile, then a public stored one, then a shipped one. Step ids share one namespace: a duplicate is INVALID_CONFIG, so prefix yours (acme., autoinstall.).

User-data templates

userdata.render renders the pinned profile's userData: { kind, template } for the workflow's node with a fresh single-use provisioned token (2 hours, or tokenTtlSeconds), stores the document through ctx.secrets, and returns only the reference. Add templates by name:

provisionModule({
  userData: {
    controlPlaneUrl: 'https://gw.eu-1.example.com',
    bcdUrl: 'https://downloads.example.com/bcd/0.2.0/bcd-linux-amd64',
    bcdSha256: '…',
    templates: { 'acme-talos.yaml': acmeTalosTemplate }, // kind 'talos' ships no template
  },
});

A template is a string with {{name}} placeholders or a function of the variables: nodeId, hostname (the node id with _ as -), joinToken, controlPlaneUrl, bcdUrl and bcdSha256. renderTemplate refuses a value with a line break or {{ (INVALID_INPUT) and a placeholder with no value (INVALID_CONFIG), so a value can never add lines to the document. Build structured formats as data and serialize them, as the built-in Ignition template does, rather than splicing strings. provisionedTargetResolver() (for the provisioned attestor) maps the token back to the node through the step result's joinTokenId, so the installed OS re-keys the same node.

Config units

A profile's hostConfig is not a step: when the workflow succeeds, its units become the node's NodeSpec host section, which bcd's host converger applies (see host configuration). A unit is { name, description, settings }, where settings is a manifest:

import { field, manifest } from 'better-compute/core';

provisionModule({
  configUnits: [
    {
      name: 'acme-nic-firmware',
      description: 'Pins the NIC firmware branch',
      settings: manifest({ branch: field.string({ pattern: '^[0-9]+\\.[0-9]+$' }) }),
    },
  ],
});

Any unit takes when:, a label expression that turns it on per node. An unknown unit or invalid settings is INVALID_INPUT when the host section is resolved. The built-ins are kernel-modules, sysctl, hugepages, nvidia-driver, fabric-manager, mig-layout, gpu-sharing, container-toolkit and k8s-join; a unit you add with the name of a built-in replaces it. A new unit is data to the control plane only: bcd applies the units it was built with, and reports any other as unit <name> is not supported by this daemon, which fails the host section terminally once the known units have had their turn.

Images

Steps that deploy an image resolve it from the catalog in the machine owner's tenant with resolveImageRef(tx, tenantId, 'name@version'), which prefers the tenant's own image, then a public one. Images are registered by digest and verified against allowlisted signers (Ed25519, or ECDSA P-256 over SHA-256); provisionModule({ imageVerifier }) replaces the check. ImageSource, a core contract for resolving other schemes, is declared but nothing uses it yet. See profiles and images.

Testing

• Shapes. tests/provision-define.test.ts covers defineStep and defineProfile refusals; assert yours throw INVALID_CONFIG with the details.path you expect.
• The engine with a stub queue. tests/extensions/talos.test.ts runs the real provision module and metal on PGlite with an in-memory CommandQueue and WorkQueue, answers commands from a simulated machine, and drives the reconcilers until nothing is left. Copy its settle() loop.
• A simulated site. createMetalSim({ modules, steps, profiles, install }) in tests/extensions/metal-sim.ts adds a site agent, per-machine BMC state and node-signed results; ubuntu-autoinstall.test.ts runs 20 deploy and release loops on it.
• Crashes. tests/metal-chaos.test.ts kills the worker at every point of a deploy and checks that no destructive command is sent twice. Do the same for a step with side effects.