Deploying machines

This page walks through the three deployments the repository proves: a GPU host commissioned and deployed over PXE on an emulated site, Ubuntu installed by its own installer from a per-machine ISO, and Talos on machines that never run bcd. Each follows the same lifecycle from bare metal; they differ in how the OS gets onto the disk and how the control plane learns it is up.

Adopt or PXE

Adoption is covered on joining nodes. The state table lets an adopted node with a BMC be released, but metal acts only on machines it enrolled, so an adopted node has no metal record and metal.undeploy answers NOT_FOUND for it today.

Before you start

A deployment that provisions metal needs, beyond the usual control plane:

• metal({ hardwareProfiles }) and provisionModule({ modules: [bareMetal], userData }), as on bare metal. The examples' shared configuration (examples/shared/compose.ts) registers both modules but no userData options and no attestors, so add them for profiles that render user data and for agents that join over PXE.
• A PKI bundle (better-compute pki init), so JoinService can admit site agents, ramdisks and installed machines, and the gateway role for them to connect to. See PKI.
• The discovery and provisioned attestors, wired as on sites.
• A site with a connected site agent, a pool on it whose policy.hardwareProfile names your hardware profile, an image in the catalog, and a deploy profile.

Walkthrough: an emulated site

examples/emulated-site stands in for a rack: QEMU/KVM virtual machines are the servers, sushy-emulator gives each a Redfish BMC over libvirt, and a bcd site agent on the libvirt bridge runs proxy DHCP, iPXE and the image cache. It needs Linux with /dev/kvm, QEMU, libvirt, sushy-tools and a bcd build, plus the iPXE builds and a ramdisk image, which the repository does not build. Nobody has run it end to end yet. Its configuration registers one hardware profile:

export const hardwareProfiles = {
  'emulated-redfish': {
    description: 'QEMU/KVM machines behind sushy-tools (Redfish), PXE-booted by the site agent',
    drivers: { power: 'redfish', management: 'redfish', boot: 'ipxe', deploy: 'image-stream' },
  },
};

virt-install --name bm-1 --memory 8192 --vcpus 4 --disk size=40 --pxe --network network=bc-site --noautoconsole
virt-install --name bm-2 --memory 8192 --vcpus 4 --disk size=40 --pxe --network network=bc-site --noautoconsole
sushy-emulator --libvirt-uri qemu:///system --port 8000

cd examples/emulated-site
better-compute pki init --trust-domain site.example.test
better-compute serve

export BETTER_COMPUTE_TENANT=root
better-compute sites create --kind shared --region lab-1 --key lab-1
better-compute site-tokens create --site site_… --ttl 3600 --key lab-1-agents --out ./site-token
echo '{"hardwareProfile":"emulated-redfish"}' > policy.json
better-compute pools create --site site_… --class metal-host --policy-file policy.json

better-compute api images.register --data @image.json
better-compute profiles create ./ubuntu-gpu-host.json
better-compute profiles validate ubuntu-24.04-gpu-host --profile-version 1

{
  "tenantId": "root",
  "siteId": "site_…",
  "poolId": "pool_…",
  "bmc": {
    "address": "https://192.168.122.1:8000/redfish/v1/Systems/<libvirt domain uuid>",
    "driver": "redfish",
    "credentials": { "username": "admin", "password": "…" }
  }
}

better-compute api metal.enroll --data @enroll-bm-1.json
better-compute workflows list --node node_… --query "items[].profileId"

better-compute metal enroll --site site_… --pool pool_… \
  --bmc-address 'https://192.168.122.1:8000/redfish/v1/Systems/<libvirt domain uuid>' --bmc-driver redfish \
  --bmc-username admin --bmc-password-file ./bmc-password

better-compute metal deploy node_… --profile ubuntu-24.04-gpu-host --profile-version 1
better-compute metal undeploy node_… --wipe quick

What the machine sees

In the lifecycle test, enrollment to ready sends exactly these commands. BMC work goes to the site agent, disk and diagnostics work to the machine's ramdisk:

The deploy then sends deploy.image (the catalog source, digest, format and compression, plus a reference to the rendered user data) and boot.kexec. The user data travels only as a reference, and the deploy's ramdisk boot gets its own discovery token.

A machine whose BMC cannot report its MACs (IPMI) and that was enrolled with only its productUuid adds a step before 4 (3d26428, 8de6e2f): metal.ensureRamdisk power-cycles it into PXE with no boot entry, a MAC first seen under its UUID within 10 minutes becomes its candidate, and nothing more is sent until an operator confirms a MAC with metal.confirmMac (better-compute metal confirm-mac <node> --mac …, after metal discovered --site … shows the candidate, 002dd95; or Confirm on the console's PXE discovery page, 15c11ee), or at once on a pool with policy.trustProvisioningNetwork (whose inventory.refresh then checks the ramdisk's UUID and NICs against the machine). See discovering new machines.

What the ramdisk does

bcd ramdisk runs these kinds (daemon/internal/agent/ramdisk.go) under metal's names. Disks are named by WWN, else serial, never by kernel name, since kernel names are not stable across boots.

inventory.refresh, diag.nvml and logs.collect run in the ramdisk too. Under the fakehw build these act on disk image files, so the Go tests run the same logic on sparse multi-terabyte files.

Walkthrough: Ubuntu autoinstall over virtual media

This is walkthrough 1 of PLAN.md §5.6: a third-party plugin, with no change to @better-compute/*, installs Ubuntu with its own installer from an ISO mounted through the BMC, with a single-use provisioned join token inside the autoinstall configuration so bcd joins afterwards.

import { createHash } from 'node:crypto';
import { definePlugin, manifest } from 'better-compute/core';
import { defineProfile, defineStep, renderTemplate } from 'better-compute/provision';

const autoinstall = `#cloud-config
autoinstall:
  version: 1
  identity: { hostname: {{hostname}} }
  late-commands:
    - curtin in-target -- sh -c "echo {{joinToken}} > /etc/bcd/join-token && chmod 600 /etc/bcd/join-token"
    - curtin in-target -- systemctl enable bcd
`;

const renderAutoinstallIso = defineStep({
  id: 'autoinstall.renderIso',
  runsOn: 'control-plane',
  timeout: '5m',
  retrySafe: false,
  input: manifest({}),
  async run(_input, context) {
    const node = context.node!;
    const { token } = await context.sql.transaction((tx) =>
      context.pki!.issueJoinToken(tx, { tenantId: 'root', role: 'provisioned', siteId: node.siteId, ttlSeconds: 7200, maxUses: 1 }),
    );
    const userData = renderTemplate(autoinstall, { hostname: node.id.replace(/_/g, '-'), joinToken: token });
    const digest = createHash('sha256').update(userData).digest('hex');
    await publishIso(digest, userData); // your ISO build and storage behind the site cache
    return { url: `https://cache.site-a/autoinstall/${digest}.iso` }; // unguessable, per node
  },
});

const ubuntuAutoinstall = defineProfile({
  id: 'ubuntu-24.04-autoinstall',
  version: 1,
  target: 'metal',
  role: 'host',
  steps: [
    { use: 'autoinstall.renderIso' },
    { use: 'media.insert', on: 'site', with: { fromStep: 'autoinstall.renderIso' } },
    { use: 'management.setBootDevice', on: 'site', with: { device: 'cd' } },
    { use: 'power.setPower', on: 'site', with: { action: 'cycle' } },
    { use: 'wait.agentJoined', with: { role: 'node' } },
    { use: 'media.eject', on: 'site' },
  ],
});

export const acmeAutoinstall = definePlugin({
  id: 'acme-autoinstall',
  extensions: [renderAutoinstallIso],
  profiles: [ubuntuAutoinstall],
});

Pass the plugin to the workflow engine next to metal, so it resolves the plugin's step and profile: provisionModule({ modules: [bareMetal, acmeAutoinstall] }). media.insert with fromStep reads the url the render step returned and mounts it through the BMC's virtual media.

On a BMC without virtual media (IPMI), this profile is refused before the workflow starts, because a fromStep ISO has no digest the site cache can verify. An ISO registered in the catalog works there (3d26428): name it with { use: 'media.insert', with: { image: 'img_…' } }, and metal pins it at the site, boots it with iPXE sanboot from the site cache, turns the later setBootDevice to cd into pxe, and has media.eject remove the boot entry. A catalog ISO is the same for every machine, so a per-node join token cannot live inside it. See sanboot instead of virtual media.

What the tests assert:

• profiles.validate resolves every step of the plugin profile.
• Enroll, ready, deploy, deployed with role host, undeploy, ready; the ISO is ejected and the token appears only inside the ISO, never in a command.
• The v0.2 acceptance in simulation: 20 consecutive unattended deploy and release loops, a fresh single-use token per install, and every transition into ready preceded by exactly one attested wipe in that workflow. A release whose wipe is not attested ends failed with failedFrom: 'releasing' and maintenance on.

Walkthrough: Talos on an agentless pool

Talos manages itself through its own API and never runs bcd. This is walkthrough 3 of PLAN.md §5.6: a pool with agent: 'none', inventory from Redfish, and a third-party step that reaches the Talos API through the site agent, because the control plane never talks to machines directly.

import { field, manifest } from 'better-compute/core';
import { httpViaSite } from 'better-compute/metal';
import { defineProfile, defineStep, stepNotReady } from 'better-compute/provision';

const talosReady = defineStep({
  id: 'wait.talosReady',
  runsOn: 'control-plane',
  relays: ['site'],
  sends: ['site.httpProxy'], // the kinds it relays: the workflow's batch scope includes them
  timeout: '20m',
  retrySafe: true,
  input: manifest({ port: field.int({ min: 1, max: 65535 }).default(50000) }),
  async run(input, context) {
    const { rows } = await context.sql.query<{ addresses: string[] }>(
      'SELECT addresses FROM metal_nodes WHERE node_id = $1', [context.node!.id],
    );
    const address = rows[0]?.addresses[0];
    if (!address) throw stepNotReady(10_000);
    const response = await httpViaSite(context, {
      url: `https://${address}:${input.port}/machine/version`,
      tlsSpkiSha256: talosCaSpki, // Talos serves its own CA: pin it
      poll: true, // each call returns the newest answer and sends the next request
    });
    if (response.status !== 200) throw stepNotReady(5_000);
    return { talos: response.json<{ version: string }>().version };
  },
});

const talos = defineProfile({
  id: 'talos-1.8',
  version: 1,
  target: 'metal',
  role: 'host',
  steps: [
    { use: 'media.insert', on: 'site', with: { url: 'https://cache.site-a/talos-1.8-metal-amd64.iso' } },
    { use: 'management.setBootDevice', on: 'site', with: { device: 'cd' } },
    { use: 'power.setPower', on: 'site', with: { action: 'cycle' } },
    { use: 'wait.talosReady' },
  ],
});

The pool opts in to what an agentless machine cannot do:

await compute.api.pools.create(credential, {
  tenantId: 'root',
  siteId,
  class: 'metal-host',
  agent: 'none',
  policy: { allowUnverifiedRelease: true, provisioningCidrs: ['10.30.0.0/16'] },
});

• Enrollment runs metal-enroll, then the out-of-band profiles: Redfish inspection records the machine's MACs and IP addresses, and commissioning only powers it off. No ramdisk ever boots.
• Deploy mounts the Talos ISO, boots it and polls the Talos API until it answers. The node is deployed with role host, and every command went to the site agent.
• Release ejects the media and powers off. It reaches ready without an attested wipe only because the pool set allowUnverifiedRelease, and metal publishes node.release.unverified. Without it the release fails with UNVERIFIED_WIPE.
• Rescue needs a ramdisk and fails the node with UNSUPPORTED.
• The node never connects, so it is never schedulable for hold reservations. PLAN.md has such nodes serve binding: 'node' leases from v0.5.

What site.httpProxy allows

The site agent is a trusted proxy on the provisioning network, so httpViaSite refuses anything beyond talking to the machine itself, with INVALID_INPUT, before a command exists:

• The host must be an IP literal among the machine's recorded addresses. Names are refused, so DNS rebinding cannot redirect it.
• https only, unless the address is inside one of the pool's policy.provisioningCidrs (IPv4).
• No credentials in the URL, no Host or hop-by-hop headers, no line breaks in header values, and secret headers only as secret:// references that the agent resolves and never logs.
• Methods GET, POST, PUT and DELETE, with a timeout from 1 to 60 seconds (30 by default).

A redirect is DRIVER_ERROR rather than followed, and so is a response over 1 MiB.

site.httpProxy is a destructive kind (0d42de8, f66e236): a request can change the machine it reaches, so with approver keys configured it runs only in an approved batch, under the leaf node:<target node>. The step must declare sends: ['site.httpProxy'] as above: the engine records the kinds a workflow can send when it starts, and metal scopes the workflow's batch from them, so with approver keys set a relaying step without it sends commands the queue refuses at enqueue: INVALID_INPUT (not in the batch's scope), or INVALID_STATE with reason batch-required when the workflow has no batch at all. See who creates batches.

bcd site enforces its own limits on each request, whatever the command says, and fails anything else with forbidden-target: http or https without credentials; GET, POST, PUT or DELETE; a host that resolves only to site-internal addresses (private, CGNAT 100.64.0.0/10 or unique-local; never loopback, link-local and so metadata services, multicast or public addresses), with the connection made to the address it checked, never a second lookup; no redirects; a timeout of at most a minute (30 seconds by default); the pinned SPKI when given; secret headers fetched with a SecretRequest. It truncates the response at 1 MiB and says so. It never reaches a BMC: every BMC host it has driven with a bmc.* command is remembered with every address it resolved to (data_dir/bmc-hosts.json) and refused, by name and by resolved address, and so is any /redfish path and any address in the site agent's own [site] bmc_networks (local configuration the control plane cannot change; daemon install-script --bmc-networks writes it, and better-compute doctor warns about site agents without it). Under the v1.0 quorum a request other than a GET is a write, which runs only under a batch leaf bound to its exact payload; readiness polls such as Talos's are GETs.

The host must also belong to the command's target_node_id (a2a853d). Metal puts the site's agentless machines (nodes of agent: 'none' pools that are not retired) into the site agent's spec as SiteSpec.proxy_targets, each with the addresses Redfish inspection recorded or metal.enroll's addresses gave (3f19859; the only source for an IPMI machine, whose inspection reports none), and never its BMC host. The list is recomputed when such a machine is enrolled with addresses, when inspection records addresses and when the machine is retired (8391d74). bcd then requires an IP literal to be one of the target node's listed addresses, and a name to match a listed name (case-insensitive, trailing dot ignored) or to resolve only to listed addresses; a target node the list does not name is refused too, all with forbidden-target. The check applies once the site's spec lists any target, which it does whenever the site has an agentless machine.

From the command line

The metal commands (171d9be) are the metal methods with flags, each decided as the token's identity:

export BETTER_COMPUTE_TENANT=root
better-compute metal enroll --site site_01k5w… --pool pool_01k5w… --bmc-address 10.20.0.14 \
  --bmc-driver redfish --bmc-credentials secret://root/bmc-r12-14 --macs b8:ce:f6:00:00:01
better-compute metal deploy node_01k5w… --profile ubuntu-24.04-gpu-host --profile-version 1
better-compute metal power node_01k5w… --action cycle
better-compute metal undeploy node_01k5w… --wipe secure-erase
better-compute metal abort node_01k5w… --reason 'wrong profile'
better-compute metal bulk --pool pool_01k5w… --verb retire --nodes-file retire.txt --reason 'EOL rack r12'

• Credentials are a secret://, vault:// or env:// reference (--bmc-credentials), or --bmc-username with --bmc-password-file. There is no password flag. See enrolling a machine.
• One verb per node: manage, inspect, provide, rescue, unrescue, recover and retire take only the node; deploy takes --profile and --profile-version, undeploy takes --wipe, abort takes --reason, and power takes --action (on, off, graceful-off, reset, graceful-restart, cycle, nmi).
• Many nodes at once: metal bulk runs one --verb (manage, inspect, provide, undeploy, rescue or retire) on the nodes of --pool listed in --nodes (comma-separated) or --nodes-file (one per line, deduplicated) as one metal.bulk batch. --reason is required: it is what the quorum's approvers see when they review the batch with batch approve. --wipe is for --verb undeploy only.
• PXE discovery: metal discovered --site ID lists what the site's agents saw PXE-booting from unknown MACs, and metal confirm-mac NODE --mac MAC binds the MAC of a machine enrolled without one (002dd95), which unblocks an IPMI machine waiting in metal.ensureRamdisk. The console's PXE discovery page does the same from a browser (15c11ee). See discovering new machines.

workflows list --node, workflows get, workflows retry --from-step and commands list --node follow what a verb started. Provision profiles have their own commands (profiles create from a JSON document, validate, render, update --visibility, delete). See command groups.

Release and retire

• metal.undeploy takes wipe: quick (the default), secure-erase, crypto-erase, or none. Every mode but none must come back node-signed with every disk's read-back passed before the machine is ready.
• metal.retire works from ready, failed, enrolling, adopting or adopted, so undeploy a deployed machine first. It crypto-erases, powers off and revokes the machine's identity; retired is final.