Drivers
Declare a driver with defineDriver, bind drivers to machines through hardware profiles, and run one out of process with the bcx driver SDK.
A is the code that does one hardware job for a machine: read and set its power
through the , pick its boot device, write an image to its disk, or run its VMs. Each job is a
driver kind (Ironic calls it an interface), and one driver may implement several kinds together, which is what
Ironic calls a hardware type: a redfish driver serves power, management and virtual-media for the same BMC.
Code registers drivers; data selects them. A driver is a defineDriver() declaration. Which driver a machine uses for
each kind is data: a binding on the node, inherited from its pool's hardware profile.
Driver kinds
DriverKinds in @better-compute/core is empty. Packages open it by declaration merging, so a kind type-checks only
where the package that declares it is imported:
| Kind | Declared by | Operations | Built-in drivers |
|---|---|---|---|
power | metal | getPower(), setPower(action) | v0.2 |
management | metal | setBootDevice(device, { persistent, uefi }), optional rotateCredential(newPassword, { username }) | v0.2 |
boot | metal | prepareRamdisk(spec), prepareInstance(boot), cleanUp() | v0.2 |
deploy | metal | deploy(spec), tearDown({ wipe }), both AsyncIterable<Progress> | v0.2 |
inspect | metal | inspect(), a NodeInventory | v0.2 |
virtual-media | metal | insert(iso), eject() | v0.2 |
console | metal | open(), a byte stream | draft until v0.3 |
bios | metal | getSettings(), applySettings(settings) | draft until v0.5 |
raid | metal | applyLayout(layout), clear() | draft until v0.5 |
firmware | metal | components(), update(component, image) | draft until v0.5 |
runtime | vps | traits, list(), apply(spec), reboot(id), remove(id), optional console(id) | v0.3 (in bcd: libvirt-kvm, 362317c, and cloud-hypervisor, 2598f0a, both written and tested against fake tools only); bcd also serves a bcx plugin of this kind as a runtime (list, apply, remove) |
Importing only @better-compute/core leaves power and runtime unusable, and tests/stubs-driver-kinds.test.ts
checks exactly that. BMC drivers and credentials describes what each metal kind is for.
A new kind is the same declaration in your own package:
declare module '@better-compute/core' {
interface DriverKinds {
'acme-cooling': { getFanCurve(): Promise<number[]>; setFanCurve(curve: number[]): Promise<void> };
}
}Declaring a driver
import { defineDriver, field, manifest } from 'better-compute/core';
import 'better-compute/metal'; // merges the metal kinds into DriverKinds
export const acmeBmc = defineDriver({
id: 'acme-bmc',
version: '1.0.0',
runsOn: 'site',
config: manifest({
address: field.url(),
credentials: field.secret(),
timeout: field.duration().default('30s'),
}),
capabilities: ['virtual-media'],
implements: {
power: (config, ctx) => ({
async getPower() {
return 'on';
},
async setPower(action) {
ctx.emit({ phase: 'power', message: `setting ${action}` });
},
}),
'virtual-media': (config) => ({ insert: async (iso) => {}, eject: async () => {} }),
},
retrySafe: { power: ['getPower'], 'virtual-media': ['insert', 'eject'] },
async validate(config, ctx) {
await ctx.secrets.resolve(config.credentials); // a live check, e.g. log in once
},
});defineDriver returns the definition frozen with extension: 'driver'.
Prop
Type
Every factory receives the parsed config and a DriverContext:
Prop
Type
Mark an operation retry-safe only when running it twice does no harm. A power cycle is not retry-safe; reading the
power state is. The flag decides whether an interrupted call may be repeated, and, over bcx, whether an error the
plugin returns is marked retryable.
Registering and listing
Pass drivers to betterCompute({ drivers }), or ship them in a module's extensions (see
modules). The registry refuses a second driver:<id>@<version> with INVALID_CONFIG.
drivers.list returns what the deployment registered, so metal.enroll callers
and the console can offer valid choices. It returns declarations only, never a configuration or a credential:
const { items } = await compute.api.drivers.list(credential, { tenantId: 'root', kind: 'power' });
// [{ id: 'acme-bmc', version: '1.0.0', runsOn: 'site', kinds: ['power', 'virtual-media'],
// capabilities: ['virtual-media'], retrySafe: { power: ['getPower'], 'virtual-media': ['eject', 'insert'] } }]It is decided as compute:drivers:read on the tenant, filters by kind (lower-case letters, digits and hyphens), and
sorts by id, then version, with kinds, capabilities and operations sorted too.
Binding drivers to machines
Machines of one pool usually share their BMC and boot hardware, so bindings live in a hardware profile that
metal({ hardwareProfiles }) names and a pool selects with policy.hardwareProfile:
import { metal } from 'better-compute/metal';
const bareMetal = metal({
hardwareProfiles: {
'gpu-redfish': {
description: 'HGX hosts with Redfish BMCs',
drivers: { power: 'redfish', management: 'redfish', boot: 'ipxe', deploy: 'image-stream' },
},
},
});A binding is a driver id, or { driver, config } when the binding carries config. At metal.enroll,
resolveDriverBindings builds the node's bindings in three layers, each overriding the one before per kind:
- The enrolled BMC binds
power,managementandvirtual-mediatobmc.driver, with{ address, credentials }as config (the credentials as asecret://reference). - The hardware profile: the enroll input's
hardwareProfile, else the pool'spolicy.hardwareProfile. An unknown name isINVALID_INPUT. - Per-node overrides.
The result is stored on the node (nodes.drivers) and served in its NodeSpec. metal.enroll checks only that
bmc.driver is present and a well-formed id string; PLAN.md has bindings validated against the registry, which is not done yet. The
hardware profile also carries the BIOS settings and RAID layout that commissioning applies (drafts until v0.5); see
profiles and images.
From a profile step to a driver
A profile step names a driver operation as <concern>.<operation> with on: set to an agent. When no registered
step has that id, the workflow engine turns it into one driver.invoke command whose args carry the concern, the
operation, the step's with, and the driver and config bound in the profile's own drivers:
{
"workflowId": "wf_…",
"stepId": "main.3",
"attempt": 0,
"targetNodeId": "node_…",
"args": { "concern": "bios", "operation": "applySettings", "args": { "…": "…" }, "driver": "acme-bmc" }
}targetNodeId appears when the command goes to the site agent rather than the machine itself. provisionModule()
registers the driver.invoke kind: lane provision, destructive (so always signed, since driver operations include
power and deploy), not retry-safe, roles node, site and ramdisk, payload DriverInvokeRequest, and a 10-minute
default timeout (each step's own timeout sets notAfter). tests/e2e/bcx.test.ts runs such steps end to end: a
module profile binds power.getPower and power.setPower to a bcx plugin's driver, and both go out as
driver.invoke through the real engine, queue and gateway to a FakeDaemon that runs them in the plugin process
(fakeDriverInvoke from @better-compute/testing). bcd runs driver.invoke with the same semantics in the
plugins it hosts (see driver plugins in bcd); no end-to-end test runs bcd for it yet. Metal's own steps
(power.setPower, media.insert, ...) are registered control-plane steps that relay typed commands instead; see
workflows.
BMCs on the site agent
The control plane never dials into a site. Metal's BMC steps run in the control plane, where the node's BMC address and
credential reference are, and send one BmcOperationRequest (driver, address, credentialsRef, operation,
argsJson, optional tlsSpkiSha256 and driverOptionsJson) to the site's agent under a kind such as bmc.query, bmc.boot,
bmc.power, bmc.rotate or bmc.firmware. The site agent fetches the credential by reference, runs <concern>.<operation> on the named driver,
and answers a BmcOperationResult. PLAN.md drives BMCs through bmclib (Redfish, IPMI and vendor BMCs); bcd site
has its own Redfish client built in (see Redfish in bcd site),
and since 51806ab it sends a bmc.* operation whose driver is anything else to a bcx plugin it hosts:
- It picks the hosted driver whose id is the request's
driverand whose manifest lists the operation's concern among its kinds (power.setPower: concernpower, operationsetPower). No such driver answersunsupported-driver("no hosted driver ipmi implements power"), and nothing is invoked. - It calls
Invokewith the concern askind, the operation, the BMCaddress,tlsSpkiSha256(hex) and the request'sdriver_options_json(asoptions) asconfig_json, the remaining arguments asargs_json, the command id as the idempotency key, the command'straceparentand itsnot_afteras the deadline. The options are nested, so they can never override the address or the key pin (7753df0). Options that are not a JSON object fail the command (invalid-payload); the built-in Redfish driver takes none, so a Redfish request with options answersinvalid-config. - The credential is fetched by reference as for Redfish and sent only as secrets (
username,password, and a rotation'snewPassword, which is taken out of the arguments), never in the arguments or the config. - The kind's operation allowlist, the destructive-batch quorum and the site agent's memory of BMC addresses (which
site.httpProxyrefuses) apply exactly as for Redfish. ADriverErrorkeeps its code; a result that is not JSON isbmc-error.
A new BMC family is therefore a bcx plugin on the site agent, whose driver id the node's BMC binding names. Its
operations can also run through profile steps sent as driver.invoke. The command kinds and credential handling are
on BMC drivers and credentials.
bcx-ipmi, the IPMI driver
daemon/cmd/bcx-ipmi is the first Go plugin in the repository (51806ab): driver ipmi, kinds power,
management and, since 7818f19, inspect, runsOn: site, over ipmitool with IPMI-over-LAN (-I lanplus).
| Operation | ipmitool command | Answer |
|---|---|---|
power.getPower | chassis power status | {"power":"on"} or {"power":"off"} |
power.setPower (action) | chassis power on, off, cycle, reset or soft (shutdown is soft) | {} |
management.setBootDevice (device, persistent) | chassis bootdev pxe, disk, cdrom or bios (hdd is disk, cd is cdrom), plus options=persistent | {} |
inspect.inspect | fru print 0 and chassis power status (required), then mc info, mc guid, lan print and sdr list (best effort) | The Redfish driver's shape, less the host's NICs (below) |
inspect.inspect answers:
| Field | From |
|---|---|
manufacturer, model, serialNumber | The builtin FRU (fru print 0): Product Manufacturer, Product Name, Product Serial, falling back to the board's (and for the serial the chassis') fields |
power | chassis power status |
macs, ipAddresses | Always empty: IPMI does not name the host's NICs reliably |
productUuid | The System GUID of mc guid (Get System GUID), lower-cased, when it is a canonical UUID other than all zeros (7cd1dd2). It is the host's SMBIOS UUID on most boards, in whichever byte order the BMC prints it; metal matches every order, so a machine enrolled without MACs is found when it PXE-boots. Left out when the BMC has none |
board | { manufacturer, product, serial } from the FRU's board fields |
bmc | { firmware, manufacturer } from mc info and { mac, ipAddress } from lan print: the BMC's own identity, not the host's |
sensors | { count, ok, problems } from sdr list; problems names at most 32 sensors whose status is neither ok nor ns |
A failed FRU read or power read fails the operation; a failed mc info, lan print or sdr list only leaves its
part out. Each ipmitool run is a separate process with the password through its own pipe.
- Config is
{ "address": "10.0.0.5[:623]", "options": { "cipherSuite": 17 } }: a host name or IP address with an optional port (ascheme://prefix is stripped; anything else isinvalid-address), and the BMC's driver options, which the site agent nests underoptionsexactly as enrollment set them (bmc.driverOptions, 7753df0). The plugin decodes them strictly:cipherSuiteis the only key, and any other key isinvalid-config. The RMCP+ cipher suite is 17 (HMAC-SHA256, AES-128) by default, or 3, 8 or 12 for older BMCs: only suites with an integrity HMAC and AES. Suite 0 (the IPMI authentication bypass), 1 and 2 (no encryption) and anything else areinvalid-config. The manifest'sconfig_schema_jsonsays the same. Enroll an old BMC withbmc.driverOptions: { "cipherSuite": 3 }; see enrolling a machine. - No key pin. IPMI has no TLS, so a BMC enrolled with a
tlsSpkiSha256pin is refused (pin-unsupported) rather than driven unpinned: use a Redfish driver, or remove the pin. - Credentials come as the secrets
usernameandpassword. The password reachesipmitoolthrough a pipe it reads once (-f /dev/fd/3), never its arguments (a process listing) nor its environment (readable in/procby any process of the same user); an IPMI 2.0 password has at most 20 bytes.ipmitoolruns by absolute path (the first of/usr/bin,/usr/sbin,/usr/local/binand/usr/local/sbinthat has it) with a fixedPATH=/usr/sbin:/usr/bin:/sbin:/binand nothing else in its environment (933c51b). A username outside[A-Za-z0-9._@-]{1,64}, a missing password, a longer one, or one with whitespace (ipmitool -fkeeps only the file's first word; b12c217) iscredentials-unavailable. - Arguments are checked against fixed sets before anything runs: an unknown action or device is
invalid-args, and any other operation isunsupported-operation. - Failures. Each
ipmitoolrun has 30 seconds (unreachable, retryable, on timeout); a failingipmitoolisbmc-error(retryable) with the first line of its output.HealthisNOT_SERVINGwhenipmitoolis not installed.
It covers power, boot device and inspection: virtual media, firmware, BIOS, RAID and credential rotation stay
Redfish's, though a catalog ISO still boots on an ipmi machine over PXE (sanboot; see
sanboot instead of virtual media). goreleaser builds it without cgo and ships it as bcx-ipmi_<version>_linux_{amd64,arm64}.tar.gz (348ff56),
and the api's downloads route serves that archive (97b2ac2; see
serving the binaries); daemon install-script --install-plugins bcx-ipmi
installs it with its pin after checking the archive against the signed release manifest (adf2ae9; see
installing plugins from the release). To use it, have it
in a site agent's [plugins] dir (below) and enroll the machine's BMC with driver ipmi, without a key pin, and in a pool
with an agent with its PXE macs or its productUuid (PXE discovery then proposes the MAC for metal.confirmMac;
3d26428, 8de6e2f), since IPMI inspection finds no MACs. Metal knows what ipmi serves and refuses
up front, with UNSUPPORTED (reason bmc-driver-unsupported), any verb whose workflows need more (a3ec8d0); see
IPMI through bcx-ipmi and
what metal checks before a workflow. Its tests
run it against a fake ipmitool only.
A driver of your own is checked the same way once metal knows its operations: metal({ bmcDrivers: { 'acme-bmc': ['power.getPower', 'power.setPower', 'management.setBootDevice'] } }). Otherwise the kinds its site agents report
(the bc.dev/driver-kind.<id>.<kind> labels each agent's Hello writes) decide, and a driver neither names is not
checked in advance.
Out-of-process drivers: bcx
bcx is one protocol, bettercompute.driver.v1.DriverPlugin, for drivers that run in their own process, in any
language, next to bcd or the control plane. It is go-plugin style: the host execs the plugin, the plugin prints one
handshake line, and the host dials it over gRPC on HTTP/2 with mutual TLS.
| RPC | Does |
|---|---|
Describe | the drivers the plugin serves, as DriverManifests, its version and the bcx protocol version (1) |
Validate | checks a config against the manifest and the driver's live checks; answers { valid, issues[] } with JSON pointers |
Invoke | runs one operation; streams Progress, then exactly one result_json or DriverError { code, message, retryable } |
Stream | a duplex byte stream (consoles); the first frame is open |
Health | SERVING or NOT_SERVING |
A DriverManifest is { id, version, kinds[], runs_on, config_schema_json, capabilities[], retry_safe_operations[] },
with the config schema as JSON Schema 2020-12 (manifest().jsonSchema()) and retry-safe operations as
<kind>.<operation>. The proto is draft and excluded from buf breaking until v0.5 ships.
The plugin side
serveDriver serves the same defineDriver object you would register in process:
#!/usr/bin/env node
import { serveDriver } from '@better-compute/driver-sdk';
import { acmeBmc } from './drivers/acme-bmc.js';
await serveDriver(acmeBmc, { pluginVersion: '1.0.0' }); // or serveDriver([acmeBmc, acmeMedia], …)Run by hand, without the magic cookie, it prints that it is a plugin and exits with code 1. Started by a host, it prints the handshake line on stdout, writes JSON log lines to stderr (the host forwards them to its own log), shuts down gracefully on SIGTERM or when its stdin closes, and ignores SIGINT.
| Option | Default | Does |
|---|---|---|
pluginVersion | '' | DescribeResponse.plugin_version |
streams | none | operations, as <kind>.<operation>, that return a byte stream; each is advertised as capability bcx.stream:<kind>.<operation> |
network | unix, or tcp on Windows | where to listen |
log | JSON lines on stderr | the plugin's logger |
handleSignals | true | the SIGTERM, stdin and SIGINT behaviour above |
shutdownGraceMs | 5,000 | how long in-flight calls may run after shutdown starts |
env, stdout | process.env, process.stdout | for tests |
Throw driverError(code, message) for an error of your own: the code must be lower-case kebab-case, and the host sees
DRIVER_ERROR with details.driverCode. A thrown ComputeError travels as its code in kebab-case (INVALID_CONFIG
becomes invalid-config); any other error is driver-error with its message.
The host side
import { externalDriver } from '@better-compute/driver-sdk';
const acme = externalDriver({ command: ['/opt/bc/plugins/acme-bmc'], sha256: '9f3e…' });
export const compute = betterCompute({ drivers: [...builtins, ...(await acme.start())] /* , … */ });
// on shutdown: await acme.close();start() verifies the pin, execs the plugin, reads the handshake within startTimeoutMs, dials it and calls
Describe. It resolves with one RemoteDriverDefinition per served driver, a normal DriverDefinition whose parts
are backed by the plugin:
implements[kind](config, ctx)returns a proxy whose every method callsInvoke. Awaiting a call sends progress toctx.emit; iterating it (for await (const p of deploy(spec))) yields the progress instead.configis a Standard Schema built from the plugin's JSON Schema. The host checks structure (types, enums, patterns, lengths, ranges, required and unknown fields) and passes the value through unchanged; the plugin parses it again, applying defaults.validatecalls the plugin'sValidate;retrySafeandcapabilitiescome from the manifest.
startExternalDrivers([a, b]) starts several hosts and returns all their drivers.
| Option | Default | Does |
|---|---|---|
command | required | the command line, run without a shell; command[0] is resolved against cwd or PATH |
sha256 | required | hex SHA-256 of the pinned file, checked before every start and restart |
env | none | extra environment for the plugin |
cwd | process.cwd() | the plugin's working directory |
startTimeoutMs | 30,000 | time to print the handshake and answer Describe |
callTimeoutMs | none | deadline of calls whose context carries no deadline |
restart | { initialDelayMs: 250, maxDelayMs: 30000 } | crash restart back-off, doubling |
shutdownGraceMs | 5,000 | between stdin EOF and SIGTERM, and SIGKILL |
log | warnings on stderr | plugin stderr, exits and restarts |
Wire rules
- Pinning.
sha256pinscommand[0]. Whencommand[0]isnode,nodejsorbun, it pins the script instead: the first later argument not starting with-(afterbun's optionalrun). Interpreter options must use the--name=valueform, and only the entry file is pinned, so bundle a JS plugin into one file. A mismatch isINVALID_CONFIG, and the host stops restarting. - Mutual TLS. Each side makes an ephemeral self-signed P-256 certificate. The plugin accepts exactly the host's
certificate from
BCX_CLIENT_CERT, not merely one signed by the same key; the host trusts only the certificate in the handshake line. - Environment. The plugin does not inherit the host's environment: it gets
PATH, the temp directories,HOME, a few OS basics, yourenv, and the threeBCX_*variables. - Arguments and results.
args_jsonis a JSON array of the positional arguments (trailingundefineds dropped); a non-array value from another host is taken as the single argument. Results are JSON, andundefinedtravels asnull. - Deadlines. The host sends
ctx.deadline, or now pluscallTimeoutMs. The plugin turns it intoctx.signaland ends the call withtimeouteven when the operation ignores the signal; the host gives up one second later. Aborting the caller'sctx.signalcancels the call. - Secrets. The host resolves only top-level config fields the schema declares secret (
x-bc.kind: 'secret'orwriteOnly), and only when their value is aSecretRef, and sends them in the request. In the plugin,ctx.secrets.resolve(ref)resolves only refs delivered that way. - Streams. For an advertised stream operation the proxy opens
Streaminstead ofInvoke. The plugin answers theopenframe with an emptydataframe, then bytes flow both ways untilclose. The deadline bounds opening the stream, not its lifetime. - Supervision. A crash or a lost connection fails calls in flight with a retryable
DRIVER_ERROR(driverCode: 'unreachable'), restarts the plugin with back-off, and makes new calls wait for the restart. A restarted plugin that describes other drivers is logged and the registered ones are kept.close()closes stdin, sends SIGTERM (not on Windows), then SIGKILL aftershutdownGraceMs.
Driver plugins in bcd
bcd hosts bcx plugins in bcd node and bcd site, with the Go side of the protocol in daemon/internal/bcx.
(bcd ramdisk accepts driver.invoke too, but it has no config file, so no plugin directory and no plugins.) Point
the agent at a plugin directory in config.toml; there is no default in bcd, and daemon install-script --plugins
writes the one below (see installing a plugin node):
[plugins]
dir = "/usr/local/lib/bcd/plugins"The directory holds the plugin binaries and a pins file in sha256sum format, one <sha256 hex> <file name> line
per plugin (# comments and blank lines are skipped). Keep both root-owned, like config.toml:
9f3e0c… acme-bmcAt start, before its session opens, the agent runs every plugin the pins file names:
- It opens the binary, hashes it through that open file and refuses to run it unless the SHA-256 equals its pin.
On Linux the plugin is then executed from the same open file (fd 3,
/proc/self/fd/3), never by path again, so what runs is what was hashed even if the file at the path is replaced in between. - It makes an ephemeral self-signed P-256 certificate and a fresh 0700 socket directory, and execs the plugin with
only
PATH,BCX_MAGIC_COOKIE,BCX_SOCKET_DIRandBCX_CLIENT_CERTin its environment, sandboxed as below. The plugin's stderr goes to bcd's. - It waits up to 10 seconds for the handshake line (
bcx|1|unix|<socket>|grpc|<cert>), dials the socket over gRPC with mTLS, pinning the plugin's certificate byte for byte, and callsDescribe; a plugin that reports a bcx protocol other than 1 is stopped. Hello.driversadvertises each driver the plugins describe, so the gateway labels the nodebc.dev/driver.<id>=<version>.
A missing pins file runs no plugin (logged as a warning); an unreadable pin line, a pin mismatch or a plugin that
fails to start is logged and skipped. Plugins are stopped with the agent. Two paths reach them: driver.invoke,
below, and, on bcd node, the instances converger, which runs a hosted driver of kind runtime whose id is not a
built-in runtime as one of its runtimes (list, apply(spec), remove(id) over Invoke; snapshots unsupported).
See walkthrough 4.
bcd supervises every plugin that started (bcx.Supervise, 9006eee, PLAN.md §5.5). When a plugin's process exits,
bcd starts it again after a back-off that begins at 1 second and doubles to at most 1 minute (back to 1 second once a
run lasted a minute), and each restart hashes the binary against its pin and sandboxes it again. An operation that
arrives during a restart fails with a retryable unavailable error. A plugin that comes back with other drivers is
logged, and the node keeps advertising the first set until bcd itself restarts. A plugin that fails its first start
is still only logged and skipped, and bcd does not call Health on its own.
The plugin sandbox
On Linux bcd runs every plugin confined, as PLAN.md §5.5 asks, set by the rest of the [plugins] section:
[plugins]
dir = "/usr/local/lib/bcd/plugins"
# user = "bcx" # unset (the default): each plugin gets its own uid
privileged = "kata" # comma-separated: plugins that run as root, unfiltered
sandbox = true # the default: no_new_privs + seccomp through bcd's shim
cgroup = "/sys/fs/cgroup/system.slice/bcd.service/plugins" # in bcd's delegated cgroup
memory_max = 1_073_741_824 # bytes per plugin; 0 or unset: no limit
pids_max = 256 # tasks per plugin; 0 or unset: no limit| Key | Default | Effect |
|---|---|---|
user | none | When bcd runs as root, unset gives each plugin its own uid and gid from a block bcd keeps for itself (0x7FFE0000 to 0x7FFEFFFF, at the top of the range systemd leaves unassigned, clear of system users, systemd's DynamicUser range and container ranges), so no plugin can read another's memory. The uid is derived from the plugin's name, moves past any uid another plugin or an existing account has, and plugins are assigned in name order, so a uid is stable across restarts and reordering the pins file changes nothing (6827df8). Set, every plugin runs as that account and its primary group. Either way no supplementary groups, and the socket directory is chowned to the plugin. root (or 0) is a configuration error, and an account that resolves to uid 0 or does not exist keeps that plugin from running (logged). Ignored when bcd is not root |
privileged | none | Plugins, by name and comma-separated, that run as root without the shim or the filter: a runtime plugin that makes namespaces and mounts (Kata). Each is logged at start as a privileged plugin. Every other plugin is confined |
sandbox | true | Runs each unprivileged plugin through bcd itself as a shim (the hidden bcd __bcx-exec <name> command, bcx.RunShim): on one locked OS thread it sets no_new_privs, installs a seccomp filter synchronised to every thread (SECCOMP_FILTER_FLAG_TSYNC), then execs the verified binary from fd 3. false drops the shim; the uid still applies |
cgroup | none | A directory under /sys/fs/cgroup inside bcd's delegated cgroup; each plugin, privileged ones included, runs in its own child bcx-<name>. bcd prepares it at start (below) |
memory_max | none | Written to the child cgroup's memory.max |
pids_max | none | Written to the child cgroup's pids.max |
The seccomp filter (amd64 and arm64) answers EPERM to what no driver plugin needs and a compromised one could abuse:
kexec, loading and deleting kernel modules, mounts (mount, umount2, pivot_root and the new mount API),
namespaces (unshare, setns, and clone with any CLONE_NEW* flag), swap, reboot, bpf, io_uring_*, ptrace
and process_vm_readv/writev, the kernel keyring, perf_event_open, userfaultfd, open_by_handle_at,
iopl/ioperm (amd64), setting the clock, acct, quotactl, lookup_dcookie and syslog. clone3 answers
ENOSYS, since its flags are out of a filter's reach and libc and Go fall back to clone. So a plugin cannot make a
user and network namespace to reach nf_tables. A system call from another ABI (x32, a foreign architecture) kills
the process. On other architectures the shim installs no filter.
Every plugin also runs in its own process group, is killed if bcd dies (Pdeathsig), and is killed as a group on
close, so whatever it started goes with it. The sandbox is Linux only: on another OS bcd runs a plugin by path and
refuses to start it when a shim, a user or a cgroup is asked for, so there a plugin runs only with
sandbox = false. The Go tests check that a sandboxed plugin's perf_event_open and unshare(CLONE_NEWUSER) get
the filter's EPERM, that /proc/<pid>/exe is the verified binary, and that the keys parse and validate.
With cgroup set, bcd prepares the directory before it starts any plugin (d902063): it creates it and enables the
memory and pids controllers in it and in its parent. cgroup v2 gives no controllers to the children of a cgroup that
holds processes itself, so when bcd finds itself running directly in the parent (systemd older than 254, without
DelegateSubgroup=main), it first moves itself into a main leaf beside the plugins. A path outside
/sys/fs/cgroup, or a cgroup bcd cannot write (not delegated), means no plugin runs, and the log says why. To set a
node up for all this, see installing a plugin node.
driver.invoke then runs in the plugins, as the FakeDaemon's fakeDriverInvoke does:
| Step | bcd |
|---|---|
| Pick the driver | the one hosted driver whose manifest lists the concern among its kinds, restricted to driver when the args name one; none is driver-not-found, several are ambiguous-driver (the step must name one) |
| Secrets | every top-level config field whose value is a secret://, vault:// or env:// reference is fetched with a node-signed SecretRequest bound to the workflow step and sent as a SecretValue; the config keeps the reference. A failed fetch is secret-unavailable |
| Request | InvokeRequest with the concern as kind, the operation, config_json, args_json (the step's args), the idempotency key wf:<workflowId>:<stepId>#<attempt>, the command's traceparent, and the command's not_after as the deadline, which also cancels the call |
| Result | DriverInvokeResult { result_json }; a DriverError keeps its code, any other failure is driver-error |
A Go plugin serves the protocol with bcx.Serve(ctx, impl, os.Stdout), where impl is a
driverv1connect.DriverPluginHandler: it checks the magic cookie, listens on plugin.sock in the host's directory,
accepts only the host's certificate from BCX_CLIENT_CERT, and prints the handshake line. It is the start of a Go
plugin SDK, internal to the daemon module for now. The Go unit tests run a pinned plugin (the test binary executed
again as one), a refused unpinned binary, and driver.invoke through a hosted driver with a secret; no end-to-end
test runs bcd with a plugin against the control plane yet.
The control plane does not merge Hello.drivers into its driver registry. What it does with them today is label
the node: at every Hello the gateway writes each advertised driver as the system label
bc.dev/driver.<id>=<version>, each kind a driver serves as bc.dev/driver-kind.<id>.<kind>=1 (14ceb1e; kinds
outside [a-z][a-z0-9-], 1 to 32 characters, skipped, at most 32 per driver), and each Hello capability as
bc.dev/cap.<name>=1 (such as bc.dev/cap.instance.network-policy), and drops the ones no longer advertised, so
placement can require a runtime, a driver's kind or a capability with PlacementRequest.nodeLabels, '*' matching
any value. The kind labels are meant for BMC steps that check a site agent's driver serves a concern before they run
(bc.dev/driver-kind.ipmi.inspect); no metal step reads them yet (see placement). The FakeDaemon
only advertises an nvidia driver of kind config on GPU hosts. PLAN.md's v0.5 acceptance includes "a third-party TS driver runs on a
node via bcx" and a Kata runtime plugin in Go (walkthrough 4).
Installing a plugin node
Two things in the stock bcd.service stop the sandbox: its capability bounding set lacks CAP_SETUID and
CAP_SETGID, and ProtectControlGroups=yes keeps the cgroup tree read-only. The install script handles both
(4225772):
better-compute daemon install-script --pool pool_vps --site fsn1 \
--gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
--bcd-version 0.1.0 --sha256 amd64=<hex> \
--plugins --privileged-plugins kata > install-bcd.sh--pluginswrites a[plugins]section withdir = "/usr/local/lib/bcd/plugins"(root-owned, 0755, so each plugin's uid can reach its binary) andcgroup = "/sys/fs/cgroup/system.slice/bcd.service/plugins", unless the node'sconfig.tomlalready has one.--privileged-plugins NAMESimplies--pluginsand addsprivileged.- On every run, the script installs the drop-in
/etc/systemd/system/bcd.service.d/plugins.confwhenconfig.tomlhas a[plugins]section, and removes it when not:CAP_SETUID CAP_SETGID CAP_KILLadded to the bounding set,Delegate=yeswithDelegateSubgroup=main,ProtectControlGroups=no, andRestrictNamespaces=noonly whenprivilegednames a plugin. - Copy the plugin binaries into the directory and list them in its
pinsfile, both root-owned, then restartbcd.service. The script installs only the plugins a release ships (--install-plugins bcx-ipmiwith--release-manifest, adf2ae9); any other plugin is yours to place.
An upgrade keeps the [plugins] section, and with it the drop-in. See
the plugin drop-in for the unit settings.
Testing a driver
Unit-test the definition in process: call implements[kind](config, ctx) with a context you build and assert on the
operations. Then test the plugin as a real subprocess, as tests/driver-sdk-*.test.ts do with
tests/fixtures/bcx-echo-driver.mjs:
import { externalDriver, pinnedFile, sha256File } from '@better-compute/driver-sdk';
const command = [process.execPath, 'dist/acme-bmc-plugin.js'];
const host = externalDriver({ command, sha256: await sha256File(pinnedFile(command)) });
const [acme] = await host.start();
const power = await acme.implements.power({ address: 'https://10.0.0.10', credentials: 'secret://root/bmc' }, context);
expect(await power.getPower()).toBe('on');
expect(await host.health()).toEqual({ serving: true, message: '' });
await host.close();tests/fixtures/bcx-loader.mjs lets a plain node subprocess load the workspace's TypeScript sources, so the fixture
runs without a build. The suites cover the handshake and pinning, Describe, Validate, Invoke with progress, errors,
deadlines, cancellation, secrets and streams, and supervision (crash restart, a pin changed under a running host,
graceful close):
pnpm vitest run tests/driver-sdk-handshake.test.ts tests/driver-sdk-host.test.ts tests/driver-sdk-supervision.test.tsBetter Compute is created by Sean Filimon
Last updated
Sinksexperimental
Event and audit sinks receive every committed row at least once, each with its own durable cursor. How dispatch works, the cursor pitfalls, poison events, and usage.
Attestorsexperimental
Write an AttestorVerifier that admits machines on your own evidence, defer a join for a second round, use evidence once, and register it with JoinService.