Site agent and ramdisk

bcd has two modes for bare metal besides bcd node. bcd site is the site agent: it drives BMCs, makes site.httpProxy requests for agentless machines, and, with a [boot] section, runs the site's boot server (proxy DHCP, TFTP, iPXE scripts, the image cache). bcd ramdisk runs inside the PXE ramdisk that boot server boots: it joins from its kernel command line and runs the commissioning and deploy kinds (wipe, image, kexec, diagnostics). Both are the same binary and the same agent core as bcd node (session, journal, signed commands); only the kinds, the configuration and the extra servers differ.

This page is the daemon-side view: flags, config.toml keys, start-up checks, what each server answers, the exact scripts and error codes. What the control plane does with all of it is in the bare-metal guide: sites and site agents (the boot lease, discovery, the discovery attestor), BMC drivers and credentials (the operations metal sends, IPMI, sanboot) and deploying machines (what each ramdisk kind does to disks).

The two modes

Both reach the gateway outbound only, like every mode (see outbound only). They were added in a302223 and 21a4647 (ramdisk) and 48cb975, 0c35daf and fc74ed9 (site).

bcd site

Flags and start-up

bcd site takes one flag:

It takes no positional arguments (exit 2). At start it checks, in this order:

Without [boot] listen it logs no [boot] listen configured: this site agent does not serve PXE machines and runs the agent alone. With it, the boot server runs beside the agent. A boot server that fails later (no kernel or initrd, a port it cannot bind) is logged as boot server stopped and does not stop bcd site: BMC work and site.httpProxy go on. Watch for that log line.

Ports 67, 69 and 4011 are privileged. The unit the install script writes is for bcd node and has no CAP_NET_BIND_SERVICE, so a site agent needs its own unit (see installing a site agent).

config.toml keys

bcd site reads the same file as bcd node (the format and the shared keys are on installing bcd). These are the keys that matter to a site agent:

A complete site agent configuration is on installing a site agent.

How it connects to the control plane

A site agent joins with a site token and then holds a session like any node: Hello and Welcome, heartbeats, the journal, signed commands. Three things are specific to it:

• Its Hello lists the bcx drivers it hosts (their manifests), which become the bc.dev/driver-kind.* labels metal checks BMC operations against, and site.bmc-networks when [site] bmc_networks is set.
• Its desired state is the site section of its NodeSpec (SiteSpec): the boot lease state (active, epoch, ttl_ms), cache_images and proxy_targets. The site converger records the lease, checks the sanboot entries, evicts and warms the cache (below).
• Its fence. Every server frame updates the time it last heard from the gateway. The agent answers PXE machines only while the site section says it holds the lease at a non-zero epoch and it heard from the gateway within the lease TTL (15 seconds when the section gives none). A partitioned agent stops answering by itself. The lease itself is the control plane's; see one active boot server per site.

The site section's status:

Kinds it runs

The kinds registered for role site (siteKinds and the shared table in kinds.go):

A node refuses the site kinds, and a site agent refuses the ramdisk's. With command-signing roots in its trust bundle, the agent refuses any command without a valid worker signature, and destructive kinds fall under the batch quorum (see signed commands). The table of kinds and timeouts metal uses is on how BMC operations run.

boot.prepareRamdisk refuses (invalid-payload) a MAC that does not parse and a sanboot_sha256 that is neither empty nor 32 bytes. Storing an entry also drops the MAC from the discovered list.

The boot server

ServeBoot starts proxy DHCP and TFTP when configured, then one HTTP server with a 10-second header timeout and 8 KiB of headers (0442561). Its routes:

/boot/ipxe answers one of these scripts, all text/plain:

For a ramdisk entry the agent fetches the discovery token with a SecretRequest bound to the entry's workflow, once per token reference (a second fetch in the same boot reuses it), then signs a fresh DiscoveryAssertion with its node key under purpose discovery-assertion. The assertion's not_after is 15 minutes out, or the entry's not_after if that is sooner; its IP is the ip query parameter, else the connection's address; its product UUID the uuid parameter, else the entry's. When [boot] url is set, bc.cache becomes <url>/cache/t/<grant>, a grant that lasts until the entry lapses (6 hours for an entry without not_after). The fields and how the control plane checks them are on the discovery attestor.

Proxy DHCP

daemon/internal/pxe answers beside the site's own DHCP server and never hands out an address (yiaddr stays 0):

• It reads requests of type DISCOVER (answered with OFFER) and REQUEST (answered with ACK), and only from clients whose vendor class (option 60) starts with PXEClient or HTTPClient. Everything else is ignored.
• From the request it takes the MAC (chaddr), the client architecture (option 93), whether the client is iPXE (user class, option 77, contains iPXE), the SMBIOS UUID (option 97, 17 bytes with type 0) and the relay (giaddr).
• An answer carries the server id (option 54, server_ip), the vendor class (HTTPClient for HTTP boot clients, else PXEClient with PXE discovery control 8 in option 43, so the firmware boots the named file with no menu), option 66 and siaddr when there is a TFTP next-server, and the boot file in option 67 (and in the BOOTP file field when it is under 128 bytes).
• A client without an address and without a relay is answered on broadcast 255.255.255.255:68; a relayed one at the address it came from.

Which file a client gets:

Before any of this, the agent must be the active boot server and the MAC must have a live boot entry. Otherwise the request goes unanswered, and a MAC without an entry is listed as discovered.

The TFTP server is read-only (a write request gets error 4), octet mode only, with blksize (capped at 1,468 bytes so a block fits a 1,500-byte MTU) and tsize negotiation. It retransmits each block up to 5 times, 2 seconds apart, from one ephemeral socket per transfer. A name that resolves outside tftp_root is "file not found". It runs at most 64 transfers at once and 4 per source address, and drops a request beyond either bound unanswered (4d8b493).

You supply the iPXE builds and the ramdisk kernel and initramfs: the repository builds neither.

Discovery

While it holds the lease, the agent lists every PXE request, and every /boot/ipxe fetch with a well-formed MAC, from a MAC without a live entry in StatusReport.discovered (0056e1e). It keeps at most 256 machines, forgets each a day after it was last seen, takes a UUID only in canonical form and an architecture only as [a-z0-9_-]{1,32} (0442561), and asks for a status report for a new machine at most once every 30 seconds. With proxy DHCP on, an HTTP fetch counts only for a MAC DHCP already showed. iPXE fetches report the architecture as ${buildarch}-${platform} (x86_64-efi); DHCP requests as bios, x86_64-efi, arm64-efi, x86_64-efi-http, arm64-efi-http or arch-<n>. What the control plane does with the list is on discovering new machines.

The image cache

The cache lives in data_dir/cache (60afcc7). Each image is a file named by its SHA-256 in hex with a .json sidecar holding the URL it was pulled from and its last use:

• Pulling. Every cache_images entry not yet cached is downloaded in the background into a .pull-* temporary file, hashed on the way, and renamed into place only when the digest matches. Only http:// and https:// URLs are pulled.
• Serving. Only under a grant (b7ed8a2): a boot grant serves any cached image until the boot entry lapses and only while the agent holds the lease; a firmware grant serves one digest for 2 hours. Grants are 128-bit random tokens held in memory, so a restart invalidates them. Serving updates the last-use time at most once an hour.
• Evicting (2091cdc). With cache_max_bytes set, each converge pass deletes least recently used images until the cache fits, skipping every digest the site section still names.
• Reporting (4735600). The inventory's images section lists ready and pulling images (at most 1,000, the most recently used), the bytes used and the budget. The control plane turns it into the site's image_locations.

BMC operations

For each bmc.* command the agent:

1. Decodes the BmcOperationRequest and refuses (invalid-payload, the command fails) an operation outside its kind's allowlist (944221c). The allowlist is on Redfish in bcd site.
2. Picks the driver: built-in Redfish for redfish or no driver; otherwise a hosted bcx driver with that id that lists the operation's concern (51806ab), or unsupported-driver.
3. Refuses driver options on Redfish (invalid-config, 7753df0).
4. Fetches the credential with a SecretRequest bound to the command's workflow step, as { username, password } JSON or a bare password paired with the step's username argument. A rotation fetches the new password the same way, by newCredentialRef.
5. For firmware.update, pulls the image into the verified cache and hands the BMC a firmware grant URL instead of the source (image-unverified without [boot] url or a valid digest).
6. Remembers the BMC host, and every address its name resolves to, in data_dir/bmc-hosts.json (da474d0).
7. Runs the operation. A hosted driver gets the address, pin and options as config and the credentials as secrets, never in its arguments.

A BMC that refuses or fails does not fail the command: it succeeds with ok: false and a code. Only a malformed payload fails the command itself.

A hosted driver's own codes (such as bcx-ipmi's pin-unsupported) pass through as they are.

The Redfish client (daemon/internal/redfish) uses TLS 1.2 or later, never an environment proxy, never follows a redirect, and reads at most 4 MiB of an answer. Power actions map to ResetType values:

power.getPower reports on for On and PoweringOff, off for Off and PoweringOn, and unknown otherwise. management.rotateCredential patches the account, then lists /redfish/v1/Systems with the new password and answers { verified }. The per-operation Redfish requests are on Redfish in bcd site.

site.httpProxy

site.httpProxy makes one request on the control plane's behalf to an agentless machine of the site. Whatever the command says, the agent enforces (0d42de8, a2a853d, 808379f, da474d0):

• an http or https URL without credentials; GET, POST, PUT or DELETE (an empty method is GET);
• no path that cleans to /redfish…, and no host that is a remembered BMC or inside [site] bmc_networks, by name or by any resolved address;
• every resolved address private, CGNAT 100.64.0.0/10 or unique-local, never loopback, link-local, multicast, unspecified or public; the connection goes to the first checked address, with no second lookup;
• the host is one of target_node_id's addresses in SiteSpec.proxy_targets (an IP literal by value, a name by a listed name or by resolving only to listed addresses); with no listed target, nothing is allowed;
• a timeout of at most a minute (30 seconds when unset or out of range), a 10-second connect timeout, no redirects, the pinned SPKI when given, secret headers fetched with a SecretRequest;
• the response body cut at 1 MiB with truncated set, its headers sorted by name.

What metal checks before it sends one is on deploying machines.

Files in data_dir

bcd ramdisk

Flags and boot parameters

bcd ramdisk is not installed on a host; it is the agent in the ramdisk image the site agent boots. Its flags:

The boot parameters are the contract between the two modes (daemon/internal/bootparams). The site agent renders them in this order; the ramdisk ignores every argument that does not start with bc.:

The token is readable in /proc/cmdline by anything on the ramdisk, which runs nothing but bcd. It is single-use, bound to the MAC, and support bundles redact it.

Start-up and join

1. It reads and parses the command line. A missing parameter exits 2 (TestRamdiskNeedsBootParameters).
2. It builds the configuration from the parameters: gateway, server name, site, pins, data_dir, a socket at <data-dir>/bcd.sock, bc.cache as the image cache, JSON logs. An http:// gateway is refused (exit 2) unless the binary is a fakehw build, which then speaks cleartext HTTP/2.
3. Without an identity in the data directory it joins: it decodes bc.jwk and checks it against the join encryption pin (exit 1 on a mismatch), collects the inventory (it joins without hardware claims if that fails, with a warning) and the boot id, and calls Join with role ramdisk, attestor discovery, the token and the assertion as the attestation. A failed join exits 1. With an identity already there (bcd restarted within the same boot), it skips the join.
4. It runs the agent until the machine kexecs into its installed OS or is powered off.

The control plane's checks on that join are on the discovery attestor and attestation.

The join configuration it installs

Before running, the ramdisk prepares the files a deploy writes into the installed OS (797915d): the same configuration with data_dir reset to /var/lib/bcd, rendered as a config.toml, and bc.jwk as join-encryption.jwk. For a ramdisk booted with bc.server_name, the installed file reads:

# Written by bcd (PLAN.md §3.5, §6 Install).
gateway_url = "https://gw.eu.example.com"
gateway_server_name = "gw.internal"
site = "site_01k5w..."
serving_ca_spki_pin = "sha256/..."
join_signing_key_pin = "sha256/..."
join_encryption_key_pin = "sha256/..."

So the installed OS's bcd join needs nothing more from its user data than the provisioned token.

Kinds it runs

What each does to the disks, step by step, is on what the ramdisk does. The daemon-side details:

• Wipe. quick (the default) zeroes the first and last MiB of each disk and reads them back. The other modes use nvme format --ses=1 --force (or --ses=2 for crypto erase) for NVMe, hdparm security erase for SATA, else blkdiscard --secure. Every disk is attempted; if any is not verified, the command fails with reason wipe-unverified and the result still lists every disk's method and outcome. Metal sends it attested, so the result is node-signed.
• Root disk. The hint's first non-empty field wins (wwn, serial, device); otherwise the smallest non-removable disk of at least 4 GiB (or the hint's min_size_bytes, if larger).
• Image stream. The ramdisk asks the site cache first, GET <bc.cache>/<sha256>, and falls back to the image's source URL on any error or non-200. The SHA-256 covers the stored bytes (before gzip or zstd), and after decompression the rest of the stream is drained so the hash covers all of it. Data goes straight to the device in 4 MiB buffers. A copy error, a failed sync or a digest mismatch zeroes the disk's ends again, so nothing bootable is left. Then the kernel rereads the partition table (blockdev --rereadpt, udevadm settle).
• User data and join files. User data is fetched with a SecretRequest bound to the step and written as a cloud-init NoCloud seed or Ignition's config.ign. ext filesystems are written in userspace with debugfs and FAT with mtools (1ae1789, d9a900f); only xfs and btrfs fall back to a nodev,nosuid mount, written through os.Root with exclusive creates.
• kexec. It journals the command as running with the boot id before it acts, then loads the kernel and initrd (the given paths, /boot/vmlinuz and /boot/initrd.img, or the newest vmlinuz-* and its initrd), copied out of an ext root with debugfs or from a read-only mount, with root=UUID=<root> ro plus the requested arguments, and runs systemctl kexec, falling back to kexec -e. It completes when the installed OS rejoins.

The ramdisk image needs bcd and the tools these call; the list is on installing the ramdisk.

What the tests cover

Under a fake root, SecureErase zeroes the file, RunDCGM passes every GPU and Kexec reboots the fake host, so the secure-erase tools, dcgmi and kexec have not run from bcd. No test has started bcd site as a process, joined a site agent or a ramdisk to a real gateway, answered a real PXE ROM, or talked to a real BMC.