Protocol

Every bcd mode (node, site agent, ramdisk) speaks one protocol to the control plane: agent/v1, a gRPC service defined in proto/bettercompute/agent/v1. The daemon always dials out, the connection carries one long-lived bidirectional stream of frames, and everything security-relevant inside it is either authenticated by TLS or signed end to end, so the gateway in the middle can relay but not forge.

This page explains how the protocol behaves. For every service, message, field number and enum value, see the generated protocol reference, built from the .proto files with each declaration's own comment; the message names below link to their entries there.

Transport

• Endpoint. The daemon dials gw.<region>:443 (the gateway_url of its config). An L4 load balancer passes TLS through, so TLS terminates in the gateway process and the gateway sees the client certificate.
• One HTTP/2 connection carries the Session stream and, from v0.3, on-demand Tunnel streams. The gateway sends an HTTP/2 PING every 20 seconds.
• Certificates. The gateway's serving certificate comes from a separate gateway serving CA (better-compute pki issue-gateway --san gw.<region>), and bcd pins that CA's SPKI hash. Nodes present client certificates from the per-region node CA, which signs client certificates only. Join is server-authenticated TLS (the node has no certificate yet); Renew, Session and Tunnel are mutual TLS.
• gRPC framing. serveGateway implements gRPC on node:http2 without a framework: POST with an application/grpc or application/grpc+proto content type, 5-byte length-prefixed messages, grpc-status and a percent-encoded grpc-message in trailers (trailers-only for immediate errors), no compression. grpc-timeout is tolerated and ignored, because sessions are long-lived by design. Other methods get HTTP 405, other content types 415.
• Daemon side. bcd (daemon/internal/transport) uses connect-go over the Go code generated into daemon/gen/ (protoc-gen-go and protoc-gen-connect-go, through daemon/buf.gen.yaml and node scripts/daemon-proto-gen.mjs; --check verifies it is current). It speaks gRPC over HTTP/2 with TLS 1.3 pinned to the serving CA's SPKI hash, never the system roots, presents its node certificate, and sends HTTP/2 PINGs every 20 seconds. When gateway_url names an IP address, gateway_server_name gives the name the gateway certificate must carry. tests/e2e/bcd.test.ts runs it against serveGateway's mTLS listener. The FakeDaemon in @better-compute/testing speaks the same protocol over real HTTP/2 or in memory.

WebSocket fallback (v0.4)

Some networks run a TLS-inspecting proxy that strips client certificates. For them the gateway also serves the same frames over a WebSocket:

A WebSocket session is marked signed-only: the gateway sends it only commands and desired states that carry a signature, and unsigned ones stay pending. The proxy can still see frame metadata (kinds, sizes, timing) and can delay or drop frames, but it cannot forge intent. PLAN.md allows this mode only once all commands and specs are signed: with pki.commandSigning both are (the gateway relays each generation's stored signed spec), so a WebSocket daemon receives its desired state. There is no long-poll or unary path before v1. The gateway side is attachGatewayWs and serveGatewayWs, and connectGatewayWs is the FakeDaemon's dialer.

Renewal without a client certificate (06cb334, PLAN.md §7.1). Behind such a proxy the mTLS Renew cannot work, and a node would lose its identity after 24 hours, so the WebSocket listener also takes POST /v1/renew: the RenewRequest bytes as application/x-protobuf, answered with the RenewResponse bytes, errors mapped as for gRPC (status, grpc-status, x-bc-reason, retry-after). No TLS identity is used: the request carries the current chain and a proof of possession by the current key, so the peer is the node the chain's leaf names, and JoinService checks chain, revocation, generation and the proof (with its replay cache). A node gets 6 renewals an hour, and a body is at most 64 KiB. gatewayRenewHandler serves it for hosts that attach the WebSocket listener to their own server.

In bcd (6cd48be). A [websocket] url in config.toml moves bcd node and bcd site to this transport: a dependency-free RFC 6455 client dials wss://.../v1/session (through the environment's HTTP CONNECT proxy when one is set), with masked binary messages of at most 16 MiB, answered pings and the close handshake. TLS is verified against the pinned serving CA, or against [websocket] proxy_ca_file for the transport only; either way the 101's x-bc-gateway-proof must chain to the pinned serving CA for server authentication, be signed by that leaf, and name this node and nonce with an iat within 60 s, or bcd never talks to the gateway. The identity is re-read at every dial, so a renewal applies at the next one, and renewals go to POST /v1/renew on the same origin, honouring the gateway's pushback. Refusals' x-bc-reason and pushback (grpc-retry-pushback-ms, retry-after) steer the redial. In WebSocket mode bcd refuses tunnels (they need the gRPC transport) and refuses to dial without command-signing roots, since the session receives signed intent only. Go tests cover the session, unproven gateways and the renewal against an in-process gateway, and tests/e2e/bcd.test.ts runs it end to end (4b2d5aa): beside a createDevStack({ signedIntent: true }) stack, the TypeScript serveGatewayWs with a serving certificate from the CA the node pins; the node restarts with [websocket] url, runs a signed inventory.refresh over the WebSocket session and renews over POST /v1/renew. daemon install-script --websocket-url writes the section (see installing bcd).

Proto packages

@better-compute/protocol carries the TypeScript code generated by protoc-gen-es (node scripts/proto-gen.mjs), plus helpers: bounded encoders and decoders, the command kind table, the signing-input layout and version negotiation. It depends only on @bufbuild/protobuf and is browser-safe. The draft packages are exported as the site and driver namespaces. protocolRegistry holds every message, for Any payloads and JSON.

The AgentGateway service

The service as session.proto declares it (see AgentGateway in the reference for each request and response message):

service AgentGateway {
  rpc Join(JoinRequest) returns (JoinResponse);
  rpc Renew(RenewRequest) returns (RenewResponse);
  rpc Session(stream AgentFrame) returns (stream ServerFrame);
  rpc Tunnel(stream TunnelFrame) returns (stream TunnelFrame);  // v0.3
}

The node id always comes from the certificate's one SPIFFE URI SAN, spiffe://<trust domain>/site/<site>/node/<node>, never from a frame (any other URI SAN is a urn:bc: statement about the key, never an identity). Join also carries the hardware claims, the attestor name (join-token in v0.1; x509pop, provisioned and discovery in v0.2; cloud-iid in v0.3; tpm in v0.5), attestor evidence, the daemon version and the role (see JoinRequest and HardwareClaims). A JoinResponseBody may carry a JoinDeferral (v0.2) instead of a certificate: the daemon then sends Join again with the same CSR, after retry_after_seconds or with its answer to a challenge. The join flow is described in joining nodes. bcd sends join-token (for bcd join, as role node or site), provisioned (for bcd join --node-id on an OS a deploy installed; --attestor names any other) and discovery (for bcd ramdisk, as role ramdisk, with the site agent's serialized SignedDiscoveryAssertion as the evidence); it answers no attestor challenge yet.

PLAN.md places JoinService and the secret broker in the api, behind internal mTLS. The gateway consumes them as a port, GatewayServices (join, renew, secret, trustBundle), so the gateway package never imports the server; better-compute serve passes the compute instance's gatewayServices.

Frames

The first AgentFrame of a session must be Hello and the first ServerFrame is Welcome. After that both sides send frames as they need to.

A gateway ignores an AgentFrame case it does not know, so a newer daemon can send frames an older gateway does not understand. Sessions are covered in sessions; commands, desired state and events in delivery.

Secrets

Commands carry secret references (secret://, vault://, env://), never values. For each reference the daemon sends SecretRequest{ref, workflow_id, step_id, allocation_id, epk, node_sig, request_id}, where epk is a fresh P-256 public key used once and node_sig covers every other field. The broker checks the signature and that the reference is bound to the agent's current step or allocation, or, for a credential the node's own spec names, to workflow host:<unit or section> and step <field> (host:k8s-join / tokenRef, host:telemetry / headersRef; see sealed secrets), and answers with a SecretResponse holding a compact JWE (ECDH-ES+A256KW, A256GCM) to epk, or with an error of not-found, not-bound, access-denied or unavailable. Gateways see ciphertext only. A session whose certificate is tenant-controlled always gets access-denied.

bcd builds the request and opens the answer in daemon/internal/pki/secrets.go (NewSecretRequest, OpenSecretResponse): the JWE has typ bcsecret+jwe and must be bound to the same reference and request id and unexpired, and a refusal surfaces its error code. The agent matches answers by request id on the session that sent the request, waits at most 10 seconds and ignores an answer to nothing. The Go tests check both directions against fixtures produced by @better-compute/pki. These bcd features send one: a site agent's BMC operations (the credential, and for bmc.rotate the new one) and site.httpProxy secret headers, bound to the command's workflow step; its boot server's discovery tokens, bound to the boot entry's workflow; the ramdisk's deploy.image user data, bound to the step; a VPS host's instance user data, bound to the instance's allocation; the k8s-join unit's bootstrap token (7579184) and the OTLP exporter's headers (27c7ee2), bound to the node's spec.

Workload tokens (v0.5)

A VPS host's metadata service gets a hosted instance's workload identity token the same way: WorkloadTokenRequest{request_id, instance_id, audience, epk, node_sig}, where audience is empty for the issuer's default, epk is a fresh P-256 key used once, and node_sig (purpose workload-token) covers the other fields. The gateway relays it to GatewayServices.workloadToken outside the frame queue, and the answer is WorkloadTokenResponse{request_id, jwe, expires_at, error}: a compact JWE of the token to epk, or an error of not-bound, disabled, audience, rate-limited, access-denied or unavailable. A gateway without the service answers unavailable, and a tenant-controlled session always gets access-denied. bcd implements the host's half the same way as secrets (NewWorkloadTokenRequest, OpenWorkloadTokenResponse, a bcworkload+jwe answer bound to the instance and the request id), and its metadata service ([imds] listen) calls it for the running instance that holds the request's source address. See workload identity.

Tunnels (v0.3)

OpenTunnel{tunnel_id, ticket} asks the daemon to open a Tunnel stream of TunnelFrames. The ticket is an ES256 JWT, valid 60 seconds, with a single-use jti, audience node:<id>, a sid and an op. The gateway verifies it before asking, and the daemon verifies it again against the JWKS in its NodeSpec access section and its own jti set. The daemon's first TunnelFrame is opened; then data on channel 0 (stdin and stdout, or the byte stream) or 1 (stderr), resize with rows and columns, and closed with an exit code. bcd refuses a ticket by opening nothing, so the gateway's open times out; it runs at most 16 tunnels at once. What it runs for each op, limits and re-authorization are in tunnels.

Message size bounds

Everything is bounded (principle P12). The bounds live in @better-compute/protocol, and both sides decode with them before parsing:

encodeMessage and decodeMessage throw ProtocolError with FRAME_TOO_LARGE or INVALID_FRAME. Decoding keeps unknown fields, so a relay that re-encodes a newer peer's message loses nothing. Bulk data never rides the session: log collection uploads to a presigned URL and images are fetched by URL.

Signed envelopes

Anything a relay must not alter travels as bytes body plus a Signature, and the signature covers the exact body bytes, so nobody ever re-encodes to verify. A Signature is ES256 in the raw 64-byte r || s form, a kid and an optional certificate chain. The signed bytes are domain-separated by purpose:

input = "bettercompute.v1/" || purpose || 0x00 || part_1 || ... || part_n
part  = uint32_be(len(bytes)) || bytes

Worker keys chain to the command-signing roots of the TrustBundle the daemon received in JoinResponse, and a newer bundle arrives in Welcome.trust, a SignedTrustBundle signed by the pinned JoinService key. The bundle also carries the v1.0 quorum's approver_keys, approver_threshold, quorum_tiers and unbatched_destructive_per_hour, empty unless pki.approvers is set, approver_set_epoch (field 8, c6214dc), the set's own epoch, separate from version and raised with every change of the set, and approver_set_signatures, by which the current approvers sign a change of those fields for that epoch (purpose approver-set, parts [uint64_be(approver_set_epoch), sha256(approverSetInput), trust_domain]), published on every version while the set is current. A daemon holding set H at epoch e takes (N, e') only when e' > e and its current threshold of H's keys signed it, N == H included; otherwise it keeps its quorum and applies the rest of the bundle. Both daemons report the set they enforce, approver_set_epoch and approver_set_sha256, in Hello (fields 18 and 19) and beside trust_version in StatusReport (10 and 11), since a bundle's version can be taken while its quorum part is refused (see changing the approver set). The daemon accepts a bundle only when its version is greater than the one it holds, and the control plane raises the version whenever the bundle's content changes, so a change always reaches the fleet. The envelopes exist from v0.1 with optional signatures, so turning them on is additive. See delivery for which kinds require them.

Errors outside frames

Refusals that end a stream use gRPC status codes:

Version skew

Negotiation. Hello carries proto_min, proto_max, daemon_version and capabilities. The gateway picks the highest version inside both ranges (negotiateProtocolVersion) and returns it in Welcome.protocol_version, along with its own capabilities. It also stores the daemon's capabilities as node_sessions.capabilities, so the API can refuse what an old daemon cannot do safely: tunnels.open opens an instance ssh or tcp tunnel only when the node advertises tunnel.instance-dial (see sessions). The gateway also writes each capability as the system node label bc.dev/cap.<name>=1, and placement gates on those labels so an older bcd never gets a spec field it would drop as unknown: vps places instances that need instance.network-policy, instance.ssh-host-only, instance.placement or instance.volumes only where they are advertised, and capacity places reservations that book host resources only on devices.limits nodes and binding: 'kubernetes' ones only on binding.kubernetes nodes (409f783). A plain GPU hold without host resources needs no capability and still places on an N-2 daemon. Today both sides speak exactly version 1 (PROTOCOL_VERSION = { min: 1, max: 1 } in TypeScript, ProtoMin and ProtoMax in Go). When the ranges do not overlap, the daemon is below the server's minimum: Welcome.upgrade_required is set, Welcome.protocol_version is the gateway's minimum, no desired state or command other than daemon.upgrade is sent, and the FakeDaemon and bcd reject anything else UNSUPPORTED. bcd runs daemon.upgrade (de9f380) when its [update] section pins release keys; without pins it refuses the update, and the binary must be replaced by hand.

The gateway's range. createGateway({ protocol: { min, max } }) sets the versions a gateway speaks; the default is PROTOCOL_VERSION. Negotiation and the Welcome use it, so a gateway at { min: 1, max: 3 } welcomes a daemon at 1..1 at version 1 and one at 1..2 at version 2, while a gateway at { min: 2, max: 3 } restricts a 1..1 daemon to upgrade_required. A range with min below 1, max below min or a non-integer bound is INVALID_CONFIG at construction. It exists for the v1.0 N-2 skew tests and staged protocol bumps (tests/gateway-protocol.test.ts); no version beyond 1 is defined yet.

The skew matrix. tests/skew-matrix.test.ts runs each supported daemon line against real dev stacks (createDevStack({ gateway: { protocol }, daemon: { profile } })). A FakeDaemon profile from daemonSkewProfiles in @better-compute/testing makes it behave like an older bcd: the version and protocol range it announces, its capabilities, the kinds it rejects UNSUPPORTED, the spec sections it reports SectionUnsupported, the inventory sections and Hello fields it never sends. The rows today are N (0.5.0), N-1 (0.4.0, no instance.snapshot) and N-2 (0.3.0, also no daemon section and no trust_version); each is welcomed at the highest shared version, adopted, runs inventory.refresh and gets its missing kinds rejected. belowMinimumProfile, and the current line against a gateway at { min: 2, max: 3 }, get upgrade_required: no spec, and ordinary commands stay pending. A release adds a row on top and drops the one that leaves the window.

What is recorded. The gateway stores Hello.daemon_version and Hello.proto_max in node_sessions (migration core/0008_session_versions), which is how the rollout controller sees a node finish an upgrade and finds nodes that need daemon.upgrade.

Policy. Daemons may lag N-1 minor versions before 1.0 and N-2 from 1.0. The range widens when a release adds a version and its minimum rises when the skew window drops one. The upgrade order is expand migrations, workers and api, gateways, daemons, then contract migrations a release later.

Additive only. agent/v1 and common/v1 were frozen at the end of P0. CI runs buf breaking with the FILE rule set against proto/baseline.binpb; the draft site and driver packages are excluded until their phases freeze them. Within the frozen packages, change is absorbed at every level:

Built-in command kinds

commands.proto holds the payloads and results of the v0.1 kinds, and commandKinds in @better-compute/protocol holds their flags; the daemon's dispatcher must carry the same table. Later kinds add their own messages: the ramdisk's diag.dcgm, disk.wipe, deploy.image and boot.kexec in bettercompute.agent.v1, the site agent's bmc.*, site.httpProxy, boot.prepareRamdisk and boot.cleanUp in bettercompute.site.v1 (their flags are metalCommandKinds in @better-compute/metal, and bcd implements them), then driver.invoke, which bcd runs in the bcx plugins it hosts (see drivers), and daemon.upgrade, which bcd runs as its self-update (de9f380).

Metal's kinds, by lane, are in delivery; every module kind and its timeout is in command kinds.