Telemetry and status

bcd reports on itself in two places. Every interval it pushes its own metrics, GPU readings, host pressure, memory and huge page pools, and the VMs and sandboxes it runs, over OTLP/HTTP straight to a collector (27c7ee2). And on the machine itself it answers bcd status on a local unix socket (a4c305b, ad8791f). Neither goes through the gateway or the session, and neither reaches PostgreSQL: principle P8 keeps telemetry off the control plane's path.

This page is the daemon's side: what bcd sends, how, and what happens when the collector is not there. The control plane's side, the nodeTelemetryModule that tells each node where to export and the doctor finding for a pool with a bad policy.telemetry, is in node metrics from bcd. The control plane's own Prometheus series are a different thing and are listed in the metrics reference.

Where the export is configured

There is no telemetry key in config.toml. The export is configured only by the telemetry section of the node's NodeSpec, so the control plane can point a fleet at a new collector without touching any host:

// Where bcd sends its own telemetry (OTLP). Never the session or Postgres (P8).
message TelemetrySection {
  // Empty disables export.
  string otlp_endpoint = 1;
  uint32 export_interval_ms = 2;
  // Extra "key=value" resource attributes.
  repeated string resource_attributes = 3;
  // Secret ref of the exporter's auth headers, fetched with SecretRequest.
  string headers_ref = 4;
}

The control plane fills the section from TelemetrySpec (packages/core/src/domain.ts). The NodeSpec controller (packages/server/src/controllers/nodespec.ts) merges each module's telemetry part field by field and renders it to the wire:

The only built-in module that sets it is nodeTelemetryModule from @better-compute/observability (53aa356). It is not registered by default. Its options, the pool override and the checks it runs are described in node metrics from bcd. Any module can return a telemetry part, and the e2e suites use a small one of their own.

The telemetry section converges like any other: see convergence and status. Every mode that runs the agent has the converger: bcd node, bcd site and bcd ramdisk.

What bcd checks

The control plane checks more than bcd does. nodeTelemetryModule allows http:// only to localhost, 127.0.0.1 or [::1], limits the interval to 5 to 3600 seconds, and keeps bcd.* and service.* attribute keys for bcd. bcd itself checks only what it needs to run safely, when it plans the section:

So bcd accepts plain http:// to any host. The e2e suites rely on this to reach a receiver on the WSL host's address. A deployment that wants TLS everywhere gets it from the module's check, not from bcd.

A terminal failure is tested in TestTelemetryExportsGPUMetrics: an ftp://x endpoint leaves the telemetry section FAILED, and the agent keeps running.

How the exporter runs

The converger keeps one exporter goroutine, under the agent's context:

• Start and restart. When the section differs from the one applied (proto.Equal), the plan has one action: export telemetry to <endpoint>, or stop the OTLP export when the endpoint is empty. Applying it cancels the running exporter, resets the last export and last error, and starts a new one if there is an endpoint. Any change, even to one resource attribute, restarts the exporter. An empty section on a node that never exported is a no-op.
• Stop. A section with an empty endpoint, or no section, stops the export. The exporter also stops with the agent.
• First export. The exporter sends once as soon as it starts, then every interval on a ticker.
• Each tick. It samples the metrics and posts them, under one deadline equal to the interval. So a slow nvidia-smi or a slow collector cannot pile up exports. The HTTP client has its own 10 s timeout.
• One batch per tick. The exporter keeps no queue. A batch that fails is not sent again. The next tick samples fresh values and sends those.

The request

Each export is one POST to <endpoint>/v1/metrics with Content-Type: application/json. The body is an ExportMetricsServiceRequest in the OTLP JSON encoding (the protobuf JSON mapping). bcd builds it itself and links no OpenTelemetry SDK. The request has one resource, one scope (bcd and bcd's version), and the metrics that have at least one point:

{
  "resourceMetrics": [{
    "resource": { "attributes": [
      { "key": "bcd.node.id", "value": { "stringValue": "n1" } },
      { "key": "service.name", "value": { "stringValue": "bcd" } }
    ] },
    "scopeMetrics": [{
      "scope": { "name": "bcd", "version": "1.0.0" },
      "metrics": [
        { "name": "bcd.gpu.utilization", "unit": "1",
          "gauge": { "dataPoints": [{ "timeUnixNano": "200000000005", "asDouble": 0.87,
            "attributes": [{ "key": "gpu.uuid", "value": { "stringValue": "GPU-1" } }] }] } },
        { "name": "bcd.gpu.ecc.errors", "unit": "{error}",
          "sum": { "aggregationTemporality": 2, "isMonotonic": true, "dataPoints": [{
            "startTimeUnixNano": "100000000000", "timeUnixNano": "200000000005", "asInt": "5",
            "attributes": [
              { "key": "gpu.uuid", "value": { "stringValue": "GPU-1" } },
              { "key": "type", "value": { "stringValue": "correctable" } }] }] } }
      ]
    }]
  }]
}

• Attributes are string values, sorted by key.
• Integer points use asInt (a string, as the JSON mapping requires); others use asDouble.
• Every point of an export has the same timeUnixNano, the time of the export.
• Sums are cumulative and monotonic (aggregationTemporality: 2). Their startTimeUnixNano is when the exporter started, not when the process started, so a restart of the exporter (any section change) starts a new series start.
• No metric carries a description.

Resource attributes

Headers

headers_ref names a secret that holds the collector's auth headers. The exporter fetches it with a SecretRequest bound to workflow host:telemetry and step headersRef. The secret broker releases it only while the node's current NodeSpec names that reference in telemetry.headersRef (binding rule 3 in packages/server/src/join/secret-broker.ts; see secrets and sealed secrets).

The secret is parsed by telemetry.ParseHeaders in one of two forms:

{"Authorization":"Bearer a"}

Authorization: Bearer a
X-Scope-OrgID: t1

A header name that is empty or contains a space, tab, CR, LF or :, or a value that contains CR or LF, is refused. The headers are set on every request after Content-Type, so a Content-Type in the secret replaces bcd's.

While the headers cannot be fetched or parsed, the exporter sends nothing. It does not export without them. It records the error (headers <ref>: …) and tries the fetch again on every tick until one succeeds. After that it keeps the headers for the life of that exporter. A rotated secret reaches bcd only when the exporter restarts: a section change, or an agent restart.

Failure behaviour

Logging is per change, not per tick. The first failure logs a warning (telemetry export, with endpoint and error). The same error again logs nothing. A different error logs again. The first success after a failure logs telemetry export resumed. The latest outcome is always in bcd status: the last successful export and the last error.

A failing export never fails the section, holds back observed_generation, or touches the session. The control plane learns nothing about it. Watch the collector's side (a node's series going stale) or bcd status on the machine.

The metrics

All metrics are sampled at the moment of the export, except host pressure (below). A metric with no points in an export is left out of it. Gauges are OTLP gauges; sums are cumulative monotonic OTLP sums.

GPUs

GPU points carry gpu.uuid, gpu.index and gpu.model, from the inventory. A sampled GPU the inventory does not list carries only gpu.uuid.

The sampled series (the first eleven, and MIG) come from one nvidia-smi call per export since 9a6fee9:

nvidia-smi --query-gpu=uuid,memory.total,memory.used,memory.reserved,memory.free,utilization.gpu,utilization.memory,temperature.gpu,power.draw,power.limit,clocks.sm,clocks.mem,pstate,mig.mode.current --format=csv,noheader,nounits
nvidia-smi --query-compute-apps=gpu_uuid --format=csv,noheader

bcd falls back to the full read (nvidia-smi -q -x and the rest of nvml.SMI.Snapshot, several seconds of driver time on a large board) when a GPU is in MIG mode, since only -q -x has the MIG instances' memory; when the driver refuses a field (memory.reserved before R510); or when a line does not have the expected columns. If nvidia-smi is missing (ErrNoDriver), there is no fallback and no sample.

The ECC, row remapping and XID series come from bcd's inventory and fault state, not from a new read, so they only change when the inventory or the fault set does.

A fakehw build samples fakehw.GPUSnapshot for its seed instead of nvidia-smi (gpusampler_fakehw.go).

Host

Host pressure is not read at export time. The agent's health loop samples it every 30 s (HealthInterval) and also sends it to the control plane in StatusReport.pressure. The export uses the latest sample. So the four pressure series are missing for the first 30 s after bcd starts, and may be up to 30 s old. The first steal reading is 0, as it needs a baseline.

Instances

These come only from a host with an instance runtime (VPS and sandbox hosts). The configured series cover every runtime, plugin runtimes included. The live usage series come only from runtimes that read it (runtime.UsageReader).

Where each runtime reads its usage (cgroup v2 and tap counters for cloud-hypervisor, virsh domstats for libvirt-kvm) and how that works under bcd helper is described in node metrics from bcd. The balloon itself is described in memory ballooning. A runtime whose List fails is left out of that export.

The status socket

bcd node, bcd site and bcd ramdisk serve their status on a unix socket (a4c305b):

• Path. socket_path in config.toml, default /run/bcd/bcd.sock. It must be absolute. bcd ramdisk uses <data dir>/bcd.sock.
• Permissions. The directory is created with mode 0700 and the socket is set to 0600, so only the account that runs the agent (root, or the unprivileged node user under bcd helper) and root can connect.
• Protocol. HTTP over the socket. One route: GET /status, which answers the agent's Status as JSON. The server's read-header timeout is 5 s.
• Lifetime. A stale socket from an earlier process is removed at start; the socket is removed again on shutdown. If the socket cannot be created, bcd logs status socket unavailable and runs without it. It is never fatal.

The socket serves only bcd status today. bcd doctor reads the host itself and does not use it. It is the only local endpoint bcd has: see outbound only.

bcd status

bcd status [--socket /run/bcd/bcd.sock] [--json]

It exits 0 when it read a status, and 1 when it could not. When there is no socket, or the connection is refused, it prints bcd status: the node agent is not running (no socket at <path>); the client times out after 5 s. In the Kubernetes DaemonSet, bcd status is the bcd container's liveness probe (see Kubernetes).

Text output

node       node_01k… (site site_1, role node)
bcd        0.4.0, boot 3f6c…
session    connected (epoch 12, clock offset -3 ms)
spec       applied 41, observed 41
  host       CONVERGED
  devices    CONVERGED
  telemetry  FAILED ConvergeFailed: converge: terminal failure: otlp endpoint "ftp://x" is not an http(s) URL
commands   0 running, 0 waiting for a spec
events     2 unacked, 0 waiting, 0 metrics dropped
identity   certificate until 2026-10-06T08:00:00Z, node key in the TPM, trust bundle v3, signed intent enforced
gpus       2
  0 GPU-5f1c…   87%   61 C   540/700 W  71.2/79.6 GiB  held by alloc_01k…
  1 GPU-9a02…    0%   34 C    72/700 W  0.0/79.6 GiB  UNHEALTHY
telemetry  https://otel.example.internal:4318, last export 2026-09-26T08:12:00Z

The values above are illustrative; the lines and their order are what runStatus prints. The section lines show each NodeSpec section's last state with the SECTION_STATE_ prefix removed, then its reason and message when there are any. The gpus, kubernetes and telemetry lines are described in bcd status. The telemetry line is printed only while the applied section has an endpoint:

metrics dropped on the events line is not about OTLP. It counts metric-class events the journal dropped when it was full (see the journal). An OTLP export that fails is never journaled.

JSON output

--json prints the same agent.Status the socket serves:

The GPU fields are read when you ask. Each bcd status runs the GPU sampler (the same nvidia-smi query as the export) with a 10 s timeout. If it fails or times out, the GPUs are still listed from the inventory, with zero readings and the inventory's memory total. Power is rounded to whole watts. Health is bcd's own verdict (no failed or pending row remap, no uncorrectable ECC, no XID this boot), the same one that keeps a GPU from new holds.

Relation to control-plane observability

The two sides do not overlap:

What the control plane does learn about a node's health travels in the session instead: StatusReport carries host pressure, section states and conditions such as XIDs, and the gateway turns them into node conditions and node.status events (see convergence and status). That path drives scheduling and health policy. The OTLP export is only for your dashboards and alerts. For alerting on the control plane, see what to alert on.

What the tests cover

Nothing has tested ECC, row remapping, MIG memory or XIDs on a real data-center GPU, the export through a TLS collector with auth headers on a real network, or a collector product. The test layout is in testing.