Troubleshooting bcd

This page maps what you see on a host (a log line, a refusal, a section that never converges) to its cause and fix, drawn from the error strings in daemon/. Each entry names the bcd doctor check that detects it, where one does. What happens to the fleet when a gateway, the database or the network fails is on failure modes; this page is about one host.

Where to look first

sudo bcd doctor --config /etc/bcd/config.toml   # one line per check; exit 1 when one fails
sudo bcd status                                 # the running agent: session, sections, certificate
journalctl -u bcd.service -e                    # bcd's JSON log lines
journalctl -u bcd-helper.service -e             # on a node installed with --unprivileged

bcd status lists every NodeSpec section with its state, reason and message. Since da60bbd the node's own log says the same: a section that cannot converge logs section retrying (WARN) with the error on every pass, about every 30 seconds, and one that failed for good logs section failed (ERROR) with its reason (AllocationFailed, InstanceFailed, UpdateFailed or ConvergeFailed). Before that commit an operator on the host saw only PROGRESSING.

{"level":"WARN","msg":"section retrying","section":"devices","generation":42,"error":"the slice of alloc_01k…: start bcd-alloc-alloc_01k….slice: …"}

Session problems log at INFO as session ended; redialing with the reason and the back-off.

bcd will not start

Joining

Server-side refusals are deliberately opaque: every refused join is ACCESS_DENIED "Join refused", and the reason is recorded as a security.join.refused event. Read the reasons in refusals. A refused join is not retried and does not consume the token.

Doctor: gateway fails when the gateway is not reachable over TCP within 5 seconds; identity warns before the first join.

Pins and TLS

bcd trusts no system root for the gateway: the chain must contain a CA whose SPKI equals serving_ca_spki_pin, and the leaf must name the host. These appear as the reason of session ended; redialing, and as the failure of bcd join.

Doctor: none checks TLS; gateway tests TCP only.

The clock

Certificates, signed intents, exec tickets and renewal proofs all compare times. Since e2a0cf0 doctor asks the kernel (adjtimex) whether the clock is synchronized.

bcd evaluates not_after through the offset the last Welcome gave it, so its own clock matters for TLS and proofs, not for command deadlines. See clock skew.

Renewals, expiry and revocation

bcd renews at 60% of the certificate's life. A failed renewal logs renewal failed (WARN) with the error, the attempt and expiresIn, and retries with back-off from 30 seconds to 10 minutes (or when the gateway's pushback says).

See certificate expiry, certificate revocation and renewal.

Sections stuck progressing

A section is PROGRESSING with reason Retrying while its converger returns an error that is not terminal, and the spec is not observed meanwhile. The log line section retrying carries the error. Common ones:

A host section that fails a unit keeps going with the others; see status and failures.

GPU holds

The devices converger fails an assignment it can never bind at once, so the control plane re-places it instead of waiting. The reason travels in AllocationFailed.

Before da60bbd an unprivileged node retried the hold every 30 seconds on systemd's "Access denied" for the slice, logging nothing above debug, and the reservation stayed binding forever. A log warning NVIDIA's CDI spec does not name held devices means the vendor spec predates the GPUs; the container-toolkit unit regenerates it.

nvidia-smi errors

bcd reads GPUs through nvidia-smi. Doctor's gpus check fails with the tool's error and, since f84f1f9, a hint for the failures seen on real hosts:

Related checks around the GPUs:

KVM and instances

Helper socket refusals

On a node installed with --unprivileged, every privileged operation goes to bcd helper. The node only sees the answer; the helper's log says why.

bcd helper refuses to start, and logs to standard error, when:

Host units through the helper fail when the helper cannot bind them to a node:

See privilege separation and signed specs.

Kubernetes and self-update

See self-update and bcd on Kubernetes.