# Observability (/docs/operations/observability) > Prometheus metrics with bounded labels, a bearer-protected /metrics route, ComputeSpan observers, trace context, health endpoints and doctor. `@better-compute/observability` holds the pieces a host uses to watch a deployment: a Prometheus registry that refuses tenant labels and caps its series, the standard `better_compute_*` metric set of PLAN.md section 14, a `GET /metrics` handler behind a bearer token, wrappers that meter IAM calls, gateway services, attestors and providers, `ComputeSpan` observers and W3C `traceparent` helpers. It is browser-safe (no Node.js imports), so every package can depend on it. The CLI host mounts it for you when the configuration's `host` export has a registry. > **Defined is not the same as recorded.** `standardMetrics(registry)` fixes the names, labels and buckets of every section 14 series, so all packages record the same thing. The built-in pieces record: the gateway series sampled by `gatewayCollector`, the gateway service calls, outbox events by family, the deployment-wide series `databaseCollector` samples from the database (below), placements through `placementObserver`, and whatever you wrap (`meteredIamPort`, `meteredAttestor`, `meteredProvider`, spans). `gateway_sessions`, `gateway_handshakes_total`, `commands_swept_total`, `reconcile_duration_seconds`, `usage_dead_letters_total` and `security_events_total` are defined but no package sets them yet; they expose their `HELP` and `TYPE` lines with no samples until your host records them. ## Wire metrics into a deployment [#wire-metrics-into-a-deployment] One registry per process. The configuration creates it, registers the standard series, passes it to what it meters, and hands it to the host through `host.metrics`: ```ts title="better-compute.config.ts" import { defineHost } from 'better-compute/cli'; import { createMetrics, eventCounter, meteredIamPort, standardMetrics } from 'better-compute/observability'; import { eventsModule } from 'better-compute/events'; import { defineConfig } from 'better-compute/server'; export const metrics = createMetrics(); // maxSeries 2000 per metric const standard = standardMetrics(metrics); export default defineConfig(({ env = process.env } = {}) => ({ // …database, secret, baseURL iam: meteredIamPort(bridge.embedded(iam, { rootTenantId }), standard), // iam_calls_total, iam_call_duration_seconds modules: [eventsModule({ secret: env.BETTER_COMPUTE_SECRET!, countEvent: eventCounter(metrics) })], })); export const host = defineHost(async ({ env }) => ({ metrics: { registry: metrics, bearerToken: env.BETTER_COMPUTE_METRICS_TOKEN }, // GET /metrics })); ``` `better-compute serve` then: * mounts `GET /metrics` (or `host.metrics.path`) on the API listener when `bearerToken` is set; without a token there is no route; * when it runs the gateway role, meters `compute.gatewayServices` with `meteredGatewayServices` and samples the gateway at every scrape with `gatewayCollector`; * when it runs any worker role, starts `databaseCollector` into the same registry: a first reading at start, then every 15 s, stopped with the server. It takes each command kind's lane from the registry and watches the host's own certificates as `cert_expiry_seconds{kind}`: `node-ca`, `command-signing` and `serving` (the gateway's TLS certificate). `better-compute dev` mounts the configuration's `host.metrics` too. The examples read the token from `BETTER_COMPUTE_METRICS_TOKEN`. Scrape it with the token as a bearer credential: ```yaml title="prometheus.yml" scrape_configs: - job_name: better-compute authorization: type: Bearer credentials_file: /etc/prometheus/better-compute.token static_configs: - targets: ['compute-api-1:3000', 'compute-api-2:3000'] ``` ## The registry [#the-registry] `createMetrics({ prefix?, maxSeries?, forbiddenLabels?, now? })` returns a registry of counters, gauges and histograms. * **Prefix.** Every name gets `better_compute_` unless it already has it. Names and labels must match `[a-z_][a-z0-9_]*`; `le` is reserved. * **No tenant labels.** A label named `tenant`, `tenant_id`, `tenantid`, `org`, `organization`, `identity` or `user` is refused with `INVALID_CONFIG` when the metric is defined. Per-tenant numbers belong in usage and audit, not in your TSDB. * **Bounded series.** Each metric holds at most `maxSeries` label sets (default 2000). New label sets past the cap fold into one series whose labels all read `(other)`, and `better_compute_metrics_collapsed_total` counts every folded recording. * **Caller-controlled values.** A label value that is not a string or is longer than 256 characters reads `(invalid)`; an empty one reads `(unknown)`. * **Collectors.** `registry.collect(fn)` runs `fn` at the start of every `expose()`, for gauges sampled at scrape time. A collector that throws is skipped for that scrape. * **Exposition.** `expose()` renders text format 0.0.4. `value(name, labels)` and `stats()` (`series`, `dropped`) serve tests and health checks. `metricsHandler({ registry, bearerToken, timingSafeEqual })` answers `GET` only (405 otherwise), requires a token of at least 16 characters, compares the SHA-256 digests of the presented and expected tokens with the constant-time function you pass (`crypto.timingSafeEqual` in Node), answers 401 with `WWW-Authenticate: Bearer` on a mismatch, and serves the text with `Cache-Control: no-store`. ## The standard series [#the-standard-series] `standardMetrics(registry)` defines the section 14 series: sessions and handshakes, command latency and sweeps, the work queue and reconcile durations, placements and booking conflicts, pool GPUs, node connectivity, backlogs, IAM calls and resolver lookups, certificate expiry, clock offset, and security events. None carries a tenant label. The [metrics reference](/docs/reference/metrics) lists every family with its type, labels, help text and buckets, generated from `metricCatalog()`, so it always matches the code. This page covers who records them. Two IAM series have hooks in the IAM bridge you can point at them: `bridge.plugin({ onSpill })` is called for every event the plugin spills, and `bridge.resolveResource(fallback, { onLookup })` reports each lookup as `hit`, `miss`, `timeout`, `unavailable`, `scope` or `fallback`. Hook failures never change a decision. ```ts const plugin = bridge.plugin({ onSpill: () => standard.iamInboxSpill.inc() }); const resolveResource = bridge.resolveResource(undefined, { onLookup: (result) => standard.resolverLookups.inc({ result }), }); ``` ### The database collector [#the-database-collector] Some section 14 series describe the whole deployment, not one process. `databaseCollector` samples them from compute's own tables with read-only, bounded statements: ```ts import { databaseCollector, standardMetrics } from 'better-compute/observability'; const collector = databaseCollector(standardMetrics(registry), { sql: compute.context.sql, laneOf: (kind) => compute.context.registry.commandKind(kind)?.lane, // default label (unknown) certificates: () => [{ kind: 'serving', notAfter }], // host certificates besides node certificates }); collector.start(); // every intervalMs (default 15 s); refresh() polls once; stop() ends it ``` | Series | Sampled from | | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `work_queue_depth{kind}` | Due, unclaimed `work_queue` rows by kind | | `backlog{queue}` | Pending `commands`, unprocessed `iam_inbox` rows, unrecorded `usage_windows`, pending `webhook_deliveries`, events a sink gave up on (`outbox_dead_letters`), and each `outbox_cursors` reader's lag as `outbox:`; each capped at 100,000 | | `node_connectivity{state}` | Nodes by connectivity | | `generation_lag_nodes` | Nodes whose observed generation is behind | | `pool_gpus{state}` | `pool_capacity_rollup` GPUs: total, allocatable, allocated, reserved, provisioning, unhealthy, maintenance, unreachable | | `cert_expiry_seconds{kind}` | Each node's newest live certificate, soonest per role as `node:`, plus the `certificates` you pass | | `clock_offset_seconds` | This process's clock minus `compute_now()` | | `command_latency_seconds{kind,lane}` | Commands settled since the last poll, `updated_at - created_at` (needs `commands.created_at`, `core/0020`) | A table an optional module did not create is skipped, and a label set that disappears reads 0 instead of going stale. Latency starts at the first poll, so a restart does not observe history twice, and at most 5,000 settled commands are observed per poll. Poll failures go to `onError` and never throw. Any number of processes may run the collector, and `serve` starts it in every process with a worker role. Each one reports the same deployment-wide gauges, so query them with `max` rather than `sum` across instances. The latency histogram counts each command once. Only the holder of the `leases` row `metrics:command-latency` observes `command_latency_seconds`: the row is taken or renewed on each poll with a TTL of three polls, and released on `stop()`. A stopped collector never takes it again. Pass `latencyLease` to name another row, or `false` to observe in every process. ### Series the instrumentation adds [#series-the-instrumentation-adds] Beyond the standard set, these come from the collectors and wrappers that record them: | Series | From | Labels | | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------------------------- | | `better_compute_gateway_open_sessions` | `gatewayCollector` | `gateway`, `role` | | `better_compute_gateway_stream_headroom`, `_draining`, `_lease_held`, `_commands_in_flight`, `_sessions_upgrade_required` | `gatewayCollector` | `gateway` | | `better_compute_gateway_loop_consecutive_failures`, `_loop_failures`, `_loop_last_ok_age_seconds` | `gatewayCollector` | `gateway`, `loop` (dispatch, lease, liveness, specs, status) | | `better_compute_gateway_service_calls_total`, `_service_seconds` | `meteredGatewayServices` | `method` (join, renew, secret), `outcome` | | `better_compute_attestor_verifications_total`, `_verification_seconds` | `meteredAttestor` | `attestor`, `outcome` (`admitted`, `deferred-`, `refused` or an error code) | | `better_compute_provider_calls_total`, `provider_call_seconds` | `meteredProvider` | `provider`, `method`, `outcome` | | `better_compute_revocation_feed_consecutive_failures` | `revocationFeedCollector` | `feed` | | `better_compute_events_total` | `eventCounter` via `eventsModule({ countEvent })` | `family`, `module` | | `better_compute_placements_total`, `booking_conflicts_total`, `placement_seconds` | `placementObserver` via `capacityModule({ onPlacement })` | `outcome`; none; `placer` (`inline`, `pool-lease`, `batch`) | | `better_compute_spans_total`, `span_duration_seconds` | `createObserver({ metrics })` | `kind`, `name`, `outcome`, `code`; `kind` | `gatewayCollector` and `meteredGatewayServices` are wired by `serve` for the gateway role; the others are yours to wrap around what you register. Placements, for example, are observed by passing capacity's hook the standard series (`booking_conflicts_total` counts replans forced by a booking conflict): ```ts capacityModule({ onPlacement: placementObserver(standardMetrics(registry), registry) }); ``` Outcome labels are bounded: ids, method names, error codes and catalog families. Help text and buckets for each of these are in the [metrics reference](/docs/reference/metrics). ## Spans and traces [#spans-and-traces] `createObserver({ onSpan?, metrics?, now? })` times units of work as a `ComputeSpan`, shaped like better-iam's `IamSpan`: ```ts interface ComputeSpan { kind: 'operation' | 'decision' | 'http' | 'command' | 'reconcile' | 'placement' | 'iam' | 'sink'; name: string; // method, route, command kind or reconciler tenantId?: string; // for hooks only, never a label outcome: 'ok' | 'denied' | 'error'; code?: string; status?: number; requestId?: string; durationMs: number; } ``` `observer.span(kind, name, fn, extra?)` runs `fn`, classifies the outcome (401, 403 and 429, or the codes `UNAUTHENTICATED`, `ACCESS_DENIED`, `CONFIRMATION_REQUIRED`, `RATE_LIMITED`, `CSRF_REJECTED` and `UNTRUSTED_ORIGIN`, are `denied`; anything else that throws is `error`), fans the span out to `onSpan` hooks and feeds the span metrics. Hooks and metrics can never affect the work: their failures are swallowed. A name the caller controls (an unknown route, `NOT_FOUND`, `INVALID_INPUT`, `METHOD_NOT_ALLOWED`) is recorded as `(invalid)` or `(unknown)`, so it cannot grow the series. The server does not create an observer itself; hosts use it to export spans to their tracing system. **Trace context.** `parseTraceparent`, `formatTraceparent` and `childTraceparent` handle W3C Trace Context version `00` (`00-<32 hex trace id>-<16 hex parent id>-<2 hex flags>`), refusing all-zero ids and version `ff`. The protocol carries `traceparent` on every `CommandBody` and `DesiredStateBody`, and the driver SDK passes it to drivers, so a daemon's work can join the request's trace. A valid `traceparent` header on an API request (version `00`, non-zero trace and parent ids) reaches the method as `MethodContext.traceparent`, and `ctx.commands.enqueue` in that method defaults `CommandBody.traceparent` to it, so the commands a call enqueues carry the caller's trace. An invalid header is dropped, not refused. Commands enqueued by reconcilers, and NodeSpecs, go out with it empty. **Request ids.** The API takes the caller's `X-Request-Id` when it is 1 to 128 characters of letters, digits and `._:-`, else it mints a `req_` id, and returns it in the response's `X-Request-Id`. IAM receives the same id in the forwarded headers, and the admission context of a mutation's audit row records it, so a support ticket with the header finds the call. **Logs.** `betterCompute({ log })` takes a `Logger`; the default writes warnings and errors as JSON to standard error. GPU telemetry does not go through the control plane (principle P8); the nodes send it straight to your collector. ## Node metrics from bcd [#node-metrics-from-bcd] bcd exports its own metrics over OTLP/HTTP, in the JSON encoding, to the collector its NodeSpec's `telemetry` section names (27c7ee2). Nothing reaches the gateway, the session or PostgreSQL. **Where to.** `nodeTelemetryModule(options)` (53aa356) gives every node that section: `options` are the platform's collector, and a pool's `policy.telemetry` wins field by field, with `resourceAttributes` merged by key. A node whose result has no endpoint gets no section, and exports nothing. The module is not registered by default; add it to your modules: ```ts title="compute.ts" import { nodeTelemetryModule } from 'better-compute/observability'; betterCompute({ // … modules: [ nodeTelemetryModule({ otlpEndpoint: 'https://otel.example.internal:4318', intervalSeconds: 30, headersRef: 'secret://root/sec_01k…', // the collector's auth headers resourceAttributes: { 'deployment.environment': 'prod' }, }), ], }); ``` | Setting | Rules | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `otlpEndpoint` | `https://`, or `http://` to `localhost`, `127.0.0.1` or `[::1]`; no credentials. bcd appends `/v1/metrics` unless the path already ends with it | | `intervalSeconds` | 5 to 3600; bcd's default is 60 seconds and its minimum 5 | | `headersRef` | A secret reference to the headers: a JSON object, or `Name: value` lines | | `resourceAttributes` | At most 32, single-line values of at most 256 characters; keys under `bcd.` and `service.` are bcd's own | | `roles` | Module option only: the node roles that export (default all, site agents included) | The module checks its options when it is built (`INVALID_INPUT`). A pool whose `policy.telemetry` fails the same checks is skipped, and its nodes keep the default; `better-compute doctor` (and the support bundle) reports such pools as `pool-telemetry-invalid`, from `nodeTelemetryFindings(sql)`. **What.** Every interval, bcd sends: | Metric | Unit | From | | --------------------------------------------------------------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `bcd.gpu.memory.used`, `bcd.gpu.memory.total`, `bcd.gpu.memory.reserved` | bytes | `nvidia-smi`, per GPU | | `bcd.gpu.utilization`, `bcd.gpu.memory.utilization` | ratio (0 to 1) | `nvidia-smi` | | `bcd.gpu.temperature` | Celsius | `nvidia-smi` | | `bcd.gpu.power.draw`, `bcd.gpu.power.limit` | watts | `nvidia-smi` | | `bcd.gpu.clock.sm`, `bcd.gpu.clock.memory` | MHz | `nvidia-smi` | | `bcd.gpu.processes` | processes | `nvidia-smi` | | `bcd.gpu.mig.memory.used`, `bcd.gpu.mig.memory.total` | bytes | `nvidia-smi`, per MIG instance (`mig.id`) | | `bcd.gpu.ecc.errors` | errors, cumulative sum | the inventory's aggregate ECC counters, `type` `correctable` or `uncorrectable` | | `bcd.gpu.rows.remapped` | rows | the inventory's row remapping, by `type` | | `bcd.gpu.xid.active` | conditions | bcd's active XID conditions per GPU | | `bcd.host.pressure.cpu`, `bcd.host.pressure.memory`, `bcd.host.pressure.io` | percent | PSI `some avg10` | | `bcd.host.cpu.steal` | percent | CPU steal | | `bcd.host.memory.total`, `bcd.host.memory.available` | bytes (`By`) | `/proc/meminfo`'s `MemTotal` and `MemAvailable` (3111284) | | `bcd.host.hugepages.total`, `bcd.host.hugepages.free` | pages (`{page}`) | `/sys/kernel/mm/hugepages`, each pool's `nr_hugepages` and `free_hugepages`, by `page.size` (the page size in bytes) | | `bcd.instances` | instances (`{instance}`) | every instance runtime's list of domains, managed or not, counted by `runtime` and `state` (`running`, `stopped`, `paused`, `error`) | | `bcd.instance.vcpu` | vCPUs (`{cpu}`) | each domain's configured vCPUs, by `instance.id`, `runtime` and `managed` | | `bcd.instance.memory` | bytes (`By`) | each domain's configured memory, by the same attributes and `hugepages` (the backing page size in bytes, `0` for normal pages) | | `bcd.instance.memory.balloon.reclaimed` | bytes (`By`) | the memory a ballooned guest lends the host now: its configured memory less what its balloon leaves it, by `instance.id` and `runtime` (9bdd43a; `bcd.instance.memory.balloon` before). Only for running, managed guests whose spec gives them a balloon (`placement.balloon`); a guest without a balloon reading counts as fully deflated (0). See [memory ballooning](/docs/daemon/delivery#memory-ballooning) | | `bcd.instance.cpu.time` | seconds (`s`), cumulative sum | live usage (0b21b4b): CPU time since the domain started, by `instance.id` and `runtime` | | `bcd.instance.memory.used` | bytes (`By`) | live usage: the memory the domain uses now | | `bcd.instance.io.read`, `bcd.instance.io.write` | bytes (`By`), cumulative sum | live usage: disk bytes read and written | | `bcd.instance.network.receive`, `bcd.instance.network.transmit` | bytes (`By`), cumulative sum | live usage: bytes the guest received and sent | **Instances.** The host and instance series come every interval from a node that has them: the memory and huge page series from any Linux host, the `bcd.instance*` series only from a host that runs an instance runtime (VPS and sandbox hosts). The configured series (`bcd.instances`, `bcd.instance.vcpu`, `bcd.instance.memory`) cover every runtime, plugin runtimes included. The live usage series come only from runtimes that can read it (`runtime.UsageReader`), for their running domains, and count from the guest's side: | Runtime | Where the usage comes from | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cloud-hypervisor` | The VMM unit's cgroup v2, `/sys/fs/cgroup/bcd-instances.slice/bcd-ch-.service`: `cpu.stat` `usage_usec`, `memory.current`, and `io.stat`'s `rbytes` and `wbytes` over every device; plus the counters of the instance's taps, where a tap's transmitted bytes are what the guest received. Managed domains that are running or paused only | | `libvirt-kvm` | `virsh domstats --raw --state-running --cpu-total --balloon --interface --block`: `cpu.time`, `balloon.rss` (the QEMU process's resident memory) and the summed `net.*` and `block.*` byte counters | A domain whose usage cannot be read is left out of the live series. Plugin runtimes report none, and a fakehw build's file-backed runtime reports synthetic usage (half the configured memory, a second of CPU per boot). Under [`bcd helper`](/docs/daemon#privilege-separation) `libvirt-kvm` reads `domstats` on libvirt's read-only socket (since 12e9ced; before it the read was refused there), and `cloud-hypervisor`, offered there since 7fe74a1, reads the same cgroup and tap counters as the node's own user, with no `ch.*` operation: since 1c9684a `TestCloudHypervisorUnderTheHelper` (`internal/runtime`) reads a helper-started domain's usage from a fake cgroup tree and tap statistics and checks that root was asked nothing. The series are tested against a fake `/proc`, sysfs, cgroup tree and captured `domstats` output (`internal/agent` `telemetry_host_test.go`, `internal/runtime` `usage_test.go`). On the control-plane side, `tests/e2e/helper-vps.test.ts` runs sandboxes on FakeDaemon VPS hosts under the helper and checks that their `bcd.instance.*` usage reaches the collector `nodeTelemetryModule` names, and leaves the next export once they are deleted (see [testing](/docs/operations/testing)). GPU points carry `gpu.uuid`, `gpu.index` and `gpu.model`. The resource carries `service.name=bcd`, `service.version`, `host.name`, `bcd.node.id` and `bcd.site.id`, then the section's own attributes (which cannot replace those). bcd fetches the headers with a `SecretRequest` that names workflow `host:telemetry` and step `headersRef`; the secret broker releases the reference only while the node's current spec names it there (binding rule 3; see [sealed secrets](/docs/operations/security#sealed-secrets)). **Failures.** An endpoint that is not an http(s) URL, or a malformed resource attribute, fails the `telemetry` section terminally; an empty endpoint stops the export. A collector's 4xx answer other than 429 drops that batch; after anything else the export runs again on the next interval, with that interval's fresh readings (a failed batch is never re-sent). bcd logs a failure once, not every interval, and logs when the export resumes. The exporter restarts whenever the section changes. `bcd status` prints the endpoint, the last export and the last error (ad8791f). It is tested against a fake collector, and since efdff1d end to end: the real fakehw bcd in `tests/e2e/bcd.test.ts` posts its GPU metrics to a local OTLP receiver the test runs, named in `NodeSpec.telemetry` (see [testing](/docs/operations/testing)). No collector product has received it yet. ## Health and readiness [#health-and-readiness] The CLI host answers two routes beside the API on every `api` listener: | Route | Answers | | -------------- | -------------------------------------------------------------------------------------- | | `GET /healthz` | 200 `{ "ok": true }` while the process serves requests | | `GET /readyz` | 200 `{ "ok": true }` when `SELECT 1` succeeds on the store, else 503 `{ "ok": false }` | Point the load balancer's health check at `/readyz`. A gateway-only or worker-only process has no HTTP listener; watch it through the gateway series and leases instead. ## doctor [#doctor] `better-compute doctor` runs `compute.selfCheck()` and adds the CLI's own findings. It only reads. `--strict` exits non-zero with `DOCTOR_FINDINGS` when any error or warning is found; `--pki PATH` names the PKI bundle to check. ```bash better-compute doctor --strict --query "findings[].check" ``` | Check | Severity | Finds | | ------------------------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `migrations` | error | A database never migrated, or migrations of this release and its modules not applied | | `btree-gist` | error | The extension the booking exclusion constraints need is missing | | `clock` | error or info | `compute_now()` is not the database clock (info when it reads the sim clock) | | `clock.skew` | warning | This process's clock differs from `compute_now()` by more than 1 s | | `catalog.*` | error or warning | Methods whose actions are uncatalogued, mismatched or undeclared | | `database` | error | The check itself could not run | | `workers.queue` | warning | Work items more than 5 minutes overdue and unclaimed, noting when no worker has heartbeated in the last minute | | `workers.sweeper` | warning | Pending commands more than 2 minutes past `not_after` and not yet expired: no worker holds the CommandSweeper's lease | | `usage.dead-letters` | warning | Unresolved rows in `usage_dead_letters` | | `usage.backlog` | warning | Closed usage windows unrecorded for more than 2 hours | | `fleet.gateway-headroom` | warning | Live, non-draining gateways holding at least 80% of their `max_streams` | | `fleet.heartbeats` | warning | Connected nodes whose session sent nothing for more than 2 minutes | | `fleet.daemon-skew` | warning | Connected daemons whose newest protocol is below the control plane's minimum | | `fleet.cert-expiry` | warning | Connected nodes with no unrevoked certificate valid for another hour: renewals are failing | | `iam.not-configured` | warning | No IAM is configured, so only public methods answer | | `iam.projection` | error | The authz projection is not loaded, so every decision is denied | | `iam.inbox-spill` | warning or error | IAM events spilled in memory, or dropped | | `iam.inbox-backlog` | warning | More than 10,000 IAM events wait in `iam_inbox` | | `iam.usage-dead-letters` | warning | Usage events that could not be recorded in IAM billing | | `iam.usage-backlog` | warning or error | Usage windows over 2 hours unrecorded (warning), or windows of a past month that IAM's `closePeriod` will refuse (error) | | `events.webhook-dead-letters` | warning | Webhook deliveries that failed every attempt | | `events.webhook-backlog` | warning | More than 1,000 deliveries overdue by 5 minutes | | `capacity.reservations.stale` | warning | Reservations more than 10 minutes past their end and not released | | `capacity.mig.stuck` | warning | Nodes that have waited more than 24 hours for a MIG layout change | | `capacity.dedicated.unoffered` | warning | Nodes dedicated to a tenant their pool is not offered to, so nobody can use them | | `capacity.device_counts.drift` | error | The trigger-kept per-node device counts the placement shortlist ranks on disagree with `devices` (a `TRUNCATE` bypasses the trigger) | | `vps.pools.runtime` | error | A `vps-host` pool runs a runtime this deployment does not list in `vps({ runtimes })` | | `vps.pools.single-tenant` | error | A container-runtime pool without `policy.singleTenant: true`, which cannot run instances | | `vps.pools.ipam` | warning | A `vps-host` pool without an IP range, so its instances get no address | | `vps.hosts.capacity` | warning | Adopted VPS hosts without vCPU, memory and disk capacity rows: no inventory yet, or no worker runs `vps.hosts` | | `vps.sandboxes.expiry` | error | Sandboxes more than two reconciler passes (30 s) past `expiresAt` and not deleting: no worker runs the `vps.sandboxes` reconciler, or it is failing | | `vps.sandboxes.deleting` | warning | Sandboxes deleting for more than 10 minutes: their instance is stuck (its host must report it gone, or be declared lost) | | `vps.sandboxes.warm` | warning | Templates below `warm.min` settled warm instances for over 10 minutes with no refill in that time (quota, capacity, a vanished flavor or image), and warm instances not settled within 10 minutes | | `vps.sandboxes.network` | error | Open sandboxes with a `private` or `none` network on a host without the `bc.dev/cap.instance.network-policy` label (placed before the capability constraint, or its bcd since downgraded): their guests have full egress. Upgrade bcd there, or delete the sandboxes | | `workflows-stuck` | warning | `provision`: active workflows with an in-flight step more than 5 minutes past its deadline. The engine fails an overdue step whenever it looks, so nobody runs their `workflow` reconciler | | `workflows-failed` | info | `provision`: workflows that failed in the last day; inspect with `workflows.get`, retry with `workflows.retry` | | `metal-failed` | warning | `metal`: machines `failed` with maintenance on, waiting for `metal.recover` or `metal.retire`, listed with their failure codes | | `metal-stalled` | warning | `metal`: machines more than 10 minutes in a workflow-driven state without an active workflow, so no worker runs the `metal` reconciler (a rescued machine resting after its rescue workflow is not stalled) | | `pool-telemetry-invalid` | warning | Pools whose `policy.telemetry` fails `nodeTelemetryModule`'s checks; their nodes export to the default collector (or none) until it is fixed | | `audit.checkpoints` | warning | Audit rows older than 3 hours not sealed by a checkpoint | | `history-partitions` | warning | `retentionModule()`: this or next month's partition of a history table is missing, so the retention pass is not running | | `history-default-partition` | warning | `retentionModule()`: rows of the last 13 months sit in a history table's default partition | | `retention-cursors` | warning | `retentionModule()`: a durable `outbox_cursors` row has not moved for a day, so its stream is not pruned behind it | | `pki.*` | error or warning | The PKI bundle unreadable or not unsealable, or a CA expired (error) or expiring (warning) | | `host.*` | error or warning | The `host` export: routes, tunnels, IAM, and whether the gateway role can authenticate daemons, reach the API's pokes, tunnels and revocations | [`system.selfCheck`](/docs/reference/api/system#selfcheck) serves the server's part of the report over the API (action `compute:fleet:admin` on `compute-fleet`); the `pki.*`, `host.*` and `iam.not-configured` findings come from the CLI only. The `workers.*`, `usage.*` and `fleet.*` checks are one read-only statement the server runs once the schema is fully migrated (they are skipped while a `migrations` finding stands). Each is a warning with a `count` and a `fix`, and each maps to an operations check of PLAN.md section 14: queues, the sweeper, usage, gateway headroom, heartbeats, daemon version skew and certificate expiry. ## What to alert on [#what-to-alert-on] These follow from the series and findings above, and from PLAN.md section 8.6 on what breaks first. * **The gateway is fenced or stuck.** `better_compute_gateway_lease_held` at 0, or `better_compute_gateway_loop_consecutive_failures` rising for any loop. A gateway that loses its lease drops its streams; nodes re-home to the others. * **Stream headroom.** `better_compute_gateway_stream_headroom` near 0 (default `maxStreams` 5,000 per gateway): add a gateway before handshakes are refused. * **The IAM lane.** `better_compute_iam_call_duration_seconds` p99 rising, or `iam_calls_total{outcome="error"}`. Every IAM call takes at least one global lock acquisition in embedded mode. * **Backlogs.** `workers.queue`, `workers.sweeper`, `iam.inbox-backlog`, `usage.backlog`, `events.webhook-backlog` and `audit.checkpoints` findings: a worker role is not running or cannot keep up. Run `doctor --strict` on a schedule and alert on a non-zero exit. * **Clock.** `clock.skew`: run NTP on every host. Business time stays anchored to the database, but ids and logs use the host clock. * **Stale reservations.** `capacity.reservations.stale`: the `controllers` role is not running reconcilers. * **Revocations.** `better_compute_revocation_feed_consecutive_failures` above 0: revoked certificates keep their sessions while the feed fails. * **Series cap.** `better_compute_metrics_collapsed_total` rising means some label is not bounded. - [Scaling](/docs/operations/scaling): The numbers behind these limits. - [Retention](/docs/operations/retention): What the housekeeping pass prunes. - [Security](/docs/operations/security): The threat model and hardening. - [The gateway](/docs/daemon/gateway): Leases, loops and admission control.