Observability

@better-compute/observability holds the pieces a host uses to watch a deployment: a Prometheus registry that refuses tenant labels and caps its series, the standard better_compute_* metric set of PLAN.md section 14, a GET /metrics handler behind a bearer token, wrappers that meter IAM calls, gateway services, attestors and providers, ComputeSpan observers and W3C traceparent helpers. It is browser-safe (no Node.js imports), so every package can depend on it. The CLI host mounts it for you when the configuration's host export has a registry.

Wire metrics into a deployment

One registry per process. The configuration creates it, registers the standard series, passes it to what it meters, and hands it to the host through host.metrics:

import { defineHost } from 'better-compute/cli';
import { createMetrics, eventCounter, meteredIamPort, standardMetrics } from 'better-compute/observability';
import { eventsModule } from 'better-compute/events';
import { defineConfig } from 'better-compute/server';

export const metrics = createMetrics(); // maxSeries 2000 per metric
const standard = standardMetrics(metrics);

export default defineConfig(({ env = process.env } = {}) => ({
  // …database, secret, baseURL
  iam: meteredIamPort(bridge.embedded(iam, { rootTenantId }), standard), // iam_calls_total, iam_call_duration_seconds
  modules: [eventsModule({ secret: env.BETTER_COMPUTE_SECRET!, countEvent: eventCounter(metrics) })],
}));

export const host = defineHost(async ({ env }) => ({
  metrics: { registry: metrics, bearerToken: env.BETTER_COMPUTE_METRICS_TOKEN }, // GET /metrics
}));

better-compute serve then:

• mounts GET /metrics (or host.metrics.path) on the API listener when bearerToken is set; without a token there is no route;
• when it runs the gateway role, meters compute.gatewayServices with meteredGatewayServices and samples the gateway at every scrape with gatewayCollector;
• when it runs any worker role, starts databaseCollector into the same registry: a first reading at start, then every 15 s, stopped with the server. It takes each command kind's lane from the registry and watches the host's own certificates as cert_expiry_seconds{kind}: node-ca, command-signing and serving (the gateway's TLS certificate).

better-compute dev mounts the configuration's host.metrics too. The examples read the token from BETTER_COMPUTE_METRICS_TOKEN.

Scrape it with the token as a bearer credential:

scrape_configs:
  - job_name: better-compute
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/better-compute.token
    static_configs:
      - targets: ['compute-api-1:3000', 'compute-api-2:3000']

The registry

createMetrics({ prefix?, maxSeries?, forbiddenLabels?, now? }) returns a registry of counters, gauges and histograms.

• Prefix. Every name gets better_compute_ unless it already has it. Names and labels must match [a-z_][a-z0-9_]*; le is reserved.
• No tenant labels. A label named tenant, tenant_id, tenantid, org, organization, identity or user is refused with INVALID_CONFIG when the metric is defined. Per-tenant numbers belong in usage and audit, not in your TSDB.
• Bounded series. Each metric holds at most maxSeries label sets (default 2000). New label sets past the cap fold into one series whose labels all read (other), and better_compute_metrics_collapsed_total counts every folded recording.
• Caller-controlled values. A label value that is not a string or is longer than 256 characters reads (invalid); an empty one reads (unknown).
• Collectors. registry.collect(fn) runs fn at the start of every expose(), for gauges sampled at scrape time. A collector that throws is skipped for that scrape.
• Exposition. expose() renders text format 0.0.4. value(name, labels) and stats() (series, dropped) serve tests and health checks.

metricsHandler({ registry, bearerToken, timingSafeEqual }) answers GET only (405 otherwise), requires a token of at least 16 characters, compares the SHA-256 digests of the presented and expected tokens with the constant-time function you pass (crypto.timingSafeEqual in Node), answers 401 with WWW-Authenticate: Bearer on a mismatch, and serves the text with Cache-Control: no-store.

The standard series

standardMetrics(registry) defines the section 14 series: sessions and handshakes, command latency and sweeps, the work queue and reconcile durations, placements and booking conflicts, pool GPUs, node connectivity, backlogs, IAM calls and resolver lookups, certificate expiry, clock offset, and security events. None carries a tenant label. The metrics reference lists every family with its type, labels, help text and buckets, generated from metricCatalog(), so it always matches the code. This page covers who records them.

Two IAM series have hooks in the IAM bridge you can point at them: bridge.plugin({ onSpill }) is called for every event the plugin spills, and bridge.resolveResource(fallback, { onLookup }) reports each lookup as hit, miss, timeout, unavailable, scope or fallback. Hook failures never change a decision.

const plugin = bridge.plugin({ onSpill: () => standard.iamInboxSpill.inc() });
const resolveResource = bridge.resolveResource(undefined, {
  onLookup: (result) => standard.resolverLookups.inc({ result }),
});

The database collector

Some section 14 series describe the whole deployment, not one process. databaseCollector samples them from compute's own tables with read-only, bounded statements:

import { databaseCollector, standardMetrics } from 'better-compute/observability';

const collector = databaseCollector(standardMetrics(registry), {
  sql: compute.context.sql,
  laneOf: (kind) => compute.context.registry.commandKind(kind)?.lane, // default label (unknown)
  certificates: () => [{ kind: 'serving', notAfter }], // host certificates besides node certificates
});
collector.start(); // every intervalMs (default 15 s); refresh() polls once; stop() ends it

A table an optional module did not create is skipped, and a label set that disappears reads 0 instead of going stale. Latency starts at the first poll, so a restart does not observe history twice, and at most 5,000 settled commands are observed per poll. Poll failures go to onError and never throw.

Any number of processes may run the collector, and serve starts it in every process with a worker role. Each one reports the same deployment-wide gauges, so query them with max rather than sum across instances. The latency histogram counts each command once. Only the holder of the leases row metrics:command-latency observes command_latency_seconds: the row is taken or renewed on each poll with a TTL of three polls, and released on stop(). A stopped collector never takes it again. Pass latencyLease to name another row, or false to observe in every process.

Series the instrumentation adds

Beyond the standard set, these come from the collectors and wrappers that record them:

gatewayCollector and meteredGatewayServices are wired by serve for the gateway role; the others are yours to wrap around what you register. Placements, for example, are observed by passing capacity's hook the standard series (booking_conflicts_total counts replans forced by a booking conflict):

capacityModule({ onPlacement: placementObserver(standardMetrics(registry), registry) });

Outcome labels are bounded: ids, method names, error codes and catalog families. Help text and buckets for each of these are in the metrics reference.

Spans and traces

createObserver({ onSpan?, metrics?, now? }) times units of work as a ComputeSpan, shaped like better-iam's IamSpan:

interface ComputeSpan {
  kind: 'operation' | 'decision' | 'http' | 'command' | 'reconcile' | 'placement' | 'iam' | 'sink';
  name: string; // method, route, command kind or reconciler
  tenantId?: string; // for hooks only, never a label
  outcome: 'ok' | 'denied' | 'error';
  code?: string;
  status?: number;
  requestId?: string;
  durationMs: number;
}

observer.span(kind, name, fn, extra?) runs fn, classifies the outcome (401, 403 and 429, or the codes UNAUTHENTICATED, ACCESS_DENIED, CONFIRMATION_REQUIRED, RATE_LIMITED, CSRF_REJECTED and UNTRUSTED_ORIGIN, are denied; anything else that throws is error), fans the span out to onSpan hooks and feeds the span metrics. Hooks and metrics can never affect the work: their failures are swallowed. A name the caller controls (an unknown route, NOT_FOUND, INVALID_INPUT, METHOD_NOT_ALLOWED) is recorded as (invalid) or (unknown), so it cannot grow the series. The server does not create an observer itself; hosts use it to export spans to their tracing system.

Trace context. parseTraceparent, formatTraceparent and childTraceparent handle W3C Trace Context version 00 (00-<32 hex trace id>-<16 hex parent id>-<2 hex flags>), refusing all-zero ids and version ff. The protocol carries traceparent on every CommandBody and DesiredStateBody, and the driver SDK passes it to drivers, so a daemon's work can join the request's trace. A valid traceparent header on an API request (version 00, non-zero trace and parent ids) reaches the method as MethodContext.traceparent, and ctx.commands.enqueue in that method defaults CommandBody.traceparent to it, so the commands a call enqueues carry the caller's trace. An invalid header is dropped, not refused. Commands enqueued by reconcilers, and NodeSpecs, go out with it empty.

Request ids. The API takes the caller's X-Request-Id when it is 1 to 128 characters of letters, digits and ._:-, else it mints a req_ id, and returns it in the response's X-Request-Id. IAM receives the same id in the forwarded headers, and the admission context of a mutation's audit row records it, so a support ticket with the header finds the call.

Logs. betterCompute({ log }) takes a Logger; the default writes warnings and errors as JSON to standard error. GPU telemetry does not go through the control plane (principle P8); the nodes send it straight to your collector.

Node metrics from bcd

bcd exports its own metrics over OTLP/HTTP, in the JSON encoding, to the collector its NodeSpec's telemetry section names (27c7ee2). Nothing reaches the gateway, the session or PostgreSQL.

Where to. nodeTelemetryModule(options) (53aa356) gives every node that section: options are the platform's collector, and a pool's policy.telemetry wins field by field, with resourceAttributes merged by key. A node whose result has no endpoint gets no section, and exports nothing. The module is not registered by default; add it to your modules:

import { nodeTelemetryModule } from 'better-compute/observability';

betterCompute({
  // …
  modules: [
    nodeTelemetryModule({
      otlpEndpoint: 'https://otel.example.internal:4318',
      intervalSeconds: 30,
      headersRef: 'secret://root/sec_01k…', // the collector's auth headers
      resourceAttributes: { 'deployment.environment': 'prod' },
    }),
  ],
});

The module checks its options when it is built (INVALID_INPUT). A pool whose policy.telemetry fails the same checks is skipped, and its nodes keep the default; better-compute doctor (and the support bundle) reports such pools as pool-telemetry-invalid, from nodeTelemetryFindings(sql).

What. Every interval, bcd sends:

Instances. The host and instance series come every interval from a node that has them: the memory and huge page series from any Linux host, the bcd.instance* series only from a host that runs an instance runtime (VPS and sandbox hosts). The configured series (bcd.instances, bcd.instance.vcpu, bcd.instance.memory) cover every runtime, plugin runtimes included. The live usage series come only from runtimes that can read it (runtime.UsageReader), for their running domains, and count from the guest's side:

A domain whose usage cannot be read is left out of the live series. Plugin runtimes report none, and a fakehw build's file-backed runtime reports synthetic usage (half the configured memory, a second of CPU per boot). Under bcd helper libvirt-kvm reads domstats on libvirt's read-only socket (since 12e9ced; before it the read was refused there), and cloud-hypervisor, offered there since 7fe74a1, reads the same cgroup and tap counters as the node's own user, with no ch.* operation: since 1c9684a TestCloudHypervisorUnderTheHelper (internal/runtime) reads a helper-started domain's usage from a fake cgroup tree and tap statistics and checks that root was asked nothing. The series are tested against a fake /proc, sysfs, cgroup tree and captured domstats output (internal/agent telemetry_host_test.go, internal/runtime usage_test.go). On the control-plane side, tests/e2e/helper-vps.test.ts runs sandboxes on FakeDaemon VPS hosts under the helper and checks that their bcd.instance.* usage reaches the collector nodeTelemetryModule names, and leaves the next export once they are deleted (see testing).

GPU points carry gpu.uuid, gpu.index and gpu.model. The resource carries service.name=bcd, service.version, host.name, bcd.node.id and bcd.site.id, then the section's own attributes (which cannot replace those). bcd fetches the headers with a SecretRequest that names workflow host:telemetry and step headersRef; the secret broker releases the reference only while the node's current spec names it there (binding rule 3; see sealed secrets).

Failures. An endpoint that is not an http(s) URL, or a malformed resource attribute, fails the telemetry section terminally; an empty endpoint stops the export. A collector's 4xx answer other than 429 drops that batch; after anything else the export runs again on the next interval, with that interval's fresh readings (a failed batch is never re-sent). bcd logs a failure once, not every interval, and logs when the export resumes. The exporter restarts whenever the section changes. bcd status prints the endpoint, the last export and the last error (ad8791f). It is tested against a fake collector, and since efdff1d end to end: the real fakehw bcd in tests/e2e/bcd.test.ts posts its GPU metrics to a local OTLP receiver the test runs, named in NodeSpec.telemetry (see testing). No collector product has received it yet.

Health and readiness

The CLI host answers two routes beside the API on every api listener:

Point the load balancer's health check at /readyz. A gateway-only or worker-only process has no HTTP listener; watch it through the gateway series and leases instead.

doctor

better-compute doctor runs compute.selfCheck() and adds the CLI's own findings. It only reads. --strict exits non-zero with DOCTOR_FINDINGS when any error or warning is found; --pki PATH names the PKI bundle to check.

better-compute doctor --strict --query "findings[].check"

system.selfCheck serves the server's part of the report over the API (action compute:fleet:admin on compute-fleet); the pki.*, host.* and iam.not-configured findings come from the CLI only.

The workers.*, usage.* and fleet.* checks are one read-only statement the server runs once the schema is fully migrated (they are skipped while a migrations finding stands). Each is a warning with a count and a fix, and each maps to an operations check of PLAN.md section 14: queues, the sweeper, usage, gateway headroom, heartbeats, daemon version skew and certificate expiry.

What to alert on

These follow from the series and findings above, and from PLAN.md section 8.6 on what breaks first.

• The gateway is fenced or stuck. better_compute_gateway_lease_held at 0, or better_compute_gateway_loop_consecutive_failures rising for any loop. A gateway that loses its lease drops its streams; nodes re-home to the others.
• Stream headroom. better_compute_gateway_stream_headroom near 0 (default maxStreams 5,000 per gateway): add a gateway before handshakes are refused.
• The IAM lane. better_compute_iam_call_duration_seconds p99 rising, or iam_calls_total{outcome="error"}. Every IAM call takes at least one global lock acquisition in embedded mode.
• Backlogs. workers.queue, workers.sweeper, iam.inbox-backlog, usage.backlog, events.webhook-backlog and audit.checkpoints findings: a worker role is not running or cannot keep up. Run doctor --strict on a schedule and alert on a non-zero exit.
• Clock. clock.skew: run NTP on every host. Business time stays anchored to the database, but ids and logs use the host clock.
• Stale reservations. capacity.reservations.stale: the controllers role is not running reconcilers.
• Revocations. better_compute_revocation_feed_consecutive_failures above 0: revoked certificates keep their sessions while the feed fails.
• Series cap. better_compute_metrics_collapsed_total rising means some label is not bounded.