Webhooks

A webhook is a tenant's subscription: an HTTPS URL and a list of event type patterns. Every committed event of that tenant whose type matches is POSTed to the URL as JSON, signed with a secret only the tenant and Better Compute know. Delivery is at least once, with retries over about two days, a dead-letter state you can inspect, and redelivery on request.

Webhooks come from the events module. Add it to your deployment with a secret of at least 32 characters, shared by every process:

import { eventsModule } from 'better-compute/events';

const compute = betterCompute({
  // …
  modules: [eventsModule({ secret: process.env.BETTER_COMPUTE_SECRET! })],
});

The module adds the webhooks API group, the tables webhooks, webhook_deliveries and event_cursors (migrations events/0001_webhooks and events/0002_delivery_retention), and three reconcilers that a worker process runs: events.fanout every second, events.deliver every two seconds and events.prune every hour. Without a worker running reconcilers (the controllers role of better-compute serve), webhooks are stored but nothing is sent.

How a delivery happens

Fan-out. events.fanout reads the outbox after its cursor in event_cursors, with the same xmin rule as every outbox reader, and queues one delivery for each active webhook of the event's tenant whose patterns match and that existed when the event was written. The queue insert and the cursor move commit together, and (webhook_id, event_id) is unique, so an event is queued at most once per webhook. The cursor is anchored at the outbox tail when the control plane starts: history from before the first start is never replayed, and a new webhook receives events from its creation on.

Delivery. events.deliver claims up to 50 due deliveries per pass and leases each for the timeout plus 30 seconds, so a crashed worker's claims come back. Within a pass, one webhook's deliveries go out one at a time in the order they were queued; different webhooks are served in parallel. Each claim counts as an attempt. A delivery of a paused or deleted webhook is cancelled instead of sent.

Create a webhook

const { webhook, secret } = await client.webhooks.create({
  tenantId: 'acme',
  url: 'https://hooks.example.com/compute',
  events: ['reservation.*', 'node.disconnected', 'node.lost'],
  description: 'Scheduler sync',
});
// Store `secret` now (whsec_...): it is returned once and never again.

A tenant can have 20 webhooks (maxPerTenant); the 21st is QUOTA_EXCEEDED. Every webhooks method is decided as compute:webhooks:manage on compute-tenant/{tenantId}.

webhooks.create takes no idempotency key, so retrying a create whose answer was lost makes a second webhook with its own secret, and both receive every matching event. Before retrying, check webhooks.list and delete the one you do not keep.

Choosing events

A pattern is * (everything), a family ending in .*, or one exact type. reservation.* matches reservation.placed and reservation.active; node.* also matches node.gpu.missing. Patterns are lower case letters, digits and hyphens between dots.

create and update check every pattern against the event catalog and refuse one that nothing can match, such as nodes.*, with INVALID_INPUT and the offenders in details.unknown, instead of letting a typo silently match nothing. webhooks.eventTypes returns the catalog: each family's type, subject, module and description. The event reference lists the same catalog, and the events page lists the individual types you will see in each family.

Modules and plugins that publish their own families register them with eventsModule({ extraEvents: [...] }) so tenants can subscribe; extraEvents: false turns the check off and accepts any well-formed pattern.

Webhook URLs

Webhook URLs are tenant input, and the workers that send deliveries sit inside your platform network, so a URL pointing at a metadata service, a BMC or a database would be a server-side request forgery. Two checks prevent it:

• When saved, the URL must be https, carry no user name or password, and not be an IP literal of a non-public address, localhost, *.localhost or *.internal.
• At every delivery, the host is resolved again and the delivery fails if any address it resolves to is loopback, private, link-local, carrier-grade NAT, multicast, a documentation range or otherwise non-public (IPv4 and IPv6, including IPv4-mapped addresses). This catches DNS names that change after the webhook was saved.

For local development only, eventsModule({ allowInsecure: true }) permits http: and allowPrivate: true permits private and loopback destinations.

Receive and verify

Each delivery is a POST with the event as its JSON body:

{
  "id": "evt_01k5y3v0m6y8x7w5t4s3r2p1n0",
  "type": "reservation.active",
  "tenantId": "acme",
  "subject": "reservation/rsv_01k5y3a0b1c2d3e4f5g6h7j8k9",
  "time": "2026-09-24T10:15:02.114Z",
  "sequence": "88213.51120",
  "data": { "state": "active", "poolId": "pool_01k5y1z2x3w4v5t6s7r8q9p0n1" }
}

The body is stored when the delivery is queued, so every attempt, and every redelivery, sends the same bytes. sequence is the outbox position in the txid.seq form; it is empty for a webhook.ping.

The signature is the scheme Better IAM uses for its own webhooks: an HMAC-SHA256, keyed with the webhook's secret, over the timestamp, a dot and the raw body, ${timestamp}.${body}, as lower-case hex. Verify it before you trust anything in the body, and check that the timestamp is recent so an old delivery cannot be replayed.

import { verifyWebhookSignature } from 'better-compute/events';

export async function POST(request: Request): Promise<Response> {
  const body = await request.text(); // the raw bytes: never re-serialize before verifying
  const valid = verifyWebhookSignature({
    secret: process.env.COMPUTE_WEBHOOK_SECRET!,
    signature: request.headers.get('x-better-compute-signature') ?? '',
    timestamp: request.headers.get('x-better-compute-timestamp') ?? '',
    body,
  });
  if (!valid) return new Response('invalid signature', { status: 401 });

  const delivery = request.headers.get('x-better-compute-delivery')!;
  if (await alreadyHandled(delivery)) return new Response(null, { status: 204 });
  const event = JSON.parse(body);
  await enqueue(event); // do the work elsewhere: answer within 10 seconds
  await markHandled(delivery);
  return new Response(null, { status: 204 });
}

verifyWebhookSignature({ secret, signature, timestamp, body, toleranceSeconds?, now? }) returns true when any v1= value in the header matches, compared in constant time, and the timestamp is within toleranceSeconds of now (300 by default, in either direction). In another language, compute the same HMAC and compare it with each v1= part:

printf '%s.%s' "$TIMESTAMP" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex

Answer quickly, and idempotently. Any 2xx response marks the delivery delivered; its body is ignored. Anything else, a redirect (redirects are not followed), a connection error or no answer within 10 seconds is a failed attempt. Because delivery is at least once, the same event can arrive twice, after a timeout that your endpoint in fact handled or after a redelivery. Deduplicate on X-Better-Compute-Delivery (one per webhook and event) or on the event id, and order by sequence, not by arrival: a delivery that failed is retried later without holding back the ones queued after it.

Retries and dead letters

A failed attempt is retried after a delay that grows with each attempt. The eighth failed attempt makes the delivery dead.

That is about 44 hours and 36 minutes from the first attempt to the last. eventsModule({ delivery }) changes the policy for a deployment: maxAttempts (8), backoffSeconds (the table above; the last value repeats), timeoutMs (10000) and limit (50 deliveries per pass). Each failure records lastStatus (the HTTP status, when there was one) and lastError.

Inspect a webhook's history, newest first, and put deliveries back in the queue once the endpoint is fixed:

const { items } = await client.webhooks.deliveries({ tenantId: 'acme', webhookId, state: 'dead', limit: 100 });
for (const delivery of items)
  await client.webhooks.redeliver({ tenantId: 'acme', deliveryId: delivery.id });

redeliver works on a dead, cancelled or delivered delivery: it becomes pending, due now, with its attempt count reset to zero, and keeps its id and body. It refuses with NOT_FOUND when the delivery does not exist in the tenant or its webhook is paused. deliveries returns at most 200 items (50 by default).

better-compute doctor reports events.webhook-dead-letters while any delivery is dead, and events.webhook-backlog when more than 1,000 deliveries are over five minutes late, which usually means no worker runs events.deliver.

How long history is kept

events.prune deletes settled delivery history once it is old enough, measured from when the delivery was queued: delivered and cancelled deliveries after 30 days, dead ones after 90, so there is time to fix an endpoint and redeliver. It deletes 5,000 rows per run and runs again a second later while a batch comes back full. pending deliveries are never deleted, whatever their age.

eventsModule({ secret, retention: { deliveredDays: 14, deadDays: 60 } }); // retention: false keeps everything

A period under one day is INVALID_CONFIG when the module is defined. pruneWebhookDeliveries(sql, retention?, limit?) runs one batch, for hosts that schedule pruning themselves. See retention.

Test, pause and delete

• ping queues a webhook.ping event to one webhook, due now, whatever its patterns. Its data is { webhookId }. The result is the queued delivery, so you can follow it with deliveries.
• update changes url, events, description or active, with the same checks as create. Setting active: false pauses the webhook and cancels its pending deliveries; events that happen while it is paused are not queued for it. Setting active: true resumes it from new events on; redeliver cancelled deliveries you still want.
• delete removes the webhook and its whole delivery history.

Rotate the secret

rotateSecret returns a new secret, once. With keepPreviousSeconds (up to 86400, one day), the old secret keeps signing too until then, so each delivery carries two signatures, the new one first. Receivers that accept either secret during the window switch without dropping a delivery:

const { secret, webhook } = await client.webhooks.rotateSecret({
  tenantId: 'acme',
  webhookId,
  keepPreviousSeconds: 3600,
});
// webhook.previousSecretExpiresAt: when the old secret stops signing

const valid = [process.env.COMPUTE_WEBHOOK_SECRET!, process.env.COMPUTE_WEBHOOK_SECRET_OLD]
  .filter((secret): secret is string => Boolean(secret))
  .some((secret) => verifyWebhookSignature({ secret, signature, timestamp, body }));

Without keepPreviousSeconds the old secret stops at once.

At rest, signing secrets are sealed with AES-256-GCM under a key derived with HKDF-SHA256 from the deployment secret and bound to the webhook id, so a sealed value copied to another row does not open. When you rotate the deployment secret itself, pass the old one in eventsModule({ secret, previousSecrets: [old] }) so existing rows still open.

The webhooks API

All ten are decided as compute:webhooks:manage on compute-tenant/{tenantId}. A webhook of another tenant is NOT_FOUND.

A WebhookDelivery has id (dlv_…), webhookId, tenantId, eventId, eventType, state, attempts, nextAttemptAt, createdAt, and lastStatus, lastError and deliveredAt when they apply. Times are epoch milliseconds.

Webhooks or watch

Webhooks suit another service of the tenant that must see every event even while it is down: the position lives in Better Compute, and failed deliveries wait for it. For a browser, a console or a script that follows events while it runs, watch streams them over one connection and resumes from a cursor the client keeps. For a platform-wide feed into a warehouse or a queue, use an event sink.