Break-glass SSH

Tunnels and commands both go through the node's agent. When the agent itself is the problem (it crashed, it cannot reach its gateway, its session is stuck), a platform operator needs a way in that does not depend on it. That is break-glass SSH.

Better Compute runs an SSH certificate authority. An operator sends the public key of their own key pair to nodes.issueSshCert; the CA signs it into an OpenSSH user certificate for one Unix account that lasts minutes, and the operator connects with plain ssh. The private key never leaves their machine, no authorized_keys file changes, and every certificate is recorded.

The certificate does not travel through a gateway: you reach the node's sshd over your own network path, such as a management network or VPN.

Set up the CA

better-compute pki init writes an SSH CA into the PKI bundle, and better-compute pki show prints its public key as sshCaPublicKey. Pass the bundle's CA to betterCompute beside the exec ticket issuer; the CA is part of the tunnels options, so it needs an issuer too (see tunnels).

export const compute = betterCompute({
  // database, secret, iam, modules, pki ...
  tunnels: {
    issuer: execTicketIssuer(bundle, { issuer: 'https://compute.example.com' }),
    ssh: bundle.sshCa,
    sshRequiresMfa: true, // the default
    sshAccounts: ['root'], // the default: the Unix accounts certificates may name
    limits: { tunnelsPerMinute: 10, sshCertsPerHour: 20 }, // the defaults, per identity and api replica
  },
});

Beyond the limits a call is RATE_LIMITED with retryAfterMs. Each api replica counts its own calls in memory.

To manage the key yourself, build the CA with createSshCertificateAuthority from better-compute/pki and give every api replica the same private key:

ca.publicKey is the OpenSSH public key line (ssh-ed25519 AAAA... better-compute-ssh-ca). The tunnels module puts it into every node's NodeSpec as access.sshCaPublicKeys.

Issue a certificate

nodes.issueSshCert is decided as compute:nodes:ssh-root on the node (see the reference).

A certificate never outlives the session that asked for it: when the caller's session has an expiry, validBefore is the earlier of now plus ttlSeconds and that expiry.

export BETTER_COMPUTE_URL=https://compute.example.com
export BETTER_COMPUTE_TOKEN=...   # a session verified with a second factor
better-compute ssh-cert --tenant root --node node_01jzq3v0m6y8x7w5t4s3r2p1na
ssh -i ~/.ssh/id_ed25519 root@10.20.0.14

better-compute ssh-cert --tenant root --node node_01jzq3v0m6y8x7w5t4s3r2p1na --connect 10.20.0.14 -- journalctl -u bcd -n 50
# runs: ssh -i <key> -o CertificateFile="<key>-cert.pub" -o IdentitiesOnly=yes -l root -- 10.20.0.14 journalctl -u bcd -n 50

better-compute api nodes.issueSshCert \
  tenantId=root nodeId=node_01jzq3v0m6y8x7w5t4s3r2p1na \
  publicKey=@$HOME/.ssh/id_ed25519.pub principal=root ttlSeconds:=900 \
  --query certificate > ~/.ssh/id_ed25519-cert.pub
ssh -i ~/.ssh/id_ed25519 root@10.20.0.14

The certificate

Every certificate is an OpenSSH cert-v01 user certificate, written without external tools:

• Type. It follows your key: ssh-ed25519-cert-v01@openssh.com or ecdsa-sha2-nistp256-cert-v01@openssh.com.
• One node-bound principal. <account>@<nodeId>, such as root@node_01jzq3…, never a bare account. Each node accepts only its own principals (NodeSpec access.sshPrincipals), so a certificate for one node opens no other node that trusts the same CA. An empty principal list, which OpenSSH reads as "any account", is never issued.
• Validity. From 1 minute before issue (for clock skew) until validBefore, which the server computes on its own clock as now plus ttlSeconds, capped at the session's expiry.
• Permissions. Only permit-pty: no agent, port or X11 forwarding, and no user rc file.
• Key id. bc:<identityId>:<nodeId>:ssh_<id>, which sshd writes to its log on every login, so a log line leads back to the person and the issue record.
• Serial. Microseconds since the epoch, bumped past the last serial the CA issued.

To read one back, run ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub, or call parseSshCertificate(line) from better-compute/pki, which returns every field and whether the signature verifies (signatureValid). Check that signatureKey is your CA separately.

Audit

Each issue writes an audit_events row with the admission context and publishes node.ssh-cert.issued (subject node/<id>) with the key id, serial, principals, identity and validBefore. Route it to a webhook if every break-glass login should page someone. The certificate itself is recorded in ssh_certificates (serial, node, tenant, identity, issuing session, principal, key id, expiry) for revocation, re-authorization and audit.

Revoke a certificate

nodes.revokeSshCert({ tenantId, nodeId, serial, reason? }) ends a certificate before validBefore. It is decided as compute:nodes:ssh-root on the node, takes the decimal serial that issueSshCert returned and a reason of up to 500 characters, and answers { nodeId, serial, principal, expiresAt, revokedAt }. It is idempotent: the first revocation stands. An unknown serial, or one issued for another node or tenant, is NOT_FOUND.

A revocation invalidates the node's desired state and publishes node.ssh-cert.revoked. From the next NodeSpec on, access.sshRevokedSerials lists the node's revoked certificates that are still within their lifetime: the node's KRL, for sshd's RevokedKeys. The hourly tunnels.ssh-certificates-gc reconciler drops records a day past their expiry, in batches of 5,000.

Revocation also happens without a call, the same way (the serial joins the node's KRL through its next NodeSpec):

On a departure, serve's host runs these revocations first (b3f7a81, review 7 M3): the identity handler revokes the SSH certificates and closes the tunnels, and the agent handler drops its cached authentications, before any module reaction (the vps sandbox reactions) runs in the same inbox transaction. Each module reaction then runs under its own savepoint (host_iam_reaction), so one that throws is undone alone and cannot roll the revocation back. The identity handler reports such a failure through host.iam.onError and the IamReconcile sweep re-runs departed identities; the agent handler rethrows after undoing the reaction, so the inbox retries the event without undoing the revocations (a3be14c), since an agent stays active in IAM and the sweep would never re-run it.

The reconciler runs every minute and re-decides the 500 longest-unchecked live certificates: each one as compute:nodes:ssh-root on its node for the session that issued it (ssh_certificates.session_id), through IamPort.authorizeSession. An allow stamps reauthorized_at. This catches what Better IAM does not announce as an event: a deleted binding, an expired role activation, a changed policy. A certificate recorded without a session id, or an IAM port without authorizeSession, is not re-decided. Tenant-wide session revocations name no identity and are left to this re-decision. Each revocation publishes node.ssh-cert.revoked in the certificate's own tenant. See IAM events.

authorizeSession answers false for everything about the holder (session gone, identity gone or inactive, the decision says no), and that revokes. It counts a NOT_FOUND from Better IAM as "gone" only when it is Better IAM's own Identity not found, Tenant not found or Resource not found; a bare 404 (a proxy's error page, or Endpoint not found on a wrong base path) is IAM trouble and throws, so an IAM incident never revokes every certificate or closes every tunnel. It throws only when compute's own read fails, so the reconciler treats only two error codes as a denial: NOT_FOUND and TENANT_SUSPENDED. A thrown ACCESS_DENIED or UNAUTHENTICATED means compute's own credential was refused, not the holder: like any other error, it keeps the certificate, stamps reauthorized_at and logs Re-authorizing an SSH certificate failed; it stays valid until it expires. Otherwise an expired service token would revoke every break-glass certificate within a minute, mid-outage. A certificate whose tenant compute_tenant_states marks suspended or deleted is revoked without asking (reason tenant-suspended or tenant-deleted).

Who has access

nodes.listSshCerts answers who holds, or held, break-glass access to a node: the certificates issued for it through the tenant, newest first. It is decided as compute:nodes:read on the node, not ssh-root, because it is audit information that the node's node.ssh-cert.issued and node.ssh-cert.revoked events already show to the same readers (see the reference).

It answers { items, nextCursor? }. Each item carries serial, nodeId, identityId (who asked for it), principal, keyId, issuedAt, expiresAt and state: active, expired or revoked. A revoked item adds revokedAt, revokedBy (an identity, or iam for a revocation driven by IAM) and revokeReason. The list never returns the certificate or any key. Without includeExpired, revoked certificates that have not yet expired are still listed.

Pages are keyed on (issuedAt, serial), so a page boundary never drops or repeats a certificate. A node the tenant cannot see is NOT_FOUND; a malformed cursor is INVALID_INPUT. Like issueSshCert, the method is added to the nodes group at run time, so call it by path:

better-compute api nodes.listSshCerts tenantId=root nodeId=node_01jzq3v0m6y8x7w5t4s3r2p1na includeExpired:=true

What the node needs

The node must trust the CA for user certificates, accept only its own principals and honour the revocation list. The control plane sends all three in NodeSpec access (sshCaPublicKeys, sshPrincipals, sshRevokedSerials), and bcd's access converger (daemon/internal/agent/access.go) turns them into sshd configuration:

# Written by bcd from NodeSpec access (break-glass SSH certificates). Do not edit.
TrustedUserCAKeys /etc/ssh/bc_ca.pub
AuthorizedPrincipalsFile /etc/ssh/bc_principals/%u
RevokedKeys /etc/ssh/bc_krl

• Only this node's principals. A principal for another node, or one whose account is empty or contains /, \, . or whitespace, is never written, so a certificate for one node cannot open another that trusts the same CA. Without an AuthorizedPrincipalsFile, sshd would accept only a principal equal to the account name, which these certificates never carry.
• A native KRL. bcd encodes the KRL itself (OpenSSH's PROTOCOL.krl, a certificate-serial section per CA key, krl_version set to the spec generation). With no revoked serials it is a valid empty KRL. The Go tests check the file with ssh-keygen -Q when it is installed.
• Reloads only when needed. sshd reads the CA, principals and KRL files at each login, so a new revocation takes effect without a reload. bcd runs systemctl reload-or-restart on ssh, then sshd, only when the drop-in itself is written or removed.
• Empty removes everything. An empty or absent access section removes every file bcd wrote, so the node stops trusting the CA when the control plane stops sending it. Stale principal files are removed on every pass.

The FakeDaemon has no such converger and still reports a non-empty access section as SectionUnsupported. On a node without bcd, configure sshd the same way by hand with the sshCaPublicKey that better-compute pki show prints, and build the KRL with ssh-keygen -k. Logging in as root also needs PermitRootLogin to allow key logins; its default, prohibit-password, does.

In the console

A platform administrator sees a Break-glass SSH panel on each node's page. It takes your public key, the account (default root) and a lifetime of 5, 15, 30 or 60 minutes, and returns the certificate with a copy button and the commands to save and use it. A pasted private key is refused in the browser before anything is sent. When the session lacks MFA, it asks you to sign in again with a second factor instead of showing the error.

Under Who holds access the panel lists the node's certificates from nodes.listSshCerts (holder, principal, issued, until, state and revocation reason, with Include expired), and every active one has a Revoke button that calls nodes.revokeSshCert after a confirmation. See the console.

Sandbox certificates are separate

Sandboxes use SSH certificates too, from a second CA: tunnels.sandboxSsh (the PKI bundle's sandbox SSH CA), never tunnels.ssh. The differences:

Because the two CAs differ, a sandbox certificate never opens a node and a break-glass certificate never opens a sandbox. The tunnels module refuses a CA public key that is not one printable line, since it is written into sshd configuration, and refuses a tunnels.sandboxSsh that is the same key as tunnels.ssh (INVALID_CONFIG), so the split cannot be configured away.

Shipped and planned