Kubernetes fleet pools

A Kubernetes fleet pool is a GPU pool whose machines are the nodes of a managed Kubernetes cluster. The pool's policy.substrate names one of the owner's provider accounts and the node shapes it wants (flavors). The kubernetes.substrates reconciler then creates the cluster and its node pools at the provider, or adopts a cluster that already exists. It runs bcd on the nodes as a DaemonSet, and those nodes join the pool. From that point compute books their GPUs like any other pool's, with binding: 'kubernetes' reservations that bcd's device plugin hands to the kubelet.

The provider runs the control plane. better-compute never runs one; it orchestrates GKE, EKS or DOKS through ProviderImpl.kubernetes. The pool substrate is provider-neutral: nothing in it is specific to Google, AWS or DigitalOcean. Only the provider account changes, along with the names that account uses for a region, a machine type and an accelerator.

Turn it on

The substrate needs the vps and kubernetes modules, built with the same provider definitions, and an agent that says what to run on the cluster. Add fleetModule() to manage the pool as FleetConfig.

import { betterCompute } from 'better-compute/server';
import { vps } from 'better-compute/vps';
import { fleetModule } from 'better-compute/fleet';
import { kubernetes } from '@better-compute/kubernetes';
import { kubernetesSubstrateAgent } from '@better-compute/provider-kubernetes';
import { awsProvider } from '@better-compute/provider-aws';
import { digitaloceanProvider } from '@better-compute/provider-digitalocean';
import { gcpProvider } from '@better-compute/provider-gcp';

const providers = [gcpProvider(), awsProvider(), digitaloceanProvider()];

export const compute = betterCompute({
  // database, secret, baseURL, iam, ...
  modules: [
    vps({ providers }),
    kubernetes({
      providers,
      // What runs bcd on the cluster: the DaemonSet options, except what the pool decides.
      substrate: { agent: kubernetesSubstrateAgent({ image, gatewayUrl, pins, controlPlaneUrl }) },
    }),
    fleetModule(),
  ],
});

Without substrate, pools that declare policy.substrate stay in error, and selfCheck reports kubernetes.substrate-agent-missing.

One document, three providers

The pool below asks for an 8 x H100 node pool of two nodes. Only providerAccounts, and what that account calls a region, a machine type and an accelerator, differ between providers.

{
  "version": 1,
  "sites": [{ "name": "k8s", "kind": "shared", "region": "us-east-1" }],
  "providerAccounts": [
    {
      "name": "managed-k8s",
      "provider": "aws",
      "config": {
        "region": "us-east-1",
        "zones": ["us-east-1a", "us-east-1b"],
        "eksClusterRoleArn": "arn:aws:iam::111111111111:role/eks-cluster",
        "eksNodeRoleArn": "arn:aws:iam::111111111111:role/eks-node",
        "eksSubnetIds": ["subnet-0000000000000000a", "subnet-0000000000000000b"]
      }
    }
  ],
  "pools": [
    {
      "name": "gpu-fleet",
      "site": "k8s",
      "class": "metal-host",
      "gpuModel": "H100",
      "policy": {
        "kubernetes": { "platformOperated": true },
        "substrate": {
          "kind": "kubernetes",
          "account": "managed-k8s",
          "cluster": { "create": { "location": "us-east-1" } },
          "flavors": [
            { "name": "h100", "machineType": "p5.48xlarge", "nodes": 2, "gpus": { "type": "nvidia-h100", "count": 8 } }
          ]
        }
      }
    }
  ]
}

{ "name": "managed-k8s", "provider": "gcp", "config": { "projectId": "my-project", "zones": ["us-central1-a"] } }

Apply it as any FleetConfig document (see Fleet configuration):

better-compute fleet plan fleet.json
better-compute fleet apply fleet.json

The account and the cluster's place (location, private nodes, or the adopted cluster's id) are the substrate's identity: a plan that changes them is blocked, so create a new pool instead. Sizing changes in place.

What happens

1. Cluster. The substrate records an ordinary kubernetesClusters cluster (substratePoolId set) with one node pool per flavor. Each node pool is labelled better-compute.dev/substrate-pool=<pool id>, and a Spot flavor's is also labelled better-compute.dev/capacity=spot. The clusters reconciler creates them at the provider.
2. bcd. Once the cluster is ready, each run mints a 600 s admin credential through the provider (ProviderKubernetes.credentials), lists the substrate's Nodes and server-side applies bcd: a Namespace, a ConfigMap and a DaemonSet that selects exactly the labelled Nodes. The credential is held in memory for that run only. It is never stored, sealed, logged or published.
3. Join. While Nodes are still to join, a hosted kubernetes join token sits in a Secret, with one use per Node whose bcd is not connected. bcd's join container reads it, and each node joins the pool. Once every Node has joined, the token is revoked and the Secret deleted.
4. Book. Reservations with binding: 'kubernetes' book the nodes' GPUs. bcd's device plugin advertises only the devices a reservation holds. The nodes of a Spot flavor carry the system label bc.dev/capacity=spot once they are matched to their Node: only reservations made with spot: true are placed on them, so nothing sold as uninterruptible sits on a machine the provider may reclaim.
5. Scale. Change a flavor's nodes in the document and apply. The provider removes Nodes. Each pool node is matched to its Kubernetes Node by the Node name bcd reports: the pod's spec.nodeName, in its inventory's os section (kubernetesNodeName), or the hostname from a bcd that does not report one yet. When a matched node is disconnected and its Node is gone, its reservations end: bound ones are released with their quota refunded once, and spot reservations and anything on a Spot node end as preempted. Then the node is retired.
6. Teardown. Remove the pool from the document and apply with prune, or archive the pool. Taking only policy.substrate out of a pool that stays tears the substrate down the same way, and also needs prune: a plain apply keeps the substrate. The token is revoked and the nodes are cordoned. The substrate waits for their bookings to end; nothing is force-released. Then it removes bcd from the cluster (only objects carrying its label), deletes a created cluster and its node pools, and retires the nodes. Nothing is left at the provider or in compute.

Read the state with kubernetesSubstrates.get({ tenantId, poolId }): mode, cluster, state and message, Nodes listed and joined, and a drain's upgrade. It never returns a token or a credential.

A substrate is ready once bcd is applied, whether or not its Nodes have joined yet. waitingToJoin says how many listed Nodes have no joined bcd and since when none has joined. If that lasts 15 minutes, the message says so and names bcd's DaemonSet: look at its pods (the image pull, the gateway URL and pins given to the substrate agent, the GPU runtime class). doctor reports the same as kubernetes.substrate-nodes-not-joining.

An error does not stop the nodes that already joined from being looked after. When the provider refuses one flavor's node pool (quota, a stockout), or a change to the document cannot be made in place, the substrate stays in error with the reason as its message, and each run still lists the Nodes, retires the ones that are gone, keeps a join token for new Nodes and re-applies bcd. It only stops at a cluster whose control plane never answered or is being deleted.

Adopt a cluster you already run

Use "cluster": { "adopt": "<provider cluster id>" } instead of create. The cluster is read and never created, changed or deleted. Each flavor adds a node pool named bc-<last 6 of the pool id>-<flavor>, so it can never take over a node pool of your own with the same name. On teardown only those node pools and bcd's labelled objects go; the cluster and its own node pools stay. Last, the substrate revokes its own access (revokeCredentials): the bc-substrate service account in bc-access, its cluster-admin binding, and the bc-access namespace once no subject is left in it (only if better-compute made it). GKE, EKS and DOKS all implement this; the cluster-wide better-compute:edit ClusterRoles and the access-guard admission policy stay, since they grant nothing on their own (an EKS access entry the tenant made stays too). If bcd or that access cannot be removed from an adopted cluster (a refused credential, a failed delete), the substrate stays deleting and retries every 2 minutes. It never leaves a privileged DaemonSet or a cluster-admin identity behind unannounced. For the same reason kubernetesClusters.credentials refuses an adopted cluster (INVALID_STATE): a kubeconfig's service account and binding would outlive the substrate on a cluster better-compute no longer manages. Reach your own cluster with the provider's credentials.

One cluster backs one pool, whichever account or deployment reaches it. Three checks hold that:

• A second pool that names the same account and cluster id is refused when its substrate is recorded.
• Before any node pool is made, the cluster's provider-global id (the GKE selfLink, the EKS ARN, the DOKS UUID) is recorded as what the pool claims. A second pool that reaches the same cluster through another provider account is refused there: its cluster goes to error, and nothing is created in the cluster.
• The substrate then claims the cluster itself with a ConfigMap, kube-system/better-compute-substrate, labelled with the pool id. It is created, never applied over an existing one, so another better-compute deployment that adopts the same cluster is refused before it adds node pools or applies bcd. Teardown deletes the claim, and a substrate that never held it removes nothing and does not revoke the bc-substrate identity.

If the provider refuses to delete a created cluster (it is in use, the account lost a permission), the cluster and its substrate go to error with the provider's message and the delete is retried with a growing wait: 5 minutes, then 10, 20, 40 and so on, at most every 6 hours. Fix the cause and the next retry finishes the teardown.

To move a cluster to another pool, archive the first pool and wait for its substrate to be deleted. Delete the ConfigMap by hand only for a cluster whose deployment is gone for good.

Provider differences

Everything above behaves the same on each provider. These are the places where providers differ:

To get the same node count everywhere, pin each flavor to one zone on GKE and EKS. Otherwise, size each flavor with its account's zone count in mind.

Check your own provider

A driver should report the cluster's provider-global id as outputs.globalId in the state ensureCluster and getCluster return: the same string whichever account reads the cluster. Without it, adoption is only checked per provider account and by the in-cluster claim.

@better-compute/testing exports kubernetesProviderConformanceCases(): what the clusters and substrate reconcilers need of any ProviderImpl.kubernetes, as cases shaped like vps' providerConformanceCases(). GKE, EKS and DOKS pass them against their emulators. The subject gives your provider's vocabulary, and the cases do the rest. They create a cluster idempotently and check its ready outputs (endpoint, version, location, nodeVersion). They check that the mode never changes in place, and that an upgrade happens in place. They add a labelled GPU node pool and a Spot one, resize them in place, and check that a node pool never changes shape. They mint credentials for the cluster endpoint, delete twice, and read and extend an adoptable cluster.

import { runConformance } from '@better-compute/core/conformance';
import { kubernetesProviderConformanceCases } from '@better-compute/testing';

const report = await runConformance(kubernetesProviderConformanceCases(), () => ({
  provider: myProvider({ fetch: emulator.fetch }),
  config: { region: 'us-east-1' },
  location: 'us-east-1',
  nodePool: { machineType: 'm7i.large', nodeCount: 1 },
  gpuNodePool: { machineType: 'g6.xlarge', nodeCount: 1, accelerators: { type: 'nvidia-l4', count: 1 } },
  accepts: (credentials) => emulator.issued.includes(credentials.token),
}));

Set zeroNodes: false if a fixed-size node pool needs at least one node; nodeCount: 0 must then be refused as INVALID_INPUT. Set spot: false if the provider has no Spot node pools.