Architecture

CosmicAC is a self-hosted platform that runs GPU workloads on your Kubernetes cluster. This page explains the components involved, how they connect to your cluster, and how each job type runs. For deployment steps, see Installation.

Deployment architecture

Setting up your cluster is separate from deploying CosmicAC. You bring a Kubernetes cluster that already has its GPU nodes and KubeVirt configured. The CosmicAC components then connect to that cluster and run your workloads on it.

wrk-server-k8s-nvidia connects to your cluster's Kubernetes API and creates the resources each job needs. For each job request it builds a graph of Kubernetes resources and applies them through the API server. For every workload it provisions a GPU VMI, a KubeVirt Virtual Machine Instance. CDI imports the VMI's root disk from a registry image, and the VMI claims one or more whole GPUs through PCI passthrough. Multi-node workloads also claim InfiniBand.

Your cluster's requirements fall into two tiers:

• Base platform — the Kubernetes, GPU, virtualization, storage, and registry foundation that every workload needs.
• Overlay networking — an optional add-on tier, needed only when a workload requires a per-instance isolated network, an OVS/VXLAN subcluster reachable through a WireGuard gateway. This capability is independent of the workload type.

CosmicAC documents the cluster requirements, not the steps to build the cluster. See Requirements.

Racks, Nodes, and GPUs

Your GPU hardware has three nested levels:

• Rack — a group of nodes registered into your CosmicAC network, in a location and with a GPU type. The orchestrator (wrk-ork) tracks it as available capacity.
• Node — a machine in your cluster that holds a pool of GPUs. Several jobs can run on the same node, each claiming different GPUs from its pool.
• GPU — a single accelerator card, such as an H100. A job claims one or more whole GPUs through passthrough.

Each job runs on a node as a Kubernetes pod that contains a VMI, and the VMI claims the job's whole GPUs. Because a node holds several GPUs, one node can host several jobs at once, each in its own pod. The GPU Container architecture diagram shows this layout on a node.

When you create a job, CosmicAC finds a rack in the requested location, picks a node with enough free GPUs, and claims whole GPUs from it for the job's VMI.

You register and manage racks with the CLI. See Manage racks in your CosmicAC network.

CosmicAC components

These components make up CosmicAC. Most run outside your cluster as part of the self-hosted platform, and the per-job agents run inside each job's VMI.

• app-ui — web interface that provides a browser dashboard for creating and managing jobs.
• cosmicac-cli — command-line interface that submits jobs, manages resources, and connects to containers from your terminal.
• app-node — application server that serves the HTTP API, authenticates requests, and routes commands to the orchestrator.
• wrk-ork — orchestrator that allocates resources, distributes jobs across the cluster, and routes requests to the workers.
• wrk-server-k8s-nvidia — Kubernetes server worker that connects to your cluster's Kubernetes API and provisions the VMIs.
• proxy-inference — inference proxy that authenticates Managed Inference requests, balances load, and routes them to model servers.
• wrk-agent-instance — GPU Container agent that runs inside a GPU Container Job's VMI and accepts shell sessions over hyperswarm-ssh.
• wrk-agent-inference — Managed Inference agent that runs inside a Managed Inference Job's VMI, serves the model with the job's runtime (vLLM or Parakeet), and registers itself in the DHT table.
• redis — in-memory data store that app-node uses for caching and runtime state, with persistence enabled.
• caddy — web entry point that serves the UI and reverse-proxies API and inference traffic on port 5173.

Caddy proxies /api to app-node and /inference to proxy-inference.

Holepunch stack

Inside CosmicAC, the components connect to each other over the Holepunch peer-to-peer (P2P) stack rather than through a central server. Components address each other directly, so there is no central broker to route, bottleneck, or expose internal traffic.

• Hyperswarm — peer-to-peer networking that lets components find and connect to each other directly, without a central broker.
• hyperswarm-rpc — RPC that carries internal calls between app-node, wrk-ork, and the workers.
• hyperswarm-ssh — an implementation that gives cosmicac-cli shell access to a running GPU Container Job over Hyperswarm.
• HRPC + Hyperswarm — bidirectional RPC that streams inference responses from the models.
• DHT table — distributed hash table where Managed Inference model servers register, and proxy-inference discovers them by topic.
• HyperDB + Autobase — distributed database that stores usage metrics and job metadata.

GPU Container architecture

A GPU Container Job runs your workload inside a KubeVirt VMI with a GPU and shell access.

How a job starts

When you submit a job from app-ui or cosmicac-cli, app-node authenticates the request and forwards it to wrk-ork, the orchestrator that places jobs. wrk-ork allocates the GPUs the job needs, choosing nodes with enough free GPUs. It then hands the job to wrk-server-k8s-nvidia, which creates the Kubernetes resources through the API. Kubernetes then creates a pod containing a VMI, with wrk-agent-instance running inside it.

How a shell connects

Once the VMI is running, cosmicac-cli connects directly to wrk-agent-instance over hyperswarm-ssh. Your commands reach the VMI over the Holepunch P2P stack rather than through app-node, so the interactive session doesn't depend on the control path that submitted the job.

Managed Inference architecture

A Managed Inference Job runs an open-source model inside a VMI, using vLLM for language models or Parakeet for speech-to-text. It exposes the model through proxy-inference as an OpenAI-compatible endpoint, which authenticates requests and balances load. You reach the model through that endpoint from any OpenAI-compatible client, or by running inference directly with cosmicac-cli.

How a job starts

When you create a Managed Inference Job from app-ui or cosmicac-cli, app-node authenticates the request and forwards it to wrk-ork. wrk-ork allocates GPUs for each replica. It then hands the job to wrk-server-k8s-nvidia, which creates the Kubernetes resources. Kubernetes runs each replica as a pod containing a VMI with wrk-agent-inference inside. As a replica spins up, wrk-agent-inference registers itself in the DHT table so proxy-inference can discover it.

How a request is served

Serving traffic follows a separate path from job creation. A client sends a request to the inference endpoint over the OpenAI-compatible API, or you run inference from cosmicac-cli. proxy-inference authenticates the request, searches the DHT table by topic to discover a model server, and balances load across the running servers. wrk-agent-inference then runs the request with the job's runtime and returns the response.

Next steps

• GPU Container Job
• Managed Inference
• Installation