NVIDIA shares architecture for shared GPU clusters

NVIDIA has outlined a joint architecture combining the KAI Scheduler and the vCluster platform to allow multiple development teams to run fully isolated virtual Kubernetes clusters on shared GPU infrastructure. This setup addresses the common dilemma where giving each team a dedicated physical cluster wastes hardware, while forcing them to share a single standard cluster leads to configuration conflicts, overlapping role-based access controls, and version clashes. By virtualizing the control plane and utilizing hierarchical GPU scheduling, organizations can preserve team autonomy without physically splitting expensive hardware.

To demonstrate the workflow, the implementation uses a single NVIDIA Brev GPU instance on Nebius equipped with one NVIDIA L40S GPU featuring 48 GB of VRAM, 40 vCPUs, 160 GiB of RAM, and a 256 GiB disk. The software stack runs on MicroK8s version v1.36.2 with the NVIDIA GPU Operator installed in the gpu-operator-resources namespace, which can be upgraded to version 26.3.x. The setup uses snap to install kubectl from the 1.35/stable channel and deploys vCluster version 0.35.1. KAI Scheduler manages resource allocation using a custom Queue CRD to establish a hierarchy. In the example, a parent queue named ml-org is allocated a total budget of 1 GPU, which is then divided among three child queues: team-nlp, team-vision, and team-recommender. Each team is guaranteed a quota of 0.33 of the GPU but can burst to a limit of 1 when other queues are idle.

For practitioners, this architecture resolves the friction of multi-tenant GPU environments. The NLP Team can install custom CRDs, the Vision Team can access cluster-admin privileges to debug scheduling, and the Recommender System Team can run a different Kubeflow version, all without risking accidental interference. Workloads are deployed using the nvidia/cuda:12.4.0-base-ubuntu22.04 image. While KAI Scheduler handles the fair scheduling of GPU slices, it does not enforce hardware-level memory isolation during time-slicing, meaning applications must manage their own memory limits or utilize NVIDIA Multi-Instance GPU technology for physical partitioning.