NVIDIA Makes the Run:ai Kubernetes GPU Orchestration Platform Fully Open Source

One of the most important layers in AI infrastructure, now as critical as the hardware itself, is making sure graphics processors (GPUs) worth thousands of dollars are used at maximum efficiency instead of sitting idle. NVIDIA, which acquired the Israel-based Kubernetes GPU orchestration startup Run:ai some time ago, has kept the promise it made during the acquisition and released the entire platform as open source under the Apache 2.0 license.

Positioned as one of the core building blocks of the NVIDIA DSX AI Factory Platform vision, this release lets platform engineering, MLOps and DevOps teams self-host, inspect and customize enterprise-grade GPU cluster management on their own servers without any commercial subscription requirement.

Why Is Standard Kubernetes Not Enough for AI Workloads?

The traditional Kubernetes architecture was designed to manage stateless microservices and web applications. But when it comes to training, fine-tuning and inference of large language models (LLMs), the standard Kubernetes scheduler runs into serious bottlenecks:

• In the standard setup, a container (pod) locks the entire physical GPU even if it uses only a few gigabytes of memory.
• In multi-node distributed training, a job should not start until all GPUs are ready at the same time (gang scheduling), but the default scheduler can assign resources piecemeal and cause deadlocks.

The new Run:ai architecture, which NVIDIA opened on GitHub as a single monorepo, solves these problems at the root with a control plane, cluster services, an installer, a CLI, a REST API and a built-in MCP (Model Context Protocol) server that lets AI assistants manage the cluster.

The 4 Modular Components of the Open-Source Run:ai Platform

NVIDIA built the architecture on 4 main modular components so that developers can use the system either as an end-to-end integrated platform or by adding only the pieces they need to their existing infrastructure:

1. KAI Scheduler (Kubernetes AI Scheduler)

KAI Scheduler, whose foundations were laid earlier as a CNCF Sandbox project, is a Kubernetes-native scheduling engine optimized from scratch for AI workloads. It has advanced capabilities built right into its core, such as gang scheduling, time-based fair-share, network topology-aware placement (taking NVLink and InfiniBand proximity into account), bin-packing and Dynamic Resource Allocation (DRA).

2. GPU Fractioning Operator (GPU Splitting With CUDA 13.4 Support)

The GPU Fractioning Operator, one of the platform's most notable new features, lets a single physical GPU be shared safely among multiple workloads. Instead of the custom abstraction layers in the old closed-source versions, it now uses GPU splitting technology directly at the native CUDA 13.4 driver level. This makes it possible to enforce hard memory limits for each workload while exporting precise hardware-level telemetry data.

3. KAI Resource Management (Enterprise Resource and Quota Management)

In large organizations, having different departments, research groups and projects share the same GPU pool requires complex authorization. This component lets you define hierarchical quotas across projects, departments and node pools. When a team is not using its guaranteed GPU quota, other teams can temporarily borrow that idle capacity (over-quota borrowing), and when the original owner submits a workload, the resources are reclaimed with priority (preemption).

4. Workload Map (Workload Abstraction Layer)

The AI ecosystem uses many different custom resource definitions (CRDs), such as PyTorchJob, RayCluster, KServe, MPIJob or the standard Deployment. Instead of writing rules hard-coded for each framework, the Workload Map component offers a flexible abstraction layer. So when a new open-source training or inference tool comes out, it can be quickly registered in the system without changing the scheduler core.

Testing on a Local Machine Without a Physical GPU

NVIDIA also included a practical developer environment (local-dev.sh) in the repo, so that developers and infrastructure architects do not need expensive GPU servers to test the system.

This script, which can run on a standard laptop with Linux or macOS, spins up a local Kubernetes (Kind) cluster and installs a fake GPU operator in it. The system simulates a virtual environment with 8 NVIDIA A10 GPUs, letting you experience GPU splitting, quota borrowing and queue management scenarios end to end without physical hardware.

Fully freed under the Apache 2.0 license, Run:ai looks set to become the strongest open standard in the Kubernetes ecosystem for companies that want to build cloud-agnostic AI infrastructure.