Auditing Run:ai on Kubernetes: GPU Quota Enforcement, Scheduler Fairness, and Exit Cost

Introduction: What to Verify in Run:ai on Kubernetes

This audit shows how to verify GPU quota enforcement, scheduler fairness, and exit cost after NVIDIA open-sourced Run:ai under Apache 2.0, based on the MarkTechPost report dated 2026-10-19 cited here. The report is the entire source material, and the useful consequence is not that "GPU scheduling is now free" — it is that a licensing conversation just ended while three operational conversations never started.

The thesis: the license removes exactly one lock-in surface. It leaves three others untouched — quota enforcement semantics, scheduler fairness under contention, and exit cost. A platform team that reads the announcement as "lock-in solved" will skip the work that actually determines whether they can run multi-tenant GPU workloads safely on this thing. A team that reads it as "procurement risk down, integration risk unchanged" runs the audit below and is right either way.

That audit is the rest of this post, written so you can run it against whatever build you end up with.

What Is Actually Confirmed, and What Is Not About Run:ai's License

The Claim and Its Single Source: The MarkTechPost Report

Confirmed by the provided source: MarkTechPost published a report dated 2026-10-19 describing NVIDIA open-sourcing Run:ai under Apache 2.0, framing it as Kubernetes GPU orchestration and noting the scheduler was previously commercial.

Thin as claims go, and it arrives through a Google News RSS redirect rather than a direct link to the article body. Treat the summary as secondhand until you check the primary trail: the NVIDIA newsroom post, the repository LICENSE file pinned to a commit, and the release notes. Do not take a repo name, tag, version range, or feature list from a news summary — including this post. I will not invent one.

You can settle the license question in under a minute once you have the URL:

git clone <repo-url> && cd <repo>
git log -1 --format='%H %ad %s' --date=short
head -5 LICENSE
git log --follow --format='%h %ad %s' --date=short -- LICENSE | tail -3

What you want to see: an Apache-2.0 header, and whether the earliest commit touching LICENSE is the first commit in the repo. A license applied at project start and one retrofitted onto an existing codebase look identical in a headline and different in a git log.

Apache 2.0 on the Scheduler Does Not Mean Apache 2.0 on the Product

Platform teams do not depend on a scheduler in isolation. They depend on a stack, and the announcement speaks to one layer of it. Honest status for each surface below, with a confirmation step rather than a verdict:

None of that is necessarily missing. The issue is that a platform team's integration surface is wider than the component that got the license.

The Three Audit Questions for Run:ai on Kubernetes

Everything below reduces to three questions you can answer before committing:

1. Does GPU quota enforcement fail closed? If the enforcement path is unavailable, do over-quota pods still schedule? If yes, quota is advisory and your multi-tenancy depends on a running pod.
2. Is scheduler fairness reproducible under contention? Run the same fixed load three times. If the distribution moves between runs, you cannot build chargeback on it.
3. What does exit cost come to in engineering hours? Count the coupled surfaces, then measure — do not estimate.

Testing GPU Quota Enforcement on Run:ai and Kubernetes

Find Where the GPU Quota Actually Lives

Two implementation families exist, and they behave very differently:

• quota expressed on a vendor project/department-style custom resource, or
• quota expressed as Kubernetes-native ResourceQuota against extended resources like nvidia.com/gpu.

kubectl get crd | grep -iE 'runai|run-ai|nvidia'
kubectl api-resources | grep -iE 'runai|project|quota|team'
kubectl get resourcequota -A

Verify the CRD names against your own install — the announcement publishes none, so any specific name you see in a blog post is a guess. Then figure out whether the quota is a spec field or a computed status field:

kubectl explain <quota-crd>.spec --recursive | head -40

A quota in status is a report. A quota in spec with an admission webhook behind it is a control. Know which one you bought.

A Two-Tenant Over-Quota Test

Set up two namespaces backed by two quota scopes with a known ceiling each. Put one greedy multi-GPU workload in tenant A and a steady, long-running workload in tenant B. Then deliberately exceed A's quota.

## tenant A already holds its full quota
kubectl -n tenant-a create deployment greedy-2 --image=<cuda-image> --replicas=4
kubectl -n tenant-a get pods -o wide
kubectl -n tenant-a get events --sort-by=.lastTimestamp | tail -20
kubectl -n tenant-b get pods -o wide   # did anything get preempted?

Three outcomes are possible, and they mean different things:

• A's second job stays Pending with a quota-related event → fail closed, queued.
• A's second job is accepted into a vendor queue object and never gets pods → fail closed, queued elsewhere; check whether the queue is visible in kubectl get pods.
• A's second job runs, or B's pods are evicted → fail open, or preemption with a policy you have not read.

Record the actual event string. "Insufficient nvidia.com/gpu" is a capacity message, not a quota message, and teams routinely confuse the two when reading kubectl describe.

Does the Quota Fail Closed?

Most teams skip this one, and it decides whether tenant isolation is a guarantee or a hope. On a lab cluster only, scale the enforcing controller to zero and try to schedule an over-quota job.

kubectl -n <vendor-namespace> scale deploy <quota-controller> --replicas=0
kubectl -n tenant-a create deployment over-quota --image=<cuda-image> --replicas=8
kubectl -n tenant-a get pods -o wide
kubectl -n tenant-a get pods -o json | jq -r '.items[] | [.metadata.name, .status.phase] | @tsv'

If those pods reach Running on real GPUs with the controller down, quota enforcement is fail-open. That is not necessarily a bug — plenty of admission-based designs are — but it changes your threat model: a crashing controller, a node drain storm, or a failed upgrade becomes a quota bypass. Restore the replicas, then check reconciliation. Does the controller kill the running over-quota pods, or only block new ones? The answer belongs in your runbook.

Scheduler Fairness Under Contention on GPU Pools

Fairness Is a Configured Policy, Not a Property

There is no abstract "fair" scheduler. Fairness is the output of weights, over-quota policy, and preemption settings applied to a specific workload mix. The audit question is therefore narrower and answerable: does the policy you configured produce the distribution you expected? If you never configured one, the answer is whatever the defaults are — find out before a research team finds out for you.

Build a Contention Test You Can Rerun

Use a fixed gang of identical jobs across two or three tenants, same node pool, same GPU type, same request size. Run it three times. Measure start-time spread and completion order per tenant — not average utilization, which hides everything that matters.

#!/usr/bin/env bash
set -euo pipefail
TENANTS=(tenant-a tenant-b tenant-c)
for run in 1 2 3; do
  for t in "${TENANTS[@]}"; do
    for i in $(seq 1 4); do
      # replace with your real GPU job manifest; keep it identical across tenants
      kubectl -n "$t" create job "probe-${run}-${i}" \
        --image=<cuda-image> --dry-run=client -o yaml | kubectl apply -f -
    done
  done
  kubectl get pods -A -o json \
    | jq -r --arg run "$run" '.items[] | select(.metadata.name|startswith("probe-\($run)-")) |
        [.metadata.name, .metadata.namespace,
         (.status.startTime // "none"), (.spec.nodeName // "unscheduled")]
        | @tsv' | sort -k3 > "run-${run}.tsv"
  for t in "${TENANTS[@]}"; do
    for i in $(seq 1 4); do
      kubectl -n "$t" wait --for=condition=complete --timeout=45m "job/probe-${run}-${i}"
    done
  done
done

I have not run this against the newly open-sourced build — the harness is the deliverable, the numbers are yours. In run-*.tsv, look for three things: whether each tenant's first start time clusters, whether the spread stays within the same order of magnitude across runs, and whether completion order is stable. If all three are stable, your policy is deterministic enough to reason about. If run 3 disagrees with runs 1 and 2, that means nondeterminism under contention — a finding worth escalating before anyone builds a chargeback model on scheduler output.

What Scheduler Misconfiguration and Bugs Look Like

Three failure modes to hunt for:

• Starvation of the low-weight tenant. The low-priority tenant never starts because higher-weight work keeps arriving. Looks fine in a short test and terrible after a week of steady submissions.
• Priority inversion around gang scheduling. If a job needs N GPUs atomically, a high-priority small job can sit behind a gang holding partial resources. Whether the open-source build implements gang scheduling at all, and how it interacts with preemption, is not settled by the public details I have.
• Granularity mismatch: whole GPU vs MIG. The quota says four GPUs, the pods each request a full device, and the MIG profile on the node cannot satisfy that combination. Quota is nominally satisfied and physically unschedulable. This is the failure mode most likely to survive a review, because every dashboard shows headroom.

Marked as inference: I would expect granularity handling to be the most common source of "the scheduler is broken" tickets after a migration, since it is a config mismatch that presents as a capacity problem.

Exit Cost — The Part the License Does Not Fix

Enumerate the Coupled Surfaces Before You Migrate

The license does not decouple any of these:

• SSO and RBAC integration, including group-to-tenant mapping
• custom resources committed into GitOps repos
• scheduler-specific fields in pod templates
• node labels, taints, and tolerations applied by the vendor stack
• dashboards and alert rules built on vendor-specific metric names
• chargeback data and the historical records your finance team has been given

Each is a migration task with a nonzero cost, and none get cheaper because the source is Apache 2.0.

A Cheap Exit-Cost Probe

Convert "how locked in are we" into a number. Stand up a plain namespace with only the NVIDIA device plugin, apply the same workload spec, and skip the vendor scheduler.

kubectl create ns plain-gpu
kubectl -n plain-gpu apply -f tenant-a-workload.yaml
kubectl -n plain-gpu get pods -o wide
kubectl -n plain-gpu describe pod <pod-name> | sed -n '/Events/,$p'

Count two categories. Hard failures: rejected fields, missing CRDs, admission denial — cheap to find and easy to estimate. Silent losses: the pod schedules and runs fine, but with no quota, no fair share, and no preemption, until a competing greedy workload shows up and takes the node. The silent ones are the expensive ones, because they do not appear in an exit-cost estimate and do appear in a post-migration incident.

Then turn it into a number for the planning meeting: count manifests referencing vendor CRDs, count dashboards, estimate days to re-run chargeback from raw metrics.

Evidence Table — What to Record During the Audit

Keep the decision rule next to the result, so a pass is not confused with an opinion. Fill the observed column from your cluster.

Where I Would Push Back on the Announcement

Three reservations, ordered by how much they should bother you.

An Apache 2.0 license is not community governance. Check who holds copyright, whether a CLA assigns rights to a single vendor, who has merge authority, who decides the roadmap. A permissively licensed project with single-vendor maintainer control is still a project where one company decides what lands. Legitimate model — just not the same thing as shared governance, and the announcement does not claim otherwise.

Open source does not ship the operational runbooks. The upgrade path, the failure modes in the previous sections, the "what do we do when the controller is down" answer — those come from your cluster, not a repo.

"Open-sourced to reduce lock-in" is an inference about motive. The reporting describes the license change. It does not establish intent. Treat motive as unconfirmed and plan against behavior you can test.

Conclusion

Treat the open-sourcing as a real reduction in procurement risk and a non-event for operational lock-in until the behavior is tested in your own clusters. The license answer is verifiable in a minute; the operational answers take a lab cluster and an afternoon.

Run these three first:

1. Scale the quota controller to zero in a lab and see whether over-quota pods schedule.
2. Run the fixed contention gang three times and compare start-time spread between runs.
3. Apply one production workload spec with only the device plugin and count the hard failures and the silent losses.

Further Reading

• Apache License 2.0 text — the actual license, for CLA and notice questions
• Kubernetes: Scheduling, Preemption and Eviction — the baseline the vendor scheduler extends
• Kubernetes: Pod Priority and Preemption — useful when reading what a vendor policy claims to do
• NVIDIA Kubernetes device plugin — the plain-namespace baseline for the exit-cost probe
• Run:ai documentation — vendor docs; check the version that matches your install, not the latest