NVIDIA's Vera Rubin NVL72 system achieved leading performance in MLPerf Inference v6.1, the industry standard for measuring AI inference systems across real-world workloads. The result matters because inference—where models answer user queries after training—now dominates data center economics. Unlike training, which happens once per model, inference scales linearly with user demand. MLPerf v6.1 tests end-to-end throughput on models like Llama-70B and Nemotron, directly measuring how many tokens a system generates per second. Higher token throughput translates directly to revenue for cloud providers; a system that generates twice as many tokens per dollar captures twice as much value from the same hardware investment.
The Vera Rubin system's performance advantage stems from NVIDIA's integration of custom silicon, CUDA software optimization, and cuDNN libraries specifically tuned for inference workloads. On Llama-70B inference tasks, the NVL72 configuration achieves latencies and throughput metrics that significantly outpace alternatives from AMD and Intel. Cloud providers like Google and Lambda Labs have begun reporting cost-per-million-tokens figures favoring NVIDIA hardware; on identical inference jobs, NVIDIA-based infrastructure costs 30-40% less per token generated compared to competing solutions. This isn't marginal advantage—it's the difference between profitable and unprofitable inference services at scale. Customers running millions of inference queries daily, from LLM APIs to retrieval-augmented generation systems, feel this gap immediately in their P&L.
Enterprise procurement reflects the reality. Major hyperscalers have already committed significant capital to NVIDIA's latest architecture; AWS, Azure, and Google Cloud all launched inference-optimized instance types around Hopper and Blackwell GPUs within weeks of availability. Analyst estimates suggest NVIDIA will capture 85-90% of discrete AI inference accelerator revenue through 2025, up from 80% in 2023. This dominance persists despite AMD's MI300 series and custom silicon efforts from Meta and others. The switching cost remains enormous—moving inference workloads off CUDA requires rewriting optimization layers, retraining quantization models, and accepting performance regression during migration. No enterprise has publicly announced a large-scale pivot away from NVIDIA inference infrastructure.
Yet vulnerabilities exist. AMD's MI300X now supports larger batch sizes in certain configurations, potentially opening cost advantages for specific use cases. Google's TPU v6e targets inference-specific workloads where NVIDIA's general-purpose architecture may be overprovisioned. Custom silicon startups are targeting narrow inference problems—speech recognition, recommendation engines—where domain-specific hardware could displace GPUs entirely. For NVIDIA, the MLPerf result validates current strategy but doesn't prevent erosion at the edges. The real test comes when major hyperscalers report actual inference workload migrations or when a credible alternative achieves within 15-20% of NVIDIA performance at substantially lower cost. That threshold hasn't been crossed yet.