NVIDIA Vera Rubin tackles AI memory with 30x throughput

On November 5, NVIDIA detailed how its upcoming NVIDIA Vera Rubin NVL72 platform bypasses the AI memory wall, delivering up to 30 times the agent throughput per dollar while per-GPU High Bandwidth Memory (HBM) capacity increases by only about 3%. Aimed at enterprise infrastructure teams and AI developers handling multi-million-token agent workflows, the architecture pivots away from stacking costly memory onto individual accelerators. Instead, it shifts the focus to full-system efficiency, even as persistent HBM supply shortages and rising component prices make brute-force memory expansion unsustainable.

The KV Cache Crisis in Agentic AI

In standard inference workloads, expensive GPU HBM must hold both the core model weights and the active key-value (KV) cache, which stores the conversational context required to process incoming requests. While basic chatbot interactions consume relatively predictable memory pools, autonomous AI agents continuously carry forward interaction histories across repeated model and tool calls. In demanding enterprise environments, these contextual footprints can swell to as many as 3 million tokens. The widespread adoption of retrieval-augmented generation (RAG) exacerbates this strain by injecting large corporate document batches directly into prompts, rapidly consuming high-speed GPU memory.

Escaping the Memory Wall Through Full-Stack Co-Design

NVIDIA likens HBM capacity to a storage warehouse and memory bandwidth to its loading docks: no matter how spacious the warehouse is, if data cannot move across the threshold quickly enough, the system hits a physical memory wall where extra gigabytes fail to produce extra inference throughput. To address this structural limit, NVIDIA is implementing a full-stack co-design strategy. By engineering GPUs, CPUs, networking fabric, local storage, and runtime software as a tightly coupled system, data is systematically compressed and tiered so that every byte delivers useful computing work.

In practice, this tiered memory flow balances high-speed access with lower-cost storage layers across the data center rack:

• NVFP4 low-precision precision formats compress static model weights and dynamic KV caches into substantially smaller memory footprints.
• NVL72 rack-scale configurations distribute massive model parameters across up to 72 interconnected GPUs to eliminate individual component bottlenecks.
• Vera CPU memory subsystems provide high-capacity LPDDR buffering directly adjacent to the GPU cluster for paused or background contexts.
• Dynamo inference runtime software identifies inactive context data and offloads it to CMX context storage, retrieving it before the GPU resumes execution.
• RAG indexing workloads are retained in CPU and storage domains rather than permanently occupying premium GPU HBM.

Measuring Token Efficiency Over Raw Capacity

The operational impact of this multi-tiered architecture shows up in throughput benchmarks rather than raw hardware capacity. According to data cited from the SemiAnalysis AgentX benchmark running at 125 tokens per second per user, the Vera Rubin NVL72 platform generates up to 30 times more agent throughput per dollar than the GB300 NVL72 platform, despite offering only an estimated 3% increase in per-GPU HBM capacity.

A similar architectural progression appears in earlier platform tiers: under a workload target of 100 tokens per second per user, the GB300 NVL72 delivers up to a 10-fold increase in throughput per dollar compared to the prior-generation H200 platform. These metrics indicate that competitive advantages in AI infrastructure are moving past sheer onboard VRAM size, focusing instead on how efficiently each byte and watt can be converted into active output tokens.

As autonomous AI agents demand longer context windows and tool integration, do you expect system-level tiering and compression to keep cloud inference economical, or will physical memory capacity eventually force hard trade-offs? Share your perspective on how your team plans for these scaling challenges.