NVIDIA's Groq 3 LPX packs 128GB of SRAM per rack to slash inference latency. The chip targets agentic AI, where hundreds of sequential steps make every millisecond compound.
Key facts
- 128GB of SRAM across the rack
- Deterministic compiler scheduling
- Targets millisecond-level latency for agents
- Hundreds of sequential inference steps per task
Agentic AI is not chat. A single task can trigger hundreds of sequential inference calls, and each decoding step adds latency that accumulates across the workflow. According to @rohanpaul_ai, the WSJ frames this as two distinct computing challenges: processing enormous context and generating tokens with extremely low latency.
Groq 3 LPX is NVIDIA's answer. The chip uses deterministic compiler scheduling to eliminate runtime variability, 128GB of SRAM across the rack to keep weights and activations local, and preplanned chip-to-chip transfers to reduce small-batch coordination overhead. These are latency optimizations aimed squarely at agentic workloads, not just raw throughput.
Why latency beats throughput for agents
Ordinary chat tolerates a few hundred milliseconds of decode time. An agent that runs 500 inference steps does not. At 50ms per step, that's 25 seconds of pure decoding — before any tool calls or context re-processing. Groq 3 LPX's millisecond-level cuts are designed to keep that total under a threshold where agents feel responsive.
The design choice is notable: instead of chasing higher batch throughput, Groq 3 LPX prioritizes deterministic, low-latency single-request paths. That's a deliberate trade against the GPU cluster norm, where utilization often trumps per-request latency. For agentic loops, the latter matters more.
The SRAM bet
128GB of SRAM across the rack is the headline number. It means the working set for a large model can stay on-chip, avoiding DRAM round-trips that dominate latency in conventional systems. Combined with compiler-driven scheduling, the chip avoids the jitter that plagues shared infrastructure.
The company did not disclose power draw or pricing. Benchmarks against competing inference accelerators are also absent from the source. What's clear is the architectural direction: latency is the new battleground for agentic AI infrastructure.
What to watch
Watch for independent benchmarks comparing Groq 3 LPX latency against Groq's LPU and NVIDIA's own H100 in multi-step agent workflows. Also track whether hyperscalers adopt the chip for agentic serving, and any disclosed power or pricing figures that affect deployment economics.
