NVIDIA has moved its Groq 3 LPX inference accelerator into full production, adding specialized hardware to its Vera Rubin platform as the chipmaker addresses a key performance problem facing agentic AI.
The new accelerator is built to generate tokens at high speed after an AI model has processed its input. This capability is particularly important for AI agents, which may perform hundreds or thousands of inference steps while completing a single assignment.
“Inference is diverging into a premium latency tier and a volume cost tier,” said Brendan Burke, Research Director at The Futurum Group. “LPX runs 3,400 tokens per second per user at long context against a field near 870, and it does not slow as context grows, because the compiler pre-schedules every transfer and the model lives in SRAM, not HBM.”
“That flat line is the proof it is a different product for the other half of the market. And NVIDIA is the only one now selling into both tiers.”
A benchmark conducted by Artificial Analysis found that Groq 3 LPX generated 3,400 output tokens per second while running the open source Gemma 4 31B model with a 100,000-token context window. NVIDIA says the result makes the system four times more responsive than the nearest competing platform for latency-sensitive workloads.
That performance addresses an important difference between generative AI applications and autonomous agents. A chatbot may produce one response to a prompt. In contrast, an agent can inspect files, reason through a problem, call external tools, check results and repeat those steps before delivering an answer. Small delays during each stage can add up to a lengthy wait.
Dividing AI Inference Work
Groq 3 LPX tackles the latency problem by dividing AI inference work across different types of processors. Within the Vera Rubin NVL72 architecture, Rubin GPUs handle large context processing, while Groq 3 LPX accelerators focus on the token-generation work where low latency is critical.
A rack-scale Groq 3 LPX deployment can contain as many as 256 LP30 accelerators connected with direct chip-to-chip links. The architecture allows LPUs and GPUs to operate together as an inference system rather than forcing one processor type to handle the entire workload.
The technology behind LPX stems from Groq Inc., the inference chip company founded by Jonathan Ross. NVIDIA obtained access to Groq technology in a $20 billion licensing deal in December and hired Ross and Groq President Sunny Madra as part of the agreement.
For NVIDIA, the launch advances its hardware strategy beyond the enormous compute demands involved in training AI models. As businesses deploy agents into production, inference speed becomes a core concern because it affects how long users wait for AI systems to complete work.
Nebius is among the first cloud providers planning to put Groq 3 LPX into production. The neocloud company will add the accelerator to Nebius Token Factory, its inference platform, giving developers access to the hardware without moving their applications to a different software stack.
NVIDIA is also expanding Vera Rubin into other large-scale AI environments. SpaceXAI plans to use Vera CPUs as part of its next-generation AI architecture, including infrastructure spanning Earth-based data centers and satellites. The CPUs are intended to handle work like orchestration, code execution, data processing, simulation and tool use while leaving GPUs available for model computation.
