SambaNovaAI reached 850 tokens/second on MiniMax M2.7 at RAISE Paris. The system pairs Nvidia H200s for prefill with SambaNova's own SN50 RDUs for decode.
Key facts
- 850 t/s on short context for MiniMax M2.7.
- 450+ t/s on long context at RAISE Paris.
- Hybrid: H200s for prefill, SN50 RDUs for decode.
- MiniMax M2.7 is a 270B-parameter MoE model.
- SN50 RDUs have 1.5 TB/s memory bandwidth.
SambaNovaAI and MiniMax released inference benchmarks at the RAISE conference in Paris, claiming 850 tokens per second on short-context workloads and 450+ tokens per second on long-context workloads for MiniMax's M2.7 model. According to @MiniMax_AI
The architecture splits the inference pipeline: Nvidia H200 GPUs handle the compute-heavy prefill phase, where the model processes the entire input prompt in parallel. SambaNova's custom SN50 Reconfigurable Dataflow Units then take over for the memory-bandwidth-limited decode phase, where autoregressive token generation stalls on memory reads.
This hybrid approach is unusual. Most inference deployments use homogeneous hardware — either all GPUs or all custom ASICs. By partitioning the workload by phase, SambaNova claims to optimise each stage on its preferred hardware: H200s for matrix-multiply-heavy prefill, SN50s for the memory-bound decode loop where RDUs' dataflow architecture can minimise data movement.
Why the split matters
The prefill-decode divide is well-known in LLM serving: prefill is compute-bound (large batch, high arithmetic intensity), while decode is memory-bandwidth-bound (small batch, low arithmetic intensity). Standard GPU clusters waste compute capacity during decode. SambaNova's RDUs, designed for dataflow execution with on-chip SRAM, may offer better memory bandwidth utilisation for the autoregressive step.
850 t/s on short context is fast — comparable to or exceeding published numbers from Groq's LPUs on similar-sized models, though direct comparisons are difficult without standardised benchmarks. [Per the announcement], the system sustained 450+ t/s on long context, where the attention mechanism's quadratic complexity typically slows prefill.
The company did not disclose whether these speeds include batch size, precision (FP8 vs FP16), or quantization details. Nor did it specify the exact hardware configuration — number of H200s vs SN50s, interconnect topology, or power draw.
Competitive context
Groq's LPU inference engine has claimed 500+ t/s on Llama 2 70B. Cerebras's Wafer-Scale Engine has demonstrated 1,800+ t/s on smaller models. SambaNova's hybrid approach differentiates by using off-the-shelf GPUs for the phase where GPUs excel, while reserving its own silicon for the phase where custom dataflow matters most. This could lower total cost of ownership compared to a fully custom system, assuming the H200s are commodity-priced.
MiniMax M2.7 is a 270-billion-parameter mixture-of-experts model. The company did not disclose the MoE routing overhead or whether the benchmark included expert parallelism. SambaNova's SN50 RDUs have 1.5 TB/s memory bandwidth and 1.2 petaops of compute, [per SambaNova's published specs], though the exact configuration used at RAISE was not detailed.
What to watch
Watch for SambaNova to publish reproducible benchmarks with batch size, precision, and power draw details. The company's next hardware generation and whether it integrates prefill acceleration natively into the RDU will determine if the hybrid approach is a bridge or a permanent architecture.
