SemiAnalysis is testing TileRT InferenceX, software claiming ultra-high interactivity on NVIDIA GPUs at batch size 1. The benchmark pits it against Cerebras, Groq LPU, and SambaNova, all custom-silicon vendors.
Key facts
- SemiAnalysis testing TileRT InferenceX on NVIDIA GPUs
- Batch size 1 inference target
- Competitors: Cerebras, Groq LPU, SambaNova
- Disaggregated prefill and decode engines
- No benchmark results disclosed
SemiAnalysis, the semiconductor research firm, is evaluating TileRT InferenceX, a software stack that claims to deliver ultra-high interactivity on NVIDIA GPUs. The test targets batch size 1 inference, the regime where latency-sensitive AI applications like real-time chatbots and agentic systems live. According to @SemiAnalysis_, the software uses a disaggregated engine: a high-throughput prefill engine and a high-interactivity decode engine.
The competitive field is stark. Cerebras, Groq LPU, and SambaNova all sell custom ASICs designed specifically to minimize per-token latency at batch 1. NVIDIA GPUs, by contrast, are optimized for throughput at high batch sizes. The claim that software alone can close that gap is a bold one—and one that flies in the face of years of hardware-centric design in the low-latency inference niche.
Why software on NVIDIA GPUs could matter
The stakes are structural. If TileRT InferenceX can match ASIC latency on commodity NVIDIA hardware, it would undercut the economic rationale for custom silicon. Data centers already run NVIDIA GPUs; adding a software layer is far cheaper than deploying new hardware. SemiAnalysis's interest is notable because the firm has deep ties to the AI supply chain—its coverage often moves markets.
But the source is thin. The tweet provides no benchmark numbers, no latency figures, no token-per-second measurements. The company did not disclose results, leaving the performance claims unverified. The tweet is a teaser, not a paper.
The technical challenge
The core problem is memory bandwidth and scheduling. ASICs like Groq LPU use static scheduling and on-chip SRAM to eliminate the overhead of dynamic batching and memory hierarchy. GPUs rely on CUDA cores and HBM, which introduces latency at batch 1. TileRT would need to bypass the standard CUDA execution model—likely via custom kernels and kernel fusion—to approach ASIC-level determinism.
The disaggregated prefill/decode split is a known technique, used by vLLM and others, but it typically targets throughput, not latency. Making decode interactive at batch 1 on a GPU is a different beast. Whether TileRT has solved that is unproven.
What to watch
SemiAnalysis has not published results. Watch for a follow-up report with concrete latency numbers—specifically time-to-first-token and inter-token latency at batch 1 on a specific NVIDIA GPU (e.g., H100 or B200). If the numbers beat or match Groq LPU's ~100 tokens/second per user, that would be a seismic shift. If not, this is another software vaporware claim.
Key Takeaways
- SemiAnalysis is testing TileRT InferenceX, software claiming batch-1 ultra-high interactivity on NVIDIA GPUs, targeting Cerebras, Groq LPU, and SambaNova.
- No benchmarks disclosed yet.
