Technique · inference

Continuous Batching

A scheduling technique that adds/removes requests at the iteration level rather than the batch level, dramatically increasing throughput for LLM serving.