What is dynamic batching in model serving, and what tradeoff does it manage?

Dynamic batching groups multiple incoming inference requests that arrive close in time into a single batch that the model processes in one forward pass. GPUs are far more efficient on batched matrix operations, so batching dramatically increases throughput and hardware utilization compared to processing requests one at a time. The tradeoff is latency versus throughput: to form a batch, the server waits a short window (e.g., a few milliseconds) for more requests to arrive, adding queuing delay to each request. A larger max batch size and longer wait window raise throughput but also tail latency. Serving frameworks like NVIDIA Triton, TorchServe, TensorFlow Serving, and vLLM (with continuous batching for LLMs) expose knobs for max batch size and max wait time so you can tune to your SLA. For LLMs, continuous or in-flight batching further improves utilization by adding and removing sequences mid-generation.