Fine-grained token scheduling for efficient pipeline parallel LLM inference serving

Abstract

Large Language Models (LLMs) are increasingly deployed in production systems, creating a growing demand for efficient distributed inference strategies. Pipeline parallelism has emerged as a predominant approach for partitioning model computation across multiple devices. While LLM Inference remains far from hardware-effective due to the autoregressive decoding process, pipeline-parallel execution introduces further inefficiencies, as workload imbalance between pipeline stages can create idle periods known as pipeline bubbles. In this work, we closely examine the LLM inference process and existing scheduling approaches for pipeline parallel execution. Based on insights derived from this analysis, we propose a scheduling algorithm for pipeline-parallel inference designed to construct balanced microbatches, mitigate pipeline bubbles, and improve resource utilization. Experimental evaluation demonstrates that the proposed approach achieves higher throughput while maintaining lower latency compared to existing scheduling strategies.