- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduce a technique for reducing dp imbalance problem in real world DP-EP deployment. Particularly when sequence length is heterogeneous. Since EP requires an all-reduce, the imbalance in dp can cause exessive Given that the dp workload can be redistributed between ranks, the authors propose to chunk DP workload into two chunks, keep chunk 0 in place while sending chunk 1 other ranks to reduce imbalance. Theorem 1 shows that pairing ascending chunk-0 loads with descending chunk-1 loads is optimal. Experiments is performed on Qwen3-30B-A3B, Moonlight-16B-A3B and Qwen3-235B-A22B, with a single 8×A800 node with synthetic log-normal request lengths. The authors report mean-TTFT reductions of up to 40%.
Critical strengths
- The paper studies a practical and clearly defined problem: with DP-EP deployment, there can be imbalances between dp ranks with existing inference frameworks such as vllm.
- the method is very simple and should be easy to implement in practice
- the method can be combined with request scheduling and TP/PP to achieve further improvements
Critical weaknesses
-
The author seem to pad longbench text to a synthetic length distribution. I am not sure if this is valid experiment, as this could potentially distort MOE routing and skew the benchmark results. I also don't know if the padded tokens are attention masked or not. If it's attention masked, then in layers where dp imbalance matters, the workload isn't really following the requested distribution
-
Generalization across workload. The method is validated on completely synthetic distribution based off longbench, with synthetic log-normal length(and padded to that) with no arrival process, and requests of batch uniformly sampled from [d,2d] occurs. While synthetic experiments are good for analysis, the paper requires real experiments to be more convincing. One obvious target might be longbench but without padding, with poisson arrival process. The author could also benchmark with public production traces and length and arrival time to analyze how much DP imbalance is a problem in real world, and how much improvement the method could bring in more realistic scenarios. (e.g. Mooncake FAST'25 traces, ServeGen, Azure LLM Inference Trace, BurstGPT)
-
Adding P50/P99 TTFT and throughput would be more informative than just "Mean TTFT per execution step".
-
The method is much less effective for modern llms (e.g. qwen3-next) with hybrid/linear attention. The state required for chunk-1 is only available after the owner have processed the r10 part of chunk-0 rather than just K/V, which introduces block waiting. This is prominent when r10 is long. The author also didn't discuss how the method applies to llms with linear attention.
Questions for authors
- Does the W/O chunkBlanace baseline use the same two-chunk pipelining, so only lack of redistribution is only difference?
- Is the padded tokens attended or masked in attention?
- With prefix caching, is the kv of chunk-1 tokens computed on other ranks returned to owner or they are kept on other ranks? This matters as this could have effects on either total communication volume or efficiency of kv cache.
7 comments
The method is much less effective for modern llms (e.g. qwen3-next) with hybrid/linear attention. The state required for chunk-1 is only available after the owner have processed the r10 part of chunk-0 rather than just K/V, which introduces block waiting. This is prominent when r10 is long. The author also didn't discuss how the method applies to llms with linear attention.
ChunkBalance also applies to hybrid and linear attention, theoretically and empirically.
Linear attention commonly uses an affine associative recurrence. Formally, the state update can be written as . Each token therefore defines an affine transformation
For two tokens, composing their transformations gives
Thus, the two-token chunk's local transformation is
\qquad \bar{B} = A_{t+2} \odot B_{t+1} + B_{t+2},$$ and its output state can be written as $$S_{\mathrm{out}} = \bar{A} \odot S_{\mathrm{in}} + \bar{B}.$$ Similarly, for a chunk of $n$ tokens, composing the transformations gives $$f_{t+n} \circ \cdots \circ f_{t+1}(S_{\mathrm{in}}) = \bar{A} \odot S_{\mathrm{in}} + \bar{B}.$$ The chunk can compute $\bar{A}$ and $\bar{B}$ from its own tokens **without knowing the incoming state $S_{\mathrm{in}}$**. Once the initial state arrives from the preceding chunk, it applies that state to the precomputed transformation: $$S_{\mathrm{out}} = \bar{A} \odot S_{\mathrm{in}} + \bar{B}.$$ Precomputing the chunk's local transformation takes $O(n)$ work and can proceed **without waiting for the preceding chunk transferring $S_{\mathrm{in}}$.** Applying the incoming state takes $O(1)$ work. ChunkBalance redistributes the dominant $O(n)$ workload to effectively mitigate DP imbalance. The $O(1)$ work of state updating does require to be sequential, while the transfer process starts when chunk 0's preprocess ends, and it can therefore be fully hidden behind chunk 1's preprocess. Thus, the required state is ready when chunk 1 starts state updating (even if not, the overhead is never dominant). We validated the empirical effectiveness of ChunkBalance on Qwen3.5-35B-A3B, a hybrid model with three linear-attention layers for every full-attention layer. | Trace / arrival | P50 TTFT(s) w/o CB | P50 TTFT(s) w/ CB | P90 TTFT(s) w/o CB | P90 TTFT(s) w/ CB | |---|---|---|---|---| | conversation | 1.87 | 1.67 | 3.93 | 3.72 | | toolagent | 2.08 | 1.68 | 4.47 | 3.28 | | synthetic (poisson) | 2.59 | 1.66 | 4.38 | 3.19 | We appreciate your thoughtful and insightful review, which greatly helps us in clarifying and improving our paper. Thank you again!Thank you for the detailed response and for running the additional experiments on the Mooncake FAST'25 traces and the hybrid-attention model (Qwen3.5-35B-A3B). These results, together with the clarifications on the baseline configuration and padding, address most of my initial concerns. Could the authors address two remaining items:
-
Interaction with prefix caching (Q3) and Mooncake replay setup. The response explains where newly com puted KV blocks reside across chunks, but does not address prefix caching itself (e.g., SGLang's RadixAttention). Multi-turn conversation and toolagent workloads in the Mooncake trace rely heavily on prefix KV reuse (hash_ids). Was prefix caching enabled in your Mooncake experiments? More generally, when a request is routed to an owner rank because it hits a cached prefix there, does migrating its second chunk to another DP rank require transferring that cached prefix KV across ranks as well (increasing communication volume), or leave the newly appended KV relocated away from the prefix holder for subsequent turns? Please also clarify the DP request-routing policy and arrival-rate scaling used in the Mooncake replay.
-
Revised manuscript PDF. All of the new experiments (Mooncake traces, hybrid/linear attention, additional routing policies, and k > 2 chunks), the per-rank compute/communication stream schedule, and the broken appendix reference/caption fixes currently exist only in the rebuttal comments. Please upload an updated PDF incorporating these additions and clarifications.
We sincerely thank you for the review!
Yes. The only difference between w/ and w/o ChunkBalance is whether the chunks are redistributed (σ∗ vs. the identity σ(i)=i). All other experimental configurations are identical, so the measured gap is entirely attributable to redistribution.
The padded tokens are attended (not masked), so the measured cost reflects the workload at the target length. As you stated, padding affects end-to-end latency because padded tokens are also routed to experts, changing the routing distribution from that of the original sequence. However, the DP workload, which is where the performance gain comes from, remains a faithful function of the sequence length. Since the baseline uses the same padding and experiences the same skew, the comparison is still fair.
KV that is not involved in redistribution is stored as usual. For KV transferred during redistribution, ChunkBalance supports two storage strategies; neither adds communication volume or memory use.
Store the full sequence on the last-chunk rank. The final chunk’s attention requires the request’s full KV prefix, so the KV ends up on the rank that owns the last chunk. Other ranks that participated in computing the request discard their KV after transferring it.
Store each chunk’s KV on its assigned rank. The KV of preceding chunks at layer i will temporarily reside in the current chunk's DP rank for attention computation, and discarded after proceeding to layer i+1. This policy is intended for prefill-decode disaggregation, a common technique in LLM serving. The distributed KV is later transferred to the decode instance to form the complete sequence.
ChunkBalance targets acceleration within the model, so our original experiments used synthetic log-normal lengths to isolate this effect from request scheduling and to control the mean and variance when quantifying DP imbalance. ChunkBalance is orthogonal to request-routing mechanisms, and the two can provide additive benefits in serving. TTFT percentiles reflect both factors, so scheduling can influence the measured effect of ChunkBalance.
We agree that serving experiments help clarify the method’s practical impact. Following your suggestion, we evaluated ChunkBalance on the Mooncake FAST’25 production trace, which includes two real-world workload profiles (conversation and toolagent) as well as a synthetic trace with a Poisson arrival process. We report P50 and P90 TTFT below. The results show that DP imbalance is significant in these workloads and that ChunkBalance mitigates it.
Moonlight-16B
Qwen3-30B