acceptodds
Under review as a conference paper at ICLR 2027

DBO++: Rethinking Communication Overlap for Dense Tensor-Parallel Prefill

Abstract

Communication overlap can reduce prefill latency in dense language models, but its value depends on more than the input batch. We present DBO++, which extends vLLM's dual-batch execution to dense tensor-parallel prefill, and study how engine configuration changes the choice between overlapping and unsplit execution. On Qwen3-32B with four H200 GPUs, changing the configured batch-token limit while keeping the input and split fixed changes the mean effect from a 10.1% slowdown to a 3.6% latency reduction. Controlled comparisons connect this reversal to startup profiling and memory allocation. In the measured setting, live allocation increases allocator retries and imposes a penalty concentrated in the overlapping path. A targeted intervention preserves existing KV capacity while changing subsequent allocation handling; measured retries disappear and a 24.6% live-allocation penalty falls to approximately zero. Separately, a matched comparison with fixed KV allocation establishes a 2.1% latency reduction over unsplit execution. A rule using workload information and the configured limit preserves a token-only rule's positive savings while avoiding its two harmful choices in an initial held-out evaluation. Independent follow-up tests expose output mismatches and slower selections, delimiting this empirical rule's support. A broader four-model, one-output-token survey records both gains and regressions. Together, these results establish configuration-dependent overlap in the studied system and show why execution selection must be evaluated with the engine conditions and output contract under which it is used.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Serving, Efficient Inference, Communication Overlap, Tensor Parallelism

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_JiBVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper attempts to optimize vLLM's dual-batch-overlap path for dense tensor-parallel prefill. It splits a prefill batch into two microbatches and interleaves their computation and all-reduce operations. The paper observes that changing vLLM's configured batch-token limit can change startup memory allocation and reverse whether overlap is faster for an otherwise fixed batch. Based on development measurements for Qwen3-32B with TP4 on H200 GPUs, it proposes a selector that enables overlap only for 10,240-20,480 scheduled tokens and when the batch uses at most 75% of the configured token limit. A fixed-cache experiment reports a 2.1% latency reduction, while broader measurements show both gains and regressions and the selector fails to transfer reliably to additional capacities.

Critical strengths
  1. The paper carefully separates unsplit execution, serial splitting, and overlapping execution, which helps distinguish splitting overhead from the effect of communication overlap.

  2. The allocation experiments are controlled and the paper is transparent about negative results, including slower selections, output mismatches, execution failures, and online-serving regressions.

  3. The paper reports repeated measurements and confidence intervals for its central controlled experiments rather than relying only on isolated timing results.

Critical weaknesses
  1. The contribution has limited novelty and unclear significance. The main systems contribution is to extend vLLM's existing dual-batch-overlap to dense TP prefill and then choose between the original and overlapping paths. The central observation that the faster execution strategy can depend on engine configuration and memory state is expected in workload-specific system tuning. In practice, one would normally profile or autotune the engine configuration and execution path jointly for the target workload. The paper does not provide a new general scheduling principle, performance model, or broadly applicable optimization that makes this configuration dependence scientifically useful or practically reusable.

  2. The selector is an ad hoc rule that does not transfer even within the configuration dimension it uses. Its thresholds are fitted to development measurements for a single Qwen3-32B BF16, TP4, H200 setting: 10,240-20,480 tokens and an n/B≤0.75n/B \leq 0.75 condition. The allocation study does not derive these thresholds. More importantly, at two new batch-token capacities, the same rule selects three cases with changed outputs and two of the six harmful cases whose outputs remain exact. Thus, the paper's own follow-up evaluation shows that the rule is not reliable when only the value of its single configuration input changes. Other relevant configuration variables, such as memory utilization, compilation mode, communication resources, and scheduling state, are not incorporated. This substantially weakens the claimed configuration-informed selection contribution.

  3. The evaluation is too narrow to support a serving-system claim. The selector is evaluated only for pure dense prefill batches with equal scheduled tokens per request on one model, GPU type, and engine. The broader four-model grid forces the overlap path and returns only one output token. It does not demonstrate that the selector generalizes. The paper does not evaluate mixed prefill/decode traffic, continuous batching, realistic arrival traces, latency SLOs, multi-turn requests, or MoE models. These restrictions make it difficult to infer value for realistic LLM serving rather than a specific eager dense-prefill microbenchmark.

  4. The practical gains are marginal and inconsistent, and comparisons with relevant systems are missing. With cache allocation fixed, the central Qwen3-32B result is only a 2.07% latency reduction. The matched TP2 long-input experiment reports reductions of 0.85-3.13%, while the wider grid contains regressions, output mismatches, OOMs, and an illegal-memory-access failure. Most concerningly, the reported Llama TP4 online-serving configurations lose 16.79-82.74% throughput. The paper does not directly compare end-to-end performance with TokenWeave, NanoFlow, or a conventional offline autotuner. The evidence therefore does not show a practically useful improvement over standard workload-specific engine tuning.

Questions for authors
  1. What is the conceptual or practical contribution beyond jointly profiling the overlap choice and engine configuration for a target workload, as is common in serving-system tuning? Why should the batch-token limit be treated as an input to a fixed rule rather than as part of an ordinary autotuning search?

  2. Can the authors derive a selector from a performance or resource model and evaluate it without refitting across unseen batch-token limits and other engine configurations? How should a deployment trust the current rule given its slower choices and output changes at the two new capacities evaluated?

  3. What are the end-to-end throughput, TTFT, and TPOT results under continuous batching and mixed prefill/decode traffic, and how do they compare with TokenWeave, NanoFlow, and a simple profile-based autotuner under the same hardware and workload?

  4. What causes the output mismatches and illegal-memory-access failure? Can the authors establish that the overlapping path is functionally equivalent to the unsplit path before using output agreement as part of selector evaluation?

Reviewer_MP9KOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the settings in which overlapping tensor parallel computation and communication is beneficial. They find that engine configuration parameters such as the batch-token limit are significant and can alter the prefered path. Decreasing the batch-token limit leads to memory pressure, and can cause issues like allocation retries when using overlapping, which in turn makes overlapping increase latency instead of reducing it. Using these insights, they present DBO++, which choses when to overlap computation and communication, and when to continue on the unsplit (not overlapped) path.

Critical strengths
  • The authors implement effective communication-computation overlapping.
  • The authors demonstrate that adding rules that depend on engine configuration to select the computational path is an effective way to acheive lower latency.
  • The study of the effects of the token limit on latency is thorough.
Critical weaknesses
  • The paper points to a specific configuration (the batch-token limit), as critical to achieving speedup with DBO++, and the paper is centered around this finding. However, it is not clear if this is inherently problematic when using communication-computation overlapping, or if it is an implementation detail. Does this happen with other TP overlapping techniques? Other engines? The paper seems more like a study of a specific implementation and a specific engine (whose exact version is not reported), not a general result. Can the authors provide evidence that (and explain why) the finding generalizes beyond the tested setting?
  • The paper talks about an 'intervention' which is only detailed very briefly in the supplementary materials (expandable segments), but is central to the paper. Furthermore, if the intervention fixes the latency penalty with a single flag, it is not clear why the batch-token-limit rule is necessary.
  • Pipelining tensor parallel computation and communication in itself is not novel (e.g., Domino, TokenWeave, ISO). There are no comparisons to these methods.
Questions for authors

See questions in weaknesses section.

Reviewer_YjFPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper extends vLLM’s dual-batch overlap machinery to dense tensor-parallel prefill. It studies when overlap is beneficial under fixed model/hardware/input conditions. The central empirical claim is that the configured batch-token limit can reverse when overlapping works. The paper proposes a pre-split selector that uses workload shape and configuration, and evaluate it on held-out cases. The paper also provides experiments on prefill latency spanning four models and TP2/TP4.

Critical strengths
  • Implemented an efficient approach of computation-communication overlapping for tensor-parallel prefill.
  • The empirical results are positive.
Critical weaknesses
  • In a nutsell, DBO++ is a straightforward implementation of overlapping the computation and communication given two batches. The idea is not new and has been frequently studied in related work like dual batch pipelining and micro-batch chunking.
  • The paper discusses TokenWeave, FlashOverlap, Flux and others, but DBO++ is not compared with any baselines.
  • The evaluation focuses on prefill latency and there are no experiments on end-to-end serving.
Questions for authors

Please address the weaknesses above.

Reviewer_vtAyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work present extends vLLM’s dual-batch execution to dense tensor-parallel prefill. In particular, the authors develop a cost model to determine the potential compute-comms overlap, and accordingly enabling their scheme (DBO++). Modest speedups are observed.

Critical strengths
  • A new knob in vLLM for compute-communication overlap
  • Valuable analysis of the benefits
Critical weaknesses
  • Incremental solution
  • Modest speedups
Questions for authors

Dear Authors, This is a nice piece of work - and I am sure the vLLM community will benefit tremendously from this feature. From the lens of a research paper at ICLR however, I find this work to be quite incremental. Dual-batch execution and TP are known ideas. This work essentially creates a cost model to determine when to enable dense TP for prefill. I value the engineering that went into enabling this, but it lacks core research novelty.

Furthermore, the speedups from Table 2 show modest gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.