- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Gated All-Reduce, a post-training modification to tensor-parallel transformers that learns a fixed-budget subset of synchronization sites to retain. Skipped sites accumulate device-local contributions in a pending residual stream, which is merged at the next retained all-reduce. Gate selection uses hard top-k with a rectangular straight-through estimator; KL distillation first learns gates with frozen weights and then jointly trains gates and LoRA adapters with a stabilizing regularizer. A MoE variant maintains consistent router inputs while allowing local expert inputs. On Llama-3.1-70B and Qwen2-57B, approximately 5.2M training tokens recover much of the average accuracy on seven downstream benchmarks. At 50% skipping, the reported retained average accuracy is 98.7% and 98.2%, respectively, with vLLM speedups on eight A100 GPUs of 12.23%/10.84% for Llama prefill/decode and 2.84%/1.65% for Qwen.
Critical strengths
-
The pending-stream formulation is a useful architectural contribution: it defers synchronization while eventually incorporating every local update, and permits skipping both attention and feed-forward synchronization sites in dense models. Compared with the reported SPD reimplementation at 50% skipping, recovered Llama average accuracy improves from 76.00 to 79.57 (base 80.59). This is meaningful evidence that aggressive communication pruning can work better than restricting pruning to attention sites.
-
The work connects the algorithm to measured inference latency in vLLM, reports absolute timings and variability, and estimates the overhead against an all-reduce-removal ceiling. The roughly 11–12% Llama speedups at 50% skipping are practically interesting. The STE and recovery ablations also make the optimization choices more interpretable than an unablated pruning recipe.
Critical weaknesses
-
The quality–latency tradeoff is not yet sufficiently established for this lossy architectural modification. Although pending updates are eventually merged, intervening nonlinear layers operate on device-local rather than fully synchronized residuals; later synchronization does not restore the original model computation. At 50% skipping, recovered Llama-3.1-70B retains 98.7% of average downstream accuracy, but WikiText perplexity increases from 3.43 to 5.69. Qwen2-57B MMLU decreases from 74.50 to 69.16 despite 98.2% retained average accuracy, while its decode speedup is only 1.65%. The paper reports these regressions transparently, but the seven-task average is insufficient to establish that broader capabilities are nearly preserved. In particular, the evaluation does not establish preservation of long-context generation, coding, or multi-step reasoning. The dense-model speedups are interesting, but stronger capability evaluation is needed to judge whether the accuracy cost is acceptable.
-
Quality validation is limited to two earlier-generation base models, Llama-3.1-70B and Qwen2-57B. This matters because the method changes model computation and requires recovery training: latency gains alone do not demonstrate a useful result on another model. Qwen3-235B is included only in the speed measurements, without corresponding recovered quality results. Thus, the paper does not establish a joint quality–latency benefit on a more recent model or an instruction-tuned/reasoning model representative of the intended deployment setting. My concern is the limited evidence for practical applicability, rather than model age by itself.
A secondary limitation is contribution attribution: Table 2 compares masks before recovery, and Figure 4 studies continued optimization starting from learned gates. A sensitivity-initialized fixed mask with the same LoRA recovery budget would clarify how much final quality comes from learning the mask rather than the pending-stream architecture and recovery. This does not invalidate the architectural contribution, and I would not treat the nonsignificant Rect-STE advantage over Identity STE as a separate major weakness. Overall, I lean toward weak rejection because the practical benefit of the lossy tradeoff remains insufficiently demonstrated, despite a plausible and technically useful design.
Questions for authors
-
Can the authors provide stronger evidence that recovered capabilities are preserved beyond the reported seven-task average, particularly for generation, multi-step reasoning, coding, or long-context use? The Llama perplexity and Qwen MMLU regressions make this consequential to the assessment. A quality–latency comparison at a more conservative skip ratio would also help identify an operating point with a clearly attractive tradeoff.
-
Are recovered quality results available for Qwen3-235B, whose speed is already measured, or for a more recent instruction-tuned/reasoning model? Paired quality and latency results on the same recovered checkpoint would materially strengthen the practical claim. I am not requesting an exhaustive model or hardware sweep.
-
As a secondary contribution-attribution question, how does a sensitivity-initialized fixed mask perform after the same LoRA recovery budget, compared with the learned mask? This would clarify the incremental value of gate learning. Please also specify the recovery hyperparameters and total training/calibration cost needed to reproduce that comparison.
We thank the reviewers for their feedback and efforts. Below, we respond to the main concerns, grouped by theme, and report new experiments. We incorporate these results into a revised version of the paper, the additions are marked.
Recovered quality for a more recent instruction-tuned model
We run our pipeline with a 50% skip ratio on Qwen-3-30B-instruct, the instruction-tuned version of a more recent MoE model. The average recovered LM-eval score for this model at 50% ratio is 97.4%. In addition, we measured speedups for this model on the same 8xA100 node, and the results are 7.86% for prefill at 2048 input sequence length and 6.66% for decode at batch size 128. The quality–latency tradeoff is not established, and all latencies come from one 8xA100 node. To measure the effect of hardware on speedup we performed similar measurements with the Llama-3.1-70B model at 50% skip ratio, on a node of 8xA6000 (slow PCIe interconnect) and on a node of 8xB200 (both, compute throughput and interconnect bandwidth are higher than A100, but the compute improved more). We added a plot comparing the speedup across different hardware to the revised paper.
In addition, we performed a longer recovery training with 84M tokens, on our 50% gated Llama-3.1-70B model to demonstrate that the accuracy can be improved with more recovery budget. This remains light recovery training in comparison with other methods, like Ladder Residual, which uses 3B tokens to recover the Llama-3.1-8B model.
The setup is impractical because TP has been deprecated in practice
TP remains standard for latency-sensitive serving of both dense and MoE models. For example, the official vLLM recipes deploy Llama 3.3 70B with TP8 and DeepSeek-R1 with TP8+EP. In addition, SPD, Ladder-Residual, and Flash Communication, all cited in our paper, are recent works targeting TP communication, which indicates that the community still treats it as a live problem.
https://recipes.vllm.ai/meta-llama/Llama-3.3-70B-Instruct
https://recipes.vllm.ai/deepseek-ai/DeepSeek-R1
The checkpoints are tied to one TP layout
Since our method is a post-training method, and only requires a short training phase, it could be applied separately for any target TP layout.
Points we have not addressed experimentally
We thank the reviewers for raising other points that we did not have enough time to address during the discussion period: accuracy evaluation on generative benchmarks, a comparison with low-precision all-reduce, a direct comparison with Ladder-Residual, the sensitivity-initialized fixed-mask ablation, and quality results for Qwen3-235B.