- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper try to address the communication bottleneck of TP (tensor parallelism) on LLM inference in bandwidth constrained settings. First, authors propose index-based all-reduce which encodes partial activations to indices into per-node, per-layer codebooks and exchange bit packed indices via all-gather. Second, authors propose focus calibration which calibrates per-layer LUTs layer by layer so that each codebook is fit on activations produced by an already quantized prefix of the model. The calibration activations are randomly subsampled before k-means.
Critical strengths
The methods show strong result on extreme bit-width. 2-bit result show less degradation compared to MX FP2 and TQ-MSE 2-bit.
Critical weaknesses
-
Rather than adding Section 3 as a separate section, merging these challenges into Section 2.2 can give more spaces for introducing more work for optimizing all-reduce. Quantization is not the only solution for optimizing all-reduce.
-
I don’t consider index-based all-reduce is a novel contribution. Since indices are not additive, gathering them and reducing locally after decoding is a natural consequence of applying VQ to all-reduce. In fact, any quantized all-reduce with a non-additive representation, including the MX and TQ-MSE baselines, requires the same gather-then-reduce scheme. Were the baselines implemented with the same all-gather and local reduction?
-
Focus calibration also has limited novelty. Layer-wise sequential calibration is standard in PTQ methods such as GPTQ[1], AdaRound[2], and BRECQ[3]. Similarly, random subsampling of data points before k-means is also common as in mini-batch k-means[4]. Faiss[5] also subsamples training points by default.
-
There are mismatches on numbers:
- Table2 Llama-2-13b MX FP2 is exactly same as Qwen3-1.7B MX FP2
- Table2 'As shown in Table 2, index-based all-reduce with VQ 2.5-bit and Focus Calibration achieves accuracy comparable to, or 1%-3.5% higher than, MX FP4 and MX FP3 across the evaluated models.’
- The proposed method’s cannot expand to larger scale of tensor parallelism due to the cost of all-gather. There wouldn’t be any benefit with TP=16 on 2-bit level due to structural limitation (With all-gather, the per-node communication amount relative to bf16 ring all-reduce is TPsize * bit / 32). Experiments only propose the results by TP=4.
[1] Frantar et al, GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [2] Nagel et al, Up or Down? Adaptive Rounding for Post-Training Quantization [3] Li et al, BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction [4] Scully, Web-scale k-means clustering [5] Johnson et al, Billion-scale similarity search with GPUs
Questions for authors
Please see section Critical Weaknesses
2 comments
Response to Reviewer VBmi (2/2)
-
W5. Scaling to larger TP groups. All-gather traffic grows with TP degree, erasing the 2-bit traffic saving at TP=16. Measurements extend to TP=4.
Thank you for pointing out this limitation. We acknowledge that the communication overhead of all-gather increases with TP size, limiting the scalability of our approach. Due to hardware resource constraints, we were unable to conduct experiments at TP=16. Nevertheless, the communication advantage decreases with TP size, our method can still provide meaningful end-to-end latency improvements at TP=8, while, based on the theoretical communication volume ratio of , this advantage disappears at TP=16.
Table 1. TP degree V.S prefill-stage latency (TTFT): 16 x 256 input tokens (16 prompts, 256 input tokens per prompts) for Qwen3-4B-Instruct-2507 (8GB parameters) in TP=2, 4, 8 Jetson ORIN devices @ 1GbE with the max model length=512, max batched size=512.
Method TP=2 TP=4 TP=8 BF16 ring all-reduce TTFT (ms) 7956 13482 17398 VQ 2.5-bits TTFT (ms) 3967 6085 12149 TTFT reduction 50.1% 54.9% 30.2% However, this scaling limitation does not diminish the demonstrated end-to-end latency benefits of our method in the evaluated TP=2,4 collaborative inference scenarios under bandwidth constraints or interconnect contention. As shown in Figure 7, our 2-bit variant reduces TTFT by 65.3%–73.9% and TPOT by 26.1%–37.1% compared with BF16 on Jetson Orin Nano and AGX Orin clusters connected via 1 GbE, across TP=2 and TP=4 configurations. Furthermore, Figure 8 demonstrates that our method also benefits server GPUs experiencing interconnect contention. On two NVIDIA A30 GPUs (TP=2) connected via PCIe 3.0 16, our 2-bit variant reduces TTFT by 24.1%–71.8% and TPOT by 15.7%–44.1% compared with BF16/FP16 across three evaluated models at 90% background bandwidth occupancy. These results demonstrate that our method provides substantial end-to-end latency improvements in its targeted scenarios, despite its limited scalability to larger TP groups. We acknowledge this limitation and will investigate more scalable communication schemes in future work.
Response to Reviewer VBmi (1/2)
We thank the reviewer for identifying the relevant prior work, and inconsistencies in the accuracy results.
W1. Related work and organisation. Incorporate the challenges into Section 2.2 and broaden the discussion beyond quantisation-based all-reduce optimisation.
We thank the reviewer for the helpful comments and suggestions. While this concern primarily relates to the organisation and the related-work discussion rather than the technical soundness of our work, we recognize its importance for clearly positioning our contributions. We acknowledge that collaborative communication can be optimised through various approaches beyond quantisation, including communication topology optimisation and hardware interconnect technologies (e.g., RDMA). Our approach is complementary to these optimisations, as activation compression can be combined with them to further improve communication efficiency. However, a detailed discussion of these alternative approaches is beyond the scope of this work. To better clarify our motivation and contributions, Section 2.2 focuses on collaborative communication schemes and activation quantisation methods, highlighting two key challenges: Performance Collapse under ultra-low-bit quantisation and Expensive Retraining. Our work specifically addresses these challenges through training-free activation compression on the TP communication path.
W2. & W3. Lack of novelty Gather-then-reduce scheme is a natural way to realise VQ-based reduction. Sequential calibration and subsampling are both established. Did MX and TQ-MSE use the same all-gather and local reduction path?
We acknowledge that gather-then-reduce, layer-wise sequential calibration, and random subsampling are established techniques. For fair comparison, we implement MX and TQ-MSE baselines using the same all-gather and local reduction scheme. However, methodological novelty does not necessarily require fundamentally new individual components. For example, GPTQ extends Optimal Brain Quantization (OBQ) with algorithmic improvements, while BRECQ builds upon AdaRound and learned step-size quantisation to improve block-wise reconstruction. Moreover, our contribution lies not in introducing a fundamentally new collective communication pattern, but in making ultra-low-bit TP activation compression practically feasible through Focus Calibration, which mitigates the severe accuracy degradation of naive quantisation without retraining. To the best of our knowledge, this is the first work to demonstrate the practical feasibility of codebook-based vector quantisation for tensor-parallel activation All-Reduce in LLM inference.
Beyond the methodological contributions, we validate our approach through vLLM integration and end-to-end evaluations on real Jetson clusters and NVIDIA A30 GPUs under bandwidth constraints and interconnect contention. As also noted by Reviewer Beio, "the edge setting is practical and the vLLM integration with real Jetson measurements is useful". We believe that enabling effective 2–2.5-bit TP communication while maintaining competitive model accuracy, together with real-hardware validation, constitutes a meaningful contribution beyond combining established techniques.
W4. Accuracy inconsistencies. The Llama-2-13B and Qwen3-1.7B MX FP2 rows are identical in Table 2. The accompanying claim of comparable or higher VQ accuracy relative to MX FP3/FP4 also conflicts with individual model results.
Thank you for pointing out these numerical inconsistencies. The MX FP2 results for Qwen3-1.7B-FP8 in Table 2 were incorrectly reported due to a copy-paste mistake. The correct results are provided below.
We also acknowledge that the reported accuracy improvement of 1.0%–3.5% was inaccurate. The corrected statement is:
"As shown in Table 2, index-based all-reduce with VQ 2.5-bit and Focus Calibration achieves accuracy comparable to, or 1.6%–2.3% higher than, MX FP4 and MX FP3 across the evaluated models."
These mistakes do not undermine our main conclusion that the proposed Focus Calibration enables 2-bit and 2.5-bit VQ to achieve better or comparable accuracy than the evaluated baselines at equivalent effective bit-widths. We will correct these numerical errors in the revised manuscript.