acceptodds
Under review as a conference paper at ICLR 2027

Focus Calibration for Ultra-Low-Bit All-Reduce in Bandwidth-Constrained Collaborative LLM Inference

21 views

Abstract

In bandwidth-constrained environments, collaborative inference of large language models (LLMs) is increasingly bottlenecked by activation communication under tensor parallelism. Although low-bit activation compression can reduce communication overhead in distributed LLM inference, further bit-width reduction is approaching practical limits, as aggressive compression often incurs substantial accuracy degradation or requires costly retraining. To address this challenge, we propose Focus Calibration, a lightweight calibration strategy for retraining-free ultra-low-bit vector quantisation, which enables an index-based all-reduce mechanism that exchanges compact centroid indices instead of full-precision activation tensors. Experimental results show that our method substantially reduces activation communication overhead, achieving up to 38.8% and 12.3% reductions in end-to-end prefill and decoding latency, respectively, while maintaining accuracy comparable to existing quantisation-based approaches at ultra-low bit-widths of 2-2.5 bits. These results establish index-based all-reduce with Focus Calibration as a practical solution for communication-efficient LLM inference in bandwidth-constrained collaborative systems.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Communication path Quantisation, Quantisation calibration, Collabrative LLM inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VBmiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper try to address the communication bottleneck of TP (tensor parallelism) on LLM inference in bandwidth constrained settings. First, authors propose index-based all-reduce which encodes partial activations to indices into per-node, per-layer codebooks and exchange bit packed indices via all-gather. Second, authors propose focus calibration which calibrates per-layer LUTs layer by layer so that each codebook is fit on activations produced by an already quantized prefix of the model. The calibration activations are randomly subsampled before k-means.

Critical strengths

The methods show strong result on extreme bit-width. 2-bit result show less degradation compared to MX FP2 and TQ-MSE 2-bit.

Critical weaknesses
  1. Rather than adding Section 3 as a separate section, merging these challenges into Section 2.2 can give more spaces for introducing more work for optimizing all-reduce. Quantization is not the only solution for optimizing all-reduce.

  2. I don’t consider index-based all-reduce is a novel contribution. Since indices are not additive, gathering them and reducing locally after decoding is a natural consequence of applying VQ to all-reduce. In fact, any quantized all-reduce with a non-additive representation, including the MX and TQ-MSE baselines, requires the same gather-then-reduce scheme. Were the baselines implemented with the same all-gather and local reduction?

  3. Focus calibration also has limited novelty. Layer-wise sequential calibration is standard in PTQ methods such as GPTQ[1], AdaRound[2], and BRECQ[3]. Similarly, random subsampling of data points before k-means is also common as in mini-batch k-means[4]. Faiss[5] also subsamples training points by default.

  4. There are mismatches on numbers:

  • Table2 Llama-2-13b MX FP2 is exactly same as Qwen3-1.7B MX FP2
  • Table2 'As shown in Table 2, index-based all-reduce with VQ 2.5-bit and Focus Calibration achieves accuracy comparable to, or 1%-3.5% higher than, MX FP4 and MX FP3 across the evaluated models.’
  1. The proposed method’s cannot expand to larger scale of tensor parallelism due to the cost of all-gather. There wouldn’t be any benefit with TP=16 on 2-bit level due to structural limitation (With all-gather, the per-node communication amount relative to bf16 ring all-reduce is TPsize * bit / 32). Experiments only propose the results by TP=4.

[1] Frantar et al, GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [2] Nagel et al, Up or Down? Adaptive Rounding for Post-Training Quantization [3] Li et al, BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction [4] Scully, Web-scale k-means clustering [5] Johnson et al, Billion-scale similarity search with GPUs

Questions for authors

Please see section Critical Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer VBmi (1/2)

We thank the reviewer for identifying the relevant prior work, and inconsistencies in the accuracy results.

  • W1. Related work and organisation. Incorporate the challenges into Section 2.2 and broaden the discussion beyond quantisation-based all-reduce optimisation.

    We thank the reviewer for the helpful comments and suggestions. While this concern primarily relates to the organisation and the related-work discussion rather than the technical soundness of our work, we recognize its importance for clearly positioning our contributions. We acknowledge that collaborative communication can be optimised through various approaches beyond quantisation, including communication topology optimisation and hardware interconnect technologies (e.g., RDMA). Our approach is complementary to these optimisations, as activation compression can be combined with them to further improve communication efficiency. However, a detailed discussion of these alternative approaches is beyond the scope of this work. To better clarify our motivation and contributions, Section 2.2 focuses on collaborative communication schemes and activation quantisation methods, highlighting two key challenges: Performance Collapse under ultra-low-bit quantisation and Expensive Retraining. Our work specifically addresses these challenges through training-free activation compression on the TP communication path.

  • W2. & W3. Lack of novelty Gather-then-reduce scheme is a natural way to realise VQ-based reduction. Sequential calibration and subsampling are both established. Did MX and TQ-MSE use the same all-gather and local reduction path?

    We acknowledge that gather-then-reduce, layer-wise sequential calibration, and random subsampling are established techniques. For fair comparison, we implement MX and TQ-MSE baselines using the same all-gather and local reduction scheme. However, methodological novelty does not necessarily require fundamentally new individual components. For example, GPTQ extends Optimal Brain Quantization (OBQ) with algorithmic improvements, while BRECQ builds upon AdaRound and learned step-size quantisation to improve block-wise reconstruction. Moreover, our contribution lies not in introducing a fundamentally new collective communication pattern, but in making ultra-low-bit TP activation compression practically feasible through Focus Calibration, which mitigates the severe accuracy degradation of naive quantisation without retraining. To the best of our knowledge, this is the first work to demonstrate the practical feasibility of codebook-based vector quantisation for tensor-parallel activation All-Reduce in LLM inference.

    Beyond the methodological contributions, we validate our approach through vLLM integration and end-to-end evaluations on real Jetson clusters and NVIDIA A30 GPUs under bandwidth constraints and interconnect contention. As also noted by Reviewer Beio, "the edge setting is practical and the vLLM integration with real Jetson measurements is useful". We believe that enabling effective 2–2.5-bit TP communication while maintaining competitive model accuracy, together with real-hardware validation, constitutes a meaningful contribution beyond combining established techniques.

  • W4. Accuracy inconsistencies. The Llama-2-13B and Qwen3-1.7B MX FP2 rows are identical in Table 2. The accompanying claim of comparable or higher VQ accuracy relative to MX FP3/FP4 also conflicts with individual model results.

    Thank you for pointing out these numerical inconsistencies. The MX FP2 results for Qwen3-1.7B-FP8 in Table 2 were incorrectly reported due to a copy-paste mistake. The correct results are provided below.

    ModelMethodBitsARC-cARC-eBoolQH.S.PIQAW.G.AVG
    Qwen3-1.7B-FP8MX FP22.2523.829.952.829.251.749.939.6

    We also acknowledge that the reported accuracy improvement of 1.0%–3.5% was inaccurate. The corrected statement is:

    "As shown in Table 2, index-based all-reduce with VQ 2.5-bit and Focus Calibration achieves accuracy comparable to, or 1.6%–2.3% higher than, MX FP4 and MX FP3 across the evaluated models."

    These mistakes do not undermine our main conclusion that the proposed Focus Calibration enables 2-bit and 2.5-bit VQ to achieve better or comparable accuracy than the evaluated baselines at equivalent effective bit-widths. We will correct these numerical errors in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer VBmi (2/2)

  • W5. Scaling to larger TP groups. All-gather traffic grows with TP degree, erasing the 2-bit traffic saving at TP=16. Measurements extend to TP=4.

    Thank you for pointing out this limitation. We acknowledge that the communication overhead of all-gather increases with TP size, limiting the scalability of our approach. Due to hardware resource constraints, we were unable to conduct experiments at TP=16. Nevertheless, the communication advantage decreases with TP size, our method can still provide meaningful end-to-end latency improvements at TP=8, while, based on the theoretical communication volume ratio of TPsize×bit/32\mathrm{TPsize} \times \mathrm{bit}/32, this advantage disappears at TP=16.

    Table 1. TP degree V.S prefill-stage latency (TTFT): 16 x 256 input tokens (16 prompts, 256 input tokens per prompts) for Qwen3-4B-Instruct-2507 (8GB parameters) in TP=2, 4, 8 Jetson ORIN devices @ 1GbE with the max model length=512, max batched size=512.

    MethodTP=2TP=4TP=8
    BF16 ring all-reduce TTFT (ms)79561348217398
    VQ 2.5-bits TTFT (ms)3967608512149
    TTFT reduction50.1%54.9%30.2%

    However, this scaling limitation does not diminish the demonstrated end-to-end latency benefits of our method in the evaluated TP=2,4 collaborative inference scenarios under bandwidth constraints or interconnect contention. As shown in Figure 7, our 2-bit variant reduces TTFT by 65.3%–73.9% and TPOT by 26.1%–37.1% compared with BF16 on Jetson Orin Nano and AGX Orin clusters connected via 1 GbE, across TP=2 and TP=4 configurations. Furthermore, Figure 8 demonstrates that our method also benefits server GPUs experiencing interconnect contention. On two NVIDIA A30 GPUs (TP=2) connected via PCIe 3.0 ×\times 16, our 2-bit variant reduces TTFT by 24.1%–71.8% and TPOT by 15.7%–44.1% compared with BF16/FP16 across three evaluated models at 90% background bandwidth occupancy. These results demonstrate that our method provides substantial end-to-end latency improvements in its targeted scenarios, despite its limited scalability to larger TP groups. We acknowledge this limitation and will investigate more scalable communication schemes in future work.

Reviewer_BeioOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Hello, authors! Thank you for the work, collaborative inference on edge devices over slow links is a practical topic, and it was nice to see real Jetson experiments. Below is my understanding of the paper.

The setting is tensor-parallel LLM inference over low bandwidth links, where the all-reduce of activations after o_proj and down_proj becomes the bottleneck. You propose to replace it with an index-based all-reduce: each rank splits its partial output into 2-dim sub-vectors, encodes each one as an index into a per-position codebook (LUT) with 2^k centroids (k = 4/5/6, so 2/2.5/3 bits per element), all-gathers the packed indices, and every rank decodes all ranks' indices with their LUTs and sums locally. The LUTs are built with Focus Calibration: k-means layer by layer, where layer l is calibrated on activations that already went through VQ in lower layers, plus random subsampling of P% of the calibration activations, with a GMM-style argument in A.3 that this reduces outlier influence. It is retraining-free.

Evaluation is on Qwen3-1.7B, Qwen3-4B, Llama-3.1-8B and Llama-2-13B, accuracy on zero-shot tasks and perplexity, latency on Jetson Orin Nano / AGX Orin (TP=2,4) over 1 GbE and 2xA30 with PCIe background traffic, against MX FP3/FP4 and TurboQuant-MSE baselines. Claims are up to 87% payload reduction, up to 38.8% lower prefill and 12.3% lower decode latency vs MX FP3, with comparable accuracy.

The code is a set of modified vLLM v0.12.0 files (RowParallelLinear and communicator) with LUTs for Qwen3-4B. In the README I found note that calibration details will be released later if the paper is accepted.

Critical strengths
  • real end-to-end integration into vLLM with Triton bit-packing for 4/5/6-bit indices, tested on real edge hardware (Jetson Orin Nano and AGX Orin over 1 GbE). the Jetson latency numbers look consistent with a simple bandwidth estimate, so the communication bottleneck and the gains there look real
  • calibrating each layer on activations after the lower layers are already quantized is a reasonable idea, and Fig. 6 shows a clear KL improvement over one-shot calibration. the method is also retraining-free, which matters for edge deployment
Critical weaknesses

Critical:

  • the core contribution can't be verified: there is no calibration code (k-means, layer-wise procedure, subsampling), README says "Calibration details will be released later if the paper is accepted", and LUTs are given only for Qwen3-4B at TP=2. also model.safetensors.index.json points to a LUT file that is not shipped
  • the released "VQ 2-bit" code seems to send 2.5 bits: VQ-INT4/linear.py has 16 centroids but packs indices with pack_int5 (pack_int4 is defined but never used). so with this code 2-bit and 2.5-bit send the same amount of data, while Fig. 7 shows different latency for them. it is not clear which code produced Fig. 7-8
  • the index-based all-reduce is an all-gather of indices plus local decode and sum, so every rank receives the indices of all other ranks and the volume grows with TP. compared to a bf16 ring all-reduce the saving is 87% only at TP=2, at TP=4 it is already about 75% and at TP=8 about 50%, while experiments go only up to TP=4. also the MX and TQ-MSE baselines in the code use the same all-gather pipeline, so the communication scheme itself is not specific to the method, only the codec is
  • calibration looks unstable and the defaults were chosen on test sets: in Fig. 14 perplexity for Qwen3-1.7B changes by several times between P = 0.5 / 0.75 / 1, sometimes more calibration data gives worse results, and the default (960 samples, P=0.75) was selected on WikiText / LAMBADA / Pile, which are also evaluation sets. so it is hard to tell if Focus Calibration is robust or P works more like a random seed

I still lean to weak accept because the edge setting is practical and the vLLM integration with real Jetson measurements is useful, but this depends on the calibration code being released and on the stability across seeds and P

Questions for authors

clarification:

  • how does communicated volume compare with bf16 ring all-reduce at TP=4 and 8? is a reduce-scatter compatible variant possible?
  • how does A.3 support lower outlier influence if uniform subsampling keeps the outlier fraction the same?
  • why MX is implemented as a uniform grid with round-based scale instead of OCP E2M1?
  • which code produced the zero-shot accuracies, the compiled path or another eager implementation?

suggestions:

  • release calibration code or at least pseudo-code with k-means settings, init, seeds and run time
  • report variance over seeds and choose P / data size on held-out data, not on the test sets
  • TQ-MSE with randomized Hadamard and single inverse rotation after sum
  • prefill-only latency on A30 without queueing
  • compare with at least one prior TP communication compression method
4 comments
AuthorsOpenReview ↗

Response to Reviewer Beio (1/4)

Hello Reviewer! Thank you very much for your recognition of this work and for your valuable suggestions.

  • W1 & S1. Missing calibration code. The supplement omits k-means, sequential calibration, and subsampling code. It provides LUTs only for Qwen3-4B at TP=2, and model.safetensors.index.json points to a LUT file that is not shipped. Please provide the calibration procedure.

    We acknowledge that the supplementary materials do not sufficiently explain the calibration procedure. Below, we clarify how Focus Calibration is implemented, including activation sampling, LUT generation, and sequential calibration.

    Regarding the missing LUT file referenced in model.safetensors.index.json, the supplementary materials already include the required LUTs, but the index file contains an outdated filename. To resolve this issue, please update the corresponding entries to reference the provided LUT file, such as vq-int5-lut-2gpu.safetensors. We will also correct these references in the revised supplementary materials.

    1. Activation sampling. We implement a sampling function, _save_tp_tensor, in RowParallelLinear.forward() to capture local activations from each TP rank before all-reduce. The sampling function is controlled by a customised environment variable: VLLM_LAYER_ID, which specifies the target layer. vLLM needs VLLM_ prefix in enveriment variable name to propagate them to vLLM worker processes. The open-source lm-evaluation-harness (lm_eval) is used to feed a fixed calibration set into vLLM under eager execution mode, with --limit controlling the number of calibration samples.

    Here is the sampling function injection example:

      # in linear.py::RowParallelLinear.forward()
      ...
      output_parallel = self.quant_method.apply(self, input_parallel, bias_)
      if self.reduce_results and self.tp_size > 1:
          
          if not torch._dynamo.is_compiling():
              _save_tp_tensor(
                  output_parallel,
                  self.tp_rank,
                  layer_id=getattr(self, "prefix", None),
                  token_size=output_parallel.shape[0],
              )
          
          # output = tensor_model_parallel_all_reduce(output_parallel)
          indices = self.encode(output_parallel, self.lut[self.tp_rank])
          indices_all = self._all_gather_lut_indices(indices, self.tp_size)
          output = self.decode_and_sum(indices_all, self.lut).view(output_parallel.shape)
      ...
    

    2. Sequential calibration and LUT generation. We initially assign randomly initialised LUTs to all TP ranks and layers. Focus Calibration then proceeds sequentially through the target layers. For each layer, we collect local activations from all TP ranks and use our customised generate_lut.py script, which primarily relies on FAISS for uniform random subsampling at ratio P and k-means clustering to generate rank-specific VQ LUTs. The generated LUTs replace the initial LUTs of the corresponding layer before proceeding to the next layer. This allows downstream calibration to account for approximation errors introduced by previously calibrated layers.

    The overall calibration procedure is summarised below:

    for layer_id in "${TARGET_LAYERS[@]}"; do
        # 1. Collect pre-all-reduce activations from all TP ranks
        export VLLM_LAYER_ID="$layer_id"
    
        lm_eval --limit "$DATA_SIZE" \
                --other-vllm-config ...
    
        # 2. Generate rank-specific LUTs and update model.safetensors.index.json correspondingly
        python generate_lut.py \
            --sample-rate "$P" \
            --layer "$VLLM_LAYER_ID"
    
    done
    
  • W2 & Q4. Packing width and experimental code paths. The supplied 16-centroid path calls pack_int5, which may make the communicated payload 2.5 bits per element. Which implementation generated Figures 7–8 and the zero-shot accuracy results, and were the zero-shot accuracy obtained with the compiled or eager path?

    Thank you for pointing out this inconsistency. This error was introduced when we manually organised the project files for the supplementary materials. Specifically, the 16-centroid VQ 2-bit implementation should use pack_int4 rather than pack_int5, as the former packs two 4-bit indices into one byte, whereas the latter is intended for 5-bit indices used by VQ 2.5-bit. We confirm that the actual implementation used for the experiments employed the correct packing functions for the corresponding bit-widths (VQ 2-bit uses pack_int4 and VQ 2.5-bit uses pack_int5). This mistake is limited to the submitted supplementary code and does not affect the reported experimental results. And all experiment results in Figures 7–8 and the zero-shot accuracy were conducted at vLLM compiled mode. We will correct the inconsistency in the revised supplementary materials.

AuthorsOpenReview ↗

Response to Reviewer Beio (2/4)

  • W3 & Q1. Collective scaling and reduce-scatter. Experiments go only up to TP=4. And the communication scheme itself is not specific to the method, only the codec is. How do the traffic savings change at TP=4 and 8 compared to bf16 ring all-reduce, and could the design use reduce-scatter?

    Thank you for raising this point. We acknowledge that gather-then-reduce is a common approach for applying non-additive quantisation to TP communication, and that the communication overhead of all-gather limits the scalability of our method as TP size increases.

    1. Communication savings and TP scaling. Compared with BF16 ring all-reduce, the theoretical per-rank communication volume ratio of our index-based all-reduce is approximately TPsize x bit/32. For VQ 2.5-bit, this corresponds to communication volume reductions of 68.75% at TP=4 and 37.5% at TP=8. Although the communication advantage decreases with TP size, our method can still provide meaningful end-to-end latency improvements at TP=8.

    Table 1. TP degree V.S prefill-stage latency (TTFT): 16 x 256 input tokens (16 prompts, 256 input tokens per prompts) for Qwen3-4B-FP8 in TP=2, 4, 8 Jetson devices @ 1GbE with the max model length=512, max batched size=512.

    MethodTP=2TP=4TP=8
    BF16 ring all-reduce TTFT (ms)79561348217398
    VQ 2.5-bits TTFT (ms)3967608512149
    TTFT reduction50.1%54.9%30.2%

    To further investigate this, we evaluate Qwen3-4B-Instruct-2507-FP8 on Jetson devices. As shown in Table 1, our VQ 2.5-bit method reduces TTFT by 54.9% and 30.2% at TP=4 and TP=8, respectively. These results demonstrate that, despite the diminishing theoretical communication savings, our method continues to provide substantial end-to-end latency improvements in bandwidth-constrained collaborative inference at TP=8.

    2. Feasibility of reduce-scatter. Incorporating reduce-scatter-like communication could potentially improve the scalability of index-based all-reduce. However, since VQ indices are non-additive, the standard reduce-scatter operation cannot directly perform reduction on the encoded indices. One possible implementation is to partition the encoded indices and exchange them through all-to-all communication, allowing each rank to decode and locally reduce its assigned partition. The reduced partitions can then be distributed through all-gather. To maintain compressed communication in this second stage, the reduced activations would need to be encoded again before all-gather and decoded after reception, introducing additional encoding and decoding overhead compared with our current implementation. Therefore, whether such a scheme can provide better end-to-end performance requires further experimental investigation, which we consider an important direction for future work.

  • W4 & S2. Calibration sensitivity. Figure 14 shows large variation across PP and calibration-set sizes. The default appears to have been chosen using datasets also used for evaluation. Please report variance and select parameters on held-out data.

    Thank you for your suggestions. To assess calibration stability, we repeated the experiments with three random seeds on the WikiText, LAMBADA (standard), and Pile Wikipedia validation sets. We now report both the mean perplexity and its standard deviation in Figure 14 of the revised manuscript. Across the evaluated models, datasets, some configurations show substantial variation across random seeds, occasionally exceeding the differences caused by changing the sampling ratio P or calibration data size. This suggests that calibration stability is configuration- and model-dependent.

    The results also show that increasing the calibration data size does not always improve perplexity. This is consistent with our analysis in Appendix A.3, as calibration subset construction also involves sampling, which requires balancing estimation variance against the influence of noisy outliers. Moreover, the optimal combination of P and data size varies across models. The original configuration (P=0.75, 960 samples) achieves competitive and relatively stable results for Qwen3-4B, Llama-3.1-8B and Llama-2-13B, but exhibits greater variability for Qwen3-1.7B-FP8. For Qwen3-1.7B-FP8, calibration using 320 samples with P=0.5 or P=0.75 achieve a better overall balance between mean perplexity and calibration stability across the three validation sets.

    We retain the original configuration (P=0.75, 960 samples) for Qwen3-1.7B-FP8 in Tables 1 and 2 to maintain consistency with the previously reported results during the rebuttal. In subsequent revisions, we will select more stable calibration configurations using a separate development split and evaluate them on independent test sets. These findings highlight the importance of considering both mean performance and variability when selecting calibration parameters, rather than relying on a single random seed.

AuthorsOpenReview ↗

Response to Reviewer Beio (3/4)

  • Q2. Interpretation of the outlier analysis. How can uniform subsampling reduce outlier influence when the expected outlier fraction is unchanged?

    We thank the reviewer for this important question. We acknowledge that uniform subsampling preserves the expected proportion of outliers. However, our analysis concerns the disproportionate influence of rare, significantly distant outliers on centroid estimation, rather than their expected proportion.

    Specifically, Appendix A.3 considers the condition where the squared distance between the clean and outlier distribution centres is significantly larger than the variance of clean samples. Under this condition, even a small number of outliers can introduce substantial centroid estimation errors. Equation (10) indicates that reducing the uniform subsampling ratio increases the probability of obtaining an outlier-free calibration subset when outliers are sparse, while Equations (11)–(17) characterise their contribution to centroid estimation error.

    More importantly, k-means-based VQ has limited codebook capacity. Reducing the sampling ratio can decrease the likelihood of including rare, calibration-specific outlier patterns during calibration, thereby reducing the risk of allocating limited codebook capacity to noise-induced activation patterns. However, excessively reducing the sampling ratio also increases estimation variance due to fewer calibration samples. This trade-off motivates the use of a moderate sampling ratio.

    Figure 11 illustrates this mechanism, while Figures 12 and 14 provide empirical evidence that subsampling can improve generalisation and perplexity compared with full-sample layer-wise calibration. Therefore, our argument is not that uniform subsampling reduces the expected proportion of outliers, but that it can mitigate their disproportionate influence on finite-codebook calibration, as supported by our theoretical analysis and experimental results.

  • Q3. Fidelity of the MX baseline. Why does the implementation use a uniform grid and round-based scaling when the manuscript describes the OCP E2M1 format?

    Thank you for pointing out this inconsistency. Our implementation uses a uniform quantisation grid of [0, 3/7, 6/7, 9/7, 12/7, 15/7, 18/7, 3] with round-based scaling, whereas the OCP E2M1 format uses a non-uniform grid of [0, 0.5, 1, 1.5, 2, 3, 4, 6] with floor-based scaling, where the scale exponent is computed as floor(log2(max_abs)) - element_max_exponent + 127 and clamped to [0, 254].

    To assess the impact of this implementation difference, we additionally evaluated the OCP-compliant E2M1 format across four models and six zero-shot benchmarks. The OCP E2M1 implementation achieves an average zero-shot accuracy of 70.7%, compared with 70.4% for our original uniform-grid FP4 implementation, a difference of only 0.3 percentage points. We further compared their end-to-end inference latency on Jetson devices and NVIDIA A30 GPUs, finding that the OCP E2M1 implementation reduces both TTFT and TPOT by less than 4.5% compared with our original implementation.

    Given the small differences in accuracy and latency, we retain the original FP4 implementation for consistency with the reported experimental results. However, we acknowledge that our implemented FP4 baseline differs from the OCP-recommended E2M1 format in both the quantisation grid and scaling strategy. We will therefore revise the manuscript to explicitly label our baseline as MX FP4 (uniform-grid variant), distinguish it from the OCP-recommended E2M1 format, and clarify the corresponding implementation details.

  • S3. Faster TQ-MSE implementation. Evaluate a randomised Hadamard transform and apply the inverse rotation once after summing the decoded vectors.

    Thank you for the suggestion. We acknowledge that including additional baselines would provide a more comprehensive evaluation of the proposed method. We consider randomised Hadamard-based quantisation methods, such as QuaRot, and the TQ-MSE baseline evaluated in our paper to belong to the broader category of transformation-based quantisation methods. These methods employ linear or non-linear transformations to reshape data distributions and mitigate the influence of outliers, thereby improving quantisation accuracy.

    In our experiments, we have included TQ-MSE as a representative transformation-based quantisation baseline. However, we acknowledge that randomised Hadamard transformations may offer different trade-offs and deserve further investigation. Given the limited rebuttal period and the existing comparison with TQ-MSE, we are unable to include additional experiments with randomised Hadamard-based quantisation at this stage. We will investigate these additional baselines, including the proposed single inverse rotation after summation, in future work.

AuthorsOpenReview ↗

Response to Reviewer Beio (4/4)

  • S4. Prefill-only latency on A30 without queueing.

    Thank you for your suggestion. We conducted additional experiments to compare the no-queueing TTFT of our proposed method against BF16/FP16 ring all-reduce on two NVIDIA A30 GPUs (TP=2) connected via PCIe 3.0 ×16, using three models: Qwen3-4B, Llama-3.1-8B, and Llama-2-13B.

    Experimental setup. We use vllm bench serve --request-rate inf to submit 16 prompts, each containing 256 input tokens, without intentional delays between requests. To allow all requests to be processed in a single prefill batch without queueing, the vLLM scheduler settings should satisfy the following constraints:

    • max_num_seqs ≥ number of prompts
    • max_num_batched_tokens ≥ number of prompts × input tokens per prompt
    • max_model_len ≥ input tokens + output tokens per prompt (set to 1 for prefill-only evaluation)

    Accordingly, we set max_num_seqs=16, max_num_batched_tokens=4096, and max_model_len=512, allowing all 16 prompts to fit within a single prefill batch, provided sufficient KV-cache capacity and simultaneous admission. We evaluate TTFT under background bandwidth occupancy ranging from 0% to 90%.

    Table 2. No-queueing TTFT (ms) on two NVIDIA A30 GPUs under varying background bandwidth occupancy.

    ModelMethod0%20%40%60%80%90%
    Llama-2-13BFP161024.23324.93308.33530.83400.03503.7
    VQ 2-bit745.7896.9972.31006.31033.41024.9
    VQ 2.5-bit874.31073.21169.21203.41227.71239.1
    Llama-3.1-8BBF16630.82059.72174.02128.12160.92237.3
    VQ 2-bit453.9426.1579.6606.0619.3615.6
    VQ 2.5-bit556.0663.2700.7742.7746.3767.8
    Qwen3-4BBF16377.11397.11482.01540.21547.71535.7
    VQ 2-bit375.2415.9320.6389.6465.9464.3
    VQ 2.5-bit449.7490.6547.2560.7544.6563.3

    As shown in Table 2, our VQ 2-bit and 2.5-bit variants reduce TTFT by up to 79.3% and 67.8%, respectively, compared with BF16/FP16 ring all-reduce under interconnect contention (20%–90% background bandwidth occupancy). These results further demonstrate that our method can substantially reduce prefill-stage latency under interconnect contention, supporting communication-efficient collaborative LLM inference.

  • S5. Comparison with prior TP communication compression. Include at least one existing method designed for TP communication.

    Thank you for this valuable suggestion. When implementing our baselines, we mainly referred to Communication Compression for Tensor Parallel LLM Inference, which employs MX-format quantisation with a gather-then-reduce scheme to compress TP communication. However, since its implementation is not publicly available, we cannot guarantee that our implementation reproduces all its technical details. Therefore, we do not claim a direct comparison with this prior work. Instead, we implement widely used quantisation methods, including MX formats (uniform-grid variant) and TQ-MSE, within the same codebase to ensure a fair and consistent comparison. This allows us to evaluate the effectiveness of different quantisation methods for TP communication while minimising differences in implementation.

Reviewer_cW9xOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the massive communication bottleneck that occurs when running large language models via Tensor Parallelism (TP) across weak networks (specifically edge clusters connected by 1 GbE). To cut down on all-reduce overhead, the authors introduce an index-based communication scheme that uses Vector Quantization (VQ) to compress activations to ultra-low bit-widths (2 to 2.5 bits). Instead of floating-point values, nodes exchange compact centroid indices via node-specific Lookup Tables (LUTs). To fix the resulting distribution drift without full retraining, they propose Focus Calibration, which uses a simple random subsampling heuristic (P%P\%) during layer-wise calibration to prevent the VQ centroids from overfitting to data noise.

Critical strengths

• Clear Bottleneck Targeting: The paper correctly identifies that in low-bandwidth or edge environments, TP communication completely kills end-to-end latency, making compression a necessity.

• Solid Latency Gains: The empirical speedups on edge hardware (up to 38.8% reduction in prefill latency) demonstrate that the system does effectively reduce payload size.

• Strong Control Baselines: Comparing against standard Micro-scaling (MX) formats and transformation-based methods (TQ-MSE) provides good context for the low-bit performance trade-offs.

Critical weaknesses

• Heuristic Novelty: Stripped of the academic framing, "Focus Calibration" is simply running layer-wise k-means using a random subset of the training data (P=75%P=75\%) to smooth out outliers. Subsampling to prevent overfitting is a text-book data technique; packaging it as a primary conference contribution feels thin.

• Massive Computational Overhead: Looking at the latency profile (Figure 13), the encoding and decoding steps are incredibly heavy—often eating up over 60% of the total time profile. This means the method only yields a net benefit in incredibly slow networking conditions (like 1 GbE). On standard high-speed interconnects or even mildly modernized networks, this compute overhead would easily wipe out the communication savings.

• Niche Hardware Focus: The evaluation relies heavily on legacy/edge hardware setups (Jetson Orin clusters over 1 GbE). While interesting for edge AI workshops, it limits the paper's broader applicability to standard enterprise or cloud-scale distributed LLM inference.

• Scale of Tested Models: Evaluating models up to 13B parameters is acceptable, but the focus on 1.7B and 4B models undermines the premise of needing heavy collaborative tensor parallelism, as these smaller models can increasingly fit onto single modern edge boards or lightweight devices without splitting layers across devices.

Questions for authors
  1. Bandwidth Tipping Point: At what network bandwidth (e.g., 10 GbE, 40 GbE) does the heavy computational overhead of VQ encoding/decoding break even with, or become slower than, just sending uncompressed BF16 tensors?

  2. Dynamic Workloads: How does the node-specific LUT approach handle dynamic runtime scenarios where background workloads cause asymmetric compute throttling across nodes?

  3. Outlier Scaling: Since the subsampling strategy explicitly downplays boundary noise, does it accidentally drop rare but critical systemic outliers that control model reasoning performance in larger LLMs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cW9x (1/2)

We thank the reviewer for recognising the communication bottleneck in bandwidth-constrained TP inference and the latency gains on Jetson. We clarify the contribution and the conditions under which the communication savings offset the cost of VQ.

  • W1. Novelty of Focus Calibration. Layer-wise calibration and random subsampling are established techniques. Their combination offers limited novelty and contribution.

    We acknowledge that sequential calibration and uniform subsampling are established techniques. Our contribution lies not in these techniques individually, but in a practical, low-overhead communication-path co-design that addresses calibration challenges arising from accumulated activation approximation errors, enabling ultra-low-bit activation compression for TP inference without retraining.

    Specifically, index-based all-reduce reduces communication overhead by transmitting compact indices instead of full-precision activations, while Focus Calibration mitigates activation distribution drift and calibration overfitting caused by accumulated quantisation errors along the TP communication path. These two components address complementary challenges: index-based all-reduce reduces communication overhead, while Focus Calibration preserves model accuracy at 2–2.5 bits. Together, they enable practical, retraining-free, ultra-low-bit collaborative LLM inference. To the best of our knowledge, this is the first work to demonstrate the practical feasibility of codebook-based vector quantisation for tensor-parallel activation All-Reduce in LLM inference.

    Beyond the methodological contributions, we validate our approach through vLLM integration and end-to-end evaluations on real Jetson clusters and NVIDIA A30 GPUs under bandwidth constraints and interconnect contention. As also noted by Reviewer Beio, "the edge setting is practical and the vLLM integration with real Jetson measurements is useful". We believe that enabling effective 2–2.5-bit TP communication while maintaining competitive model accuracy, together with real-hardware validation, constitutes a meaningful contribution beyond combining established techniques.

  • W2 & Q1. Computational overhead and break-even bandwidth. Figure 13 shows substantial encode/decode cost. At what bandwidth, for example on a 10 or 40 GbE link, does VQ cease to outperform BF16?

    We acknowledge that encoding and decoding overhead may outweigh the communication savings when sufficient bandwidth is available. However, we would like to clarify that our method primarily targets communication bottlenecks caused by either limited link bandwidth or contention on shared interconnects, aiming to improve the capability of legacy and lightweight devices to support increasingly demanding AI workloads. Moreover, Figure 8 already evaluates our method under a higher-bandwidth interconnect, using two NVIDIA A30 GPUs (TP=2) connected via PCIe 3.0 ×\times 16, with a measured unloaded bandwidth of 10.5 GB/s (84 Gbps), which exceeds the nominal bandwidth of both 10 GbE and 40 GbE.

    Figure 7 and 8 demonstrates that the communication savings of index-based all-reduce can outweigh its quantisation overhead, leading to lower end-to-end prefill and decoding latency under bandwidth constraints or interconnect contention. However, the exact break-even bandwidth depends on the workload, hardware, and available communication bandwidth, and cannot be determined from these experiments alone.

  • W3. Hardware coverage. The emphasis on Jetson clusters with 1 GbE limits the relevance to server deployments.

    Our method targets communication bottlenecks caused by either limited link bandwidth or contention on shared interconnects, aiming to improve the capability of legacy and lightweight devices to support increasingly demanding AI workloads. We evaluate the proposed method on Jetson clusters connected via 1 GbE to assess its effectiveness in bandwidth-constrained collaborative inference. We have also valuated activation compression on two A30 GPUs (TP=2) under varying levels of background bandwidth occupancy, as shown in Figure 8, to emulate interconnect contention from co-located workloads. The Figure 8 demonstrate that our method benefits not only bandwidth-constrained edge clusters but also enabling more efficient TP collaboration between server-level GPUs with interconnect contention.

AuthorsOpenReview ↗

Response to Reviewer cW9x (2/2)

  • W4. Model size and the need for TP. The 1.7B and 4B models may fit on a single modern edge device.

    We acknowledge that 1.7B and 4B models can run on a single device with sufficient memory and appropriate inference settings. However, whether a model can be deployed on a single device depends not only on its parameter count but also on the memory required for the KV cache and runtime buffers, as well as the memory available on the device. The memory requirements vary with inference settings such as maximum sequence length and batch size, while background workloads and vLLM engine itself can further reduce the available memory on shared edge devices.

    As shown in Figure 1, Qwen3-1.7B-FP8 runs out of memory on a single 8 GB Jetson Orin Nano under the evaluated inference settings, while the same workload runs successfully at TP=2 and TP=4. This demonstrates that TP can be necessary even for relatively small models on devices whose nominal memory capacity exceeds the memory required to store the model weights. In this work, we use 1.7B and 4B models to quantify the end-to-end latency benefits of activation compression during TP inference on Jetson devices. We further extend the evaluation to models ranging from 4B to 13B parameters on two A30 GPUs under interconnect contention, as shown in Figure 8.

  • Q2. Asymmetric compute throttling. How do node-specific LUTs behave if background load slows some nodes more than others?

    In tensor-parallel inference, the overall latency is constrained by the slowest rank due to synchronisation during collective communication. Our index-based all-reduce performs the same operations on every rank: encoding local activations, gathering encoded indices, decoding the received indices, and summing the reconstructed activations. On homogeneous devices with the same tensor shapes and codebook sizes, these operations are expected to have similar latency in the absence of asymmetric background workloads. Node-specific LUTs do not change this symmetry. They are used because activation distributions differ across ranks, requiring each rank to encode its local activations using a separately calibrated codebook. However, asymmetric compute throttling caused by heterogeneous devices or background workloads may introduce execution imbalances and increase synchronisation delays. Our work focuses on enabling effective deployment of index-based codebook compression in TP inference, while supporting fully dynamic operating conditions, including asymmetric compute throttling across heterogeneous devices, remains a direction for future work.

  • Q3. Rare but important activation outliers. Could subsampling remove rare activation patterns that matter for reasoning in larger models?

    Thank you for raising this point. We acknowledge that uniform random subsampling may miss rare but important systematic outliers. However, a sufficiently large calibration dataset and an appropriate sampling ratio increase the likelihood of retaining representative samples of recurring outlier patterns, allowing VQ to capture them in the codebook. To assess whether subsampling compromises downstream task performance, we evaluate zero-shot accuracy across four models and six benchmarks, as reported in Table 2 of Appendix A.6. Both the 2.5-bit and 2-bit variants with our proposed calibration method achieve accuracy comparable to the uncompressed baseline. These results suggest that uniform random subsampling, with sufficient calibration data and an appropriate sampling ratio, can preserve downstream task accuracy even under ultra-low-bit activation compression.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary

We thank the reviewers for their time and careful evaluation of our submission. In response to their comments, we have addressed the following major concerns:

  1. Clarify the novelty. Our contribution lies not in introducing fundamentally new collective communication or calibration techniques, but in a low-overhead communication-path co-design that addresses accumulated activation approximation errors, enabling ultra-low-bit TP activation compression without retraining. To the best of our knowledge, this is the first work to demonstrate the practical feasibility of codebook-based vector quantisation for tensor-parallel activation All-Reduce in LLM inference.

  2. Clarify the scalability. We demonstrate that our method provides substantial latency benefits on high-bandwidth interconnects, such as NVIDIA A30 GPUs @ PCIe 3.0 ×16 (10.5 GBps). Our method achieves a 30.2% reduction in prefill-stage TTFT at TP=8 on Jetson Orin devices @ 1 GbE. While we acknowledge the scalability limitations of all-gather at larger TP degrees, these limitations do not diminish the demonstrated end-to-end latency benefits in our targeted bandwidth-constrained and interconnect-contended scenarios. Improving scalability to larger TP groups remains a direction for future work.

  3. Provide implementation details. We provide additional implementation details and pseudocode including activation sampling, calibration data collection, LUT generation, and the overall sequential calibration procedure to improve reproducibility.

  4. Evaluate calibration stability. We report the mean and standard deviation of perplexity across the validation sets of three benchmarks and update the corresponding discussion in Appendix A.5. The results suggest that calibration stability is both configuration- and model-dependent.

  5. Correct inconsistencies in manuscript and code. We correct the numerical inconsistencies in Table 2 and clarify the implementation discrepancies in the supplementary code. We confirm that these errors do not affect the reported experimental conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.