- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
Quantar is an approach for quantized tensor-parallel communication. The issue the authors address is that with dynamic scaling, scales must be communicated, and accumulation must be done after descaling and in high precision. This makes direct in-switch accumulation impossible. They solve this by using a single static scale shared across all ranks. The authors supply a theoretical section that explains why static quantization is sufficient for quantizing communication, along with an analysis of the accumulated residual error.
Critical strengths
- Quantar is a tensor-parallel communication quantization method with overall very good downstream accuracy results.
- The motivation for static quantization to speed up communication is well articulated and explained.
Critical weaknesses
- The theoretical analysis establishes a checkpoint-dependent bound on the communicated activations and uses it to motivate static quantization. However, the same normalization-based argument appears applicable to the outputs of other transformer layers, regardless of whether they are communicated. It is therefore unclear what the analysis establishes beyond the general feasibility of static activation quantization, which is already widely used and is well established. How does the theoretical analysis differ from what one could derive for static quantization of the output of each MLP or attention layer?
- The main elements that differentiate Quantar from other communication compression methods, and from static quantization in general, are that it chooses one scale for all ranks (Eq. 1) and that accumulation is done on scaled tensors. This can cause in-range rounding error or accumulation error. The consequences of these choices are not discussed in the theoretical section.
- The main text focuses on the open-loop analysis of transport error, reporting a peak error of 2.89% and a final error of 1.31%. However, Appendix C.6 shows that when subsequent layers process the perturbed residual stream, the residual trajectory deviates by 29.0–37.3% at its peak. While the paper notes that this is an internal-state metric rather than a quality metric, this substantial difference raises questions about which error measure best characterizes Quantar's numerical behavior and whether the open-loop analysis adequately captures the effects of quantization in the full model.
- The downstream accuracy results are overall very good across models and datasets. The only caveat is that MMLU-Pro drops and WikiText-2 perplexity rises on all five models. Accuracy comparisons to other communication quantization methods would be beneficial (Flash Communication, Flash Communication V2, QuickReduce, FlashInfer’s quantized all-reduce, and others). For me, to show that Quantar is novel and significant, the necessary comparison is to show that it is faster than other communication quantization methods. The claim that communicating scales and dequantizing before accumulation is slower is asserted, not shown.
- Figure 1 is only loosely related to the proposed method and addresses a specific part of the theoretical proof.
Questions for authors
Please refer to the weaknesses above.
6 comments
Response to Reviewer B1EU (2/2)
W3. Open-loop error and the closed-loop model.
"this substantial difference raises questions about which error measure best characterizes Quantar's numerical behavior and whether the open-loop analysis adequately captures the effects of quantization in the full model."
Each of the three measures in the submission answers a different question—and §3 already states explicitly which one is the full-model test:
- Transport error (open loop, §3, L. 298–318): what Quantar itself adds. §3 “isolates the transport error itself” (L. 299): κ = 1.114 vs. a 121× aligned worst case (L. 313–315), so errors behave like a random walk rather than piling up (2.89% peak, 1.31% final).
- Trajectory sensitivity (closed loop, App. C.6, L. 841–845): how strongly 121 sites react to any arithmetic perturbation. For clarity, a new unquantized BF16 switch control (FP32 accumulation, zero FP8) moves the residual stream by almost the same amount:
| Metric | WikiText | Code | Multilingual |
|---|---|---|---|
| BF16-switch deviation | 35.0% | 32.9% | 29.4% |
| Quantar deviation | 37.3% | 34.7% | 29.0% |
| Top-1 agreement (switch / Quantar, %) | 94.9 / 94.4 | 98.9 / 98.8 | 92.2 / 91.9 |
The 29–37% drift reflects deep-network sensitivity to reduction order—even in unquantized BF16—not an FP8 instability (next-token agreement within 0.5 pp).
- Task outcomes (served evaluations, §3, L. 318–320; §4, Tables 1–4): Task shifts stay inside our 1 pp paired non-inferiority margin (see W4).
W4. Comparisons with quantized communication methods.
“The claim that communicating scales and dequantizing before accumulation is slower is asserted, not shown.”
On the same B300 TP8 boundary (L. 770–775), Quantar is 1.22–1.27× faster than FlashInfer v0.7.3 and 1.13–1.86× faster than each scale-handling regime in §2.3 and Appendix A (L. 114–121, 124–135, 174–179, 708–724):
| Method / arm | Scale handling | Reduction path | Cost vs. Quantar |
|---|---|---|---|
| Quantar | Frozen shared scale; nothing computed or sent | In-switch (NVSwitch) FP8 sum, FP16 accumulation | 1.00× |
| Reconstruct-first control (Flash Comm., QuickReduce, Hansen-Palmus, Dong) | Scale work charged at zero | GPU two-shot, FP32 accumulation (L. 115–121) | 1.17–1.24× slower (8k–32k); Quantar's forward is 1.08× faster at 32k |
| FlashInfer v0.7.3 (direct run) | Per-block scales computed and sent (L. 715–718) | GPU two-shot, reconstruct before sum | 1.22–1.27× slower (8k–32k) |
| Agree-per-batch control (FP8-LM pattern) | Runtime absmax + cross-rank agreement (L. 174–179) | Quantar's in-switch kernel and epilogue | 1.26–1.86× slower (2k–32k) at identical error (3.2450% vs 3.2454%) |
| NCCL FP8 (Tables 11–12) | Shared scale; nothing sent | Ring; re-rounds the running sum to FP8 at each hop (L. 720–722) | 1.13–1.24× slower; forward 1.10× vs. 1.20× at 32k; no serving gain |
The reconstruct-first control runs the two-shot dataflow without paying for scales, so it bounds that regime from below in our build. FlashInfer, which does compute and send scales, lands just above it at every length (32k: control 1440.2 µs, FlashInfer 1486.0 µs, Quantar 1221.2 µs). QuickReduce is ROCm-only, Dong et al. implement no collective (L. 129–132) and we found no public code for Flash Comm. or Hansen-Palmus.
Why each alternative loses on the speed–accuracy trade-off:
- Reconstruct-first pays 1.17–1.27× latency for a bounded quality upside. Its lower tensor error (2.31% vs. 3.58%, Table 6) can recover at most 0.52 pp on Qwen3-32B MMLU-Pro: the lower end of Quantar's paired rerun, −0.14 pp (90% CI [−0.52, +0.24]). Elsewhere Quantar already matches BF16:
- MMLU-Pro −0.10 to −0.42 pp and WikiText-2 +0.06–0.88% (Table 1; −0.11% on Nemotron-3-Ultra NVFP4, L. 369–370);
- eight-task means within 0.1 pp (Table 2);
- RULER paired lower bounds of at least −0.81 pp through 128k;
- GSM8K-CoT with FP8 forced on every decoded token, +0.08 pp [−0.98, +1.13];
- LongBench-E and GSM8K-CoT parity (Table 4).
- Agree-per-batch pays 1.26–1.86× latency for zero accuracy gain. It uses the same kernel and has the same error, so the gap is the absmax scan and scale agreement alone.
- Round-per-hop loses on both speed and accuracy. It is 1.13–1.24× slower at the boundary, gives no serving gain (Tables 11–12), and degrades DeepSeek-V3.1 further: MMLU-Pro −0.45 vs. −0.27 pp, perplexity +0.32% vs. +0.18%.
W5. Figure 1. We will add a better diagram to explain scale and all-reduce mechanics for our final version.
We ask the reviewer to reconsider the Soundness score of 1 and the overall recommendation. If a remaining concern is decisive for rejection, please identify the affected conclusion and the evidence needed to resolve it, so we can address that point directly.
Thank you for addressing my points. My main remaining concern is that, while the theoretical section seems to be correct, I don’t find it to be a very convincing explanation for why Quantar works. Specifically, most of the claims in the theoretical section do not address anything that is specific to Quantar. For example, the bound (W1) and the in-range rounding error (W2) are general arguments for FP8 static quantization. The main difference between Quantar and other static quantization methods is selecting the maximum scale over all ranks. I would have liked to see the authors address this, for example with an analysis of how the scales/ranges differ across ranks. If one rank’s activations have a much larger magnitude than the others’, activations on other ranks might be pushed into the subnormal range and flushed to zero.
Regarding W3 - Thank you for adding the trajectory sensitivity measurement control.
Regarding W4 - Thank you for adding speedup measurements. Can you please specify how much faster Quantar’s forward pass is compared to the competing methods, for a few models?
Regarding W5 - Thank you for addressing this.
I originally gave a soundness score of 1 because of the problems with the theoretical section. In light of the rebuttal, I will increase it.
Response to Reviewer B1EU (1/2)
We thank the reviewer for describing Quantar's downstream accuracy as “overall very good” and its motivation as “well articulated and explained.”
W1. Novelty of the communication formulation.
“How does the theoretical analysis differ from what one could derive for static quantization of the output of each MLP or attention layer?”
Our claim. The activation bound is standard: RMSNorm bounds any sublayer’s input, and we don’t claim the bound as new. The contribution is how we enable communication quantization with it. The bound holds for every rank’s partial and for their sum, so one per-channel scale can be fixed offline and shared by all ranks. That is what lets FP8 partials be summed directly, including inside NVSwitch, with no runtime scale exchange. Per-layer static quantization only needs a scale valid on one device, so it never faces this cross-rank requirement.
From the view of quantization itself, static scales are usually treated as a calibration gamble, on the assumption that activation ranges depend on the input (L. 180–187). We show that in pre-norm transformers the ceiling is fixed by the weights. The question is no longer whether calibration saw the worst case, but where to place the FP8 grid under a known ceiling (L. 269–273). It also separates two errors that are usually lumped together:
Each step from submission:
Dong et al.'s offline rank-local scales still reconstruct and sum in BF16 (L. 129–132).
W2. Rounding and accumulation.
“This can cause in-range rounding error or accumulation error. The consequences of these choices are not discussed in the theoretical section.”
Both effects are addressed in the original submission. §3 discusses in-range rounding, defines transport error, and analyzes its accumulation in the open-loop replay (L. 211–213, 294–318). Appendices C.3 and C.5 specify the reduction arithmetic and derive the normal-range rounding bound. We collect the exact passages and locations below to make these connections explicit.
The 6.25% bound is per normal-range value; cancellation between rank partials can amplify error relative to the completed sum. Table 6 (L. 433–440) measures the net effect on natural activations: 3.58% median error for Quantar, 2.31% for reconstruct-first, and 41.32% for rank-local direct summation.
Across ranks, Table 9 (L. 865–869) holds the installed scales fixed: Quantar's median error stays at 3.16–3.72% over W = 2–64, while per-hop FP8 grows from 3.72% to 15.24%. Appendix D (L. 853–862) uses checkpoint weights and captured activations: W ≤ 8 is cross-checked against genuine partials and W = 16-64 uses numerical repartitioning on those real activations to evaluate reduction arithmetic and error only, with no latency claims.