acceptodds
Under review as a conference paper at ICLR 2027

Quantar: Quantizing All-Reduce for Efficient Tensor-Parallel Serving

12 views

Abstract

Quantization has reached nearly every tensor a language model stores or computes, yet tensor-parallel all-reduce still communicates in BF16. We find that this communication accounts for as much as 37% of prefill device time across models ranging from an 8B dense transformer to a 671B mixture-of-experts model, and its share grows as lower-precision computation makes the rest of inference faster. We introduce Quantar to quantize the reduction itself. Existing dynamic quantization, if applied to all-reduce, would require an extra communication round for scale agreement; without it, the reduced values cannot be mapped back to the correct sum. Quantar instead freezes communication scales calibrated once and shares them across ranks and requests. We show that transformer normalization places an input-independent bound on all-reduce activations, making one frozen scale practical across ranks. On eight B300 GPUs, Quantar speeds up the complete communication boundary by up to 1.81× and prefill by up to 1.28×, and on eight RTX PRO 6000 GPUs, performance measurements reach 1.92× and 1.61×. At matched settings, Quantar outperforms per-hop FP8 by up to 1.24× at the communication boundary and 1.10× in prefill, while avoiding its 1.54× error growth from two to eight ranks. We also show that quality remains at BF16 parity: full 12,032-question MMLU-Pro scores change by at most 0.42 percentage points across five models, and all nine benchmarks and 33 subtasks evaluated across three models remain within 2σ. Quantar also matches BF16 on LongBench-E, GSM8K-CoT, and RULER from 8k through 128k, including 85.4 versus 85.0 at 128k. Quantar's advantage grows as models become larger, more sharded, and more heavily quantized, extending low-bit gains across the inference path.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Inference, Quantization, Tensor Parallel, Communication Collectives

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_B1EUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

Quantar is an approach for quantized tensor-parallel communication. The issue the authors address is that with dynamic scaling, scales must be communicated, and accumulation must be done after descaling and in high precision. This makes direct in-switch accumulation impossible. They solve this by using a single static scale shared across all ranks. The authors supply a theoretical section that explains why static quantization is sufficient for quantizing communication, along with an analysis of the accumulated residual error.

Critical strengths
  • Quantar is a tensor-parallel communication quantization method with overall very good downstream accuracy results.
  • The motivation for static quantization to speed up communication is well articulated and explained.
Critical weaknesses
  • The theoretical analysis establishes a checkpoint-dependent bound on the communicated activations and uses it to motivate static quantization. However, the same normalization-based argument appears applicable to the outputs of other transformer layers, regardless of whether they are communicated. It is therefore unclear what the analysis establishes beyond the general feasibility of static activation quantization, which is already widely used and is well established. How does the theoretical analysis differ from what one could derive for static quantization of the output of each MLP or attention layer?
  • The main elements that differentiate Quantar from other communication compression methods, and from static quantization in general, are that it chooses one scale for all ranks (Eq. 1) and that accumulation is done on scaled tensors. This can cause in-range rounding error or accumulation error. The consequences of these choices are not discussed in the theoretical section.
  • The main text focuses on the open-loop analysis of transport error, reporting a peak error of 2.89% and a final error of 1.31%. However, Appendix C.6 shows that when subsequent layers process the perturbed residual stream, the residual trajectory deviates by 29.0–37.3% at its peak. While the paper notes that this is an internal-state metric rather than a quality metric, this substantial difference raises questions about which error measure best characterizes Quantar's numerical behavior and whether the open-loop analysis adequately captures the effects of quantization in the full model.
  • The downstream accuracy results are overall very good across models and datasets. The only caveat is that MMLU-Pro drops and WikiText-2 perplexity rises on all five models. Accuracy comparisons to other communication quantization methods would be beneficial (Flash Communication, Flash Communication V2, QuickReduce, FlashInfer’s quantized all-reduce, and others). For me, to show that Quantar is novel and significant, the necessary comparison is to show that it is faster than other communication quantization methods. The claim that communicating scales and dequantizing before accumulation is slower is asserted, not shown.
  • Figure 1 is only loosely related to the proposed method and addresses a specific part of the theoretical proof.
Questions for authors

Please refer to the weaknesses above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer B1EU (1/2)

We thank the reviewer for describing Quantar's downstream accuracy as “overall very good” and its motivation as “well articulated and explained.”

W1. Novelty of the communication formulation.

“How does the theoretical analysis differ from what one could derive for static quantization of the output of each MLP or attention layer?”

Our claim. The activation bound is standard: RMSNorm bounds any sublayer’s input, and we don’t claim the bound as new. The contribution is how we enable communication quantization with it. The bound holds for every rank’s partial and for their sum, so one per-channel scale can be fixed offline and shared by all ranks. That is what lets FP8 partials be summed directly, including inside NVSwitch, with no runtime scale exchange. Per-layer static quantization only needs a scale valid on one device, so it never faces this cross-rank requirement.

From the view of quantization itself, static scales are usually treated as a calibration gamble, on the assumption that activation ranges depend on the input (L. 180–187). We show that in pre-norm transformers the ceiling is fixed by the weights. The question is no longer whether calibration saw the worst case, but where to place the FP8 grid under a known ceiling (L. 269–273). It also separates two errors that are usually lumped together:

  1. range errors (clipping, underflow), which the bound and headroom control;
  2. in-range rounding, which is at most 6.25% per normal value and cancels across depth (κ = 1.114; L. 193–233, 314–318).

Each step from submission:

ClaimPaper evidence
Request-independent channel boundsEq. 2, L. 235–269; Table 8, L. 757–763: a 4× residual change leaves communicated magnitudes at 0.999–1.000×
Frozen common unit enables direct reduction and fusionEq. 1, L. 165–173; partial/sum calibration, L. 274–282; scan/agreement removal, L. 174–179; producer fusion, L. 321–323
Calibration transfer and error geometry are testedCross-domain/equal-budget tables, Table 5, L. 406–424; error-energy decomposition and κ = 1.114, Eq. 3, L. 300–318

Dong et al.'s offline rank-local scales still reconstruct and sum in BF16 (L. 129–132).

W2. Rounding and accumulation.

“This can cause in-range rounding error or accumulation error. The consequences of these choices are not discussed in the theoretical section.”

Both effects are addressed in the original submission. §3 discusses in-range rounding, defines transport error, and analyzes its accumulation in the open-loop replay (L. 211–213, 294–318). Appendices C.3 and C.5 specify the reduction arithmetic and derive the normal-range rounding bound. We collect the exact passages and locations below to make these connections explicit.

  • In-range rounding — §3, L. 211–213: “Between the two thresholds, round-to-nearest moves the value to the nearest point on the FP8 grid. The error is at most half the adjacent spacing, which for normal E4M3 is at most 2⁻⁴ = 6.25% of the value” (derivation: Appendix C.5, L. 818–832).
  • Transport error — §3, L. 296: “This difference includes rounding the rank-local partials and the completed sum.”
  • Depth accumulation — §3, L. 306–308: “If the perturbations remain weakly aligned, their pairwise terms cancel and comparable local errors grow as √N, not N. Whether deterministic rounding follows this behavior is an empirical question.” Eq. 3 (L. 300–304) decomposes the error energy; L. 314–318 reports κ = 1.114 and mean pairwise cosine 8.10 × 10⁻⁴. The controlled exponents are 0.500 versus 0.992 (Appendix C.6, L. 846–849).
  • Reduction accumulation — Appendix C.3, L. 807–808: “Quantar instead widens the shared-scale FP8 operands inside the reduction, accumulates them with FP16 semantics, and converts only the completed sum back to FP8.”

The 6.25% bound is per normal-range value; cancellation between rank partials can amplify error relative to the completed sum. Table 6 (L. 433–440) measures the net effect on natural activations: 3.58% median error for Quantar, 2.31% for reconstruct-first, and 41.32% for rank-local direct summation.

Across ranks, Table 9 (L. 865–869) holds the installed scales fixed: Quantar's median error stays at 3.16–3.72% over W = 2–64, while per-hop FP8 grows from 3.72% to 15.24%. Appendix D (L. 853–862) uses checkpoint weights and captured activations: W ≤ 8 is cross-checked against genuine partials and W = 16-64 uses numerical repartitioning on those real activations to evaluate reduction arithmetic and error only, with no latency claims.

AuthorsOpenReview ↗

Response to Reviewer B1EU (2/2)

W3. Open-loop error and the closed-loop model.

"this substantial difference raises questions about which error measure best characterizes Quantar's numerical behavior and whether the open-loop analysis adequately captures the effects of quantization in the full model."

Each of the three measures in the submission answers a different question—and §3 already states explicitly which one is the full-model test:

  • Transport error (open loop, §3, L. 298–318): what Quantar itself adds. §3 “isolates the transport error itself” (L. 299): κ = 1.114 vs. a 121× aligned worst case (L. 313–315), so errors behave like a random walk rather than piling up (2.89% peak, 1.31% final).
  • Trajectory sensitivity (closed loop, App. C.6, L. 841–845): how strongly 121 sites react to any arithmetic perturbation. For clarity, a new unquantized BF16 switch control (FP32 accumulation, zero FP8) moves the residual stream by almost the same amount:
MetricWikiTextCodeMultilingual
BF16-switch deviation35.0%32.9%29.4%
Quantar deviation37.3%34.7%29.0%
Top-1 agreement (switch / Quantar, %)94.9 / 94.498.9 / 98.892.2 / 91.9

The 29–37% drift reflects deep-network sensitivity to reduction order—even in unquantized BF16—not an FP8 instability (next-token agreement within 0.5 pp).

  • Task outcomes (served evaluations, §3, L. 318–320; §4, Tables 1–4): Task shifts stay inside our 1 pp paired non-inferiority margin (see W4).

W4. Comparisons with quantized communication methods.

“The claim that communicating scales and dequantizing before accumulation is slower is asserted, not shown.”

On the same B300 TP8 boundary (L. 770–775), Quantar is 1.22–1.27× faster than FlashInfer v0.7.3 and 1.13–1.86× faster than each scale-handling regime in §2.3 and Appendix A (L. 114–121, 124–135, 174–179, 708–724):

Method / armScale handlingReduction pathCost vs. Quantar
QuantarFrozen shared scale; nothing computed or sentIn-switch (NVSwitch) FP8 sum, FP16 accumulation1.00×
Reconstruct-first control (Flash Comm., QuickReduce, Hansen-Palmus, Dong)Scale work charged at zeroGPU two-shot, FP32 accumulation (L. 115–121)1.17–1.24× slower (8k–32k); Quantar's forward is 1.08× faster at 32k
FlashInfer v0.7.3 (direct run)Per-block scales computed and sent (L. 715–718)GPU two-shot, reconstruct before sum1.22–1.27× slower (8k–32k)
Agree-per-batch control (FP8-LM pattern)Runtime absmax + cross-rank agreement (L. 174–179)Quantar's in-switch kernel and epilogue1.26–1.86× slower (2k–32k) at identical error (3.2450% vs 3.2454%)
NCCL FP8 (Tables 11–12)Shared scale; nothing sentRing; re-rounds the running sum to FP8 at each hop (L. 720–722)1.13–1.24× slower; forward 1.10× vs. 1.20× at 32k; no serving gain

The reconstruct-first control runs the two-shot dataflow without paying for scales, so it bounds that regime from below in our build. FlashInfer, which does compute and send scales, lands just above it at every length (32k: control 1440.2 µs, FlashInfer 1486.0 µs, Quantar 1221.2 µs). QuickReduce is ROCm-only, Dong et al. implement no collective (L. 129–132) and we found no public code for Flash Comm. or Hansen-Palmus.

Why each alternative loses on the speed–accuracy trade-off:

  • Reconstruct-first pays 1.17–1.27× latency for a bounded quality upside. Its lower tensor error (2.31% vs. 3.58%, Table 6) can recover at most 0.52 pp on Qwen3-32B MMLU-Pro: the lower end of Quantar's paired rerun, −0.14 pp (90% CI [−0.52, +0.24]). Elsewhere Quantar already matches BF16:
    • MMLU-Pro −0.10 to −0.42 pp and WikiText-2 +0.06–0.88% (Table 1; −0.11% on Nemotron-3-Ultra NVFP4, L. 369–370);
    • eight-task means within 0.1 pp (Table 2);
    • RULER paired lower bounds of at least −0.81 pp through 128k;
    • GSM8K-CoT with FP8 forced on every decoded token, +0.08 pp [−0.98, +1.13];
    • LongBench-E and GSM8K-CoT parity (Table 4).
  • Agree-per-batch pays 1.26–1.86× latency for zero accuracy gain. It uses the same kernel and has the same error, so the gap is the absmax scan and scale agreement alone.
  • Round-per-hop loses on both speed and accuracy. It is 1.13–1.24× slower at the boundary, gives no serving gain (Tables 11–12), and degrades DeepSeek-V3.1 further: MMLU-Pro −0.45 vs. −0.27 pp, perplexity +0.32% vs. +0.18%.

W5. Figure 1. We will add a better diagram to explain scale and all-reduce mechanics for our final version.

We ask the reviewer to reconsider the Soundness score of 1 and the overall recommendation. If a remaining concern is decisive for rejection, please identify the affected conclusion and the evidence needed to resolve it, so we can address that point directly.

Thank you for addressing my points. My main remaining concern is that, while the theoretical section seems to be correct, I don’t find it to be a very convincing explanation for why Quantar works. Specifically, most of the claims in the theoretical section do not address anything that is specific to Quantar. For example, the bound (W1) and the in-range rounding error (W2) are general arguments for FP8 static quantization. The main difference between Quantar and other static quantization methods is selecting the maximum scale over all ranks. I would have liked to see the authors address this, for example with an analysis of how the scales/ranges differ across ranks. If one rank’s activations have a much larger magnitude than the others’, activations on other ranks might be pushed into the subnormal range and flushed to zero.

Regarding W3 - Thank you for adding the trajectory sensitivity measurement control.

Regarding W4 - Thank you for adding speedup measurements. Can you please specify how much faster Quantar’s forward pass is compared to the competing methods, for a few models?

Regarding W5 - Thank you for addressing this.

I originally gave a soundness score of 1 because of the problems with the theoretical section. In light of the rebuttal, I will increase it.

Reviewer_pqisOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Quantar addresses All-Reduce communication in tensor-parallel LLM inference, which often remains in BF16 even when model computation uses low precision. It proposes a frozen per-channel E4M3 scale shared across ranks and requests, allowing quantized partial outputs to be reduced in a common unit. The implementation uses a representability guard, FP16 accumulation, NVLink SHARP/NVLS, and fused residual/RMSNorm operations. The paper motivates the fixed scale using RMSNorm-based activation bounds and evaluates the system on B300 and RTX PRO 6000 platforms with several dense and MoE models. It reports up to 1.81× communication and 1.28× prefill speedups on B300, with small measured changes on MMLU-Pro, long-context, and other benchmarks.

Critical strengths
  1. The paper targets a real bottleneck that becomes more important as computation moves to lower precision. It reports end-to-end measurements on modern multi-GPU systems, rather than inferring speedups only from the reduction in byte count. The use of NVLS, FP16 accumulation, and fused producer/consumer operations makes the systems contribution concrete.
  2. The shared-scale formulation is simple and well motivated. The paper clearly explains why direct summation with rank-local scales is invalid, and it evaluates the resulting method across several models, long-context settings, and downstream tasks. These experiments make the method more credible than a communication-only simulation.
Critical weaknesses
  1. The central static-scale choice is not sufficiently justified. RMSNorm establishes a finite, checkpoint-dependent bound, but the deployed scale is the calibration maximum multiplied by a fixed headroom of η=32. The paper does not connect this choice to the theoretical bound or provide a systematic sweep over η, calibration size/domain, clipping, underflow, and tensor-parallel world size. Results for TP=16–64 are virtual re-partitionings, and the guard already increases one site’s error at TP=64. The robustness of a frozen scale table in real deployments therefore remains uncertain.
  2. The comparison with prior quantized communication methods is not sufficiently apples-to-apples. The main baselines are BF16 NCCL and per-hop FP8; the paper does not report end-to-end latency and quality against dynamic per-block systems such as Flash Communication, FlashInfer, or QuickReduce under the same hardware and transport. Quantar’s speed includes NVLS, FP16 accumulation, fusion, and a different collective, while its error is mostly compared at tensor level. This makes the claimed advantage over existing methods difficult to isolate.
Questions for authors
  1. Can the authors provide an apples-to-apples comparison with Flash Communication, FlashInfer, or QuickReduce using the same GPU, TP size, context length, collective path, and fusion boundary, reporting both end-to-end latency and downstream quality?
  2. How sensitive are accuracy, clipping/underflow, guard activation, and latency to the headroom η, calibration-token budget, calibration domain, and TP world size? In particular, are there real-hardware results beyond TP=8 rather than virtual re-partitionings?
  3. Does the method support decode, dynamic batching, and mixed request lengths, and what exact GPU/software features are required? Please also clarify whether the custom kernel and scale-table generation procedure will be released.
5 comments
AuthorsOpenReview ↗

Response to Reviewer pqis (1/3)

We thank the reviewer for recognizing that Quantar “targets a real bottleneck,” that the shared-scale formulation is “simple and well motivated,” and that the evaluations establish credibility beyond communication-only simulation.

W1 / Q2. Why a frozen scale is viable, and how the bound relates to η.

Dynamic scaling tracks input-dependent ranges. Residual-stream magnitude varies across tokens and layers; §3 (L. 235–269) shows why that magnitude need not determine the communicated range:

  1. RMSNorm bounds the input norm. Each attention and MLP/MoE sublayer input satisfies ∥y∥≤∥γ∥∞d\|y\| \le \|\gamma\|_\infty\sqrt d, independently of residual magnitude (L. 235–248).
  2. Fixed weights carry this bound through the sublayer. Cauchy–Schwarz bounds each projection (L. 250–252). Attention averages bounded values with nonnegative weights summing to one, so sequence length does not enlarge the bound (L. 253–254). In SwiGLU, ∣SiLU⁡(u)∣≤∣u∣|\operatorname{SiLU}(u)|\le |u|; MoE combines finitely many bounded experts with bounded router weights (L. 255–256). The output projection then bounds every communicated channel (Eq. 2, L. 260–268).
  3. Each pre-norm sublayer renormalizes its own input (L. 235–237). Residual-amplitude growth therefore does not compound this bound across depth. Each site has its own ceiling, fixed by its checkpoint weights; requests change alignment with projection rows, not that ceiling (L. 265–272).

Table 8 (L. 757–763) isolates this mechanism: multiplying the residual by four before RMSNorm leaves communicated magnitudes at 0.999–1.000×. Applying the same factor after normalization produces a largest branch response of 30.2–34.4×, measured as a median per-channel absmax ratio. Normalization keeps the communicated range from following residual magnitude.

The bound establishes a ceiling; calibration and η choose the operating range (L. 269–288). Sizing to the worst-case ceiling would spend E4M3's range on values absent from calibration and push small values toward subnormals. Calibration instead places the grid around observed activations. With s=ηm/Cs=\eta m/C, the format ceiling is sC=ηmsC=\eta m, and the calibration maximum maps to C/ηC/\eta. Headroom covers unseen directional variation. Increasing η expands the range and moves more small values into subnormals, while preserving the 6.25% worst-case relative-rounding bound for normal values (Appendix C.5, L. 818–832). To further validate the installed η = 32 (L. 283), the held-out Llama-3.1-8B sweep, including non-power-of-two settings, measures both sides:

  • Clipping: 0.00395% at η = 4; zero at η ≥ 12.
  • Subnormals / zeros: 2.92% / 0.18% at η = 32; 11.30% / 0.74% at η = 128.
  • Median tensor error: 2.60–2.62% across η = 6–128.

η = 32 lies on this plateau and is 2.7× the first tested zero-clipping setting, η = 12. Tables 1–5 evaluate downstream quality at the installed η; the sweep further validates that choice at the tensor level. Changing η changes stored constants while retaining the same runtime operations. The same frozen table is evaluated from 8k to 128k (Table 3).

  • Calibration domain. One WikiText table holds on code, multilingual, and QA tasks; separately, tables calibrated on three corpora at equal budget agree within reported downstream sampling uncertainty and span 0.0052 in WikiText perplexity (Table 5, L. 406–424; Appendix B, L. 735–744). Sampled telemetry records no guard engagements for the equal-budget tables (L. 423–424). Each checkpoint retains one table across all tasks and context lengths (L. 354–355).
AuthorsOpenReview ↗

Response to Reviewer pqis (2/3)

W1 / Q2 (continued).

Calibration budget: chosen per model, as in standard post-training quantization, with held-out guard telemetry as a direct sufficiency check. DeepSeek-V3.1 uses 266,240 calibration tokens and Llama-3.1-8B 1,048,576; both record zero guard clamps at deployment (DeepSeek on WikiText traffic, Llama on held-out data). On the Llama table, the η sweep in Part 1 quantifies the margin: clamps fall to zero from η = 12 through η = 128, including the deployed η = 32. Since budget enters the scale only through each channel's maximum mcm_c (L. 274–282), and a smaller sample can only lower it, the same sweep bounds budget sensitivity: each channel's maximum can fall to 37.5% (12/32) of ours before any held-out partial clamps at η = 32.

Guard: guarantees a finite FP8 sum for any input, with zero clamps on both models at deployment (App. C.2, L. 779–793). E4M3's largest value is 448; unguarded aligned inputs reach 896–3,584 at W = 2–8 and return non-finite values. Clamping each rank to 448/W keeps the sum finite independent of calibration. It changed 0 of 1.18 × 10¹¹ DeepSeek-V3.1 WikiText values (8 of 2.97 × 10¹⁰ under random-token stress) and 0 of 5.37 × 10⁸ held-out Llama-3.1-8B partials at η = 32, at ≤0.5% boundary latency.

TP scaling.

  • TP16–64 is error-only, computed with Quantar's deployed arithmetic on real activations and weights. Our node has 8 GPUs, so for W > 8 we split captured activations and checkpoint weights into W shards (matching real 8-GPU partials for W ≤ 8). We claim no speedup beyond TP8.
  • A scale table belongs to a checkpoint and TP partition, built in one offline BF16 pass (the maximum in L. 274–282 is over that partition's rank partials and completed sum; §3 will state this). Reusing a TP8 table holds through TP32 (3.16–3.72% median error, Table 9).
  • Per-rank guard margin is η/W. Calibration maps the largest observed value to 448/η and the guard caps each rank at 448/W, so calibrated values have η/W headroom: 4× at TP8 with η = 32, and 1× at W = 32, where error stays flat. Reusing the TP8 table at W = 64 gives η/W = 0.5 (cap 7 vs. calibrated maximum 14) and raises one MLP site to 7.78% (L. 860–862); rebuilding with η = 64 brings that site to 3.14%. We therefore build with η ≥ W.

W1 / Q2 at a glance: η, clipping/underflow, and calibration domain (Part 1); budget, guard activation, and TP degree (above); latency does not depend on η, which changes only stored constants, and the guard costs ≤0.5%.

W2 / Q1. Apples-to-apples comparisons and component isolation.

New measurements on the same B300 TP8 boundary (reduction, residual add, RMSNorm, casts; L. 770–775): Quantar is 1.22–1.27× faster than FlashInfer v0.7.3 and 1.13–1.86× faster than each scale-handling regime (§2.3, App. A):

Method / armScale handlingReduction pathCost vs. Quantar
QuantarFrozen shared scale; nothing computed or sentIn-switch FP8 sum, FP16 accumulation1.00×
8-bit reconstruct-first control (new; regime of Flash Comm., QuickReduce, Hansen-Palmus, Dong)Scale work charged at zeroGPU two-shot, FP32 accumulation1.17–1.24× slower (8k–32k); forward 1.02×/1.08× slower (8k/32k, both unfused)
FlashInfer v0.7.3 (new, direct run)Per-block scales computed and sent (L. 715–718)GPU two-shot, reconstruct before sum1.22–1.27× slower (8k–32k)
Agree-per-batch control (new, FP8-LM)Runtime absmax + cross-rank agreement (L. 174–179)Quantar's in-switch kernel and epilogue1.26–1.86× slower (2k–32k), same error
NCCL FP8 (Tables 11–12)Shared scale; nothing sentRing; re-rounds the sum to FP8 per hop (L. 720–722)1.13–1.24× slower; forward 1.10× vs. 1.20× at 32k; no serving gain

The control charges zero cost for scales, and shipped FlashInfer, which computes and sends them, lands above it at every length (32k: control 1440.2 µs, FlashInfer 1486.0 µs, Quantar 1221.2 µs); FlashInfer was measured at the boundary only. Sub-8-bit variants (INT4 QuickReduce, Flash Comm. V2) trade precision for smaller payloads; QuickReduce is ROCm-only, Dong et al. implement no collective (L. 129–132), and we found no public code for Flash Comm. or Hansen-Palmus.

AuthorsOpenReview ↗

Response to Reviewer pqis (3/3)

W2 / Q1 (continued).

  • Batch-1 forward vs. concurrent serving. With scale work free, the reconstruct-first control has the GPU two-shot dataflow of shared FP8 (P2P), so its boundary gap to Quantar is GPU FP32 vs. in-switch FP16 reduction. In batch-1 forward, with producer fusion off in both arms, Quantar is 1.02×/1.08× faster at 8k/32k. Separately, in 8k concurrent serving, shared FP8 (P2P) is within 1.2% of Quantar's TTFT (Table 12).

  • What Quantar's speed includes. In-switch reduction, direct FP8 summation, and the fused producer cast are not separate optimizations; each of them needs the frozen shared scale. In-switch reduction requires one scale across ranks (L. 119–121), and a fused producer cast requires the scale before the producer runs (L. 321–323). Where a component can be held fixed, we hold it for extra clarification:

    • Freezing the scale alone is worth 1.26–1.86×: agree-per-batch keeps Quantar's kernel and epilogue and changes only runtime scale handling (Part 2).
    • Fusion is a minor share, about 3% of forward time (Table 10: 1.20× with vs. 1.16–1.17× without, DeepSeek NVFP4, 16k–32k).
    • FP16 accumulation is fixed by the toolchain, not chosen by us: with CUDA 13.0, the assembler accepts only FP16 accumulation for FP8 multimem.ld_reduce on B300.
    • The gain does not only depend on NVSwitch: at matched NVLS, Quantar's forward gain over BF16 is 1.17× at 32k (1.02× at 8k, Table 11); without NVLS, shared FP8 (P2P) is 1.24× faster than NCCL BF16 at the 32k boundary, and PCIe-only RTX PRO 6000 GPUs reach 1.56–1.92× (Table 7).
  • Transport-only vs. reduction quantization: Reconstruct-first methods, FlashInfer included, quantize only the transport, reconstructing each partial before summing (L. 115–116); Quantar quantizes the reduction itself. Reconstructing first is why their tensor error is lower (2.31% vs. 3.58%, Table 6); rank-local scales are why they cannot sum compressed values in-switch (L. 119–121). Quantar's task scores leave little room for that precision to matter: Qwen3-32B MMLU-Pro is −0.14 pp vs. BF16 (paired 90% CI [−0.52, +0.24]; Tables 1–4), and per-site errors are nearly uncorrelated in the open-loop replay (§3). Per-hop FP8, which also sums in FP8, has higher error, reaching 15.24% vs. 3.52% at W = 64 (Table 9).

Q3. Decode, batching, requirements, code.

  • Batching and mixed lengths: a scale table belongs to the checkpoint and TP partition, so batch size and request lengths need no runtime adjustment. This is exercised throughout Tables 1–4 and 12 (64-concurrent suites, concurrency 16–128, 8k–128k contexts) and in mixed prefill/decode (4,096-token prompts, 256-token outputs: +7.0% output throughput at concurrency 128).
  • Decode dispatch and quality: decode stays on stock BF16 below the 8 MiB gate (256 Qwen3-32B requests = 2.5 MiB; 3.93 vs. 3.99 ms/token at 1 request, 9.86 vs. 9.78 at 256, overlapping ranges). Served quality suites ran at 64 concurrent requests (>8 MiB), logging 10⁴–10⁵ FP8 all-reduces per suite. With FP8 forced onto every decoded token, Qwen3-32B GSM8K-CoT meets the 1 pp non-inferiority margin (92.87% vs. 92.80%, paired 90% CI [−0.98, +1.13]), so on this workload the gate is a latency choice.
  • Requirements & code: CUDA, PyTorch, and NCCL already stated in App. G. The kernel and scale-table builder will be released at a future date.

Overall: These controls reinforce Quantar's central result: one offline-calibrated shared scale per checkpoint and TP partition quantizes the reduction itself, speeding up prefill and serving on both NVSwitch and PCIe while staying close to BF16 across models and 8k–128k contexts. The review characterizes its soundness concerns as “minor or fixable” and calls the formulation “simple and well motivated.” We respectfully ask that the recommendation be reconsidered in light of this evidence. If an unresolved concern remains decisive for rejection, please identify the conclusion it affects and the evidence needed to resolve it, so we can address that point directly.

Reviewer_GHWJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Quantar installs offline-calibrated, per-channel scales shared across tensor-parallel ranks and requests. This allows partial outputs to be reduced in a common quantized unit without runtime scale agreement. The implementation uses E4M3 communication with widened accumulation, applies the shared scale after reduction, and integrates quantization with the surrounding inference boundary. A pre-norm argument establishes finite request-independent activation bounds; practical calibration uses headroom rather than those worst-case bounds. Experiments span model quality, long contexts, cross-corpus calibration transfer, communication latency, and prefill on B300 and RTX PRO 6000 systems.

Critical strengths
  1. A coherent algorithm–system solution to a concrete bottleneck. Shared scales make direct reduction arithmetically meaningful, while widened accumulation avoids repeatedly rounding partial sums to FP8. Appendix C specifies FP16 accumulation and a representability guard. Table 7 and Appendix F connect this design to real prefill and communication-boundary measurements rather than compression ratios alone.
  2. Substantial validation, including useful controls. The paper evaluates full MMLU-Pro, additional tasks, long contexts, calibration transfer, and matched transport. In particular, Table 11 separates some transport effects: on DeepSeek at 32k, the reported 1.20× overall forward speedup versus 1.03× for BF16 NVLS corresponds to about 1.17× over the matched transport. The paper also labels larger-world-size experiments as virtual repartitions, which is appropriate transparency (pp. 7–9; Appendices D–F).
Critical weaknesses
  1. Decoding doesn't count into the paper; I am worried about we need better policy here.
  2. Some mechanistic and equivalence conclusions are stronger than the corresponding tests. The open-loop replay isolates additive transport errors, whereas Appendix C.6 reports closed-loop residual deviations reaching 29.0–37.3%. This does not contradict good output quality, but means the near-random-walk replay is not a general closed-loop stability argument. Likewise, remaining within twice one arm's standard error is not a paired equivalence test. These are limitations of interpretation, not evidence that the reported accuracy gains or parity are false (Appendices C.6 and E).
Questions for authors
  1. What changes across a headroom sweep in clipping frequency, subnormal/zero frequency, task quality, and boundary latency? When changing TP degree, is recalibration required, recommended, or unnecessary?
  2. Can you report paired task-score confidence intervals with a predefined acceptable quality loss, instead of interpreting the current 2σ check as equivalence?
  3. For the PCIe implementation, can you separate the benefit of shared scales from widened accumulation and kernel fusion under identical transport and accumulation settings? The existing B300 matched-transport results are useful and should be retained.
  4. Could you report a per-component time breakdown (communication boundary vs. all other kernels) for each arm of Table 11 at 8k and 32k, for DeepSeek-V3.1 and Llama-3.1-8B, together with the spread over repetitions? Under Amdahl's law, Table 11 implies very different boundary shares across arms (≈0.90 for BF16 over NVLS vs. ≈0 for per-hop FP8 at 8k). Llama's 1.28× would require a share of at least ≈0.48 even at a 1.81× boundary speedup, above the stated 37% maximum. Please also report the measurement behind the 1.81× headline (L. 466 places it within 2k–32k). Which weights does the upper DeepSeek block of Table 10 use, given that its 16k/32k speedups (1.20×) match neither weight-precision row?
  5. For checkpoints other than DeepSeek-V3.1, in particular Llama-3.1-8B at 8k (the 1.28× headline), what forward speedup does BF16 over NVLS give? Does the BF16-NVLS arm use the same fused scale/residual/RMSNorm kernel as Quantar? Given that shared FP8 over P2P is within about 1% of Quantar in Table 12, how much of Quantar's gain do you attribute to the in-switch reduction itself?
  6. why no decode? Could you add decode results?
  7. Several headline efficiency numbers cannot be traced to reported measurements. The abstract's 1.81× boundary speedup (L. 022; also L. 065) is placed in a "2k to 32k" range (L. 466), but no table reports boundary results at 2k or at any length other than 8k and 32k, where the only per-arm values (Table 11, DeepSeek-V3.1) are 1.54× and 1.46×. L. 466–467 states that "the full context-length, weight-precision, and transport sweeps are reported in Section F", yet Tables 10–12 contain no boundary sweep over context length. why such happens?
  8. The latency results contain internal inconsistencies of a few percent and report no dispersion. Under a simple time-share (Amdahl) model, a forward speedup F obtained from a boundary speedup B requires the boundary to occupy f = (1 − 1/F)/(1 − 1/B) of the baseline time. Applied to Table 11 (L. 949–952), this gives f ≈ 0.90 for BF16 over NVLS at 8k (collective 1.09×, forward 1.08×) but f ≈ 0 for per-hop FP8 (1.24×, 1.00×), and f ≈ 0.53 for Quantar at 32k (1.46×, 1.20×). The last value exceeds the 29–37% All-Reduce share stated in the Introduction (L. 049–050), and all three conclusions survive rounding of the two-decimal speedups. Likewise, if the 37% maximum and the boundary speedups measure the same quantity, a 1.81× boundary speedup could raise prefill speed by at most ≈1.20×, yet Table 7 reports 1.28× for Llama-3.1-8B at 8k, which would require f ≥ 0.48. This points to measurement noise, differences between the isolated collective benchmark and in-situ behavior, or forward-time effects outside the boundary (e.g., the fused producer path, L. 462). In Table 10, the upper DeepSeek block reports 1.20× at both 16k and 32k, whereas the weight-precision block reports 1.09×/1.08× (FP8) and 1.16×/1.17× (NVFP4) at the same lengths (L. 926–935), so the upper block matches neither. Llama-3.1-8B's speedup is non-monotonic in context length (1.28×, 1.08×, 1.20×, 1.10×, 1.07× from 8k to 128k), and the main text shows only 8k and 32k. The protocol reports medians of "at least three repetitions" (L. 979), and Table 12 ratios of medians over three repetitions, without any spread. Effects as small as the 8k matched-NVLS gain (1.10/1.08 ≈ 1.02×, L. 464) therefore cannot be distinguished from run-to-run variation.

Other issues:

  • L. 023: "eight RTX 6000 Pros" should read "eight RTX PRO 6000 GPUs", the name used in Table 7 and Appendices F–G.
  • BF16/FP8 are typeset in small capitals in some places and in regular capitals in others, sometimes within one sentence (e.g., L. 012 vs. L. 065; L. 356, "BF16 All-Reduce" vs. "per-hop FP8"). The collective is written "all-reduce" (L. 012, 017, 020), "All-Reduce" (from L. 046), and "AllReduce" (FlashInfer reference, L. 543–544). Please standardize.
  • The same object goes by different names: "sum-safety guard" (L. 285) vs. "representability guard" (L. 424, 742, 860); "complete communication boundary" (L. 022) vs. "the All-Reduce" (L. 065) vs. "the collective itself" (L. 465); "Quantar (shared FP8)" in Table 11 vs. a separate "shared FP8 (P2P)" arm in Table 12; "Multi-hop (1)" (Table 3, L. 384) vs. "variable tracking" (L. 380) vs. "multi-hop tracing" (L. 883). Please define the boundary once and use one label per arm.
  • Appendices are cited as "Section A", "Section C.5", etc. (e.g., L. 053, 132, 212, 285, 320, 373, 424, 429, 459, 467); "Appendix A" would avoid confusion with main-text sections.
  • Notation: e denotes both the reconstruction error e(x; s) (L. 205–206) and the exponent in Appendix C.5, and both meanings appear in one display (L. 824–825). z denotes both the output-projection input (L. 257) and the tree-node sums (L. 800). Rounding is written 𝒬(·) at L. 081 but ℛ₈(·) at L. 800, and C (largest E4M3 magnitude) and 𝒞 (calibration corpus, L. 274) are easy to confuse. Figure 1 labels the normalized vector u, which the text never defines. Only three displays are numbered, so the calibration and scale formulas (L. 276–281) cannot be referenced by number.
  • Figure 1 (L. 216–229): the caption's second sentence (projection rows carrying the bound to each channel) is not depicted, and the panel labels use a sans-serif, non-math font unlike the body. A diagram of the actual data path (epilogue cast → multimem.ld_reduce with FP16 accumulation → multimem.st → scale/residual/RMSNorm kernel) would be more informative.
  • Table 2 (L. 342–349) is set in a much smaller font than the other tables and is hard to read at print size. It also omits the standard errors on which the 2σ rule (L. 373) depends.
  • Table 3 caption (L. 379–380) ends mid-list without a period and mentions neither the QA category nor the Llama row. In the Llama "Average" row, the 128k cell is the 11-task core rather than the 13-task average (L. 376–377); please mark it.
  • Table 5 (L. 409–414): the two rows labeled "WikiText-103" (L. 412), presumably the deployed table and the equal-budget refit, are indistinguishable. The text cites "reported sampling uncertainty" (L. 423) and "measured task uncertainty" (L. 741–742), but the table shows no uncertainties.
  • L. 422–424: the line break leaves ", and multilingual-calibrated" at the start of a line after "code-"; rephrasing (e.g., "tables calibrated on WikiText, code, and multilingual text") avoids this. "representability- guard" (L. 424) has a stray space.
  • L. 461: "reduces prefill latency by 1.10–1.28×" mixes a reduction with a speedup factor. Suggested: "speeds up prefill by 1.10–1.28× (9–22% lower latency)".
5 comments
AuthorsOpenReview ↗

Response to Reviewer GHWJ (1/3)

We thank the reviewer for calling Quantar “a coherent algorithm–system solution to a concrete bottleneck” with “substantial validation, including useful controls,” and for noting that the quality points are “limitations of interpretation, not evidence that the reported accuracy gains or parity are false.”

W1 / Q6. Policy for decode.

  • Size-gated collective dispatch is standard practice in distributed runtimes: small messages are latency-bound; large messages are bandwidth-bound. NCCL selects algorithms and protocols by message size (NVIDIA). Quantar follows this principle: above 8 MiB it selects FP8; below that threshold it retains the stock path.
  • Prefill exposes the bandwidth bottleneck. Each 8k Qwen3-32B collective carries about 80 MiB before compression. On DeepSeek-V3.1, all-reduce occupies 37% of prefill device time with NVFP4 weights, up from 29% with FP8 (L. 049–050). Table 7 (L. 443–455) reports 1.09–1.28× prefill speedup; Table 12 (L. 961–968) reports 1.07–1.12× TTFT speedup.
  • Small decode messages favor the stock path. They carry about 10 KiB per request. Forcing BF16 and FP8 through the switch gives 5.08 and 5.22 ms/token, versus 3.99 ms on the stock path (Qwen3-32B, one request). Both one-request and 256-request decode batches select the stock BF16 collective: at 256 requests the Qwen3-32B payload is 256 × 5,120 × 2 bytes = 2.5 MiB, below the 8 MiB gate. Latency is 3.93 vs 3.99 ms/token at one request and 9.86 vs 9.78 at 256 requests (Quantar vs BF16); the 256-request repetition ranges overlap. These runs test stock-path dispatch.
  • Mixed prefill/decode serving: with 4,096-token prompts and 256-token outputs, Quantar gives 7.0% higher output throughput at concurrency 128.
  • FP8 quality during autoregression. We forced FP8 onto every decoded token, Qwen3-32B's GSM8K-CoT change is +0.08 pp (paired 90% CI [−0.98, +1.13]), meeting the 1 pp non-inferiority criterion. This validates the frozen-scale formulation in the tested autoregressive workload; small-message dispatch is chosen for latency.

W2 (closed loop). What the 29–37% measures.

§3 isolates transport-error accumulation by holding branch outputs fixed (L. 298–299). Its replay reports peak error 2.89% and κ = 1.114; served evaluations measure closed-loop quality (L. 316–320). Appendix C.6 (L. 841–845) characterizes the trajectory deviation, while Table 4 (L. 397–403) measures generated outputs directly.

To further validate the trajectory interpretation in Appendix C.6 (L. 841–845), we compare against BF16 switch reduction with FP32 accumulation. Peak residual deviations and top-1 agreement with the BF16 reference are:

PromptBF16-switch deviationQuantar deviationTop-1 agreement with BF16 (switch / Quantar)
WikiText35.0%37.3%94.9% / 94.4%
Code32.9%34.7%98.9% / 98.8%
Multilingual29.4%29.0%92.2% / 91.9%

Trajectory displacement alone does not identify an FP8-specific instability: the BF16 control produces comparable displacement, with next-token agreement within 0.5 pp of Quantar. Open loop isolates transport-error accumulation; served benchmarks measure its downstream effect.

W2 (quality) / Q2. Paired confidence intervals.

Tables 1–3 quantify downstream quality (L. 367–377): MMLU-Pro changes ≤0.42 pp across five models, eight-task mean changes ≤0.1 pp across three models, and DeepSeek's RULER-core changes ≤0.1 pp through 128k. Each checkpoint uses one frozen table (L. 354–355). The 2σ check in Appendix E (L. 874–880) serves as a per-task screen; the paired intervals below quantify acceptable loss directly.

All served quality runs executed at 64 concurrent requests (and 8k–128k prompts on RULER), where batched prefills exceed 8 MiB and rank-0 telemetry logged 104–10510^4\text{–}10^5 FP8 all-reduces per suite.

To further validate the quality results in Tables 1–3, the requested paired tests use a 1 pp acceptable-loss margin. For Quantar − BF16, the paired 90% confidence intervals give:

  • MMLU-Pro, Qwen3-32B, paired rerun: −0.14 pp [−0.52, +0.24].
  • RULER, 8k–128k: lowest lower endpoints −0.63 pp (DeepSeek-V3.1) and −0.81 pp (Llama-3.1-8B).
  • GSM8K-CoT, Qwen3-32B, FP8 throughout decode: +0.08 pp [−0.98, +1.13].

The original MMLU-Pro run retained aggregate scores; the paired rerun logged per-question outputs. Table 1's −0.39 pp lies within the rerun's 90% CI [−0.52, +0.24].

Every lower endpoint exceeds −1 pp, establishing non-inferiority within the stated margin at one-sided α = 0.05 for each tested comparison. These paired bounds provide the quantitative basis for quality preservation in these comparisons; 2σ screens task-level outliers.

AuthorsOpenReview ↗

Response to Reviewer GHWJ (2/3)

Q1. Headroom and changes in TP degree.

Headroom: η = 32 sits on a measured plateau. The checkpoint fixes the ceiling, calibration places the grid, and η supplies headroom (L. 266–288). On held-out Llama-3.1-8B:

  • Guard clamps: 0.00395% at η = 4, and zero from η = 12 onward, so η = 32 leaves a 2.7× margin. At deployment, the guard clamped no values on DeepSeek-V3.1 WikiText traffic or on held-out Llama.
  • Subnormals / zeros: 2.92% / 0.18% at η = 32, rising to 11.30% / 0.74% at η = 128.
  • Median tensor error: flat at 2.60–2.62% across η = 6–128.
  • Quality / latency: Tables 1–5 evaluate quality at η = 32. Changing η changes only stored scale values, not runtime operations, so latency does not depend on it.

TP degree: recommended per partition; in our tests, reusing a table held while W ≤ η and needed a rebuild once W > η. Standard deployment calibrates once per checkpoint and target TP partition, with η ≥ W; calibration records that partition's rank partials and completed sum (L. 274–282). The ratio η/W gives the guard's headroom over that observed range: 4× at TP8 with η = 32. Separately, Table 9 tests cross-partition reuse numerically: the unchanged TP8 table stays nearly flat through W = 32, and at W = 64 one site reaches 7.78% (L. 860–862), which rebuilding with η = 64 reduces to 3.14%. Results beyond TP8 are virtual repartitions, not hardware measurements. The guard keeps the sum finite regardless of calibration (L. 779–788).

Q3. Shared scales versus accumulation and fusion without NVLS.

Without the switch, when doing reduction on the GPUs, we isolate each factor on B300: on that non-NVLS path (GPU two-shot over P2P):

  • Shared scales alone: with transport, epilogue, and FP32 accumulation identical in both arms, and no fusion (the boundary times the cast as a standalone kernel), shared FP8 is 1.44–1.50× faster than BF16 at the 8k–32k boundary. Neither accumulation width nor fusion contributes to this gain.
  • Fusion is part of the technique. As is standard in quantized inference, the FP8 cast is fused into the producer GEMM so quantization delivers its full benefit; Quantar's frozen scale is what makes this possible for communication (L. 321–323). In the full model (DeepSeek, NVLS), it adds about 3% of forward time: 1.20× with vs. 1.16–1.17× without at 16k–32k, and 1.10× either way at 8k (Table 10; blocks as described in Q4).

Q4. In-model boundary shares and Table 10 configurations.

Table 11's caption (L. 944–946) distinguishes the communication boundary from forward wall time; L. 979–980 specifies separate forward and CUDA-event measurements. The isolated boundary includes reduction, residual addition, RMSNorm, and casts (L. 770–775). In the model, Quantar's send-side cast fuses into the producer GEMM's epilogue (L. 321–323, 462) and is accounted for with the producer; the isolated benchmark times it as a standalone kernel. The BF16 NCCL baseline needs no FP8 send cast and has similar per-site boundary time at 32k (≈1.7 ms in-model versus ≈1.8 ms isolated). Producer fusion therefore contributes to the higher in-model boundary speedup (1.78× versus 1.46×); net of the cost it adds to the producer, fusion is worth about 3% of forward time (Table 10, Q3). Wall-clock time also includes time outside GPU kernels—synchronization, framework runtime, and idle periods—which differs by arm.

One instrumented DeepSeek-V3.1 session gives the component snapshot below: boundary / other GPU work, in ms. Table 7 and Q8 report the dedicated forward wall-time campaign and its repetition spread. The component snapshot supplies attribution; its counters are kept separate from those wall-time medians.

ContextBF16 NCCLBF16 NVLSPer-hop FP8Quantar
8k54.2 / 86.249.5 / 86.244.4 / 86.131.5 / 86.2
32k207.4 / 380.1188.2 / 380.0147.4 / 384.9118.6 / 386.4

The relevant comparisons are:

  • DeepSeek, 32k: the original Table 7 campaign gives an in-model boundary of 207.9 → 117.0 ms (1.78×) within a 591.6 ms baseline forward: 207.9 / 591.6 = 35.1% of wall time. The f ≈ 0.53 estimate combines the 1.20× forward result with the isolated 1.46× benchmark. Using the measured in-model 1.78× gives f ≈ 0.38 under the fixed-remainder assumption; wall time also includes arm-specific execution outside the boundary. The isolated ratio therefore cannot determine the in-model boundary share.
AuthorsOpenReview ↗

Response to Reviewer GHWJ (3/3)

Q4 (continued).

  • Llama, 8k: its smaller compute footprint leaves a larger relative communication share: 16.53 / 36.76 = 45.0%. The collective decreases to 8.88 ms (1.86×), giving 1 / (0.55 + 0.45 / 1.86) = 1.26×, close to the measured 1.28×. A separate four-arm 8k trace gives collective / other GPU time (ms): BF16 NCCL 16.5 / 15.4; BF16 NVLS 12.7 / 15.4; per-hop FP8 14.2 / 16.5; Quantar 9.0 / 15.6. The abstract's "as much as 37%" figure (L. 013–015) refers to DeepSeek-V3.1, as quantified in L. 049–050; we will name DeepSeek-V3.1 explicitly in that phrase.
  • Table 10 (L. 926–935) reports two experiments: the context-length block's DeepSeek rows use NVFP4 with producer-side cast fusion; the weight-precision block compares FP8 and NVFP4 without that fusion, each against its matched BF16-communication baseline. The caption will identify both.

Q5. Matched kernels and the switch's contribution.

Llama-3.1-8B at 8k: the 1.28× is stated against BF16 NCCL, the unmodified baseline (L. 356). In a new 8k-only run (two launches, 14 warm passes per arm), medians [min–max] are BF16 NCCL 36.4 [35.3–53.4], BF16 NVLS 32.1 [31.5–34.4], and Quantar 28.5 [27.9–29.7] ms: BF16 NVLS gives 1.13× over BF16 NCCL, and Quantar adds 1.13× over matched NVLS. In the four-arm trace above, the collective falls 16.5 → 12.7 → 9.0 ms, so transport and shared FP8 each account for about half of the collective-time saving. On Llama, BF16 NVLS and Quantar share the in-switch reduction path. On DeepSeek, BF16 NVLS runs the same in-switch reduction and fused scale/residual/RMSNorm kernel as Quantar, compiled for BF16; in the 32k profile in (2/3), switching transport saves 19.2 ms of boundary time (207.4 → 188.2), and the shared-FP8 path, including producer-side fusion, saves a further 69.6 ms (188.2 → 118.6).

P2P versus in-switch paths: at the 32k isolated boundary, Quantar is 1.46× and shared FP8 over P2P 1.24×, both vs. BF16 NCCL. In batch-one forward, Quantar is 1.02× (8k) and 1.08× (32k) faster than the reconstruct-first GPU control, with producer fusion disabled in both arms; this comparison includes GPU FP32 versus in-switch FP16 reduction (pqis, 3/3). In 8k concurrent serving (Table 12), the two transports differ by at most 1.2% TTFT.

Q7. The 1.81× measurement.

It is the 4k isolated boundary at DeepSeek's hidden size, 7,168: 342.1 [342.0–352.5] → 189.2 [188.1–190.2] µs (n = 4), or 1.81×, under the CUDA-event protocol in L. 979–980. The 2k / 4k / 8k / 16k / 32k sweep is 1.70 / 1.81 / 1.53 / 1.47 / 1.46×. Table 11's 1.54× at 8k is a separate run. Thank you for pointing this out; we will add this boundary sweep to Appendix F.

Q8. Repetition spread and long-context scaling.

The forward protocol interleaves configurations and drops cold passes (L. 978–979). For the reported cells, median [min–max] latency in ms is:

  • DeepSeek, 32k: 591.6 [590.5–602.7] → 492.3 [491.9–493.0], nine repetitions.
  • Llama, 8k / 16k / 32k: 36.8 [36.3–37.4] → 28.6 [27.6–28.7]; 66.5 [66.4–67.8] → 61.3 [60.5–61.7]; 122.0 [121.6–123.6] → 102.0 [101.8–102.6], three repetitions each.

The listed BF16 and Quantar ranges do not overlap. Llama's 16k cell (1.08×) is the largest prefill size covered by SGLang's piecewise CUDA graphs in this configuration; we will flag it in Table 10. Beyond 16k, Llama's speedup follows the expected long-context trend, from 1.20× at 32k to 1.10× at 64k and 1.07× at 128k, as attention's quadratic work outgrows the linear all-reduce volume; both models retain 1.07–1.13× at 128k (Table 10).

We will standardize GPU names, all-reduce terminology, and symbols (one name per object and arm, numbered displays, "Appendix" citations), and replace Figure 1 with a data-path diagram. Table 2 will show standard errors at a readable size; Table 3 will identify the 128k core; Table 5 will distinguish deployed/refitted tables and show uncertainties. The caption and wording corrections at L. 023, 379–380, 422–424, and 461 will be incorporated.

The review characterizes its soundness concerns as “minor or fixable” and credits the paper's validation and controls. We ask that the score be reconsidered in light of the evidence and clarifications above. If any unresolved concern remains decisive for a rejection recommendation, we would appreciate knowing which conclusion it calls into question and what evidence would resolve it, so we can address that point directly.

Comments on the paper

1 comment

As only a few days remain in the author-reviewer discussion period, I encourage reviewers to review the authors' rebuttal and actively participate in the discussion. In particular, please clarify which concerns have been addressed and identify any remaining concerns or questions that may require further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.