- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RONA, a post-training quantization framework for universal multimodal embedding (UME) models under the challenging W4A4 setting. The authors observe that existing PTQ methods stay near full precision at milder bit-widths but degrade substantially at W4A4. They argue that preserving retrieval geometry, rather than only layer-wise reconstruction error, is essential. RONA combines group-wise learned orthogonal rotations, a quantization-noise-aware safe subspace for additive weight reparameterization, and learned per-layer absorption coefficients. These are calibrated with full-precision query/target embedding anchoring and positive-pair attraction. On Qwen3-VL-Embedding-8B across 78 MMEB-V2 tasks, RONA reaches 73.55 under W4A4, which is 3.65 points above the strongest baseline and 94.4% of full precision. It also improves the strongest baselines on two 2B backbones by 6.20 and 2.29 points, supported by ablations and retrieval-geometry analyses.
Critical strengths
The main strengths of the paper are that (i) it studies an important and underexplored problem with a well-motivated, retrieval-specific perspective on quantization error, and (ii) its experimental comparison is carefully controlled and supported by informative mechanistic analysis.
First, the paper addresses aggressive W4A4 quantization of universal multimodal embedding models, a practically important setting that has received little attention. Its central observation is well motivated: an embedding encoder represents each input as a single point used for nearest-neighbor retrieval, so errors of similar magnitude can affect retrieval margins very differently depending on their direction. The paper also points out that directions that look safe under clean activations can become harmful once coupled with activation-quantization noise. This offers a useful perspective beyond minimizing layer-wise reconstruction error.
Second, the empirical evaluation is unusually well controlled. All baselines share the same checkpoint, calibration records, token allocation, quantizer definitions, grouped-GPTQ backend, clipping-calibration objective, and refresh schedule, while each retains its own transformation (Appendix B.7). This makes the comparison considerably more credible than ones using unmatched implementations. The norm-matched intervention directly supports the paper's main hypothesis: removing the activation-noise coupling component while keeping the perturbation magnitude fixed reduces the ranking-flip rate from 1.14% to 0.56%. Consistent gains on three backbones (+3.65, +6.20, and +2.29 points) further support the robustness of the results.
Critical weaknesses
Most of the gain appears to come from components other than the proposed noise-aware absorption. With absorption fully disabled, RONA already reaches 72.66 under the shared protocol (Table B.5). That is 2.76 points above DuQuant and 3.69 points above SpinQuant, which also learns rotations. Noise-aware absorption, the paper's central contribution, adds only 0.89 points. Roughly three quarters of the headline 3.65-point improvement thus seems to come from the rotation and calibration pipeline, yet the paper does not analyze why this pipeline outperforms existing learned-rotation methods. This is consequential because it leaves unclear what the actual source of the improvement is, and whether the paper's main claim about noise-aware directions is what drives the results. The results also come from a single run with 25 calibration pairs, so the small absorption-related gains are hard to assess without variance estimates.
The deployment motivation is not supported by measured evidence. The paper is motivated by reducing serving cost, but all results use simulated quantization: weights are reconstructed before PyTorch Linear operations, and activations are quantized and then dequantized. RONA also keeps dense FP32 group rotations (about 0.8 GB for the 8B model) in the inference path. No end-to-end latency, throughput, or peak memory is reported. This is consequential because the extra rotation cost could offset part of the W4A4 efficiency benefit once optimized low-bit kernels are used. The paper therefore demonstrates improved accuracy, but not the practical serving advantage it claims.
Questions for authors
-
What explains the large gain of RONA's rotation pipeline over the learned-rotation baselines? With absorption disabled, RONA reaches 72.66, compared with 68.97 for SpinQuant and 69.90 for DuQuant under the shared protocol. Could the authors clarify which design choices are responsible for this gap, for example, the 128-channel group-wise block structure, the Hadamard initialization, the Cayley parameterization, or the A4/RTN-aware rotation objective? An ablation that varies these factors individually would help readers understand where the main improvement comes from. It would also clarify whether the noise-aware absorption or the rotation design is the primary contribution.
-
Can the authors report measured deployment benefits? Could the authors report latency, throughput, and peak device memory for BF16, a standard W4A4 baseline, and RONA, using either a native or a representative fused implementation? Relatedly, could they evaluate the noise-aware absorption pipeline on top of a fixed fast rotation (e.g., Hadamard) instead of learned dense rotations? If most of the accuracy remains, this would provide a more deployment-friendly variant. If not, report the accuracy-versus-runtime trade-off for the dense rotations.
-
How robust are the results to calibration sampling, seeds, and initialization? Could the authors report the mean and standard deviation over several random calibration draws and seeds on Qwen3-VL-Embedding-8B? Could they also compare all-zero, random, and the stored initialization of the absorption coefficients? It would also help to clarify how the initial coefficients for Qwen3-VL-Embedding-2B were obtained. Finally, could the authors report the compute cost of the preparatory initialization search, which the baselines do not receive?
-
Could the authors provide a more direct comparison with OSAQ-style subspaces? Even without an official OSAQ implementation, could the authors compare the following under the same absorption solver and calibration pipeline: (i) a curvature/Hessian-based subspace, (ii) the clean-activation-only subspace, and (iii) the proposed clean-plus-noise subspace? This would isolate the contribution of the noise-aware direction criterion.
-
Why does the calibration objective not use negative candidates or margins? Have the authors considered adding hard negatives or a direct margin-preservation term based on the full-precision encoder's nearest irrelevant target? A comparison would clarify whether the positive-only objective was chosen for simplicity or stability, or because explicit ranking supervision adds little benefit.
-
How sensitive is the method to the safe-subspace rank and the noise coefficient? The main configuration uses rank 32 and γ = 1. The γ = 0 ablation is useful, but could the authors show results for several positive γ values and subspace ranks to indicate whether the chosen operating point is robust?
-
Does the method generalize beyond the MMEB-V2 calibration distribution? Could the authors calibrate on one subset of task families and evaluate on held-out families, or on an independent multimodal retrieval benchmark? This would indicate whether the estimated safe directions reflect general quantization structure rather than benchmark-specific activation statistics.
13 comments
Response to Reviewer SGuB (2/8): W1 & Q1, the role of noise-aware absorption; variance
W1 / Q1 (continued): noise-aware absorption decides whether absorption helps or hurts.
2. The 0.89 understates the effect of the noise-aware criterion. Tables 2 and B.5 of the submission give (78 tasks):
| Configuration | All (78) | Source |
|---|---|---|
| RONA rotation, no absorption | 72.66 | Table B.5 |
| + absorption along clean-only directions (γ = 0) | 72.25 | Table 2 |
| + absorption along noise-aware directions, fixed α | 73.03 | Table B.5 |
| + learned α (full RONA) | 73.55 | Table 1 |
- Same absorption, same solver, only the direction criterion changes. With clean-only directions, absorption is 0.41 below no absorption; with the noise term it is 0.89 above. The noise term thus accounts for a 1.30-point swing, which is the contribution of the noise-aware criterion itself.
- Strictly monotone chain. No absorption 72.66 < fixed α 73.03 < learned α 73.55, consistently across the three modalities (+0.79 / +1.17 / +0.84 on image / video / document tasks).
- Share of the remaining gap. Without absorption the model is still 5.27 points below full precision (77.93 − 72.66); noise-aware absorption recovers about 17% of it, a substantial fraction of the error left after a strong rotation.
- Direct mechanistic evidence. The equal-norm intervention of Figure 4 fixes the norm of the perturbation and changes only its direction. Removing the noise-coupled component halves the ranking-flip rate (1.14% → 0.56%), whereas a random direction of the same norm barely changes it (1.09%). Whether retrieval breaks is decided by whether the perturbation lies in noise-sensitive directions.
- New evidence.
- Across all 12,303 quantization groups, the clean-only safe basis carries as much quantization-noise energy as a random direction (0.988, random = 1); RONA's basis lowers it to 0.794, in every group.
- OSAQ, re-implemented from its paper with the clean-curvature criterion, reaches only 56.52 under the same W4A4 setting, against 73.55 for RONA, which wins on all 78 tasks.
3. The advantage grows with activation noise. If RONA's gain came only from a strong rotation, its lead would be roughly constant across activation precisions. Instead it grows as activation noise grows:
| Setting | RONA vs. strongest baseline | Source |
|---|---|---|
| W4A16 (activations not quantized) | on par (75.58 vs. 75.93) | Table E.1 |
| W4A8 | +0.08 | Table E.1 |
| W4A4 | +3.65 | Table 1 |
| W3A3 | +44.95 (59.83 vs. DuQuant 14.88) | new; RONA and DuQuant change only the bit width, all else as in W4A4; 78 tasks |
Without activation quantization the noise term vanishes and RONA reduces to an ordinary reparameterization with no advantage; as activation noise appears and grows, the advantage grows with it. This is exactly what the noise-aware-direction argument predicts.
4. Variance. We drew two further independent calibration sets (one pair per subset, 25 pairs), recalibrated from scratch and evaluated all 78 tasks. Over the three calibration draws RONA scores 73.55, 73.82 and 73.59 (mean 73.65 ± 0.15), leading DuQuant (69.90) by +3.65, +3.92 and +3.69. The standard deviation is far smaller than the lead and than the 1.30-point swing produced by the noise term.
Response to Reviewer SGuB (3/8): W2 & Q2, measured deployment benefit; absorption on a fixed fast rotation
W2 / Q2: measured latency, throughput and peak memory for BF16, standard W4A4 and RONA.
1. Measured deployment benefit. We implemented the deployed W4A4 path with real low-bit kernels and measured the full forward pass of Qwen3-VL-Embedding-8B on one A800-80GB.
Implementation
- Activation kernel. Reads each activation once, applies the rotation per 128-channel group on tensor cores, then the asymmetric per-token, per-group 4-bit quantizer. With the rotation switched off it is the standard W4A4 input path, which isolates the rotation cost.
- GEMM kernel. INT4 tensor cores (
mma.sync.m16n8k64.u4.s4); after every 128-wide K group the int32 tile is rescaled in fp32 by that group's scales. - Correctness. 0.16% relative error against a float64 reference. With all 368 Linear layers replaced, the mean query/target cosine to the BF16 model is 0.915/0.933, versus 0.918/0.933 for the simulated quantization used in the paper, so the measured speed represents the paper's model.
Measurements (batch 16)
| batch 16 | BF16 | Standard W4A4 (no rotation) | RONA |
|---|---|---|---|
| Image latency (ms) | 235.0 | 284.8 | 304.7 |
| Document-page latency (ms) | 990.6 | 1185.0 | 1268.8 |
| Video latency (ms) | 1109.5 | 1331.3 | 1426.7 |
| Image throughput (items/s) | 68.1 | 56.2 | 52.5 |
| Linear weight memory (GiB) | 14.00 | 3.94 | 4.32 |
| Peak memory, video (GiB) | 18.09 | 8.07 | 8.46 |
- Memory. Weights −69%, peak memory −53% to −63%.
- Storage of the rotation. The implementation stores the rotation blocks in BF16, and the paper's accuracies are obtained with this storage. They take 0.38 GiB in total; the 0.8 GB of Appendix E.2 assumed FP32 storage and overestimates it by about 2×.
- Runtime cost of the rotation. On the same kernels, RONA adds −1% to 7% latency over standard W4A4 without rotation. The rotation acts only inside each 128-channel group and is fused with activation quantization in one kernel, so it adds no memory round trip. For comparison, DuQuant's rotation–permutation–rotation adds 17–121% on the same kernels.
- Speed relative to BF16. On A800 every group-wise W4A4 method is slower than BF16; RONA runs at 0.72–0.78× BF16 speed. A breakdown shows that INT4 arithmetic alone is 2–3× faster than BF16 (162 vs. 482 ms for all 368 Linear GEMMs at 8192 tokens with one scale per token), but the fp32 rescale after every 128-channel group takes about two thirds of the group-wise GEMM time (612 ms exact, 237 ms with the rescale removed). APEX4 (arXiv:2606.08761) shows this is set by the ratio ρ between INT4 tensor-core and FP32 CUDA-core throughput: on A100/A800 (ρ = 64) the same format reaches only 0.43–0.47× FP16, whereas on RTX 3090 (ρ = 16) and L40S (ρ = 8) it reaches 1.9–2.5×. Combined with our measured rotation cost (about 17% of the BF16 Linear time) and the 68% share of Linear layers in the BF16 forward pass, RONA's expected end-to-end speed-up on such GPUs is about 1.3–1.4× (1.4–1.7× in the Linear layers).
The extra rotation therefore does not offset the W4A4 benefit: it costs at most a few percent over standard W4A4, while the memory saving is fully realized and the remaining latency gap on A800 is a hardware property shared by all group-wise W4A4 methods.
W2 / Q2 (continued): noise-aware absorption on top of a fixed fast rotation.
2. RONA-H. We replaced the learned rotation by a fixed randomized Hadamard-128 (no training at all) and kept the noise-aware absorption and the rest of the pipeline unchanged; all 78 tasks:
| Rotation | Absorption | Image | Video | VisDoc | All |
|---|---|---|---|---|---|
| Learned block rotation (RONA, Table 1) | noise-aware | 74.77 | 63.79 | 79.06 | 73.55 |
| Fixed Hadamard-128 (RONA-H) | noise-aware | 74.36 | 62.58 | 79.32 | 73.17 |
| DuQuant (Table 1) | — | 70.64 | 60.09 | 76.16 | 69.90 |
- Most of the accuracy is retained. Over all 78 tasks RONA-H is only 0.38 below the learned rotation and still 3.27 above DuQuant; on document tasks it is even slightly higher. The deployment-friendly variant suggested by the reviewer is therefore viable: the learned rotation gives the highest accuracy, and RONA-H gives the simplest deployment.
- Runtime. A fixed Hadamard-128 can be applied with a fast Walsh–Hadamard transform, O(128 log 128) additions per group, and needs no stored rotation matrix (only a sign vector). Since the learned dense block rotation already costs only −1% to 7% over standard W4A4, the runtime difference between the two is small, and either can be chosen according to deployment needs.
Response to Reviewer SGuB (1/8): W1 & Q1, where the gain comes from (rotation)
W1 / Q1: the source of the gain, and why RONA's rotation outperforms learned-rotation baselines.
We thank the reviewer for the careful analysis. We first clarify that RONA's rotation is not a borrowed off-the-shelf component but the first half of the method: the "RO" in ROtation and Noise-aware Absorption. All of RONA's components follow one principle: every transform is fitted under the deployed quantizers, inside the quantization groups. The rotation and the noise-aware absorption apply this principle at two levels, and together they form the contribution of the paper.
1. Why RONA's rotation outperforms existing learned rotations. As the reviewer notes among the strengths, all baselines share RONA's checkpoint, calibration data, quantizers, grouped GPTQ, clipping-calibration objective and refresh schedule (Appendix B.7, Table B.6). The gap between 72.66 and SpinQuant 68.97 / DuQuant 69.90 therefore comes from the rotation design itself, not from the shared calibration pipeline. RONA's rotation differs from existing ones in three ways:
To answer the question directly, we varied these factors one at a time with absorption disabled and everything else fixed (60 image and document tasks; change relative to RONA's rotation):