acceptodds
Under review as a conference paper at ICLR 2027

RONA: ROTATION AND NOISE-AWARE ABSORPTION FOR LOW-BIT UNIVERSAL MULTIMODAL EMBEDDING

12 views

Abstract

Universal multimodal embedding (UME) encodes text, images, video, and visually rich documents into a shared space for cross-modal retrieval. Although reasoning-enhanced methods improve embedding quality, their additional inference-time computation limits low-latency, high-throughput retrieval, leaving direct encoders as the more deployment-friendly choice. Their large backbones, however, still incur substantial serving cost. We study whether post-training quantization (PTQ) can compress UME without sacrificing retrieval accuracy. Across 78 MMEB-V2 tasks, strong PTQ methods remain near full precision at W4A16, W8A16, W4A8, and W8A8, yet collapse at W4A4, a failure not explained by layer-wise reconstruction error. We show that low-bit UME must preserve retrieval geometry and that directions safe under clean activations become harmful when coupled with activation quantization noise. We therefore propose RONA, which combines function-preserving rotations with a group-aligned, quantization-noise-aware safe subspace for weight reparameterization. A per-layer absorption coefficient, calibrated against grouped-GPTQ weight reconstructions and query–target embedding geometry, controls how strongly each layer uses these safe directions while the backbone remains frozen. RONA achieves 73.55 at W4A4, exceeding the strongest PTQ baseline by 3.65 points and retaining 94.4% of full-precision performance. Gains of 6.20 and 2.29 points on two 2B backbones demonstrate its effectiveness across model scales and architectures.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Universal Multimodal Embedding, Post-Training Quantization, Low-Bit Quantization, Embedding Geometry, Multimodal Retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SGuBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RONA, a post-training quantization framework for universal multimodal embedding (UME) models under the challenging W4A4 setting. The authors observe that existing PTQ methods stay near full precision at milder bit-widths but degrade substantially at W4A4. They argue that preserving retrieval geometry, rather than only layer-wise reconstruction error, is essential. RONA combines group-wise learned orthogonal rotations, a quantization-noise-aware safe subspace for additive weight reparameterization, and learned per-layer absorption coefficients. These are calibrated with full-precision query/target embedding anchoring and positive-pair attraction. On Qwen3-VL-Embedding-8B across 78 MMEB-V2 tasks, RONA reaches 73.55 under W4A4, which is 3.65 points above the strongest baseline and 94.4% of full precision. It also improves the strongest baselines on two 2B backbones by 6.20 and 2.29 points, supported by ablations and retrieval-geometry analyses.

Critical strengths

The main strengths of the paper are that (i) it studies an important and underexplored problem with a well-motivated, retrieval-specific perspective on quantization error, and (ii) its experimental comparison is carefully controlled and supported by informative mechanistic analysis.

First, the paper addresses aggressive W4A4 quantization of universal multimodal embedding models, a practically important setting that has received little attention. Its central observation is well motivated: an embedding encoder represents each input as a single point used for nearest-neighbor retrieval, so errors of similar magnitude can affect retrieval margins very differently depending on their direction. The paper also points out that directions that look safe under clean activations can become harmful once coupled with activation-quantization noise. This offers a useful perspective beyond minimizing layer-wise reconstruction error.

Second, the empirical evaluation is unusually well controlled. All baselines share the same checkpoint, calibration records, token allocation, quantizer definitions, grouped-GPTQ backend, clipping-calibration objective, and refresh schedule, while each retains its own transformation (Appendix B.7). This makes the comparison considerably more credible than ones using unmatched implementations. The norm-matched intervention directly supports the paper's main hypothesis: removing the activation-noise coupling component while keeping the perturbation magnitude fixed reduces the ranking-flip rate from 1.14% to 0.56%. Consistent gains on three backbones (+3.65, +6.20, and +2.29 points) further support the robustness of the results.

Critical weaknesses

Most of the gain appears to come from components other than the proposed noise-aware absorption. With absorption fully disabled, RONA already reaches 72.66 under the shared protocol (Table B.5). That is 2.76 points above DuQuant and 3.69 points above SpinQuant, which also learns rotations. Noise-aware absorption, the paper's central contribution, adds only 0.89 points. Roughly three quarters of the headline 3.65-point improvement thus seems to come from the rotation and calibration pipeline, yet the paper does not analyze why this pipeline outperforms existing learned-rotation methods. This is consequential because it leaves unclear what the actual source of the improvement is, and whether the paper's main claim about noise-aware directions is what drives the results. The results also come from a single run with 25 calibration pairs, so the small absorption-related gains are hard to assess without variance estimates.

The deployment motivation is not supported by measured evidence. The paper is motivated by reducing serving cost, but all results use simulated quantization: weights are reconstructed before PyTorch Linear operations, and activations are quantized and then dequantized. RONA also keeps dense FP32 group rotations (about 0.8 GB for the 8B model) in the inference path. No end-to-end latency, throughput, or peak memory is reported. This is consequential because the extra rotation cost could offset part of the W4A4 efficiency benefit once optimized low-bit kernels are used. The paper therefore demonstrates improved accuracy, but not the practical serving advantage it claims.

Questions for authors
  1. What explains the large gain of RONA's rotation pipeline over the learned-rotation baselines? With absorption disabled, RONA reaches 72.66, compared with 68.97 for SpinQuant and 69.90 for DuQuant under the shared protocol. Could the authors clarify which design choices are responsible for this gap, for example, the 128-channel group-wise block structure, the Hadamard initialization, the Cayley parameterization, or the A4/RTN-aware rotation objective? An ablation that varies these factors individually would help readers understand where the main improvement comes from. It would also clarify whether the noise-aware absorption or the rotation design is the primary contribution.

  2. Can the authors report measured deployment benefits? Could the authors report latency, throughput, and peak device memory for BF16, a standard W4A4 baseline, and RONA, using either a native or a representative fused implementation? Relatedly, could they evaluate the noise-aware absorption pipeline on top of a fixed fast rotation (e.g., Hadamard) instead of learned dense rotations? If most of the accuracy remains, this would provide a more deployment-friendly variant. If not, report the accuracy-versus-runtime trade-off for the dense rotations.

  3. How robust are the results to calibration sampling, seeds, and initialization? Could the authors report the mean and standard deviation over several random calibration draws and seeds on Qwen3-VL-Embedding-8B? Could they also compare all-zero, random, and the stored initialization of the absorption coefficients? It would also help to clarify how the initial coefficients for Qwen3-VL-Embedding-2B were obtained. Finally, could the authors report the compute cost of the preparatory initialization search, which the baselines do not receive?

  4. Could the authors provide a more direct comparison with OSAQ-style subspaces? Even without an official OSAQ implementation, could the authors compare the following under the same absorption solver and calibration pipeline: (i) a curvature/Hessian-based subspace, (ii) the clean-activation-only subspace, and (iii) the proposed clean-plus-noise subspace? This would isolate the contribution of the noise-aware direction criterion.

  5. Why does the calibration objective not use negative candidates or margins? Have the authors considered adding hard negatives or a direct margin-preservation term based on the full-precision encoder's nearest irrelevant target? A comparison would clarify whether the positive-only objective was chosen for simplicity or stability, or because explicit ranking supervision adds little benefit.

  6. How sensitive is the method to the safe-subspace rank and the noise coefficient? The main configuration uses rank 32 and γ = 1. The γ = 0 ablation is useful, but could the authors show results for several positive γ values and subspace ranks to indicate whether the chosen operating point is robust?

  7. Does the method generalize beyond the MMEB-V2 calibration distribution? Could the authors calibrate on one subset of task families and evaluate on held-out families, or on an independent multimodal retrieval benchmark? This would indicate whether the estimated safe directions reflect general quantization structure rather than benchmark-specific activation statistics.

13 comments
AuthorsOpenReview ↗

Response to Reviewer SGuB (1/8): W1 & Q1, where the gain comes from (rotation)

W1 / Q1: the source of the gain, and why RONA's rotation outperforms learned-rotation baselines.

We thank the reviewer for the careful analysis. We first clarify that RONA's rotation is not a borrowed off-the-shelf component but the first half of the method: the "RO" in ROtation and Noise-aware Absorption. All of RONA's components follow one principle: every transform is fitted under the deployed quantizers, inside the quantization groups. The rotation and the noise-aware absorption apply this principle at two levels, and together they form the contribution of the paper.

1. Why RONA's rotation outperforms existing learned rotations. As the reviewer notes among the strengths, all baselines share RONA's checkpoint, calibration data, quantizers, grouped GPTQ, clipping-calibration objective and refresh schedule (Appendix B.7, Table B.6). The gap between 72.66 and SpinQuant 68.97 / DuQuant 69.90 therefore comes from the rotation design itself, not from the shared calibration pipeline. RONA's rotation differs from existing ones in three ways:

  • Fitted under the deployed quantizers. RONA fits the rotation jointly under the deployed A4 quantizer and the W4 grid (Eq. 1), directly minimizing the quantized output error. SpinQuant learns its rotation for activation quantization with full-precision weights, and DuQuant builds its rotation from an outlier heuristic; neither puts the W4 grid into the objective.
  • Aligned with the quantization groups. RONA's rotation blocks are exactly 128-dimensional and coincide with the A4/W4 groups, so the rotation never moves magnitude across groups and each group's scale depends only on its own channels.
  • Structured initialization. It starts from Hadamard-128 and is optimized on the orthogonal group through a Cayley parameterization, which keeps the full-precision output exactly unchanged.

To answer the question directly, we varied these factors one at a time with absorption disabled and everything else fixed (60 image and document tasks; change relative to RONA's rotation):

Factor changedvs. RONA's rotationTasks where RONA's rotation wins / loses (of 78)
Hadamard init → identity init−1.1355 / 23
A4+W4 objective → A4-only objective (SpinQuant-style)−0.6150 / 28
Both changed (identity init + A4-only objective)−0.8452 / 26
  • Hadamard initialization is the largest factor: removing it costs 1.13 and hurts 55 of 78 tasks.
  • Putting the W4 grid into the rotation objective adds a further 0.61 on top of Hadamard initialization (50 vs. 28 wins). The two work best together: only from a well-structured Hadamard starting point does the joint A4+W4 objective find a rotation that fits both quantization grids; from identity the joint objective is harder to optimize, so removing both (−0.84) is close to removing the initialization alone (−1.13). RONA's advantage comes from the combination of structured initialization and fitting under the deployed quantizers, not from any single trick.
  • The Cayley parameterization is the means of keeping the rotation orthogonal. Replacing the learned rotation by a fixed Hadamard-128 (also group-aligned) while keeping noise-aware absorption leaves the accuracy almost unchanged (78 tasks: 73.17 vs. 73.55; full results in our reply to W2/Q2 below). The gain comes from group alignment and the quantizer-in-the-loop design rather than from a particular parameterization.
  • 128-channel group alignment. Conversely, misaligning the subspace with the quantization groups lowers the score from 73.55 to 72.82 (Table 2).
AuthorsOpenReview ↗

Response to Reviewer SGuB (2/8): W1 & Q1, the role of noise-aware absorption; variance

W1 / Q1 (continued): noise-aware absorption decides whether absorption helps or hurts.

2. The 0.89 understates the effect of the noise-aware criterion. Tables 2 and B.5 of the submission give (78 tasks):

ConfigurationAll (78)Source
RONA rotation, no absorption72.66Table B.5
+ absorption along clean-only directions (γ = 0)72.25Table 2
+ absorption along noise-aware directions, fixed α73.03Table B.5
+ learned α (full RONA)73.55Table 1
  • Same absorption, same solver, only the direction criterion changes. With clean-only directions, absorption is 0.41 below no absorption; with the noise term it is 0.89 above. The noise term thus accounts for a 1.30-point swing, which is the contribution of the noise-aware criterion itself.
  • Strictly monotone chain. No absorption 72.66 < fixed α 73.03 < learned α 73.55, consistently across the three modalities (+0.79 / +1.17 / +0.84 on image / video / document tasks).
  • Share of the remaining gap. Without absorption the model is still 5.27 points below full precision (77.93 − 72.66); noise-aware absorption recovers about 17% of it, a substantial fraction of the error left after a strong rotation.
  • Direct mechanistic evidence. The equal-norm intervention of Figure 4 fixes the norm of the perturbation and changes only its direction. Removing the noise-coupled component halves the ranking-flip rate (1.14% → 0.56%), whereas a random direction of the same norm barely changes it (1.09%). Whether retrieval breaks is decided by whether the perturbation lies in noise-sensitive directions.
  • New evidence.
    • Across all 12,303 quantization groups, the clean-only safe basis carries as much quantization-noise energy as a random direction (0.988, random = 1); RONA's basis lowers it to 0.794, in every group.
    • OSAQ, re-implemented from its paper with the clean-curvature criterion, reaches only 56.52 under the same W4A4 setting, against 73.55 for RONA, which wins on all 78 tasks.

3. The advantage grows with activation noise. If RONA's gain came only from a strong rotation, its lead would be roughly constant across activation precisions. Instead it grows as activation noise grows:

SettingRONA vs. strongest baselineSource
W4A16 (activations not quantized)on par (75.58 vs. 75.93)Table E.1
W4A8+0.08Table E.1
W4A4+3.65Table 1
W3A3+44.95 (59.83 vs. DuQuant 14.88)new; RONA and DuQuant change only the bit width, all else as in W4A4; 78 tasks

Without activation quantization the noise term vanishes and RONA reduces to an ordinary reparameterization with no advantage; as activation noise appears and grows, the advantage grows with it. This is exactly what the noise-aware-direction argument predicts.

4. Variance. We drew two further independent calibration sets (one pair per subset, 25 pairs), recalibrated from scratch and evaluated all 78 tasks. Over the three calibration draws RONA scores 73.55, 73.82 and 73.59 (mean 73.65 ± 0.15), leading DuQuant (69.90) by +3.65, +3.92 and +3.69. The standard deviation is far smaller than the lead and than the 1.30-point swing produced by the noise term.

AuthorsOpenReview ↗

Response to Reviewer SGuB (3/8): W2 & Q2, measured deployment benefit; absorption on a fixed fast rotation

W2 / Q2: measured latency, throughput and peak memory for BF16, standard W4A4 and RONA.

1. Measured deployment benefit. We implemented the deployed W4A4 path with real low-bit kernels and measured the full forward pass of Qwen3-VL-Embedding-8B on one A800-80GB.

Implementation

  • Activation kernel. Reads each activation once, applies the rotation per 128-channel group on tensor cores, then the asymmetric per-token, per-group 4-bit quantizer. With the rotation switched off it is the standard W4A4 input path, which isolates the rotation cost.
  • GEMM kernel. INT4 tensor cores (mma.sync.m16n8k64.u4.s4); after every 128-wide K group the int32 tile is rescaled in fp32 by that group's scales.
  • Correctness. 0.16% relative error against a float64 reference. With all 368 Linear layers replaced, the mean query/target cosine to the BF16 model is 0.915/0.933, versus 0.918/0.933 for the simulated quantization used in the paper, so the measured speed represents the paper's model.

Measurements (batch 16)

batch 16BF16Standard W4A4 (no rotation)RONA
Image latency (ms)235.0284.8304.7
Document-page latency (ms)990.61185.01268.8
Video latency (ms)1109.51331.31426.7
Image throughput (items/s)68.156.252.5
Linear weight memory (GiB)14.003.944.32
Peak memory, video (GiB)18.098.078.46
  • Memory. Weights −69%, peak memory −53% to −63%.
  • Storage of the rotation. The implementation stores the rotation blocks in BF16, and the paper's accuracies are obtained with this storage. They take 0.38 GiB in total; the 0.8 GB of Appendix E.2 assumed FP32 storage and overestimates it by about 2×.
  • Runtime cost of the rotation. On the same kernels, RONA adds −1% to 7% latency over standard W4A4 without rotation. The rotation acts only inside each 128-channel group and is fused with activation quantization in one kernel, so it adds no memory round trip. For comparison, DuQuant's rotation–permutation–rotation adds 17–121% on the same kernels.
  • Speed relative to BF16. On A800 every group-wise W4A4 method is slower than BF16; RONA runs at 0.72–0.78× BF16 speed. A breakdown shows that INT4 arithmetic alone is 2–3× faster than BF16 (162 vs. 482 ms for all 368 Linear GEMMs at 8192 tokens with one scale per token), but the fp32 rescale after every 128-channel group takes about two thirds of the group-wise GEMM time (612 ms exact, 237 ms with the rescale removed). APEX4 (arXiv:2606.08761) shows this is set by the ratio ρ between INT4 tensor-core and FP32 CUDA-core throughput: on A100/A800 (ρ = 64) the same format reaches only 0.43–0.47× FP16, whereas on RTX 3090 (ρ = 16) and L40S (ρ = 8) it reaches 1.9–2.5×. Combined with our measured rotation cost (about 17% of the BF16 Linear time) and the 68% share of Linear layers in the BF16 forward pass, RONA's expected end-to-end speed-up on such GPUs is about 1.3–1.4× (1.4–1.7× in the Linear layers).

The extra rotation therefore does not offset the W4A4 benefit: it costs at most a few percent over standard W4A4, while the memory saving is fully realized and the remaining latency gap on A800 is a hardware property shared by all group-wise W4A4 methods.

W2 / Q2 (continued): noise-aware absorption on top of a fixed fast rotation.

2. RONA-H. We replaced the learned rotation by a fixed randomized Hadamard-128 (no training at all) and kept the noise-aware absorption and the rest of the pipeline unchanged; all 78 tasks:

RotationAbsorptionImageVideoVisDocAll
Learned block rotation (RONA, Table 1)noise-aware74.7763.7979.0673.55
Fixed Hadamard-128 (RONA-H)noise-aware74.3662.5879.3273.17
DuQuant (Table 1)—70.6460.0976.1669.90
  • Most of the accuracy is retained. Over all 78 tasks RONA-H is only 0.38 below the learned rotation and still 3.27 above DuQuant; on document tasks it is even slightly higher. The deployment-friendly variant suggested by the reviewer is therefore viable: the learned rotation gives the highest accuracy, and RONA-H gives the simplest deployment.
  • Runtime. A fixed Hadamard-128 can be applied with a fast Walsh–Hadamard transform, O(128 log 128) additions per group, and needs no stored rotation matrix (only a sign vector). Since the learned dense block rotation already costs only −1% to 7% over standard W4A4, the runtime difference between the two is small, and either can be chosen according to deployment needs.
Reviewer_P5tzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies low-bit post-training quantization (PTQ) for Universal Multimodal Embedding (UME) models. It identifies a W4A4 collapse in retrieval accuracy and attributes it to the sensitivity of specific directions to activation quantization noise. It proposes RONA, a PTQ framework using function-preserving rotations, safe weight reparameterization restricted to a quantization-noise-aware subspace, and learned layer-wise absorption coefficients. RONA significantly improves W4A4 retrieval accuracy on 78 MMEB-V2 tasks.

Critical strengths
  1. Insightful diagnostic analysis. The paper provides a insight that W4A4 quantization failures in multimodal retrieval depend not only on reconstruction error magnitude but also on directional embedding drift. This insight is further supported by quantitative analyses and experimental results.

  2. Extensive experimental solid. The method is evaluated across three backbones, consistently outperforming strong W4A4 baselines. Ablation studies further support the effectiveness of its key components.

Critical weaknesses
  1. Unclear and overly complicated presentation. The paper is difficult to follow for several reasons. (i) It uses excessive technical terminology, such as retrieval-preserving, group-aligned, group-wise, task-adaptive, and quantization-consistent, which increases the reading burden. (ii) Figure 2 contains too much information and text, failing to highlight the key novelty. (iii) Some symbols, such as YY, uu, lack clear explanations when first introduced, while extensive definitions are deferred to the appendix. Moreover, the Method section does not clearly connect the proposed designs to the problems they aim to solve. The authors should briefly introduce preliminary techniques and focus more on explaining their novel contributions.

  2. Insufficient justification of methodological complexity. RONA introduces multiple components, but their necessity and combined benefits beyond the core noise-aware subspace design are not sufficiently justified. Despite the ablation studies, the paper could better distinguish essential innovations from auxiliary design choices.

Questions for authors
  1. Inference efficiency. Can the authors report inference latency, throughput, and per-module runtime overhead compared with existing methods?

  2. Validation of directional insights. Can the authors provide additional visualizations (e.g., PCA or t-SNE) or controlled quantitative analyses to better illustrate how the learned safe subspace preserves retrieval geometry?

  3. Lower-bit generalization. Does RONA remain effective under W3A3 or W2A2 quantization compared with existing methods?

8 comments
AuthorsOpenReview ↗

Response to Reviewer P5tz (1/6): W1, presentation

W1: unclear and overly complicated presentation.

We thank the reviewer for these concrete comments. We clarify each point below.

(i) Terminology. Each modifier names one concrete operation:

  • group-aligned / group-wise: computed inside the quantizer's 128-channel groups;
  • quantization-consistent: computed with the deployed quantizer in the loop;
  • task-adaptive: one strength learned per layer;
  • retrieval-preserving: the calibration target keeps quantized query and target embeddings close to their full-precision versions.

Only two named concepts are needed to follow the method: the noise-aware safe subspace and the absorption strength α.

(ii) Symbols.

  • Y = XWᵀ + 1bᵀ is the output of a Linear layer (Sec. 3.1).
  • u is one 128-channel activation group of a token; the activation quantizer Q_A(u; c) computes its scale and zero-point on this group (Eq. B.1).

(iii) RONA in four sentences.

  1. Rotate each layer's inputs and weights inside every quantization group; the rotation is fitted under the deployed 4-bit quantizers, keeps the full-precision output unchanged and makes the inputs easier to quantize.
  2. In each group, find the weight directions to which both the clean input and its 4-bit quantization error respond weakly.
  3. Move weight outliers only along those directions and fold the change into the 4-bit weights.
  4. Learn one strength per layer, with GPTQ in the loop, so that quantized query and target embeddings stay close to the full-precision ones.

(iv) Problem-to-design map.

ProblemDesignSection
Activation-quantization noise can turn directions that are safe for clean activations into harmful onesNoise-aware safe subspace3.3
Group-wise rounding breaks cancellation across groupsOne group partition shared by rotation, subspace, absorption and GPTQ3.2–3.4
Layers tolerate different amounts of modificationLearned per-layer strength α3.5–3.6
Small reconstruction error does not imply preserved retrieval marginsCalibration on embedding geometry3.7

(v) Figure 2. Figure 2 is a complete pipeline reference, which is why it is dense; the core idea is shown on its own in Figure 1(c). When reading Figure 2, only blocks 2.1–2.4 and 3 are new; block 1 (calibration data and BF16 forward) and block 4 (export) are the standard PTQ setup.

AuthorsOpenReview ↗

Response to Reviewer P5tz (2/6): W2, necessity of the components

W2: justification of the methodological complexity.

RONA's components are not a stack of independent tricks; they apply one principle at different places: every transform is fitted under the deployed quantizers, inside the quantization groups. Existing rotation methods (QuaRot, SpinQuant, DuQuant) design their transforms for full precision or for activation quantization only, and outlier-absorption methods such as OSAQ look only at the curvature of clean activations. RONA's contribution is to put the deployed quantizer into both the rotation objective and the absorption criterion.

Tables 2 and B.5 of the submission remove the components one at a time, with the same calibration data, quantizers and calibration schedule (78 tasks):

ConfigurationAll (78)vs. full RONA
Full RONA73.55—
No noise term (clean activations only, γ = 0)72.25−1.30
No absorption72.66−0.89
Absorption strength α fixed, not learned73.03−0.52
Subspace misaligned with the quantization groups72.82−0.73
Embedding-geometry calibration replaced by layer-wise reconstruction72.55−1.00

Removing any component lowers the score, and each component addresses a specific problem:

  1. Noise-aware safe subspace (core). With clean-only directions, absorption is 0.41 below no absorption; with the noise term it is 0.89 above, a swing of 1.30 points. Once activations are quantized, the noise term decides whether outlier absorption helps or hurts.
  2. Group-wise rotation fitted under the deployed quantizers. A new factor-by-factor ablation (absorption disabled; 60 image and document tasks) shows that including the W4 grid in the rotation objective is worth +0.61 over an A4-only objective (40 vs. 20 task wins), and Hadamard initialization +1.13 over identity (46 vs. 14). This is the quantizer-in-the-loop principle applied to the rotation, and one of the main sources of RONA's lead over SpinQuant and DuQuant.
  3. Shared quantization groups. Rotation, subspace, absorption and GPTQ use the same 128-channel partition. Misaligning them lowers the score to 72.82, because group-wise rounding breaks cross-group cancellation.
  4. Learned per-layer strength α. Layers tolerate different amounts of modification: 73.03 with a fixed α, 73.55 when learned. Initializing every α at zero gives the same result (78 tasks: 73.77 vs. 73.55), so it does not depend on an initialization search.
  5. Embedding-geometry calibration. Retrieval depends on the relative positions of queries and targets, not on per-layer reconstruction error; layer-wise reconstruction calibration drops to 72.55.

The chain "no absorption 72.66 < fixed α 73.03 < learned α 73.55" is strictly monotone and consistent across the three modalities, so the components add up rather than substitute for one another.

The complexity is offline only. Safe bases, ΔW and calibration gradients are discarded after export; at inference RONA adds only a block rotation. Calibrating the 8B model takes about 30 minutes on one A800: 8.5 min for the geometry of all 368 layers, 17 min for four α/c steps with GPTQ refreshes, and 1 min for the export check.

AuthorsOpenReview ↗

Response to Reviewer P5tz (3/6): Q1, inference efficiency (measured)

Q1: inference latency, throughput and per-module overhead compared with existing methods.

We implemented the deployed W4A4 path with real INT4 kernels and measured the full forward pass of Qwen3-VL-Embedding-8B on one A800-80GB (the GPUs available to us). Below are the measurements, followed in the next comment by the per-module breakdown and the comparison with existing transforms.

1. Implementation

  • Activation kernel. Reads each activation once, applies the rotation per 128-channel group on tensor cores, then the asymmetric per-token, per-group 4-bit quantizer. With the rotation switched off it is the standard W4A4 input path, which isolates the cost of the rotation.
  • GEMM kernel. INT4 tensor cores (mma.sync.m16n8k64.u4.s4); after every 128-wide K group the int32 tile is rescaled in fp32 by that group's scales.
  • Correctness. 0.16% relative error against a float64 reference. With all 368 Linear layers replaced, the mean query/target cosine to the BF16 model is 0.915/0.933, versus 0.918/0.933 for the simulated quantization used in the paper.

2. Memory (measured, GiB)

BF16W4A4, no rotationRONA
Linear weights14.003.944.32 (−69%)
Whole model15.185.175.56 (−63%)
Peak, text, batch 1615.415.375.75 (−63%)
Peak, video, batch 1618.098.078.46 (−53%)

The rotation blocks are stored in BF16 and take 0.38 GiB.

3. Latency and throughput (A800, batch 16)

InputBF16 (ms)RONA (ms)RONA vs. W4A4 without rotationBF16 (items/s)RONA (items/s)
Text query82.9105.9+6.4%192.9151.1
Image235.0304.7+7.0%68.152.5
Document page990.61268.8+7.1%16.212.6
Video, 8 frames1109.51426.7+7.2%14.411.2
  • Cost of the online rotation. On the same kernels, RONA adds −1% to 7% latency over W4A4 without rotation (batch 1 and 16). The rotation acts only inside each 128-channel group and is fused with activation quantization, so it adds no extra memory round trip.
  • A800 is not faster than BF16. RONA runs at 0.72–0.78× BF16 speed. The cause is the GPU, not the rotation. Total GEMM time of the 368 Linear layers at 8192 tokens (ms):
GEMMms
BF16 cuBLAS482
INT4, one scale per token162
INT4, per-128 groups, fp32 rescale removed (timing only)237
INT4, per-128 groups, exact612

INT4 arithmetic itself is 2–3× faster than BF16, but the per-group fp32 rescale takes about two thirds of the group-wise GEMM time. APEX4 (arXiv:2606.08761) shows that this cost is set by the ratio ρ between INT4 tensor-core and FP32 CUDA-core throughput: on A100/A800 (ρ = 64), W4A4-g128 reaches only 0.43–0.47× FP16 when compute-bound, consistent with our measurement, while the same format reaches 2.0–2.5× on RTX 3090 (ρ = 16) and 1.9–2.1× on L40S (ρ = 8). This is a property of group-wise W4A4 on this hardware and applies to every group-wise W4A4 method.

4. Expected speed-up on low-ρ GPUs

  • RONA's rotation plus activation quantization costs about 17% of the BF16 Linear time on A800 (84 ms vs. 482 ms at 8192 tokens).
  • With APEX4's 1.9–2.5× group-wise GEMM speed-up, RONA's Linear time becomes 1/1.9 + 0.17 ≈ 0.70 to 1/2.5 + 0.17 ≈ 0.57 of BF16, i.e. 1.4–1.7× faster.
  • Linear layers account for about 68% of the BF16 forward pass (our profiling, images and document pages); the remainder (attention, norms, copies) is unchanged. End to end this gives about 1.26–1.41×, with throughput at the same batch rising by the same factor; the 53–69% memory reduction also allows larger batches.
  • This estimate agrees with APEX4's measured end-to-end speed-up of the same format on L40S (1.16–1.66×).
Reviewer_sPVvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies an interesting and practical system optimization problem: compressing universal multimodal embedding models to W4A4 while preserving retrieval performance.

The proposed RONA method focuses on preserving embedding geometry rather than simply minimizing layer-wise reconstruction error. It consists of four main components:

  1. Function-preserving Rotation: Applies orthogonal rotations to activations and weights to improve quantization behavior.
  2. Noise-Aware Safe Subspace: Identifies low-sensitivity directions by considering both activation statistics and quantization noise.
  3. Weight Absorption: Suppresses weight outliers within the safe subspace, with adaptive coefficients controlling the modification strength.
  4. Embedding Geometry Calibration: Uses full-precision embeddings as references to preserve query-target representations and positive-pair similarity during quantization.

The paper evaluates RONA on 78 MMEB-V2 tasks, covering image, video, and visual-document retrieval, using Qwen3-VL-Embedding-8B and two additional 2B models. The results demonstrate promising W4A4 accuracy retention and improvements over existing PTQ methods.

Critical strengths
  1. Practical and meaningful problem. The paper addresses low-bit quantization for multimodal embedding models. Preserving embedding geometry and retrieval relationships, rather than only minimizing reconstruction errors, is an interesting and useful direction.

  2. Strong empirical results with relatively comprehensive validation. The experiments cover 78 MMEB-V2 tasks across image, video, and visual-document modalities. RONA retains 94.4% of FP16 performance on Qwen3-VL-Embedding-8B and improves over the strongest evaluated W4A4 PTQ baseline by 3.65 points. The additional results on two 2B models provide evidence of generalization across model scales and architectures. The ablation studies also support the effectiveness of the proposed components.

Critical weaknesses
  1. Missing end-to-end efficiency evaluation. The motivation is to reduce the serving cost of multimodal embedding models, but the paper mainly reports accuracy retention and theoretical efficiency analysis. Actual inference latency, throughput, and peak memory are not evaluated. Since RONA introduces online rotation operations, the practical speedup remains unclear.

  2. The advantage over closely related quantization methods could be better established. The method combines rotation, safe-subspace optimization, and outlier absorption, which are related to existing Hessian-aware and outlier-absorption approaches. Although the ablation studies are reasonably comprehensive, a more direct comparison with closely related methods such as OSAQ would help clarify the incremental contribution of RONA.

Questions for authors
  1. Could the authors provide end-to-end inference latency, throughput, and memory measurements? In particular, how much practical speedup can RONA achieve after accounting for the overhead of online rotations?

  2. How does RONA compare with existing Hessian-aware or outlier-absorption methods, particularly OSAQ, under matched W4A4 settings? It would be helpful to better understand the advantage of noise-aware subspace selection over these related approaches.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sPVv (1/3): W1 & Q1, end-to-end efficiency

W1 / Q1: end-to-end latency, throughput and memory, including the online rotation.

We thank the reviewer for this suggestion. The GPUs available to us are A800-80GB, so we implemented the deployed W4A4 path with real INT4 kernels, measured the full forward pass of Qwen3-VL-Embedding-8B on one A800, and combined these measurements with published cross-GPU results to estimate the speed-up on GPUs with a lower tensor-core/CUDA-core ratio ρ.

1. Implementation

  • Activation kernel. Reads each activation once, applies the rotation per 128-channel group on tensor cores, then the asymmetric per-token, per-group 4-bit quantizer. With the rotation switched off it is the standard W4A4 input path, which isolates the cost of the rotation.
  • GEMM kernel. INT4 tensor cores (mma.sync.m16n8k64.u4.s4); after every 128-wide K group the int32 tile is rescaled in fp32 by that group's scales.
  • Correctness. 0.16% relative error against a float64 reference. With all 368 Linear layers replaced, the mean query/target cosine to the BF16 model is 0.915/0.933, versus 0.918/0.933 for the simulated quantization used in the paper.

2. Memory (measured, GiB)

BF16W4A4, no rotationRONA
Linear weights14.003.944.32 (−69%)
Whole model15.185.175.56 (−63%)
Peak, text, batch 1615.415.375.75 (−63%)
Peak, video, batch 1618.098.078.46 (−53%)

The rotation blocks are stored in BF16 and take 0.38 GiB.

3. Latency and throughput (A800, batch 16)

InputBF16 (ms)RONA (ms)RONA vs. W4A4 without rotationBF16 (items/s)RONA (items/s)
Text query82.9105.9+6.4%192.9151.1
Image235.0304.7+7.0%68.152.5
Document page990.61268.8+7.1%16.212.6
Video, 8 frames1109.51426.7+7.2%14.411.2
  • Cost of the online rotation. On the same kernels, RONA adds −1% to 7% latency over W4A4 without rotation (batch 1 and 16). The rotation acts only inside each 128-channel group and is fused with activation quantization, so it adds no extra memory round trip.
  • A800 is not faster than BF16. RONA runs at 0.72–0.78× BF16 speed. The cause is the GPU, not the rotation. Total GEMM time of the 368 Linear layers at 8192 tokens (ms):
GEMMms
BF16 cuBLAS482
INT4, one scale per token162
INT4, per-128 groups, fp32 rescale removed (timing only)237
INT4, per-128 groups, exact612

INT4 arithmetic itself is 2–3× faster than BF16, but the per-group fp32 rescale takes about two thirds of the group-wise GEMM time. APEX4 (arXiv:2606.08761) shows that this cost is set by the ratio ρ between INT4 tensor-core and FP32 CUDA-core throughput: on A100/A800 (ρ = 64), W4A4-g128 reaches only 0.43–0.47× FP16 when compute-bound, consistent with our measurement, while the same format reaches 2.0–2.5× on RTX 3090 (ρ = 16) and 1.9–2.1× on L40S (ρ = 8). This is a property of group-wise W4A4 on this hardware and applies to every group-wise W4A4 method.

4. Expected speed-up on low-ρ GPUs

  • RONA's rotation plus activation quantization costs about 17% of the BF16 Linear time on A800 (84 ms vs. 482 ms at 8192 tokens).
  • With APEX4's 1.9–2.5× group-wise GEMM speed-up, RONA's Linear time becomes 1/1.9 + 0.17 ≈ 0.70 to 1/2.5 + 0.17 ≈ 0.57 of BF16, i.e. 1.4–1.7× faster.
  • Linear layers account for about 68% of the BF16 forward pass (our profiling, images and document pages); the remainder (attention, norms, copies) is unchanged. End to end this gives about 1.26–1.41×, with throughput at the same batch rising by the same factor; the 53–69% memory reduction also allows larger batches.
  • This estimate agrees with APEX4's measured end-to-end speed-up of the same format on L40S (1.16–1.66×).

Summary. Measured: 53–69% lower memory, and the online rotation costs −1% to 7%. On A800, group-wise W4A4 is limited by the per-group rescale and is not faster than BF16; on GPUs with ρ ≤ 16 we expect about 1.3–1.4× end to end and 1.4–1.7× in the Linear layers.

AuthorsOpenReview ↗

Response to Reviewer sPVv (2/3): W2 & Q2, comparison with OSAQ

W2 / Q2: comparison with Hessian-aware and outlier-absorption methods, in particular OSAQ, under matched W4A4 settings.

We thank the reviewer for this suggestion. OSAQ has no public code, so we re-implemented it from the paper (arXiv:2605.04738) and placed it in exactly the same W4A4 pipeline as RONA.

1. The two methods differ in how the subspace is chosen. RONA and OSAQ share a form: both add a correction inside a low-sensitivity subspace to absorb weight outliers, and both solve it with a softmax-weighted closed form (Appendix F). They differ in the direction criterion:

  • OSAQ takes the null space of the clean-activation Hessian XᵀX over the full input width. It only guarantees that the clean response Xv is small. OSAQ mainly targets weight-only quantization, where activations stay in full precision at inference, and there this choice is sound.
  • Under W4A4, the input at inference is X + E, where E is the activation-quantization noise. The absorbed correction ΔW multiplies E and produces an extra error EΔWᵀ. The null space of XᵀX does not see the noise response Ev.
  • RONA requires the clean response Xv and the noise response Ev to be small separately, inside each 128-channel quantization group (Eq. 3). The absorbed correction is therefore safe under the deployed quantizers, and, being group-local, it never moves weight magnitude across quantization groups.

2. Setup.

  • OSAQ, following the paper. Per Linear layer, the full-width clean Hessian accumulated over all tokens of the 25 calibration pairs; the eigenvectors with the smallest |eigenvalues|; the Softmax-∞ closed-form solve with the μ₁ and μ₂ regularizers; the correction folded entirely into the weights; absorption applied before the rotation, as OSAQ prescribes.
  • Hyperparameters not given in the paper. OSAQ does not report numeric values for its energy threshold or regularizers. We set the null-space rank to 1/4 of the input width, the same budget as RONA's 32 of every 128 channels, and used RONA's τ, μ₁, μ₂, rescaled to the row width.
  • Everything else identical to RONA: Qwen3-VL-Embedding-8B, the same block rotation, GPTQ W4, asymmetric A4, activation-clipping calibration and 25 calibration pairs, evaluated on all 78 tasks. The only difference is the subspace criterion.

Table R3. W4A4, Qwen3-VL-Embedding-8B, MMEB-V2 (78 tasks)

MethodImageVideoVisDocAll
OSAQ (our re-implementation)54.4143.0969.7756.52
RONA74.7763.7979.0673.55

RONA is higher on all 78 tasks (78 wins, 0 losses; two-sided sign test p = 6.6×10⁻²⁴), by 17.03 points overall and by 20.36 / 20.70 / 9.29 on image / video / document tasks.

AuthorsOpenReview ↗

Response to Reviewer sPVv (3/3): W2 & Q2, continued

W2 / Q2 (continued): why the clean-curvature criterion fails under W4A4, and cost.

3. Why the OSAQ criterion fails under W4A4.

  1. A clean-only null space is no longer safe once activations are quantized. After absorption a Linear layer outputs (X + E)(W + ΔW)ᵀ = XWᵀ + XΔWᵀ + E(W + ΔW)ᵀ. OSAQ only makes XΔWᵀ small. With weight-only quantization E = 0, so any ΔW inside the null space of XᵀX is harmless, which is OSAQ's setting. Under W4A4, E multiplies ΔW and creates the error EΔWᵀ, which grows with ‖ΔW‖ and is not constrained at all by the null space of XᵀX. In our re-implementation the median ‖ΔW‖ reaches 51% of ‖W‖ under the matched rank budget, and all of it is exposed to quantization noise at inference. RONA's Eq. (3) requires both Xv and Ev to be small along the correction, which directly bounds EΔWᵀ.
  2. A full-width correction breaks group-wise scales. W4 and A4 are quantized per 128-channel group, each with its own scale. OSAQ's null-space directions span the whole input width, so the correction moves weight magnitude between groups and changes their maxima and scales. Cancellation across groups holds in full precision but not after group-wise rounding. RONA's subspace, absorption, rotation and GPTQ share one 128-channel partition, so the correction stays inside its group.
  3. A full-width Hessian null space is unreliable with little calibration data. The 25 pairs give about 9,500 tokens per layer. For a 12,288-wide down_proj this is fewer than the Hessian dimension, so XᵀX is rank-deficient in 36 layers, and much of its "null space" consists of directions the calibration data simply does not cover but test inputs do. RONA estimates 128×128 matrices per group, which the same data determines well. Consistently, OSAQ reaches a lower calibration loss than RONA on the 25 pairs (0.642 vs. 0.705) yet loses on all 78 test tasks: its correction fits the calibration data but does not transfer to test inputs.
  4. The absorption happens in the wrong coordinates. OSAQ computes its null space and absorbs before the rotation; the rotation then redistributes channels, so directions found in the original coordinates no longer correspond to the deployed quantization grid. RONA computes the subspace in the rotated coordinates with the deployed quantizer.

4. Cost. OSAQ needs a full-width eigendecomposition per layer (12,288 × 12,288 for down_proj); its geometry stage over 368 layers took 88 minutes on one A800. RONA decomposes only 128×128 matrices and takes 8.5 minutes.

Summary. Under the same W4A4 setting, where the subspace criterion is the only difference, RONA outperforms OSAQ on all 78 tasks. When activations are also quantized, a null space of the clean-activation curvature does not make absorption safe; the criterion must constrain both the clean and the quantization-noise responses inside each quantization group, which is RONA's noise-aware safe subspace.

Area_Chair_nU1bOpenReview ↗

please examine rebuttal response

Hello Reviewer sPVv,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal and comment and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: thank you to all reviewers

We thank the three reviewers for their time and for detailed reviews that were specific enough to act on. Before the individual responses, we summarize what the reviewers found valuable, since it frames what we set out to strengthen during the discussion period.

The problem. sPVv considers low-bit quantization of multimodal embedding models, with the goal of preserving embedding geometry and retrieval relations rather than only minimizing reconstruction error, a practical and meaningful direction. SGuB describes it as an important but under-explored setting and finds the retrieval-specific view of quantization error well motivated.

The insight. P5tz highlights the diagnostic finding that W4A4 failures in multimodal retrieval depend not only on the size of the quantization error but also on the direction in which embeddings drift. SGuB notes that directions which look safe under clean activations can become harmful once they couple with activation-quantization noise, and finds this a useful perspective beyond layer-wise reconstruction.

The evaluation. sPVv and P5tz find the experiments solid and comprehensive: 78 MMEB-V2 tasks over image, video and visual-document retrieval, three backbones, and consistent gains over strong W4A4 baselines. SGuB finds the comparison unusually well controlled, since all baselines share the checkpoint, calibration records, quantizers, grouped-GPTQ backend and calibration schedule, and notes that the equal-norm intervention directly supports the main hypothesis.

What we added during the discussion period.

  1. Measured efficiency with real INT4 tensor-core kernels on A800: end-to-end latency, throughput and peak memory for BF16, standard W4A4 and RONA; per-module cost; the cost of RONA's rotation versus DuQuant's transform; and the expected speed-up on GPUs with a lower tensor-core/CUDA-core throughput ratio (sPVv W1/Q1, P5tz Q1, SGuB W2/Q2).
  2. A direct comparison with OSAQ, re-implemented from the paper because no code is released, under exactly RONA's W4A4 pipeline on all 78 tasks (sPVv W2/Q2, SGuB Q4).
  3. Sources of the gain: a factor-by-factor rotation ablation, the three subspace criteria under one solver, and a direct measurement of clean and noise energy in every safe basis (SGuB W1/Q1 and Q4, P5tz W2).
  4. Robustness and sensitivity: three independent calibration draws, three initializations of the absorption coefficients, and sweeps over the safe-subspace rank and the noise coefficient γ (SGuB Q3 and Q6).
  5. Geometry and generalization: a quantitative analysis of how well the BF16 retrieval geometry is preserved, PCA and margin visualizations, and calibration on images only with evaluation on held-out document and video families (P5tz Q2, SGuB Q5 and Q7).
  6. Lower bit widths: W3A3 and W2A2 for RONA, DuQuant and SpinQuant (P5tz Q3).
  7. Presentation: plain-language definitions, symbols, a four-sentence summary of RONA and a problem-to-design map (P5tz W1).

How to read the replies. We have not revised the submitted PDF during the discussion period. Every new result and clarification is given in full in the replies under each review, one comment per weakness/question (or per weakness–question pair when both ask the same thing). Table, figure, equation and appendix numbers refer to the submission. Unless stated otherwise, new results are on Qwen3-VL-Embedding-8B, W4A4, all 78 MMEB-V2 tasks, with the same 25 calibration pairs and evaluation protocol as the paper. Two new figures are provided through an anonymous link that does not track visitors.

We will incorporate all of these new experiments, analyses and clarifications, together with the presentation changes requested by P5tz, into the final version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.