acceptodds
Under review as a conference paper at ICLR 2027

Fixing the Errors That Flip Answers: Output-Weighted Low-Rank Correction for Quantized Vision-Language Models

Abstract

A small low-rank correction added to a low-bit vision-language model (VLM) can repair only part of its quantization error. We ask which error directions such a limited correction budget should repair to preserve the answers of the full-precision (FP16) teacher. Starting from the observation that flips concentrate where the teacher's answer margin is small, so that small output changes decide them, we weight the low-rank reconstruction of the quantization residual by input-activation statistics and by the second moment of an output gradient that measures how strongly each output direction moves the answer; we call this output-weighted quantization error reconstruction OQER. With the same correction storage (rank 64, bits/weight) on a 3-bit LLaVA-1.5-7B, it removes (POPE) and (MME) of the teacher disagreements, against and for activation-weighted reconstruction, and at rank 32, half the extra storage, it still exceeds activation-weighted rank 64. The gain holds on top of the MBQ and QIG quantizers, on LLaVA-1.5-13B and Qwen2.5-VL-7B. In the tested settings, retaining the structure and alignment of the output weighting improves fidelity, and several output gradients are effective.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
post-training quantization, vision-language models, low-rank error correction, quantization error reconstruction, teacher fidelity, answer flips, decision margins, model compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_z7JmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which quantization-error directions should receive a limited low-rank correction budget to preserve a full-precision VLM teacher's answers. OQER weights the residual of a fixed quantized checkpoint by both input-activation and output-gradient moments, then constructs a training-free correction using truncated SVD. On LLaVA-1.5-7B at 3 bits and rank 64, net recovery on POPE-adversarial/MME increases from 34.3%/16.6% for activation weighting to 57.5%/33.1% (Table 2). Experiments across quantizers, backbones and short generated answers provide further support. The contribution is primarily the post-hoc correction setting and fidelity evidence, rather than a new weighted-SVD closed form.

Critical strengths
  1. Direct, controlled validation of fidelity repair at equal correction storage. With the same quantized checkpoint, rank, calibration questions and factor precision, OQER achieves 57.5%/33.1% net recovery on POPE-adversarial/MME versus 34.3%/16.6% for activation weighting; rank 32 still reaches 46.0%/29.6%. Table 10 implies mean remaining disagreements of 57 versus 88 and approximately 187 versus 233. Paired statistics and correction-strength controls support the difference (Tables 2, 10, 13; Appendices A, H). This establishes a useful application of known weighting principles to fixed-residual repair and is the main acceptance argument; it does not establish a unique margin-gradient benefit.

  2. Structure controls and cross-setting results give the empirical contribution substance beyond one main table. The diagonal, sign and permutation controls support the role of output-coordinate structure and alignment (Table 6; Appendices I, K). Tables 16-19 extend the fidelity evidence across quantizers, backbones and short generated answers. Full output moments are not new, but the repair experiments provide concrete evidence of their usefulness. Some Qwen MME effects remain unresolved, so the finding should remain limited to the tested settings.

Critical weaknesses
  1. Limited methodological differentiation. The formulation draws on QERA-style reconstruction of a fixed quantization residual and LASER-style two-sided output-gradient weighting. In Table 7, whole-prompt loss weighting achieves most of the improvement over activation weighting, while the additional margin advantage is not statistically significant. The authors acknowledge this overlap, but the extent of new knowledge beyond these established ingredients remains a principal limitation. The acceptance case therefore depends primarily on the significance of the controlled fidelity and structure findings.

  2. The basis for the default configuration and its supported scope require clarification. Gradient rankings vary with the evaluated setting: Table 11 includes a lower OQER point estimate in a partial-layer, rank-8 configuration, without a significance mark, and the ScienceQA pilot in Appendix I favors likelihood over nearest-rival margin weighting. Appendix O explicitly acknowledges the absence of a general selection rule. These observations do not invalidate the main all-layer, rank-64 results, but leave uncertainty about how the recommended configuration is selected and the budget/readout regimes to which the reported benefits should be generalized.

  3. Incomplete evidence for the deployment trade-off. Accuracy and teacher fidelity are evaluated with simulated quantization, whereas actual kernels are measured only for resource use. Table 8 reports similar peak memory for corrected 3-bit and the lowest-memory 4-bit reference, while Table 17's simulated MME results favor 4-bit in teacher fidelity. Since these measurements do not jointly characterize actual execution, they do not yet establish the practical fidelity-memory-latency trade-off under the intended deployment constraints. This is an evidence gap, rather than a demonstrated failure of the real-kernel implementation.

Questions for authors
  1. Could the authors delineate the contribution relative to QERA-style residual reconstruction and LASER-style output-gradient weighting? Given that Table 7 does not establish a statistically significant advantage of the margin gradient over whole-prompt loss, which methodological distinction or controlled empirical finding constitutes the principal advance beyond these existing approaches? Please also contextualize this advance relative to other residual reconstruction methods like Quant Experts and SVDQuant.

  2. How were the output-gradient objective and correction rank selected for the main experiments? Table 11 reports a lower POPE net-recovery point estimate for OQER at rank 8 when only layers 0-11 are corrected; the ScienceQA results in Appendix I also favors likelihood over nearest-rival margin weighting. Could the authors clarify whether the margin/rank-64 defaults were specified a priori or selected using calibration or development data, and delineate the budget and readout settings supported by the current evidence?

  3. Using the existing evaluation items at the same correction budget, please report teacher-disagreement rates by teacher-margin bin for activation weighting, whole-prompt-loss weighting and OQER. Given the limitations in Appendices P/Q, which differences between these corrections does the flip analysis explain?

  4. Could the authors characterize the fidelity-memory-latency trade-off of 3-bit plus OQER (ranks 32/64) relative to the 3 or 4-bit reference under comparable evaluation conditions? Indicate when adding a correction is preferable to increasing bitwidth at comparable memory or latency budgets.

If the authors satisfactorily address these concerns, I would be willing to raise my rating.

7 comments
AuthorsOpenReview ↗

[1/4] Response to Reviewer z7Jm

Thank you for the precise review. Your summary states the contribution as we intend it. We answer the four questions in order, and Q1, Q2 and Q4 also address the three weaknesses. Table numbers below refer to the revised PDF. Our summary comment lists all changes and the renumbering.

Q1. Contribution relative to QERA and LASER, and Quant Experts and SVDQuant

The principal advance is empirical. The paper asks whether, under a limited correction budget, minimizing the reconstruction error of a quantized VLM preserves its answers. Three controlled findings show that at the same budget a correction with a lower reconstruction error can preserve fewer of them.

  1. Reconstructing the residual better does not preserve more answers. The table compares the two corrections on one checkpoint and one seed (Table 23, rank 64, seed 0, net recovery on POPE-adversarial / MME).
Rank-64 correctionSquared weight residual vs. no correctionNet recovery
Activation-weighted (exact optimum for the layer-output error)−3.3%38.1% / 15.1%
OQER+9.1%54.5% / 31.9%

Over three seeds the net recovery is 34.3% / 16.6% against 57.5% / 33.1% (Table 2), and the activation-weighted correction also captures more of the layer-output error than OQER (0.609 against 0.539, Table 6). It is already the exact optimum for its criterion, so the limit lies in the criterion. Table 32 describes the difference per layer for one seed. Of the residual energy weighted by the margin-gradient moment, the activation-weighted correction captures 61% and OQER 91%. In the attention projections of layers 22 to 31 the output moment has an effective rank of 1.6 to 3.4 against 14.6 to 16.1 for the input moment, and there the activation-weighted correction captures only 49 to 58%. OQER is ahead under its own metric by construction, so the numbers of Table 32 characterize the two corrections and are not an independent proof.

  1. On LLaVA-1.5-7B the gain depends on the full, aligned output moment. With the same margin gradient (Table 6, three POPE splits / MME), LLaVA-1.5-7B recovers 37.0% / 16.6% without output weighting, 25.0% / 0.0% with the diagonal of the moment, 25.3% / −5.6% with flipped signs and 59.7% / 33.1% with the full moment. On Qwen2.5-VL-7B the POPE effect has the same direction and is significant (6.0%, 11.0%, 24.1%, 38.0%). MME does not resolve it, and we keep that limit in the text.
  2. The gain holds at half the storage and on three quantizers. Rank-32 OQER exceeds the rank-64 activation-weighted correction on every seed (+11.7† / +13.0†), and teacher disagreement falls with AWQ, MBQ and QIG alike (Table 3). With QIG on MME, for example, it is 10.95% uncorrected, 10.83% with the activation-weighted correction and 8.93% with OQER.

Ingredients and prior work. From QERA we take the object (the residual of a fixed quantized checkpoint) and from LASER the two-sided weighting. Neither is claimed. While preparing the revision we also found OBD-LLM (arXiv, April 2026), which is closer than either. As an extension it fits a low-rank adapter to the residual of a GPTQ-quantized LLM with the same two-sided whitening, using the output-gradient moment of the next-token loss, and reports perplexity. We now cite it as the closest prior work (Section 2, Appendix Y), and the whole-prompt loss arm of Table 7 is the counterpart of that weighting in our setting. The contribution is therefore the evidence for quantized VLMs with teacher fidelity as the target.

We do not rely on the margin gradient. Its lead over the whole-prompt loss is within resampling noise (Table 7, +5.0 / +4.5), as Section 6.3.2 states, while its lead over the LASER-inspired last-24-token loss is significant (+11.9† / +6.8†).

Quant Experts and SVDQuant (Appendix Y, with SVDQuant also in Table 33). SVDQuant keeps the leading singular components of the weight in a 16-bit branch and quantizes the remainder afterwards, without weighting. It belongs to the preserve-then-quantize family that SRR represents in Table 18. Quant Experts compensates weight and activation quantization of VLMs with a shared adapter and token-routed experts, weighting the input side. OQER fits one static term to the residual of a checkpoint that is already quantized and weights the output side as well, so it is complementary to both. We compare with both at the level of the objective and did not run them on our checkpoints.

(Continued in Next Response)

AuthorsOpenReview ↗

[2/4] Response to Reviewer z7Jm

(Continued from Previous Response)

Q2. How the gradient and the rank were selected, and the supported settings

The answer margin was not chosen from results, but rank 64 was first compared on the test sets. An independent selection on held-out data is consistent with the same configuration, and a new all-layer sweep states for which ranks the gain holds.

The two defaults were not chosen in the same way. The answer margin was the first output gradient we ran. It follows from the flip analysis of Section 4, and the other gradient sources of Table 7 were implemented afterwards as controls, so no result was used to prefer it. The rank was not fixed in advance. Ranks 8, 32 and 64 were first compared on POPE-adversarial and MME (layers 0 to 11, seed 0, the runs of Table 12) before the all-layer rank-64 experiments, so the choice of rank 64 is not free of test-set selection.

To test whether the choice holds without the test sets, we ran an independent selection on held-out data. We used the 3,000 COCO dev questions of Appendix H (1,500 images). None of these images is in POPE or in the calibration sets. Six of them also appear in the COCO-based subtasks of MME, and the 12 questions on these images are excluded, which leaves 2,988 questions. Teacher disagreement needs no labels. In Table 28 the dev columns are measured on a re-quantized checkpoint (checkpoint C of Appendix R) at ranks 32 and 64, and the test column repeats Table 7 (the checkpoint of the submission). The activation-weighted and the answer-margin rows of one seed share one job and one set of quantized weights.

Output moment built fromDev disagreement (%), rank 32Dev disagreement (%), rank 64Test net recovery at rank 64, POPE-adversarial / MME (Table 7)
nothing (activation-only)2.352.2934.3% / 16.6%
LASER-inspired last-24-token loss2.061.8645.5% / 26.3%
whole-prompt loss2.011.8652.5% / 28.6%
teacher-response likelihood1.651.5554.0% / 28.0%
answer likelihood1.701.4553.7% / 30.0%
answer margin (OQER default)1.741.4557.5% / 33.1%

At rank 64 the answer margin and the answer likelihood tie for the lowest dev disagreement (1.45% each, difference 0.00 points with a 95% interval of [−0.19, +0.19]). At rank 32 the teacher-response likelihood is lowest (1.65%) and the answer margin is close behind (1.74%). Without a correction the disagreement is 3.05%. The candidates of this selection are the six gradient sources at ranks 32 and 64. Among them, rank 64 with the answer margin or the answer likelihood gave the lowest observed dev disagreement, and in the same-checkpoint comparison of Table 7 these two do not differ significantly on the test sets. Four of the gradient sources ran as separate jobs (Appendix R), so we read this as consistent with the default and not as a ranking of all six. It does not make the original choice of the rank independent of the test sets.

Supported settings. Table 27 adds an all-layer rank sweep over three seeds on LLaVA-1.5-7B with AWQ-style 3-bit weights. The cells give the teacher disagreement of the activation-weighted correction minus that of OQER, in points (uncorrected model: 4.83% on POPE-adversarial, 10.87% on MME). For each seed the two corrections were built and evaluated in one job on the same quantized weights, so the comparison is paired.

Rank (extra bits/weight)8 (+0.05)16 (+0.10)32 (+0.20)64 (+0.40)128 (+0.79)
POPE-adversarial+1.29†+1.74†+0.73†+1.04†+0.81†
MME−0.56+0.63+0.55+2.11†+2.20†
  • Supported on LLaVA-1.5-7B with AWQ-style 3-bit weights: all-layer correction from rank 8 on POPE and from rank 64 on MME (Table 27). On MME the difference at ranks 16 and 32 has the same sign but is not significant, and at rank 8 OQER is not ahead.
  • Supported at rank 64: binary readouts on three backbones and with three quantizers at 3 bits (Tables 3 and 4), and short generated answers on the three backbones with AWQ-style 3-bit weights (Table 17, seed 0).
  • Benchmark-dependent: rank 8 on layers 0 to 11. In Table 12 the difference to activation weighting is −10.4 points on POPE (not significant) and +8.6† on MME, so we do not claim a gain for such budgets.
  • Mixed: multiple-choice readouts. On MMBench OQER is ahead of activation weighting in fidelity and in accuracy (+2.18 pp†), and on the full ScienceQA-IMG test set the two are level (Table 30). In the ScienceQA pilot a candidate-normalised likelihood is ahead of the fixed-rival margin (43.2% against 32.6%, and 24.2% for activation-only weighting), because a K-way answer can flip to any rival. We therefore claim no general gain for K-way readouts and give no rule for the gradient there.

(Continued in Next Response)

AuthorsOpenReview ↗

[3/4] Response to Reviewer z7Jm

(Continued from Previous Response)

Q3. Teacher disagreement by teacher-margin bin, and what the flip analysis explains

The binned rates follow the prediction of the flip analysis, and OQER has the lowest disagreement in every low-margin bin of POPE. The margin model explains where the flips occur and the gap between activation-only and output weighting, but not the order among the output-weighted variants.

Table 31 (Appendix W), LLaVA-1.5-7B, rank 64, three-seed means, teacher disagreement in %. All columns come from the stored outputs of the main experiments and share one teacher and one uncorrected model. POPE-adv. is the adversarial split (3,000 questions), the POPE split that the whole-prompt run covers.

BenchmarkTeacher margin (questions)3-bitActivation-weightedWhole-prompt lossOQER
POPE-adv.below 0.5 (263)31.1825.6020.4119.90
POPE-adv.0.5 to 1 (266)15.417.143.261.50
POPE-adv.1 to 2 (537)2.050.310.250.12
POPE-adv.above 2 (1,934)0.000.000.000.00
MMEbelow 0.5 (504)36.7131.6129.9627.78
MME0.5 to 1 (447)13.6511.338.659.17
MME1 to 2 (642)4.673.121.350.73
MME2 to 4 (634)0.470.420.160.16
MMEabove 4 (147)0.000.000.000.00

Bold marks the lowest value in each row. Most disagreements concentrate below a teacher margin of 1, and no model disagrees above a margin of 2 on POPE. The lead of OQER over the whole-prompt loss is largest in the 0.5 to 1 bin of POPE (1.76 points), where the 3-bit model disagrees on 15.41% of the questions, the activation-weighted correction on 7.14%, the whole-prompt loss on 3.26% and OQER on 1.50%. The whole-prompt loss stays close to OQER (20.41% against 19.90% in the lowest bin) and is ahead in one MME bin, in line with Table 7.

Fit of the margin model to each corrected model (POPE-adversarial, seed 0). The numbers are those of Table 24, which has a new row for the whole-prompt loss.

CorrectionSlope βLow-margin residual σ_Q1Flips predictedFlips observed
Activation-weighted0.8940.4453.8%2.8%
Whole-prompt loss0.8700.3762.8%2.2%
OQER0.9210.3542.7%2.0%
Likelihood variant0.9150.3592.8%1.8%
Diagonal output moment0.8760.4273.5%3.6%

On MME the model predicts 11.9% of flips for the activation-weighted correction and 9.9% for OQER, and 10.0% and 8.0% are observed (Table 24).

  • Explained: where disagreements arise (small teacher margins) and the difference between activation-only and full output weighting (lower σ_Q1, fewer predicted and fewer observed flips).
  • Not explained: the order among the output-weighted variants (the likelihood variant has more predicted and fewer observed flips than OQER) and the diagonal arm (fewer predicted than activation-only, more observed).
  • We therefore use the model as motivation and as a description of the corrected models, not as the mechanism that ranks every arm (Appendices Q and W).

(Continued in Next Response)

Reviewer_choCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Output-Weighted Quantization Error Reconstruction (OQER), a training-free low-rank correction method for quantized vision-language models. Instead of minimizing weight reconstruction error or intermediate activation error alone, OQER prioritizes quantization error directions that have a stronger impact on the model's final answer decisions. The method incorporates input activation statistics and output-gradient statistics into a weighted low-rank reconstruction objective. The authors evaluate OQER across different quantization methods and VLM backbones, mainly using POPE and MME. The experimental results demonstrate improved agreement with the full-precision teacher compared with activation-weighted reconstruction under comparable correction budgets.

Critical strengths
  1. Meaningful problem formulation. The paper investigates how a limited low-rank correction budget should be allocated to preserve important output decisions rather than simply minimizing weight reconstruction error. This is a reasonable and practically relevant research question. The observation that better weight reconstruction does not necessarily translate into better preservation of model decisions provides a useful perspective on quantization error compensation.

  2. Comprehensive experimental evaluation. The paper presents extensive experiments and carefully designed control studies. In particular, the authors investigate full versus diagonal output-gradient covariance matrices, coordinate permutations, sign perturbations, different gradient sources, and different low-rank correction budgets. These experiments provide useful evidence for understanding the role of output weighting and strengthen the empirical analysis.

Critical weaknesses
  1. Limited evaluation scope and an unclear connection between teacher fidelity and downstream task performance. Although the paper argues that low-rank correction should prioritize quantization errors affecting final answer decisions, the main experiments focus on binary-answer benchmarks, primarily POPE and MME. While the appendix includes limited evaluations involving short generated answers, the effectiveness of the method on more general open-ended VLM tasks remains insufficiently demonstrated. More importantly, the substantial improvements in teacher-answer fidelity do not consist ently translate into improvements in standard task accuracy. In some settings, the corrected models achieve better agreement with the full-precision teacher without improving downstream accuracy. Although preserving teacher behavior is a legitimate objective, the practical benefit of prioritizing this metric remains insufficiently established, particularly when improvements in fidelity do not lead to corresponding gains in task performance.

  2. Limited methodological and theoretical novelty relative to existing work. The proposed OQER method is closely related to existing loss-aware low-rank factorization methods, particularly LASER. Both approaches rely on a two-sided weighted low-rank reconstruction objective, and the underlying weighted SVD formulation is not fundamentally new. Compared with LASER, OQER primarily applies this formulation to the quantization residual of a fixed checkpoint and introduces answer-related output gradients to guide the correction. However, it remains unclear what fundamentally new algorithmic or theoretical insight is introduced beyond these modifications. Furthermore, the experimental results indicate that alternative output gradients can achieve performance comparable to the proposed answer-margin gradient, which further weakens the distinctiveness of the specific formulation.

Questions for authors
  1. How does OQER fundamentally differ from LASER beyond applying a similar two-sided weighted reconstruction objective to a fixed quantization residual and changing the gradient source? What new algorithmic or theoretical insight does OQER provide? In particular, given that alternative output-gradient sources achieve comparable results in Table 7, what distinguishes the proposed decision-oriented formulation from more general gradient-weighted reconstruction?

  2. Can the authors provide additional evidence that the improvements in teacher-answer fidelity translate into meaningful benefits on more general VLM tasks, especially open-ended visual question answering or longer-form generation? How should the practical value of the method be assessed when fidelity improves substantially but standard task accuracy does not consistently improve?

4 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer choC

Thank you for the detailed review. Your two weaknesses correspond to your two questions, so we answer them together. Table numbers below refer to the revised PDF. Our summary comment lists all changes and the renumbering.

Q1. Difference from LASER, what is new, and decision-oriented versus general gradient weighting

The new insight is empirical. The paper asks whether, under a limited correction budget, minimizing the reconstruction error of a quantized VLM preserves its answers. Our experiments show that at the same budget a correction with a lower reconstruction error can preserve fewer of the teacher's answers, and they show when and why. The two-sided weighted SVD that we use is not new, and point 4 states exactly how it relates to LASER and to OBD-LLM.

1. Better reconstruction does not preserve more answers. The activation-weighted correction is the exact optimum for the layer-output error. At the same rank and on the same checkpoint it captures more of that error than OQER (0.609 against 0.539, Table 6), and it still recovers fewer of the teacher's answers, 34.3% / 16.6% against 57.5% / 33.1% net recovery on POPE-adversarial / MME (a gap of +23.1† / +16.5† points, Table 2).

Reviewers Y3RJ and z7Jm name this controlled comparison as a strength of the paper. Reviewer Y3RJ appreciates that "the quantized weights, correction rank, and calibration data are kept the same", and Reviewer z7Jm describes it as a "direct, controlled validation of fidelity repair at equal correction storage". Only the weighting changes between the two rows.

2. The two criteria select different corrections. Of the residual energy weighted by the margin-gradient moment, the activation-weighted correction captures 61% and OQER 91% (new Table 32, seed 0). In the attention projections of layers 22 to 31 the output moment has an effective rank of 1.6 to 3.4 against 14.6 to 16.1 for the input moment, and there the activation-weighted correction captures only 49 to 58%. The structure of the output moment matters as well. On LLaVA-1.5-7B, keeping only its diagonal or flipping its signs brings recovery below activation-only weighting (Table 6), and permuting its coordinates lowers the captured decision-weighted energy from 0.91 to 0.53 (Appendix I). OQER is ahead under its own metric by construction, so the captured energies describe how the two corrections differ. The independent evidence for the fidelity gain is the paired comparison of Table 2 and the structure controls of Table 6.

3. The finding is not tied to one setting. It holds as an add-on to three quantizers and on three backbones (Tables 3 and 4). It holds at half the storage, where rank-32 OQER exceeds the rank-64 activation-weighted correction on every seed (Table 2). It also holds with generated answers on GQA, TextVQA and MMBench (Q2 below).

4. Relation to prior work. Neither the weighted SVD nor its use on a quantization residual is our contribution.

LASEROQER
What is factorizedthe weights of a model, for loss-aware low-rank compression with rank allocation across layersthe residual ΔW between a full-precision model and an already quantized checkpoint
Rolea compression method of its owna training-free add-on to an existing quantizer (AWQ, MBQ, QIG)
Targetthe task lossthe answers of the full-precision teacher

The closest prior work is OBD-LLM (arXiv, April 2026), which we found while preparing the revision and now cite (Section 2, Appendix Y). As an extension it fits a low-rank adapter to the residual of a GPTQ-quantized LLM with the same two-sided whitening, using the output-gradient moment of the next-token loss, and reports perplexity. Our claim is therefore limited to what these works do not study, quantized VLMs with agreement with the teacher's answers as the target. For this setting the paper shows which reconstruction criterion preserves the answers and under which conditions.

Decision-oriented versus general gradient weighting (Table 7, net recovery on POPE-adversarial / MME). The whole-prompt loss row is the counterpart of the OBD-LLM weighting in our setting.

Output moment built fromNet recoveryLead of the answer margin
nothing (activation-only)34.3% / 16.6%+23.1 / +16.5, significant
LASER-inspired last-24-token loss45.5% / 26.3%+11.9 / +6.8, significant
whole-prompt loss52.5% / 28.6%+5.0 / +4.5, not significant
answer margin (OQER default)57.5% / 33.1%

The answer-directed gradient is significantly better than the LASER-inspired last-24-token loss and not separable from the whole-prompt loss. The paper states this (Section 6.3.2). It uses the margin as the default and does not claim a margin-specific benefit.

(Continued in the next comment)

AuthorsOpenReview ↗

[2/2] Response to Reviewer choC

(Continued from Previous Response)

Q2. Broader VLM tasks, and how to assess the method when fidelity improves and accuracy does not

Broader tasks. We ran the open-ended VQA evaluation you ask for. Table 29 (Appendix V) adds GQA and TextVQA with generated answers and the official metrics, and Table 30 adds two multiple-choice benchmarks. The corrections are the ones of the main experiments, built from yes/no calibration questions and not re-calibrated (rank 64, three-seed means). On both open-ended benchmarks OQER has the lowest teacher disagreement (GQA 12.29% against 13.42% for the activation-weighted correction and 15.81% without a correction, TextVQA 27.09% against 28.27% and 30.96%) and the fewest broken answers, and on TextVQA its accuracy is significantly higher (+0.64 pp†).

BenchmarkTeacher disagreement (%): 3-bit / act.-weighted / OQERBroken: 3-bit / act.-weighted / OQERAccuracy (%): 3-bit / act.-weighted / OQEROQER − act.-weighted (pp)
GQA testdev (open-ended)15.81 / 13.42 / 12.29719 / 574.7 / 521.759.07 / 59.57 / 59.69+0.12
TextVQA val (open-ended)30.96 / 28.27 / 27.09381 / 347.3 / 312.045.40 / 45.62 / 46.26+0.64†
ScienceQA-IMG (multiple-choice)18.05 / 16.13 / 15.98134 / 128.3 / 122.065.20 / 65.38 / 65.03−0.35
MMBench-EN dev (multiple-choice)33.8 / 32.5 / 30.1107 / 106.0 / 79.356.87 / 56.90 / 59.08+2.18†

† 95% CI excludes 0. TextVQA accuracy is the VQA accuracy with partial credit, and for the broken count an answer is correct when its credit is at least 0.3. For MMBench all columns are per circular question (1,164 questions, each asked with rotated options), and a model agrees with the teacher only if it gives the same answer on every copy.

  • OQER has the lowest teacher disagreement and the fewest broken answers on all four benchmarks.
  • Its accuracy is significantly above the activation-weighted correction on TextVQA and MMBench and not separable from it on GQA and ScienceQA-IMG. On ScienceQA-IMG the two corrections are also close in fidelity (15.98% against 16.13%), which matches the limit for multiple-choice readouts that the paper reports for its ScienceQA pilot (Appendix I).
  • The paper already had a smaller test of this kind. With generated answers of up to eight tokens (Appendix J, Table 17), OQER has the fewest teacher disagreements and the fewest broken teacher-correct answers on all three backbones.
  • Generation of long responses such as captions is not evaluated, and Appendix O states this as a limit of the supported scope.

How to assess. We propose three numbers per configuration. Teacher disagreement shows how much behaviour changed. Broken answers (teacher right, model wrong) are the regressions that a user of the replaced model sees. Accuracy is the third. On a yes/no question the remaining disagreements are chance-correct (teacher wrong, changed answer right), which is why accuracy can stay flat while fidelity improves. The table below gives the three quantizers on LLaVA-1.5-7B (the runs of Table 3, with accuracies from Table 18, POPE broken counts from Table 19, the QIG difference on MME from Table 20 and the MME broken counts from Appendix U). Table 8 in the new Section 6.4 gives the three backbones.

Quantizer, benchmarkBroken: 3-bitBroken: act.-weightedBroken: OQERAccuracy (%): 3-bit / act.-weighted / OQEROQER − act.-weighted (pp)
AWQ, POPE229128.785.384.66 / 85.46 / 85.55+0.09
AWQ, MME182159.0125.074.98 / 74.96 / 75.89+0.93†
MBQ, POPE199103.780.084.90 / 85.75 / 85.64−0.11
MBQ, MME167141.7125.075.15 / 75.62 / 76.02+0.39
QIG, POPE274106.389.084.28 / 85.59 / 85.42−0.17
QIG, MME166174.0138.075.53 / 74.73 / 75.86+1.14†

† 95% CI excludes 0.

  • OQER breaks the fewest teacher-correct answers in every cell, and in all ten cells including the other backbones.
  • The cases where the activation-weighted correction is slightly more accurate are explained by chance-correct answers. With MBQ on POPE it breaks 104 answers against 80 for OQER but keeps 90 chance-correct ones against 57. The differences (−0.11 and −0.17 pp) are not significant.
  • The accuracy of OQER is never significantly below the activation-weighted correction and is significantly above it on MME with AWQ and QIG. With QIG on MME the activation-weighted correction breaks more answers than no correction (174 against 166).

We hope the open-ended evaluation you asked for and the clarified relation to prior work address your two concerns.

AuthorsOpenReview ↗

Brief Summary of Our Response to Reviewer choC

Dear Reviewer choC,

Thank you again for your thoughtful review. Your question on whether fidelity turns into benefit on broader tasks led us to the open-ended evaluation in the revision, and we are grateful for it. As the discussion period ends soon, here is a short summary of our answers.

Q1. The new insight is empirical: a correction with lower reconstruction error can preserve fewer of the teacher's answers. Activation weighting is the exact optimum for the layer-output error and captures more of it (0.609 vs. 0.539), yet it recovers fewer answers (34.3% / 16.6% vs. 57.5% / 33.1%, Table 2). On LLaVA-1.5-7B the gain also needs the full output moment: with only its diagonal or with flipped signs, recovery falls below activation-only weighting (Table 6). The weighted SVD is not claimed, and we now cite OBD-LLM as the closest prior work; our claim is limited to quantized VLMs with the teacher's answers as the target.

Q2. On GQA and TextVQA with generated answers, using the same corrections without re-calibration, OQER has the lowest teacher disagreement and the fewest broken answers (GQA: 719 / 574.7 / 521.7, TextVQA: 381 / 347.3 / 312.0 for 3-bit / activation-weighted / OQER). Its accuracy is significantly above the activation-weighted correction on TextVQA (+0.64 pp) and MMBench (+2.18 pp), and not separable from it on GQA and ScienceQA-IMG. For assessment we propose reporting teacher disagreement, broken answers and accuracy together, since chance-correct answers explain why accuracy can stay flat while fidelity improves.

We would be grateful to hear whether this addresses your concerns, and we are happy to answer any further questions.

Thank you, Authors

AuthorsOpenReview ↗

Gentle Reminder to Reviewer choC

Dear Reviewer choC,

Thank you again for your time and for the careful review.

As the discussion period is closing soon, we wanted to check whether our response addressed your concerns. If anything remains unclear, we would be glad to clarify it in the time that is left.

Thank you, Authors

Reviewer_Y3RJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OQER, a training-free low-rank correction method for quantized vision-language models. Instead of simply reconstructing the quantization error in the weights or intermediate activations, OQER uses output-gradient information to prioritize the error directions that are more likely to change the model's answers.

Critical strengths
  1. I think the motivation makes sense. When the correction budget is limited, it seems more useful to focus on errors that affect the final answer rather than simply minimizing the overall weight reconstruction error. The analysis showing that quantization-induced answer flips mostly happen on low-margin examples provides a good motivation for the method.

  2. The experimental evaluation is quite solid. On LLaVA-1.5-7B, OQER improves net flip recovery from 34.3% to 57.5% on POPE and from 16.6% to 33.1% on MME compared with activation-weighted reconstruction. The authors also test different quantizers, model backbones, correction ranks, and output gradients. I particularly appreciate the controlled comparisons where the quantized weights, correction rank, and calibration data are kept the same. This makes the improvement more convincing.

Critical weaknesses
  1. My main concern is that the improvement is mostly in teacher agreement rather than actual task performance. I understand why preserving the FP16 model's behavior is useful, especially when the quantized model is meant to replace an existing model. However, matching the teacher also means recovering its mistakes, and the standard benchmark results do not always improve. For example, in Table 21, the official MME score for LLaVA-1.5-7B with AWQ drops from 1715.9 without correction to 1695.9 with OQER. On LLaVA-1.5-13B, the score also drops from 1665.8 to 1647.3. The authors acknowledge this distinction, which I appreciate, but it makes me less convinced about the practical benefit. Most of the experiments also focus on yes/no or short-answer predictions, while longer free-form generation is not evaluated. I would like to see whether the improved teacher fidelity translates into more useful behavior in broader VLM tasks.

  2. Another concern is whether OQER provides a clear practical advantage over simply using a slightly higher quantization precision. Appendix C reports that the 3-bit model with the rank-64 correction uses 4.7 GB of peak GPU memory, compared with 4.8 GB for a regular 4-bit model. However, the 4-bit model still has considerably better MME teacher fidelity and accuracy. The low-rank correction also introduces additional decoding overhead. This makes me wonder whether the extra complexity is worthwhile in a real deployment, especially since the main accuracy experiments use simulated quantization rather than actual low-bit kernels. I also think the novelty is somewhat incremental, as the two-sided weighted SVD formulation is closely related to existing loss-aware methods such as LASER. The paper is fairly clear about this relationship, but the results suggest that generic output-gradient weighting, rather than the answer-margin formulation specifically, is responsible for much of the gain.

Questions for authors

N/A

6 comments
AuthorsOpenReview ↗

[1/4] Response to Reviewer Y3RJ

Thank you for the careful review. Table numbers below refer to the revised PDF. Our summary comment lists all changes and the renumbering.

1. Teacher agreement versus task performance, and the official MME score

You are right that the official MME score falls with OQER in the two settings you name, from 1715.9 to 1695.9 on LLaVA-1.5-7B and from 1665.8 to 1647.3 on LLaVA-1.5-13B (Table 22). The per-question results show what happens underneath. OQER is the correction that breaks the fewest answers the FP16 model gets right in all six comparisons of Table 8. Accuracy moves less than fidelity because part of the 3-bit model's disagreements are correct by chance.

On a yes/no question, every disagreement with the teacher is either broken (teacher right, model wrong) or chance-correct (teacher wrong, so the changed answer is right). Table 8 (new Section 6.4) separates the two (rank 64, three-seed means).

Backbone, benchmarkBroken: 3-bitBroken: act.-weightedBroken: OQERAccuracy (%): 3-bit / act.-weighted / OQER
LLaVA-1.5-7B, POPE229128.785.384.66 / 85.46 / 85.55
LLaVA-1.5-7B, MME182159.0125.074.98 / 74.96 / 75.89
LLaVA-1.5-13B, POPE119100.768.785.28 / 85.31 / 85.71
LLaVA-1.5-13B, MME143117.3106.376.03 / 75.67 / 75.60
Qwen2.5-VL-7B, POPE171170.087.386.44 / 86.30 / 87.28
Qwen2.5-VL-7B, MME11091.382.786.02 / 86.68 / 87.01
  • 30 to 44% of the 3-bit disagreements are chance-correct, and restoring the teacher removes part of them. This is the LLaVA-1.5-13B MME case you point to. OQER lowers the broken answers from 143 to 106 and also removes 47 of the 109 chance-correct ones, so accuracy changes by −0.44 pp (95% CI [−1.35, +0.53], not significant).
  • The accuracy of OQER is significantly above the 3-bit model in four of the six cells and significantly above the activation-weighted correction in three (intervals in Appendix U). It is never significantly below either.
  • The official MME score gives each of the 14 subtasks the same weight, so the eight subtasks with at most 60 questions hold 17% of the questions and 57% of the points (Appendix M), and a few answers move the total by tens of points. On LLaVA-1.5-7B the perception score rises with OQER (1367.0 to 1370.9) and the fall comes from the four cognition subtasks (348.9 to 325.0), while the per-question MME accuracy goes from 74.98% to 75.89% (+0.91 pp, not significant). In Table 22, OQER is above the uncorrected model in two of five settings and above the activation-weighted correction in four of five. We do not claim a gain on the official score, and the main text reports per-question accuracy for this reason.

(Continued in Next Response)

AuthorsOpenReview ↗

[2/4] Response to Reviewer Y3RJ

(Continued from Previous Response)

2. Broader tasks and free-form generation

  • Already in the paper: with generated answers of up to eight tokens scored by the official parsers (Appendix J, Table 17), OQER has the fewest teacher disagreements and the fewest broken teacher-correct answers on all three backbones.
  • New: Tables 29 and 30 (Appendix V) add two open-ended and two multiple-choice benchmarks with generated answers and the official metrics (rank 64, three-seed means).
BenchmarkTeacher disagreement (%): 3-bit / act.-weighted / OQERBroken: 3-bit / act.-weighted / OQERAccuracy (%): 3-bit / act.-weighted / OQEROQER − act.-weighted (pp)
GQA testdev (open-ended)15.81 / 13.42 / 12.29719 / 574.7 / 521.759.07 / 59.57 / 59.69+0.12
TextVQA val (open-ended)30.96 / 28.27 / 27.09381 / 347.3 / 312.045.40 / 45.62 / 46.26+0.64†
ScienceQA-IMG (multiple-choice)18.05 / 16.13 / 15.98134 / 128.3 / 122.065.20 / 65.38 / 65.03−0.35
MMBench-EN dev (multiple-choice)33.8 / 32.5 / 30.1107 / 106.0 / 79.356.87 / 56.90 / 59.08+2.18†

† 95% CI excludes 0. TextVQA accuracy is the VQA accuracy with partial credit, and for the broken count an answer is correct when its credit is at least 0.3. For MMBench all columns are per circular question (1,164 questions, each asked with rotated options), and a model agrees with the teacher only if it gives the same answer on every copy.

  • OQER has the lowest teacher disagreement and the fewest broken answers on all four benchmarks. Nothing was re-calibrated for these tasks. The corrections are the ones built from the yes/no calibration questions.
  • Its accuracy is significantly above the activation-weighted correction on TextVQA and MMBench and not separable from it on GQA and ScienceQA-IMG. On ScienceQA-IMG the two corrections are also close in fidelity (15.98% against 16.13%), which matches the limit for multiple-choice readouts that the paper reports for its ScienceQA pilot (Appendix I).
  • Generation of long responses such as captions is not evaluated, and Appendix O states this as a limit of the supported scope.

(Continued in Next Response)

AuthorsOpenReview ↗

[3/4] Response to Reviewer Y3RJ

(Continued from Previous Response)

3. Is the correction worth it compared with a higher bit-width? Simulated quantization and overhead

We agree that a 4-bit model is the better choice when it fits. With the kernels we measured it decodes faster and has the higher fidelity and accuracy on MME, while on POPE the two are similar. The correction is meant for the other cases, a quantization format that is fixed or a storage budget below one extra bit, and there it restores fidelity at a small cost. Table 26 (Appendix S, summarised in Section 6.4) now measures fidelity, accuracy, memory and latency together with real auto-gptq kernels. Memory and latency are measured on one RTX A6000.

Configurationbits/weightStored (GiB)Peak memory (GiB)ms/tokenTeacher disagreement (%), POPE / MMEAccuracy (%), POPE / MMESame answer as simulated (%), POPE / MME
W33.152.384.59161.24.41 / 11.9684.27 / 74.43100.00 / 99.87
W3 + OQER, rank 323.352.524.73163.22.59 / 8.1385.38 / 75.8299.80 / 99.92
W3 + act.-weighted, rank 643.552.674.88164.22.84 / 9.6585.39 / 74.8199.90 / 99.83
W3 + OQER, rank 643.552.674.88164.21.88 / 7.8385.56 / 76.1299.99 / 99.96
W44.163.145.09102.81.80 / 5.2785.50 / 77.1799.97 / 99.87
W4 + OQER, rank 324.363.285.24102.00.96 / 3.4585.72 / 78.05100.00 / 99.92
W4 + OQER, rank 644.553.435.39102.30.94 / 3.7985.44 / 78.0599.96 / 99.87
  • The real kernels and the simulated pipeline, run from the same packed integer weights, give the same answer on 99.8 to 100% of the items, and the largest difference between their answer margins on any of the 11,374 questions is 0.078 log-odds. In this setting (LLaVA-1.5-7B, one checkpoint, seed 0) the simulated pipeline reproduces the real kernels. These rows use a re-quantized checkpoint, so the values differ slightly from Table 3, and all rows of one bit-width share the same quantized weights. The correction factors come from the server that runs our calibration, and the packed model for the real kernels was built on another machine, where the quantization is not bit-identical. The factors were therefore fitted to a slightly different residual than the one they correct (Appendix R). This works against both corrections alike, and the shares of removed disagreements stay close to those on the checkpoint of the submission. OQER at rank 64 removes 57% (POPE) and 35% (MME) of the 3-bit disagreements here, against 60% and 33% in Table 18.
  • With real kernels the 3-bit model with rank-64 OQER has a POPE accuracy and disagreement rate similar to those of the 4-bit model (85.56% against 85.50%, and 1.88% against 1.80%) with 15% less stored weight (2.67 GiB against 3.14 GiB). The 4-bit model stays ahead on MME (accuracy 77.17% against 76.12%, disagreement 5.27% against 7.83%). The uncorrected 3-bit model reaches 84.27% and 74.43%.
  • The correction is a fidelity repair for two situations, a quantization format that is fixed and a storage budget below one extra bit. Rank 32 costs +0.2 bits/weight, lowers the disagreement of the 3-bit model from 4.41% to 2.59% on POPE and from 11.96% to 8.13% on MME, and is ahead of the finer quantization grids that we tested at a similar budget (Table 13).
  • A correction also helps on top of the 4-bit model, where rank 32 lowers disagreement from 1.80% to 0.96% on POPE and from 5.27% to 3.45% on MME. Over three seeds in simulation, both corrections lower the disagreement of the 4-bit model on MME from 4.80% to 3.72%, and OQER is ahead of activation weighting only on POPE (0.98% against 1.39%†). The advantage of output weighting is therefore mainly a result for 3 bits.
  • Latency. With the available kernels the 4-bit model decodes faster than the 3-bit model (102.8 against 161.2 ms/token, and 164.2 with a rank-64 correction), because the 3-bit CUDA kernel is the slower one. On these kernels the unfused correction branch adds at most 3%. On the 3-bit model it adds 1.2% at rank 32 and 1.9% at rank 64. On the 4-bit model the three medians are close on this host (102.0 and 102.3 against 102.8), and on a second host rank 64 is 3% above the uncorrected model (101.9 against 99.0 ms/token). With the faster exllamav2 kernel, which exists for 4 bits only, the 4-bit model takes 22.2 ms/token and the unfused branch raises this by about 50%, to 33.3 ms/token at rank 32 and 33.6 at rank 64 (second host). On a fast kernel the branch is therefore a large part of the decoding time. We make no speed claim, and when latency is the binding constraint the higher bit-width is the better choice.

(Continued in Next Response)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers,

We sincerely thank all reviewers for their careful reading and constructive feedback. We are encouraged that the reviewers found the motivation and the problem formulation sound (Reviewers Y3RJ, choC), valued the controlled comparisons in which the quantized checkpoint, the rank and the calibration data are held fixed (Reviewers Y3RJ, z7Jm), and recognized the control studies on the structure of the output weighting and the results across quantizers and backbones (Reviewers choC, z7Jm).

Following your suggestions, we added experiments and clarifications and uploaded a revised PDF. All changes are in blue, and the sentences of the submitted version are kept. The main text now uses 10 pages. Below we summarize the key updates.

1. What the fidelity gain means in practice

  • [Section 6.4, Table 8, Appendix U] Broken and chance-correct answers. Each disagreement with the teacher is split into an answer the teacher had right and one it had wrong. OQER breaks the fewest teacher-correct answers in all six backbone and benchmark settings. (Y3RJ, choC, z7Jm)
  • [Section 6.4, Appendix S, Table 26] Real low-bit kernels and the comparison with 4 bits. Fidelity, accuracy, memory and latency are measured together with real kernels. The real kernels and the simulated pipeline give the same answer on at least 99.8% of the questions. (Y3RJ, z7Jm)
  • [Appendix V, Tables 29 and 30] Broader tasks. We added two open-ended benchmarks (GQA, TextVQA) and two multiple-choice benchmarks (ScienceQA-IMG, MMBench) with generated answers and the official metrics. (Y3RJ, choC)

2. Choice of the configuration and analysis

  • [Appendix T, Tables 27 and 28] Supported ranks and a held-out check. An all-layer rank sweep over three seeds shows from which rank the gain holds, and the default gradient and rank are checked on held-out questions that play no part in the evaluation. (z7Jm)
  • [Appendix W, Table 31, new row of Table 24] Teacher-margin bins. Teacher disagreement is reported per bin of the teacher margin for the uncorrected model and three corrections. (z7Jm)
  • [Appendix X, Table 32] Layer-wise view of the two weightings. The table shows how much of the residual each correction captures under the activation metric and under the decision metric. (choC, z7Jm)

3. Positioning and scope

  • [Section 2, Appendix Y, Table 33] Prior work. We now cite OBD-LLM as the closest prior work and discuss SVDQuant and Quant Experts. The first contribution states that the weighting has precedents. (Y3RJ, choC, z7Jm)
  • [Appendix O] Supported scope. A new paragraph lists the settings that the evidence supports and those it does not. (Y3RJ, z7Jm)
  • [Appendix R, Table 25] Reading guide. One table lists the arm names, splits and seeds behind every table. The appendix also names the quantized checkpoints of the added runs and reports one calibration image that is also an MME image, with its negligible effect. (z7Jm)

4. Notes on the revised PDF

  • Table numbers. Section 6.4 contains a new Table 8, so Tables 8 to 23 of the submission are Tables 9 to 24 in the revision. The new appendix tables are Tables 25 to 33. Section and appendix letters are unchanged, and our responses use the numbers of the revised PDF.
  • Qualified statements. Five statements of the submitted text are qualified in blue where the added material made them incomplete: the first contribution, the use of GQA, the disjointness of calibration and evaluation images, the scope of simulated quantization and the hardware statement of Section 3.
  • Reference name. The work previously cited as LASER in the submission and our responses now appears as LSVD (Wang et al., 2026) in the revised manuscript, reflecting its updated title. Both refer to the same work, and all comparisons remain unchanged.

We hope that the revised manuscript and our individual responses address your concerns. We thank you again for the time you spent on our work and would be glad to answer further questions.

Sincerely, Authors of Submission 54850

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.