acceptodds
Under review as a conference paper at ICLR 2027

Beyond Local Error: Functional Reconstruction for Vision-Language Model Quantization

Abstract

Post-training quantization (PTQ) aims to reduce the cost of vision-language models while retaining their capabilities. Reconstruction-based methods pursue this goal by matching quantized intermediate outputs to their full-precision counterparts, with modality-aware approaches further balancing visual and textual errors. Yet an error’s impact depends on what happens next: subsequent operations can respond differently to perturbations of similar magnitude. A smaller local error therefore need not better preserve the computation that follows. We introduce Modality- Native Functional Equalization (MNFE), a PTQ method based on multimodal functional reconstruction. MNFE evaluates quantization errors at module-specific states formed by downstream operations and balances their contributions across visual evidence, textual context, and answer information. A structured search refines channel scales through bounded group-wise adjustments. Extensive ex- periments on Qwen2.5-VL-7B and InternVL2.5-8B show that MNFE achieves the highest nine-benchmark average among evaluated quantized methods under W3A16, W4A8, and W4A6. Notably, under W4A6, MNFE retains 94.8% of the full-precision average score on InternVL2.5-8B. Our code will be made publicly available upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, vision-language models, multimodal large language models, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HYy8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Modality-Native Functional Equalization (MNFE), a post-training quantization method for vision-language models. To address these limitations, the authors introduce MNFE, which consists of three components. Functional reconstruction partitions the projections within each Transformer block into three groups and employs group-specific reconstruction targets that incorporate subsequent operations. Hierarchical role-balanced reconstruction distinguishes visual, textual context, and answer tokens, assigning adaptive reconstruction weights based on their reference-state energy characteristics, while structured scale search improves the calibration procedure. Extensive experiments across multiple VLMs and quantization configurations demonstrate that MNFE achieves SOTA average performance among the evaluated methods in most settings.

Critical strengths
  1. The paper identifies limitations of existing PTQ methods for VLMs and addresses them using a unified framework that integrates functional reconstruction, role-adaptive weighting, and structured scale search. These complementary designs demonstrate methodological novelty and achieve strong quantization performance.
Critical weaknesses
  1. Despite its relatively complex design, hierarchical allocation improves the average score by only 0.1 points over fixed role weighting, while degrading performance on 5 out of 9 benchmarks. This marginal and inconsistent improvement raises concerns about whether the additional complexity is justified.
  2. Insufficient ablation studies. The paper lacks an isolated ablation of the structured scale search and a complete component-wise ablation of the three proposed components. Existing results suggest that functional reconstruction accounts for most of the performance gains, while the other components provide only marginal improvements or lack independent validation. Consequently, the main demonstrated contribution appears to lie primarily in functional reconstruction, rather than the full framework.
Questions for authors
  1. The effectiveness of hierarchical role-balanced reconstruction and structured scale search is not sufficiently supported by the current ablation studies. Could the authors provide additional experiments to isolate and quantify the contributions of these two components?
  2. Although MNFE emphasizes going “beyond local error,” the downstream operations considered in functional reconstruction are largely confined to the current Transformer block and the input RMSNorm of the subsequent block. This appears to be a form of short-range reconstruction rather than modeling long-range error propagation. Could the authors clarify the extent to which MNFE captures downstream effects and provide evidence that these local functional targets adequately reflect quantization errors propagated through deeper layers?
1 comment
AuthorsOpenReview ↗

Response to Reviewer HYy8

We sincerely thank the reviewer for the careful evaluation of MNFE and for raising important questions about the individual contributions of its components and the scope of its downstream reconstruction objectives. We particularly appreciate the concern that the original presentation did not sufficiently distinguish the central contribution from the complementary components. We conducted additional component-wise experiments and cross-layer diagnostics to address these concerns.

Q1. Independent contributions of hierarchical weighting and structured scale search.

We agree that the original ablations did not fully isolate the contributions of all three components. We therefore conducted a complete 2×2×2 component ablation on Qwen2.5-VL-7B/W4A8, varying reconstruction targets (Local vs. Functional), role weighting (Fixed vs. Adaptive), and scale search (Initialization only vs. Initialization + Refinement), while keeping the calibration data and evaluation protocol fixed. The eight configurations are:

TargetWeightSearchAverage
LocalFixedInit.73.89
LocalFixedRefine74.21
LocalAdaptiveInit.73.87
LocalAdaptiveRefine74.35
FunctionalFixedInit.74.07
FunctionalFixedRefine74.57
FunctionalAdaptiveInit.74.17
FunctionalAdaptiveRefine74.81

Across the four matched comparisons for each factor, the average effects are:

ComponentAverage gain
Functional Reconstruction+0.33
Adaptive Role Weighting+0.11
Group-wise Refinement+0.49

These results clarify the complementary roles of the three components. Functional Reconstruction provides the central change in the reconstruction objective, adaptive weighting further balances multimodal reconstruction errors, and group-wise refinement improves scale selection beyond initialization. We acknowledge that adaptive weighting provides a modest improvement and does not improve every benchmark. However, its additional three-seed comparison demonstrates consistently positive gains of +0.28, +0.41, and +0.23 points over fixed weighting. Group-wise refinement also provides gains of +0.39 to +0.64 points across the tested configurations. We believe these controlled experiments provide stronger evidence for the individual and complementary contributions of MNFE.

Q2. Does MNFE model long-range error propagation?

We thank the reviewer for raising this important conceptual question. We clarify that MNFE does not explicitly simulate quantization-error propagation through the entire remaining Transformer stack. Instead, it selects module-specific reconstruction targets that incorporate relevant subsequent operations at carefully chosen functional boundaries, including residual addition, normalization, and multiplicative gating. To investigate whether these targets better reflect deeper computational deviations, we conducted an additional cross-layer diagnostic on Qwen2.5-VL-7B/W4A8. Using 18 matched intervention positions, we measured Spearman correlations between reconstruction objectives and hidden-state deviations at progressively deeper Transformer blocks.

Comparison targetLocalFunctionalDifference
Subsequent block 1 output0.93420.9425+0.0083
Subsequent block 2 output0.88970.9068+0.0172
Subsequent block 4 output0.86100.8828+0.0218
Subsequent block 8 output0.81710.8326+0.0155

Functional Reconstruction achieves higher mean correlations at all four examined depths. This suggests that module-specific functional targets provide a modestly better proxy for certain deeper hidden-state deviations.

We also examined final-output metrics. The correlation with final-answer KL is 0.7228 for Functional Reconstruction versus 0.7459 for Local Reconstruction; the corresponding labeled-answer NLL correlations are 0.0222 and 0.0185. We therefore interpret the results as evidence that module-specific functional targets better reflect selected downstream hidden-state deviations, rather than claiming that MNFE explicitly models or consistently predicts full-network error propagation. We will clarify this scope in the revised manuscript. Importantly, this distinction does not change the central motivation of MNFE: reconstruction targets should account for relevant downstream computations rather than only local projection outputs.

Closing remarks. We sincerely thank the reviewer for the thoughtful comments on component-wise validation and the conceptual scope of downstream reconstruction. In particular, the question about long-range error propagation prompted us to conduct additional cross-layer diagnostics and sharpen the methodological positioning of MNFE. We greatly appreciate the careful reading and constructive suggestions, and would be delighted to address any further questions or continue the technical discussion.

Reviewer_cQneOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MNFE, a post-training quantization method for vision-language models. It selects module-specific downstream reconstruction targets for attention, MLP gate/up, and down projections, and combines them with normalized, role-weighted errors for visual, context, and answer tokens. Channel scales are selected through statistics-based initialization and bounded group-wise search. Experiments cover two model families, four model sizes, and three precision settings, with fixed-candidate-pool ablations examining reconstruction targets and role weighting.

Critical strengths
  1. The reconstruction targets have a reasonable motivation: normalization, gating, and residual connections can change the effect of quantization errors. Controlled ablations with shared candidates and fixed role weights support Functional over the selected Local baseline, particularly at W4A8 and W4A6.

  2. The evaluation covers multiple models and precisions. The supplement clarifies role scores, search settings, and parameter folding, and provides additional calibration-sampling results that strengthen the evidence in one model–precision setting.

Critical weaknesses
  1. The selected targets may miss relevant downstream errors, and their advantage over block-level reconstruction remains unclear. For attention, the objective reconstructs NMLP(r)N_{\mathrm{MLP}}(r), while rr also directly contributes to the block output r+dr+d. A change in residual magnitude can be largely removed by RMSNorm while still affecting the block output. This identifies a potential limitation of the target, although it does not establish an observed failure. The current ablations compare Functional with Local but do not include a matched block-output reconstruction control. Consequently, the benefit of the particular target choices remains insufficiently established.

  2. The independent value of group-wise search and adaptive role weighting is unclear. Search is presented as a main contribution, but accuracy and calibration time are not compared against initialization alone. The adaptive fixed-pool control and the full method both report a W4A8 average of 74.8, leaving the additional benefit of search unclear. Adaptive weighting improves over fixed weights by only about 0.1 points. The multi-seed comparison between MNFE and MBQ does not establish the stability of these component gains or the smaller margins over stronger baselines.

Questions for authors
  1. Under matched candidates, role weights, and scoring budgets, how does a common block-output reconstruction target perform? Does the attention target overlook residual-magnitude errors that normalization suppresses?

  2. What accuracy and measured calibration time are obtained with initialization alone and with group-wise refinement? The similar averages of the fixed-pool control and full method make the additional value of search unclear.

  3. Are the gains from adaptive weighting over fixed weights, and from MNFE over the strongest baseline in the tested setting, consistent across calibration draws? The current MNFE–MBQ comparison does not resolve these narrower questions.

1 comment
AuthorsOpenReview ↗

Response to Reviewer cQne

We sincerely thank the reviewer for the careful technical assessment of MNFE. We particularly appreciate the insightful observation regarding residual-magnitude errors under RMSNorm and the questions about the independent contribution and computational cost of group-wise search. These comments motivated targeted experiments that helped us better characterize our reconstruction objectives.

Q1. Does the attention target overlook residual-magnitude errors?

We thank the reviewer for identifying this subtle issue. We agree that RMSNorm can suppress certain residual-magnitude perturbations. Consequently, matching only the normalized attention target NMLP(r)N_{\mathrm{MLP}}(r) does not necessarily minimize the pre-normalization residual error. To investigate this concern, we conducted a matched ablation on Qwen2.5-VL-7B/W4A8, using the same 128 calibration examples, fixed V/A/C weights, and 3,584 attention candidates. Gate/up and down use identical Functional scales across variants.

VariantAverageResidual norm error (×10⁻⁵)
Local73.958.86
Functional74.219.45
Residual-only73.828.28
Combined74.118.92
Attention block output74.087.80
Block-All74.057.80

Functional Reconstruction achieves the highest nine-benchmark average, although block-output reconstruction obtains lower residual-norm error. Adding an explicit residual reconstruction term also reduces this error but does not improve the benchmark average over Functional Reconstruction.

This result highlights an important distinction: minimizing residual-magnitude discrepancy alone is not equivalent to improving downstream task performance. Our attention target is designed to preserve the normalized representation consumed by the subsequent MLP, rather than to minimize every intermediate-state discrepancy. We will clarify this design rationale and its trade-off in the revised manuscript.

Q2. Independent contribution and calibration cost of group-wise refinement.

We appreciate the reviewer pointing out that the original experiments did not sufficiently isolate initialization from refinement. We therefore conducted paired experiments on Qwen2.5-VL-7B, comparing initialization alone (A) with initialization followed by bounded group-wise refinement (B). Both variants start from the same initialization procedure and use the same reconstruction targets, calibration examples, and role-weighting strategy.

SettingInitializationWith refinementGain
W4A8, adaptive74.1774.81+0.64
W4A8, fixed74.0774.57+0.50
W3A16, adaptive74.3974.78+0.39

The improvement is not solely driven by OCRBench: excluding OCRBench, the remaining eight benchmarks still improve by 0.33 points under adaptive weighting and 0.25 points under fixed weighting at W4A8. We also measured calibration time. For the fixed-weight W4A8 pair, refinement increases calibration time from 31.34 to 97.60 minutes. This demonstrates an accuracy–calibration-cost trade-off. We further compared refinement with the fixed-candidate-pool control:

W4A8 settingRefinementFixed pool
Adaptive74.8174.75
Fixed74.5774.68

Thus, group-wise refinement provides additional gains over its initialization family, while the fixed candidate pool remains competitive. We position refinement as an additional optimization mechanism, rather than claiming that it universally dominates every candidate-selection strategy.

Q3. Stability of adaptive weighting across calibration draws.

We agree that the previous MNFE-versus-MBQ comparison did not directly establish the stability of the smaller adaptive-versus-fixed weighting gain. We therefore conducted a paired comparison on Qwen2.5-VL-7B/W4A8. Within each seed, both variants use identical calibration examples, Functional Reconstruction targets, and a frozen candidate pool, with only the role weights changed.

SeedFixed V/A/CEnergy-adaptiveGain
4274.2174.49+0.28
4373.7574.16+0.41
4474.3074.53+0.23
Mean ± sample SD74.09 ± 0.3074.40 ± 0.20+0.31 ± 0.10

The gain is consistently positive across all three calibration draws. Although modest, this provides direct evidence that adaptive weighting offers a reproducible complementary benefit in the tested model–precision setting.

Closing remarks. We are particularly grateful for the reviewer's technically insightful observation about RMSNorm and residual-magnitude preservation. It prompted us to investigate alternative reconstruction targets and better understand the relationship between intermediate-state fidelity and downstream task performance. We also sincerely appreciate the suggestions regarding search ablations and calibration cost. We would be very happy to address any further questions and welcome continued technical discussion.

Reviewer_v98rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MNFE, a post-training quantization method for vision-language models that combines module-specific downstream functional reconstruction, hierarchical role-balanced error weighting, and channel-scale optimization. Experimental results show that MNFE achieves better overall performance than the evaluated quantization baselines, without introducing additional inference-time operations.

Critical strengths
  1. Well-motivated method design. The proposed module-specific functional reconstruction considers downstream transformations of quantization errors, providing a reasonable alternative to conventional local reconstruction objectives.
  2. Extensive experimental evaluation. The experiments cover multiple VLM architectures, model sizes, and quantization precisions, demonstrating strong overall performance compared with existing baselines.
Critical weaknesses
  1. Limited justification and validation of hierarchical role weighting. The proposed hierarchical weighting relies on reference target-to-input energy ratios, but their relationship to quantization sensitivity or task-level importance is not sufficiently justified. Moreover, Appendix Table 8 shows only a 0.1-point improvement over fixed V/A/C weights. Although MNFE outperforms MBQ and QIG in the main experiments, these comparisons involve different quantization pipelines and cannot isolate the contribution of the weighting scheme. Could the authors compare MBQ- and QIG-style weighting strategies under the same functional reconstruction objective and scale-search settings?
  2. Comparisons with existing reconstruction methods are somewhat limited. Prior methods such as BRECQ and OmniQuant have already explored block-wise reconstruction to account for downstream computational dependencies. Although MNFE further introduces module-specific reconstruction targets, the experiments primarily demonstrate improvements over local reconstruction, without comparisons against these established reconstruction approaches. Consequently, the distinct benefits of the proposed module-specific targets remain insufficiently demonstrated.
Questions for authors
  1. Unusual RTN performance degradation. In Appendix Table 9, RTN achieves an average score of only 3.3 on Qwen2.5-VL-32B under W4A6, with near-zero scores on several benchmarks, whereas it achieves 72.0 on InternVL2.5-38B under the same precision. Could the authors explain this unusually large discrepancy and provide diagnostic evidence for the observed performance collapse?
  2. Calibration data sensitivity. MNFE uses only 128 COCO image–caption pairs for calibration, while the evaluation spans diverse tasks such as OCR, chart understanding, and document VQA. How sensitive is MNFE to the calibration data distribution? In particular, would using document- or chart-oriented calibration samples change the learned role weights and downstream performance?
1 comment
AuthorsOpenReview ↗

Response to Reviewer v98r

We sincerely thank the reviewer for recognizing MNFE's motivation and extensive experimental evaluation. We particularly appreciate the insightful questions on role weighting, reconstruction granularity, and calibration robustness, which motivated additional controlled experiments.

Q1. Justification and validation of hierarchical role weighting.

We clarify that reference target-to-input energy ratios are not intended as direct estimates of task importance or quantization sensitivity. Rather, they provide candidate-independent signals to adaptively balance reconstruction errors across visual, answer, and context roles.

To isolate this contribution, we conducted matched experiments on Qwen2.5-VL-7B/W4A8, keeping calibration samples, Functional Reconstruction targets, candidate scales, and scoring budgets identical within each seed, while changing only the weighting strategy.

SeedMBQ-styleQIG-styleFixed V/A/CEnergy-adaptive
4273.3173.5274.2174.49
43––73.7574.16
44––74.3074.53
Mean (42–44)––74.0974.40

Energy-adaptive weighting outperforms MBQ-style and QIG-style weighting in the seed-42 comparison. More importantly, its gains over fixed V/A/C weighting are consistently positive: +0.28, +0.41, and +0.23 points across three seeds, yielding +0.31 ± 0.10 points (sample SD). These results support the effectiveness of reference-energy-based hierarchical allocation as a complementary component of MNFE.

Q2. Comparison with existing reconstruction methods.

To further examine the benefit of module-specific target selection, we conducted a matched comparison on Qwen2.5-VL-7B/W4A8 across three calibration seeds. All variants share the same calibration samples, role-weighting protocol, candidate scales, and scoring budgets within each seed.

SeedLocal outputCommon block outputModule-specific Functional
4274.1274.0574.21
4373.7473.4573.75
4474.1274.0874.30
Mean73.9973.8674.09

Module-specific Functional Reconstruction outperforms the common block-output control in all three draws (+0.16, +0.30, and +0.23 points). This supports our motivation that different projection groups benefit from reconstruction targets tailored to their downstream computations. We clarify that this is a controlled block-output ablation, not a full reimplementation of BRECQ or OmniQuant.

Q3. Unusual RTN degradation on Qwen2.5-VL-32B/W4A6.

We thank the reviewer for identifying this anomaly. We audited 21,673 matched evaluation items and found:

  • W4A6 RTN produced 17,099 empty decoded responses (78.90%).
  • W4A8 RTN produced zero empty responses.
  • MNFE W4A6 and InternVL2.5-38B/W4A6 RTN also produced zero empty responses.

We also corrected the handling of positive partial-credit ANLS scores in DocVQA. The seven-task average changes from 3.3308 to 3.3201, still rounding to 3.3. Thus, the evaluator issue does not explain the collapse.

These findings indicate a severe generation failure specific to the archived Qwen2.5-VL-32B/W4A6 RTN run. We are conducting further diagnostics, including generated-token traces, first-step EOS statistics, stopping reasons, and layer-wise activation statistics, to investigate its mechanism. We will report the findings without attributing the failure to an unverified cause.

Q4. Calibration data sensitivity.

Our protocol follows the 128-example COCO image-caption calibration setting used in MBQ, enabling controlled comparisons with existing VLM quantization methods.

We additionally evaluated three independently sampled COCO calibration draws on Qwen2.5-VL-7B/W4A8. MNFE consistently outperforms MBQ, with an average gain of 2.8 points. We acknowledge that different COCO sampling draws do not establish robustness to calibration-domain shifts. We will clarify this distinction and identify document- and chart-oriented calibration as an important direction for future investigation.

Closing remarks. We sincerely thank the reviewer for these valuable suggestions. The questions on weighting comparisons, reconstruction granularity, and the RTN anomaly have helped us strengthen our experimental analysis. We greatly appreciate the careful assessment and constructive discussion, and would be delighted to address any further questions or continue the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the motivation and empirical potential of MNFE. Their insightful comments have motivated additional controlled experiments and helped us further clarify the central contribution of our work.

The central contribution of MNFE is Module-specific Functional Reconstruction. Our key insight is that the impact of quantization error depends not only on its local magnitude, but also on the downstream computations that process it. Different projection groups participate in different subsequent operations, such as residual addition, normalization, and multiplicative gating. Therefore, a uniform local-output or block-output reconstruction criterion may not adequately reflect their distinct functional effects. MNFE addresses this by designing module-specific downstream functional targets for different projection groups, providing a more computation-aware objective for channel-scale selection.

Our existing controlled ablations provide strong evidence for this central design. On Qwen2.5-VL-7B, under matched candidate pools and fixed role weights, Functional Reconstruction consistently improves over Local Reconstruction:

PrecisionLocalFunctionalGain
W3A1673.774.4+0.63
W4A871.374.7+3.34
W4A666.369.0+2.65

Notably, all nine benchmarks improve under W4A8 and W4A6. Additional matched experiments further show that module-specific functional targets outperform a common block-output target across three calibration draws (+0.16, +0.30, and +0.23 points).

Beyond this central contribution, MNFE integrates two complementary mechanisms:

  1. Hierarchical Role Balancing adaptively allocates reconstruction errors across visual, answer, and context roles. Additional paired experiments demonstrate consistent gains over fixed V/A/C weighting across three calibration draws, averaging +0.31 points.
  2. Bounded Group-wise Scale Search refines channel scales under the proposed functional objective. Additional experiments show improvements of +0.39 to +0.64 points over initialization across the tested configurations, with calibration costs explicitly quantified.

We have also conducted additional residual-target ablations, cross-layer diagnostics, and an audit of the anomalous RTN result. These results further clarify the effectiveness and scope of MNFE.

We sincerely appreciate the reviewers' valuable suggestions. We believe the additional experiments and clarifications provide stronger evidence for our central contribution and a more comprehensive understanding of the complementary roles of the proposed components. We address each reviewer's comments below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.