acceptodds
Withdrawn from ICLR 2027

Beyond Reconstruction Loss in Post-Training Quantization: Balanced Fitting for Large Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training quantization (PTQ) enables efficient deployment of large vision-language models (LVLMs), but is typically calibrated on a small set while expected to generalize across diverse downstream tasks. Although recent PTQ methods for LVLMs incorporate sensitivity signals, they still minimize reconstruction loss with respect to the full-precision model, potentially over-preserving FP behavior and calibration-specific bias. Rather than treating quantization solely as an error to be minimized, we observe that it can also provide beneficial regularization for certain layers and modalities. Motivated by this observation, we propose Balanced Fitting, a quantization effect-based framework that balances precision and regularization beyond reconstruction-based optimization. By measuring layer- and component-wise quantization effects for weights, vision activations, and text activations, Balanced Fitting combines fine-grained fitting for sensitive components with coarser fitting to exploit potential regularization benefits. Experiments on multiple LVLMs show that our method consistently outperforms prior PTQ approaches under both weight-only and weight-activation quantization, while lower reconstruction loss does not reliably translate into better downstream performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-Training Quantization, Model Compression, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PrZXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Balanced Fitting for LVLM post-training quantization. It measures how quantizing individual layer components changes calibration-task NLL, then uses these effects to assign different scaling-search budgets across layers while keeping the reconstruction objective. It achieves better average performance than the compared baselines across different LVLMs in the reported results.

Critical strengths

Using task-NLL changes to allocate layer-wise scaling-search budgets is an interesting idea. Table 12 also shows that selectively using fewer candidates can improve downstream performance, suggesting that uniformly increasing search granularity is not always beneficial for LVLMs quantization.

Critical weaknesses
  1. Adaptive search-budget allocation is the core contribution of Balanced Fitting, but its incremental benefit is less clear than the main results in Tables 1 and 2 suggest. Uniform CWE-20 in Table 8 appears to be the most direct control, since it removes adaptive allocation. It also appears to be a configuration of AWQ-style scaling search, but its differences from the AWQ implementation in Table 7 are not explained. If so, AWQ and other scaling-based LVLM-specific methods should be evaluated under the same configuration where applicable. Uniform CWE-20 should replace RTN, or at least be added, as the primary basic baseline in the main results, since it better isolates the proposed contribution. Some gains over this control are very small: on Qwen2-VL W3A16, the average increases from 70.47 to 70.56. Without clearer implementation details and evidence that these small gains are reliable, it is difficult to judge how much improvement comes specifically from the proposed allocation strategy.

  2. The practical value beyond the small-calibration setting remains unclear. The main experiments use only 64 calibration pairs. If Balanced Fitting helps primarily by limiting calibration overfitting, increasing the calibration set could reduce its advantage. Given the small gains over Uniform CWE-20 in Table 8, it is important to establish whether these gains persist with more calibration data.

These concerns limit my confidence in the method’s independent contribution and practical advantage, so I lean toward rejection.

Questions for authors

My questions mainly reflect the concerns in Critical Weaknesses. I hope the authors can address these concerns either by answering the questions below or in another proper way.

  1. What exactly differs between Uniform CWE-20 in Table 8 and the AWQ implementation in Table 7?

  2. How do LVLM-specific methods (MBQ and QIG) perform under the same underlying CWE configuration?

  3. What is the actual gain of Balanced Fitting when the comparison is fair enough (against Uniform CWE-20 version of AWQ, MBQ and QIG)? I would currently consider the results in Table 8 are more fair, which actually seems marginal on Qwen2-VL.

  4. How do these fairer comparisons change with larger calibration sets?

  5. Minor: Is there a better term than “positive quantization effect”? It sounds unintuitive to imply that quantization inherently improves model quality.

Reviewer_q5rqOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Balanced Fitting, a post-training quantization framework that adaptively controls the granularity of channel-wise equalization to improve quantization performance in large vision-language models. Rather than uniformly minimizing reconstruction error, the method estimates layer- and component-wise quantization effects for weights, vision activations, and text activations. These effects are transformed into importance scores, augmented with a layer-wise conflict score, and used to adapt the granularity of CWE search across layers. The method is evaluated on several LVLMs under W3A16 and W4A8 settings, with results suggesting improvements over existing PTQ baselines.

Critical strengths
  1. The paper discusses a relevant problem in LVLM quantization: uniform fitting strategies may not adequately account for heterogeneous quantization sensitivity across layers and modalities. The idea of using measured quantization effects to allocate search resources adaptively is intuitive and potentially useful.

  2. The experiments and visualization are comprehensive covering various baselines, benchmarks and LVLM models.

Critical weaknesses
  1. The claimed departure from reconstruction-based PTQ appears overstated. Despite the framing of “Beyond Reconstruction Loss,” the proposed method retains the existing CWE quantization procedure and reconstruction objective, primarily changing the number of search candidates allocated to each layer. Thus, the main contribution appears to be an adaptive search-budget allocation strategy built on top of an existing reconstruction-based PTQ method, rather than a fundamentally different quantization objective or framework beyond reconstruction-based optimization. The authors should clarify precisely in what sense the method goes beyond reconstruction loss, and distinguish the use of an additional signal for allocating search resources from directly optimizing a new objective.

  2. The effectiveness of the method appears limited, particularly when scaling to larger models. The gains appear relatively modest in several settings, and its effectiveness is less convincing on the 26B model. The authors should provide additional experiments across model sizes and settings, along with statistical analyses to establish whether the observed improvements are consistent and statistically meaningful.

Questions for authors
  1. The paper only emphysizes the importance of analyzing the granularity, but lacks direct connection between performance and specific measurement. Quantitative/intuitive analysis about why the proposed method can boost the performance should be investigated in depth.

  2. The allocation procedure relies on several manually designed components, including the piecewise transformation of quantization effects into importance scores, the standard-deviation-based conflict score, and the budget-allocation rule involving a percentile threshold and a concentration factor. The paper does not sufficiently establish why these particular formulations are necessary or whether they are consistently better than simpler alternatives. More systematic ablations are needed to isolate the contribution of each design choice. In particular, the authors should compare the proposed importance and conflict scores against simpler alternatives and evaluate the allocation strategies under matched total search budgets.

  3. Sensitivity to key hyperparameters should be analyzed in detail. For example, in Table 3, the performance remains stable even if the values of hyperparameters become zero, questioning their necessity.

  4. The paper reports calibration wall-clock times using three RTX A6000 GPUs concurrently for the proposed method's effect analyses, compared with one GPU for each baseline. These measurements do not establish lower total computational cost or resource requirements.

  5. Figure 5 provides limited insight into the layer-wise behavior of the method. It visualizes layer-wise budget allocation and downstream performance. However, the paper does not adequately explain what the resulting budget distribution reveals about the model. In particular, it remains unclear why specific layers receiving substantially more search budget correspond to identifiable architectural or multimodal-processing characteristics, and whether the allocated budgets correlate with independently measured downstream sensitivity. The proposed interpretation and allocation mechanism are not yet equally well validated. The authors should examin the relationship between layer depth, module type, component-wise quantization effects, allocated budgets, and downstream performance changes.

  6. The counterintuitive results in Table 4 are insufficiently explained. The Low setting outperforms the High setting, which is unexpected given that a larger fitting budget allows for finer-grained CWE search and is presumably intended to improve quantization performance. The authors attribute this result to “excessive fitting can be detrimental,” which remains vague and lacks supporting evidence.

Reviewer_AkvdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Balanced Fitting, a post-training quantization framework for large vision-language models. By evaluating the quantization effects of weights, visual activations, and textual activations across different layers, the method allocates different CWE search budgets to different components, performs finer-grained fitting on sensitive components, and explores potential regularization benefits by reducing the fitting precision of some components. The authors conduct experiments on multiple vision-language models, two quantization settings, and five downstream benchmarks. The results demonstrate that the proposed method achieves certain performance advantages.

Critical strengths
  1. This paper addresses the discrepancy between traditional reconstruction loss and downstream task performance. By allocating search budgets differentially, it balances fitting precision against potential regularization benefits, which has certain research value.

  2. The paper conducts experiments across multiple models and quantization settings and provides search budget comparisons, quantization effect stability analyses, and calibration data domain shift experiments, offering some support for the effectiveness of the proposed method.

  3. The proposed method does not require model retraining and achieves differential fitting primarily by adjusting the search budgets in the existing CWE optimization process. Its design is relatively simple and has certain practical application potential.

Critical weaknesses
  1. Comparisons with recent quantization methods are insufficient. Although the paper compares the proposed method with methods such as QIG, direct comparisons with other recent vision-language model quantization methods from 2026 remain limited. Therefore, the current experiments are insufficient to fully demonstrate the competitiveness of the proposed method relative to the latest research advances.

  2. The coverage and recency of the evaluated models are insufficient. The paper primarily uses LLaVA-OneVision-7B, InternVL2-8B/26B, and Qwen2-VL-7B, without validation on newer-generation vision-language models. Although the existing experiments cover different model scales, they do not sufficiently demonstrate the generalization capability of the method to newer model architectures.

  3. The effectiveness of the quantization-effect-guided budget allocation strategy across different quantization settings requires further validation. The authors have conducted budget-matched block-sensitivity-based allocation comparisons, providing preliminary evidence that the QE-guided strategy outperforms alternative budget allocation criteria. However, these comparisons are conducted primarily under the W4A8 setting, and it remains unclear whether this advantage generalizes to other quantization settings.

Questions for authors
  1. Could the authors provide comparisons with recent quantization methods published in 2026, such as VLM-PTQ, and evaluate the competitiveness of the proposed method under comparable model, quantization bit-width, and benchmark settings where possible?

  2. Could the authors extend the experiments to newer-generation vision-language models, such as Qwen2.5-VL-7B, Qwen3-VL-8B, or InternVL3.5-8B, to further evaluate the cross-model generalization capability of the proposed method?

  3. The authors conducted budget-matched block-sensitivity-based allocation comparisons under the W4A8 setting. Could the authors provide corresponding budget-matched comparison experiments under other quantization settings, or further clarify the scope of applicability of the allocation strategy, to verify the effectiveness and robustness of the QE-guided budget allocation strategy across different quantization settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.