- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Balanced Fitting for LVLM post-training quantization. It measures how quantizing individual layer components changes calibration-task NLL, then uses these effects to assign different scaling-search budgets across layers while keeping the reconstruction objective. It achieves better average performance than the compared baselines across different LVLMs in the reported results.
Critical strengths
Using task-NLL changes to allocate layer-wise scaling-search budgets is an interesting idea. Table 12 also shows that selectively using fewer candidates can improve downstream performance, suggesting that uniformly increasing search granularity is not always beneficial for LVLMs quantization.
Critical weaknesses
-
Adaptive search-budget allocation is the core contribution of Balanced Fitting, but its incremental benefit is less clear than the main results in Tables 1 and 2 suggest. Uniform CWE-20 in Table 8 appears to be the most direct control, since it removes adaptive allocation. It also appears to be a configuration of AWQ-style scaling search, but its differences from the AWQ implementation in Table 7 are not explained. If so, AWQ and other scaling-based LVLM-specific methods should be evaluated under the same configuration where applicable. Uniform CWE-20 should replace RTN, or at least be added, as the primary basic baseline in the main results, since it better isolates the proposed contribution. Some gains over this control are very small: on Qwen2-VL W3A16, the average increases from 70.47 to 70.56. Without clearer implementation details and evidence that these small gains are reliable, it is difficult to judge how much improvement comes specifically from the proposed allocation strategy.
-
The practical value beyond the small-calibration setting remains unclear. The main experiments use only 64 calibration pairs. If Balanced Fitting helps primarily by limiting calibration overfitting, increasing the calibration set could reduce its advantage. Given the small gains over Uniform CWE-20 in Table 8, it is important to establish whether these gains persist with more calibration data.
These concerns limit my confidence in the method’s independent contribution and practical advantage, so I lean toward rejection.
Questions for authors
My questions mainly reflect the concerns in Critical Weaknesses. I hope the authors can address these concerns either by answering the questions below or in another proper way.
-
What exactly differs between Uniform CWE-20 in Table 8 and the AWQ implementation in Table 7?
-
How do LVLM-specific methods (MBQ and QIG) perform under the same underlying CWE configuration?
-
What is the actual gain of Balanced Fitting when the comparison is fair enough (against Uniform CWE-20 version of AWQ, MBQ and QIG)? I would currently consider the results in Table 8 are more fair, which actually seems marginal on Qwen2-VL.
-
How do these fairer comparisons change with larger calibration sets?
-
Minor: Is there a better term than “positive quantization effect”? It sounds unintuitive to imply that quantization inherently improves model quality.