- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ONPTQ calibrates multimodal models on prefixes generated by the current quantized model. It combines decision-boundary erosion and short counterfactual branch divergence to prioritize calibration states, with context anchoring and iterative trajectory refresh. Experiments on Qwen models show the clearest gains.
Critical strengths
-
Well-motivated. On-policy prefixes and counterfactual branches address autoregressive distribution shift beyond static reconstruction.
-
W4A6 average accuracy improves over the strongest listed baseline by 2.8 and 1.8 points on the two VLM backbones (Table 1), without changing the deployed inference graph.
Critical weaknesses
-
Practical advantage and robustness remain unclear.
- Added offline calibration cost is not quantified in GPU-hours or relative compute.
- Evaluation is limited to 3B–8B Qwen-family models.
- Five refreshed rounds reduce MMMU/ScienceQA accuracy from 48.8/89.7 to 44.2/86.1 (Table 4), raising concerns about stopping and stability.
- Qwen2.5-VL baselines are imported while ONPTQ is evaluated separately (Appendix C.1); higher-precision gains are small or inconsistent.
-
The benefit of combining the risk signals remains unclear. Consequence-only selection achieves a lower joint-critical flip rate in Table 9, while ONPTQ improves erosion. A matched downstream comparison would clarify the value of the combined risk.
Questions for authors
- What are the calibration GPU-hours, base quantizer, trainable parameters, and optimization schedule? Does the gain persist under equal compute?
- How were the imported Qwen2.5-VL baselines verified to match ONPTQ's evaluation protocol?
- Why does five-round refresh degrade performance in Table 4? Was the total update budget controlled, and how was the default round count selected?
- Does ONPTQ outperform consequence-only selection under matched downstream calibration? Please explain the flip-rate trade-off in Table 9.