- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper formulates answer revision for a frozen VLM as two supervised decisions: selecting the most promising proposal from a fixed pool of five candidates, and deciding whether replacing the initial answer with that proposal has positive relative utility. The FER ranker uses features derived from generation confidence, entropy, consensus, image-intervention stability, and question-relevant regions, with the official per-example score as supervision. A Pairwise Utility Gate (UG) then combines proposal features, baseline features, and their differences to regress the official-score change caused by replacement; a threshold calibrated on labeled data chooses between revision and retention. For Qwen2.5-VL-3B, the final policy improves an archived OK-VQA subset by 1.198 points and VQAv2 by 0.6145 points, while the 0.064-point change on GQA is statistically inconclusive. The paper also reports revision coverage, wrong-to-correct and correct-to-wrong transitions, oracle headroom, cross-backbone results, and matched controls for 6- versus 27-field feature schemas.
Critical strengths
-
The formulation correctly distinguishes a good proposal from a worthwhile replacement.
-
The reporting does not conceal negative or inconclusive outcomes.
-
The evaluation includes several useful controls, and clearly states that “frozen” applies only to the VLM, not to the supervised decision system.
Critical weaknesses
-
The core two-stage claim lacks a supervision- and capacity-matched joint baseline. Both FER and UG are supervised gradient-boosting models, and UG has access to FER scores, proposal and baseline features, and their differences. Comparing UG with scalar thresholds based on confidence, entropy, or FER margin does not show that “rank first, then estimate relative utility” is better than a comparably expressive model that directly predicts the final utility of every candidate, including the baseline.
-
The improvements are small and strongly dataset-dependent, limiting broad reliability claims. For Qwen-3B, the statistically resolved gains are only +1.198 on OK-VQA and +0.6145 on VQAv2, while GQA is inconclusive. The final policy recovers only about 0.6%-9.3% of candidate-pool oracle headroom, and the LLaVA results include two inconclusive changes and one slight regression.
-
The datasets use different evaluation constructions that are not directly comparable.
Questions for authors
-
Under exactly the same candidate pool, 28-dimensional relative features, labels, image-group splits, and tuning budget, how does a joint model perform if it ranks the baseline and all candidates together, or directly predicts each candidate's utility relative to the baseline and selects the maximum? Is the difference from FER+UG supported by paired confidence intervals?
-
What are the separate gains, acceptance coverage, and W2C/C2W counts for the VQAv2 out-of-fold and disjoint-complement strata? How does performance vary as target-domain calibration labels increase from a small sample to the current scale, and is there a transferable operating point that does not require relabeling every target dataset?