acceptodds
Under review as a conference paper at ICLR 2027

Beyond Confidence: Evidence-Guided Selective Answer Revision for Frozen Vision-Language Model

Abstract

A frozen vision-language model (VLM) may contain a better sampled answer than its first response, yet indiscriminate revision can overwrite a correct baseline. We formulate answer revision as proposal selection followed by baseline-relative acceptance. An evidence-aware candidate ranker (FER) selects a proposal from generation, visual-intervention, and region-conditioned alignment evidence; a Pairwise Utility Gate (UG) estimates replacement utility. We apply a common image-group-separated calibration principle, while the final evaluation construction follows each benchmark’s split structure. The method improves OK-VQA by +1.198 points and VQAv2 by +0.615; on GQA it recovers most of the FER-only degradation and yields a statistically inconclusive +0.064. A matched-data, matched-fold comparison places a compact six-cue schema within one standard error of the expanded FER schema, with small dataset-dependent end-to-end differences. Cross-scale Qwen2.5-VL and model-specific BLIP-2 and LLaVA experiments further show that transfer depends on the backbone and candidate-pool headroom.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
ML: Multimodal & Large Multimodal Models (LMMs), NLP: Interpretability, Analysis & Evaluation (incl. Factuality & Hallucination), NLP: Language Grounding & Multi-modal NLP

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_x2KTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper formulates answer revision for a frozen VLM as two supervised decisions: selecting the most promising proposal from a fixed pool of five candidates, and deciding whether replacing the initial answer with that proposal has positive relative utility. The FER ranker uses features derived from generation confidence, entropy, consensus, image-intervention stability, and question-relevant regions, with the official per-example score as supervision. A Pairwise Utility Gate (UG) then combines proposal features, baseline features, and their differences to regress the official-score change caused by replacement; a threshold calibrated on labeled data chooses between revision and retention. For Qwen2.5-VL-3B, the final policy improves an archived OK-VQA subset by 1.198 points and VQAv2 by 0.6145 points, while the 0.064-point change on GQA is statistically inconclusive. The paper also reports revision coverage, wrong-to-correct and correct-to-wrong transitions, oracle headroom, cross-backbone results, and matched controls for 6- versus 27-field feature schemas.

Critical strengths
  1. The formulation correctly distinguishes a good proposal from a worthwhile replacement.

  2. The reporting does not conceal negative or inconclusive outcomes.

  3. The evaluation includes several useful controls, and clearly states that “frozen” applies only to the VLM, not to the supervised decision system.

Critical weaknesses
  1. The core two-stage claim lacks a supervision- and capacity-matched joint baseline. Both FER and UG are supervised gradient-boosting models, and UG has access to FER scores, proposal and baseline features, and their differences. Comparing UG with scalar thresholds based on confidence, entropy, or FER margin does not show that “rank first, then estimate relative utility” is better than a comparably expressive model that directly predicts the final utility of every candidate, including the baseline.

  2. The improvements are small and strongly dataset-dependent, limiting broad reliability claims. For Qwen-3B, the statistically resolved gains are only +1.198 on OK-VQA and +0.6145 on VQAv2, while GQA is inconclusive. The final policy recovers only about 0.6%-9.3% of candidate-pool oracle headroom, and the LLaVA results include two inconclusive changes and one slight regression.

  3. The datasets use different evaluation constructions that are not directly comparable.

Questions for authors
  1. Under exactly the same candidate pool, 28-dimensional relative features, labels, image-group splits, and tuning budget, how does a joint model perform if it ranks the baseline and all candidates together, or directly predicts each candidate's utility relative to the baseline and selects the maximum? Is the difference from FER+UG supported by paired confidence intervals?

  2. What are the separate gains, acceptance coverage, and W2C/C2W counts for the VQAv2 out-of-fold and disjoint-complement strata? How does performance vary as target-domain calibration labels increase from a small sample to the current scale, and is there a transferable operating point that does not require relabeling every target dataset?

Reviewer_ropdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses studying when a VLM should replace its first answer, with one of multiple candidate answers sampled by the same VLM. Their method consists of a learned ranker that ranks each answer candidate using several signals, and a gate model that decides whether to keep the base answer or replace it with the best answer candidate. The method is tested with three datasets, and reported minimal gains above the base and several compared methods.

Critical strengths

The framing is sensible. The paper separates two questions: which candidate looks best, and whether it is actually better than the first answer. Learning the score change relative to the baseline directly, rather than candidate quality alone, is a reasonable target, and Figs. 1–2 motivate it clearly.

  • Evaluation is broad: it covers three benchmarks, and four backbones.
Critical weaknesses
  • the paper is hard to follow because several symbols in the core equations are undefined (e.g., eᵢ, s*, t(·), G_ψ, μ̂), notation and terminology are inconsistent
  • key details such as how the baseline and pool are decoded and how features are computed are frequently deferred to the supplement while the paper is under the page limits.
  • improvement over simply keeping the baseline answer is minimal. The method does not consistently beat simple alternatives. Since it requires labeled data and probes per candidate, the practical value of the method is unclear.
  • BLIP-2 baselines are far below the model’s known zero-shot performance (major). In Table 3, the BLIP-2 (OPT-2.7B) baseline scores 20.8 on OK-VQA, 13.6 on GQA and 19.6 on VQAv2. The standard zero-shot results for this model are roughly 31, 31–34 and 50+, respectively [1]. Even after revision, the final scores remain below those standard numbers. This suggests the baseline is misconfigured.

[1] Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." International conference on machine learning. PmLR, 2023.

Questions for authors
  1. FER training data. What data was the deployed FER-27 trained on for each benchmark?
  2. If UG estimates the expected score change, why does the best threshold fall below zero (−0.10, −0.02)? What are the results at τ = 0?
  3. What percentage of final answers differ from the baseline answer on each dataset?
Reviewer_Ca5iOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats post-hoc answer revision for a frozen VLM as two supervised decisions over a cached pool of five sampled candidates. An evidence-aware ranker (FER, a gradient-boosted regressor over 27 generation, image-intervention and region-probe features) proposes a candidate c⋆c^\star. A Pairwise Utility Gate (UG, a gradient-boosted regressor over 28 proposal-vs-baseline features) predicts the official-score change Δ=S(c⋆)−S(a0)\Delta=S(c^\star)-S(a_0) and replaces the first answer a0a_0 when the prediction exceeds a threshold calibrated on labeled, image-group-separated data; a plug-in Bayes argument motivates this target. The authors frame the contribution as the baseline-relative target plus a controlled empirical study. With Qwen2.5-VL-3B, the final policy improves VQAv2 val by +0.61 and a 3,789-question OK-VQA subset by +1.20, while the GQA result is inconclusive (+0.06). The ranker, trained on the OK-VQA subset, supplies the OK-VQA gain but hurts GQA and VQAv2, where the in-domain gate recovers the loss. The supplement adds pool statistics, provenance tables, a matched FER-6/FER-27 control, gain–harm frontiers, oracle headroom (at most 9.3% recovered) and results on other backbones: gains on Qwen2.5-VL-7B, large gains on BLIP-2 from very low baselines, and no gain on LLaVA-1.5. But still there are many concerns about experiments so currently I recommand reject.

Critical strengths

A principled, metric-aligned target with one robust result. The gate regresses the realized score change from replacing a0a_0, so KEEP is exactly neutral and the decision is tied to the metric rather than to proposal confidence. On VQAv2 the gain holds across rankers on 214K questions) with the deployed ranker. On GQA and VQAv2 the gate also stops an out-of-domain proposer from hurting. Treating acceptance as a safety layer is a useful lesson for revision and self-correction pipelines.

Critical weaknesses

W1. Key numbers cannot be traced to a clean protocol, and the OK-VQA gain may not survive a held-out ranker.

  • OK-VQA ranker. The deployed FER-27 was trained on 22,729 rows from the same 3,789 OK-VQA questions that are evaluated (Table S3, M.2). OK-VQA is described as cross-fitted, yet Table S3 documents a single model, scripts/train_fer.py saves only a model refit on all rows, and the training set has ≈6 rows per question although K=5K=5. With a FER trained on disjoint data (Table S10), the OK-VQA gain falls from +1.20 to +0.40 (FER-27) / +0.58 (FER-6). Since UG is inactive on OK-VQA, the headline result rests entirely on this model.
  • Inconsistent reporting. For 7B VQAv2, the gain and harm totals in Table S14 imply +0.811, not the reported +0.684. That value matches the excluded Q-FER run in core_results_overview.csv. The same file lists 7B OK-VQA as +1.478 (frozen transfer, 5.2% replacement), whereas the paper reports +1.288 (cross-fitting, 98.76% coverage).
  • Confounded backbones. BLIP-2 is prompted and decoded differently from its standard zero-shot setup ("Question: {} Answer:" with beam search; Li et al., 2023). Its baselines are 20.8 / 13.6 / 19.6, so its large gains plausibly reflect answer-format repair. Qwen-7B uses one third of the 3B max_pixels (200,704 vs. 602,112), which plausibly explains why it scores below 3B.
  • Reproducibility. As written, the released code cannot produce the candidate pools: it calls int(config["seed"]) on UNRESOLVED_NOT_PERSISTED and ignores max_pixels. Region selection, prompt variants, how a0a_0 is decoded, how the OK-VQA subset was chosen and how the Table 1 baselines were built are not documented anywhere.

W2. Even at face value, the central thesis is not isolated, and the gains are small.

  • The gate's gain over FER on GQA/VQAv2 (+0.95 / +0.75) compares a gate trained in-domain with a ranker trained only on OK-VQA. Since 96–99% of pools already contain a0a_0 (Table S1), the missing control is an in-domain ranker over C(x)∪{a0}\mathcal{C}(x)\cup\{a_0\} trained on the same labels; the authors concede it is absent (M.4, M.6).
  • On OK-VQA, every fold picks the lowest threshold in the grid (−0.10; 98.8% acceptance), which amounts to always revising.
  • The Table 1 baselines are undocumented and mostly missing on VQAv2, and temperature scaling beats FER+UG on GQA (+0.19 vs. +0.06). No matched ablation tests the evidence families.
  • Only VQAv2 shows a robust gain (≈ +0.6), and inference cost is not reported. The chosen operating points still overwrite many correct answers (C2W/W2C = 247/255 on GQA).
Questions for authors

See weekness.

Reviewer_hMK8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Beyond Confidence studies answer revision for a frozen vision-language model: given an image and a question, the model produces a first answer and a pool of 5 sampled candidates. The paper's point is that revision needs two judgments, not one: which candidate is best supported, and whether it is better than keeping the baseline. It therefore separates proposal selection from acceptance: an evidence-aware ranker (FER) scores candidates with generation, visual-intervention, and region-conditioned evidence, and a Pairwise Utility Gate (UG) predicts the official-score change from replacement, revising only when the predicted gain exceeds a calibrated threshold. The final policy improves the evaluated OK-VQA subset by +1.198 and VQAv2 by +0.6145; GQA yields an inconclusive +0.064. The idea — making baseline-relative comparison an explicit learned target — is good; the question is whether the evidence establishes the method's effectiveness.

Critical strengths
  • The baseline-relative two-decision framing is a clean idea: defending the first answer unless a challenger is demonstrably better, with the gain over the baseline as an explicit learning target.
  • Honest experiments with clear attribution: image-group-separated calibration, paired bootstrap intervals, and inconclusive (GQA) and negative (LLaVA) outcomes retained rather than hidden; ranker and gate gains are separated (FER supplies OK-VQA; UG recovers GQA/VQAv2 degradation), with gain–harm frontiers and oracle headroom quantified.
  • Cross-backbone portability is tested, including negative transfer — not reported on a single backbone only.
Critical weaknesses
  • The gains are small and inconsistent. Macro +0.625 rests on OK-VQA (+1.198, significant but a 3,789-question subset) and VQAv2 (+0.6145, significant but a hybrid estimate); GQA (+0.064) is inconclusive by the paper's own interval. Most pool headroom remains unrecovered (9.3% / 0.6% / 8.5%). The gate itself is a safety device, not a gain device: on OK-VQA it slightly reduces FER-only (+1.25 → +1.20). A method whose benefit is significant on two datasets (one a subset, one hybrid) and inconclusive on the third does not yet look like a general solution.
  • Two key controls are missing, and both need re-run experiments. First, the comparison direction is only explored with regression models: whether other models or methods (for example an LLM-judge gate) in the same comparison direction would do better is untested. Second, the gate is only ever tested against the model's own first answer: if others' methods first find the best answer and that becomes the default, can this method still add gains — significantly, and consistently across datasets? Until those runs exist, the method's effectiveness remains in doubt.
  • Evaluation protocol and deployment costs are unclear. The VQAv2 score mixes two gate variants (an image-excluded gate for 39,996 questions and a full-calibration gate for the rest), OK-VQA is a subset, stratum scores are unreported, and no gate-increment interval is given — so how much the gate itself adds, and whether it is significant, is unclear from the main text. The paper also does not show that two stages beat a joint ranker that includes the baseline among the candidates, does not establish which evidence families matter, reports no calibration-size curve or full latency, and cannot reproduce the pool exactly because decoder settings are missing; the gate needs target-domain labels and accepts 90.33% on average.
Questions for authors
  1. If the best answer from others' methods becomes the default, can your gate still add gains — significantly, and consistently across datasets? (Re-run experiments needed.)
  2. What happens with non-regression gates in the same comparison direction, for example an LLM judge deciding revise-or-keep?
  3. Can you report gate-increment intervals, pure out-of-fold VQAv2 results with stratum scores, and the calibration-size curve (label cost vs gain)?
Reviewer_Lw8qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies supervised answer revision for frozen vision-language models. An evidence-aware ranker selects a proposal from five cached candidate records, potentially containing duplicates, and a Pairwise Utility Gate estimates whether replacing the original answer would improve the official benchmark score. The decision models combine generation statistics, image-transformation responses, and region-conditioned evidence. The primary Qwen2.5-VL-3B evaluation shows improvements on a 3,789-question OK-VQA subset and full VQAv2 validation under image-group-separated evaluation, while the GQA test-dev improvement is statistically inconclusive. Additional backbone experiments show mixed results. The VLM remains frozen, but the decision models require labelled supervision and calibration.

Critical strengths

The paper addresses a practical reliability problem in vision-language models by separating candidate-answer selection from the decision to replace the original response. Its baseline-relative utility formulation considers whether a proposed revision improves on the original answer, using generation statistics, image-transformation responses, and region-conditioned evidence. The empirical study includes image-group-separated evaluation, paired confidence intervals, matched feature-schema comparisons, and analyses of beneficial and harmful revisions and candidate-pool headroom. Improvements on the evaluated OK-VQA subset and VQAv2 demonstrate potential without modifying the underlying VLM. The paper also distinguishes the contributions of ranking and acceptance and transparently reports mixed cross-backbone results and methodological limitations.

Critical weaknesses

The authors are suggested to address the following:

  1. The advantage of the two-stage design is not fully established. The supplement includes a direct all-candidate gate experiment, but it uses an older fixed-threshold protocol rather than the main calibrated setup. Would the two-stage method outperform a single-stage alternative under matched features, supervision, capacity, and calibration?
  2. Benefits vary across datasets and backbones. Qwen-3B improves on the evaluated OK-VQA subset and VQAv2, but its GQA gain is inconclusive. LLaVA results show no statistically established improvement. Adding UG also slightly lowers the FER-only OK-VQA point estimate; this does not establish a significant degradation. What determines when the gate adds value beyond ranking alone?
  3. The additional value of each evidence source is unclear. The paper provides feature-schema comparisons and historical evidence combinations, but these do not isolate each evidence family under matched conditions. How does performance change when generation, global visual-intervention, or region-conditioned evidence is removed separately?
  4. Calibration requirements limit deployment flexibility. The primary gate requires target-domain labelled calibration, and the amount needed is not studied. Some frozen Qwen cross-scale transfer is demonstrated, so it would be inaccurate to say that every experiment requires recalibration. How much labelled calibration data is necessary, and how robust is deployment without target-domain recalibration?
  5. The accuracy–computation trade-off is unmeasured. Candidate generation and evidence probes require additional model calls, but total inference cost and full pipeline latency are not reported. This leaves efficiency unknown rather than proving the method is inefficient. Do the gains persist against simpler alternatives at the same total inference budget?
  6. Reporting limits interpretation and exact reproduction. The disclosed OK-VQA subset limits the scope of its result. More importantly, separate scores for VQAv2’s two evaluation portions are missing, and some decoder settings cannot be recovered. These issues do not establish evaluation leakage. Can you report the VQAv2 portions separately and verify the findings using a fully specified candidate-generation recipe?
Questions for authors

Please see weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.