acceptodds
Under review as a conference paper at ICLR 2027

Ask a Better Question: Unlocking What Vision-Language Models Already Get Right

Abstract

Vision-language models are queried through language, yet a frozen model's answer can vary substantially across questions intended to express the same visual request. This sensitivity creates both a practical failure mode and an opportunity: the same model may answer an alternative formulation correctly after failing on the original question. We ask whether the model's own first response can guide a better formulation at inference time. We introduce Inference-time response-CONditioned reframing (ICON), a lightweight text rewriter that observes the original question, answer options when present, and the frozen VLM's initial response, then produces a single revised question for a second pass through the same model. is trained offline from supervised rewrite pairs, but deployment requires neither gold labels nor candidate search, and the target VLM remains frozen throughout. On Robo2VLM, relative to question-only rewriting, the full response-conditioned rewriter increases the mean accuracy gain from to percentage points with a B rewriter and from to points with an B rewriter. In the matched comparison, adding the initial response to question-and-options-only rewriting contributes an additional and points, respectively. Across eight benchmarks, separately trained rewriters yield gains of to percentage points, and a Robo2VLM-trained rewriter transfers across several frozen VLMs on the same task. Together, these results identify the model's own response as a useful conditioning signal for targeted question reformulation, enabling the same frozen VLM to produce correct answers under revised formulations when the original wording leads to an error.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mKA9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ICON is a learned text rewriter that conditions on the original question, answer options when present, and a frozen VLM's initial response, then issues one revised question to the same VLM. Gold labels are used offline to construct and filter rewrite targets; deployment uses a single rewrite without labels or candidate search. Matched question-and-options-only versus response-conditioned experiments test the value of the initial response. Further evaluations train separate rewriters on eight benchmarks and transfer a fixed Robo2VLM rewriter across frozen answerers on the same task.

Critical strengths
  1. The matched QO/QOA comparison provides clear evidence for the main contribution. Adding the initial response improves Robo2VLM accuracy by 4.84 and 4.30 percentage points for the two rewriter sizes, beyond the question-and-options-only controls. This supports a specific empirical result about response conditioning rather than only the observation that question wording matters, and is the main reason I lean toward acceptance.

  2. The method addresses the practical risk that rewriting can break initially correct answers through identity targets and fix-versus-break analysis. The distinction between offline label use and label-free deployment is explicit, and standard inference requires only one learned rewrite. These choices make the limited contribution practically interpretable.

Critical weaknesses
  1. Successful rewrites have not been independently shown to preserve the original requested judgment. Training proposals see the gold answer, while disclosure rules and black-image checks do not establish semantic equivalence. The paper acknowledges this limitation, and the current evidence does not establish leakage. A modest blinded audit would nevertheless strengthen the interpretation that the gains reflect better elicitation of the same visual judgment.
  2. The broader results support benchmark-specific usefulness rather than a universally transferable rewriter: Table 3 explicitly trains a separate rewriter for each benchmark, and several gains are small. Means over three seeds do not show uncertainty, which matters particularly for VSR's 0.20-point gain with the 8B rewriter. Appendix A establishes disjoint question IDs but leaves image/video grouping worth clarifying; actual contamination is not established. These qualifications limit the breadth of the claim without negating the main Robo2VLM improvement.
Questions for authors
  1. Focused validation of semantic fidelity: What fraction of a blinded sample of successful and unsuccessful deployed rewrites preserves the original requested judgment under independent assessment?
  2. Clarification and focused uncertainty analysis: Can the authors provide seed variation or paired uncertainty for the reported gains, especially the small gains, and clarify whether custom partitions separate shared images or videos as well as question IDs?
Reviewer_foxcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at how much a frozen VLM's answer depends on the wording of the question, and proposes ICON: a trained text rewriter that reads the original question, the answer options (when present) and the VLM's own first answer, then writes one revised question that is sent back to the same frozen VLM.

Section 3 first measures the effect. With GPT-5.5 paraphrases, Qwen3-VL-2B changes its answer on 73.2% of Robo2VLM items and 43.0% of LingoQA items, and label-aware best-of-K accuracy goes from 51.8 to 74.5 and from 45.8 to 69.0. The authors present this as an upper bound, not a deployable number.

ICON is a LoRA adapter (r=64) on a frozen text backbone. Training pairs are built offline with gold labels (Sec. 4): if the first answer is correct the target is the unchanged question; if it is wrong, the target is a paraphrase that the frozen VLM answers correctly. Candidates go through a rule-based answer-disclosure filter and a black-image check. At test time there are no labels and no candidate search; the cost is two VLM calls and one rewriter call.

Main results: on Robo2VLM the full input (QOA) gives +8.46 / +8.54 points with 4B / 8B rewriters, against +3.62 / +4.24 for question+options (QO) and +1.65 / +2.12 for question only (Table 2). All zero-label prompting baselines are zero or negative, and trained baselines (PRewrite, ReQueR, Advisor) are lower. Separately trained rewriters give +0.20 to +8.54 on eight benchmarks (Table 3), and the Robo2VLM rewriter transfers to other answerers, including three commercial APIs, with +0.10 to +5.54 (Table 4). Section 6 reports that the rewriter changes 71.2% of initially wrong questions and 6.9% of initially correct ones, and gives a taxonomy of successful rewrites.

Critical strengths
  1. The QO vs QOA comparison (Sec. 5.2, Table 2) is a real controlled ablation. Training data, rewriter backbone, evaluation items and answerer are held fixed, and only the initial response a_ori is added. The extra +4.84 / +4.30 points are large next to every other delta in that table. This is the evidence behind the paper's main claim, and it is the part I expect to hold up best, at least on Robo2VLM.

  2. The paper is careful about what is and is not deployable. Best-of-K is labeled as label-aware throughout, Sec. 6.1 reports fix/break counts (perceptual grounding fixes 130 items and breaks 183), and Appendix C openly says the RL-trained rewriter raised black-image answerability, so those results were not used. This makes the claims much easier to check, and it is the reason I think the concerns below can be resolved with experiments rather than a rewrite of the paper.

Critical weaknesses
  1. Test-time answer leakage or answer elimination is not ruled out for the reported numbers. The disclosure and black-image filters are applied only to training targets (Sec. 4, "Filtering answer-leaking candidates"), and the authors themselves say these checks "do not establish semantic equivalence." At inference the rewriter sees a_ori and the options. Sec. 6.1 shows that, from text alone, it rewrites 71.2% of wrong items and only 6.9% of correct ones, so it has learned a fairly good text-only error detector. A rewriter that can tell a_ori is probably wrong can raise accuracy by steering the new question away from a_ori, without the question being any "better" as a visual request. Figure 1 (middle) is arguably a case in point: the reframed question adds "if pointing right", which narrows the candidate arrows instead of restating the request. Appendix C shows this failure mode does appear once the rewriter is optimized harder. The only reported leak rate (0%) is for the pooled study in Sec. 7, not for Tables 2 to 4. I would need: (a) black-image accuracy of original vs rewritten questions on the evaluation split, for Q, QO and QOA; (b) how often the corrected final answer differs from a_ori, compared against a simple baseline that, whenever the rewriter decides to rewrite, re-asks the original question with a_ori removed from the options; (c) a small human check (around 100 items) that rewrites keep the original request. If the QOA over QO gain mostly disappears under (a) or (b), the central claim does not hold. If it survives, I would raise my score.

  2. Outside Robo2VLM the gains are small and come with no uncertainty estimates. Every table reports a mean over three seeds with no standard deviation, confidence interval or paired test. Six of eight benchmarks in Table 3 are between +0.20 and +2.17 points; +0.20 on VSR (N=494) is about one question, and +1.02 on MMMU (N=423) is about four. Transfer to the strongest answerers is +0.10 / +0.47 for GPT-5.5 and +0.97 / +1.57 for Qwen3-VL-8B (Table 4). The large Robo2VLM gain comes from a templated benchmark where the rewriter is trained on 5,341 items from the same templates, so it may partly reflect template-specific fixes. Two further issues affect the main comparison. The rewriter backbone was chosen on the evaluation split (Table 1 uses base accuracies 48.8 and 72.8, which are the evaluation-split numbers from Sec. 3 and Table 3). And the baselines omit the obvious one for an open 2B answerer: LoRA fine-tuning the answerer on the same 5,341 labeled items. Without per-seed variance, paired significance tests (bootstrap or McNemar over items) and that baseline, I cannot tell whether "positive gains across all eight benchmarks" (Sec. 5.3) is a real effect or how the +8.5 compares to the cheapest alternative that uses the same labels.

Smaller points that did not drive my score: Sec. 6.1 cites Table 2 for the fixed-transformation results, but those numbers are not in Table 2; Sec. 6.2 says it follows "the same setting in Table 3" for Idefics3 and Molmo, which appear in Table 4; the "base-anchored cross-rewrite consensus" in Sec. 7 is never defined and its results are not in a table; the MMBench, GQA and A-OKVQA results (Sec. 7) and the Appendix C results are described without numbers; K, the proposer model used for training pairs, and the answerer's decoding settings are not stated the OpenReview abstract contains a raw \icon{} macro; quotation marks in the Figure 1 caption are mismatched.

Questions for authors
Reviewer_7SupOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VLMs are sensitive to question wording: given the same visual input and intended question, a frozen VLM may give different answer under different phrasings. The paper proposes ICON, a text rewriter that takes the original question, the answer options and the frozen VLM’s initial answer to reformulate the question for a second pass through the same VLM. With Qwen3-VL-2Ben3-VL-2B answerer, ICON improves accuracy by +0.20 to +8.54 points across 8 benchmarks.

Critical strengths

Strengths:

  • The method is simple and can be applied to closed-source answerers.
  • The Q -> QO -> QOA ablation isolates the contribution of each rewriter input, though not its mechanism behind it.
Critical weaknesses

Weaknesses:

  • Gating and rewrite quality are not disentangled. The paper acknowledges that effective reformulation requires both a good decision about when to rewrite and useful rewrite content, but it does not measure their relative contributions. On Robo2VLM, the rewriter changes 71.2% of initially incorrect questions but only 6.9% of initially correct questions, and a fixed rewrite when correct items are protected achieves up to +9.74 points. This suggests that the gain may mainly come from selective rewriting rather than better rewriting questions. If so, the paper should reframe its contribution more clearly and include gating then rewriting baselines.
  • Training time rewrite generation may leak gold answer information. The proposer sees the gold answer when generating rewrite candidates, so the training targets may still contain implicit hints or intermediate reasoning steps that the two filters cannot catch. The rewriter may learn this hinting behavior and apply to test time. The black image test should also be applied to test time rewrites.
  • Generalization is limited. Most main results come from rewriters trained separately for each benchmark. In cross benchmark setting, a single pooled rewriter yields no net gain since help and hurt nearly cancel out. An additional consensus rule (which is not defined in the paper) yields only a small gain. In the cross answerer setting, gains are also modest beyond the small 2B answerer.
  • Cost-effectiveness is not quantified. ICON needs 3 sequential calls but neither compute nor latency is reported.
Questions for authors

Questions:

  1. Among rewritten questions, what fraction of initially incorrect questions become correct and initially correct become incorrect (for Q, QO and QOA)? Please compare performance of QO with QOA given the same keep or rewrite decision as QOA.
  2. What is the black-image accuracy of test-time rewrites compared with the original questions?
  3. Could the authors include an accuracy-cost trade off comparision between ICON and directly using a stronger answerer (e.g., Qwen3-VL-4B/8B)?
  4. Please report the number of candidate rewrites, the proposer model and prompts, the leakage-filter rules, the decoding settings, and the training hyperparameters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.