- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Test-Time Hinting (TTH), motivated by the observation that errors made by different proprietary VLMs on the A-OKVQA benchmark tend to concentrate around common failure modes. The authors train a smaller Qwen3-VL model that, given an image-question pair, generates hints to guide black-box target models toward more accurate predictions. They demonstrate improvements across the evaluated models and benchmarks. The hints are curated through an iterative agentic pipeline in which GPT-5 generates and refines candidate hints, which are subsequently used to train the smaller hint generator.
Critical strengths
-
The proposed method is practical, especially considering that the hints are ultimately generated by a smaller model and the larger target model is queried only once. This makes the approach more efficient than majority-voting or Best-of-N (BoN) methods.
-
I appreciate the inclusion of both repair rate and harm rate as evaluation metrics. I think these provide a useful and informative assessment of the method's effectiveness.
-
The empirical gains, particularly the transferability to unseen models and the improvements on weaker target models, appear quite substantial relative to the baseline performance.
Critical weaknesses
-
The main contribution appears to be TTH, which is essentially a form of structured, input-conditioned prompting. However, I find that the novelty claims and literature review do not fully reflect the state of the field as of September 2026. There has been considerable work on prompt optimization, including Black-Box Prompt Optimization, TEMPERA: Test-Time Prompting via Reinforcement Learning, and more recent genetic prompt optimization methods such as GEPA. In my opinion, the positioning of the proposed method relative to this literature is somewhat limited, and the novelty claims could be more clearly articulated.
-
Related to the previous point, by September 2026, several works have explored test-time scaling (TTS) for VLMs, including efficient TTS approaches such as SPARC: Separating Perception and Reasoning Circuits for Test-time Scaling of VLMs, Efficient Test-Time Scaling for Small Vision-Language Models, and AVIS: Adaptive Test-Time Scaling for Vision-Language Models, among others. The paper does not sufficiently discuss this body of work or provide comparisons with these approaches. Even if the primary motivation is to support black-box models, I believe it would still be valuable to discuss these methods and potentially compare against some of them using open-source models, since TTH itself is not inherently limited to black-box settings.
-
The paper argues that existing TTS methods have paid relatively little attention to vision-centric benchmarks. However, the evaluation is limited to a relatively small set of visual benchmarks, without sufficiently motivating their selection. Given the availability of benchmarks spanning perception, mathematical reasoning, real-world understanding, and other capabilities, I believe a broader evaluation or a clearer justification for the selected benchmarks would strengthen the empirical study.
-
This is a relatively minor concern. In Figure 1, upon closer inspection of the input image, the vehicle appears to be an ATV/quad bike rather than a tractor. As such, the original model's prediction of "bike" may actually be defensible, while TTH changes the answer to "tractor." If this interpretation is correct, the example may illustrate a case of harm rather than a successful repair. I acknowledge that this could be due to ambiguity or noise in the benchmark annotations.
-
The role of the prompting strategy itself seems somewhat underexplored, and the choice of prepending the current hint format is not fully justified. It would be useful to include additional controls, such as a simple instruction prompt encouraging the model to carefully inspect the image regions relevant to the question and ground its response in visual evidence. Another potentially informative comparison would be against simple visual prompting techniques, such as those explored in Visual Structures Help Visual Reasoning: Addressing the Binding Problem in LVLMs. These comparisons could help clarify how much of the improvement comes from the learned hints rather than from more general prompting or visual-grounding strategies.
Questions for authors
-
Why was A-OKVQA selected as the primary benchmark for the failure-mode analysis and method development? Additionally, what motivated the selection of only three benchmarks for the transfer evaluation?
-
Could the authors evaluate TTH on open-source VLMs and compare it with existing test-time scaling approaches for VLMs?
9 comments
Response to Reviewer mH4p (2/3) on W2 and W3
W2 and Q2 (continued)
Open-weight target. Following your suggestion, we evaluated TTH on the open-weight Inkling-Small from Thinking Machines Lab on all four benchmarks, with Inkling as the external judge. is used without retraining, with the model's name as the target identifier, so both the target and its identifier are unseen in training. We also include two methods you cite. TTAug augments each question into 16 inputs and averages their next-token probabilities at every decoding step, following its paper's default, and VISER is the training-free visual prompting method you suggested in W5.
| Method | A-OKVQA | RealWorldQA | VCR | Visual7W |
|---|---|---|---|---|
| Base | 70.39 [67.77, 73.01] | 67.04 [63.38, 70.70] | 50.22 [48.84, 51.60] | 67.38 [66.08, 68.68] |
| CoT | 69.34 [66.64, 71.97] | 65.45 [61.78, 69.11] | 51.28 [49.90, 52.66] | 68.46 [67.16, 69.74] |
| Self-refine | 75.11 [72.58, 77.55] | 67.99 [64.33, 71.66] | 51.44 [50.06, 52.82] | 71.26 [70.00, 72.52] |
| External judge | 75.72 [73.19, 78.17] | 69.11 [65.45, 72.77] | 53.86 [52.48, 55.24] | 74.96 [73.76, 76.16] |
| TTAug | 76.33 [73.89, 78.78] | 68.47 [64.81, 72.13] | 54.30 [52.92, 55.68] | 72.56 [71.32, 73.80] |
| VISER | 67.34 [64.63, 70.04] | 66.72 [63.06, 70.38] | 51.40 [50.02, 52.78] | 68.92 [67.64, 70.20] |
| TTH | 85.85 [83.84, 87.86] | 73.41 [69.90, 76.75] | 57.74 [56.36, 59.10] | 79.18 [78.04, 80.30] |
TTH is significantly more accurate than Base and every other method on all four benchmarks. TTAug improves over Base by a few points, in line with the gains reported in its paper, while TTH queries the target once instead of decoding 16 inputs in parallel. Without retraining, TTH therefore extends to an open-weight target and outperforms every baseline there.
W3 and Q1. Choice of benchmarks
Thank you for pushing us to motivate this choice. We should have phrased our point about prior work more precisely. We did not mean that test-time scaling overlooks vision-centric benchmarks, and the methods you cite evaluate on several of them. Rather, much of this literature emphasizes mathematical, scientific and knowledge-intensive reasoning, while general visual understanding has received comparatively less focus, and this is what TTH targets. We will clarify this in the introduction. Our selection of benchmarks follows from four considerations.
- General rather than capability-specific. We wanted to show that learned hints improve visual understanding in general, without tailoring TTH to one capability. General VQA questions mix recognition, attributes, counting, spatial relations and everyday knowledge, so they test the approach broadly.
- Failures a compact generator can address. We target questions that frontier VLMs can answer with everyday knowledge and short reasoning but sometimes get wrong because they overlook or misread something in the image. This is where a 4B generator can plausibly help a much larger target, and it separates the effect of anticipatory hints from gaps in knowledge or multi-step reasoning.
- Diversity. The four benchmarks vary the image source, question style and answer distribution, from everyday scenes in A-OKVQA to movie stills in VCR and mostly driving scenes in RealWorldQA. Together they span knowledge-based VQA, visual commonsense reasoning, grounded perception questions and real-world spatial understanding.
- Scale and measurability. All four are established multiple-choice benchmarks with large test sets, including 5,000 questions each for VCR and Visual7W, larger than many capability-specific benchmarks. These sizes support paired significance tests, and exact correctness gives GRPO a clear reward and makes repair, harm and answer leakage directly measurable.
Why A-OKVQA. A-OKVQA meets two needs of our training pipeline. First, learning to repair errors requires many questions that the targets get wrong, yet frontier targets answer most questions correctly, as about 71% of A-OKVQA training questions are answered correctly by all three primary targets. With 17,056 training questions, A-OKVQA still leaves enough errors to support SFT and RL, which would be much harder with a smaller dataset. Second, its questions come with ground-truth rationales, short explanations that narrate the visual evidence and knowledge behind each answer. The hint proposer (in the agentic hint optimization stage) relies on them to ground its hints in the right evidence, which makes the hints more reliable, and the failure-mode annotator of Section 2 uses them to label why a target failed.
We will add this rationale to the paper. Extending TTH to capability-specific benchmarks, such as perception and mathematical reasoning, is a natural next step that we plan to pursue. Within the current scope, TTH transfers zero-shot to all three unseen benchmarks with significant gains over Base for every target.
Response to Reviewer mH4p (3/3) on W4 and W5
W4. Figure 1 example
Thank you for the careful inspection. We agree that the vehicle can also be read as an ATV, in which case "bike" is defensible. The example follows the A-OKVQA label, "tractor", and reflects the ambiguity and annotation noise you mention. The other examples in Appendix J are less open to such interpretation, and we will replace this example, in Figure 1 and in Appendix J.3, with an unambiguous one so that it illustrates a clear repair.
W5. Prompting controls
We agree that these controls are needed to isolate what the learned hints contribute, and ran both on the A-OKVQA test set with the three primary targets. We also added a third control that keeps TTH's learned hints but breaks their link to the image.
- Grounding instruction. A static instruction, "Inspect the relevant regions and ground your answer in visual evidence."
- VISER. The visual prompting method you suggested, from "Visual Structures Help Visual Reasoning". Following its default setup, we draw three equally spaced 1-pixel black lines that split the image into four bands numbered 1 to 4 on the left, and add the prompt "Scan the image sequentially based on horizontal lines".
- Mismatched-image hint. writes the hint from a mismatched image, and the target answers with this hint and the correct image. This tests whether the hints work through the visual evidence of each image or through their form alone.
| Target | Method | Accuracy | Repair | Harm |
|---|---|---|---|---|
| Claude 4.5 Haiku | Base | 78.25 [75.89, 80.61] | - | - |
| Grounding instruction | 76.24 [73.80, 78.69] | 18.88 [14.23, 23.94] | 7.81 [6.09, 9.63] | |
| VISER | 75.20 [72.75, 77.73] | 14.06 [9.88, 18.47] | 7.81 [6.12, 9.62] | |
| Mismatched-image hint | 74.85 [72.31, 77.38] | 20.88 [15.94, 26.02] | 10.16 [8.19, 12.16] | |
| TTH | 84.28 [82.09, 86.45] | 49.40 | 6.03 | |
| Gemini 2.5 Flash Lite | Base | 82.88 [80.70, 85.06] | - | - |
| Grounding instruction | 85.50 [83.41, 87.51] | 32.14 [25.69, 38.91] | 3.48 [2.34, 4.69] | |
| VISER | 79.56 [77.21, 81.83] | 22.96 [17.13, 29.13] | 8.75 [6.98, 10.61] | |
| Mismatched-image hint | 81.40 [79.13, 83.58] | 12.24 [7.83, 17.02] | 4.32 [3.07, 5.66] | |
| TTH | 90.57 [88.73, 92.31] | 61.73 | 3.48 | |
| GPT-5 Nano | Base | 84.37 [82.27, 86.47] | - | - |
| Grounding instruction | 84.63 [82.53, 86.64] | 31.84 [25.26, 38.81] | 5.59 [4.18, 7.09] | |
| VISER | 83.76 [81.57, 85.85] | 29.05 [22.58, 35.80] | 6.11 [4.65, 7.68] | |
| Mismatched-image hint | 84.02 [81.83, 86.11] | 29.61 [23.03, 36.42] | 5.90 [4.46, 7.44] | |
| TTH | 88.12 [86.11, 90.04] | 45.81 | 4.04 |
TTH is significantly more accurate than all three controls on all three targets, and repairs more errors on every target at equal or lower harm. The grounding instruction significantly helps Gemini 2.5 Flash Lite but significantly hurts Claude 4.5 Haiku, and VISER significantly lowers accuracy on both, with no significant change on GPT-5 Nano for either. We suspect that VISER's lines, designed for binding problems such as counting and visual search, distract the target on our questions, which mostly require general reasoning about scenes and knowledge. When the hint is written from a mismatched image, TTH's gain disappears and accuracy falls to or below Base on all three targets, so the hints work through the visual evidence of each image rather than through their form.
We prepend the hint so that it acts before the target answers, in a single call, and make it contrastive so that it points to discriminating evidence without revealing the answer. Together with the static-hint and untrained-generator ablations in Table 4, these controls indicate that TTH's gains come from the learned, input-specific content of its hints rather than from generic prompting or visual-grounding strategies.
We sincerely thank you again for the thoughtful review and the valuable pointers! We will add the new results, the extended related-work discussion and the benchmark rationale to the revision.
Response to Reviewer mH4p (1/3) on W1 and W2
Thank you for the thoughtful and constructive review, and for recognizing the practicality of TTH, the value of the repair and harm metrics and the transfer to unseen models! We especially appreciate the pointers to related work and the concrete controls you suggested. We ran these controls and an evaluation on an open-weight VLM, and report them across three comments because of the length limit. W1 and W2 (with Q2) are addressed in comments 1 and 2, W3 (with Q1) in comment 2, and W4 and W5 in comment 3. Brackets are 95% CIs, and significance refers to the paired bootstrap used in the paper.
W1. Positioning relative to prompt optimization
Thank you for bringing these works together. We agree that TTH acts through the prompt and should be positioned within the prompt-optimization literature more explicitly. We group these methods by what they optimize and contrast each group with TTH.
Task-level prompt optimization. APE, OPRO, DSPy and GEPA search offline for instructions or demonstrations (GEPA by evolving them through LLM reflection on execution traces) and then apply one optimized prompt to every input of a task. TTH instead generates a new hint for every input, conditioned on its specific image, question and answer options and on the target model. This matters in natural-image VQA, where the target already understands the task and its likely error depends on the particular input. Our controls support this. Fixed prompts do not reliably help, whether the failure-mode checklist in Table 4 or the grounding instruction and VISER's visual prompt in W5, and TTH is significantly more accurate than both W5 controls on every target. A hint written for a mismatched image also loses TTH's gain entirely (see response to W5), so the gain comes from conditioning on the actual input.
Learned input-level prompt rewriting. BPO trains a prompt rewriter on human preference data so that black-box LLMs give better-aligned responses. Like TTH, it adapts each input's prompt for a black-box model, but it is about alignment and is demonstrated only on text. TTH works on a different axis, capability. It trains a small model to make a larger one more accurate on visually grounded tasks, where a hint must point to evidence in the image, and we measure both its repairs and its harms on nine VLMs from six providers. TEMPERA trains an RL policy that edits each query's instruction, in-context examples and label words, using the target's hidden states and label probabilities. Closed-weight models expose neither, which is exactly the setting TTH is built for.
Beyond conditioning on the input, TTH is trained to anticipate and prevent the target's own failure, which Section 2 shows is largely shared across models and predictable from the input alone. Its reward is defined relative to the target's base answer and penalizes turning a correct answer into a wrong one, its hints are verified with the target in the loop and must not reveal the answer, and at test time it calls the black-box target once. We will add these works to Related Work, state the contribution in these terms, and narrow the "first learned, single-call method" claim accordingly. We believe this makes TTH's contribution clearer, which is to learn from the input alone how a target is likely to fail, and to prevent that failure with a hint given before a single call.
W2 and Q2. Test-time scaling for VLMs and open-weight models
We agree that these methods should be discussed, and that TTH is not inherently limited to black-box models. Beyond discussing them, we ran the comparison you ask for. We evaluated TTH on an open-weight model, and on the same model we ran TTAug, one of the methods you cite that is applicable to this setting. The results are in comment 2. We see test-time scaling for open-weight and for closed-weight VLMs as two design pathways that allow different levels of intervention. The methods you cite were developed and evaluated on open-weight models. SPARC, the test-time augmentation and adaptation methods for small VLMs (TTAug and TTAdapt) and AVIS sample the target several times, through repeated relevance detection before a cropped re-answer, averaged predictions over augmented inputs, or an adaptive self-consistency budget, and, as evaluated, they rely on open weights or internals such as token probabilities, gradients or attention keys. Because open-weight models expose strictly more information, such methods often cannot be applied to closed-weight models, whereas a method that needs only black-box access applies to both. As practitioners increasingly rely on frontier models whose weights they cannot access, we designed TTH for the closed-weight pathway, with one pass of a compact generator and a single call to the target. Its hints can also be prepended to the inputs of open-weight methods. We will add this discussion to Related Work and make this motivation clearer in the introduction.