acceptodds
Withdrawn from ICLR 2027

Test-Time Hinting for Black-Box Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time scaling (TTS) methods have proven highly effective for LLMs, yet their application to vision-language models (VLMs) remains relatively underexplored. Existing VLM TTS methods largely require open-weight model access or expensive repeated sampling, and are evaluated primarily on multimodal mathematical and scientific reasoning benchmarks rather than natural-image understanding. In this paper, we propose Test-Time Hinting, a method that improves VLM performance with a single call to the target VLM and requires only black-box API access, which makes it broadly applicable to frontier closed-weight models. Our method is motivated by the observation that VLM errors tend to cluster around recurring failure patterns that are shared across models and predictable from the input alone. We therefore train a lightweight hint generator model to predict, for a given test input, which "hint" should be prepended to the prompt, providing targeted contextual or procedural guidance that steers the VLM away from its characteristic failure modes without revealing the answer. We show that Test-Time Hinting significantly improves the accuracy of multiple closed-weight VLMs on natural-image VQA benchmarks and that these gains generalize to unseen benchmarks and VLMs without retraining the hint generator.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time scaling, vision-language models, black-box models, visual question answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mH4pOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Test-Time Hinting (TTH), motivated by the observation that errors made by different proprietary VLMs on the A-OKVQA benchmark tend to concentrate around common failure modes. The authors train a smaller Qwen3-VL model that, given an image-question pair, generates hints to guide black-box target models toward more accurate predictions. They demonstrate improvements across the evaluated models and benchmarks. The hints are curated through an iterative agentic pipeline in which GPT-5 generates and refines candidate hints, which are subsequently used to train the smaller hint generator.

Critical strengths
  1. The proposed method is practical, especially considering that the hints are ultimately generated by a smaller model and the larger target model is queried only once. This makes the approach more efficient than majority-voting or Best-of-N (BoN) methods.

  2. I appreciate the inclusion of both repair rate and harm rate as evaluation metrics. I think these provide a useful and informative assessment of the method's effectiveness.

  3. The empirical gains, particularly the transferability to unseen models and the improvements on weaker target models, appear quite substantial relative to the baseline performance.

Critical weaknesses
  1. The main contribution appears to be TTH, which is essentially a form of structured, input-conditioned prompting. However, I find that the novelty claims and literature review do not fully reflect the state of the field as of September 2026. There has been considerable work on prompt optimization, including Black-Box Prompt Optimization, TEMPERA: Test-Time Prompting via Reinforcement Learning, and more recent genetic prompt optimization methods such as GEPA. In my opinion, the positioning of the proposed method relative to this literature is somewhat limited, and the novelty claims could be more clearly articulated.

  2. Related to the previous point, by September 2026, several works have explored test-time scaling (TTS) for VLMs, including efficient TTS approaches such as SPARC: Separating Perception and Reasoning Circuits for Test-time Scaling of VLMs, Efficient Test-Time Scaling for Small Vision-Language Models, and AVIS: Adaptive Test-Time Scaling for Vision-Language Models, among others. The paper does not sufficiently discuss this body of work or provide comparisons with these approaches. Even if the primary motivation is to support black-box models, I believe it would still be valuable to discuss these methods and potentially compare against some of them using open-source models, since TTH itself is not inherently limited to black-box settings.

  3. The paper argues that existing TTS methods have paid relatively little attention to vision-centric benchmarks. However, the evaluation is limited to a relatively small set of visual benchmarks, without sufficiently motivating their selection. Given the availability of benchmarks spanning perception, mathematical reasoning, real-world understanding, and other capabilities, I believe a broader evaluation or a clearer justification for the selected benchmarks would strengthen the empirical study.

  4. This is a relatively minor concern. In Figure 1, upon closer inspection of the input image, the vehicle appears to be an ATV/quad bike rather than a tractor. As such, the original model's prediction of "bike" may actually be defensible, while TTH changes the answer to "tractor." If this interpretation is correct, the example may illustrate a case of harm rather than a successful repair. I acknowledge that this could be due to ambiguity or noise in the benchmark annotations.

  5. The role of the prompting strategy itself seems somewhat underexplored, and the choice of prepending the current hint format is not fully justified. It would be useful to include additional controls, such as a simple instruction prompt encouraging the model to carefully inspect the image regions relevant to the question and ground its response in visual evidence. Another potentially informative comparison would be against simple visual prompting techniques, such as those explored in Visual Structures Help Visual Reasoning: Addressing the Binding Problem in LVLMs. These comparisons could help clarify how much of the improvement comes from the learned hints rather than from more general prompting or visual-grounding strategies.

Questions for authors
  1. Why was A-OKVQA selected as the primary benchmark for the failure-mode analysis and method development? Additionally, what motivated the selection of only three benchmarks for the transfer evaluation?

  2. Could the authors evaluate TTH on open-source VLMs and compare it with existing test-time scaling approaches for VLMs?

9 comments
AuthorsOpenReview ↗

Response to Reviewer mH4p (1/3) on W1 and W2

Thank you for the thoughtful and constructive review, and for recognizing the practicality of TTH, the value of the repair and harm metrics and the transfer to unseen models! We especially appreciate the pointers to related work and the concrete controls you suggested. We ran these controls and an evaluation on an open-weight VLM, and report them across three comments because of the length limit. W1 and W2 (with Q2) are addressed in comments 1 and 2, W3 (with Q1) in comment 2, and W4 and W5 in comment 3. Brackets are 95% CIs, and significance refers to the paired bootstrap used in the paper.


W1. Positioning relative to prompt optimization

Thank you for bringing these works together. We agree that TTH acts through the prompt and should be positioned within the prompt-optimization literature more explicitly. We group these methods by what they optimize and contrast each group with TTH.

Task-level prompt optimization. APE, OPRO, DSPy and GEPA search offline for instructions or demonstrations (GEPA by evolving them through LLM reflection on execution traces) and then apply one optimized prompt to every input of a task. TTH instead generates a new hint for every input, conditioned on its specific image, question and answer options and on the target model. This matters in natural-image VQA, where the target already understands the task and its likely error depends on the particular input. Our controls support this. Fixed prompts do not reliably help, whether the failure-mode checklist in Table 4 or the grounding instruction and VISER's visual prompt in W5, and TTH is significantly more accurate than both W5 controls on every target. A hint written for a mismatched image also loses TTH's gain entirely (see response to W5), so the gain comes from conditioning on the actual input.

Learned input-level prompt rewriting. BPO trains a prompt rewriter on human preference data so that black-box LLMs give better-aligned responses. Like TTH, it adapts each input's prompt for a black-box model, but it is about alignment and is demonstrated only on text. TTH works on a different axis, capability. It trains a small model to make a larger one more accurate on visually grounded tasks, where a hint must point to evidence in the image, and we measure both its repairs and its harms on nine VLMs from six providers. TEMPERA trains an RL policy that edits each query's instruction, in-context examples and label words, using the target's hidden states and label probabilities. Closed-weight models expose neither, which is exactly the setting TTH is built for.

Beyond conditioning on the input, TTH is trained to anticipate and prevent the target's own failure, which Section 2 shows is largely shared across models and predictable from the input alone. Its reward is defined relative to the target's base answer and penalizes turning a correct answer into a wrong one, its hints are verified with the target in the loop and must not reveal the answer, and at test time it calls the black-box target once. We will add these works to Related Work, state the contribution in these terms, and narrow the "first learned, single-call method" claim accordingly. We believe this makes TTH's contribution clearer, which is to learn from the input alone how a target is likely to fail, and to prevent that failure with a hint given before a single call.

W2 and Q2. Test-time scaling for VLMs and open-weight models

We agree that these methods should be discussed, and that TTH is not inherently limited to black-box models. Beyond discussing them, we ran the comparison you ask for. We evaluated TTH on an open-weight model, and on the same model we ran TTAug, one of the methods you cite that is applicable to this setting. The results are in comment 2. We see test-time scaling for open-weight and for closed-weight VLMs as two design pathways that allow different levels of intervention. The methods you cite were developed and evaluated on open-weight models. SPARC, the test-time augmentation and adaptation methods for small VLMs (TTAug and TTAdapt) and AVIS sample the target several times, through repeated relevance detection before a cropped re-answer, averaged predictions over augmented inputs, or an adaptive self-consistency budget, and, as evaluated, they rely on open weights or internals such as token probabilities, gradients or attention keys. Because open-weight models expose strictly more information, such methods often cannot be applied to closed-weight models, whereas a method that needs only black-box access applies to both. As practitioners increasingly rely on frontier models whose weights they cannot access, we designed TTH for the closed-weight pathway, with one pass of a compact generator and a single call to the target. Its hints can also be prepended to the inputs of open-weight methods. We will add this discussion to Related Work and make this motivation clearer in the introduction.

AuthorsOpenReview ↗

Response to Reviewer mH4p (2/3) on W2 and W3

W2 and Q2 (continued)

Open-weight target. Following your suggestion, we evaluated TTH on the open-weight Inkling-Small from Thinking Machines Lab on all four benchmarks, with Inkling as the external judge. GθG_\theta is used without retraining, with the model's name as the target identifier, so both the target and its identifier are unseen in training. We also include two methods you cite. TTAug augments each question into 16 inputs and averages their next-token probabilities at every decoding step, following its paper's default, and VISER is the training-free visual prompting method you suggested in W5.

MethodA-OKVQARealWorldQAVCRVisual7W
Base70.39 [67.77, 73.01]67.04 [63.38, 70.70]50.22 [48.84, 51.60]67.38 [66.08, 68.68]
CoT69.34 [66.64, 71.97]65.45 [61.78, 69.11]51.28 [49.90, 52.66]68.46 [67.16, 69.74]
Self-refine75.11 [72.58, 77.55]67.99 [64.33, 71.66]51.44 [50.06, 52.82]71.26 [70.00, 72.52]
External judge75.72 [73.19, 78.17]69.11 [65.45, 72.77]53.86 [52.48, 55.24]74.96 [73.76, 76.16]
TTAug76.33 [73.89, 78.78]68.47 [64.81, 72.13]54.30 [52.92, 55.68]72.56 [71.32, 73.80]
VISER67.34 [64.63, 70.04]66.72 [63.06, 70.38]51.40 [50.02, 52.78]68.92 [67.64, 70.20]
TTH85.85 [83.84, 87.86]73.41 [69.90, 76.75]57.74 [56.36, 59.10]79.18 [78.04, 80.30]

TTH is significantly more accurate than Base and every other method on all four benchmarks. TTAug improves over Base by a few points, in line with the gains reported in its paper, while TTH queries the target once instead of decoding 16 inputs in parallel. Without retraining, TTH therefore extends to an open-weight target and outperforms every baseline there.

W3 and Q1. Choice of benchmarks

Thank you for pushing us to motivate this choice. We should have phrased our point about prior work more precisely. We did not mean that test-time scaling overlooks vision-centric benchmarks, and the methods you cite evaluate on several of them. Rather, much of this literature emphasizes mathematical, scientific and knowledge-intensive reasoning, while general visual understanding has received comparatively less focus, and this is what TTH targets. We will clarify this in the introduction. Our selection of benchmarks follows from four considerations.

  • General rather than capability-specific. We wanted to show that learned hints improve visual understanding in general, without tailoring TTH to one capability. General VQA questions mix recognition, attributes, counting, spatial relations and everyday knowledge, so they test the approach broadly.
  • Failures a compact generator can address. We target questions that frontier VLMs can answer with everyday knowledge and short reasoning but sometimes get wrong because they overlook or misread something in the image. This is where a 4B generator can plausibly help a much larger target, and it separates the effect of anticipatory hints from gaps in knowledge or multi-step reasoning.
  • Diversity. The four benchmarks vary the image source, question style and answer distribution, from everyday scenes in A-OKVQA to movie stills in VCR and mostly driving scenes in RealWorldQA. Together they span knowledge-based VQA, visual commonsense reasoning, grounded perception questions and real-world spatial understanding.
  • Scale and measurability. All four are established multiple-choice benchmarks with large test sets, including 5,000 questions each for VCR and Visual7W, larger than many capability-specific benchmarks. These sizes support paired significance tests, and exact correctness gives GRPO a clear reward and makes repair, harm and answer leakage directly measurable.

Why A-OKVQA. A-OKVQA meets two needs of our training pipeline. First, learning to repair errors requires many questions that the targets get wrong, yet frontier targets answer most questions correctly, as about 71% of A-OKVQA training questions are answered correctly by all three primary targets. With 17,056 training questions, A-OKVQA still leaves enough errors to support SFT and RL, which would be much harder with a smaller dataset. Second, its questions come with ground-truth rationales, short explanations that narrate the visual evidence and knowledge behind each answer. The hint proposer (in the agentic hint optimization stage) relies on them to ground its hints in the right evidence, which makes the hints more reliable, and the failure-mode annotator of Section 2 uses them to label why a target failed.

We will add this rationale to the paper. Extending TTH to capability-specific benchmarks, such as perception and mathematical reasoning, is a natural next step that we plan to pursue. Within the current scope, TTH transfers zero-shot to all three unseen benchmarks with significant gains over Base for every target.

AuthorsOpenReview ↗

Response to Reviewer mH4p (3/3) on W4 and W5

W4. Figure 1 example

Thank you for the careful inspection. We agree that the vehicle can also be read as an ATV, in which case "bike" is defensible. The example follows the A-OKVQA label, "tractor", and reflects the ambiguity and annotation noise you mention. The other examples in Appendix J are less open to such interpretation, and we will replace this example, in Figure 1 and in Appendix J.3, with an unambiguous one so that it illustrates a clear repair.

W5. Prompting controls

We agree that these controls are needed to isolate what the learned hints contribute, and ran both on the A-OKVQA test set with the three primary targets. We also added a third control that keeps TTH's learned hints but breaks their link to the image.

  • Grounding instruction. A static instruction, "Inspect the relevant regions and ground your answer in visual evidence."
  • VISER. The visual prompting method you suggested, from "Visual Structures Help Visual Reasoning". Following its default setup, we draw three equally spaced 1-pixel black lines that split the image into four bands numbered 1 to 4 on the left, and add the prompt "Scan the image sequentially based on horizontal lines".
  • Mismatched-image hint. GθG_\theta writes the hint from a mismatched image, and the target answers with this hint and the correct image. This tests whether the hints work through the visual evidence of each image or through their form alone.
TargetMethodAccuracyRepairHarm
Claude 4.5 HaikuBase78.25 [75.89, 80.61]--
Grounding instruction76.24 [73.80, 78.69]18.88 [14.23, 23.94]7.81 [6.09, 9.63]
VISER75.20 [72.75, 77.73]14.06 [9.88, 18.47]7.81 [6.12, 9.62]
Mismatched-image hint74.85 [72.31, 77.38]20.88 [15.94, 26.02]10.16 [8.19, 12.16]
TTH84.28 [82.09, 86.45]49.406.03
Gemini 2.5 Flash LiteBase82.88 [80.70, 85.06]--
Grounding instruction85.50 [83.41, 87.51]32.14 [25.69, 38.91]3.48 [2.34, 4.69]
VISER79.56 [77.21, 81.83]22.96 [17.13, 29.13]8.75 [6.98, 10.61]
Mismatched-image hint81.40 [79.13, 83.58]12.24 [7.83, 17.02]4.32 [3.07, 5.66]
TTH90.57 [88.73, 92.31]61.733.48
GPT-5 NanoBase84.37 [82.27, 86.47]--
Grounding instruction84.63 [82.53, 86.64]31.84 [25.26, 38.81]5.59 [4.18, 7.09]
VISER83.76 [81.57, 85.85]29.05 [22.58, 35.80]6.11 [4.65, 7.68]
Mismatched-image hint84.02 [81.83, 86.11]29.61 [23.03, 36.42]5.90 [4.46, 7.44]
TTH88.12 [86.11, 90.04]45.814.04

TTH is significantly more accurate than all three controls on all three targets, and repairs more errors on every target at equal or lower harm. The grounding instruction significantly helps Gemini 2.5 Flash Lite but significantly hurts Claude 4.5 Haiku, and VISER significantly lowers accuracy on both, with no significant change on GPT-5 Nano for either. We suspect that VISER's lines, designed for binding problems such as counting and visual search, distract the target on our questions, which mostly require general reasoning about scenes and knowledge. When the hint is written from a mismatched image, TTH's gain disappears and accuracy falls to or below Base on all three targets, so the hints work through the visual evidence of each image rather than through their form.

We prepend the hint so that it acts before the target answers, in a single call, and make it contrastive so that it points to discriminating evidence without revealing the answer. Together with the static-hint and untrained-generator ablations in Table 4, these controls indicate that TTH's gains come from the learned, input-specific content of its hints rather than from generic prompting or visual-grounding strategies.


We sincerely thank you again for the thoughtful review and the valuable pointers! We will add the new results, the extended related-work discussion and the benchmark rationale to the revision.

Reviewer_5aMROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Test-Time Hinting (TTH), which generates input-specific guidance before a black-box vision-language model answers. A compact 4B hint generator is trained through supervised distillation of agentically constructed hints, followed by reinforcement learning using downstream correctness across multiple target models. Inference requires one generator pass and one target-model call. Experiments show improvements on natural-image VQA, including transfer to unseen target models and benchmarks.

Critical strengths

The idea is intuitive and practically useful: anticipating likely mistakes before answering avoids the repeated target calls required by post-hoc correction. The framework coherently combines hint construction, distillation, and outcome-based optimization. The evaluation across multiple models and datasets, together with comparisons against static hints, an untrained generator, and different training stages, provides meaningful support for the approach. Reporting both repair and harm rates is also valuable.

Critical weaknesses
  1. The rationale for excluding mixed question–model pairs is insufficiently established. Consistently incorrect examples may include both repairable mistakes and failures beyond the target’s capabilities, while inconsistently answered examples could provide useful supervision for improving reliability. An ablation including mixed pairs, ideally at a matched training budget, would clarify whether the current selection improves learning or primarily simplifies data construction.

  2. The three categories are defined using only three responses at API-default decoding settings. Their stability under resampling, different sample counts, or changes to temperature and prompting is unclear. Please report category proportions and reassignment rates, and assess whether these choices materially affect downstream performance. Repeated evaluation or an unhinted resampling control would also help separate hint-induced repair and harm from ordinary response variability.

  3. The ablations support the effectiveness of the compact generator, but they do not fully establish what it learns. Although the paper includes leakage checks and a qualitative harmful-hint analysis, stronger visual-grounding controls would help distinguish useful guidance from superficial correlations. For example, image ablations or controlled changes to relevant visual evidence could test whether hints respond appropriately; quantifying irrelevant or fabricated guidance would strengthen the analysis beyond answer accuracy.

  4. Transfer to unseen target models yields modest net gains despite substantially higher repair rates. In Table 2, improvements over the strongest baseline are only 0.17–0.70 percentage points, while harm rates generally increase. The paper’s limited-error-headroom explanation is reasonable, but explicit paired significance results for these comparisons and further analysis of the repair–harm trade-off would clarify the practical benefit.

Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5aMR (1/2) on W1 to W3

Thank you for the thoughtful and constructive review! We especially appreciate that each concern came with a concrete experiment that could address it. We ran new experiments for each concern and report them across two comments because of the length limit. W1 and W2 are addressed in comment 1, W3 in comments 1 and 2, and W4 in comment 2. Brackets are 95% CIs, and significance refers to the paired bootstrap used in the paper.


W1. Excluding mixed question-model pairs

Thank you for this thoughtful intuition. We did not benchmark this choice in our original experiments. Following your suggestion, we retrained GθG_\theta with the mixed pairs included, to encourage hints that do not harm the instances a target already answers correctly in mixed pairs while repairing the instances it answers incorrectly. To match the training budget, the mixed pairs replace part of the base-correct pairs, so training runs for the same number of steps and only the composition of the data differs, giving a head-to-head comparison.

TargetTTHTTH trained with mixed pairs
Claude 4.5 Haiku84.28 [82.09, 86.45]84.54 [82.45, 86.64]
Gemini 2.5 Flash Lite90.57 [88.73, 92.31]90.66 [88.91, 92.31]
GPT-5 Nano88.12 [86.11, 90.04]88.47 [86.55, 90.31]

Including the mixed pairs improves learning, with a mild gain on all three targets on the A-OKVQA test set, although none of the gains is statistically significant. This supports your intuition that mixed pairs provide useful supervision, and we will add this ablation to the paper.

W2. Stability of the three-response categories

We agree that response variability is a well-known property of frontier-model APIs. Our limited API budget had to cover experiments on large VQA test sets (1,145 questions for A-OKVQA, 5,000 each for VCR and Visual7W, and 628 for RealWorldQA), so we could not afford the multiplying API cost of quantifying this variability across combinations of sample counts, temperatures and prompts. This is why we used each API's default settings, the setting most practitioners use, and why we bound all results with 95% CIs and paired significance tests. We acknowledge this as a limitation of the study and will add it to the discussion. Within this budget, we report the category proportions and the unhinted resampling control you suggested.

Category proportions. Of the training question-model pairs, 7.2% are mixed. Including mixed pairs in training (W1) changes accuracy by at most 0.35 points, and only upward, so downstream performance is not materially affected by how these pairs are categorized.

Unhinted resampling control. We drew a second unhinted sample for the three primary targets on A-OKVQA at the same settings and scored it against the first, which measures the resampling noise floor.

TargetAccuracyRepairHarmTTH repairTTH harm
Claude 4.5 Haiku77.21 [74.76, 79.65]6.02 [3.24, 9.15]3.01 [1.94, 4.18]49.406.03
Gemini 2.5 Flash Lite82.53 [80.35, 84.72]8.16 [4.52, 12.20]2.11 [1.26, 3.07]61.733.48
GPT-5 Nano84.98 [82.88, 86.99]15.64 [10.50, 21.15]2.17 [1.32, 3.14]45.814.04

This control separates hint-induced repair and harm from ordinary response variability. TTH's repair is 30 to 54 points above the floor, while its harm is only 1.4 to 3.0 points above it. By contrast, the repair rates of CoT and self-refine on GPT-5 Nano in Table 1 (18.44 and 17.88) fall within the resampling interval. Beyond ordinary response variability, the hints therefore add far more repairs than harms.

W3. What the hint generator learns and visual grounding

We agree that answer accuracy alone does not show what GθG_\theta learns.

Irrelevant or fabricated guidance. To quantify this, we extended the analysis of Appendix I to 100 harmful cases, in which TTH turned a correct Base answer into a wrong one. As in Appendix I, fabricated guidance is the least frequent pattern, appearing in only 6 of the 100 cases, so fabrication is a rare source of harm. We will add these counts to Appendix I.

Visual grounding. Following your suggestion, we ran two image ablations on the A-OKVQA test set with the three primary targets, changing the image given to GθG_\theta and the image given to the target separately.

  • Hinter-blinded grounding test. GθG_\theta writes the hint from a mismatched image, and the target answers with this hint and the correct image. This tests whether the hints depend on the visual evidence.
  • Target-blinded leakage test. GθG_\theta writes the hint from the correct image, but the target sees only the hint, the question and the options, without the image. This tests whether the hint alone reveals the answer.
AuthorsOpenReview ↗

Response to Reviewer 5aMR (2/2) on W3 and W4

W3 (continued)

TargetConditionAccuracyRepairHarm
Claude 4.5 HaikuBase78.25 [75.89, 80.61]--
TTH84.28 [82.09, 86.45]49.406.03
Hinter-blinded74.85 [72.31, 77.38]20.88 [15.94, 26.02]10.16 [8.19, 12.16]
Target-blinded53.19 [50.31, 56.07]--
Gemini 2.5 Flash LiteBase82.88 [80.70, 85.06]--
TTH90.57 [88.73, 92.31]61.733.48
Hinter-blinded81.40 [79.13, 83.58]12.24 [7.83, 17.02]4.32 [3.07, 5.66]
Target-blinded53.54 [50.66, 56.42]--
GPT-5 NanoBase84.37 [82.27, 86.47]--
TTH88.12 [86.11, 90.04]45.814.04
Hinter-blinded84.02 [81.83, 86.11]29.61 [23.03, 36.42]5.90 [4.46, 7.44]
Target-blinded54.41 [51.53, 57.29]--

When GθG_\theta sees a mismatched image, its hints lose much of their repair and cause more harm, so TTH's gain disappears and accuracy falls to or below Base on all three targets. When the target does not see the image, it stays far below TTH even with the hint, so the hint does not simply reveal the answer. TTH's gains therefore depend on the correct image on both sides, which indicates that the hints are grounded in the visual evidence rather than in the question alone.

W4. Paired tests and the repair-harm trade-off on unseen targets

Repair-harm trade-off. We agree that TTH has the highest or tied-highest harm on the unseen A-OKVQA targets, and we will state this explicitly. TTH changes the target's answer more often than the baselines, which raises repair and harm together, and the net effect on accuracy is ΔAcc=(1−B)R−BH\Delta\mathrm{Acc} = (1-B)R - BH. Because repair is weighted by 1−B1-B and harm by BB, one point of harm offsets 5 to 11 points of repair at the base accuracies of Table 2. This is why a repair rate 1.5 to 3.8 times that of the strongest baseline yields margins of only 0.17 to 0.70 points. On a less saturated benchmark (results shown below), the same repair advantage should yield much larger and more observable gains, since at a base accuracy of about 77% one point of harm offsets only about 3 points of repair.

Paired tests for Table 2. We agree that these comparisons need explicit paired tests, and we will add them to Table 9. On A-OKVQA, TTH has the highest accuracy on all five unseen targets, but its margin over the strongest baseline is not significant on any of them, and no baseline is significantly better than TTH either. Against Base, TTH's gain is significant only for Grok 4 Fast Reasoning, and the baselines face the same limit, as none improves significantly over Base except self-refine on Grok. Rather than concluding that no method can improve these targets, we read this as a statistical limitation of high base accuracy on 1,145 questions, where few errors remain from which to obtain a significant net gain, as our headroom analysis in Appendix H shows. Although we cannot show significance on Table 2, the same paired tests on a less saturated benchmark give statistically significant results, which we report next on RealWorldQA.

Unseen frontier targets on a less saturated benchmark. We evaluated three of these unseen frontier targets on RealWorldQA, which is also unseen in training and where their base accuracy is about 77% rather than 90 to 92%. BCM is the best competing method among CoT, self-refine and the external judge.

TargetMethodAccuracyRepairHarm
Claude 4.5 OpusBase76.91 [73.57, 80.10]--
BCM (external judge)78.18 [74.84, 81.37]14.48 [8.96, 20.42]2.69 [1.41, 4.23]
TTH78.03 [74.84, 81.21]33.79 [26.19, 41.73]8.70 [6.28, 11.27]
Gemini 2.5 FlashBase76.75 [73.41, 79.94]--
BCM (CoT)77.71 [74.36, 80.89]15.75 [10.00, 21.97]3.53 [1.94, 5.29]
TTH81.85 [78.82, 84.87]33.56 [26.00, 41.35]3.53 [1.99, 5.26]
GPT-5.2Base77.23 [73.89, 80.41]--
BCM (self-refine)78.18 [75.00, 81.37]20.28 [13.79, 27.14]4.74 [2.95, 6.73]
TTH83.44 [80.57, 86.31]49.65 [41.50, 57.69]6.60 [4.47, 8.88]

TTH significantly outperforms Base and every benchmarked method for Gemini 2.5 Flash and GPT-5.2, and its margin over BCM grows from 0.17 and 0.26 points on A-OKVQA to 4.14 and 5.26 points. For Claude 4.5 Opus, TTH more than doubles the repair of the external judge (33.79 vs 14.48), but its higher harm (8.70 vs 2.69) offsets this, and the two are indistinguishable while both improve over Base by more than 1 point. This establishes joint transfer to an unseen benchmark and unseen targets for two of the three, where TTH's repair advantage turns into a significant practical benefit.


We sincerely thank you again for the thoughtful review and the concrete suggestions! We will add all new results and paired tests to the revision, together with the discussion of API variability.

AuthorsOpenReview ↗

Revised paper uploaded for Reviewer 5aMR

Thank you again for your thoughtful review and the concrete suggestions! We have now uploaded the revised paper. All new results, paired tests and changes described in our response are in it, and all new or changed text is marked in blue so that it is easy to find.

Summary of the main conclusions. Training with the mixed pairs at a matched budget gives a mild but not significant gain on all three primary targets, so how these pairs are categorized does not materially affect performance (W1 and W2). A second unhinted sample repairs far fewer errors than TTH, so the repairs go well beyond ordinary response variability. The gain of TTH disappears when the hint is written from a mismatched image, and without the image the hinted target stays far below TTH, so the hints work through the visual evidence and do not simply reveal the answer (W3). Fabricated guidance is a rare source of harm. On the unseen targets on A-OKVQA (W4), where few errors remain, TTH's margin over the strongest baseline is not significant and its harm rate is the highest or tied-highest. On the less saturated RealWorldQA, TTH is significantly more accurate than Base and every baseline for two of the three unseen targets.

The table below shows where each of your points is addressed in the revision.

Your pointWhere it is in the revision
W1 Training with mixed pairsAppendix N.2, Table 20, noted in Step 1 of Section 3
W2 Category proportionsStep 1 of Section 3 and Appendix N.2
W2 Unhinted resampling controlAppendix N.1, Table 19. The second sample is also a reference row in Table 4, and Section 5.2 compares the repair rates of TTH with this floor
W2 API variability as a limitationLimitations in Section 6
W3 Irrelevant or fabricated guidanceAppendix I, now based on 100 harmful cases
W3 Hinter-blinded grounding testRow "Mismatched-image hint" of Table 4, paragraph "The hints depend on the image" in Section 5.5, and Appendix L.3, Table 15
W3 Target-blinded leakage testAppendix L.4, Table 16, now on all four benchmarks, summarized in the leakage audit of Section 5.5
W4 Repair-harm trade-offStated in Section 5.3, with the arithmetic in Appendix H
W4 Paired tests for Table 2Table 9 and "Paired-test conclusions" in Appendix D, summarized in Section 5.3
W4 Unseen targets on a less saturated benchmarkNew paragraph in Section 5.3 and Appendix M.1, Table 17

We hope the revised paper resolves your concerns. If it does, we would be grateful if you would consider updating your rating. If anything remains unclear, we would be glad to address it before the discussion period ends.

We sincerely thank you again for your time and for helping us improve the paper!

Reviewer_PTiROpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

mm, and produces a short contrastive natural-language hint that is prepended to the target’s prompt. Training data come from an agentic loop in which GPT-5 proposes and edits hints using the ground truth and the target’s wrong answer, and the target verifies them. The generator is trained with SFT on verified hints and then with GRPO, using a reward averaged over three targets that scores repair, no-op, harm, and unrepaired outcomes. The method is motivated by an A-OKVQA analysis showing that errors of three budget-tier proprietary VLMs overlap, share failure-mode labels, and are partly predictable from a 4B model’s hidden state. On A-OKVQA, TTH improves Claude 4.5 Haiku, Gemini 2.5 Flash Lite, and GPT-5 Nano by 3.8 to 7.7 points and significantly beats CoT, self-refine, and an external judge on two of the three targets. The same checkpoint is evaluated zero-shot on five unseen targets and on VCR, Visual7W, and RealWorldQA. The paper also includes ablations, leakage audits, bootstrap confidence intervals, and cost estimates.

Critical strengths

S1. Consistent gains under a realistic deployment constraint. The single-call, black-box setting matches how many applications use proprietary VLMs, and the baselines TTH beats need two or three sequential calls. On the three training targets, TTH is significantly more accurate than Base in all 12 target-benchmark cells. For Claude 4.5 Haiku and Gemini 2.5 Flash Lite it is significantly more accurate than every baseline on all four benchmarks, including RealWorldQA, which differs in image domain from the training data. This is the main reason I consider the method useful in practice.

S2. Careful evaluation on most axes. The repair/harm decomposition is tied to accuracy through ΔAcc=(1−B)R−BH\Delta\text{Acc} = (1-B)R - BH. All main results have paired bootstrap intervals. The paper also adds an untrained hint-writer control, a manual analysis of harmful hints, a headroom-normalized gain analysis, and explicit training cost ($260 in API calls, about 11 H200 hours). The untrained-generator control in particular shows that the training pipeline, rather than the mere presence of a hint, drives the gain. These additions make most claims checkable and raise my confidence in the numbers for the primary targets.

Critical weaknesses

W1. The source of the gains is not established. GθG_\theta is a 4B VLM fine-tuned on 17K in-domain questions to imitate hints that GPT-5 wrote with access to the ground truth. Its hints can therefore carry its own reading of the answer without naming an option. Appendix J.1 shows this with “just casual summer clothes and sunglasses” and “Focus on the people, not the truck branding”. The oracle failure-category ablation (Table 4) does not support failure-mode anticipation as the mechanism. The text-only probes cannot detect this effect. They never see the question, their options-only baseline is at chance, and two of the end-to-end intervals reach about 38%. The missing controls are the direct accuracy of GθG_\theta and of a LoRA-tuned 4B answerer, a text-only LLM probe that also sees the question, and same-supervision baselines (answer injection, kNN few-shot). If a fine-tuned 4B model alone approaches TTH, the contribution is small-model-guided ensembling rather than failure anticipation.

W2. Claims exceed the evidence, and the closest prior work is missing. Directional Stimulus Prompting (Li et al., NeurIPS 2023) already trains a small model with SFT and RL against a black-box LLM to generate instance-specific hints. LM-Guided CoT (Lee et al., 2024) and BPO (Cheng et al., 2024) are also close, so the claim to be “the first learned, single-call method” should be narrowed. The claim that gains generalize to unseen VLMs has no paired tests (none for Table 9). On frontier-tier targets the gains are only 0.17 to 0.87 points, and TTH has the highest or tied-highest harm rate on all five unseen targets. All four benchmarks are multiple choice, yet the abstract frames the method around natural-image VQA and frontier models. The demonstrated value is for budget-tier models on multiple-choice questions.

Minor Points:

  • There is no re-run Base control, so repair and harm rates include an unmeasured resampling noise floor.
  • The Stage 2 reward is averaged over all three targets and ignores mm, so target conditioning is untested.
  • The Table 9 interval for TTH on Gemini 2.5 Flash ([90.73, 93.34]) is narrower than a bootstrap over 1,145 questions allows (about ±1.6 points).
Questions for authors

Clarifications:

  • Is Base at test time a single sample? If so, what are the repair and harm rates of a second unhinted sample?
  • Which mm is used for Grok and Nova, and how do results change with a swapped or empty mm?
  • Can you verify the Table 9 interval for TTH on Gemini 2.5 Flash?

Requested experiments:

  • What is the direct accuracy of GθG_\theta, and of a LoRA-tuned Qwen3-VL-4B answerer, on all four benchmarks? How often does the hinted target follow GθG_\theta’s answer when that answer is wrong?
  • Does a capable text-only LLM, given the question and options, gain accuracy when the hint is added?
  • How does TTH compare with injecting the small model’s answer and confidence, and with kNN few-shot prompting from the same training set?
  • Can you report paired tests for Table 9 and for SFT versus SFT+GRPO? Can you also evaluate the unseen targets on a less saturated unseen benchmark such as RealWorldQA?

Addressing these can change my rating.

11 comments
AuthorsOpenReview ↗

Response to Reviewer PTiR (1/4) on W1

Thank you for the careful and technically detailed review! We especially appreciate that each concern came with a concrete experiment that could resolve it. We ran the requested experiments and report them across four comments because of the length limit. W1 is addressed in comments 1 and 2, W2 in comments 3 and 4, and the minor points and questions in comment 4. Brackets are 95% CIs, and significance refers to the paired bootstrap used in the paper.


W1. Do the hints carry GθG_\theta's own answer?

We agree that an individual hint, such as the one in Appendix J.1, can carry part of GθG_\theta's reading of the image, and we do not claim that no hint ever does. The question W1 raises is whether such answer signal explains TTH's gains. Since J.1 is a single example, we used the controls you proposed to test this quantitatively across benchmarks and targets in (a) to (d) below. Together, they show that it does not account for most of TTH's gains.

(a) Direct accuracy of GθG_\theta and of a LoRA-tuned 4B answerer. The answerer is Qwen3-VL-4B-Instruct fine-tuned with LoRA to answer directly, using the same SFT recipe and the same training budget as GθG_\theta on the A-OKVQA training split.

BenchmarkGθG_\theta (hint generator)LoRA-tuned 4B answerer
A-OKVQA77.55 [75.11, 80.00]85.76 [83.67, 87.77]
VCR66.00 [64.68, 67.30]62.12 [60.78, 63.46]
Visual7W78.24 [77.10, 79.38]81.74 [80.66, 82.80]
RealWorldQA66.72 [63.06, 70.22]69.27 [65.61, 72.77]

Training for hinting makes GθG_\theta a better hinter (Table 4) but a worse answerer than its initialization. This is the intended trade-off, since GθG_\theta is optimized to steer the target rather than to answer.

The TTH-hinted target is more accurate than this answerer in 9 of the 12 target-benchmark cells (Tables 1 and 3), by 2.36 to 7.32 points, and is at most 1.5 points below it in the other 3, all for Claude 4.5 Haiku. TTH's gains are therefore not explained simply by a 4B model having been trained on this dataset, and are more plausibly due to the hints themselves.

(b) How targets answer when GθG_\theta is wrong. Values are percentages of the questions that GθG_\theta answers incorrectly.

BenchmarkTargetTarget correct (TTH / Base)Target picks GθG_\theta's wrong option (TTH / Base)
A-OKVQAClaude 4.5 Haiku59 / 5726 / 28
Gemini 2.5 Flash Lite70 / 6020 / 24
GPT-5 Nano62 / 6323 / 23
VCRClaude 4.5 Haiku42 / 3144 / 44
Gemini 2.5 Flash Lite41 / 2845 / 49
GPT-5 Nano36 / 3545 / 48
Visual7WClaude 4.5 Haiku52 / 4235 / 40
Gemini 2.5 Flash Lite58 / 4634 / 36
GPT-5 Nano55 / 5431 / 31
RealWorldQAClaude 4.5 Haiku50 / 4034 / 42
Gemini 2.5 Flash Lite56 / 4134 / 44
GPT-5 Nano51 / 4138 / 44

When GθG_\theta is wrong, the hinted target is generally right more often than Base, so the hints do not simply pass or imply GθG_\theta's answer on to the target.

AuthorsOpenReview ↗

Response to Reviewer PTiR (2/4) on W1

W1 (continued)

(c) Is TTH small-model-guided ensembling? The question is not whether a 4B model can answer well, but whether TTH works by passing a small model's answer to the target. Three results show that it does not.

  1. Injecting the LoRA-tuned answerer's answer and confidence (Prophet-style) and kNN few-shot prompting from the same training set are both significantly worse than TTH on all three targets (table below).
  2. On A-OKVQA, GθG_\theta answers well below this answerer (77.55 vs 85.76), yet the targets it hints reach 84.28 to 90.57.
  3. On RealWorldQA, three unseen frontier targets already answer better than this answerer at Base (about 77 vs 69.27), and TTH still lifts them to 78.03 to 83.44, with significant gains on two of them (comment 3).
TargetBaseAnswer injectionkNN few-shot (k=8k=8)TTH
Claude 4.5 Haiku78.25 [75.89, 80.61]75.02 [72.49, 77.47]77.55 [75.11, 80.00]84.28 [82.09, 86.45]
Gemini 2.5 Flash Lite82.88 [80.70, 85.06]86.11 [84.10, 88.12]84.89 [82.79, 86.90]90.57 [88.73, 92.31]
GPT-5 Nano84.37 [82.27, 86.47]85.07 [82.97, 87.07]84.89 [82.79, 86.90]88.12 [86.11, 90.04]

Answer injection gives the target the answerer's choice and its softmax confidence over the option letters, through the same prompt as TTH, so only the hint content differs. A typical injected hint reads "A smaller vision-language model (Qwen3-VL-4B) answered B (bike) with 87% confidence." kNN few-shot uses the 8 training questions nearest to each test question by CLIP ViT-L/14 image and question similarity, drawn from the same 17,056-question pool used to train GθG_\theta.

Answer injection pushes Claude 4.5 Haiku below Base even though the injected answers are more accurate than its own. Haiku has the highest harm rate among our targets in most settings (Tables 1 and 3) and appears sensitive to injected content. We suspect it sometimes treats another model's answer as a distractor and second-guesses a correct answer.

(d) Leakage test with a text-only LLM that sees the question. This test asks whether the hint itself leaks the answer to the target. We gave each target the question and options without the image, with and without the hint. These are capable LLMs and the same models that read the hints in TTH.

Benchmark (chance)TargetText onlyText + hintTTH with image
A-OKVQA (25)Claude 4.5 Haiku50.83 [47.95, 53.71]53.19 [50.31, 56.07]84.28
Gemini 2.5 Flash Lite49.00 [46.11, 51.88]53.54 [50.66, 56.42]90.57
GPT-5 Nano50.66 [47.77, 53.54]54.41 [51.53, 57.29]88.12
VCR (25)Claude 4.5 Haiku39.06 [37.70, 40.42]40.22 [38.86, 41.58]64.98
Gemini 2.5 Flash Lite39.80 [38.44, 41.16]40.74 [39.38, 42.10]67.04
GPT-5 Nano42.68 [41.32, 44.06]41.74 [40.38, 43.10]68.68
Visual7W (25)Claude 4.5 Haiku39.54 [38.18, 40.90]43.00 [41.62, 44.38]81.26
Gemini 2.5 Flash Lite40.08 [38.72, 41.44]46.36 [44.98, 47.74]85.22
GPT-5 Nano38.94 [37.60, 40.30]40.02 [38.66, 41.38]84.44
RealWorldQA (38.2)Claude 4.5 Haiku40.92 [37.10, 44.75]45.06 [41.24, 48.89]69.11
Gemini 2.5 Flash Lite38.22 [34.39, 42.04]41.24 [37.42, 45.06]76.59
GPT-5 Nano43.63 [39.81, 47.45]42.52 [38.69, 46.34]75.64

To your question, a capable text-only LLM does gain accuracy from the hint, but only mildly. Without the image, the models already score well above chance on A-OKVQA, VCR and Visual7W, so the questions themselves carry some bias toward certain answers. The hint adds a few points in most settings, so it can carry a mild cue usable without the image. If the hints leaked GθG_\theta's answer, however, a capable reader could recover it from the text and approach GθG_\theta's own accuracy in (a). Instead, Text + hint stays 22 to 38 points below it on every benchmark, and far below TTH with the image, so this cue does not account for most of TTH's gains.

(e) Oracle failure-category ablation (Table 4). We agree that this ablation does not support categorical failure prediction as the mechanism, and we did not intend it to. It was meant to show that knowing the category of a likely failure, such as a spatial-relation error, is not enough to help the target. Useful anticipation has to be fine-grained and specific, pointing to the context of the question or the image evidence where the target is likely to go wrong. The ablation therefore shows that a categorical predictor does not work, not that failure anticipation does not work. We will rewrite this paragraph to make the distinction clear.

AuthorsOpenReview ↗

Response to Reviewer PTiR (3/4) on W2

W2. Prior work, unseen targets and scope

DSP, LM-Guided CoT and BPO. Thank you for these references. We agree they are the closest prior work and should have been discussed. Like TTH, each trains a small model to steer a frozen black-box LLM in a single call.

  • DSP is the closest. A small policy model such as T5, trained with SFT and RL, generates instance-specific stimuli such as keywords that steer a text-only LLM toward a desired output in summarization, dialogue and reasoning.
  • LM-Guided CoT trains a small model by distillation and RL to write a rationale for each input, from which a frozen large model predicts the answer.
  • BPO trains a prompt rewriter on human preference data so that black-box LLMs give better-aligned responses.

TTH differs in what its guidance is trained to do. DSP and BPO are mainly about alignment, steering a text LLM toward desired outputs such as summaries with given keywords or responses that match human preferences. TTH works on a different axis, capability. It trains a small model to make a larger one more accurate, in a controlled setting where repair, harm and overall accuracy are all reported. It works on images, so a hint must point to visual evidence. It is trained to anticipate and prevent the target's own failure on each input, with a reward defined relative to the target's base answer that penalizes turning a correct answer into a wrong one, whereas DSP rewards the quality of the LLM's output. Its hints are verified with the target in the loop, must contrast plausible alternatives and must not reveal the answer, which leaves the reasoning and the answer to the target, whereas LM-Guided CoT supplies the reasoning itself. We will add this comparison to Related Work and narrow the "first learned, single-call method" claim accordingly.

Paired tests for Table 9. On A-OKVQA, TTH has the highest accuracy on all five unseen targets, but its gain over Base is significant only for Grok 4 Fast Reasoning. The baselines face the same limit, as none improves significantly over Base except self-refine on Grok. Rather than concluding that no method can improve these targets, we read this as a statistical limitation of high base accuracy on 1,145 questions, where few errors remain from which to obtain a significant net gain, as our headroom analysis in the appendix shows. We will add these tests to Table 9.

Unseen frontier targets on a less saturated benchmark. Following your suggestion, we evaluated three unseen frontier targets on RealWorldQA, which is also unseen in training and where their base accuracy is about 77% rather than 90 to 92%. BCM is the best competing method among CoT, self-refine and the external judge.

TargetMethodAccuracyRepairHarm
Claude 4.5 OpusBase76.91 [73.57, 80.10]--
BCM (external judge)78.18 [74.84, 81.37]14.48 [8.96, 20.42]2.69 [1.41, 4.23]
TTH78.03 [74.84, 81.21]33.79 [26.19, 41.73]8.70 [6.28, 11.27]
Gemini 2.5 FlashBase76.75 [73.41, 79.94]--
BCM (CoT)77.71 [74.36, 80.89]15.75 [10.00, 21.97]3.53 [1.94, 5.29]
TTH81.85 [78.82, 84.87]33.56 [26.00, 41.35]3.53 [1.99, 5.26]
GPT-5.2Base77.23 [73.89, 80.41]--
BCM (self-refine)78.18 [75.00, 81.37]20.28 [13.79, 27.14]4.74 [2.95, 6.73]
TTH83.44 [80.57, 86.31]49.65 [41.50, 57.69]6.60 [4.47, 8.88]

TTH significantly outperforms Base and every benchmarked method for Gemini 2.5 Flash and GPT-5.2, gaining 5.10 and 6.21 points over Base, compared with 0.17 to 0.87 for the same targets on A-OKVQA. For Claude 4.5 Opus, TTH and the external judge (BCM) are indistinguishable. This establishes joint transfer to an unseen benchmark and unseen targets for two of the three.

Harm on unseen targets. We agree that TTH has the highest or tied-highest harm on the unseen A-OKVQA targets, and we will state this explicitly. TTH changes the target's answer more often than the baselines, which raises repair and harm together, and the net effect on accuracy is ΔAcc=(1−B)R−BH\Delta\mathrm{Acc} = (1-B)R - BH. Claude 4.5 Opus on RealWorldQA illustrates this. Its harm under TTH is higher than under the external judge (8.70 vs 2.69), but its repair is more than twice as high (33.79 vs 14.48). On Gemini 2.5 Flash and GPT-5.2, the repair advantage outweighs any extra harm. On these two targets, TTH's harm is not significantly higher than that of the best competing method, even though its repair and its eventual gain in accuracy are significantly higher.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.