acceptodds
Under review as a conference paper at ICLR 2027

Can Visual Chain-of-Thought Trust Its Visual Evidence? Pre-Consumption Evidence Verification for Interleaved VLMs

18 views

Abstract

Interleaved vision-language models revisit images during reasoning, yet typically consume newly selected evidence without checking whether it matches the current visual request. We show through controlled evidence interventions that such mismatches are consequential: replacing erroneous evidence with ground-truth evidence substantially improves answer accuracy, whereas counterfactual evidence degrades it. Moreover, correcting errors after consumption poses a dilemma: appending correct evidence only partially recovers accuracy, while rollback requires discarding already-generated reasoning. These findings motivate a simple principle: verify visual evidence before consumption. We introduce VERIFYCOT, which checks whether selected evidence matches the model’s current visual request. This request is derived directly from the model’s hidden states, eliminating the need to extract a textual reference. Evidence that passes is kept, and evidence that fails is corrected before reasoning continues. Across interleaved VLM architectures and five visual reasoning benchmarks, VERIFYCOT improves end-to-end accuracy in 14 of 15 evaluated settings, with gains of up to 15.7 percentage points. Compared with unconditional grounding, it achieves higher accuracy with fewer grounding calls, while also offering a better accuracy–compute trade-off than the evaluated post-hoc correction methods. We provide code in the supplementary material and will release it on GitHub.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Chain-of-Thought, Interleaved Vision-Language Models, Multimodal Reasoning, Pre-Consumption Verification, Evidence Verification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TsmcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This submission asks whether interleaved vision-language models can rely on the visual evidence they pick up in the middle of reasoning. Controlled interventions on the selected evidence show that swapping in ground-truth regions yields large accuracy gains, whereas counterfactual evidence hurts; fixing mistakes after consumption is unsatisfying in both directions — appending the correct region recovers accuracy only partially, while rollback throws away reasoning already produced. These findings motivate a "verify before you consume" principle, realized as VerifyCoT.

Critical strengths
  1. The causal analysis is the core asset. The oracle/counterfactual intervention design supplies direct, quantitative evidence that selection mistakes drive downstream errors, so the central verify-first principle is empirically established.
  2. Thoughtful engineering and broad evaluation. Deriving requests from hidden states sidesteps fragile, model-specific textual-reference extraction, and the verify-then-selectively-correct pipeline avoids wasteful grounder calls. Results span multiple architectures and five benchmarks with consistent, often sizable, improvements.
Critical weaknesses
  1. The framework needs annotated target regions to train its request interfaces and verifier — it is not training-free. Portability to new backbones or tasks lacking such annotations is therefore limited, and the deployment cost relative to plug-and-play alternatives is under-discussed.
  2. Gains shrink noticeably on RL-trained variants, and TreeBench slightly regresses under Visual CoT, indicating that verification cannot repair reasoning failures that occur after evidence consumption. Sensitivity to the verification threshold τ and the verifier's own error modes (the non-zero Harm column in Table 2) receive only thin analysis.
Questions for authors

Need more analysis around the critical Weaknesses.

7 comments

Thank you for the thoughtful review and for recognizing the motivation behind our work. Your main concerns are adaptation cost and portability, performance after RL training and on TreeBench, whether model performance is sensitive to the verification threshold, and the main sources of harmful changes. We report adaptation costs and cheaper alternatives, analyze the results after RL training and on TreeBench, examine threshold sensitivity, and identify the sources of harmful changes through manual inspection.

Tsmc-Q1: Adaptation and deployment costs

The framework needs annotated target regions to train its request interfaces and verifier — it is not training-free. Portability to new backbones or tasks lacking such annotations is therefore limited, and the deployment cost relative to plug-and-play alternatives is under-discussed.

Response. VerifyCoT is not training-free, but its adaptation costs are modest. With available SFT data, we train the request interfaces and verifier directly. Even without the model’s original SFT data, we can adapt VerifyCoT using annotations from another dataset and generated trajectories. Even this training can be skipped by using LLM-cleaned references as a plug-and-play alternative, which enables VerifyCoT adaptation with almost no additional training cost.

1. Adapting with the model's SFT data

For Pixel-Reasoner and Visual CoT, we reuse the trajectories and target-region annotations in their SFT data, so no additional manual region annotation is required in this setting. Adaptation consists of preparing features and training the request interfaces and verifier. The base VLM, semantic encoder, and grounding backbone remain frozen. The table reports the total SFT images processed and the associated human effort and compute.

Base modelSFT imagesHuman effort (h)Preparation (GPU-h)Training (GPU-h)
Pixel-Reasoner (SFT)2,7041.31.102.28
Visual CoT8,0001.10.484.65

GPU-h denotes hours of GPU use on a single NVIDIA RTX 4090.

Human effort. Two authors each repeated the workflow with AI assistance. We report their average active time, including data preparation, checking, and writing training scripts.

Preparation. We replay SFT trajectories through the frozen VLM to extract hidden states before evidence selection and pair them with visual features from the corresponding crops.

Training. The reported time covers all training of the verifier and grounder.

2. Adapting without the model's SFT data

Inspired by LLaVA (2023), we use GPT-6.1 Sol to generate PR trajectories from Visual CoT's questions and annotated regions. A few correct PR inference trajectories serve as in-context examples, so PR's original SFT data are not required. We manually check a sample to ensure that the generated visual requests match the annotated regions.

We then feed the generated trajectories into frozen PR and extract the hidden states before each evidence selection. These hidden states and the corresponding regions are used to train VerifyCoT as described in Part 1. This avoids annotating new regions and manually writing the trajectories.

Training source for PRTraining imagesHuman effort (h)Total API tokens (input / output)Preparation (GPU-h)Training (GPU-h)
Visual CoT annotations + generated trajectories2,0004.137.12M / 0.634M0.881.72

Human effort. Compared with using existing SFT data, the additional time is mainly spent preparing the generated trajectories and manually checking a sample for correctness.

See Additional details: Examples of generated trajectories at the end of this response.

3. Adapting without training new request interfaces (plug-and-play)

The principle of verification before consumption can also be implemented in a plug-and-play manner using an LLM-cleaned reference (here we use Qwen2.5-VL-7B). Verification uses cosine similarity between frozen SigLIP2 text and crop embeddings, without a learned binary head. Frozen Grounding DINO uses the same text to propose regions.

Comparison on V* Bench with Pixel-Reasoner (SFT).

MethodRequest sourceAccuracy (%)Time per question (s)
Native—77.494.9
VerifyCoTLatent (PR SFT data)83.255.4
Latent (Visual CoT data + generated trajectories)81.155.4
Plug-and-playLLM-cleaned reference80.108.4

Reusing annotations from another dataset achieves 81.15% accuracy at the same inference time as the original setup. The plug-and-play variant also improves over Native, offering an alternative when training new interfaces is impractical. It takes 8.4 s per question, compared with 5.4 s for latent VerifyCoT.

We also discuss a possible direction for further reducing the cost of using VerifyCoT in Additional details: Future directions.

Tsmc-Q2: Evidence verification and failure analysis

Gains shrink noticeably on RL-trained variants, and TreeBench slightly regresses under Visual CoT, indicating that verification cannot repair reasoning failures that occur after evidence consumption. Sensitivity to the verification threshold τ and the verifier's own error modes (the non-zero Harm column in Table 2) receive only thin analysis.

Response. Thank you for encouraging further analysis of these results. Below, we discuss Visual CoT's decline on TreeBench, evidence verification and subsequent reasoning, sensitivity to the verification threshold, and the sources of harmful changes.

1. Performance decline on TreeBench

Many TreeBench questions require complex reasoning across multiple steps to reach the correct answer (Wang et al., 2026). In our evaluated setup, Visual CoT selects a region once and then answers without generating explicit CoT. We believe this limits opportunities for further evidence selection and explicit reasoning, thus contributing to the performance decline on TreeBench. See Additional details for an example.

For Pixel-Reasoner, which supports multiple evidence steps and explicit reasoning, VerifyCoT improves TreeBench accuracy by 2.0 pp (SFT) and 5.4 pp (RL). These gains support the value of VerifyCoT even on tasks that require complex reasoning beyond localization.

2. Evidence verification and subsequent reasoning

Evidence selection errors occur across the evaluated base models and training approaches. Correcting these errors improves overall answer accuracy, even though correct evidence does not guarantee a correct final answer in every case.

The table below shows overall answer accuracy when visual evidence is replaced with GT or counterfactual evidence, or corrected by VerifyCoT. For each model and dataset, the best result is bold and the second best is underlined.

Base modelEvidenceV* Bench (%)ZoomBench (%)TreeBench (%)
Visual CoTNative52.435.530.5‾\underline{30.5}
GT evidence83.838.533.3
Counterfactual evidence47.127.830.0
VerifyCoT68.1‾\underline{68.1}36.8‾\underline{36.8}29.9
PRNative77.534.938.5
GT evidence84.353.540.7
Counterfactual evidence69.633.234.1
VerifyCoT83.2‾\underline{83.2}48.8‾\underline{48.8}40.5‾\underline{40.5}

The table shows that correct visual evidence improves overall performance, but even GT evidence does not guarantee correct subsequent reasoning or final answers. As discussed in lines 309–310, the base model can still make errors when interpreting and reasoning with correct evidence. For example, the model selects the correct vehicle region on its own but misidentifies its color (see Additional details).

VerifyCoT improves the accuracy of Pixel-Reasoner (RL) on all five evaluated benchmarks. The gain on TreeBench reaches 5.4 percentage points, compared with 2.0 points for the SFT model. These results show that VerifyCoT’s gains vary across benchmarks rather than consistently decreasing after RL training, supporting the value of matching intermediate visual evidence to the current reasoning step in RL-trained models.

BenchmarkPixel-Reasoner SFT gain (pp)Pixel-Reasoner RL gain (pp)
V* Bench+5.8+1.6
HR4K+2.5+2.1
HR8K+4.6+2.9
TreeBench+2.0+5.4
ZoomBench+13.9+0.2

Moreover, Table 4(b) shows that appending GT evidence after incorrect visual evidence has been consumed only partially recovers performance. This highlights the importance of checking evidence before use, so that incorrect visual information does not mislead subsequent reasoning.

3. Sensitivity to the verification threshold

Our threshold sweep shows that performance is relatively stable around the default setting, and all tested settings substantially improve over Native. Appendix D.2 reports the following results on Visual CoT/V* Bench. Each threshold is calibrated on SFT development pairs to meet a target acceptance rate for valid evidence, then fixed before test evaluation.

Dev. acceptance target (%)Accuracy (%)Gain over Native (pp)Fix / Harm (questions)Replacements
7065.97+13.6134 / 8108
8068.59+16.2334 / 397
85 (default)68.06+15.7133 / 387
9067.02+14.6630 / 277
9563.35+10.9922 / 158

For acceptance targets between 80% and 90%, accuracy ranges from 67.02% to 68.59%, showing that the gains do not depend on fine threshold tuning. Across all tested targets, accuracy remains 10.99–16.23 percentage points above Native. Lower targets trigger more replacements, while higher targets reduce harmful changes but also miss opportunities to correct errors.

Reviewer_Vzx3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies errors in intermediate visual evidence. The evidence selected by a model may not match its current request, yet the model may still use it in later reasoning. The authors replace intermediate image regions while keeping the preceding context and current request fixed. These experiments show that incorrect evidence can affect final answers, while correct regions can recover some lost accuracy.

The proposed method, VerifyCoT, uses hidden states from before the evidence-selection action to construct verification and grounding requests. It keeps the original evidence if verification succeeds. Otherwise, it calls Grounding DINO to generate alternatives and verifies the highest-scoring candidate. If this candidate also fails verification, the method falls back to the original evidence. The base VLM and external visual models remain frozen. Only the request interfaces and verification module are trained. Experiments cover the SFT and RL checkpoints of Pixel-Reasoner, as well as Visual CoT.

Critical strengths
  1. The evidence interventions directly support the motivation.

The authors examine 100 Visual CoT examples from V* Bench. Among the 67 examples with invalid original evidence, replacing it with ground-truth (GT) regions raises accuracy from 28.4% to 77.6%. Among the 33 examples with valid original evidence, injecting incorrect regions lowers accuracy from 90.9% to 69.7% (Table 3). VerifyCoT repairs 31 of the 67 invalid selections. All 33 examples with initially valid evidence still receive valid support. These results support studying evidence-selection errors directly.

  1. The role of selective verification is supported by meaningful controls.

Table 2 compares unconditional grounding, random gating, NLL gating, and VerifyCoT. These variants use the same grounding specialist and candidate-selection policy. On Visual CoT, VerifyCoT improves accuracy from 64.9% to 68.1% compared with unconditional grounding. It also reduces grounding calls per question from 1.00 to 0.46. Pixel-Reasoner shows a similar trend.

The random and NLL gates use matched calling budgets. These controls show that the gains cannot be explained simply by making more specialist calls. In addition, Appendix C.3 holds correction decisions fixed and confirms that the final answers are identical. This comparison supports the value of avoiding wasted continuation generation.

Critical weaknesses
  1. The accuracy advantage over post-hoc correction still combines differences in timing, supervision, and visual resources.

Appendix C.2 states that post-hoc self-correction and LookBack can only use the original image and text trajectory. They cannot reattach cropped images, call new tools, or use an external critic. VerifyCoT, however, uses interfaces trained with region-level supervision and separate visual specialists.

Table 4(a) therefore shows that VerifyCoT outperforms these baseline implementations. It does not isolate correction before evidence use as the sole source of the accuracy advantage.

  1. Evidence for broad applicability and consistent gains remains limited.

The end-to-end experiments mainly cover two 7B architectures that rely on region cropping or visual tools. VoCoT is evaluated only at the component level. The manual evidence audit covers 100 single-step examples from one benchmark. It includes only cases whose evidence validity could be reliably assessed and excludes uncertain cases. This audit does not establish how evidence errors are distributed across complex, multi-step reasoning trajectories.

Questions for authors
  1. Please report the numbers of images and evidence steps used for interface training, development, and testing for each base model. Do any of these data overlap with the five external benchmarks, including identical or near-duplicate images?

  2. In Table 5, do the text-based and hidden-state-based grounding requests use the same region-level supervision and comparable training budgets? Appendix D.1 states that the text-based verifiers are also trained, but the corresponding setup for the grounding branches remains unclear. Could you add a post-hoc correction baseline on one existing benchmark that uses the same verifier, grounder, and evidence budget? This would help separate the contribution of additional visual resources from that of correction timing.

  3. Could you provide a small-scale audit of existing Pixel-Reasoner test trajectories with multiple evidence steps? Please distinguish cases where the request itself is wrong from cases where the request is appropriate but the selected evidence is wrong. Please also report evidence repairs and harmful changes at each step.

5 comments

Thank you for your careful review and for recognizing the value of our evidence interventions and the controls supporting selective verification. Your main concerns are the training data and benchmark overlap, the supervision and visual resources used in our comparisons, and evidence errors in multi-step trajectories. We clarify these settings, add a comparison with the same evidence budget, and audit existing PR trajectories.

Vzx3-Q1: Data sizes and benchmark overlap

Please report the numbers of images and evidence steps used for interface training, development, and testing for each base model. Do any of these data overlap with the five external benchmarks, including identical or near-duplicate images?

Response. Thank you for raising this question. No confirmed overlaps remain between our training, development, and test data and the five evaluation benchmarks. We checked for overlap when we first introduced the benchmarks, before training the verifier and grounder.

We use exact and perceptual image hashes to compare the Visual CoT and PR SFT data with all five benchmarks, then manually check possible matches. The table shows the results before removing overlapping images.

BenchmarkVisual CoT: hashVisual CoT: manualPR: hashPR: manual
V* Bench001212
HR4K0000
HR8K0000
TreeBench0030
ZoomBench0030
Total001812

We manually inspect all 18 pairs. Six are false positives, while the remaining 12 are confirmed as near duplicates between PR SFT and V* Bench. We exclude these 12 images before training the verifier and grounder.

The table below gives the numbers of original images and evidence steps in the final training, development, and test sets.

Base modelSplitImagesEvidence steps
Visual CoTTraining6,4006,400
Development800800
Test800800
Pixel-ReasonerTraining2,1633,680
Development270446
Test271446

Vzx3-Q2: Grounding supervision and a post-hoc baseline with the same evidence budget

1. Grounding supervision and training budgets

In Table 5, do the text-based and hidden-state-based grounding requests use the same region-level supervision and comparable training budgets? Appendix D.1 states that the text-based verifiers are also trained, but the corresponding setup for the grounding branches remains unclear.

Response. Both branches use the same frozen Grounding DINO model. Text requests pass through its pretrained text encoder, with no additional training. To let Grounding DINO use the VLM's hidden states, we train only an interface (resampler and adaptor) that maps them into the same text embedding space. We first train this interface to match reference embeddings from Grounding DINO's text encoder, then train it with annotated target boxes using the bounding box loss.

We believe the pretrained model already works well. Our aim is to connect the VLM's current visual request to Grounding DINO's existing ability to locate the target region. We keep its pretrained components unchanged and do not add any extra modules on the visual side.

2. Post-hoc correction under the same evidence budget

Could you add a post-hoc correction baseline on one existing benchmark that uses the same verifier, grounder, and evidence budget? This would help separate the contribution of additional visual resources from that of correction timing.

Response. With the same verifier, grounder, and evidence budget, post-hoc correction improves accuracy from 77.49% to 79.58%, while VerifyCoT reaches 83.25%. This supports our “verify before consumption” principle: correction timing matters beyond the benefit of additional visual resources.

In Table 4(b), we compare the two settings using the same correct visual evidence (GT). On TreeBench, providing GT before use achieves 40.7% accuracy, while adding GT after the model has used an incorrect crop, even with explicit correction instructions, reaches only 37.5–38.0%. Thus, even GT evidence does not fully recover the performance achieved by correcting evidence before use.

Following your suggestion, we extend this comparison to V* Bench using the same verifier, grounder, and evidence budget as VerifyCoT. For visual evidence that VerifyCoT replaces, we provide the same replacement evidence after the model gives its initial answer. Inspired by CRITIC (Gou et al., 2024), we ask the model to recheck its visual conclusions and revise the reasoning that depends on them. The full prompt is provided in Additional details: Post-hoc correction prompt.

Comparison on V* Bench with Pixel-Reasoner (SFT).

MethodUse of visual evidenceAccuracy (%)
NativeNo extra expert crop77.49
Native + ReviewReview after the initial answer; no extra expert crop77.49
Post-hoc correctionSame evidence, after the initial answer79.58
VerifyCoTSame evidence, before use83.25

Post-hoc correction improves accuracy to 79.58%. This shows that providing correct visual evidence can help the model correct its mistakes and reach the right answer. VerifyCoT reaches 83.25% with the same visual evidence, about 3.7 percentage points higher than post-hoc correction. This supports our “verify before consumption” principle: correcting evidence before use offers benefits beyond providing additional visual evidence.

One possible reason is that adding correct evidence does not remove the incorrect crop or the reasoning already based on it. The model may therefore continue to follow its earlier reasoning when revising its answer, even with the new evidence. We share an interesting observation from our attention analysis related to this explanation in Additional details: Attention to earlier incorrect evidence.

Vzx3-Q3: Multi-step Pixel-Reasoner evidence audit

Could you provide a small-scale audit of existing Pixel-Reasoner test trajectories with multiple evidence steps? Please distinguish cases where the request itself is wrong from cases where the request is appropriate but the selected evidence is wrong. Please also report evidence repairs and harmful changes at each step.

Response. Thank you for suggesting this audit. In the 45 audited PR trajectories, evidence selection errors are much more common than request errors and occur at both early and later steps. On the same questions, VerifyCoT repairs 17 evidence errors and introduces 1, while answer accuracy increases from 40.0% to 46.7%.

1. Distribution of evidence errors across multi-step reasoning trajectories

We audit all PR baseline trajectories with multiple evidence steps on V* Bench and TreeBench.

Through our audit of these 45 trajectories, we group them into three categories: localization retries, rechecking or zooming in on the same target, and collecting or comparing evidence across multiple objects. These trajectories contain 101 evidence steps. At each step, we assess whether the request is appropriate for the question and preceding context, and whether the returned evidence meets the request.

Request and evidence errors by visual evidence step in the PR baseline.

Baseline stepRequest ✓ / Evidence ✓Request ✓ / Evidence ✗Request ✗ / Evidence ✓Request ✗ / Evidence ✗Total
19360045
28350245
≥3290011
Total198002101

Request ✓ means the request is appropriate; Evidence ✓ means the returned evidence meets the request. The ≥3 row combines all third and later visual evidence steps.

Evidence mismatches occur in 36 of 45 first steps, 37 of 45 second steps, and 9 of 11 third or later steps. Their presence at both the first and later stages supports verifying each new piece of visual evidence throughout multi-step reasoning.

2. Request errors versus evidence selection errors

Only 2 of the 101 requests are inappropriate. In 80 steps, the request is appropriate, but the returned evidence does not meet it. Most errors in this audit therefore arise from evidence selection: the model requests relevant information but fails to obtain it. This suggests that the model can often use its reasoning to identify what it needs to inspect, while its ability to locate the correct region remains a bottleneck. This leaves substantial room to improve the evidence available to the model by verifying and correcting it before consumption.

3. Evidence repairs and harmful changes with VerifyCoT

We then audit VerifyCoT's actual replacement events on the same 45 questions. At each event, we compare the original and replacement crops against the current request. Step numbers refer to VerifyCoT's own trajectories, which may differ in length and content from the baseline trajectories.

Evidence repairs and harmful replacements in VerifyCoT on the same 45 questions.

Step in VerifyCoTTrajectories reaching this stepReplacementsEvidence ✗ → ✓Evidence ✓ → ✗Evidence ✗ → ✗Evidence ✓ → ✓
1452212046
226123135
3111000
4111000
Total73 steps36171711

Across 36 replacements, VerifyCoT corrects the evidence in 17 cases and changes correct evidence to incorrect evidence in 1 case. Repairs occur at Step 1 and later steps. Across the same 45 questions, the total number of evidence steps decreases from 101 to 73, and 19 trajectories use only 1 evidence step.

Answer accuracy increases from 40.0% to 46.7%, with 3 answers changing from incorrect to correct and 0 changing from correct to incorrect. We manually examine the 3 improved cases below.

Cases with improved final answers.

CaseAnswer changeIntervention stepEvidence changeBaseline total stepsVerifyCoT total steps
Tree-0108✗ → ✓1✗ → ✓41
Tree-0177✗ → ✓1✗ → ✓41
Tree-0319✗ → ✓1✗ → ✓21

In all 3 cases, VerifyCoT replaces incorrect evidence with correct evidence at Step 1. The model then gives the correct answer with only 1 evidence step, compared with 2 or 4 steps in the baseline. This is consistent with Table 3(b): correcting visual evidence can help the model reach the correct answer. These cases also suggest that providing correct evidence early can reduce the need for further evidence requests.

We also share valuable observations and possible directions for future work in Additional details: Observations from trajectory inspection.

Reviewer_RBBaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies evidence-selection errors in interleaved VLMs. Through controlled interventions, the authors show that replacing the model's selected evidence with oracle evidence improves accuracy, while counterfactual evidence degrades it. They further show that appending correct evidence after an erroneous crop has been consumed only recovers partially. Motivated by these findings, they propose VerifyCoT, which verifies selected evidence before the model consumes it, and corrects if needed. The method is evaluated on Pixel-Reasoner and Visual CoT across five benchmarks. The benchmarking results demonstrate the method effectiveness.

Critical strengths
  1. Clear motivation and presentation. The paper is well motivated. Controlled interventions that inject oracle or counterfactual evidence show that selection errors affect final answers, and they also show that appending correct evidence after consumption recovers only partially. This gives the "verify before consumption" a concrete motivation.
  2. Extensive evaluation. The method is evaluated on five benchmarks across different models. The evaluation covers answer quality and efficiency.
Critical weaknesses
  1. Applicability and annotation cost. Both evaluated models reuse step-level target boxes that already exist in their SFT data. Extending the method to other models would require collecting trajectories, annotating step-level targets, and training additional components for each model. The paper's practical relevance would be strengthened by quantifying this cost or testing a cheaper option.
  2. Ablations on the pipeline complexity. The method relies on several trained components and design choices. Apart from latent vs. text requests, it would be valuable to assess individual contribution, and adding simple alternatives (an MLP probe on H_t or a question-conditioned GDINO) would help justify the added complexity.
Questions for authors
  1. When the verifier rejects a crop, how does VerifyCOT compare with simply replacing it with the full image, or, for Pixel-Reasoner, returning the full image and letting the model re-crop by itself. This would separate the gain of verifier only from the gain of supplying correct evidence.
6 comments

Thank you for your careful review and for recognizing the motivation and evaluation of our work. Your main concerns are adaptation cost, component complexity, and whether returning the full image is sufficient for recovery. We report adaptation costs and evaluate simpler request interfaces, question-conditioned grounding, and full-image alternatives.

RBBa-Q1: Applicability and annotation cost

Applicability and annotation cost. Both evaluated models reuse step-level target boxes that already exist in their SFT data. Extending the method to other models would require collecting trajectories, annotating step-level targets, and training additional components for each model. The paper's practical relevance would be strengthened by quantifying this cost or testing a cheaper option.

Response. Our measurements show that VerifyCoT can be adapted at low cost, even without the target model's original SFT data. When training new interfaces is impractical, an LLM-cleaned reference provides a plug-and-play alternative.

1. Adapting with the model's SFT data

For Pixel-Reasoner and Visual CoT, we reuse the trajectories and target-region annotations in their SFT data, so no additional manual region annotation is required in this setting. Adaptation consists of preparing features and training the request interfaces and verifier; the base VLM, semantic encoder, and grounding backbone remain frozen.

Base modelSFT imagesHuman effort (h)Preparation (GPU-h)Training (GPU-h)
Pixel-Reasoner (SFT)2,7041.31.102.28
Visual CoT8,0001.10.484.65

GPU-h denotes hours of GPU use on a single NVIDIA RTX 4090.

Human effort. Two authors each repeat the workflow with AI assistance. We report their average active work time, including data preparation, data checks, and writing training scripts.

Preparation. We replay SFT trajectories through the frozen VLM to extract hidden states before evidence selection and pair them with visual features from the corresponding crops.

Training. The reported time covers all training of the verifier and grounder.

2. Adapting without the model's SFT data

Inspired by LLaVA (2023), we use GPT-6.1 Sol to generate PR trajectories from Visual CoT's questions and annotated regions. A few correct PR inference trajectories serve as in-context examples, so PR's original SFT data are not required. We manually check a sample to ensure that the generated visual requests match the annotated regions.

We then feed the generated trajectories into frozen PR and extract the hidden states before each evidence selection. These hidden states and the corresponding regions are used to train VerifyCoT as described in Part 1. This avoids annotating new regions and manually writing the trajectories.

Training source for PRTraining imagesHuman effort (h)Total API tokens (input / output)Preparation (GPU-h)Training (GPU-h)
Visual CoT annotations + generated trajectories2,0004.137.12M / 0.634M0.881.72

Human effort. Compared with using existing SFT data, the additional time is mainly spent preparing the generated trajectories and manually checking a sample for correctness.

See Additional details: Qualitative examples at the end of this response.

3. Adapting without training new request interfaces

The principle of verification before consumption can also be implemented with lower adaptation cost using an LLM-cleaned reference (here we use Qwen2.5-VL-7B). Verification uses cosine similarity between frozen SigLIP2 text and crop embeddings, without a learned binary head. Frozen Grounding DINO uses the same text to propose regions.

Comparison on VSTAR with Pixel-Reasoner (SFT).

MethodRequest sourceAccuracy (%)Time per question (s)
Native—77.494.9
VerifyCoTLatent (PR SFT data)83.255.4
Latent (Visual CoT data + generated trajectories)81.155.4
Text requestsLLM-cleaned reference80.108.4

Reusing annotations from another dataset achieves 81.15% accuracy at the same inference time as the original setup. The LLM-cleaned reference also improves over Native, offering an alternative when training new interfaces is impractical. It takes 8.4 s per question, compared with 5.4 s for latent VerifyCoT.

We also discuss a possible direction for further reducing the cost of using VerifyCoT in Additional details: Future directions.

RBBa-Q2: Component ablations and simpler alternatives

Ablations on the pipeline complexity. The method relies on several trained components and design choices. Apart from latent vs. text requests, it would be valuable to assess individual contribution, and adding simple alternatives (an MLP probe on H_t or a question-conditioned GDINO) would help justify the added complexity.

Response. Our interface design achieves higher accuracy with fewer parameters than the MLP. Question-conditioned GDINO improves over Native, while using the current visual request further improves accuracy, especially on questions requiring more complex reasoning, such as spatial relations. Making VerifyCoT simpler and more effective is also a goal we share.

Existing ablations. Table 2 examines verification through comparisons with always grounding and with Random and NLL gates at matched calling budgets. Table 5(c) shows that latent requests improve verification and grounding separately, with the highest QA accuracy when both use latent requests. These experiments support the effectiveness of our verifier and grounder design. We further evaluate the MLP request interface and question-conditioned GDINO below.

1. MLP performance

We initially considered an MLP, but chose the resampler and adaptor for two reasons: information extraction and parameter efficiency. CoT varies in wording and structure, so the resampler uses learned queries and attention to extract the current visual request across token positions. The adaptor then maps it into the representation required by the verifier or grounder. Directly applying an MLP to concatenated hidden states requires many parameters because of the large input dimension.

MLP settings. We use MLPs with two linear layers and GELU activation on the same L = 16 concatenated hidden states.

Parameter counts and accuracy on V* Bench with Pixel-Reasoner (SFT).

Request interfaceMLP hidden widthTotal interface parametersAccuracy (%)
MLP647.46M75.39
MLP12814.91M—
MLP25629.82M—
MLP51259.64M—
Resampler + adaptor—6.54M83.25

We compare our design with an MLP of hidden width 64, which has a comparable parameter count. Compressing the concatenated hidden states from 57,344 dimensions to 64 may lose useful information and help explain the MLP's lower accuracy. These results suggest that the resampler and adaptor extract the current visual request more effectively. Our design achieves higher accuracy with fewer interface parameters.

2. Question-conditioned GDINO performance

We use the original question as the text input to Grounding DINO and evaluate on V* Bench. We report overall accuracy and separate results for spatial relation and direct attribute questions.

MethodOverallSpatial relationsDirect attributes
Native77.4972.3780.87
Verification + question-conditioned GDINO80.1073.6884.35
VerifyCoT83.2580.2685.22

Question-conditioned GDINO improves accuracy from 77.49% to 80.10%, while VerifyCoT reaches 83.25%. The original question already provides useful guidance for direct attribute questions, but brings a smaller improvement on spatial relations. Using the current grounding request provides a further gain of 6.58 percentage points on spatial relations, compared with 0.87 points on direct attributes.

The original question describes the overall task, while the current reasoning helps specify what the model needs to inspect. For example, when the model needs to locate the car, the question “What is the spatial relationship between the street lamp and the car?” mentions both objects without specifying which one is the current grounding target. A question can contain several objects and relations, while reasoning helps narrow down the region needed at the current step. Using this more specific request to guide grounding helps explain the larger improvement on spatial relations. We therefore expect requests that identify the target at each step to be more effective than a fixed question on tasks requiring multiple reasoning steps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.