- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This submission asks whether interleaved vision-language models can rely on the visual evidence they pick up in the middle of reasoning. Controlled interventions on the selected evidence show that swapping in ground-truth regions yields large accuracy gains, whereas counterfactual evidence hurts; fixing mistakes after consumption is unsatisfying in both directions — appending the correct region recovers accuracy only partially, while rollback throws away reasoning already produced. These findings motivate a "verify before you consume" principle, realized as VerifyCoT.
Critical strengths
- The causal analysis is the core asset. The oracle/counterfactual intervention design supplies direct, quantitative evidence that selection mistakes drive downstream errors, so the central verify-first principle is empirically established.
- Thoughtful engineering and broad evaluation. Deriving requests from hidden states sidesteps fragile, model-specific textual-reference extraction, and the verify-then-selectively-correct pipeline avoids wasteful grounder calls. Results span multiple architectures and five benchmarks with consistent, often sizable, improvements.
Critical weaknesses
- The framework needs annotated target regions to train its request interfaces and verifier — it is not training-free. Portability to new backbones or tasks lacking such annotations is therefore limited, and the deployment cost relative to plug-and-play alternatives is under-discussed.
- Gains shrink noticeably on RL-trained variants, and TreeBench slightly regresses under Visual CoT, indicating that verification cannot repair reasoning failures that occur after evidence consumption. Sensitivity to the verification threshold τ and the verifier's own error modes (the non-zero Harm column in Table 2) receive only thin analysis.
Questions for authors
Need more analysis around the critical Weaknesses.
7 comments
3. Adapting without training new request interfaces (plug-and-play)
The principle of verification before consumption can also be implemented in a plug-and-play manner using an LLM-cleaned reference (here we use Qwen2.5-VL-7B). Verification uses cosine similarity between frozen SigLIP2 text and crop embeddings, without a learned binary head. Frozen Grounding DINO uses the same text to propose regions.
Comparison on V* Bench with Pixel-Reasoner (SFT).
| Method | Request source | Accuracy (%) | Time per question (s) |
|---|---|---|---|
| Native | — | 77.49 | 4.9 |
| VerifyCoT | Latent (PR SFT data) | 83.25 | 5.4 |
| Latent (Visual CoT data + generated trajectories) | 81.15 | 5.4 | |
| Plug-and-play | LLM-cleaned reference | 80.10 | 8.4 |
Reusing annotations from another dataset achieves 81.15% accuracy at the same inference time as the original setup. The plug-and-play variant also improves over Native, offering an alternative when training new interfaces is impractical. It takes 8.4 s per question, compared with 5.4 s for latent VerifyCoT.
We also discuss a possible direction for further reducing the cost of using VerifyCoT in Additional details: Future directions.
Tsmc-Q2: Evidence verification and failure analysis
Gains shrink noticeably on RL-trained variants, and TreeBench slightly regresses under Visual CoT, indicating that verification cannot repair reasoning failures that occur after evidence consumption. Sensitivity to the verification threshold τ and the verifier's own error modes (the non-zero Harm column in Table 2) receive only thin analysis.
Response. Thank you for encouraging further analysis of these results. Below, we discuss Visual CoT's decline on TreeBench, evidence verification and subsequent reasoning, sensitivity to the verification threshold, and the sources of harmful changes.
1. Performance decline on TreeBench
Many TreeBench questions require complex reasoning across multiple steps to reach the correct answer (Wang et al., 2026). In our evaluated setup, Visual CoT selects a region once and then answers without generating explicit CoT. We believe this limits opportunities for further evidence selection and explicit reasoning, thus contributing to the performance decline on TreeBench. See Additional details for an example.
For Pixel-Reasoner, which supports multiple evidence steps and explicit reasoning, VerifyCoT improves TreeBench accuracy by 2.0 pp (SFT) and 5.4 pp (RL). These gains support the value of VerifyCoT even on tasks that require complex reasoning beyond localization.
2. Evidence verification and subsequent reasoning
Evidence selection errors occur across the evaluated base models and training approaches. Correcting these errors improves overall answer accuracy, even though correct evidence does not guarantee a correct final answer in every case.
The table below shows overall answer accuracy when visual evidence is replaced with GT or counterfactual evidence, or corrected by VerifyCoT. For each model and dataset, the best result is bold and the second best is underlined.
| Base model | Evidence | V* Bench (%) | ZoomBench (%) | TreeBench (%) |
|---|---|---|---|---|
| Visual CoT | Native | 52.4 | 35.5 | |
| GT evidence | 83.8 | 38.5 | 33.3 | |
| Counterfactual evidence | 47.1 | 27.8 | 30.0 | |
| VerifyCoT | 29.9 | |||
| PR | Native | 77.5 | 34.9 | 38.5 |
| GT evidence | 84.3 | 53.5 | 40.7 | |
| Counterfactual evidence | 69.6 | 33.2 | 34.1 | |
| VerifyCoT |
The table shows that correct visual evidence improves overall performance, but even GT evidence does not guarantee correct subsequent reasoning or final answers. As discussed in lines 309–310, the base model can still make errors when interpreting and reasoning with correct evidence. For example, the model selects the correct vehicle region on its own but misidentifies its color (see Additional details).
VerifyCoT improves the accuracy of Pixel-Reasoner (RL) on all five evaluated benchmarks. The gain on TreeBench reaches 5.4 percentage points, compared with 2.0 points for the SFT model. These results show that VerifyCoT’s gains vary across benchmarks rather than consistently decreasing after RL training, supporting the value of matching intermediate visual evidence to the current reasoning step in RL-trained models.
| Benchmark | Pixel-Reasoner SFT gain (pp) | Pixel-Reasoner RL gain (pp) |
|---|---|---|
| V* Bench | +5.8 | +1.6 |
| HR4K | +2.5 | +2.1 |
| HR8K | +4.6 | +2.9 |
| TreeBench | +2.0 | +5.4 |
| ZoomBench | +13.9 | +0.2 |
Moreover, Table 4(b) shows that appending GT evidence after incorrect visual evidence has been consumed only partially recovers performance. This highlights the importance of checking evidence before use, so that incorrect visual information does not mislead subsequent reasoning.
3. Sensitivity to the verification threshold
Our threshold sweep shows that performance is relatively stable around the default setting, and all tested settings substantially improve over Native. Appendix D.2 reports the following results on Visual CoT/V* Bench. Each threshold is calibrated on SFT development pairs to meet a target acceptance rate for valid evidence, then fixed before test evaluation.
| Dev. acceptance target (%) | Accuracy (%) | Gain over Native (pp) | Fix / Harm (questions) | Replacements |
|---|---|---|---|---|
| 70 | 65.97 | +13.61 | 34 / 8 | 108 |
| 80 | 68.59 | +16.23 | 34 / 3 | 97 |
| 85 (default) | 68.06 | +15.71 | 33 / 3 | 87 |
| 90 | 67.02 | +14.66 | 30 / 2 | 77 |
| 95 | 63.35 | +10.99 | 22 / 1 | 58 |
For acceptance targets between 80% and 90%, accuracy ranges from 67.02% to 68.59%, showing that the gains do not depend on fine threshold tuning. Across all tested targets, accuracy remains 10.99–16.23 percentage points above Native. Lower targets trigger more replacements, while higher targets reduce harmful changes but also miss opportunities to correct errors.
Thank you for the thoughtful review and for recognizing the motivation behind our work. Your main concerns are adaptation cost and portability, performance after RL training and on TreeBench, whether model performance is sensitive to the verification threshold, and the main sources of harmful changes. We report adaptation costs and cheaper alternatives, analyze the results after RL training and on TreeBench, examine threshold sensitivity, and identify the sources of harmful changes through manual inspection.
Tsmc-Q1: Adaptation and deployment costs
Response. VerifyCoT is not training-free, but its adaptation costs are modest. With available SFT data, we train the request interfaces and verifier directly. Even without the model’s original SFT data, we can adapt VerifyCoT using annotations from another dataset and generated trajectories. Even this training can be skipped by using LLM-cleaned references as a plug-and-play alternative, which enables VerifyCoT adaptation with almost no additional training cost.
1. Adapting with the model's SFT data
For Pixel-Reasoner and Visual CoT, we reuse the trajectories and target-region annotations in their SFT data, so no additional manual region annotation is required in this setting. Adaptation consists of preparing features and training the request interfaces and verifier. The base VLM, semantic encoder, and grounding backbone remain frozen. The table reports the total SFT images processed and the associated human effort and compute.
GPU-h denotes hours of GPU use on a single NVIDIA RTX 4090.
Human effort. Two authors each repeated the workflow with AI assistance. We report their average active time, including data preparation, checking, and writing training scripts.
Preparation. We replay SFT trajectories through the frozen VLM to extract hidden states before evidence selection and pair them with visual features from the corresponding crops.
Training. The reported time covers all training of the verifier and grounder.
2. Adapting without the model's SFT data
Inspired by LLaVA (2023), we use GPT-6.1 Sol to generate PR trajectories from Visual CoT's questions and annotated regions. A few correct PR inference trajectories serve as in-context examples, so PR's original SFT data are not required. We manually check a sample to ensure that the generated visual requests match the annotated regions.
We then feed the generated trajectories into frozen PR and extract the hidden states before each evidence selection. These hidden states and the corresponding regions are used to train VerifyCoT as described in Part 1. This avoids annotating new regions and manually writing the trajectories.
Human effort. Compared with using existing SFT data, the additional time is mainly spent preparing the generated trajectories and manually checking a sample for correctness.
See Additional details: Examples of generated trajectories at the end of this response.