- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To address the issue in vision-language models where tools are invoked without obtaining or utilizing valid visual evidence, this paper proposes FaithEyes. This method employs a shared-parameter sub-agent to assess the utility of images generated by tools, feeding the assessment and rationale back to the main agent while filtering out useless images and adjusting reinforcement learning rewards based on the ratio of valid tool invocations. Trained using SFT and GRPO and evaluated across two model backbones and six visual benchmarks, the method demonstrates enhanced high-resolution perception and improved relevance of intermediate images, alongside a reduction in inference token overhead.
Critical strengths
- The problem is of practical significance, and the methodological design is coherent. Tool-use assessment is applied to both inference-time feedback and reward optimization, addressing both the training and inference phases; the use of a shared-parameter sub-agent eliminates the need for an external judge model during inference, offering practical value.
- The improvements in tool relevance and efficiency are convincing. Even when retaining images deemed useless, the 7B model achieves a tool fidelity score of 53.1% on HR-Bench 8K—significantly higher than CodeV’s 12.1%—demonstrating that the gains do not stem solely from filtering. Token overhead on this benchmark is reduced by 54.1%, a figure that accounts for the sub-agent's own overhead (Figure 2, Table 3).
- The ablation study is comprehensive. The authors examine judgment feedback, reward scaling, reward coefficients, and training stages; Table 4 demonstrates the complementary nature of the two core components. Additionally, the appendix analyzes the effects of decoupling tool rewards from answer correctness.
Critical weaknesses
- Core metrics are insufficient to prove that the model genuinely relies on tool-generated evidence. Current "faithfulness" measures primarily whether the correct reasoning trajectory contains at least one process image depicting the target object, thereby reflecting only the relevance of visual evidence. The absence of intervention experiments—such as removing, replacing, or perturbing process images—makes it impossible to rule out the possibility that the model still relies mainly on the original image or text to generate answers, even after retrieving relevant process images. The attention analysis in Appendix B also explicitly acknowledges that it provides evidence of correlation only.
- The judge model and supervision labels lack independent quantitative validation. SFT labels are determined based on trajectory structures, requiring the large model to generate rationales that align with these predefined labels; furthermore, the final evaluation employs the same Qwen3-VL-235B model used to construct the data. Although the authors mention manual spot-checks, they report no details regarding sample size, consistency rates, or error rates; simply varying the prompts is insufficient to rule out shared judgment biases.
- The source of gains from the feedback mechanism remains unclear. Reasoning rationales may directly contain problem-solving information—such as color or spatial location—implying that performance gains could stem from the additional visual descriptions provided by sub-agents rather than the feedback mechanism itself. Existing ablation studies fail to adequately disentangle the respective contributions of binary usefulness judgments, answer-related information within rationales, and image filtering.
- Gains in complex reasoning tasks are inconsistent. On the MathVision benchmark, the 8B model’s performance dropped from a baseline of 53.9 to 46.1, while the 7B version scored 29.9—falling short of CodeV’s 33.6. Improved tool localization has not yet consistently translated into enhanced capabilities for complex visual reasoning.
Questions for authors
See Weaknesses.
4 comments
We sincerely thank you for all your valuable comments. Below are our responses to each weakness and question.
Weakness 2: quantitative validation of supervision labels and risk of judgment bias
Thanks for this important question. We provide quantitative validation of supervision labels and the faithfulness ratios judged by models from different families below.
Firstly, the helpful/unhelpful labels are derived from trajectory structure in the publicly released Thyme SFT data and we generate rationales using Qwen3-VL-235B-A22B. Since grounding-style assessment is relatively easy, two authors independently checked 300 randomly sampled examples against the image, assessing whether the crop contains the target, whether the rationale is supported by the image and label, and for two-call trajectories, whether the relative focus of the crops supports the trajectory-derived label. We report inter-annotator agreement and human verification accuracy that is the proportion of examples for which both authors agreed with the trajectory-derived label and its rationale. We also use DeepSeek-V4.1-Flash-1017 to check the label and rationale of 10,000 samples. The results are shown below:
| Metric | Inter-annotator agreement | Human verification, N=300 | DeepSeek-V4.1-Flash-1017, N=10000 |
|---|---|---|---|
| Value | 95.3% | 93.0% | 94.2% |
The residual disagreements are concentrated in borderline cases (e.g., crops that partially clip the target).
Secondly, to assess whether the reported ranking depends on Qwen3-VL-235B-A22B as the evaluation judge, we re-evaluate the tool faithfulness ratio with DeepSeek-V4.1-Flash-1017 using the same judging prompts. This model is from a different family, with no shared weights or training pipeline. The results are shown below:
| Tool Faithfulness Ratio | V* | HR-Bench 4K | HR-Bench 8K |
|---|---|---|---|
| DeepEyes | 76.1 | 27.8 | 5.6 |
| Thyme | 80.7 | 34.1 | 4.5 |
| CodeV | 92.5 | 52.3 | 10.7 |
| FaithEyes (keep unhelpful) | 94.0 | 80.1 | 55.9 |
| FaithEyes | 94.3 | 87.2 | 79.5 |
The relative pattern is similar to that obtained with Qwen3-VL-235B-A22B, and FaithEyes has the highest reported ratio across all three benchmarks under this second judge.
Weakness 3: answer information from rationale
Thanks for this insightful question. We test whether rationale text contributes to accuracy by removing it while retaining the binary verdict and image-filtering behavior. The results below suggest that rationale text is not a major source of the observed accuracy gains.
Firstly, the subagent sees only the question and the local crop, not the main agent's chain-of-thought or code. For a helpful verdict, the process image is returned together with the rationale, so any attribute it mentions is simultaneously visible in the image itself, although expressing it in text is easier to use. For an unhelpful verdict, the crop is discarded and the rationale explains the rejection, such as "the queried target is not visible".
Secondly, we conduct a new ablation that removes all rationale text from the tool observations while keeping the binary verdict and image-filtering behavior unchanged during evaluation. As shown below, the accuracy differences are small across all six benchmarks. This provides evidence that rationale text has little effect on accuracy.
| V* | HR-Bench 4K | HR-Bench 8K | MathVista | MathVerse | MathVision | |
|---|---|---|---|---|---|---|
| FaithEyes-7B | 87.4 | 77.8 | 72.9 | 73.1 | 51.0 | 29.9 |
| w/o Rationale | 87.5 | 77.1 | 72.5 | 73.0 | 50.4 | 30.2 |
Thirdly, we further measure what fraction of rationales actually contain explicit answer fragments, using DeepSeek-V4.1-Flash-1017 as the judge model. The ratios on all six benchmarks are:
| V* | HR-Bench 4K | HR-Bench 8K | MathVista | MathVerse | MathVision | |
|---|---|---|---|---|---|---|
| Ratio (%) | 65.30 | 50.80 | 17.20 | 29.00 | 18.00 | 3.30 |
The high ratios on perception benchmarks are plausible because judging whether a crop helps answer a question about an attribute such as color may involve mentioning that attribute, while the main agent can inspect the same crop directly. The small accuracy changes after rationale removal, even on V* with the highest answer-related ratio, suggest rationale text is not the main source of the gains.
We sincerely thank you for all your valuable comments. Below are our responses to each weakness and question.
Weakness 4: weak results on MathVision
Thanks for raising this concern, and we will explain it below.
(1) On MathVision, FaithEyes-7B improves accuracy from 21.4 to 29.9 and outperforms four of the five same-scale agentic baselines, ranking second to CodeV. However, FaithEyes outperforms CodeV on MathVista and MathVerse, so their relative performance varies across reasoning benchmarks.
(2) For Qwen3-VL-8B, Table 8 of paper shows the progression “base 53.9 → SFT 40.8 → RL 46.1” on MathVision. The largest decrease occurs during SFT, whereas RL effectively recovers performance by 5.3 points. One possible reason is an imbalance in mathematical supervision. Our SFT data are adapted from Thyme, whose authors explicitly report relatively limited mathematical-computation data compared with image-manipulation data. Fine-tuning Qwen3-VL-8B, which is already exceptionally strong at mathematics, on such trajectories shifts its output distribution.
We have incorporated the corresponding clarifications and additional results into the revised manuscript, with revisions marked in red.
The author’s response addressed my concerns; however, taking everything into account, I still consider this paper to be at a borderline level (e.g., a score of 2.5).
We sincerely thank you for all your valuable comments. Below are our responses to each weakness and question.
Weakness 1: evidence for dependence on process images
We agree that the faithfulness ratio alone cannot establish dependence on process images. We therefore conduct intervention experiments following the protocol of [1]. For each example, we truncate the recorded trajectory immediately before the final answer-producing assistant turn, apply image interventions to the retained context, and regenerate the final answer. We remove or randomize the process images, the original image, or both. For randomization, we replace all pixel values with Gaussian noise. Results for the data-comparable agentic baselines and FaithEyes on V*, HR-Bench 4K and HR-Bench 8K are shown below.
"Full" denotes the regenerated accuracy without image intervention. The "Remove/Random Orig&Process" results reflect what the model can achieve by relying solely on common sense and the textual content in the trajectory. From the results, we can observe:
(1) Under process-image interventions, FaithEyes shows the largest accuracy degradation among all compared models for both removal and randomization. Admittedly, the absolute drop is moderate (up to 4.6 points), but this does not mean the process images are ignored or meaningless when provided. The model could fall back on the original image and other context when process images are unavailable. Randomization hurts more than removal for FaithEyes on all three benchmarks, consistent with the model using the process-image content. In contrast, DeepEyes and Thyme are insensitive to process-image interventions, which is consistent with their low faithfulness ratios.
(2) When the original image is removed, FaithEyes retains the highest accuracy and exhibits the smallest accuracy drop on all three benchmarks. Comparing "Remove Orig" with "Remove Orig&Process" further shows that process images provide an additional +10.5/+5.9/+6.4 points of accuracy given the retained trajectory context. These results support an incremental contribution of process images during final-turn generation.
(3) As shown in the attention analysis (Table 6 of paper), the answer tokens of FaithEyes attend to process images, although less than to the original image, consistent with the text- and initial-image-centric attention bias of VLMs. Injecting the subagent judgement further increases attention to helpful images and decreases it for unhelpful ones. While attention itself is only a correlational proxy, the attention analysis and the causal intervention results above jointly indicate that the process images are genuinely consulted during answer generation.
Overall, the intervention results provide evidence that FaithEyes' predictions are sensitive to the availability and content of process images, and these images can contribute to answering even when the original image is unavailable. Although the original image remains an important source of evidence, the fine-grained process images from tool calls can yield additional gains.
[1] On the Faithfulness of Visual Thinking: Measurement and Enhancement. Liu et al. 2025.