acceptodds
Withdrawn from ICLR 2027

FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Self-Verification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic vision-language models (VLMs), which interleave textual reasoning with explicit tool calls such as cropping and code-based image manipulation, have emerged as a compelling paradigm for reliable and interpretable multi-modal reasoning. However, recent studies have revealed that such models often use tools unfaithfully. Many process images are irrelevant to the question (e.g., the crops miss the queried target), yet the tool call still receives full credit and the model still answers correctly. Such decorative or misaligned tool calls waste computation and reveal that the model does not faithfully use the evidence it retrieves. This may stem from two limitations of prevailing methods: the tool reward fails to distinguish useful from useless calls, and tool feedback carries no signal of usefulness. To this end, we introduce FaithEyes, a multi-agent self-judging framework. Concretely, we use a VLM to judge whether each process image helps answer the question. The judgement is injected into the reasoning context as part of the tool observation to help subsequent reasoning, and meanwhile is used to scale the tool reward by the helpful-tool ratio to suppress reward hacking. To keep judgement available at evaluation, we further design a multi-agent framework where the model itself serves as a subagent to judge the tool calls from the main agent, eliminating any dependence on external models at inference. Training via a two-stage SFT + RL pipeline on adapted open-source data, FaithEyes attains competitive or superior accuracy across visual perception and reasoning benchmarks, while substantially improving tool faithfulness and reducing inference cost.

Primary area:
reinforcement learning
Keywords:
Agentic VLMs, Multi-Agent, Reinforcement Learning, Faithful Tool Use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UCZeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

To address the issue in vision-language models where tools are invoked without obtaining or utilizing valid visual evidence, this paper proposes FaithEyes. This method employs a shared-parameter sub-agent to assess the utility of images generated by tools, feeding the assessment and rationale back to the main agent while filtering out useless images and adjusting reinforcement learning rewards based on the ratio of valid tool invocations. Trained using SFT and GRPO and evaluated across two model backbones and six visual benchmarks, the method demonstrates enhanced high-resolution perception and improved relevance of intermediate images, alongside a reduction in inference token overhead.

Critical strengths
  1. The problem is of practical significance, and the methodological design is coherent. Tool-use assessment is applied to both inference-time feedback and reward optimization, addressing both the training and inference phases; the use of a shared-parameter sub-agent eliminates the need for an external judge model during inference, offering practical value.
  2. The improvements in tool relevance and efficiency are convincing. Even when retaining images deemed useless, the 7B model achieves a tool fidelity score of 53.1% on HR-Bench 8K—significantly higher than CodeV’s 12.1%—demonstrating that the gains do not stem solely from filtering. Token overhead on this benchmark is reduced by 54.1%, a figure that accounts for the sub-agent's own overhead (Figure 2, Table 3).
  3. The ablation study is comprehensive. The authors examine judgment feedback, reward scaling, reward coefficients, and training stages; Table 4 demonstrates the complementary nature of the two core components. Additionally, the appendix analyzes the effects of decoupling tool rewards from answer correctness.
Critical weaknesses
  1. Core metrics are insufficient to prove that the model genuinely relies on tool-generated evidence. Current "faithfulness" measures primarily whether the correct reasoning trajectory contains at least one process image depicting the target object, thereby reflecting only the relevance of visual evidence. The absence of intervention experiments—such as removing, replacing, or perturbing process images—makes it impossible to rule out the possibility that the model still relies mainly on the original image or text to generate answers, even after retrieving relevant process images. The attention analysis in Appendix B also explicitly acknowledges that it provides evidence of correlation only.
  2. The judge model and supervision labels lack independent quantitative validation. SFT labels are determined based on trajectory structures, requiring the large model to generate rationales that align with these predefined labels; furthermore, the final evaluation employs the same Qwen3-VL-235B model used to construct the data. Although the authors mention manual spot-checks, they report no details regarding sample size, consistency rates, or error rates; simply varying the prompts is insufficient to rule out shared judgment biases.
  3. The source of gains from the feedback mechanism remains unclear. Reasoning rationales may directly contain problem-solving information—such as color or spatial location—implying that performance gains could stem from the additional visual descriptions provided by sub-agents rather than the feedback mechanism itself. Existing ablation studies fail to adequately disentangle the respective contributions of binary usefulness judgments, answer-related information within rationales, and image filtering.
  4. Gains in complex reasoning tasks are inconsistent. On the MathVision benchmark, the 8B model’s performance dropped from a baseline of 53.9 to 46.1, while the 7B version scored 29.9—falling short of CodeV’s 33.6. Improved tool localization has not yet consistently translated into enhanced capabilities for complex visual reasoning.
Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

We sincerely thank you for all your valuable comments. Below are our responses to each weakness and question.

Weakness 1: evidence for dependence on process images

We agree that the faithfulness ratio alone cannot establish dependence on process images. We therefore conduct intervention experiments following the protocol of [1]. For each example, we truncate the recorded trajectory immediately before the final answer-producing assistant turn, apply image interventions to the retained context, and regenerate the final answer. We remove or randomize the process images, the original image, or both. For randomization, we replace all pixel values with Gaussian noise. Results for the data-comparable agentic baselines and FaithEyes on V*, HR-Bench 4K and HR-Bench 8K are shown below.

V*FullRemove ProcessRandom ProcessRemove OrigRandom OrigRemove Orig&ProcessRandom Orig&Process
DeepEyes83.783.2 (-0.5)83.2 (-0.5)69.5 (-14.2)68.0 (-15.7)64.0 (-19.7)63.0 (-20.7)
Thyme82.981.6 (-1.3)82.2 (-0.7)68.6 (-14.3)67.6 (-15.3)61.4 (-21.5)61.4 (-21.5)
CodeV82.880.4 (-2.4)80.5 (-2.3)75.4 (-7.4)73.8 (-9.0)65.9 (-16.9)62.6 (-20.2)
FaithEyes86.982.7 (-4.2)82.3 (-4.6)79.6 (-7.3)78.0 (-8.9)69.1 (-17.8)61.5 (-25.4)
HR-Bench 4KFullRemove ProcessRandom ProcessRemove OrigRandom OrigRemove Orig&ProcessRandom Orig&Process
DeepEyes73.672.9 (-0.7)73.2 (-0.4)56.9 (-16.7)56.5 (-17.1)53.0 (-20.6)52.7 (-20.9)
Thyme74.875.7 (+0.9)76.1 (+1.3)55.7 (-19.1)56.9 (-17.9)52.7 (-22.1)52.8 (-22.0)
CodeV75.474.9 (-0.5)73.2 (-2.2)62.4 (-13.0)60.0 (-15.4)59.5 (-15.9)56.0 (-19.4)
FaithEyes77.174.5 (-2.6)73.9 (-3.2)64.2 (-12.9)61.9 (-15.2)58.3 (-18.8)55.5 (-21.6)
HR-Bench 8KFullRemove ProcessRandom ProcessRemove OrigRandom OrigRemove Orig&ProcessRandom Orig&Process
DeepEyes70.069.5 (-0.5)69.7 (-0.3)53.4 (-16.6)53.0 (-17.0)52.5 (-17.5)51.2 (-18.8)
Thyme69.369.4 (+0.1)69.0 (-0.3)55.9 (-13.4)53.3 (-16.0)52.4 (-16.9)51.9 (-17.4)
CodeV69.670.1 (+0.5)69.8 (+0.2)58.5 (-11.1)56.0 (-13.6)54.5 (-15.1)53.8 (-15.8)
FaithEyes72.271.4 (-0.8)70.4 (-1.8)61.7 (-10.5)58.7 (-13.5)55.3 (-16.9)53.9 (-18.3)

"Full" denotes the regenerated accuracy without image intervention. The "Remove/Random Orig&Process" results reflect what the model can achieve by relying solely on common sense and the textual content in the trajectory. From the results, we can observe:

(1) Under process-image interventions, FaithEyes shows the largest accuracy degradation among all compared models for both removal and randomization. Admittedly, the absolute drop is moderate (up to 4.6 points), but this does not mean the process images are ignored or meaningless when provided. The model could fall back on the original image and other context when process images are unavailable. Randomization hurts more than removal for FaithEyes on all three benchmarks, consistent with the model using the process-image content. In contrast, DeepEyes and Thyme are insensitive to process-image interventions, which is consistent with their low faithfulness ratios.

(2) When the original image is removed, FaithEyes retains the highest accuracy and exhibits the smallest accuracy drop on all three benchmarks. Comparing "Remove Orig" with "Remove Orig&Process" further shows that process images provide an additional +10.5/+5.9/+6.4 points of accuracy given the retained trajectory context. These results support an incremental contribution of process images during final-turn generation.

(3) As shown in the attention analysis (Table 6 of paper), the answer tokens of FaithEyes attend to process images, although less than to the original image, consistent with the text- and initial-image-centric attention bias of VLMs. Injecting the subagent judgement further increases attention to helpful images and decreases it for unhelpful ones. While attention itself is only a correlational proxy, the attention analysis and the causal intervention results above jointly indicate that the process images are genuinely consulted during answer generation.

Overall, the intervention results provide evidence that FaithEyes' predictions are sensitive to the availability and content of process images, and these images can contribute to answering even when the original image is unavailable. Although the original image remains an important source of evidence, the fine-grained process images from tool calls can yield additional gains.

[1] On the Faithfulness of Visual Thinking: Measurement and Enhancement. Liu et al. 2025.

AuthorsOpenReview ↗

We sincerely thank you for all your valuable comments. Below are our responses to each weakness and question.

Weakness 2: quantitative validation of supervision labels and risk of judgment bias

Thanks for this important question. We provide quantitative validation of supervision labels and the faithfulness ratios judged by models from different families below.

Firstly, the helpful/unhelpful labels are derived from trajectory structure in the publicly released Thyme SFT data and we generate rationales using Qwen3-VL-235B-A22B. Since grounding-style assessment is relatively easy, two authors independently checked 300 randomly sampled examples against the image, assessing whether the crop contains the target, whether the rationale is supported by the image and label, and for two-call trajectories, whether the relative focus of the crops supports the trajectory-derived label. We report inter-annotator agreement and human verification accuracy that is the proportion of examples for which both authors agreed with the trajectory-derived label and its rationale. We also use DeepSeek-V4.1-Flash-1017 to check the label and rationale of 10,000 samples. The results are shown below:

MetricInter-annotator agreementHuman verification, N=300DeepSeek-V4.1-Flash-1017, N=10000
Value95.3%93.0%94.2%

The residual disagreements are concentrated in borderline cases (e.g., crops that partially clip the target).

Secondly, to assess whether the reported ranking depends on Qwen3-VL-235B-A22B as the evaluation judge, we re-evaluate the tool faithfulness ratio with DeepSeek-V4.1-Flash-1017 using the same judging prompts. This model is from a different family, with no shared weights or training pipeline. The results are shown below:

Tool Faithfulness RatioV*HR-Bench 4KHR-Bench 8K
DeepEyes76.127.85.6
Thyme80.734.14.5
CodeV92.552.310.7
FaithEyes (keep unhelpful)94.080.155.9
FaithEyes94.387.279.5

The relative pattern is similar to that obtained with Qwen3-VL-235B-A22B, and FaithEyes has the highest reported ratio across all three benchmarks under this second judge.

Weakness 3: answer information from rationale

Thanks for this insightful question. We test whether rationale text contributes to accuracy by removing it while retaining the binary verdict and image-filtering behavior. The results below suggest that rationale text is not a major source of the observed accuracy gains.

Firstly, the subagent sees only the question and the local crop, not the main agent's chain-of-thought or code. For a helpful verdict, the process image is returned together with the rationale, so any attribute it mentions is simultaneously visible in the image itself, although expressing it in text is easier to use. For an unhelpful verdict, the crop is discarded and the rationale explains the rejection, such as "the queried target is not visible".

Secondly, we conduct a new ablation that removes all rationale text from the tool observations while keeping the binary verdict and image-filtering behavior unchanged during evaluation. As shown below, the accuracy differences are small across all six benchmarks. This provides evidence that rationale text has little effect on accuracy.

V*HR-Bench 4KHR-Bench 8KMathVistaMathVerseMathVision
FaithEyes-7B87.477.872.973.151.029.9
w/o Rationale87.577.172.573.050.430.2

Thirdly, we further measure what fraction of rationales actually contain explicit answer fragments, using DeepSeek-V4.1-Flash-1017 as the judge model. The ratios on all six benchmarks are:

V*HR-Bench 4KHR-Bench 8KMathVistaMathVerseMathVision
Ratio (%)65.3050.8017.2029.0018.003.30

The high ratios on perception benchmarks are plausible because judging whether a crop helps answer a question about an attribute such as color may involve mentioning that attribute, while the main agent can inspect the same crop directly. The small accuracy changes after rationale removal, even on V* with the highest answer-related ratio, suggest rationale text is not the main source of the gains.

AuthorsOpenReview ↗

We sincerely thank you for all your valuable comments. Below are our responses to each weakness and question.

Weakness 4: weak results on MathVision

Thanks for raising this concern, and we will explain it below.

(1) On MathVision, FaithEyes-7B improves accuracy from 21.4 to 29.9 and outperforms four of the five same-scale agentic baselines, ranking second to CodeV. However, FaithEyes outperforms CodeV on MathVista and MathVerse, so their relative performance varies across reasoning benchmarks.

(2) For Qwen3-VL-8B, Table 8 of paper shows the progression “base 53.9 → SFT 40.8 → RL 46.1” on MathVision. The largest decrease occurs during SFT, whereas RL effectively recovers performance by 5.3 points. One possible reason is an imbalance in mathematical supervision. Our SFT data are adapted from Thyme, whose authors explicitly report relatively limited mathematical-computation data compared with image-manipulation data. Fine-tuning Qwen3-VL-8B, which is already exceptionally strong at mathematics, on such trajectories shifts its output distribution.

We have incorporated the corresponding clarifications and additional results into the revised manuscript, with revisions marked in red.

The author’s response addressed my concerns; however, taking everything into account, I still consider this paper to be at a borderline level (e.g., a score of 2.5).

Reviewer_iqjyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FaithEyes to reduce the number of vision-linguistic model calls to tools that are either irrelevant to the question or make the calls only to obtain rewards. It uses the same parameters to be both the main agent and the checking sub-agent. The main agent calls a code tool to generate intermediate images, while the sub-agent evaluates if the images help answer the question and explains its judgment. The judgments are then added to the main agent’s context, and unhelpful images are filtered out. In addition, the judgments are used to determine a tool reward based on the ratio of the number of useful calls.

The authors train the model using supervised fine-tuning and gradient regularization with policy optimization and test their method on two base models across several visual perception and mathematical reasoning benchmarks. The method achieves reasonably consistent gains on high-resolution perception tasks, improves the paper’s tool faithfulness metric, and reduces token consumption. Their ablation study shows that both judgment feedback and tool rewards contribute to the results, albeit with substantial drops on some mathematical tasks.

Critical strengths
  1. I think the paper is addressing a practical problem. Rewarding model for calling a tool and answering correctly does not ensure that the tool output is actually useful. FaithEyes leverages judgement of tool outputs to guide subsequent reasoning plus training time rewards, thus allowing model to identify unhelpful crops and adapt. Using the same model as checker avoids needing to rely on another large model at inference time, the overall approach seems simple and practical.

  2. Experiments give some grounds for believing that the approach is effective. The authors tested it on two base models and reported largely consistent gains on high resolution perception benchmarks. The ablations suggest that both judgement feedback and reward shaping contribute. Notably, the tool faithfulness metric stays above baselines even when all the tool images are kept, suggesting that the gains are not solely due to dropping images that the model judged as unhelpful. I also like that the cost accounting includes the sub-agent's prompts and responses.

Critical weaknesses
  1. The current evaluation more directly supports the claim that tool images are more relevant than the claim that the model uses tool evidence more faithfully. The metrics in Section 4.2 and Appendix E.3 mainly check if the crops contain question-relevant targets and are reasonably focused, rather than if the answer is dependent on the images provided. Even with a correct crop, the main agent may answer based on the original image. The sub-agent's rationale may also contain information that directly helps answer the question, such as an object's color. Accuracy gains could therefore be achieved through an additional round of visual analysis and its textual output. The existing judgment-injection ablation does not separate the effects of the binary judgment, the information in the rationale, and the image itself. The attention analysis in the appendix offers some supporting observations but does not resolve this issue. Since more faithful tool use is the paper's central claim, I think it requires more direct experiments on evidence dependence, or a narrower interpretation of the results.

  2. The self-evaluator determines both rewards and context content, but its reliability has not been well-vetted. The method uses a shared-parameter model to generate tool calls and judge their usefulness, but the paper is unclear whether the evaluator is updated along with the policy during RL, or whether the sub-agent's outputs are included in the optimization. If the evaluator were updated during training, the authors need to rule out the possibility that tool rewards rise partly because its judgments become more lenient. The SFT judgment labels are mainly built from the order of tool calls in existing trajectories, with a large model then generating rationales for the assigned labels. Although the authors mention manual spot checks, they do not report quantitative label quality or judgment accuracy. The final evaluation also uses Qwen3-VL-235B, which generated the SFT rationales; using a different prompt does not fully rule out the influences of shared judgment preferences. I would therefore like to see an independent human evaluation or cross-validation with a different model family to establish whether the gains reflect better tool outputs rather than closer alignment with a particular evaluator's preferences.

Questions for authors
  1. During RL, does the checking sub-agent use the current policy or a frozen SFT model? Are its judgment and rationale tokens included in the policy loss? If the checking sub-agent's parameters are updated during the training process, have the authors evaluated its judgment accuracy and the portion of tool outputs it deems useful?

  2. The authors mention manually spot-checking the generated data. Could they please report how many examples they spot-checked, what criteria they used for spot-checking, and what accuracy they achieved? For the final tool faithfulness results, are there alternative human annotations or model judgments from other families that could corroborate the given findings?

3 comments
AuthorsOpenReview ↗

We sincerely thank you for all your valuable comments. Below are our responses to each weakness and question.

Weakness 1.1: direct evidence for dependence on process images

We agree that the faithfulness ratio alone cannot establish dependence on process images. We therefore conduct intervention experiments following the protocol of [1]. For each example, we truncate the recorded trajectory immediately before the final answer-producing assistant turn, apply image interventions to the retained context, and regenerate the final answer. We remove or randomize the process images, the original image, or both. For randomization, we replace all pixel values with Gaussian noise. Results for the data-comparable agentic baselines and FaithEyes on V*, HR-Bench 4K and HR-Bench 8K are shown below.

V*FullRemove ProcessRandom ProcessRemove OrigRandom OrigRemove Orig&ProcessRandom Orig&Process
DeepEyes83.783.2 (-0.5)83.2 (-0.5)69.5 (-14.2)68.0 (-15.7)64.0 (-19.7)63.0 (-20.7)
Thyme82.981.6 (-1.3)82.2 (-0.7)68.6 (-14.3)67.6 (-15.3)61.4 (-21.5)61.4 (-21.5)
CodeV82.880.4 (-2.4)80.5 (-2.3)75.4 (-7.4)73.8 (-9.0)65.9 (-16.9)62.6 (-20.2)
FaithEyes86.982.7 (-4.2)82.3 (-4.6)79.6 (-7.3)78.0 (-8.9)69.1 (-17.8)61.5 (-25.4)
HR-Bench 4KFullRemove ProcessRandom ProcessRemove OrigRandom OrigRemove Orig&ProcessRandom Orig&Process
DeepEyes73.672.9 (-0.7)73.2 (-0.4)56.9 (-16.7)56.5 (-17.1)53.0 (-20.6)52.7 (-20.9)
Thyme74.875.7 (+0.9)76.1 (+1.3)55.7 (-19.1)56.9 (-17.9)52.7 (-22.1)52.8 (-22.0)
CodeV75.474.9 (-0.5)73.2 (-2.2)62.4 (-13.0)60.0 (-15.4)59.5 (-15.9)56.0 (-19.4)
FaithEyes77.174.5 (-2.6)73.9 (-3.2)64.2 (-12.9)61.9 (-15.2)58.3 (-18.8)55.5 (-21.6)
HR-Bench 8KFullRemove ProcessRandom ProcessRemove OrigRandom OrigRemove Orig&ProcessRandom Orig&Process
DeepEyes70.069.5 (-0.5)69.7 (-0.3)53.4 (-16.6)53.0 (-17.0)52.5 (-17.5)51.2 (-18.8)
Thyme69.369.4 (+0.1)69.0 (-0.3)55.9 (-13.4)53.3 (-16.0)52.4 (-16.9)51.9 (-17.4)
CodeV69.670.1 (+0.5)69.8 (+0.2)58.5 (-11.1)56.0 (-13.6)54.5 (-15.1)53.8 (-15.8)
FaithEyes72.271.4 (-0.8)70.4 (-1.8)61.7 (-10.5)58.7 (-13.5)55.3 (-16.9)53.9 (-18.3)

"Full" denotes the regenerated accuracy without image intervention. The "Remove/Random Orig&Process" results reflect what the model can achieve by relying solely on common sense and the textual content in the trajectory. From the results, we can observe:

(1) Under process-image interventions, FaithEyes shows the largest accuracy degradation among all compared models for both removal and randomization. Admittedly, the absolute drop is moderate (up to 4.6 points), but this does not mean the process images are ignored or meaningless when provided. The model could fall back on the original image and other context when process images are unavailable. Randomization hurts more than removal for FaithEyes on all three benchmarks, consistent with the model using the process-image content. In contrast, DeepEyes and Thyme are insensitive to process-image interventions, which is consistent with their low faithfulness ratios.

(2) When the original image is removed, FaithEyes retains the highest accuracy and exhibits the smallest accuracy drop on all three benchmarks. Comparing "Remove Orig" with "Remove Orig&Process" further shows that process images provide an additional +10.5/+5.9/+6.4 points of accuracy given the retained trajectory context. These results support an incremental contribution of process images during final-turn generation.

(3) As shown in the attention analysis (Table 6 of paper), the answer tokens of FaithEyes attend to process images, although less than to the original image, consistent with the text- and initial-image-centric attention bias of VLMs. Injecting the subagent judgement further increases attention to helpful images and decreases it for unhelpful ones. While attention itself is only a correlational proxy, the attention analysis and the causal intervention results above jointly indicate that the process images are genuinely consulted during answer generation.

Overall, the intervention results provide evidence that FaithEyes' predictions are sensitive to the availability and content of process images, and these images can contribute to answering even when the original image is unavailable. Although the original image remains an important source of evidence, the fine-grained process images from tool calls can yield additional gains.

[1] On the Faithfulness of Visual Thinking: Measurement and Enhancement. Liu et al. 2025.

AuthorsOpenReview ↗

We sincerely thank you for all your valuable comments. Below are our responses to each weakness and question.

Weakness 1.2: answer information from rationale

Thanks for this insightful question. We test whether rationale text contributes to accuracy by removing it while retaining the binary verdict and image-filtering behavior. The results below suggest that rationale text is not a major source of the observed accuracy gains.

Firstly, the subagent sees only the question and the local crop, not the main agent's chain-of-thought or code. For a helpful verdict, the process image is returned together with the rationale, so any attribute it mentions is simultaneously visible in the image itself, although expressing it in text is easier to use. For an unhelpful verdict, the crop is discarded and the rationale explains the rejection, such as "the queried target is not visible".

Secondly, we conduct a new ablation that removes all rationale text from the tool observations while keeping the binary verdict and image-filtering behavior unchanged during evaluation. As shown below, the accuracy differences are small across all six benchmarks. This provides evidence that rationale text has little effect on accuracy.

V*HR-Bench 4KHR-Bench 8KMathVistaMathVerseMathVision
FaithEyes-7B87.477.872.973.151.029.9
w/o Rationale87.577.172.573.050.430.2

Thirdly, we further measure what fraction of rationales actually contain explicit answer fragments, using DeepSeek-V4.1-Flash-1017 as the judge model. The ratios on all six benchmarks are:

V*HR-Bench 4KHR-Bench 8KMathVistaMathVerseMathVision
Ratio (%)65.3050.8017.2029.0018.003.30

The high ratios on perception benchmarks are plausible because judging whether a crop helps answer a question about an attribute such as color may involve mentioning that attribute, while the main agent can inspect the same crop directly. The small accuracy changes after rationale removal, even on V* with the highest answer-related ratio, suggest rationale text is not the main source of the gains.

AuthorsOpenReview ↗

We sincerely thank you for all your valuable comments. Below are our responses to each weakness and question.

Question 1 & Weakness 2.1: update dynamics of the evaluator

Thanks for this important question. We first clarify the update dynamics of the subagent, then report evidence relevant to the concern that its judgements may become more lenient.

Firstly, the subagent is not a separate model but the policy model itself, instantiated with a specific subagent prompt. Its weights are shared with, and updated along with, the main policy. However, all tool observations, including the subagent's judgements, are excluded from the policy loss and advantage estimation. The judgements therefore never enter the policy loss, and the model receives gradients only on the main agent's tokens (reasoning, code, and answers), with no direct gradients on how it judges.

Secondly, role-specific masking removes the direct optimization path through the subagent's judgement tokens. The policy is not explicitly rewarded for producing lenient judgements. Since shared-parameter updates could still change its judging behavior, we evaluate the subagent at different training steps throughout RL by comparing its judgements with those of DeepSeek-V4.1-Flash-1017 on V* and HR-Bench 4K. The consistency ratios remain high and stable, with no downward trend over training:

Consistency ratio (%)Step-0Step-50Step-100Step-150Step-200Step-250Step-300Step-380
V*86.486.185.485.884.985.686.186.9
HR-Bench 4K88.688.287.487.886.987.788.389.1

These results indicate that the subagent's judgements remain closely aligned with a strong external model on grounding-style evaluation throughout RL, with no empirical sign of increasing leniency. Additionally, the "Tool Reward" curve in Figure 5 of paper shows the helpful-tool ratio rising from below 60% to about 90% during RL.

Question 2 & Weakness 2.2: details of manual checks and judgement results from other model families

Thanks for this important question. We provide details of the manual checks and the faithfulness ratios assessed by models from different families below.

Firstly, the helpful/unhelpful labels are derived from trajectory structure in the publicly released Thyme SFT data and we generate rationales using Qwen3-VL-235B-A22B. Since grounding-style assessment is relatively easy, two authors independently checked 300 randomly sampled examples against the image, assessing whether the crop contains the target, whether the rationale is supported by the image and label, and for two-call trajectories, whether the relative focus of the crops supports the trajectory-derived label. We report inter-annotator agreement and human verification accuracy that is the proportion of examples for which both authors agreed with the trajectory-derived label and its rationale. We also use DeepSeek-V4.1-Flash-1017 to check the label and rationale of 10,000 samples. The results are shown below:

MetricInter-annotator agreementHuman verification, N=300DeepSeek-V4.1-Flash-1017, N=10000
Value95.3%93.0%94.2%

The residual disagreements are concentrated in borderline cases (e.g., crops that partially clip the target).

Secondly, to assess whether the reported ranking depends on Qwen3-VL-235B-A22B as the evaluation judge, we re-evaluate the tool faithfulness ratio with DeepSeek-V4.1-Flash-1017 using the same judging prompts. This model is from a different family, with no shared weights or training pipeline. The results are shown below:

Tool Faithfulness RatioV*HR-Bench 4KHR-Bench 8K
DeepEyes76.127.85.6
Thyme80.734.14.5
CodeV92.552.310.7
FaithEyes (keep unhelpful)94.080.155.9
FaithEyes94.387.279.5

The relative pattern is similar to that obtained with Qwen3-VL-235B-A22B, and FaithEyes has the highest reported ratio across all three benchmarks under this second judge.

We have incorporated the corresponding clarifications and additional results into the revised manuscript, with revisions marked in red.

Reviewer_TiR3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studied the problem of the problem of faithful tool use in agentic vision-language model (VLM), wher recent studies have shown that VLMs often use tools unfaithfully. That is, many tool uses on images are irrelevant to the question, yet the tool call still receives full credit and the model still answers correctly. To address this problem, the authors proposed to use a VLM to judge whether each process image helps answer the question. The judgement is injected into the reasoning context as part of the tool observation to help subsequent reasoning, and meanwhile is used to scale the tool reward by the helpful-tool ratio to suppress reward hacking. To keep judgement available at evaluation, the authors further designed a multi-agent framework where the model itself serves as a subagent to judge the tool calls from the main agent. The authors proposed to training the VLM via a two-stage SFT + RL pipeline on adapted open-source data. The authors conducted experiments to verify the performance of their proposed approach and training methods.

Critical strengths
  1. The faithful tool use problem in VLMs is a timely and important problem.

  2. The proposed main agent and subagent approach for judgement injected into the reasoning context as part of the tool observation to help subsequent reasoning is new and interesting.

Critical weaknesses
  1. Although the multi-agent judgement framework is an interesting idea, it may also introduce the risk of inaccurate judgements in faithful tool use. If such inaccurate judgement arises, how to ensure the faithful tool use and the overall performance?

  2. The SFT and GRPO training and the associated reward function design are all very standard, and thus the novelty in these aspects is marginal. Could the authors highlight any specific differences and challenges in using these methods in VLM faithful tool uses?

Questions for authors

Please see the weakness section above.

1 comment
AuthorsOpenReview ↗

We sincerely thank you for all your valuable comments. Below are our responses to each weakness and question.

Weakness 1: risk of inaccurate self-judgement

Thank you for raising this important concern about inaccurate self-judgement. We address it in three parts.

Firstly, inaccurate self-judgements can affect the trajectory, but the design provides possible recovery paths. For a false rejection, a helpful image is dropped while its judgement is returned. Because the main agent is trained for judgement-driven reasoning during SFT, it may respond by making another, better-targeted crop or by reasoning from the original image when it is sufficient. This may incur an additional tool call, but could allow the agent to recover by recropping or using the original image. For a false acceptance, an unhelpful image is retained and may add noise to subsequent reasoning, as can happen in baseline agentic VLMs where the main agent will re-examine the image. The tool-reward coefficient is small, which limits the direct contribution of this reward term.

Secondly, although inaccurate self-judgement may not lead to catastrophic collapse, they still have a clear impact on both faithfulness and accuracy. To this end, we explicitly cold-start this capability during SFT using a large number of judgement trajectories. In fact, the judging task is easier than the solving task. The subagent only needs to answer a grounding-style question rather than solve the original problem.

Thirdly, we further assess whether the self-judge's capability degrades during RL by comparing its judgements at different training steps with those of DeepSeek-V4.1-Flash-1017, a strong model for this grounding-style evaluation. The consistency ratios remain high and stable throughout training:

Consistency ratio (%)Step-0Step-50Step-100Step-150Step-200Step-250Step-300Step-380
V*86.486.185.485.884.985.686.186.9
HR-Bench 4K88.688.287.487.886.987.788.389.1

Agreement remains high with only modest fluctuations, providing evidence against a progressive decline in self-judging capability or increasing leniency during RL. Additionally, as shown in Table 4 of paper, replacing the subagent with Qwen3-VL-235B-A22B also yields only a limited improvement in accuracy and faithfulness ratio, suggesting that our model itself is sufficient for judging.

Weakness 2: novelty in SFT and GRPO training

We agree that SFT+RL is a standard recipe. However, faithful tool use raises task-specific challenges that motivate several design choices.

Firstly, the standard accuracy-dependent tool reward design collapses in our setting. The tool execution failure rate spikes to approximately 18%, and the number of tool calls drops toward zero, as shown in Figure 6 of paper, since no tool reward is available on hard questions regardless of tool quality. This failure motivates our two deviations: decoupling the tool reward from answer correctness, and computing it as a helpful-tool ratio rather than a per-call sum to avoid a reward bonus that grows linearly with call frequency.

Secondly, the open-source SFT trajectories do not provide explicit helpfulness rationales or downstream reasoning conditioned on such judgements. We adapt these existing data by deriving helpful/unhelpful labels from trajectory structure, generating label-consistent rationales, and rewriting the downstream reasoning to condition on the judgement.

Thirdly, the RL rollout is a multi-agent interaction rather than a standard single-policy rollout, which demands role-specific masking. Unlike standard RL, where the environment is fixed, our rollout environment includes the policy model itself, which acts as the subagent and produces the judgements. All tool observations (including the subagent's judgements) are therefore excluded from the loss and advantage estimation. Otherwise, the policy would receive gradients on its own judgements and be incentivized to update its own environment, for example, by generating more lenient rather than more accurate judgements.

Overall, we treat SFT+RL as established infrastructure and concentrate our novelty on what faithful tool use demands beyond it: a decoupled, ratio-based tool reward validated by the failure of the standard design, the construction of judgement supervision, and role-specific masking rules.

We have incorporated the corresponding clarifications and additional results into the revised manuscript, with revisions marked in red.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewers,

Thank you again for your thoughtful and constructive feedback. We have provided point-by-point responses to your concerns in our rebuttal and would welcome your thoughts on whether these responses adequately address them.

If any concerns remain unresolved, we would appreciate the opportunity to better understand them and provide further clarification during the discussion period. If our responses have addressed your concerns, we would be grateful if you could reconsider your assessment in light of the rebuttal.

Thank you for your time and consideration. We look forward to the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.