- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies rationale hallucination in multimodal safety guards that produce explanations alongside harmful/benign predictions, decomposing the failure into visual evidence hallucination and safety attribution hallucination. Across 4 benchmarks, 3 rationale-generation paradigms, and 2 Qwen2.5-VL backbones at 7B/32B, it finds that visual hallucination persists across methods and scales, that higher classification accuracy does not imply better grounding, that hallucination concentrates in Attribute/Relation atomic facts, and that Think-with-Image tools are used too rarely or too loosely. Building on this diagnosis, it proposes TCG-RL, a GRPO variant that rewards tool use only when it improves atomic-fact grounding relative to a no-tool reference, reporting VES gains with preserved accuracy on Qwen2.5-VL-7B.
Critical strengths
- The paper targets a gap: as safety guards move from binary detection to explanation-augmented decisions, whether the accompanying rationale is faithful is largely unexamined, and the visual-vs-attribution split (Figure 1) is intuitive and reusable.
- The decoupling result — correct safety prediction does not imply a grounded rationale, consistent across datasets, methods, and model scales — is the most transferable finding and is well supported.
- The root-cause analysis gives concrete mechanistic hypotheses rather than a generic "models hallucinate" claim: hallucination concentrates in Attribute/Relation atomic facts, and DyFo's tools are invoked infrequently and focused poorly, and the TCG-RL w/o-TC ablation correctly isolates the contribution of the contrastive reward.
Critical weaknesses
- VES and SAS are both LLM-judged, but the main text never states which judge is used and reports no human agreement or cross-judge consistency; with the entire empirical story resting on these scores, judge reliability is load-bearing but deferred entirely to the appendix.
- The TCG-RL reward uses F_atomic — the same atomic-fact failure score that defines VES at evaluation — creating reward–evaluation circularity with no human evaluation of TCG-RL rationales and no held-out judge, so the paper cannot rule out that the model has learned to score well on the judge rather than to be actually grounded.
- TCG-RL is validated only on Qwen2.5-VL-7B despite the paper's own 7B-vs-32B analysis, compares only against Direct CoT/GRPO/Thyme rather than the U-CoT, MIND, ALARM, Visual CoT, DyFo, and ZoomEye baselines characterized in Section 3, and includes no "always use tools" arm to show that selective tool use actually beats more tool use.
- The no-tool-reference contrastive reward is presented as the novel mechanism, but concurrent work already rewards tool use based on whether it helps — CodeV/TAPO (Hou et al., 2025) uses process-level rewards on visual tool outputs, and AdaTooler-V (Wang et al., ACL 2026) uses a Tool Benefit Score; the paper must position TCG-RL against these and explain why its safety-rationale target and atomic-fact reward are distinct.
- The root-cause claims about Think-with-Image tools rest solely on DyFo's tool-use statistics, which may not generalize to Visual CoT, ZoomEye, DeepEyes, or Thyme; and the ground-truth safety attributions used in Eqs. 2–4 are never described as human- or LLM-derived, leaving SAS with the same reliability concern as VES.
- The "approximately 20% over strong baselines" claim (lines 111–112) is not traceable to Table 3, where VES gains over Thyme are only ~3–7 absolute points, and the abstract's "substantially reduces rationale hallucination" should be scoped to visual evidence, since SAS gains are smaller and accuracy drops slightly on MAMI and PrideMM.
Questions for authors
The method's evidence — judge reliability, reward–metric circularity, single-backbone and incomplete baselines, missing related-work positioning, and over-stated gains — does not yet support TCG-RL as a headline contribution. See weakness for more details.
4 comments
Response to Reviewer JCaQ (2/3)
W3: Baseline Coverage and Selective Tool Use
Thank you for these suggestions. Computational constraints currently prevent additional training experiments on larger backbones. We have nevertheless consolidated the complete comparison between TCG-RL and all evaluated baselines (described in Section 3) on Qwen2.5-VL-7B. The expanded comparison is consistent with the findings against Thyme: TCG-RL achieves the highest rationale reliability while maintaining leading classification performance.
- Comparison with the full set of baselines
| Qwen2.5-7B-VL | Harm-P | PrideMM | ||||
|---|---|---|---|---|---|---|
| Acc. | VES | SAS | Acc. | VES | SAS | |
| Direct CoT | 56.9 | 76.4 | 76.6 | 60.2 | 75.7 | 72.7 |
| U-CoT+ | 57.1 | 68.4 | 75.9 | 60.9 | 68.5 | 63.0 |
| MIND | 57.2 | 78.4 | 71.8 | 64.7 | 74.5 | 62.0 |
| ALARM | 60.3 | 74.6 | 76.0 | 64.3 | 70.8 | 66.4 |
| Visual CoT | 59.2 | 75.6 | 73.6 | 60.2 | 75.6 | 71.4 |
| DyFo | 56.9 | 74.2 | 78.1 | 63.1 | 74.5 | 71.9 |
| ZoomEye | 57.5 | 72.9 | 76.2 | 60.9 | 65.5 | 69.1 |
| Thyme | 65.1 | 76.2 | 72.0 | 74.6 | 77.5 | 65.3 |
| TCG-RL | 65.3 | 97.0 | 76.8 | 76.8 | 95.8 | 78.5 |
We further compare three tool-use settings: Selective tools allows TCG-RL to decide whether and how to use tools; Always-tools requires at least one visual tool operation per input; and No tools disables visual tools. Tool-use Rate is the percentage of samples with at least one executed visual tool operation.
As shown in Table below, Selective tools outperforms Always-tools on both classification accuracy and rationale reliability. These results show that TCG-RL benefits from using visual tools appropriately, rather than simply invoking them more often.
- Comparison of tool-use strategies
| Qwen2.5-7B-VL | Harm-P | PrideMM | ||||||
|---|---|---|---|---|---|---|---|---|
| Acc. | VES | SAS | Tool-use Rate | Acc. | VES | SAS | Tool-use Rate | |
| Selective tools (TCG-RL) | 65.3 | 97.0 | 75.8 | 78.2 | 76.8 | 95.8 | 78.5 | 72.6 |
| No tools | 61.6 | 80.2 | 64.6 | 0 | 72.9 | 83.7 | 62.7 | 0 |
| Always-tools | 64.0 | 90.9 | 73.8 | 100 | 74.0 | 86.3 | 72.6 | 100 |
W4: Relationship to Existing Tool-Benefit Reward Methods
Thank you for highlighting CodeV/TAPO and AdaTooler-V. While CodeV/TAPO rewards relevant intermediate tool outputs and AdaTooler-V measures tool benefit through accuracy differences, TCG-RL rewards improved support for atomic visual facts in generated rationales relative to a fixed no-tool reference.
This objective addresses our central finding: correct safety predictions and relevant tool observations do not guarantee grounded safety rationales. Our contribution lies in systematically diagnosing this gap and optimize rationale grounding. The existing w/o TC ablation and the always-tools comparison support the value of the contrastive component and selective tool use. We will clarify these connections and add the relevant citations in the revision.
Response to Reviewer JCaQ (3/3)
W5: Coverage of the Tool-Use Analysis and Safety-Attribution Labels
Thank you for this suggestion. Since DyFo has a relatively autonomous tool-use policy, we use it as the main analysis case in the paper. We additionally provide results for ZoomEye and Visual CoT. For ZoomEye, the tool-use strategy is fixed, and the model only decides whether to invoke a tool. We therefore report the Tool-use rate. For Visual CoT, the cropping procedure is fixed, but the model determines the size of the cropped region, so we report the Focused Zoom rate.
- ZoomEye Tool-use rate (%)
| Tool use | HarmC | HarmP | MAMI | PrideMM |
|---|---|---|---|---|
| Qwen2.5-7B-VL | 4.8 | 8.8 | 16.5 | 41.0 |
| Qwen2.5-32B-VL | 78.3 | 80.7 | 69.6 | 68.4 |
| Qwen3-8B-VL | 1.7 | 1.2 | 1.5 | 5.4 |
| Qwen3-32B-VL | 28.5 | 15.4 | 1.7 | 8.1 |
| Qwen3.5-27B | 54.3 | 64.8 | 78.4 | 76.0 |
- Visual CoT Focused Zoom rate (%)
| Focused Zoom | HarmC | HarmP | MAMI | PrideMM |
|---|---|---|---|---|
| Qwen2.5-7B-VL | 51.3 | 46.6 | 47.7 | 52.8 |
| Qwen2.5-32B-VL | 54.6 | 50.0 | 52.8 | 56.9 |
| Qwen3-8B-VL | 29.1 | 21.6 | 31.1 | 46.3 |
| Qwen3-32B-VL | 20.3 | 20.6 | 45.5 | 57.5 |
| Qwen3.5-27B | 43.6 | 42.7 | 53.7 | 59.1 |
The low Tool-use rate of ZoomEye and the Focused Zoom rate of Visual CoT are consistent with the main finding in Section 4 that tool access alone does not ensure fine-grained grounding: models must also decide when and where to inspect, and how to use the resulting observations in their rationales.
Regarding the reference safety-attribution labels, Appendix A (Lines 702–732) specifies that these labels come from dataset annotations: harmful-target categories for Harm-C/Harm-P, misogyny types for MAMI, and stance labels for PrideMM. SAS is extracted and judged by an LLM judge, using Qwen3-32B. The independent reliability analysis of this judge is provided in our response to W1.
W6: Clarification of the Reported Gains.
Thank you for pointing this out. We rechecked Table 3 and clarified the comparison. Compared with to Thyme, TCG-RL improves VES by 23.1, 20.8, 17.1, and 18.3 percentage points on Harm-C, Harm-P, MAMI, and PrideMM, respectively, confirming an average improvement of 19.8 percentage points. Thus, the original “approximately 20%” refers to this average gain in VES. The corresponding average SAS improvement is 9.0 percentage points. Meanwhile, classification accuracy decreases slightly on MAMI (−1.0), while it actually improves on PrideMM (+2.2); overall classification performance remains comparable. We will scope the abstract’s claim to the reduction in visual evidence hallucination and report the improvement in safety attribution separately.
We will incorporate the clarifications and additional results into the revised manuscript. We thank the reviewer again for helping us improve the paper.
A Gentle Reminder
Dear Reviewer JCaQ,
Thank you again for your thoughtful comments and constructive suggestions throughout the discussion. As the author–reviewer discussion period is coming to an end, we would like to kindly ask whether our responses have addressed your main concerns. If there are still aspects that remain unclear or require further clarification, we would be very happy to discuss them further.
We sincerely appreciate your time and valuable feedback.
Response to Reviewer JCaQ (1/3)
We thank the reviewer for the thorough and constructive feedback. We are glad that the reviewer found the distinction between visual evidence hallucination and safety attribution hallucination useful, and valued our findings on the gap between classification accuracy and rationale grounding. We address each concern below.
Thank you for raising this concern.
Appendix C.5 (Lines 906–907)specifies that VES uses Qwen3-32B for descriptive-clause identification and atomic-fact extraction, and Qwen3-VL-32B-Instruct for visual verification. SAS is judged by Qwen3-32B. We will make the evaluation models and their reliability explicit in the main experimental setup.VES follows the atomic-fact verification framework of FaithScore (Jing et al., EMNLP 2024), which was evaluated against human judgments. We further validate reliability with our evaluators on safety rationales. We randomly sampled 100 atomic fact–image pairs across four datasets and five tested models for VES, and 100 rationale–image pairs for SAS, covering over-flagging (OF), under-flagging (UF), and mis-attribution (MA).Three researchers independently annotated these 200 cases. We also used an independent gpt-6.1-sol judge from a different model family for cross-judge validation.
The table reports unanimous agreement and Fleiss’ κ among the three researchers, alongside agreement rates and Cohen’s κ for Qwen versus the human majority vote and Qwen versus GPT.
Qwen agrees with the human majority on 90%–96% of cases (κ=0.73–0.89) and with GPT on 85%–89% (κ=0.69–0.74), supporting judge reliability in our setting (Landis and Koch, 1977). We will include the annotation protocol and validation results in the revision.
Thank you for raising this concern. We add independent validation and output analyses to distinguish genuine grounding improvements from evaluator adaptation.
First, we report rationale Text Length and Fact Count, the average number of atomic facts extracted per rationale during VES computation, indicating how much factual content it contains. TCG-RL maintains text lengths comparable to Thyme and produces 13–14 facts per rationale, showing that its gains do not rely on short or factually sparse explanations.
Second, we examine Attribute and Relation facts, which are harder to verify and closely related to safety interpretation (
Section 4). TCG-RL achieves higher visual support rates than Thyme across all datasets, supporting improved grounding of fine-grained evidence.Third, we re-evaluate the same rationales with an independent gpt-6.1-sol judge not involved in TCG-RL training. VES and SAS gains persist across all datasets, providing evidence that the improvements generalize beyond the training evaluator.
Finally, three researchers blindly compared TCG-RL and Thyme on 100 paired examples from the four test sets. Based on the majority vote, TCG-RL achieved a 73% win rate. Together, these results support genuine improvements in rationale grounding rather than merely adaptation to the judge. We will include these analyses and the human evaluation protocol in the revision.