acceptodds
Under review as a conference paper at ICLR 2027

When Safety Rationales Hallucinate: Diagnosing and Grounding Multimodal Safety Guards

Abstract

Multimodal safety guards increasingly generate rationales alongside safety predictions, yet whether these rationales are reliable remains largely unexplored. We systematically study *rationale hallucination* from two perspectives: visual rationale hallucination, where rationales contain unsupported visual facts, and safety attribution hallucination, where grounded content is assigned an incorrect safety interpretation. Across rationale-generation methods, multiple vision-language backbones, and safety datasets, we find that rationale hallucination is widespread and does not consistently diminish with higher classification accuracy or increased model scale. Further analysis of the Think-with-Image paradigm shows that hallucination is concentrated in fine-grained attributes and relations, while visual tools are often invoked infrequently or without sufficient spatial focus. Based on these findings, we propose *Tool-Contrastive Grounding Reinforcement Learning* (TCG-RL), which rewards tool use only when it improves rationale grounding over a tool-free reference. TCG-RL substantially reduces rationale hallucination while maintaining competitive safety prediction performance. By revealing the gap between prediction correctness and rationale reliability, our study motivates a shift in multimodal safety guarding from outcome-only evaluation toward safety decisions that are both accurate and verifiably grounded.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety, Multimodal Safety Guard, Hallucination

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JCaQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies rationale hallucination in multimodal safety guards that produce explanations alongside harmful/benign predictions, decomposing the failure into visual evidence hallucination and safety attribution hallucination. Across 4 benchmarks, 3 rationale-generation paradigms, and 2 Qwen2.5-VL backbones at 7B/32B, it finds that visual hallucination persists across methods and scales, that higher classification accuracy does not imply better grounding, that hallucination concentrates in Attribute/Relation atomic facts, and that Think-with-Image tools are used too rarely or too loosely. Building on this diagnosis, it proposes TCG-RL, a GRPO variant that rewards tool use only when it improves atomic-fact grounding relative to a no-tool reference, reporting VES gains with preserved accuracy on Qwen2.5-VL-7B.

Critical strengths
  1. The paper targets a gap: as safety guards move from binary detection to explanation-augmented decisions, whether the accompanying rationale is faithful is largely unexamined, and the visual-vs-attribution split (Figure 1) is intuitive and reusable.
  2. The decoupling result — correct safety prediction does not imply a grounded rationale, consistent across datasets, methods, and model scales — is the most transferable finding and is well supported.
  3. The root-cause analysis gives concrete mechanistic hypotheses rather than a generic "models hallucinate" claim: hallucination concentrates in Attribute/Relation atomic facts, and DyFo's tools are invoked infrequently and focused poorly, and the TCG-RL w/o-TC ablation correctly isolates the contribution of the contrastive reward.
Critical weaknesses
  1. VES and SAS are both LLM-judged, but the main text never states which judge is used and reports no human agreement or cross-judge consistency; with the entire empirical story resting on these scores, judge reliability is load-bearing but deferred entirely to the appendix.
  2. The TCG-RL reward uses F_atomic — the same atomic-fact failure score that defines VES at evaluation — creating reward–evaluation circularity with no human evaluation of TCG-RL rationales and no held-out judge, so the paper cannot rule out that the model has learned to score well on the judge rather than to be actually grounded.
  3. TCG-RL is validated only on Qwen2.5-VL-7B despite the paper's own 7B-vs-32B analysis, compares only against Direct CoT/GRPO/Thyme rather than the U-CoT, MIND, ALARM, Visual CoT, DyFo, and ZoomEye baselines characterized in Section 3, and includes no "always use tools" arm to show that selective tool use actually beats more tool use.
  4. The no-tool-reference contrastive reward is presented as the novel mechanism, but concurrent work already rewards tool use based on whether it helps — CodeV/TAPO (Hou et al., 2025) uses process-level rewards on visual tool outputs, and AdaTooler-V (Wang et al., ACL 2026) uses a Tool Benefit Score; the paper must position TCG-RL against these and explain why its safety-rationale target and atomic-fact reward are distinct.
  5. The root-cause claims about Think-with-Image tools rest solely on DyFo's tool-use statistics, which may not generalize to Visual CoT, ZoomEye, DeepEyes, or Thyme; and the ground-truth safety attributions used in Eqs. 2–4 are never described as human- or LLM-derived, leaving SAS with the same reliability concern as VES.
  6. The "approximately 20% over strong baselines" claim (lines 111–112) is not traceable to Table 3, where VES gains over Thyme are only ~3–7 absolute points, and the abstract's "substantially reduces rationale hallucination" should be scoped to visual evidence, since SAS gains are smaller and accuracy drops slightly on MAMI and PrideMM.
Questions for authors

The method's evidence — judge reliability, reward–metric circularity, single-backbone and incomplete baselines, missing related-work positioning, and over-stated gains — does not yet support TCG-RL as a headline contribution. See weakness for more details.

4 comments
AuthorsOpenReview ↗

Response to Reviewer JCaQ (1/3)

We thank the reviewer for the thorough and constructive feedback. We are glad that the reviewer found the distinction between visual evidence hallucination and safety attribution hallucination useful, and valued our findings on the gap between classification accuracy and rationale grounding. We address each concern below.

W1: Evaluation models and their reliability

Thank you for raising this concern. Appendix C.5 (Lines 906–907) specifies that VES uses Qwen3-32B for descriptive-clause identification and atomic-fact extraction, and Qwen3-VL-32B-Instruct for visual verification. SAS is judged by Qwen3-32B. We will make the evaluation models and their reliability explicit in the main experimental setup.

VES follows the atomic-fact verification framework of FaithScore (Jing et al., EMNLP 2024), which was evaluated against human judgments. We further validate reliability with our evaluators on safety rationales. We randomly sampled 100 atomic fact–image pairs across four datasets and five tested models for VES, and 100 rationale–image pairs for SAS, covering over-flagging (OF), under-flagging (UF), and mis-attribution (MA).Three researchers independently annotated these 200 cases. We also used an independent gpt-6.1-sol judge from a different model family for cross-judge validation.

The table reports unanimous agreement and Fleiss’ κ among the three researchers, alongside agreement rates and Cohen’s κ for Qwen versus the human majority vote and Qwen versus GPT.

ConsistencyVESSAS-OFSAS-UFSAS-MA
Unanimous human agreement (%)88899392
Qwen–human majority agreement (%)94959690
Qwen–GPT agreement (%)89858785
κ
Inter-human Fleiss’κ0.830.820.870.81
Qwen–human majority Cohen’s κ0.850.880.890.73
Qwen–GPT Cohen’s κ0.740.690.710.72

Qwen agrees with the human majority on 90%–96% of cases (κ=0.73–0.89) and with GPT on 85%–89% (κ=0.69–0.74), supporting judge reliability in our setting (Landis and Koch, 1977). We will include the annotation protocol and validation results in the revision.

W2: Reward–evaluation circularity

Thank you for raising this concern. We add independent validation and output analyses to distinguish genuine grounding improvements from evaluator adaptation.

First, we report rationale Text Length and Fact Count, the average number of atomic facts extracted per rationale during VES computation, indicating how much factual content it contains. TCG-RL maintains text lengths comparable to Thyme and produces 13–14 facts per rationale, showing that its gains do not rely on short or factually sparse explanations.

  • Rationale Length (words) and Fact Count
Qwen2.5-7B-VLHarm-PPrideMM
Text lengthFact countText lengthFact count
Direct CoT178.0618.41187.2222.83
GRPO191.3911.55178.2613.87
Thyme141.4315.95150.4414.76
TCG-RL140.6814.42141.5114.26

Second, we examine Attribute and Relation facts, which are harder to verify and closely related to safety interpretation (Section 4). TCG-RL achieves higher visual support rates than Thyme across all datasets, supporting improved grounding of fine-grained evidence.

  • Attribute and Relation Fact Accuracy
Qwen2.5-7B-VLHarm-PPrideMM
AttributeRelationAttributeRelation
Direct CoT87.1587.7286.2988.54
GRPO80.4987.1382.8487.54
Thyme86.9287.8485.2288.64
TCG-RL w/o TC94.7194.1986.6792.57
TCG-RL92.5994.9290.9194.18

Third, we re-evaluate the same rationales with an independent gpt-6.1-sol judge not involved in TCG-RL training. VES and SAS gains persist across all datasets, providing evidence that the improvements generalize beyond the training evaluator.

  • Evaluation by gpt-6.1-sol Judge
Qwen2.5-7B-VLHarm-PPrideMM
VESSASVRSSAS
ThymeVanilla judge76.272.077.565.3
gpt-6.1-sol judge77.172.380.465.0
TCG-RLVanilla judge97.076.895.878.5
gpt-6.1-sol judge98.074.597.378.1

Finally, three researchers blindly compared TCG-RL and Thyme on 100 paired examples from the four test sets. Based on the majority vote, TCG-RL achieved a 73% win rate. Together, these results support genuine improvements in rationale grounding rather than merely adaptation to the judge. We will include these analyses and the human evaluation protocol in the revision.

AuthorsOpenReview ↗

Response to Reviewer JCaQ (2/3)

W3: Baseline Coverage and Selective Tool Use

Thank you for these suggestions. Computational constraints currently prevent additional training experiments on larger backbones. We have nevertheless consolidated the complete comparison between TCG-RL and all evaluated baselines (described in Section 3) on Qwen2.5-VL-7B. The expanded comparison is consistent with the findings against Thyme: TCG-RL achieves the highest rationale reliability while maintaining leading classification performance.

  • Comparison with the full set of baselines
Qwen2.5-7B-VLHarm-PPrideMM
Acc.VESSASAcc.VESSAS
Direct CoT56.976.476.660.275.772.7
U-CoT+57.168.475.960.968.563.0
MIND57.278.471.864.774.562.0
ALARM60.374.676.064.370.866.4
Visual CoT59.275.673.660.275.671.4
DyFo56.974.278.163.174.571.9
ZoomEye57.572.976.260.965.569.1
Thyme65.176.272.074.677.565.3
TCG-RL65.397.076.876.895.878.5

We further compare three tool-use settings: Selective tools allows TCG-RL to decide whether and how to use tools; Always-tools requires at least one visual tool operation per input; and No tools disables visual tools. Tool-use Rate is the percentage of samples with at least one executed visual tool operation.

As shown in Table below, Selective tools outperforms Always-tools on both classification accuracy and rationale reliability. These results show that TCG-RL benefits from using visual tools appropriately, rather than simply invoking them more often.

  • Comparison of tool-use strategies
Qwen2.5-7B-VLHarm-PPrideMM
Acc.VESSASTool-use RateAcc.VESSASTool-use Rate
Selective tools (TCG-RL)65.397.075.878.276.895.878.572.6
No tools61.680.264.6072.983.762.70
Always-tools64.090.973.810074.086.372.6100

W4: Relationship to Existing Tool-Benefit Reward Methods

Thank you for highlighting CodeV/TAPO and AdaTooler-V. While CodeV/TAPO rewards relevant intermediate tool outputs and AdaTooler-V measures tool benefit through accuracy differences, TCG-RL rewards improved support for atomic visual facts in generated rationales relative to a fixed no-tool reference.

This objective addresses our central finding: correct safety predictions and relevant tool observations do not guarantee grounded safety rationales. Our contribution lies in systematically diagnosing this gap and optimize rationale grounding. The existing w/o TC ablation and the always-tools comparison support the value of the contrastive component and selective tool use. We will clarify these connections and add the relevant citations in the revision.

AuthorsOpenReview ↗

Response to Reviewer JCaQ (3/3)

W5: Coverage of the Tool-Use Analysis and Safety-Attribution Labels

Thank you for this suggestion. Since DyFo has a relatively autonomous tool-use policy, we use it as the main analysis case in the paper. We additionally provide results for ZoomEye and Visual CoT. For ZoomEye, the tool-use strategy is fixed, and the model only decides whether to invoke a tool. We therefore report the Tool-use rate. For Visual CoT, the cropping procedure is fixed, but the model determines the size of the cropped region, so we report the Focused Zoom rate.

  • ZoomEye Tool-use rate (%)
Tool useHarmCHarmPMAMIPrideMM
Qwen2.5-7B-VL4.88.816.541.0
Qwen2.5-32B-VL78.380.769.668.4
Qwen3-8B-VL1.71.21.55.4
Qwen3-32B-VL28.515.41.78.1
Qwen3.5-27B54.364.878.476.0
  • Visual CoT Focused Zoom rate (%)
Focused ZoomHarmCHarmPMAMIPrideMM
Qwen2.5-7B-VL51.346.647.752.8
Qwen2.5-32B-VL54.650.052.856.9
Qwen3-8B-VL29.121.631.146.3
Qwen3-32B-VL20.320.645.557.5
Qwen3.5-27B43.642.753.759.1

The low Tool-use rate of ZoomEye and the Focused Zoom rate of Visual CoT are consistent with the main finding in Section 4 that tool access alone does not ensure fine-grained grounding: models must also decide when and where to inspect, and how to use the resulting observations in their rationales.

Regarding the reference safety-attribution labels, Appendix A (Lines 702–732) specifies that these labels come from dataset annotations: harmful-target categories for Harm-C/Harm-P, misogyny types for MAMI, and stance labels for PrideMM. SAS is extracted and judged by an LLM judge, using Qwen3-32B. The independent reliability analysis of this judge is provided in our response to W1.

W6: Clarification of the Reported Gains.

Thank you for pointing this out. We rechecked Table 3 and clarified the comparison. Compared with to Thyme, TCG-RL improves VES by 23.1, 20.8, 17.1, and 18.3 percentage points on Harm-C, Harm-P, MAMI, and PrideMM, respectively, confirming an average improvement of 19.8 percentage points. Thus, the original “approximately 20%” refers to this average gain in VES. The corresponding average SAS improvement is 9.0 percentage points. Meanwhile, classification accuracy decreases slightly on MAMI (−1.0), while it actually improves on PrideMM (+2.2); overall classification performance remains comparable. We will scope the abstract’s claim to the reduction in visual evidence hallucination and report the improvement in safety attribution separately.

We will incorporate the clarifications and additional results into the revised manuscript. We thank the reviewer again for helping us improve the paper.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer JCaQ,

Thank you again for your thoughtful comments and constructive suggestions throughout the discussion. As the author–reviewer discussion period is coming to an end, we would like to kindly ask whether our responses have addressed your main concerns. If there are still aspects that remain unclear or require further clarification, we would be very happy to discuss them further.

We sincerely appreciate your time and valuable feedback.

Reviewer_Qme7OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on the hallucination problem in multimodal safety guards. It first presents a taxonomy and cause analysis of hallucinations in multimodal safety guards, illustrated through comprehensive experiments. Building upon these analyses, the paper proposes TCG-RL, which encourages the use of tools to obtain more meaningful grounding information. Experimental results demonstrate the effectiveness of this approach.

Critical strengths
  1. Rich experimental results. The extensive experiments provide a detailed demonstration of the hallucination problem in multimodal safety guards.
  2. Clear and fluent presentation. The paper clearly articulates the VES and SAS metrics, as well as the key takeaways derived from the experiments.
Critical weaknesses
  1. Ambiguous mathematical notation. In Section 5.1, three distinct function notations appear: Fatomic(τ,I)F_{\text{atomic}}(\tau, I), Fref(q)F_{\text{ref}}(q), and F(τ)F(\tau). Since these functions share the same base notation F(⋅)F(\cdot) yet accept different arguments, it is confusing whether they represent the same underlying concept, particularly in the case of F(τ)F(\tau).
Questions for authors

Regarding the TCG reward in TCG-RL, why is it formulated as a ternary discrete reward based on Δ(τ,q)\Delta(\tau, q), rather than directly introducing weighted λΔΔ(τ,q)\lambda_{\Delta} \Delta(\tau, q) within R(τ)R(\tau)?

4 comments

We thank the Reviewer for the careful and constructive review. We appreciate the reviewer’s positive assessment of our experimental analysis, presentation, and overall soundness. Below, we clarify the notation in Section 5.1 and explain the design of the ternary contrastive reward.

W1: Notation in Section 5.1.

We thank the reviewer for identifying this ambiguity. We use Fatomic(τ,I)F_{\mathrm{atomic}}(\tau,I) to denote the atomic-fact faithfulness score of the rationale generated by trajectory τ\tau for image II, and Fref(q)F_{\mathrm{ref}}(q) to denote the score of the fixed no-tool reference rationale for question qq. In Eq. 7, F(τ)F(\tau) is intended as shorthand for the same rollout faithfulness score Fatomic(τ,I)F_{\mathrm{atomic}}(\tau,I), rather than a distinct function. We agree that this shorthand is confusing and will use a consistent notation and clarify the definitions in the revision.

Q1: Ternary contrastive reward.

We thank the reviewer for this suggestion. The tolerance margin ϵ\epsilon maps small score differences to zero, reducing the influence of minor fluctuations, while the bounded ternary signal rewards sufficiently large improvements and penalizes sufficiently large degradations relative to the no-tool reference. We agree that a continuous reward such as λΔΔ(τ,q)\lambda_{\Delta}\Delta(\tau,q) is a reasonable alternative that preserves the magnitude of the change.

We also tested this continuous variant using the same training setup. As shown in the table, it remains competitive with the baselines, but the ternary-reward variant achieves higher VES and SAS on all four benchmarks, while their classification accuracies are similar. We will add this comparison and clarify the design rationale in the revision.

  • Comparison between the ternary and continuous contrastive rewards.
Qwen2.5-7B-VLHarm-CHarm-PMAMIPrideMM
Acc.VESSASAcc.VESSASAcc.VESSASAcc.VESSAS
Direct CoT66.473.380.756.976.476.671.481.475.960.275.772.7
GRPO74.071.280.661.474.869.476.679.975.969.675.672.4
Thyme78.971.479.965.176.272.078.380.770.774.677.565.3
TCG-RL w/o TC74.881.982.064.588.175.378.290.087.471.886.775.0
TCG-RL (continuous reward)77.693.480.364.994.172.377.494.284.176.090.874.6
TCG-RL78.094.582.765.397.075.877.397.886.076.895.878.5

We thank the reviewer again for raising these important points. We believe these clarification will make the reward formulation and its empirical motivation substantially clearer.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer Qme7,

Thank you again for your thoughtful comments and constructive suggestions throughout the discussion. As the author–reviewer discussion period is coming to an end, we would like to kindly ask whether our responses have addressed your main concerns. If there are still aspects that remain unclear or require further clarification, we would be very happy to discuss them further.

We sincerely appreciate your time and valuable feedback.

Reviewer_Qme7OpenReview ↗

Reviewer Response

Thank you for the additional experimental results and the detailed clarification of notation. Furthermore, I am curious about the specific value and selection criteria for λΔ\lambda_{\Delta} in TCG-RL (continuous reward). The authors' response has satisfactorily addressed my concerns, and I will raise my rating to 4.

Thank you for your positive assessment and for the follow-up question. In the continuous-reward variant, λΔ=0.5\lambda_{\Delta}=0.5, the same value as λcf\lambda_{\mathrm{cf}} in the ternary-reward setting. We used this value while keeping the other training settings unchanged. Thank you again for your thoughtful feedback.

Reviewer_aGsLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks into the Multimodal LLM hallucination issue, specifically from their generated rationales, rather than just the accuracy of the final safety judgements. The authors categorize two types of rationale hallucination: (1) visual evidence hallucination where the rationale involves some incorrect/hallucinated visual content, stated as if it is found in the image; (2) safety attribution hallucination where the rationale can be underflagging, overflagging or incorrectly flagging. Experiments on a set of Qwen models across 4 safety datasets reveal a persistent issue in both types of hallucination. Based on these findings the authors proposed tcg-rl, tool contrastive grounding RL, which contrasts the model's performance scores with tool (for visual cropping etc.) against the reference scores obtained with a fixed tool-free model, and the reward encourages the model to use tool when it is getting a better result doing so. Training results show that the proposed TCGRL effectively improves rationale reliability, while maintaining classification accuracy.

Critical strengths

S1: The work provides well-motivated empirical analysis of how current models possess rationale hallucination. The authors identify an important gap between classification scores and the actual correctness of generated rationales. Systematic experiments is conducted across multiple recent methods on rationale generation, a set of qwen models, and 4 safety datasets. The results show that a higher classification accuracy do not necessarily mean more reliable rationales. These findings strongly motivates the proposed method.

S2: This work includes relevant ablation studies supporting substantial gains obtained through the proposed methods. TCG-RL achieves substantial improvements in visual grounding scores across the datasets. The ablation provides evidence that the proposed reward contributes to the improvements beyond the other existing components. Thus the empirical contribution of the paper is solid.

Critical weaknesses

W1: LLM as a judge is heavily used during fact extraction and verification. If we are questioning the rationale reliability of the model under examination, why are we adopting llm as the main judge for the core decisions? Some ablation or human evaluation/alignment studies is missing here, we need to first show that these judges are fair and reliable.

Questions for authors

Q1: see W1, any additional studies that can show the reliability of llm to be used as a judge?

Q2: The claim that Scaling alone does not resolve rationale hallucination might be insufficiently supported if we are just looking at Qwen 2.5 VL 7B and 32B. Any additional results that can support this claim? consider other model architecture, or more size variants of qwen.

4 comments

We thank the reviewer for recognizing the motivation, empirical analysis, and ablation results of our work. We appreciate the questions about evaluation reliability and model scaling. We address both points below and will incorporate the additional validation and model results into the revision.

W1 & Q1: Reliability of LLM-based evaluation

Thank you for raising this concern. We use LLM judges because evaluating open-ended rationales requires identifying freely phrased factual claims and checking their visual support and safety attribution at scale. VES follows the atomic-fact verification framework of FaithScore (Jing et al., EMNLP 2024), which was evaluated against human judgments. We further validate our evaluators on safety rationales through human annotation and cross-judge comparison.

For VES, we randomly sampled 100 atomic fact–image pairs from outputs spanning four datasets and five tested models. For SAS, we sampled 100 rationale–image pairs covering over-flagging (OF), under-flagging (UF), and mis-attribution (MA). Three researchers independently annotated the 200 sampled cases. We also compared the Qwen evaluators with an independent gpt-6.1-sol judge from another model family. Table 1 reports inter-annotator agreement and agreement between evaluators.

  • Table 1: Human and cross-judge agreement for VES and SAS judgments
consistencyVESSAS-OFSAS-UFSAS-MA
Unanimous human agreement (%)88899392
Qwen–human majority agreement (%)94959690
Qwen–GPT agreement (%)89858785
kappa κ
Inter-human Fleiss’κ0.830.820.870.81
Qwen–human majority Cohen’s κ0.850.880.890.73
Qwen–GPT Cohen’s κ0.740.690.710.72

Qwen agrees with the human majority on 90%–96% of cases (κ = 0.73–0.89) and with the independent GPT judge on 85%–89% (κ = 0.69–0.74). Together with the high inter-human agreement, these results support the reliability of the sampled judgments in our evaluation setting. We will add the annotation protocol and validation results to the revised paper.

Q2: Does scaling alone resolve rationale hallucination?

We thank the reviewer for suggesting a broader test of this claim. We additionally evaluated three Gemma 4 models—gemma-4-E4B-it, gemma-4-12B-it, and gemma-4-31B-it—as well as qwen3.5-397b-a17b on Harm-C, Harm-P, and MAMI. Tables 2 and 3 report accuracy (Acc.), VES, and SAS across rationale-generation methods.

  • Table 2: Additional results for the Gemma 4 series.
Harm-CHarm-PMAMI
Acc.VESSASAcc.VESSASAcc.VESSAS
gemma-4-E4B-it
Direct CoT70.671.389.457.776.379.860.280.280.2
Visual CoT72.675.488.859.776.284.262.680.679.1
DyFo70.171.588.758.072.680.759.375.977.6
ZoomEye71.272.189.959.476.379.860.777.580.7
gemma-4-12B-it
Direct CoT64.169.371.755.871.578.064.377.879.7
Visual CoT64.169.481.357.571.485.066.679.675.9
DyFo63.867.578.355.868.184.961.472.977.2
ZoomEye64.769.977.856.370.879.063.277.472.3
gemma-4-31B-it
Direct CoT63.373.583.758.676.480.968.983.474.2
Visual CoT65.373.489.661.177.184.672.685.478.1
DyFo63.069.587.757.572.980.366.078.575.7
ZoomEye63.072.976.959.777.567.768.783.877.6
  • Table 3: Additional results for qwen3.5-397b-a17b.
qwen3.5-397b-a17bHarm-CHarm-PMAMI
Acc.VESSASAcc.VESSASAcc.VESSAS
Direct CoT67.266.289.460.070.188.276.480.189.3
Visual CoT68.669.092.063.470.791.874.778.985.2
DyFo67.566.092.059.768.186.271.676.291.1
ZoomEye63.667.192.661.470.187.371.775.386.6
U-CoT+83.956.695.960.059.384.863.071.885.2
MIND65.361.692.960.866.583.577.977.585.6
ALARM64.763.580.463.765.881.175.974.674.8

The Gemma 4 results show that visual grounding does not improve consistently across the three model sizes or reasoning methods. Hallucinated visual evidence also persists with qwen3.5-397b-a17b, further illustrating that a more accurate safety decision need not come with a better-grounded rationale. These additional results support our intended claim: across the evaluated settings, increasing model capacity alone does not consistently eliminate rationale hallucination.

We thank the reviewer again for these suggestions. We believe these clarifications will make the reliability of our evaluation and the scope of our scaling claim substantially clearer.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer aGsL,

Thank you again for your thoughtful comments and constructive suggestions throughout the discussion. As the author–reviewer discussion period is coming to an end, we would like to kindly ask whether our responses have addressed your main concerns. If there are still aspects that remain unclear or require further clarification, we would be very happy to discuss them further.

We sincerely appreciate your time and valuable feedback.

Thank you for the detailed responses, I believe my concerns are addressed, especially the llm-judge part where the agreement score showed some degree of reliability. I will keep my recommendation for accept.

Thank you for your thoughtful follow-up and continued support. We appreciate your time and constructive feedback.

Comments on the paper

1 comment

Dear ACs,

We would like to summarize the strengths of this work acknowledged by the reviewers, and the responses we have made to address the reviewers' concerns.


Strengths acknowledged by the reviewers:

  1. Important and underexplored problem. (Reviewers aGsL, JCaQ)
  • Reviewers agree that the paper identifies an important gap: correct safety predictions do not guarantee grounded rationales. The proposed hallucination taxonomy is intuitive and reusable.
  1. Comprehensive experiments and mechanistic analysis. (All Reviewers)
  • Reviewers praise the breadth of the experiments and the concrete root-cause analysis, which strongly motivates the proposed method.
  1. Effective method with supporting ablations. (Reviewers aGsL, JCaQ)
  • TCG-RL substantially improves rationale grounding, and the ablation isolates the contribution of the tool-contrastive reward.
  1. Clear presentation. (Reviewers aGsL, Qme7)
  • The paper is clear and easy to follow, with well-defined metrics and takeaways.

There are four main concerns raised by reviewers:

  1. Reliability of LLM judges and reward–evaluation circularity. (Reviewers aGsL and JCaQ)
  • Rebuttal: Human annotation shows strong agreement between our judge and human annotators (κ = 0.73–0.89). TCG-RL's gains persist under an independent held-out judge, and TCG-RL achieves a 73% win rate over Thyme in blind human comparison. Reviewer aGsL confirmed their concerns were addressed and maintained their accept recommendation.
  1. Generality of the findings. (Reviewers aGsL and JCaQ)
  • Rebuttal: We added Gemma 4 (E4B/12B/31B) and Qwen3.5-397B, along with tool-use analyses of ZoomEye and Visual CoT. All results are consistent with our original findings. Reviewer aGsL confirmed their concerns were addressed; Reviewer JCaQ did not respond.
  1. Baseline coverage and positioning of TCG-RL. (Reviewer JCaQ)
  • Rebuttal: TCG-RL outperforms all seven baselines from Section 3 as well as an always-use-tools variant. We also clarified how it differs from CodeV/TAPO and AdaTooler-V: TCG-RL rewards rationale-level atomic-fact grounding rather than tool outputs or accuracy gains. No further response was received.
  1. Notation and reward design. (Reviewer Qme7)
  • Rebuttal: We clarified the notation and showed that the ternary reward outperforms a continuous variant. Reviewer Qme7 confirmed that the concerns were satisfactorily addressed and raised the rating to 4.

We sincerely appreciate your valuable time!

Thanks and regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.