acceptodds
Under review as a conference paper at ICLR 2027

Tool Mediation Alters Refusal Mechanisms in Large Language Models

Abstract

Large language models (LLMs) are increasingly deployed with access to external tools, yet harmful tool-mediated interactions are less likely to be refused when compared to regular conversational ones. As this change in refusal behavior remains underexplored, we investigate its underlying mechanisms across a diverse set of open-weight language models. We find that information about the harmfulness of a request remains strongly encoded in the model's representations and transfers across conversational and tool-mediated inputs. Evidence from representation geometry and neuron-level analysis further indicates that the two interaction modes systematically distribute harm-related computation differently. Crucially, while conversational inputs can be refused at relatively low levels of perceived harmfulness, tool-mediated inputs remain permissive until harmfulness crosses a substantially higher effective refusal threshold. Moreover, tool-mediated refusal is also more brittle: progressively weakening the refusal computation disrupts tool-mediated refusal at lower intervention strengths than conversational refusal, even when benign capabilities remain intact. Together, our findings indicate that tool mediation does not simply reduce the internal perception of harm, but instead impacts its conversion into refusal. Overall, this suggests tool-mediated environments may intrinsically reduce robustness of models to harmful requests, and that conventional safety evaluations may not fully transfer to LLM agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models (LLMs), Alignment, AI Safety, Interpretability, Agentic AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Tib9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates differences in safety refusal between conversational and tool-calling settings. Across eight open-weight models, the authors find that models refuse harmful requests less frequently when tools are available. Through representation analysis and intervention experiments, they further argue that tool mediation preserves information about harmfulness but changes how that information is translated into refusal. The research question is important, and the paper provides substantial evidence of a behavioral gap. However, I find the evidence for the proposed mechanistic explanation less conclusive. My main concerns involve the comparability of refusal measurements, the causal connection between the identified internal components and refusal behavior, and the controls used to isolate the effect of tool exposure.

Critical strengths
  1. Important and practically relevant research question. As models increasingly perform actions through tools, understanding whether conversational safety transfers to tool use is valuable.
  2. Broad behavioral evaluation. The study covers eight models and compares different presentations of matched intents. The consistent reduction in tool-side refusal suggests that the phenomenon deserves attention.
  3. Useful controls and supplementary analyses. Multiple directive paraphrases, cross-fitting by intent, matched random-neuron controls, multi-turn evaluation, and the PrivacyLens experiments strengthen the study. The authors also acknowledge capability limitations and exceptions across architectures.
Critical weaknesses
  1. Refusal is measured differently across the two settings

According to Section 4 and Appendix B.2, conversational refusal is primarily assessed by an LLM judge, whereas tool-side compliance is primarily determined by whether the model calls a target function. These procedures do not necessarily measure the same behavior. For example, the absence of a target call may reflect refusal, insufficient tool-use capability, or malformed output. Conversely, a target-function call should be evaluated in context to establish whether it attempts a harmful action. The measured gap may therefore combine differences in safety behavior with differences in execution capability and output format. The capability-related analysis and multi-turn evaluation are helpful. However, because refusal labels also inform the subsequent representation and threshold analyses, a common evaluation standard is needed. I would like to see both settings assessed using the same categories: explicit safety refusal, harmful assistance, capability failure, and formatting failure, with harmful tool-call attempts reported separately.

  1. Changes in internal signals do not yet establish changes in the refusal mechanism

In Section 5.1 and Tables 15–16, the main outcome of the neuron-ablation experiments is the change in the final-layer harmfulness projection. This establishes that the selected neurons contribute to the measured representation, but it does not directly establish that they determine whether the model refuses a harmful request. An alternative explanation remains possible: tool mediation changes how information is encoded, producing different neuron contributions and projection values, while the computation responsible for refusal remains substantially shared. Appendix C.7 does include directional interventions that measure refusal behavior. However, these interventions differ from the neuron sets identified earlier, leaving a gap between locating channel-related components and demonstrating their behavioral role. The mechanistic claim would be stronger if ablating the identified neurons selectively changed refusal in the corresponding channel while preserving benign tool-use capability. A restoration or activation-patching experiment would provide additional support, although a clear behavioral validation of the existing neuron sets would already be valuable.

  1. The effects of tool availability and execution instructions are not fully separated

The system prompt for tool exposure₁ in Table 7 already instructs the model to issue tool calls when appropriate. Results under this condition therefore provide limited support for the claim that tool exposure triggers the shift before the model is instructed to act. The tool exposure₂ condition removes this explicit instruction, which is a useful improvement. However, the paper lacks a direct no-tool control using exactly the same system prompt as exposure₂. Moreover, the main representational analysis in Figure 3 is based on exposure₁. A more controlled comparison would hold the system prompt and user instruction fixed while changing only tool availability. Repeating the key behavioral and representational analyses under this comparison would clarify how much of the observed shift can be attributed to tool exposure itself.

Questions for authors
  1. Does the main refusal gap persist when both settings are evaluated under a common annotation scheme distinguishing safety refusal, harmful assistance, capability failure, and formatting failure? How are tool-side outputs without a target call classified?

  2. How does ablating the identified channel-related neuron sets affect actual refusal and harmful tool-call rates? Are these effects channel-selective, and do they occur while benign tool-use capability remains intact?

  3. What happens in a no-tool condition using exactly the same system prompt as tool exposure₂? Can the representational findings in Figure 3 be reproduced under this more controlled comparison?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Tib9 (1/3)

We thank the reviewer for taking the time to provide detailed feedback on our work. We appreciate the helpful suggestions and address each of the points raised below.

Common annotation scheme. We re-graded harmful intents using Qwen3.5-27B as an LLM judge, applying the same evaluation procedure to conversational and tool-mediated outputs. The refusal drop reported in Table 2 persists across all eight models and is close to the results obtained using our original grading.

ModelPaperQwen3.5-27BAgreement (C)Agreement (T)
Qwen3-8B−0.43−0.490.981.00
Qwen3-32B−0.67−0.700.981.00
Llama-3.1-8B−0.84−0.800.971.00
Llama-3.1-70B−0.28−0.250.971.00
Ministral-3-8B−0.28−0.180.871.00
Gemma-4-12B−0.15−0.200.971.00
Gemma-4-31B−0.27−0.310.981.00
gpt-oss-20b−0.20−0.200.991.00

We also extend the capability comparison in the paper to cover both conversational and tool-mediated outputs, distinguishing the categories requested, using both the original grading procedure used in the paper, as well as Qwen3.5-27B instructed to distinguish between those categories, with the following results:

Original grading, Conversational → Tool:

ModelExplicit refusal (harm+both)Capability (cap+both)Bare refusal (none)Harmful assistanceFormat failureHarmful tool-call attempt (T)
Qwen3-8B0.96→0.530.13→0.020.01→0.000.01→0.460.02→0.040.46
Qwen3-32B0.54→0.170.08→0.010.43→0.120.01→0.700.00→0.080.71
Llama-3.1-8B0.29→0.020.03→0.000.65→0.080.05→0.900.03→0.070.90
Llama-3.1-70B0.18→0.250.04→0.010.71→0.360.09→0.390.04→0.050.39
Ministral-3-8B0.69→0.400.09→0.010.03→0.060.26→0.550.06→0.110.55
Gemma-4-12B0.93→0.780.25→0.000.00→0.000.01→0.220.01→0.030.22
Gemma-4-31B0.95→0.670.30→0.000.00→0.000.01→0.330.01→0.010.33
gpt-oss-20b0.99→0.790.00→0.000.01→0.010.00→0.210.02→0.060.21

LLM judge, Conversational → Tool:

ModelExplicit refusalCapabilityBare refusalHarmful assistanceFormat failureHarmful tool-call attempt (T)
Qwen3-8B0.96→0.520.05→0.010.00→0.000.03→0.470.00→0.000.48
Qwen3-32B0.52→0.170.04→0.000.43→0.100.02→0.730.00→0.000.73
Llama-3.1-8B0.38→0.020.02→0.000.52→0.060.08→0.910.00→0.000.92
Llama-3.1-70B0.24→0.350.03→0.000.62→0.250.11→0.390.00→0.000.39
Ministral-3-8B0.61→0.390.02→0.000.00→0.050.37→0.560.00→0.000.56
Gemma-4-12B0.93→0.780.18→0.000.00→0.000.01→0.220.00→0.000.22
Gemma-4-31B0.94→0.670.23→0.000.00→0.000.02→0.330.00→0.000.33
gpt-oss-20b0.78→0.710.01→0.000.20→0.070.02→0.220.00→0.000.22

Results show in both procedures that the reduction in explicit safety refusals in the tool-mediated setting is accompanied by a corresponding increase in harmful assistance, while capability and formatting failures remain negligible and cannot account for the observed refusal drop. We will extend Appendix B.2 to include these additional evaluations and the corresponding categorical taxonomy.

Tool-side outputs without a function call. We classify those instances as refusal if they contain an explicit refusal statement, and otherwise as a format failure, which we exclude for the analysis. Counting all indeterminate outputs as refusals raises tool-side refusal rates by at most 0.06, still below conversational refusal rates for every model. Moreover, the new LLM judge evaluation classifies 89–100% of these outputs as harmful assistance for all models except Ministral-3-8B (54%), so their exclusion might understate the refusal gap. We will clarify this rule in the revised manuscript.

AuthorsOpenReview ↗

Reviewer Tib9 (2/3)

Behavioral effects of neuron experiments. We agree that testing the identified neurons behaviorally would complement the representational results in the paper. We conducted two additional experiments to further examine this: (1) an ablation, and (2), a restoration where we restore activations to the corresponding conversational mean values. In both cases, we intervene on the identified tool-selective and conversation-selective populations (K=8000, four late MLP layers) and compare the refusal rates in the tool channel against a random control set. We present the results for the smaller models in the following table, where we show refusal rates before and after intervention (on all intents, including those with execution hints) and the resulting benign capability after the intervention. We omit the ablation for Llama-3.1-8B as its baseline refusal rate is 0.05.

ModelBaseline refusalAfter ablationAfter restoration
RandomTool-sel.Conv-sel.BenignRandomTool-sel.Conv-sel.Benign
Llama-3.1-8B0.05----0.050.080.160.98 / 0.98
Qwen3-8B0.280.270.200.220.99 / 0.990.270.350.430.99 / 0.98
Ministral-3-8B0.340.340.300.361.00 / 1.000.370.380.531.00 / 0.99
Gemma-4-12B0.630.630.150.640.34 / 0.950.640.920.640.46 / 0.97

Restoring the conversation-selective population increases refusal in all three 8B models. Conversely, ablating the tool-selective population reduces refusal in Qwen3-8B and Ministral-3-8B, and collapses benign capability in Gemma-4-12B (we also tested this model at K=800, where there is a 0.07 drop in refusal without capability degradation). Benign tool execution remains largely unaffected across interventions, except for tool-selective interventions in Gemma-4-12B, where there is a collapse in benign tool execution. Taken together, these results indicate that, at least in the smaller models, the channel-specific populations also play a role in tool-channel refusal, and that tool mediation does not simply weaken a single shared population, but instead relies partly on a different one.

Ablating the conversational channel leaves refusal unchanged, consistent with the robustness of conversational refusal in Appendix C.7 and Table 4. In the larger models, we do not observe major changes in refusal from these interventions. A fixed set of 8,000 neurons in four layers covers a smaller fraction of the network in these models, which limits the strength of the intervention. We will add these experiments in the appendix, summarise them in Sections 1 and 5.1, and state their scope explicitly in the Abstract and Introduction.

AuthorsOpenReview ↗

Response to Reviewer Tib9 (3/3)

Effect of execution instructions in refusal. We ran the requested control, considering the exact system prompt from tool-exp₂ with no tools attached. Across all eight models, refusal in this condition remains close or higher than the conversational level, and attaching tools reduces refusal in every model. This effect is comparable to that under tool-exp₁, suggesting that the explicit instruction to execute tool calls in the system prompt is not necessary for the behavioral shift.

ModelReferencetool-exp₂ prompt, no toolstool-exp₂ prompt, with toolsΔ\Delta tool-exp₂Δ\Delta tool-exp₁ - agentic
Qwen3-8B0.9710.9930.787−0.206−0.245
Qwen3-32B0.9680.9850.811−0.174−0.239
Llama-3.1-8B0.9410.9860.551−0.435−0.568
Llama-3.1-70B0.8920.9730.715−0.258−0.332
Ministral-3-8B0.7250.7710.642−0.129−0.271
Gemma-4-12B0.9350.9670.871−0.095−0.090
Gemma-4-31B0.9450.9600.857−0.103−0.131
gpt-oss-20b0.9980.9950.889−0.106−0.155

Representational results. We repeated the analyses from Figure 3 using tool-exp₂, with the results are shown in the table below. The findings are consistent with Figure 3 (despite tool-action using the tool-exp₁ prompt), and confirm that the representational findings also hold without the explicit instruction to execute tool calls in the system prompt.

ModelExposure fraction, tool-exp₁ (Figure 3)Exposure fraction tool-exp₂Axis agreement ÷ ceiling, exposure₁ / exposure₂Transfer AUROC, exposure₁ / exposure₂
Qwen3-8B0.760.650.87 / 0.850.98 / 0.98
Qwen3-32B0.710.670.58 / 0.450.78 / 0.74
Llama-3.1-8B0.860.820.87 / 0.840.99 / 0.99
Llama-3.1-70B0.720.480.81 / 0.771.00 / 1.00
Ministral-3-8B0.800.690.95 / 0.920.92 / 0.91
Gemma-4-12B0.630.530.90 / 0.870.98 / 0.99
Gemma-4-31B0.600.500.92 / 0.890.97 / 0.98
gpt-oss-20b0.780.640.96 / 0.940.95 / 0.95

We additionally decompose the fractions relative to the matched no-tool condition (tool-exp₂ system prompt without tools) to further isolate the representation impact of tool availability. Attaching tools is the dominant effect to the representational shift in all models, although the agentic framing already shifts the representation by 4–26% in some. Overall, this indicates that exposure is the primary driver of the representational shift, while the agentic framing makes a smaller, model-dependent contribution.

Modeltool-exp₂ prompt, no toolstool-exp₂
Qwen3-8B−10%65%
Qwen3-32B−7%67%
Llama-3.1-8B−10%82%
Llama-3.1-70B−20%48%
Ministral-3-8B4%69%
Gemma-4-12B26%53%
Gemma-4-31B15%50%
gpt-oss-20b26%64%

We will add the behavioral and representational results in Appendix C.2. We will also state that the observed representational effects persist without the explicit instruction to execute tool calls in the system prompt, while noting the contribution of agentic prompt framing in some models.

Reviewer_aonVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper conducts a mechanistic study of how LLMs' refusal behavior varies by interacting contexts' tool accessibility. The authors present mechanistic evidence that

  • LLM-perceived harm (mechanistically extracted harm directions) are encoded differently for tool-use and conversational contexts
  • the LLMs' refusal behavior is influenced by the LLMs' internally perceived harm, but the threshold of perceived harm to gate refusal is different for tool-use and conversational contexts
  • moreover, the refusal in tool-mediated contexts is less robust to adversarial interventions to bypass the refusal
Critical strengths

Strength 1. Soundness

  • The paper's mechanistic approach and interpretation are largely sound.
  • The test of multiple experimental conditions (e.g., Agentic framing, Tool exp.2, Tool exp.1, and Tool action) was particularly insightful.
  • The empirical breadth of using eight open-source LLMs of varying families adds reliability to the analyses.
  • The comprehensive mechanistic ablation studies adds causal rigor to the analyses

Strength 2. Novel findings on an important topic

  • The finding that refusals in tool-mediated and conversational contexts may have different effective refusal thresholds in LLM internals was particularly interesting.
  • The set of evidence that harm directions may be encoded differently for tool-mediated and conversational contexts is also interesting.
  • The findings' direct implication addresses an important topic in AI safety
Critical weaknesses

In my view, the only major weakness is the limited scope of the contribution.

  • The studied phenomenon is not strictly new, as it has been repeatedly reported in previous studies. The authors did not provide a new set of tools or frameworks to study the phenomenon.
  • This leaves the new mechanistic explanation as the primary contribution.
  • While I agree that the findings are important, I believe a mechanistic explanation being the only major contribution is an important weakness. Because a mechanistic approach's role is largely explanatory, gaining useful insights on how to address the studied phenomenon is also non-trivial.

A stronger version of this paper could, e.g.,

  • (1) present a new solution based on the mechanistic findings
  • (2) provide a deeper analysis of why the phenomenon occurs (e.g., attributing the cause to the post-training regime)
  • or (3) explore a new important behavioral phenomenon, driven by the mechanistic findings
Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer aonV

We thank the reviewer for the positive assessment, and we are glad they found the condition decomposition insightful and the finding of distinct refusal thresholds across channels interesting.

We agree that prior work has reported text-versus-tool safety gaps and that we do not propose a complete mitigation. Nevertheless, our study goes beyond earlier behavioral reports and yields novel empirical and mechanistic findings. Through paired decomposition, we provide direct evidence that tool exposure alone accounts for much of the refusal drop. Furthermore, our mechanistic analysis reveals that the gap reflects shifted refusal thresholds and greater fragility, keeping the recognition of harm intact. This explains why current text-based safety evaluations fail to predict agentic behavior. Isolating the impact of tool exposure and uncovering the internal mechanism behind refusal failure offer actionable guidance for both safety benchmark design and targeted mitigations.

We also agree that turning these insights into interventions is an important next step, and we ran two proof-of-concept experiments to illustrate potential applications of our findings. As the first one, and in response to complementary reviewer feedback (in our response to Reviewer Tib9), we ran a new intervention experiment showing that patching the identified neuron profiles can partially restore refusal in the tool channel for smaller models. This preliminary result suggests that our findings could inform patching-based approaches to defensive steering and behavioral analysis.

The second proof-of-concept attempts to build a monitor of harmful requests in the tool channel that doesn’t require harmful agentic traces. It builds on our findings in Section 5.1, showing that harm representations persist under tool mediation, even when refusal is degraded. For that purpose, we use the harm direction, estimated on conversational requests only. However, since tool mediation shifts the harm representation (Appendix C.5), we also calibrate the detection threshold using benign agentic traces. We evaluate it on held-out behaviours, with about 500 harmful and 400 benign tool requests per model (excluding Llama3-70B). As a reference, we include a logistic regression probe trained directly on the tool channel, which has access to harmful agentic traces, as well as an uncalibrated conversational probe. The calibrated monitor's detection rate at the chosen FPR appears higher than the model's refusal rate in most models, although the two are not directly comparable as they are evaluated on different subsets of intents, and it performs similarly to a probe fitted in the tool channel directly.

ModelTool refusal[email protected]: Conv. probe / tool-trained[email protected] Conv. probe (conv. threshold)
Qwen3-8B0.540.80 / 0.770.46
Llama-3.1-8B0.100.61 / 0.730.21
Ministral-3-8B0.450.68 / 0.760.31
Gemma-4-12B0.790.78 / 0.780.77
Qwen3-32B0.300.78 / 0.830.12
Gemma-4-31B0.680.79 / 0.630.32
gpt-oss-20b0.810.67 / 0.700.16

We emphasise that neither experiment is envisioned as a complete deployable defense. Both rely on established techniques, are evaluated at limited scale, and have not been tested against adaptive attacks. Their purpose is rather to illustrate how the mechanistic account developed in the paper can inform further research. The finding that refusal degrades while harm representation is preserved motivates focusing on interventions on the conversion of harm into refusal, or on the harm representation itself, which remains accessible in both channels. Likewise, tool mediation shifting internal representations globally identifies what must be adapted when safety mechanisms developed for conversational settings are applied to agents. In the revised manuscript, we will report both experiments in the appendix, and discuss them in Section 6 as directions for future work.

I thank the authors for engaging with my concerns. I believe the paper is broadly solid, interesting, and timely, and the new experiments are promising. I maintain my positive review.

Reviewer_Q2FHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why open-weight LLMs refuse harmful requests less often through tools than in conversation. Using AgentHarm requests and eight models, the authors decompose the conversational-to-tool transition into cumulative conditions (agentic framing, tool exposure, tool action) and probe the two channels (conversational, tool-mediated) with harm directions, neuron ablations and GCG (Greedy Coordinate Gradient) attacks. They report that tool exposure alone accounts for a substantial part of the refusal drop, and that harm remains decodable in both channels. The authors conclude that the tool channel applies a higher refusal threshold to a preserved harm signal, and that its refusal is more brittle.

Critical strengths
  • The paper addresses a question of direct relevance to the safety of LLM agents: whether refusal relies on the same internal mechanisms in the conversational and tool-mediated channels. Holding the weights and the request fixed while changing only the delivery channel, across eight models from five families, is a controlled design that can be reused for other studies.
  • The behavioral decomposition is a clear and practically relevant result. Adding the conditions one at a time shows that agentic framing alone barely changes refusal, whereas tool exposure lowers refusal significantly in all eight models, even when the system prompt does not ask for a tool call. This suggests that refusal evaluations conducted only in the conversational setting may overestimate the safety of tool-using agents.
Critical weaknesses

-The evidence does not yet establish that tool mediation changes how harm is converted into refusal rather than reducing perceived harm. The harm and refusal directions are nearly collinear (in sec. 5.1), so the harm projection is close to a refusal readout, and fig. 2 cannot separate perceived harm from the refusal state as the x-axis is not a clean measure of harm. Tool renderings also project lower on the harm direction for most intents, although random directions show a comparable displacement. In five of eight models, the conversational 50% threshold lies below the observed range, so the threshold comparison is not observed in those models, and the paper does not define how Δˉ\bar{\Delta} is computed. A per-sample regression of refusal on channel and harm projection would test the claim directly.

  • The claim that tool-mediated refusal is more brittle is stated more strongly than the evidence supports. It is mainly based on directional ablation, which section 5.2 itself treats as suggestive evidence, so in practice the claim rests on the GCG experiment. That experiment attacks 8 to 12 intents per model with a single seed and reports no confidence intervals, and the two channels are optimized toward different targets. More intents and seeds and comparable targets would make the comparison interpretable.
Questions for authors
  • How are the fig. 2 curves and 50% thresholds estimated, and does the threshold displacement Δˉ\bar{\Delta} persist in the tool exposure$_2$ condition, which has no tool-call instruction, or at a layer where the harm and refusal directions are less collinear?
  • For the GCG experiment, what criterion defines a broken refusal, and what is the exact number of intents attacked for each model?
  • The tool exposure and tool action conditions are scored by parsing tool calls, whereas the reference and agentic framing conditions are scored by an LLM judge. How large is the refusal drop when the same judge scores all conditions?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Q2FH (1/2)

We thank the reviewer for their thoughtful and constructive feedback. We have conducted additional analyses and expanded our experiments to address the concerns raised. We respond to each point below and summarize the corresponding changes to the manuscript.

Figure 2 and 50% thresholds. Figure 2 shows isotonic regressions of refusal probability on harm projection, with the 50% threshold marking the harm reading at which the model refuses half of the requests. Δ\Delta is the refusal gap (conversational minus tool), computed over all harmful AgentHarm intents, including those with specific tool-call hints (unlike the conservative values in Table 2 excluding hinted intents). We will further clarify these definitions in the manuscript. In some models, as noted by the reviewer, the conversational refusal threshold lies below the plotted range, meaning that the actual shift to the tool refusal threshold is even larger than shown. Importantly, the tool refusal threshold is higher in all models, indicating that a stronger harm representation is required to trigger refusal in the tool channel in all cases.

Analysis on tool-exp₂ and less collinear layers. We extend our analysis as requested under both conditions: (1) tool-exp₂, where tools are available but the system prompt does not instruct the model to call them, and (2) seven probed layers per model, including those where harm and refusal directions are less collinear. The tool threshold remains higher in every model under tool-exp₂ and at every probed layer. The table reports the threshold shift (shift in projection onto the harm direction at the 50% threshold) under tool-exp₂ and at each model's least collinear layer:

ModelShift tool-exp₂ (SD)Layer (cosine sim)Shift at that layer (SD)
Llama-3.1-8B2.226 (0.55)≥ 4.8
Qwen3-8B≥ 1.8711 (0.63)≥ 4.6
Qwen3-32B≥ 1.6720 (0.73)≥ 4.3
Ministral-3-8B1.0812 (0.51)≥ 2.4
Llama-3.1-70B1.5715 (0.47)≥ 1.2
Gemma-4-12B≥ 1.8517 (0.80)≥ 1.2
Gemma-4-31B≥ 1.8821 (0.68)≥ 0.3
gpt-oss-20b≥ 0.4012 (0.77)≥ 0.7

"≥" indicates a lower bound, as the conversational threshold lies below the observed range. The shift persists under tool-exp₂ and at earlier layers that are less collinear, showing that the Figure 2 result is not an artifact of harm–refusal direction collinearity. We emphasize that this does not challenge the use of harm projection as a proxy for refusal in our other analyses, but rather shows that the observed channel difference persists even when the two directions are less aligned. We will add both analyses to the appendix.

Per-sample regression. We ran the per-sample regression suggested by the reviewer (to be included in the appendix). Comparing the same intents at matched harm projections, the tool channel exhibits lower refusal rates in all models (with gaps of 0.20–0.87). When each rendering is evaluated at its own harm projection, the gap persists in Qwen3-8B, Qwen3-32B and Llama-3.1-8B (0.19–0.34), but is small (≤ 0.05) or slightly reversed in the other models. Importantly, this second comparison does not match the same intents, as tool renderings project lower on the harm direction, they are compared against conversational intents perceived as more harmful. As the reviewer notes, this downward shift also occurs along random directions and is not specific to harm. Together with the threshold results under tool exposure and at less collinear layers, these findings suggest that tool mediation changes how harm translates into refusal, and reduced perceived harm may contribute in some models, but does not fully explain the effect in al scenarios. We will therefore revise our "rather than a reduction in perceived harm" claim in Section 5.1 to "not only a reduction in perceived harm."

AuthorsOpenReview ↗

Response to Reviewer Q2FH (2/2)

Expanded GCG evaluation. The experimental details of our GCG experiment are provided in Appendix B.4. We use the same methodology for evaluating refusal as in our other experiments: in the tool channel, a refusal is considered broken when the model executes the targeted tool, while in the conversational channel we rely on an LLM judge. We acknowledge that the optimization targets differ between channels, but this is intentional and reflects their different output modalities. In the tool channel, success requires actual tool execution, and in the conversational channel, following standard GCG practice, success is defined as generating a response judged not to be a refusal. We agree that evaluating more intents and seeds strengthens the analysis. In our original experiment, we selected one intent from a reduced set AgentHarm base behaviors, leading to 8–12 intents per model. We now have expanded to a broader set of 20 AgentHarm behaviors, with several intents per behavior, and now report results across three random seeds. The number of eligible intents is constrained by requiring default refusal in both channels, and by computational constraints. We also corrected a bug in the GCG evaluation of Ministral-3-8B. The results are consistent with our original findings, further supporting the conclusion that refusal is more brittle in the tool channel.

Model# Intents (# behaviours)Break ≤40, TBreak ≤40, CMedian TMedian C
Qwen3-8B25 (20)92%26.7 ± 10.1%640+
Qwen3-32B18 (15)92.6 ± 3.2%1.9 ± 3.2%1240+
Llama-3.1-8B12 (7)88.9 ± 9.6%2.8 ± 4.8%440+
Llama-3.1-70B27 (20)100%0%440+
Ministral-3-8B27 (20)96.3 ± 3.7%19.8 ± 10.7%640+
Gemma-4-12B26 (20)6.4 ± 2.2%0%40+40+
Gemma-4-31B26 (20)28.8 ± 8.2%0%40+40+

Common-judge evaluation. We re-graded every harmful intent using Qwen3.5-27B as an LLM judge applied identically to conversational and tool-mediated outputs. We report below the change in refusal rate from the conversational reference to tool action using both our original grading and the LLM judge. The refusal gap persists in all eight models under the LLM judge, and is close to the results obtained with our original channel-specific evaluation (agreement with our original grading is 0.97-1 in all models, with the exception of Ministral-3-8B at 0.87), indicating that the observed refusal gap is not an artifact of using different evaluation procedures across channels. We will include the additional evaluation and results in the appendix.

ModelPaperQwen3.5-27BAgreement (C)Agreement (T)
Qwen3-8B−0.43−0.490.981.00
Qwen3-32B−0.67−0.700.981.00
Llama-3.1-8B−0.84−0.800.971.00
Llama-3.1-70B−0.28−0.250.971.00
Ministral-3-8B−0.28−0.180.871.00
Gemma-4-12B−0.15−0.200.971.00
Gemma-4-31B−0.27−0.310.981.00
gpt-oss-20b−0.20−0.200.991.00

Comments on the paper

1 comment
AuthorsOpenReview ↗

Meta-Response & Summary of Changes.

We sincerely thank the reviewers for their thoughtful and constructive feedback. Following the reviews, we addressed all the points raised by conducting additional experiments, expanding our analyses, and clarifying several aspects of our methodology and interpretation.

We summarize the main changes below (the full details and results are provided in our responses to each review).

First, we strengthened the robustness of our behavioral evaluations. We introduced a unified LLM-based judge across both settings to test for evaluation artifacts and we expanded our safety analyses with categorical breakdowns of refusals. These results further support our conclusion that the refusal gap is not an artifact of grading, and reflects substantive differences in safety behavior across channels rather than execution or formatting limitations. Moreover, we expanded our GCG evaluations with additional intents and runs across multiple random seeds, further strengthening the robustness of our results.

Second, we reinforced our behavioral and mechanistic claims regarding the effect of tool exposure on refusal behavior through additional controls that replicate our findings. We additionally conducted further neuron ablation and restoration analyses to investigate the distinct roles of tool and conversation-selective neurons.

Finally, we expanded the discussion of the broader implications of our findings, outlining avenues for future research, extending our mechanistic insights to the study of safety defenses.

Thanks to the reviewers’ feedback, we believe the study is now considerably stronger, and we will update the manuscript to incorporate these changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.