acceptodds
Under review as a conference paper at ICLR 2027

Tool-Call Safety Evaluation Depends on What Labels Represent

Abstract

A tool-call safety monitor scores proposed calls before execution. Evaluating those scores against eventual attack success can credit or penalize the monitor for later actions. We distinguish whether the pending calls complete the attacker's goal from whether the full run succeeds, applying the benchmark's official success check immediately before and after the calls. These labels disagree on 19.6% of validated public AgentDojo decisions and 26.2% for two newly run agents, mostly because completion occurs later. With predictions fixed, changing the label moves TS-Guard's new-system AUROC from 0.628 to 0.841 and changes held-out monitor selection. ToolSafe's step annotations on a separate benchmark reproduce the sensitivity. On the Qwen agent, at the compared operating points, the locally selected monitor completes 6.1 percentage points more user tasks without attack success than the terminally selected monitor; its attack success is 1.1 points higher, with an interval covering zero. A blocking-disabled arm separates blocking from execution configuration. The event assigned to a score is part of monitor evaluation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, tool use, prompt injection, safety monitoring, evaluation methodology, evaluation targets

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JRBROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at how the choice of evaluation labels affects tool-call safety monitors in prompt injection settings. It distinguishes between whether the current calls complete an attack and whether the full run eventually succeeds, showing that this choice can really change monitor scores and which monitors get selected. It also tests the chosen monitors in blocking experiments, where the safety-utility tradeoff depends on the agent and the threshold used.

Critical strengths

I like how the paper draws a clear line between predicting eventual attack success and identifying calls that actually complete an attack. Keeping the predictions fixed and changing only the labels makes that especially obvious.

The analysis doesn’t stop at metric shifts, it checks monitor selection and calibration, then tests the selected monitors in actual blocking experiments.

The blocking-disabled control is a solid design choice, since it isolates the effect of blocking from any changes in the execution setup.

Critical weaknesses

Here are my main concerns:

  • the contribution feels a bit narrow. The paper does a good job showing that evaluating different events can affect monitor scores and selection, but the broader takeaway isn’t fully clear. It doesn’t really establish when local completion is the better defense target, especially since earlier attempts that don’t fully complete an attack may still be worth blocking. Does this only apply to monitors in prompt injection?

  • The practical implications are underdeveloped. The local labels depend on knowing the attacker’s goal, having an executable success checker, and being able to replay the environment. Those are useful for benchmarks, but often aren’t available in production, and the paper doesn’t give much guidance on how to use these findings for monitor selection or calibration in practice.

  • The motivation for the defense setting could be stronger. The paper assumes a pre-execution monitor without really explaining when this is preferable to other prompt injection defenses. If the broader point is about evaluating monitors, it would also help to say whether the framing applies to loss-of-control settings, where the agent itself is pursuing an unwanted objective instead of following injected instructions.

Questions for authors

see above

1 comment

Thank you for your thoughtful feedback on the paper's contribution, practical implications, and scope.

Comment to W1

We agree that earlier calls can deserve blocking even when they do not complete an attack. We do not establish local completion L as a better target for choosing which calls to block. Our contribution is a way to distinguish recognition of immediate completion from forecasting eventual success, and evidence that this distinction changes how monitors are evaluated.

For example, reading an injected message may enable a later leak while having L=0. A monitor could correctly recognize that the read does not complete the goal, yet receive a positive terminal label Y because the leak happens later. This does not make Y wrong for forecasting, or the read safe to allow. The benefit of blocking it needs separate execution tests.

We constructed L using official goal checks before and after the pending calls. With TS-Guard scores fixed, new-agent AUROC was 0.628 against Y versus 0.841 against L: +0.213 [0.085, 0.327], persisting after excluding source runs with native Mistral call text. Repeating monitor selection and calibration under system resampling gave excess L loss 0.224 [0.133, 0.325] from using Y. The interval included zero when tasks and goals were also resampled. Changing from Y to L did not consistently favor a different monitor across candidate pools. Brackets are 95% intervals under the respective procedures. We believe these results make the distinction useful for assessing what capability a reported score actually supports.

The issue also arose without an attacker: MANTRA's operating-rule checks gave terminal/local disagreement on 9-30% of decisions. Its monitors remained weak, and selecting using L did not improve on selecting using Y and recalibrating to L. This supports applicability beyond injection, without establishing better prevention.

Comment to W2

You are right that exact L is unavailable without a known goal and executable checker. Where those are missing, we recommend selecting and calibrating for an event that can be labeled, such as a specified operating-rule violation, and stating what those labels measure. We do not solve unknown-goal labeling in production.

The practical steps are to define the event and eligible calls, obtain checker-based or clearly identified proxy labels, select and calibrate on development data, choose thresholds on separate validation data, then measure executed security and utility against a blocking-disabled control. Proxy labels need validation and should not be treated as exact L. Our original thresholds were chosen after seeing execution outcomes, so our experiments do not prospectively validate this whole workflow.

Replay reconstructs offline labels from existing traces; instrumented benchmark runs can instead log the checks directly. The deployed scorer need not run the checker: our paired classifier used checker-derived training labels without privileged goal text or checks in scoring inputs. AgentDojo-L provides benchmark labels and validation/exclusion records. Our judges that labeled calls without replay still needed the known goal and gave mixed calibration results, so they did not remove the limitation you identify.

Comment to W3

We chose pre-execution monitoring because it offers an external gate before tool side effects, without retraining the agent. It is useful where that intervention point is available and can complement other defenses. We did not compare defense classes or establish general superiority.

The same evaluation framing can apply when the agent itself pursues an unwanted objective, provided an observable criterion is defined on the relevant snapshots. The objective need not come from an injected instruction. We tested injection and MANTRA rule violations, not intentionally deceptive or loss-of-control agents; transfer to those settings remains untested.

Reviewer_5ENeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at how tool-call safety evaluation changes depending on what the labels actually describe. The authors separate two things - whether the proposed calls finish an attack right away, and whether the whole run succeeds in the end. The authors rebuild the agentdojo environments and check the attackers goal before and after the calls. The paper also adds a separate annotation check and some experiments where monitors block calls.

Critical strengths

The paper points at a real evaluation problem. I like that the authors keep the predictions the same, so you can clearly see what happens when only the labels change. I also liked that the authors are open about the uncertainty in the monitor rankings and that they show the practical results depend a lot on the blocking threshold.

Critical weaknesses
  1. The blocking thresholds were chosen using the evaluation data. the practical benefit needs to be tested on separate data.
  2. The monitor rankings are still uncertain. most of the improvement comes from recalibrating scores, so the benefit of choosing a different monitor is unclear.
  3. Replay test removes calls but keeps later arguments fixed. those arguments may still contain information from the removed calls, making them look less necessary than they were.
Questions for authors

a) Is the safe-task success edge maintained when the baseline is measured on separate validation runs? b) How stable is the monitor selection when there are more candidate monitors? c) How does the skip-replay result still hold if the agent regenerates its later actions from the changed history? d) Does the label effect still matter when the monitor scores every decision in a run, not just the first one?

1 comment

Thank you for your careful reading and specific questions about validation, monitor selection, and replay.

Comment to W1/Q(a)

We ran both policies again in separate Qwen executions and observed higher safe-task success with Mistral-monitor 0.9 than TS-Guard 0.5. This supports their subsequent performance, but does not validate the original threshold-selection procedure: Mistral's thresholds had been chosen using submitted execution outcomes.

We kept the policies fixed and changed the attacks on the same 620 task-goal pairs. Entries below are Mistral minus TS-Guard in percentage points; safe-task success means user completion without attack success. Intervals are 95%: identity intervals resampled task-goal pairs, while cluster intervals resampled tasks and goals separately.

AttackSafe task: identity intervalTask/goal-cluster intervalAttack success: identity interval
tool_knowledge+5.3 [2.7, 7.9][0.0, 11.5]+0.8 [-0.2, 1.9]
ignore_previous+5.8 [2.3, 9.4][-1.3, 12.8]+1.6 [0.3, 2.9]
Equal pool+5.6 [3.2, 8.0][0.9, 10.3]+1.2 [0.3, 2.1]

Only the pooled safe-task cluster interval excluded zero. Attack success was also higher; its pooled cluster interval was [-0.5, 3.7], so we cannot claim an advantage at matched security. Pooling was specified after the tool_knowledge result; server configurations differed across attacks, and that study's cluster analysis was post hoc. All 949 available task-goal pairs had appeared in earlier data, so these tests did not provide validation on untouched pairs.

Retrospective split-half threshold selection gave +4.9 safe-task points [-5.8, 8.9], an interval describing resampling and split variability.

A separate Mistral monitor with tool schemas did worse in its primary comparison. Its flag-rate rule selected threshold 1.0 (0.30% development flags versus the TS-Guard 1.0 reference's 9.15%). Against the specified TS-Guard 0.5 comparator, attack success was 46.1% versus 2.7%, and safe-task success fell 13.1 points, identity interval [-16.9, -9.2], clusters [-22.8, -4.0]. A threshold 0.9 sensitivity, specified before the schema-aware execution outcomes, gave +6.5 safe-task points [3.7, 9.2] but +1.3 attack points [0.2, 2.6], with identity intervals; it did not replace the primary result.

We excluded defective Mistral-agent executions; corrected validation failed its pilot. The Qwen follow-ups were unaffected by the submitted Qwen trace defect. These reruns tested the fixed policies, not how they were originally selected.

Comment to W2/Q(b)

Adding candidates did not establish a consistent change in monitor choice, and we agree that recalibration explains much of the reduction in L log loss. L records pending-call goal completion and Y terminal attack success. Repeating selection and calibration under system resampling gave a total gap of 0.224 [0.133, 0.325]; keeping the Y-selected monitor and recalibrating it to L reduced the gap to 0.069 [0.028, 0.120]. Both intervals included zero when tasks and goals were also resampled.

In the exploratory nine-candidate prompt pool, each label selected the same Mistral variant in 11/12 folds (not necessarily the same folds); changing monitor choice contributed -0.005 and recalibration 0.220 to the loss difference. These variants added no model families. In the 15-candidate panel on 1,434 decisions, 13 had higher AUROC against L, but the highest-AUROC monitor differed between Y and L in only 71.6% of draws, below the specified 95% criterion (73.0% after excluding source runs with native Mistral call text). Mistral ranked first under both labels on new-agent data alone. The evaluation target therefore mattered even when choosing a different monitor was not supported.

Comment to W3/Q(c)

We cannot show that skip-replay would give the same result if the agent regenerated later actions. You are right that recorded arguments may retain information from omitted calls, so we have withdrawn the causal interpretation and limited the result to the recorded suffix. Blocking executions regenerated actions but could block repeatedly; they did not test the isolated intervention you describe. Neither the fixed-score comparisons nor the complete-policy comparisons depend on that causal assumption.

Comment to Q(d)

The AUROC difference between L and Y persisted beyond the first decision. Fixed TS-Guard scores gave L-minus-Y AUROC +0.119 [0.056, 0.193] on 1,725 eligible Qwen decisions and +0.107 [0.041, 0.180] on 1,190 later decisions. These task/goal-cluster intervals gave runs equal weight. We excluded replay failures and decisions after goal completion, so this was not an evaluation of every assistant turn.

Reviewer_pmfNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how the choice of evaluation labels changes the assessment and selection of safety monitors for agents that use tools. It distinguishes whether a decision contains executable calls, whether those calls immediately complete an attacker’s goal, and whether the attack eventually succeeds. The authors construct local completion labels through environment replay, checking whether each proposed call sequence changes the attacker’s goal from unsatisfied to satisfied. On AgentDojo, they hold monitor scores fixed to isolate label effects and separately examine recalibration and monitor selection. In the public cohort, terminal labels select TS-Guard, whereas local completion labels select Mistral, showing that the evaluation target can change which monitor appears preferable. An additional analysis on ToolSafe’s ASB-Traj benchmark examines the same issue using step annotations. The authors then deploy the selected monitors with blocking, agent continuation, and a matched control with blocking disabled. In one Qwen comparison, the locally selected monitor increases safe task completion under attack from 46.8% to 52.9% and reduces blocking. In contrast, attack success changes from 1.9% to 3.1%, connecting the offline evaluation choices to concrete security and utility outcomes.

Critical strengths
  1. The paper clearly distinguishes intervention eligibility, local completion, and terminal success, identifying an important ambiguity in tool-call safety evaluation.

  2. The empirical analysis isolates label effects by holding predictions fixed and strengthens its findings through replay validation, calibration controls, and an external benchmark.

  3. The execution study includes a control with blocking disabled and jointly reports security and task outcomes, helping separate intervention effects from execution configuration differences.

Critical weaknesses
  1. The confidence intervals keep the selected monitors and calibration models fixed, so it remains unclear whether repeating the full selection and calibration process on resampled data would consistently reproduce the reported change in monitor choice and performance gain.

  2. Although the paper reports offline flagging rates on clean examples, it lacks a matched execution comparison of the two selected monitors on tasks without attacks, leaving their effects on benign task completion and recovery from false blocks unclear.

  3. The paper shows that label choice changes monitor rankings, but provides limited evidence that existing evaluations use terminal outcomes to assess individual tool calls or that local labels improve monitor selection over directly measuring attack success and task completion, leaving the practical relevance of the main comparison unclear.

Questions for authors

See weakness

1 comment

Thank you for your careful reading and constructive suggestions on selection stability, benign execution, and practical relevance.

Comment to W1

We repeated monitor selection and calibration in every bootstrap sample to test whether the reported change in monitor choice persisted when both steps were rerun. Here, L records whether the pending calls complete the attacker's goal, and Y records whether the attack eventually succeeds.

When we resampled systems, Y selected TS-Guard in 99.96% of folds and L selected Mistral in 99.7%. Using Y rather than L for selection and calibration increased held-out L log loss by 0.224 [0.133, 0.325]. Keeping the Y-selected monitor but recalibrating it against L reduced this gap to 0.069 [0.028, 0.120]. This supports the change in monitor choice, while showing that recalibration explains much of the difference.

When we also resampled user tasks and attacker goals, the intervals widened to [-0.018, 0.562] for the total gap and [-0.060, 0.292] for the gap after recalibration. Both included zero, so this analysis did not establish a positive gap after accounting for those sources of variation. It retained 906/1,000 samples with both classes in every calibration split.

We also tested the procedures selected and calibrated on public data on 932 new-agent decisions, without refitting. The total gap was 0.285 [0.117, 0.535] and persisted after excluding source runs containing native Mistral call text. The benefit of changing the monitor alone remained uncertain. These analyses strengthen the evidence beyond the submission's fixed-prediction intervals; they did not involve re-executing each resampled blocking policy. Brackets are 95% intervals under the stated procedures.

Comment to W2

We ran a matched comparison on all 97 user tasks without injection. On Qwen, both selected policies reduced benign task completion, and neither had an established advantage over the other:

PolicyTask successRuns with a block
Blocking disabled70.1%0%
TS-Guard 0.557.7%23.7%
Mistral monitor 0.957.7%17.5%

Both lost 12.4 percentage points versus blocking disabled, with paired task-resampling intervals [-19.6, -6.2] and [-18.6, -6.2]. Their difference was 0.0 [-7.2, 7.2], which does not establish equivalence. This provides actual benign completion outcomes in addition to the original flagging rates.

About 12-13% of blocked runs still completed the task. This does not estimate recovery caused by continuation, because the monitors blocked different runs. Each task ran once per arm. We excluded defective Mistral-agent executions; the Mistral monitor here scored Qwen calls through a separate adapter. The submitted Qwen trace defect did not affect these benign runs.

Comment to W3

We agree that changing monitor rankings is not enough to show better policy selection. We have not documented prior tool-call studies treating terminal outcomes as immediate-completion ground truth. Outcome forecasting and local assessment can both be legitimate; our contribution is to show what changes when a run's outcome is used to evaluate its pending calls.

This was not confined to the first decision. With TS-Guard scores fixed, L-minus-Y AUROC was +0.119 [0.056, 0.193] on 1,725 replay-valid Qwen decisions whose goal was unmet before the calls, and +0.107 [0.041, 0.180] on 1,190 later decisions. These task/goal-cluster intervals gave runs equal weight and excluded replay failures and post-completion decisions.

We also compared direct selection among six policies with L-based screening that shortlisted two. Both used 60 execution observations, with separate tasks for selection and evaluation. Mean signed safe-task regret was 4.40 versus 5.91 points; lower is better. Direct minus L-screening was -1.51, with split-variability interval [-16.25, 14.65], not a population CI. The criterion for improved policy selection was not met. Violations of the held-out 5% attack constraint were 8.65% versus 0.10%, reflecting L-screening's low-attack shortlist; signed regret can favor infeasible policies. Candidates were partly outcome-informed and offline scoring costs were excluded.

We therefore do not claim that L-screening beats direct selection. We believe its value here is to clarify what monitor evaluations establish, while leaving deployment choices to executed security and utility tests. L=0 does not mean a call is safe to execute.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.