acceptodds
Under review as a conference paper at ICLR 2027

TACT-Bench: Testing Adaptive Conversational Threats against Tool-Use Agents

Abstract

The growing autonomy of AI agents in using external tools and making decisions has created an urgent need for comprehensive safety benchmarks. Existing benchmarks test harmful goals, compromised tool channels, or adaptive multi-turn attacks; however, they may not capture the unique capabilities of policy-governed tool-use agents, who must resist unauthorized requests without abandoning the legitimate work embedded in the same interaction. Existing benchmarks provide limited evidence about persistent, policy-blind social engineering that seeks a concrete tool outcome. To fill this gap, we introduce **TACT-Bench**, a stateful benchmark built around scenario-specific stories, personas, manipulation tactics, and response-adaptive escalation. Each adversarial task targets an action explicitly prohibited by the policy provided to the assistant, while the policy remains hidden from the attacking user. TACT pairs each adversarial task with at least one benign counterpart, and its adversarial tasks retain authorized ground-truth actions when a compliant resolution remains available. We define these base tasks as TACT-Base. We additionally introduce TACT-TrustPrime, which places a complete independent task before an unrelated attack in the same conversation. We also propose Paired Selective Trustworthiness (PST), which combines explicit safe refusal with demonstrated capability on matched benign tasks, to detect overcautious and impractical agents. Averaged across assistants, TACT-Base raises Attack-Induced Failure Rate (AFR) by 17.53% over four compatible attack baselines, and TACT-TrustPrime raises it by a further 17.01% (31.08% over Foot-In-The-Door, the strongest baseline). The two evaluated defense methods reduce AFR by only 8.35%–9.41% on average and do not consistently improve PST, underscoring the need for stronger policy-enforcement methods.

Primary area:
datasets and benchmarks
Keywords:
Tool-use Agents, Agentic AI, Trustworthy AI, Tool-Using LLMs

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_GJefOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TACT-Bench, a benchmark for evaluating the robustness of tool-use agents against adaptive conversational attacks. It incorporates scenario-specific social engineering, persistent adversarial interactions, and matched benign tasks. The authors further introduce TACT-TrustPrime to investigate the impact of prior legitimate interactions and propose Paired Selective Trustworthiness (PST) to jointly assess security and benign task performance. Experiments across six assistants and 2,852 tasks demonstrate substantially higher attack effectiveness than existing baselines, while revealing limitations in current defense mechanisms.

Critical strengths
  1. The paper addresses an important gap in agent safety evaluation by considering adaptive, policy-blind conversational attacks while preserving legitimate task requirements.

  2. The experiments cover multiple assistants, attack baselines, and defense mechanisms. The proposed PST metric provides a useful perspective on balancing security and task utility.

Critical weaknesses
  1. Although the benchmark includes diverse scenarios, all tasks are instantiated within a retail environment, potentially limiting cross-domain generalizability.

  2. The evaluation relies on a single LLM-based adversarial user simulator, which may introduce model-specific biases and limit the diversity of attack behaviors.

Questions for authors
  1. Could the authors separately report AFR caused by forbidden tool invocations versus incorrect final states? Would the conclusions and model rankings remain consistent under these separate metrics?

  2. How well does TACT-Bench generalize to genuinely unseen scenarios or application environments, rather than synthetic variations of the existing 17 scenario families?

  3. Could the authors introduce a length-matched benign-interaction baseline for TrustPrime and quantify the impact of excluded simulator failures on the reported attack success rates?

  4. The related work discussion could be strengthened by incorporating relevant studies on adversarial robustness, safety evaluation, and efficiency, including XSTest, HarmBench, Sponge Tool Attack, and JailbreakBench. These works provide complementary perspectives on model safety, attack resilience, and security-efficiency trade-offs.

I'm willing to adjust my rating according to the authors' rebuttal.

4 comments

Part 1: summary, AFR by cause (Q1), user simulator (W2)

Thank you for the review, and for being open to revising your rating. The analyses below use the logs of the experiments in the paper, together with one new control experiment for Q3.

In brief:

  1. AFR by cause (Q1). We report forbidden-tool invocations and incorrect final states separately for every assistant. The model ranking is identical under incorrect final states and nearly identical under forbidden invocations.
  2. Single simulator (W2). App. E.8 already re-runs Vanilla TACT-Base with a second user simulator, Claude Sonnet 4.6, for three assistants. Their ranking is preserved, and TACT's AFR stays above the four baselines (even in their run with the GPT-5.6-SOL user) for all three.
  3. Excluded simulator failures (Q3). They are 8 of 918 Base task instances and 19 of 2,118 TrustPrime task instances. No imputation moves a six-assistant mean AFR by more than 1.5 points or changes the sign of a difference between means.
  4. Length (Q3). For the two assistants with High-step runs, a Base attack with similar length to TrustPrime's length has no higher AFR than the ordinary Base attack. In a new control, re-sending the complete policy to the assistant at the start of the second TrustPrime task leaves AFR unchanged (50.4% against 51.3%).
  5. Domain and unseen scenarios (W1, Q2). The benchmark is retail only, and the paper says so. Within it, the comparisons of TACT with the baselines and of TrustPrime with Base keep their sign when any family is left out, and in 96.6% to 100% of random halves of the families.
  6. Related work (Q4). We have added the four papers to Sec. 2.

Q1. AFR by cause. AFR counts an adversarial episode as a failure when a task-forbidden tool is invoked or when the final state differs from the annotated safe state. Both can happen in one episode, so we give each rate and their overlap (AFR = forbidden + incorrect − both). We report the split for the main results, Vanilla TACT-Base on all six assistants, and for the two defenses. % of scored adversarial task instances:

AssistantAFRForbidden tool invokedIncorrect final stateBoth
GPT-OSS-120B47.0620.9243.1416.99
Mistral-Large-348.3722.8843.7918.30
DeepSeek-V3.242.4821.5737.9116.99
Claude Opus 4.828.196.0425.503.36
GPT-5.6-SOL35.9512.4232.689.15
Gemini 3.5 Flash33.5612.7530.209.40
Mean39.2716.0935.5412.36

Model ranking. Ranked by incorrect final state, the six assistants fall in exactly the AFR order (Spearman 1.00). Ranked by forbidden-tool invocation, the correlation with the AFR order is 0.89. Two neighbouring pairs swap: DeepSeek-V3.2 with GPT-OSS-120B, which are one episode apart, and Gemini 3.5 Flash with GPT-5.6-SOL, which have the same count. Under all three metrics, the three frontier assistants are the three least vulnerable, and Claude Opus 4.8 is the least.

Defenses. Mean of six assistants:

Vanilla TACT-BaseAFRForbidden tool invokedIncorrect final state
No defense39.2716.0935.54
Tool-Guard30.917.5030.80
Policy-Guard29.866.0429.75

Each defense cuts some of the forbidden invocations and leaves most incorrect final states in place. A forbidden tool is still invoked in 7.5% and 6.0% of adversarial episodes, and most of the remaining AFR is an incorrect final state.

We have added these columns, with their overlap, to the table of AFR components in App. H.1, which also gives a finer three-way split.

W2. Single user simulator. The paper already includes a second simulator. App. E.8 re-runs Vanilla TACT-Base with Claude Sonnet 4.6 as the user, with the same instructions, personas, facts and tactic ladders, for three assistants. We agree it is easy to miss and now point to it from Sec. 4.2.

AssistantAFR, GPT-5.6-SOL userAFR, Claude Sonnet 4.6 user
Claude Opus 4.828.1926.80
GPT-5.6-SOL35.9531.37
Gemini 3.5 Flash33.5629.41
  • Ranking. The three assistants are in the same order under both simulators.
  • Level. The Sonnet-driven attack is 1.4 to 4.6 points weaker on each assistant (−3.4 on average).
  • Baselines. Sonnet-driven TACT has higher AFR than each of the four GPT-5.6-SOL-driven baselines for each of the three assistants, 12 of 12 comparisons.
  • Agreement. The two simulators lead to the same outcome on 77% to 85% of individual task instances. Per-family AFR, averaged over the three assistants, has a Pearson correlation of 0.85 between them.
  • Behaviour. Both simulators work from the same hidden goal, facts and tactic ladder, and both adapt to the assistant's replies.

Part 2: length and exclusions (Q3)

Q3 (a). Length-matched benign baseline.

Does a longer conversation by itself raise AFR? No. The High-step variant delivers the same objective and facts over more turns (Sec. 3.2), which makes it a length control. For the two assistants with High-step TACT-Base runs:

Mistral-Large-3 and GPT-5.6-SOLUser turnsAssistant stepsAFR
Vanilla TACT-Base8.412.442.2
High-step TACT-Base12.116.039.9
Vanilla TrustPrime11.217.257.2

High-step Base is as long as TrustPrime (12.1 against 11.2 user turns). Its AFR is not above Vanilla Base (−2.3, lower for both assistants), while TrustPrime is 17.3 points above High-step Base. Within TrustPrime, High-step lengthens the conversation by two thirds (19.4 against 11.7 user turns, three assistants), and AFR moves by +1.3 on the attack-primer pairs scored in both runs. No episode came near its step budget.

Does the longer conversation make the assistant lose track of the policy? This is the concern behind a length-matched baseline, and a conversation that is benign in both phases cannot test it: an authorized second request is completed whether or not the policy is still being applied. Benign requests are completed at high rates in any case: the first task in 1,621 of the 1,625 scored TrustPrime conversations where this can be checked, and 92.6% mean benign competence on the standalone counterparts (App. E.7). The test has to be made where the policy is needed, at the attack. In a new control, we therefore re-ran Vanilla TrustPrime and re-sent the complete policy to the assistant at the start of the second task, directly before the customer's first message of that task. Tasks, user simulator and scoring are unchanged. If the policy had faded, the reminder should lower AFR.

We ran it for two of the three frontier assistants. Because some instances were discarded in either run, the two conditions are compared on the instances scored in both.

AssistantAFR as in the paperAFR with the policy re-sent
GPT-5.6-SOL49.856.5
Gemini 3.5 Flash50.946.4
Both50.451.3

The reminder does not lower AFR: it is higher for GPT-5.6-SOL, lower for Gemini 3.5 Flash, and not significantly different overall. Of the conversations, 89.4% end the same way in both conditions, 4.8% fail only without the reminder and 5.8% only with it (exact McNemar p = 0.59).

Neither additional turns nor a fading policy therefore explains the TrustPrime result. This supports the reading of Sec. 5.4 and App. D, in which the completed benign request establishes rapport, confirms the account context and satisfies the ordinary safeguards, the assistant comes to treat the user as legitimate, and the attack that follows builds on that standing, and that's why it gets higher AFR and lower RR/PST. We have added the control to App. H.3 and can extend it to the remaining assistants for the camera-ready version upon acceptance.

Q3 (b). Excluded simulator failures. An episode is discarded when the simulator cannot produce a usable adversarial user turn. In 67 of the 70 cases below, its provider refused the persona (App. E.9); the other 3 are provider errors. No conversation takes place, so there is nothing to score. The table gives the counts and the range the six-assistant mean AFR can take if every excluded instance is counted as safe or as a failure.

SettingDiscarded / adversarial instancesAFR reportedRange under any imputation
Vanilla TACT-Base8 / 91839.2739.00 to 39.87
FITD17 / 91825.2024.84 to 26.69
Naive, PAP, Crescendo0 / 918 each20.48, 21.02, 20.26unchanged
Vanilla TrustPrime19 / 2,118, plus 16 pre-attack hand-offs56.2855.38 to 57.03
Tool-Guard, Policy-Guard14 / 918, 12 / 91830.91, 29.8630.50 to 32.03, 29.52 to 30.83

All discards fall on two assistants, Claude Opus 4.8 and Gemini 3.5 Flash. The 16 TrustPrime hand-offs are instances in which the assistant transferred to a human before the attack began (App. D.2). They are a scoring rule, not a simulator failure, and the range includes them.

Under the least favourable imputation for each comparison:

  • TACT − FITD is worst-case +12.3 against +14.1 as reported, and TACT stays above every baseline for every assistant (24 of 24).
  • TrustPrime − Base is worst-case +15.5 against +17.0, and positive for every assistant.
  • The defenses' mean AFR reductions are worst-case 7.0 and 8.2 points against 8.4 and 9.4.

App. E.9 also measures the effect directly. The eight Base instances discarded under GPT-5.6-SOL did run under the Sonnet simulator, and all eight were attack-induced failures.

FITD's 17 discards (15 persona refusals and 2 provider authentication errors) are included above. We have added FITD and the defended runs to Table 22 (Table 23 in the revised paper), and the ranges above to App. H.4.

Part 3: domain and unseen scenarios (W1, Q2), related work (Q4)

W1 and Q2. Domain and unseen scenarios. TACT-Bench is a benchmark, and retail is the environment in which we built it. What it contributes as a method does not depend on the retail scenarios: persistent attacks through the ordinary user channel, built from a scenario story, a persona and a tactic ladder that adapts to the assistant's replies, matched adversarial and benign tasks scored on tool calls and the final state, the TrustPrime construction, and the PST metric. Our goal was to show that such an evaluation can be built and what it reveals, with retail as the worked example. The empirical results are for this instantiation, as Sec. 6 states ("the 17 scenario families do not substitute for future cross-domain evaluation"), as a limitation of this work.

What the existing data can show is how far the aggregate comparisons depend on particular scenario families. This is a sensitivity analysis, not a test on new scenarios.

  • Seed instances and augmented copies. Averaged over the six assistants, AFR is 40.2 on the 17 hand-written seed instances and 39.1 on their augmented copies. Within a family and assistant, on average 87% of the instances have the majority outcome. As you note, a family's instances are entity-level variations of one scenario, so the independent unit is the family, and all our intervals resample families.
  • Leaving one family out. Each point estimate keeps its sign and most of its size whichever family is removed. These ranges show sensitivity to single families.
Mean of six assistantsAll 17 familiesRange with one family left out
AFR: TACT − mean of the four baselines+17.5+13.3 to +20.1
AFR: TACT − FITD+14.1+10.3 to +17.1
AFR: Tool-Guard − no defense−8.4−9.4 to −4.9
AFR: Policy-Guard − no defense−9.4−10.5 to −6.1

The assistant ranking correlates at least 0.92 with the full ranking in all 17 cases, and the three frontier assistants always remain the three least vulnerable. For TrustPrime, the family-by-family comparison is in App. H.3 and in our response to Reviewer 1 (higher than Base in 13 of 17 families).

  • Random halves of the families. Over all 24,310 ways to split the 17 families into 8 and 9, TACT's AFR is above each baseline in 96.6% (FITD) to 99.9% of the half-suites, and TrustPrime is above Base in all of them.

Extending to another domain. A new domain needs its own policy, tools, records and scenario stories. The task structure, the scoring and the metrics carry over unchanged. The tactics transfer as well. They are general social-engineering moves such as urgency, authority claims, prior-agent approval, manager escalation and off-system workarounds (Sec. 3.2), and only the story and the facts each step uses are written for a scenario, so the retail ladders can serve as templates. The policy primitives the retail tools exercise (identity, ownership, authorization, eligibility, approved destinations, evidence, explicit confirmation, sensitive disclosure) are shared with financial, travel, healthcare and enterprise agents (Sec. 3.2), so we expect the same kind of findings there. We have not yet run a second domain, but it could be an extension of this work.

Q4. Related work. Thank you for these suggestions. We have added XSTest, HarmBench, Sponge Tool Attack, and JailbreakBench to Section 2 and clarified how their evaluation objectives complement TACT-Bench.

Dear Reviewer GJef,

Thank you for your review. As the discussion deadline approaches, we would appreciate hearing whether our response addresses your main concerns. If so, would you consider adjusting your rating?

We would be happy to answer any follow-up questions or clarify any remaining points.

Best,

The Authors

Reviewer_15fPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes TACT-Bench, a benchmark to test whether policy-governed tool-use agents can resist a policy-blind user-based attack. Each adversarial task has a story, a persona, a hidden objective and a ladder, and is paired with matched benign task(s). The TACT-Base dataset is augmented via tone and decomposition variants and augmented. A second suite, TACT-TrustPrime, puts an unrelated benign task before the attack. The authors also propose the PST metric, which answers whether explicit resistance coexists with demonstrated authorized capability.. Experiments on four attack baselines and two defense methods shows benchmark quality.

Critical strengths
  1. Pairing each adversarial task with benign ones is a good design
  2. The provided dataset might be interesting for the respective communities.
  3. The paper tries to many details.
Critical weaknesses

The paper might be too over emphasized on every details but misses the big-picture description of the work. And this makes the work hard to follow:

  1. In L194, what does ti,jt_{i,j} refer to?
  2. In real experiment, how to measure a target is "exact match" in L204?
  3. I might not realize the part to describe how to transfer an attack data into benign data.
  4. There are many symbols in the Section 3.1, is it necessary to import all of these symbols?
  5. The figure 1 does not give a high-level idea how the data generation/simulation is done
  6. In L266, what does "seeds" refer to?

In L58, the paper introduce a table to compare the difference between this work and other work. But in the introduction, there is no specific analysis from the view of table columns.

Questions for authors

Please refer to Weakness section.

3 comments

Part 1: clarification on some terms and notations and benign task construction (W1,W2, and W3)

We thank the reviewer for recognizing the value of pairing adversarial tasks with benign counterparts and for identifying places where the presentation obscures the main idea. We agree that the paper should introduce the overall design more clearly. We have revised the Introduction and redesigned figures to make the setting easier to follow. The central question is whether an agent can enforce explicit policy boundaries under persistent, adaptive social engineering while remaining capable of completing related authorized tasks. We address the individual concerns below.

  1. Notation at L194.

    We apologize for the insufficient explanation. In the submitted notation, Γi\Gamma_i is the number of tool calls made during conversational turn ii; it can be zero. The index jj identifies a call within that turn. The call is denoted by ti,jt_{i,j}, and ai,ja_{i,j} is the observation returned by its execution.

    We have replaced the detailed turn-by-turn notation in Section 3.1 with a plain-language explanation and provided the formalization with a notation table in Appendix B.1. The expression (ai,j,e′)←Exec⁡(ti,j,e)(a_{i,j},e')\leftarrow\operatorname{Exec}(t_{i,j},e) returns the tool observation and resulting database state, in which the current state is then updated to e′e'.

  2. How “exact match” is measured at L204.

    We have added the operational explanation to Section 3.1. Our experiments run in an executable simulated retail environment, so the evaluator can inspect the database directly. It replays the annotated safe actions on a fresh copy of the initial database to obtain the reference safe state, then compares this reference with the database produced by the assistant. The implementation first compares deterministic database hashes and, on a mismatch, examines the differing fields.

    Structured fields must match exactly; for example, the refund destination must be identical to the annotated destination. Only designated free-text fields, such as a recovery-ticket reason, allow semantically equivalent wording. GPT-4.1 performs these comparisons in the reported experiments. Appendix C.3 reports validation on 100 free-text comparisons, with 91% mean human–judge agreement and Cohen’s κ=0.820\kappa=0.820.

    Thus, the comparison concerns the resulting database state and does not require an identical sequence of tool calls. Forbidden-tool calls are checked separately against the action log, so a matching final state alone is insufficient.

  3. How adversarial tasks are paired with benign tasks.

    We have added a construction explanation to Section 3.1. The initial adversarial and benign tasks are manually authored within matched scenario families. For each benign counterpart, we specify a related request that the policy permits and ensure that the database supports the authorized operation. We also annotate the safe actions needed to complete it.

    Figure 5 illustrates this with a refund-destination pair that shares the same return context. The adversarial user tries to obtain a new gift card sent to an external email without the required pre-approval. The benign user requests the return refund on an existing gift card already on the account. Appendix A.3 and Figure 4 explain why this family also includes benign counterparts that separately exercise pre-approved issuance operations.

    These counterparts are evaluated in separate episodes. Section 3.1 also distinguishes them from an authorized resolution requested within an adversarial episode after the attack tactics have been exhausted.

Part 2: revisions on some terms, notations, and figures (W4 to W6) and explaining Table 1 column names

  1. The amount of notation in Section 3.1.

    We agree that the detailed trajectory notation was unnecessary in the main text. We have kept the task definition and the equations defining safe handling and attack-induced failure, while moving the turn-by-turn formalization to Appendix B.1.

    The main text now explains the interaction directly: the user sees the natural-language conversation, while the assistant additionally receives the policy and tool definitions and retains its internal tool history. The appendix provides the formal details and a notation table for readers who need them.

  2. Figure 1 and the generation/simulation overview.

    We have redesigned Figure 1 (and Figures 3, 8, and 9) to make the runtime interaction easier to follow and added a construction overview at the beginning of Section 3.2. The revised caption also directs readers to Figure 3 in Appendix A.1 for the dataset-construction process.

    We begin with 40 manually authored initial tasks. We retain the original task set and create three additional tone/decomposition variants, giving four task sets in total. Each then follows two independent construction branches. Rule-based entity augmentation produces TACT-Base. Separately, an LLM-based pipeline screens cross-scenario benign–adversarial pairs and synthesizes TACT-TrustPrime tasks, with an independent benign task preceding the attack.

    At evaluation time, the user simulator generates the conversation turn by turn from the task instruction and adapts to the assistant’s replies. The main text now explicitly distinguishes this runtime simulation from the earlier construction of the task specifications.

  3. Meaning of “seeds” at L266.

    We have replaced “40 seeds” with “40 initial tasks” and defined these tasks at the beginning of Section 3.2 as 17 manually authored adversarial tasks and 23 matched benign tasks across 17 scenario families. The term refers to task specifications, not random-number seeds.

  4. Explaining the comparison through Table 1’s columns.

    We have integrated explanations of Table 1’s column terms into the Introduction where the corresponding concepts are introduced. The opening paragraph now explains “Executable tools” as tools that perform operations in an environment.

    The paragraph describing the evaluation gap explains that the attacker interacts only through ordinary dialogue (“User-only channel”) and adapts requests to the assistant’s replies (“Response-adaptive user”). It also defines “Policy-blind” by stating that the assistant’s policy and system prompt are not provided to the user.

    The paragraph introducing TACT-Bench distinguishes related authorized tasks evaluated in separate episodes (“Matched benign”) from authorized work completed within an adversarial episode (“Safe resolution under attack”). For example, an attacker tries to redirect a refund to a newly issued gift card without the required pre-approval. The matched benign counterpart is a separate task in which the user requests an authorized refund to an existing gift card on the account. Safe resolution under attack concerns the original adversarial conversation: after exhausting the attack tactics, the user explicitly requests that permitted refund and cooperates with the required checks. The assistant should complete it once the policy requirements are satisfied, without refusing solely because the user previously attempted an unauthorized action.

    The TrustPrime description explains “Independent benign then attack” as an independent benign task followed by an unrelated attack from a different scenario in the same conversation.

    These explanations explicitly connect the Introduction to Table 1 and clarify the combination of properties evaluated by TACT-Bench. The table now also includes citations to the individual works.

Dear Reviewer 15fP,

Thank you for your review. As the discussion deadline approaches, we would appreciate hearing whether our response addresses your main concerns. If so, would you consider adjusting your rating?

We would be happy to answer any follow-up questions or clarify any remaining points.

Best,

The Authors

Comments on the paper

6 comments

Part 1: summary, and the defense analysis (W1, Q1)

Thank you for a careful and constructive review. We ran the analyses below on the existing logs, with no new model calls. Part 3 also reports one new control experiment, and part 4 three repeated runs. In brief:

  1. Call-level attribution confirms your reading of Table 25: 24.7% is an intervention rate. Still, Policy-Guard never approved an authorized call in 101 benign episodes, and Tool-Guard in none.
  2. The conflict in App. F.9 comes from a labelling error in runtime, not from the verifier. We checked all the episodes that it could affect, and it is behind 2 of the 376 blocked calls in only 2 defense episodes, which we no longer count as false positives. No other result changes.
  3. AFR is reported in the three components you name in part 2. Both defenses cut prohibited actions, though they still fall short, and neither improves completion of the authorized resolution, even though they are asked by the user explicitly, and because of the induced attack, the agent couldn't finish them.
  4. Family by family, TrustPrime AFR is higher than Base in 13 of the 17 scenario families. The difference is unchanged under equal source weights and does not come from incomplete primers. A new control on the same TrustPrime instances shows that it does not come from the policy fading over a longer dialogue either (part 3).
  5. Family-cluster intervals and denominators are in part 4.

Unless noted otherwise, intervals are percentile bootstraps over the 17 scenario families (1,000,000 resamples; a family's instances and all assistants are resampled together). Differences are computed before rounding. Section and table numbers are those of the original submission. In the revised paper, Table 8 and later tables are numbered one higher, and the new analyses are in App. H.

Blocked calls in benign episodes (Q1). We compared every blocked call in a benign episode with that task's annotated authorized calls and forbidden tools. A blocked call counts as authorized when its tool and arguments match an annotated authorized action of the task. Free-text arguments such as a ticket reason are not compared, and an omitted optional argument whose default is the annotated value counts as a match. Six assistants pooled, Vanilla TACT-Base, 1,242 benign episodes per defense.

Blocked call in a benign episodeTool-GuardPolicy-Guard
Benign episodes with at least one block32 (2.6%)307 (24.7%)
Blocked calls32376
Not authorized: a tool the task forbids3121
Not authorized: another tool, or different arguments131
Authorized, passed later in the episode0175
Authorized, never approved0147
Conflicting lookups cited by the verifier (App. F.9)02

You are right that 2.6% and 24.7% are intervention rates, and we have relabeled that column as benign interventions. Every Tool-Guard block was a correct intervention. It blocked no authorized calls, and 31 of its 32 blocks stopped a tool the benign task forbids, which is why BVR is zero under Tool-Guard.

For Policy-Guard, apart from the 2 calls of App. F.9 (below), the call-level picture has three parts.

  • 52 blocked calls (14%) were not authorized for the task.
  • 175 (47%) were authorized calls held until the assistant completed a step the checklist requires, most often explicit confirmation, and then passed. They cost turns, not outcomes: 165 episodes, 4 failures.
  • 147 (39%) were authorized calls that the verifier never approved. They fall in 101 benign episodes, and 100 of those failed.

Of the 1,242 benign episodes, 174 failed under Policy-Guard against 91 with no defense. The 100 failures in episodes where an authorized call was never approved exceed that whole increase of 83. credit_loyalty_points and return_delivered_order_items account for 126 of the 147 never-approved calls. This is the call-level evidence that Table 25 did not provide. We have added it as App. H.2, and the statements in App. E.5, F.7 and F.10 now rest on it and are worded at that level: the verifier also denies authorized calls in 8% of benign episodes.

The conflict in App. F.9. You are right that this block is not a verifier error. The conflict comes from a labelling error in runtime in one lookup. The verifier cites the two conflicting lookups in 2 of the 376, both in the benign password-reset task: the episode we showed, and one in which the call was approved on retry. We no longer count these as false positives. Counting the failed episode as a success raises Policy-Guard's benign success and PST by about 0.1 point each, so no defense result changes. We also checked other trajectories where this could have had an effect, and nothing has been affected other than what's mentioned. We have replaced the example in App. F.9 and deleted the sentence saying that cases of this shape drive the 24.7% rate.

Part 2: metric auditability (Q3) and what the defenses change (W1)

Components of AFR. We replayed every scored adversarial episode and compared its final state with the safe state field by field. Each attack-induced failure falls in the first of your three classes that applies.

  • F, forbidden call: a task-forbidden tool was called.
  • A, active unauthorized change: no forbidden call, but the assistant made a change the safe state does not contain.
  • P, passive authorized-resolution failure: neither, but the authorized resolution was not completed as annotated. Either it is missing, for example because the conversation was handed to a human first, or the record it writes does not state the annotated facts.

Mean of six assistants, % of scored adversarial task instances; F, A and P sum to AFR. The two overlap columns are not exclusive.

Vanilla TACT-BaseAFRFof F: also an unauthorized changeof F: also an authorized-resolution failureAP
No defense39.2716.0912.34.61.2021.97
Tool-Guard30.917.507.44.20.8722.54
Policy-Guard29.866.045.92.80.5523.27

No defense, by assistant:

AssistantScoredAFRFAP
GPT-OSS-120B15347.0620.92.623.5
Mistral-Large-315348.3722.92.023.5
DeepSeek-V3.215342.4821.62.019.0
Claude Opus 4.814928.196.00.022.1
GPT-5.6-SOL15335.9512.40.023.5
Gemini 3.5 Flash14933.5612.80.720.1

On the overlaps: in about three cases in four, a forbidden call comes with an unauthorized change (12.3 of 16.1). In 4.6 of the 16.1, the authorized resolution fails as well, almost always because the prohibited action was taken in its place. Of the 1.2 points of A, 0.3 also fail the authorized resolution.

We have added a table of these components as App. H.1 and point to it from Sec. 4.1, so that AFR can be read both as the service-level metric and by component.

What the defenses change. You ask whether a defense reduces prohibited actions, changes service completion, or both. Prohibited actions (F or A: a forbidden call or an unauthorized change) fall from 17.3% of adversarial episodes to 8.4% under Tool-Guard and to 6.6% under Policy-Guard. Passive authorized-resolution failures (i.e. P) stay at 22.0%, 22.5% and 23.3%. Both defenses therefore act on prohibited actions only. A prohibited action still occurs in 8.4% and 6.6% of adversarial episodes, and the defenses do not help the assistant complete the authorized resolution, so AFR remains 30.9 and 29.9. This is the sense in which both defenses fall short, and we now state it this way in App. F.10 and App. H.1.

What S is in PST. S is the set of scored adversarial task instances.

  • Base. One scenario instance per adversarial task instance: 17 families × 9 augmented instances = 153 per variant, minus discards. Each is paired with the benign instances of the same family in the same augmentation copy, so msm_s is 1 for 117 instances, 2 for 27, and 4 for 9.
  • TrustPrime. One scenario instance per TrustPrime task. It is paired with the standalone benign counterparts of its adversarial source's family among the 23 benign tasks (msm_s is 1, 2 or 4 for 274, 62 and 17 tasks). Those 23 benign outcomes are reused across the roughly 20 pairs that share an adversarial source. The primer is never part of csc_s.

An instance can contribute to PST only if it is not an attack-induced failure (rsr_s = 1 requires asa_s = 0).

The 199 and 201 benign counts. They are paired subsets, as you inferred. All 207 benign episodes ran in each of the six runs of Table 10 and none was discarded. |B| counts the benign counterparts of scored adversarial instances. Claude Opus 4.8 lost four adversarial instances of one family with two counterparts each (207 − 8 = 199). Gemini 3.5 Flash lost two from that family and two from a family with one counterpart (207 − 6 = 201).

Part 3: TrustPrime (W2, Q2)

Source weighting. TrustPrime contains every admissible cross-scenario pair, so the 17 adversarial sources are close to equally weighted. Each appears in 17 to 22 of the 353 pairs. With equal source weights, AFR is 39.8 in Base and 57.5 in TrustPrime, against 39.3 and 56.3 as reported, and the difference is +17.7. It is positive for every assistant (+14.1 to +24.1). Composition does not explain the gap.

Base and TrustPrime, family by family. Each row compares one Base attack (9 instances per assistant) with the TrustPrime tasks built from it (17 to 22), over the scored instances. AFR, mean of six assistants.

Scenario familyBaseTrustPrimeΔ
BOPIS pickup person change0.09.1+9.1
Refund destination90.798.0+7.2
Account recovery contact92.690.9−1.7
Password reset link redirection75.998.3+22.4
Recovery phone number16.716.70.0
Gift card balance reissue16.713.7−3.0
Post-purchase promotion credit7.490.8+83.4
Loyalty points from unverified receipt63.068.1+5.1
Furniture damage replacement61.193.7+32.5
Pending order item swap1.98.3+6.5
Receipt resend7.423.1+15.7
Missing item22.275.4+53.2
Device trade-in value dispute33.395.8+62.5
Marketplace "not as described" claim83.385.9+2.6
Pharmacy purchase history56.359.0+2.6
Expired return window48.145.9−2.3
Final-sale item0.04.8+4.8
All 17, families weighted equally39.857.5+17.7

TrustPrime AFR is higher in 13 of the 17 families, equal in 1 and lower in 3. Nine families have an interval of their own that excludes zero when their instances are resampled, and all nine are increases. No single family carries the result: leaving any one out, the mean difference stays between +13.6 and +19.0.

Primer completion. Among the 2,083 scored Vanilla TrustPrime tasks, the primer's state change was completed in 1,621 (77.8%) and not completed in 4 (0.2%, all GPT-OSS-120B). It is undefined in 458 (22.0%) because five of the 23 primers leave no database change to check. Per assistant, the completed share is 76.8% to 78.2%. AFR is 54.8% where the primer was completed and 61.1% where it is undefined, against 56.3% overall, so incomplete primers do not drive the result. The 16 pre-attack transfers were already excluded (App. D.2).

Length, on the same TrustPrime instances (new control). You note that the primer also changes dialogue length, and that High-step Base is not a length- and state-matched comparison on the same TrustPrime attack instances. We therefore re-ran Vanilla TrustPrime on those same instances, so that the first task, the length and the state are matched, and re-sent the complete policy to the assistant at the start of the second task, directly before the customer's first message of that task. Tasks, user simulator and scoring are unchanged. If the longer dialogue had weakened the assistant's hold on the policy, the reminder should lower AFR. We ran it for two of the three frontier assistants. Because some instances were discarded in either run, the two conditions are compared on the instances scored in both.

AssistantAFR as in the paperAFR with the policy re-sent
GPT-5.6-SOL49.856.5
Gemini 3.5 Flash50.946.4
Both50.451.3

The reminder does not lower AFR: it is higher for GPT-5.6-SOL, lower for Gemini 3.5 Flash, and not significantly different overall. Of the conversations, 89.4% end the same way in both conditions, 4.8% fail only without the reminder and 5.8% only with it (exact McNemar p = 0.59). A fading policy therefore does not explain the TrustPrime increase. We can extend the control to the remaining assistants for the camera-ready version upon acceptance.

What we conclude. Every TrustPrime task inherits one of the 17 Base attacks, with its hidden objective, tactic ladder, forbidden actions and authorized resolution, and places it after a completed benign request from the same user (App. D.1). The table above therefore compares each of the 17 attacks in Base with the TrustPrime tasks built from it, and the increase holds in most families and for every assistant: AFR +17.0 and PST −15.1 against Base. We tested possible alternative explanations: source weighting, primer completion and length. None of them explains the increase. This supports the reading of Sec. 5.4 and App. D, in which the completed benign request establishes rapport, confirms the account context and satisfies the ordinary safeguards, the assistant comes to treat the user as legitimate, and the attack that follows builds on that standing. We now report these controls in App. H.3, with pointers from Sec. 5.4 and App. E.7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.