- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TACT-Bench, a benchmark for evaluating the robustness of tool-use agents against adaptive conversational attacks. It incorporates scenario-specific social engineering, persistent adversarial interactions, and matched benign tasks. The authors further introduce TACT-TrustPrime to investigate the impact of prior legitimate interactions and propose Paired Selective Trustworthiness (PST) to jointly assess security and benign task performance. Experiments across six assistants and 2,852 tasks demonstrate substantially higher attack effectiveness than existing baselines, while revealing limitations in current defense mechanisms.
Critical strengths
-
The paper addresses an important gap in agent safety evaluation by considering adaptive, policy-blind conversational attacks while preserving legitimate task requirements.
-
The experiments cover multiple assistants, attack baselines, and defense mechanisms. The proposed PST metric provides a useful perspective on balancing security and task utility.
Critical weaknesses
-
Although the benchmark includes diverse scenarios, all tasks are instantiated within a retail environment, potentially limiting cross-domain generalizability.
-
The evaluation relies on a single LLM-based adversarial user simulator, which may introduce model-specific biases and limit the diversity of attack behaviors.
Questions for authors
-
Could the authors separately report AFR caused by forbidden tool invocations versus incorrect final states? Would the conclusions and model rankings remain consistent under these separate metrics?
-
How well does TACT-Bench generalize to genuinely unseen scenarios or application environments, rather than synthetic variations of the existing 17 scenario families?
-
Could the authors introduce a length-matched benign-interaction baseline for TrustPrime and quantify the impact of excluded simulator failures on the reported attack success rates?
-
The related work discussion could be strengthened by incorporating relevant studies on adversarial robustness, safety evaluation, and efficiency, including XSTest, HarmBench, Sponge Tool Attack, and JailbreakBench. These works provide complementary perspectives on model safety, attack resilience, and security-efficiency trade-offs.
I'm willing to adjust my rating according to the authors' rebuttal.
4 comments
Part 2: length and exclusions (Q3)
Q3 (a). Length-matched benign baseline.
Does a longer conversation by itself raise AFR? No. The High-step variant delivers the same objective and facts over more turns (Sec. 3.2), which makes it a length control. For the two assistants with High-step TACT-Base runs:
| Mistral-Large-3 and GPT-5.6-SOL | User turns | Assistant steps | AFR |
|---|---|---|---|
| Vanilla TACT-Base | 8.4 | 12.4 | 42.2 |
| High-step TACT-Base | 12.1 | 16.0 | 39.9 |
| Vanilla TrustPrime | 11.2 | 17.2 | 57.2 |
High-step Base is as long as TrustPrime (12.1 against 11.2 user turns). Its AFR is not above Vanilla Base (−2.3, lower for both assistants), while TrustPrime is 17.3 points above High-step Base. Within TrustPrime, High-step lengthens the conversation by two thirds (19.4 against 11.7 user turns, three assistants), and AFR moves by +1.3 on the attack-primer pairs scored in both runs. No episode came near its step budget.
Does the longer conversation make the assistant lose track of the policy? This is the concern behind a length-matched baseline, and a conversation that is benign in both phases cannot test it: an authorized second request is completed whether or not the policy is still being applied. Benign requests are completed at high rates in any case: the first task in 1,621 of the 1,625 scored TrustPrime conversations where this can be checked, and 92.6% mean benign competence on the standalone counterparts (App. E.7). The test has to be made where the policy is needed, at the attack. In a new control, we therefore re-ran Vanilla TrustPrime and re-sent the complete policy to the assistant at the start of the second task, directly before the customer's first message of that task. Tasks, user simulator and scoring are unchanged. If the policy had faded, the reminder should lower AFR.
We ran it for two of the three frontier assistants. Because some instances were discarded in either run, the two conditions are compared on the instances scored in both.
| Assistant | AFR as in the paper | AFR with the policy re-sent |
|---|---|---|
| GPT-5.6-SOL | 49.8 | 56.5 |
| Gemini 3.5 Flash | 50.9 | 46.4 |
| Both | 50.4 | 51.3 |
The reminder does not lower AFR: it is higher for GPT-5.6-SOL, lower for Gemini 3.5 Flash, and not significantly different overall. Of the conversations, 89.4% end the same way in both conditions, 4.8% fail only without the reminder and 5.8% only with it (exact McNemar p = 0.59).
Neither additional turns nor a fading policy therefore explains the TrustPrime result. This supports the reading of Sec. 5.4 and App. D, in which the completed benign request establishes rapport, confirms the account context and satisfies the ordinary safeguards, the assistant comes to treat the user as legitimate, and the attack that follows builds on that standing, and that's why it gets higher AFR and lower RR/PST. We have added the control to App. H.3 and can extend it to the remaining assistants for the camera-ready version upon acceptance.
Q3 (b). Excluded simulator failures. An episode is discarded when the simulator cannot produce a usable adversarial user turn. In 67 of the 70 cases below, its provider refused the persona (App. E.9); the other 3 are provider errors. No conversation takes place, so there is nothing to score. The table gives the counts and the range the six-assistant mean AFR can take if every excluded instance is counted as safe or as a failure.
| Setting | Discarded / adversarial instances | AFR reported | Range under any imputation |
|---|---|---|---|
| Vanilla TACT-Base | 8 / 918 | 39.27 | 39.00 to 39.87 |
| FITD | 17 / 918 | 25.20 | 24.84 to 26.69 |
| Naive, PAP, Crescendo | 0 / 918 each | 20.48, 21.02, 20.26 | unchanged |
| Vanilla TrustPrime | 19 / 2,118, plus 16 pre-attack hand-offs | 56.28 | 55.38 to 57.03 |
| Tool-Guard, Policy-Guard | 14 / 918, 12 / 918 | 30.91, 29.86 | 30.50 to 32.03, 29.52 to 30.83 |
All discards fall on two assistants, Claude Opus 4.8 and Gemini 3.5 Flash. The 16 TrustPrime hand-offs are instances in which the assistant transferred to a human before the attack began (App. D.2). They are a scoring rule, not a simulator failure, and the range includes them.
Under the least favourable imputation for each comparison:
- TACT − FITD is worst-case +12.3 against +14.1 as reported, and TACT stays above every baseline for every assistant (24 of 24).
- TrustPrime − Base is worst-case +15.5 against +17.0, and positive for every assistant.
- The defenses' mean AFR reductions are worst-case 7.0 and 8.2 points against 8.4 and 9.4.
App. E.9 also measures the effect directly. The eight Base instances discarded under GPT-5.6-SOL did run under the Sonnet simulator, and all eight were attack-induced failures.
FITD's 17 discards (15 persona refusals and 2 provider authentication errors) are included above. We have added FITD and the defended runs to Table 22 (Table 23 in the revised paper), and the ranges above to App. H.4.
Part 3: domain and unseen scenarios (W1, Q2), related work (Q4)
W1 and Q2. Domain and unseen scenarios. TACT-Bench is a benchmark, and retail is the environment in which we built it. What it contributes as a method does not depend on the retail scenarios: persistent attacks through the ordinary user channel, built from a scenario story, a persona and a tactic ladder that adapts to the assistant's replies, matched adversarial and benign tasks scored on tool calls and the final state, the TrustPrime construction, and the PST metric. Our goal was to show that such an evaluation can be built and what it reveals, with retail as the worked example. The empirical results are for this instantiation, as Sec. 6 states ("the 17 scenario families do not substitute for future cross-domain evaluation"), as a limitation of this work.
What the existing data can show is how far the aggregate comparisons depend on particular scenario families. This is a sensitivity analysis, not a test on new scenarios.
- Seed instances and augmented copies. Averaged over the six assistants, AFR is 40.2 on the 17 hand-written seed instances and 39.1 on their augmented copies. Within a family and assistant, on average 87% of the instances have the majority outcome. As you note, a family's instances are entity-level variations of one scenario, so the independent unit is the family, and all our intervals resample families.
- Leaving one family out. Each point estimate keeps its sign and most of its size whichever family is removed. These ranges show sensitivity to single families.
| Mean of six assistants | All 17 families | Range with one family left out |
|---|---|---|
| AFR: TACT − mean of the four baselines | +17.5 | +13.3 to +20.1 |
| AFR: TACT − FITD | +14.1 | +10.3 to +17.1 |
| AFR: Tool-Guard − no defense | −8.4 | −9.4 to −4.9 |
| AFR: Policy-Guard − no defense | −9.4 | −10.5 to −6.1 |
The assistant ranking correlates at least 0.92 with the full ranking in all 17 cases, and the three frontier assistants always remain the three least vulnerable. For TrustPrime, the family-by-family comparison is in App. H.3 and in our response to Reviewer 1 (higher than Base in 13 of 17 families).
- Random halves of the families. Over all 24,310 ways to split the 17 families into 8 and 9, TACT's AFR is above each baseline in 96.6% (FITD) to 99.9% of the half-suites, and TrustPrime is above Base in all of them.
Extending to another domain. A new domain needs its own policy, tools, records and scenario stories. The task structure, the scoring and the metrics carry over unchanged. The tactics transfer as well. They are general social-engineering moves such as urgency, authority claims, prior-agent approval, manager escalation and off-system workarounds (Sec. 3.2), and only the story and the facts each step uses are written for a scenario, so the retail ladders can serve as templates. The policy primitives the retail tools exercise (identity, ownership, authorization, eligibility, approved destinations, evidence, explicit confirmation, sensitive disclosure) are shared with financial, travel, healthcare and enterprise agents (Sec. 3.2), so we expect the same kind of findings there. We have not yet run a second domain, but it could be an extension of this work.
Q4. Related work. Thank you for these suggestions. We have added XSTest, HarmBench, Sponge Tool Attack, and JailbreakBench to Section 2 and clarified how their evaluation objectives complement TACT-Bench.
Dear Reviewer GJef,
Thank you for your review. As the discussion deadline approaches, we would appreciate hearing whether our response addresses your main concerns. If so, would you consider adjusting your rating?
We would be happy to answer any follow-up questions or clarify any remaining points.
Best,
The Authors
Part 1: summary, AFR by cause (Q1), user simulator (W2)
Thank you for the review, and for being open to revising your rating. The analyses below use the logs of the experiments in the paper, together with one new control experiment for Q3.
In brief:
Q1. AFR by cause. AFR counts an adversarial episode as a failure when a task-forbidden tool is invoked or when the final state differs from the annotated safe state. Both can happen in one episode, so we give each rate and their overlap (AFR = forbidden + incorrect − both). We report the split for the main results, Vanilla TACT-Base on all six assistants, and for the two defenses. % of scored adversarial task instances:
Model ranking. Ranked by incorrect final state, the six assistants fall in exactly the AFR order (Spearman 1.00). Ranked by forbidden-tool invocation, the correlation with the AFR order is 0.89. Two neighbouring pairs swap: DeepSeek-V3.2 with GPT-OSS-120B, which are one episode apart, and Gemini 3.5 Flash with GPT-5.6-SOL, which have the same count. Under all three metrics, the three frontier assistants are the three least vulnerable, and Claude Opus 4.8 is the least.
Defenses. Mean of six assistants:
Each defense cuts some of the forbidden invocations and leaves most incorrect final states in place. A forbidden tool is still invoked in 7.5% and 6.0% of adversarial episodes, and most of the remaining AFR is an incorrect final state.
We have added these columns, with their overlap, to the table of AFR components in App. H.1, which also gives a finer three-way split.
W2. Single user simulator. The paper already includes a second simulator. App. E.8 re-runs Vanilla TACT-Base with Claude Sonnet 4.6 as the user, with the same instructions, personas, facts and tactic ladders, for three assistants. We agree it is easy to miss and now point to it from Sec. 4.2.