TACT-Bench: Testing Adaptive Conversational Threats against Tool-Use Agents

The growing autonomy of AI agents in using external tools and making decisions has created an urgent need for comprehensive safety benchmarks. Existing benchmarks test harmful goals, compromised tool channels, or adaptive multi-turn attacks; however, they may not capture the unique capabilities of policy-governed tool-use agents, who must resist unauthorized requests without abandoning the legitimate work embedded in the same interaction. Existing benchmarks provide limited evidence about persistent, policy-blind social engineering that seeks a concrete tool outcome. To fill this gap, we introduce **TACT-Bench**, a stateful benchmark built around scenario-specific stories, personas, manipulation tactics, and response-adaptive escalation. Each adversarial task targets an action explicitly prohibited by the policy provided to the assistant, while the policy remains hidden from the attacking user. TACT pairs each adversarial task with at least one benign counterpart, and its adversarial tasks retain authorized ground-truth actions when a compliant resolution remains available. We define these base tasks as TACT-Base. We additionally introduce TACT-TrustPrime, which places a complete independent task before an unrelated attack in the same conversation. We also propose Paired Selective Trustworthiness (PST), which combines explicit safe refusal with demonstrated capability on matched benign tasks, to detect overcautious and impractical agents. Averaged across assistants, TACT-Base raises Attack-Induced Failure Rate (AFR) by 17.53% over four compatible attack baselines, and TACT-TrustPrime raises it by a further 17.01% (31.08% over Foot-In-The-Door, the strongest baseline). The two evaluated defense methods reduce AFR by only 8.35%–9.41% on average and do not consistently improve PST, underscoring the need for stronger policy-enforcement methods.