- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The author propose the AgentBoundary framework for tool-using agent safety. It builds four-way families of executable agent tasks around a shared tool by enabling controlled comparisons of the risky-looking but authorised tasks.
Critical strengths
(1) The four way counterfactual setup does a much better job of separating out over-refusal from unsafe compliance than the benign/harmful pairs. (2) The paper reports on the aligner’s failures and includes a tool gating baseline and an untrained wrapper control.
Critical weaknesses
(1) The central assumption that permissibility is determined by evidence that arises during execution is not supported by the data. In the case study the Hard- Benign prompt contains no authorization at all and the plain model' s refusal is a perfectly sensible reaction to that prompt. The tool observations in the case study make no difference to what is permitted. The aligned decisions in both cases are a direct function of the prompt text rather than of any evidence that emerges during execution. The environment is LM- emulated and the rosters produced by the different runs of the prompt are different so the evidence is not held constant across arms. The labels are derived from the generator' s intent and human validation is performed on families not on the actual trajectories that are realised. (2) The measurement chain is not validated at the level that the metrics are applied. HB utility and CM safety are binarised at score> = 2 from an LLM judge. The judge validation reports within- one agreement on a 0 to 3 scale. A one point disagreement at the 1 to 2 boundary will change the binary outcome so 91. 0% and 96. 7% within- one agreement tells us very little about pass/fail agreement. Appendix E reports human- human exact agreement of 49. 6% on helpfulness and within- one agreement of 68. 2% which is lower than the human judge figure and that needs to be explained. The identity of the judge and any potential overlap with the models being evaluated is not discussed in the main text. The aligner is trained on a pool constructed by the same pipeline and on the same action boundary principle as the benchmark and is tested only on that benchmark. The improvements may be due to learning the generator' s disguise patterns. No confidence intervals, seeds or repeated runs are reported.
Questions for authors
- Please report binary threshold agreement (pass/fail at >= 2) between humans and the judge, with kappa, per split.
- What proportion of Hard-Benign refusals happen prior to any tool call and in how many of those is it an observation that provides the authorisation that leads to a change of decision?
- Does the aligner transfer to an external benchmark such as AgentHarm or to a differently generated family set.
- Which model acts as the judge and the emulator and does any of the evaluated configurations share a vendor with it;
- How much run-to-run variance is there for one configuration?
16 comments
4. What the two aligned decisions demonstrate.
Both aligned decisions follow prompt wording, not execution evidence.
Permission evidence can be present in the user request or supplied by tools. In I.2, the user's request delegates the benign announcement workflow within the stipulated legitimate-use context. The roster provides the factual content needed to execute it. AgentBoundary asks whether the action respects the applicable permission and effect. It does not require every task to reveal a new permission after the first tool call.
5. The basis for an LM-emulated environment.
The environment is LM-emulated.
Lines 250–253 in Section 4.1 explicitly follows ToolEmu's LM-emulated sandbox methodology. Related published implementations include:
- ToolEmu emulates observations from tool specifications, call arguments and interaction history to evaluate sensitive tool-use workflows [1].
- PrivacyLens builds on ToolEmu and uses GPT-4 to generate privacy-sensitive observations for calendar, email, messaging and notebook tools [2].
- HAICOSYSTEM uses GPT-4o as an environment engine for tool responses under scenario-specific constraints, including healthcare and finance settings [3].
- StableToolBench uses cached API responses and falls back to LM simulation when neither cached nor live responses are available [4].
These works provide methodological support for AgentBoundary's LM-emulated sandbox. Our evaluation also includes human validation of task semantics and realized tool-use trajectories. Professional annotators assess interaction coherence, tool misuse, task completion and tool-call risk, with repeated annotations and quality review; automatic judge scores are checked against human ratings .
References
[1] Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., Dubois, Y., Maddison, C. J., and Hashimoto, T. . Identifying the Risks of LM Agents with an LM-Emulated Sandbox. International Conference on Learning Representations (ICLR).
[2] Shao, Y., Li, T., Shi, W., Liu, Y., and Yang, D. . PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.
[3] Zhou, X., Kim, H., Brahman, F., Jiang, L., Zhu, H., Lu, X., Xu, F., Lin, Y., Choi, Y., Mireshghallah, N., Le Bras, R., and Sap, M. . HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Interactive AI Agents. Conference on Language Modeling (COLM).
[4] Guo, Z., Cheng, S., Wang, H., Liang, S., Qin, Y., Li, P., Liu, Z., Sun, M., and Liu, Y. . StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024, 11143–11156.
6. Different rosters across arms.
the same prompt yields different rosters across runs , so evidence is not held fixed across arms.
Repeated-run stability. To demonstrate stability directly, we conducted 100 multi-run experiment cases across seven backbones. It showed that the result remained stable across runs. Minor information differences don't affect the result.
| Backbone | Utility pass/fail agreement | Safety pass/fail agreement |
|---|---|---|
| GPT-5.5 | 97.00% | 96.00% |
| Gemini-3.5-Flash | 100.00% | 98.00% |
| Qwen397 | 99.00% | 97.00% |
| Qwen3.8-27B | 97.00% | 99.00% |
| DeepSeek-V4-Flash | 97.00% | 96.00% |
| GLM-5 | 95.00% | 95.00% |
| Nemotron-Nano-9B-v2 | 100.00% | 100.00% |
Independent human validation. The human study checks family semantics and permission boundaries: HB/CM boundary contrast passes 98.0%, and split-safety semantics pass 99.1%. Repeated independent annotations agree on these checks at 96.4% and 99.5%, respectively.
7. Validation of the boundary labels.
Labels (HB authorized, CM forbidden) come from the generator’s intent
Labels are checked through construction constraints and independent human judgments. HB preserves EB's authorized workflow; CM preserves HS's harmful endpoint while changing its framing . The human questionnaire assesses the resulting task families and permission boundaries (Appendix E, Tables 7–8):
| Construction check | Pass rate, 95% CI |
|---|---|
| Sibling consistency | 97.0% |
| |Natural wording|99.7%
| |Harmful-source severity|98.1%
| |HB/CM boundary contrast|98.0%
| |Split-safety semantics|99.1%
|
Human evaluation supports the intended four-way construction and boundary labels. The HB/CM boundary contrast passes 98.0%, and split-safety semantics passes 99.1%.
8. Human validation of realized trajectories.
Human validation checks families, not realized trajectories.
The submitted study includes 660 trajectory-annotation records covering Bare and Aligner runs . We ask annotators who are expertised in agent safety to inspect the actual prompts, tool definitions and realized trajectories. They are required to assess coherence, tool misuse, task completion and risk under common guidelines and quality review.
Moreover, the annotation finds that 97.88% of records mark the interactions as coherent, and 99.24% report no obvious tool misuse. These results support the coherence and tool-use plausibility of the audited interactions. The same annotations also provide a human reference for validating the automatic judge through their completion and risk ratings.
Q1Please report binary-threshold agreement between humans and the judge, with kappa, per split.
Below, we report pass/fail agreement and κ separately for each split at the actual decision threshold. We also compare human–human and human–judge agreement under the same rules on a common archive population. For EB and HB safety, all human reference labels are “safe”; We therefore mark Cohen’s κ for these single-class reference subsets.
| Split / metric | Pass/fail agreement % | κ |
|---|---|---|
| EB utility | 84.18% |
|0.632| |EB safety|97.34%
|N/A| |HB utility|92.94%
|0.854| |HB safety|98.40%
|N/A| |CM utility|88.70%
|0.708| |CM safety|90.48%
|0.618| |HS utility|97.38%
|0.794| |HS safety|98.43%
|0.842|
Human–human versus human–judge agreement on the same episodes. We use the same pass/fail rule and the same repeatedly annotated archive episodes for both comparisons.
| Metric | Human–human pass/fail agreement | Human–judge pass/fail agreement |
|---|---|---|
| Utility | 91.41% | 89.24% |
| Safety | 97.37% | 94.61% |
Dear Reviewer 9KQG,
Thank you for the careful and detailed review. We are greatly encouraged by** your recognition that the four-way counterfactual layout offers a clearer diagnostic of over-refusal and unsafe compliance, and that reporting intervention controls alongside safety trade-offs is valuable. **Below, we reply to all your conerete comments one by one.
W1. Execution evidence, the LM-emulated environment, and human validation
Thank you for the detailed and constructive questions. AgentBoundary evaluates action boundaries using the user's delegation, tool observations and the effects of the proposed action.
1. Permission evidence can be initial or revealed during execution.
The trajectories below show tool-revealed permission facts explicitly informing subsequent decisions. The table gives three original DeepSeek-V4-Flash examples. Permission can also be established via the initial request. New authorization need not emerge in every task (§3; Appendix B).
The agent uses these observations to select permitted resources or withhold restricted actions. The provided examples are trajectory-level examples of evidence use, which do not establish a counterfactual causal effect alone.
2. Semantic authorization in the bulletin case.
This family follows ordinary assistant use. Users delegate tasks through a request and its application context, without necessarily adding a separate authorization statement. The request is to identify a student, describe her achievements and distribute a school announcement through specified channels. Under the family's legitimate-user assumption, task semantics convey delegation for that workflow.
The HB rewrite introduces verbs such as “penetrate,” “zero in,” and “exploit” while preserving the benign bulletin objective. The bare agent interprets them as intrusion and refuses before tool use, which we view as the misclassification under that delegation assumption.
We will state this delegation assumption explicitly in I.2. It is not an additional credential in the archived input. Workspace policy, privacy and recipient restrictions still apply, and the CM harassment workflow remains forbidden. I.1 gives an explicit complementary contrast: The $50,000 invoice workflow operates inside a quarantined test environment in HB, while its harmful counterpart changes and uploads a financial record to the company system.
3. The role of roster observation.
In I.2, the prompt and use context provide the intended delegation. In the example, the roster identifies the student and grounds the bulletin's content. The lookup does not grant publication authority. We will state these roles directly in the case description. Other families reveal further constraints during execution, including the license, verification and disclosure facts shown in point 1.