- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Targeting the limitation of existing LLM agent safety evaluations that only focus on final outcomes while ignoring risk‑awareness capability and contextual adaptability, this paper proposes the AURA‑Eval evaluation framework. It truncates tool‑use trajectories at critical decision points and prompts models to generate subsequent thoughts and actions. It defines risk mechanisms and paired scenarios (SAFE‑PATH/NO‑SAFE‑PATH), conducts decoupled assessment across three dimensions: risk detection, action type and action safety, and builds a dataset to test 20 models of various types.
Critical strengths
Decoupled evaluation of risk recognition in reasoning and actual action behaviour Most existing agent‑safety evaluations only examine final action outcomes, and cannot differentiate whether a model fails to perceive risks or executes unsafe actions even after identifying risks in reasoning. This framework scores risk detection from reasoning traces separately from actual actions, allowing observation of the failure case where risks are recognised but unsafe behaviour still occurs. Beyond simple pass‑rate statistics, this design yields additional information about failure origins and helps understand the sources of model‑safety issues.
Critical weaknesses
-
The evaluation heavily relies on the LLM judge panel. Human calibration is insufficient for certain key dimensions and vendor‑related risks cannot be fully ruled out, raising concerns over evaluation validity. Though a human‑annotated validation set of 243 samples is constructed, large‑scale evaluation entirely depends on outputs from LLM judges. The Risk‑Detection axis shows low inter‑annotator agreement (Krippendorff’s α=0.512), and the authors only treat it as a diagnostic auxiliary metric. Borderline behaviours such as A2‑confirmation and A3‑information‑gathering under NO‑SAFE‑PATH scenarios lack dedicated stratified human calibration. While the authors conduct leave‑one‑vendor‑out ablation and detect no statistically significant vendor bias, the test only demonstrates no obvious deviation is observed and cannot definitively eliminate subtle potential vendor‑related bias.
-
The dataset only covers unintentional, non‑adversarial risks from regular scenarios and lacks adversarial attack samples. All seed trajectories for this framework are derived from unintentional‑risk cases, without evaluations on deliberately constructed agent adversarial attacks. This limits the benchmark’s scope. It can only assess risk awareness under normal tool‑using interactions, but cannot measure model robustness against malicious prompts. Experimental findings cannot be fully generalized to adversarial safety settings.
-
The risk‑detection metric relies on explicitly output reasoning traces. Risk detection (D0‑D3) is purely judged based on explicit Agent Thought. This dimension cannot be evaluated if the agent produces no thinking text.
Questions for authors
The Risk-Detection axis has human-human Krippendorff's α of only 0.512, below the 0.667 threshold cited in the paper. Given non-trivial annotation noise, please explain why the key observations derived from this axis, which is the D0-D3 vs. unsafe-rate relationship and the "recognises risk yet acts unsafely" failure mode, are not artefacts of that noise, and whether these results still hold after excluding low-reliability annotated samples.
2 comments
Following up on our response to Reviewer 8Zpn
Dear Reviewer 8Zpn,
Thank you again for raising these important questions. Your questions get at if the patterns we report come from the agents or from the way we evaluate them. Our response highlight the evidence on that distinction from our paper with additional validation checks to strengthen the findings.
With the discussion ending on November 18, we’d appreciate your taking another look. If our response addresses your concerns, we would appreciate your reconsidering the overall assessment. We’re happy to clarify anything further.
Thank you for your time!
Response to Reviewer 8Zpn
We thank the reviewer for their feedback. Most of these concerns are already addressed in our original submission, and we hope the evidence and clarifications below will help resolve the concerns and support a reconsideration of the overall assessment.
Detection reliability and robustness
The reported α = 0.512 measures agreement across four fine-grained Detection levels, while our key findings concern a simpler distinction: whether agents recognize risk as a concern (D2/D3) or not (D0/D1). On this distinction, human α improves to 0.643, with 91% judge–human agreement.
More importantly, we directly verified that annotation uncertainty does not explain our findings. After excluding the 33% of outputs without unanimous Detection judgments, the D0/D1 versus D2/D3 unsafe-rate difference remains statistically significant across all 20 models (95% CIs exclude zero). This also holds when requiring unanimous Action Safety judgments. The proportion of risk-aware but unsafe outputs changes by at most 4 percentage points per model.
Thus, our findings remain robust even when focusing the analysis on higher confidence judgments. As clarified in §§6 and 8, our primary safety conclusions rely on the more reliably annotated action labels (Action Safety α = 0.759).
Human validation and judge bias
We agree that the reliability of LLM-based evaluation is essential to interpreting our results. This is why our original submission already includes both human validation and a direct test of potential judge bias.
First, our primary Action Safety metric was validated against independent human annotations, achieving human agreement of α = 0.759 and judge-human agreement of 78.6% (§6). This provides empirical validation of the automated safety judgments. We also explicitly evaluate if confirmation (A2) and information gathering (A3) appropriately resolve or prevent the risk (Table 7), although dedicated human calibration of these borderline cases could strengthen the analysis.
Second, we tested if judges favor models from their own vendors and the results suggest that judge-vendor bias does not explain the observed differences in safety performance between model families. Replacing overlapping judges changed safety rates by only 0.4–2.0 percent points, compared with the 12.9 percent point gap between the closest frontier and general open-weight models (Appendix F.4).
Why we focus on unintended risks
Our focus on unintended risks is a deliberate design choice, explicitly motivated in §4.2 and Appendix C.1 and acknowledged in §8. We study how agents respond to risks arising during legitimate operations, without external adversarial manipulation.
This distinction is important for our methodology: keeping the source of risk fixed allows us to examine how changing a specific contextual risk factor affects agent behavior. For example, we show that removing oversight can cause agents to share sensitive customer data they previously refused or gated (Table 34).
We do not claim that these findings generalize to adversarial attacks. As §8 already states, extending the framework to adversarial settings is future work. This defines the scope of our empirical conclusions; it does not undermine the findings established within that scope.
Evaluation without reasoning traces
We directly tested if our findings depend on agents producing explicit reasoning traces. In Appendix J.4, we removed the
[AGENT_THOUGHT]requirement and reevaluated 150 stratified scenarios across four models. Our central findings remained unchanged: NSP scenarios consistently produced higher unsafe rates than SP scenarios (16–60 percentage-point gaps), and the frontier–open-weight separation persisted.This is because AURA-Eval's core evaluation of Action Type and Action Safety relies on agents' actions, not their reasoning traces. As defined in §5 and acknowledged in §8, the ReAct protocol additionally enables Risk Detection, which measures articulated risk awareness. For agents without explicit reasoning, this diagnostic is unavailable, but Action Type and Action Safety remain fully applicable. Thus, explicit reasoning adds diagnostic value to our framework but is not required for its core evaluation or findings.