acceptodds
Under review as a conference paper at ICLR 2027

AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories

Abstract

LLM agents are increasingly taking actions with real consequences, yet safety evaluations often reduce their behavior to a final safe or unsafe outcome, obscuring whether the agent recognized the risk and how it responded. We introduce AURA-Eval, a framework for evaluating whether agents recognize contextual risk and adapt their behavior accordingly. AURA-Eval truncates tool-use trajectories at safety-critical decision points and asks the evaluated model to generate the next thought and action. It creates controlled scenario variations along six risk mechanisms: harm intensity, scale, target susceptibility, system dependency, oversight, and reversibility. It also creates paired scenarios that preserve the underlying goal but differ in whether safe fulfillment is available, allowing unsafe compliance and over-refusal to be evaluated together. An LLM-judge panel, validated against human annotations, evaluates each continuation along three axes: risk detection from the reasoning trace, and action type and safety from the action. From 157 R-Judge seed trajectories, we construct 1,249 evaluation items and evaluate 20 frontier, open-weight, and safety-tuned models. Across all 20 models, unsafe-action rates are higher when no safe path for request fulfillment is available than when one is; for frontier models, the rates are 27-40% versus 5-7%. When a safe path exists, frontier models rarely refuse; when none exists, they most often propose an alternative. Open-weight models, by contrast, more often execute the request than propose alternatives and act unsafely on 58-83% of items. Controlled risk-mechanism analyses further show particularly high unsafe rates under changes to oversight and scale; removing pre-execution oversight flips 52.3% of cases that were safe on the corresponding original scenario to unsafe.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent safety, risk awareness, evaluation framework

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_8ZpnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Targeting the limitation of existing LLM agent safety evaluations that only focus on final outcomes while ignoring risk‑awareness capability and contextual adaptability, this paper proposes the AURA‑Eval evaluation framework. It truncates tool‑use trajectories at critical decision points and prompts models to generate subsequent thoughts and actions. It defines risk mechanisms and paired scenarios (SAFE‑PATH/NO‑SAFE‑PATH), conducts decoupled assessment across three dimensions: risk detection, action type and action safety, and builds a dataset to test 20 models of various types.

Critical strengths

Decoupled evaluation of risk recognition in reasoning and actual action behaviour Most existing agent‑safety evaluations only examine final action outcomes, and cannot differentiate whether a model fails to perceive risks or executes unsafe actions even after identifying risks in reasoning. This framework scores risk detection from reasoning traces separately from actual actions, allowing observation of the failure case where risks are recognised but unsafe behaviour still occurs. Beyond simple pass‑rate statistics, this design yields additional information about failure origins and helps understand the sources of model‑safety issues.

Critical weaknesses
  1. The evaluation heavily relies on the LLM judge panel. Human calibration is insufficient for certain key dimensions and vendor‑related risks cannot be fully ruled out, raising concerns over evaluation validity. Though a human‑annotated validation set of 243 samples is constructed, large‑scale evaluation entirely depends on outputs from LLM judges. The Risk‑Detection axis shows low inter‑annotator agreement (Krippendorff’s α=0.512), and the authors only treat it as a diagnostic auxiliary metric. Borderline behaviours such as A2‑confirmation and A3‑information‑gathering under NO‑SAFE‑PATH scenarios lack dedicated stratified human calibration. While the authors conduct leave‑one‑vendor‑out ablation and detect no statistically significant vendor bias, the test only demonstrates no obvious deviation is observed and cannot definitively eliminate subtle potential vendor‑related bias.

  2. The dataset only covers unintentional, non‑adversarial risks from regular scenarios and lacks adversarial attack samples. All seed trajectories for this framework are derived from unintentional‑risk cases, without evaluations on deliberately constructed agent adversarial attacks. This limits the benchmark’s scope. It can only assess risk awareness under normal tool‑using interactions, but cannot measure model robustness against malicious prompts. Experimental findings cannot be fully generalized to adversarial safety settings.

  3. The risk‑detection metric relies on explicitly output reasoning traces. Risk detection (D0‑D3) is purely judged based on explicit Agent Thought. This dimension cannot be evaluated if the agent produces no thinking text.

Questions for authors

The Risk-Detection axis has human-human Krippendorff's α of only 0.512, below the 0.667 threshold cited in the paper. Given non-trivial annotation noise, please explain why the key observations derived from this axis, which is the D0-D3 vs. unsafe-rate relationship and the "recognises risk yet acts unsafely" failure mode, are not artefacts of that noise, and whether these results still hold after excluding low-reliability annotated samples.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8Zpn

We thank the reviewer for their feedback. Most of these concerns are already addressed in our original submission, and we hope the evidence and clarifications below will help resolve the concerns and support a reconsideration of the overall assessment.

Detection reliability and robustness

The reported α = 0.512 measures agreement across four fine-grained Detection levels, while our key findings concern a simpler distinction: whether agents recognize risk as a concern (D2/D3) or not (D0/D1). On this distinction, human α improves to 0.643, with 91% judge–human agreement.

More importantly, we directly verified that annotation uncertainty does not explain our findings. After excluding the 33% of outputs without unanimous Detection judgments, the D0/D1 versus D2/D3 unsafe-rate difference remains statistically significant across all 20 models (95% CIs exclude zero). This also holds when requiring unanimous Action Safety judgments. The proportion of risk-aware but unsafe outputs changes by at most 4 percentage points per model.

Thus, our findings remain robust even when focusing the analysis on higher confidence judgments. As clarified in §§6 and 8, our primary safety conclusions rely on the more reliably annotated action labels (Action Safety α = 0.759).

Human validation and judge bias

We agree that the reliability of LLM-based evaluation is essential to interpreting our results. This is why our original submission already includes both human validation and a direct test of potential judge bias.

First, our primary Action Safety metric was validated against independent human annotations, achieving human agreement of α = 0.759 and judge-human agreement of 78.6% (§6). This provides empirical validation of the automated safety judgments. We also explicitly evaluate if confirmation (A2) and information gathering (A3) appropriately resolve or prevent the risk (Table 7), although dedicated human calibration of these borderline cases could strengthen the analysis.

Second, we tested if judges favor models from their own vendors and the results suggest that judge-vendor bias does not explain the observed differences in safety performance between model families. Replacing overlapping judges changed safety rates by only 0.4–2.0 percent points, compared with the 12.9 percent point gap between the closest frontier and general open-weight models (Appendix F.4).

Why we focus on unintended risks

Our focus on unintended risks is a deliberate design choice, explicitly motivated in §4.2 and Appendix C.1 and acknowledged in §8. We study how agents respond to risks arising during legitimate operations, without external adversarial manipulation.

This distinction is important for our methodology: keeping the source of risk fixed allows us to examine how changing a specific contextual risk factor affects agent behavior. For example, we show that removing oversight can cause agents to share sensitive customer data they previously refused or gated (Table 34).

We do not claim that these findings generalize to adversarial attacks. As §8 already states, extending the framework to adversarial settings is future work. This defines the scope of our empirical conclusions; it does not undermine the findings established within that scope.

Evaluation without reasoning traces

We directly tested if our findings depend on agents producing explicit reasoning traces. In Appendix J.4, we removed the [AGENT_THOUGHT] requirement and reevaluated 150 stratified scenarios across four models. Our central findings remained unchanged: NSP scenarios consistently produced higher unsafe rates than SP scenarios (16–60 percentage-point gaps), and the frontier–open-weight separation persisted.

This is because AURA-Eval's core evaluation of Action Type and Action Safety relies on agents' actions, not their reasoning traces. As defined in §5 and acknowledged in §8, the ReAct protocol additionally enables Risk Detection, which measures articulated risk awareness. For agents without explicit reasoning, this diagnostic is unavailable, but Action Type and Action Safety remain fully applicable. Thus, explicit reasoning adds diagnostic value to our framework but is not required for its core evaluation or findings.

AuthorsOpenReview ↗

Following up on our response to Reviewer 8Zpn

Dear Reviewer 8Zpn,

Thank you again for raising these important questions. Your questions get at if the patterns we report come from the agents or from the way we evaluate them. Our response highlight the evidence on that distinction from our paper with additional validation checks to strengthen the findings.

With the discussion ending on November 18, we’d appreciate your taking another look. If our response addresses your concerns, we would appreciate your reconsidering the overall assessment. We’re happy to clarify anything further.

Thank you for your time!

Reviewer_jEJYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that current LLM agent safety evals create fixed scenarios that may or may not be representative of deployment time scenarios. To address this they propose AURA-Eval, a new framework for agent safety evaluation that is constructed to study nuanced risks that are context dependent and arise in ordinary use of these agents. The goal is not only to study if agents' responses are safe or unsafe, but also to see if they can recognize the risks and adapt accordingly.

Critical strengths
  • The framework construction is well motivated and appears grounded in well studied risk frameworks with a focus on contextual risk. Specifically, the authors start with a seed agent trajectory and introduce risk at different locations while making minimal changes to the overall semantics of the trajectory.
  • To evaluate the understanding of the risks, the authors also create paired scenarios, one where there is no safe path and another where a safe path exists and refusal is unnecessary.
  • Authors evaluate a broad range of frontier and open-weights models. And the results seem insightful. According to the paper, frontier models are better at recognizing the existence of safe-path and choose them more often than open-weight models.
Critical weaknesses
  • The main weakness is the scale of the eval dataset. It is seeded from 157 R-judge trajectories. It is not clear how representative these are of realistic scenarios we use agents currently.
  • Additionally, since the no-safe-path labels and other metadata are generated synthetically, it is not clear how to reliably scale this approach to a bigger more diverse eval framework.
  • The related work section does have a discussion on how the proposed framework differs, but it may be useful to give some more insights on how this benchmark is more suitable than others in certain situations.
Questions for authors

Could models specifically trained to be guardian or judge models be better during the dataset construction or detection? The agreement even among human laborers on detection seems to not be that high. A discussion on how hard the task is for human and how to scale it would be useful.

Comments on the paper

1 comment

Apologies for the delayed reply. I appreciate the authors’ detailed response and additional analyses. The stratified results address my concern about request-changing pairs, and the mechanism-level analyses clarify the benchmark’s diagnostic value. However, my concern about novelty remains, as the methodological advance beyond prior benchmarks is still limited. Therefore,I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.