acceptodds
Under review as a conference paper at ICLR 2027

StepSafe: Learning Timely Intervention with Context-Aware Safety Reasoning

Abstract

Safety in agentic systems is inherently sequential: whether an action is safe depends on prior actions, tool outputs, information disclosures, and interactions among agents. Recent work has shifted agent safety guardrails from post-hoc trajectory-level assessment toward step-level action diagnosis and proactive risk prediction, yet existing training objectives provide limited supervision for how step evidence accumulates over time and when it becomes sufficient to justify intervention. We propose \methodlong (ASPO): a training framework for step-level timely intervention. Pre-fix level stage alignment allows crediting each decision at its own step: it compares each response with the other responses sampled at the same prefix and rewards the action that fits the stage, continuing before the risk is visible, asking while clarification can still avert harm, and stopping before harm is irreversible. Adaptive feedback self-distillation further credits the reasoning behind each decision, with the model itself as a teacher that sees the step annotations on intervention steps and rules mined from its own false alarms on safe steps. Its training data carry a violation taxonomy and an intervention window at every step and are extended through self-evolving multi-agent red-teaming. Finally we present StepSafe, a step-level reasoning guardrail with timely intervention, that achieves the state-of-the-art peformance across five agent-safety benchmarks on safety-utility pareto curve among open-source guardrails with Youden's of 50.6 (4B) and 53.2 (8B) against 46.2.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agentic safety, safety monitoring, post-training, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vTZZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper trains a step-level safety monitor that judges each proposed agent action before it executes, with the goal of intervening not only correctly but in time. The key observation is that existing labels say whether a trajectory or step is unsafe, but not when the risk first becomes visible or until when intervention can still prevent harm. The authors address this by annotating every step with a four-axis violation taxonomy and an intervention window (signal s, boundary b, last preventable step h), and extending the corpus through multi-agent red-teaming with matched benign twins. They then propose ASPO, which combines stage-aligned policy optimization (SAPO) — scoring each decision by its stage, comparing it only against responses sampled at the same prefix, and balancing safe and intervention steps — with adaptive feedback self-distillation (AFSD), where the model acts as its own teacher using step annotations on intervention steps and rules mined from its own false alarms on safe steps. The resulting monitor, STEPSAFE, achieves J = 50.6 (4B) and 53.2 (8B) across five agent-safety benchmarks, outperforming open-source guardrails such as StepGuard (46.2).

Critical strengths

The reformulation from "whether an action is unsafe" to "when intervention becomes warranted" is well motivated, and the intervention window (s, b, h) is a clean and reusable formalization of timeliness.

SAPO's prefix-local credit assignment directly targets known failure modes of trajectory-level and naive step-level rewards, and the ablations (Table 3) isolate the contribution of each design choice convincingly.

The false-alarm rubric is an elegant answer to the lack of supervision on safe steps: rules mined from the monitor's own false alarms, admitted only if they clear no known violations, including the unsafe counterparts of matched benign twins.

The evaluation is fairly comprehensive: five benchmarks, eight published monitors plus API models, and ablations covering credit estimators, feedback components, model scale, seed variance, and operating points.

Critical weaknesses

The training annotations rely largely on GPT-5.6 Luna, and no human verification of label accuracy is reported. This may introduce annotator bias into training. To my knowledge, some of the safe/unsafe labels in the training data are inaccurate, so human verification would be important.

The paper does not include a case study. It is difficult for readers to see how StepSafe operates — how it prevents harm and provides early warnings. Some good examples would make the method much easier to follow.

Figure 3 may be somewhat inaccurate or biased, because the benchmarks authors utilize havde different accuracy and quality. I would suggest drawing the Pareto frontier separately for each benchmark. The ASSE benchmark in particular has apparent quality issues: in ASSE-Security, many agent responses appear to be API error messages such as "OpenAI STATUS error 500: (e.response)", while ATBench and R-Judge look considerably cleaner.

Figures 1 and 2 are somewhat hard to follow and generated by AI. I would suggest redrawing them in a revised version to improve readability.

Questions for authors

The raw data formats of these benchmarks appear to differ considerably. Could the authors clarify how the trajectories from different benchmarks were converted into a unified format, and how the evaluation was kept consistent across them?

1 comment

Thank you for your response. My main reason for raising the third weakness was that inaccuracies in the benchmark could introduce substantial measurement error, making it difficult to draw reliable conclusions about model performance from the benchmark scores. Overall, most of my concerns have been addressed, I have raised my score to 3.

Reviewer_kuzkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on agent safety in multi-step workflows. Existing methods mostly look at the whole trajectory, which makes it hard to tell which step should be responsible and when to intervene. The authors propose StepSafe trained using ASPO with two main components. Stage-Aligned Policy Optimization (SAPO) uses rewards for different stages and compares decisions at the same prefix to better learn when to intervene. Adaptive Feedback Self-Distillation (AFSD) uses additional feedback to improve the decisions and reduce false alarms. The method is trained on annotated trajectories, with extra data from multi-agent red-teaming. The experimental results are generally good, especially compared with other open-source guardrails.

Critical strengths
  1. The focus on step-level safety and intervention timing is valuable. The stage-aware design is well motivated and fits the problem.

  2. The experimental evaluation is thorough, covering multiple benchmarks and detailed ablations. The results show consistent improvements over existing open-source methods.

Critical weaknesses
  1. The method depends quite a lot on annotated intervention windows and predefined rewards. It is not clear how well it can generalize to new tasks or safety settings without more annotations.

  2. Most of the evaluation is offline. It is still unclear whether the improvements would hold in real interactive settings, where interventions may change what happens next.

  3. The organization feels a bit inconsistent. The abstract, introduction, and method section each present the contributions in a different way, so the structure does not quite line up across sections.

Questions for authors
  1. The method relies quite heavily on annotated intervention windows and predefined rewards. How well does it generalize to new tasks or safety settings without additional annotations? How much new annotation would be needed to adapt it?

  2. SAPO optimizes step-level rewards on fixed trajectories, but interventions like ASK may change what happens next. Would the improvements still hold in an interactive setting? Is there any evidence that the lower offline ASR translates to fewer actual harmful outcomes?

  3. In Table 3, stage-aware rewards improve utility but also increase ASR compared to binary rewards. Is this mainly a shift in the operating point, or does it actually improve the safety-utility Pareto frontier? A comparison at matched utility or ASR would help.

  4. How well does StepSafe handle risks that only become harmful when several individually safe actions are combined, especially across multiple agents? It would be useful to see results on this type of case.

  5. For adaptive red-teaming, how much of the improvement comes from the adaptive process rather than simply adding more synthetic data? A comparison with non-adaptive generation under the same data budget would help. Also, can it discover new failure types beyond the predefined taxonomy?

Reviewer_H5QvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces STEPSAFE, a step-level safety guardrail for agentic systems that aims to intervene at the appropriate time before an unsafe action causes harm. The authors propose Adaptive Stage-Aligned Policy Optimization (ASPO), which combines Stage-Aligned Policy Optimization (SAPO) for assigning decision-level credit according to intervention timing with Adaptive Feedback Self-Distillation (AFSD) for providing fine-grained reasoning supervision on both unsafe and safe steps. The approach also constructs a taxonomy-grounded dataset with intervention-window annotations and expands training coverage through multi-agent red-teaming.

Critical strengths

The proposed intervention-window formulation explicitly models when risk becomes observable and when intervention can still prevent harm, providing a meaningful training signal beyond trajectory-level labels. The method combines stage-aware decision optimization with token-level self-distillation, and the ablation studies provide useful evidence for the contributions of these components. Evaluation across five benchmarks, multiple model scales, safety–utility operating points, and temporal intervention metrics supports the practical potential of the proposed guardrail.

Critical weaknesses

The proposed framework combines several complex components, including stage-dependent rewards, class-balanced policy optimization, self-distillation, and dynamically mined false-alarm rules, making it difficult to isolate the source of the overall improvement and assess the generality of the method.

Questions for authors

How sensitive is STEPSAFE's performance to errors in the automatically generated intervention-window annotations, particularly the signal and last-preventable-step labels? Can the authors provide human-annotation agreement or an evaluation of robustness to annotation noise?

How much does each component of ASPO contribute under controlled comparisons, particularly SAPO's stage-dependent reward schedule, class-balanced weighting, and AFSD's adaptive false-alarm rubric? Have the authors evaluated whether simpler combinations achieve comparable results?

The false-alarm rubric is mined from the model's own errors and checked using the same model family; how do the authors prevent incorrect or overgeneralized rules from suppressing legitimate safety interventions on unseen cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.