acceptodds
Withdrawn from ICLR 2027

CAST: Cascaded Adaptive Safety Triage for Agent Trajectories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety classification of tool-use trajectories requires distinguishing unsafe behavior from legitimate actions whose safety depends on the surrounding request, permissions, and outcomes. However, trajectory-level binary labels do not explicitly reveal these safety-relevant dependencies, creating two challenges: learning a context-sensitive safety boundary from limited supervision and deciding when a prediction requires stronger review. We present CAST (Cascaded Adaptive Safety Triage), a cascaded safety monitor that addresses both. Evidence Construction and Counterfactual Editing creates evidence records and targeted label-flipping trajectory edits to expose local safety boundaries. Privileged Expert Distillation transfers evidence-informed supervision into an Expert that requires only the original trajectory at inference time. Three-Action Routing then learns to predict safe, predict unsafe, or defer difficult cases to the stronger Expert. On AgentSafety-Bench, CAST-Pro achieves 89.82% Macro-F1 and outperforms the compared guardrail and post-training baselines. CAST-Flash retains 86.19% Macro-F1 while reducing Expert calls to 28.50% and Expert-token usage to 30.79% of Pro. Across four additional benchmarks, CAST also improves average Macro-F1 over same-backbone post-training baselines. These results show the value of jointly learning context-sensitive safety judgments and selective Expert review. Code and data are available at https://anonymous.4open.science/r/CAST-ICLR2027.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent safety, trajectory safety monitoring, supervised fine-tuning, on-policy distillation, learning to defer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AfpLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

CAST (Cascaded Adaptive Safety Triage) classifies the safety of agent trajectories by learning context-sensitive safety boundaries and employing an adaptive model cascading strategy. CAST first introduces Evidence Construction and Counterfactual Editing (ECCE) to construct evidence linking the trajectory and counterfactual trajectories with flipped safety labels. The constructed data is then used in two training stages: (1) Privileged Expert Distillation (PED), which trains an expert model through SFT on the original and counterfactual trajectories, followed by privileged distillation using the constructed evidence; and (2) Three-Action Routing (TAR), which trains a lightweight routing model through binary warm-up, expert-informed route supervision, and GRPO. CAST provides two inference modes: CAST-Flash, which uses TAR to either directly classify trajectories or defer them to the expert model for classification, and CAST-Pro, which relies entirely on the expert model for safety classification.

Experiments demonstrate that CAST achieves competitive performance against existing guardrail and post-training baselines. On Agent-SafetyBench, CAST-Pro and CAST-Flash achieve 89.82% and 86.19% Macro-F1 with Qwen3-8B, and 85.29% and 82.77% Macro-F1 with Llama3.1-8B, respectively.

Critical strengths
  1. Effective safety classification with low cost. Safety classification is an important problem. Compared with existing methods, CAST improves safety classification performance at a relatively low cost and achieves promising results across multiple benchmarks.

  2. Comprehensive experimental evaluation. The paper conducts extensive ablation studies to demonstrate the contributions of ECCE, PED, and TAR to the overall performance. Robustness experiments show that CAST is more robust to perturbations than directly using larger models or existing methods. Furthermore, the boundary-edit experiments provide evidence that CAST performs better on safety classification involving context-sensitive safety boundaries.

Critical weaknesses
  1. Counterfactual editing and evidence-informed supervision are established ideas. CAST applies these approaches to safety classification and achieves promising results. However, the paper provides limited discussion of the challenges involved in adapting these approaches to safety boundary identification, particularly the rationale for the eight editing categories and how the validity of the generated counterfactuals is ensured.

  2. The reasons behind the performance improvements are not sufficiently analyzed. Compared with SFT, OPD, and GRPO, CAST-Pro significantly improves Safe Recall while maintaining high Unsafe Recall. Although the ablation studies demonstrate the contributions of ECCE and PED to the improvement in Safe Recall, the paper provides limited insight into why these components lead to such improvements.

Questions for authors
  1. Could the authors further clarify the definition of context-sensitive safety boundaries and which contextual changes affect the safety judgments of agent trajectories? Does counterfactual editing follow any systematic principles?

  2. Could the authors further analyze why CAST improves Safe Recall while maintaining high Unsafe Recall?

Reviewer_CH68OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAST, a monitor that reviews completed AI-agent trajectories and judges whether they were safe or unsafe. Noting that the same action can be safe or unsafe depending on context, it combines (1) targeted label-flipping edits (ECCE), (2) distillation from a teacher that sees supporting evidence (PED), and (3) a small router that decides directly or defers to an expert (TAR), aiming to balance accuracy and monitoring cost. The authors offer two variants: CAST-Pro, which applies the expert to every trajectory, and CAST-Flash, which uses the router to call the expert only when needed. They evaluate CAST mainly on Agent-SafetyBench and also on four additional benchmarks (ATBench, ATBench-Codex, R-Judge, and ToolSafe), comparing it with guardrail methods, post-training methods using the same backbone, and closed-source models. They also report ablations, tests against edits and surface changes to trajectories, and an analysis of routing behavior. Based on these results, the authors claim that CAST-Pro achieves the best Macro-F1 on Agent-SafetyBench, that CAST-Flash keeps most of this performance while calling the expert for under 30% of trajectories, and that CAST generalizes better than the post-training baselines. The reviewer understands that the risks it targets are mainly careless mistakes, not deliberate “rogue AI” behavior.

Critical strengths

The paper identifies a real problem: whether an agent’s action is safe often depends on the situation, not just on the action itself. It also explains why simple safe/unsafe labels are not enough, since a model trained only on them may learn shortcuts such as specific words or tool names. To fix this, the authors create pairs of nearly identical trajectories with opposite labels, and they had humans check the quality of this generated data. The extra explanations are used only during training, so they are not needed in real use. The faster version (Flash) calls the large model for only about 30% of cases while keeping accuracy reasonably close to the full version (Pro). The paper also (1) reports results for each class, (2) tests the effect of removing each component, and (3) openly reports the results.

Critical weaknesses

Although CAST is a monitor, its main evaluation uses Agent-SafetyBench, which was originally designed to test agents, not monitors. Training, threshold tuning, and testing all use data from this same benchmark (Table 3). The test set has only 400 cases, and the comparison with closed models not trained on this data is unequal (Appendix I.1). ASSEBench, a common benchmark for monitors, is not used or even mentioned, and ATBench and R-Judge are used only as subsets without explaining how they were selected (Table 3). CAST appears to be built mainly by combining known techniques, such as counterfactual data augmentation, knowledge distillation, and learning to defer. Moreover, RQ5 (Section 4.6; Appendix J.2) suggests that TAR performs no better than simple uncertainty-based routing, so the benefit of its complex training remains unclear.

Questions for authors

The reviewer summarizes the questions for authors here.

  1. Why Agent-SafetyBench as the primary benchmark? CAST addresses a monitoring task, judging recorded agent trajectories, yet its primary evaluation uses Agent-SafetyBench, whose main purpose is to evaluate the safety of agents themselves. We understand that this benchmark also includes human-labeled interaction records. However, given that benchmarks designed for monitoring already exist (e.g., R-Judge, ASSEBench, and ATBench), could you explain why Agent-SafetyBench was chosen as the primary evaluation?

  2. Why were ASSEBench and AgentDoG not included? The paper uses AgentAuditor as a baseline but does not mention ASSEBench, which was introduced in the same work. Could you explain why CAST was not evaluated on ASSEBench? In addition, the paper cites AgentDoG in its related work and uses ATBench, which is associated with the same research group, for generalization. Could you explain why AgentDoG, a trajectory-level safety judge, was not included as a baseline?

  3. Origin of the Agent-SafetyBench trajectories Which agents produced the 1,997 trajectories derived from Agent-SafetyBench? Whether these are the human-labeled interaction records from the original paper or trajectories generated by the authors affects how the results should be interpreted.

  4. Construction of the generalization test sets ATBench and R-Judge are each reduced to 300 cases (150 safe and 150 unsafe). How were these subsets sampled? Could you explain why the classes were balanced 1:1 rather than kept at their original ratios, and share results on the full datasets if available? We would also like to know why ToolSafe, which is evaluated at the action level, is averaged with equal weight alongside the three trajectory-level datasets.

  5. Isolating the effect of TAR training In the matched-budget comparison in RQ5, TAR showed no significant difference from uncertainty-based routing. However, according to Appendix J.2, this uncertainty baseline uses the probabilities of the TAR-trained Flash router, so it does not isolate the effect of TAR training itself. Since the paper notes that ablation A8 uses a different budget and selection rule, could you report results for routing by the entropy of a binary router without TAR training, under the same 30% expert budget?

  6. Optimistic bias in expert correctness when constructing route labels The route labels (Equation 3) are assigned over the full augmented training set, with defer assigned when the expert is correct. However, the expert was trained on this same data in Stage A, so judging its correctness there is likely to overestimate its ability. The cached expert predictions used for the GRPO reward appear to share this issue. In addition, the warm-up router is trained on the same data and then scored on it. How do you think these biases affect the defer labels and the learned routing policy?

  7. Inconsistent deferral counts The abstract and Section 4.3 report an expert-call rate of 28.50%, which matches the 114 deferrals (94 resolved correctly by the expert) in Table 14 of Appendix G.2 and in Appendix D.2. However, Section 4.6 and Figure 4(b) report 117 deferrals (29.25%), of which the expert correctly resolves 100. If these come from different runs, could you clarify which numbers correspond to which run?

Reviewer_Rhq9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CAST is a post-hoc safety monitor that classifies completed agent tool-use trajectories as safe or unsafe. The problems that the paper targets are binary trajectory labels don’t reveal which contextual factors determine safety, and running a strong monitor on every trajectory is costly.

CAST has three stages - ECCE (Evidence Construction and Counterfactual Editing), PED (Privileged Expert Distillation), and TAR (Three-Action Routing)

There are two deployment modes. CAST-Pro runs the expert on every input. CAST-Flash calls the expert only when the router defers.

The paper evaluates on Agent-SafetyBench (400-trajectory test split) and four generalization sets (ATBench, ATBench-Codex, R-Judge, ToolSafe). It compares against re-implemented guardrails (AgentAuditor, AGrail, SafeHarbor, LlamaFirewall), post-training baselines (SFT, OPD, GRPO) and eight closed-source zero-shot judges. CAST-Pro reaches 89.82 Macro-F1, compared with 87.03 for the best baseline (GRPO). CAST-Flash reaches 86.19 while calling the expert on 28.5% of inputs at about 0.53× Pro’s token cost.

Critical strengths

None

Critical weaknesses
  1. Unequal training data distribution for different methods - CAST and baselines are not trained on same data
  2. Overstated “joint” learning: the expert and router are trained one after the other with the expert frozen, and CAST-Pro doesn’t use the router at all.
  3. Small test set and limited statistics: the primary test set has 400 examples, there are only 3 seeds, and closed-source judges are single runs with unspecified prompts.
  4. Only DeepSeek-v4-Flash is used to generate all the explanations.
  5. the motivation to use above method is not properly stated
  6. Internal inconsistency: the deferral count is 117 (29.25%) in Section 4.6 but 114 (28.50%) in Table 14 and App. D.2.
Questions for authors

The motivation behind using different dataset for training CAST vs baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.