- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
CAST (Cascaded Adaptive Safety Triage) classifies the safety of agent trajectories by learning context-sensitive safety boundaries and employing an adaptive model cascading strategy. CAST first introduces Evidence Construction and Counterfactual Editing (ECCE) to construct evidence linking the trajectory and counterfactual trajectories with flipped safety labels. The constructed data is then used in two training stages: (1) Privileged Expert Distillation (PED), which trains an expert model through SFT on the original and counterfactual trajectories, followed by privileged distillation using the constructed evidence; and (2) Three-Action Routing (TAR), which trains a lightweight routing model through binary warm-up, expert-informed route supervision, and GRPO. CAST provides two inference modes: CAST-Flash, which uses TAR to either directly classify trajectories or defer them to the expert model for classification, and CAST-Pro, which relies entirely on the expert model for safety classification.
Experiments demonstrate that CAST achieves competitive performance against existing guardrail and post-training baselines. On Agent-SafetyBench, CAST-Pro and CAST-Flash achieve 89.82% and 86.19% Macro-F1 with Qwen3-8B, and 85.29% and 82.77% Macro-F1 with Llama3.1-8B, respectively.
Critical strengths
-
Effective safety classification with low cost. Safety classification is an important problem. Compared with existing methods, CAST improves safety classification performance at a relatively low cost and achieves promising results across multiple benchmarks.
-
Comprehensive experimental evaluation. The paper conducts extensive ablation studies to demonstrate the contributions of ECCE, PED, and TAR to the overall performance. Robustness experiments show that CAST is more robust to perturbations than directly using larger models or existing methods. Furthermore, the boundary-edit experiments provide evidence that CAST performs better on safety classification involving context-sensitive safety boundaries.
Critical weaknesses
-
Counterfactual editing and evidence-informed supervision are established ideas. CAST applies these approaches to safety classification and achieves promising results. However, the paper provides limited discussion of the challenges involved in adapting these approaches to safety boundary identification, particularly the rationale for the eight editing categories and how the validity of the generated counterfactuals is ensured.
-
The reasons behind the performance improvements are not sufficiently analyzed. Compared with SFT, OPD, and GRPO, CAST-Pro significantly improves Safe Recall while maintaining high Unsafe Recall. Although the ablation studies demonstrate the contributions of ECCE and PED to the improvement in Safe Recall, the paper provides limited insight into why these components lead to such improvements.
Questions for authors
-
Could the authors further clarify the definition of context-sensitive safety boundaries and which contextual changes affect the safety judgments of agent trajectories? Does counterfactual editing follow any systematic principles?
-
Could the authors further analyze why CAST improves Safe Recall while maintaining high Unsafe Recall?