- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the distinction between risk exposure and unsafe enactment in LLM-based agent safety monitoring. The authors propose Evidence-Dynamics Alignment (EDAlign), a post-hoc framework that extracts Exposure and Enactment evidence from intermediate representations of frozen safety monitors. These signals are aggregated across model layers into a structured feature representation and leveraged by a lightweight alignment head to refine trajectory-level safety predictions. Experiments on ATBench and ASSEBench across four LLM backbones show improvements in safety classification accuracy and F1, particularly by reducing false positives on risk-exposed but behaviorally safe trajectories. Additional analyses examine evidence recoverability, prediction errors, and feature ablations.
Critical strengths
- The paper highlights an important distinction between encountering risky information and actually performing unsafe actions. This perspective is meaningful for trajectory-level safety monitoring.
Critical weaknesses
- Limited methodological novelty and insufficiently controlled comparisons. As described in Sections 3, EDAlign primarily combines supervised linear probes, handcrafted statistical features, and a lightweight post-hoc classification head. Although the Exposure-Enactment decomposition is intuitively reasonable, the approach does not directly modify or causally align the internal decision process. Furthermore, the direct unsafe-probe baseline in Appendix B.8 adopts a 10-dimensional feature representation, compared with the 24-dimensional representation of EDAlign. Without stronger capacity-matched comparisons, it remains unclear whether the improvements arise from the proposed evidence factorization or simply from richer supervised features.
- Model-dependent effectiveness and insufficient justification of the feature design. The results in Table 1 show that EDAlign does not consistently outperform the baselines across different backbones and datasets. In particular, for Llama3.1-8B on ATBench, EDAlign achieves an F1 of 67.75%, below 72.93% of AgentDoG. More importantly, Table 2 shows that removing any individual feature group improves ATBench F1 for Llama3.1-8B, with a substantial 7.02% improvement when removing evidence endpoints. Although Section 5.4 acknowledges model-dependent feature contributions, it provides limited explanation for this behavior. These findings raise concerns about the necessity and robustness of the proposed structured evidence representation across different models and datasets.
Questions for authors
- Could the authors provide a capacity-matched comparison between EDAlign and direct unsafe probing, including comparable feature dimensionality and classification-head complexity? This would help establish the independent contribution of Exposure-Enactment factorization.
- Table 1 shows that EDAlign underperforms AgentDoG for Llama3.1-8B on ATBench, while Table 2 indicates that removing any individual feature group improves its F1, particularly evidence endpoints (+7.02 points). How do the authors explain these model-specific results, and what do they imply about the necessity and generalizability of the proposed feature design?
3 comments
Looking forward to your reply and further discussion.
Looking forward to your reply and further discussion.
Thank you for the detailed rebuttal and additional experiments.
The newly introduced U-Probe-Dynamics (24D) baseline provides stronger evidence for the effectiveness of Exposure-Enactment factorization, partially addressing my concerns. However, the explanation of the model-dependent ablation results remains insufficient, particularly why removing individual feature groups consistently improves performance for Llama3.1-8B on ATBench. The proposed framework still appears methodologically incremental, and the robustness of its feature design remains questionable.
Additionally, I noticed that the initial submission already exceeds the 9-page main-text limit, with the Conclusion extending onto page 10, and no corresponding revisions to the main text have been made during the rebuttal period. Considering these concerns, I maintain my original rating of 1/4.
W1&Q1: Capacity-matched comparison between EDAlign and direct unsafe probing
We thank the reviewer for this important suggestion. U-Probe trains an independent binary unsafe probe at each layer. Since it models a single unsafe evidence stream, applying our structured feature construction naturally yields a 10D representation. EDAlign instead learns two probe families: Exposure (D00 vs. D10) and Enactment (D10 vs. D11), yielding 24D features that capture both streams, their discrepancies, interactions with the native decision, and cross-layer dynamics. Thus, the dimensionality difference reflects the evidence structures modeled, rather than an arbitrary restriction of U-Probe.
To address this concern, we introduce U-Probe-Dynamics (24D). It retains a single unsafe probe per layer and aggregates its layer-wise scores Sl into 24 features: native-decision features, three anchor-point scores, segment-wise changes, global/segmented path statistics, and native-decision/evidence interactions. It uses the same middle-to-final layer range and classification head as EDAlign, but has no separate Exposure/Enactment channels.
Table X. Capacity-matched comparison on ATBench (%, ↑ higher is better except FPR):
EDAlign consistently outperforms U-Probe-Dynamics on ATBench, with F1 gains of 1.28–5.37 points (mean: 4.17).
We further evaluate cross-benchmark transfer from ATBench to ASSEBench. Probes are trained on ATBench and frozen; 80 ASSEBench samples recalibrate the alignment head, and the remaining 682 are used for testing.
Table Y. Cross-benchmark transfer on ASSEBench (%, ↑ higher is better except FPR):
EDAlign outperforms U-Probe-Dynamics on three of four backbones, with F1 gains of 3.73–11.04 points; U-Probe-Dynamics is 0.81 points better on Llama3.1-8B. Across eight settings, EDAlign wins 7/8 with a mean F1 gain of 4.36 points. These results show that the gains are not solely due to higher dimensionality and support the benefit of Exposure–Enactment factorization under limited target-domain alignment data.
W2&Q2: Model-specific results and generalizability of feature design
We thank the reviewer for raising this concern. The observed differences reflect distinct native-monitor biases and backbone-dependent internal safety evidence.
On Llama3.1-8B/ATBench, AgentDoG has higher Recall than EDAlign (82.32% vs. 52.53%) but much higher FPR (56.58% vs. 3.29%), yielding F1 of 72.93% vs. 67.75%. We acknowledge this trade-off: EDAlign favors substantially lower FPR.
More broadly, Vanilla Qwen3-4B/8B have high FPRs (79.61%/55.92%), indicating over-flagging of safe trajectories, whereas Llama3.1-8B has low Recall (38.89%), indicating missed unsafe behaviors. Relative to Vanilla, EDAlign reduces FPR while increasing Recall for Qwen, and improves both Recall and FPR for Llama. Thus, EDAlign can correct different native decision biases rather than simply shifting predictions in one direction.
Performance also depends on backbone-specific evidence distributions. For Llama3.1-8B, All-layer evidence improves F1 over Mid-to-final by 0.45 points on ATBench and 2.32 on ASSEBench, suggesting useful evidence in earlier layers. Feature ablations further show dataset-dependent effects: removing evidence endpoints improves ATBench F1 by 7.02 points, whereas removing path features reduces ASSEBench F1 by 2.79 points.
Overall, EDAlign can mitigate different native-monitor biases through post-hoc alignment of internal safety evidence, but its final performance remains dependent on backbone-specific evidence, feature interactions, and the resulting decision boundary and threshold. We therefore do not claim every feature group is universally beneficial or that EDAlign consistently outperforms every baseline.