- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies prospective detection of the first anomalous transition in LLM-agent execution. It combines trajectory encoding, task-conditioned normal references, and action-conditioned next-state prediction. Predicted state and transition deviations are fused with direct task–action evidence to estimate risk before execution. Experiments assess next-state prediction, anomaly-onset detection, and computational efficiency across several transfer settings.
Critical strengths
-
Assessing a candidate action through its predicted consequences relative to normal process support is a meaningful approach to prospective risk estimation.
-
The reported lightweight auditor achieves a useful performance–cost trade-off. Evaluation across multiple holdout settings and random seeds, with a shared FAR-calibration protocol, provides substantial empirical coverage.
Critical weaknesses
-
R-A omits trajectory history and normal-reference evidence, while the TS baselines differ in their representation front end and architecture. These comparisons therefore leave the incremental value of explicit next-state prediction unclear. Separately, the value of cross-executor alignment as a distinct contribution is not sufficiently established by the current geometric analysis.
-
The method introduces several intermediate representations, statistical features, and fusion components, but the numerical encoding of raw events and the construction of the 29-dimensional state remain insufficiently specified. These omissions hinder understanding of the method and make the source of its performance–cost differences from frozen-observer baselines less clear.
Questions for authors
-
Could the authors clarify which existing results support the added value of explicit next-state prediction over directly using trajectory history and normal-reference information for risk estimation? If available, a direct-risk comparison retaining these information sources would help clarify this point.
-
In Table 5, distances between different tasks also contract, proportionally more than same-task cross-executor distances. Could the authors provide either a further interpretation of how these geometric results support cross-executor alignment, or, where feasible, a downstream comparison with and without the alignment term on held-out executors at the same FAR budget to support its practical value?
-
How are raw events encoded for the GRU, and observations mapped to the 29-dimensional state? Could the TS baselines adopt the proposed encoding front end, retaining their downstream decision structures with appropriate retraining, to assess its performance–cost impact? If not, what prevents this adaptation?
We thank the reviewer for the constructive comments and recognition of the value of prospective, lightweight process-risk auditing. We have added controlled evaluations, clarified implementation details, and revised the relevant claims in the manuscript.
1. "Added value of explicit next-state prediction … trajectory history and normal-reference information" (W1, Q1)
Our world model learns action-conditioned normal process dynamics and predicts the future structural state induced by a candidate action before execution. By comparing this predicted state and transition with task-conditioned normal support, the risk estimator can incorporate prospective process evidence beyond the observed trajectory history.
To evaluate this contribution, we conducted additional frozen-upstream controlled ablations using the gated fusion mechanism of R-G, progressively incorporating completed history, retrieved normal support, and explicit prediction.
The results show that history and normal support improve exact-onset detection by 16.82 pp and 2.43 pp, respectively. Importantly, adding explicit next-state prediction provides a further 2.27 pp improvement after retaining both information sources.
This directly supports the utility of action-conditioned future-state representations as complementary evidence for prospective risk estimation. Together with the action-conditioning experiments, these findings connect the world model's predictive capability with its contribution to downstream onset detection.
The original 17.3 pp improvement over R-A represents the benefit of the complete integrated framework, while the additional controlled comparison further identifies the incremental contribution of explicit prediction.
Revised: Sec.4.4, Sec.5.4, Appendix C.2.
2. "Geometric results … downstream comparison with and without the alignment term" (W1, Q2)
We thank the reviewer for highlighting the distinction between geometric alignment and downstream detection performance.
Cross-executor alignment serves as an auxiliary representation regularizer, rather than a primary contribution of our framework. Our central contributions lie in task-conditioned local normal support, action-conditioned future-state prediction, and onset-aware risk estimation.
To examine alignment's effectiveness, we conducted an additional downstream ablation on six actual held-out executors (18 paired evaluations), alongside our geometric analysis:
These results show that alignment does not yield consistent downstream detection gains. In contrast, task-conditioned normal support provides measurable complementary evidence, as shown in Section 1. Improving alignment while preserving task-specific structure remains future work.
Revised: Sec.5.2, Appendix C.3.
3. "Raw events … 29-dimensional state … TS frontend" (W2, Q3)
(a) "Raw events … GRU … 29-dimensional state?"
Each completed event uses separate 29D action and 29D outcome features, projected to 64D and processed by a 64-unit GRU. The outcome comprises flags (10D), metrics (4D), status (3D), delta (5D), dependency (4D), and side effects (3D). These differ from the learned history state. We also clarify the 304D task/action semantics, lexical hashing, and task/event-compatible normal-reference retrieval. A candidate action is scored before its outcome is observed.
Revised: Sec.4.2–4.3, Appendix B.1.
(b) "TS front end … performance-cost impact?"
We retrained both TS-Traj and TS-Onset decision heads with R-G's structured inputs across 108 contexts:
TS-Onset retains the highest Hit, while R-G exceeds TS-Traj and benefits from prediction. Each adapted TS head has 223,941 trainable parameters (excluding the shared frozen frontend) and omits the original 7B observer.
Under a unified FP32 evaluation of the original frozen-observer configurations, R-G achieves 23.34×/23.43× lower p50 latency than TS-Traj/TS-Onset. Peak allocated GPU memory is 0.0104 GiB versus 31.19 GiB. This supports an accuracy–efficiency trade-off under matched precision.
Revised: Appendix C.5, Sec.5.5, Appendix B.5.
We appreciate the reviewer's suggestions, which have strengthened the evaluation and clarified the scope of our claims.