- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies safe-only online early failure detection for LLM-based agentic systems, aiming to identify execution failures before tasks are completed without requiring extensive failure trajectory annotations. The authors propose Learning from Success (LFS), which combines a parametric safe-dynamics predictor with a non-parametric memory of successful transitions to detect deviations from successful execution patterns. When limited failed trajectories are available, LFS incorporates decisive failure transitions as a contrastive memory without retraining the predictor. Experiments on AgentProcessBench and AFTraj-2K demonstrate competitive detection and localization performance, with additional improvements from limited failure supervision.
Critical strengths
- The proposed framework is intuitive and computationally lightweight, combining temporal prediction with context-based retrieval to capture complementary signals of execution anomalies.
- The results show promising performance in the safe-only setting and demonstrate that a relatively small failure memory can substantially improve detection and localization on certain domains.
Critical weaknesses
- The methodological novelty appears limited, as next-step prediction, nearest-neighbor transition retrieval, and memory-based failure comparison are established techniques whose combination may not constitute a sufficiently distinctive contribution.
- The transition retrieval mechanism primarily uses the immediately preceding step embedding, leaving it unclear whether it adequately captures the longer-range, task-dependent context emphasized in the paper's motivation.
- The evaluation metrics focus partly on ranking annotated decisive failure steps against successful execution steps, which may not fully characterize practical online monitoring performance, particularly false alarms and missed detections.
Questions for authors
-
How does LFS differ fundamentally from existing safe-only trajectory modeling and anomaly detection methods, particularly OAT and ATS, beyond combining a predictor with transition retrieval?
-
Why does the transition memory retrieve neighbors using only the immediately preceding step embedding rather than the task query and a longer execution-history representation? Have the authors evaluated these alternatives?
-
How much does each component contribute relative to simpler baselines, including the dynamics predictor alone, nearest-neighbor transition anomaly detection alone, and a straightforward safe-versus-failure memory comparison?
-
Can the authors provide additional online evaluation results measuring false-alarm rates, missed detections, and detection timing under distribution shifts or varying task and trajectory-length distributions?
Dear Reviewer taam,
Thank you very much for your time and effort throughout the review process.
We sincerely appreciate your valuable suggestions and comments. Regarding all your concerns, we have provided detailed response and supplemented new experimental results, including:
1. Additional FAR evaluations on both benchmarks.
2. Ablation studies on the "Transition memory" and the "Safe Dynamics" component.
3. An analysis on Coding and Agentic domains of the non-monotonic performance improvements with respect to the failure memory.
4. A fully supervised comparison between AgentForesight-7B and a fine-tuned Qwen2.5-3B classifier.
We have also provided detailed explanations addressing your other questions.
We fully understand that you may have a busy schedule. Nevertheless, we would greatly appreciate it if you could take a moment to review our responses and share any further feedback you may have.
Thank you again for your time and consideration.
Sincerely,
Authors