- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents PrefixGuard, which uses a GRU to predict agent failures online and a learned DFA to replay execution traces and check manually specified rules. Although the prediction results are promising, most gains appear to come from the input representation and GRU, while the DFA provides limited additional diagnostic value. The evaluation also lacks generalization to unseen tasks and agents.
Critical strengths
-
The paper addresses the important practical problem of detecting agent failures early enough for intervention.
-
The evaluation covers four diverse benchmarks and multiple warning horizons.
-
The experimental protocol is careful, with separate training, validation, calibration, and test splits and several complementary metrics.
-
The paper is transparent, providing extensive ablations and reporting limitations and failure cases of the DFA-based monitor.
Critical weaknesses
-
This paper's structure is loose. The relationship between GRU, discrete even, DFA is weak. The improvement of prediction comes from data and GRU. In section3.2 and Table1, author mentioned use stepview as input directly will be better than event-based GRU and mentioned the improve comes from step information. So I do not know the motivation of discrete events in this paper.
-
Meanwhile, DFA is used to re-organize the current trajectories, the failed reason should be checked by human. Section 4 mentioned abstract events can not explain the failure case, it needs human to do task-grounded specifications.
-
This paper has 3 contributions, but on this above limitations, only prediction is proved, they other two's motivation is weak, especially DFA, because discrete events may decrease performance, it seems introduce discrete events is to propose A, not for prediction.
-
The baselines are weak, the baselines are used traditional error detection method like PCA, DeepLog. Author only discuss recent paper but do not compare with them. Also, in section3.2 PrefixGuard use full step view but PPM-LSTM only use tool|status
-
The generalization is a problem, although author mentioned "Performance on unseen tasks and agents remains unmeasured", OOD tasks is important for audit tasks. In Appendix B.4, every new tasks need a new GRU, the cost is high.
-
In Appendix F.2, the 5 rules get from 94 failure case and then do retrospective audit on 359 trajectories including 94 case, without training set and test set.
-
DFA may not effectiveness, in Table21, DFA in WebArena is all silent, long-horizon in SkillsBench is also silent.
-
The presentation of this paper is poor, x-axis of figure3 and figure 4 doesn't align
Questions for authors
See weaknesses