acceptodds
Withdrawn from ICLR 2027

PrefixGuard: Online Failure Warning and Trace-Grounded Diagnosis for LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM) agents execute multi-step tasks, but terminal verdicts arrive too late for intervention. Agent traces mix messages, tool calls, and feedback, making it difficult to learn failure signals from terminal outcomes. Even if a model learns to predict failure from execution prefixes, its predictions alone cannot show which actions, feedback, or unmet conditions need inspection. Existing work studies online failure prediction and diagnosis from completed trajectories, yet few methods connect timely warning to a compact symbolic model of execution behavior. We introduce PrefixGuard, a neuro-symbolic method that uses learned events to connect online failure warning with finite-state execution diagnosis. A gated recurrent unit (GRU) predicts horizon-specific failure risk from these events. A deterministic finite automaton (DFA) organizes their histories into compact risk-labeled paths for replay and rule-monitor composition. Across all 24 benchmark–horizon settings, PrefixGuard exceeds the strongest evaluated baseline in mean test area under the precision–recall curve (AUPRC). Average gains range from 12.0 to 21.9 percentage points. At a nominal 10% FAR budget, averaged test recall spans 56.1%–93.0% across benchmarks. On one frozen -Bench DFA, model checking five specifications identifies violations in 21 of 265 benchmark-successful runs.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-Symbolic, LLM Agents, Online Failure Warning, Finite-State Audit

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_f73VOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PrefixGuard, which uses a GRU to predict agent failures online and a learned DFA to replay execution traces and check manually specified rules. Although the prediction results are promising, most gains appear to come from the input representation and GRU, while the DFA provides limited additional diagnostic value. The evaluation also lacks generalization to unseen tasks and agents.

Critical strengths
  • The paper addresses the important practical problem of detecting agent failures early enough for intervention.

  • The evaluation covers four diverse benchmarks and multiple warning horizons.

  • The experimental protocol is careful, with separate training, validation, calibration, and test splits and several complementary metrics.

  • The paper is transparent, providing extensive ablations and reporting limitations and failure cases of the DFA-based monitor.

Critical weaknesses
  • This paper's structure is loose. The relationship between GRU, discrete even, DFA is weak. The improvement of prediction comes from data and GRU. In section3.2 and Table1, author mentioned use stepview as input directly will be better than event-based GRU and mentioned the improve comes from step information. So I do not know the motivation of discrete events in this paper.

  • Meanwhile, DFA is used to re-organize the current trajectories, the failed reason should be checked by human. Section 4 mentioned abstract events can not explain the failure case, it needs human to do task-grounded specifications.

  • This paper has 3 contributions, but on this above limitations, only prediction is proved, they other two's motivation is weak, especially DFA, because discrete events may decrease performance, it seems introduce discrete events is to propose A, not for prediction.

  • The baselines are weak, the baselines are used traditional error detection method like PCA, DeepLog. Author only discuss recent paper but do not compare with them. Also, in section3.2 PrefixGuard use full step view but PPM-LSTM only use tool|status

  • The generalization is a problem, although author mentioned "Performance on unseen tasks and agents remains unmeasured", OOD tasks is important for audit tasks. In Appendix B.4, every new tasks need a new GRU, the cost is high.

  • In Appendix F.2, the 5 rules get from 94 failure case and then do retrospective audit on 359 trajectories including 94 case, without training set and test set.

  • DFA may not effectiveness, in Table21, DFA in WebArena is all silent, long-horizon in SkillsBench is also silent.

  • The presentation of this paper is poor, x-axis of figure3 and figure 4 doesn't align

Questions for authors

See weaknesses

Reviewer_CmtcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PrefixGuard, a monitor for LLM-agent executions with two parts. The first part uses the current execution prefix to predict if a run will fail withing N further steps. The second creates a risk labeled DFA for replaying runs and checking them against rules. The model is evaluated on traces from WebARena, tau^2 Bench, SkillsBench and TerminalBench.

Critical strengths
  1. The eval is rigorous. Four benchmarks, many agents, frozen splits and multiple seeds. The authors also include even the unfavorable findings.
  2. The problem is useful, and the paper includes a concrete example where tool-grounded specification flag five successful run without any runtime rejection.
Critical weaknesses
  1. The comparison is apples to oranges for the GRU performance. The strongest baseline, PPM-LSTM, only sees tool and status codes as inputs, while PrefixGuard see the full step text. On top of this, every task appears in every split, providing no insight into how this method generalizes to unseen tasks.

  2. There's not much support for the claim that the DFA has any diagnostic or verification value. For example, the calibrated threshold for the DFA produces no alerts in every WebArena setting. The argument for interpretability consists of hand selected cases.

Questions for authors

1, How do two simple baselines perform on the same splits? a. a prior only baseline: the training-set failure rate per task and the per task X agent/model assigned to every prefix. b. Logistic regression on the prefix TF-IDF. 2. Can you report results on a task-disjoint split and on a leave one agent out split? 3. Why is the StepView MLP exluded from figure 3 and the headline comparison, given table 18? 4. For the 21 flagged successful runs, how many would the same monitors find if run directly on the raw traces, with the DFA? And how many of the 21 involved only runtime-rejected calls?

Reviewer_Y87tOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes PrefixGuard, a monitor that predicts failure risk before an LLM agent completes execution and supports execution trace inspection. Using historical traces and final success or failure labels, it learns event representations and a GRU predictor, then constructs a risk-annotated DFA from the same representations. The DFA supports trace replay and incorporates explicit task rules to identify violations. Experiments on four agent benchmarks evaluate early-warning performance and the preservation of predictive information in the DFA, while case studies illustrate the diagnostic utility of the neural-symbolic framework.

Critical strengths

S1. Clear presentation. The paper is well written and easy to follow.

S2. Important problem. Moving beyond success prediction to failure diagnosis addresses an important problem.

Critical weaknesses

W1. The added value of the GRU-based neural-symbolic design and learned warning signals is insufficiently established. The GRU variant without discrete events achieves better average predictive performance. However, direct comparisons with Transformer predictors or LLM judges under matched input and supervision settings, as well as monitors that apply rules directly to raw logs without a DFA, are lacking.

W2. The practical utility of warnings for intervention remains unvalidated. The paper does not evaluate whether warning-triggered interventions improve task success or increase costs through unnecessary interventions. Controlled comparisons are needed to establish whether the framework provides more actionable feedback and improves agent performance sufficiently to justify its additional training and design costs.

W3. Task overlap prevents distinguishing generalizable failure prediction from task-specific pattern learning. Every evaluation task also appears in the training data, potentially simplifying prediction through familiar success and failure trajectories or recurring responses. Although this setting does not necessarily constitute data leakage, the absence of a task-disjoint evaluation limits conclusions about failure prediction on unseen tasks.

Questions for authors

Please address the concerns raised in the critical weaknesses. (new experiments should be included, including ablation studies, task held-out experiments, and application to llm agent's pipeline.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.