- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for repairing LLM agent failures by selecting where to restart based on belief provenance, predicted repair success, and downstream cost. Experiments on multiple benchmarks and human-annotated failures show improved repair success over existing self-correction methods and reduced reuse of unsupported beliefs.
Critical strengths
-
The proposed method is well motivated and has a clear intuition.
-
The empirical evaluation is broad.
Critical weaknesses
-
Although the authors compare their method with self-correction baselines, there is no direct comparison made between their method and others like CausalFlow, RIR, and dependency guided rollback repair.
-
In the restart criterion, the normalized plausibility score is combined with the descendant cost in an equally weighted manner. However, the relative significance of each component is subject to the number of candidates available, thus the dominance of the cost component for large belief networks.
Questions for authors
-
Could the authors add a workflow diagram showing PASTA pipeline?
-
How sensitive is the restart selection to the number of candidate nodes and the relative weighting of plausibility and cost? Could the authors provide a sensitivity analysis?
-
Since the counterfactual proposals are not directly used during regeneration, how well do the predicted repair outcomes align with actual repair success? Have the authors evaluated this consistency?
2 comments
Workflow, Graph-Size and Cost Sensitivity, and Counterfactual Validity
We thank the reviewer for identifying three connected presentation and mechanism questions: whether cost overwhelms plausibility on large graphs, whether the complete execution flow is understandable, and whether counterfactual proposals predict actual repairs. We now address all three with Figure 2 and Tables 15–16 (Sections 2 and 4.5; Appendix K).
Q1: Workflow (Figure 2). The five-stage diagram makes Algorithm 1 explicit: (1) verify the task/profile constraints and extract a provenance-typed belief graph ; (2) construct candidate set from beliefs at the violated step, their ancestors, and the final-decision node; (3) generate minimal counterfactual replacement probes and predicted repair indicators; (4) select a cost-aware restart , preserve the original prefix, and prune the downstream commitments; (5) regenerate the suffix from fixed observations and verify again. Unsuccessful candidates are excluded on subsequent attempts, with their initial plausibility mass down-weighted. The proposal is used to rank candidates, not inserted as a required edit into the regeneration prompt.
W2/Q2: Meaning of equal weighting. For candidate , normalized descendant cost is ; selection maximizes , with published . Although sums to one, is normalized within each graph, so graph size alone need not determine the trade-off. Retried candidates are excluded; for remaining candidates , where . This increases the relative influence of plausibility versus cost on later attempts (Proposition 3).
Direct cost-weight sweep (Table 16b). With (plausibility only), first-pick/three-attempt repair is 14.52%/16.43%, saving 24.1% of regenerated tokens versus root restart. At , the rates are 14.39%/16.31% with 40.2% saving. At , they become 13.50%/15.41% with 48.6% saving; choosing the cheapest node alone yields 10.83%/13.12% with 61.4% saving. Hence the default sacrifices just 0.12 pp three-attempt repair relative to pure plausibility in exchange for 16.1 pp additional regenerated-token savings. The trade-off is measured, rather than assumed optimal for every application.
Graph-size stratification. For small graphs (, ), PASTA's first pick repairs 15.06% against 10.96% for uniform random (+4.10 pp). For medium graphs (, ), the rates are 14.37% versus 10.02% (+4.35 pp); for large graphs (, ), 12.88% versus 8.90% (+3.98 pp). This stable relative margin does not support the concern that descendant cost inevitably collapses selection quality at larger , while acknowledging that absolute repair becomes harder.
Q3: Proposal versus actual outcome. We regenerated from every one of 3,191 nodes (11,979 outcomes; Table 16). 378 nodes truly repair their case. Counterfactual Boolean proposals identify 254/378 (67.2% recall) and correctly reject 2,427/2,813 (86.3% specificity); TP/FP/TN/FN are 254/386/2,427/124. The Boolean predictor's Brier score is 0.1598, improving modestly to 0.1541 for provenance-weighted normalized . First-pick PASTA repairs 113/785 (14.39%) versus random 10.20% (gain +4.20 pp, 95% CI [+2.15,+6.28], ). The retry-context comparison is 13.12% versus 9.26% ().
Finally, on 785 cases a three-arm test (Table 15) finds 16.31% repair with prefix-only truncation, 16.05% after adding the proposed replacement (McNemar ), and 15.16% with neutral length-matched text. Accordingly the counterfactual is a useful selection signal, but does not become an edit that the model must obey. This distinction is now explicit throughout the workflow description.
Specialized Repair Baselines, Paired Significance, and Oracle Limits
We thank the reviewer for pointing out that a comparison solely against generic self-correction methods does not distinguish PASTA from targeted rollback or counterfactual-intervention approaches. We added direct, matched evaluations of the requested repair paradigms (Section 4.5; Appendix K, Table 14a), and clarified their mechanisms and relationship to PASTA in Section 5.
Common experimental setting. We start with the same 785 initially failed, answer-blind trajectories, provide each method the same task, user profile, fixed observations, and constraint-only violation feedback, and permit at most three repair attempts. Each item combines 3–5 profile/task predicates, with mean 2.6 co-occurring violations. The outcome is strict conjunctive verifier acceptance: leaving any predicate violated counts as failure. This is not the same population or outcome as the 300 natural failures in Table 3a, which average 1.2 violated predicates and are judged on the primary failure by blinded evaluators. The difference matters when interpreting the absolute percentages.
Results and paired tests. PASTA repairs 128/785 (16.31%). Direct LLM step attribution (ATTR, informed by CORRECT) repairs 105/785 (13.38%), a PASTA improvement of +2.93 percentage points (95% CI [+0.64,+5.22], Holm p=0.0280). A DoVer-inspired diagnose/intervene/verify adaptation repairs 102/785 (12.99%) (+3.31 pp, CI [+0.51,+5.99], Holm p=0.0280). Three-attempt matched full-output retry repairs 101/785 (12.87%) (+3.44 pp, CI [+1.15,+5.73], Holm p=0.0114). Dependency-guided rollback repairs 97/785 (12.36%) (+3.95 pp, CI [+1.40,+6.50], Holm p=0.0114); RIR-inspired interval checkpointing repairs 91/785 (11.59%) (+4.71 pp, CI [+2.29,+7.13], Holm p=0.0011). These are paired comparisons with multiplicity correction, not differences between unrelated runs.
CausalFlow comparison. We also implemented a CausalFlow-inspired minimal-intervention baseline for a 100-case candidate-matrix subset. Its repair rate is 17/100 (17.0%), compared with 20/100 (20.0%) for PASTA; its observed mean is 2.34 attempts (Table 14a). This smaller subset provides directional evidence rather than a claim of statistically established superiority. We explicitly label all specialist baselines as single-agent adaptations of the relevant paradigms, not verbatim reproductions of systems designed for other execution environments.
Why the absolute rate is low. We exhaustively evaluated restart candidates at all 3,191 nodes in these 785 belief graphs (11,979 regeneration outcomes; Table 16a). In 626/785 cases (79.75%), no single candidate restart succeeds with the original observations held fixed, often because independent constraints fail simultaneously. Thus even an oracle that sees all candidate outcomes has a 159/785 (20.25%) ceiling within this intervention family. PASTA's 128 successes amount to 80.5% of that oracle-repairable subset, not 16.31% of an unconstrained population. Independent external-gold correctness is 107/746 (14.34%) and strict joint pass is 103/746 (13.81%); we report these to distinguish constraint acceptance from reference correctness.
Distinct mechanism. Attribution, checkpointing, and full retry can identify a step or re-run from a checkpoint, but PASTA explicitly distinguishes user-stated facts from unsupported model-inferred commitments. It combines a provenance prior, counterfactual predicted repair, and normalized descendant cost to choose a restart that may differ from the earliest causal fault. The claim is therefore not that PASTA invented rollback, but that user-relative provenance improves which partial restart to select. These experiments directly address the missing specialist comparisons while retaining the fixed-observation scope of the evaluation.