- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to turn failed agent trajectories into usable training data. The authors collect 50,228 "failed trajectory + diagnosis" pairs across 33 environments, 19 harness families, and 23 models. All failures arise naturally from agent runs; no errors are injected.
Each diagnosis is generated by an LLM and contains the erroneous step, the responsible agent, an explanation with trace citations, and a proposed corrective action. In environments that support replay, the authors branch from the state before the erroneous step: one branch substitutes the corrective action and continues, the other re-executes the original action and continues, and the two branches are compared on whether they pass the verifier.
The authors run three types of experiments, each on a different subset of the data:
- Correction utility: across 3,062 replay pairs, first-proposal corrections raise the pass rate from 18.4% to 51.1%.
- Diagnosis model: after SFT, Qwen3-8B's exact-step agreement with teacher labels on the internal holdout rises from 47.2% to 63.6%.
- Acting agent: adding repair data on top of success-only SFT improves WebShop-lite by 6.67 points.
Critical strengths
- The problem is valuable and the framing is clear. The authors split failed experience into training views by strength of evidence: views that only require a diagnosis, views that require an executed and passing correction, and views that require comparing two actions from the same state. They also count the collected data, the training data, and the replay data separately.
- The control design of the replay experiment is in the right direction. The corrective action and the original action are paired from the same state, and the "both pass" and "only the original passes" outcomes are retained.
- The appendix is thorough. Negative results, multiple-comparison corrections, task-clustered confidence intervals, and overlap audits are all reported, which makes the results easy to check.
Critical weaknesses
W1. The diagnosis labels are not independently validated, and the reported experiments do not use the 50k collection.
- The 50,228 pairs pass only structural checks and support only coverage statistics. The diagnosis experiments use an earlier frozen release of 4,319 rows, whose 943-row holdout is labeled by the same multi-model pipeline as the training data, with no human labels. When the authors' own fifth rubric version is applied, only 18 holdout rows and 53 of 3,157 training rows remain (Table 9).
- The human review covers only 80 historical records, which are not drawn from the test set and were reviewed with AI assistance. Both reviewers accept the diagnosis content on only 41/80 records (Appendix W). In addition, 603 labels in the 948-task paired cohort were constructed with ground-truth content in the prompt (Appendix S), information the student never sees.
- The 63.6% exact-step accuracy therefore measures agreement with the teacher pipeline rather than diagnostic ability. The main text acknowledges this, but the abstract and Figure 1 place this number next to frontier models, which readers can easily take as a capability ranking.
W2. The evidence that the data helps correction, diagnosis, and acting is not robust.
- Correction: the main text reports a 32.7-point gain from first-proposal corrections, but under the cleaner setting of Appendix M (a single proposal and a fixed actor), the correction beats its control by only 12.9 points. A prompt with the diagnosis gains only 5.6 points (interval 0.0 to 11.3) over the single sentence "Reconsider your next action." (Table 27), so the contribution of the diagnosis content itself is not established.
- Diagnosis model: under the shared prompt, the trained model falls 31.5 points below the base on Who&When AG and 20.5 points on TrajErrBench (Table 16). Only the unified Who&When protocol shows a significant gain.
- Acting agent: compared with success-only SFT, repair data improves only WebShop-lite significantly (+6.67), while TextQuest drops by 8.3 to 9.7 points. On real ALFWorld and ScienceWorld, the preventive arm, the only one tested there, falls from 5.97% and 9.67% to 0.75% and 0.34%.
Questions for authors
- Given that Table 1 lists Exec. fix and Ctrl. as distinguishing features of AED, how does the replay verification differ from DoVer (arXiv:2512.06749), which also intervenes on failure hypotheses, re-executes from the intervention point, and uses the outcome to validate them, and from A2P (arXiv:2509.10401), which simulates counterfactual interventions for step-level attribution on Who&When? Should CORRECT (arXiv:2509.24088) also be included in Table 1?
- Since Appendix F.1 states that ground-truth access was permitted during diagnosis, were the corrective actions in the replay study generated with access to ground truth or reference answers, and could you report these cases separately with each branch sampled more than once, given that re-executing the original action already passes 18.4% of the time?
- Since only 871 of the 3,062 pairs were replayed under the original harness and 1,846 were continued with a single-agent ReAct loop (Table 14), what is the first-proposal effect on these 871 pairs, and what mainly explains the gap between the 32.7 points in the main text and the 12.9 points in Appendix M?
- Given that the same base model reaches 56.90% agent accuracy on Who&When HC when continuation is allowed but 22.41% when only the first response is scored (Tables 2 and 3), how do the frontier models perform under the continuation protocol, and does the student still surpass them?
- Given that training only on the compact localization target leaves the model below the base on the holdout (46.59% vs. 47.19%, Appendix I.13) while training on the full diagnosis reaches 63.56%, could you compare the two under matched data and conditions?
- Could you measure diagnosis accuracy on a test set labeled independently by humans without AI assistance, and how are cases with multiple plausible erroneous steps handled in evaluation?
5 comments
W2.1 and Q3. 32.7 points in the main text vs. 12.9 in Appendix M; the 871 same-harness pairs. These two metrics evaluate distinct stages: main-text replay validates construction-time corrections (our training targets), whereas Appendix M tests an autonomous test-time proposer without reference data on one fixed actor across 150 failures (one per task). To verify that harness substitution did not inflate the main-text replay gain, we re-analysed all 3,062 replay pairs at the pair level (App. Y.2):
| Replay subset | Pairs | First-proposal gain (pts) [95% CI] | Discordant pairs |
|---|---|---|---|
| All pairs (main text) | 3,062 | +32.7 [28.4, 37.0] | — |
| Original harness | 871 | +38.6 [33.7, 43.5] | 346 vs. 10 |
| Original harness, selected contrast | 871 | +47.1 [42.3, 51.7] | — |
| Reference payload but no verifier output | 1,757 | +34.6 [28.5, 40.8] | — |
| Excluding the 99 pairs that used a reference trajectory | 2,963 | +31.9 [27.8, 36.5] | — |
| Exactly one trial per arm | 2,410 | +29.3 [26.6, 31.9] | 788 vs. 83 |
The first-proposal gain is actually higher on the 871 original-harness pairs (+38.6) than on the full set (+32.7), so substituting the ReAct continuation does not inflate the replay gain.
Q2. Ground truth in replay proposals and repeated sampling. Yes: replay corrections are construction-time training targets proposed with access to the task's reference payload (as noted in App. F.1 and now stated upfront in Sec. 5.1). However, the table above shows that the gain does not rely on verifier leakage or multi-sampling: without verifier output, the gain remains +34.6, and with strictly one trial per arm, the gain is +29.3 (788 vs. 83 discordant pairs).
W2.3. Acting agent. We ran an exposure-matched, five-seed actor comparison with 5× enlarged procedural test sets (full table in our response to Reviewer AAaz, W3). Under matched exposure, the earlier TextQuest loss disappears, so it came from unequal task exposure, and GridWorld shows a significant gain from repair supervision (1,886 vs. 1,728 solved across five seeds, p < 0.001). On real ScienceWorld, success-only training drops just as far as every repair arm under both the original recipe (3 of 300 solved vs. 29 untrained) and the matched recipe, so the ScienceWorld drop comes from fine-tuning on the development environments rather than from repair data. Sec. 5.3 now presents these five-seed results per environment.
Q1. Positioning relative to DoVer, A2P, and CORRECT. We integrated all three works into Table 1 and Sec. 2 (summarized in Table B of the General Response). DoVer is the closest: it branches from checkpoints to execute edited orchestrator messages across three trials and compares against the original failed log. AED adds the control DoVer lacks: it re-executes the original action from the identical state, so stochastic retries are not miscounted as valid repairs. This matters because original actions pass 18.4% of the time upon retry in our data. By contrast, A2P only simulates counterfactuals in-prompt without execution, and CORRECT relies on LLM-injected errors across >2,000 trajectories, whereas all AED failures arise naturally and are verified against a matched execution control.
Q4. Frontier models under the continuation protocol. The continuation protocol only triggers when a local model hits the generation token limit mid-reasoning block. Hosted frontier models never hit this limit in our runs (App. V.1), so their scores are identical under both protocols. Our trained student outperforms frontier models on hand-crafted responsible-agent accuracy (70.69 vs. 56.90, and 68.97 vs. 60.34), while GPT-5.6 Sol remains ahead on exact step localization.
Q5. Compact vs. full targets under matched data and conditions. The earlier gap between compact and full targets was caused by training the compact baseline with LoRA rather than full fine-tuning. We retrained the compact-target model under strictly matched conditions: the same 1,656 rows, byte-identical inputs, and the full fine-tuning recipe (Sec. 5.2, App. Y.3).
| Arm (1,656 tasks, holdout exact step) | Accuracy |
|---|---|
| Full diagnosis, full fine-tuning (3-seed mean) | 63.56 |
| Compact, full fine-tuning (new, matched recipe) | 63.1 |
| Compact, LoRA (original) | 46.59 |
| Untrained base (same evaluation run) | 47.6 |
Under matched full fine-tuning, compact targets reach 63.1%, within 0.5 points of full targets and 15.5 points above the base model, so the learning signal comes from AED's step-level supervision under either target format.
We hope these new experiments and analyses answer your questions, and we would be grateful if you would consider raising your score.
We thank the reviewer for a rigorous, detailed review and for highlighting our evidence-graded training views, paired replay control, and thorough appendices. We address every point below, with corresponding updates marked in blue in the revised PDF.
W1.1. Structural checks on the 50k pairs vs. Table 9 row counts. The 50,228-pair collection is the full released dataset, whereas the frozen release is the controlled subset used for our experiments; we now state this clearly in Sec. 3.3 and Sec. 5.2. Table 9 applies our strictest current rubric (v5) to the frozen release (with prompts and tool lists restored), leaving 18 holdout and 53 training rows under that conservative audit rather than defining the boundary of usable training data. To measure label quality directly, we pair lenient evaluation with an independent human study (W1.2).
W1.2 and Q6. Unassisted human labels and multiple plausible error steps. We are finishing an unassisted human annotation study with no AI-generated content. Blind annotators read the same trace the model sees, record both their single preferred error step and all defensible error steps, lock their answers, and only then view our diagnosis. Covering the 80 audited records plus a fresh stratified holdout sample, this provides a clean human benchmark for test-set accuracy, reporting strict accuracy (preferred step), lenient accuracy (any defensible step), and teacher accuracy split by ground-truth access. Results will be posted in a few days.
For multiple plausible steps on the full holdout set, lenient scoring using the teacher's recorded alternative steps confirms AED's clear advantage:
W1.3. Presenting 63.6% alongside frontier models in the abstract and Fig. 1. We updated the abstract, Fig. 1b, and Sec. 5.2 to state explicitly that 63.6% measures agreement with teacher labels.
W2.2. Losses under the shared prompt. Format-agnostic rescoring shows that the shared-prompt Who&When AG drop is driven by output formatting: format compliance drops from 94.4% to 18.8% on the unseen prompt, whereas on cases both models format validly, localization accuracy is comparable (42.0 vs. 43.7). After overlap-clean retraining across three seeds, our model improves unified Who&When by +6.70 points [0.71, 12.89].