acceptodds
Under review as a conference paper at ICLR 2027

Agent Error Dataset: Scaling 50,000 Error–Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

Abstract

LLM agents learn from demonstrations and reward feedback, yet an unsuccessful rollout does not specify which decision should change. How can failed expe- rience become reusable supervision for diagnosing errors and improving agent behavior? We introduce the Agent Error Dataset (AED), comprising 50,228 error–diagnosis pairs across 33 environments, 19 harness families and 23 policy models in text-based agent systems for failure analysis and post-training. Our five-stage Agentic Error-to-Training (AET) pipeline generates trace-grounded diagnoses and proposed corrections, tests corrections against original-action re- tries from the same checkpoint where replay is supported, and constructs separate training views for diagnosis and recovery. Across 3,062 matched replay pairs, first-proposal corrections improve verifier pass rates by 32.7 percentage points over their original-action controls. Training on up to 1,656 source tasks from a separately frozen diagnosis release raises Qwen3-8B’s exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds. In a single-seed actor recipe comparison, action-only repair training scores 6.67 per- centage points higher on WebShop-lite than success-only training, with gains that depend on the environment.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Failure Analysis, Post-train LLM, Synthetic Dataset

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QcCjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to turn failed agent trajectories into usable training data. The authors collect 50,228 "failed trajectory + diagnosis" pairs across 33 environments, 19 harness families, and 23 models. All failures arise naturally from agent runs; no errors are injected.

Each diagnosis is generated by an LLM and contains the erroneous step, the responsible agent, an explanation with trace citations, and a proposed corrective action. In environments that support replay, the authors branch from the state before the erroneous step: one branch substitutes the corrective action and continues, the other re-executes the original action and continues, and the two branches are compared on whether they pass the verifier.

The authors run three types of experiments, each on a different subset of the data:

  1. Correction utility: across 3,062 replay pairs, first-proposal corrections raise the pass rate from 18.4% to 51.1%.
  2. Diagnosis model: after SFT, Qwen3-8B's exact-step agreement with teacher labels on the internal holdout rises from 47.2% to 63.6%.
  3. Acting agent: adding repair data on top of success-only SFT improves WebShop-lite by 6.67 points.
Critical strengths
  1. The problem is valuable and the framing is clear. The authors split failed experience into training views by strength of evidence: views that only require a diagnosis, views that require an executed and passing correction, and views that require comparing two actions from the same state. They also count the collected data, the training data, and the replay data separately.
  2. The control design of the replay experiment is in the right direction. The corrective action and the original action are paired from the same state, and the "both pass" and "only the original passes" outcomes are retained.
  3. The appendix is thorough. Negative results, multiple-comparison corrections, task-clustered confidence intervals, and overlap audits are all reported, which makes the results easy to check.
Critical weaknesses

W1. The diagnosis labels are not independently validated, and the reported experiments do not use the 50k collection.

  1. The 50,228 pairs pass only structural checks and support only coverage statistics. The diagnosis experiments use an earlier frozen release of 4,319 rows, whose 943-row holdout is labeled by the same multi-model pipeline as the training data, with no human labels. When the authors' own fifth rubric version is applied, only 18 holdout rows and 53 of 3,157 training rows remain (Table 9).
  2. The human review covers only 80 historical records, which are not drawn from the test set and were reviewed with AI assistance. Both reviewers accept the diagnosis content on only 41/80 records (Appendix W). In addition, 603 labels in the 948-task paired cohort were constructed with ground-truth content in the prompt (Appendix S), information the student never sees.
  3. The 63.6% exact-step accuracy therefore measures agreement with the teacher pipeline rather than diagnostic ability. The main text acknowledges this, but the abstract and Figure 1 place this number next to frontier models, which readers can easily take as a capability ranking.

W2. The evidence that the data helps correction, diagnosis, and acting is not robust.

  1. Correction: the main text reports a 32.7-point gain from first-proposal corrections, but under the cleaner setting of Appendix M (a single proposal and a fixed actor), the correction beats its control by only 12.9 points. A prompt with the diagnosis gains only 5.6 points (interval 0.0 to 11.3) over the single sentence "Reconsider your next action." (Table 27), so the contribution of the diagnosis content itself is not established.
  2. Diagnosis model: under the shared prompt, the trained model falls 31.5 points below the base on Who&When AG and 20.5 points on TrajErrBench (Table 16). Only the unified Who&When protocol shows a significant gain.
  3. Acting agent: compared with success-only SFT, repair data improves only WebShop-lite significantly (+6.67), while TextQuest drops by 8.3 to 9.7 points. On real ALFWorld and ScienceWorld, the preventive arm, the only one tested there, falls from 5.97% and 9.67% to 0.75% and 0.34%.
Questions for authors
  1. Given that Table 1 lists Exec. fix and Ctrl. as distinguishing features of AED, how does the replay verification differ from DoVer (arXiv:2512.06749), which also intervenes on failure hypotheses, re-executes from the intervention point, and uses the outcome to validate them, and from A2P (arXiv:2509.10401), which simulates counterfactual interventions for step-level attribution on Who&When? Should CORRECT (arXiv:2509.24088) also be included in Table 1?
  2. Since Appendix F.1 states that ground-truth access was permitted during diagnosis, were the corrective actions in the replay study generated with access to ground truth or reference answers, and could you report these cases separately with each branch sampled more than once, given that re-executing the original action already passes 18.4% of the time?
  3. Since only 871 of the 3,062 pairs were replayed under the original harness and 1,846 were continued with a single-agent ReAct loop (Table 14), what is the first-proposal effect on these 871 pairs, and what mainly explains the gap between the 32.7 points in the main text and the 12.9 points in Appendix M?
  4. Given that the same base model reaches 56.90% agent accuracy on Who&When HC when continuation is allowed but 22.41% when only the first response is scored (Tables 2 and 3), how do the frontier models perform under the continuation protocol, and does the student still surpass them?
  5. Given that training only on the compact localization target leaves the model below the base on the holdout (46.59% vs. 47.19%, Appendix I.13) while training on the full diagnosis reaches 63.56%, could you compare the two under matched data and conditions?
  6. Could you measure diagnosis accuracy on a test set labeled independently by humans without AI assistance, and how are cases with multiple plausible erroneous steps handled in evaluation?
5 comments

We thank the reviewer for a rigorous, detailed review and for highlighting our evidence-graded training views, paired replay control, and thorough appendices. We address every point below, with corresponding updates marked in blue in the revised PDF.

W1.1. Structural checks on the 50k pairs vs. Table 9 row counts. The 50,228-pair collection is the full released dataset, whereas the frozen release is the controlled subset used for our experiments; we now state this clearly in Sec. 3.3 and Sec. 5.2. Table 9 applies our strictest current rubric (v5) to the frozen release (with prompts and tool lists restored), leaving 18 holdout and 53 training rows under that conservative audit rather than defining the boundary of usable training data. To measure label quality directly, we pair lenient evaluation with an independent human study (W1.2).

W1.2 and Q6. Unassisted human labels and multiple plausible error steps. We are finishing an unassisted human annotation study with no AI-generated content. Blind annotators read the same trace the model sees, record both their single preferred error step and all defensible error steps, lock their answers, and only then view our diagnosis. Covering the 80 audited records plus a fresh stratified holdout sample, this provides a clean human benchmark for test-set accuracy, reporting strict accuracy (preferred step), lenient accuracy (any defensible step), and teacher accuracy split by ground-truth access. Results will be posted in a few days.

For multiple plausible steps on the full holdout set, lenient scoring using the teacher's recorded alternative steps confirms AED's clear advantage:

Holdout, lenient exact stepAccuracy
AED (3-seed mean)74.48
Frontier models61.6–66.7
Base56.95

W1.3. Presenting 63.6% alongside frontier models in the abstract and Fig. 1. We updated the abstract, Fig. 1b, and Sec. 5.2 to state explicitly that 63.6% measures agreement with teacher labels.

W2.2. Losses under the shared prompt. Format-agnostic rescoring shows that the shared-prompt Who&When AG drop is driven by output formatting: format compliance drops from 94.4% to 18.8% on the unseen prompt, whereas on cases both models format validly, localization accuracy is comparable (42.0 vs. 43.7). After overlap-clean retraining across three seeds, our model improves unified Who&When by +6.70 points [0.71, 12.89].

W2.1 and Q3. 32.7 points in the main text vs. 12.9 in Appendix M; the 871 same-harness pairs. These two metrics evaluate distinct stages: main-text replay validates construction-time corrections (our training targets), whereas Appendix M tests an autonomous test-time proposer without reference data on one fixed actor across 150 failures (one per task). To verify that harness substitution did not inflate the main-text replay gain, we re-analysed all 3,062 replay pairs at the pair level (App. Y.2):

Replay subsetPairsFirst-proposal gain (pts) [95% CI]Discordant pairs
All pairs (main text)3,062+32.7 [28.4, 37.0]—
Original harness871+38.6 [33.7, 43.5]346 vs. 10
Original harness, selected contrast871+47.1 [42.3, 51.7]—
Reference payload but no verifier output1,757+34.6 [28.5, 40.8]—
Excluding the 99 pairs that used a reference trajectory2,963+31.9 [27.8, 36.5]—
Exactly one trial per arm2,410+29.3 [26.6, 31.9]788 vs. 83

The first-proposal gain is actually higher on the 871 original-harness pairs (+38.6) than on the full set (+32.7), so substituting the ReAct continuation does not inflate the replay gain.

Q2. Ground truth in replay proposals and repeated sampling. Yes: replay corrections are construction-time training targets proposed with access to the task's reference payload (as noted in App. F.1 and now stated upfront in Sec. 5.1). However, the table above shows that the gain does not rely on verifier leakage or multi-sampling: without verifier output, the gain remains +34.6, and with strictly one trial per arm, the gain is +29.3 (788 vs. 83 discordant pairs).

W2.3. Acting agent. We ran an exposure-matched, five-seed actor comparison with 5× enlarged procedural test sets (full table in our response to Reviewer AAaz, W3). Under matched exposure, the earlier TextQuest loss disappears, so it came from unequal task exposure, and GridWorld shows a significant gain from repair supervision (1,886 vs. 1,728 solved across five seeds, p < 0.001). On real ScienceWorld, success-only training drops just as far as every repair arm under both the original recipe (3 of 300 solved vs. 29 untrained) and the matched recipe, so the ScienceWorld drop comes from fine-tuning on the development environments rather than from repair data. Sec. 5.3 now presents these five-seed results per environment.

Q1. Positioning relative to DoVer, A2P, and CORRECT. We integrated all three works into Table 1 and Sec. 2 (summarized in Table B of the General Response). DoVer is the closest: it branches from checkpoints to execute edited orchestrator messages across three trials and compares against the original failed log. AED adds the control DoVer lacks: it re-executes the original action from the identical state, so stochastic retries are not miscounted as valid repairs. This matters because original actions pass 18.4% of the time upon retry in our data. By contrast, A2P only simulates counterfactuals in-prompt without execution, and CORRECT relies on LLM-injected errors across >2,000 trajectories, whereas all AED failures arise naturally and are verified against a matched execution control.

Q4. Frontier models under the continuation protocol. The continuation protocol only triggers when a local model hits the generation token limit mid-reasoning block. Hosted frontier models never hit this limit in our runs (App. V.1), so their scores are identical under both protocols. Our trained student outperforms frontier models on hand-crafted responsible-agent accuracy (70.69 vs. 56.90, and 68.97 vs. 60.34), while GPT-5.6 Sol remains ahead on exact step localization.

Q5. Compact vs. full targets under matched data and conditions. The earlier gap between compact and full targets was caused by training the compact baseline with LoRA rather than full fine-tuning. We retrained the compact-target model under strictly matched conditions: the same 1,656 rows, byte-identical inputs, and the full fine-tuning recipe (Sec. 5.2, App. Y.3).

Arm (1,656 tasks, holdout exact step)Accuracy
Full diagnosis, full fine-tuning (3-seed mean)63.56
Compact, full fine-tuning (new, matched recipe)63.1
Compact, LoRA (original)46.59
Untrained base (same evaluation run)47.6

Under matched full fine-tuning, compact targets reach 63.1%, within 0.5 points of full targets and 15.5 points above the base model, so the learning signal comes from AED's step-level supervision under either target format.

We hope these new experiments and analyses answer your questions, and we would be grateful if you would consider raising your score.

Reviewer_yZfNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Agent Error Dataset (AED), comprising 50,228 error–diagnosis pairs across 33 environments, 19 agent harness families, and 23 policy models. Its five-stage AET pipeline collects failures, generates and checks diagnoses, tests corrections through paired replay where supported, and constructs training examples for diagnosis and action learning. Experiments show improved verifier pass rates with proposed corrections and improved agreement with internal teacher labels after diagnosis SFT. External diagnosis results vary across benchmarks, while repair-based action training produces environment-dependent gains and losses in task success.

Critical strengths

AED provides a substantial resource for diagnosing errors, covering diverse environments, agent frameworks, and policy models. Its construction pipeline combines trace-grounded diagnosis checks with paired replay where supported, distinguishing evidence-supported diagnoses from empirically tested corrections. The dataset supports multiple training objectives, and the paper evaluates correction utility, diagnosis learning, and action-policy training while documenting data-quality limitations and negative results.

Critical weaknesses
  1. The correctness of the diagnostic annotations is not sufficiently established. Agreement with internal teacher labels does not independently demonstrate diagnostic correctness. Successful replay provides evidence that a correction works, but does not necessarily establish that its accompanying diagnosis is correct. The annotation problems reported in the audits further motivate stronger independent validation of the dataset’s labels.

  2. As acknowledged in Section 5.3, the experiments on training agents with repair data vary in task coverage, training exposure, and optimizer updates. They therefore evaluate complete training recipes rather than isolate the contribution of repair data. Although this limitation is clearly disclosed, it leaves the dataset’s specific benefit for training agents insufficiently established. A controlled comparison with matched task coverage and training budgets is needed to attribute improvements more confidently to repair data.

  3. The paper reports diversity across agent frameworks, but the number of agents within the multi-agent systems is unclear. Could the authors summarize the distribution of agent counts and clarify whether diagnosis or correction performance varies with system size? A controlled scaling experiment is not essential, but descriptive statistics would help characterize the dataset’s multi-agent coverage.

Questions for authors
  1. It is difficult to follow this paper. For example,

    a). Section 3.2 is difficult to follow because it introduces several terms, such as student-visible trace and build views, without sufficiently explaining their operational meaning. I think a running example would substantially improve clarity.

    b). In Section 3.3 Source tasks, stored source-trace blobs, and error–diagnosis pairs are related and should be clearly distinguished. A schematic illustrating the relationships among them would be beneficial.

    c). The spacing around lines 256–269 appears unusually large, particularly between the takeaway box and the subsequent paragraphs. The standalone sentence referring to Appendix B.1 also feels abrupt, as it concerns source coverage and diagnosis multiplicity rather than the surrounding discussion of failure profiles.

    d). Equation (1) would benefit from a clearer explanation of LdiagL_{\mathrm{diag}} and LactorL_{\mathrm{actor}}, explicitly linking them to diagnosis generation and action prediction, respectively. Although the text briefly describes their normalization, it does not explain why diagnosis training averages token-normalized losses across examples, whereas actor training averages over all target tokens. Please clarify the rationale for this distinction and its implications for weighting examples of different lengths.

7 comments

We thank the reviewer for the thorough feedback and constructive presentation suggestions. We have implemented all four presentation improvements in the revised PDF (in blue) and address each technical point below.

W1. Correctness of the diagnostic annotations. The revision now explicitly distinguishes our two verification mechanisms: replay verifies that a proposed correction succeeds in execution, while diagnosis checks verify that the explanation is grounded in the trace. To independently validate error localization itself, we are completing an unassisted human annotation study in which annotators identify the error step blind before seeing our diagnosis; results will be posted in a few days. Meanwhile, lenient scoring, which accepts any step the teacher recorded as defensible, yields 74.5 for AED versus 57.0 for the base model and 61.6–66.7 for frontier models, so our gains do not rely on a single rigid step label.

W2. Controlled comparison with matched task coverage and training budget. We ran this controlled comparison across five seeds: success-only and repair arms train on the same 588 tasks, with tokens and turns matched within 1% per environment and 736 updates each, and are evaluated on 5× enlarged GridWorld and Warehouse test sets (full table in our response to Reviewer AAaz, W3). Matching exposure shows where repair data helps: on GridWorld, it solves 1,886 vs. 1,728 tasks across five seeds (p < 0.001), while the earlier TextQuest drop disappears once exposure is equalized. Sec. 5.3 now reports these five-seed results per environment.

W3. Number of agents in multi-agent systems. We added a complete agent-count census across all 50,228 pairs (Sec. 3.3, App. Y.5). AED's core contribution is verified diagnosis and correction supervision built from naturally occurring failures; multi-agent rollouts are not its main focus, so we did not design elaborate multi-agent systems.

Acting agents per traceShare of pairs
193.4%
2 (mostly planner–executor systems)6.6%

Both diagnosis and replay gains hold on two-agent traces:

SubsetTrained modelBaseClaude Opus 5
Two-agent holdout cases (125), exact step60.0–60.8%42.4%48.8%

In replay, corrections achieve a +50.7 point gain on the 345 multi-agent pairs, compared to +43.5 and +47.1 on single-agent pairs.

Q1(a). Running example for the Sec. 3.2 terms. Sec. 3.1 now tracks a concrete running example (Case A) across all five pipeline stages. The student-visible trace is the exact input to the diagnosis model: task description, available tools, numbered trajectory steps, and final outcome, excluding any reference answer, verifier output, or replay result. Build views are the supervised training formats derived from a single record: diagnosis (trace → error step, agent, explanation, and correction), recovery (prefix before the error → corrected command), and preference (corrected vs. original command from the identical state).

Q1(b). Counting units. Sec. 3.3 now states the data hierarchy explicitly: 9,961 source tasks → 38,278 stored failed runs (source-trace blobs) → 50,228 error–diagnosis pairs.

Q1(c). Spacing and the Appendix B.1 sentence. Fixed. We corrected the spacing and moved the Appendix B.1 sentence into the counting-units paragraph in Sec. 3.3.

Q1(d). Eq. (1). Sec. 4 now explicitly defines l_i as the summed negative log-likelihood over the target tokens of example i. For diagnosis, we average per example so every trajectory weighs equally; because diagnosis targets vary in length due to quoted trace evidence, token-level averaging would overweight verbose explanations relative to the concise step prediction we evaluate. For the acting agent, we follow the success-only baseline and average per token, so a short termination action is not weighted equally to a full reasoning step. Because arms with longer turns receive higher gradient weight under token averaging, we explicitly report and match token budgets across arms.

We hope these revisions and controlled experiments address your questions, and we would be grateful if you would consider raising your score.

Thank you again for the detailed presentation feedback. We revised the paper along each of your comments (changes in blue) and have uploaded the new version:

Your commentWhat we changedWhere
Q1(a): Sec. 3.2 terms; running exampleOne failure (Case A) now runs through all five pipeline stages; "student-visible trace" and "view" are definedSec. 3.1–3.2; App. F.1
Q1(b): tasks, traces and pairs; schematicThe three counting units are defined, with an inline schematic: 9,961 source tasks → 38,278 source traces → 50,228 pairsSec. 3.3
Q1(c): spacing; Appendix B.1 sentenceSpacing fixed; the B.1 pointer moved into the counting paragraphSec. 3.3
Q1(d): Eq. (1)The two losses, and why they average differently, are explained in wordsSec. 4
W1: annotation correctnessA lenient-scoring check is added; the unassisted human study will be posted in the next few daysSec. 5.2; App. Y.8
W2: controlled comparisonExposure-matched comparison over five seedsSec. 5.3; App. Y.7
W3: number of agentsAgent counts reported: 93.4% single-agent, 6.6% two-agentSec. 3.3; App. Y.5

We also corrected two details in our earlier reply: the replay gains by group under W3 (+41.4 / +38.6 / +28.3 for the first proposal, App. Y.2) and the definition of the student-visible trace under Q1(a).

We hope the paper is now easier to follow, and we are glad to clarify anything that remains unclear. Thank you again.

Reviewer_AAazOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AED collects 50,228 error–diagnosis pairs across 33 environments and introduces a pipeline linking failures, diagnoses, proposed corrections, and optional replay evidence. Separate experiments assess correction utility, diagnosis training, and acting-policy training.

Critical strengths

The resource addresses a useful gap between unsuccessful trajectories and actionable training supervision. Matched original-action retries are a strong experimental control: first-proposal corrections improve verifier pass rates by 32.7 percentage points, with a task-clustered interval of [28.4, 37.0]. Three-seed diagnosis-training experiments also show systematic internal learning gains. The separation of replay success from learned autonomous recovery is appropriately explicit.

Critical weaknesses

Dataset quality and the evaluated pipeline are not sufficiently connected. The diagnosis experiments use an older release whose construction omitted policy system prompts and tool lists and predates current semantic checks. The historical, AI-assisted human audit cannot establish the quality of the final collection. Thus, the headline collection scale and learning results do not yet validate a single, consistently quality-controlled resource.

Internal teacher agreement is stronger than the evidence for general diagnostic ability. The 47.2%→63.6% improvement measures agreement with construction labels, not independent correctness. External results depend on protocol and task overlap; some Who&When settings deteriorate, and mean TrajErrBench accuracy remains below the base model. The existing external evaluations are valuable, but do not establish robust transfer.

Actor improvements are not causally attributable to repair supervision. Training recipes differ in task pools, exposure, and optimizer updates and use one seed. WebShop-lite gains coexist with significant TextQuest losses, while the tested preventive recipe loses to the untrained policy on real ALFWorld and ScienceWorld. Controlled, exposure-matched experiments are necessary to isolate the value of the proposed supervision.

Questions for authors

Can the updated, fully contextualized release receive an independent audit and support multi-seed, exposure-matched training evaluations on overlap-clean external benchmarks?

4 comments

We thank the reviewer for recognizing the matched original-action control, the three-seed diagnosis gains, and our clear separation of replay success from autonomous recovery. We address each point below, with all revisions marked in blue in the updated PDF.

W1. Connecting dataset quality to the evaluated pipeline. The 50,228-pair collection is our full released resource, while the frozen release is the controlled subset used for training and evaluation; we now state this relationship explicitly in Sec. 3.3 and Sec. 5.2. In Table 9, the re-export already restores policy prompts and tool lists, so the numbers reflect our strictest content checks rather than missing context.

To measure label quality directly, we are completing an unassisted human annotation study. Annotators who have never seen the records first locate the error blind, lock their answer, and only then evaluate our diagnosis. This covers the 80 previously audited records plus a fresh, environment-stratified sample of holdout cases. We will report strict and lenient human–teacher agreement, human–human agreement as a baseline, and teacher accuracy split by ground-truth access. We will post the results in a few days.

W2. Teacher agreement vs. general diagnostic ability. 63.6% indeed measures agreement with teacher labels, and we now state this explicitly in the abstract, Fig. 1, and Sec. 5.2. Three new analyses show that the gains are not an artefact of scoring against a single teacher preference:

AnalysisResult
Lenient scoring: any step the teacher recorded as acceptable counts (317 of 943 cases have alternatives)Trained model still gains +15.9 to +18.4 pts over the base (App. Y.4)
Overlap-clean retraining, 3 seeds, unified Who&When+6.70 pts [0.71, 12.89] over the base
Format-agnostic rescoring of the shared-prompt Who&When AG dropFormat compliance falls from 94.4% to 18.8%; on cases both models answer, accuracy is on par (42.0 vs. 43.7)

The shared-prompt drops stem primarily from formatting mismatches on an unseen prompt template, rather than weaker error localization.

W3. Attributing actor improvements to repair supervision. We ran the exposure-matched experiment the reviewer asked for, with five seeds. Success-only and post-error repair training use the same 588 source tasks, with supervised tokens and turns matched within 1% per environment and the same 736 optimizer updates (App. Y.7). Because the procedural test sets originally had 100 tasks, we enlarged them fivefold (499 GridWorld and 500 Warehouse tasks, subsuming the original 100 and disjoint from training). Below are the solved tasks summed over all five seeds, with paired exact McNemar tests on the pooled pairs:

Environment (tasks × 5 seeds)Success-onlyWith repair datap
GridWorld (499)1,7281,886< 0.001
Text-to-SQL (1,014)3,9393,9580.50
TextQuest (300)1,3491,3290.18
ALFWorld-lite (300)1,4341,4440.33
WebShop-lite (300)1,3201,2680.003
Warehouse (500)865245< 0.001
Real ALFWorld (134)4917< 0.001
Real ScienceWorld (300)16110.42

Controlling exposure isolates the direct effect of repair supervision. On GridWorld, repair data improves performance in four of five seeds, solving 158 more tasks overall. The previous TextQuest drop is no longer statistically significant, so it arose from unequal task exposure rather than from repair supervision. Because effects vary by environment, Sec. 5.3 now reports per-environment results across all five seeds rather than a single aggregate. Finally, evaluating all original arms on real ALFWorld and ScienceWorld (App. Y.6) shows that on ScienceWorld, success-only training drops just as much as every repair arm (3 of 300 solved vs. 29 for the untrained policy), and both matched arms remain near 1% (16 vs. 11 over five seeds). The ScienceWorld drop therefore comes from fine-tuning on the development environments, not from repair data.

Q. Independent audit and multi-seed, exposure-matched evaluation on overlap-clean benchmarks.

  • Independent audit: The unassisted human study is finishing; we will post results in a few days (W1).
  • Multi-seed, exposure-matched actor evaluation: Completed across five seeds with 5× enlarged procedural test sets (W3).
  • Overlap-clean external diagnosis evaluation: Completed across three seeds, showing a +6.70 pt gain on unified Who&When (W2).
  • Contextualized retraining: Retraining on the fully contextualized release will be included in the camera-ready version.

We hope these controlled experiments address your concerns, and we would be grateful if you would consider raising your score.

Dear Reviewer AAaz,

Thank you again for your careful review. As the discussion period ends in a few days, we would like to briefly follow up.

Our responses above address each of your points, and General Response (3/3) now adds the latest human-evaluation results, including a review with no AI assistance in which the rater localizes the error before seeing our diagnosis. This speaks directly to your concern about independent evidence for the diagnosis labels. The revised PDF also includes the exposure-matched, five-seed actor comparison you asked for.

We would be grateful to know whether these responses resolve your concerns. If they do and you have no further questions, we would appreciate it if you could consider updating your score. If anything remains unclear, we are happy to provide further clarification or analysis in the remaining time.

Thank you for your time and consideration.

Best regards, The Authors

Dear Reviewer,

We really appreciate your feedback, which helped us improve our paper a lot.

As the Discussion period comes to a close. we are still sad not to have heard from you.

We are eager to address any remaining concerns and would like to know whether our response fully addressed them.

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.