acceptodds
Under review as a conference paper at ICLR 2027

ReproSaddler: Reproducing Agent Failures for Harness Optimization

Abstract

Failures of deployed language-model agents reveal concrete weaknesses in their prompts, tool interfaces, and execution logic, but they are difficult to reuse for harness optimization. The interaction is often observable while the persistent environment that conditioned the failure stays private. Copying that state is not permitted, and an error label alone discards the context that made the failure informative. We introduce ReproSaddler, a framework for converting such evidence into fresh, executable failure scenarios. A trusted compiler fixes a de-identified oracle–agent failure relation and a value-free workflow blueprint, which the generator cannot revise. The generator establishes a scaffold for the required workflow in a retrieved donor environment, then proposes individually switchable task and environment conditions. Strict oracle validation and an all-conditions-OFF success control distinguish an executable task from the conditions under which the seed harness reproduces the target relation. Trace-guided refinement and conservative ablation reduce those conditions, so an accepted scenario targets the specific observed discrepancy rather than an arbitrary error. We formulate a six-arm evaluation that compares generated scenarios with observable-only evidence, random tasks, unrelated failures, privileged original tasks, and the unmodified harness. The primary endpoint is repair of the original user failures, which makes failure reproduction a testable data-generation claim rather than a proxy for agent improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agent, Agentic AI, Harness optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mSQhOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ReproSaddler turns anonymized failure descriptions and action records into executable cases in public donor environments, without supplying the original environment state. The method specifies a typed failure condition, compares configurable conditions with an all-off baseline, and uses cases that pass validation to improve a fixed agent harness. Studies on GAIA2 and Workspace-Bench include random tasks, unrelated tasks, observable-only variants, and privileged original tasks as comparisons.

Critical strengths
  1. Comparing all-off with all-on conditions while keeping the failure matcher fixed helps distinguish reproduction of the intended error from general task difficulty. A case is accepted only if the preserved workflow produces the specified failure (Section 4, Eq. 9).
  2. Table 1 evaluates the value of reconstructed cases through downstream results and appropriate baselines. In Workspace-Bench, the generated-task condition raises target-failure avg@3 from 43.81 to 56.19 and the score on an unseen environment group from 39.96 to 49.85. These outcomes reveal more than the number of successful reproductions alone.
Critical weaknesses
  1. Appendix B.3 indicates that the reported means and standard deviations come from three executions of one chosen harness, rather than three separate runs of reconstruction and optimization. Consequently, the table does not quantify how consistently the complete procedure selects an effective harness. Independent repetitions of the full procedure, or uncertainty estimates across source tasks and environment groups, would better support the performance claim.
  2. The motivating use case involves field reports subject to privacy limits, whereas the experiments start from benchmarks that provide oracle action sequences; Appendix G notes that actual reports may not include these. Moreover, the enforced data boundary is a software rule rather than proof of privacy, and the earlier Workspace-Bench generators accessed privileged materials (Appendix C). An access record for each stage and experiments with independently gathered, incomplete reports would help establish relevance to the proposed setting.
Questions for authors
  1. If reconstruction and AutoSaddler optimization are each repeated from new seeds, what variation appears in final performance? The current three executions of a selected harness do not answer this.
  2. Which generator version and source fields produced each Workspace-Bench condition in Table 1? Could you document that reconstructed scenarios had no access to source file contents or paths that identify the source?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We are deeply grateful to the reviewer for their thorough review and valuable feedback. In the following, we provide detailed responses to the specific points raised by the reviewer.


W1 & Q1

To address the weakness, we repeat reconstruction and optimization in two additional settings.

  • Setting 1. We repeat the full procedure of ReproSaddler 10 times, from data generation through harness optimization to evaluation.
  • Setting 2. We fix the generated data and repeat harness optimization and evaluation 10 times.

GPT-5.5 was deprecated on our serving platform, so the new experiments use GPT Luna for the agent, the optimizer, and the judge, chosen over GPT Sol for cost. Absolute scores and their variance may therefore differ from Table 1 of the submission.

Setting 1

RunGAIA2 originalGAIA2 held-outWorkspace-Bench originalWorkspace-Bench held-out
132.55253.347.4
237.55453.645.3
3355252.450.6
4305656.544.6
537.55055.748.5
632.55453.648.5
7355854.552.4
837.55656.252.5
927.55653.247.2
1037.55455.547.5
Mean34.2554.2054.4548.45
Std3.552.391.432.69

Setting 2

RunGAIA2 originalGAIA2 held-outWorkspace-Bench originalWorkspace-Bench held-out
1355253.350.1
232.55252.249.6
3305853.546.4
437.55651.850.7
527.55056.152.3
632.55655.351.8
7355253.247.2
837.55655.247.9
932.55057.349.9
1037.55656.151.9
Mean33.7553.8054.4049.78
Std3.392.901.852.04

Since these results show little difference from the previous results reported in the paper, data generation adds little variance to the final performance. Together with the fact that all executable arms share the same optimizer and differ only in their training data, this indicates that the effect of our method comes from the generated data rather than from the training process. The larger standard deviation on GAIA2 comes from the small test sets, where a single task in the GAIA2 original test set changes the score by 2 to 2.5 points. Across all settings, ReproSaddler effectively recovers the performance gain obtained with the original data without accessing any private information.


W2 & Q2

Our motivation is to improve the harness without accessing private user information, and having an oracle action does not conflict with this goal. An oracle action can come directly from the user's correction, such as "I wanted A, but you did B", where A is the oracle action. This information is stated by the user rather than read from the user's environment. Reproducing failures when even this oracle is unavailable is a natural next step, and we will add it to the limitations as future work.

Furthermore, the table below documents exactly which information the generator can access when it generates data.

InformationGenerator accessNote
Source user requestNoReplaced by a de-identified abstract task
Names, addresses, and titles in the requestNoReplaced by placeholders in a de-identified summary
Source app state and file contentsNo
File names and file pathsNo
Failure (difference from the oracle)Yes
Donor stateYesA separate environment with no private user information
Harness codeNo
Held-out test setNo

We agree that filtering private information is not a proof of privacy. However, blocking access to the source environment alone is effective in practice. We audited all 46 generated scenarios in the submission (26 on GAIA2 and 20 on Workspace-Bench) against the original data.

CheckMatches
File names identical to the original data0
8-gram overlap with the original user requests1

The single 8-gram match is a generic phrase ("The first order was canceled, so I am"), which we consider coincidental. These results show that blocking access to the source environment alone keeps file names and user requests of the original data out of the generated data.

Lastly, all Workspace-Bench results come from the restricted-input generator described in Appendix B.4, which does not use privileged source inputs. We will revise the parts of the paper that may cause confusion on this point.

AuthorsOpenReview ↗

Gentle Reminder on Our Response

Dear Reviewer mSQh,

Thank you again for your valuable feedback. As the discussion period ends soon (Nov 18), we would like to kindly remind you that we have posted our response addressing your comments.

We would greatly appreciate it if you could take a look and let us know whether it resolves your concerns, or if there is anything further we can clarify.

Best regards, Authors

Reviewer_WZqWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Harness optimization needs executable tasks, but deployed failures may depend on environment state that cannot be shared. ReproSaddler fixes an oracle-agent discrepancy and available workflow constraints through a trusted projection, then builds a fresh donor scenario with switchable conditions. The proposed method is primarily a prompt+workflow approach to generate data to drive the AutoSaddler harness optimization framework: refinement, condition deletion, and recovery supply harness-optimization data with underlying LLM weights fixed. Experiments on GAIA2 and Workspace-Bench evaluate original-failure repair and held-out transfer, and produce positive if limited improvements.

Critical strengths
  1. The paper connects field-failure reproduction under restricted access with harness optimization for stochastic, stateful agents (Section 2). This is an important problem for real-world agent improvement using field data. Fixing the target before generation, then requiring reference replay and workflow-preserving OFF/ON controls, gives practitioners a concrete way to obtain repair data when source environments are unavailable.
  2. The evaluation reports positive, if limited gains on a restricted set of benchmarks. Table 1 reports held-out gains over Seed on both benchmarks. However, the improvement confidence bands across the treatment arms overlap strongly, and it is difficult to be positive that the proposed approach beats alternatives (e.g. Random). See below.
Critical weaknesses
  1. Section 5.2 attributes the gains specifically to executable reproductions, but Observable uses a separate controller, and actual per-arm optimization exposure and costs are not reported. Its result supports the utility of the complete procedure, but cannot identify the separate contribution of executable evidence. It'd be helpful to implement one or more of (i) more seeds to narrow the confidence bands, (ii) more benchmarks, (iii) more careful statistical analysis (e.g. mixed-effects model) to tease out per-arm fixed effects from per-task random ones.
  2. Generation provenance needs clarification to make the restricted-access result auditable. Appendix B.4 explicitly describes restricted-input Workspace generation, but its linkage to Table 1's training cohort is unclear across the inventories in D.2 and historical variants in C and G. GAIA2's donor description also contradicts itself: U21-U27 is said to include U28, which Figure 2 includes (p. 17).
Questions for authors

What are the actual per-arm task exposure and optimization/generation costs? Distinguish conclusions supported by the shared-optimizer executable comparisons from those involving Observable's different controller, and narrow Section 5.2's attribution accordingly.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We would like to express our sincere gratitude to the reviewer for insightful feedback to improve the quality of our manuscript. We have carefully addressed each of the weaknesses identified by the reviewer.


W1 & Q1

AutoSaddler improves a harness by executing training tasks. In each iteration, it runs a minibatch with the parent harness, diagnoses the failure from the execution traces, reruns the same minibatch after patching to measure the change, and reflects on the before and after results. Observable has no executable environment by definition, so none of these steps can run, and a separate controller is unavoidable. The controller difference is therefore a direct consequence of having no executable evidence, which is exactly the factor we compare.

We designed the Observable controller to follow AutoSaddler as closely as possible, and the table below summarizes the setup.

AutoSaddler (executable arms)Observable controller
Minibatch size33
Epochs33
Sampler seed2026073020260730
Session modelSame model through the Copilot SDKSame model through the Copilot SDK
Editable harness surfaceSame allowlistSame allowlist
Candidate selectionSame development setSame development set
Execution of training tasksYesNo (not possible)
Before and after scores on the minibatchYesNo (not possible)
Sessions per iterationSelection, diagnosis and patch, reflectionOne repair session

On GAIA2, the Observable optimizer also receives the same de-identified projection format as the ReproSaddler generator, so the two arms start from the same evidence.

Moreover, our main claim, that ReproSaddler achieves an effect similar to Original without accessing private information, relies only on comparisons among executable arms that share AutoSaddler. This claim is therefore not affected by the controller difference. We will clarify in Section 5.2 which conclusions come from comparisons among executable arms and which involve Observable.

Additionally, The table below reports the training task exposure and the cost of ReproSaddler per run of the full procedure. Costs are in AIU, the cost unit of our serving platform (20 × uncached input + 2 × cached input + 120 × output, per 1M tokens).

Per run (mean)GAIA2Workspace-Bench
Training tasks of Original4020
Training tasks of ReproSaddler (accepted scenarios)2620
Generation cost of ReproSaddler (AIU)6,23023,964
Generation cost per accepted scenario (AIU)6771,536
Optimization cost of ReproSaddler (AIU)457288

ReproSaddler is exposed to fewer training tasks than Original on both benchmarks. On GAIA2, all executable arms share the same AutoSaddler setup and the number of proposals grows with the number of training tasks, so ReproSaddler does not use more optimization than Original.

The main additional cost is data generation, which is a one-time offline cost per accepted scenario. This cost replaces access to the private environment, which Original requires but a deployed system cannot obtain.


W2

All Workspace-Bench results come from the restricted-input generator described in Appendix B.4, which does not use privileged source inputs. Furthermore, the table below documents exactly which information the generator can access when it generates data.

InformationGenerator accessNote
Source user requestNoReplaced by a de-identified abstract task
Names, addresses, and titles in the requestNoReplaced by placeholders in a de-identified summary
Source app state and file contentsNo
File names and file pathsNo
Failure (difference from the oracle)Yes
Donor stateYesA separate environment with no private user information
Harness codeNo
Held-out test setNo

We also audited all 46 generated scenarios in the submission (26 on GAIA2 and 20 on Workspace-Bench) against the original data.

CheckMatches
File names identical to the original data0
8-gram overlap with the original user requests1

The single 8-gram match is a generic phrase ("The first order was canceled, so I am"), which we consider coincidental.

On GAIA2, we used only U21 to U27 as donors, and U28 was used only as the development set.

We will revise all parts of the text and figures that may cause confusion on these points.

AuthorsOpenReview ↗

Gentle Reminder on Our Response

Dear Reviewer WZqW,

Thank you again for your valuable feedback. As the discussion period ends soon (Nov 18), we would like to kindly remind you that we have posted our response addressing your comments.

We would greatly appreciate it if you could take a look and let us know whether it resolves your concerns, or if there is anything further we can clarify.

Best regards, Authors

Reviewer_ehRJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Agents fail because of private user state, not always agent errors or harness issues. ReproSaddler recreates these kinds of faulty environments and allows you to optimize harness on them.

Critical strengths
  • Interesting framing that we need to adversarially build these environments in addition to harnesses
  • ReproSaddler wins on Workspace-Bench
Critical weaknesses
  • Framing of results and interpretation becomes messier when some results are within noise
Questions for authors

How can you reduce the noise?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

We greatly appreciate the reviewer for the thoughtful assessment and constructive comments on our work. We are happy to address each of the weaknesses that the reviewer raised.


To analyze the results more thoroughly, we ran additional experiments with 10 runs each. GPT-5.5 was deprecated on our serving platform, so the new experiments use GPT Luna for the agent, the optimizer, and the judge, chosen over GPT Sol for cost. Absolute scores and their variance may therefore differ from Table 1 of the submission.

GAIA2, original test set

RunSeedRandomOther-FOriginalObservableReproSaddler
1303532.532.527.530
232.532.532.532.522.532.5
327.5353537.53035
427.53532.53527.535
51532.527.542.532.530
62532.532.5403032.5
73027.525352532.5
827.52527.537.527.535
92535253027.527.5
1027.527.527.532.532.532.5
Mean26.7531.7529.7535.5028.2532.25
Std4.723.743.623.873.132.49

GAIA2, held-out test set

RunSeedRandomOther-FOriginalObservableReproSaddler
1525852525656
2525852585458
3505854565456
4545452605854
5545456585056
6505458565452
7525456565456
8546054505652
9546254584854
10506252565058
Mean52.2057.4054.0056.0053.4055.20
Std1.753.272.112.983.132.15

Workspace-Bench, original test set

RunSeedRandomOther-FOriginalObservableReproSaddler
14851.650.459.84953.8
249.151.65862.849.753.3
346.65257.656.148.452.5
447.559.556.95548.856.9
54851.554.45757.856.4
652.453.153.256.451.258.3
748.455.753.355.348.659.1
849.152.350.856.153.252.8
948.853.449.754.150.556.9
1047.549.351.15649.852
Mean48.5453.0053.5456.8650.7055.20
Std1.572.813.102.572.892.60

Workspace-Bench, held-out test set

RunSeedRandomOther-FOriginalObservableReproSaddler
141.351.144.547.74451.7
243.749.253.850.84348.6
344.147.248.35249.650.1
4424951.151.74448.5
541.948.250.648.345.151.2
640.948.353.449.24741.9
741.648.550.15044.548.6
844.545.138.95142.949.6
946.852445344.949.6
1045.747.744.246.648.245.9
Mean43.2548.6347.8950.0345.3248.57
Std2.011.934.822.052.242.84
AuthorsOpenReview ↗

Official Comment by Authors (2)

Furthermore, to rigorously clarify the effectiveness of reconstruction and optimization, we conduct and present further experimental results from additional settings.

  • Setting 1. We repeat the full procedure of ReproSaddler 10 times, from data generation through harness optimization to evaluation.
  • Setting 2. We fix the generated data and repeat harness optimization and evaluation 10 times.

Setting 1

RunGAIA2 originalGAIA2 held-outWorkspace-Bench originalWorkspace-Bench held-out
132.55253.347.4
237.55453.645.3
3355252.450.6
4305656.544.6
537.55055.748.5
632.55453.648.5
7355854.552.4
837.55656.252.5
927.55653.247.2
1037.55455.547.5
Mean34.2554.2054.4548.45
Std3.552.391.432.69

Setting 2

RunGAIA2 originalGAIA2 held-outWorkspace-Bench originalWorkspace-Bench held-out
1355253.350.1
232.55252.249.6
3305853.546.4
437.55651.850.7
527.55056.152.3
632.55655.351.8
7355253.247.2
837.55655.247.9
932.55057.349.9
1037.55656.151.9
Mean33.7553.8054.4049.78
Std3.392.901.852.04

Without access to any private information, we observe that ReproSaddler effectively recovers the performance gain obtained with the original data across all settings. The remaining gap is expected because Original trains directly on the original test set itself. A The standard deviation is relatively large on GAIA2 because the test sets contain only 20 to 50 tasks, so a single task in the GAIA2 original test set changes the score by 2.5 points. Although the results may appear noisy, Figure 3 of the paper clearly demonstrates the effectiveness of our method when the test set is split into target failures and other failures. We emphasize again that our contribution is to generate data that achieves an effect similar to optimizing the harness on the original data, without accessing the private information in the original data, and these results sufficiently support this contribution.

AuthorsOpenReview ↗

Gentle Reminder on Our Response

Dear Reviewer ehRJ,

Thank you again for your valuable feedback. As the discussion period ends soon (Nov 18), we would like to kindly remind you that we have posted our response addressing your comments.

We would greatly appreciate it if you could take a look and let us know whether it resolves your concerns, or if there is anything further we can clarify.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.