- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ReproSaddler turns anonymized failure descriptions and action records into executable cases in public donor environments, without supplying the original environment state. The method specifies a typed failure condition, compares configurable conditions with an all-off baseline, and uses cases that pass validation to improve a fixed agent harness. Studies on GAIA2 and Workspace-Bench include random tasks, unrelated tasks, observable-only variants, and privileged original tasks as comparisons.
Critical strengths
- Comparing all-off with all-on conditions while keeping the failure matcher fixed helps distinguish reproduction of the intended error from general task difficulty. A case is accepted only if the preserved workflow produces the specified failure (Section 4, Eq. 9).
- Table 1 evaluates the value of reconstructed cases through downstream results and appropriate baselines. In Workspace-Bench, the generated-task condition raises target-failure avg@3 from 43.81 to 56.19 and the score on an unseen environment group from 39.96 to 49.85. These outcomes reveal more than the number of successful reproductions alone.
Critical weaknesses
- Appendix B.3 indicates that the reported means and standard deviations come from three executions of one chosen harness, rather than three separate runs of reconstruction and optimization. Consequently, the table does not quantify how consistently the complete procedure selects an effective harness. Independent repetitions of the full procedure, or uncertainty estimates across source tasks and environment groups, would better support the performance claim.
- The motivating use case involves field reports subject to privacy limits, whereas the experiments start from benchmarks that provide oracle action sequences; Appendix G notes that actual reports may not include these. Moreover, the enforced data boundary is a software rule rather than proof of privacy, and the earlier Workspace-Bench generators accessed privileged materials (Appendix C). An access record for each stage and experiments with independently gathered, incomplete reports would help establish relevance to the proposed setting.
Questions for authors
- If reconstruction and AutoSaddler optimization are each repeated from new seeds, what variation appears in final performance? The current three executions of a selected harness do not answer this.
- Which generator version and source fields produced each Workspace-Bench condition in Table 1? Could you document that reconstructed scenarios had no access to source file contents or paths that identify the source?
2 comments
Gentle Reminder on Our Response
Dear Reviewer mSQh,
Thank you again for your valuable feedback. As the discussion period ends soon (Nov 18), we would like to kindly remind you that we have posted our response addressing your comments.
We would greatly appreciate it if you could take a look and let us know whether it resolves your concerns, or if there is anything further we can clarify.
Best regards, Authors
Official Comment by Authors
We are deeply grateful to the reviewer for their thorough review and valuable feedback. In the following, we provide detailed responses to the specific points raised by the reviewer.
W1 & Q1
To address the weakness, we repeat reconstruction and optimization in two additional settings.
GPT-5.5 was deprecated on our serving platform, so the new experiments use GPT Luna for the agent, the optimizer, and the judge, chosen over GPT Sol for cost. Absolute scores and their variance may therefore differ from Table 1 of the submission.
Setting 1
Setting 2
Since these results show little difference from the previous results reported in the paper, data generation adds little variance to the final performance. Together with the fact that all executable arms share the same optimizer and differ only in their training data, this indicates that the effect of our method comes from the generated data rather than from the training process. The larger standard deviation on GAIA2 comes from the small test sets, where a single task in the GAIA2 original test set changes the score by 2 to 2.5 points. Across all settings, ReproSaddler effectively recovers the performance gain obtained with the original data without accessing any private information.
W2 & Q2
Our motivation is to improve the harness without accessing private user information, and having an oracle action does not conflict with this goal. An oracle action can come directly from the user's correction, such as "I wanted A, but you did B", where A is the oracle action. This information is stated by the user rather than read from the user's environment. Reproducing failures when even this oracle is unavailable is a natural next step, and we will add it to the limitations as future work.
Furthermore, the table below documents exactly which information the generator can access when it generates data.
We agree that filtering private information is not a proof of privacy. However, blocking access to the source environment alone is effective in practice. We audited all 46 generated scenarios in the submission (26 on GAIA2 and 20 on Workspace-Bench) against the original data.
The single 8-gram match is a generic phrase ("The first order was canceled, so I am"), which we consider coincidental. These results show that blocking access to the source environment alone keeps file names and user requests of the original data out of the generated data.
Lastly, all Workspace-Bench results come from the restricted-input generator described in Appendix B.4, which does not use privileged source inputs. We will revise the parts of the paper that may cause confusion on this point.