- Rating
- 4(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReTask, a framework that converts existing agent execution records into reusable executable tasks for evaluation and supervision. ReTask first infers task specifications from execution records, then reconstructs unsolved initial environments and generates corresponding evaluators. Experiments on Workspace-Bench show that the generated tasks preserve model rankings consistent with the original benchmark, achieving a Spearman correlation of 0.964. Furthermore, experiments on SWE-ReBench demonstrate that the generated tasks can produce verified trajectories for supervised fine-tuning, improving downstream performance on multiple SWE-bench benchmarks.
Critical strengths
-
Novel and practical task compilation framework. The paper proposes an interesting approach to converting existing agent trajectories into reusable executable tasks, enabling more effective reuse of historical agent execution data.
-
Comprehensive evaluation across different settings. The authors evaluate ReTask on both Workspace-Bench and SWE-ReBench, demonstrating its effectiveness for both agent evaluation and training data generation.
-
Strong downstream training results. The generated trajectories improve the performance of two base models across all six SWE-bench evaluation settings, demonstrating the practical value of the proposed framework.
Critical weaknesses
I appreciate the motivation behind this work. However, I have some concerns about how the authors validate the effectiveness of their proposed method.
-
Based on my experience, directly using LLMs to generate test cases and environments can introduce substantial hallucination issues. In other words, both the generated tasks and their evaluators may contain errors. I believe that benchmarks constructed using this method still require human verification to ensure their quality. Simply demonstrating that the generated benchmark can distinguish between different models is not sufficient to establish the quality and reliability of the benchmark.
-
Using trajectories from an existing benchmark to construct new tasks and then evaluating the trained model on the same benchmark may introduce a risk of data leakage. I believe this potential issue needs to be carefully addressed.
Overall, these concerns are the main reasons why I cannot give this paper a score of 4.
Questions for authors
None
4 comments
Response 2
W2. Separation of Training and Evaluation Tasks
We clarify that the tasks used for SFT are distinct from those used for downstream evaluation. Our experiments use datasets with separate roles (§4.1, Appendix C.2):
| Dataset | Experimental role |
|---|---|
| Workspace-Bench | Task compilation and comparative agent evaluation only. No model fine-tuning. |
| SWE-ReBench | Source execution records and original software issues for compiling tasks and collecting new SFT trajectories. |
| SWE-bench Verified / Pro / Multilingual | Downstream evaluation of the fine-tuned models. |
Appendix C.2 confirms zero original-task identity overlap between the 1,123 SFT training tasks and each of the three downstream evaluation sets. This check uses the original software issues underlying the Compiled tasks, rather than relying solely on different dataset names.
Native and Compiled trajectories are paired entirely within the training corpus, with one successful trajectory per original issue and condition collected using the same teacher and agent protocol. This controls the supervision comparison without introducing overlap with downstream evaluation tasks.
Thus, the fine-tuned models are evaluated on different original tasks from those used to collect their supervision, directly addressing the concern about training and evaluation instance overlap.
Thanks again for your support for our work and its motivation!! We hope the additional evidence addresses your concerns. Best, Authors
It addresses my concerns
Thank you so much for your time! We really appreciate your support for our work and are glad that our responses addressed your concerns.
Response 1
We thank the reviewer for recognizing the motivation and practical value of converting historical execution records into reusable tasks, as well as our downstream training results.
W1. Quality of Generated Tasks and Evaluators
We agree that model discrimination alone does not establish the quality of generated tasks and evaluators. We address this concern through
two complementary assessments:ReTask derives task requirements from the actionable request and evidence about the initial state, using a shared specification to coordinate workspace and evaluator construction (§§3.1–3.3).
Evaluation protocol. For each source record, an LLM identifies the required resources, task requirements, and completion conditions from the actionable request and eligible initial-state evidence. This forms a reference specification, fixed before task construction and used to evaluate all methods on that record.
An LLM judge, blinded to method labels and without access to recorded solutions or solver scores, compares each generated task and evaluator against this reference. It checks which requirements are preserved, which completion conditions are implemented, and whether the evaluator introduces unsupported checks.
We compare Full ReTask with two method ablations:
Requirement recall measures the fraction of reference requirements preserved. Criterion coverage measures the weighted fraction of reference completion conditions judged to be implemented by the evaluator. Unsupported checks are the fraction of substantive evaluator checks not supported by the reference.
These results support the contribution of our construction design to preserving task requirements and evaluator completion conditions, beyond merely producing executable tasks.Following your suggestion, we conducted a human audit of 95 new DeepSeek-V4.1-Flash outcomes on Full ReTask tasks. Five annotators reviewed approximately equal portions of the outcomes, with a second annotator reviewing cases flagged as problematic. They examined evaluator decisions and supporting reasoning, consulting task instructions, final responses, and delivered artifacts as needed. The audited outcomes were not used to construct or repair the evaluators. 93 outcomes received high-confidence audit labels.
Two outcomes lacked high-confidence audit labels and are excluded. False discovery and false omission rates are calculated among evaluator-accepted and evaluator-rejected outcomes, respectively.
Human–evaluator agreement is 81.7% on the audited outcomes. Among 84 accepted outcomes, 16 were judged erroneous; among nine rejected outcomes, one was judged erroneous. The audit also identifies artifact evidence handling and aggregation of completion checks as areas for our future evaluator refinement.
Together, these evaluations provide complementary evidence about construction fidelity and grading correctness,addressing both the preservation of task requirements and the reliability of evaluator decisions.