- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DataWorkLearn, which allows data agents to generate practice tasks from a target workspace and adapt before receiving specific tasks. The method first generates executable solutions, obtains reference results through execution, and then constructs tasks and graders. These are used for training after reliability checks and difficulty filtering. The authors compare weight, skill, and harness adaptation on four data-analysis and data-engineering benchmarks, with LoRA and GRPO weight updates achieving the best overall results.
Critical strengths
- Generating executable practice tasks from the workspace itself has practical value. The method combines execution-based verification, mutant checks, and difficulty filtering into a reasonably complete training-data construction pipeline.
- Gains on complex analytical tasks are substantial. The 27B model improves by approximately 22.5 and 22.4 percentage points on DAB and DABstep-hard, respectively. Comparing different adaptation methods on the same generated tasks also provides useful empirical findings.
Critical weaknesses
- Generalization boundaries and overfitting risks remain unclear. Training and testing within the same workspace are consistent with the paper's setting and do not in themselves constitute data leakage. However, the experiments do not clearly distinguish transferable operational abilities from adaptation to fixed data and task templates. The appendices also disclose that some task families were informed by aggregate failure categories observed in benchmark runs. Ruling out overlapping questions is insufficient to rule out excessive adaptation to the test-task distribution.
- The evaluation of parameter adaptation is limited in scope. The experiments use only a fixed LoRA configuration, without a rationale for its selection, rank-sensitivity analysis, or comparison with full-parameter fine-tuning. The results therefore support the effectiveness of the current LoRA + GRPO configuration, but provide limited analysis of the relationship between performance, cost, and overfitting in parameter adaptation.
- Insufficient ablations of core modules. The paper compares adaptation methods and task sources, but does not directly establish the independent contributions of key mechanisms such as difficulty filtering and mutant checks. The staged DAB experiments lack a control that applies RL directly to the original harness, and therefore do not sufficiently establish complementarity between harness optimization and RL.
Questions for authors
- Could you evaluate on unseen task templates or operation combinations, updated data, and new workspaces to clarify the generalization boundaries? Do the gains persist without priors derived from benchmark failure categories?
- Why was the current LoRA configuration chosen? Could you compare performance and cost across different ranks and discuss the trade-offs relative to full-parameter fine-tuning?
- Could you ablate difficulty filtering and mutant checks while holding the number of tasks and training budget fixed, and report results for RL applied directly to the original harness?
Thank you for your constructive review, and for noting the practical value of generating executable practice tasks from the workspace, our "reasonably complete training-data construction pipeline", and the substantial gains on complex analytical tasks. Our responses to each point are below.
Thank you for the suggestions. We ran three additional evaluations, and RL keeps its gain in all of them: on unseen operation combinations, on paraphrased questions, and on generated tasks it never trained on.
On updated data and new workspaces: our method targets specialization to the workspace the agent will work in. We will state this scope in the revision and discuss new workspaces and continual adaptation as future directions.
We used one standard configuration without tuning it, and it works unchanged on all four benchmarks and both models.
Following your suggestion, we will add a rank sweep (r = 8, 32, 128), a full fine-tuning comparison with Pass@1 and GPU-hours, and training curves to App. C.
Thank you for the suggestion. Below is what each component does and the evidence for it; on DAB, the gain comes from RL rather than the harness stage.
Due to the time constraints of the rebuttal period, we will add the fixed-budget ablations of both components and RL on the original harness to §5.4 and App. B in the final version.