acceptodds
Desk rejected at ICLR 2027

DataWorkLearn: The Workspace Is the Training Ground for Data Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Data agents, LLM agents that query, transform, and maintain data on a user's behalf, work inside workspaces of interdependent sources: databases and files, documentation, and layered pipelines whose definitions depend on one another, so correctness depends on how these pieces fit together. We introduce DataWorkLearn, which lets an agent learn a workspace before the first question arrives, from only the workspace and a short description of the expected work, with no labeled tasks, hand-written verifiers, or human in the loop. The agent explores the workspace, writes tasks together with executable solutions over its sources and pipelines, and verifies every solution by execution rather than by a model's judgment. Because every task is an instruction, a starting environment, and a verifier, we use the same tasks to adapt the agent through reinforcement learning on its weights, through learned skills, and through harness optimization, and compare the three on identical tasks. Across four data-analysis and data-engineering benchmarks, DataWorkLearn improves a 27B open-weight model by up to 22.5 points on the Data Agent Benchmark (DAB) and 22.4 on DABstep-hard, outperforming GEPA, EvoSkill, and Meta-Harness, each adapted on the same generated tasks, by up to 18.8 points. A data workspace can thus serve as verified, label-free training signal before any question is asked, not only as context at inference time.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Data agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_N1SEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DataWorkLearn, which allows data agents to generate practice tasks from a target workspace and adapt before receiving specific tasks. The method first generates executable solutions, obtains reference results through execution, and then constructs tasks and graders. These are used for training after reliability checks and difficulty filtering. The authors compare weight, skill, and harness adaptation on four data-analysis and data-engineering benchmarks, with LoRA and GRPO weight updates achieving the best overall results.

Critical strengths
  1. Generating executable practice tasks from the workspace itself has practical value. The method combines execution-based verification, mutant checks, and difficulty filtering into a reasonably complete training-data construction pipeline.
  2. Gains on complex analytical tasks are substantial. The 27B model improves by approximately 22.5 and 22.4 percentage points on DAB and DABstep-hard, respectively. Comparing different adaptation methods on the same generated tasks also provides useful empirical findings.
Critical weaknesses
  1. Generalization boundaries and overfitting risks remain unclear. Training and testing within the same workspace are consistent with the paper's setting and do not in themselves constitute data leakage. However, the experiments do not clearly distinguish transferable operational abilities from adaptation to fixed data and task templates. The appendices also disclose that some task families were informed by aggregate failure categories observed in benchmark runs. Ruling out overlapping questions is insufficient to rule out excessive adaptation to the test-task distribution.
  2. The evaluation of parameter adaptation is limited in scope. The experiments use only a fixed LoRA configuration, without a rationale for its selection, rank-sensitivity analysis, or comparison with full-parameter fine-tuning. The results therefore support the effectiveness of the current LoRA + GRPO configuration, but provide limited analysis of the relationship between performance, cost, and overfitting in parameter adaptation.
  3. Insufficient ablations of core modules. The paper compares adaptation methods and task sources, but does not directly establish the independent contributions of key mechanisms such as difficulty filtering and mutant checks. The staged DAB experiments lack a control that applies RL directly to the original harness, and therefore do not sufficiently establish complementarity between harness optimization and RL.
Questions for authors
  1. Could you evaluate on unseen task templates or operation combinations, updated data, and new workspaces to clarify the generalization boundaries? Do the gains persist without priors derived from benchmark failure categories?
  2. Why was the current LoRA configuration chosen? Could you compare performance and cost across different ranks and discuss the trade-offs relative to full-parameter fine-tuning?
  3. Could you ablate difficulty filtering and mutant checks while holding the number of tasks and training budget fixed, and report results for RL applied directly to the original harness?
1 comment

Thank you for your constructive review, and for noting the practical value of generating executable practice tasks from the workspace, our "reasonably complete training-data construction pipeline", and the substantial gains on complex analytical tasks. Our responses to each point are below.


Could you evaluate on unseen task templates or operation combinations, updated data, and new workspaces to clarify the generalization boundaries? Do the gains persist without priors derived from benchmark failure categories?

Thank you for the suggestions. We ran three additional evaluations, and RL keeps its gain in all of them: on unseen operation combinations, on paraphrased questions, and on generated tasks it never trained on.

  • Unseen operation combinations. Using the operation labels of Fig. 3a, we split the DAB questions by whether any generated task uses the same combination of operations (51 questions have labels). RL improves both groups by about 22 points.
  • Paraphrased questions. We rewrote all 450 official DABstep questions, keeping every number, date, identifier and name verbatim. The gain is unchanged, so it does not depend on the wording of the questions.
  • Unseen generated tasks. We evaluated on 424 verified generated DABstep tasks outside the training set, where RL improves from 5.0 to 41.5.
Test (27B)Before RLAfter RL
DAB, seen operation combination (43 questions)47.769.8
DAB, unseen operation combination (8 questions)43.865.6
DABstep-hard, original questions (Table 1)19.341.6
DABstep-hard, paraphrased questions20.741.6
DABstep, unseen generated tasks (424)5.041.5
  • Priors. The DAB and DABstep task-family catalogues were informed by aggregate failure classes of the agent on the benchmark; no official question, answer or per-task learning signal is given to the task author (App. B.5). The last two tests show that the gain is not tied to the wording of the official questions or to the tasks seen in training.

On updated data and new workspaces: our method targets specialization to the workspace the agent will work in. We will state this scope in the revision and discuss new workspaces and continual adaptation as future directions.


Why was the current LoRA configuration chosen? Could you compare performance and cost across different ranks and discuss the trade-offs relative to full-parameter fine-tuning?

We used one standard configuration without tuning it, and it works unchanged on all four benchmarks and both models.

  • Why this configuration. We use rank 32 and α=64 on all linear layers, a standard GRPO recipe, everywhere. Since it was not tuned on any benchmark, the gains in Table 1 do not come from picking a favorable rank.
  • Trade-off against full fine-tuning. We chose LoRA for cost: the adapter is 0.47 GB and many adapters share one served base model, whereas full fine-tuning stores and serves a separate copy of the model for every workspace.
  • Overfitting. RL carries 57% of its gain on generated tasks over to the official DAB questions, whereas the gains of prompt and harness search do not transfer (Fig. 2b).

Following your suggestion, we will add a rank sweep (r = 8, 32, 128), a full fine-tuning comparison with Pass@1 and GPU-hours, and training curves to App. C.


Could you ablate difficulty filtering and mutant checks while holding the number of tasks and training budget fixed, and report results for RL applied directly to the original harness?

Thank you for the suggestion. Below is what each component does and the evidence for it; on DAB, the gain comes from RL rather than the harness stage.

  • Difficulty filtering. GRPO learns only from tasks that the model sometimes solves and sometimes fails. On DAB, 337 of the 917 generated tasks are of this kind, and the filter keeps them (App. B.1). Tasks that the model already solves give little signal: the baseline generators' tasks start at a training reward of 0.7–0.9 and improve the 9B agent by at most +6.4 points, versus +13.4 for ours (§5.3).
  • Mutant checks. A task is kept only if its verifier rejects wrong answers (perturbed values, swapped fields, dropped rows). Otherwise the verifier would reward wrong trajectories.
  • RL and the harness. Harness learning alone moves DAB from 45.6 to 48.4, and RL then adds 19.7 points (68.06). Of the 22 questions whose pass rate rises by more than 0.25, 16 improve only under RL (§5.4).

Due to the time constraints of the rebuttal period, we will add the fixed-budget ablations of both components and RL on the original harness to §5.4 and App. B in the final version.

Reviewer_8o1LOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces DataWorkLearn, which generates executable tasks from a target data workspace and uses them to support model, skill and harness adaption. Experiments with 9B and 27B agents across four benchmarks yield substantial gains using workload-informed priors without providing downstream task instances.

Critical strengths
  1. The same practice tasks can be used to train the model, teach it reusable skills, or improve its prompts and workflow. This makes the different approaches easier to compare.
  2. The method improves performance across four benchmarks and both model sizes. For the 27B model, the gains reach about 22.5 percentage points on DAB and 22.4 on DABstep-hard.
Critical weaknesses
  1. Re-running a solution does not show whether the model-generated labels in the 31 tasks are correct. The paper also needs to clarify which data were used to choose checkpoints and what was changed in the benchmark answers and scoring code.
  2. The comparisons use different agent setups, model-serving settings, and amounts of training. This makes it hard to tell how much of the improvement comes from updating the model or from the way the practice tasks are generated.
Questions for authors
  1. Which data determine checkpoint selection, and which fixed task, answer-key and scorer versions produce the Main Results?
  2. Do the interface and generator conclusions hold when harness, serving, GRPO grouping and total training budgets are controlled?
  3. Any checks that validate semantic labels and instruction-program alignment beyond replay and mutant tests?
3 comments

Thank you for your detailed review, and for noting that the same practice tasks can train the model, teach it reusable skills, or improve its prompts and workflow, which "makes the different approaches easier to compare", and that the method improves all four benchmarks at both model sizes. Our responses to each point are below.


Re-running a solution does not show whether the model-generated labels in the 31 tasks are correct.

We ran a blind audit of all 31 tasks: the labels agree with independent sources, and they affect 2 of the 521 tasks behind the 27B result.

  • For all other tasks, labels are not produced by a model: each label is the output of executing the authored program (a typed graph of deterministic operators, or a project edit) in the workspace.
  • Only 26 of the 31 tasks carry model-produced labels (news topic classification). The other 5 apply deterministic rules (file extension, phrase match, exchange mapping); re-deriving them from the raw databases reproduces every answer.
  • Blind relabeling: four models (two Claude, two GPT) relabeled all 1,911 classified articles without seeing the questions, answers or original labels, and we add each article's category in the public AG News dataset as a fifth, non-LLM source. DAB's own official answers are computed from these categories. Agreement is high (Fleiss κ = 0.90).
  • Effect on the reward: re-executing every task with the consensus labels changes few answers, each from a tie or a one-article gap.
Training setTasksAnswers changed
27B5212 (0.4%)
9B3978 (2.0%)

We will release the corrected answers and add the audit to App. B.1.


Any checks that validate semantic labels and instruction-program alignment beyond replay and mutant tests?

Yes: the labels are audited as above, and the question text is checked against its program.

  • The reference answer is fixed by executing the program, and the question is then written from the program. A checklist stage verifies that the question asks for the quantity the program computes, states comparisons in the program's direction, asks for a single target, and states every entity, period and filter that scopes the computation.
  • On DABstep this stage removed 134 of 921 candidates (App. B). We applied the same four checks to the 31 DAB tasks: all 31 pass, and no answer changes.
  • Other existing checks (App. B): rejection of tasks whose answer a shallower program also produces; on ADE-Bench, an independently written equivalent solution plus a separate model review.

We will add these checks on the DAB tasks to App. B.


Which data determine checkpoint selection, and which fixed task, answer-key and scorer versions produce the Main Results?

Checkpoints are selected from generated tasks alone. Every change to the benchmarks is listed below.

  • Checkpoint selection. The selection rule uses generated tasks only: we take the first checkpoint whose reward on the generated tasks (5-step mean) is not improved by more than 2 points in the following 15 steps. We will report every RL row and add the reward curves to App. C.
  • DAB (54 tasks) and ADE-Bench (64 tasks): official tasks, answers and scorers, unchanged.
  • DABstep: gold answers are only available through the leaderboard, so we score with an answer key reconstructed from publicly graded submissions, which agrees with the official verdicts on 99.96% of held-out submissions. The 26 tasks whose answer cannot be recovered are excluded (424 of 450 remain: 353 hard, 71 easy).
  • DAComp-DE-Verified: several released gold projects do not score 100 under the original scorer. We fixed reproducibility defects in the scorer and harness (scoring now runs in an isolated process so that the official evaluator's timeouts take effect, score parsing no longer breaks on database progress output, and every trial has a memory and wall-clock limit) and kept the 43 tasks whose gold project scores reproducibly.

We will state this in App. A and release the task lists, the answer key, and the scorer and harness diffs.


Do the interface and generator conclusions hold when harness, serving, GRPO grouping and total training budgets are controlled?

Yes. Both comparisons use the same model, harness and RL recipe, and are compared at matched budgets.

  • Harness. Every method in §5.2 and every task source in §5.3 starts from the same harness.
  • Serving. All methods use the same base model. With the same harness, it scores 48.4 through OpenRouter (used by GEPA, EvoSkill and Meta-Harness) and 46.8 on our serving stack, so serving does not explain the gap.
  • GRPO grouping. Every task source is trained with the same RL recipe and 256 rollouts per step.
  • Budget. Fig. 2a compares the interfaces at equal inference tokens: at 1.4B tokens, the full budget of Meta-Harness, RL reaches 54.6, while GEPA and Meta-Harness stay between 40 and 50 even at two to three times that budget. In §5.3, every source provides 308 tasks to the same 9B model.

We will state these controls in §5.2 and §5.3.

Reviewer_8o1LOpenReview ↗

reviewer response

Thank you for the clarification. I will raise the my soundness score.

Reviewer_rDLkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DataWorkLearn, a framework for adapting data agents to a new workspace before downstream tasks are known. It generates executable practice tasks by having an author agent write and run solutions, then derives instruction-environment-verifier triples for adaptation. Across four data-agent benchmarks, RL-based weight adaptation improves two Qwen models and generally outperforms skill and harness adaptation methods trained on the same generated tasks.

Critical strengths
  1. The paper addresses a practically relevant problem: adapting data agents to a new workspace before downstream user tasks are available. The execution-grounded task-generation framework provides a clear way to obtain verified training signals without relying on manually labeled tasks or LLM-based judges.
  2. The paper presents an empirical evaluation across four data-agent benchmarks. Comparing RL, skill, and harness adaptation using the same generated-task interface is particularly useful, and the consistent gains of RL adaptation on the harder settings support the practical value of the proposed approach.
Critical weaknesses

The task-family design is informed by failure categories observed across benchmark runs, and DAComp-DE-Verified additionally uses aggregate complexity targets. Although the authors do not train directly on official test instances, these priors may reveal the likely target task distribution and enable targeted data construction. Thus, the setting appears closer to synthesizing data for a partially specified benchmark workload, which limits the novelty and value.

Questions for authors

n/a

1 comment

Thank you for your positive review, and for recognizing that our work addresses "a practically relevant problem", that the execution-grounded framework "provides a clear way to obtain verified training signals without relying on manually labeled tasks or LLM-based judges", and that comparing RL, skill and harness adaptation on the same generated tasks is "particularly useful".

The task-family design is informed by failure categories observed across benchmark runs, and DAComp-DE-Verified additionally uses aggregate complexity targets. [...] Thus, the setting appears closer to synthesizing data for a partially specified benchmark workload, which limits the novelty and value.

The setting is a partially specified workload by design, and the contribution is how verified supervision is obtained from the workspace.

  • Setting. The agent knows the kind of work it will do, but none of the tasks (App. B.5). This matches deployment: a team knows its agent will, e.g., answer cross-database analytical questions over its warehouses, but has no labeled tasks, reference solutions or graders for future requests. Our question is whether the workspace can supply verified training signal in this setting.
  • What the priors contain. A short description of the scenario and a catalogue of task families. The DAB and DABstep catalogs were informed by aggregate failure classes of the agent on the benchmark, and DAComp-DE-Verified uses aggregate complexity targets. No official question, answer or per-task learning signal is given to the task author.
  • Novelty. The catalogue guides which kinds of tasks are generated; the supervision itself comes from the workspace: every label is the result of executing an authored program. With the same environments, harness and number of training tasks, existing generators (DataMind-12K, OmniSQL, AWM) improve the 9B agent by at most +6.4 points on DAB, versus +13.4 for ours (§5.3).
  • The generated tasks differ from the benchmark's questions. 67% of our tasks are cross-database versus 31% of DAB questions (Fig. 3a), and RL also solves questions that no generated task asks, such as PATENTS/2 (§5.4).
  • New tests. RL keeps its gain when all official DABstep questions are paraphrased (every number, date, identifier and name kept verbatim), and it also solves generated tasks that were never used in training.
DABstep, 27BBaseRL
Official hard questions (Table 1)19.341.6
Official hard questions, paraphrased20.741.6
Generated tasks never used in training (424)5.041.5

We will move the description of the priors from App. B.5 into the problem setting in the main text, and add both tests to the revision.

Comments on the paper

1 comment

We thank the AC and all reviewers for their time and constructive feedback. We are encouraged that the reviewers find the problem relevant (rDLk), the pipeline "reasonably complete" (N1SE), the comparison of RL, skill and harness adaptation on the same generated tasks useful (rDLk, 8o1L, N1SE), and the gains on complex analytical tasks substantial (8o1L, N1SE).

We have responded to each reviewer individually. Below is a summary of the main points.

1. New evaluations of generalization (rDLk, N1SE). RL keeps its gain on DAB questions whose operation combination appears in no generated task, on paraphrased DABstep questions, and on generated tasks never used in training.

Test (27B)Before RLAfter RL
DAB, seen operation combination (43 questions)47.769.8
DAB, unseen operation combination (8 questions)43.865.6
DABstep-hard, original questions (Table 1)19.341.6
DABstep-hard, paraphrased questions20.741.6
DABstep, unseen generated tasks (424)5.041.5

2. Label audit (8o1L). We ran a blind audit of the 31 tasks with semantic labels. Four models and the public AG News categories agree with the labels (Fleiss κ = 0.90), and re-executing every task with the consensus labels changes 2 of the 521 answers behind the 27B result.

3. Checkpoint selection and benchmark versions (8o1L). The selection rule uses generated tasks only; on DABstep-hard the selected checkpoint scores 40.5, 21 points above the base agent. We also list the exact task, answer-key and scorer versions of all four benchmarks.

4. Controls in the comparisons (8o1L). Both the interface and the generator comparison use the same model, harness and RL recipe, and are compared at matched budgets.

5. Setting and priors (rDLk, N1SE). The priors are a short scenario description and a catalogue of task families; no official question, answer or per-task learning signal is given to the task author (App. B.5). We will move this description into the problem setting in the main text.

Planned additions to the final version. The new evaluations and the audit above; reward curves and selected checkpoints (App. C); a LoRA rank sweep and a full fine-tuning comparison (App. C); fixed-budget ablations of difficulty filtering and mutant checks, and RL on the original harness (§5.4, App. B). We will also release the task lists, the DABstep answer key, and the scorer and harness diffs.

We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.