acceptodds
Under review as a conference paper at ICLR 2027

AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?

Abstract

Recent gains in language model capability have come more from data than from architecture. Frontier labs and data companies produce verifiable agentic tasks, which supervised finetuning and reinforcement learning then turn into capability. This production line still rests on human labour and on human-in-the-loop collaboration. Agentic tasks are also the hardest kind of data to produce, because each one needs an executable environment, a reliable verifier, and a difficulty matched to the model being trained. Automating task creation would let data production scale with compute rather than with expert headcount, would extend to more domains, and would enable a key step in recursive self-improvement (RSI). Current evaluations of an agent's ability to write such tasks measure how a model performs after training on what the agent produced. That does not match common practice in the data industry, where a delivery is accepted or rejected one sample at a time, before any training. No existing evaluation asks whether an individual task meets the acceptance criteria of a data pipeline. We therefore introduce AutoDataBench. Given an original benchmark task and a record of the target model attempting it, an agent must write a new task that follows the same benchmark conventions and meets practical acceptance standards on validity, novelty, difficulty and behavioural coverage. We make the job as easy as we can. The agent is given the conventions, a starting task and the behaviour to target, and it needs only to write a task of similar structure that exercises the same behavioural modes. Across three benchmarks of executable agent tasks, no agent we evaluate scores above 20 out of 100 at the default time budget of 45 minutes. Giving the strongest agent four times as long improves its score substantially, while the cost of one usable task stays almost unchanged. Current agents can write training tasks of the required quality, but not efficiently. \sys provides a direct measure of an agent's capacity for autonomous data synthesis: one artifact at a time, judged against the criteria a production pipeline would apply, and without a training run.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agents, llm, data, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R3tpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AutoDataBench, a benchmark for evaluating agents that generate new tasks for target models based on existing datasets. The system contains an author agent, which receives an original task and the target model’s attempts, then produces a new task. The benchmark then evaluates the generated tasks through defect checks, difficulty measures, and coverage of hidden behavioural rubrics. The authors conduct experiments with five agents based on different language models on 24 original tasks selected from Terminal-Bench 4.0, Terminal-Bench-Science, and AutomationBench. The target model they choose is DeepSeek-V4-Pro. The results show that under a 45-minute budget, scores range from 0.097 to 0.184. For Kimi-K3, when the budget increases to 180 minutes, the score rises to 0.541. The authors find that difficulty calibration is the main bottleneck in producing tasks that meet their requirements.

Critical strengths

The authors carry out a substantial number of experiments, evaluating five agents across three task suites with 240 authoring episodes and additional experiments with a larger budget. The authors provide some insight into agents’ difficulties in producing usable tasks, specifically in meeting the difficulty requirement.

Critical weaknesses
  1. This benchmark measures whether agents can generate new tasks that meet the authors’ acceptance criteria, but it does not establish whether meeting these criteria leads to better training results. Even tasks that pass novelty checks and meet the difficulty requirement may still provide little additional learning value. This is also acknowledged in the conclusion. But as the central goal of this benchmark is to test whether agents can generate new data for training, this limitation cannot be overlooked.

  2. The study provides very limited evidence that the reported scores and domain-specific rankings generalize to other tasks or target models. In this paper, the evaluation uses only 24 manually selected original tasks and only one target model, DeepSeek-V4-Pro. Since difficulty is measured against DeepSeek-V4-Pro, the low scores do not imply that agents would also perform poorly when generating tasks for models with different capabilities. A broader task sample and more target models, especially smaller models, would help support the general conclusions.

Questions for authors
  1. Could the authors clarify how the pass-rate interval and the 60% coverage target were selected?

  2. Could the authors provide evidence on the reliability of the LLM-generated rubrics and judge decisions? For example, do they agree with human assessments? This is important because the benchmark aims to evaluate whether agents can perform the task-authoring work of a human data team. Can the judge reliably distinguish defective tasks, such as tasks that cannot be completed, from valid ones?

Reviewer_b2h1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces AutoDataBench, which tests whether agents can create executable training tasks from an existing task and a target model’s attempts. Following the industry practice described in the paper, each sample is graded individually before training, based on validity, novelty, difficulty, and behavioral coverage. Five agents are evaluated on 24 tasks across three suites. Scores are low at the default budget, but improve substantially when both time and reasoning effort increase.

Critical strengths
  • The paper identifies a useful evaluation target. Creating a task requires producing instructions, an executable environment, and a reliable verifier. Evaluating each delivered sample makes these requirements explicit and helps separate data-production failures from issues in the subsequent training process.

  • The scoring framework captures a meaningful tension in task creation. A generated task must preserve relevant behavioral challenges while introducing something new and remaining solvable by the target model. Requiring evidence from actual solver trajectories gives this objective a concrete basis.

Critical weaknesses
  • The paper does not establish that its score measures useful, weakness-targeted training data. A task can score well without targeting the weakness that makes the model fail. The rubric rewards bringing the model to particular decision points, and covering three of five modes is enough for full quality credit. For example, a generated task could satisfy those criteria, then introduce an unrelated difficult step to achieve the required pass rate. It could therefore satisfy both behavioral coverage and difficulty without providing anything useful to address the original weakness. The paper does not test whether its scoring rules distinguish this case from genuinely targeted tasks.

  • The experimental evidence is too limited to support the main conclusions confidently. Difficulty is accepted or rejected based on only six attempts, without establishing the stability of those decisions.

  • The longer-budget experiment increases both available time and reasoning effort. It demonstrates that this combined change improves performance, but does not isolate additional calibration time as the explanation. Moreover, the dollar-cost analysis excludes also calibration calls, evaluation rollouts, and judging, which in my view limit the conclusion about efficiency.

  • The premise is worthwhile, but the current experiments leave the central scoring assumptions insufficiently validated. I would say the paper below the standard I would expect for acceptance at ICLR.

Questions for authors
  1. Does providing the target model’s attempts improve results over providing only the original task and suite conventions?
  2. What independent evidence supports the acceptance judgments and their relationship to training value?
  3. How stable are acceptance decisions and author rankings when delivered tasks receive more evaluation attempts?
  4. Does the budget improvement persist when reasoning effort is held fixed, and what is the cost per accepted task when calibration and evaluation costs are included?
Reviewer_auVvOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AutoDataBench tests whether an agent can do the authoring job of a data team: given one original task from a public suite and sanitised transcripts of the target model (deepseek-v4-pro) attempting it, the agent writes one new executable task, scored before any training as gate times difficulty times quality. Across 24 original tasks from Terminal-Bench 4.0, Terminal-Bench-Science, and AutomationBench, five author agents score 0.097 to 0.184 out of 1.0 at a 45 minute budget, with difficulty calibration, not mode coverage, as the binding constraint; giving the best agent 180 minutes raises its score to 0.541 at a roughly unchanged cost per usable task, though whether passing this acceptance test means a task is worth training on is left untested.

Critical strengths
  • The unit of evaluation is the right one, and it is new. Scoring one delivered task, before training, against explicit acceptance criteria matches how data is actually commissioned and paid for, and Table 1 positions this precisely against prior benchmarks. Difficulty is also measured by execution, not by a judge: the target model attempts the delivered task K = 6 times under the task's own verifier (Section 2.3), so the term that dominates the score is grounded in behaviour.
  • The gate and the judge are both constrained to evidence. The gate blocks the obvious shortcut (the surface swap defect, Appendix C, backed by a mechanical diff in Appendix D) and defects must be evidenced against a file and line before they fire; for a mode to count as present, the judge must cite an attempt, a step, and a verbatim quote from the transcripts, and unreachable modes are removed from the score.
  • The analysis is candid, and the economics ask the production question. Finding 3 reports that agent rankings invert across suites instead of hiding it, and Appendix F discloses that the 180 minute arm also changed reasoning effort. Table 3 prices one usable delivery ($4.81 to $27.69 at the default budget) rather than one episode, and shows that score and cost orderings disagree.
Critical weaknesses
  • My main concern: the acceptance criteria are never validated as a proxy for training value. No experiment, not even a small one, trains on accepted versus rejected deliveries to check that acceptance predicts improvement, and the authors concede in Section 6 that this "remains untested". Until some check like that exists, the benchmark measures conformance to acceptance criteria that are plausible but unvalidated.
  • The analyst and the judge are a single unvalidated model. Both roles are claude-opus-5 (Section 2.2): the rubric of 125 modes is one model's reduction of the transcripts, and mode presence is one model's judgement, with no human agreement study for either. Coverage is also thin for the conclusions drawn: 24 hand picked tasks (Appendix A), one target model, 16 episodes per agent per suite, and no uncertainty estimate on the Table 2 ranking; and with K = 6, difficulty classification near the band edges is sampling noise as much as design.
  • The cost and time numbers are not clean. Table 3 counts only the author agent's own tokens, excluding the calibration calls that Section 3.3 shows buying the score improvement, the K official attempts, and the judge; the glm-5.3 figure is a $168.20 estimate replacing an actual spend of $746.55. The headline 2.9x time scaling result is also confounded, since the 180 minute arm ran at maximum reasoning effort while the 45 minute arms used default effort (Appendix F).
Questions for authors
  1. Can you provide any evidence, even at small scale, that acceptance predicts training value? For example, a short SFT or RL run for one suite comparing target model improvement from accepted deliveries against rejected ones.
  2. What is the human agreement with the analyst's rubrics and the judge's mode coverage verdicts on a sample of episodes? Do the Table 2 rankings change if a different model family serves as analyst and judge?
  3. If the excluded costs are counted (calibration calls, the K official attempts, and the judge), what is the cost per usable task for each agent, and does the ordering in Table 3 survive?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.