- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AutoDataBench, a benchmark for evaluating agents that generate new tasks for target models based on existing datasets. The system contains an author agent, which receives an original task and the target model’s attempts, then produces a new task. The benchmark then evaluates the generated tasks through defect checks, difficulty measures, and coverage of hidden behavioural rubrics. The authors conduct experiments with five agents based on different language models on 24 original tasks selected from Terminal-Bench 4.0, Terminal-Bench-Science, and AutomationBench. The target model they choose is DeepSeek-V4-Pro. The results show that under a 45-minute budget, scores range from 0.097 to 0.184. For Kimi-K3, when the budget increases to 180 minutes, the score rises to 0.541. The authors find that difficulty calibration is the main bottleneck in producing tasks that meet their requirements.
Critical strengths
The authors carry out a substantial number of experiments, evaluating five agents across three task suites with 240 authoring episodes and additional experiments with a larger budget. The authors provide some insight into agents’ difficulties in producing usable tasks, specifically in meeting the difficulty requirement.
Critical weaknesses
-
This benchmark measures whether agents can generate new tasks that meet the authors’ acceptance criteria, but it does not establish whether meeting these criteria leads to better training results. Even tasks that pass novelty checks and meet the difficulty requirement may still provide little additional learning value. This is also acknowledged in the conclusion. But as the central goal of this benchmark is to test whether agents can generate new data for training, this limitation cannot be overlooked.
-
The study provides very limited evidence that the reported scores and domain-specific rankings generalize to other tasks or target models. In this paper, the evaluation uses only 24 manually selected original tasks and only one target model, DeepSeek-V4-Pro. Since difficulty is measured against DeepSeek-V4-Pro, the low scores do not imply that agents would also perform poorly when generating tasks for models with different capabilities. A broader task sample and more target models, especially smaller models, would help support the general conclusions.
Questions for authors
-
Could the authors clarify how the pass-rate interval and the 60% coverage target were selected?
-
Could the authors provide evidence on the reliability of the LLM-generated rubrics and judge decisions? For example, do they agree with human assessments? This is important because the benchmark aims to evaluate whether agents can perform the task-authoring work of a human data team. Can the judge reliably distinguish defective tasks, such as tasks that cannot be completed, from valid ones?