- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The benchmark tests whether agents can improve training data for tool use, retrieval, and knowledge injection under fixed training pipelines.
Critical strengths
S1: Fixing the training pipeline makes the effect of data changes easier to assess.
S2: An important problem.
S3: The authors made an effort to do a meaningful analysis of the experimental results.
Critical weaknesses
W1: The distinction from Curation-Bench needs more discussion. That benchmark already evaluates iterative data curation with fixed models, training recipes, and evaluation. In the related work section, it was briefly mentioned, but AutoDataBench should explain which research questions its additional tasks enable that the earlier setting cannot address.
W2: The authors wanted to "explore whether iterative feedback helps LLMs better understand how changes to training data affect model performance", but the current experimental setup doesn't seem to control for feedback vs. solely relying on iterations (?). Does feedback help agents make better decisions, or does trying more iterations explain the gains? A no-feedback search baseline is needed.
W3: On the methods/evidence supporting another claim or question to investigate, I'm not quite convinced that how well the LLM predicts the overall score indicates how well the LLM understands the data intervention. For example, if the agent sees previous runs and scores (as mentioned in lines 420- 422), how much of the prediction comes from extrapolating previous observations vs. actual understanding of the intervention? More meaningful things to ask the agent could instead be: Can it predict the direction and size of the change? Can it rank the alternative interventions before trying them? etc.
Overall the benchmark is coherent, but the claims about understanding and learning from experimentation need much stronger evidence.
Questions for authors
Please address my questions in W1-3.
Response to Reviewer ppQX
Thank you for your careful reading and insightful questions. We appreciate the opportunity to clarify how our work differs from Curation-Bench and share additional results on feedback and forecasting.
W1. What questions does AutoDataBench add to Curation-Bench?
We cite Curation-Bench in §2 and regard it as closely related concurrent work. Both benchmarks fix the training pipeline and allow agents to revise data using evaluation feedback. Curation-Bench studies how agents explore curation strategies and how the scaffold and search budget affect that exploration. AutoDataBench asks how well the same agents handle three different decisions about training data. In tool use, the agent must recover valid supervision from corrupted examples. In retrieval, it must decide which examples to sample and which passages to pair as positives and negatives. In knowledge injection, it must select facts from source documents and write supervision that teaches them to the target model while preserving existing knowledge. Evaluating the same agents across these settings lets us examine where a successful data strategy transfers and where its gains are confined to the feedback distribution.
Two analyses extend this comparison. In §4.4, we record an agent’s prediction after it submits its data and before training begins, so we can compare what it expects from its intervention with what the model learns. In §4.5, we use the research trajectories themselves as mid-training data and test their value for downstream coding at a fixed token budget and with identical subsequent SFT. We will make these research questions explicit in Related Work.
W2. Does feedback add value beyond repeated attempts and checkpoint selection?
We added a no-feedback retrieval condition for GPT-5.6-Sol and Kimi-3. Both conditions used the same data pool, target model, training recipe, auxiliary models, and configured resource limits. In the no-feedback condition, agents retained their conversation history, code, processed data, and auxiliary quality checks. They could build on earlier attempts. We withheld target-model scores, training loss, and best-checkpoint information. After each run, the evaluator selected the best ID checkpoint using the same rule in both conditions and scored it on hidden OOD data.
Mean retrieval scores are reported as nDCG@10 × 100. Both agents scored higher with feedback on ID and OOD, with OOD differences of 1.96 and 1.20 points, respectively. Repeated attempts, reuse of earlier work, and retrospective selection were available in both conditions. The comparison therefore supports an additional contribution from performance feedback. Hardware and realized compute differed between the original and supplementary runs, which limits the precision of that estimate. We will report this control and interpret the original first-to-best gains as gains from the full iterative procedure.
W3. Can agents anticipate what their chosen intervention will do?
After committing a dataset and before training, the agent records a predicted score, an interval, and a brief rationale. We can then check whether it anticipated the direction and size of the change from the previous round. We compared its forecast with the previous score and a two-score linear trend on the same eligible rounds. The table reports MAE in points (0–100).
On tool use, both agents beat both historical baselines on MAE. Kimi-3 predicted the direction of change correctly on 80% of rounds, versus 50% for always predicting an increase. GPT-5.6-Sol matched that baseline. On retrieval, both agents matched the direction baseline, and the two-score trend gave lower MAE than either agent.
The tasks offer different clues. In tool use, the agent can check a repair against the request or tool schema, providing some evidence that the intervention may help. In retrieval, a harder negative may teach a useful distinction or penalize a relevant passage, and its effect depends on how training reorders competing passages across queries. This may make the effect of individual interventions difficult to anticipate even when score history predicts the overall performance level well. Overall, the clearest evidence of predictive information beyond score history is on tool use. We will revise §4.4 to reflect this task dependence.