acceptodds
Under review as a conference paper at ICLR 2027

AutoDataBench: A Data-centric Testbed for Accelerating Auto Research

Abstract

Existing auto-research benchmarks often entangle multiple sources of improvement, including training frameworks, hyperparameters, compute budgets, and data, making it difficult to attribute why one frontier agent outperforms another to specific research capabilities. In this work, we isolate and systematically evaluate Data Intelligence: an agent's ability to understand, manipulate, and improve the data that shapes model capabilities. We introduce AutoDataBench, a controlled testbed built on a conceptual framework of data intelligence spanning data diagnosis, data organization, and data construction, instantiated through three highly curated optimization tasks while holding non-data factors fixed. Across tool use, retrieval, and knowledge injection, we evaluate frontier LLMs' ability to improve training data through iterative experimentation under task-specific resource budgets. Beyond optimization performance, we ask: do LLMs understand what their data interventions do? We compare predictions made before training with observed outcomes to seek evidence of data-effect reasoning beyond trial and error, and explore whether iterative feedback helps LLMs better understand how changes to training data affect model performance. Finally, we show that reusing AutoDataBench trajectories for mid-training improves downstream coding performance, highlighting its value in both evaluating data intelligence and generating high-quality training data.

Primary area:
datasets and benchmarks
Keywords:
Auto Research, Agent Benchmark, Data Optimization, Data-centric

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ppQXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The benchmark tests whether agents can improve training data for tool use, retrieval, and knowledge injection under fixed training pipelines.

Critical strengths

S1: Fixing the training pipeline makes the effect of data changes easier to assess.

S2: An important problem.

S3: The authors made an effort to do a meaningful analysis of the experimental results.

Critical weaknesses

W1: The distinction from Curation-Bench needs more discussion. That benchmark already evaluates iterative data curation with fixed models, training recipes, and evaluation. In the related work section, it was briefly mentioned, but AutoDataBench should explain which research questions its additional tasks enable that the earlier setting cannot address.

W2: The authors wanted to "explore whether iterative feedback helps LLMs better understand how changes to training data affect model performance", but the current experimental setup doesn't seem to control for feedback vs. solely relying on iterations (?). Does feedback help agents make better decisions, or does trying more iterations explain the gains? A no-feedback search baseline is needed.

W3: On the methods/evidence supporting another claim or question to investigate, I'm not quite convinced that how well the LLM predicts the overall score indicates how well the LLM understands the data intervention. For example, if the agent sees previous runs and scores (as mentioned in lines 420- 422), how much of the prediction comes from extrapolating previous observations vs. actual understanding of the intervention? More meaningful things to ask the agent could instead be: Can it predict the direction and size of the change? Can it rank the alternative interventions before trying them? etc.

Overall the benchmark is coherent, but the claims about understanding and learning from experimentation need much stronger evidence.

Questions for authors

Please address my questions in W1-3.

1 comment
AuthorsOpenReview ↗

Response to Reviewer ppQX

Thank you for your careful reading and insightful questions. We appreciate the opportunity to clarify how our work differs from Curation-Bench and share additional results on feedback and forecasting.

W1. What questions does AutoDataBench add to Curation-Bench?

We cite Curation-Bench in §2 and regard it as closely related concurrent work. Both benchmarks fix the training pipeline and allow agents to revise data using evaluation feedback. Curation-Bench studies how agents explore curation strategies and how the scaffold and search budget affect that exploration. AutoDataBench asks how well the same agents handle three different decisions about training data. In tool use, the agent must recover valid supervision from corrupted examples. In retrieval, it must decide which examples to sample and which passages to pair as positives and negatives. In knowledge injection, it must select facts from source documents and write supervision that teaches them to the target model while preserving existing knowledge. Evaluating the same agents across these settings lets us examine where a successful data strategy transfers and where its gains are confined to the feedback distribution.

Two analyses extend this comparison. In §4.4, we record an agent’s prediction after it submits its data and before training begins, so we can compare what it expects from its intervention with what the model learns. In §4.5, we use the research trajectories themselves as mid-training data and test their value for downstream coding at a fixed token budget and with identical subsequent SFT. We will make these research questions explicit in Related Work.

W2. Does feedback add value beyond repeated attempts and checkpoint selection?

We added a no-feedback retrieval condition for GPT-5.6-Sol and Kimi-3. Both conditions used the same data pool, target model, training recipe, auxiliary models, and configured resource limits. In the no-feedback condition, agents retained their conversation history, code, processed data, and auxiliary quality checks. They could build on earlier attempts. We withheld target-model scores, training loss, and best-checkpoint information. After each run, the evaluator selected the best ID checkpoint using the same rule in both conditions and scored it on hidden OOD data.

ModelID with feedbackID without feedbackOOD with feedbackOOD without feedback
GPT-5.6-Sol40.4640.0426.1524.19
Kimi-340.0738.0029.8528.65

Mean retrieval scores are reported as nDCG@10 × 100. Both agents scored higher with feedback on ID and OOD, with OOD differences of 1.96 and 1.20 points, respectively. Repeated attempts, reuse of earlier work, and retrospective selection were available in both conditions. The comparison therefore supports an additional contribution from performance feedback. Hardware and realized compute differed between the original and supplementary runs, which limits the precision of that estimate. We will report this control and interpret the original first-to-best gains as gains from the full iterative procedure.

W3. Can agents anticipate what their chosen intervention will do?

After committing a dataset and before training, the agent records a predicted score, an interval, and a brief rationale. We can then check whether it anticipated the direction and size of the change from the previous round. We compared its forecast with the previous score and a two-score linear trend on the same eligible rounds. The table reports MAE in points (0–100).

TaskModelAgent forecastPrevious scoreTwo-score trend
Tool useGPT-5.6-Sol1.702.093.77
Tool useKimi-32.733.566.69
RetrievalGPT-5.6-Sol0.640.610.58
RetrievalKimi-31.551.000.80

On tool use, both agents beat both historical baselines on MAE. Kimi-3 predicted the direction of change correctly on 80% of rounds, versus 50% for always predicting an increase. GPT-5.6-Sol matched that baseline. On retrieval, both agents matched the direction baseline, and the two-score trend gave lower MAE than either agent.

The tasks offer different clues. In tool use, the agent can check a repair against the request or tool schema, providing some evidence that the intervention may help. In retrieval, a harder negative may teach a useful distinction or penalize a relevant passage, and its effect depends on how training reorders competing passages across queries. This may make the effect of individual interventions difficult to anticipate even when score history predicts the overall performance level well. Overall, the clearest evidence of predictive information beyond score history is on tool use. We will revise §4.4 to reflect this task dependence.

Reviewer_UcR9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AutoDataBench is a controlled testbed for evaluating LLM agents' ability to improve training data while keeping target models and training recipes fixed. It instantiates data diagnosis and repair, organization, and construction through tool-use data curation, retrieval training, and knowledge injection. Agents inspect data, implement interventions, train models, and use evaluation feedback to revise their strategies under resource limits. Seven models are evaluated with three standard runs per task, including hidden assessments of transfer or knowledge retention. The paper additionally probes forecasts of intervention outcomes recorded before training and tests whether collected research trajectories improve downstream coding performance when incorporated into a matched-token mid-training mixture.

Critical strengths
  1. A useful controlled evaluation setting. Fixing the training pipelines while allowing actual feedback-driven data revisions focuses attention on an important practical capability. The three tasks span different data operations and learning paradigms, providing a substantive starting point for evaluating data-research agents.

  2. The evaluation exposes limitations beyond target scores. Hidden assessments reveal that retrieval ID gains do not reliably transfer: all agent mean OOD scores fall below the simple baseline. Reporting repeated standard runs and these negative transfer results is informative. The matched-token trajectory-training experiment also provides an encouraging, though preliminary, demonstration of reuse.

Critical weaknesses
  1. The evaluation does not sufficiently separate adaptive research from repeated search and standard data processing. The first-to-best gains in Section 4.3 and Appendix A.4 combine feedback use, multiple attempts, checkpoint selection, and training variability, as the authors acknowledge. Missing equal-budget no-feedback search and fixed curation/generation baselines make it difficult to determine how much the research loop itself contributes. Reference comparisons do not resolve this: the tool-use expert uses pre-corruption data, whereas the knowledge expert uses rule-based mining without LLM-assisted construction. These are useful references, but do not establish superiority over equally equipped data-processing workflows. This limits the interpretation of the benchmark scores as evidence of adaptive data-research competence.

  2. The additional evidence for understanding data effects and for distinctive trajectory value is preliminary. Section 4.4 uses only one prediction-enabled run per model-task pair for two models and lacks a last-score or trend baseline. Accurate absolute-score forecasts can result from stable scores rather than understanding the current intervention; improvement-direction prediction and interval calibration would be more diagnostic. Section 4.5 controls token budget and subsequent SFT, but reports one matched training pair without repeated-seed uncertainty or an equal-token alternative code/agent-trajectory control. The results support feasibility, but not yet a robust or distinctive benefit of the data-research structure. I recognize that these contributions are framed as explorations; the missing controls nevertheless limit the overall evidential strength.

Questions for authors
  1. Under the same resource budget, how does feedback-driven optimization compare with independent no-feedback proposals and a fixed curation/generation pipeline using the same auxiliary models? Can repeated training on unchanged data quantify the variability component?

  2. Do the forecasts outperform predicting the previous score or a simple historical trend? Can you report accuracy on the sign/magnitude of score changes and coverage of the requested uncertainty intervals?

  3. What resources and trial budgets were used for the expert references? Is there a knowledge-construction reference that uses the same auxiliary LLM without an adaptive research loop?

  4. How does this work differ from PostTrainBench and Curation-Bench? I think this is a crucial question, as many of the capabilities it evaluates are already covered by PostTrainBench, Curation-Bench, and related work.

2 comments
AuthorsOpenReview ↗

(part 1/2) Response to Reviewer UcR9

Thank you for your careful review and constructive comments. We appreciate the opportunity to address these points with additional controls, analyses, and clarifications.

W1 and Q1. Feedback, search, and training variability

We added a retrieval control that preserves repeated attempts, cross-round memory, and best-of-run selection while withholding target-model feedback. Agents had the same data pool, target model, training recipe, auxiliary models, and configured resource limits. They could inspect their previous proposals, reuse intermediate results, and consult auxiliary quality checks, but received no evaluation scores, training loss, or best-checkpoint information. The evaluator selected the highest-scoring ID checkpoint after each run and evaluated it on hidden OOD data. This design tests the value of revealing scores during a continuing search. Each proposal can draw on previous work.

ModelID with feedbackID without feedbackOOD with feedbackOOD without feedback
GPT-5.6-Sol40.4640.0426.1524.19
Kimi-340.0738.0029.8528.65

Scores are mean nDCG@10 × 100. Both agents scored higher with feedback, including on hidden OOD evaluation (+1.96 for GPT-5.6-Sol and +1.20 for Kimi-3). This supports a contribution from feedback beyond the opportunity to try and select multiple proposals. The original and supplementary runs differed in hardware and realized compute, so we interpret these differences as supporting evidence. The first-to-best gain in §4.3 measures the full iterative procedure.

The trajectories also show what an agent can change within a standard processing workflow. In one no-feedback Kimi-3 run, three proposals each used 600,000 training rows and four negatives per row. Sampling in the original source proportions scored 34.77. Reweighting sources and rotating the negatives assigned to repeated examples scored 38.38. A subsequent proposal that also incorporated auxiliary-LLM checks on negatives scored 39.02. These comparisons show how much the choice of data strategy can matter at a fixed size. A separately specified, equal-budget pipeline would be needed to estimate the advantage over that particular workflow.

For training variability, we keep the target model, training configuration, and training seed fixed across proposals. In retrieval, every training call resets the seed to 42. This controls seed changes across rounds. Estimating residual variability on unchanged data would require repeated training runs on the same submission.

W2 and Q2. Forecast baselines, direction of change, and interval coverage

We evaluated each pre-training forecast against the previous score and a linear extrapolation of the last two scores on the same eligible rounds. The first three numeric columns report MAE on a 0–100 scale. The last two summarize the intervals the agents supplied before training.

TaskModelAgent MAEPrevious-score MAETrend MAEInterval coverageMean width (points)
Tool useGPT-5.6-Sol1.702.093.7777.8%3.97
Tool useKimi-32.733.566.6970.0%6.25
RetrievalGPT-5.6-Sol0.640.610.5892.3%3.13
RetrievalKimi-31.551.000.8085.7%4.50

Both agents beat the historical baselines on tool use. For the direction of change, Kimi-3 reached 80% accuracy on tool use against 50% for always predicting an increase. GPT-5.6-Sol matched that baseline at 50%, and both agents matched it on retrieval. The MAE of the predicted signed change equals the score MAE because the same previous score is subtracted from the prediction and the observation. Interval coverage is the observed fraction of outcomes inside the forecast interval. A calibration claim would require a common nominal coverage level. Given one prediction-enabled run per model–task pair, we will present the results in §4.4 as exploratory. In particular, the low retrieval MAEs are also attainable by extrapolating score history.

W2. What does the trajectory-training comparison establish?

Each arm in §4.5 uses exactly 5B mid-training tokens from a code-oriented training setup. The control uses the original mixture. In the treatment, AutoDataBench trajectories occupy 80M tokens (1.6% of the same budget), obtained by upsampling an 8M-token corpus tenfold. Both checkpoints then receive identical SFT. All five coding scores improved in this matched pair, with the largest gains on SWE-bench Multilingual (27.76 to 33.33) and CRUXEval input prediction (72.00 to 74.12). This establishes the feasibility of reusing research trajectories in that mixture. Distinguishing the contribution of their data-research content from that of other agent trajectories would require an equal-token comparison with an alternative trajectory corpus and replication across training seeds.

AuthorsOpenReview ↗

(part 2/2) Response to Reviewer UcR9

Q3. Expert resources and access to feedback

The tool-use reference uses the original data before corruption. After schema validation and deduplication, we allocated 16,000 places in proportion to category size and selected examples using the source dataset’s quality scores. Agents had no access to those quality scores.

For retrieval and knowledge injection, human domain experts worked under the same task budgets and had the same opportunity as the agents to run multiple trials. They used the same round-by-round ID feedback to revise their proposals. They knew that an OOD evaluation would take place, while its contents remained hidden. The retrieval expert organized training examples and mined negatives. The knowledge expert used keyword- and rule-based mining. A fixed knowledge-generation pipeline using the same auxiliary LLM remains an open comparison. We will clarify the construction and interpretation of each reference.

Q4. Relationship to PostTrainBench and Curation-Bench

PostTrainBench evaluates end-to-end post-training decisions, including the choice of data sources, training methods, and hyperparameters. AutoDataBench fixes the target model and recipe for each task so that the agent’s experimental decisions concern the data.

Curation-Bench is closely related concurrent work. Both settings allow feedback-driven data revision under a fixed pipeline. Curation-Bench examines how agents explore curation strategies and how the scaffold and search budget affect that process. AutoDataBench evaluates how the same agents handle three tasks that involve repairing corrupted tool-call supervision, organizing positive and negative examples for contrastive training, and turning source documents into new knowledge supervision. Hidden evaluations then probe transfer or retention. We also study whether agents can anticipate an intervention’s outcome before training and whether their research trajectories can serve as mid-training data. We will make these distinctions explicit in Related Work.

Reviewer_yajmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AutoDataBench, a benchmark for evaluating how well LLM agents improve training data while the model, training recipe and evaluation are held fixed. It covers three dimensions of “data intelligence” (diagnosis and repair, organization, construction) through three tasks: tool-use fine-tuning on corrupted data, contrastive training of an embedding model, and knowledge injection into a model with an early training cutoff. Seven LLMs are compared within a shared agent framework on in-distribution scores and on hidden out-of-distribution evaluations. The paper also tests whether agents can predict the effect of their data changes, and whether the agents’ trajectories help as mid-training data for coding.

Critical strengths
  • Isolating the data side of auto-research is a useful framing. Fixing everything except the data makes differences between agents easier to attribute, and covering three training paradigms (SFT, contrastive learning, distillation) gives broader coverage than most related benchmarks.

  • The hidden OOD evaluation in each task is a valuable design choice. The retrieval results, where strong in-distribution performance does not carry over to held-out data, are an interesting finding that most agent benchmarks would not be able to surface.

Critical weaknesses
  • The positioning against recent work could be sharper. Curation-Bench uses a very similar setup (fixed pipeline, agent edits data, iterative feedback), and a more explicit comparison would help clarify what is new beyond the broader task coverage. Related data-agent work such as LLM-AutoDP is not discussed.

  • The evaluation design makes the ID and OOD results hard to interpret. The ID scores used as feedback and for checkpoint selection appear to be computed on the reported evaluation sets themselves, selected as the best of many rounds, so the ID numbers are likely optimistic. The task prompts are also not reported, so it is hard to tell how much of the observed behavior is discovered by the agents versus guided by the instructions.

  • The model comparison is hard to rely on. The lineup mixes flagship and non-flagship or older models, and some major model families are missing. Results come from three runs with high variance, the tool-use task is close to saturated, and the Overall score is largely driven by the knowledge task.

  • The evaluation measures only the final model scores, not the data skills the benchmark is built around (for example, how well agents detect and repair corruption). The agent framework is also not described in enough detail to reproduce or judge how much it affects the results.

Questions for authors
  1. Since the tool-use task has ground-truth corruption labels, I was wondering if the authors have analyzed how well agents actually detect and repair corrupted examples (for example, precision and recall per corruption type, or how often they filter versus repair). This would make the diagnosis dimension measurable directly instead of only through downstream accuracy.

  2. I was wondering whether agents receive feedback on a split separate from the reported ID evaluation. My understanding from the paper is that the ID evaluation set serves as both the feedback signal and the checkpoint-selection criterion. If so, results with a separate validation split, and ID and OOD scores across rounds, would help.

  3. Regarding retrieval behavior: I was curious what the final training mixtures looked like compared to the original source distribution, and whether reweighting toward the ID sources explains the OOD drop. Have the authors measured how often mined negatives are actually relevant passages (false negatives), and how much of the gain comes from training-set size alone? It would also help to know which corpora negatives were mined from, to rule out overlap with the evaluation collections.

  4. Regarding knowledge injection setup: Could the authors clarify whether Retention scores are visible to the agent during optimization, and whether checkpoint selection uses Novel alone or a combination of Novel and Retention? Since the agent receives source documents that correspond directly to the evaluated facts, results on paraphrased questions or facts not directly targeted would help separate knowledge acquisition from memorization.

  5. Could the authors describe the shared agentic framework in more detail (agent loop, how context is managed over 24 hours, memory across rounds)? I was also wondering if the authors have checked whether the rankings hold under a different scaffold, and whether they could report inference settings (reasoning effort, model versions), token usage and cost, and how many rounds each model completed within the time limit.

  6. Model selection and aggregation. I was curious about the reasoning behind the model choices, for example, using a lighter DeepSeek variant, which Claude model “Claude-4.7” refers to, and why no Gemini model is included. A normalized aggregate (for example, the fraction of the baseline-to-expert gap closed in each task) with confidence intervals would make the ranking easier to interpret.

  7. How were the expert references built, and did they use the same ID feedback or any knowledge of the OOD sets? Separately, have the authors considered varying or removing the data budget cap in the tool-use task? This would show the trade-off between data quantity and quality, which the current setup largely fixes.

2 comments
AuthorsOpenReview ↗

(part 1/2) Response to Reviewer yajm

Thank you for your detailed review and helpful suggestions. We appreciate the opportunity to clarify the benchmark design and provide further examples and results.

Relationship to Curation-Bench and LLM-AutoDP

AutoDataBench is a benchmark for evaluating how LLM agents improve training data. Curation-Bench is closely related concurrent work, cited in §2. Both benchmarks allow iterative revision under a fixed training pipeline. Curation-Bench studies how agents explore curation strategies under different scaffolds and search budgets. AutoDataBench evaluates the same agents across tool-call repair, retrieval data organization, and knowledge construction, then asks whether the gains transfer. Our tasks are complementary. In tool use, agents repair unreliable labels. In retrieval, they organize positive and negative examples for contrastive learning. In knowledge injection, they construct new supervision from raw documents. All seven models in Table 1 exceeded the simple retrieval baseline on mean ID score, yet all seven fell below it on mean hidden OOD score when evaluated at their ID-selected checkpoints. The benchmark also lets us record forecasts before training and test whether the resulting trajectories improve downstream coding at a fixed mid-training token budget.

LLM-AutoDP proposes a method for generating and optimizing data-processing pipelines. Its agent selects and orders predefined processing modules and uses evaluation feedback to improve their composition. In AutoDataBench, agents inspect the data and implement their own interventions within each task’s fixed training pipeline. We will add LLM-AutoDP to Related Work and clarify this relationship.

ID feedback and checkpoint selection

Yes. The same ID evaluation set supplies round-by-round feedback and determines checkpoint selection. Table 1 reports the mean of the best ID scores from three independent runs. Its OOD entry is the mean hidden-evaluation score of those three selected checkpoints. The ID score therefore measures the outcome of optimization against a visible feedback signal and can be optimistic as an estimate of generalization. Figure 2 already shows best-so-far ID curves for selected standard runs, and Figure 3 shows the observed score at each round of the forecasting experiments. We will supplement these figures in the appendix with results on a separate ID test split or with round-by-round OOD curves.

Tool-use diagnosis and repair

Agents both filter and repair training examples. In one run, GLM-5.2 filtered out 5,031 examples because the tool calls supplied arguments the tools did not accept, and 544 because a required argument was missing. Other errors require the LLM to interpret the user’s request. In a separate run, a user asked for the next ten playlist entries after the first ten. The original training label skipped only one entry. The agent changed offset from 1 to 10 and preserved the correct user and playlist IDs. The filtering counts describe actions on the dataset, and the before-and-after example shows a repair whose correctness can be checked directly against the request.

Retrieval data size and negative construction

All mined negatives came from the supplied training pool, which is disjoint from the evaluation corpora. Agents sometimes increased the weight of sources represented in the ID evaluation. They also adjusted the difficulty of the negatives paired with each query. In one Qwen-3.8-Max run, two rounds used the same 150,564 query–positive pairs and two negatives per pair. Selecting the most similar eligible candidates yielded an ID score of 29.92. In the other round, the agent required mined negatives to score at least 0.03 below the positive in query similarity and filled any remaining slots by sampling from the same source. The score rose to 34.24. The 4.32-point difference occurred with training size, source proportions, and negatives per pair held fixed. The task thus requires decisions both about which sources to sample and about what distinctions the model should learn within those sources.

Knowledge acquisition and retention

During optimization, the agent can read the source documents and receives aggregate Novel accuracy after each round. The Novel questions and Retention scores remain hidden throughout the run. We select the checkpoint solely by Novel accuracy, breaking ties in favor of the earliest round, and evaluate retention at that checkpoint.

The source documents contain the facts to be learned. The agent must decide which facts to teach and how to turn them into training questions and answers. At evaluation time, the target model answers independently constructed Novel questions without access to the documents. Novel accuracy therefore reflects fact selection, the effectiveness of the resulting supervision, and the model’s ability to answer when the evaluation wording differs from its training examples.

AuthorsOpenReview ↗

(part 2/2) Response to Reviewer yajm

Model selection and supplementary Gemini results

Flash was the DeepSeek-V4 variant available to us when we ran the original experiments. The “Claude-4.7” entry in Table 1 is Claude Opus 4.7 (claude-4.7-opus). We have since evaluated Gemini-3.5-Flash (gemini-3.5-flash) in three runs per task. The following summarizes the results:

TaskID metricMean IDHidden metricHidden mean
Function callingFinal ID74.85BFCL OOD55.86
RetrievalID38.90OOD30.44
Knowledge injectionNovel51.90Retention47.05

Using the equal-weight average of the three task ID scores from the above table, Gemini-3.5-Flash obtains an Overall of 55.22.

Aggregate ranking and uncertainty

Although Overall weights the three tasks equally, their observed score ranges differ. Among the original seven models, the ranges of mean ID scores are 1.16 points for tool use, 4.24 for retrieval, and 12.20 for knowledge injection. Kimi-3 leads Overall on the strength of knowledge injection, while GPT-5.6-Sol leads on tool use and retrieval. Table 1 reports per-task means and standard deviations across three runs.

Following your suggestion, we also computed the fraction of the baseline-to-expert gap each model closes. For each task we divide the agent’s mean ID gain over baseline by the expert’s ID gain over baseline, then average the three fractions equally. The denominators are 11.20, 6.24, and 9.30 points for tool use, retrieval, and knowledge injection, respectively. The baseline corresponds to 0% and the expert to 100%. Scores above the expert can exceed 100%.

Model in Table 1Original OverallNormalized OverallApproximate 95% CI
Kimi-360.68139.95%[119.49%, 160.40%]
GPT-5.6-Sol58.85120.59%[95.52%, 145.65%]
GLM-5.259.10120.41%[86.77%, 154.05%]
Qwen-3.8-Max58.56115.66%[76.07%, 155.25%]
Claude-4.758.31112.79%[83.26%, 142.32%]
Qwen-3.7-Max56.8097.07%[67.65%, 126.50%]
DeepSeek-V4-Flash55.2074.48%[10.83%, 138.13%]

We used the per-run best ID scores in Appendix Tables 4–6, treating the three tasks and their reference scores as fixed and the runs within each task as independent. We propagated the three task-specific sample variances through the normalized mean and used a Welch–Satterthwaite approximation for the degrees of freedom. Kimi-3 retains the highest point estimate. GPT-5.6-Sol and GLM-5.2 exchange places and differ by only 0.18 percentage points after normalization. With three runs per task, the intervals support a cautious interpretation of close rankings.

Expert references and the 16,000-example cap

For tool use, we constructed the reference from the data before corruption. After tool-schema validation and deduplication, we allocated places in proportion to category size and used the original dataset’s quality scores to select 16,000 examples. Agents had no access to those quality scores. The 16,000-example limit is a ceiling on each training submission. Agents may submit smaller datasets. Agents can select fewer than 16,000 examples to prioritize quality. At least four models did so, including Qwen-3.8-Max (11,655 examples) and Claude Opus 4.7 (12,357). We kept that ceiling fixed throughout the experiment, so the comparison evaluates selection, repair, and data composition under a common size constraint.

For retrieval and knowledge injection, domain experts could iterate under the same resource limits and used the same round-by-round ID feedback as the agents. The retrieval expert organized examples and mined negatives. The knowledge expert used keyword- and rule-based mining. Both knew an OOD evaluation would take place, and neither saw its contents.

Shared scaffold, prompts, and inference settings

Each run continues in one conversation. The shared agent framework manages memory and context across rounds. After training, it returns the ID result to that conversation so the agent can inspect what happened and revise its next submission. The Evals column in Appendix A.2, Tables 4–6, gives the number of training rounds that returned an ID score for each model, task, and run.

The task prompts specify the objective and operational constraints and leave the processing strategy to the agent. We will include the full prompts in the appendix. All models in Table 1 used the same scaffold and a reasoning-effort setting of high. Across all iterations of one complete run, the research agent typically generated hundreds of thousands of output tokens, with usage varying by model. Evaluating how rankings change across scaffolds is a direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.