- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DataSense-Bench, which is a benchmark that tests whether frontier AI agents can judge the training value of data before any training happens. Within the designed pipeline, the CLI agent receives a candidate data pool, inference-only access to Qwen3-4B, a fixed SFT recipe, one H100 GPU, and four hours. It must form five disjoint, equal-sized groups and rank them by expected post-training performance. The benchmark then finetunes a fresh model on each group and measures the score of the top-ranked group against random selection. The benchmark covers two tasks: terminal problem solving and multi-turn tool use. Ten agents are tested on TBLite and six on BFCL, with three runs each. The authors find that gains over random selection are small, rankings are unreliable and inconsistent across tasks, and the same agent often changes its selection rule across runs.
Critical strengths
-
This paper explores a timely and well-motivated problem. As agents apply to research tasks, it matters whether they can predict the value of an experimental choice before conducting training. Data selection is a concrete and checkable example of this.
-
The paper reports results for every group in every run, summarizes each agent's selection recipe, and reports API cost and time. This makes it easy to see how agents reason and where they disagree.
Critical weaknesses
-
Limited novelty and insight. The benchmark combines existing data pools, evaluations, models, and agents through a simple protocol. This protocol (select a subset from a fixed pool, train with a fixed recipe, and measure the result) is the core design of existing benchmarks such as DataComp, DataComp-LM, and DataPerf, which the paper does not cite. PostTrainBench already tests agents on post-training choices, including data. The main new element is using AI agents as the selectors. The paper also shows that agents select and rank poorly, but it does not test any way to improve this, such as limited training feedback or combining multiple runs, and it does not analyze which of the agents' choices lead to better outcomes. So it diagnoses a gap but offers little guidance on how to close it.
-
Limited experimental evidence. Agents are compared only with random selection and a "more data" reference. There are no standard selection methods (e.g., loss filtering, deduplication, LESS) or simple heuristics run under the same protocol, so we cannot tell whether agents are worse than simple algorithms or whether the task is just hard. The benchmark also uses one base model, one SFT recipe, and two tasks, and only six of the ten agents are tested on BFCL. It is unclear whether the conclusions hold more broadly.
Questions for authors
-
Would agents select and rank better with limited feedback, such as a small proxy training run or a few validation results? Does combining an agent's three runs give more stable or better selections? Which of the agents' choices (e.g. how they use base-model loss) are linked to better outcomes?
-
How does DataSense-Bench differ from DataComp, DataComp-LM, DataPerf, and PostTrainBench? What does it reveal that these benchmarks cannot?
-
How do standard data-selection methods (e.g., loss filtering, deduplication, LESS) and simple heuristics (e.g., completed trajectories only) perform under the same protocol? Do agents outperform them?
-
Why are only six of the ten agents tested on BFCL? Do the conclusions hold for other base models or training recipes?
8 comments
Response to Weakness 2:
Limited experimental evidence. Agents are compared only with random selection and a "more data" reference. There are no standard selection methods (e.g., loss filtering, deduplication, LESS) or simple heuristics run under the same protocol, so we cannot tell whether agents are worse than simple algorithms or whether the task is just hard. The benchmark also uses one base model, one SFT recipe, and two tasks, and only six of the ten agents are tested on BFCL. It is unclear whether the conclusions hold more broadly.
We added Qwen3.5-2B experiments on BFCL, evaluated the four previously missing agents to complete BFCL coverage for all ten agents, and added standard selection baselines. We will also extend the additional-model evaluation to TBLite.
Protocol. We expand Qwen3-4B BFCL coverage to all ten agents and evaluate six agents on Qwen3.5-2B. Each agent contributes three selections of five groups, with 50 trajectories per group. Every checkpoint is evaluated three times on BFCL800.
| BFCL evaluation | Qwen3-4B | Qwen3.5-2B |
|---|---|---|
| Average g1 accuracy range (%) | 34.76–36.93 | 16.83–20.29 |
| Random accuracy (%) | 35.58 ± 0.81 | 18.38 ± 0.82 |
| Highest average g1 accuracy (%) | 36.93, GPT-5.6 Sol | 20.29, Kimi |
| Average rank-correlation range | −0.27 to 0.83 | −0.10 to 0.93 |
| Identifies the best group in all three runs | Astra | None |
| Agent / reference | BFCL 4B accuracy (%) | BFCL 2B accuracy (%) |
|---|---|---|
| GPT-5.6 Sol | 36.93 | — |
| DeepSeek V4 Pro | 36.53 | 19.89 |
| GPT-6 Astra | 36.49 | 19.92 |
| Kimi K3 | 36.29 | 20.29 |
| Opus 5 | 35.94 | 19.64 |
| Random | 35.58 | 18.38 |
| Fable 5.1 | 35.21 | 19.15 |
| Gemini 3.5 Flash | 35.10 | 16.83 |
| Fable 5 | 35.01 | — |
| Sonnet 5 | 34.96 | — |
| GPT-5.5 | 34.76 | — |
Agent entries report average g1 accuracy across three independent selection runs.
Five agents exceed the respective Random average in each model panel, but reliably identifying the best group remains difficult on both models. We hold the training recipe fixed within each task to isolate differences in data selection.
We also evaluate low- and high-loss selection, diversity, deduplication, LESS, and completion flags under the same group sizes and downstream protocols. The tested loss-, diversity-, and quality-based selectors do not consistently improve on Random across tasks, motivating evaluation of how agents choose and combine these signals. Agent recipes themselves often combine these simple criteria: Kimi and DeepSeek use loss and diversity constraints, while Opus and Astra incorporate quality or execution-evidence checks, as analyzed in Section 4.4. Please refer to Section 4.2 and Table 1 for the full results and Appendix D.1–D.2 for the protocols.
We use task-specific SFT configurations rather than one shared recipe. TBLite uses the hyperparameters provided by the training-data authors for 1K trajectories, with a learning rate of 4×10⁻⁵ for seven epochs; BFCL uses 5×10⁻⁶ for two epochs, chosen through limited manual tuning rather than a systematic hyperparameter search. Appendix A.1 documents the final configurations, which are fixed across selections within each task to isolate data effects. Given the cost of downstream training and repeated evaluation, we focus on two tasks and ten agents.
Revision: Section 4.2 and Table 1 report the expanded comparisons; Appendix D.1–D.2 give the protocols. Section 4.4 analyzes agent recipes, and Appendix A.1 lists the training configurations.
Response to Question 1:
Would agents select and rank better with limited feedback, such as a small proxy training run or a few validation results?
As mentioned in our response to Weakness 1, validation feedback gives modest average test gains, but early improvements on the feedback set are not consistently sustained and transfer only weakly to unseen evaluation data. Development-loss feedback gives no average improvement. These experiments assess selection, not changes in five-group ranking accuracy.
Revision: Section 4.5 and Figure 5 report iterative feedback and its transfer to the unseen evaluation set. Appendix D.4 and Table 12 report the paired Kimi development-loss feedback results; Appendix D.5 details the iterative protocol.
Does combining an agent's three runs give more stable or better selections?
As mentioned in our response to Weakness 1, combining Kimi’s three selections improves observed BFCL accuracy by 1.03%. Therefore, combining an agent's multiple runs is a potential way to improve the performance, although it bring additional cost on data selection.
Revision: Section 4.5 reports the consensus result, and Appendix D.3 describes aggregation.
Which of the agents' choices (e.g. how they use base-model loss) are linked to better outcomes?
As mentioned in our response to Weakness 1, no single choice consistently improves selection across tasks. Section 4.4 analyzes how agents apply different preferences and normalization schemes to similar signals. The baselines show the same lack of consistent improvement over Random: low loss matches Random on BFCL but falls below it on TBLite, while diversity shows the opposite pattern.
Revision: Section 4.4 analyzes agent choices; Section 4.2 and Table 1 report the baseline comparisons. Section 5 discusses possible improvements.
Response to Question 2:
How does DataSense-Bench differ from DataComp, DataComp-LM, DataPerf, and PostTrainBench? What does it reveal that these benchmarks cannot?
As mentioned in our response to Weakness 1, we isolate agents’ data sense by fixing the model and training recipe and evaluating both selection quality and ranking ability. This reveals whether agents can predict data value in addition to finding useful subsets. The results separate these capabilities: for example, Kimi reaches 36.29% average first-group BFCL accuracy but only 0.02 average rank correlation, while Opus reaches 0.77 rank correlation without identifying the best group in any of its three runs.
Our shared-pair value-prediction experiment directly separates ranking ability from selection: agents predict which of the same already-evaluated subsets will perform better, with downstream scores hidden. This isolates forecasting ability from group construction, revealing failures that selection performance alone does not expose.
All agents rank the same 64 BFCL pairs and 32 TBLite pairs, containing 50 and 1,000 trajectories per group, respectively. Measured scores and evaluation examples are hidden. Each agent completes three prediction runs with a budget of one H100 for up to five hours per run. The table reports average prediction accuracy:
| Agent / control | BFCL | TBLite |
|---|---|---|
| DeepSeek | 81.25% | 42.71% |
| Sonnet 5 | 78.65% | 48.96% |
| Fable 5.1 | 78.13% | 48.96% |
| GPT-6 Astra | 76.56% | 46.88% |
| Kimi K3 | 73.44% | 44.79% |
| GPT-5.6 Sol | 73.44% | 45.83% |
| Opus 5 | 71.88% | 47.92% |
| Fable 5 | 70.31% | 41.67% |
| GPT-5.5 | 67.19% | 45.83% |
| Gemini | 61.46% | 45.83% |
| Choose the group with fewer answer tokens as higher-scoring | 59.375% | 59.375% |
Average prediction accuracy across agents is 73.23% on BFCL and 45.94% on TBLite, showing that reliable ranking data groups according to their post-training performance remains challenging. Please refer to Section 4.3 and Figure 4 for the results, and Appendix D.6 for the shared data construction.
Revision: Section 2, Related Work, now cites DataComp, DataComp-LM, and DataPerf and explains how our benchmark differs in evaluating both selection quality and ranking ability. Section 1 clarifies this motivation, and Section 4.2 discusses the cross-task baseline comparisons.
Thank you for taking the time to provide detailed feedback on the motivation and experimental scope of our work. Your suggestions broaden the baseline comparisons and motivate us to investigate aggregation and feedback as potential improvements. We address each of your concerns below.
Response to Weakness 1:
Thank you for highlighting these related benchmarks.
First, we want to clarify our motivation: to examine agents’ data sense through selection quality and ranking ability. Using agents to select data before costly training brings our benchmark closer to practical workflows in frontier labs and real-world applications. We view the ability to select useful data and anticipate its value as a foundational step toward AI scientists that can make autonomous experimental decisions.
Difference from prior benchmarks
DataComp and DataComp-LM benchmark training-data curation, while DataPerf evaluates a broader range of data-centric methods. We examine whether agents can both select useful data and predict its downstream value by asking them to commit to selected groups and their rankings before training.
To isolate this ability, we fix the downstream model and training recipe. This distinguishes our setting from PostTrainBench, which evaluates end-to-end post-training with data choices, training methods, and hyperparameters jointly determining performance. We added these citations and comparisons in Section 2 and clarified our motivation in Section 1.
Consensus and iterative improvement with limited feedback
Consensus. We select 50 trajectories by frequency across Kimi’s three original top-group subsets, breaking ties by pool index without using scores, and train on the merged subset.
Iterative improvement with limited feedback. We explicitly prompt Kimi K3 and GPT-6 Astra to iteratively improve their selections using reward from 98 validation tasks, with three independent runs per agent and up to five rounds. Checkpoints are selected by validation reward, and BFCL800 test scores remain hidden to the agents.
Consensus improves the observed BFCL accuracy from 36.29% to 37.32%.
Iterative improvement with feedback can improve performance on the validation set in early rounds, but these gains are not consistently sustained and transfer poorly to the unseen evaluation set: validation-selected checkpoints improve average test accuracy by 0.74% for Kimi and 0.14% for Astra, with one Astra run declining by 1.00%.
Additionally, our analysis in Section 4.4 shows that selection based on loss, diversity, or quality signals alone does not consistently improve performance across tasks. Therefore, future work could explore memory, skills, and agent harness design to better use feedback and iteratively improve selection.
Revision: Section 4.5 presents consensus and iterative feedback; Figure 5 shows the improvement on the validation set and evaluation set. Appendix D.3 gives consensus details, and D.5 specifies the iterative protocol.