acceptodds
Under review as a conference paper at ICLR 2027

DatasetResearch: Benchmarking LLM Agents For Demand-Driven Dataset Discovery

Abstract

As high-quality public data becomes scarce, sustaining model progress increasingly requires agents to discover datasets tailored to specific needs. Yet it remains unclear whether autonomous agents can reliably search for, transform, or synthesize useful training data. We introduce DatasetResearch, comprising 208 reference- grounded demand specifications, each paired with a held-out reference dataset and detailed metadata. Our evaluation combines metadata alignment with downstream task performance under few-shot and fine-tuning protocols. Across open and proprietary systems, synthesis substantially outperforms search on Reasoning- intensive fine-tuning (75.46% vs. 34.52%), whereas Knowledge-intensive fine- tuning is close (43.67% vs. 45.29%) and the search advantage is clearer in few- shot evaluation. On the difficult 20-task DatasetResearch-Pro subset, a Retrieve–Transform–Synthesize workflow reaches 32.4% normalized fine-tune performance, compared with 24.1% for the strongest pure workflow; demand- only human reconstruction reaches 52.3%, demonstrating measurable headroom. These results establish DatasetResearch as a discriminative benchmark for data-research agents while clarifying the scope and attainability of its reference- normalized scores.

Primary area:
datasets and benchmarks
Keywords:
dataset, large language models, agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yiynOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper benchmarks agents on an important problem of whether agents could come up with datasets based on requests. The authors evaluate agents on several different settings, such as search for existing datasets, synthesizing, deep research, and retrieve transform and synthesize. The benchmark consists of more than 200 tasks, where each contains a demand specification, a reference dataset, and metadata. The benchmark instances are split into reasoning-intensive and knowledge-intensive. The authors evaluated agents on these tasks by comparing against the reference dataset in various dimensions. The authors found that there is still room for improvement for existing agents.

Critical strengths
  • This paper evaluates agents on an important problem of whether they could discover datasets given users' demands. This is very important for researchers because researchers need to discover or develop datasets for their own training. Also the benchmark covers multiple domains, enabling comprehensive evaluation.
  • The evaluation protocol is well-developed. The authors evaluated on not only similarity or match rate of the discovered dataset compared to the reference dataset, but also examined whether the resulting dataset the agent discovered actually improves model abilities through training.
Critical weaknesses
  • The benchmark lacks validity. The authors include reference datasets in the benchmark, but did not validate that the reference dataset is the only correct datasets matching the users' requests. Thus, it is hard to tell whether low score on the benchmark is due to the agent not being able to come up with good datasets, or whether the reference datasets are not comprehensive/representative.
  • The writing is a bit difficult to follow. Additionally, some details are not clear in the paper. For example, it is hard to understand which part of each dataset was used for training and which was used for eval, what is the split rate.
Questions for authors
  • For DTP-Finetune, what is the train test split?
  • Why is there only one reference dataset for each demand request?
Reviewer_6UU6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes to benchmark agents' ability to retrieve or synthesize training datasets for 208 NLP tasks and evaluates metadata alignment and downstream few-shot and fine-tuned performance.

Critical strengths

S1: Finding useful datasets is a meaningful problem.

S2: I appreciate that the evaluation combines metadata alignment with downstream task performance, rather than a single dimension.

Critical weaknesses

W1: It's still unclear to me what this benchmark is actually measuring. The paper combines finding existing datasets, generating synthetic data, and training a downstream model. These are related but different tasks. If a model performs poorly after training, how do we know whether the agent found unsuitable data or the model failed to learn from actually good data? The current eval does not clearly separate these possibilities.

W2: How representative is the setting of actual dataset discovery? The authors generate demand specifications from existing datasets and deliberately prevent agents from accessing the reference datasets. This seems closer to finding substitute training data than answering real users' data demands. Also, if another dataset satisfies the demand but differs from the reference distribution, would the eval fairly recognize it?

W3: I'm not quite convinced that the practical value of this benchmark is established. Using o3 to generate data for fine-tuning LLaMA-8B can make sense if the goal is cheaper inference (which was not mentioned in the paper though), but when is this worthwhile compared with directly using the teacher? I would like to see quality and total cost comparisons, including data generation and training. Otherwise, it is unclear what the synthesis results tell us about dataset discovery beyound transferring capabilities from a larger model to a smaller one.

Questions for authors

Please address my questions in W1-3.

Reviewer_h69iOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DatasetResearch evaluates agents that search for or synthesize training data from a demand specification. It constructs 208 tasks from reference datasets and scores outputs using metadata similarity and downstream model performance. The paper compares search, synthesis, and a hybrid workflow, with additional human comparisons on a 20-task subset.

Critical strengths

S1: Turning a data need into a useful training set is a practical problem. Including downstream fine-tuning performance makes the evaluation more informative than retrieval relevance alone.

S2: The human reconstruction study provides a useful reference point. The retrieved-union oracle and query-rewrite experiments also help identify limitations of the discovery pipeline.

Critical weaknesses

W1: The benchmark's task does not fully match its demand-driven framing. It appears closer to reconstructing a hidden dataset than finding data that satisfies a real user need: demands are generated backward from reference datasets, and the source can be identified in 66% of cases (Section 4.5). The natural-query rewrites are a useful check, but they remain tied to the same references. Independently collected practitioner needs would better establish that the benchmark measures useful dataset discovery.

W2: The search–synthesis comparison is not sufficiently controlled. In the main results, synthesis receives a reference example and generates 500 samples, whereas search returns up to 1,000 samples that require format conversion (Table 5; Appendix D). Synthesis can directly follow the reference format, so format alignment may explain part of its advantage even with standardized downstream processing. The abstract claims that “synthesis substantially outperforms search on Reasoning-intensive fine-tuning (75.46% vs. 34.52%).” These scores show a gap between the evaluated workflows, but do not establish that synthesis itself causes the advantage. The o3 reference-example ablation is helpful, but does not resolve the other differences or directly validate the headline GPT-5 comparison. Matched sample counts, reference access, and post-processing, together with label normalization or format-insensitive scoring, are needed.

W3: The search setting is confusing and may disadvantage retrieval by construction. The paper describes references as gated, while search on the full benchmark is restricted to public HuggingFace data (Section 3.2; Appendix D). Preventing direct retrieval of the reference is understandable, but evaluating against an inaccessible reference may penalize useful alternatives. The paper needs to justify why this setting fairly measures useful dataset discovery and clarify whether suitable alternatives are available within the permitted search space.

Questions for authors

Q1: Can you validate the benchmark using independently collected practitioner needs, beyond rewrites of reference-derived specifications? How would you distinguish satisfying a real need from reconstructing the hidden reference?

Q2: Does the GPT-5 synthesis advantage on reasoning-intensive fine-tuning remain when both workflows receive the same reference information, use equal sample counts, and undergo equivalent post-processing? How much of the gap remains after label normalization or format-insensitive scoring?

Q3: Why is search restricted to public HuggingFace datasets while the references are gated? Have you verified that suitable alternatives exist within this search space, and does expanding the search scope change the comparison with synthesis?

2 comments

Thank you for the detailed response and the additional experiments, which address most of my concerns about the framing and the controlled comparison, though I still have some reservations about the search setting. Taking everything into account, I’ve raised my score from 2 to 3.

Thank you for your thoughtful follow-up and for raising your score. Your comments on the framing and the controlled comparison shaped the most important changes in our revision, and the paper is clearer and more rigorous as a result. We have noted your remaining reservations about the search setting. In the final version, we will state the search scope and its limitations more clearly. If you have any further suggestions, we would be glad to discuss them.

Comments on the paper

1 comment
Area_Chair_g5mROpenReview ↗

Please join the discussion and put acknowledge on authors' rebuttal

As the discussion period is approaching its end, please join the discussion and put acknowledge or feedback on authors' rebuttal, thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.