- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper benchmarks agents on an important problem of whether agents could come up with datasets based on requests. The authors evaluate agents on several different settings, such as search for existing datasets, synthesizing, deep research, and retrieve transform and synthesize. The benchmark consists of more than 200 tasks, where each contains a demand specification, a reference dataset, and metadata. The benchmark instances are split into reasoning-intensive and knowledge-intensive. The authors evaluated agents on these tasks by comparing against the reference dataset in various dimensions. The authors found that there is still room for improvement for existing agents.
Critical strengths
- This paper evaluates agents on an important problem of whether they could discover datasets given users' demands. This is very important for researchers because researchers need to discover or develop datasets for their own training. Also the benchmark covers multiple domains, enabling comprehensive evaluation.
- The evaluation protocol is well-developed. The authors evaluated on not only similarity or match rate of the discovered dataset compared to the reference dataset, but also examined whether the resulting dataset the agent discovered actually improves model abilities through training.
Critical weaknesses
- The benchmark lacks validity. The authors include reference datasets in the benchmark, but did not validate that the reference dataset is the only correct datasets matching the users' requests. Thus, it is hard to tell whether low score on the benchmark is due to the agent not being able to come up with good datasets, or whether the reference datasets are not comprehensive/representative.
- The writing is a bit difficult to follow. Additionally, some details are not clear in the paper. For example, it is hard to understand which part of each dataset was used for training and which was used for eval, what is the split rate.
Questions for authors
- For DTP-Finetune, what is the train test split?
- Why is there only one reference dataset for each demand request?
Thank you for the detailed response and the additional experiments, which address most of my concerns about the framing and the controlled comparison, though I still have some reservations about the search setting. Taking everything into account, I’ve raised my score from 2 to 3.