- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark for agents that construct visual data collections from the open web rather than retrieve answers or select images from a fixed pool. It contains 46 tasks across seven domains, evaluates text-only and reference-image specifications, and combines provenance checks with task-specific validity and quality rubrics. Experiments examine agent frameworks, collection size, and joint-task execution.
Critical strengths
The paper makes a worthwhile shift from evaluating an agent’s answer to evaluating the artifacts it acquires. Requiring images, source information, and factual annotations creates a concrete deliverable whose correctness can be inspected.
Critical weaknesses
-
Collection-level success is not yet collection-level quality. The paper motivates data construction as a collection-level capability, but its principal metrics largely aggregate individual-image judgments. BPR measures the proportion of valid images, QS averages image-level quality, and TSR measures whether enough images pass. Duplicate checking introduces a relational constraint, but these metrics do not explicitly
-
Candidate tasks are filtered using an OpenClaw agent powered by GPT-5.5, with tasks removed when their mean preliminary score exceeds 4.0. GPT-5.5 also drafts preliminary rubrics and performs final semantic evaluation, although humans subsequently refine the benchmark. This is defensible as a method for constructing a challenge set, but it conditions the benchmark on one system’s failures. Consequently, I would be cautious about interpreting the reported performance as representative reliability across research-driven data needs. This is a selection-bias concern, not evidence that the benchmark is invalid.
-
Ultimately, I am concerned about the practical utility of the proposed evaluation framework. In my view, a benchmark should not only measure current performance but also provide actionable signals for improving models through training or at least expose model capability issues. The paper does not yet demonstrate whether its evaluation criteria serve these purposes.
Questions for authors
NA
4 comments
Response to Weakness 3.
We thank the reviewer for raising this important question. First, DataResearchBench employs three metrics, BPR, QS, and TSR@, to reveal different capabilities and limitations of agents. In addition, trajectory analysis reveals how agents allocate their efforts across planning, search, verification, acquisition, and quality assurance. Our Collection Size and Multi-Task experiments further challenge agents in long-horizon data collection and multi-task execution. The experimental results demonstrate that agent decisions are crucial to task performance. As further demonstrated in our response to Reviewer roT1's Weakness 2, we observe that Kimi exhibits relatively robust performance at larger collection sizes and benefits from subagent-based task decomposition in joint-task execution. These results expose several limitations of current agents and demonstrate that our benchmark can be used to evaluate and compare concrete agent-level improvement strategies. In our subsequent experiments, we have also observed substantial positive gains from improving agent strategies on these tasks.
Since retraining closed-source models or very large open-source backbones is often costly or difficult to implement, we believe that evaluating and improving agent-level strategies and frameworks provides a practical complementary direction. This is because agent capabilities emerge from the coupled interaction of the backbone's underlying capabilities, the harness, and the strategies governing how the backbone is utilized. DataResearchBench provides a common set of challenging tasks and evaluation criteria for investigating these differences and comparing potential improvements, while offering a natural long-horizon task setting.
Finally, we believe that open-web data collection has important practical relevance to emerging recursive self-improvement (RSI) workflows. An important component of such workflows is identifying current capability limitations and acquiring task-relevant data to address them, followed by data verification and organization, and ultimately using appropriate frameworks for model post-training. High-quality data is essential to this process. DataResearchBench directly evaluates whether agents can discover, acquire, verify, and organize such visual data resources from the open web. Its collection-size settings also provide a natural testbed for evaluating long-horizon data acquisition, while its multi-task setting examines coordination across multiple concurrent collection objectives.
Therefore, we believe that DataResearchBench not only reveals certain limitations of current agents but also provides practical research and evaluation value. We thank the reviewer for raising these concerns and will further clarify these practical implications in the revised manuscript.
Thanks for authors for the response. My score remain unchanged since I am still unconvinced for my weakness 3. The authors mention retraining closed or large open models can be costly. I agree with this, but post-trainning models at 7b or 13b with LORA should be very doable and would significantly strengthen the results, which I highly recommend the authors to pursue.
We agree that applying LoRA-based post-training to 7B or 13B open-source models is feasible and would provide a valuable way to demonstrate the practical utility of our benchmark. We appreciate this suggestion and believe that it is an important direction to explore.
At the same time, we believe that one important value of a benchmark is providing a systematic way to evaluate agents, identify their capability limitations, and reveal potential directions for improvement. Model post-training is one important application of such a benchmark, rather than a prerequisite for establishing its evaluation value. DataResearchBench already provides diagnostic signals through its complementary metrics and trajectory analysis, revealing where current agents struggle and how different execution strategies affect task outcomes. Furthermore, we applied the Dream-RSI[1] approach to five tasks and conducted additional experiments while changing only the model's execution strategy. All five tasks showed improvements in their scores, although the magnitude of improvement varied across tasks. This provides further empirical evidence that agent strategies can have a substantial impact on task performance.
Regarding the proposed LoRA experiments, we believe that effective post-training for open-web data sourcing involves several challenges beyond the parameter-efficient fine-tuning process itself. For SFT, constructing high-quality training trajectories is hard because these tasks are open-ended and admit multiple valid sourcing strategies. Directly using synthetic trajectories may introduce a bias toward particular execution paths, which are not necessarily optimal or generalizable. Their quality and effectiveness therefore need to be further validated on real-world web tasks. For RL, online rollouts introduce substantial runtime costs and are affected by network instability, tool availability, and changes in website content and structure. Even when synthetic environments or trajectories are used to reduce these costs, their effectiveness and generalization to real open-web environments still require validation.
We have also conducted preliminary evaluations using Qwen3.5-9B with the Qwen agent framework. The results were unsatisfactory, particularly on long-horizon tasks, where the agent struggled to complete the workflow and even failed to consistently satisfy the basic output-format requirements. This suggests that effective post-training may need to be conducted in stages, first establishing basic instruction-following and structured-output capabilities, and then improving the core data sourcing capabilities. Corresponding ablation studies would also be necessary to distinguish improvements in basic format compliance from genuine gains in search, verification, and collection quality. Otherwise, performance improvements after fine-tuning may not necessarily reflect improvements in the capabilities that DataResearchBench aims to evaluate.
We therefore view LoRA-based post-training as a valuable but distinct research direction that deserves dedicated investigation beyond the scope of the current benchmark study. Such work would involve not only fine-tuning but also training data construction, training strategy design, and careful evaluation of generalization in real-world environments. We fully agree with the reviewer that exploring this direction would further strengthen the practical utility of DataResearchBench, and we will pursue it in our future work. We sincerely thank the reviewer again for this valuable recommendation.
[1] Dream-RSI: Recursive Self-Improvement through Evolving Worlds
Response to Weakness 1.
We thank the reviewer for pointing out that our existing metrics do not explicitly capture the category-level coverage of an entire collection. We agree that evaluating individual-image validity and quality alone is insufficient to fully characterize the overall properties of a collection. To complement the existing metrics, we conducted an additional analysis of task-mentioned category coverage across the collected images.
Specifically, we constructed a category inventory for each task based on the categories explicitly mentioned in its original task description and task-specific Task Relevance criteria. Thirty tasks contain sufficiently explicit categories to support this analysis. For each task, we then attempted to map all images passing the Base criteria to the corresponding categories. We first averaged category coverage across the relevant dimensions within each task and then calculated the macro average across all applicable tasks, measuring the proportion of task-mentioned categories covered by at least one valid image. The results are shown below.
The results reveal clear differences in collection-level category coverage across agents. In most cases, providing reference images has a positive effect on coverage. This additional metric provides a complementary perspective to BPR, QS, and TSR, reflecting the extent to which a collection covers the categories specified by its task. We will incorporate this additional analysis into the revised manuscript, clarify the respective roles of image-level and collection-level evaluation, and further refine this evaluation in future work.
Response to Weakness 2.
We thank the reviewer for this insightful observation. We agree that using a single reference agent to filter candidate tasks introduces a certain degree of selection bias. To examine the extent of this bias and whether the resulting benchmark primarily reflects the failure patterns of the reference agent, we conducted an additional cross-agent analysis involving the five agents in our main experiments. The results are summarized below.
These results reveal both shared challenges and system-dependent failure patterns. The mean failure-set Jaccard is higher than the corresponding random baseline, indicating that different agents encounter some common difficult tasks. Meanwhile, bidirectional cross-over occurs in all 10 agent pairs under the Visual Specification setting and in 9 out of 10 pairs under the Text-only setting, showing that their failure sets are not simply nested. Therefore, the selected tasks expose performance differences across multiple agent systems rather than merely reproducing the failure patterns of the reference agent. We acknowledge that this additional analysis does not eliminate the original selection bias, but it further supports the value of DataResearchBench for cross-system evaluation.
Regarding the use of GPT-5.5 in rubric construction and semantic evaluation, we emphasize that GPT-5.5 was only used to draft the preliminary scoring criteria. These criteria were subsequently revised and cross-checked by three human reviewers until agreement was reached. During this process, at least one criterion was completely rewritten by human reviewers for the vast majority of tasks, and the proportion of criteria revised also exceeded half. We also report the agreement between the automated judge and human reviewers. Although these procedures provide additional human quality control, we acknowledge that residual model-related evaluation bias may still exist. Our current analysis further supports the value of DataResearchBench.
We will clarify the challenge-oriented scope of DataResearchBench, the role of reference-agent filtering, and the above limitations in the revised manuscript. We will also incorporate additional analyses and experiments to further address concerns regarding potential bias.