acceptodds
Under review as a conference paper at ICLR 2027

DataResearchBench: Benchmarking Visual Data Sourcing with Generalist Agents on the Open Web

Abstract

Current agents are increasingly capable of retrieving information and synthesizing answers, but many research workflows require constructing data resources that are not already available in a suitable form. We introduce DataResearchBench to evaluate agents' capability of transforming high-level requirements into structured visual data collections through source discovery, acquisition, verification, and annotation. The benchmark contains 46 tasks across seven domains and requires agents to acquire images from public web resources without a predefined candidate pool. It also studies reference images as visual specifications for requirements that are difficult to express through text alone. We evaluate collected data using duplicate and provenance checks together with human-reviewed task-specific rubrics that separate base validity from additional quality. Experiments with five generalist agents (e.g., Codex, Claude Code) reveal a substantial gap between finding valid examples and constructing consistently valid collections. The strongest agent reaches 78.3% BPR (Base Pass Rate) under visual specification, yet only 45.7% of tasks contain five valid images. Visual specifications improve most agents, while fixed-backbone comparisons reveal substantial framework-dependent variation in collection quality and sourcing behavior. Larger collection requests yield more valid images overall but lower validity and quality at the largest scale, and multi-task execution further exposes differences in agent orchestration. DataResearchBench provides a testbed for studying how specification, framework, and workload shape reliable visual data construction. We will open-source the dataset and code and continuously update DataResearchBench to support emerging research needs across a broader range of data types.

Primary area:
datasets and benchmarks
Keywords:
Agent Evaluation, Open-Web Data Sourcing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gMXvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a benchmark for agents that construct visual data collections from the open web rather than retrieve answers or select images from a fixed pool. It contains 46 tasks across seven domains, evaluates text-only and reference-image specifications, and combines provenance checks with task-specific validity and quality rubrics. Experiments examine agent frameworks, collection size, and joint-task execution.

Critical strengths

The paper makes a worthwhile shift from evaluating an agent’s answer to evaluating the artifacts it acquires. Requiring images, source information, and factual annotations creates a concrete deliverable whose correctness can be inspected.

Critical weaknesses
  1. Collection-level success is not yet collection-level quality. The paper motivates data construction as a collection-level capability, but its principal metrics largely aggregate individual-image judgments. BPR measures the proportion of valid images, QS averages image-level quality, and TSR measures whether enough images pass. Duplicate checking introduces a relational constraint, but these metrics do not explicitly

  2. Candidate tasks are filtered using an OpenClaw agent powered by GPT-5.5, with tasks removed when their mean preliminary score exceeds 4.0. GPT-5.5 also drafts preliminary rubrics and performs final semantic evaluation, although humans subsequently refine the benchmark. This is defensible as a method for constructing a challenge set, but it conditions the benchmark on one system’s failures. Consequently, I would be cautious about interpreting the reported performance as representative reliability across research-driven data needs. This is a selection-bias concern, not evidence that the benchmark is invalid.

  3. Ultimately, I am concerned about the practical utility of the proposed evaluation framework. In my view, a benchmark should not only measure current performance but also provide actionable signals for improving models through training or at least expose model capability issues. The paper does not yet demonstrate whether its evaluation criteria serve these purposes.

Questions for authors

NA

4 comments

Response to Weakness 1.

We thank the reviewer for pointing out that our existing metrics do not explicitly capture the category-level coverage of an entire collection. We agree that evaluating individual-image validity and quality alone is insufficient to fully characterize the overall properties of a collection. To complement the existing metrics, we conducted an additional analysis of task-mentioned category coverage across the collected images.

Specifically, we constructed a category inventory for each task based on the categories explicitly mentioned in its original task description and task-specific Task Relevance criteria. Thirty tasks contain sufficiently explicit categories to support this analysis. For each task, we then attempted to map all images passing the Base criteria to the corresponding categories. We first averaged category coverage across the relevant dimensions within each task and then calculated the macro average across all applicable tasks, measuring the proportion of task-mentioned categories covered by at least one valid image. The results are shown below.

AgentVisual Spec. (%)Text-only (%)Difference (pp)
Claude Code74.3564.63+9.72
Codex v258.9360.37-1.44
Kimi Code50.2447.11+3.13
MiniMax42.0138.78+3.23
Grok70.3257.78+12.54

The results reveal clear differences in collection-level category coverage across agents. In most cases, providing reference images has a positive effect on coverage. This additional metric provides a complementary perspective to BPR, QS, and TSR, reflecting the extent to which a collection covers the categories specified by its task. We will incorporate this additional analysis into the revised manuscript, clarify the respective roles of image-level and collection-level evaluation, and further refine this evaluation in future work.

Response to Weakness 2.

We thank the reviewer for this insightful observation. We agree that using a single reference agent to filter candidate tasks introduces a certain degree of selection bias. To examine the extent of this bias and whether the resulting benchmark primarily reflects the failure patterns of the reference agent, we conducted an additional cross-agent analysis involving the five agents in our main experiments. The results are summarized below.

MetricVisual Spec.Text-only
Task-level disagreement (%)69.0573.17
Pairwise outcome disagreement (%)34.2935.12
Mean failure-set Jaccard (%)26.1338.60
Random Jaccard baseline (%)16.0421.26
Bidirectional cross-over pairs10/109/10

These results reveal both shared challenges and system-dependent failure patterns. The mean failure-set Jaccard is higher than the corresponding random baseline, indicating that different agents encounter some common difficult tasks. Meanwhile, bidirectional cross-over occurs in all 10 agent pairs under the Visual Specification setting and in 9 out of 10 pairs under the Text-only setting, showing that their failure sets are not simply nested. Therefore, the selected tasks expose performance differences across multiple agent systems rather than merely reproducing the failure patterns of the reference agent. We acknowledge that this additional analysis does not eliminate the original selection bias, but it further supports the value of DataResearchBench for cross-system evaluation.

Regarding the use of GPT-5.5 in rubric construction and semantic evaluation, we emphasize that GPT-5.5 was only used to draft the preliminary scoring criteria. These criteria were subsequently revised and cross-checked by three human reviewers until agreement was reached. During this process, at least one criterion was completely rewritten by human reviewers for the vast majority of tasks, and the proportion of criteria revised also exceeded half. We also report the agreement between the automated judge and human reviewers. Although these procedures provide additional human quality control, we acknowledge that residual model-related evaluation bias may still exist. Our current analysis further supports the value of DataResearchBench.

We will clarify the challenge-oriented scope of DataResearchBench, the role of reference-agent filtering, and the above limitations in the revised manuscript. We will also incorporate additional analyses and experiments to further address concerns regarding potential bias.

Response to Weakness 3.

We thank the reviewer for raising this important question. First, DataResearchBench employs three metrics, BPR, QS, and TSR@mm, to reveal different capabilities and limitations of agents. In addition, trajectory analysis reveals how agents allocate their efforts across planning, search, verification, acquisition, and quality assurance. Our Collection Size and Multi-Task experiments further challenge agents in long-horizon data collection and multi-task execution. The experimental results demonstrate that agent decisions are crucial to task performance. As further demonstrated in our response to Reviewer roT1's Weakness 2, we observe that Kimi exhibits relatively robust performance at larger collection sizes and benefits from subagent-based task decomposition in joint-task execution. These results expose several limitations of current agents and demonstrate that our benchmark can be used to evaluate and compare concrete agent-level improvement strategies. In our subsequent experiments, we have also observed substantial positive gains from improving agent strategies on these tasks.

Since retraining closed-source models or very large open-source backbones is often costly or difficult to implement, we believe that evaluating and improving agent-level strategies and frameworks provides a practical complementary direction. This is because agent capabilities emerge from the coupled interaction of the backbone's underlying capabilities, the harness, and the strategies governing how the backbone is utilized. DataResearchBench provides a common set of challenging tasks and evaluation criteria for investigating these differences and comparing potential improvements, while offering a natural long-horizon task setting.

Finally, we believe that open-web data collection has important practical relevance to emerging recursive self-improvement (RSI) workflows. An important component of such workflows is identifying current capability limitations and acquiring task-relevant data to address them, followed by data verification and organization, and ultimately using appropriate frameworks for model post-training. High-quality data is essential to this process. DataResearchBench directly evaluates whether agents can discover, acquire, verify, and organize such visual data resources from the open web. Its collection-size settings also provide a natural testbed for evaluating long-horizon data acquisition, while its multi-task setting examines coordination across multiple concurrent collection objectives.

Therefore, we believe that DataResearchBench not only reveals certain limitations of current agents but also provides practical research and evaluation value. We thank the reviewer for raising these concerns and will further clarify these practical implications in the revised manuscript.

Thanks for authors for the response. My score remain unchanged since I am still unconvinced for my weakness 3. The authors mention retraining closed or large open models can be costly. I agree with this, but post-trainning models at 7b or 13b with LORA should be very doable and would significantly strengthen the results, which I highly recommend the authors to pursue.

We agree that applying LoRA-based post-training to 7B or 13B open-source models is feasible and would provide a valuable way to demonstrate the practical utility of our benchmark. We appreciate this suggestion and believe that it is an important direction to explore.

At the same time, we believe that one important value of a benchmark is providing a systematic way to evaluate agents, identify their capability limitations, and reveal potential directions for improvement. Model post-training is one important application of such a benchmark, rather than a prerequisite for establishing its evaluation value. DataResearchBench already provides diagnostic signals through its complementary metrics and trajectory analysis, revealing where current agents struggle and how different execution strategies affect task outcomes. Furthermore, we applied the Dream-RSI[1] approach to five tasks and conducted additional experiments while changing only the model's execution strategy. All five tasks showed improvements in their scores, although the magnitude of improvement varied across tasks. This provides further empirical evidence that agent strategies can have a substantial impact on task performance.

Regarding the proposed LoRA experiments, we believe that effective post-training for open-web data sourcing involves several challenges beyond the parameter-efficient fine-tuning process itself. For SFT, constructing high-quality training trajectories is hard because these tasks are open-ended and admit multiple valid sourcing strategies. Directly using synthetic trajectories may introduce a bias toward particular execution paths, which are not necessarily optimal or generalizable. Their quality and effectiveness therefore need to be further validated on real-world web tasks. For RL, online rollouts introduce substantial runtime costs and are affected by network instability, tool availability, and changes in website content and structure. Even when synthetic environments or trajectories are used to reduce these costs, their effectiveness and generalization to real open-web environments still require validation.

We have also conducted preliminary evaluations using Qwen3.5-9B with the Qwen agent framework. The results were unsatisfactory, particularly on long-horizon tasks, where the agent struggled to complete the workflow and even failed to consistently satisfy the basic output-format requirements. This suggests that effective post-training may need to be conducted in stages, first establishing basic instruction-following and structured-output capabilities, and then improving the core data sourcing capabilities. Corresponding ablation studies would also be necessary to distinguish improvements in basic format compliance from genuine gains in search, verification, and collection quality. Otherwise, performance improvements after fine-tuning may not necessarily reflect improvements in the capabilities that DataResearchBench aims to evaluate.

We therefore view LoRA-based post-training as a valuable but distinct research direction that deserves dedicated investigation beyond the scope of the current benchmark study. Such work would involve not only fine-tuning but also training data construction, training strategy design, and careful evaluation of generalization in real-world environments. We fully agree with the reviewer that exploring this direction would further strengthen the practical utility of DataResearchBench, and we will pursue it in our future work. We sincerely thank the reviewer again for this valuable recommendation.

[1] Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Reviewer_roT1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

DataResearchBench evaluates agents on 46 visual data-sourcing tasks across seven domains, requiring image collections with sources and annotations. It studies reference-image guidance, agent frameworks, and collection workloads, revealing a substantial gap between finding valid individual images and consistently delivering complete, valid collections.

Critical strengths
  1. The benchmark addresses a meaningful data-preparation capability. Acquiring usable image collections requires sustained attention to relevance, quality, provenance, and annotations. The distinction between sample-level validity and collection-level success is particularly useful: the strongest main agent achieves 78.3% image validity, but only 45.7% of tasks contain five valid images. This exposes an important limitation in reliable data collection.

  2. The evaluation protocol and controlled comparisons provide a solid foundation. The benchmark combines provenance auditing, duplicate detection, human-reviewed task rubrics, and collection-level metrics. Unified tools and fixed-backbone framework comparisons help investigate differences beyond model identity. For example, with GPT-5.6-Sol under text-only instructions, BPR varies from 58.3% for Codex to 65.2% for OpenClaw, motivating further study of sourcing and verification strategies.

Critical weaknesses
  1. Uncontrolled resource budgets limit the framework comparisons. Tasks impose no limits on time, tokens, tool calls, or expenditure. Consequently, higher performance may partly reflect greater resource consumption, even with the same backbone. The reported cost analysis also excludes external-tool costs. Budget-matched comparisons are needed to establish the relative effectiveness of different frameworks.

  2. The scaling and coordination findings have limited generality and causal support. The collection-size study covers one agent and seven tasks. In the joint-task experiment, observing delegation alongside improved performance does not isolate delegation’s contribution. Broader evaluation and within-system delegation ablations would strengthen conclusions about scaling and task coordination.

Questions for authors
  1. With fixed backbones and matched budgets, do the framework rankings persist when both model and external-tool costs are included?
  2. Do the scaling trends hold across more tasks and agents, and does enabling delegation improve joint-task performance within the same system?
2 comments

Response to Weakness 1 and Question 1.

We thank the reviewer for this insightful comment. Our original unconstrained setting was designed to investigate agents' autonomous resource-allocation decisions in open-ended data sourcing tasks. This allows us to observe the resource-management behaviors of different agent frameworks without imposing an external execution policy. Nevertheless, we agree that evaluating frameworks under a shared resource budget provides an important complementary perspective.

To investigate this, we conducted an additional experiment using the same Kimi K3 backbone across Kimi Code, Pi, and OpenClaw under Visual Specification. We imposed a 1M-token budget per task and prompted agents to consolidate their collected results and produce the final output when token consumption reached 90% of the budget. To reduce the substantial runtime overhead, we omitted the time-consuming web-based provenance verification step and report the raw base-pass rate and high-quality score.

FrameworkBudgetBPR (%)Hi-Q (%)
Kimi CodeUnconstrained58.2651.95
Kimi CodeLimited43.4839.50
PiUnconstrained51.7447.57
PiLimited51.7448.59
OpenClawUnconstrained61.7455.18
OpenClawLimited32.1729.73

The ranking change is closely related to their different resource consumption patterns. Pi's unconstrained execution already stays within the 1M-token budget, whereas Kimi Code and OpenClaw exceed this threshold, with OpenClaw incurring the highest model-token cost among the three. Consequently, the budget constrains OpenClaw and Kimi Code more substantially, explaining their larger performance drops. External-tool costs are relatively low in our setting and are therefore omitted from this supplementary analysis. Overall, framework rankings can vary across different settings, depending on resource requirements.

Response to Weakness 2 and Question 2.

We thank the reviewer for this valuable comment. We conducted two additional experiments to investigate these aspects. As in the previous supplementary experiment, web-based provenance verification was omitted due to its substantial runtime overhead, and we report the resulting raw metrics.

Collection-size scaling.

We extended the collection-size experiment to a broader set of agents/frameworks, including Kimi, Grok, OpenClaw, and Pi.

AgentN=5 (BPR / Hi-Q)N=20 (BPR / Hi-Q)N=50 (BPR / Hi-Q)
Kimi71.43 / 58.2955.00 / 43.0061.14 / 53.81
Grok68.57 / 55.4351.43 / 45.0049.14 / 41.28
OpenClaw71.43 / 59.6256.43 / 44.8145.14 / 35.64
Pi65.71 / 55.2458.57 / 47.8141.43 / 35.49

The results show that BPR and Hi-Q at N=50 are lower than at N=5 across all four agents, although their scaling trajectories differ. Notably, Kimi, which employs a subagent strategy at N=50, partially recovers its performance, whereas the other three agents continue to decline. We also repeated the Codex scaling experiment and reproduced its performance improvement at N=20. In contrast, Pi and OpenClaw, despite sharing the GPT-5.6-Sol backbone with Codex, do not exhibit the same pattern. These findings suggest that scaling behavior is not strictly monotonic and depends substantially on the agent and its framework. Our seven selected tasks cover all seven benchmark domains, while their average completion performance varies across agents. In future work, we will progressively extend the experiments to all benchmark tasks and larger collection sizes.

Delegation ablation.

To directly examine the contribution of delegation, we conducted a within-system ablation of Kimi on the same 45 tasks under Joint-5 execution, comparing configurations with and without subagents, as well as a configuration using another framework (Pi).

ExperimentBPR (%)QS (%)TSR@1 (%)TSR@2 (%)TSR@3 (%)TSR@4 (%)TSR@5 (%)
Kimi (Single-task)58.6752.2184.4477.7862.2244.4424.44
Kimi (Joint-5)60.0055.3784.4473.3362.2251.1128.89
Kimi (Joint-5, no subagents)56.0047.4786.6771.1160.0044.4417.78
Kimi (Joint-5, Pi)57.7852.2384.4475.5666.6740.0022.22

The results show that enabling subagents increases BPR by 4.00 percentage points and QS by 7.90 points compared with the configuration without subagents. The improvements are particularly evident at higher TSR thresholds, with TSR@4 increasing by 6.67 points and TSR@5 by 11.11 points. Meanwhile, Kimi retains good multi-task performance even without subagents, which is also reflected in its performance under the Pi framework. Together with Kimi's relatively stable performance at N=50, these results support the positive role of subagent-based task decomposition in multi-task coordination and large-scale data collection. We also highlight Kimi's strong performance in multi-task and large-scale settings.

We will add relevant experiments and further discussion of these two aspects in the revised manuscript.

Reviewer_M3oKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduce DataResearchBench, which evaluates how well models on agentic visual data research from open web resources. Experiments show 4 findings inlcuding imbalanced improvements from visual specifications, framework-dependent differences on performance, etc.

Critical strengths
  • The topic of agentic visual data research is a very practical and important topic, which can be a bottleneck in real-world applications of agent systems.
Critical weaknesses
  • According to figure 1, the tasks primarilly derives from acedemic papers, this may introduce some bias toward acedemic-oriented collection needs, while ignoring other subject domains like commercial, operational and creative scenarios.

  • The reference-agent run is employed in filtering, and subsequent curation demands public-web feasibility, clear requirements, and verifiability. Could these steps seamlessly exclude genuinely challenging but realistic scenarios, leaving tasks that are more agent-friendly (easier for agents to accomplish)?

Questions for authors

please address the weaknesses first.

1 comment

Response to Weakness 1.

We thank the reviewer for pointing out the potential domain bias arising from the fact that our tasks are primarily derived from academic papers. We acknowledge this concern. We would like to clarify that academic papers serve as the primary source for identifying data requirements during benchmark construction, but this does not mean that the resulting tasks are limited to academic research. For example, domains such as Business & Economics, Art & Design, and Entertainment & Culture already cover various visual data collection needs beyond traditional academic research. Therefore, the sources from which tasks are derived and the application domains covered by the resulting tasks are not equivalent concepts.

More importantly, DataResearchBench focuses on a fundamental capability in broader data-driven workflows, like discovering, acquiring, verifying, and organizing visual data from the open web. The resulting collections can serve as foundational data resources for various downstream applications. For example, in scenarios such as business analysis, operational workflows, and creative production, the acquired raw visual data can be further classified, filtered, processed, and organized to meet specific application requirements. Therefore, our goal is to evaluate this foundational data acquisition capability, which has cross-scenario reusability, rather than directly cover the complete workflows of all downstream application scenarios.

Nevertheless, we acknowledge that the current task collection does not yet fully reflect the diversity of data collection needs in non-academic scenarios. We will further clarify the scope and limitations of our benchmark in the revised manuscript and consider introducing more tasks directly derived from real-world industry scenarios in future extensions.

Response to Weakness 2.

We thank the reviewer for raising this important question. We agree that task selection and manual curation criteria may influence the distribution of tasks in the final benchmark. Therefore, it is important to clarify the purposes of these procedures and their relationship to task difficulty.

First, we would like to clarify the actual filtering direction of the Reference-agent Filtering stage. We used OpenClaw (powered by GPT-5.5) to execute all 738 candidate tasks and scored the returned images based on five preliminary Base Criteria. Higher scores indicate better satisfaction of the basic requirements. We removed tasks whose average scores exceeded 4.0, meaning that the reference agent was already able to complete these tasks relatively well. Thus, this filtering step was designed to exclude tasks that the reference agent could already handle effectively, rather than to retain easy tasks, allowing the subsequent human curation stage to focus more on challenging data acquisition requirements.

Second, regarding the criteria of Public-web Feasibility, Clear Requirements, and Verifiability used during human curation, we emphasize that these criteria were assessed manually during task construction. Their primary purpose was to ensure task executability and evaluation validity, rather than to determine whether tasks were easy for the reference agent to complete. Therefore, clear requirements and verifiable results do not necessarily imply simple execution.

Finally, the existing experimental results demonstrate that the retained tasks still pose substantial challenges to agents, particularly when agents are required to collect all five valid images for a task. Furthermore, the Collection Size experiment imposes increasingly demanding collection requirements. These results indicate that consistently acquiring a sufficient number of valid images remains a challenging capability, which is precisely one of the aspects that DataResearchBench aims to evaluate.

In summary, task filtering and human curation are primarily intended to ensure the executability of data collection tasks and the evaluability of their outcomes, rather than to systematically simplify tasks into ones that agents can easily complete. We will further clarify the roles of these selection criteria and distinguish them from task difficulty in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.