acceptodds
Under review as a conference paper at ICLR 2027

DataClawBench: An Agent Benchmark for Exploratory Real-World Financial Data Analysis

Abstract

Realistic analysis often begins without knowing where the relevant evidence lies, raising a fundamental question for autonomous data analysis: can agents discover and use the required evidence with limited human guidance? However, existing benchmarks typically evaluate such agents in human-guided settings, providing selected data sources, explicit data schemas, or cleaned data, thereby understating the exploratory burden. To evaluate autonomous exploratory analysis in a more realistic setting, we introduce DataClawBench, a benchmark built from financial think tank consulting scenarios where agents must independently explore unfamiliar, noisy, cross-domain data and produce verifiable conclusions. DataClawBench provides a unified real-world data environment containing approximately 2.06 million records across enterprise, industry, and policy domains while preserving native data imperfections. On this environment, it defines 492 multi-step cross-domain tasks that withhold data source hints, complete schema documentation, and explicit noise descriptions. To reveal exploratory progress and failure modes beyond final outcomes, we annotate critical intermediate milestones and introduce a process-oriented evaluation framework. A systematic evaluation of ten advanced LLMs under the OpenClaw agent reveals that exploratory data analysis breaks agent reliability: more exploration does not reliably translate into task-relevant progress or correct final answers

Primary area:
datasets and benchmarks
Keywords:
Autonomous Agent, Data Analysis, long-horizon task benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_32NFOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for evaluating data agents in an unfamiliar financial data environment. It contains 492 tasks across seven categories and approximately 2.06 million records spanning enterprise, industry, and policy data. Expert-annotated answers, reference steps, and intermediate milestones support outcome accuracy, execution efficiency, and process-level evaluation through Goal Progress Rate (GPR) and Temporal Progress Efficiency (TPE). Ten LLMs are evaluated, with the strongest achieving 68.3% overall accuracy. Validation includes environmental interventions, human–judge agreement studies, cross-judge comparisons, and trajectory case studies.

Critical strengths

None

Critical weaknesses
  1. The Milestone Coverage Rate in §4.3 checks whether the independently identified indispensable results satisfy Ct⊆GtC_t\subseteq G_t. This establishes coverage, but does not exclude unnecessary, path-specific milestones in GtG_t. For example, Gt=Ct∪{r}G_t=C_t\cup\{r\} passes the coverage test even if a valid solution never needs rr.

  2. Table 5 computes GPR and TPE only on tasks that each model answers incorrectly. Consequently, each model is evaluated on a different subset, and a stronger model may fail mainly on intrinsically harder tasks, while a weaker model also fails on easier ones.

  3. Figure 2a jointly removes noise and irrelevant sources, then adds schema guidance. This design cannot distinguish the contribution of cleaning from that of reducing the search space. Moreover, tasks are selected because they previously failed, so the native-environment baseline is zero by construction.

Questions for authors
  1. Case 2 states that all four indicators are min–max normalized before weighting. It then attributes a change in the winning province to using 602 instead of 80 as the common national denominator for policy intensity. Let pip_i be province ii's relevant policy count and d>0d>0 the common denominator. For a fixed set of provinces with nonconstant counts,

    =\frac{p_i-\min_j p_j}{\max_j p_j-\min_j p_j}.$$ The denominator cancels. Replacing 80 with 602 therefore cannot compress the normalized policy contribution, change its effective weight, or reverse the ranking. The statements at lines 1637–1645 are incompatible with the specified calculation.
5 comments

Thank you for your comments. We believe W1, W2, and Q1 involve some misunderstandings, which we clarify below. For W3, we conducted additional experiments to address the request for more detailed causal analysis. Please feel free to reach out if any further clarification is needed.

W1. Does Milestone Coverage Rate Exclude Redundant Milestones?\color{blue}{\text{W1. Does Milestone Coverage Rate Exclude Redundant Milestones?}}

Reviewer: The Milestone Coverage Rate in §4.3 checks whether the independently identified indispensable results satisfy Ct⊆GtC_t\subseteq G_t. This establishes coverage, but does not exclude unnecessary, path-specific milestones in GtG_t. For example, Gt=Ct∪rG_t=C_t\cup r passes the coverage test even if a valid solution never needs rr.

Clarification of Milestone Coverage Rate.\color{blue}{\text{Clarification of Milestone Coverage Rate.}} We believe this weakness reflects a misunderstanding of our evaluation design. Milestone Coverage Rate measures the coverage of essential intermediate results, rather than proving the absence of redundant milestones. The gold milestone set may contain redundant milestones relative to individual valid paths. This is an intentional part of our milestone judgment design.

We improve evaluation consistency and robustness across different solution paths by combining high milestone coverage with a judgment mechanism based on logical and temporal dependencies. Specifically, milestones can be achieved directly or inferred from correctly completed downstream milestones (L161–162, L1516–1526). Together with high coverage of critical intermediate results, this mechanism reduces missed milestone achievements caused by different solution paths.

Lack of a Specific Concern.\color{blue}{\text{Lack of a Specific Concern.}} The review points out the possible existence of redundant milestones but does not identify any concrete negative impact of such redundancy. Therefore, we find it difficult to determine the specific concern being raised.

W2. Different Difficulty Distributions Among Failed Tasks\color{blue}{\text{W2. Different Difficulty Distributions Among Failed Tasks}}

Reviewer: Table 5 computes GPR and TPE only on tasks that each model answers incorrectly. Consequently, each model is evaluated on a different subset, and a stronger model may fail mainly on intrinsically harder tasks, while a weaker model also fails on easier ones.

Clarification of the Evaluation Design.\color{blue}{\text{Clarification of the Evaluation Design.}} We believe this weakness also reflects a misunderstanding of our evaluation design. First, GPR and TPE in Table 5 measure conditional progress on each model's own failed tasks. Evaluating different subsets of failed tasks is therefore inherent to their intended purpose. Second, the review overlooks our evaluation across predefined difficulty levels. Tasks within the same difficulty level fall within a comparable range of difficulty.

Lack of a Specific Concern and Logical Connection.\color{blue}{\text{Lack of a Specific Concern and Logical Connection.}} The observation that "a stronger model may perform well on harder tasks while a weaker model performs poorly on easier tasks" appears disconnected from the preceding criticism. More importantly, the review does not identify a concrete negative consequence of this observation. We therefore do not consider it a well supported weakness.

W3. More Detailed Analysis of Environmental Factors\color{blue}{\text{W3. More Detailed Analysis of Environmental Factors}}

Reviewer: Figure 2a jointly removes noise and irrelevant sources, then adds schema guidance. This design cannot distinguish the contribution of cleaning from that of reducing the search space. Moreover, tasks are selected because they previously failed, so the native-environment baseline is zero by construction.

This concern requests finer causal analysis beyond the claims made in our paper. Following the reviewer's suggestion, we conducted additional experiments that separately remove noisy evidence and irrelevant data sources.

The results show that agents are affected by multiple sources of uncertainty, including noisy evidence, irrelevant data sources, and incomplete schema understanding. In our data environment, noisy evidence has a greater impact than irrelevant data sources.

ModelOriginal DataWithout Noisy EvidenceWithout Irrelevant SourcesWithout BothWith Schema Guidance
Qwen3.5-Plus0%13.3%7.76%17.77%35.00%
GLM50%23.9%7.8%28.33%48.33%

Q1. Clarification of Case 2.\color{blue}{\text{Q1. Clarification of Case 2.}}

Reviewer: Replacing 80 with 602 therefore cannot compress the normalized policy contribution, change its effective weight, or reverse the ranking. The statements at lines 1637–1645 are incompatible with the specified calculation.

The statement in L1637–1645 is consistent with the specified calculation. The reviewer's denominator cancellation formula is correct when the pharmaceutical policy counts for each province remain fixed and only the common national denominator changes. However, the Agent's industry filtering error changed both the national total and the pharmaceutical policy counts for individual provinces.

Specifically, let pip_i denote the number of pharmaceutical policies in province ii, and bib_i denote the number of other policies. The Agent incorrectly used pi+bip_i+b_i as the pharmaceutical policy count for each province. Therefore, its normalized policy score becomes

\frac{p_i+b_i-\min_j(p_j+b_j)} {\max_j(p_j+b_j)-\min_j(p_j+b_j)}.$$ Although the common denominator cancels out, including unrelated policies changes the relative distribution of policy counts across provinces. Thus, the normalized differences between provincial scores can still change. In Case 2, Shanghai's policy score advantage over Jiangsu decreases from 0.181818 to 0.081356. The original description summarized this error as a change in the national denominator without explicitly explaining that the provincial numerators and their distribution also changed. **We have revised the explanation in Case 2 to clarify the actual source of the ranking change.**
AuthorsOpenReview ↗

Follow-up on Our Responses and Further Experimental Analysis

Dear Reviewer 32NF,

Thank you for taking the time to review our submission and provide your feedback. As the discussion period draws to a close, we would like to kindly follow up on our responses and additional clarifications.

We have clarified the milestone evaluation mechanism and the intended interpretation of GPR/TPE. We also conducted additional experiments to separately examine the effects of noisy evidence and irrelevant data sources, and clarified the calculation issue in Case 2.

We would greatly appreciate it if you could revisit our responses and let us know whether these clarifications address your concerns. If you find our responses satisfactory, we would sincerely appreciate your consideration in updating your evaluation.

Thank you again for your time and consideration.

Best regards,

The Authors of Submission 29107

Reviewer_FnumOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DataClawBench, a benchmark for evaluating autonomous agents on exploratory financial data analysis with limited data guidance, incomplete schema information, and noisy real-world inputs. It comprises 492 tasks across seven categories, built on approximately 2.06 million real-world records spanning enterprise, industry, and policy domains. The benchmark introduces milestone-based metrics, GPR and TPE, to assess intermediate progress and diagnose failures beyond final-answer accuracy. Experiments with ten LLMs using OpenClaw show that exploratory data analysis remains challenging, with the best model achieving 68.3% accuracy.

Critical strengths

1.The paper is well-presented with clear figures.

2.The paper provides a thorough validation of evaluation reliability, including human annotation agreement, cross-judge consistency, and milestone coverage audits.

3.The benchmark offers a comprehensive evaluation of agent performance across different task types and complexities.

Critical weaknesses

1.The data environment consists of only 18 files, primarily in CSV format, which limits the diversity of data formats and storage systems represented in the benchmark.

2.The novelty of DataClawBench is not sufficiently established. The paper argues that existing benchmarks largely rely on human-guided data settings, but overlooks recent works such as KramaBench (ICLR 2026) and LakeQA (ICML 2026), which already evaluate autonomous data exploration under limited guidance.

3.The process evaluation metrics contain flaws. GPR and TPE rely on predefined gold milestones, but different valid solution paths may involve different intermediate results. Although the authors conduct a Milestone Coverage Audit, it only verifies whether the annotated milestones cover essential intermediate results, without establishing robustness across different reasoning paths. Moreover, GPR and TPE are only evaluated on failed tasks, overlooking cases where agents reach correct answers through flawed reasoning.

Questions for authors

1.How long does it take to run the full DataClawBench benchmark, and how many tasks are executed in parallel?

2.Since the gold answers and milestones are constructed in a cleaned data environment, while agents are evaluated on the original noisy data, how do the authors ensure that all tasks remain solvable and that the required evidence can be recovered from the original environment?

2 comments

We thank you for recognizing the paper's clear presentation, broad experimental coverage, and the value of human agreement validation and milestone evaluation. We address the three main concerns and two questions below.

W1. Limited Diversity of Data Formats and Storage Systems\color{blue}{\text{W1. Limited Diversity of Data Formats and Storage Systems}}

Reviewer: The data environment consists of only 18 files, primarily in CSV format, which limits the diversity of data formats and storage systems represented in the benchmark.

Why We Use a Unified CSV Format:\color{blue}{\text{Why We Use a Unified CSV Format:}} DataClawBench primarily evaluates autonomous data discovery and evidence analysis across domains, rather than file format parsing or storage system adaptation. We therefore organize the data into 18 CSV files based on their semantic content. This controls differences in file parsing and focuses the evaluation on autonomous exploration without source hints, complete schema documentation, or noise descriptions.

Convert to the Original Format:\color{blue}{\text{Convert to the Original Format:}} The 18 CSV files indeed limit the diversity of data formats and storage backends, but this does not imply a lack of diversity in the underlying data or analytical tasks. Our 2.06 million records originate from 12,119 source files, including 9,965 enterprise reports (PDF), 19 MySQL tables (SQL), 633 policy documents (HTML), and 1,502 policy attachments (PDF, DOC, XLS). These sources contain structured, partially structured, and unstructured information. Users who require greater diversity in data formats or storage systems can convert the data into PDFs, databases, or other storage formats without changing the benchmark tasks or evaluation criteria. This flexibility is an advantage of our evaluation framework.

W2. Insufficient Articulation of Novelty Due to Missing Related Work\color{blue}{\text{W2. Insufficient Articulation of Novelty Due to Missing Related Work}}

Reviewer: The paper overlooks recent works such as KramaBench (ICLR 2026) and LakeQA (ICML 2026), which already evaluate autonomous data exploration under limited guidance.

Related Work:\color{blue}{\text{Related Work:}} KramaBench and LakeQA are important related works. We have added discussions of these studies in the revised Abstract (L15–18), Introduction (L43–52), and Related Work (L478–485). We have also revised statements that might imply autonomous exploration was first introduced by our work and clarified our core novelty.

Novelty Clarification:\color{blue}{\text{Novelty Clarification:}} Although recent benchmarks have begun to evaluate autonomous exploratory analysis, they mainly show that agents still struggle to complete such tasks reliably. They offer limited insight into a more fundamental question: How effectively do data agents turn exploration into progress? To investigate the exploration–progress relationship in a complex, realistic setting, we introduce DataClawBench, which combines financial think tank consulting tasks with a trajectory-aligned semantic progress evaluation framework. The core novelty of DataClawBench is not the introduction of autonomous data exploration, but the quantitative evaluation of how exploration translates into correct, timely progress toward the original task goal during autonomous execution.

W3. Milestone Robustness Across Solution Paths and Evaluation of Failed Tasks\color{blue}{\text{W3. Milestone Robustness Across Solution Paths and Evaluation of Failed Tasks}}

Reviewer: GPR and TPE rely on predefined gold milestones, but different valid solution paths may involve different intermediate results. Although the authors conduct a Milestone Coverage Audit, it only verifies whether the annotated milestones cover essential intermediate results, without establishing robustness across different reasoning paths.

Milestone Judgment Across Solution Paths:\color{blue}{\text{Milestone Judgment Across Solution Paths:}} We improve evaluation consistency across different solution paths by combining gold milestones with high coverage and a judgment mechanism based on logical and temporal dependencies. Gold milestones capture critical intermediate results and their logical and temporal dependencies. Therefore, correctly achieving a downstream milestone also implies the achievement of its prerequisite milestones, even if they are not explicitly recorded (L161–162, L1516–1526). Combined with high coverage of critical intermediate results, this mechanism reduces missed milestone achievements caused by differences in solution paths. We have clarified this design in the revised paper (L163–168).

Additional Validation of Robustness Across Solution Paths:\color{blue}{\text{Additional Validation of Robustness Across Solution Paths:}} Our original Milestone Coverage Audit verifies that the gold milestones cover essential intermediate results across different paths. To further validate judgment robustness, we conducted an additional human validation experiment. Two annotators independently examined the same trajectories used in the coverage audit. They labeled each gold milestone as directly achieved, indirectly achieved, or not achieved, and calculated the corresponding GPR scores. We then compared the proportion of trajectories whose GPR scores agreed with human annotations. The results show that the high coverage of critical intermediate results and the judgment mechanism based on logical dependencies jointly support robustness progress evaluation across different valid solution paths.

Judgment SettingGPR Agreement
Direct Only24.5%
Direct + Indirect91.75%

Reviewer: GPR and TPE are only evaluated on failed tasks, overlooking cases where agents reach correct answers through flawed reasoning.

Evaluating GPR and TPE only on failed tasks is an intentional choice based on their purpose. Under the current milestone dependency rules, trajectories with correct final answers are treated as having achieved all target milestones. Therefore, GPR and TPE are mainly used to distinguish different levels of meaningful progress among failed tasks.

We have acknowledged in Limitations (L1805-1808) that this design cannot identify cases where agents reach correct final answers through flawed intermediate reasoning. To reduce such cases, we design tasks with sufficiently large answer spaces and avoid unnecessary binary questions, thereby reducing the likelihood of obtaining correct answers through guessing.

Q1. Full Evaluation Time and Parallel Execution\color{blue}{\text{Q1. Full Evaluation Time and Parallel Execution}}

Reviewer: How long does it take to run the full DataClawBench benchmark, and how many tasks are executed in parallel?

The time required for a complete evaluation run varies depending on model configurations and provider service latency. For example, evaluating all 492 tasks, takes approximately 4 hours for Claude Opus 5 with 10 concurrent tasks, and approximately 15.5 hours for GPT-5.6-SOL with one task executed at a time.

Q2. Task Solvability in the Original Noisy Environment\color{blue}{\text{Q2. Task Solvability in the Original Noisy Environment}}

Reviewer: Since the gold answers and milestones are constructed in a cleaned data environment, while agents are evaluated on the original noisy data, how do the authors ensure that all tasks remain solvable and that the required evidence can be recovered from the original environment?

The information relevant to each task is preserved in both the original evaluation data and the cleaned data used for annotation, ensuring that the required evidence is available.

Specifically, human experts first clean and organize the original data used for evaluation. They then perform the analysis and annotate gold answers and critical intermediate results. Cleaning changes the organization and accessibility of the data, but does not introduce evidence absent from the original environment. Therefore, the experts' data processing and analysis provide a reproducible solution path.

Reviewer_CH22OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The benchmark evaluates agents on 492 financial analysis tasks requiring discovery, cleaning, and integration of unfamiliar data, while milestones are assessed alongside final-answer scoring.

Critical strengths

S1: Real-world data. S2: Important problem. S3: Intermediate milestones help distinguish partial progress from complete failure.

Critical weaknesses

W1: The novelty discussion misses relevant work. KramaBench (https://arxiv.org/pdf/2506.06541, ICLR 2026, arxiv'ed in June 2025) already evaluates discovery, cleaning, integration, and analysis over real-world data lakes, without human guidance. KramaBench also has a sub-task design with intermediate answers, almost identical to your reference steps/milestones. I'm very surprised that the related work missed that.

Other than that, the related work section needs a significant update. It's unclear to me why the following benchmarks were not discussed in the paper. It feels that the novelty comparison still lives in 2025. It's unclear what new capability is tested beyond the financial setting and particular metrics.

W2: I stumbled upon Table 9 and got very confused by the model settings. Why are the LLMs set to different thinking modes, especially with thinking off for GPT 5.4? If this factor is not controlled for in the experiments, how could we draw any meaningful conclusion? This could substantially affect scores, rankings, and conclusions about the systems' behaviour.

W3: Why are numerical answers also evaluated by an LLM judge? I understand using an LLM to assess textual answers, but many questions have numerical answers (e.g., in your example, what's the ratio...) that could be checked programmatically. Using an LLM for these comparisons introduces an unnecessary source of scoring error. The metric part of the benchmark is extremely important, and its current state lacks the needed care. How are things like rounding and equivalent units handled? And what if the answer is a set (of either strings or numbers)? It's hard to judge whether the reported accuracy is accurate.

Questions for authors

Please address my questions in W1-3.

5 comments

We thank you for recognizing the value of our real-world data, the importance of our research question, and the role of milestones in process evaluation. We address W1–W3 below. Please feel free to reach out if any further clarification is needed.

W1. Insufficient Articulation of Novelty Due to Missing Related Work\color{blue}{\text{W1. Insufficient Articulation of Novelty Due to Missing Related Work}}

Novelty Clarification:\color{blue}{\text{Novelty Clarification:}} Although recent benchmarks have begun to evaluate autonomous exploratory analysis, they mainly show that agents still struggle to complete such tasks reliably. They offer limited insight into a more fundamental question: How effectively do data agents turn exploration into progress? To investigate the exploration–progress relationship in a complex, realistic setting, we introduce DataClawBench, which combines financial think tank consulting tasks with a trajectory-aligned semantic progress evaluation framework. The core novelty of DataClawBench is not the introduction of autonomous data exploration, but the quantitative evaluation of how exploration translates into correct, timely progress toward the original task goal during autonomous execution.

Related Work:\color{blue}{\text{Related Work:}} KramaBench, DAComp, CoDA-Bench, and LakeQA are important related works. We have added discussions of these studies in the revised Abstract (L15–18), Introduction (L43–52), and Related Work (L478–485). We have also revised statements that might imply autonomous exploration was first introduced by our work.

Reviewer: KramaBench already evaluates discovery, cleaning, integration, and analysis over real-world data lakes, without human guidance. KramaBench also has a sub-task design with intermediate answers, almost identical to your reference steps/milestones.

We recognize that KramaBench already evaluates autonomous data discovery, cleaning, integration, and analysis over real-world data lakes. The key distinction lies in what is evaluated. KramaBench evaluates Sub-task performance by separately providing explicit subquestions and corresponding data subsets, rather than assessing progress within the original autonomous execution. In contrast, DataClawBench directly evaluates actual end-to-end execution trajectories to determine which critical intermediate results are correctly achieved and when. This enables us to examine exploration efficiency and cascading failures, which are difficult to capture through independent subtask evaluation.

Reviewer: It's unclear what new capability is tested beyond the financial setting and particular metrics.

DataClaw measures the effectiveness, efficiency, and failure modes of agent exploration. It further characterizes the exploration–progress gap (L338–368) and shows that exploration failures arise not only from incorrect actions, but also from agents failing to recognize when their actions have invalidated their plans (L408–431).

W2. Different Thinking Configurations\color{blue}{\text{W2. Different Thinking Configurations}}

Reviewer: Why are the LLMs set to different thinking modes, especially with thinking off for GPT 5.4?

The Thinking settings in Table 9 follow the default reasoning configurations of the respective APIs (L797), consistent with GameplayQA (ACL 2026) [1] and MCPMark (ICLR 2026) [2]. Model performance can vary with proprietary parameters such as temperature and Thinking intensity. Moreover, identically named Thinking levels are not necessarily comparable across models. Due to computational constraints, standardizing these proprietary parameters across models is difficult. We therefore use common API provider recommended settings, keep other Agent framework configurations consistent, and explicitly document the settings used for each evaluation.

[1] GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents. https://aclanthology.org/2026.acl-long.1585.pdf

[2] MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use. https://arxiv.org/abs/2509.24002

Reviewer: If this factor is not controlled for in the experiments, how could we draw any meaningful conclusion?

Our main finding is that agent exploration does not always translate into correct answers or meaningful progress in the evaluated environment. This research question and the diagnostic value of our benchmark do not depend on interpreting model rankings as absolute measures of capability.

W3. Reliability of LLM-as-Judge\color{blue}{\text{W3. Reliability of LLM-as-Judge}}

Reviewer: Why are numerical answers also evaluated by an LLM judge? ...could be checked programmatically.

Our primary reason for using LLM-as-Judge is not the difficulty of comparing numerical values, but the systematic errors that can arise when extracting answers from Agent outputs [3, 4].

Programmatic comparison is straightforward for standardized numerical answers. However, agents may not strictly follow the required output format. Fixed parsing rules can therefore extract answers incorrectly and misjudge correct results. An LLM judge can mitigate such errors through semantic understanding, especially for complex answers involving different units, mixed text and numbers, or sets containing multiple elements.

[3] Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering. https://aclanthology.org/2025.findings-acl.950.pdf
[4] Maslab: A unified and comprehensive codebase for llm-based multi-agent systems. https://arxiv.org/abs/2505.16988.

Reviewer: How are things like rounding and equivalent units handled?

We ensure rigorous scoring through explicit output requirements. Each task includes Output guidelines (L1433) specifying units, significant digits, rounding, and output formats.

Reviewer: And what if the answer is a set (of either strings or numbers)? It's hard to judge whether the reported accuracy is accurate.

For answers containing multiple parts, the judge evaluates each part against its corresponding gold answer using explicit Rubric rules (L1469), rather than assigning a single subjective score.

For example, given the gold answer ["Large", 166.33], each part receives a binary score of 0 or 1, and the total score is their average. Outputs that cannot be parsed into comparable parts receive a score of 0 according to the scoring rules. Independent human validation in Table 8 further supports the reliability of our judge.

AuthorsOpenReview ↗

Follow-up on Our Responses and Additional Clarifications

Dear Reviewer CH22,

Thank you again for your thoughtful feedback on our submission. As the discussion period draws to a close, we would like to kindly follow up on our response and revisions.

We have substantially expanded the related work discussion to include KramaBench, DAComp, CoDA-Bench, and LakeQA, and clarified the distinction between our trajectory-level evaluation and existing subtask-based benchmarks. We have also clarified the rationale behind the model-specific default reasoning configurations and provided further explanations of our LLM-based evaluation, including answer extraction, numerical precision, unit handling, and multi-part answer scoring, supported by independent human validation.

We would greatly appreciate it if you could take another look at our responses and let us know whether these clarifications adequately address your concerns, particularly regarding the novelty and evaluation reliability of DataClawBench. If you find our responses satisfactory, we would sincerely appreciate your consideration in updating your evaluation accordingly.

Thank you very much for your time and consideration.

Best regards,

The Authors of Submission 29107

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.