- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a benchmark for evaluating data agents in an unfamiliar financial data environment. It contains 492 tasks across seven categories and approximately 2.06 million records spanning enterprise, industry, and policy data. Expert-annotated answers, reference steps, and intermediate milestones support outcome accuracy, execution efficiency, and process-level evaluation through Goal Progress Rate (GPR) and Temporal Progress Efficiency (TPE). Ten LLMs are evaluated, with the strongest achieving 68.3% overall accuracy. Validation includes environmental interventions, human–judge agreement studies, cross-judge comparisons, and trajectory case studies.
Critical strengths
None
Critical weaknesses
-
The Milestone Coverage Rate in §4.3 checks whether the independently identified indispensable results satisfy . This establishes coverage, but does not exclude unnecessary, path-specific milestones in . For example, passes the coverage test even if a valid solution never needs .
-
Table 5 computes GPR and TPE only on tasks that each model answers incorrectly. Consequently, each model is evaluated on a different subset, and a stronger model may fail mainly on intrinsically harder tasks, while a weaker model also fails on easier ones.
-
Figure 2a jointly removes noise and irrelevant sources, then adds schema guidance. This design cannot distinguish the contribution of cleaning from that of reducing the search space. Moreover, tasks are selected because they previously failed, so the native-environment baseline is zero by construction.
Questions for authors
-
Case 2 states that all four indicators are min–max normalized before weighting. It then attributes a change in the winning province to using 602 instead of 80 as the common national denominator for policy intensity. Let be province 's relevant policy count and the common denominator. For a fixed set of provinces with nonconstant counts,
=\frac{p_i-\min_j p_j}{\max_j p_j-\min_j p_j}.$$ The denominator cancels. Replacing 80 with 602 therefore cannot compress the normalized policy contribution, change its effective weight, or reverse the ranking. The statements at lines 1637–1645 are incompatible with the specified calculation.
5 comments
Reviewer: Replacing 80 with 602 therefore cannot compress the normalized policy contribution, change its effective weight, or reverse the ranking. The statements at lines 1637–1645 are incompatible with the specified calculation.
The statement in L1637–1645 is consistent with the specified calculation. The reviewer's denominator cancellation formula is correct when the pharmaceutical policy counts for each province remain fixed and only the common national denominator changes. However, the Agent's industry filtering error changed both the national total and the pharmaceutical policy counts for individual provinces.
Specifically, let denote the number of pharmaceutical policies in province , and denote the number of other policies. The Agent incorrectly used as the pharmaceutical policy count for each province. Therefore, its normalized policy score becomes
\frac{p_i+b_i-\min_j(p_j+b_j)} {\max_j(p_j+b_j)-\min_j(p_j+b_j)}.$$ Although the common denominator cancels out, including unrelated policies changes the relative distribution of policy counts across provinces. Thus, the normalized differences between provincial scores can still change. In Case 2, Shanghai's policy score advantage over Jiangsu decreases from 0.181818 to 0.081356. The original description summarized this error as a change in the national denominator without explicitly explaining that the provincial numerators and their distribution also changed. **We have revised the explanation in Case 2 to clarify the actual source of the ranking change.**Follow-up on Our Responses and Further Experimental Analysis
Dear Reviewer 32NF,
Thank you for taking the time to review our submission and provide your feedback. As the discussion period draws to a close, we would like to kindly follow up on our responses and additional clarifications.
We have clarified the milestone evaluation mechanism and the intended interpretation of GPR/TPE. We also conducted additional experiments to separately examine the effects of noisy evidence and irrelevant data sources, and clarified the calculation issue in Case 2.
We would greatly appreciate it if you could revisit our responses and let us know whether these clarifications address your concerns. If you find our responses satisfactory, we would sincerely appreciate your consideration in updating your evaluation.
Thank you again for your time and consideration.
Best regards,
The Authors of Submission 29107
Thank you for your comments. We believe W1, W2, and Q1 involve some misunderstandings, which we clarify below. For W3, we conducted additional experiments to address the request for more detailed causal analysis. Please feel free to reach out if any further clarification is needed.
W1. Does Milestone Coverage Rate Exclude Redundant Milestones?
Clarification of Milestone Coverage Rate. We believe this weakness reflects a misunderstanding of our evaluation design. Milestone Coverage Rate measures the coverage of essential intermediate results, rather than proving the absence of redundant milestones. The gold milestone set may contain redundant milestones relative to individual valid paths. This is an intentional part of our milestone judgment design.
We improve evaluation consistency and robustness across different solution paths by combining high milestone coverage with a judgment mechanism based on logical and temporal dependencies. Specifically, milestones can be achieved directly or inferred from correctly completed downstream milestones (L161–162, L1516–1526). Together with high coverage of critical intermediate results, this mechanism reduces missed milestone achievements caused by different solution paths.
Lack of a Specific Concern. The review points out the possible existence of redundant milestones but does not identify any concrete negative impact of such redundancy. Therefore, we find it difficult to determine the specific concern being raised.
W2. Different Difficulty Distributions Among Failed Tasks
Clarification of the Evaluation Design. We believe this weakness also reflects a misunderstanding of our evaluation design. First, GPR and TPE in Table 5 measure conditional progress on each model's own failed tasks. Evaluating different subsets of failed tasks is therefore inherent to their intended purpose. Second, the review overlooks our evaluation across predefined difficulty levels. Tasks within the same difficulty level fall within a comparable range of difficulty.
Lack of a Specific Concern and Logical Connection. The observation that "a stronger model may perform well on harder tasks while a weaker model performs poorly on easier tasks" appears disconnected from the preceding criticism. More importantly, the review does not identify a concrete negative consequence of this observation. We therefore do not consider it a well supported weakness.
W3. More Detailed Analysis of Environmental Factors
This concern requests finer causal analysis beyond the claims made in our paper. Following the reviewer's suggestion, we conducted additional experiments that separately remove noisy evidence and irrelevant data sources.
The results show that agents are affected by multiple sources of uncertainty, including noisy evidence, irrelevant data sources, and incomplete schema understanding. In our data environment, noisy evidence has a greater impact than irrelevant data sources.