- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces QF-Bench, an execution-grounded benchmark comprising 86 real-world quantitative finance tasks across diverse financial domains. The authors develop a hierarchical verification framework with over 3,300 assertions to evaluate numerical correctness and financial consistency, along with Finance-Zero as a non-agentic baseline and an automated error attribution mechanism.
Critical strengths
-
The paper evaluates a broad range of models, including 38 Finance-Zero configurations and 9 CLI agent configurations.
-
The paper recognizes the importance of benchmark verification and establishes a detailed quality assurance process with multiple stages and validation steps to ensure task correctness and evaluation reliability.
Critical weaknesses
-
Insufficiently Justified Motivation
The paper's motivation relies on an overly simplified characterization of existing financial benchmarks as primarily focusing on predictive tasks or document-based reasoning rather than numerical computation. However, prior work such as FinQA[1] explicitly evaluates multi-step numerical reasoning over financial data, QuantEval[2] covers quantitative mathematical reasoning and executable strategy coding. Therefore, numerical computation and executable financial tasks are not entirely overlooked by existing benchmarks.
-
Insufficient Justification of Benchmark Construction.
Although the paper presents a detailed quality assurance pipeline and task types, the methodology used to construct the benchmark remains unexplained. In particular, the authors provide limited information about how the tasks were originally collected, selected, and organized across different quantitative finance domains. The proposed pipeline primarily focuses on validating and refining submitted tasks, rather than explaining the underlying task selection strategy.
-
Limited and Unfair Baseline Comparison. First, the authors only evaluate general-purpose CLI agents without considering existing finance-specific agents. Second, the comparison between Finance-Zero and CLI agents may not be fair, as they operate under substantially different settings, including execution time limits (120 seconds vs. 30 minutes), interaction budgets, and opportunities for iterative debugging. Therefore, the reliability of Finance-Zero as a reference baseline is also questionable.
-
Poor Writing. The paper suffers from lack of logical coherence, particularly in the Introduction and Related Work sections. The authors frequently use bolded opening sentences as informal paragraph headings rather than organizing the discussion into proper subsections or establishing smooth transitions between paragraphs. This writing makes the narrative appear fragmented without sufficient logical connections.
[1] FinQA: A dataset of numerical reasoning over financial data
[2] QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models
Questions for authors
- How were the easy, medium, and hard difficulty levels determined? Why do some Easy tasks have extremely low pass rates (e.g., 0.9%), while some Hard tasks achieve much higher pass rates (e.g., 27.8%)?
- Could the authors provide a complete end-to-end example, including the original task instruction, input data, expected deliverables, and verification criteria, rather than only summarizing representative tasks in Appendix C?
Other concerns listed in weakness part