acceptodds
Under review as a conference paper at ICLR 2027

QuantitativeFinance-Bench: Benchmarking AI Agents on Real-World Quantitative Finance Tasks

60 views

Abstract

Evaluating language models on quantitative finance tasks is challenging because such tasks require simultaneous mathematical rigor, domain expertise, and data engineering in a single executable artifact. We introduce QuantitativeFinance-Bench (QF-Bench), an execution-grounded benchmark for the computational core of professional quantitative finance. We have four main contributions. First, we curate 86 practitioner-authored tasks spanning derivatives pricing, fixed income, credit, risk management, factor research, backtesting, microstructure, and financial NLP. Each task is executed in a network-isolated Docker container with real financial data. Second, we introduce a hierarchical verification framework with more than 3,300 assertions across 86 tasks. These assertions encode domain-specific checks, including no-arbitrage bounds, put-call parity, and convergence diagnostics that are not captured by generic code-level tests. Third, we define Finance-Zero, a non-agentic baseline for difficulty calibration, and introduce automated error attribution for Computation, Convention, and Mislabeling errors. Fourth, we evaluate 38 Finance-Zero base-model configurations and 9 production CLI-agent configurations, finding that the best system achieves 61.7% pass@1 while the single-call baseline reaches 42.6%. A run passes only when all required deliverables and verifier checkpoints succeed. The benchmark and evaluation resources are available at https://anonymous.4open.science/r/QuantitativeFinance-Bench-07BB.

Primary area:
datasets and benchmarks
Keywords:
Quantitative Finance, Autonomous Agents, Evaluation and Benchmarking, Large Language Models, Financial Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NQviOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces QF-Bench, an execution-grounded benchmark comprising 86 real-world quantitative finance tasks across diverse financial domains. The authors develop a hierarchical verification framework with over 3,300 assertions to evaluate numerical correctness and financial consistency, along with Finance-Zero as a non-agentic baseline and an automated error attribution mechanism.

Critical strengths
  1. The paper evaluates a broad range of models, including 38 Finance-Zero configurations and 9 CLI agent configurations.

  2. The paper recognizes the importance of benchmark verification and establishes a detailed quality assurance process with multiple stages and validation steps to ensure task correctness and evaluation reliability.

Critical weaknesses
  1. Insufficiently Justified Motivation

    The paper's motivation relies on an overly simplified characterization of existing financial benchmarks as primarily focusing on predictive tasks or document-based reasoning rather than numerical computation. However, prior work such as FinQA[1] explicitly evaluates multi-step numerical reasoning over financial data, QuantEval[2] covers quantitative mathematical reasoning and executable strategy coding. Therefore, numerical computation and executable financial tasks are not entirely overlooked by existing benchmarks.

  2. Insufficient Justification of Benchmark Construction.

    Although the paper presents a detailed quality assurance pipeline and task types, the methodology used to construct the benchmark remains unexplained. In particular, the authors provide limited information about how the tasks were originally collected, selected, and organized across different quantitative finance domains. The proposed pipeline primarily focuses on validating and refining submitted tasks, rather than explaining the underlying task selection strategy.

  3. Limited and Unfair Baseline Comparison. First, the authors only evaluate general-purpose CLI agents without considering existing finance-specific agents. Second, the comparison between Finance-Zero and CLI agents may not be fair, as they operate under substantially different settings, including execution time limits (120 seconds vs. 30 minutes), interaction budgets, and opportunities for iterative debugging. Therefore, the reliability of Finance-Zero as a reference baseline is also questionable.

  4. Poor Writing. The paper suffers from lack of logical coherence, particularly in the Introduction and Related Work sections. The authors frequently use bolded opening sentences as informal paragraph headings rather than organizing the discussion into proper subsections or establishing smooth transitions between paragraphs. This writing makes the narrative appear fragmented without sufficient logical connections.

[1] FinQA: A dataset of numerical reasoning over financial data

[2] QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

Questions for authors
  1. How were the easy, medium, and hard difficulty levels determined? Why do some Easy tasks have extremely low pass rates (e.g., 0.9%), while some Hard tasks achieve much higher pass rates (e.g., 27.8%)?
  2. Could the authors provide a complete end-to-end example, including the original task instruction, input data, expected deliverables, and verification criteria, rather than only summarizing representative tasks in Appendix C?

Other concerns listed in weakness part

Reviewer_UMfQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces QuantitativeFinance-Bench (QF-Bench), an execution-grounded benchmark for evaluating language model agents on the computational core of professional quantitative finance. The benchmark comprises 86 practitioner-authored tasks spanning 8 core domains: derivatives pricing, fixed income, credit risk, risk management, factor research, backtesting, market microstructure, and financial NLP. All tasks run in network-isolated Docker containers with real market data to replicate real-world data challenges including missing values and holiday calendars. The paper makes four primary contributions:

  1. A curated task set authored by quant and risk practitioners, grounded in real desk workflows rather than textbook problems.
  2. A five-stage hierarchical verification framework with over 3,300 domain-specific assertions, including financial invariants and convergence diagnostics that are absent from generic code-level benchmarks.
  3. Finance-Zero, a single-call non-agentic baseline for difficulty calibration, plus an automated error attribution taxonomy classifying failures into Computation, Convention, and Mislabeling errors.
  4. Large-scale evaluation of 38 base model configurations and 9 production CLI-agent configurations, showing that the best system achieves 61.7% pass@1, and that agentic iteration primarily resolves software engineering failures while finance-specific knowledge errors persist across iterations. Overall, the work addresses a critical gap in AI evaluation by benchmarking executable numerical correctness and financial consistency, rather than generic coding ability or document-level financial reasoning.
Critical strengths
  1. Domain-specific verification as a foundational innovation. The five-stage hierarchical verification framework—especially the inclusion of financial invariants and industry convention checks—represents a major advance beyond generic code execution benchmarks. This design aligns evaluation with actual quantitative finance practice, where numerically plausible outputs can be materially wrong if they violate financial first principles or standard conventions. This approach is broadly applicable to benchmark design in other professional domains.
  2. Deep, mechanistically grounded failure mode analysis. The decomposition of failures into Layer 1 (engineering) and Layer 2 (finance knowledge, split into theoretical and conventional errors), combined with the empirical discovery that finance errors exhibit strong iteration stickiness (64.9% of initial finance errors persist across revisions), is a highly impactful insight. It explains why agentic iteration delivers diminishing returns in knowledge-intensive domains and provides clear, actionable directions for model improvement and agent design.
Critical weaknesses
  1. Limited task scale and incomplete domain coverage. With 86 total tasks and only 35 Hard-tier tasks, some difficulty-tier and sub-domain analyses have limited statistical robustness. Several quantitatively important areas of modern quantitative finance—including high-frequency market making, Basel IV regulatory capital calculation, structured credit waterfall mechanics, and climate financial risk stress testing—are not yet covered, narrowing the benchmark's scope relative to the full breadth of industry practice.
  2. Insufficient agent diversity and contamination validation. The evaluation is restricted to two closed-source CLI agent frameworks, excluding open-source agent scaffolds and other agent paradigms . This limits how broadly we can generalize the observed failure patterns and performance rankings. Additionally, while canary strings detect verbatim code leakage, there is no systematic analysis of textual overlap between task descriptions and public training data, leaving residual uncertainty about whether performance scores are inflated by data contamination.
Questions for authors
  1. Data contamination: Have you quantified the textual overlap between task specifications/instructions and the public training corpora of the evaluated models? Could you provide specific metrics or detection results regarding data contamination beyond the canary string checks?
  2. Agent generalization: Do you plan to extend the evaluation to open-source agent frameworks and other agent paradigms beyond CLI scaffolds? How do you expect performance patterns and failure modes might differ across agent architectures?
  3. Finance-Zero calibration: The single-call baseline uses a 120-second timeout. Have you run control experiments to test how pass rates change with longer timeouts? Are there tasks that fail only due to execution timeout rather than insufficient model capability?
Reviewer_gfeDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces QF-Bench, an executable benchmark containing 86 practitioner-written quantitative finance tasks. The tasks cover derivatives pricing, fixed income, risk management, factor modeling, and backtesting, using real financial data in Docker environments. A hierarchical verifier with over 3,300 assertions checks numerical correctness, financial constraints, and convergence. The authors evaluate 38 Finance-Zero configurations and 9 CLI-agent configurations, with the best agent achieving 61.7% pass@1. The failure analysis shows that finance-specific errors can persist even after multiple rounds of debugging.

Critical strengths
  • Useful benchmark with broad task coverage. QF-Bench evaluates executable financial computations rather than just financial QA or trading performance. Its coverage across multiple quant finance domains, together with real data and domain-specific checks, makes it a useful evaluation resource.
  • Informative failure analysis. The experiments reveal that financial formula and convention errors can persist through iterative debugging, even when engineering errors are resolved. The concrete cases in Section 5.3 and Appendix I make this finding convincing.
Critical weaknesses
  • The incremental value over existing finance benchmarks is not sufficiently established. Existing benchmarks already include financial correctness checks: QuantEval compares backtest metrics against reference results and enforces financial constraints, while QuantCode-Eval uses executable property checks for strategy implementations. Although QF-Bench covers more financial domains, the paper does not clearly establish what its verifier adds beyond these approaches. In particular, a staged ablation would help quantify how many additional errors are detected by financial invariants and convergence checks beyond basic numerical verification, and whether these checks change model rankings.
  • Some financial verification rules need clearer assumptions. Appendix B.4 lists Asian option price ≤\leq European option price as a consistency check, but this does not hold universally; certain fixed-strike Asian puts can be more expensive than their European counterparts. Similarly, \texttt{mc-greek-surface-1} requires the pathwise Delta estimator to have lower variance than the likelihood-ratio estimator, although empirical variance rankings can be affected by sampling noise. These checks may be valid for the chosen settings, but their assumptions and tolerances are not sufficiently explained. Since passing requires satisfying every check, this raises concerns about potential false rejections.
Questions for authors
  • Could the authors clarify the differences from QuantEval and QuantCode-Eval, and provide an ablation showing the additional errors detected by domain-specific checks?

  • What assumptions and tolerances are used for the Asian option pricing inequality and the Monte Carlo variance comparison? How are boundary cases and sampling noise handled?

  • Figure 2 includes a Finance-Zero failure gate. How many candidate tasks were revised or excluded by this criterion, and could it bias the benchmark toward tasks that favor iterative agents?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.