acceptodds
Withdrawn from ICLR 2027

FinREALBench: The Execution Reliability Gap Beyond Reasoning and Acting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Professional agents must execute workflows correctly and deliver results supported by the evidence they use. We introduce FinREALBench, a benchmark of 100 executable financial workflows spanning ten professional categories. Rather than assessing trajectories and answers in isolation, its evaluation contract follows evidence from source acquisition through transformation and repair into the submitted artifact. Process, Outcome, and Integrity assess distinct parts of this contract, while Oracle replay and targeted self-tests exercise the implemented evaluators. Across 16 model–harness systems, we examine where execution goes wrong and whether subsequent repairs reach the delivered answer. In a source-audited ownership task, all 16 submissions are schema-valid and arithmetically consistent, yet three retain an incorrect denominator. Same-model trajectories reveal two repair paths: correction from an observed filing passage and substitution of an unsupported value. Beyond this task, 133 of 868 schema-valid, fully complete artifacts receive evidence-consistency scores at most 0.5, spanning 56 tasks and 12 settings. These analyses characterize the Execution Reliability Gap: agents struggle to perform the work correctly, and complete submissions can remain unsupported. FinREALBench localizes failures along the evidence-to-delivery chain, identifying stronger execution skills, requirement-aligned checks, and evidence-grounded repair as complementary development targets.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agent Evaluation, Financial Agents, Execution Reliability, ReAct, Process-Aware Evaluation, Real-World Workflows, Model–Harness Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FdX5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FinREALBench, a benchmark for evaluating agents on real-world financial workflows, comprising 100 executable tasks across 10 professional categories. Unlike approaches that focus solely on final answers or evaluate execution trajectories in isolation, the authors propose Evidence-to-Commitment Traceability, aiming to track evidence throughout the workflow, from source acquisition and data transformation to error repair and final delivery.

The evaluation framework consists of three components: Process, Outcome, and Integrity. It combines deterministic checks with LLM-as-a-Judge evaluation and validates the evaluators through Oracle replay, 1,702 targeted mutation tests, and positive controls. Experiments across 16 model-harness configurations reveal substantial weaknesses in execution validation, source correctness, and evidence consistency. Further case studies show that even structurally complete and arithmetically consistent answers may contain incorrect or unsupported values.

Critical strengths
  1. Rather than merely answering financial questions, agents are required to perform information retrieval, document processing, code execution, financial calculations, and structured result delivery in a real execution environment. Compared with static financial question-answering tasks, this setting provides a more comprehensive assessment of agents' capabilities in professional workflows. The use of real data sources, financial temporal constraints, and accounting requirements also provides concrete and verifiable grounds for reliability evaluation. I consider this the paper's primary practical contribution.

  2. The authors validate their evaluators through Oracle replay, targeted mutation testing, and positive controls, testing not only whether the intended correct solutions are accepted but also whether predefined errors are detected and valid implementation variations are tolerated. All 100 tasks pass Oracle replay, and all 1,702 targeted mutants are detected. This systematic approach to testing the evaluators themselves is commendable and strengthens the engineering credibility of the benchmark.

Critical weaknesses
  1. The novelty over existing trajectory-level failure diagnosis research is not sufficiently established. The paper proposes Evidence-to-Commitment Traceability to identify errors across evidence acquisition, transformation, repair, and delivery. However, related work such as FinTrace, AgentRx, and TELBench/DRIFT has already explored trajectory evaluation, failure localization, and error propagation. While the financial task contracts and source verification mechanisms are valuable, it remains unclear what additional diagnostic capabilities the proposed framework provides beyond existing approaches.

  2. The central evidence-chain diagnostic capability lacks systematic experimental validation. Although the paper introduces four evidential transitions, the experiments primarily report Process, Outcome, and Evidence Consistency scores rather than directly evaluating failure localization across these transitions. For example, the incorrect denominators in FTB-049 can be identified by comparing final answers with source documents, without examining execution trajectories. While the repair-path analysis uses trajectory information, it remains limited to a few case studies. The paper therefore does not sufficiently demonstrate the unique diagnostic value of Evidence-to-Commitment Traceability beyond conventional source verification and trajectory analysis.

Questions for authors
  1. Could the authors provide a controlled comparison to demonstrate the incremental diagnostic value of Evidence-to-Commitment Traceability? In particular, it would be helpful to distinguish the benefits of incorporating execution trajectories from those of the proposed evidence-linked evaluation methodology itself.

  2. How were the 0.5/0.5, 0.75/0.25, and 0.7/0.3 weights used in Process, Outcome, and Overall determined? Were these choices supported by empirical evidence or a calibration procedure? Could the authors provide a sensitivity analysis showing whether the main model rankings change substantially under different weighting schemes?

Reviewer_3XDROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FinREALBench proposes a benchmark of 100 executable financial workflows across ten categories. It evaluates agent behavior through deterministic and LLM-based Process and Outcome metrics with an Integrity gate. The paper emphasizes evidence traceability across source acquisition, computation, error repair and final artifact generation. It evaluates 16 model harness configurations and reports that some fully complete artifacts exhibit weak evidence consistency.

Critical strengths

(1) Realistic workflow evaluation: The paper evaluates actual agent execution, including tool use, computations, and final artifacts. This is valuable for understanding failures in real-world financial tasks.

(2) Evidence based failure analysis: Paper shows how agents can produce complete and arithmetically consistent answers while relying on incorrect inputs or unsupported repairs. Case studies illustrate failure modes.

Critical weaknesses

(1) Evaluator validation: It remains unclear whether the LLM-based judgments have been independently validated on real agent outputs.

(2) Failure localization: Analysis mainly relies on a few selected case studies. It is unclear whether these examples are representative of the broader benchmark or whether the proposed approach can reliably identify similar failures across other tasks.

Questions for authors

(Q1) Have the LLM-based judgments(e.g. evidence consistency) been validated against independent expert annotations of real agent outputs? If so, how well do they agree?

(Q2) Failure localization: Beyond the selected case studies, how consistently can the framework identify which stage of a workflow caused a failure?

(Q3) Evaluation sensitivity: How were the weights for Process, Outcome, and Overall selected? Do the system rankings remain consistent under alternative weighting choices?

minor question:

(Q4) Integrity gate is described as a hard gate, but it is not included in the overall scoring equation. How Integrity violations affect the final scores?

Reviewer_4SogOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FinREALBench evaluates agents on 100 executable financial workflows, separating evaluation process of finance into quality, outcome quality, and integrity. It validates deterministic graders using oracle replay, 1702 targeted mutations, and positive controls, then evaluates 16 mode harness settings. The author claims a high yielding rate, of 133 of 868 schema valid, fully complete artifacts receive evidence-consistency scores at most 0.5, spanning 56 tasks and 12 settings

Critical strengths

S1: Timely and important topics: people are quickly building long-horizon financial agent benchmarks for the real world, for robust evaluation of more and more powerful LLM agents. This paper could provide some add-on value to the community in general.

S2: separating evaluation into quality, outcome quality, and integrity and have different checker and expert quality control is good to have, proposing useful framework for distinguishing arithmetic correctness from source-grounded correctness.

Critical weaknesses

W1: NO disclosure of data licensing and access conditions. The paper does not provide any licensing information for its data sources, some of which do not appear to be publicly accessible. Further details are provided in the ethical concerns section. Given the sensitivity and access restrictions associated with some financial data, the authors should clearly document these terms.

W2: Limited task and rubric quality. I manually inspected several task descriptions in the supplementary materials and found them brief, simple, and insufficiently specified. In my assessment, their quality falls substantially below that of concurrent finance benchmarks such as BigFinanceBench and FrontierFinanceBench. The associated rubrics also appear insufficiently rigorous, particularly in assessing factual correctness. These issues weaken confidence in the benchmark’s ability to evaluate substantive financial workflows.

W3: Presentation and readability. The manuscript appears hastily prepared, does not consistently follow ICLR formatting conventions, and is difficult to follow. I encourage the authors to devote more human attention to the paper readability.

Questions for authors

Q1: How can the authors systematically select and curate high-quality financial data while ensuring that the relevant licenses permit its intended use and release? Although fully addressing this concern may be difficult within the rebuttal period, clear data provenance and appropriate permissions are prerequisites for a credible dataset contribution.

Q2: What constitutes a high quality rubric for financial tasks, particularly with respect to specificity, factual correctness, and evidence grounding? I encourage the authors to explore a structured expert review process that jointly improves task and rubric quality while increasing the proportion of candidate tasks that meet these standards.

These are fundamental building blocks for any data research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.