- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces FinREALBench, a benchmark for evaluating agents on real-world financial workflows, comprising 100 executable tasks across 10 professional categories. Unlike approaches that focus solely on final answers or evaluate execution trajectories in isolation, the authors propose Evidence-to-Commitment Traceability, aiming to track evidence throughout the workflow, from source acquisition and data transformation to error repair and final delivery.
The evaluation framework consists of three components: Process, Outcome, and Integrity. It combines deterministic checks with LLM-as-a-Judge evaluation and validates the evaluators through Oracle replay, 1,702 targeted mutation tests, and positive controls. Experiments across 16 model-harness configurations reveal substantial weaknesses in execution validation, source correctness, and evidence consistency. Further case studies show that even structurally complete and arithmetically consistent answers may contain incorrect or unsupported values.
Critical strengths
-
Rather than merely answering financial questions, agents are required to perform information retrieval, document processing, code execution, financial calculations, and structured result delivery in a real execution environment. Compared with static financial question-answering tasks, this setting provides a more comprehensive assessment of agents' capabilities in professional workflows. The use of real data sources, financial temporal constraints, and accounting requirements also provides concrete and verifiable grounds for reliability evaluation. I consider this the paper's primary practical contribution.
-
The authors validate their evaluators through Oracle replay, targeted mutation testing, and positive controls, testing not only whether the intended correct solutions are accepted but also whether predefined errors are detected and valid implementation variations are tolerated. All 100 tasks pass Oracle replay, and all 1,702 targeted mutants are detected. This systematic approach to testing the evaluators themselves is commendable and strengthens the engineering credibility of the benchmark.
Critical weaknesses
-
The novelty over existing trajectory-level failure diagnosis research is not sufficiently established. The paper proposes Evidence-to-Commitment Traceability to identify errors across evidence acquisition, transformation, repair, and delivery. However, related work such as FinTrace, AgentRx, and TELBench/DRIFT has already explored trajectory evaluation, failure localization, and error propagation. While the financial task contracts and source verification mechanisms are valuable, it remains unclear what additional diagnostic capabilities the proposed framework provides beyond existing approaches.
-
The central evidence-chain diagnostic capability lacks systematic experimental validation. Although the paper introduces four evidential transitions, the experiments primarily report Process, Outcome, and Evidence Consistency scores rather than directly evaluating failure localization across these transitions. For example, the incorrect denominators in FTB-049 can be identified by comparing final answers with source documents, without examining execution trajectories. While the repair-path analysis uses trajectory information, it remains limited to a few case studies. The paper therefore does not sufficiently demonstrate the unique diagnostic value of Evidence-to-Commitment Traceability beyond conventional source verification and trajectory analysis.
Questions for authors
-
Could the authors provide a controlled comparison to demonstrate the incremental diagnostic value of Evidence-to-Commitment Traceability? In particular, it would be helpful to distinguish the benefits of incorporating execution trajectories from those of the proposed evidence-linked evaluation methodology itself.
-
How were the 0.5/0.5, 0.75/0.25, and 0.7/0.3 weights used in Process, Outcome, and Overall determined? Were these choices supported by empirical evidence or a calibration procedure? Could the authors provide a sensitivity analysis showing whether the main model rankings change substantially under different weighting schemes?