acceptodds
Under review as a conference paper at ICLR 2027

FinancialAuditBench: Benchmark Construction under Differential Privacy Using Real-World Priors

Abstract

As AI agents are becoming widely adopted in the financial services industry, careful measurement is essential to understand where they can be reliably deployed and where oversight and professional review remain necessary. Such measurement, however, is constrained by limited access to proprietary or privacy-sensitive data. Existing benchmarks therefore often rely on publicly available data, human- and/or LLM-authored tasks, or simplified settings. We introduce FinancialAuditBench, a benchmark for evaluating agents on financial statement audit tasks, along with a framework for systematically generating synthetic engagements. Our task generation framework leverages differentially private aggregate statistics from historical audits along with audit expertise contributed through over 1,100 hours of benchmark development and review. FinancialAuditBench consists of 90 tasks spanning workpaper completion and review across six synthetic audit engagements, each containing an average of 179 files. Evaluation on eleven frontier models shows that while agents complete substantial portions of staff-level audit tasks well, they sometimes perform inappropriate procedures or produce incorrect documentation. Beyond financial auditing, our framework offers an approach for systematically generating synthetic tasks for model evaluation and training in privacy-sensitive domains.

Primary area:
datasets and benchmarks
Keywords:
AI systems, agents, datasets, benchmarks, differential privacy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fWMPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the author introduced FinancialAuditBench that used differential privacy related metrics to construct financial statement audit tasks and benchmark frontier model performance on top of it.

Critical strengths
  1. The dataset is constructed with a differential privacy guaranteed method, which could to some extent expose a more real-world dataset for evaluations.

  2. The dataset is evaluated by domain experts, which increases the reliability of this benchmark dataset.

  3. The author provides an in depth failure analysis , providing important insights to future model improvement.

Critical weaknesses

In this paper, the author develop a computational graph that generates synthetic companies and corresponding documents, and the auditors help define the graph. My main concern is that this form of construction may carry over a strong inductive bias, which is testing whether the models are functioning the same way as the auditors.

Questions for authors

My main question would be how do you scale up with the framework if the fidelity of the benchmark depends on feedback with domain expert.

2 comments

Critical Weaknesses:

Inductive bias: You are correct that our synthetic data generation framework relies on a graph that is partially manually defined and extensively revised by domain experts. This is indeed meant to ensure that our tasks reflect the work auditors perform in actual client engagements. However, since the purpose of our benchmark is to measure how well agents can perform the work of human auditors as assessed by domain experts, we do not see this similarity (or "inductive bias") as a weakness, but rather as an inherent strength of our method. Synthetic data that is too dissimilar to real-world data does not allow us to assess how well agents would perform on realistic data, and we do not require them to follow any individual auditor’s reasoning process.

Questions for Authors:

Scaling the framework: Existing methods for constructing high quality benchmark tasks usually involve domain experts writing each task manually. This becomes incredibly challenging for tasks that span hundreds of documents. Our computational graph improves upon this by allowing for audit relationships and rules to be codified. Each derived relationship in the graph (the nodes and vertices) should always be reviewed by a domain expert to ensure sufficient reliability and accuracy. This is a constraint that scales with the size and density of the graph, but once such a graph is constructed, generating more samples can be done with less human oversight.

Thank you for the clarification and answering my questions. My recommendation to the acceptance remains the same.

Reviewer_78jsOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a benchmark for financial statement auditing by collecting several real world audit documents, synthesizing realistic and private scenarios using DP and approving them with human experts (licensed auditors). This paper largely describes the procedures of creating this benchmark, and several frontier models are evaluated on this benchmark, showing that while the agents alone can perform well on most aspects, they fail to meet the full criteria in most cases. The agent evaluations are also examined by human experts

Critical strengths

This seems to be indeed a previously-unexplored area, and the authors are proposing a useful artifact (the benchmark) that future works can use. The benchmark seems well constructed, and the evaluation of frontier models is a nice addition.

Critical weaknesses

The benchmark is quite niche. While I think it is indeed a useful benchmark, the proposal of a benchmark alone with some preliminary evaluations is not a strong enough contribution.

Questions for authors
  • Time saving is one, but cost is another. is there a comparison of the cost of the tokens from the agent usage vs. typical cost of a human auditor?
  • Did the authors try tweaking the setup of the agents to see if the performance can be improved? Custom harness or different prompts/tones?
2 comments

Critical Weaknesses:

Contribution of a benchmark and prelim evaluations: The intended scope of our benchmark is to evaluate language models on real-world tasks performed by financial auditors. The practice of proposing datasets for specific domains has been established as being acceptable for evaluation datasets (not pretraining datasets) provided that there is strong construct validity [1]. Furthermore, ICLR has traditionally welcomed datasets and benchmarks papers such as SWE-Bench [2], Tau-Bench [3], and Terminal-Bench [4], whose core contributions are a benchmark and corresponding evaluations.

Questions for Authors:

Time savings versus costs: We provide the cost of tokens, wall clock times, and estimates for how long human auditors take to complete each task in Appendix D.1. Typical costs for human auditors are intentionally not provided as each audit firm has their own unit economics, thus we do not make any claims regarding the cost comparison.

Setup of agents for improving performance: We evaluate all models with a simple harness and a standard set of prompts written by auditors to ensure that all models are evaluated in a controlled setting. Indeed, custom harnesses, improved prompts, and model specific harnesses like Codex could be implemented or used to further drive improved performance (which we did observe!), but those experiments are outside of the scope of this paper. We do, however, include results using Codex computer-use as an ablation in Section 6.6 (L422-431).

[1] https://openreview.net/forum?id=mdA5lVvNcU

[2] https://openreview.net/forum?id=VTF8yNQM66

[3] https://openreview.net/forum?id=roNSXZpUDN

[4] https://openreview.net/forum?id=a7Qa4CcHak

Thank you for your clarifications. I will raise my score to 3.

Reviewer_KjWeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FinancialAuditBench, a benchmark of 45 workpaper-completion tasks and 45 matching review tasks for AI agents doing staff-level audit fieldwork, built on six synthetic engagements from two industries.

The engagement binders are generated by an expert-designed computational graph that keeps documents mutually consistent, with some source variables drawn from priors estimated on confidential historical binders under binder-level (3.65,10−5)(3.65, 10^{-5})-DP.

Across 11 frontier models, the best one meets 95.84% of rubric criteria but fully passes only 69.17% of tasks. The authors conclude that agents still need professional review.

Critical strengths

The paper targets an under-explored domain. Unlike CPA-exam evaluations, the benchmark reflects audit fieldwork, requiring models to navigate large data rooms (an average of 179 files per engagement), trace amounts to supporting evidence, and document their conclusions. The privacy design is principled: access to confidential data is confined to a single aggregation phase, the privacy unit is an entire binder, a privacy ceiling was set before data access, and the queries and privacy ledger are released. The computational graph maintains consistency across more than a hundred documents without requiring the estimation of high-dimensional joint distributions. The benchmark draws on over 1,100 hours of auditor effort, and the evaluation spans strict and partial-credit scoring, pass^k, review tasks, computer use, and auditor-led failure analysis. The authors also explicitly acknowledge that a passing score does not guarantee an error-free workpaper.

Critical weaknesses

My main concern is the lack of quantitative validation of the synthetic data, despite the generation framework being presented as a core contribution. Although the authors acknowledge that realism and fidelity remain open challenges (Section 7), the paper describes the benchmark as realistic and representative and states, without supporting evidence, that performance on it correlates with performance in production audit settings (Appendix A.3). The evidence for realism appears limited to iterative feedback from the auditors who helped develop the generator; no blinded or independent assessment is reported. Beyond the average number of files per binder, the paper provides no descriptive statistics on revenue, balances, or transaction counts, nor an analysis of variation across the six companies. It also does not compare generated values with the DP priors from which they are sampled. This comparison is particularly important because propagation through the graph and resampling after failed LLM-generated feasibility tests may alter the resulting distributions.

Using DP to release statistics from confidential binders is a sensible design choice, but the added value of the resulting priors is not quantitatively established. This matters because the title, abstract and conclusion foreground these priors, and the introduction states that they are meant to reduce the expert feedback needed to correct implausible assumptions from public sources or LLMs. This benefit is supported only qualitatively (Section 4.3). The priors cover six ratios, sixteen record features and a few summaries, while the graph’s roughly 250 nodes and 1,000 edges were expanded using public resources, auditor expertise and LLMs, so it is unclear how much they shape the generated data. Reporting the number of historical binders and the estimation error of the released statistics, and even a rough measure of the effect (e.g., expert corrections needed with and without the priors on one engagement), would make this claim easier to assess.

Robustness to the sampled synthetic instances is another concern. With only six engagements, it is unclear whether model scores and rankings would remain stable if the binders were regenerated with different seeds. Repeating the evaluation across regenerated instances would test this sensitivity and help substantiate the claim that the framework supports scalable task construction. The binders were also designed to limit errors and ambiguities, and rubric items affected by unintended document inconsistencies are excluded (Appendix A.2), without reporting how many. Since incomplete, inconsistent, and poorly organized client data are an important source of difficulty in real audits, as the authors acknowledge, these design choices may cause the benchmark to underestimate task difficulty.

Evaluation reliability also needs stronger validation. Each workpaper is graded on about ten selected checks, some delegated to an LLM grader, but agreement with independent human judgment is not reported. The rubrics were revised after inspecting model submissions, leaving unclear whether the revisions were validated on held-out submissions. Model comparisons also appear sensitive to inference settings: GPT-6 Astra scores 43.61% at medium reasoning effort in the main table and 71.11% at maximum effort in Table 7, exceeding the best result in the main table. This sensitivity makes cross-model rankings difficult to interpret without comparable inference budgets. Finally, the paper reports no confidence intervals that account for the clustering of the 45 tasks within six binders.

Overall, the paper is most convincing as an expert-curated audit benchmark that incorporates private real-world statistics through DP. Its realism claims would be stronger with quantitative validation of the generated data, and even a limited comparison with alternative priors would clarify the role of the DP priors. I currently lean toward acceptance, as the benchmark is a valuable contribution, and would raise my score further with quantitative validation of the synthetic binders and evidence that model rankings are stable across regenerated instances.

Questions for authors
  1. What is the empirical basis and quality of the DP priors (e.g., the number of historical binders and the estimation error of the released statistics), and is there evidence of their added value over LLM-generated or public priors?

  2. Has the realism of the synthetic binders been assessed independently, for example by auditors not involved in construction, and what evidence supports the stated correlation with performance in production audit settings?

  3. How stable are model scores and rankings when the binders are regenerated with different seeds?

  4. How many rubric items were excluded, and how many unintended generator errors were found during expert review?

  5. How well does the LLM grader agree with independent human graders?

3 comments

Critical Weaknesses: Thank you for all the detailed feedback. Below, we address the weaknesses you mentioned that are not covered in your questions for authors.

Quantitative validation of synthetic data and dp priors: As this benchmark targets a specific domain (financial auditing), we rely on domain experts (financial auditors) as our validation and ground truth. This practice is well established for other domains (e.g. code generation, deep research, general science, etc.) where datasets are constructed either from publicly available sources (e.g., code repositories) and/or manually written. In practice, auditors reviewed all the tasks we generated, and we released only tasks they considered realistic enough to encounter in actual client engagements. Furthermore, we focused on validating the correctness and realism of the final audit documents and tasks over measuring how closely generated values match the DP priors as agents interact with these documents and are what auditors can assess directly using their professional expertise. Please see also our answers to your questions 1 and 2 below.

Descriptive statistics: We provide limited audit-specific quantitative metrics for each task, so that readers may focus more on the methods proposed in our work. However, metrics such as the number of transactions, entities, employees, end of month balances, variation between binders, etc. can be easily derived by looking at our released benchmark tasks. For instance, across the six binders, each binder contains an average of 30.5 distinct customers, 40 active employees, 5.2 distinct vendors, 1,133 journal entries, and 1,014 employee payroll records.

Incomplete, inconsistent, and poorly organized client data: We do model the messiness of real world data that naturally occurs in the financial audit process. Auditors provided feedback on how different companies format their financials, how information may be (confusingly) recorded across documents, what essential documents may at times be missing, among others. As we point out in our failure analysis section (Section 6.3), frontier models still sometimes hallucinate supporting documents that are not provided, and our benchmark captures those errors. We purposely excluded ambiguous or unclear tasks that would require the agent to ask clarification questions, and leave the question of how performance on such tasks can be measured for future work.

Model comparisons and effect of reasoning effort: Our main results use a constant reasoning effort (medium) to ensure fair comparisons between all models. The ablation on reasoning effort involving GPT-6 Astra in Appendix D.4 shows that as reasoning effort increases, so does task accuracy. Model ranking stability is addressed in the next section.

Questions for authors:

1 What is the empirical basis and quality of the DP priors: The financial datasets that currently exist in the public domain are largely centered around publicly traded companies, whose financials are very different from privately held companies. We make two contributions on this front: (1) we provide aggregated statistics on a subset of privately held companies and (2) we provide data on the audit processes for each of these companies. Audit data, even for public companies, is not fully part of the public domain. More broadly, the distribution of financials across the full set of all companies that exist is large and hard to empirically approximate. We use a dataset of ~1,000 historical binders as the empirical basis for our priors, and naturally there is significant variance company to company as each company has its own intricacies and nuances. Thus, we rely on domain experts (licensed auditors) rather than statistical methods to validate and correct the seeds we used for generating this benchmark. The claim we make is that the quality and utility of these seeds are more realistic compared to priors provided by LLMs or other publicly available data sources. Auditors reported that our priors required fewer corrections than LLM-generated or public-data alternatives, providing qualitative evidence of their added value. We did not quantitatively measure these time savings due to resource constraints.

2 Third-party realism assessment and correlation with production settings: We did not perform independent third-party analysis on the realism of the dataset we released due to financial constraints. Realism more broadly can be evaluated both at a high and low level by evaluating aspects such as the complexity, shape, and scale of each task as well as more specifically how distributions of say individual entity names are. The main claim regarding this that we make in our work is that FinancialAuditBench represents a significant step forward in the overall realism, complexity, and scale of tasks relative to prior work. Realism here refers to how similar the tasks we construct are compared to real-world tasks that auditors perform, complexity refers to features such as the number of types of documents we release, and scale refers to things like the number of files the agent must reason over. Domain experts reported that the shape and complexity of our tasks more closely align with their day-to-day work than those of prior benchmarks such as APEX-Accounting [1] and AuditFraudBench [2]. We also evaluated correlations with production settings using a set of proprietary data sources, and while we cannot release these sources due to their proprietary nature, we report a positive correlation.

3 Stability of model scores and rankings: We only release and evaluate a fixed number of companies' binders due to the cost required to review and revise each generated binder. We evaluate each model eight times on each binder we release and find that model rankings are relatively stable on this set of evaluation tasks. Confidence intervals over the eight runs we do for each model (which can be calculated from the results we provide in the submission) are provided below. We will add this table to the manuscript:

ModelStrict pass rateApprox. 95% CI
Claude Opus 5.569.17%66.1–72.3%
Claude Fable 5.165.83%62.6–69.1%
Grok 4.647.22%44.1–50.3%
GPT-5.6 Sol44.17%40.7–47.6%
GPT-6 Astra43.61%39.6–47.6%
GPT-5.6 Terra40.28%36.7–43.9%
Muse Spark 1.339.44%36.0–42.9%
DeepSeek V4.1 Flash33.06%29.5–36.6%
GLM-5.229.44%26.5–32.4%
Kimi K328.89%25.3–32.5%
Gemini 3.8 Flash27.50%25.0–30.0%

4 Rubric items and unintended generator issues: As the nature of the graph construction process involved identifying and resolving generator errors over many rounds, naturally the number of errors went down over time. At the end, domain experts worked to resolve all generator issues, and we accounted for remaining errors in the rubrics such that agents would not be penalized or affected by them.

5 Agreement between LLM graders and domain experts: After the rubrics were constructed, reviewed, and revised, we conducted an independent qualitative review of two models’ submissions for one binder. Domain experts reviewed all criteria that the LLM graders judged these submissions had failed to satisfy, and for this set of criteria, the domain experts agreed with the judgments reached by the LLM graders.

[1] https://arxiv.org/abs/2607.27189

[2] https://arxiv.org/abs/2606.08345

Thank you for the response.

The added details on the corpus size and the confidence intervals are helpful.

However, the response still relies on expert review, while the authors note that experts could not always judge the realism of modeled values without the private document (Section 4.3). The synthetic binders therefore need quantitative validation, and the added value of the DP priors is supported only qualitatively.

Overall, I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.