My main concern is the lack of quantitative validation of the synthetic data, despite the generation framework being presented as a core contribution. Although the authors acknowledge that realism and fidelity remain open challenges (Section 7), the paper describes the benchmark as realistic and representative and states, without supporting evidence, that performance on it correlates with performance in production audit settings (Appendix A.3). The evidence for realism appears limited to iterative feedback from the auditors who helped develop the generator; no blinded or independent assessment is reported. Beyond the average number of files per binder, the paper provides no descriptive statistics on revenue, balances, or transaction counts, nor an analysis of variation across the six companies. It also does not compare generated values with the DP priors from which they are sampled. This comparison is particularly important because propagation through the graph and resampling after failed LLM-generated feasibility tests may alter the resulting distributions.
Using DP to release statistics from confidential binders is a sensible design choice, but the added value of the resulting priors is not quantitatively established. This matters because the title, abstract and conclusion foreground these priors, and the introduction states that they are meant to reduce the expert feedback needed to correct implausible assumptions from public sources or LLMs. This benefit is supported only qualitatively (Section 4.3). The priors cover six ratios, sixteen record features and a few summaries, while the graph’s roughly 250 nodes and 1,000 edges were expanded using public resources, auditor expertise and LLMs, so it is unclear how much they shape the generated data. Reporting the number of historical binders and the estimation error of the released statistics, and even a rough measure of the effect (e.g., expert corrections needed with and without the priors on one engagement), would make this claim easier to assess.
Robustness to the sampled synthetic instances is another concern. With only six engagements, it is unclear whether model scores and rankings would remain stable if the binders were regenerated with different seeds. Repeating the evaluation across regenerated instances would test this sensitivity and help substantiate the claim that the framework supports scalable task construction. The binders were also designed to limit errors and ambiguities, and rubric items affected by unintended document inconsistencies are excluded (Appendix A.2), without reporting how many. Since incomplete, inconsistent, and poorly organized client data are an important source of difficulty in real audits, as the authors acknowledge, these design choices may cause the benchmark to underestimate task difficulty.
Evaluation reliability also needs stronger validation. Each workpaper is graded on about ten selected checks, some delegated to an LLM grader, but agreement with independent human judgment is not reported. The rubrics were revised after inspecting model submissions, leaving unclear whether the revisions were validated on held-out submissions. Model comparisons also appear sensitive to inference settings: GPT-6 Astra scores 43.61% at medium reasoning effort in the main table and 71.11% at maximum effort in Table 7, exceeding the best result in the main table. This sensitivity makes cross-model rankings difficult to interpret without comparable inference budgets. Finally, the paper reports no confidence intervals that account for the clustering of the 45 tasks within six binders.
Overall, the paper is most convincing as an expert-curated audit benchmark that incorporates private real-world statistics through DP. Its realism claims would be stronger with quantitative validation of the generated data, and even a limited comparison with alternative priors would clarify the role of the DP priors. I currently lean toward acceptance, as the benchmark is a valuable contribution, and would raise my score further with quantitative validation of the synthetic binders and evidence that model rankings are stable across regenerated instances.
Critical Weaknesses:
Inductive bias: You are correct that our synthetic data generation framework relies on a graph that is partially manually defined and extensively revised by domain experts. This is indeed meant to ensure that our tasks reflect the work auditors perform in actual client engagements. However, since the purpose of our benchmark is to measure how well agents can perform the work of human auditors as assessed by domain experts, we do not see this similarity (or "inductive bias") as a weakness, but rather as an inherent strength of our method. Synthetic data that is too dissimilar to real-world data does not allow us to assess how well agents would perform on realistic data, and we do not require them to follow any individual auditor’s reasoning process.
Questions for Authors:
Scaling the framework: Existing methods for constructing high quality benchmark tasks usually involve domain experts writing each task manually. This becomes incredibly challenging for tasks that span hundreds of documents. Our computational graph improves upon this by allowing for audit relationships and rules to be codified. Each derived relationship in the graph (the nodes and vertices) should always be reviewed by a domain expert to ensure sufficient reliability and accuracy. This is a constraint that scales with the size and density of the graph, but once such a graph is constructed, generating more samples can be done with less human oversight.