acceptodds
Under review as a conference paper at ICLR 2027

TaxLab: An AI Tax-Filing Benchmark Built on Field-Level Operationalization of Tax Law

Abstract

AI agents are increasingly capable of completing complex, multi-step tasks, making high-stakes applications, such as automated tax preparation, a promising direction. Evaluating tax-filing agents, however, remains difficult: it is hard to determine what range of tax situations a benchmark actually covers, how diverse those situations are, and where an agent's mistakes originate when a completed return is incorrect. We introduce TaxLab, a benchmark for interactive U.S. federal individual tax filing built on field-level operationalization of tax law. We construct TaxLab-Rule, an executable, source-grounded rule system comprising 3,142 rules, each linking a specific reporting field to the official sources that determine its treatment. This representation provides a common foundation for constructing gold returns, measuring benchmark coverage and diversity, and tracing filing errors back to the rules and sources involved. Using this system, we construct 55 CPA-reviewed tax-year-2025 scenarios and an interactive environment in which agents must identify missing taxpayer information, query a simulated taxpayer, consult offline IRS guidance, and complete the required tax forms. We evaluate nine models and find that tax preparation remains challenging: the strongest fully evaluated model, Claude Opus 5.5, achieves only 64.0% perfect filing accuracy despite substantially higher field-level accuracy. Our error diagnosis further reveals distinct issue-level failure patterns across models, showing that even strong models remain disproportionately error-prone on specific tax treatments. Together, these results show that substantial progress is still needed before AI agents can reliably prepare complete tax returns, and that improving aggregate accuracy alone is insufficient without addressing the specific tax treatments that drive filing failures.

Primary area:
datasets and benchmarks
Keywords:
Benchmarks, Tax preparation, Financial reasoning, Agentic Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_c1YkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper describes TAXLAB, a benchmark for measuring how agents can comply and accurately produce correct tax returns in the context of US law for federal taxation which comprises of 3,142 rules covering 63 tax issues from 2025 tax year snapshot. The paper also introduces a rule-based system for validating taxpayer facts and guidance to the reporting fields to properly evaluate accuracy and errors of AI agents. Results show that frontier AI agents like Fable and Astra perform well in terms of final return and compliance accuracy but ultimately achieve lower perfect filing accuracy which presents that tax filing with agents remains a challenge that the benchmark aims to shed light on and encourage more work.

Critical strengths

The benchmark tackles a very important and practical but difficult problem of tax filing and investigating if AI agents are empirically good at this. I positively see benchmark and evaluation contributions on this area of legal compliance and accuracy as necessary as long as a high-level of technical rigor is achieved. I see value from the TAXLAB benchmark given that it’s expert-designed and it seems arguably thorough in terms of the resource building process.

Critical weaknesses

The paper does not properly justify the quality of its contribution as a benchmark. The paper states that one of its contributions is “TAXLAB a dataset comprising 55 taxpayer scenarios whose gold returns are derived from our rule system for tax year 2025” and ends there. The first question that comes to mind is, is this even representative? Is this data substantial enough to leverage performance of AI agents for tax filing? How large were the datasets from previous works in order for readers to know that this 55-case dataset is a substantial contribution? The paper provides very limited context at all. Moreover, remember that readers from the ICLR community with no legal background might question how small this dataset is and whether it deserves a presentation to the conference.

The data is generated from a 2025 snapshot of US tax scenarios (tax-year-2025), how does the benchmark handle temporal shifts in tax scenarios from previous years and for years to come? Performance of any AI agent for this snapshot is unlikely to hold if tax scenarios are volatile (which the paper does not clarify).

Why is the addition of legal context experiments only applied to Astra? To me this is a more important primary-level result that deserves better spotlight than Table 2. Before you analyze the actions of an agent, you evaluate first the high-level performance of compliance and how additional (legal or policy) context and/or guidance improves this.

To me, Gemini 3.8 Flash, Claude Sonnet 5, and any other model that did not complete the runs must be omitted from the paper as there is no logic to compare them with models that did complete the runs. Moreover, even a small range of deducted runs (e.g., 2-3) can flip the results since the entire benchmark results are anchored on a single trial. I also do not get the time limit set for the models to finish the task if the paper’s main motivation is to look at their performance as the primary variable. Efficiency can be discussed later in the paper but without the uniform comparison of performance, it makes the results questionable. This overall setup by the authors does not make me confident with the results.

Questions for authors

See questions raised above.

5 comments

We thank the reviewer for the valuable feedback. In the following, we first clarify three points regarding the paper and evaluation setup, then address the remaining concerns.

AuthorsOpenReview ↗

Clarifications to Misunderstanding

M1. Comparison with previous benchmarks

How large were the datasets from previous works in order for readers to know that this 55-case dataset is a substantial contribution? The paper provides very limited context at all. Moreover, remember that readers from the ICLR community with no legal background might question how small this dataset is and whether it deserves a presentation to the conference.

We would like to clarify that Table 1 explicitly reports the number of cases in previous benchmarks in its "Cases" column. Two prior benchmarks contain 50 and 32 cases, respectively. The other two do not publicly release their datasets or report the number of cases, preventing a direct comparison.

Our 55-case benchmark is therefore comparable in scale to existing benchmarks with publicly reported case counts, while providing broader rule-level coverage and diversity.

M2. Legal context in the main evaluation

Why is the addition of legal context experiments only applied to Astra? To me this is a more important primary-level result that deserves better spotlight than Table 2.

We would like to clarify that all agents in Table 2 are already evaluated with access to legal context (IRS documentation), as described in Figure 1 and Section 3.3. The Astra-specific experiment is an additional ablation examining the effect of removing this context.

As stated in our introduction, our primary goal is to evaluate agents' ability to reliably prepare complete tax returns. We therefore argue that the main evaluation should prioritize the more realistic setting in which agents have access to official IRS guidance, rather than focusing primarily on measuring the impact of providing such guidance.

M3. Motivation for execution timeouts

I also do not get the time limit set for the models to finish the task if the paper’s main motivation is to look at their performance as the primary variable. Efficiency can be discussed later in the paper but without the uniform comparison of performance, it makes the results questionable.

We would like to clarify that the timeout is intended to ensure termination, not to evaluate efficiency. Imposing execution limits is common practice in agent benchmarks, such as Terminal-Bench, MLE-bench, and RE-Bench.

This safeguard is necessary because agents may become stuck in repetitive reasoning or tool-use loops, as discussed by Yang et al. (2025). We also observed such repetitive execution patterns in our evaluations of Gemini and Sonnet 5.

More fundamentally, there is no general method to guarantee termination of arbitrary programs (the halting problem), and the use of black-box AI models introduces additional uncertainty. Without execution limits, some evaluation runs could continue indefinitely, making reliable benchmark execution impractical.

We will clarify this motivation in the revised paper.

AuthorsOpenReview ↗

Responses to Other Concerns

C1. Representativeness of the 55 cases

Thank you for raising this important question. Establishing whether a benchmark statistically represents the distribution of real-world tasks is a recognized challenge in benchmark construction, as discussed by Dehghani et al. (2021).

As in other expert-designed benchmarks, we worked with CPA to construct cases informed by their extensive experience with real-world tax filing.

Beyond case count and representativeness, our rule system provides explicit measurements of tax-rule coverage and diversity. These measurements allow us to assess which tax treatments are exercised by the benchmark, rather than relying solely on the number of cases.

C2. Generalization across tax years

Thank you for this question. Our rule system is designed to facilitate annual updates by construction. Specifically, each rule records its official source, allowing us to identify which rules are affected by changes in tax law or IRS guidance. The IRS also publishes annual "What's New" summaries in the Form 1040 instructions and Publication 17, which help identify relevant changes.

For example, most routine annual adjustments are numerical parameter changes. For example, the standard deduction for married filing jointly increased from 29,200 USD in 2024 to 31,500 USD in 2025. Such changes can be incorporated by simply updating the corresponding parameters.

C3. Fair comparison across models with incomplete runs

We report model accuracy with timed-out and failed runs included in the denominator (i.e., these runs are counted as incorrect answers rather than excluded from the denominator), as shown in the attached results.

ModelFRA %CFA %PFA %
Claude Opus 5.561.876.958.2
Claude Fable 5.181.891.961.8
GPT-6 Astra69.186.561.8
GPT-5.6 Sol61.875.330.9
GPT-5.547.363.216.4
Gemini 3.8 Flash9.19.71.8
Claude Sonnet 533.345.76.1
Claude Haiku 4.53.611.70.0
GPT-5.6 Luna0.09.60.0

Additionally, we report model accuracy on the 46 cases successfully completed by all compared models (excluding Gemini and Sonnet 5 which failed to complete more than 50% of the benchmark cases). This enables a direct, apples-to-apples comparison across models on the same set of cases.

ModelFRA (%)CFA (%)PFA (%)MCP callsInput (k)Output (k)Time (min)
Claude Opus 5.567.485.663.09.67620.983.319.4
Fable 5.182.691.360.910.52360.263.917.9
GPT-6 Astra69.685.560.913.32689.220.212.8
GPT-5.6 Sol65.276.234.810.55440.741.322.5
GPT-5.552.266.717.410.04260.132.818.3
Claude Haiku4.311.90.07.44463.144.622.0
GPT-5.6 Luna0.09.80.08.32209.915.211.9

C4. Variability across repeated runs

Thank you for highlighting this concern. To assess consistency, we conducted an additional experiment in which 10 sampled cases were evaluated three times per model, excluding Gemini and Sonnet 5. Model rankings remained consistent across all three runs. The table below reports the mean ± standard deviation across runs.

ModelFRA (%)CFA (%)PFA (%)
Fable 5.180.0 ± 0.088.6 ± 7.160.0 ± 10.0
Claude Opus 5.573.3 ± 5.882.9 ± 2.256.7 ± 5.8
GPT-6 Astra60.0 ± 0.082.7 ± 3.250.0 ± 10.0
GPT-5.6 Sol53.3 ± 5.867.7 ± 7.230.0 ± 10.0
GPT-5.530.0 ± 0.060.6 ± 5.813.3 ± 5.8
GPT-5.6 Luna0.0 ± 0.012.9 ± 5.40.0 ± 0.0
Claude Haiku0.0 ± 0.07.1 ± 6.60.0 ± 0.0
Reviewer_YN8kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TaxLab, an agentic benchmark consisting of 55 tax filing scenarios. The evaluation pipeline has three main components: (1) identifying tax issues from NTA-reported cases and extracting applicable tax rules; (2) generating scenarios based on the identified issues and executing the rule system on LLM-generated taxpayer facts to produce gold returns; and (3) evaluating agents using the harnesses of nine models, including Codex, Claude Code, and Antigravity. During evaluation, agents are provided with scenario information, fixed offline IRS rules, and tools.

The paper reports evaluation results across nine models, showing that tax filing remains challenging even for frontier models. For example, Claude Opus 5.5 achieves only 64.0% perfect filing accuracy. The paper also compares TaxLab with existing tax filing benchmarks in terms of measurable coverage and diversity. TaxLab achieves a higher average pairwise Jaccard distance, suggesting greater diversity among its scenarios.

Overall, this paper addresses an important problem in applying LLM-based agents to tax filing. The research question is novel and valuable, and the work helps address gaps in existing research.

Critical strengths

This paper has several major strengths:

  1. Novel, well-motivated, and meaningful research problem. This paper focuses on evaluating agents' capabilities in tax filing, an important research area that has not been fully explored in existing work. The authors identify three limitations of current expert-knowledge benchmarks: (1) coverage and diversity are difficult to measure; (2) manually constructed ground truth is prone to errors (the authors cite more than 10 errors reported in TaxCalcBench); and (3) evaluation failures are difficult to trace to their underlying causes.

  2. Realistic evaluation environment. The evaluation environment closely reflects real-world tax preparation. The paper separates information into initially available and withheld facts, with a simulated taxpayer who only provides information when asked. This setup evaluates whether agents can identify missing information and ask relevant follow-up questions. Together with offline IRS guidance and editable tax form workbooks, the environment provides a reasonable approximation of end-to-end tax filing rather than a simple tax question-answering task.

  3. Substantial benchmark development and infrastructure. The benchmark includes 3,142 rules covering 63 tax issues, with each rule verified by a CPA against its cited source. The 55 CPA-reviewed scenarios and tool-based agent evaluation environment also represent substantial engineering efforts.

Critical weaknesses

This paper has the following major weaknesses:

  1. Limitations in the evaluation setup and reliability of reported model comparisons. The evaluation excludes 56 timed-out runs and 33 runtime failures from accuracy calculations, resulting in substantially different numbers of graded cases across models. This affects the reported rankings: for example, Claude Opus 5.5's PFA drops from 64.0% to 58.2% when unsuccessful runs are counted over all 55 tasks, below Fable 5.1 and GPT-6 Astra. In addition, the simulated taxpayer oracle relies on GPT-6 Astra without a reported accuracy or consistency audit, and different models use different agent harnesses. These factors make it difficult to separate model capability from evaluation setup effects and raise concerns about the reliability of cross-model comparisons.

  2. Insufficient validation of benchmark coverage claims and inconsistent reporting. The paper reports that TaxLab covers all 63 tax issues while TaxCalcBench covers only 47.6%, but the completeness and representativeness of the underlying 63-issue taxonomy have not been independently established. Since TaxLab scenarios are generated specifically to cover this taxonomy, the reported coverage primarily demonstrates completeness relative to the authors' own issue set rather than broader coverage of federal tax law. Additionally, the Introduction reports TaxCalcBench coverage as 57.1%, contradicting the 47.6% reported in Table 1 and Related Work. These issues weaken the interpretation and clarity of the benchmark coverage comparison.

  3. Writing quality and related work. The paper also has several issues with its presentation and literature review. First, the reference list contains approximately 20–30 entries, more than one-third of which are vendor blog posts, marketing pages, or GitHub issue trackers. The paper omits several relevant lines of research, including agentic benchmarks (e.g., AgentBench and τ-bench, the latter of which uses an LLM-simulated user similar to the taxpayer oracle in this work), legal and tax NLP benchmarks beyond SARA (e.g., LegalBench and work by Blair-Stanek et al. and Nay et al.), and prior research on benchmark label errors and dataset documentation. These works are particularly relevant to the three challenges identified by the authors. Second, the paper contains several typos and formatting issues. For example, "Inland Revenue Code" should likely be "Internal Revenue Code," and line 286 contains a broken sentence.

Questions for authors
  1. How are anchor fields selected, and how many are used per case (including their distribution across cases)? How does the grader handle legitimate alternative tax elections (e.g., standard vs. itemized deductions) that produce different but legally valid returns?

  2. Which coverage figure for TaxCalcBench is correct (57.1% or 47.6%)? More broadly, how did the authors validate the completeness of the 63-issue taxonomy relative to the scope of federal individual income tax? How was the accuracy of the manually reconstructed TaxCalcBench cases verified?

  3. Was the accuracy of the taxpayer oracle audited (e.g., by checking a sample of its responses against the hold-back facts)? How does the oracle handle questions whose answers are not included in the hold-back information?

  4. How do the model rankings change when timed-out and failed runs are included in the denominator? Can the authors report end-to-end task success rates alongside accuracy on completed cases, particularly given the substantial differences in completion rates across models?

I would be willing to raise my score if the authors address the major numerical inconsistency and experiment setup limitations.

6 comments

We thank the reviewer for the constructive feedback, particularly for identifying a numerical inconsistency in our paper. Below, we address the questions and concerns. We will incorporate the clarifications and additional results into the revised paper.

AuthorsOpenReview ↗

Responses to Questions 1 and 2

Q1a: How are anchor fields selected, and how many are used per case?

Anchor fields are selected by the CPA based on two considerations: (1) their importance in professional tax-return review and (2) minimizing redundant scoring of fields affected by the same underlying error. Because tax fields are interdependent, complete independence is not always possible.

Across the 55 cases, the benchmark scores 1,746 anchor fields, with 11 to 57 per case (25th percentile 22, median 32, 75th percentile 40; mean 31.7, SD 10.6).

Q1b: How does the grader handle legitimate alternative tax elections?

Thank you for raising this question. We would like to clarify that TaxLab evaluates not only whether an agent produces a legally valid return, but also whether it identifies the most beneficial filing choices for the taxpayer.

We explicitly design cases to exercise such choices. For example, agents must decide between standard and itemized deductions, or between deducting state and local income taxes and sales taxes under Schedule A.

Our rule system also accounts for legal constraints on these choices. For example, under IRC Section 63(c)(6)(A), a married taxpayer filing separately cannot claim the standard deduction if their spouse itemizes.

We will clarify this evaluation objective and the treatment of alternative elections.

Q2a: Which TaxCalcBench coverage figure is correct?

47.6% is the correct figure. We apologize for the inconsistency. The 57.1% reported in the Introduction is an outdated result. We will correct it.

TaxCalcBench includes cases from different tax years. In our initial analysis, we mistakenly evaluated some non-TY2025 cases under our TY2025 rule system. After correcting this issue and restricting the comparison to applicable TY2025 cases, the coverage decreased to 47.6%.

Q2b: How was the completeness of the 63-issue taxonomy validated?

The taxonomy was developed iteratively with CPA. During rule development, CPA reviewed the covered tax situations and identified additional situations requiring new issues or rules.

We further assess coverage by mapping our rules to official IRS forms and schedules. Our benchmark covers all relevant forms and schedules, excluding those related to foreign taxation (which is out of scope, as indicated in the paper).

Q2c: How was the accuracy of the reconstructed TaxCalcBench cases verified?

We would like to clarify that we do not report TaxCalcBench's filing accuracy. Our discussion of TaxCalcBench's known errors includes only errors confirmed by its authors or reported by other users and subsequently verified by our CPAs. We deliberately adopt a conservative approach when reporting errors in prior benchmarks.

We also note that converting TaxCalcBench's cases (to our rule engine) is straightforward, as it only involves translating basic taxpayer information from their format into ours, without requiring tax expertise. The conversion was performed by authors and independently cross-checked.

Additionally, we had tax professionals verify the correctness of the conversions during the rebuttal period.

AuthorsOpenReview ↗

Responses to Questions 3 and 4

Q3a: Was the taxpayer oracle's accuracy audited?

We constrain the simulated taxpayer to a simple information-retrieval task, i.e., answering individual factual questions using the held-out information or indicating that the requested information is unavailable. We use GPT-6 Astra with maximum reasoning effort for this relatively straightforward task.

To assess reliability, we randomly sampled 350 of the 3,899 simulated responses for manual verification. Following Cochran's sample-size formula, this sample size provides a 95% confidence level with a worst-case margin of error of 5 percentage points.

All 350 responses were substantively correct. Three contained minor representation differences (e.g., 0.45% versus 0.0045), without changing the underlying information.

Q3b: How does the oracle handle questions whose answers are unavailable?

When the requested information is not available in the held-out facts, the oracle responds "I don't know" rather than inventing an answer. This models situations in which a real taxpayer cannot provide the requested information.

Q4: How do model rankings change when timed-out and failed runs are included?

Thank you for this suggestion.

we report model accuracy with timed-out and failed runs included in the denominator (i.e., these runs are counted as incorrect answers rather than excluded from the denominator), as shown in the attached results.

ModelFRA %CFA %PFA %
Claude Opus 5.561.876.958.2
Claude Fable 5.181.891.961.8
GPT-6 Astra69.186.561.8
GPT-5.6 Sol61.875.330.9
GPT-5.547.363.216.4
Gemini 3.8 Flash9.19.71.8
Claude Sonnet 533.345.76.1
Claude Haiku 4.53.611.70.0
GPT-5.6 Luna0.09.60.0

Additionally, we report model accuracy on the 46 cases successfully completed by all compared models (excluding Gemini and Sonnet 5 which failed to complete more than 50% of the benchmark cases). This enables a direct, apples-to-apples comparison across models on the same set of cases.

ModelFRA (%)CFA (%)PFA (%)MCP callsInput (k)Output (k)Time (min)
Claude Opus 5.567.485.663.09.67620.983.319.4
Fable 5.182.691.360.910.52360.263.917.9
GPT-6 Astra69.685.560.913.32689.220.212.8
GPT-5.6 Sol65.276.234.810.55440.741.322.5
GPT-5.552.266.717.410.04260.132.818.3
Claude Haiku4.311.90.07.44463.144.622.0
GPT-5.6 Luna0.09.80.08.32209.915.211.9
Reviewer_ZWa3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents TAXLAB, a benchmark for evaluating AI agents on U.S. individual tax filing. Its key contributions are:

  1. TAXLAB, a benchmark of 55 CPA-reviewed tax-year-2025 scenarios with systematically generated gold returns and execution traces.
  2. TAXLAB-RULE, a system of 3,142 rules organized around tax-form fields that links reporting decisions to taxpayer facts, forms, and official sources.
  3. An interactive tax-filing environment in which agents identify missing information, ask for needed facts, consult IRS guidance, and complete the required federal tax forms.

The evaluation covers nine models and shows that tax filing remains challenging even for frontier models, with different agents showing different patterns of error.

Critical strengths
  • The benchmark includes CPA-reviewed tax scenarios, gold returns, and execution traces. -The rule system connects gold-return generation, coverage measurement, and error analysis through explicit links to tax rules and official sources.
  • The interactive environment tests whether agents recognize missing taxpayer information and ask appropriate follow-up questions.
  • The evaluation covers nine frontier models, with detailed error analysis highlighting where each model struggles.
  • As expected, providing IRS documents improves GPT-6 Astra’s accuracy; the ablation quantifies this improvement.
Critical weaknesses
  • Accuracy is reported only for completed and graded cases, and these numbers vary considerably across models. For example, Opus gets 32 perfect returns out of 50 graded cases, while Fable and Astra each get 34 out of 55. This makes the rankings difficult to compare.
  • Covering all 63 tax issues does not necessarily mean good coverage of the 3,142 rules, their branches, boundary cases, exceptions, or interactions. Relational and higher-order metamorphic tests could complement the individual scenarios (see [1–3]).
  • Relevant work on metamorphic testing of tax-preparation software appears to be missing, including LLM-based generation of metamorphic specifications and agentic development with higher-order metamorphic testing (see, [1–3]). Discussing this work would help place the contribution in context, especially regarding the oracle problem.
  • With only 55 scenarios and no repeated runs or uncertainty estimates reported, it is hard to judge small performance differences. Some issue-level findings could also be based on very few cases.
  • The error analysis shows where models make mistakes, but not necessarily why. Examples separating missing information, misunderstanding of tax rules, calculation errors, and errors propagated from earlier steps would be useful.

[1] Metamorphic Testing and Debugging of Tax Preparation Software. https://arxiv.org/abs/2205.04998

[2] On the Potential and Limitations of Few-Shot In-Context Learning to Generate Metamorphic Specifications for Tax Preparation Software. https://arxiv.org/abs/2311.11979

[3] An LLM Agentic Approach for Legal-Critical Software: A Case Study for Tax Prep Software. https://arxiv.org/abs/2509.13471

Questions for authors
  1. What fraction of the 3,142 rules is exercised by the benchmark? How well do the scenarios cover boundary cases, exceptions, and interactions between rules?
  2. How many CPAs were involved in validation? Were the gold returns independently recomputed or checked against other tax software?
  3. How do you check that the simulated taxpayer answers accurately, without omitting requested facts or providing unintended hints?
  4. How consistent are the results across repeated runs? For each tax issue, how many cases were evaluated and how many had errors?
  5. How would the benchmark be updated as tax laws, thresholds, and forms change? How much expert effort would this require?
4 comments

We thank the reviewer for the constructive feedback. Below, we address the questions and concerns. We will incorporate the additional results and discussions into the revised paper.

AuthorsOpenReview ↗

Responses to Questions

Q1. Rule coverage and boundary cases.

Thank you for raising this important point.

Our benchmark exercises 72.7% of the 3,142 rules. Some unexercised rules are outside the benchmark's scope by design (e.g., rules determining whether a taxpayer needs to file are not exercised because all our cases require filing). Other unexercised rules represent similar tax treatments that differ mainly in numerical thresholds or rates. Our CPA considered these variants largely redundant and did not prioritize them.

Our rule system also allows us to programmatically measure coverage of exceptions, boundary cases, and rule interactions:

  • Exceptions: Exception rules are explicitly encoded in our rule system and assigned priority tier 90. Our benchmark exercises 41.8% of these rules.
  • Boundary cases: We define a boundary case as one in which a rule's input exactly matches a numerical threshold in its guard. Our benchmark exercises 71 boundary hits.
  • Rule interactions: We identify an interaction when one rule's output is directly used as another rule's input, and the two rules belong to different tax issues. On average, each case exercises 191.4 such interaction pairs.

Q2. CPA validation and independent verification.

Given the high hourly cost of CPA review, we engaged one CPA to validate all 3,142 tax rules and 55 cases, providing reasonable assurance. Two tax professionals also reviewed the cases.

We further cross-checked the gold returns using FreeTaxUSA. We successfully verified 49 of the 55 cases. The remaining six involve tax treatments unsupported by FreeTaxUSA, such as like-kind exchanges (Form 8824), ISO-related alternative minimum tax adjustments (Form 6251) and at risk limitations (Form 6198).

Q3. Accuracy of simulated taxpayer responses.

We constrain the simulated taxpayer to an information-retrieval task (i.e., answering single-fact questions using the held-out information or stating the information is missing). We use GPT-6 Astra with maximum reasoning effort for this relatively straightforward task.

To assess reliability, we randomly sampled 350 of the 3,899 simulated responses for manual verification. Following Cochran's formula, this sampling provides a 95% confidence level.

All 350 responses were correct. Three contained minor representation differences (e.g., 0.45% versus 0.0045).

Q4a. Repeated-run consistency.

To assess run-to-run consistency, we evaluated each model (excluding Gemini and Sonnet 5) three times on the same 10 randomly sampled cases. Model rankings remained consistent across all three runs. Details are shown in the table below.

ModelFRA (%)CFA (%)PFA (%)
Fable 5.180.0 ± 0.088.6 ± 7.160.0 ± 10.0
Claude Opus 5.573.3 ± 5.882.9 ± 2.256.7 ± 5.8
GPT-6 Astra60.0 ± 0.082.7 ± 3.250.0 ± 10.0
GPT-5.6 Sol53.3 ± 5.867.7 ± 7.230.0 ± 10.0
GPT-5.530.0 ± 0.060.6 ± 5.813.3 ± 5.8
GPT-5.6 Luna0.0 ± 0.012.9 ± 5.40.0 ± 0.0
Claude Haiku0.0 ± 0.07.1 ± 6.60.0 ± 0.0

Q4b. issue-level errors

Figure 4 in our paper reports tax issues covered by each case.

We additionally provide per-model error rates for each tax topic (i.e., calculated over all cases exercising that topic) in the table below (shown as %).

TopicOpus 5.5Fable 5.1AstraSolGPT-5.5FlashSonnetHaikuLuna
Filing & Comp.33182436539376100100
Income1379203087437891
Deductions302226436794668993
Property16003242959010095
Credits & Taxes3833334575958598100
Entities211414507186579371

Note that each tax topic covers multiple tax issues. We will also include an issue-level breakdown in the revised paper, as OpenReview space limits prevent us from presenting the full data here.

Q5. Updating the benchmark for future tax years.

Our rule system is designed to make annual updates easier. Each rule records its official source, allowing us to identify affected rules when tax guidance changes. The IRS also publishes annual "What's New" summaries in its Form 1040 instructions, which help identify relevant changes.

Most routine updates involve numerical thresholds. Such changes can be straightforwardly incorporated by updating the corresponding parameters.

Substantive changes require more expert effort but are relatively infrequent. For example, the 2025 tax law introduced a new deduction of up to $25,000 for qualified tips, subject to eligibility requirements and phaseouts. Such changes require expert review and potentially new rules. However, such changes occur much less frequently.

AuthorsOpenReview ↗

Clarification to Weaknesses

W1. Comparability across models with different numbers of completed cases.

Thank you for this suggestion.

We report model accuracy with timed-out and failed runs included in the denominator (i.e., these runs are counted as incorrect answers rather than excluded from the denominator), as shown in the attached results.

ModelFRA %CFA %PFA %
Claude Opus 5.561.876.958.2
Claude Fable 5.181.891.961.8
GPT-6 Astra69.186.561.8
GPT-5.6 Sol61.875.330.9
GPT-5.547.363.216.4
Gemini 3.8 Flash9.19.71.8
Claude Sonnet 533.345.76.1
Claude Haiku 4.53.611.70.0
GPT-5.6 Luna0.09.60.0

Additionally, we report model accuracy on the 46 cases successfully completed by all compared models (excluding Gemini and Sonnet 5 which failed to complete more than 50% of the benchmark cases). This enables a direct, apples-to-apples comparison across models on the same set of cases.

ModelFRA (%)CFA (%)PFA (%)MCP callsInput (k)Output (k)Time (min)
Claude Opus 5.567.485.663.09.67620.983.319.4
Fable 5.182.691.360.910.52360.263.917.9
GPT-6 Astra69.685.560.913.32689.220.212.8
GPT-5.6 Sol65.276.234.810.55440.741.322.5
GPT-5.552.266.717.410.04260.132.818.3
Claude Haiku4.311.90.07.44463.144.622.0
GPT-5.6 Luna0.09.80.08.32209.915.211.9

W2. Related work on metamorphic testing.

Thank you for pointing out these relevant studies! We agree that metamorphic testing is a valuable approach that could further strengthen our benchmark. We will include and discuss these studies in the revised paper.

We will also clarify that our rule system is designed to be accessible to CPAs and other tax professionals. Specifically, the rules are written largely in natural language and include explicit provenance annotations, making them easier for tax professionals to review, author, and maintain.

W3. Root-cause analysis of agent errors.

Thank you for this suggestion. We conducted an additional manual root-cause analysis of 181 sampled agent trials (using the same confidence level described above) to understand why errors occur, rather than only where they appear.

The analysis identified several recurring causes:

Root causeDescriptionFailing runs affected
Missing informationAgent did not request necessary facts or made unsupported assumptions62%
Tax-rule misunderstandingIncorrect applicability, parameters, or classification36%
Calculation errorsCorrect rule and inputs but incorrect arithmetic or rounding13%
Source-document misreadingIncorrect values extracted from provided documents3%

For example, one agent incorrectly excluded a $261,949 home-sale gain without accounting for the taxpayer's filing status. This is a misunderstanding of the applicable tax treatment rather than an arithmetic error.

We will include these findings in the revised paper.

Thank you for the detailed response. The rule coverage numbers and the comparison on the same set of cases were especially helpful.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

Summary of Rebuttal and Revisions

We thank all three reviewers for their constructive feedback and thoughtful discussions. Below, we summarize how we addressed their main concerns.


  • Reviewer ZWa3 provided exceptionally thoughtful and constructive feedback, raising important questions about rule coverage and comparisons across the same set of cases. These suggestions helped us identify opportunities to strengthen both the benchmark analysis and the evaluation methodology. In response, we provided detailed coverage measurements, showing that TaxLab exercises 72.7% of its 3,142 rules, along with additional analyses of boundary cases, exceptions, and rule interactions. We also reported results on 46 matched cases to enable fairer comparisons across models. The reviewer acknowledged that these additions were particularly helpful. We are especially grateful for the reviewer's careful assessment and actionable suggestions, which have meaningfully improved the quality and rigor of our paper.

  • Reviewer YN8k provided valuable and thoughtful feedback that helped us further strengthen our benchmark. In the follow-up discussion, the reviewer raised two remaining questions regarding CPA involvement and optimal filing choices. We clarified that one CPA reviewed all 3,142 rules and 55 cases, with two additional tax professionals also reviewing the cases. We further cross-checked 49 of the 55 gold returns using FreeTaxUSA. Given the substantial time and cost of professional review, we believe this represents an unusually rigorous validation effort. To our knowledge, no prior tax-filing benchmark has undertaken comparable professional validation covering both its entire rule system and all benchmark cases. Regarding optimality, our rule system systematically explores legally available filing choices to identify the optimal return, which is further checked through professional review. We sincerely appreciate the reviewer's careful consideration and constructive suggestions.

  • Reviewer c1Yk initially raised two concerns that we clarified by pointing to information already provided in the original paper: the comparison with prior benchmark sizes in Table 1 and the fact that all agents in our main evaluation have access to IRS guidance. The reviewer subsequently raised two further questions: "Does agent filing performance generalize across different tax years?" and "Why does the benchmark need execution timeouts?" Although we did not receive further feedback on our proposed experiments, we completed both to provide additional empirical evidence. First, we adapted 10 cases to tax year 2024 and evaluated Opus 5.5 and GPT-6 Astra. Both models achieved the same FRA as on their 2025 counterparts, providing evidence of consistent performance across the two tax years. Second, we evaluated three alternative task framings on 10 sampled cases using Gemini. None resolved the repetitive reasoning loops that prevented termination, further supporting the need for execution limits in reliable agent evaluation. We hope these additional results help address the reviewer's remaining concerns.

We have attached the revised manuscript, with all changes highlighted in red.

We sincerely appreciate the reviewers' time and feedback, which have helped us improve the paper. We believe the additional experiments, analyses, and clarifications have substantially strengthened our work and addressed the key concerns raised during the review process. We thank the reviewers again for their thoughtful consideration of our revised submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.