- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper describes TAXLAB, a benchmark for measuring how agents can comply and accurately produce correct tax returns in the context of US law for federal taxation which comprises of 3,142 rules covering 63 tax issues from 2025 tax year snapshot. The paper also introduces a rule-based system for validating taxpayer facts and guidance to the reporting fields to properly evaluate accuracy and errors of AI agents. Results show that frontier AI agents like Fable and Astra perform well in terms of final return and compliance accuracy but ultimately achieve lower perfect filing accuracy which presents that tax filing with agents remains a challenge that the benchmark aims to shed light on and encourage more work.
Critical strengths
The benchmark tackles a very important and practical but difficult problem of tax filing and investigating if AI agents are empirically good at this. I positively see benchmark and evaluation contributions on this area of legal compliance and accuracy as necessary as long as a high-level of technical rigor is achieved. I see value from the TAXLAB benchmark given that it’s expert-designed and it seems arguably thorough in terms of the resource building process.
Critical weaknesses
The paper does not properly justify the quality of its contribution as a benchmark. The paper states that one of its contributions is “TAXLAB a dataset comprising 55 taxpayer scenarios whose gold returns are derived from our rule system for tax year 2025” and ends there. The first question that comes to mind is, is this even representative? Is this data substantial enough to leverage performance of AI agents for tax filing? How large were the datasets from previous works in order for readers to know that this 55-case dataset is a substantial contribution? The paper provides very limited context at all. Moreover, remember that readers from the ICLR community with no legal background might question how small this dataset is and whether it deserves a presentation to the conference.
The data is generated from a 2025 snapshot of US tax scenarios (tax-year-2025), how does the benchmark handle temporal shifts in tax scenarios from previous years and for years to come? Performance of any AI agent for this snapshot is unlikely to hold if tax scenarios are volatile (which the paper does not clarify).
Why is the addition of legal context experiments only applied to Astra? To me this is a more important primary-level result that deserves better spotlight than Table 2. Before you analyze the actions of an agent, you evaluate first the high-level performance of compliance and how additional (legal or policy) context and/or guidance improves this.
To me, Gemini 3.8 Flash, Claude Sonnet 5, and any other model that did not complete the runs must be omitted from the paper as there is no logic to compare them with models that did complete the runs. Moreover, even a small range of deducted runs (e.g., 2-3) can flip the results since the entire benchmark results are anchored on a single trial. I also do not get the time limit set for the models to finish the task if the paper’s main motivation is to look at their performance as the primary variable. Efficiency can be discussed later in the paper but without the uniform comparison of performance, it makes the results questionable. This overall setup by the authors does not make me confident with the results.
Questions for authors
See questions raised above.
5 comments
Clarifications to Misunderstanding
M1. Comparison with previous benchmarks
How large were the datasets from previous works in order for readers to know that this 55-case dataset is a substantial contribution? The paper provides very limited context at all. Moreover, remember that readers from the ICLR community with no legal background might question how small this dataset is and whether it deserves a presentation to the conference.
We would like to clarify that Table 1 explicitly reports the number of cases in previous benchmarks in its "Cases" column. Two prior benchmarks contain 50 and 32 cases, respectively. The other two do not publicly release their datasets or report the number of cases, preventing a direct comparison.
Our 55-case benchmark is therefore comparable in scale to existing benchmarks with publicly reported case counts, while providing broader rule-level coverage and diversity.
M2. Legal context in the main evaluation
Why is the addition of legal context experiments only applied to Astra? To me this is a more important primary-level result that deserves better spotlight than Table 2.
We would like to clarify that all agents in Table 2 are already evaluated with access to legal context (IRS documentation), as described in Figure 1 and Section 3.3. The Astra-specific experiment is an additional ablation examining the effect of removing this context.
As stated in our introduction, our primary goal is to evaluate agents' ability to reliably prepare complete tax returns. We therefore argue that the main evaluation should prioritize the more realistic setting in which agents have access to official IRS guidance, rather than focusing primarily on measuring the impact of providing such guidance.
M3. Motivation for execution timeouts
I also do not get the time limit set for the models to finish the task if the paper’s main motivation is to look at their performance as the primary variable. Efficiency can be discussed later in the paper but without the uniform comparison of performance, it makes the results questionable.
We would like to clarify that the timeout is intended to ensure termination, not to evaluate efficiency. Imposing execution limits is common practice in agent benchmarks, such as Terminal-Bench, MLE-bench, and RE-Bench.
This safeguard is necessary because agents may become stuck in repetitive reasoning or tool-use loops, as discussed by Yang et al. (2025). We also observed such repetitive execution patterns in our evaluations of Gemini and Sonnet 5.
More fundamentally, there is no general method to guarantee termination of arbitrary programs (the halting problem), and the use of black-box AI models introduces additional uncertainty. Without execution limits, some evaluation runs could continue indefinitely, making reliable benchmark execution impractical.
We will clarify this motivation in the revised paper.
Responses to Other Concerns
C1. Representativeness of the 55 cases
Thank you for raising this important question. Establishing whether a benchmark statistically represents the distribution of real-world tasks is a recognized challenge in benchmark construction, as discussed by Dehghani et al. (2021).
As in other expert-designed benchmarks, we worked with CPA to construct cases informed by their extensive experience with real-world tax filing.
Beyond case count and representativeness, our rule system provides explicit measurements of tax-rule coverage and diversity. These measurements allow us to assess which tax treatments are exercised by the benchmark, rather than relying solely on the number of cases.
C2. Generalization across tax years
Thank you for this question. Our rule system is designed to facilitate annual updates by construction. Specifically, each rule records its official source, allowing us to identify which rules are affected by changes in tax law or IRS guidance. The IRS also publishes annual "What's New" summaries in the Form 1040 instructions and Publication 17, which help identify relevant changes.
For example, most routine annual adjustments are numerical parameter changes. For example, the standard deduction for married filing jointly increased from 29,200 USD in 2024 to 31,500 USD in 2025. Such changes can be incorporated by simply updating the corresponding parameters.
C3. Fair comparison across models with incomplete runs
We report model accuracy with timed-out and failed runs included in the denominator (i.e., these runs are counted as incorrect answers rather than excluded from the denominator), as shown in the attached results.
| Model | FRA % | CFA % | PFA % |
|---|---|---|---|
| Claude Opus 5.5 | 61.8 | 76.9 | 58.2 |
| Claude Fable 5.1 | 81.8 | 91.9 | 61.8 |
| GPT-6 Astra | 69.1 | 86.5 | 61.8 |
| GPT-5.6 Sol | 61.8 | 75.3 | 30.9 |
| GPT-5.5 | 47.3 | 63.2 | 16.4 |
| Gemini 3.8 Flash | 9.1 | 9.7 | 1.8 |
| Claude Sonnet 5 | 33.3 | 45.7 | 6.1 |
| Claude Haiku 4.5 | 3.6 | 11.7 | 0.0 |
| GPT-5.6 Luna | 0.0 | 9.6 | 0.0 |
Additionally, we report model accuracy on the 46 cases successfully completed by all compared models (excluding Gemini and Sonnet 5 which failed to complete more than 50% of the benchmark cases). This enables a direct, apples-to-apples comparison across models on the same set of cases.
| Model | FRA (%) | CFA (%) | PFA (%) | MCP calls | Input (k) | Output (k) | Time (min) |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 67.4 | 85.6 | 63.0 | 9.6 | 7620.9 | 83.3 | 19.4 |
| Fable 5.1 | 82.6 | 91.3 | 60.9 | 10.5 | 2360.2 | 63.9 | 17.9 |
| GPT-6 Astra | 69.6 | 85.5 | 60.9 | 13.3 | 2689.2 | 20.2 | 12.8 |
| GPT-5.6 Sol | 65.2 | 76.2 | 34.8 | 10.5 | 5440.7 | 41.3 | 22.5 |
| GPT-5.5 | 52.2 | 66.7 | 17.4 | 10.0 | 4260.1 | 32.8 | 18.3 |
| Claude Haiku | 4.3 | 11.9 | 0.0 | 7.4 | 4463.1 | 44.6 | 22.0 |
| GPT-5.6 Luna | 0.0 | 9.8 | 0.0 | 8.3 | 2209.9 | 15.2 | 11.9 |
C4. Variability across repeated runs
Thank you for highlighting this concern. To assess consistency, we conducted an additional experiment in which 10 sampled cases were evaluated three times per model, excluding Gemini and Sonnet 5. Model rankings remained consistent across all three runs. The table below reports the mean ± standard deviation across runs.
| Model | FRA (%) | CFA (%) | PFA (%) |
|---|---|---|---|
| Fable 5.1 | 80.0 ± 0.0 | 88.6 ± 7.1 | 60.0 ± 10.0 |
| Claude Opus 5.5 | 73.3 ± 5.8 | 82.9 ± 2.2 | 56.7 ± 5.8 |
| GPT-6 Astra | 60.0 ± 0.0 | 82.7 ± 3.2 | 50.0 ± 10.0 |
| GPT-5.6 Sol | 53.3 ± 5.8 | 67.7 ± 7.2 | 30.0 ± 10.0 |
| GPT-5.5 | 30.0 ± 0.0 | 60.6 ± 5.8 | 13.3 ± 5.8 |
| GPT-5.6 Luna | 0.0 ± 0.0 | 12.9 ± 5.4 | 0.0 ± 0.0 |
| Claude Haiku | 0.0 ± 0.0 | 7.1 ± 6.6 | 0.0 ± 0.0 |
We thank the reviewer for the valuable feedback. In the following, we first clarify three points regarding the paper and evaluation setup, then address the remaining concerns.