acceptodds
Under review as a conference paper at ICLR 2027

ALawBench: Benchmarking Frontier Agents in IPO Legal Due Diligence

Abstract

Legal due diligence for an initial public offering (IPO) requires reviewing extensive records across an issuer's corporate history, consumes thousands of lawyer hours, and demands professional judgement. Advances in frontier agents raise the prospect of conducting such matters end to end, but testing this prospect requires a realistic workspace, materials grounded in real disclosures, and rubrics grounded in legal practice. We introduce ALawBench, the first benchmark to evaluate agents on comprehensive legal due diligence for IPOs in the Chinese mainland and Hong Kong, a task usually undertaken by legal teams. Its legal workspace combines scanned documents, corporate databases with controlled distractors, and professional tools for the full IPO due diligence workflow. Built by 37 legal experts from public disclosures, ALawBench comprises 63 matters, 12,151 PDFs, 30,243 pages, and 15,557 rubrics. The rubrics measure diligence through review, verification, and report delivery, and judgement through risk identification and professional assessment. Across 21 frontier models, the best net score rate is 40.39%, and 12 models score below zero. Review, delivery, and risk identification improve with capability, but verification errors barely decline and professional assessment improves only marginally. Reading nearly all key documents leaves verification errors unchanged and adds little assessment credit. Frontier agents cannot yet deliver issuable IPO legal due diligence reports, as their diligence falls short at verification and their judgement stalls at assessment.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, IPO Legal Due Diligence, Benchmark, Long-Horizon Tasks, Rubric-Based Evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_YuCAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ALawBench evaluates agents on comprehensive IPO legal due diligence using 63 expert-constructed matters, 12,151 scanned PDFs, external databases with controlled distractors, and 15,557 expert-authored rubrics. Across 21 models, the best reported net score is 40.39%, and the authors conclude that verification and professional assessment remain difficult despite broader evidence access. The released repository provides five complete sample matters and runnable evaluation infrastructure, but not the full dataset or archived results supporting the main leaderboard.

Critical strengths
  1. Realistic and substantial professional-agent benchmark supported by genuine implementation artifacts: ALawBench evaluates complex legal investigations involving scanned data rooms, corporate-record databases, controlled distractors, and final due-diligence report production. The released sample contains five complete matters with 983 PDFs, 2,384 pages, 1,131 rubrics, and an 18-tool interface; its release checker passes, providing concrete evidence of a substantial implementation rather than merely a proposed benchmark.
  2. Useful capability-level evaluation linked to document-access behavior: The five-action scoring framework distinguishes Review, Verify, Deliver, Identify, and Assess, while the recorded document-access traces support analysis of how evidence coverage relates to professional performance. This combination offers a meaningful approach to identifying downstream limitations in complex legal-agent workflows, although high document-access coverage does not by itself establish correct evidence extraction or comprehension.
Critical weaknesses
  1. The scoring structure may distort the paper's central verification and assessment conclusions: Verify measures accumulated error penalties rather than errors per factual assertion, allowing more comprehensive reports to retain similar total penalties despite potentially lower error rates. More importantly, Identify and Assess have different maximum score contributions (67.61 versus 25.76 percentage points), yet their raw gains are compared directly; the strongest model achieves approximately 47.5% and 48.1% of these respective maxima. Without action-normalized results, it is unclear whether Assess is genuinely disproportionately weaker or whether its smaller absolute contribution largely reflects the evaluation design.
  2. The full experimental results are not independently reproducible from the released artifact, and professional-quality calibration remains insufficient: The repository contains only five of the 63 benchmark matters and provides no archived trajectories or scored outputs for the 21-model comparison, preventing independent verification of the reported 40.39% maximum score and the Verify/Assess trends. The automatic grader's overall agreement with expert judgments is reported, but action-specific reliability is unavailable, and no expert-written reference report establishes an empirical threshold for an issuable professional deliverable. These limitations weaken confidence in the two headline capability bottlenecks without demonstrating that the underlying reported results are incorrect.
Questions for authors
  1. Can the authors provide reviewer access to the complete 63-matter dataset, archived 21-model outputs, and evaluation scores? If the full data cannot be released, what independent verification procedure supports the reported leaderboard and action-level findings beyond the five provided sample matters?
  2. Can the authors report Identify and Assess scores normalized by their respective available scoring budgets, and Verify errors normalized by factual assertions or legal citations? Do the claimed assessment plateau and persistent verification limitation remain after these adjustments?
  3. What is the grader's agreement with legal experts separately for Review, Verify, Deliver, Identify, and Assess? Can independent expert scoring confirm the small differences in Assess among the strongest agents, rather than attributing them to grader uncertainty?
  4. What score would an expert-written due-diligence report receive under the same evaluation framework, and what threshold constitutes an issuable report? How do the authors distinguish failures in evidence localization or comprehension from downstream legal judgment when document-access coverage is high?
1 comment

We sincerely thank you for the detailed review. We added an action table on a common scale, a trigger rate for Verify rules, and action-level agreement results.

Q1: About Data Access. We will release the data with the camera-ready version.

Q2: About Normalized Results. Both findings hold on a common scale. The four bounded action scores are rescaled by their theoretical intervals (Appendix Table 9), so Identify and Assess are normalized by their budgets. The Verify rate is the share of applicable Verify rules that a report triggers.

ModelReviewDeliverIdentifyAssessVerify rate (%)
GPT 6 Astra92.4688.3847.5348.063.12
Qwen 3.8 Max80.1380.5645.6044.337.50
GLM 5.382.8481.1438.2743.683.70
Kimi K382.6282.0635.4544.042.74
Claude Opus 562.3972.7039.2544.283.25
Span30.0715.6912.084.39

Rescaling is linear, so the Spearman coefficients in Section 4.2 are unchanged.

  • (i) Assess plateaus. The five models differ by only 4.39 on Assess, compared with 12.08 on Identify. The Identify span exceeds the Assess span by 7.69, with a 95% CI of [1.77, 11.18] from 10,000 matter resamples. Per-matter normalization gives spans of 4.66 for Assess and 11.99 for Identify, a 7.32 gap with a 95% CI of [1.33, 10.90].
  • (ii) Verify errors persist. The Verify rate ranges from 1.51% to 8.00% across the 21 models and shows no monotonic decline as net score rises (Spearman ρ=−0.19, p=0.42). Legal citation errors are scored by a separate retrieval pipeline and are not included in this rate.

Q3: About Action-Level Agreement. The table gives the agreement between the grader and the three-expert majority for each action.

ReviewVerifyDeliverIdentifyAssess
κ0.6550.7050.8630.6700.648

Independent expert labels confirm that the small Assess span among the leading models is not caused by grader uncertainty. The grader agrees with the experts about as well on Assess (κ=0.648) as on Identify (κ=0.670), yet the leading models differ by 12.08 on Identify and by only 4.39 on Assess.

Q4: About the Expert Score, Threshold, and Localization.

(a) Expert score. Each matter is built from the prospectus, legal opinions, and lawyers' work reports of a real IPO, so all rubrics take this human work as their reference. With sufficient time, the human baseline is the full score of 100%, against 40.39% for GPT 6 Astra, the best-performing model in our evaluation. IPO legal due diligence takes a legal team months of work. The discussion period is too short for our legal experts to write due diligence reports for all 63 matters.

(b) Threshold. Legal experts judged that none of the current reports is issuable. Our benchmark is built around the requirements for an issuable report, including the full report required by the task (Appendix G.1) and the duties in IPO legal-practice and listing rules (Appendix A.2, Table 4). The scores show how far a report falls short of being issuable.

(c) Localization. We distinguish these failure types by checking the trajectory against the rubrics used to grade the report:

FailureRubricsPattern
LocalizationReviewThe trajectory shows that a required file was not read
ComprehensionFact, VerifyThe file was read, but the report misstates or omits the fact
Legal judgmentIssue, AssessThe fact is stated correctly, but the issue, risk grade, or remedy is wrong

For files recorded as read in the trajectory, we distinguish comprehension failures from legal judgment failures by whether the report states the facts correctly.

W1: About the Scoring Structure. As in Q2, the Verify finding holds as a rate, not only as a total penalty. The Assess plateau also remains on a common scale. We keep raw action scores because they show how much each action contributes to the net score. Verify counts every error because a lawyer must fix each one before issuing the report.

W2: About Reproducibility and Calibration.

  • As in Q1, we will release the data with the camera-ready version.
  • As in Q3, grader agreement by action ranges from κ=0.648 to 0.863.
  • As in Q4, with sufficient time the human baseline is the full score of 100%, and legal experts judged that none of the current reports is issuable.

The camera-ready version will add these results to the appendix.

Reviewer_zb8GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ALawBench, an end-to-end benchmark for IPO legal due diligence, comprising 63 cases, 12,151 PDF documents, and 15,557 expert-authored rubrics. The benchmark evaluates five professional actions across 21 frontier models. The results suggest that current agents remain far from delivering high-quality legal due diligence, with particular limitations in verification and risk assessment.

Critical strengths
  1. Compared with existing legal benchmarks, ALawBench covers the complete workflow from raw-document investigation and cross-document verification to final report generation. This setting better reflects real-world legal practice and offers substantial practical value.
  2. The dataset covers multiple listing markets and industries and involves 37 legal experts in construction and validation. Cross-review procedures and human-grader agreement analysis support its quality, while positive and negative rubrics help capture consequential legal errors.
  3. Beyond overall performance comparisons, the paper decomposes scores into five actions and analyzes them alongside document-access records. This provides useful insights into potential failure modes of agents in complex professional workflows.
Critical weaknesses
  1. Experts derive case-specific legal issues and rubrics from historical IPO disclosures, but the paper does not sufficiently explain how the coverage of major legal risks is ensured. Moreover, rubric-based scores are used to assess professional report quality without validating their agreement with experts' holistic judgments of report issuability. This weakens the benchmark's construct validity.
  2. The five actions have asymmetric scoring structures: Verify contains only penalties, whereas Identify and Assess contain only positive rewards, with maximum contributions of 67.61 and 25.76 percentage points, respectively. Directly comparing their raw scores and improvements can therefore be misleading. In particular, the assessment bottleneck claim requires stronger evidence based on normalized performance.
  3. The authors observe limited Verify and Assess improvements at high document coverage and attribute the remaining limitations primarily to verification and reasoning rather than retrieval. However, accessing a document does not guarantee successful extraction, understanding, or use of its critical evidence. Without adequately controlling for model and case differences, the results establish correlations rather than identifying the underlying causes.
Questions for authors

How do the authors validate that the case-specific rubrics adequately cover major legal risks? Is there any independent holistic expert evaluation to establish that rubric scores reflect the professional issuability of complete reports?

2 comments

We sincerely thank you for the careful review. We added a common action scale and coverage regressions with fixed effects.

Q1: About Validating Rubric Coverage. We use common review guidelines and a second expert's check of each matter's rubrics to ensure coverage of major legal risks.

  • Before rubric writing began, partners and senior lawyers helped set the review guidelines, which specify what an LDD report must cover and how each item is judged.
  • One expert wrote the rubrics for each matter and another checked them. Rubrics on which the two experts disagreed were returned for revision.
  • All rubric writers and checkers have IPO experience.

Q2: About Rubric Scores and Issuability. Legal experts evaluated the reports as a whole and judged that none of the current reports is issuable. The evaluation covers both the content and the form of an issuable report.

  • Content and analysis.
    • Lawyers with extensive IPO experience led the design of the rubric dimensions and mapped the content requirements of real LDD reports into the rubrics.
    • The dimensions cover what matters in practice, including risk identification, factual accuracy, application of law, complete disclosure, and diligent verification.
    • The rubrics evaluate how a model investigates each matter's corporate history, equity changes, major assets, and claims and debts, and how it identifies and analyzes the related legal risks.
    • The scores therefore measure the gap between a model's report and one a law firm could deliver directly to its client.
  • Form and presentation. Scoring criteria address the cover page, table of contents and definitions, heading hierarchy and numbering, and fonts and layout. They measure whether a report follows the format conventions of LDD reports issued by lawyers in practice.

Three legal experts labeled 3,321 sampled rubric decisions. Cohen's κ between their majority-vote labels and the grader's decisions is 0.72.

W1: About Risk Coverage and Construct Validity.

  • As in Q1, rubrics follow common review guidelines and are checked by a second expert.
  • As in Q2, legal experts judged that none of the current reports is issuable, and the evaluation covers both the content and the form of an issuable report.

W2: About Asymmetric Action Scoring. Raw action scores show how much each action actually contributes to the net score. For comparison across actions we now add a common scale. The four bounded action scores are rescaled by their theoretical intervals (Appendix Table 9). Verify has no finite lower bound because every error counts, so we report the share of applicable Verify rules triggered.

ModelReviewDeliverIdentifyAssessVerify rate (%)
GPT 6 Astra92.4688.3847.5348.063.12
Qwen 3.8 Max80.1380.5645.6044.337.50
GLM 5.382.8481.1438.2743.683.70
Kimi K382.6282.0635.4544.042.74
Claude Opus 562.3972.7039.2544.283.25
Span30.0715.6912.084.39

Rescaling is linear, so the Spearman coefficients in Section 4.2 are unchanged. On this scale, both findings hold.

  • (i) Assess plateaus. The five models differ by only 4.39 on Assess, the smallest span of the four actions. The Review, Deliver, and Identify spans exceed it by 25.68, 11.30, and 7.69. Based on 10,000 matter resamples, all three gaps have 95% intervals above zero. Assess therefore distinguishes these five models less clearly than each of the other three actions does.
  • (ii) Verify errors persist. Across all 21 models, the Verify rate shows no clear decline as models get stronger (ρ=−0.19, p=0.42).

W3: About Access, Use, and Controls. We added an analysis with model and matter fixed effects, which control for differences in model ability and matter difficulty, and the coverage findings still hold. The table gives the score change associated with each 10-percentage-point increase in coverage:

ActionChange (points)95% CI
Review0.685[0.262, 1.108]
Deliver0.553[0.067, 1.039]
Identify0.737[0.155, 1.319]
Assess0.179[−0.004, 0.363]
Verify−0.002[−0.121, 0.118]

The Review, Deliver, and Identify intervals are above zero, and the Assess and Verify intervals contain zero. We will list causal analysis as future work in the camera-ready version.

We will also add these results to the appendix.

Thank you for the detailed reply and supplementary experiments from the author. I believe the newly added analysis has substantially improved the paper, especially the regression analysis that unified the scoring scale and controlled for fixed effects of the model and cases, which has provided more solid support for the conclusion that the errors in the Assess platform and Verify persist.

Reviewer_X7qMOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether an AI agent can independently complete an IPO legal due diligence project, identifying facts and legal risks from extensive raw materials and ultimately delivering a comprehensive due diligence report. The primary contribution of the paper is a complex legal workflow benchmark: ALawBench. The authors extract facts and timelines from disclosed materials such as prospectuses and legal opinions, anonymize them, and then generate corresponding documents with the assistance of experts and LLMs to construct a simulated workspace. Controlled defects and database distractors are subsequently introduced. The paper proposes two dimensions and five actions, yielding a series of findings: 1. Under this scoring system, overall performance remains limited. 2. Higher-scoring models show more significant improvements in review, delivery, and risk identification. 3. Accessing a large number of key documents still does not guarantee report accuracy.

Critical strengths
  1. The task selection is valuable and significantly surpasses ordinary legal Q&A.
  2. The benchmark development involves substantial workload with extensive expert participation. It encompasses 63 projects, over 30,000 pages of documents, and 15,000 scoring rules.
  3. The scoring granularity is relatively high.
  4. The case reveals some specific failures.
Critical weaknesses
  1. No expert baseline. Experts created questions but didn't answer them. 40.39% is unclear whether it's worse than humans or close to human performance.
  2. Single run, no variance.
Questions for authors
  1. Could you provide the token/second cost for each model? This is crucial for the practical usability of the agent benchmark.
  2. Could you provide an expert baseline?
1 comment

We sincerely thank you for the positive assessment. We added a matter-level bootstrap analysis of the leaderboard and per-model token costs.

Q1: About Token Cost. The table gives the average input and output tokens per report for all 21 models, in leaderboard order.

ModelAvg. input tokens (M)Avg. output tokens (K)
GPT 6 Astra85.36118.8
Qwen 3.8 Max136.673,019.5
GLM 5.376.391,123.3
Kimi K334.15274.0
Claude Opus 5134.74697.4
Grok 4.611.61296.4
GPT 5.6 Sol46.52211.1
Claude Fable 524.44561.1
DeepSeek V4 Flash140.50210.0
GPT 5.551.87275.5
Grok 4.58.78135.7
MiniMax M388.14162.7
GLM 5.230.27220.8
Gemini 3.6 Flash45.03116.9
Doubao Seed 2.1 Pro86.76664.0
DeepSeek V4 Pro43.00161.3
Qwen 3.7 Max18.77179.6
Gemini 3.7 Flash48.07134.1
HY331.74234.5
Gemini 3.1 Pro88.99201.7
MiMo V2.5 Pro20.28161.8

Q2: About an Expert Baseline. Each matter is built from the prospectus, legal opinions, and lawyers' work reports of a real IPO, so all rubrics take this human work as their reference. With sufficient time, the human baseline is the full score of 100%, against 40.39% for GPT 6 Astra, the best-performing model in our evaluation. IPO legal due diligence takes a legal team months of work. The discussion period is too short for our legal experts to write due diligence reports for all 63 matters.

W1: About the Expert Baseline. As in Q2, with sufficient time the human baseline is the full score of 100%, so GPT 6 Astra's 40.39% is far below human performance.

W2: About Single Runs and Variance. As Q1 shows, the benchmark is costly to run. The 21 models average 59.62 million input tokens and 436.2 thousand output tokens per report, so each model was run once per matter. We also quantified matter-sampling uncertainty by resampling the 63 matters 10,000 times and recomputing every score and rank:

ModelNet (%)95% CI95% rank interval
GPT 6 Astra40.39[36.92, 43.87][1, 1]
Qwen 3.8 Max27.32[22.89, 31.54][2, 4]
GLM 5.327.11[22.89, 31.30][2, 4]
Kimi K326.89[22.56, 31.08][2, 4]
Claude Opus 518.35[12.11, 24.21][5, 5]
Grok 4.69.80[6.09, 13.53][6, 7]
MiMo V2.5 Pro−34.50[−39.08, −29.96][21, 21]

The model tiers are stable. The camera-ready version will report bootstrap results for all 21 models in the appendix and state the single-run protocol in the Limitations section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.