- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ALawBench evaluates agents on comprehensive IPO legal due diligence using 63 expert-constructed matters, 12,151 scanned PDFs, external databases with controlled distractors, and 15,557 expert-authored rubrics. Across 21 models, the best reported net score is 40.39%, and the authors conclude that verification and professional assessment remain difficult despite broader evidence access. The released repository provides five complete sample matters and runnable evaluation infrastructure, but not the full dataset or archived results supporting the main leaderboard.
Critical strengths
- Realistic and substantial professional-agent benchmark supported by genuine implementation artifacts: ALawBench evaluates complex legal investigations involving scanned data rooms, corporate-record databases, controlled distractors, and final due-diligence report production. The released sample contains five complete matters with 983 PDFs, 2,384 pages, 1,131 rubrics, and an 18-tool interface; its release checker passes, providing concrete evidence of a substantial implementation rather than merely a proposed benchmark.
- Useful capability-level evaluation linked to document-access behavior: The five-action scoring framework distinguishes Review, Verify, Deliver, Identify, and Assess, while the recorded document-access traces support analysis of how evidence coverage relates to professional performance. This combination offers a meaningful approach to identifying downstream limitations in complex legal-agent workflows, although high document-access coverage does not by itself establish correct evidence extraction or comprehension.
Critical weaknesses
- The scoring structure may distort the paper's central verification and assessment conclusions: Verify measures accumulated error penalties rather than errors per factual assertion, allowing more comprehensive reports to retain similar total penalties despite potentially lower error rates. More importantly, Identify and Assess have different maximum score contributions (67.61 versus 25.76 percentage points), yet their raw gains are compared directly; the strongest model achieves approximately 47.5% and 48.1% of these respective maxima. Without action-normalized results, it is unclear whether Assess is genuinely disproportionately weaker or whether its smaller absolute contribution largely reflects the evaluation design.
- The full experimental results are not independently reproducible from the released artifact, and professional-quality calibration remains insufficient: The repository contains only five of the 63 benchmark matters and provides no archived trajectories or scored outputs for the 21-model comparison, preventing independent verification of the reported 40.39% maximum score and the Verify/Assess trends. The automatic grader's overall agreement with expert judgments is reported, but action-specific reliability is unavailable, and no expert-written reference report establishes an empirical threshold for an issuable professional deliverable. These limitations weaken confidence in the two headline capability bottlenecks without demonstrating that the underlying reported results are incorrect.
Questions for authors
- Can the authors provide reviewer access to the complete 63-matter dataset, archived 21-model outputs, and evaluation scores? If the full data cannot be released, what independent verification procedure supports the reported leaderboard and action-level findings beyond the five provided sample matters?
- Can the authors report Identify and Assess scores normalized by their respective available scoring budgets, and Verify errors normalized by factual assertions or legal citations? Do the claimed assessment plateau and persistent verification limitation remain after these adjustments?
- What is the grader's agreement with legal experts separately for Review, Verify, Deliver, Identify, and Assess? Can independent expert scoring confirm the small differences in Assess among the strongest agents, rather than attributing them to grader uncertainty?
- What score would an expert-written due-diligence report receive under the same evaluation framework, and what threshold constitutes an issuable report? How do the authors distinguish failures in evidence localization or comprehension from downstream legal judgment when document-access coverage is high?
We sincerely thank you for the detailed review. We added an action table on a common scale, a trigger rate for Verify rules, and action-level agreement results.
Q1: About Data Access. We will release the data with the camera-ready version.
Q2: About Normalized Results. Both findings hold on a common scale. The four bounded action scores are rescaled by their theoretical intervals (Appendix Table 9), so Identify and Assess are normalized by their budgets. The Verify rate is the share of applicable Verify rules that a report triggers.
Rescaling is linear, so the Spearman coefficients in Section 4.2 are unchanged.
Q3: About Action-Level Agreement. The table gives the agreement between the grader and the three-expert majority for each action.
Independent expert labels confirm that the small Assess span among the leading models is not caused by grader uncertainty. The grader agrees with the experts about as well on Assess (κ=0.648) as on Identify (κ=0.670), yet the leading models differ by 12.08 on Identify and by only 4.39 on Assess.
Q4: About the Expert Score, Threshold, and Localization.
(a) Expert score. Each matter is built from the prospectus, legal opinions, and lawyers' work reports of a real IPO, so all rubrics take this human work as their reference. With sufficient time, the human baseline is the full score of 100%, against 40.39% for GPT 6 Astra, the best-performing model in our evaluation. IPO legal due diligence takes a legal team months of work. The discussion period is too short for our legal experts to write due diligence reports for all 63 matters.
(b) Threshold. Legal experts judged that none of the current reports is issuable. Our benchmark is built around the requirements for an issuable report, including the full report required by the task (Appendix G.1) and the duties in IPO legal-practice and listing rules (Appendix A.2, Table 4). The scores show how far a report falls short of being issuable.
(c) Localization. We distinguish these failure types by checking the trajectory against the rubrics used to grade the report:
For files recorded as read in the trajectory, we distinguish comprehension failures from legal judgment failures by whether the report states the facts correctly.
W1: About the Scoring Structure. As in Q2, the Verify finding holds as a rate, not only as a total penalty. The Assess plateau also remains on a common scale. We keep raw action scores because they show how much each action contributes to the net score. Verify counts every error because a lawyer must fix each one before issuing the report.
W2: About Reproducibility and Calibration.
The camera-ready version will add these results to the appendix.