acceptodds
Under review as a conference paper at ICLR 2027

RegBench: Source-Grounded Benchmarks for Regulatory Cross-Reference Reasoning

14 views

Abstract

Compliance work in regulated domains requires tracking cross-references across sections and applying the resulting chain to numeric or categorical decisions. Existing benchmarks test long-document retrieval and multi-hop QA, but the benchmarks reviewed here do not isolate traversal over explicit regulatory cross-reference graphs. We introduce RegBench, a specialised benchmark comprising **827 expert-level questions grounded in 4,766 atomic-fact propositions** from DNV Ship Rules and the U.S. Basel III capital regulation, 12 CFR Part 217. Frontier systems reach only **17–49%** strict accuracy on DNV and **17–57%** on Basel, with clear tier degradation as required context scope increases, and an independent judge from a third vendor reproduces the ranking. Three published graph-RAG methods further collapse to ≤17% strict at pilot scale. To scale such evaluations, we present a novel framework that converts regulatory cross-reference graphs into source-grounded QA items with minimal human effort. This is achieved through a graph-anchored generation pipeline that samples reference chains to synthesise scenarios, and a calibrated audit (v5d) that bounds quality without expert review of every question. Our automated audit agrees with full-set SME annotation on **99.0%** of audited items and **99.1%** of atomic facts on DNV, and transfers to 12 CFR Part 217 without tuning. Together with independently validated answer grading, these results support strict atomic-fact evaluation at benchmark scale. Code and data are available at https://github.com/regbench/regbench.

Primary area:
datasets and benchmarks
Keywords:
benchmark, regulatory reasoning, cross-reference reasoning, multi-hop reasoning, long-context evaluation, LLM-as-judge, dataset auditing, graph RAG, ship classification rules, Basel III

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kyvbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RegBench evaluates whether language models can work through cross-references in regulatory documents and produce grounded compliance decisions. The benchmark draws on maritime and banking regulations and combines graph-guided question construction with fact-level checking and expert review. This is a worthwhile setting, and the care taken with data auditing is a strength. My concern is that the evaluation does not yet clearly separate cross-reference reasoning from finding relevant provisions and producing a complete answer.

Critical strengths

The paper tackles a concrete problem that general long-document QA benchmarks often miss: applying rules spread across clauses, tables, and sections. The construction pipeline is also more careful than a typical LLM-generated QA dataset. In particular, the expert checks, judge calibration, and sensitivity analyses give readers useful information about annotation quality.

Critical weaknesses

1.

The questions do not always appear to require the sampled reference chain. The paper builds questions around paths through a regulation graph, but a path used to generate a question is not necessarily a path needed to answer it. In Appendix G, example F500_0015 combines a numerical compliance decision about a discharge opening with a separate question asking where a hatch-cover requirement is specified. Both involve regulations, but the latter does not feed into the former. More broadly, 339 of the 780 leaderboard questions are T0/T1 and do not require crossing sections. Those are reasonable tasks to include, but aggregate accuracy should not be presented as a direct measure of cross-section reasoning. A small expert audit of whether each hop is actually necessary would help considerably.

Strict accuracy makes the tier results hard to interpret. An answer is marked wrong if it omits any required atomic fact, even when its calculation and compliance verdict are correct. As questions require more facts, this rule alone can lower accuracy. That makes it difficult to tell how much of the T0-to-T4 decline reflects harder reasoning rather than longer answer keys. The answer-key sensitivity test adds to this concern: alternative expert-approved decompositions change 55 of 715 item-level decisions (Appendix R). I would like to see verdict accuracy and evidence completeness reported separately, with tier comparisons controlled for the number of required facts. For DNV T4, it would also help to know whether the provided context actually contains every gold provision after the context expansion and length cap.

2.

The choice of the final audit prompt needs an explanation. Table 12 reports identical recall for v5b and v5d (81.2%), but v5b has better precision (52.0% versus 50.0%) and agreement (89.9% versus 89.2%). This seems at odds with the stated no-regression freeze criterion. There may be an operational reason to prefer v5d, but it is not evident from the reported results.

A smaller point: the Graph-RAG comparison covers only 47 pilot questions and changes more than the retrieval strategy. It is useful as a pilot, but too limited to support a broad conclusion about graph-based retrieval.

I see substantial work in building and checking this benchmark. The missing piece is stronger evidence that the reported scores capture the specific reasoning ability the paper sets out to test. I therefore lean Weak rejection (2/4).

Questions for authors
  1. In a representative sample of T2–T4 questions, how often is every hop in the gold reference chain necessary for the final decision?
  2. Does the tier-level performance decline remain after controlling for required-fact count and gold-evidence coverage? Could you report final-verdict accuracy separately from strict completeness?
  3. What criterion led to selecting v5d over v5b, given the results in Table 12?
7 comments
AuthorsOpenReview ↗

Response to Reviewer kyvb (1/3): Hop-Necessity Audit

We thank the reviewer for a precise and well-argued review. Its central question is the right one: do the scores measure the reasoning we claim? We answer each point below with new analyses. All changes are in the revised PDF.

W1a / Q1. Do questions require the sampled chain?

The reviewer is right about F500_0015. Its second part ("where is the bolt-area requirement specified?") does not feed the freeing-port verdict. We have replaced it in App. G with an item whose chain feeds the decision.

To measure how common this is, we ran the audit the reviewer suggested. Domain experts (ship designers for DNV, finance reviewers for Basel; research-team members, blinded to model outputs and audit flags) labelled every hop in tier-stratified samples of T2–T4 items: 60 DNV items (112 hops) and 60 Basel items (132 hops). Each item was labelled by one expert. A hop is labelled necessary if its section can guide the reader to the right answer or holds a condition that could change the judgement. It is labelled not necessary only if it cannot affect the judgement in any way. The full definition and ruling notes are in App. S.7.

DNVBasel
items where every hop is necessary (expert)24/60 = 40.0% [28.6, 52.6]35/60 = 58.3% [45.7, 69.9]
items where every hop is necessary or needed for a side part55/60 = 91.7% [81.9, 96.4]52/60 = 86.7% [75.8, 93.1]
hops labelled necessary70/112 = 62.5%103/132 = 78.0%
hops needed only for a side part of the question37/112 = 33.0%21/132 = 15.9%

Across both corpora, nearly every hop in the gold chains is used by the question: every hop is necessary or needed for a side part in 92% of DNV items (T2 17/20, T3 19/20, T4 19/20) and 87% of Basel items (T2 25/30, T3 27/30). Fewer items have every hop bearing on the final judgement: 40% on DNV (T2 12/20, T3 7/20, T4 5/20) and 58% on Basel (T2 18/30, T3 17/30). Most of the remaining hops serve a side part of the question, as in F500_0015. The revised paper makes per-tier results the primary view and reports the audit per corpus and tier.

On T0/T1. The revised §4 makes the per-tier results the primary view and describes the aggregate as an overall benchmark score.

AuthorsOpenReview ↗

Response to Reviewer kyvb (2/3): Fact Count, Coverage, Verdict vs Completeness

W1b / Q2. Tier decline vs number of facts, gold-evidence coverage, and verdict vs completeness.

Fact count. The number of required facts rises only modestly with tier: DNV T0–T4 average 5.36 → 6.03 facts; Basel T0–T3 average 5.68 → 6.03. In a logistic regression of strict correctness on tier, fact count and system (item-clustered SEs), tier remains strongly significant:

  • DNV: OR 0.54 per tier, p < 10⁻²⁸. Without the fact-count term the OR is 0.51.
  • Basel: OR 0.55 per tier, p < 10⁻¹³.

The decline also persists at a fixed fact count. Among DNV items with exactly 6 facts, strict accuracy falls 58% → 51% → 37% → 23% → 6% across T0–T4. Longer answer keys therefore do not explain the decline.

Gold-evidence coverage (including DNV T4). The reviewer's concern is justified. At DNV T4, the full-context input does not contain the complete gold chain: T4 chains cross Parts, and the context is truncated at the length cap. Every system receives the same practitioner-scope context, capped at 600K characters due to context limit. The full-context decline at the deepest tiers partly reflects evidence that was not in the context, and we have revised our presentation accordingly. We ran a gold-chain oracle on all 780 items (per-model table in our reply to L82R), and the revised paper presents the tier results under both conditions.

With coverage guaranteed, the results still support our claim. A significant decline nevertheless remains: under the oracle, DNV falls from 71–81% at T0 to 37–55% at T4, and Basel from 65–90% to 32–65%. Controlling for required-fact count and gold-context length, tier stays significant (OR 0.86 per tier on DNV, p = 0.036; 0.68 on Basel, p = 0.001; item-clustered). The tier decline under full context thus combines missing evidence with a genuine composition cost. We now report both, and no longer attribute the whole decline to reasoning.

Verdict accuracy and evidence completeness, separately and controlled for fact count and evidence coverage. Using the judge's existing fact-level decisions, we tagged each item's final-answer fact(s). Verdict accuracy requires only those facts to be present; evidence completeness is the share of the remaining derivation facts present. To control for coverage as well, we pool the five systems that ran both full context and the gold-chain oracle (%; per-tier change controlling for fact count):

T0T1T2T3T4per tier
DNV strict, full context666147335−13.9pp
DNV strict, oracle7568545845−5.7pp
DNV verdict, full context8579815219−14.9pp
DNV verdict, oracle8984878576−1.6pp
DNV completeness, full context8782776438−11.5pp
DNV completeness, oracle9087818477−2.8pp
Basel strict, full context64643726–−13.9pp
Basel strict, oracle77735046–−11.2pp
Basel verdict, full context86796854–−10.6pp
Basel verdict, oracle92877274–−6.7pp
Basel completeness, full context84847368–−6.8pp
Basel completeness, oracle91898182–−4.1pp

Controlling for fact count barely changes any slope (DNV verdict under full context: −15.9pp per tier uncontrolled, −14.9pp controlled), so longer answer keys do not explain the decline. With every gold section given, strict accuracy still falls with tier on both corpora (DNV 75% → 45%, Basel 77% → 46%), and verdict accuracy falls as well, less steeply (DNV 89% → 76%, Basel 92% → 74%). Under the oracle, T2 and T3 score about the same on both corpora. We attribute this to the oracle supplying each section on the chain in full and side by side: it removes the distance a T3 chain spans across the rulebook while keeping the reference text intact, so the reader does not follow a wrong reference. Part of the deep-tier verdict loss under full context therefore reflects evidence the input lacked; with the evidence given, what remains on DNV is mainly a loss in stating the full derivation, which strict scoring captures and verdict-only scoring would miss.

AuthorsOpenReview ↗

Response to Reviewer kyvb (3/3): Answer Keys, v5d, Graph-RAG

Answer-key sensitivity (55/715). The 55 changed decisions are out of 715 system–item pairs for five systems, which is 92.3% agreement. We have extended the study to all seven leaderboard systems: 911/1001 = 91.0% agreement, with rank correlation ρ = 0.90. The one discordant pair lies within a group of four systems 1.4pp apart, and no system's score changes significantly. On Basel, the ranking is unchanged (ρ = τ = 1).

W2 / Q3. Why v5d over v5b?

We thank the reviewer for raising this. The freeze paragraph in App. M stated a different criterion from the one we actually used to choose v5d, and we have rewritten it. The choice was made on the SME-labelled 100-item subset of the development pool (the SME-3 set), with an objective that weights recall above precision (recall + precision/5, subject to recall ≥ 80%). On that subset, v5d scored 0.951 against 0.944 for v5b, with precision 58.8% vs 55.6%, flag rate 17% vs 18%, and fewer false flags at T3/T4 (5 vs 7).

On the full 148-item pool reported in Table 13, the two variants are effectively tied. They catch the same 13 SME-flagged items and differ only on non-flagged items, where v5d raises one additional false flag. This is the 52.0% vs 50.0% precision difference the reviewer notes. We now report both pools in Table 13, state the selection objective explicitly, and no longer claim that v5d dominates every ablation.

Graph-RAG Agreed. We have replaced the pilot with a controlled comparison on all 499 DNV and 281 Basel items in which only the retrieval differs (same Sonnet 4.6 reader, prompt and judge; tables in our reply to A9Ju). On both corpora, HippoRAG does not differ significantly from BM25 (DNV 47.5% vs 44.5%, p = 0.22; Basel 53.7% vs 56.9%, p = 0.34), while GraphRAG's local-search context is substantially weaker (26.3% and 40.6%).

We are grateful for these specific questions. They prompted analyses that make the paper's construct claims more precise, and we would welcome further comments.

Reviewer_L82ROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RegBench, a benchmark for evaluating source-grounded reasoning over regulatory documents. RegBench focuses on problems that require models to follow cross-references across regulatory provisions and derive numerical or categorical decisions. The benchmark contains questions from DNV Ship Rules and 12 CFR Part 217 and evaluates answers through required intermediate atomic facts rather than final-answer correctness alone.

Critical strengths
  • The paper addresses an important practical challenge in regulatory QA: relevant rules can be distributed across multiple cross-referenced provisions, requiring models to combine information rather than simply retrieve a single passage.
  • The structured representation of each problem through a reference chain, required atomic facts, and a final verdict provides richer supervision than conventional question-answer benchmarks.
  • Evaluating the framework across two substantially different regulatory corpora is useful evidence of its portability.
Critical weaknesses
  • The novelty of the problem formulation appears overstated. The paper presents regulatory cross-reference traversal as a largely unaddressed benchmark setting. However, closely related prior work already studies essentially this structural challenge. In particular, SearchFireSafety (Evaluating Structure-Aware Retrieval and Safety in Statute-Centric Legal QA, Chae et al., ACL 2026) models statutory regulations as citation-linked structures and evaluates multi-hop retrieval/reasoning over hierarchically fragmented regulatory evidence. The distinctive contribution of RegBench therefore seems narrower than currently presented: namely, evaluating complete source-grounded derivations through atomic intermediate facts, together with a scalable auditing procedure. I believe the paper should discuss SearchFireSafety and related work directly and clarify this distinction.

  • The experiments do not clearly separate reasoning difficulty from retrieval and long-context navigation. This is my main technical concern. The T0~T4 tiers are defined primarily by the scope over which relevant information is distributed, rather than by reasoning depth alone. Consequently, the observed degradation can result from at least three factors: failure to retrieve relevant provisions, failure to locate them within a large context, or failure to reason correctly after the evidence has been identified. An important missing experiment is therefore a gold-chain oracle condition, where models receive exactly the source clauses required for the intended derivation. Comparing gold-chain oracle, BM25 retrieval, and full-context settings would reveal whether RegBench primarily exposes a reasoning bottleneck or an evidence-discovery/context-navigation bottleneck. The interpretation of the BM25 results is similarly unclear. BM25 provides only the top-10 chunks, whereas full-context exposes the model to a much larger amount of potentially distracting text. Thus, the claim that retrieval "scales better" with increasing chain depth is difficult to establish from this comparison alone. Reporting gold-chain retrieval recall for BM25 at each tier would substantially improve the analysis.

  • The synthetic construction and filtering pipeline may bias the benchmark toward easier-to-generate regulatory cases. RegBench questions are LLM-generated from sampled reference chains, and a substantial fraction of generated candidates is filtered before release. The paper also acknowledges that some exception-heavy and table-arithmetic cases are removed during construction/auditing. However, exceptions, conditional branches, tables, and calculations may be precisely the cases that make real regulatory reasoning difficult. This raises a potential selection bias: RegBench may represent the subset of regulatory reasoning problems that the generation/auditing pipeline can reliably construct, rather than the broader distribution of real regulatory tasks. A characterization of rejected versus retained examples would help assess this issue. The claim that the questions are "expert-level" also deserves additional support, particularly for Basel, where scenario realism was not validated by an in-the-loop banking practitioner.

Questions for authors
  • How do the authors distinguish RegBench from SearchFireSafety (Chae et al., ACL 2026) and other work that already models statutory/regulatory documents as citation graphs and evaluates multi-hop reasoning over linked provisions? Would the authors agree that RegBench’s main novelty lies more specifically in atomic-fact-level derivation evaluation and scalable auditing?

  • Could the authors provide a gold-chain oracle experiment, where models are given exactly the clauses required for the derivation? Does the strong tier-wise degradation remain when retrieval and long-context navigation are removed?

  • What fraction of the gold reference chain is retrieved by BM25 top-10 at each tier? In particular, how does accuracy differ between cases where BM25 retrieves the complete gold chain and cases where it does not?

  • How different are the examples rejected by the generation/confidence/audit pipeline from those retained in RegBench? Are rejected examples disproportionately associated with exceptions, conditional branches, tables, formulas, or more complex reference structures?

  • Overall, I find the benchmark construction and validation effort substantial. However, the paper would be considerably stronger if it more carefully positioned its novelty relative to existing regulatory citation-graph benchmarks and experimentally separated cross-reference reasoning from retrieval and long-context evidence discovery.

7 comments
AuthorsOpenReview ↗

Response to Reviewer L82R (1/3): SearchFireSafety and Contribution

We thank the reviewer for a detailed and constructive review. We respond in three parts: positioning relative to SearchFireSafety; the main technical concern (whether the tier decline comes from retrieval, long-context navigation or reasoning), with new experiments; and selection bias. All changes are in the revised PDF.

W1 / Q1. Relation to SearchFireSafety, and what RegBench contributes.

In brief: we agree that explicit cross-reference structure is not a new benchmark setting, and we now credit SearchFireSafety directly. However, SearchFireSafety works over short statute units, while RegBench grades derivations over long regulatory documents, the setting our introduction describes. The submitted abstract did not state this, and the revised abstract now does.

Prior work. We appreciate the reference. SearchFireSafety (Chae et al., ACL 2026) already models statutory regulation as a citation graph, over the full hierarchy of Korean fire-safety law. It evaluates structure-aware, multi-hop retrieval over linked provisions, showing that graph-guided retrieval recovers evidence that flat retrieval misses (Recall@10 from 23.4 with BM25 to 54.7 with a dense retriever plus its graph reranker), and its consultation set is built from 876 real petitions with official answers. To our knowledge it is the first benchmark of multi-hop retrieval over a statutory citation graph, and for citation-linked retrieval over statute units it precedes us.

Short units vs long documents. However, those units are short: SearchFireSafety segments its 131 statutes into 4,467 units of about 480 characters, and a question rests on one or two of them. RegBench was built for long documents, as the introduction of the submission states ("Question answering (QA) over long technical documents is the natural test of this read-and-derive capability: it requires a model to locate the relevant clauses, follow them through the document…"), and Table 1 compares benchmarks first on long-document input. The submitted abstract left this setting implicit, which made the two benchmarks look closer than they are.

What we changed. We have revised the abstract to name both settings and credit SearchFireSafety, and added it to Related Work, alongside RegOps-Bench (Ju & Lee, 2026), which follows cross-document citations in national R&D regulations and scores per-rule attribution, and ObliQA / ObliQA-MP (RegNLP) and STARD.

Our contribution. The paper lists three contributions (§1):

  1. a portable graph-anchored generation pipeline that turns a corpus's cross-reference graph into chain-anchored, source-grounded items, applied unchanged to two regulatory regimes;
  2. a calibrated selective audit (v5d) that gates release quality without per-item expert authoring, with locked rules carried to held-out splits and a second corpus;
  3. the benchmark itself, which grades complete derivations by strict conjunction over audited atomic facts.

The revised introduction now states the contribution in these terms.

How the two benchmarks differ. They serve different goals. SearchFireSafety targets practitioner consultation: identifying the governing provision, answered either as an option number (multi-hop set: 3,395 items, each on one delegation edge) or as a consultation answer graded holistically against the official reply (about 1.5 relevant documents per question). RegBench targets specialist verification over long documents:

  • The source documents are long. A RegBench item's gold evidence has a median of 58K characters on DNV (121K at T4) and 18K on Basel, and the full-context condition gives the model about 600K characters of rule text, so locating the governing clauses is part of the task.
  • The output is a computed result. Each scenario is carried through a chain of 1–4 sections to a computed value and a verdict; 303/780 items span three or more sections.
  • Every intermediate fact is graded, so the score shows where a derivation breaks.
  • Evaluation adds a long-context condition to retrieval, with a gold-evidence oracle as a diagnostic.
AuthorsOpenReview ↗

Response to Reviewer L82R (2/3): Gold-Chain Oracle

W2 / Q2. Separating reasoning from retrieval and long-context navigation (main concern).

In brief: both factors matter. Giving the model the gold evidence removes part of the collapse at the deepest tiers, but a significant tier decline remains even then. RegBench therefore measures both locating the provisions a cross-reference chain points to and composing them once found.

Pilot evidence in the submission. The pilot panel (App. N, Table 14) includes a gold-chain oracle condition, which the revision now defines explicitly: the reader receives only the full text of the item's anchor section and of every section on its gold reference chain, in document order, with the same prompt and judge and no retrieval. Under this condition, strict accuracy is 55.3% (Opus 4.6) and 46.8% (Haiku 4.5), which suggests that evidence access is not the only bottleneck.

Full-scale oracle, as the reviewer requested. We ran the oracle on all 780 leaderboard items for five models that also have full-context runs (the sixth leaderboard model, MiniMax-M2.7, has since been retired by its provider), with the same items, prompt and judge as those runs. Strict accuracy (%), full context → gold-chain oracle:

DNV (n = 499)T0T1T2T3T4
GPT-5.6-sol73 → 7366 → 7155 → 6043 → 649 → 50
GPT-5.6-terra64 → 7158 → 5746 → 4930 → 534 → 37
GPT-5.465 → 7763 → 6848 → 5632 → 617 → 44
Sonnet 4.673 → 8167 → 8549 → 6628 → 603 → 55
Qwen3.6-Plus57 → 7253 → 6136 → 4129 → 524 → 41
Basel (n = 281)T0T1T2T3
GPT-5.6-sol72 → 8171 → 7949 → 5632 → 48
GPT-5.6-terra71 → 6561 → 6139 → 3427 → 42
GPT-5.459 → 7261 → 7037 → 5124 → 42
Sonnet 4.671 → 9069 → 8438 → 7031 → 65
Qwen3.6-Plus46 → 7457 → 7124 → 3715 → 32

(Per-model interfaces and settings are in App. S.)

What the results show.

  1. Supplying the gold evidence removes part of the collapse at the deepest tiers. DNV T4 rises from 3–9% to 37–55% for every model, and T3 gains 20–31pp (paired McNemar, p ≤ 2×10⁻⁴ for every model at T3 and T4; Basel T3 p ≤ 0.035). At T0–T2, the changes are smaller: mostly not significant for the three GPT models, but significant for Sonnet 4.6 (DNV T1–T2, Basel T0–T2) and Qwen3.6-Plus (DNV T0, Basel T0–T2).
  2. A significant decline remains even when the evidence is given. Under the oracle, DNV accuracy still falls from 71–81% at T0 to 37–55% at T4, and Basel from 65–90% at T0 to 32–65% at T3. We fit a logistic regression on the oracle condition, controlling for required-fact count and gold-context length, with system fixed effects and standard errors clustered by item. Tier remains significant: OR 0.86 per tier on DNV (p = 0.036) and 0.68 on Basel (p = 0.001).

The revised paper therefore says "cross-reference evidence discovery and composition" instead of "reasoning", and adds the oracle table and the recall analysis below.

AuthorsOpenReview ↗

Response to Reviewer L82R (3/3): BM25 Recall, Selection Bias and Basel Realism

W2 / Q3 (continued). BM25 recall.

In brief: the reviewer is right that the BM25 vs full-context comparison did not support our claim that retrieval "scales better". We have replaced that claim with the measurements below.

BM25 gold-chain recall. Section-level recall of the gold chain by BM25 top-10, and the share of items whose full chain is retrieved:

DNV tierT0T1T2T3T4
gold-chain recall89%89%63%79%65%
full chain retrieved89%89%30%50%15%
Basel tierT0T1T2T3
gold-chain recall84%76%79%75%
full chain retrieved84%76%45%32%

Retrieving the full chain makes a large difference: BM25+Sonnet scores 62.9% on DNV and 73.5% on Basel when the full chain is retrieved, and 22.5% and 33.0% when it is not.

W3 / Q4. Selection bias from generation and filtering.

In brief: the released set under-represents table-heavy deep-chain items, but not exception- or condition-heavy reasoning. We also correct a sentence of ours that the reviewer refers to.

A correction first. The sentence ("the audit removes exception-heavy and table-arithmetic items", App. C) is inaccurate. The audit-and-repair loop removed only 10 items across both corpora, and table-cell defects were mostly repaired, not removed (§3.4, §3.6). We have rewritten the sentence.

Where the filtering happens. The main filter sits upstream, in the generation confidence check. On DNV, 169 of 681 generated candidates were rejected, concentrated in the deeper tiers (T0: 2, T1: 5, T2: 63, T3: 51, T4: 48). The pipeline does find deeper chains harder to construct reliably, and the released set under-represents some of them. So that this tier skew does not drive the comparison, we compared rejected and retained candidates within each tier:

  • Table-heavy and deeper-tier items are rejected more often. More rejected than retained candidates reference a table (T2: 57% vs 40%; T4: 67% vs 47%). Navigation-style questions have the highest rejection rate (57%).
  • Exception language is more common among retained items. Exception language ("unless", "except", "provided that", "notwithstanding", "otherwise", "in lieu" and similar; full list in the appendix) is less frequent among rejected candidates than retained ones (T2: 16% vs 28%; T3: 18% vs 25%; T4: 23% vs 34%). Formula-branch and gating-condition questions are almost always retained (rejection 6–8%), and edge-case questions always are.

We have added this finding to the limitations as a property of the construction method.

"Expert-level" (Basel). Agreed that the support differs by corpus. As stated in §3.2, the realism claim rests on DNV, whose scope was set with practising ship designers; Basel serves as evidence that the construction pipeline transfers, and its scenarios were not validated by an in-the-loop banking practitioner. What Basel does have is expert review of its answer keys. Four finance-credentialed reviewers audited 79 Basel items against the regulation text: three re-reviewed the 59 items v5d flagged (49 repaired, 7 removed, 3 spurious flags), and one audited a stratified sample of 20 passed items (App. B). Finance reviewers also labelled every hop in 60 Basel items in the hop audit (App. S.7).

We hope the clarified positioning and the new oracle and recall results address the reviewer's concerns, and we would be glad to clarify anything further.

Reviewer_A9JuOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RegBench is an 827-item benchmark (546 DNV ship-rule items, 281 from 12 CFR 217) where each item is a compliance scenario whose answer requires following an explicit cross-reference chain. Items are LLM-generated over chains sampled from the corpus graph, filtered by an LLM audit (v5d) calibrated against in-house SME review, and graded by strict conjunction over LLM-written atomic facts, using an LLM judge validated against SME labels and a second-vendor judge. Frontier models score 36–57% strict (MiniMax 17%), accuracy falls with chain scope, and BM25+reader beats full-context at the deepest tiers.

Critical strengths

The authors have done a lot of checks which people normally skip in LLM generated benchmarks. Held out audit split, second judge from a different vendor on all the answers (40 out of 42 orderings came out same), generator swap, perturbed answer keys, closed book 0%. Since Sonnet 4.6 is generating the items, writing the answer keys, judging, and also getting evaluated, I would not have trusted this leaderboard without these checks. So this is good.

Critical weaknesses

Strict accuracy is mixing two different things, a model which followed the wrong chain and a model which followed the right chain but wrote it in short. Appendix N itself says this, but nowhere is it separated. Table 3 only reports strict, and the prompt which is given to the model is not in the paper. So if the model is not told to write every intermediate fact, then some part of this 17–57% number is basically a penalty for writing less, and the level of detail is decided by how Sonnet itself breaks down the facts. I would like to see, for each system, how many of the strict failures actually had the correct final verdict.

Questions for authors

The graph-RAG collapse is in the abstract, but Appendix N says the three systems use different readers and different retrieval, so the pilot does not tell where the gap is coming from. Can you run them with the same reader on the 499 DNV items? Otherwise I think this claim should not be in the abstract.

5 comments
AuthorsOpenReview ↗

Response to Reviewer A9Ju (1/2): Strict vs Verdict Accuracy, Prompt

We thank the reviewer for the careful reading and for recognising the validation checks. Both requests were concrete and useful, and we address each below with what the reviewer asked for. All changes are in the revised PDF.

W1. Strict accuracy mixes wrong-chain answers with right-chain-but-concise answers; the answer prompt is not in the paper.

In brief: the gap between strict and verdict accuracy is substantial, but it does not come from bare verdicts: every answer states a derivation. We now include the prompt and report verdict accuracy for every system. The top system on DNV and the last system on both corpora stay the same.

Prompt. The prompt is already disclosed in our GitHub repository, and we have now added it to the appendix (App. I). The prompt asks the model to locate the relevant clauses, reason through them step by step, cite the clause numbers it uses, and give concrete numeric results, and the answers do so: no system returns a bare verdict, every system's median answer exceeds 900 characters, and at least 98% of each system's answers cite clause numbers.

Verdict accuracy. For every released item, we marked which required fact(s) state the final answer the question asks for: the compliance decision or final value, one per part for multi-part questions. An LLM proposed the tags from the question and fact list only (no model answers), and the authors checked a sample. A response is verdict-correct if every such verdict fact is judged present. This is computed from the judge's existing fact-level decisions; no answers are re-scored.

SystemDNV strictDNV verdictstrict failures with correct verdictBasel strictBasel verdictstrict failures with correct verdict
GPT-5.6-sol49.368.538%56.276.546%
BM25 + Sonnet 4.644.561.931%56.972.236%
Sonnet 4.644.158.726%52.069.436%
GPT-5.443.163.336%45.270.546%
GPT-5.6-terra40.564.140%49.574.449%
Qwen3.6-Plus35.961.340%35.667.349%
MiniMax-M2.716.626.111%17.124.29%

Size of the gap. For most systems, 26–49% of strict failures still reach the correct verdict. These answers state a derivation but omit or misstate at least one required intermediate fact. The two scores therefore answer different questions: strict accuracy asks whether the derivation is stated, and verdict accuracy asks whether the decision is right. Auditable compliance work needs the first, and the table now makes the gap between them visible.

Effect on the ranking. Under verdict accuracy, the systems in the middle of the ranking change order, and their differences are not statistically significant. On DNV, GPT-5.6-sol remains first and is significantly ahead of every other system (paired McNemar, p ≤ 0.008 against each, uncorrected), and MiniMax remains last on both corpora. The five systems in between lie within 5.4pp on DNV and 7.1pp on Basel, and no pair among them differs significantly after multiple-comparison correction. The revised paper reports verdict accuracy alongside strict accuracy in a new appendix table, referenced from §4.2.

Systems also differ in how much of the derivation they state: among the six systems above MiniMax, the two with a Sonnet 4.6 reader have the lowest share of strict failures with a correct verdict (DNV 26% and 31% vs 36–40%; Basel 36% vs 46–49%). Our DeepSeek-v4-pro cross-judge (§4.4) shows the same pattern (DNV 27% and 30% vs 37–40%; Basel 34% and 36% vs 44–48%; App. S), which rules out a judge preference and displays the actual measurement difference between our derivation-level scoring and pure result-based measurement.

Fact decomposition. The reviewer's concern that the level of detail follows Sonnet's fact decomposition is fair, and the answer-key sensitivity study (App. R) tests it directly. It re-scores all seven leaderboard systems against SME-certified alternative decompositions: on DNV, 911/1001 item-level decisions agree (91.0%, ρ = 0.90); on Basel, the ranking is unchanged (ρ = τ = 1). Within the alternative decompositions tested, the ranking is therefore robust to how facts are segmented.

AuthorsOpenReview ↗

Response to Reviewer A9Ju (2/2): Same-Reader Graph-RAG

Q. Graph-RAG claim in the abstract.

Agreed: the 47-item pilot cannot support an abstract-level claim, because the systems differ in both retrieval and reader (HippoRAG and GraphRAG used a Claude Haiku 4.5 reader, and ArchRAG used Sonnet 4.6). We have replaced the pilot-based graph-RAG sentence in the abstract with the controlled result below, and removed it from the contribution list. App. N now states each system's reader and labels the comparison as a pilot that does not isolate retrieval.

We also ran the controlled comparison the reviewer asked for. On all 499 DNV and all 281 Basel items, HippoRAG and GraphRAG use their own retrieval over the same corpus as BM25. All three arms share the same reader (Sonnet 4.6), answer prompt (the BM25 arm's) and judge:

Strict accuracy (DNV, n = 499)AllT0T1T2T3T4
HippoRAG retrieval + Sonnet47.5%6666423429
BM25 retrieval + Sonnet44.5%6565353720
GraphRAG local-search context + Sonnet26.3%493620206
Strict accuracy (Basel, n = 281)AllT0T1T2T3
HippoRAG retrieval + Sonnet53.7%78733827
BM25 retrieval + Sonnet56.9%70705435
GraphRAG local-search context + Sonnet40.6%62502724

With the reader held fixed, the two graph methods diverge:

  • HippoRAG does not differ significantly from BM25 on either corpus (paired McNemar: DNV p = 0.22, Basel p = 0.34). At the deepest tier it is ahead on DNV (T4: 29% vs 20%, p = 0.14) and behind on Basel (T3: 27% vs 35%, p = 0.24).
  • GraphRAG's local-search context, which supplies entity, relation and community summaries alongside text units, falls well below BM25 on both corpora (DNV p < 10⁻¹², Basel p < 10⁻⁶) and at every tier.

App. N now reports this comparison in place of the pilot claim, and we no longer say that graph-RAG methods as a class "collapse". We would be glad to clarify anything further.

AuthorsOpenReview ↗

Thank you, and looking forward to your thoughts

Thank you again for the constructive review. Following your suggestions, the revision reports verdict accuracy for every system (App. S.3, Table 22), includes the answer prompt (App. I), and replaces the graph-RAG pilot claim with a same-reader comparison on all 499 DNV and 281 Basel items (App. S.5, Table 24). As the discussion closes on November 18 (AoE), we would be glad to hear whether these address your questions, and to clarify anything further.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of New Analyses and Revisions

We thank all three reviewers for careful and specific reviews. Three concerns were shared, and we answer each with new analyses on the full leaderboard. All changes are in the revised PDF; details are in the per-reviewer replies.

  1. What the tiers measure (L82R, kyvb). We ran a gold-chain oracle on all 780 items for five models. The deepest-tier collapse partly reflects evidence that was not in the context: DNV T4 rises from 3–9% to 37–55%. A significant tier decline remains with the evidence given, controlling for required-fact count and context length (OR 0.86 per tier on DNV, 0.68 on Basel). We now describe the tiers as measuring cross-reference evidence discovery and composition.
  2. Strict vs verdict accuracy (A9Ju, kyvb). We tagged each item's verdict facts and now report verdict accuracy for every system alongside strict accuracy. The tier decline holds for verdict accuracy too, and the top and bottom of the ranking are unchanged.
  3. Graph-RAG (A9Ju, kyvb). The abstract's pilot claim is replaced by a controlled comparison. With one shared reader on all 499 DNV and 281 Basel items, HippoRAG is on par with BM25 (DNV 47.5% vs 44.5%, p = 0.22; Basel 53.7% vs 56.9%, p = 0.34), and GraphRAG's local-search context is well below (26.3% and 40.6%).

We also position RegBench against SearchFireSafety as a benchmark over long regulatory documents (L82R), and report an expert audit of hop necessity (kyvb).

AuthorsOpenReview ↗

Closing note

We thank all three reviewers and the AC for an engaged discussion. We summarise what the revised paper now establishes.

RegBench grades the complete source-grounded derivation that carries a scenario along a cross-reference chain through long regulatory documents to a verdict: 827 expert-level questions with 4,766 atomic facts from DNV Ship Rules and 12 CFR Part 217.

  1. What the tiers measure. A gold-chain oracle on all 780 items for five models shows that the deepest-tier collapse partly reflects evidence missing from the context (DNV T4 rises from 3–9% to 37–55%), while a significant tier decline remains with the evidence given. A length-matched oracle for the same five models holds input length equal across tiers, and the decline remains (pooled odds ratio per tier 0.81 on DNV, 0.60 on Basel; App. S.1). The tiers therefore measure cross-reference evidence discovery and composition.
  2. Verdict and completeness. Verdict accuracy is reported for every system alongside strict accuracy, and the oracle decomposition separates verdict accuracy from evidence completeness (App. S.3–S.4).
  3. Hop necessity. In an expert audit of 120 T2–T4 items, every hop of the gold chain is used by the question in 92% of DNV and 87% of Part 217 items (App. S.7).
  4. Graph-RAG. With one shared reader on all 780 items, HippoRAG is on par with BM25, and GraphRAG's local-search context falls 16–18pp below it on both corpora (App. S.5).
  5. Answer-key robustness. The answer-key sensitivity study now covers all seven leaderboard systems (App. R).
  6. Positioning. We credit SearchFireSafety and related statute benchmarks directly, and the abstract now states the long-document setting.

Status of the discussion. Reviewer kyvb found that the verdict and completeness metrics, the two oracles and the hop audit address their main concern about how to interpret the tier effects, and that the narrower framing around evidence discovery and composition is supported by the new analyses. Reviewer L82R noted that the oracle experiments address most of their concerns; their remaining question, on context length and the T2–T3 pattern, is answered in our follow-up with the length-matched oracle. Reviewer A9Ju's two requests, verdict accuracy with the answer prompt and a controlled graph-RAG comparison, are answered in App. S.3, App. I and App. S.5.

A highlighted copy of the revised paper, with every change since submission marked, is in the supplementary material. We are grateful to the reviewers for questions that made the paper substantially clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.