- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
RegBench evaluates whether language models can work through cross-references in regulatory documents and produce grounded compliance decisions. The benchmark draws on maritime and banking regulations and combines graph-guided question construction with fact-level checking and expert review. This is a worthwhile setting, and the care taken with data auditing is a strength. My concern is that the evaluation does not yet clearly separate cross-reference reasoning from finding relevant provisions and producing a complete answer.
Critical strengths
The paper tackles a concrete problem that general long-document QA benchmarks often miss: applying rules spread across clauses, tables, and sections. The construction pipeline is also more careful than a typical LLM-generated QA dataset. In particular, the expert checks, judge calibration, and sensitivity analyses give readers useful information about annotation quality.
Critical weaknesses
1.
The questions do not always appear to require the sampled reference chain. The paper builds questions around paths through a regulation graph, but a path used to generate a question is not necessarily a path needed to answer it. In Appendix G, example F500_0015 combines a numerical compliance decision about a discharge opening with a separate question asking where a hatch-cover requirement is specified. Both involve regulations, but the latter does not feed into the former. More broadly, 339 of the 780 leaderboard questions are T0/T1 and do not require crossing sections. Those are reasonable tasks to include, but aggregate accuracy should not be presented as a direct measure of cross-section reasoning. A small expert audit of whether each hop is actually necessary would help considerably.
Strict accuracy makes the tier results hard to interpret. An answer is marked wrong if it omits any required atomic fact, even when its calculation and compliance verdict are correct. As questions require more facts, this rule alone can lower accuracy. That makes it difficult to tell how much of the T0-to-T4 decline reflects harder reasoning rather than longer answer keys. The answer-key sensitivity test adds to this concern: alternative expert-approved decompositions change 55 of 715 item-level decisions (Appendix R). I would like to see verdict accuracy and evidence completeness reported separately, with tier comparisons controlled for the number of required facts. For DNV T4, it would also help to know whether the provided context actually contains every gold provision after the context expansion and length cap.
2.
The choice of the final audit prompt needs an explanation. Table 12 reports identical recall for v5b and v5d (81.2%), but v5b has better precision (52.0% versus 50.0%) and agreement (89.9% versus 89.2%). This seems at odds with the stated no-regression freeze criterion. There may be an operational reason to prefer v5d, but it is not evident from the reported results.
A smaller point: the Graph-RAG comparison covers only 47 pilot questions and changes more than the retrieval strategy. It is useful as a pilot, but too limited to support a broad conclusion about graph-based retrieval.
I see substantial work in building and checking this benchmark. The missing piece is stronger evidence that the reported scores capture the specific reasoning ability the paper sets out to test. I therefore lean Weak rejection (2/4).
Questions for authors
- In a representative sample of T2–T4 questions, how often is every hop in the gold reference chain necessary for the final decision?
- Does the tier-level performance decline remain after controlling for required-fact count and gold-evidence coverage? Could you report final-verdict accuracy separately from strict completeness?
- What criterion led to selecting v5d over v5b, given the results in Table 12?
7 comments
Response to Reviewer kyvb (2/3): Fact Count, Coverage, Verdict vs Completeness
W1b / Q2. Tier decline vs number of facts, gold-evidence coverage, and verdict vs completeness.
Fact count. The number of required facts rises only modestly with tier: DNV T0–T4 average 5.36 → 6.03 facts; Basel T0–T3 average 5.68 → 6.03. In a logistic regression of strict correctness on tier, fact count and system (item-clustered SEs), tier remains strongly significant:
- DNV: OR 0.54 per tier, p < 10⁻²⁸. Without the fact-count term the OR is 0.51.
- Basel: OR 0.55 per tier, p < 10⁻¹³.
The decline also persists at a fixed fact count. Among DNV items with exactly 6 facts, strict accuracy falls 58% → 51% → 37% → 23% → 6% across T0–T4. Longer answer keys therefore do not explain the decline.
Gold-evidence coverage (including DNV T4). The reviewer's concern is justified. At DNV T4, the full-context input does not contain the complete gold chain: T4 chains cross Parts, and the context is truncated at the length cap. Every system receives the same practitioner-scope context, capped at 600K characters due to context limit. The full-context decline at the deepest tiers partly reflects evidence that was not in the context, and we have revised our presentation accordingly. We ran a gold-chain oracle on all 780 items (per-model table in our reply to L82R), and the revised paper presents the tier results under both conditions.
With coverage guaranteed, the results still support our claim. A significant decline nevertheless remains: under the oracle, DNV falls from 71–81% at T0 to 37–55% at T4, and Basel from 65–90% to 32–65%. Controlling for required-fact count and gold-context length, tier stays significant (OR 0.86 per tier on DNV, p = 0.036; 0.68 on Basel, p = 0.001; item-clustered). The tier decline under full context thus combines missing evidence with a genuine composition cost. We now report both, and no longer attribute the whole decline to reasoning.
Verdict accuracy and evidence completeness, separately and controlled for fact count and evidence coverage. Using the judge's existing fact-level decisions, we tagged each item's final-answer fact(s). Verdict accuracy requires only those facts to be present; evidence completeness is the share of the remaining derivation facts present. To control for coverage as well, we pool the five systems that ran both full context and the gold-chain oracle (%; per-tier change controlling for fact count):
| T0 | T1 | T2 | T3 | T4 | per tier | |
|---|---|---|---|---|---|---|
| DNV strict, full context | 66 | 61 | 47 | 33 | 5 | −13.9pp |
| DNV strict, oracle | 75 | 68 | 54 | 58 | 45 | −5.7pp |
| DNV verdict, full context | 85 | 79 | 81 | 52 | 19 | −14.9pp |
| DNV verdict, oracle | 89 | 84 | 87 | 85 | 76 | −1.6pp |
| DNV completeness, full context | 87 | 82 | 77 | 64 | 38 | −11.5pp |
| DNV completeness, oracle | 90 | 87 | 81 | 84 | 77 | −2.8pp |
| Basel strict, full context | 64 | 64 | 37 | 26 | – | −13.9pp |
| Basel strict, oracle | 77 | 73 | 50 | 46 | – | −11.2pp |
| Basel verdict, full context | 86 | 79 | 68 | 54 | – | −10.6pp |
| Basel verdict, oracle | 92 | 87 | 72 | 74 | – | −6.7pp |
| Basel completeness, full context | 84 | 84 | 73 | 68 | – | −6.8pp |
| Basel completeness, oracle | 91 | 89 | 81 | 82 | – | −4.1pp |
Controlling for fact count barely changes any slope (DNV verdict under full context: −15.9pp per tier uncontrolled, −14.9pp controlled), so longer answer keys do not explain the decline. With every gold section given, strict accuracy still falls with tier on both corpora (DNV 75% → 45%, Basel 77% → 46%), and verdict accuracy falls as well, less steeply (DNV 89% → 76%, Basel 92% → 74%). Under the oracle, T2 and T3 score about the same on both corpora. We attribute this to the oracle supplying each section on the chain in full and side by side: it removes the distance a T3 chain spans across the rulebook while keeping the reference text intact, so the reader does not follow a wrong reference. Part of the deep-tier verdict loss under full context therefore reflects evidence the input lacked; with the evidence given, what remains on DNV is mainly a loss in stating the full derivation, which strict scoring captures and verdict-only scoring would miss.
Response to Reviewer kyvb (3/3): Answer Keys, v5d, Graph-RAG
Answer-key sensitivity (55/715). The 55 changed decisions are out of 715 system–item pairs for five systems, which is 92.3% agreement. We have extended the study to all seven leaderboard systems: 911/1001 = 91.0% agreement, with rank correlation ρ = 0.90. The one discordant pair lies within a group of four systems 1.4pp apart, and no system's score changes significantly. On Basel, the ranking is unchanged (ρ = τ = 1).
W2 / Q3. Why v5d over v5b?
We thank the reviewer for raising this. The freeze paragraph in App. M stated a different criterion from the one we actually used to choose v5d, and we have rewritten it. The choice was made on the SME-labelled 100-item subset of the development pool (the SME-3 set), with an objective that weights recall above precision (recall + precision/5, subject to recall ≥ 80%). On that subset, v5d scored 0.951 against 0.944 for v5b, with precision 58.8% vs 55.6%, flag rate 17% vs 18%, and fewer false flags at T3/T4 (5 vs 7).
On the full 148-item pool reported in Table 13, the two variants are effectively tied. They catch the same 13 SME-flagged items and differ only on non-flagged items, where v5d raises one additional false flag. This is the 52.0% vs 50.0% precision difference the reviewer notes. We now report both pools in Table 13, state the selection objective explicitly, and no longer claim that v5d dominates every ablation.
Graph-RAG Agreed. We have replaced the pilot with a controlled comparison on all 499 DNV and 281 Basel items in which only the retrieval differs (same Sonnet 4.6 reader, prompt and judge; tables in our reply to A9Ju). On both corpora, HippoRAG does not differ significantly from BM25 (DNV 47.5% vs 44.5%, p = 0.22; Basel 53.7% vs 56.9%, p = 0.34), while GraphRAG's local-search context is substantially weaker (26.3% and 40.6%).
We are grateful for these specific questions. They prompted analyses that make the paper's construct claims more precise, and we would welcome further comments.
Response to Reviewer kyvb (1/3): Hop-Necessity Audit
We thank the reviewer for a precise and well-argued review. Its central question is the right one: do the scores measure the reasoning we claim? We answer each point below with new analyses. All changes are in the revised PDF.
W1a / Q1. Do questions require the sampled chain?
The reviewer is right about F500_0015. Its second part ("where is the bolt-area requirement specified?") does not feed the freeing-port verdict. We have replaced it in App. G with an item whose chain feeds the decision.
To measure how common this is, we ran the audit the reviewer suggested. Domain experts (ship designers for DNV, finance reviewers for Basel; research-team members, blinded to model outputs and audit flags) labelled every hop in tier-stratified samples of T2–T4 items: 60 DNV items (112 hops) and 60 Basel items (132 hops). Each item was labelled by one expert. A hop is labelled necessary if its section can guide the reader to the right answer or holds a condition that could change the judgement. It is labelled not necessary only if it cannot affect the judgement in any way. The full definition and ruling notes are in App. S.7.
Across both corpora, nearly every hop in the gold chains is used by the question: every hop is necessary or needed for a side part in 92% of DNV items (T2 17/20, T3 19/20, T4 19/20) and 87% of Basel items (T2 25/30, T3 27/30). Fewer items have every hop bearing on the final judgement: 40% on DNV (T2 12/20, T3 7/20, T4 5/20) and 58% on Basel (T2 18/30, T3 17/30). Most of the remaining hops serve a side part of the question, as in F500_0015. The revised paper makes per-tier results the primary view and reports the audit per corpus and tier.
On T0/T1. The revised §4 makes the per-tier results the primary view and describes the aggregate as an overall benchmark score.