- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CovenantBench, a benchmark and reinforcement-learning environment for legal reasoning over corporate debt contracts. It has three layers: (i) agentic document-navigation tasks over real indentures and credit agreements, with 4,042 evaluation tasks scored by deterministic checkers; (ii) outcome prediction on real covenant disputes, including a hand-audited held-out set of 101 issue prompts across 89 disputes; and (iii) an adversarial debate protocol in which advocate policies argue before a judge anchored to recorded court outcomes. The authors train a 9B Qwen3.5 model with QLoRA and GRPO under roughly $150 of compute. They report that training doubles zero-shot agentic pass@1 on held-out training-family tasks, lifts the base on every navigation family, and matches frontier-scale models on outcome prediction at lower inference cost. Additional audits probe memorization, clause sensitivity, calibration, and argument content. The paper releases the corpus, environment, checkers, and adapters.
Critical strengths
The benchmark design is a genuine resource contribution. CovenantBench combines real instruments, agentic sandbox navigation, deterministic checkers, and court-outcome anchoring. This is a meaningful step beyond passage-classification legal NLP benchmarks, and the released environment/checkers make the contribution reusable.
The auditing is unusually careful and transparent. The paper reports quote audits, constant-answer baselines, closed-book memorization probes, clause counterfactuals, calibration, key-noise analysis, and matched-compute ablations. Even where results are null or modest, the authors state them clearly. This substantially increases trust in the reported navigation gains and the released benchmark.
Critical weaknesses
The outcome-prediction and debate layers do not yet establish a clear scientific claim. On the 101 pristine prompts, all arms land between 0.89 and 0.91 balanced accuracy with overlapping intervals. The trained judge is not clearly better than the base or frontier models, and the debate self-play ablation finds no detectable advantage over plain anchoring at matched compute. The paper is honest about this, but it means the headline “judge anchoring” and “debate” contributions are better read as infrastructure/protocol contributions than as demonstrated performance improvements.
Label and key provenance still limits what the benchmark can claim. Outcome labels are LLM-assisted under human review, with no attorney audit; the blind second annotation is another LLM pipeline. The evaluation-suite key has a measured noise band, and the corrected generator changes 228 of 936 held-out dependency keys. Fourteen of 43 mounts are proxies or inconclusive. These issues are documented, but they bound the strength of claims about legal correctness and make the benchmark’s key quality a first-order concern.
Questions for authors
Can the authors provide an attorney-audited subsample of the 101 pristine outcome prompts, even if small, with reported agreement and error analysis? This would materially strengthen the outcome layer.
Given the saturated outcome-prediction results, what exactly should readers conclude from the judge-anchoring stage? Is the claim that it produces a cheaper calibrated judge, a clause-sensitive judge, or a training signal for debate, rather than higher accuracy?
Can the authors report the navigation results under the corrected key generator for the full held-out draw, not only the stable-key subset? The 228 changed keys are a central robustness concern.
Since the debate ablation does not detect an advantage over plain anchoring at matched compute, what is the intended scientific takeaway of the debate protocol? Is it primarily a citation-discipline mechanism rather than an accuracy-improving method?
Can the authors clarify how much the 14 proxy/inconclusive mounts affect the L2 and dispute-suite results, and whether the main conclusions hold on the verified-mount subset alone?
Response to Reviewer 5QWJ
Thank you for your review. We adopt your framing of the outcome and debate layers as infrastructure and protocol contributions rather than demonstrated performance improvements. One of the checks in the general response answers your first weakness: the 0.89–0.91 band across all arms was produced by a posture line that stated the disposition in 59 prompts; with it removed the arms separate, and the 9B judges trail the 397B by 10 points (general response §2).
Q2 — what to conclude from Stage 3 (judge anchoring). Against the base on the same harness, Stage 3 left accuracy unchanged (0.802 vs 0.811 on posture-stripped prompts), calibration unchanged (vote-matched ECE gap −0.009 [−0.052, +0.042]), and clause sensitivity mostly unchanged (both move correctly on 11 of 14 patterns; training removes two wrong-direction cells). What it bought is reliability: the share of calls that end without a usable verdict drops from 9–10% to ~2%. The revision reports Stage 3 as a negative result and corrects the conclusion, which credits training with these judge properties.
Q4 — takeaway of the debate protocol. As you rightly noted, the debate mechanism bought citation discipline (0.679 → 0.766 over the rounds, against 0.719 for plain anchoring at matched compute) and usable submissions, not accuracy. The scientific takeaway is a negative result: self-play against a judge whose verdict is mostly fixed by the record (general response §3) cannot reward argument substance, so it reproduces plain anchoring on every metric except the one term the reward checks mechanically. This places the result inside the debate literature rather than against it: debate as oversight depends on a judge who needs the debaters for the decision-relevant information (Irving et al., 2018), and its advantage over the judge answering alone appears where that asymmetry exists and is mixed where it does not (Kenton et al., 2024). Our judge receives the record — facts and quoted provisions — which already carries most of the outcome (general response §3: the record alone scores 0.74).
Q3 — corrected keys on the full draw. Switching keys moves every arm by ≤0.003. On corrected keys, trained − base is +0.028 [+0.002, +0.054]; the 397B lead is unchanged; the trained model is within noise of Kimi and DeepSeek-V3.2; and its gain over base is at depth 1 (+0.086 [+0.022, +0.152]), with no separation at depth 2 (+0.008 [−0.029, +0.047]) or depth 3 (−0.011 [−0.049, +0.029]) under the 10-call budget. The +0.058 in Appendix E was the gain on the 183 tasks whose keys the correction left unchanged, where the gain concentrates; the full draw was +0.029 at submission. The revision replaces that subset analysis with the full-draw result.
Q5 — proxy and inconclusive mounts. Judge anchoring reads no mount, so Fig. 2 is unaffected. Holding prediction does. Re-checking every held-out mount on instrument identity (issuer, series, date) and re-evaluating quote overlap found two misses: a shelf indenture mounted in place of the instrument the court construed, and a related party's near-identical indenture. No proxy or inconclusive mount enters the dispute suite. One of its ten mounts is the related party's indenture noted above; it serves five tasks that ask for section numbers, a quoted passage and a put price, and every one of those answers is the same in the court's instrument.
Both construed instruments are now mounted from the public filings and pass the quote audit; every arm was re-run on the three affected tasks; and the revision's column is all 13 tasks on verified mounts, with the ordering of arms unchanged (general response §2). Proxy mounts (a different series by the same issuer, or a companion document of the same deal, where the construed instrument is not public) and inconclusive mounts remain in the navigation suite only, where the task is structural and the keys are derived from the mounted text, so neither can produce a wrong key; neither enters holding prediction. The provenance appendix lists, per mount, the instrument the opinion describes, the document mounted, and the count of opinion quotations matched.
W2 / Q1 — label provenance and an attorney-audited subsample. Your description is accurate and the revision will adopt it. We agree an attorney review would materially strengthen the layer and have begun arranging a blind review of a subsample of the held-out prompts. We cannot commit to completing it within the discussion period; the limitations section will state the status. In the meantime the paper will describe the labels for what they are — the courts' recorded outcomes, adjudicated by us rather than by attorneys — and claim nothing beyond that.