acceptodds
Under review as a conference paper at ICLR 2027

CovenantBench: Navigating Corporate Debt Contracts with Litigation-Verified Rewards

Abstract

We present CovenantBench, a benchmark and reinforcement-learning environment for legal reasoning over corporate debt instruments, anchored to verifiable outcomes of real covenant litigation. The benchmark has three layers: (i) agentic document-navigation tasks over actual indentures and credit agreements pulled from public filings — a 4,042-task evaluation suite with audited keys — scored by deterministic checkers; (ii) outcome prediction on real covenant disputes, with a hand-audited held-out evaluation set — 101 issue prompts across 89 disputes, every record reviewed line-by-line against the deciding opinion — each labeled by the recorded outcome and the kind of motion or appeal the court was deciding; and (iii) an adversarial debate protocol in which advocate policies argue both sides of contested clauses before a judge whose reward is anchored to the true outcome. We train a 9B-parameter model (QLoRA + GRPO, under 150 of training pays for itself across every layer: it doubles zero-shot agentic pass@1 on 825 held-out tasks of its training families (0.283 →0.558 composite, 0.234 →0.485 fullmarks, thrice re-evaluated), lifts the base on every family of the audited evaluation suite, and matches frontier-scale models on outcome prediction at roughly one-fifth their inference cost. Second, the trained judge’s predictions respond to the clause, not just the facts: holding the facts fixed and outcome-scrubbed, swapping a protective clause for a permissive one moves P(creditor) in the doctrinally correct direction. Third, our arms do not answer from memory, while frontier arms partly do: closed-book probes place both our base and trained models at chance, while the comparison model 44× larger in total parameters recites actual rulings from memory. We release the corpus, environment, checkers, and adapters.

Primary area:
datasets and benchmarks
Keywords:
benchmark, legal reasoning, LLM agents, document understanding, tool use, reinforcement learning, verifiable rewards, GRPO, debate, financial contracts

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5QWJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CovenantBench, a benchmark and reinforcement-learning environment for legal reasoning over corporate debt contracts. It has three layers: (i) agentic document-navigation tasks over real indentures and credit agreements, with 4,042 evaluation tasks scored by deterministic checkers; (ii) outcome prediction on real covenant disputes, including a hand-audited held-out set of 101 issue prompts across 89 disputes; and (iii) an adversarial debate protocol in which advocate policies argue before a judge anchored to recorded court outcomes. The authors train a 9B Qwen3.5 model with QLoRA and GRPO under roughly $150 of compute. They report that training doubles zero-shot agentic pass@1 on held-out training-family tasks, lifts the base on every navigation family, and matches frontier-scale models on outcome prediction at lower inference cost. Additional audits probe memorization, clause sensitivity, calibration, and argument content. The paper releases the corpus, environment, checkers, and adapters.

Critical strengths

The benchmark design is a genuine resource contribution. CovenantBench combines real instruments, agentic sandbox navigation, deterministic checkers, and court-outcome anchoring. This is a meaningful step beyond passage-classification legal NLP benchmarks, and the released environment/checkers make the contribution reusable.

The auditing is unusually careful and transparent. The paper reports quote audits, constant-answer baselines, closed-book memorization probes, clause counterfactuals, calibration, key-noise analysis, and matched-compute ablations. Even where results are null or modest, the authors state them clearly. This substantially increases trust in the reported navigation gains and the released benchmark.

Critical weaknesses

The outcome-prediction and debate layers do not yet establish a clear scientific claim. On the 101 pristine prompts, all arms land between 0.89 and 0.91 balanced accuracy with overlapping intervals. The trained judge is not clearly better than the base or frontier models, and the debate self-play ablation finds no detectable advantage over plain anchoring at matched compute. The paper is honest about this, but it means the headline “judge anchoring” and “debate” contributions are better read as infrastructure/protocol contributions than as demonstrated performance improvements.

Label and key provenance still limits what the benchmark can claim. Outcome labels are LLM-assisted under human review, with no attorney audit; the blind second annotation is another LLM pipeline. The evaluation-suite key has a measured noise band, and the corrected generator changes 228 of 936 held-out dependency keys. Fourteen of 43 mounts are proxies or inconclusive. These issues are documented, but they bound the strength of claims about legal correctness and make the benchmark’s key quality a first-order concern.

Questions for authors

Can the authors provide an attorney-audited subsample of the 101 pristine outcome prompts, even if small, with reported agreement and error analysis? This would materially strengthen the outcome layer.

Given the saturated outcome-prediction results, what exactly should readers conclude from the judge-anchoring stage? Is the claim that it produces a cheaper calibrated judge, a clause-sensitive judge, or a training signal for debate, rather than higher accuracy?

Can the authors report the navigation results under the corrected key generator for the full held-out draw, not only the stable-key subset? The 228 changed keys are a central robustness concern.

Since the debate ablation does not detect an advantage over plain anchoring at matched compute, what is the intended scientific takeaway of the debate protocol? Is it primarily a citation-discipline mechanism rather than an accuracy-improving method?

Can the authors clarify how much the 14 proxy/inconclusive mounts affect the L2 and dispute-suite results, and whether the main conclusions hold on the verified-mount subset alone?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5QWJ

Thank you for your review. We adopt your framing of the outcome and debate layers as infrastructure and protocol contributions rather than demonstrated performance improvements. One of the checks in the general response answers your first weakness: the 0.89–0.91 band across all arms was produced by a posture line that stated the disposition in 59 prompts; with it removed the arms separate, and the 9B judges trail the 397B by 10 points (general response §2).

Q2 — what to conclude from Stage 3 (judge anchoring). Against the base on the same harness, Stage 3 left accuracy unchanged (0.802 vs 0.811 on posture-stripped prompts), calibration unchanged (vote-matched ECE gap −0.009 [−0.052, +0.042]), and clause sensitivity mostly unchanged (both move correctly on 11 of 14 patterns; training removes two wrong-direction cells). What it bought is reliability: the share of calls that end without a usable verdict drops from 9–10% to ~2%. The revision reports Stage 3 as a negative result and corrects the conclusion, which credits training with these judge properties.

Q4 — takeaway of the debate protocol. As you rightly noted, the debate mechanism bought citation discipline (0.679 → 0.766 over the rounds, against 0.719 for plain anchoring at matched compute) and usable submissions, not accuracy. The scientific takeaway is a negative result: self-play against a judge whose verdict is mostly fixed by the record (general response §3) cannot reward argument substance, so it reproduces plain anchoring on every metric except the one term the reward checks mechanically. This places the result inside the debate literature rather than against it: debate as oversight depends on a judge who needs the debaters for the decision-relevant information (Irving et al., 2018), and its advantage over the judge answering alone appears where that asymmetry exists and is mixed where it does not (Kenton et al., 2024). Our judge receives the record — facts and quoted provisions — which already carries most of the outcome (general response §3: the record alone scores 0.74).

Q3 — corrected keys on the full draw. Switching keys moves every arm by ≤0.003. On corrected keys, trained − base is +0.028 [+0.002, +0.054]; the 397B lead is unchanged; the trained model is within noise of Kimi and DeepSeek-V3.2; and its gain over base is at depth 1 (+0.086 [+0.022, +0.152]), with no separation at depth 2 (+0.008 [−0.029, +0.047]) or depth 3 (−0.011 [−0.049, +0.029]) under the 10-call budget. The +0.058 in Appendix E was the gain on the 183 tasks whose keys the correction left unchanged, where the gain concentrates; the full draw was +0.029 at submission. The revision replaces that subset analysis with the full-draw result.

Q5 — proxy and inconclusive mounts. Judge anchoring reads no mount, so Fig. 2 is unaffected. Holding prediction does. Re-checking every held-out mount on instrument identity (issuer, series, date) and re-evaluating quote overlap found two misses: a shelf indenture mounted in place of the instrument the court construed, and a related party's near-identical indenture. No proxy or inconclusive mount enters the dispute suite. One of its ten mounts is the related party's indenture noted above; it serves five tasks that ask for section numbers, a quoted passage and a put price, and every one of those answers is the same in the court's instrument.

Both construed instruments are now mounted from the public filings and pass the quote audit; every arm was re-run on the three affected tasks; and the revision's column is all 13 tasks on verified mounts, with the ordering of arms unchanged (general response §2). Proxy mounts (a different series by the same issuer, or a companion document of the same deal, where the construed instrument is not public) and inconclusive mounts remain in the navigation suite only, where the task is structural and the keys are derived from the mounted text, so neither can produce a wrong key; neither enters holding prediction. The provenance appendix lists, per mount, the instrument the opinion describes, the document mounted, and the count of opinion quotations matched.

W2 / Q1 — label provenance and an attorney-audited subsample. Your description is accurate and the revision will adopt it. We agree an attorney review would materially strengthen the layer and have begun arranging a blind review of a subsample of the held-out prompts. We cannot commit to completing it within the discussion period; the limitations section will state the status. In the meantime the paper will describe the labels for what they are — the courts' recorded outcomes, adjudicated by us rather than by attorneys — and claim nothing beyond that.

Reviewer_5KPSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new benchmark called Covenant Bench. It tests LLMs on long corporate debt contracts in a sandbox environment. The benchmark has three levels: L1 tests finding sections in raw contracts, L2 tests predicting court decisions without reading the opposing arguments, and L3 tests two models debating each other with an LLM judge. The authors test Qwen3.5-9B and train it with four stages: SFT on tool use, RL, judge alignment, and self-play debate. They report that this small model works well, and they show it does not just memorize case answers by using a closed-book test.

Critical strengths

1.Most existing legal benchmarks like CUAD or LegalBench only use short text snippets. This paper puts real, full contracts into a terminal sandbox so the model has to run commands to look up clauses itself, which feels much closer to how real lawyers work.
2.The closed-book test is quite a innovative check. By giving the model zero context, they show its accuracy drops to around 50% (random guess), which proves the model is actually reading the document instead of just memorizing famous court cases.

Critical weaknesses
  1. The reward metric in Stage 4 debate self-play is decoupled from substantive legal reasoning and remains confined to surface-level formatting: The reward formulation used in Stage 4, presents notable issues. This metric merely checks whether cited sections exist within the contract, while the judge model does not inspect the underlying contract documents when adjudicating, leaving the system entirely unable to verify whether a cited section constitutes an operative provision central to the dispute. (Although the authors acknowledge this phenomenon in Appendices K and L, empirical results demonstrate that after two rounds of self-play, the proportion of citations pointing to real sections rises to 94%, yet the proportion landing on provisions actually in dispute remains stalled at 27%-29%. Consequently, the adversarial debate protocol fails to improve this probability or resolve the core issue, suggesting its practical efficacy is far less substantial than claimed.)
  2. Although the authors identify the length-bias artifact in judicial opinions—where longer arguments systematically prevail-and introduce real-brief prompts from pre-trial filings as a remedy, the current mitigation remains severely limited in scale (covering only 21 queries across 17 cases), failing to comprehensively replace the 101-query primary test set with unbiased original materials.
  3. Insufficient literature citations: The references listed at the end of the main paper total only 18, which is markedly sparse for a comprehensive benchmark paper.
Questions for authors

Regarding Weakness 1: Why not simply add a penalty term to the reward function-if the clause cited by the model actually exists but is completely irrelevant to the dispute in this case, penalize it directly (e.g., -0.1)?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5KPS

Thank you for the review and for the concrete proposal.

W1 — the reward checks existence, not relevance. Yes. The judge's verdict, the other 90% of the reward, is also insensitive to citation content: swapping every section one side cites for a real but irrelevant section of the same instrument does not move the frozen judge (general response §3). So nothing in the Stage 4 reward responds to relevance, which is why citation validity rose from 0.70 to 0.94 from base to stage 4 while the share of citations on the disputed provisions stayed at 0.27–0.29 throughout. The revision will state this up front and will describe the debate protocol as a released environment whose reward moved the two dimensions it paid for (reliability and validity) rather than as a method that improved argumentation.

Q1 — why not a relevance penalty. The proposed penalty needs a reference for 'relevant to this dispute'. About half the opinions in the held-out set cite no section number of the instrument at all. So the only reference available is the one to three sections of the mounted instrument whose text matches the provisions the opinion quotes — the list the 0.27–0.29 alignment figure is computed against. A sound covenant argument also cites the definitions the operative clause depends on and the remedies provisions it triggers, which the court may not have "construed". That penalty would teach the advocate to cite only the listed sections and would work against the navigation skill the benchmark is built around. Two verifiable forms avoid this: a positive alignment term (share of cited sections in the construed list, or the alignment-F1 the paper names as headroom), and a penalty only for citations to structurally irrelevant sections (notices, counterparts, governing law, and the like, identifiable from section titles), which is exactly how we built the irrelevant-citation attack in the general response §3.

On your question of whether −0.1 would be enough: yes. The existing 0.1 validity term moved validity from 0.679 to 0.766 over the debate rounds. A relevance term at the same weight should carry comparable signal and raise alignment.

We cannot retrain and re-evaluate Stage 4 credibly inside the discussion period, so we report no result; the revision will scope it for future work, and the release will ship the two inputs the reward term needs: the construed-provision lists and the title-based boilerplate filter from the citation-swap test.

W2 — brief-sourced control. Indeed, the brief-sourced control is small. It covers 21 issues across 17 cases, bounded by the availability of two-sided merits briefing, which the paper's sourcing appendix quantifies. We cannot replace the 101-prompt set with brief-sourced material inside the discussion period. Two things reduce the weight on that control. First, the posture correction (general response §2) removes the largest non-reading channel we have found in the primary set, and the revised Fig. 2 is on stripped prompts. Second, the control asked whether the judge picks up on the summaries' tilt toward the winning side. Three further results — posture, citation swap, argument deletion — now answer the same question the same way: the judge reads the record, not the arguments. The control's conclusion no longer rests on 21 issues alone.

W3 — references. The revision makes two changes to related work. First, it adds Kenton et al. (2024). They compare debate with a judge answering on its own. Debate helps when the judge lacks information that the debaters have; when the judge has the same information, the results are mixed. Our judge is given the full record, so it is in the second case, and no gain over plain anchoring is the expected result. Second, it corrects the sentence on debate. The submitted paper positioned the outcome-anchored judge as a contribution over the debate literature. The revision clarifies what it did: a judge trained to predict the court from the record decides from the record, so debate could not improve its accuracy over plain anchoring; what debate did improve is citation validity.

A small clarification on the closed-book probe. The chance-level result is for the two 9B arms; on the held-out issues the frontier arms answer 0.56–0.82 closed-book (397B 0.82, Kimi 0.71, DeepSeek-V4-Pro 0.56). We will make the abstract's sentence unambiguous.

Thank you to the authors for their detailed responses and additional explanations. I acknowledge the authors’ clarification regarding the positioning of the debate environment, as well as the supplementary comparative experiments on multiple sets of length biases. The above issues fall within the acceptable scope of research limitations and future work, and the original score remains unchanged.

Reviewer_64JKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CovenantBench, a benchmark and training environment for reasoning over corporate debt contracts. It includes document-navigation tasks with programmatic scoring, outcome prediction for real legal disputes, and a debate protocol evaluated by a learned judge. The authors also train a 9B model through supervised tracing, navigation RL, judge anchoring, and debate self-play. Experiments show improvements on several navigation tasks, while additional controls examine memorization, prompt construction, tool budgets, and the contribution of debate training.

Critical strengths

1.The benchmark addresses a useful problem beyond short-passage question answering. Finding operative clauses and following definitions across long contracts are meaningful document-grounded tasks. The sandbox environment and programmatic checkers provide a concrete way to study these abilities.

2.The paper includes useful controls and openly reports negative results. The demonstration and tool-budget controls show how much agent performance depends on the evaluation setup. The authors also report that debate does not clearly outperform additional direct anchoring, and that outcome-prediction accuracy does not improve over the base model. These findings help distinguish execution improvements from stronger legal reasoning.

Critical weaknesses

1.The judge changes tasks between training and use, but its validity for the new task is not sufficiently established. In Stage 3, the judge learns to predict recorded court outcomes from facts, provisions, and curated arguments. In Stage 4, it evaluates newly generated argument pairs and provides the advocate’s reward. These tasks are related, but they are not equivalent. Correctly predicting the historical winner does not show that the judge can distinguish a sound argument from a persuasive but unsupported one. The paper includes held-out outcome prediction, clause counterfactuals, and a brief-sourced control. These are useful, but they do not directly establish agreement with independent expert judgments on unseen generated argument pairs. The brief-sourced result also raises concerns about distribution sensitivity: on the 21-issue subset, the trained judge’s accuracy drops from 0.905 with curated arguments to 0.714 with actual pre-decision briefs. Appendix K states that the judge does not open the document when ruling on arguments. The reward therefore does not directly verify whether a cited provision supports the argument. The separate argument-content audit evaluates the advocates, rather than validating this judge as a reward model. My concern is not that reward exploitation has been demonstrated, but that the meaning and reliability of the debate reward remain unclear.

2.The results mainly support better navigation and more reliable output, rather than stronger substantive legal reasoning or a distinct benefit from debate. Table 1 shows no clear outcome-prediction improvement over the untrained base. Section 5.2 attributes about 72% of the gain on held-out training-family tasks to SFT, while Appendix K finds that additional direct anchoring performs similarly to debate under the matched step budget. Table 12 shows a meaningful increase in clean arguments, from 24/52 to 45/52. However, point recall conditional on a clean argument is similar for the base and final model: 0.693 versus 0.705. This supports improved reliability in producing usable arguments, but does not clearly demonstrate better argument content. The paper should separate navigation accuracy, output reliability, outcome prediction, and argument quality when stating its contributions. Navigation conclusions also remain partly provisional because Appendix E reports changes to 228 of 936 held-out dependency keys without full corrected-key rescoring. The stable-key sensitivity analysis is helpful, but does not settle performance on the complete corrected set.

Questions for authors

1.Judge validation: What evidence directly validates the frozen debate judge on unseen, model-generated argument pairs? Has its judgment been compared with independent legal-expert assessments, especially for fluent arguments that cite real but irrelevant provisions? Please distinguish this from court-outcome accuracy and the separate audit of advocate content. 2.Corrected references: Can the submitted model outputs be rescored against the corrected dependency keys? Which reported improvements and model comparisons remain supported? If this is not possible, which conclusions should be treated as provisional? 3.Dispute independence: How many held-out outcome-prediction prompts belong to disputes represented in training at another procedural stage? Can you report existing predictions separately for these prompts and the remaining disputes?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 64JK

Thank you for the review and for the suggestion to test whether the judge can tell a sound argument from a fluent one. We tested it directly; the answer is no. Below we add detail to the numbers in the general response.

W1 / Q1 — judge validity on generated pairs. We have no comparison with independent expert judgments on generated argument pairs. For fluent arguments that cite real but irrelevant provisions, the frozen judge's ruling does not change (general response §3: pooled Δ −0.011 [−0.057, +0.039] over 160 pairs; flips at the noise floor). Combined with the posture finding (§2), the deletion test (removing the winning side's argument moves P(gold) by 0.11 and flips 13 of 101 verdicts; the record alone scores 0.74), and the brief-sourced drop you cite, the judge is best described as a record-reader: facts, posture and provisions determine most of its verdict, the arguments' prose moves it by about 0.1, and citations do not move it at all. Appendix K's observation that it never opens the document is therefore not a gap in an otherwise valid grader; the grader does not police argument content at all.

Consequently, an argument is rewarded when the record favors it and citations exist. The revision will say so, and remove any language implying the judge validates argument quality. The Stage 4 gains can be summarized as usable submissions, win rate before this judge, and citation validity. Citation validity remains the only metric on which debate beats plain anchoring at matched compute (0.766 vs 0.719). We will clarify our contribution as better navigation and reliability, not stronger legal reasoning, and no benefit from debate beyond citation validity. The revision will report navigation accuracy, output reliability, outcome prediction, and argument quality separately as you suggest. The attribution you cite also stands: supervised tracing accounts for about 72% of the navigation gain and the two RL stages for the rest, the matched-compute ablation finds no debate-specific advantage beyond citation validity, and point recall conditional on a clean argument is flat (0.693 vs 0.705). The revision will report these numbers in the contributions list rather than leaving them in the appendices.

Q2 — corrected-key rescore. Per your comment, here is what the corrected-key re-run settles. On the full 300-task draw, the trained agent beats its base by +0.028 [+0.002, +0.054] (+0.029 on the old keys at submission); the gain is at depth 1 (0.140 → 0.226); at depth 3 the trained agent is level with its base (0.049 vs 0.060, −0.011 [−0.049, +0.029]). The arm ordering in Table 1 is unchanged, and no arm moves by more than 0.003 under any of the three key sets (published, corrected, v1.0). The +0.058 in Appendix E was the gain on the 183 tasks whose keys the correction left unchanged, not on the full draw; on the other 117 tasks the arms tie under either key set. "228 of 936" counts changed keys across the whole held-out pool; 117 of the 300 draw tasks behind Table 1 are affected. No Table 1 navigation conclusion now rests on the uncorrected keys.

Q3 — dispute independence. Four of 101 held-out prompts (three disputes: Marblegate, Serta, Charter) belong to a dispute represented in training at another procedural stage; nine more share a debtor with a training record but concern a different dispute. Existing predictions excluding first the four, then all thirteen, move no arm by more than 0.022 (general response §1). In three of the four straddling prompts the training-stage court ruled the other way, and every 9B arm predicted the held-out outcome, so the straddles do not act as an outcome-copy channel. The paper's "roughly seven" is replaced by these counts.

On the posture correction (§2). This was not raised in your review; we report it because it bears on the same question. The Fig. 2 accuracy of ~0.90 included prompts whose posture line stated the disposition. On stripped prompts the 9B judges score ~0.80, and ~0.90 on the 42 prompts whose record gives nothing away; the 397B and DeepSeek arms do not fall with them (0.905; 0.87–0.88), so the apparent parity across scale was the leak. The same line, removed from the holding-prediction task prompts, costs the 9B arms about 0.2 and the 397B nothing. On the 13 tasks now on verified mounts the 397B leads the trained 9B 0.683 to 0.596, a lead that does not exclude zero. Both results point the same way as your W2: training did not lift outcome prediction. The revision uses the stripped prompts throughout.

We hope these answers, and the revised framing, address the soundness concerns; we would welcome any further test you think would sharpen the judge question.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (1 of 3)

We thank all reviewers for their close reading and the time they invested. Our response addresses questions raised across reviewers: whether the navigation result holds under the corrected keys, what the judge and debate stages actually establish, and the extent to which the held-out outcome set is independent of training and of the mounted documents. We ran every check requested and added two of our own, prompted by the reviews. The checks confirmed the navigation result and the independence of the outcome set, and found two errors in the submission — a posture line that disclosed the outcome in 59 of 101 judge prompts, and two held-out mounts that were not the instrument the court construed. Both are corrected and re-run below; the revised PDF follows by Nov 13.

1. Checks

CheckRequested byResult
Navigation rescored under corrected keys, full 300-task held-out draw, all five arms64JK, 5QWJevery arm moves ≤ 0.003; trained − base +0.028 [+0.002, +0.054]
Held-out outcome prompts sharing a dispute (4) or a debtor (9 more) with training excluded64JKno arm moves by more than 0.022
Every mount read by a case-dependent evaluation checked against the opinion's description of the instrument (issuer, series, date) and its verbatim quotations5QWJTwo of eight holding-prediction mounts were not the construed instrument; both re-mounted from the public filings and the three affected tasks re-run for every arm; no arm ordering changes
Judge tested on model-generated argument pairs with citations swapped for irrelevant provisions64JKjudge does not move: Δ −0.011 [−0.057, +0.039]
Posture line removed from the judge prompts, and from the holding-prediction task promptsours, while building the test abovejudge: 9B arms and Kimi 0.90 → 0.80–0.82; DeepSeek arms 0.87–0.88; 397B unchanged at 0.905. Holding prediction: 9B arms fall ~0.2, 397B unchanged
Anchoring-only arm at matched compute, run on the 50-task dispute suite (Appendix G) and the 13 holding-prediction tasksours, for 5QWJ Q4 and 64JK W2ties the debate arm on both: 0.754 vs 0.755 (paired +0.001 [−0.06, +0.06]); 0.606 vs 0.596

Four of the six checks changed no comparison between arms, and the mount audit replaced two documents without changing any ordering. The posture check changed one ordering and reverses the paper's claim that the 9B judge matches frontier-scale models on outcome prediction. It also answers 5QWJ's saturation point: the 0.89–0.91 band was the posture line, not a ceiling. With the line removed, the arms separate by scale (9B arms and Kimi 0.80–0.82, DeepSeek 0.87–0.88, 397B 0.905).

AuthorsOpenReview ↗

General Response (2 of 3)

2. Two corrections

The 9B judge does not match the frontier arms on outcome prediction. The held-out judge prompts carry a one-line procedural posture from the case record. On 59 of 101 prompts it states the outcome ("affirmed dismissal", "vacated and remanded"). The facts were scrubbed of outcome language; the posture line was not. Every arm was re-run on the same prompts with the line removed (k=8):

armas submittedposture removedpaired Δ [95% CI]
Qwen3.5-397B0.9010.905+0.004 [−0.048, +0.061]
DeepSeek-V4-Pro0.8970.880−0.017 [−0.075, +0.029]
DeepSeek-V3.20.8920.872−0.021 [−0.065, +0.023]
Kimi-K2.60.9010.819−0.082 [−0.154, −0.025]
untrained 9B0.9100.811−0.099 [−0.189, −0.022]
trained 9B0.8970.802−0.095 [−0.186, −0.013]

The line gave every arm the answer on the same 59 prompts, so every arm sat at ~0.90. When removed, the 9B judges trail the 397B by 10 points and the DeepSeek arms by 7–8, level with Kimi. The loss comes entirely from the 59 leaky prompts (0.90 → 0.72–0.75), with the 42 clean prompts unchanged at ~0.90. The 397B does not move because it knows these rulings (0.82 closed-book). The revision will evaluate the stripped prompts throughout and ship them as the canonical set. We will remove the claim that the trained model matches frontier-scale models on outcome prediction, and report the ablation shown above in an appendix.

The same line sits in the holding-prediction task prompts (the sandbox task: facts plus the mounted instrument, arguments withheld). We removed it and re-ran every arm on all 13 tasks on verified mounts (two re-mounted; next paragraph); the three arms below ran eight episodes per task. The 397B scores 0.683 [0.54, 0.82], the trained 9B 0.596 [0.41, 0.77] and the base 0.519 [0.34, 0.69]. The trained leads the base by +0.077 [−0.07, +0.21], and the 397B leads the trained 9B by +0.087 [−0.14, +0.30]. As submitted (Table 1, two episodes per task) the order was trained (0.769) > 397B (0.731) > base (0.615). The posture line accounts for the reversal: on the ten tasks whose mounts did not change, removing it costs the trained 9B 0.21, the base 0.16, and the 397B nothing. The judged end-to-end layer is unaffected: re-judging the same stage-4 argument pairs with the line removed from the judge's prompt changes mean P(company) by +0.010 [−0.098, +0.128].

Two held-out mounts were not the construed instrument. The audit in the submitted paper matched exhibits by issuer and verified them by quote overlap with the mounted text; it did not check the series and date against the instrument the opinion describes, and its quote rule accepted a single match. Re-checking every mount in the held-out set against the instrument identity (issuer, series, date) and re-evaluating quote overlap with the mounted instrument found two misses: a shelf indenture mounted in place of the instrument the court construed and a related party's near-identical indenture. Both construed instruments are now mounted from the public filings and pass the quote audit, and every arm was re-run on the three affected tasks. The re-mount changes no ordering: on the 13 tasks the 397B still leads, the trained 9B still edges its base, and trained and anchoring remain tied. The release will ship both re-mounted instruments.

AuthorsOpenReview ↗

General Response (3 of 3)

3. What the judge responds to

All reviewers ask whether the judge grades arguments. We ran three tests:

  • Replace every provision an advocate cites with a real but irrelevant section of the same instrument, title visible ("Section 7.10 (Eligibility) creates an exception for dispositions…"): the verdict does not move (160 pairs, 8 votes each; flips at the noise floor).
  • Delete the winning side's argument entirely: P(gold) falls 0.11 and 13 of 101 verdicts flip; delete both arguments, and the record alone (facts + provisions, no posture) still scores 0.74.
  • Delete the posture line: accuracy falls 0.10; keeping only the caption, posture line and question, the 9B judge still scores 0.80 — the line alone is worth as much as the whole record without it.

The judge's verdict is therefore set mostly by the record: the arguments' prose moves it by about 0.1, and citations do not move it at all. Consequently, a side is rewarded when the record favors it and it makes valid citations. The debate rounds moved exactly what that pays for: usable submissions (empties 0.32 → 0.13), citation validity (0.679 → 0.766) and win rate before this judge (0.20 → 0.29). Plain anchoring at matched compute reproduces the reliability and win-rate gains and the end-to-end gain itself (0.754 to debate's 0.755 on the dispute suite), leaving citation validity as the only gain specific to debate (0.766 against 0.719).

We adopt 5QWJ's interpretation. Judge anchoring is a negative result: against the untrained base it changed neither accuracy nor calibration nor clause sensitivity, and bought output reliability (calls ending without a usable verdict 9–10% → 2%). The trained agent still leads its base on navigation (+0.028 [+0.002, +0.054]), on the dispute suite (0.755 vs 0.549, +0.21 [+0.11, +0.28]), and by +0.077 [−0.07, +0.21] on the 13 holding tasks. Against plain anchoring at matched compute it shows no advantage on either end-to-end suite (+0.001 [−0.06, +0.06]; −0.010 [−0.08, +0.05]). The paper's conclusion will be rewritten accordingly, and the contributions will be stated along the four axes 64JK named: navigation accuracy, output reliability, outcome prediction, and argument quality.

4. Remaining items

  • Attorney review (5QWJ Q1). Labels are model-drafted and human-adjudicated against the primary record; the blind second annotation is a second model pipeline; no human-only label exists today. A blind attorney review of a subsample is being arranged. We cannot commit to completing it in the discussion period.
  • Relevance penalty (5KPS Q1). Agreed as the next reward; design and the reason it was not in the submitted reward are in our reply to 5KPS.

5. Revision

Revised PDF by Nov 13, including the stripped-prompt judge evaluation and citation-swap appendix, corrected-key column in Table 1, dispute-overlap counts, posture-stripped 13-task holding-prediction column and Table 3 reclassification, rewritten abstract and conclusion, the related-work changes in our reply to 5KPS, and minor presentation changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.