acceptodds
Under review as a conference paper at ICLR 2027

RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications

17 views

Abstract

We study reasoning in **Rule-Governed Decision Tasks (RGDTs)**, where models apply external rules to case facts and justify decisions, as required in policy, contract, and compliance settings. Beyond the deductive capability emphasized by standard mathematical and logical reasoning tasks, RGDTs require interpreting rules and their applicability, assessing conditions from evidence, combining judgments under rules and exceptions, and providing checkable justifications. These demands motivate a new benchmark that assesses existing LLMs' reasoning abilities by evaluating both their decisions and their stated grounds. We introduce ***RGDT-Bench***, providing 202.1K condition-level supervision slots across four task tracks and eight supported task-probe combinations that vary access to supporting information. Label-blind extraction and deterministic checks produce labels for *warrant completeness*: source-referenced coverage and consistency of stated decision grounds. The benchmark attributes failures to four process layers: rule use, condition, evidence, and aggregation, and checks the final outcome. Among evaluable correct responses, warrant incompleteness averages 40.2% across six evaluated LLMs and supported task-probe combinations. Such warrant incompleteness poses potential safety risks and remains difficult to detect: the best of seventeen existing evaluators reaches only 57.69% (random: 50%) task-averaged area under the receiver operating characteristic curve (AUROC). To address this difficulty, we train a simple reward model with warrant supervision. It achieves 69.24% task-averaged AUROC among correct answers, exceeding the matched outcome-supervised baseline by 10.37 pp (percentage points) and the best existing evaluator by 11.55 pp. Beyond completeness assessment, the model outperforms both outcome-supervised baselines across nearly all response-selection comparisons, supporting ***RGDT-Bench***'s warrant supervision for RGDT reasoning.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Reasoning Evaluation, Benchmarking, Rule-Governed Decision Tasks, Warrant Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MddnOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces RGDT-Bench to evaluate LLM reasoning completeness across four dimensions: rule use, condition assessment, evidence support, and decision aggregation. Experiments reveal that 40.2% of correct responses have incomplete warrants, which existing evaluators struggle to detect. A warrant-supervised reward model substantially improves completeness assessment and downstream response selection.

Critical strengths
  1. The paper addresses an important limitation of outcome-based evaluation, showing that correct answers can conceal incomplete or unsupported reasoning.
  2. RGDT-Bench provides fine-grained reasoning diagnostics across four process layers and multiple task domains, enabling systematic analysis of reasoning failures.
  3. Extensive experiments with 17 existing evaluators demonstrate the difficulty of assessing warrant completeness. Warrant supervision improves AUROC by 10.37 percentage points over the matched outcome-supervised baseline among correct answers.
Critical weaknesses
  1. The connection between warrant completeness and decision reliability remains unclear. The paper shows that correct answers can contain incomplete warrants, but does not establish whether improving warrant completeness leads to more reliable decisions. The downstream evaluation is limited to Best-of-N selection on the same benchmark tasks, without validation in independent decision-making scenarios.
  2. The benchmark has limited case diversity. Despite collecting 86,784 model responses, RGDT-Bench contains only 452 original cases, including 90 test cases (Table 6). This raises concerns about whether the findings generalize to more diverse rules, documents, and decision scenarios.
  3. The reliability of automatic warrant annotations remains a concern. Warrant labels depend on LLM-based claim extraction, yet only 360 responses are manually validated, with a Krippendorff's α of 0.687. Potential annotation errors may propagate into both benchmark evaluation and reward-model supervision.
  4. The practical benefits of warrant supervision are limited. Despite substantial improvements in AUROC, Warrant-RM achieves only a 1.23 percentage-point gain in warrant completeness and a 0.46 percentage-point gain in answer accuracy over Full-response ORM in the Best-of-16 evaluation across all response groups (Table 4). These modest gains leave the practical value of warrant supervision insufficiently demonstrated.

Overall, the paper systematically quantifies incomplete warrants in correct answers, but their practical significance remains insufficiently established. Although the proposed framework identifies reasoning deficiencies and warrant supervision improves response selection, it remains unclear to what extent warrant incompleteness reflects consequential reasoning failures or predicts decision reliability. Moreover, the modest gains in Best-of-N selection provide limited evidence of substantial downstream benefits. Thus, the connection between warrant completeness and reliable decision-making remains insufficiently validated.

Questions for authors

Please refer to Weaknesses.

9 comments
AuthorsOpenReview ↗

Opening response

To test the practical value of warrant supervision, we test transfer of the learned scoring signal from pointwise assessment to pairwise comparison and listwise selection on held-out RGDT tasks. M1 connects concrete reasoning failures to assessment, M2 explains case coverage and split design, M3 checks the findings through balanced human review, and M4 examines ranking, candidate retention and final selection. Together, these tests connect the supervision signal to successive decisions in a response-review workflow.

AuthorsOpenReview ↗

M1. Checking Explanations and Comparing Candidates Reveal the Value of Warrant Supervision

Response

To connect warrant completeness to practical review, we test whether the same learned signal helps assess one explanation, compare alternatives and retain or select candidates. Thank you for asking us to clarify this connection. Sections 5.4–5.5 now follow this pointwise → pairwise → listwise progression with the same fitted models.

1. Pointwise: identifying reasoning errors that outcome checks miss

Consider the Regulation example in Appendix F.6. The evidence reports a leg injury but says nothing about discharge from military service. The response infers “He lost a leg, likely discharged,” yet returns the correct answer because another eligibility condition remains unresolved. Checking only the final answer misses this unsupported judgment. Warrant supervision makes such stated grounds available for checking. Section 3 defines the coverage target, and Appendix F.6 illustrates rule-scope, evidence and decision-consistency failures across the four tasks.

A reviewer deciding whether to accept an explanation therefore needs a signal about its grounds. To isolate the supervision target, Section 5.4 matches the backbone, data, weights and training budget (Appendix D.3). We test discrimination across all responses, then hold outcome correctness fixed in the correct-only view to ask whether the score still distinguishes complete from incomplete grounds. Table 4 shows the advantage in both views.

Completeness AUROC (%)Final-only ORMFull-response ORMWarrant-RMGain over Final / Full (pp)
Full67.3172.9979.93+12.62 / +6.94
Correct-only51.8658.8769.24+17.38 / +10.37

The gains also extend to AUPRC and balanced accuracy. Relative to Full-response ORM, these improve by 10.16 and 3.13 pp in the full view, and 9.59 and 5.00 pp among correct answers. Because Warrant-RM and Full-response ORM receive the same explanation, the advantage identifies the value of warrant supervision even when the final answer is already correct.

2. Pairwise and listwise: using the assessment signal to choose responses

Because assessment and selection place different demands on a score [1,2], we test transfer explicitly with the same models. Warrant-ranking AUROC improves by 7.486/3.799 pp over Final-only/Full-response ORM, with statistical support for both comparisons. Outcome ranking has positive mean gains of 4.694/0.472 pp (Section 5.5, Table 5).

When review capacity is limited, retaining k=1, 2, 4 or 8 of 16 candidates increases mean warrant precision by 1.685–4.405 pp over Final-only ORM and 0.866–1.265 pp over Full-response ORM. Gains are positive at every evaluated k<16 (Table 6), supporting prioritization of explanations for review.

Returning only the highest-scored answer is a more demanding transfer test. In Best-of-16 opportunity pools, completeness/accuracy gains are +11.02/+8.11 pp over Final-only ORM and +2.18/+1.66 pp over Full-response ORM. Across all groups, the corresponding gains remain +4.40/+1.69 and +1.23/+0.46 pp (Table 7). Opportunity pools contain both target labels. The all-group view also includes uniform-label pools, where selection leaves the target unchanged. These differences use the displayed means.

These results connect the assessment signal to improved candidate prioritization and positive mean final-selection gains on held-out RGDT tasks. M4 presents the full ranking and selection tables. Further source collections and decision environments would extend this validation.

Conclusion: On held-out RGDT tasks, warrant supervision connects identifiable reasoning errors to better assessment, comparison and candidate retention.

References

[1] Zhou, E., et al. (2025). RMB: Comprehensively Benchmarking Reward Models in LLM Alignment. ICLR.

[2] Kim, S., et al. (2025). Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization. ACL, pp. 13252–13280.

AuthorsOpenReview ↗

M2. Four Distinct Rule Settings Enable Controlled Tests of Reasoning and Supervision

Response

To make reasoning failures verifiable, we organize the benchmark around four distinct decision settings, each with source-grounded requirements for its judgments, evidence and decision logic. Thank you for raising case diversity. Sections 3–4 and Appendix B now clarify how the sources and controlled comparisons support the findings.

1. Grounding each decision makes the reasoning question testable

Our motivating question is whether an answer's stated grounds are complete and supported even when its final decision is correct. Answer labels alone cannot resolve this question. Each case therefore needs a reference warrant specifying the required judgments, their supporting rules and evidence, and how they combine into a decision (Section 3.2). This structure makes it possible to locate an omitted condition, an unsupported inference or an incorrect aggregation, and then turn those diagnoses into supervision. This design makes variation in the governing rules and required reasoning observable at the source-case level. The probes and generated responses then allow controlled comparisons within those cases.

2. The four tracks vary the rules, evidence and decisions being assessed

The task coverage extends from applying policies to questions, through inference from contracts and administrative rules, to checking basketball transactions against a rule manual. These settings require different judgments, support and decision logic. Sections 3–4 and Appendix C.2 specify how the common evaluation framework preserves these task-specific requirements.

Track and sourceRule and evidence settingDistinct reasoning demand
Policy / ConditionalQAPolicy rules and question-specific evidenceEstablish whether the supplied grounds justify the answer to the policy question
Contract / ContractNLIContract clauses and annotated supportJudge multiple inference conditions, use required or accepted alternative evidence, and combine the judgments
Regulation / ShARCAdministrative rules and case evidenceResolve required conditions, retain unresolved judgments, and apply conjunctions or alternatives in the decision logic
Transaction / RuleArena–NBATransaction facts and a basketball rule manualIdentify the proposed action, apply an accepted governing rule and justify overall compliance

These differences matter because a score that recognizes a well-supported policy answer must also handle clause-level evidence, unresolved regulatory conditions and action-specific compliance. The shared representation makes these distinct demands comparable while preserving their source-specific requirements.

3. Repeated cases isolate support-access and model effects

The probes ask how reasoning changes when the same decision requires locating support in different contexts. P1 supplies matched support, P2 introduces confusable passages, and P3 retains longer natural document context. All four tracks support P1. Contract and Transaction additionally support P2/P3 because their documents contain the relevant distractors and surrounding material. The construction preserves the facts, required support and correct decision (Section 4.1 and Appendix B.3). Comparing six generators on these shared inputs then separates model differences from changes in the underlying problem. Repeated responses provide the candidates needed to test whether warrant supervision improves assessment, comparison and selection. This connects support-access diagnosis to downstream evaluation.

4. Source-level separation tests the signal on held-out cases

To prevent related cases from leaking across splits, cases sharing documents, related operations or similar content are grouped first, with every probe and response kept in its group's split (Appendix B.1). Test data remain separate from fitting and model selection. To account for dependence among responses, uncertainty analyses retain these groups as the resampling units (Appendix D.1). The revised audit supplies a separate reliability check across every test case, with balanced task and generator coverage (Appendix C.3).

Expanding to additional policy collections, contracts and rule manuals is the next step toward broader source-level generalization, using the same reference-grounded framework.

Conclusion: Across four task families and controlled support settings, held-out comparisons connect identifiable reasoning failures to useful supervision.

Reviewer_hNJGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses not only on whether LLMs get the final answer right, but also on what they explicitly take into account to reach it, in contexts where the reasons and the justification matter (they use four tracks: policy, regulation, contracts, and transactions). The main goal is to assess the completeness of the justification (warrant) behind each response and to identify which elements or conditions are missing. The authors also test whether existing evaluators can detect incomplete justifications, and train their own reward model to improve on them. The main finding is that 40.2% of correct final answers have incomplete warrants across the evaluated models, and that existing evaluators struggle to detect this.

Critical strengths

I think that the idea of separating the final answer from its justification is valuable, especially in the kind of settings chosen by the authors where the reasons matter as much as the decision itself. I like the structured way in which they approach it, breaking the justification into its components (rule, evidence, the integration of them), and making the assessment transparent by pinpointing where a justification fails. They also add some components that enrich the evaluation: organizing cases into three input settings to test whether models can identify and rely only on the relevant information, and comparing various evaluators (LLM judges and reward models). This matters for my assessment because it turns a valid concern (correct answers for the wrong reasons) into something that can be measured and diagnosed in detail, which is a useful contribution.

Critical weaknesses

The results depend on how strict the definition of a complete explanation is, and I am not convinced that definition or criteria is the right one, especially if LLMs are not asked for exhaustive explanations. If I understood the example of Figure 11 correctly, the justification is marked incomplete if it fails to consider the three conditions in the integration, even when the rule makes the two omitted conditions irrelevant. The example is presented in a way that is difficult to follow, there is only one across the four task tracks that the paper covers and it is in the Appendix, which makes all the pipeline difficult to understand in concrete. Regarding validation, the human review only compares automatic labels with human judgments. I would have liked to see agreement between independent human annotators as well, since this would show whether humans actually agree on what a complete justification requires, which is the foundation of the whole benchmark.

Questions for authors
  1. Could you provide examples from the other task tracks, especially examples of failures?
  2. Could you provide examples in which the LLM evaluator judges a justification as complete when it is actually incomplete according to the reference?
  3. Were the humans who validated the responses experts in each domain?
9 comments
AuthorsOpenReview ↗

Opening response

A benchmark for checkable decisions needs a clear criterion that independent reviewers can apply consistently. H1 tests this through explicit instructions, expert agreement and human labels. H2 provides four-task examples, H3 examines judge errors, and H4 details the audit. Sections 5.4–5.5 further test the learned score in pointwise assessment, pairwise ranking and listwise selection on held-out RGDT tasks.

AuthorsOpenReview ↗

H1. Explicit Coverage and Balanced Expert Review Support the Main Findings

Response

To make the criterion useful for reviewing decisions, we clarify what it checks and test the findings under explicit instructions and human labels. Thank you for motivating these checks.

1. Reference coverage serves a review purpose

The original example left room for ambiguity between decision sufficiency and reference coverage. Section 3, Appendix A and revised Figure 12 now distinguish why a contradicted Contract condition suffices for rejection from which reference-required aggregation grounds remain unstated. Concise paraphrases and combined statements count as coverage. H2 and Appendix F provide four-task examples and the worked pipeline.

CWG measures warrant incompleteness among evaluable correct responses. Ignoring aggregation checks reduces it only from 40.2% to 35.7%, preserving the main pattern (Section 5.1, Table 1). This tests sensitivity to the assessment criterion, complementing the O/F/S instruction test.

2. Explicit instructions test whether coverage requests explain the gap

To distinguish the effect of requesting coverage from that of organizing the response, we compare three matched settings. O retains the original instruction. F requests all required conditions, rules, evidence and their role in the decision in free prose. S requests the same grounds in structured fields. Models identify the conditions themselves, with task inputs and generation settings fixed (Section 5.1, Table 2 and Appendix G).

InstructionCWG ↓Change from O (pp)
O: original40.08%—
F: explicit, free prose38.09%−1.99
S: explicit, structured38.81%−1.27

F and S each reduce CWG for five of six generators (mean reductions of only 1.99 and 1.27 percentage points, respectively), yet CWG remains close to 39%. Making the coverage requirement explicit modestly reduces the measured warrant gap, which remains substantial under aligned instructions. This matched cohort differs from the cohort underlying the original 40.2% result.

3. Independent expert agreement and human labels check the findings

The original audit sampled the full response collection randomly without task or generator balance. We therefore reviewed 720 test responses with balanced margins and coverage of every test case. To test whether experts apply the criterion consistently, three senior legal professionals reviewed each response independently. All three agree on 657/720 responses (91.25%), with nominal Krippendorff’s α = 0.8830 across their individual labels (details in H4). Automatic–human agreement is 627/720 (87.08%), with Krippendorff’s α and Cohen’s κ both 0.739 (Table 14).

To isolate sensitivity to annotation differences, we replace automatic labels with human labels while fixing the trained models, responses and evaluation weights, including all disagreements (Table 15). Gap = human-label CWG − automatic-label CWG.

TaskAutomatic CWGHuman CWGGap (pp)
Transaction35.63%37.39%+1.76
Policy42.96%41.87%−1.09
Contract30.62%28.93%−1.68
Regulation33.33%33.07%−0.26

Human review preserves both the task ordering and the substantial gap among correct answers. CWG changes by less than 2 pp in every task and remains 28.93–41.87%, supporting the finding beyond the automatic labels.

To check whether the learned signal also matches expert judgments, we compare WRM with both outcome-supervised baselines under human labels, across all responses and with answer correctness held fixed (Section 5.4, Table 16):

Human-label AUROCFinal-only ORMFull-response ORMWRM
Full65.06%69.94%78.19%
Correct-only47.32%53.08%66.84%

The gains over Full-response ORM are 8.25/13.77 pp, and over Final-only ORM 13.13/19.52 pp. The learned assessment advantage therefore persists when independent human judgments determine the evaluation target.

4. Stronger checks support the central findings

We took your concerns seriously and updated the experiments and manuscript through these more rigorous checks.

Conclusion: Independent expert agreement supports consistent application of the criterion, while explicit instructions and human-label validation preserve the warrant-gap and assessment findings.

AuthorsOpenReview ↗

H2. Four Task Examples Make the Reasoning Failures Concrete

Response

We added examples from all four tasks so that readers can follow each diagnosis from the supplied facts to the model's statement and final decision. Thank you for requesting this. Each example connects the diagnosis to the governing rule, available evidence and actual model response.

1. Four task examples connect facts, responses and diagnoses

Appendix F.5 now presents the task-specific prompt templates, and Appendix F.6 pairs actual test responses with the facts and rules needed to understand their failures. The examples cover distinct review situations: interpreting the scope of a contract obligation, checking the applicability of a transaction rule, assessing an eligibility condition, and ensuring that the returned decision agrees with the explanation.

Task and review situationSupplied facts or ruleWhat the response doesWhat the diagnosis reveals
Contract: confidentiality scopeCommercial secrets and other information designated confidential are protected as separate categories.Extends the designation requirement to both categories, then returns insufficient evidence instead of negative.A rule-scope error changes the assessment of a stated condition.
Transaction: sign-and-trade eligibilityThe facts give Team B's salary, while Team A's salary is unspecified.States “we can assume Team A's team salary is less than cap” and returns positive instead of negative.An unsupported premise is used to determine rule applicability.
Regulation: program eligibilityInjury is documented, while discharge status is unspecified.Infers “He lost a leg, likely discharged,” while correctly returning insufficient evidence because another requirement remains unresolved.A correct answer coexists with an unsupported condition judgment.
Policy: VAT recoveryThe computer is exclusively for private use, for which the rule disallows VAT recovery.The explanation concludes that the answer should be negative, but the submitted final label is positive.The final decision contradicts the model's own stated reasoning.

2. What the examples explain about reasoning failures

These cases expose errors in rule scope, evidence and decision consistency that a reviewer can locate directly. In particular, the Regulation example explains why checking the final answer alone can overlook a reasoning failure: the unresolved treatment condition leads to the correct insufficient-evidence answer, even though the discharge judgment is unsupported. This example complements the Contract workflow in Figure 12, which explains the separate issue of reference-required coverage in an otherwise valid short-circuit argument.

Conclusion: The four-task examples make the diagnoses directly inspectable and distinguish missing reference coverage from concrete errors in rules, evidence and decision consistency.

Reviewer_A8ESOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces RGDT-Bench, a benchmark for evaluating LLM reasoning in rule-governed decision tasks, where models must apply rules to case facts and justify their decisions. The benchmark evaluates reasoning completeness across four layers: rule use, condition assessment, evidence support, and decision aggregation, in addition to final-answer correctness. Experiments across six LLMs show that many correct answers still have incomplete justifications, while existing evaluators struggle to identify these issues. To address this limitation, the authors train a warrant-supervised reward model (Warrant-RM), which outperforms outcome-supervised baselines in assessing warrant completeness.

Critical strengths

The paper addresses an interesting and practically important problem. This is particularly relevant to applying LLMs to rule-governed decision tasks, where the reasoning process can matter as much as the final decision. I also appreciate the transparency of the methodology and the effort to construct a benchmark spanning multiple task domains, with fine-grained evaluations of different aspects of reasoning.

Critical weaknesses

The definition of warrant completeness appears overly restrictive and is not sufficiently justified. As explicitly stated in Appendix A, a response can be labeled incomplete even when fewer stated grounds are logically sufficient to support the correct decision. This criterion may conflate missing information with genuine reasoning failures and penalize concise but valid explanations. Moreover, Appendix C.2 acknowledges that generation prompts do not uniformly request exhaustive coverage. Therefore, the measured warrant gaps may not accurately reflect weaknesses in LLM reasoning, but may instead partially reflect a mismatch between the benchmark's completeness requirements and the generation instructions.

This concern also extends to the evaluation of existing reasoning judges. The evaluation criteria used for existing LLM judges are not fully aligned with the benchmark's completeness criterion. Appendix D explicitly states that the LLM-judge rubric permits harmless omissions, whereas the proposed warrant verifier requires coverage of all reference-required conditions. Therefore, the relatively poor performance of existing evaluators may partially result from a mismatch in evaluation objectives. Similarly, the improvement of Warrant-RM over outcome-supervised models primarily demonstrates better prediction of the proposed warrant labels, rather than necessarily better identification of practically consequential reasoning failures. Although the authors also report improvements in downstream response selection, the gains in final-answer accuracy are relatively small.

Questions for authors
  1. Appendix C.3 reports 84.2% agreement between automatic annotations and human judgments. Given that both the benchmark analysis and reward model training rely on these automatically generated labels, how sensitive are the main findings to annotation errors? Additionally, do the reported warrant gaps and evaluator improvements remain consistent when assessed using human-verified labels?

2 In Table 4, Warrant-RM improves answer accuracy over Full-response ORM by only 0.46 percentage points at (N=16) in the all-response-group setting. Given the uncertainty estimates in Figure 8, this gain appears marginal. Are the improvements consistent across training runs and statistically distinguishable from zero? How should the practical significance of these relatively small accuracy gains be interpreted?

8 comments
AuthorsOpenReview ↗

Opening response

To test whether warrant supervision supports response assessment and selection, we strengthen its validation and examine transfer of the same learned score across evaluation objectives on held-out RGDT tasks. A1 tests explicit coverage instructions, A2 aligns the judge rubric, and A3 checks the findings with balanced human review. A4 follows the score from assessing one explanation (pointwise), through comparing alternatives (pairwise), to retaining and selecting candidates (listwise), connecting the validation to practical use.

AuthorsOpenReview ↗

A1. Explicit Coverage Instructions Preserve the Warrant-Gap Finding

Response

Substantial warrant gaps persist when models are explicitly asked to provide the assessed grounds. Thank you for highlighting the need to clarify how generation instructions relate to the coverage criterion and test this relationship directly. We clarified the criterion in Section 3 and Appendix A and added the matched experiment to Section 5.1 and Appendix G.

1. Separate decision sufficiency from reviewable coverage.

A decisive condition may settle the answer, while a reviewer also needs to check the other judgments required by the task and their connection to the rule. RGDT-Bench measures this source-referenced coverage and consistency, accepting concise paraphrases and combined statements. The original presentation left room for ambiguity between decision sufficiency and reference coverage. We revised Figure 12 to distinguish why the decision follows from C3 from which reference-required aggregation grounds remain unstated. The extracted claims now appear beside the corresponding checks.

CWG measures warrant incompleteness among evaluable correct responses.

Ignoring aggregation checks reduces mean CWG only from 40.2% to 35.7%, preserving the main pattern (Section 5.1, Table 1). This tests sensitivity to the assessment criterion, complementing the O/F/S instruction test.

Appendix F.6 also illustrates a concrete error: a Regulation response infers discharge from military service from a leg injury, although discharge is unspecified. Another unresolved condition leads to the correct final answer. Checking the grounds thus reveals an unsupported judgment that answer correctness misses.

2. Test explicit coverage with matched generation.

To test whether coverage requests or reporting format explain the gap, O retains the original instruction. F requests every required condition, rule, supporting evidence and contribution to the decision in free prose. S requests the same grounds in organized fields. Models still identify the relevant conditions themselves. To isolate these instruction changes, we fix task inputs and generation settings across six generators, with four responses per prompt in each setting.

Generation settingHow the required grounds are requestedCWG ↓Change from O (pp)Answer accuracy
O: originalOriginal instruction40.08%—72.74%
F: explicit free proseFull required coverage, freely organized38.09%−1.9972.03%
S: explicit structured reportingSame coverage, organized into fields38.81%−1.2773.09%

3. Interpret the controlled comparison.

F and S lower CWG for five of six generators (mean reductions of only 1.99 and 1.27 percentage points, respectively), showing a modest reduction in the measured warrant gap under explicit coverage instructions. Substantial gaps remain under both reporting formats, supporting the need to assess and supervise expressed warrants even when coverage is requested explicitly. Table 2 reports the changes, Table 26 gives absolute results, and Figure 20 provides the instructions. This new matched cohort explains the small difference between its O baseline and the original 40.2% result. We took your concern seriously and updated both the experimental evidence and the assessment description.

Conclusion: Substantial warrant gaps persist under explicit coverage requests, supporting assessment and supervision of the grounds stated in correct answers.

AuthorsOpenReview ↗

A2. An Explicit, Aligned Rubric Preserves the Assessment Advantage

Response

An explicit, aligned rubric preserves Warrant-RM's assessment advantage across both views. Thank you for highlighting ambiguity in how the assessment criterion was presented. To make the coverage requirements explicit, we refined the rubric and re-evaluated all fifteen LLM judges.

1. State what a reviewer needs to check.

To assess whether an explanation supplies its decision grounds, the rubric explicitly checks condition judgments, evidence support, rule application, aggregation, and outcome correctness and consistency. It accepts concise or combined statements and explains unresolved conditions and required findings that do not change the final decision. Judges receive the visible task and response, estimate the probability that all requirements hold, and explain their assessment. Reference annotations remain separate. Section 5.3 explains the design, and Appendix F.4/Figure 13 gives the wording.

2. Test whether recent model advances narrow the assessment gap.

To examine whether recent advances in model capability translate into better warrant assessment, we expanded the judge panel with GPT-6.1 Sol and Gemini-3.8-Flash. Three native reward models retain their original scoring interfaces. Resource constraints delayed the GLM-5.2 rerun, whose results will be added at camera-ready once complete. To separate general assessment from recognizing missing grounds in already correct answers, we evaluate both the full response set and a correct-only view that fixes answer correctness.

To measure sensitivity to the refined rubric, the table compares each judge on the same responses. Δ = aligned − original rubric, in percentage points. AUC = AUROC, PR = AUPRC, BA = balanced accuracy. Absolute results are in Table 3.

LLM judgeFull AUCFull PRFull BACorrect AUCCorrect PRCorrect BA
GPT-5.6 Sol-0.57-0.81-1.65+1.18-0.01-0.25
GPT-6.1 Sol-0.51+0.02-5.15+1.56+1.03-0.91
Gemini-2.5-Pro-1.45-0.85-1.54-0.51-0.26-0.41
Gemini-3.8-Flash+0.59+0.12-0.86+1.64+0.57+0.30
Qwen3.5-0.8B+1.94+1.46-0.05+0.58+0.25+0.67
Qwen3.5-4B-1.18-0.31+0.35-1.93-0.49+0.82
Qwen3.5-9B-1.82-1.24-0.59-1.95-0.70-0.27
Qwen3.5-27B-1.49-1.25-0.24-0.53-0.11+0.31
Qwen3.5-35B-A3B-3.17-2.00-1.04-4.12-1.69-0.35
gpt-oss-20B+1.65+3.51-0.71+2.10+3.03-0.01
gpt-oss-120B-2.80-2.28-0.40-4.39-2.54-0.15
DeepSeek-R1-Distill-Llama-8B+0.47+0.01+0.45+0.78+0.07+0.40
DeepSeek-R1-Distill-Llama-70B+1.22+1.02-0.35+1.78+1.15-0.65
Llama-3.1-8B-Instruct-0.84-0.39-3.13-0.93-0.05-3.34
Llama-3.1-70B-Instruct-1.10-0.60+0.49-1.84-0.76+0.03
Max. absolute Δ (pp)3.173.515.154.393.033.34

3. Connect the result to the supervision contribution.

Rubric changes move judges in both directions. Thirteen full-view and twelve correct-only AUROC changes are within 2 pp. Under the refined rubric, Warrant-RM reaches 79.93%/69.24% AUROC in full/correct-only views, versus 70.54%/57.26% for the best external scorers and 72.99%/58.87% for matched Full-response ORM (Tables 3–4). Its gains are +9.38/+11.98 pp over the best external scorers and +6.94/+10.37 pp over the matched ORM. The correct-only external comparison has a paired jackknife 95% interval of [4.68, 19.29] pp (Section 5.3).

Recognizing missing grounds in correct answers remains difficult under the explicit coverage rubric, supporting the need for warrant supervision. Human-label evaluation retains a 13.77 pp correct-only advantage over Full-response ORM (A3). The score also improves same-question warrant ranking by 3.799 pp and top-four warrant completeness by 1.265 pp over that baseline (A4, Section 5.5), connecting assessment to comparing explanations and retaining candidates for review.

Conclusion: The refined rubric preserves Warrant-RM's advantage in both views, supporting warrant supervision as an assessment signal beyond outcome supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.