acceptodds
Under review as a conference paper at ICLR 2027

Beyond Semantic Relevance: Learning Decision Authority over Retrieved Precedents

Abstract

Large language models increasingly use retrieved examples and cases to support prediction, but semantic relevance alone does not determine whether a past decision should transfer to a new query. This limitation is especially important in case-augmented reasoning, where retrieved examples carry historical decisions that can directly steer the current prediction. We formulate this problem as decision-authority learning: determining when a retrieved precedent is applicable enough for its prior decision to influence the current query. A key challenge is that outcome supervision cannot distinguish valid from invalid precedent transfer when both lead to the same correct label, a failure mode we call analogical reward aliasing. To address this, we construct contrastive intervention groups (CIGs) that approximately match retrieval relevance while varying applicability, and introduce Grounded Analogical Reward Optimization (GARO) together with its intervention-grouped variant, IG-GARO, to separate task correctness from precedent-transfer quality. A separate utility model controls when the trained specialist should replace Query-only. On natural similarity-matched COLD pairs, IG-GARO improves authority discrimination accuracy from 73.6% to 87.6%. The full system reaches 88.1 Macro-F1 while invoking the specialist on 19.7% of queries, and achieves the highest Macro-F1 point estimate across all evaluated moderation and legal settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Content Moderation, Case-Based Reasoning, Analogical Reasoning, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rBwbOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates decision-authority learning, which asks when a retrieved precedent's historical decision may transfer to a new query, and identifies analogical reward aliasing as the reason outcome supervision alone cannot teach that boundary. It constructs similarity-matched contrastive intervention groups and trains a specialist to predict USE, LIMIT, or REJECT using GARO and IG-GARO, which separate task correctness from groundedness, materiality, and structural-authority rewards, while a separate cross-fitted utility model decides when to invoke the specialist instead of Query-only. On COLD, authority discrimination accuracy improves from 73.6 to 87.6 and the full system reaches 88.1 Macro-F1 at a 19.7% invocation rate, with the highest Macro-F1 point estimates across all evaluated moderation and legal settings.

Critical strengths

1. A new formulation of decision transfer with direct evidence that the boundary is learnable The paper isolates a problem that retrieval similarity cannot settle, teaches a specialist to decide when a precedent's historical decision may transfer, and measures that decision directly through authority discrimination accuracy and Conflict F1 rather than through end-to-end labels alone. The targeted results are strong. Authority discrimination rises from 73.6 to 87.6, Conflict F1 reaches 85.3 against 79.3 for the strongest prior method, and the gains survive shortcut controls, including 84.7 accuracy when both precedents share a historical label and 85.9 with natural-only groups. Authority learning also adds 1.8 points under a fixed deployment gate, with a 95% CI of [+1.1, +2.5]. This matters because the central claim is precisely that applicability boundaries can be learned beyond similarity, and this evidence measures that boundary with controls against label copying and construction artifacts, making it the most defensible contribution and the main basis for acceptance.

2. A deployment-oriented system that turns a conditional capability into an efficient, stable one The paper diagnoses when precedent reasoning helps and hurts, with the specialist gaining 13.5 accuracy points under top-2 retrieval disagreement and losing 4.6 under agreement, and converts this into an end-to-end system through a disagreement screen and a cross-fitted utility model. The resulting system reaches 88.1 Macro-F1 while invoking the specialist on 19.7% of queries, with mean latency of 2.72 seconds against 5.95 seconds for always-on deployment, and the invocation rate is stable from out-of-fold data to the final test (20.4% to 19.7%). This matters because it shows the capability is usable under a real budget and that its selective behavior reflects the data rather than a threshold artifact, which distinguishes a deployable method from a diagnostic observation.

Critical weaknesses

1. Attribution confounded between authority learning and selective deployment The final system combines the authority-trained specialist with a separate gate and utility model, reaching 88.1 Macro-F1 on COLD while invoking the specialist on 19.7% of queries, and the closest CARO and CHAIRO baselines are evaluated at 100% invocation. With the same fixed specialist, routing alone lifts the always-on score from 82.4 to 88.1, and authority learning contributes 1.8 points under a fixed gate over Single-case GARO (CI [+1.1, +2.5]). The paper reports no control that applies the same gate and utility routing to CARO, CHAIRO, or Outcome-GRPO, and no Gate + Outcome-GRPO result, so the margin over the strongest baseline (84.1) cannot be fully attributed to decision-authority learning. Selective invocation, which could be layered onto other specialists, remains a candidate explanation. Always-on IG-GARO scores 82.4, below CARO and CHAIRO, so the method's value depends heavily on routing. This attribution issue bears directly on the value of the core contribution.

2. Cross-setting and cross-domain evidence weaker than claimed Four of the seven settings in Table 2 show a margin of at most 1.0 point over the best baseline (USClassActions +0.5, Ex-ToxiCN-MM +0.5, HED-COLD +0.9, Gemma +0.7). All seven settings report point estimates without confidence intervals or significance tests, and the two legal test sets contain only 600 and 500 examples. In the legal domains, authority learning alone does not exceed CARO or CHAIRO. Gate + IG-GARO is below them on USClassActions and level with them on CaseFacts, and the highest point estimates appear only after utility routing is added. The paper lists consistent end-to-end gains across moderation and legal settings as one of its core contributions, and the evidence supports a narrower claim of effectiveness on COLD-style moderation data with small, routing-dependent cross-domain gains. The cross-domain generalization claim is therefore not supported at the stated strength.

3. Same-family critic and teacher leave the process reward untested The specialist is Qwen3.5-4B, the groundedness and materiality scores are produced by a frozen Qwen3.8-27B critic, and the supervised fine-tuning traces and CIG relation signatures come from a Qwen3.5-35B-A3B teacher, so policy, critic, and teacher share one model family. If the critic systematically favors same-family outputs, part of the authority improvement may reflect alignment with the critic instead of transferable applicability judgments. A 300-trace human audit shows critic-annotator agreement (κ = 0.80 for groundedness, 0.72 for materiality), but no cross-family replacement control is reported and the paper does not discuss this coupling. The relation component of the process reward carries weight 0.4 and is scored by this critic, so the independence of the training signal is untested and the authority gains cannot be separated from same-family preference alignment.

Questions for authors

How much of the cross-setting advantage remains if the same gate and utility routing are applied to CARO, CHAIRO, or Outcome-GRPO, and what is the Gate + Outcome-GRPO result? Are the cross-setting gains, especially on the two legal tasks, statistically significant, and what is the minimum detectable effect at test sizes of 600 and 500? Do the authority discrimination accuracy and Conflict F1 results survive with a cross-family critic and teacher or human process labels? How are the routing features (label disagreement, entropy, margin) obtained without labeled precedents or under distribution shift, and when does the gate fail? Please add controls that apply the same routing to the baselines and a Gate + Outcome-GRPO ablation, with significance tests for cross-setting results. Repeat the pipeline with a cross-family critic and teacher or stronger human process annotation. Add a limitations section covering dependence on labeled retrieval archives, language and domain coverage, gate failure conditions, and misuse risk. Report routing stability under threshold calibration and distribution shift, together with its cost.

Reviewer_bT83OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a specialist to decide whether a retrieved case's decision applies to a new query. GARO rewards both applicability judgments and answer correctness; IG-GARO compares rollouts across contrasting precedent conditions. A separate router controls specialist invocation. Experiments on moderation and legal tasks report better predictions at lower inference cost than always-on deployment.

Critical strengths
  1. The moderation example makes the problem clear: endorsing and condemning a discriminatory statement can involve almost identical wording but require opposite labels.

  2. CIGs compare applicable and inapplicable precedents for the same query. The separate applicability reward can distinguish correct answers supported by different transfer judgments, providing supervision that outcome-only rewards lack.

  3. Evaluation on natural pairs tests applicability beyond the edited training examples. The audits separately check the constructed targets and the critic's groundedness/materiality scores.

Critical weaknesses
  1. Tables 1 and 9 do not seem to fit together. Under Eq. (12), most test queries keep their Query-only predictions because the retrieved labels agree. Given the accuracy reported for this group, reaching 88.5% overall accuracy requires the full system to answer about 95% of the remaining conflict queries correctly. Yet Table 1 reports a conflict Macro-F1 of 86.2%. The Query-only results on that same subset show that the classes are not imbalanced enough to explain this gap. Allowing for rounding and variation across the three seeds, the conflict Macro-F1 would still need to be at least about 91.8%. If both tables use the same subsets, Query-only checkpoints, and metric definitions, these results cannot all hold.

  2. The 84.7 to 86.5 Macro-F1 comparison under transparent gating supports IG-GARO over Single-case GARO. The full system's 88.1 also includes learned routing. I could not identify a corresponding Single-case GARO + utility result; Figure 3's "same deployment rule" does not specify one. There is a separate supervision issue with Flat Case-SFT: the paper does not establish that it receives the same applicability targets.

  3. Eq. (4) interprets the router score as an estimate of conditional expected utility. SmoothL1 in A.7 does not generally estimate that mean, so the sign of the fitted score need not match the sign of expected utility. Neither the transition parameter nor calibration evidence is given.

  4. A.6 already gives each candidate output a label. A.7 then selects a USE candidate, but does not say whether its label is returned or another answer is generated. LIMIT is described as providing context or constraints, yet its place in this procedure is unspecified.

    ADA and transfer-state accuracy test the applicability judgment. They do not directly test how the historical decision affects the final prediction.

  5. In Section 4.1, "preserving the label" could refer to the unchanged query or the edited precedent. The latter may need relabeling, and A.8 does not explain how this is handled.

    The same section treats similarity matching as making applicability the main source of variation. Close retrieval scores alone do not establish this.

  6. A.7 omits the SFT/RL learning rates, training steps, batch sizes, generation settings, and number of candidate precedents.

    Section 5 states that paired bootstrap confidence intervals are used, and Section 6.1 reports intervals for the gains. A.13 describes only seed means and standard deviations, leaving the resampling procedure undocumented.

Questions for authors

Q1:For each seed, please give the agreement/disagreement confusion matrices for Query-only and the full system, identifying the checkpoints and evaluation subsets.

Q2:Does Figure 3 include Single-case GARO with the learned utility router? Which supervision and routing settings are shared across the compared methods?

Q3:Do the two Gate systems share the Query-only checkpoint, retrieval results, margin calibration, and threshold, and what explains their 24.8% versus 23.2% call rates?

Q4:What SmoothL1 transition parameter is used, and what evidence supports interpreting a positive router score as positive expected utility?

Q5:Two worked examples would help with the method description: one controlled edit showing both labels before and after editing, and one inference trace showing candidate states, the selected precedent, and the returned label. Include where a LIMIT candidate contributes, if it does.

Reviewer_Rr7uOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies decision authority learning: determining when a retrieved precedent is sufficiently applicable for its historical decision to influence a new prediction. It introduces Contrastive Intervention Groups (CIGs) and GARO/IG-GARO to distinguish task correctness from valid precedent transfer, alongside a utility-based expert router. The proposed system reports improved authority discrimination and strong macro-F1 with selective expert invocation.

Critical strengths

Clear and practically meaningful problem. The paper clearly distinguishes semantic similarity from valid precedent transfer. Its formulation of analogical reward aliasing explains why rewarding final-answer correctness alone cannot distinguish sound analogies from invalid transfers that happen to produce the correct answer. This motivates a valuable research direction for case-augmented reasoning.

Critical weaknesses

Sources of the final gains are insufficiently isolated. The strongest results combine IG-GARO with the Utility router, whereas CARO and CHAIRO are primarily evaluated with always-on invocation. Without equipping these baselines with routers trained and evaluated under the same protocol, it remains unclear how much improvement comes from applicability learning versus selective combination of two models.

The necessity of the reinforcement learning design remains unproven. Missing controls include supervised training and applicability classification or ranking using the same CIGs, teacher data, and structured labels, with matched training budgets. Consequently, the gains may partly reflect better data, more explicit supervision, or additional computation rather than the IG-GARO objective itself.

Applicability discrimination does not establish control over final predictions. ADA primarily measures whether applicable precedents rank above inapplicable ones; it does not directly demonstrate that invalid precedents are prevented from influencing predictions. Behavioral tests involving precedent deletion, replacement, and historical-label perturbation are needed to substantiate the claimed decision authority control.

Questions for authors

Could the authors equip CARO and CHAIRO with utility routers trained and evaluated under the same protocol and budget, and report whether IG-GARO retains its advantage? This would help distinguish gains from applicability learning from those due to selective expert routing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.