- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formulates decision-authority learning, which asks when a retrieved precedent's historical decision may transfer to a new query, and identifies analogical reward aliasing as the reason outcome supervision alone cannot teach that boundary. It constructs similarity-matched contrastive intervention groups and trains a specialist to predict USE, LIMIT, or REJECT using GARO and IG-GARO, which separate task correctness from groundedness, materiality, and structural-authority rewards, while a separate cross-fitted utility model decides when to invoke the specialist instead of Query-only. On COLD, authority discrimination accuracy improves from 73.6 to 87.6 and the full system reaches 88.1 Macro-F1 at a 19.7% invocation rate, with the highest Macro-F1 point estimates across all evaluated moderation and legal settings.
Critical strengths
1. A new formulation of decision transfer with direct evidence that the boundary is learnable The paper isolates a problem that retrieval similarity cannot settle, teaches a specialist to decide when a precedent's historical decision may transfer, and measures that decision directly through authority discrimination accuracy and Conflict F1 rather than through end-to-end labels alone. The targeted results are strong. Authority discrimination rises from 73.6 to 87.6, Conflict F1 reaches 85.3 against 79.3 for the strongest prior method, and the gains survive shortcut controls, including 84.7 accuracy when both precedents share a historical label and 85.9 with natural-only groups. Authority learning also adds 1.8 points under a fixed deployment gate, with a 95% CI of [+1.1, +2.5]. This matters because the central claim is precisely that applicability boundaries can be learned beyond similarity, and this evidence measures that boundary with controls against label copying and construction artifacts, making it the most defensible contribution and the main basis for acceptance.
2. A deployment-oriented system that turns a conditional capability into an efficient, stable one The paper diagnoses when precedent reasoning helps and hurts, with the specialist gaining 13.5 accuracy points under top-2 retrieval disagreement and losing 4.6 under agreement, and converts this into an end-to-end system through a disagreement screen and a cross-fitted utility model. The resulting system reaches 88.1 Macro-F1 while invoking the specialist on 19.7% of queries, with mean latency of 2.72 seconds against 5.95 seconds for always-on deployment, and the invocation rate is stable from out-of-fold data to the final test (20.4% to 19.7%). This matters because it shows the capability is usable under a real budget and that its selective behavior reflects the data rather than a threshold artifact, which distinguishes a deployable method from a diagnostic observation.
Critical weaknesses
1. Attribution confounded between authority learning and selective deployment The final system combines the authority-trained specialist with a separate gate and utility model, reaching 88.1 Macro-F1 on COLD while invoking the specialist on 19.7% of queries, and the closest CARO and CHAIRO baselines are evaluated at 100% invocation. With the same fixed specialist, routing alone lifts the always-on score from 82.4 to 88.1, and authority learning contributes 1.8 points under a fixed gate over Single-case GARO (CI [+1.1, +2.5]). The paper reports no control that applies the same gate and utility routing to CARO, CHAIRO, or Outcome-GRPO, and no Gate + Outcome-GRPO result, so the margin over the strongest baseline (84.1) cannot be fully attributed to decision-authority learning. Selective invocation, which could be layered onto other specialists, remains a candidate explanation. Always-on IG-GARO scores 82.4, below CARO and CHAIRO, so the method's value depends heavily on routing. This attribution issue bears directly on the value of the core contribution.
2. Cross-setting and cross-domain evidence weaker than claimed Four of the seven settings in Table 2 show a margin of at most 1.0 point over the best baseline (USClassActions +0.5, Ex-ToxiCN-MM +0.5, HED-COLD +0.9, Gemma +0.7). All seven settings report point estimates without confidence intervals or significance tests, and the two legal test sets contain only 600 and 500 examples. In the legal domains, authority learning alone does not exceed CARO or CHAIRO. Gate + IG-GARO is below them on USClassActions and level with them on CaseFacts, and the highest point estimates appear only after utility routing is added. The paper lists consistent end-to-end gains across moderation and legal settings as one of its core contributions, and the evidence supports a narrower claim of effectiveness on COLD-style moderation data with small, routing-dependent cross-domain gains. The cross-domain generalization claim is therefore not supported at the stated strength.
3. Same-family critic and teacher leave the process reward untested The specialist is Qwen3.5-4B, the groundedness and materiality scores are produced by a frozen Qwen3.8-27B critic, and the supervised fine-tuning traces and CIG relation signatures come from a Qwen3.5-35B-A3B teacher, so policy, critic, and teacher share one model family. If the critic systematically favors same-family outputs, part of the authority improvement may reflect alignment with the critic instead of transferable applicability judgments. A 300-trace human audit shows critic-annotator agreement (κ = 0.80 for groundedness, 0.72 for materiality), but no cross-family replacement control is reported and the paper does not discuss this coupling. The relation component of the process reward carries weight 0.4 and is scored by this critic, so the independence of the training signal is untested and the authority gains cannot be separated from same-family preference alignment.
Questions for authors
How much of the cross-setting advantage remains if the same gate and utility routing are applied to CARO, CHAIRO, or Outcome-GRPO, and what is the Gate + Outcome-GRPO result? Are the cross-setting gains, especially on the two legal tasks, statistically significant, and what is the minimum detectable effect at test sizes of 600 and 500? Do the authority discrimination accuracy and Conflict F1 results survive with a cross-family critic and teacher or human process labels? How are the routing features (label disagreement, entropy, margin) obtained without labeled precedents or under distribution shift, and when does the gate fail? Please add controls that apply the same routing to the baselines and a Gate + Outcome-GRPO ablation, with significance tests for cross-setting results. Repeat the pipeline with a cross-family critic and teacher or stronger human process annotation. Add a limitations section covering dependence on labeled retrieval archives, language and domain coverage, gate failure conditions, and misuse risk. Report routing stability under threshold calibration and distribution shift, together with its cost.