acceptodds
Under review as a conference paper at ICLR 2027

Training LLM Judges from Language Feedback via Position-Selective Self-Distillation

Abstract

We study training LLM judges from natural language feedback, especially for subjective tasks where the verdict depends strongly on which evaluation criteria the judge invokes and how it weighs them. The dominant approach, outcome-supervised RL (e.g., GRPO), credits every token in the rollout with a single scalar determined only by the accuracy of the final verdict, providing no separate credit at the criterion-choice tokens and ignoring the rich language feedback (e.g., preference rationales) that naturally accompanies preference labels. Self-Distillation (SD) is one natural way to use this language feedback: the same model, conditioned on this feedback, acts as a teacher providing dense, position-level supervision. However, not all positions carry equally useful signal. Using the per-position entropy shift between teacher and student, we identify two regimes: context sharpening, where the teacher concentrates probability on a particular feedback-aligned criterion expression, and context spreading, where the teacher distributes probability across multiple feedback-aligned alternatives. We interpret these patterns as follows: sharpening encourages memorization of a particular criterion expression, whereas spreading promotes semantic understanding by preserving these alternatives. Motivated by this asymmetry, we introduce position masking based on the entropy shift that retains the lower tail of the entropy-shift distribution. Experiments show that masking higher-entropy-shift positions improves out-of-distribution generalization over naive SD. The resulting self-distilled judges outperform judges trained with outcome-supervised RL by 2-9 percentage points on the evaluated subjective subcategories, while remaining competitive on objective ones.

Primary area:
reinforcement learning
Keywords:
learning from language feedback, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gyMsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper trains pairwise LLM judges from preference data in which each label comes with a short human rationale. Instead of rewarding only the final verdict, as Dr. GRPO does, the authors use on-policy self-distillation (SD): the same model, conditioned on the rationale, acts as a teacher at every position of the student's rollout. They measure the per-position entropy shift Delta H_t between teacher and student, and call large positive shifts "context sharpening" and large negative shifts "context spreading". Their interpretation is that sharpening encourages memorization of specific criterion wording, while spreading supports semantic understanding. On this basis, SD+mask discards the 70% of positions with the largest Delta H_t in each rollout. With Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507 trained on HelpSteer3-Preference, SD+mask improves RM-Bench and RewardBench V2 over naive SD (no masking) and Dr. GRPO. The paper reports gains of 2 to 9 points over Dr. GRPO on the Chat, Factuality and Focus subcategories, which it calls subjective. SD+mask also gives higher win rates than Dr. GRPO and naive SD when its 30B judge selects responses in a best-of-8 tournament on Arena-Hard-v2 creative writing.

Critical strengths

The paper separates criterion choice from the rigor with which criteria are applied. This explains why a verdict-only reward gives little signal on the tokens that name criteria, and why that should matter more for subjective judging. The analysis of where the distillation signal sits is also informative: per-position reverse KL is U-shaped in Delta H_t, and the tokens that name and define criteria are 16.7% of response tokens but 44.4% of the KL mass.

The selector ablations are the strongest evidence in the paper that the proposed rule beats the alternatives, subject to W1 and Q2 below. Five alternatives are compared at matched rho on both model sizes, including a direction flip (keeping only the highest-Delta H_t positions) and random masking, and the proposed selector is best in both. At 30B it is 2.8 points of Total Avg. ahead of the best alternative.

The data processing is careful: duplicates and train/validation leakage in HelpSteer3-Preference are removed. The appendix also tests a second kind of feedback.

Critical weaknesses

W1. The main comparisons appear to rest on single runs. Appendix E lists six main training runs (two models, three methods), and I could not find seeds or variances. Evaluation uses one sample per prompt at temperature 0.7. Several comparisons depend on small differences. In Table 3, Total Avg. at 30B is 80.17, 80.63 and 80.68 for rho in {0, 0.3, 0.5}. Naive SD and Dr. GRPO are also only 0.14 apart at 30B (80.17 and 80.31). Some RewardBench V2 subsets are small: Precise IF has 160 prompts, so one prompt is 0.625 points. Section 3.2 highlights a Precise IF gain from 40.62 to 48.75 (naive SD to SD+mask at 30B). I cannot tell whether differences of this size exceed training and sampling noise. Could the authors report means and standard deviations over several seeds? Given the stated cost of roughly one day per run, the 4B setting alone would be informative. Confidence intervals over evaluation prompts would help too.

W2. SD sees the preference rationale and Dr. GRPO does not, so the 2 to 9 point gains on the subcategories the paper calls subjective could come from the extra information rather than from distillation. There is no controlled baseline that uses the rationales without distillation, for example the RL methods with natural-language critiques described in Appendix A (Critique-GRPO, Text2Grad). Table 1 also has no rows for the untrained Qwen3 judges, so one cannot see how much each method gains over its starting point on RewardBench V2 or on the RM-Bench subcategories (Figure 1 gives only RM-Bench Overall). I would like to see the untrained rows and such a baseline, or an explanation of why none applies.

W3. Appendix E gives Dr. GRPO a learning rate of 1e-6 at 4B and 2e-6 at 30B with 8 rollouts per prompt, and gives naive SD and SD+mask 5e-6 and 1e-5 (five times larger) with 4 rollouts. How were these values chosen? Apart from the rho sweep for SD+mask, the only tuning I could find is for the prompt template (Section 3.1). In Figure 1, the smoothed 4B Dr. GRPO curve stays within about one point of its starting accuracy (roughly 81.7 to 82.6 on RM-Bench). I cannot tell whether the baseline is well tuned. If Dr. GRPO was not tuned, please add a small learning-rate sweep for it.

W4. I could not find a direct test of the memorization-versus-understanding explanation that motivates masking (Abstract, Sections 2.3, 2.4 and 3.2). Section 2.3 reports that the teacher's candidate criterion names are closer in embedding space to the decisive criterion extracted from the rationale than when the rationale comes from another example (sharpening 0.713 vs 0.647, spreading 0.646 vs 0.610). This shows that the teacher uses the rationale, which is the expected effect of conditioning on it, not that sharpening leads to memorization or spreading to understanding. For spreading, the score is a mean over the top-10 candidate names, so on its own it does not show that probability is spread over several feedback-aligned alternatives, although the paper pairs it with the higher teacher entropy. The samples are small (87 and 71 positions), and no validation of the GPT-5.5 annotation is reported. The Appendix B examples also allow other readings: in the sharpening cases the teacher names the criterion that a key phrase of the rationale points to, and in the spreading cases the student has already committed with high probability to a criterion that the rationale does not treat as decisive. Figure 3 (more distinct criterion clusters for SD+mask than for naive SD) is offered as further support, but the paper does not say whether the number of extracted criteria per judgment is the same for the two judges, and more clusters need not mean more appropriate criteria. A direct test is possible with existing models: adding the 30B direction-flip model of Table 4 to Figure 3 would show whether it has fewer distinct criterion clusters than naive SD and SD+mask, as the paper's interpretation would imply. Otherwise, I suggest presenting the interpretation as a hypothesis.

Questions for authors

Q1. Removing 70% of positions outright is a drastic choice. In Table 3 the gain in Total Avg. over naive SD at 30B is concentrated at rho = 0.7 (+2.33, against +0.46 and +0.51 at rho = 0.3 and 0.5), and at 4B it starts at rho = 0.5 (+0.82, Table 6). Did you test a soft weighting by Delta H_t instead of zeroing positions? Appendix A cites a weighting-based on-policy distillation method [21], so such a comparison seems natural.

Q2. Were all rows in Tables 4, 5 and 7 evaluated on the same prompts? The Precise IF values 40.67 and 44.00 (Table 4), 36.00 and 48.00 (Table 5) and 40.67 (Table 7) are multiples of 1/150, whereas all other Precise IF values in the paper are multiples of 1/160 (the size of the official subset). The same pattern appears for Focus, Math and Safety in these rows, as if ten prompts were missing from each of these four subsets.

Reviewer_5neEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a method that makes use of the rationales in preference datasets to improve the judge / reward model derived from the data. They achieve that by creating token-level supervision during RL rollouts. The supervision is derived from a model that is also conditioned on the rationale of the preference pair, acting as a teacher, with the student distilling the output distribution. Thereafter, the derived judge (the student) can be used without a need for a rationale to be provided during inference. The authors also identify an important dimension of the problem, the teacher's entropy at each token, and particularly its difference from the entropy of the student. They use this difference to filter samples where the teacher is more confident than the student by percentile, which further improves the performance of the student.

Critical strengths
  • The main method is intuitive and simple to implement and replicate, making it particularly relevant to the scientific community: they provide a very simple mechanism to allow researchers and practitioners to make use of rationales collected from annotators during preference annotations.
  • The authors present a plethora of experiments, and verify their findings with multiple models, baselines, and on various datasets.
  • The paper is mostly easy to follow.
Critical weaknesses
  • The intuition as to why, when the teacher is certain, the distillation is masked is not entirely clear to me. The results, of course, show that this is indeed a more useful direction than no filtering or filtering lower-entropy signals, yet the "theory" behind it appears as post-hoc explanation / rationalization of the results rather than a true motivation to pursue the technique. Of course, the authors did identify an interesting dimension of the problem, the relative entropy of the teacher, but the presentation of the reasoning I believe may be misleading.
  • Most results are not presented with error bars / confidence intervals / stddevs.
Questions for authors
  • When sweeping rho, it does not seem to yield monotonic (or even U/inverse-U shaped) behavior. Is this perhaps an indication that the identified criterion is not the "correct" filtering approach but only correlates decently well with that?
  • Section 3.3 is very interesting, I believe the paper would benefit from more evidence like this, e.g., simply more datasets, or even using the judge to train the models. Why was only that one experiment performed?
  • Could the authors elaborate on the small effect sizes of the similarity when using proper/control rationales in Section 2.3?
Reviewer_tN54OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a self-distillation method for training LLM judges with the rationale that comes with each preference label. The same model is run as a teacher with the rationale in its prompt and as a student without it, and the student is trained to match the teacher's token distribution on its own rollouts. The authors further measure how much the rationale changes the teacher's entropy at each position, and mask out the positions where the teacher becomes more confident, so that only the remaining positions are distilled. The method is tested on various judge benchmarks with two model sizes, along with ablations on the mask fraction and the selection rule.

Critical strengths
  • The paper addresses a timely and relevant problem.
  • The writing is clear, and easy to follow.
Critical weaknesses
  • One of my main concerns is that there is no proper baseline for using the rationale. The paper argues that the rationale is useful, but the only method that uses it is the authors' own naive SD, and naive SD is no better than Dr. GRPO at 30B. Other simple ways to use the rationale are not tried, for example fine-tuning the judge to write the rationale, or giving a reward when the judge's reasoning agrees with the rationale as in RationaleRM [39]. RationaleRM-30B appears in Table 1 only as a number taken from a different model and data.
  • The masking part does not seem to be about the rationale. To my understanding, it is a token selection rule for on-policy distillation, and similar rules already exist (Appendix A). In particular, the positions kept here look like the overconfident tokens of TIP [46], but no such method is compared or discussed. Appendix C also shows that the rule works the same with generated rubrics, so the real contribution seems to be a general token selection rule rather than a way of using language feedback.
  • The explanation of why the masking works is not well supported. The paper says that sharpening positions lead to memorization and spreading positions teach understanding, but the analysis in Section 2.3 is on one model with about 160 positions, and it only checks the teacher side. Whether the student is actually wrong at the kept positions is never checked. The method may simply work because it only trains on positions that raise the student's entropy.
  • I am also not sure whether the comparison is fully fair. The mask fraction is chosen on the test benchmarks (Table 3), and the sweep stops at 0.7 while the accuracy is still increasing. In Figure 1, the curves for the 30B model are still rising at the end of training, so the numbers are at a fixed budget rather than at convergence. SD and Dr. GRPO also use different learning rates and rollout counts (Appendix E) without explanation.
Questions for authors

Refer to the above weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.