- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper trains pairwise LLM judges from preference data in which each label comes with a short human rationale. Instead of rewarding only the final verdict, as Dr. GRPO does, the authors use on-policy self-distillation (SD): the same model, conditioned on the rationale, acts as a teacher at every position of the student's rollout. They measure the per-position entropy shift Delta H_t between teacher and student, and call large positive shifts "context sharpening" and large negative shifts "context spreading". Their interpretation is that sharpening encourages memorization of specific criterion wording, while spreading supports semantic understanding. On this basis, SD+mask discards the 70% of positions with the largest Delta H_t in each rollout. With Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507 trained on HelpSteer3-Preference, SD+mask improves RM-Bench and RewardBench V2 over naive SD (no masking) and Dr. GRPO. The paper reports gains of 2 to 9 points over Dr. GRPO on the Chat, Factuality and Focus subcategories, which it calls subjective. SD+mask also gives higher win rates than Dr. GRPO and naive SD when its 30B judge selects responses in a best-of-8 tournament on Arena-Hard-v2 creative writing.
Critical strengths
The paper separates criterion choice from the rigor with which criteria are applied. This explains why a verdict-only reward gives little signal on the tokens that name criteria, and why that should matter more for subjective judging. The analysis of where the distillation signal sits is also informative: per-position reverse KL is U-shaped in Delta H_t, and the tokens that name and define criteria are 16.7% of response tokens but 44.4% of the KL mass.
The selector ablations are the strongest evidence in the paper that the proposed rule beats the alternatives, subject to W1 and Q2 below. Five alternatives are compared at matched rho on both model sizes, including a direction flip (keeping only the highest-Delta H_t positions) and random masking, and the proposed selector is best in both. At 30B it is 2.8 points of Total Avg. ahead of the best alternative.
The data processing is careful: duplicates and train/validation leakage in HelpSteer3-Preference are removed. The appendix also tests a second kind of feedback.
Critical weaknesses
W1. The main comparisons appear to rest on single runs. Appendix E lists six main training runs (two models, three methods), and I could not find seeds or variances. Evaluation uses one sample per prompt at temperature 0.7. Several comparisons depend on small differences. In Table 3, Total Avg. at 30B is 80.17, 80.63 and 80.68 for rho in {0, 0.3, 0.5}. Naive SD and Dr. GRPO are also only 0.14 apart at 30B (80.17 and 80.31). Some RewardBench V2 subsets are small: Precise IF has 160 prompts, so one prompt is 0.625 points. Section 3.2 highlights a Precise IF gain from 40.62 to 48.75 (naive SD to SD+mask at 30B). I cannot tell whether differences of this size exceed training and sampling noise. Could the authors report means and standard deviations over several seeds? Given the stated cost of roughly one day per run, the 4B setting alone would be informative. Confidence intervals over evaluation prompts would help too.
W2. SD sees the preference rationale and Dr. GRPO does not, so the 2 to 9 point gains on the subcategories the paper calls subjective could come from the extra information rather than from distillation. There is no controlled baseline that uses the rationales without distillation, for example the RL methods with natural-language critiques described in Appendix A (Critique-GRPO, Text2Grad). Table 1 also has no rows for the untrained Qwen3 judges, so one cannot see how much each method gains over its starting point on RewardBench V2 or on the RM-Bench subcategories (Figure 1 gives only RM-Bench Overall). I would like to see the untrained rows and such a baseline, or an explanation of why none applies.
W3. Appendix E gives Dr. GRPO a learning rate of 1e-6 at 4B and 2e-6 at 30B with 8 rollouts per prompt, and gives naive SD and SD+mask 5e-6 and 1e-5 (five times larger) with 4 rollouts. How were these values chosen? Apart from the rho sweep for SD+mask, the only tuning I could find is for the prompt template (Section 3.1). In Figure 1, the smoothed 4B Dr. GRPO curve stays within about one point of its starting accuracy (roughly 81.7 to 82.6 on RM-Bench). I cannot tell whether the baseline is well tuned. If Dr. GRPO was not tuned, please add a small learning-rate sweep for it.
W4. I could not find a direct test of the memorization-versus-understanding explanation that motivates masking (Abstract, Sections 2.3, 2.4 and 3.2). Section 2.3 reports that the teacher's candidate criterion names are closer in embedding space to the decisive criterion extracted from the rationale than when the rationale comes from another example (sharpening 0.713 vs 0.647, spreading 0.646 vs 0.610). This shows that the teacher uses the rationale, which is the expected effect of conditioning on it, not that sharpening leads to memorization or spreading to understanding. For spreading, the score is a mean over the top-10 candidate names, so on its own it does not show that probability is spread over several feedback-aligned alternatives, although the paper pairs it with the higher teacher entropy. The samples are small (87 and 71 positions), and no validation of the GPT-5.5 annotation is reported. The Appendix B examples also allow other readings: in the sharpening cases the teacher names the criterion that a key phrase of the rationale points to, and in the spreading cases the student has already committed with high probability to a criterion that the rationale does not treat as decisive. Figure 3 (more distinct criterion clusters for SD+mask than for naive SD) is offered as further support, but the paper does not say whether the number of extracted criteria per judgment is the same for the two judges, and more clusters need not mean more appropriate criteria. A direct test is possible with existing models: adding the 30B direction-flip model of Table 4 to Figure 3 would show whether it has fewer distinct criterion clusters than naive SD and SD+mask, as the paper's interpretation would imply. Otherwise, I suggest presenting the interpretation as a hypothesis.
Questions for authors
Q1. Removing 70% of positions outright is a drastic choice. In Table 3 the gain in Total Avg. over naive SD at 30B is concentrated at rho = 0.7 (+2.33, against +0.46 and +0.51 at rho = 0.3 and 0.5), and at 4B it starts at rho = 0.5 (+0.82, Table 6). Did you test a soft weighting by Delta H_t instead of zeroing positions? Appendix A cites a weighting-based on-policy distillation method [21], so such a comparison seems natural.
Q2. Were all rows in Tables 4, 5 and 7 evaluated on the same prompts? The Precise IF values 40.67 and 44.00 (Table 4), 36.00 and 48.00 (Table 5) and 40.67 (Table 7) are multiples of 1/150, whereas all other Precise IF values in the paper are multiples of 1/160 (the size of the official subset). The same pattern appears for Focus, Math and Safety in these rows, as if ten prompts were missing from each of these four subsets.