- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
When one model is both policy and judge in self-rewarding training, does training the policy role shift the judge toward responses agreeing with a user's stated answer, and does that shift change which preference pairs are selected next? On MMLU-Pro with three models, Experiment 1 trains only the policy role in arms differing only in stance pairing and finds agreement-paired > neutral > disagreement-paired in all nine model and seed runs. Experiment 2 replaces the stance-sensitive (odd) part of the judge score with the base model's before DPO, which changes many selected pairs but has a mixed effect on the next policy, and trains four rounds with stance-averaged rewards, lowering measured dependence by 10.75 points at an uncertain capability cost.
Critical strengths
- The matched neutral arm makes the first result credible. The neutral arm sees the same prompts and responses as the other arms, differing only in pairing, so it isolates the stance association. Every trained arm ends below base while the ordering holds in every run, so a comparison with base alone would have pointed the opposite way, a reusable lesson for studies of self-evaluating models.
- The component intervention is clean and candidly reported. The two branches share the first-round adapter, candidates, prompt order and pre-substitution live scores, so later differences trace to the replaced component. The paper discloses the reversed seed, the unmet capability margin and the protocol amendments, and supports H3 with two readouts independent of its answer extractor.
Critical weaknesses
- The two links are tested separately, and the second link's behavioral effect is not stable across seeds. The motivating claim is a coupling: training shifts the judge, which then changes what the next policy learns, so each link must hold reliably on its own. H2 is a primary result with a Holm p of 0.0007, yet one of three seeds is slightly negative and the interval includes zero, as it does in the Llama replication. The small p comes from a bootstrap that centers every item at the pooled mean (Eq. 8), so it reflects item resampling within seeds and, as the paper notes, says little about stability across seeds, the right replication unit for a training intervention. Progressive amplification of the odd score is also untested. The supported finding is that this component alters pair selection, not that a judge shift reliably changes the next policy.
- The H3 reduction cannot yet be separated from output degeneration, capability change and answer extraction. Dependence is computed from extracted answers, so reading H3 as reduced sycophancy requires both paths to produce comparably valid answers; they do not. Length termination, rare at base, becomes common on both paths; in one seed the stance-averaged model gives an extractable answer far less often than the live-judge model and shows a large capability deficit, while unextracted answers can move the metric either way. The arbitrating answer readings come from AI readers of unrecorded version and a study-aware AI adjudicator; varying the unresolved readings moves H3 from -2.11 to 18.58 points (Table 22). The protocol, including the length budget, was amended mid-execution. H3 therefore shows lower measured dependence, not yet lower sycophancy, which is what the mitigation claim needs.
Questions for authors
- Could you report seed-level inference for H1 to H3, such as exact permutation over seeds, and explain why H2 keeps a primary Holm p-value given your own caveat about what it measures?
- Since a pair changes whenever either member differs, how often does it change under a substitution that should not matter, such as rescoring the same first-round candidates with the same-stance paraphrase used for ParaAvg?
- With the checkpoints retained, can you report the fixed-cohort judge interaction for the live-judge path across all rounds of Experiment 2?
- For H3, what is the dependence gap on items where neither path hit the length limit, and does a human reading of length-terminated stance-averaged responses in seed 2 confirm the extracted answers?
3 comments
5. Which links are supported (W1)
The stance-odd component alters pair selection, while its next-update behavioral effect is heterogeneous across seeds. The revision adds seed-level inference to the mixed H2 outcome stated in the abstract and conclusion (pp. 1, 10; §4.3, p. 7). The matched-exposure comparison is positioned alongside the before–after and accuracy-based judge evaluations used in self- and meta-rewarding studies (Yuan et al., 2024; Wu et al., 2025; Wang et al., 2025; §2, p. 3; §5, p. 9). In the new post hoc no-stance SFT control (Qwen9, Llama8), the shared decrease is absent in Qwen9 and much smaller in Llama8, which reinforces the matched neutral arm's role in isolating the pairing effect (§4.1, pp. 6–7; Appendix B.3, Table 12(c), pp. 21–22).
The new analyses show consistent changes in judge scores and selected pairs, followed by mixed next-update behavioral effects in Qwen9. The first self-rewarding update raises the fixed-cohort judge interaction above base in every seed (§4.2, p. 7; Appendix G.1, Table 30, p. 34). Replacement then moves selected pairs away from the wrong stated answer in every seed (§4.2, p. 7; Appendix C.3, Table 16, pp. 24–25). Next-policy H2 is +3.55 pp; seed contrasts are +5.21/+5.71/−0.29 pp (§4.3, p. 7).
Removing the whole stance input gives a different pattern from replacing the learned component. The new post hoc stance-free (SF) baseline has lower next-update Dep than the live-judge path in all six model–seed comparisons: by 6.43/7.96/5.96 pp for Qwen9 and 13.18/13.50/11.86 pp for Llama8 (§4.3, p. 8; Appendix D.3, Table 19, p. 27). Its training pairs have wrong-stance agreement contrasts of −11.63/−16.60/−12.96 pp for Qwen9 and −4.58/−8.54/−6.69 pp for Llama8, all negative and lower than V (Appendix D.3, p. 28).
Llama8 differs at pair selection: replacement moves pairs toward the wrong stated answer (§4.3, p. 8; Appendix C.3, p. 25). Its V−OF contrast is mixed, +11.79/+2.89/−1.68 pp; on naturally ended, extracted items it is +6.82/+2.64/+0.09 pp, with the last interval [−1.55, 1.68] including zero (Appendix D.2, pp. 26–27; Table 18, p. 26). Experiments 1 and 2 identify separately trained interventions (§5, p. 10).
6. Interpretation of H3 and the response audit (W2)
H3 shows lower measured dependence, with capability preservation and answer validity unresolved; the revised §4.4 (p. 9) and the conclusion (p. 10) use that interpretation.
The audit description specifies the annotation procedure (§3.4, p. 6; Appendix F.2, p. 30). GPT-6 Luna annotated each sampled response twice through separate API calls, without sharing predictions. The annotation calls saw only the question, its options, and the full response; condition labels (model, arm, seed), the gold answer, the extractor's prediction, and the finish reason were hidden from them. Responses with disagreement between annotations, uncertainty in either annotation, or disagreement between an annotation and the rule-based extractor were adjudicated by GPT-6 Terra (84 of 960 responses). The task was to identify the response's selected answer rather than solve the question; responses without an identifiable final choice received a null label. The audit served evaluation only and played no role in training or reward computation. Its unresolved-reading range for H3 is [−2.11, 18.58] pp (Figure 2, p. 8; Appendix F.3, Table 25 (Table 22 in the submission), p. 31), and the abstract (p. 1) states that unresolved readings permit either sign. The requested human reading covers audited seed-2 items (Section 4 above; Appendix F.5, Table 28, p. 33).
The capability criterion is unmet: −1.35 pp [−12.57, 4.70]; seed 2 −12.73 pp (§4.4, p. 9; Tables 1 and 3, pp. 6 and 8). Seed-2 BSR4 has 52.14% extraction versus V4's 72.57%, and 57.86% length termination (Appendix E.1, Table 20, p. 28). The protocol amendments, including the 2,048-token response budget, remain disclosed in §3.4 (pp. 5–6) and Appendix A.5 (p. 19), which also reports the regeneration of responses from fixed checkpoints. The timing of the amendments relative to outcomes is incompletely documented (§3.4, p. 6; §5, p. 10), and the analyses added in revision are labeled post hoc in §3.4 (p. 6), Appendix A.5 (p. 19), and the appendix sections that report them.
Thank you again for the careful reading and the precise requests, which made the revision considerably stronger.
References
- Yuan, W. et al. Self-Rewarding Language Models. ICML 2024.
- Wu, T. et al. Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge. EMNLP 2025.
- Wang, Z. et al. CREAM: Consistency Regularized Self-Rewarding Language Models. ICLR 2025.
Reminder: Request for Feedback on Our Rebuttal
Thank you again for your review. As the discussion period is coming to a close, we wanted to briefly follow up on our rebuttal.
If you have a chance to review our response, we would appreciate any further feedback, particularly if you have any remaining concerns. We would also appreciate your consideration of our response in your final assessment.
Thank you for your time and consideration.
Thank you for the thorough and constructive review, and for highlighting the matched neutral arm as a reusable lesson and the clean component intervention. The four concrete questions directly shaped the new analyses. Each comment is answered point by point below, and the corresponding revisions are incorporated in the updated PDF, with new analyses and substantive additions highlighted in blue.
1. Seed-level inference and the H2 p-value (Q1)
Table 1 (p. 6) replaces the bootstrap Holm column with post hoc exact one-sided sign-flip p-values over the three seed contrasts, since the training seed is the replication unit: 0.125/0.250/0.125 for H1–H3. The minimum attainable p is 0.125; Holm-adjusted values are 0.375 each.
The new 95% t intervals, with two degrees of freedom, are [0.072, 0.123] for H1, [−4.7, 11.8] pp for H2, and [1.6, 19.9] pp for H3 (Appendix H.2, Table 36, p. 39). They assume normally distributed seed contrasts and exclude answer-reading uncertainty. H1 and H3 are positive in every seed; H2 is not. All three intervals now appear in the main text (§4.1, p. 6; §4.3, p. 7; §4.4, p. 9). H2 remains a primary contrast; the abstract (p. 1) now reports its seed-level t interval, and §4.3 (p. 7) states that a reproducible behavioral benefit is not established.
The protocol-specified bootstrap p-values for H1–H3 remain in the Table 1 caption (p. 6) and Appendix H.2, Table 35 (p. 38). The new explanation of that calculation confirms your diagnosis: after centering every item at the pooled mean, a null replicate reaches the observed estimate only when its uncentered mean is at least twice that estimate. For H2, 7.10 pp exceeds the largest seed contrast, 5.71 pp, so exceedances require item resampling within seeds. This does not establish stability across training runs.
2. Pair changes under reference substitutions (Q2)
A same-stance paraphrase changes a comparable share of pairs to replacement: 34.4–36.7% against 33.40–41.60%; re-scoring identical inputs changes 19.9–22.7%. These new controls are defined in §3.3 (p. 5) and reported in §4.2 (p. 7) and Appendix C.3, Table 16 (pp. 24–25). The abstract (p. 1) and contribution list (p. 2) describe the comparable paraphrase rate; §4.2 states that a stance-preserving substitution changes a comparable share of pairs. Attribution in the V/OF comparison rests on identical live scores and substitution of only the odd component (§3.3, pp. 4–5; Appendix C.1, Table 14, p. 24).
The new composition analysis distinguishes direction: on wrong-stance prompts, the chosen-minus-rejected agreement contrast changes by −6.20/−5.53/−12.96 pp under replacement, −1.55/−3.16/−4.81 pp under paraphrasing, and +1.94/+0.40/−0.37 pp under re-scoring (§4.2, p. 7; Appendix C.3, p. 25). The paraphrase also lowers this statistic in every seed, and replacement lowers it by more.
3. Fixed-cohort judge interaction across rounds (Q3)
The new trajectory has no common direction across seeds along the live-judge path (§4.4, p. 9; Appendix G.1, Table 30, p. 34). OR on the same 978 natural pairs is:
The first self-rewarding update raises OR above base (0.173) in every seed (§4.2, p. 7). From C1 to V4 it rises in seed 0, falls in seed 1, and is non-monotone in seed 2. V4 exceeds BSR4 in every seed; BSR4 lies below base. Each checkpoint has a single evaluation, and the base comparison is descriptive.
4. Natural termination and human answer reading (Q4)
The new natural-termination analysis retains a positive gap in every seed. When all four V4/BSR4 responses end naturally, H3 is +14.11/+8.88/+5.95 pp on 535/1,059/370 items, with nominal 95% item-bootstrap intervals [11.21, 17.20]/[6.70, 11.05]/[3.51, 8.38]. Requiring extraction leaves seed 1 at +8.88 pp on 1,058 items (§4.4, p. 9; Appendix F.4, Table 26, p. 32). These post-treatment subsets do not identify the full-cohort effect; seed 2 is below its full-cohort gap of 9.43 pp.
Two annotators, each with at least four years of NLP research experience, independently read and reviewed 100 responses from 25 seed-2 items, including the 38 sampled length-terminated BSR4 responses. Their labels retain all 70 answers found by the rule-based answer extractor, including the 18 among those 38 responses, and recover 17 previously unextracted answers (11 BSR4, 6 V4). With the original sampling weights, seed-2 H3 moves from +9.43 to +11.77 pp (equal-seed mean +11.53 pp); the seed-2 measurement-only 95% interval is [−8.84, 32.38] pp, which includes zero. The calculation changes labels only for these 100 responses and keeps extractor labels for the rest of the audit sample (§4.4, p. 9; Appendix F.5, Table 28, Set 3, p. 33). The reading covers 25 of the 1,400 seed-2 behavioral items.