acceptodds
Under review as a conference paper at ICLR 2027

Controlled Audits of Policy–Judge Coupling

Abstract

Self-rewarding language models reuse a policy as the judge supplying its next training signal. We distinguish changes in evaluator preferences from changes in evaluated responses through two controlled experiments on factual multiple-choice questions. First, policy fine-tuning varies the association between a user's stated answer (the stance) and the response while matching prompt and response marginals, without judge-format examples. On fixed responses, the judge's relative preference for agreement follows agreement-paired neutral disagreement-paired training in all nine model–seed combinations across Qwen3.5-9B, Qwen3.5-4B, and Llama-3.1-8B-Instruct. Every trained condition nevertheless has a lower stance-dependent score contrast than base. Second, at an identical policy checkpoint and candidate set, replacing the learned stance-sensitive reward component with its base-model counterpart changes 33.40–41.60% of selected preference pairs across three Qwen3.5-9B seeds. The next-update stance-dependence contrast is +3.55 percentage points, with mixed seed signs and a 95% interval of [−0.02, 6.23]. Averaging rewards over both stances for four rounds yields 10.75 percentage points lower measured dependence; capability preservation is unestablished, and unresolved answer readings permit either sign. Matched exposure reveals evaluator differences that base-only audits miss, while reward interventions require separate validation of selected supervision and policy behavior.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
self-rewarding language models, LLM alignment, evaluator drift, reward modeling, sycophancy, preference optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QEmJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When one model is both policy and judge in self-rewarding training, does training the policy role shift the judge toward responses agreeing with a user's stated answer, and does that shift change which preference pairs are selected next? On MMLU-Pro with three models, Experiment 1 trains only the policy role in arms differing only in stance pairing and finds agreement-paired > neutral > disagreement-paired in all nine model and seed runs. Experiment 2 replaces the stance-sensitive (odd) part of the judge score with the base model's before DPO, which changes many selected pairs but has a mixed effect on the next policy, and trains four rounds with stance-averaged rewards, lowering measured dependence by 10.75 points at an uncertain capability cost.

Critical strengths
  1. The matched neutral arm makes the first result credible. The neutral arm sees the same prompts and responses as the other arms, differing only in pairing, so it isolates the stance association. Every trained arm ends below base while the ordering holds in every run, so a comparison with base alone would have pointed the opposite way, a reusable lesson for studies of self-evaluating models.
  2. The component intervention is clean and candidly reported. The two branches share the first-round adapter, candidates, prompt order and pre-substitution live scores, so later differences trace to the replaced component. The paper discloses the reversed seed, the unmet capability margin and the protocol amendments, and supports H3 with two readouts independent of its answer extractor.
Critical weaknesses
  1. The two links are tested separately, and the second link's behavioral effect is not stable across seeds. The motivating claim is a coupling: training shifts the judge, which then changes what the next policy learns, so each link must hold reliably on its own. H2 is a primary result with a Holm p of 0.0007, yet one of three seeds is slightly negative and the interval includes zero, as it does in the Llama replication. The small p comes from a bootstrap that centers every item at the pooled mean (Eq. 8), so it reflects item resampling within seeds and, as the paper notes, says little about stability across seeds, the right replication unit for a training intervention. Progressive amplification of the odd score is also untested. The supported finding is that this component alters pair selection, not that a judge shift reliably changes the next policy.
  2. The H3 reduction cannot yet be separated from output degeneration, capability change and answer extraction. Dependence is computed from extracted answers, so reading H3 as reduced sycophancy requires both paths to produce comparably valid answers; they do not. Length termination, rare at base, becomes common on both paths; in one seed the stance-averaged model gives an extractable answer far less often than the live-judge model and shows a large capability deficit, while unextracted answers can move the metric either way. The arbitrating answer readings come from AI readers of unrecorded version and a study-aware AI adjudicator; varying the unresolved readings moves H3 from -2.11 to 18.58 points (Table 22). The protocol, including the length budget, was amended mid-execution. H3 therefore shows lower measured dependence, not yet lower sycophancy, which is what the mitigation claim needs.
Questions for authors
  1. Could you report seed-level inference for H1 to H3, such as exact permutation over seeds, and explain why H2 keeps a primary Holm p-value given your own caveat about what it measures?
  2. Since a pair changes whenever either member differs, how often does it change under a substitution that should not matter, such as rescoring the same first-round candidates with the same-stance paraphrase used for ParaAvg?
  3. With the checkpoints retained, can you report the fixed-cohort judge interaction for the live-judge path across all rounds of Experiment 2?
  4. For H3, what is the dependence gap on items where neither path hit the length limit, and does a human reading of length-terminated stance-averaged responses in seed 2 confirm the extracted answers?
3 comments

Thank you for the thorough and constructive review, and for highlighting the matched neutral arm as a reusable lesson and the clean component intervention. The four concrete questions directly shaped the new analyses. Each comment is answered point by point below, and the corresponding revisions are incorporated in the updated PDF, with new analyses and substantive additions highlighted in blue.

1. Seed-level inference and the H2 p-value (Q1)

Table 1 (p. 6) replaces the bootstrap Holm column with post hoc exact one-sided sign-flip p-values over the three seed contrasts, since the training seed is the replication unit: 0.125/0.250/0.125 for H1–H3. The minimum attainable p is 0.125; Holm-adjusted values are 0.375 each.

The new 95% t intervals, with two degrees of freedom, are [0.072, 0.123] for H1, [−4.7, 11.8] pp for H2, and [1.6, 19.9] pp for H3 (Appendix H.2, Table 36, p. 39). They assume normally distributed seed contrasts and exclude answer-reading uncertainty. H1 and H3 are positive in every seed; H2 is not. All three intervals now appear in the main text (§4.1, p. 6; §4.3, p. 7; §4.4, p. 9). H2 remains a primary contrast; the abstract (p. 1) now reports its seed-level t interval, and §4.3 (p. 7) states that a reproducible behavioral benefit is not established.

The protocol-specified bootstrap p-values for H1–H3 remain in the Table 1 caption (p. 6) and Appendix H.2, Table 35 (p. 38). The new explanation of that calculation confirms your diagnosis: after centering every item at the pooled mean, a null replicate reaches the observed estimate only when its uncentered mean is at least twice that estimate. For H2, 7.10 pp exceeds the largest seed contrast, 5.71 pp, so exceedances require item resampling within seeds. This does not establish stability across training runs.

2. Pair changes under reference substitutions (Q2)

A same-stance paraphrase changes a comparable share of pairs to replacement: 34.4–36.7% against 33.40–41.60%; re-scoring identical inputs changes 19.9–22.7%. These new controls are defined in §3.3 (p. 5) and reported in §4.2 (p. 7) and Appendix C.3, Table 16 (pp. 24–25). The abstract (p. 1) and contribution list (p. 2) describe the comparable paraphrase rate; §4.2 states that a stance-preserving substitution changes a comparable share of pairs. Attribution in the V/OF comparison rests on identical live scores and substitution of only the odd component (§3.3, pp. 4–5; Appendix C.1, Table 14, p. 24).

The new composition analysis distinguishes direction: on wrong-stance prompts, the chosen-minus-rejected agreement contrast changes by −6.20/−5.53/−12.96 pp under replacement, −1.55/−3.16/−4.81 pp under paraphrasing, and +1.94/+0.40/−0.37 pp under re-scoring (§4.2, p. 7; Appendix C.3, p. 25). The paraphrase also lowers this statistic in every seed, and replacement lowers it by more.

3. Fixed-cohort judge interaction across rounds (Q3)

The new trajectory has no common direction across seeds along the live-judge path (§4.4, p. 9; Appendix G.1, Table 30, p. 34). OR on the same 978 natural pairs is:

SeedC1V2V3V4BSR4
00.3110.4230.4920.5910.116
10.2670.2250.1910.1740.101
20.2830.2650.2860.2520.108

The first self-rewarding update raises OR above base (0.173) in every seed (§4.2, p. 7). From C1 to V4 it rises in seed 0, falls in seed 1, and is non-monotone in seed 2. V4 exceeds BSR4 in every seed; BSR4 lies below base. Each checkpoint has a single evaluation, and the base comparison is descriptive.

4. Natural termination and human answer reading (Q4)

The new natural-termination analysis retains a positive gap in every seed. When all four V4/BSR4 responses end naturally, H3 is +14.11/+8.88/+5.95 pp on 535/1,059/370 items, with nominal 95% item-bootstrap intervals [11.21, 17.20]/[6.70, 11.05]/[3.51, 8.38]. Requiring extraction leaves seed 1 at +8.88 pp on 1,058 items (§4.4, p. 9; Appendix F.4, Table 26, p. 32). These post-treatment subsets do not identify the full-cohort effect; seed 2 is below its full-cohort gap of 9.43 pp.

Two annotators, each with at least four years of NLP research experience, independently read and reviewed 100 responses from 25 seed-2 items, including the 38 sampled length-terminated BSR4 responses. Their labels retain all 70 answers found by the rule-based answer extractor, including the 18 among those 38 responses, and recover 17 previously unextracted answers (11 BSR4, 6 V4). With the original sampling weights, seed-2 H3 moves from +9.43 to +11.77 pp (equal-seed mean +11.53 pp); the seed-2 measurement-only 95% interval is [−8.84, 32.38] pp, which includes zero. The calculation changes labels only for these 100 responses and keeps extractor labels for the rest of the audit sample (§4.4, p. 9; Appendix F.5, Table 28, Set 3, p. 33). The reading covers 25 of the 1,400 seed-2 behavioral items.

5. Which links are supported (W1)

The stance-odd component alters pair selection, while its next-update behavioral effect is heterogeneous across seeds. The revision adds seed-level inference to the mixed H2 outcome stated in the abstract and conclusion (pp. 1, 10; §4.3, p. 7). The matched-exposure comparison is positioned alongside the before–after and accuracy-based judge evaluations used in self- and meta-rewarding studies (Yuan et al., 2024; Wu et al., 2025; Wang et al., 2025; §2, p. 3; §5, p. 9). In the new post hoc no-stance SFT control (Qwen9, Llama8), the shared decrease is absent in Qwen9 and much smaller in Llama8, which reinforces the matched neutral arm's role in isolating the pairing effect (§4.1, pp. 6–7; Appendix B.3, Table 12(c), pp. 21–22).

The new analyses show consistent changes in judge scores and selected pairs, followed by mixed next-update behavioral effects in Qwen9. The first self-rewarding update raises the fixed-cohort judge interaction above base in every seed (§4.2, p. 7; Appendix G.1, Table 30, p. 34). Replacement then moves selected pairs away from the wrong stated answer in every seed (§4.2, p. 7; Appendix C.3, Table 16, pp. 24–25). Next-policy H2 is +3.55 pp; seed contrasts are +5.21/+5.71/−0.29 pp (§4.3, p. 7).

Removing the whole stance input gives a different pattern from replacing the learned component. The new post hoc stance-free (SF) baseline has lower next-update Dep than the live-judge path in all six model–seed comparisons: by 6.43/7.96/5.96 pp for Qwen9 and 13.18/13.50/11.86 pp for Llama8 (§4.3, p. 8; Appendix D.3, Table 19, p. 27). Its training pairs have wrong-stance agreement contrasts of −11.63/−16.60/−12.96 pp for Qwen9 and −4.58/−8.54/−6.69 pp for Llama8, all negative and lower than V (Appendix D.3, p. 28).

Llama8 differs at pair selection: replacement moves pairs toward the wrong stated answer (§4.3, p. 8; Appendix C.3, p. 25). Its V−OF contrast is mixed, +11.79/+2.89/−1.68 pp; on naturally ended, extracted items it is +6.82/+2.64/+0.09 pp, with the last interval [−1.55, 1.68] including zero (Appendix D.2, pp. 26–27; Table 18, p. 26). Experiments 1 and 2 identify separately trained interventions (§5, p. 10).

6. Interpretation of H3 and the response audit (W2)

H3 shows lower measured dependence, with capability preservation and answer validity unresolved; the revised §4.4 (p. 9) and the conclusion (p. 10) use that interpretation.

The audit description specifies the annotation procedure (§3.4, p. 6; Appendix F.2, p. 30). GPT-6 Luna annotated each sampled response twice through separate API calls, without sharing predictions. The annotation calls saw only the question, its options, and the full response; condition labels (model, arm, seed), the gold answer, the extractor's prediction, and the finish reason were hidden from them. Responses with disagreement between annotations, uncertainty in either annotation, or disagreement between an annotation and the rule-based extractor were adjudicated by GPT-6 Terra (84 of 960 responses). The task was to identify the response's selected answer rather than solve the question; responses without an identifiable final choice received a null label. The audit served evaluation only and played no role in training or reward computation. Its unresolved-reading range for H3 is [−2.11, 18.58] pp (Figure 2, p. 8; Appendix F.3, Table 25 (Table 22 in the submission), p. 31), and the abstract (p. 1) states that unresolved readings permit either sign. The requested human reading covers audited seed-2 items (Section 4 above; Appendix F.5, Table 28, p. 33).

The capability criterion is unmet: −1.35 pp [−12.57, 4.70]; seed 2 −12.73 pp (§4.4, p. 9; Tables 1 and 3, pp. 6 and 8). Seed-2 BSR4 has 52.14% extraction versus V4's 72.57%, and 57.86% length termination (Appendix E.1, Table 20, p. 28). The protocol amendments, including the 2,048-token response budget, remain disclosed in §3.4 (pp. 5–6) and Appendix A.5 (p. 19), which also reports the regeneration of responses from fixed checkpoints. The timing of the amendments relative to outcomes is incompletely documented (§3.4, p. 6; §5, p. 10), and the analyses added in revision are labeled post hoc in §3.4 (p. 6), Appendix A.5 (p. 19), and the appendix sections that report them.

Thank you again for the careful reading and the precise requests, which made the revision considerably stronger.


References

  • Yuan, W. et al. Self-Rewarding Language Models. ICML 2024.
  • Wu, T. et al. Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge. EMNLP 2025.
  • Wang, Z. et al. CREAM: Consistency Regularized Self-Rewarding Language Models. ICLR 2025.
AuthorsOpenReview ↗

Reminder: Request for Feedback on Our Rebuttal

Thank you again for your review. As the discussion period is coming to a close, we wanted to briefly follow up on our rebuttal.

If you have a chance to review our response, we would appreciate any further feedback, particularly if you have any remaining concerns. We would also appreciate your consideration of our response in your final assessment.

Thank you for your time and consideration.

Reviewer_1gBVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the relationships among response training, judging preferences, and subsequent training in self-rewarding language models whose policy and judge share weights. First, the authors construct agreement-paired, neutral, and disagreement-paired fine-tuning data with matched prompt and response exposure, then evaluate the resulting judges on fixed responses. Second, they decompose judge scores into an even component that is invariant under a stance flip and an odd component that changes sign. At a shared checkpoint and candidate set, they replace the odd component and examine changes in preference pairs and next-update policy behavior. Finally, they run four rounds of self-rewarding training using scores averaged over both user stances and evaluate stance dependence, answer accuracy, and output validity.

Critical strengths
  1. The preference-transfer experiment is well controlled. Matching prompt and response exposure, changing only their pairing, and evaluating fixed responses makes the evidence stronger than a simple before–after comparison. The consistent ordering across nine model–seed combinations adds credibility.
  2. Component replacement provides a clear comparison. Holding the checkpoint and candidates fixed isolates its effect on pair selection. The authors also report negative results and measurement limitations transparently.
Critical weaknesses
  1. The experiments are elaborate, but the additional insight and practical relevance are limited. It is reasonably expected that response training affects a shared judge and that changing scores changes selected data. Experiment 1 provides cleaner evidence for the former, which has value; reporting how many pairs change in Experiment 2 offers less insight on its own. The paper also does not sufficiently explain which practical training workflows its particular self-generation, self-scoring, and self-training setup can inform.

  2. The more informative observations remain unexplained. OR falls below base in every trained condition in Experiment 1. In Experiment 2, selected-response correctness improves without consistently reducing next-update Dep: both Qwen-9B and Llama-8B show mixed seed signs. These observations do not invalidate the local findings, but the paper mainly reports differences without establishing when or why the intervention helps.

  3. The accumulated intervention has not demonstrated reliable benefits. Four-round BSR lowers measured Dep by 10 pp, but one seed loses 12 pp in Acc_stance, reaches 57% length termination, and has only 52% answer extraction. Dep is also zero when neither stance-conditioned response yields an extractable answer, so some apparent improvement may reflect response failure. Supplementary measurements provide support but do not resolve this alternative explanation or establish capability preservation.

Questions for authors
  1. Beyond confirming these reasonably expected relationships, what is the main new insight? Which practical training workflows can this self-rewarding setup inform?
  2. Why does OR fall below base in every trained condition? Why does odd-component replacement improve chosen-response correctness without consistently lowering subsequent Dep?
  3. Can existing outputs and stronger answer validation distinguish genuine behavioral improvement from truncation, missing answers, and capability changes? Which conclusions remain supported under these measurement uncertainties?
3 comments

Thank you for the thorough evaluation and for recognizing the controlled design of the preference-transfer experiment and the component-replacement comparison. The questions on the main insight, the shared OR decrease, and answer validity were very helpful. Each comment is answered point by point below, and the corresponding revisions are incorporated in the updated PDF, with new analyses and substantive additions highlighted in blue.

1. New insight and practical workflows (Q1/W1)

A before–after audit here would read the judges in all three stance-bearing arms as less agreement-prone than base, yet matched exposure shows that agreement-paired training raised agreement preference in all nine model–seed runs (+0.09743 for Qwen3.5-9B; §4.1, p. 6; Table 2, p. 7). This distinction matters for the before–after and accuracy-based judge evaluations used in self-/meta-rewarding studies: overall change and correctness ranking do not isolate the learned association (§2, p. 3; §5, p. 9).

For self- and meta-rewarding loops that reuse an updated policy as judge, three practical checks follow:

  1. Audit the judge against matched exposure. Keep prompt and response exposure fixed while changing their pairing; Table 2 (p. 7) separates the association effect from the decrease shared by the stance-bearing arms. The new no-stance SFT control in Section 2 below shows that the comparison with base changes with the training prompts.
  2. Validate selected pairs and next-policy behavior separately. Qwen9 seed 2 gains 20 correct chosen responses out of 512 and has the largest shift away from the wrong stated answer (−12.96 pp), yet next-update dependence does not fall. Llama8 selects fewer correct answers and favors the wrong stated answer more in every seed, yet Dep falls in two (§4.2–4.3, pp. 7–8; Appendix C.3, pp. 24–25; Table 18, p. 26).
  3. Withhold a separable stance from the judge. The new post hoc stance-free (SF) baseline has lower next-update Dep than the live-judge path (V2) in all six model–seed comparisons: by 6.43/7.96/5.96 pp for Qwen9 and 13.18/13.50/11.86 pp for Llama8 (§4.3, p. 8; Appendix D.3, Table 19, p. 27).

The pair-change controls reinforce the second check: re-scoring identical inputs changes 19.9–22.7% of pairs and a same-stance paraphrase changes 34.4–36.7%, comparable to replacement's 33.40–41.60% (§4.2, p. 7; Appendix C.3, p. 24; Table 16, p. 25).

2. Shared OR decrease and mixed behavioral effects (Q2/W2)

A new post hoc no-stance SFT control repeats Exp 1 SFT on the same 2,000 items and 4,000 target responses with the stance sentence omitted. OR is 0.260/0.285/0.260 for Qwen9 (base 0.174) and 0.473/0.449/0.444 for Llama8 (base 0.486), whereas all three stance-bearing arms fall below base in every seed (Appendix B.3, Table 12(c), pp. 21–22). The decrease is absent in Qwen9 and much smaller in Llama8 when the stance sentence is omitted; score normalization also excludes uniform rescaling (Appendix B.3, p. 21). The control pairs each item's single prompt with both responses, so it changes the conditional target structure together with the prompt text. The neutral arm balances agreement and disagreement mappings within each subject and ends near zero in every model: 0.02247/−0.00248/0.01064 for Qwen9/Qwen4/Llama8, against base 0.17401/0.36326/0.48621. Agreement pairing raises OR above that level by +0.09743/+0.05682/+0.11006, less than the base-to-neutral gap in every model, which is why even the syc arm lies below base (Table 2, p. 7). The matched comparison isolates the stance–response association; comparisons with base or the no-stance SFT control change exposure together with pairing.

Higher chosen correctness summarizes only part of the DPO signal: the gains are 8/5/20 responses out of 512 (§4.2, p. 7), and chosen/rejected responses share an answer in 76–85% of Qwen9 pairs, so answer-level contrasts distinguish only the minority whose answers differ while DPO learns from both full texts. OF anchors the odd component to base, which is not uniformly less stance-sensitive: on wrong-stance prompts, the mean odd-score advantage of agreeing over non-agreeing candidates, in score units, is lower for base than live in every Qwen9 seed (0.334/0.336/0.360 versus 0.569/0.520/0.588), but higher for base in two Llama8 seeds (Appendix C.3, p. 25). H2 is +3.55 pp with mixed seed signs (+5.21/+5.71/−0.29 pp), exact seed-level p = 0.250, and 95% t interval [−4.7, 11.8] pp (§4.3, p. 7; Appendix H.2, Table 36, p. 39). A consistent reduction appears when the stance is removed from the judge input altogether (SF: all six comparisons; Table 19, p. 27), whereas replacing only the learned component leaves the base judge's agreement advantage in the reward and gives mixed signs (§4.3, pp. 7–8). SF can also change the even component, so this contrast is descriptive; it indicates when a reward-side intervention lowered Dep in these runs.

3. Response failure, answer validation, and supported conclusions (Q3/W3)

BSR has lower measured dependence in every seed on both the full cohorts and naturally ended, extracted subsets, and the human labels retain all 70 extracted answers among the 100 annotated responses. When all four responses per item (both stances, both paths) end naturally and are extracted, H3 is +14.11/+8.88/+5.95 pp on 535/1,058/370 items, versus +14.93/+7.89/+9.43 pp on the full cohorts (Table 3, p. 8; §4.4, p. 9; Appendix F.4, Table 26, p. 32). The gap remains among items whose responses end naturally and yield extracted answers in both paths. The smaller seed-2 gap leaves room for output failure to contribute to its full-cohort value; these post-treatment subsets do not identify the full-cohort effect.

For stronger answer validation, two human annotators read the seed-2 responses most exposed to truncation: 100 responses from 25 items that include the 38 sampled length-terminated BSR4 outputs. The extractor did not misread any answer it had found (all 70 were confirmed), and the annotators identified answers in 17 responses the extractor had left blank, 11 of them from BSR4. Re-weighting H3 with these labels raises the seed-2 estimate from +9.43 to +11.77 pp, so the answers recovered from truncated outputs do not erase the BSR advantage; the seed-2 measurement-only interval, [−8.84, 32.38] pp, remains wide (§4.4, p. 9; Appendix F.5, Table 28, p. 33).

The fixed-response likelihood contrast also favors BSR in every seed without new generation or answer extraction (Appendix G.1, Table 29, p. 34). The capability criterion is unmet: −1.35 pp [−12.57, 4.70]; seed 2 −12.73 pp (§4.4, p. 9; Tables 1 and 3, pp. 6 and 8).

Under these measurement uncertainties, the supported conclusions are: (a) the Exp 1 ordering, its opinion-question replication, and the no-stance SFT control, which use judge scores on fixed responses and extract no answers from trained-model outputs; (b) the round-2 change in selected pairs under component replacement, which rests on identical candidates and live scores; (c) lower measured dependence under BSR in every seed, including on naturally ended, extracted items; and (d) lower next-update dependence under stance-free judging in all six comparisons, also on naturally ended, extracted items (Appendix D.3, Table 19, p. 27). Capability preservation and superiority of BSR over same-stance averaging or stance-free judging remain unresolved (§4.4, p. 9; §5, p. 10).

4. Relation to prior self-rewarding studies (Q1/W1)

Prior self-rewarding studies assess the reused judge by agreement with human or stronger-model rankings across iterations (Yuan et al., 2024; Wu et al., 2025) or by ranking consistency with the previous iteration and accuracy on labeled preferences (Wang et al., 2025). These before–after and accuracy comparisons summarize a judge's total change, which can also include shifts from fine-tuning unrelated to evaluation (Panickssery et al., 2024). In the present runs, such an audit would read every stance-bearing judge as less agreement-prone than base, whereas the matched-exposure comparison shows that agreement pairing raised agreement preference in all nine model–seed runs. The second finding complements evidence that the effect of a preference pair depends on the model being trained (Zhang et al., 2026): selected-pair diagnostics did not determine next-update dependence here. Both points are now stated in §2 (p. 3) and §5 (pp. 9–10).

Thank you again for the constructive questions, which substantially improved the clarity of the paper.


References

  • Yuan, W. et al. Self-Rewarding Language Models. ICML 2024.
  • Wu, T. et al. Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge. EMNLP 2025.
  • Wang, Z. et al. CREAM: Consistency Regularized Self-Rewarding Language Models. ICLR 2025.
  • Panickssery, A., Bowman, S. R., and Feng, S. LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024.
  • Zhang, Z. et al. Towards Understanding Valuable Preference Data for Large Language Model Alignment. ICLR 2026.
AuthorsOpenReview ↗

Reminder: Request for Feedback on Our Rebuttal

Thank you again for your review. As the discussion period is coming to a close, we wanted to briefly follow up on our rebuttal.

If you have a chance to review our response, we would appreciate any further feedback, particularly if you have any remaining concerns. We would also appreciate your consideration of our response in your final assessment.

Thank you for your time and consideration.

Reviewer_pv2HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how policy fine-tuning and reward-component interventions affect judge preferences and downstream behavior in self-reward LLMs. Using MMLU-Pro questions with user-stance hints, the authors run two main experiments to show that 1) SFT on stance-paird QA dataset shifts models' judge preferences even without judge-format training data, and 2) decomposing judge score into a stance-even component and stance-odd component during iterative DPO.

Critical strengths
  • The experiments are cleanly designed in Experiment 1. Matching the prompt and response marginals in Exp 1 isolates the effect of stance-response pairing on the judge. The transferring effect vs base model is very interesting.

  • The ablation is good and well-controlled. For example, the round-2 branching experiment cleanly isolates how much the stance sensitive reward component changes DPO pair selection.

Critical weaknesses
  • The paper's core method of self-judging and BSR averaging cannot seem to be applied in open-ended tasks where no ground-truth answer exists.

  • The most basic baseline seems to be missing from the study by simply stripping the user's hint from the judge prompt which removes prompt-side stance bias in a single call. Without this baseline, it's unclear why we need BSR's judge calls or synthetic stance flipping

Questions for authors

Why not simply evaluate candidate responses with the stance-free prompt when judging, rather than feeding the user's stance to the judge and averaging over artificial stance flips (BSR)?

2 comments

Thank you for the careful review and for recognizing the clean design of Experiment 1 and the well-controlled round-2 comparison. The question about stance-free judging was especially helpful. Each comment is answered point by point below, and the corresponding revisions are incorporated in the updated PDF, with new analyses and substantive additions highlighted in blue.

1. Requested stance-free baseline (question and W2)

Thank you for suggesting this baseline. The stance-free (SF) judge is the most direct baseline when the stance is separable, and the revision ran it: SF lowered next-update dependence relative to the live-judge path (V2) in all six model–seed comparisons (§4.3, p. 8; Appendix D.3, Table 19, p. 27).

This post hoc baseline removes the user's stated answer from the judge prompt in one call. SF shares the checkpoint, candidates, and prompts with V and, in both models, reuses V2's adapter-initialization seed (§3.3, p. 5; Appendix D.3, p. 27). Dep is lower than under V2 by:

ModelSeed 0 (pp)Seed 1 (pp)Seed 2 (pp)
Qwen3.5-9B6.437.965.96
Llama-3.1-8B-Instruct13.1813.5011.86

Qwen9 V2 values come from the original evaluation; Llama8 V2 was re-evaluated alongside SF2. The gap persists on items whose four responses end naturally and yield extracted answers in all six comparisons.

The capability comparisons use 4,832 items and a separate comparator for each model: odd-component replacement (OF2) for Qwen9 and V2 for Llama8. SF2 accuracy is within 0.66 pp of the comparator without a stance and differs from it by −0.29 to +2.89 pp under stance. These differences are descriptive; they establish neither noninferiority nor an accuracy gain (§4.3, p. 8; Appendix D.3, Table 19, p. 27).

SF2 Dep is also below that of OF2 in all six comparisons, a descriptive comparison (Appendix D.3, p. 28). The abstract (p. 1), the contribution list (p. 2), the Discussion (§5, p. 10), and the Conclusion (p. 10) now state the stance-free result.

2. Why the stance flip is still used

A judge that never sees the stance cannot show whether policy training changed how that judge responds to a stance, which is the question of Experiment 1 and the target of the round-2 replacement (Appendix D.3, p. 28). Stance flipping is therefore retained as a measurement instrument: it measures how the judge's score changes with the stated answer and lets OF anchor that component to base, isolating the learned change while fixing the remaining score, checkpoint, and candidates. This is the round-2 comparison your review praised for cleanly isolating changes in pair selection. SF removes base and learned stance sensitivity together and can also change the remaining score; V−OF isolates the learned change, whose next-update effect varies in sign across seeds (§3.3, pp. 4–5; §4.3, pp. 7–9; Appendix D.3, p. 28).

BSR is reported as a probe of accumulated intervention effects: it removes the odd component at every round from base, while SF trains one update from the shared C1 (§3.3, p. 5; §5, p. 10; Appendix D.3, p. 28). Section 4.4 (p. 9) reports lower measured dependence under BSR with the capability criterion unmet, and §5 (p. 10) states that BSR and stance-free judging are not compared at a matched state.

3. Open-ended tasks and questions without a gold answer (W1)

Neither the judge call nor the even/odd decomposition uses a gold label; stance averaging requires a stated position and a defined alternative that can be exchanged while holding the task fixed (§3.3, p. 4). The revision applies the Exp 1 judge readout to two-option opinion questions without a gold answer: the same 27 trained checkpoints judge fixed responses to 89/97/95 questions for Qwen9/Qwen4/Llama8, and agreement-paired > neutral > disagreement-paired holds in all nine model–seed combinations, with syc−neutral means +0.134/+0.086/+0.124 (§3.2, p. 4; §4.1, p. 7; Appendix B.3, pp. 21–23; Table 12(b), p. 22).

This extension is a judge audit of fixed responses; its scope is narrower than general open-ended self-rewarding (§5, p. 10). Thank you for raising this point, which led to the opinion-question audit.

4. New control for Experiment 1

The transfer result relative to base, listed as a strength in your review, now has a direct control. The new post hoc no-stance SFT control uses the same 4,000 responses with the stance sentence omitted and leaves OR at 0.260/0.285/0.260 for Qwen9 (base 0.174) and 0.473/0.449/0.444 for Llama8 (base 0.486), whereas all three stance-bearing arms fall below base in every seed (§3.2, p. 4; §4.1, pp. 6–7; Appendix B.3, Table 12(c), pp. 21–22). The decrease is absent in Qwen9 and much smaller in Llama8 when the stance sentence is omitted, and the matched neutral arm isolates the pairing effect among the stance-bearing arms.

AuthorsOpenReview ↗

Reminder: Request for Feedback on Our Rebuttal

Thank you again for your review. As the discussion period is coming to a close, we wanted to briefly follow up on our rebuttal.

If you have a chance to review our response, we would appreciate any further feedback, particularly if you have any remaining concerns. We would also appreciate your consideration of our response in your final assessment.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.