acceptodds
Under review as a conference paper at ICLR 2027

When Higher Scores Yield Less Feedback: Score Saturation in Self-Rewarding Language Models

Abstract

Self-rewarding language models can improve through an iterative feedback loop in which the model generates multiple responses, evaluates them using a discrete scoring rubric, and trains on the resulting preferences. A known failure mode is , where generated responses increasingly receive scores near the top of the rubric, producing more ties and fewer informative comparisons. We develop a theoretical analysis of when and how preference optimization contributes to saturation and reshapes the feedback available to subsequent rounds. We first show that the preference optimization update direction inherently contains a component that increases the probability of the highest score, while other components of the preference gradient can either reinforce or counteract this effect. We further decompose score changes into effects arising from changes in the model's response distribution and changes in its evaluation behavior, and derive conditions under which these effects accumulate to produce score saturation. As saturation progresses, the preference comparisons increasingly reflect the gradient structure associated with increasing top-score probability, even as informative comparisons become increasingly rare. Consequently, maintaining a fixed number of them requires a sampling budget that grows inversely with the remaining probability of a non-top score, and asymptotically exponentially in the accumulated gains. Experiments on self-rewarding language models validate these theoretical results.

Primary area:
learning theory
Keywords:
LLM-as-a-Judge, Preference learning, Self-rewarding language models, Score saturation, Direct Preference Optimization(DPO)

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_s9y9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies score saturation in iterative self-rewarding language models trained with DPO and discrete rubric-based feedback. Under max–min preference selection, it decomposes the round-start update into a component associated with increasing top-score probability and a residual. It then separates score changes over a completed round into actor and judge gains, and gives sufficient conditions for positive gains. The paper also relates score concentration to the declining availability of non-tied preference pairs and derives the sampling budget needed to maintain their expected supply. Experiments with Llama-3-8B-Instruct on OASST compare self-judge and fixed-judge training, supplemented by gradient, preference-allocation, and sampling diagnostics.

Critical strengths
  1. The analysis makes several useful distinctions that are easy to overlook in self-rewarding systems. In particular, Theorem 1 separates an allocation-determined component of the DPO update from the residual, while Proposition 4 shows how within-score redistribution separates preference-margin improvement from changes in score-level probability. The paper is also careful to distinguish absolute gradient approximation from directional alignment.
Critical weaknesses
  1. The theoretical conditions are not shown to explain the observed training dynamics. Theorem 1 identifies a component of the update that favors top-score probability, but the sign of the full update still depends on the residual. The paper acknowledges this, yet it remains unclear when the subsequent sufficient conditions provide useful control. Theorem 2 depends on within-score KL terms and allocation-dependent loss bounds, and Theorem 3 requires alignment between the completed update and the judge's sensitivity. These quantities are not evaluated in the reported experiments. This matters especially for the large early judge gain in Figure 1G, which drives much of the observed concentration. By definition, ΛT=logit⁡(pT,L)−logit⁡(p0,L).\Lambda_T=\operatorname{logit}(p_{T,L})-\operatorname{logit}(p_{0,L}). Thus, the saturation threshold and the exponential budget dependence on ΛT\Lambda_T characterize consequences of the realized score distribution; their ability to explain training rests on understanding how those gains arise. Appendix D extends the analysis beyond initialization, but introduces further drift and geometric conditions whose usefulness in the experimental setting is unclear. The limitations are stated in Appendix B, but I would still like to see a concrete regime or quantitative evidence showing that the conditions explain the observed gains. This is my main reservation about the depth of the theoretical contribution.

  2. The training procedure differs from the preference-selection rule used in the proofs. Section 2 assumes strict max–min selection with uniform tie-breaking. Appendix C.1, lines 868–879, instead describes averaging five scores, choosing a response using length control among candidates within 0.250.25 of the maximum mean score, and applying additional filtering. Averaging could be handled by redefining the score space, but its correspondence with the five-category diagnostics in Appendix C.1.1 needs clarification. The length-based and near-maximum selection rules are more consequential: they can change the selected score pair and the distribution of responses selected within a score category. The factorization Γt,m,ℓ(N)=πtm⊗πtℓ\Gamma^{(N)}_{t,m,\ell}=\pi_t^m\otimes\pi_t^\ell in Proposition 1, and the gradient decomposition that relies on it, therefore do not automatically describe the implemented pipeline. This leaves a gap between the analyzed mechanism and the training experiments. A controlled comparison using the exact theoretical selector would help establish whether the main findings persist when these assumptions are respected.

  3. The budget experiment relies on derived quantities rather than an independent measurement of the predicted supply. Appendix C.2 estimates the score frequencies and then calculates

\qquad \widehat B_t^*(100)=\left\lceil\frac{100}{\widehat a_{N,t}}\right\rceil.$$ Figure 3 is useful for illustrating the formulas on empirical distributions, but it does not independently test whether the predicted budgets produce the expected number of accepted pairs. The bootstrap intervals quantify uncertainty in these derived estimates. The independent-set experiment in Appendix C.3 addresses allocation matching, which is a different claim. Since sampling cost is one of the main conclusions, I would like to see predicted acceptance rates and budgets checked against newly sampled groups at the existing checkpoints. This would give more direct evidence for the practical accuracy of the budget predictions.
Questions for authors
  1. Which of the sufficient conditions in Theorems 2–3 or Appendix D are satisfied, or yield useful bounds, along the reported trajectories? Could you provide quantitative evidence from the existing checkpoints, or a nontrivial setting in which the conditions can be established? In particular, can the analysis account for the large early judge gain in Figure 1G?

  2. Please clarify whether each diagnostic uses individual integer scores, averaged scores, or discretized averages, and report how often the length-controlled selector differs from strict max–min selection. Could you include a controlled experiment using the exact selection rule in Section 2, with uniform tie-breaking and consistent score definitions? Keeping the initial checkpoint, prompts, group size, and optimization settings fixed, the relevant comparisons would be top-score probability, actor/judge gains, and accepted-pair counts across rounds.

  3. At the existing fixed prompts and checkpoints t∈{0,1,6}t\in\{0,1,6\}, could you estimate acceptance probabilities from one candidate set and test the predictions on independently generated groups? Comparing predicted and observed acceptance rates, and accepted-pair counts at budgets chosen using B^t∗(M)\widehat B_t^*(M), would directly address the budget-validation concern. Please include uncertainty estimates and distinguish the strict theoretical selector from the practical data builder.

Reviewer_7qHiOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies score saturation in iterative self-rewarding language models. It theoretically analyzes top-score reinforcement, decomposes score evolution into Actor and Judge gains, and argues that score saturation reduces informative preference pairs and increases sampling costs.

Critical strengths
  • Relevant problem. Score saturation may limit the availability of informative preference pairs in iterative self-rewarding systems.

  • Coherent analysis. The Actor–Judge decomposition provides a useful perspective on how response quality and scoring behavior jointly affect score evolution.

Critical weaknesses
  • Limited theoretical novelty. Several conclusions, particularly the reduction in informative preference pairs under score concentration, appear to follow naturally from max–min sampling. The paper does not sufficiently establish what fundamentally new insight its theoretical framework provides.

  • Insufficient connection to practical training dynamics. The local gradient analysis does not by itself establish that top-score probabilities increase or eventually saturate in realistic training. The required conditions are not sufficiently validated empirically.

  • Unclear practical impact. This is the most significant weakness. Even if all theoretical results are correct, it remains unclear how they should change actual training practice. The paper proposes no validated intervention and demonstrates no measurable improvement in model quality, training efficiency, or robustness. Consequently, the work risks characterizing an expected property of the sampling mechanism without establishing a meaningful practical or scientific advance.

  • Limited empirical validation. The experiments cover a narrow setting and do not establish generality across models, tasks, or scoring configurations. The relationship between score saturation and actual model capability or training degradation also remains unclear.

Questions for authors
  1. Even if the theory is entirely correct, what concrete decisions should practitioners change when training self-rewarding models? Can the analysis lead to a validated method that improves model quality or reduces training costs?

  2. How do the authors distinguish genuine capability improvement from changes in Judge behavior when the top-score probability increases?

  3. How novel are the theoretical results beyond the expected consequences of max–min sampling under a concentrated discrete score distribution?

  4. Can the authors validate the predicted dynamics across different models and tasks, and demonstrate a measurable impact on downstream performance or training efficiency?

Reviewer_weA8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies score saturation in self-rewarding language models that construct preference pairs from discrete rubric scores using Max-min selection and DPO training. It derives an allocation-based decomposition of the round-start DPO direction, separates completed-round changes in top-score log odds into actor and judge gains. Under fixed candidate-group size and diverging cumulative gains, the required budget for a fixed expected preference count grows exponentially in those gains. Experiments with Llama-3-8B-Instruct on OASST compare self-judging and fixed-judging over eight rounds, validating the theoretical assumptions.

Critical strengths
  1. This paper defines score saturation as concentration in the on-policy actor–judge score distribution and explicitly separates it from response quality and judging accuracy which is a new aspect in terms of theory in self-rewarding area.
  2. This paper provides a clear explanation of the relationship between score saturation and preference allocation, and the conditions under which DPO can increase top-score probability.
  3. I believe the theoretical analysis in Appendix is mostly correct where I used AI to check.
Critical weaknesses
  1. The empirical setup is relatively weak: using one model, one corpus and evaluating on the same one benchmark. Would like to see more results on downstream benchmarks (I think evaluation is relatively cheap? no need to re-train any models), also comparison with existing methods for mitigating score saturation in self-rewarding. Would also like to see the behavior analysis of those methods in terms of the proposed allocation decomposition and actor-gain analysis.
  2. This paper does not measure response quality, so the harm of saturation is established only as a reduction in usable preference pairs. A win-rate or human-preference comparison between round-0 and the most saturated checkpoint would show whether the loss of feedback actually degrades outputs, which is the motivation stated in the introduction.
  3. This paper lacks analysis on the distribution of trajectories and DPO data pairs when uses exact max–min rule with uniform endpoint tie-breaking.
Questions for authors

Please see the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.