- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies score saturation in iterative self-rewarding language models trained with DPO and discrete rubric-based feedback. Under max–min preference selection, it decomposes the round-start update into a component associated with increasing top-score probability and a residual. It then separates score changes over a completed round into actor and judge gains, and gives sufficient conditions for positive gains. The paper also relates score concentration to the declining availability of non-tied preference pairs and derives the sampling budget needed to maintain their expected supply. Experiments with Llama-3-8B-Instruct on OASST compare self-judge and fixed-judge training, supplemented by gradient, preference-allocation, and sampling diagnostics.
Critical strengths
- The analysis makes several useful distinctions that are easy to overlook in self-rewarding systems. In particular, Theorem 1 separates an allocation-determined component of the DPO update from the residual, while Proposition 4 shows how within-score redistribution separates preference-margin improvement from changes in score-level probability. The paper is also careful to distinguish absolute gradient approximation from directional alignment.
Critical weaknesses
-
The theoretical conditions are not shown to explain the observed training dynamics. Theorem 1 identifies a component of the update that favors top-score probability, but the sign of the full update still depends on the residual. The paper acknowledges this, yet it remains unclear when the subsequent sufficient conditions provide useful control. Theorem 2 depends on within-score KL terms and allocation-dependent loss bounds, and Theorem 3 requires alignment between the completed update and the judge's sensitivity. These quantities are not evaluated in the reported experiments. This matters especially for the large early judge gain in Figure 1G, which drives much of the observed concentration. By definition, Thus, the saturation threshold and the exponential budget dependence on characterize consequences of the realized score distribution; their ability to explain training rests on understanding how those gains arise. Appendix D extends the analysis beyond initialization, but introduces further drift and geometric conditions whose usefulness in the experimental setting is unclear. The limitations are stated in Appendix B, but I would still like to see a concrete regime or quantitative evidence showing that the conditions explain the observed gains. This is my main reservation about the depth of the theoretical contribution.
-
The training procedure differs from the preference-selection rule used in the proofs. Section 2 assumes strict max–min selection with uniform tie-breaking. Appendix C.1, lines 868–879, instead describes averaging five scores, choosing a response using length control among candidates within of the maximum mean score, and applying additional filtering. Averaging could be handled by redefining the score space, but its correspondence with the five-category diagnostics in Appendix C.1.1 needs clarification. The length-based and near-maximum selection rules are more consequential: they can change the selected score pair and the distribution of responses selected within a score category. The factorization in Proposition 1, and the gradient decomposition that relies on it, therefore do not automatically describe the implemented pipeline. This leaves a gap between the analyzed mechanism and the training experiments. A controlled comparison using the exact theoretical selector would help establish whether the main findings persist when these assumptions are respected.
-
The budget experiment relies on derived quantities rather than an independent measurement of the predicted supply. Appendix C.2 estimates the score frequencies and then calculates
Questions for authors
-
Which of the sufficient conditions in Theorems 2–3 or Appendix D are satisfied, or yield useful bounds, along the reported trajectories? Could you provide quantitative evidence from the existing checkpoints, or a nontrivial setting in which the conditions can be established? In particular, can the analysis account for the large early judge gain in Figure 1G?
-
Please clarify whether each diagnostic uses individual integer scores, averaged scores, or discretized averages, and report how often the length-controlled selector differs from strict max–min selection. Could you include a controlled experiment using the exact selection rule in Section 2, with uniform tie-breaking and consistent score definitions? Keeping the initial checkpoint, prompts, group size, and optimization settings fixed, the relevant comparisons would be top-score probability, actor/judge gains, and accepted-pair counts across rounds.
-
At the existing fixed prompts and checkpoints , could you estimate acceptance probabilities from one candidate set and test the predictions on independently generated groups? Comparing predicted and observed acceptance rates, and accepted-pair counts at budgets chosen using , would directly address the budget-validation concern. Please include uncertainty estimates and distinguish the strict theoretical selector from the practical data builder.