- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the fragility of self-rewarding optimization (SRO), where the same language model acts as both an answer generator and a reward generator. The authors first introduce two prompt-level attacks, Re-Ranking and Preamble, which manipulate the training-prompt distribution while leaving the SRO algorithm unchanged. These attacks substantially degrade answer quality across multiple models and two SRO methods, suggesting that SRO can be highly sensitive to the reliability of its own reward signal.
Critical strengths
- The self-rewarding collapse is an important reearch topic.
- The empirical failure mode is strong and easy to interpret. The proposed attacks are simple at the prompt level, yet they cause large degradation across several model families and SRO algorithms.
- The frozen-judge analysis is informative, which provides useful evidence that evaluator co-evolution amplifies initial preference corruption, rather than the observed collapse being solely due to noisy preference optimization.
Critical weaknesses
Major Question
- SRF requires an oracle, which limits its practical role. The appendix contains oracle-free attack variants, but the paper does not provide a convincing oracle-free estimator of SRF that could be used to detect impending collapse during normal SRO training.
- The protected subspace is constructed from a small fixed probe set sampled from the same training dataset as the SRO task. It is unclear whether the method would remain effective under probe-distribution shift. For example, a subspace estimated from safety judgments such as SAFE/UNSAFE may not span the same directions as a reward generator judging mathematical correctness.
- Computational overhead is reported only in absolute terms, in addition, if I understand correctly, one SRO step is one training step, which will introduce a large additional overhead, more detail or normalized cost would be more informative.
- The clean-training results suggest that this trade-off is modest at but the degradation in clean AQ/SRF for larger indeed indicates reduced plasticity. It would be useful to analyze what fraction of projected components correspond to beneficial versus harmful changes in reward fidelity, or to consider an adaptive/sign-aware projection mechanism.
- The relationship between SRF and downstream performance is not uniformly tight. For example, on MMLU-Pro, some stronger Qwen models maintain relatively high SRF even while AQ deteriorates substantially under attack.
- The paper does not discuss [1], which is highly relevant because it also studies collapse and reward hacking in self-rewarding training and proposes complementary supervision to stabilize the loop. The current paper should more clearly distinguish its SRF-based failure analysis and Projection Decoupling Defense from Co-rewarding’s self-consistent-illusion perspective and multi-view/teacher-based stabilization.
Minor Question
- The “question difficulty” definition in Eq. (1) is misleading. This equation quantity does not measure the difficulty of the question for the answer generator. It measures the disagreement between the oracle and the self-reward generator on policy-generated answers.
- The “phase transition” claim appears stronger than the current evidence. The paper states that SRO exhibits a phase-transition-like behavior around an empirical SRF threshold, approximately 76.3% in one setting. However, this value appears to be inferred from a relatively coarse sweep of corruption levels rather than from a formal change-point analysis.
[1] Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026)
Questions for authors
See weakness
6 comments
The clean-training results suggest that this trade-off is modest at but the degradation in clean AQ/SRF for larger indeed indicates reduced plasticity. It would be useful to analyze what fraction of projected components correspond to beneficial versus harmful changes in reward fidelity, or to consider an adaptive/sign-aware projection mechanism.
Following the reviewer’s suggestion, we classify the iteration-3 Adam updates (Llama-3.1-8B, SRLM, SafeRLHF, and 16 probes) according to their first-order effect on the reward-fidelity surrogate loss. Components that decrease the loss are classified as beneficial, those that increase it as harmful, and those with near-zero or uncertain effects as unresolved. For this analysis only, we use a separate, fixed set of oracle-labeled responses. As shown in the table below, harmful components account for 76.0-77.3% of the update under the Re-Ranking attack, whereas beneficial components account for 43.5-45.2% in clean training. Also, the results show a security-utility trade-off. For instance, raising $\alpha$ from 0.5 to 1 improves attacked AQ by 0.3% but lowers clean AQ by 1.4%.
| Training condition | α | Beneficial (%) | Harmful (%) | Unresolved (%) | Iter. 3 AQ (%) | ΔAQ vs. α = 0 (%) |
|---|---|---|---|---|---|---|
| Clean | 0.5 | 45.2 | 40.1 | 14.7 | 91.9 | −0.3 |
| Clean | 0.75 | 43.9 | 41.2 | 14.9 | 90.8 | −1.4 |
| Clean | 1.0 | 43.5 | 41.5 | 15.0 | 90.5 | −1.7 |
| Re-Ranking | 0.5 | 15.1 | 76.0 | 8.9 | 85.2 | +10.7 |
| Re-Ranking | 0.75 | 14.6 | 76.9 | 8.5 | 85.4 | +10.9 |
| Re-Ranking | 1.0 | 14.4 | 77.3 | 8.3 | 85.5 | +11.0 |
AQ is measured at the end of iteration 3; $\alpha$ = 0 baselines are 92.2% (clean) and 74.5% (Re-Ranking).
We also propose an adaptive projection mechanism. We calculate a new $alpha$ for step $s$ as $\alpha_s = \frac{u_s}{1+u_s} $, where $u_s$ is the average sensitivity of the protected directions, weighted by how much of the update moves along directions that change the model's self-evaluation. We detail this formulation in Appendix D.6. Intuitively, more sensitive updates receive stronger projection. This adaptive mechanism requires no external labels or drift-budget calibration. The table below reports iteration-3 AQ/SRF (%) and an average $\alpha$ across the training steps.
Observe that compared with fixed = 0.5, adaptive projection increases attacked AQ/SRF by 0.7/0.8%, while clean AQ/SRF only decreases by 0.2/0.1%. The results indicate that adaptive projection is a promising next step for our proposed defense.
| Method | Clean final AQ | Clean final SRF | Re-Ranking final AQ | Re-Ranking final SRF | Re-Ranking mean α |
|---|---|---|---|---|---|
| Standard SRO | 92.2 | 91.1 | 74.5 | 74.4 | 0 |
| Fixed \alpha=0.5 | 91.9 | 89.9 | 85.2 | 82.2 | 0.5 |
| Adaptive | 91.7 | 89.8 | 85.9 | 83.0 | 0.67 |
The relationship between SRF and downstream performance is not uniformly tight. For example, on MMLU-Pro, some stronger Qwen models maintain relatively high SRF even while AQ deteriorates substantially under attack.
Across our experiments with 4 models, 3 domains, and 2 attacks, SRF generally tracks AQ, with one exception: Qwen3-4B on MMLU-Pro under CREAM. A possible explanation is that the pretraining of the stronger Qwen model supports robust correctness judgments on MMLU-Pro, and CREAM’s reward-consistency regularization also helps preserve these judgments across iterations. Meanwhile, attack-driven updates may shift the generation policy away from behaviors that effectively elicit this knowledge, lowering AQ while SRF remains relatively high. We have revised the abstract and §4.1.1 to describe the empirical trend more precisely and clarify that high SRF alone does not guarantee successful self-improvement.
The paper does not discuss [1], which is highly relevant because it also studies collapse and reward hacking in self-rewarding training and proposes complementary supervision to stabilize the loop. The current paper should more clearly distinguish its SRF-based failure
Thanks for pointing us to this relevant work. We now cite it and discuss its relationship to our work in §2 of the revison.
Specifically, Co-rewarding attributes collapse to a self-consistent illusion, where confidence or consistency rises without gains in correctness. It stabilizes training by changing how rewards are constructed: Co-rewarding-I uses agreement across semantically equivalent questions, and Co-rewarding-II uses pseudo-labels from an EMA-updated reference model.
In contrast, we study evaluator co-evolution in SRO methods (SRLM, CREAM), where the same model generates and judges answers. Across diverse domains, we show that updates to shared parameters can degrade the evaluator and amplify attack-induced reward corruption. Our Projection Decoupling Defense suppresses updates along directions to which reward judgments are sensitive.
Thus, Co-rewarding improves the source of supervision, while our defense limits evaluator drift, making the two potentially complementary.
The “question difficulty” definition in Eq. (1) is misleading. This equation quantity does not measure the difficulty of the question for the answer generator. It measures the disagreement between the oracle and the self-reward generator on policy-generated answers.
We have replaced “question difficulty” with “reward disagreement score” throughout the paper, and clarified that higher scores indicate greater reward disagreement on answers sampled from the current policy.
The “phase transition” claim appears stronger than the current evidence. The paper states that SRO exhibits a phase-transition-like behavior around an empirical SRF threshold, approximately 76.3% in one setting. However, this value appears to be inferred from a relatively coarse sweep of corruption levels rather than from a formal change-point analysis.
In the revision, we have updated “phase transition” to “phase-transition region”, over which SRO behavior shifts from improvement toward degradation as initial SRF decreases. We use this term descriptively, without claiming a formally established phase transition or a precisely identified critical threshold. In particular, 76.3% is a sampled initial SRF value within this region, rather than a statistically estimated critical point. We have also clarified that this region depends on the model and task in §4.1.1.
Again, we thank the reviewer for the valuable feedback and welcome any further suggestions or comments.
Authors
We thank the reviewer for the insightful comments. Below, we respond to each of the reviewer’s questions, and we have incorporated the corresponding changes into the revised manuscript.
We propose SRF primarily as a metric for explaining the collapse phenomenon in SRO training, linking the initial SRF to subsequent improvement or collapse. In practice, SRF can detect potential collapse by estimating it on a small amount of labeled data (annotated by humans or a stronger model), rather than on the large pool of unlabeled training data. To assess how reliable such estimates are, the table below compares the SRF from full evaluation with estimates based on 32–256 oracle-labeled responses, for Llama-3.1-8B with SRLM on SafeRLHF at iteration 3 with 10 repeats. Standard deviations are shown in parentheses.
Observe that with 256 labels, the mean estimated SRF is within 0.1% of the full-evaluation reference, and the standard deviation stays around 2.0% in both conditions. Even with only 32 labels, the estimates still separate the clean and attack conditions. These results suggest that SRF can be applied in practice through low-budget labeling.
Thank you for the insightful comment. Different domains may draw on different model capabilities (e.g., mathematical reasoning vs. safety reasoning) and thus activate different neurons. As a result, probe gradients estimated in one domain may transfer only partially to other domains. The table below shows the transferability of probes from SafeRLHF to other domains (measured by AQ/SRF) under the Re-Ranking attack: they improve AQ by 2.8% on MMLU-Pro and 3.2% on ARC, compared with 10.7% on SafeRLHF itself. We therefore recommend using probes that are representative of the target domain.
To clarify, each SRO round consists of multiple optimizer steps. As specified in Algorithm 1 and Appendix E.3, the probe gradients and the projection basis are computed once per round and then reused to filter the Adam update at every optimizer step within that round. As shown in the table below, the default setting of 16 probes adds only 58.0s per SRO round, increasing the total runtime per round from 2395.4s to 2453.4s (a 2.42% overhead). With 8 and 32 probes, the overhead is 1.25% and 4.53%, respectively.