acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Mitigating the Fragility of Self-Rewarding Optimization

Abstract

Self-rewarding optimization (SRO) promises to improve language models by enabling them to generate their own reward signals. Yet when and how SRO succeeds or fails remains poorly understood. We reveal that SRO is fundamentally fragile: even seemingly benign perturbations to training prompts can corrupt the self-generated preference signal and trigger iterative collapse. We demonstrate this through two novel, prompt-level attacks: a Re-Ranking attack that concentrates training on prompts where self-reward diverges from an oracle, and a Preamble attack that learns short prefixes inflating self-reward without any gain in oracle quality. To explain why these attacks succeed, we introduce self-reward fidelity (SRF), a metric quantifying on-policy agreement between self-evaluated and oracle rewards. We show that SRF is tightly coupled to answer quality through shared model parameters, and that controlled SRF perturbations induce a phase-transition behavior: SRO improves when SRF remains sufficiently high, but collapses once SRF drops below a model- and task-dependent threshold. This analysis motivates Projection Decoupling Defense, which suppresses SRO update components along directions to which the reward generator is locally sensitive, improving SRO robustness under adversarial conditions. This work sheds light on the fragility of SRO and charts promising directions toward making it more robust. The artifacts are available: https://anonymous.4open.science/r/SRO-dilemma-5ED7

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Self-Rewarding Optimization, Language model robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cMQGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the fragility of self-rewarding optimization (SRO), where the same language model acts as both an answer generator and a reward generator. The authors first introduce two prompt-level attacks, Re-Ranking and Preamble, which manipulate the training-prompt distribution while leaving the SRO algorithm unchanged. These attacks substantially degrade answer quality across multiple models and two SRO methods, suggesting that SRO can be highly sensitive to the reliability of its own reward signal.

Critical strengths
  1. The self-rewarding collapse is an important reearch topic.
  2. The empirical failure mode is strong and easy to interpret. The proposed attacks are simple at the prompt level, yet they cause large degradation across several model families and SRO algorithms.
  3. The frozen-judge analysis is informative, which provides useful evidence that evaluator co-evolution amplifies initial preference corruption, rather than the observed collapse being solely due to noisy preference optimization.
Critical weaknesses

Major Question

  1. SRF requires an oracle, which limits its practical role. The appendix contains oracle-free attack variants, but the paper does not provide a convincing oracle-free estimator of SRF that could be used to detect impending collapse during normal SRO training.
  2. The protected subspace is constructed from a small fixed probe set sampled from the same training dataset as the SRO task. It is unclear whether the method would remain effective under probe-distribution shift. For example, a subspace estimated from safety judgments such as SAFE/UNSAFE may not span the same directions as a reward generator judging mathematical correctness.
  3. Computational overhead is reported only in absolute terms, in addition, if I understand correctly, one SRO step is one training step, which will introduce a large additional overhead, more detail or normalized cost would be more informative.
  4. The clean-training results suggest that this trade-off is modest at α=0.5\alpha=0.5 but the degradation in clean AQ/SRF for larger α\alpha indeed indicates reduced plasticity. It would be useful to analyze what fraction of projected components correspond to beneficial versus harmful changes in reward fidelity, or to consider an adaptive/sign-aware projection mechanism.
  5. The relationship between SRF and downstream performance is not uniformly tight. For example, on MMLU-Pro, some stronger Qwen models maintain relatively high SRF even while AQ deteriorates substantially under attack.
  6. The paper does not discuss [1], which is highly relevant because it also studies collapse and reward hacking in self-rewarding training and proposes complementary supervision to stabilize the loop. The current paper should more clearly distinguish its SRF-based failure analysis and Projection Decoupling Defense from Co-rewarding’s self-consistent-illusion perspective and multi-view/teacher-based stabilization.

Minor Question

  1. The “question difficulty” definition in Eq. (1) is misleading. This equation quantity does not measure the difficulty of the question for the answer generator. It measures the disagreement between the oracle and the self-reward generator on policy-generated answers.
  2. The “phase transition” claim appears stronger than the current evidence. The paper states that SRO exhibits a phase-transition-like behavior around an empirical SRF threshold, approximately 76.3% in one setting. However, this value appears to be inferred from a relatively coarse sweep of corruption levels rather than from a formal change-point analysis.

[1] Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026)

Questions for authors

See weakness

6 comments

We thank the reviewer for the insightful comments. Below, we respond to each of the reviewer’s questions, and we have incorporated the corresponding changes into the revised manuscript.


SRF requires an oracle, which limits its practical role. The appendix contains oracle-free attack variants, but the paper does not provide a convincing oracle-free estimator of SRF that could be used to detect impending collapse during normal SRO training.

We propose SRF primarily as a metric for explaining the collapse phenomenon in SRO training, linking the initial SRF to subsequent improvement or collapse. In practice, SRF can detect potential collapse by estimating it on a small amount of labeled data (annotated by humans or a stronger model), rather than on the large pool of unlabeled training data. To assess how reliable such estimates are, the table below compares the SRF from full evaluation with estimates based on 32–256 oracle-labeled responses, for Llama-3.1-8B with SRLM on SafeRLHF at iteration 3 with 10 repeats. Standard deviations are shown in parentheses.

Oracle labelsClean SRF (%)Re-Ranking SRF (%)
Full evaluation91.174.4
3290.5 (3.0)73.3 (4.3)
6490.7 (2.2)73.6 (2.9)
12890.9 (1.3)74.6 (2.3)
25691.2 (1.1)74.4 (1.7)

Observe that with 256 labels, the mean estimated SRF is within 0.1% of the full-evaluation reference, and the standard deviation stays around 2.0% in both conditions. Even with only 32 labels, the estimates still separate the clean and attack conditions. These results suggest that SRF can be applied in practice through low-budget labeling.


The protected subspace is constructed from a small fixed probe set sampled from the same training dataset as the SRO task. It is unclear whether the method would remain effective under probe-distribution shift. For example, a subspace estimated from safety judgments such as SAFE/UNSAFE may not span the same directions as a reward generator judging mathematical correctness.

Thank you for the insightful comment. Different domains may draw on different model capabilities (e.g., mathematical reasoning vs. safety reasoning) and thus activate different neurons. As a result, probe gradients estimated in one domain may transfer only partially to other domains. The table below shows the transferability of probes from SafeRLHF to other domains (measured by AQ/SRF) under the Re-Ranking attack: they improve AQ by 2.8% on MMLU-Pro and 3.2% on ARC, compared with 10.7% on SafeRLHF itself. We therefore recommend using probes that are representative of the target domain.

Probe source domain ↓ / Target domain →Safety (SafeRLHF)Knowledge (MMLU-Pro)Reasoning (ARC)
None (no defense)74.5 / 74.427.7 / 32.216.3 / 40.4
Safety (SafeRLHF)85.2 / 82.230.5 / 35.519.5 / 45.1

Computational overhead is reported only in absolute terms, in addition, if I understand correctly, one SRO step is one training step, which will introduce a large additional overhead, more detail or normalized cost would be more informative.

To clarify, each SRO round consists of multiple optimizer steps. As specified in Algorithm 1 and Appendix E.3, the probe gradients and the projection basis are computed once per round and then reused to filter the Adam update at every optimizer step within that round. As shown in the table below, the default setting of 16 probes adds only 58.0s per SRO round, increasing the total runtime per round from 2395.4s to 2453.4s (a 2.42% overhead). With 8 and 32 probes, the overhead is 1.25% and 4.53%, respectively.

MethodProjection computation (s/round)Projection application (s/round)Training time (s/round)Total time (s/round)Peak GPU memory (GB)Peak CPU memory (GB)
Vanilla SRO——2395.42395.466.12.12
Projection Decoupling Defense (8 probes)9.5320.42425.3367.27.06
Projection Decoupling Defense (16 probes)15.5042.52453.4070.315.06
Projection Decoupling Defense (32 probes)33.2075.22503.8075.931.06

The clean-training results suggest that this trade-off is modest at α=0.5\alpha=0.5 but the degradation in clean AQ/SRF for larger α\alpha indeed indicates reduced plasticity. It would be useful to analyze what fraction of projected components correspond to beneficial versus harmful changes in reward fidelity, or to consider an adaptive/sign-aware projection mechanism.

Following the reviewer’s suggestion, we classify the iteration-3 Adam updates (Llama-3.1-8B, SRLM, SafeRLHF, and 16 probes) according to their first-order effect on the reward-fidelity surrogate loss. Components that decrease the loss are classified as beneficial, those that increase it as harmful, and those with near-zero or uncertain effects as unresolved. For this analysis only, we use a separate, fixed set of oracle-labeled responses. As shown in the table below, harmful components account for 76.0-77.3% of the update under the Re-Ranking attack, whereas beneficial components account for 43.5-45.2% in clean training. Also, the results show a security-utility trade-off. For instance, raising $\alpha$ from 0.5 to 1 improves attacked AQ by 0.3% but lowers clean AQ by 1.4%.

Training conditionαBeneficial (%)Harmful (%)Unresolved (%)Iter. 3 AQ (%)ΔAQ vs. α = 0 (%)
Clean0.545.240.114.791.9−0.3
Clean0.7543.941.214.990.8−1.4
Clean1.043.541.515.090.5−1.7
Re-Ranking0.515.176.08.985.2+10.7
Re-Ranking0.7514.676.98.585.4+10.9
Re-Ranking1.014.477.38.385.5+11.0

AQ is measured at the end of iteration 3; $\alpha$ = 0 baselines are 92.2% (clean) and 74.5% (Re-Ranking).

We also propose an adaptive projection mechanism. We calculate a new $alpha$ for step $s$ as $\alpha_s = \frac{u_s}{1+u_s} $, where $u_s$ is the average sensitivity of the protected directions, weighted by how much of the update moves along directions that change the model's self-evaluation. We detail this formulation in Appendix D.6. Intuitively, more sensitive updates receive stronger projection. This adaptive mechanism requires no external labels or drift-budget calibration. The table below reports iteration-3 AQ/SRF (%) and an average $\alpha$ across the training steps.

Observe that compared with fixed α\alpha = 0.5, adaptive projection increases attacked AQ/SRF by 0.7/0.8%, while clean AQ/SRF only decreases by 0.2/0.1%. The results indicate that adaptive projection is a promising next step for our proposed defense.

MethodClean final AQClean final SRFRe-Ranking final AQRe-Ranking final SRFRe-Ranking mean α
Standard SRO92.291.174.574.40
Fixed \alpha=0.591.989.985.282.20.5
Adaptive91.789.885.983.00.67

The relationship between SRF and downstream performance is not uniformly tight. For example, on MMLU-Pro, some stronger Qwen models maintain relatively high SRF even while AQ deteriorates substantially under attack.

Across our experiments with 4 models, 3 domains, and 2 attacks, SRF generally tracks AQ, with one exception: Qwen3-4B on MMLU-Pro under CREAM. A possible explanation is that the pretraining of the stronger Qwen model supports robust correctness judgments on MMLU-Pro, and CREAM’s reward-consistency regularization also helps preserve these judgments across iterations. Meanwhile, attack-driven updates may shift the generation policy away from behaviors that effectively elicit this knowledge, lowering AQ while SRF remains relatively high. We have revised the abstract and §4.1.1 to describe the empirical trend more precisely and clarify that high SRF alone does not guarantee successful self-improvement.

The paper does not discuss [1], which is highly relevant because it also studies collapse and reward hacking in self-rewarding training and proposes complementary supervision to stabilize the loop. The current paper should more clearly distinguish its SRF-based failure

Thanks for pointing us to this relevant work. We now cite it and discuss its relationship to our work in §2 of the revison.

Specifically, Co-rewarding attributes collapse to a self-consistent illusion, where confidence or consistency rises without gains in correctness. It stabilizes training by changing how rewards are constructed: Co-rewarding-I uses agreement across semantically equivalent questions, and Co-rewarding-II uses pseudo-labels from an EMA-updated reference model.

In contrast, we study evaluator co-evolution in SRO methods (SRLM, CREAM), where the same model generates and judges answers. Across diverse domains, we show that updates to shared parameters can degrade the evaluator and amplify attack-induced reward corruption. Our Projection Decoupling Defense suppresses updates along directions to which reward judgments are sensitive.

Thus, Co-rewarding improves the source of supervision, while our defense limits evaluator drift, making the two potentially complementary.


The “question difficulty” definition in Eq. (1) is misleading. This equation quantity does not measure the difficulty of the question for the answer generator. It measures the disagreement between the oracle and the self-reward generator on policy-generated answers.

We have replaced “question difficulty” with “reward disagreement score” throughout the paper, and clarified that higher scores indicate greater reward disagreement on answers sampled from the current policy.


The “phase transition” claim appears stronger than the current evidence. The paper states that SRO exhibits a phase-transition-like behavior around an empirical SRF threshold, approximately 76.3% in one setting. However, this value appears to be inferred from a relatively coarse sweep of corruption levels rather than from a formal change-point analysis.

In the revision, we have updated “phase transition” to “phase-transition region”, over which SRO behavior shifts from improvement toward degradation as initial SRF decreases. We use this term descriptively, without claiming a formally established phase transition or a precisely identified critical threshold. In particular, 76.3% is a sampled initial SRF value within this region, rather than a statistically estimated critical point. We have also clarified that this region depends on the model and task in §4.1.1.


Again, we thank the reviewer for the valuable feedback and welcome any further suggestions or comments.

Authors

Reviewer_HW9rOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the fragility of self-rewarding optimization. It introduces two prompt-level attacks and self-reward fidelity (SRF). SRF is a measure of agreement between self-generated and oracle rewards. Through controlled reward-label perturbations, frozen-judge controls, and local gradient analysis, the paper examines how unreliable feedback can be during training. It further proposes Projection Decoupling Defense, which suppresses self-rewarding optimization update components along directions to which the reward generator is locally sensitive.

Critical strengths
  1. The paper focuses on an interesting problem and is overall well-organized. It examines the risks when training changes the evaluator that provides following supervision. After discovering the overall issues, it proposes SRF to explain the potential causes.

  2. The attacks are evaluated across multiple settings. Experiments cover four models, two SRO methods, and safety, knowledge, and reasoning tasks. Additional evaluations consider oracle-free attacks, cross-model transfer, and sensitivity to evaluator choice.

  3. The method is intuitive and supported by comparative experiments. Comparisons with random projection and a frozen judge help assess the effectiveness of the proposed defense.

Critical weaknesses
  1. The label-corruption experiment (Lines 299–310) does not separate the damage caused by incorrect labels from the additional damage caused by the self-rewarding loop. Appendix G.2 seems to provide a frozen-judge control (Lines 1495–1511), but the main text does not explicitly reference or discuss these results. Thus, the whole description of this part can be a bit confusing and the corresponding analysis can be a bit abrupt due to the lack of references.

  2. Lines 1071–1078 report the additional probe computation time, but it is unclear whether this includes constructing the projection basis and applying the projection at every training step. Total training time and peak memory usage, compared with standard SRO, didn't seem to be included in the experiments.

  3. It is unclear whether fixed probes will act like in different training phases. The probes are generated by the initial model and remain fixed, although their gradients and projection basis are recomputed each round (Lines 1209–1213). As the model’s responses change during training, these initial examples may become less representative.

Questions for authors

See weakness

  1. Could you connect the frozen-judge results in Appendix G.2 to Section 4.1.1 and clarify how the self-rewarding loop adds to the harm caused by incorrect labels?

  2. What are the total training time and peak memory overhead compared with standard SRO?

  3. Can 16 or more sampled probes adequately cover a diverse dataset, and does the defense generalize to distributions not represented by these probes?

4 comments

We thank the reviewer for the insightful comments. Below, we respond to each of the reviewer’s questions, and we have incorporated the corresponding changes into the revised manuscript.


The label-corruption experiment (Lines 299–310) does not separate the damage caused by incorrect labels from the additional damage caused by the self-rewarding loop. Appendix G.2 seems to provide a frozen-judge control (Lines 1495–1511), but the main text does not explicitly reference or discuss these results. Thus, the whole description of this part can be a bit confusing and the corresponding analysis can be a bit abrupt due to the lack of references. Could you connect the frozen-judge results in Appendix G.2 to Section 4.1.1 and clarify how the self-rewarding loop adds to the harm caused by incorrect labels?

Thank you for this suggestion. We have connected the frozen-judge control (Appendix G.2) to §4.1.1 and now discuss it in the main text. The control separates the two sources of harm. Both branches share the same initial model, corrupted first-round labels, and first update, so the change from Iteration 0 to 1 reflects the direct damage from incorrect labels. They differ only afterward: standard SRO keeps using the updated model as its judge, whereas the frozen branch keeps the original judge. The further change from Iteration 1 to 3 (Eq. (25)) therefore isolates the additional damage from the self-rewarding loop.

Corruption levelInitial SRFFurther AQ loss (SROFurther AQ loss (frozen judge)
Mild ($\epsilon$=0.15)72.1%12.80.1
Severe ($epsilon$=0.8)17.0%48.86.0

Incorrect labels cause harm with either judge, but only when the judge co-evolves with the policy does the damage keep compounding. The self-rewarding loop thus amplifies the initial corruption more than just propagating it.


Lines 1071–1078 report the additional probe computation time, but it is unclear whether this includes constructing the projection basis and applying the projection at every training step. Total training time and peak memory usage, compared with standard SRO, didn't seem to be included in the experiments. What are the total training time and peak memory overhead compared with standard SRO?

To clarify, each SRO round consists of multiple optimizer steps. As specified in Algorithm 1 and Appendix E.3, the probe gradients and the projection basis are computed once per round and then reused to filter the Adam update at every optimizer step within that round. As shown in the table below, the default setting of 16 probes adds only 58.0 s per SRO round, increasing the total runtime per round from 2395.4 s to 2453.4 s (a 2.42% overhead). It also increases peak memory usage from 66.1GB to 70.3 GB of GPU memory and from 2.12 GB to 15.06 GB of CPU memory, compared with standard SRO.

MethodProjection computation (s/round)Projection application (s/round)Training time (s/round)Total time (s/round)Peak GPU memory (GB)Peak CPU memory (GB)
Vanilla SRO0 (no probes)—2395.42395.466.12.12
Projection Decoupling Defense (8 probes)9.5320.42395.42425.3367.27.06
Projection Decoupling Defense (16 probes)15.5042.52395.42453.4070.315.06
Projection Decoupling Defense (32 probes)33.2075.22395.42503.8075.931.06

It is unclear whether fixed probes will act like in different training phases. The probes are generated by the initial model and remain fixed, although their gradients and projection basis are recomputed each round (Lines 1209–1213). As the model’s responses change during training, these initial examples may become less representative. Can 16 or more sampled probes adequately cover a diverse dataset, and does the defense generalize to distributions not represented by these probes?

Fixed vs. recomputed probes. We compare probe configuration strategies on SafeRLHF in the table below. The three strategies (fixed probes, fixed prompts with regenerated responses, and resampled prompts with regenerated responses) differ by less than 0.4% across all reported metrics, indicating that fixed probes are a viable and efficient choice.

Probe configurationClean AQ (%)Re-Ranking AQ (%)Re-Ranking SRF (%)
Fixed probes91.985.282.2
Fixed prompts + regenerated responses92.285.582.2
Resampled prompts + regenerated responses92.185.081.8

Probe number. As shown in Appendix D.4, 16 probes offer a practical trade-off: attacked AQ rises from 81.1% with 8 probes to 85.2% with 16, while additional probes yield only marginal further gains.

Probe source ↓ / Target domain →Safety (SafeRLHF)Knowledge (MMLU-Pro)Reasoning (ARC)
None (no defense)74.5 / 74.427.7 / 32.216.3 / 40.4
Safety (SafeRLHF)85.2 / 82.230.5 / 35.519.5 / 45.1

Cross-domain transferability. Different domains may draw on different model capabilities (e.g., mathematical reasoning vs. safety reasoning) and thus activate different neurons. As a result, probe gradients estimated in one domain may transfer only partially to other domains. The table above shows the transferability of probes from SafeRLHF to other domains (measured by AQ/SRF) under the Re-Ranking attack: they improve AQ by 2.8% on MMLU-Pro and 3.2% on ARC, compared with 10.7% on SafeRLHF itself. We therefore recommend using probes that are representative of the target domain.


Again, we thank the reviewer for the valuable feedback and welcome any further suggestions or comments.

Authors

Thank you for the detailed responses and additional experiments. The clarification addresses most of my concerns. I will maintain my positive score.

Thank you for your engagement and for acknowledging our clarifications and additional experiments! We truly appreciate your support of our work. We will incorporate these clarifications and new results into the revision.

Reviewer_Cy9oOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary
  • Studied problem: Study self-rewarding optimization (SRO) (LMs generate their own reward signal) and its failure modes and find that seemingly benign perturbations to training prompts can corrupt the self-generated preference signal and trigger iterative collapse.
  • Submission approach: Assume that If SRO is to serve as a scalable alignment mechanism, it should remain stable under variations in training data, such as reordering, filtering, or adversarial modification of prompts. The authors derive two prompt-level attacks (access to input prompts and target model outputs) to highlight the fragility of SRO, introduce a metric for predicting SRO improvement or collapses, and derive a defense to suppress collapse-inducing updates.
  • Approach realization: The authors evaluate four models (Llama-3.2-3B, Llama-3.1-8B,, Qwen-3-4B, and Qwen-2.5-7B), use MMLU-Pro, ARC, and SafeRLHF datasets with gpt5 as orcale evaluator for their attack construction. The authors evaluate two SRO methods (SRLM and CREAM).
  • Claims/Contributions: Claim to reveal SRO is fundamentally fragile, claim to have added two new prompt-level attacks (Re-ranking: find training prompts where self reward diverges from oracle; Preamble: find prefix that inflate self-reward), claim to introduce self-reward-fidelity (quantifying on-policy agreement between self-evaluated and oracle rewards) and resulting Projection Decoupling Defense to avoid low SRF updates
Critical strengths
  • Interesting topic and understanding SRO collapse is important
  • The proposed SRO training method (algorithm 1, SRO with Projection Decoupling Defense) seems to be more robust to the chosen attacks and seems to be competitive to vanilla SRO training (Figure 7).
Critical weaknesses
  • The Re-Ranking attack seems to just filter a dataset for prompts the target model is bad at and then starts training on prompts the model is more likely to get wrong. That this results in collapse does not seem insightful. I have similar reservations about the Preamble attack, as adversarially finding prefixes that reduce model accuracy to destabilize the training loop is not surprising. How realistic and representative are these attacks of actual training loops? Do these attacks provide insights about realistic training data distributions? To me, this risks being read as “If you start self improvement training loops on data the target model is bad at (intentionally or unintentionally, for whatever reason) you get collapse”.
  • It is also unclear to what extend this attacks are meaningful as threat vectors. Specifically, the attacks require access to the target model to get the self-rewards. This requires access or assumptions about the reward-task prompt when using the target model as reward generator. How robust is the attack to different prompts for the reward generator that might be different? This issue could also be mitigated by calling them “ablations” and not attacks.
  • On the self-reward fidelity metric: Using it to predict when training is successful or collapsing seems problematic, as I think they are not independent of each other. Specifically, the SRF metric is a function of the current model’s outputs on the data, so it is a function of the same model state that calculates loss/accuracy, no? (The oracle judged AQ measure is essentially equivalent to accuracy). The metrics seem to be near-monotone transforms of each other. The SRF threshold claim for the discovered “phrase transition” is therefore also in question, as a threshold is just another monotone relation. If they are not dependent, I would have reservations about the causal claim that SRF predicts collapse or not given the reported evidence.
Questions for authors
  • See Weaknesses above.
  • Also: In the introduction, you mention that an issue with SRO is that what works in one setting does not work in others (robustness and context-sensitivity issues). The new training method (SRO with Projection Decoupling Defense) seems to have similar issues (Figure 7) when used in non-adversarial settings. Can you explain when it improves training and how this depends on the training data distribution or specific target model attributes?
9 comments

We thank the reviewer for the insightful comments. Below, we respond to each of the reviewer’s questions, and we have incorporated the corresponding changes into the revision.


The Re-Ranking attack seems to just filter a dataset for prompts the target model is bad at and then starts training on prompts the model is more likely to get wrong. That this results in collapse does not seem insightful. I have similar reservations about the Preamble attack, as adversarially finding prefixes that reduce model accuracy to destabilize the training loop is not surprising. How realistic and representative are these attacks of actual training loops? Do these attacks provide insights about realistic training data distributions?

We agree that training on data the model handles poorly will hurt it, and we don’t present that as our finding. Our focus is self-rewarding optimization (SRO). In SRO, damage from bad data compounds, because each policy update also degrades the evaluator that supplies the next round’s supervision.

Compounding effect of SRO. To isolate this effect, we compare SRO with standard preference training that uses a frozen judge. Both branches share the same first corrupted update and the same training settings, and both policies remain trainable. The only difference is that SRO’s evaluator co-evolves with the policy (Appendix G.1 and Appendix G.2). The table reports $ \Delta AQ = AQ_3 − AQ_1$, the change in answer quality from Iteration 1 to Iteration 3.

Experiment/conditionSRO $ \Delta $ AQ (%)Frozen-judge $ \Delta $ AQ (%)
Controlled initial SRF: 72.1%−12.8−0.1
Controlled initial SRF: 17.0%−48.8−6.0
Re-Ranking attack−12.9+0.3
Preamble attack−9.2+1.6

As shown above, under both attacks, the two variants suffer similar initial damage. After that, the frozen-judge model holds steady or recovers slightly (+0.3, +1.6), while SRO keeps declining (−12.9, −9.2). With a fixed evaluator, the corrupted data causes a one-time drop. When the evaluator co-evolves with the policy, that drop becomes a continued decline. Under severe controlled corruption (17.0% initial SRF), the frozen judge does not prevent all further degradation (−6.0), but SRO makes it roughly eight times larger (−48.8).

Risk realism. We see the Re-Ranking and Preamble attacks as stress tests. They are controlled ways to shift supervision quality at the start of training, and they let us measure how each training paradigm responds. Similar shifts can arise without an adversary, for example from difficulty-based curricula or hard-example mining. The practical implication is that SRO lacks a fixed reference point for its supervision. As a result, any early supervision error, whether adversarial or accidental, can compound over iterations.


It is also unclear to what extend this attacks are meaningful as threat vectors. Specifically, the attacks require access to the target model to get the self-rewards. This requires access or assumptions about the reward-task prompt when using the target model as reward generator. How robust is the attack to different prompts for the reward generator that might be different? This issue could also be mitigated by calling them “ablations” and not attacks.

Access to reward generator. In practice, we can lift the requirement for the target model’s self-rewards. In Appendix F.4, we construct each attack once on a surrogate model, either Llama-3.2-3B (same family) or Qwen2.5-7B (cross-family), and reuse it unchanged for three SRLM iterations on an unseen Llama-3.1-8B target on SafeRLHF. The target is never queried during construction. Both Re-Ranking and Preamble transfer, reducing target AQ and SRF whether or not an oracle is used during construction. At Iteration 3, the AQ gap between same- and cross-family surrogates is only 1.2-2.2%. These attacks are therefore meaningful under a black-box threat model.

Access to reward-task prompts. To test prompt sensitivity directly, we construct attacks using the original reward prompt and hold them fixed. We then run three SRLM iterations on Llama-3.1-8B with SafeRLHF under 3 different reward prompts: the original and two held-out variants. The first variant paraphrases the instructions, and the second restructures the rubric (see §E.5 for details).

Attack construction promptSRO Training reward promptCleanFixed Re-RankingFixed Preamble
OriginalOriginal92.2 / 91.174.5 / 74.471.3 / 68.3
OriginalParaphrased92.4 / 91.375.5 / 74.972.4 / 70.3
OriginalRestructured92.5 / 91.377.1 / 74.873.1 / 71.1

Under the paraphrased and restructured prompts, the fixed Re-Ranking and Preamble attacks reduce AQ by 15.4–16.9% and 19.4–20.0%, respectively, relative to each prompt’s clean baseline. Both attacks therefore transfer to the held-out reward prompts without re-optimization.

“Attack” vs. “ablation”. Following the reviewer's suggestion, we have clarified in §3.2 that these perturbations also serve as controlled stress tests of reward fidelity.

On the self-reward fidelity metric: Using it to predict when training is successful or collapsing seems problematic, as I think they are not independent of each other. Specifically, the SRF metric is a function of the current model’s outputs on the data, so it is a function of the same model state that calculates loss/accuracy, no? (The oracle judged AQ measure is essentially equivalent to accuracy). The metrics seem to be near-monotone transforms of each other. The SRF threshold claim for the discovered “phrase transition” is therefore also in question, as a threshold is just another monotone relation. If they are not dependent, I would have reservations about the causal claim that SRF predicts collapse or not given the reported evidence.

We agree that SRF and AQ are statistically dependent, since both are computed from the same model. However, they measure different properties, and SRF is not a monotone transform of AQ. AQ measures whether the model’s answers are correct, while SRF measures whether its self-evaluations agree with the oracle. A model that answers incorrectly but recognizes the error lowers AQ yet raises SRF.

To separate SRF’s effect from AQ, we intervene on reward fidelity directly (section 4.1.1). Starting from the same initial model, and thus the same initial AQ, we vary only the fidelity of the training reward labels and obtain markedly different trajectories. After an identical corrupted first update, AQ deteriorates far more when the evaluator co-evolves with the policy than when it is frozen, for example a further 48.8% vs. 6.0% at 17.0% initial SRF.

We agree that this evidence supports SRF as a diagnostic of self-supervision reliability rather than as a general predictor of collapse. In the revision, we have qualified the prediction claim accordingly in section 4.1.1 and now describe the observed threshold as an empirical transition specific to our settings.


In the introduction, you mention that an issue with SRO is that what works in one setting does not work in others (robustness and context-sensitivity issues). The new training method (SRO with Projection Decoupling Defense) seems to have similar issues (Figure 7) when used in non-adversarial settings. Can you explain when it improves training and how this depends on the training data distribution or specific target model attributes?

We agree that the defense is context-sensitive. Its effect depends on one quantity that both the model and the data influence: how reliable the model's self-reward is on the prompts it trains on. Projection removes update components that change self-evaluation. It can block harmful changes to the evaluator but can also block some beneficial ones, so it helps when the removed components are mostly harmful.

Dependence on the target model. The relevant attribute is the model's initial self-reward fidelity relative to its transition region.

ModelInitial SRFTransition region (section 4.1.1)Clean SRO without defenseFinal clean AQ: no defense → defense
Llama-3.2-3B71.8%≈ 77.4–74.6%collapses45.4% → 84.4%
Llama-3.1-8B84.8%≈ 80.6–72.1%stable92.2% → 91.9%

Llama-3.2-3B starts below its transition region, so clean SRO itself degrades the evaluator, and the defense recovers most of the loss (AQ still ends below its initial 88.6%). Llama-3.1-8B starts above its region, clean SRO is already stable, and the defense costs 0.3 points.

Dependence on the training data distribution. With the model fixed (Llama-3.1-8B), the prompt distribution changes what projection removes (α=0.5\alpha = 0.5, iteration 3; Appendix D.5, Table 3):

Training promptsHarmful share of removed updateBeneficial shareAQ change from defense
Clean40.1%45.2%−0.3 points
Re-Ranking76.0%15.1%+10.7 points

On clean prompts, the removed components roughly cancel out, so the net effect is close to zero. Re-Ranking, by contrast, concentrates training on prompts with high reward disagreement, where about three-quarters of the removed components would otherwise have degraded reward fidelity.

The benefit is also smaller when the probes come from a different task than the training data. SafeRLHF probes improve attacked AQ by 10.7% on SafeRLHF, but only 2.8% on MMLU-Pro and 3.2% on ARC (Appendix D.7).

Iteration-3 AQ / SRF (%) under Re-Ranking, Llama-3.1-8B, SRLM:

Probe source domain ↓ / Target domain →Safety (SafeRLHF)Knowledge (MMLU-Pro)Reasoning (ARC)
None (no defense)74.5 / 74.427.7 / 32.216.3 / 40.4
Safety (SafeRLHF)85.2 / 82.230.5 / 35.519.5 / 45.1

Practical use. Both dependencies can be checked before training by estimating the initial model's SRF on a small labeled sample of the intended training prompts. In our experiments, the defense helped most where this fidelity was low (Llama-3.2-3B at 71.8%; prompts selected for high reward disagreement). Where SRF was high and training was stable, a small α\alpha or no projection was sufficient. Appendix B.4 shows that such estimates need few labels: with 256 oracle labels, the estimate is within 0.1% of the full evaluation.


We thank the reviewer again for the valuable feedback and welcome any further suggestions or comments.

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their constructive feedback. We summarize the main questions raised in the reviews and how our original manuscript and our rebuttal address them.


1. SRO collapse is more than "bad data hurts"

The collapse of self-rewarding optimization (SRO) goes beyond "bad data hurts": SRO compounds an initial error, because the evaluator co-evolves with the policy. As introduced in Appendix G of our original manuscript, we use a frozen-judge control experiment to quantify this. SRO updates share-parameter judge with the policy, while the frozen-judge control keeps it fixed. We show the results as follows (details in Appendix G).

ConditionSROFrozen judge
Re-Ranking attack−12.9+0.3
Preamble attack−9.2+1.6
Controlled initial SRF 72.1%−12.8−0.1
Controlled initial SRF 17.0%−48.8−6.0

From Iteration 1 to 3, answer quality (AQ) changes little with a frozen judge but keeps falling under SRO.

2. The attacks are realistic threat vectors

We discussed this question in our original manuscript along two dimensions. First, our threat model assumes that the adversary can only manipulate training prompts, which covers deployments where externally collected prompts or filtered data pools decide what enters the self-rewarding loop. Second, we show that the attacks can be built without an oracle, and that attacks built on a surrogate model still transfer to an unseen target that is never queried (Appendix F.4), with a gap of only 1.2–2.2 AQ points between same- and cross-family surrogates.

We further examine another setting in our rebuttal. Attacks built with one reward prompt are reused unchanged under two held-out reward prompts and still reduce AQ by 15.4–16.9 (Re-Ranking) and 19.4–20.0 (Preamble) points, so the attacks do not depend on the exact reward prompt.

We also clarify that the attacks serve as controlled stress tests of reward fidelity: similar shifts in training prompts can arise without any adversary, e.g., from difficulty-based curricula or hard-example mining.

3. SRF explains when SRO improves or collapses

In our original manuscript, we show that SRF can serve as a diagnostic of self-supervision reliability, which explains when SRO improves or collapses. We emphasize two points. First, SRF and AQ measure different properties: AQ is whether the model's answers are correct, while SRF is whether its self-rewards agree with the oracle. Second, reward fidelity alone changes the outcome of SRO: from the same model, hence the same initial AQ, perturbing only the reward labels yields trajectories ranging from improvement to collapse.

We further examine the practical use of SRF in our rebuttal, testing whether it can be estimated from a small labeled sample. With 256 oracle labels, the estimate is within 0.1 points of the full evaluation, and 32 labels already separate clean from attacked training, so SRF is practical with a small labeling budget.

4. A small set of fixed probes is sufficient

As described in our original manuscript, our defense uses 16 fixed probes sampled from the training questions. We validate this setting with a probe-count sweep and find that a small probe set is sufficient: going from 8 to 16 probes raises attacked AQ from 81.1% to 85.2%, while going to 128 adds only 1.3 points.

We further examine probe drift in our rebuttal. Refreshing the probes during training, by regenerating responses or resampling prompts, changes every reported metric by at most 0.5 points, so fixed probes remain representative.

5. The defense helps under attacks

In our original manuscript, we study this question in two ways. First, we sweep the protection strength α\alpha: α=0.5\alpha = 0.5 raises attacked AQ to 85.2% while keeping clean AQ within 0.3 points of undefended SRO, and larger α\alpha adds at most 0.3 points under attack at a clean cost of up to 1.4 points. Second, we evaluate the defense without any attack: it stays within 0.2–0.3 points of undefended SRO on the three stable models, and raises the final AQ of the fragile Llama-3.2-3B from 45.4% to 84.4%.

We further examine why the defense behaves this way in our rebuttal. We classify the update components that projection removes by their effect on reward fidelity. Under Re-Ranking, 76.0% are harmful and 15.1% are beneficial, while in clean training the shares are 40.1% and 45.2%, which explains the large gain under attack and the near-zero cost on clean data.


In short, our paper shows that SRO collapse comes from the evaluator co-evolving with the policy. We use attacks to expose this weakness under various access assumptions, and we use SRF to examine when the collapse occurs. We propose Projection Decoupling Defense, which mitigates the collapse with a small fixed probe set. Our rebuttal strengthens each of these points with additional experiments and analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.