- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Self-rewarding test-time reinforcement learning adapts models on unlabeled deployment data using rewards derived from the model’s own outputs, such as majority voting or entropy. While this has shown promise for reasoning, real-world prompt streams are heterogeneous and may mix reasoning tasks, benign instructions, and jailbreak attempts. This paper asks whether these proxy rewards remain reliable in such settings, or whether they can unintentionally reinforce unsafe or otherwise undesirable response patterns.
The experiments show that heterogeneous test-time training can produce three linked effects: apparent safety amplification, harmfulness amplification, and a reasoning tax, depending on the method, model, and prompt mixture. The authors trace these effects to reward-compatible response patterns. For example, TTRL can collapse onto repeated refusal templates, while RENT can amplify confident or transferred response styles. The paper also shows with HarmInject that an adversarially constructed mixed prompt can make harmful behavior reward-compatible.
Critical strengths
-
The paper is clearly written and easy to follow.
-
The analysis is thorough and the experiment design looks solid to me.
-
Earlier self-training work often focuses on homogenous data sources (math-only or reasoning-only), whereas most real-world LLM training has to deal with heterogeneous data sources. This paper therefore is a significant next step towards understanding how self-training interacts with more realistic data mixtures.
-
I really liked the idea of HarmInject, and the careful study of the effect of adversarially constructed mixed prompts on self-training.
Critical weaknesses
(What happens under extended RL training?)
The paper mainly focuses on self-training runs for a very few number of RL steps. However, multiple prior works show the weakness of self-training RL methods when exposed to a large number of RL steps. None of these papers are cited or discussed, and the RL runs are very short (only 250-350 steps). For example, SRT (identical to TTRL) [1] already shows that long training runs with majority voting reward leads to complete collapse in performance, which is not cited in this paper. [2] also discusses this, and is cited in the paper, but without any discussion on prolonged self-reward RL training. These results were also discussed in [3], but it is not mentioned in the current manuscript.
My primary concern is this: since self-training RL algorithms are already known to be prone to model collapse and failure, and therefore unlikely to be adopted at scale, What is the utility of studying their usefulness with respect to safety concerns?
(Analyzing outdated self-training algorithms)
The paper analyzes TTRL and RENT as representatives of self-rewarded RL training objectives. However, both of these are more than 1 year old at this point, and it is unclear to me whether they can be good representatives of this class of algorithms. Could the authors also run their analysis with more modern algorithms, like RESTRAIN [3] or more recent self-training objectives that seem more stable?
(Outdated models)
All experiments use outdated base models like Llama-3 or Qwen2.5. Rerunning the analysis on modern open-source LLMs can make the results more practical.
References
[1] Can Large Reasoning Models Self-Train? https://arxiv.org/abs/2505.21444
[2] How Far Can Unsupervised RLVR Scale LLM Training? https://arxiv.org/abs/2603.08660
[3] RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization, https://arxiv.org/abs/2603.08660
Questions for authors
Please clarify the weaknesses mentioned above.
3 comments
I thank the authors for their detailed response!
However, I disagree with the authors about the significance of this paper. SRT and RESTRAIN showed that majority voting like training is prone to not scale without model collapse. Hence it is unclear to me why would one use similar training algorithms in the first place, and that makes the associated safety risk a mute point.
While I understand the compute constraints during the rebuttal, my concerns remain, and hence I will keep my current score.
Thanks,
Reviewer
Response to reviewer VysQ comment
We thank the reviewer for engaging with our response. We respectfully see the significance differently. Training collapse has not led the field to abandon RL methods; it has led to stabilized versions. GRPO with ground-truth rewards also collapses in long or multi-turn training [1, 2, 3], and the response has been stabilized variants. Self-rewarding RL is following the same path: RESTRAIN [4], which the reviewer cites, builds on majority-vote training but weights all of the model's answers by vote frequency and penalizes low-agreement prompts, and it reports results close to gold-label training, and He et al. [5] find intrinsic rewards remain useful for test-time training on small datasets, the setting we study.
As these methods become more stable and more widely used, their safety matters more, not less. Our results show this risk is not visible through reasoning metrics: across three runs, AMC accuracy rose by 3.4–9.6 pp while ASR rose by 32.4–48.4 pp. A user would notice collapse, but would see these runs as successful. We have not tested RESTRAIN and do not claim it shares these failures. We will add this discussion to the paper. We hope this clarifies our view of the paper's relevance, and we thank the reviewer for the helpful references and discussion.
References
[1] Z. Wang et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073, 2025.
[2] Z. Xue et al. SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning. arXiv:2509.02479, 2025.
[3] C. Zheng et al. Group Sequence Policy Optimization. arXiv:2507.18071, 2025.
[4] Z. Yu et al. RESTRAIN: From Spurious Votes to Signals — Self-Driven RL with Self-Penalization. ICLR 2026. arXiv:2510.02172.
[5] B. He et al. How Far Can Unsupervised RLVR Scale LLM Training? ICLR 2026. arXiv:2603.08660.
Response to reviewer VysQ
We thank the reviewer for their feedback. We address the three concerns below.
W1: Extended training and the value of studying safety
We agree that prolonged self-rewarded RL can cause reasoning collapse. We already cite He et al., but will expand that discussion and add SRT and RESTRAIN. Importantly, SRT Appendix H and He et al., Section 5 also report small-dataset test-time settings where reasoning improves without observed accuracy collapse. These settings remain relevant to safety analysis. Our results show why safety still needs separate measurement. Across the same three Qwen-1.5B RENT + Instruct runs, AMC accuracy increased by 3.4–9.6 percentage points from each run's starting model, while ASR increased by 32.4–48.4 points. Both changes compare initialization with the final five evaluations, steps 346–350. The AMC gains are changes from initialization, not reasoning taxes against a clean reference.
Thus, reasoning improvements can coexist with worsening safety. Our contribution concerns these safety changes under mixed prompts, not the discovery of eventual reasoning collapse. We do not establish behavior under substantially longer training and will state this distinction explicitly.
W2: Newer training objectives
We chose TTRL and RENT to examine majority-agreement and entropy-based reward designs. We do not claim that they cover all current self-rewarding test-time training methods. RESTRAIN changes how answers, rollouts, and prompts contribute to training through weighting and self-penalization. We will discuss these differences. Since we have not evaluated RESTRAIN, we cannot claim that it shares the observed failures.
W3: Newer models
Our conclusions apply to the tested Qwen2.5 and Llama-3 models. The selected 7B–8B experiments address model size, but do not establish generalization to newer model families. We agree that newer-model experiments would strengthen practical relevance. Our current compute budget prevents adding the requested algorithm and model comparisons during rebuttal. We will state these limits explicitly and restrict broader claims to the evidence from the tested implementations and models.