acceptodds
Under review as a conference paper at ICLR 2027

Safety and Reasoning Vulnerabilities in Self-Rewarding Test-Time Reinforcement Learning

Abstract

Test-time reinforcement learning adapts a language model on unlabeled test inputs by deriving rewards from the model's own generations. This label-free property makes test-time RL attractive when verifiable feedback is unavailable, and recent methods have been shown to improve reasoning when the test-time stream contains only reasoning problems. We study what happens in heterogeneous streams, i.e., when reasoning problems are mixed with jailbreak attempts or benign instructions. We evaluate two representative methods: TTRL, which rewards majority-vote agreement, and RENT, which rewards low entropy. We find that mixed streams produce 3 linked effects: safety amplification, harmfulness amplification, and a reasoning tax. These effects follow a common mechanism: self-rewarding test-time RL reinforces the response pattern that its reward can score most easily. Under TTRL, jailbreak prompts often collapse to a short refusal template; on held-out jailbreak prompts, 54-78% of responses can become the same refusal sentence. This lowers the measured attack success rate, producing apparent safety amplification, while math responses become more self-consistent without becoming more correct, producing the reasoning tax. RENT does not show the same refusal-template collapse, and its safety direction depends more strongly on the model's initial behavior. We further show harmfulness amplification with an adversarial prompt, "HarmInject”, that couples a jailbreak to a reasoning problem, enabling rewards for harmful compliance. These results show that consistency and confidence are unsafe proxies for correctness and safety in heterogeneous test-time RL.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Test-time training, AI Safety, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VysQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Self-rewarding test-time reinforcement learning adapts models on unlabeled deployment data using rewards derived from the model’s own outputs, such as majority voting or entropy. While this has shown promise for reasoning, real-world prompt streams are heterogeneous and may mix reasoning tasks, benign instructions, and jailbreak attempts. This paper asks whether these proxy rewards remain reliable in such settings, or whether they can unintentionally reinforce unsafe or otherwise undesirable response patterns.

The experiments show that heterogeneous test-time training can produce three linked effects: apparent safety amplification, harmfulness amplification, and a reasoning tax, depending on the method, model, and prompt mixture. The authors trace these effects to reward-compatible response patterns. For example, TTRL can collapse onto repeated refusal templates, while RENT can amplify confident or transferred response styles. The paper also shows with HarmInject that an adversarially constructed mixed prompt can make harmful behavior reward-compatible.

Critical strengths
  1. The paper is clearly written and easy to follow.

  2. The analysis is thorough and the experiment design looks solid to me.

  3. Earlier self-training work often focuses on homogenous data sources (math-only or reasoning-only), whereas most real-world LLM training has to deal with heterogeneous data sources. This paper therefore is a significant next step towards understanding how self-training interacts with more realistic data mixtures.

  4. I really liked the idea of HarmInject, and the careful study of the effect of adversarially constructed mixed prompts on self-training.

Critical weaknesses

(What happens under extended RL training?)

The paper mainly focuses on self-training runs for a very few number of RL steps. However, multiple prior works show the weakness of self-training RL methods when exposed to a large number of RL steps. None of these papers are cited or discussed, and the RL runs are very short (only 250-350 steps). For example, SRT (identical to TTRL) [1] already shows that long training runs with majority voting reward leads to complete collapse in performance, which is not cited in this paper. [2] also discusses this, and is cited in the paper, but without any discussion on prolonged self-reward RL training. These results were also discussed in [3], but it is not mentioned in the current manuscript.

My primary concern is this: since self-training RL algorithms are already known to be prone to model collapse and failure, and therefore unlikely to be adopted at scale, What is the utility of studying their usefulness with respect to safety concerns?

(Analyzing outdated self-training algorithms)

The paper analyzes TTRL and RENT as representatives of self-rewarded RL training objectives. However, both of these are more than 1 year old at this point, and it is unclear to me whether they can be good representatives of this class of algorithms. Could the authors also run their analysis with more modern algorithms, like RESTRAIN [3] or more recent self-training objectives that seem more stable?

(Outdated models)

All experiments use outdated base models like Llama-3 or Qwen2.5. Rerunning the analysis on modern open-source LLMs can make the results more practical.

References

[1] Can Large Reasoning Models Self-Train? https://arxiv.org/abs/2505.21444

[2] How Far Can Unsupervised RLVR Scale LLM Training? https://arxiv.org/abs/2603.08660

[3] RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization, https://arxiv.org/abs/2603.08660

Questions for authors

Please clarify the weaknesses mentioned above.

3 comments
AuthorsOpenReview ↗

Response to reviewer VysQ

We thank the reviewer for their feedback. We address the three concerns below.

W1: Extended training and the value of studying safety

We agree that prolonged self-rewarded RL can cause reasoning collapse. We already cite He et al., but will expand that discussion and add SRT and RESTRAIN. Importantly, SRT Appendix H and He et al., Section 5 also report small-dataset test-time settings where reasoning improves without observed accuracy collapse. These settings remain relevant to safety analysis. Our results show why safety still needs separate measurement. Across the same three Qwen-1.5B RENT + Instruct runs, AMC accuracy increased by 3.4–9.6 percentage points from each run's starting model, while ASR increased by 32.4–48.4 points. Both changes compare initialization with the final five evaluations, steps 346–350. The AMC gains are changes from initialization, not reasoning taxes against a clean reference.

Thus, reasoning improvements can coexist with worsening safety. Our contribution concerns these safety changes under mixed prompts, not the discovery of eventual reasoning collapse. We do not establish behavior under substantially longer training and will state this distinction explicitly.

W2: Newer training objectives

We chose TTRL and RENT to examine majority-agreement and entropy-based reward designs. We do not claim that they cover all current self-rewarding test-time training methods. RESTRAIN changes how answers, rollouts, and prompts contribute to training through weighting and self-penalization. We will discuss these differences. Since we have not evaluated RESTRAIN, we cannot claim that it shares the observed failures.

W3: Newer models

Our conclusions apply to the tested Qwen2.5 and Llama-3 models. The selected 7B–8B experiments address model size, but do not establish generalization to newer model families. We agree that newer-model experiments would strengthen practical relevance. Our current compute budget prevents adding the requested algorithm and model comparisons during rebuttal. We will state these limits explicitly and restrict broader claims to the evidence from the tested implementations and models.

I thank the authors for their detailed response!

However, I disagree with the authors about the significance of this paper. SRT and RESTRAIN showed that majority voting like training is prone to not scale without model collapse. Hence it is unclear to me why would one use similar training algorithms in the first place, and that makes the associated safety risk a mute point.

While I understand the compute constraints during the rebuttal, my concerns remain, and hence I will keep my current score.

Thanks,

Reviewer

AuthorsOpenReview ↗

Response to reviewer VysQ comment

We thank the reviewer for engaging with our response. We respectfully see the significance differently. Training collapse has not led the field to abandon RL methods; it has led to stabilized versions. GRPO with ground-truth rewards also collapses in long or multi-turn training [1, 2, 3], and the response has been stabilized variants. Self-rewarding RL is following the same path: RESTRAIN [4], which the reviewer cites, builds on majority-vote training but weights all of the model's answers by vote frequency and penalizes low-agreement prompts, and it reports results close to gold-label training, and He et al. [5] find intrinsic rewards remain useful for test-time training on small datasets, the setting we study.

As these methods become more stable and more widely used, their safety matters more, not less. Our results show this risk is not visible through reasoning metrics: across three runs, AMC accuracy rose by 3.4–9.6 pp while ASR rose by 32.4–48.4 pp. A user would notice collapse, but would see these runs as successful. We have not tested RESTRAIN and do not claim it shares these failures. We will add this discussion to the paper. We hope this clarifies our view of the paper's relevance, and we thank the reviewer for the helpful references and discussion.

References

[1] Z. Wang et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073, 2025.

[2] Z. Xue et al. SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning. arXiv:2509.02479, 2025.

[3] C. Zheng et al. Group Sequence Policy Optimization. arXiv:2507.18071, 2025.

[4] Z. Yu et al. RESTRAIN: From Spurious Votes to Signals — Self-Driven RL with Self-Penalization. ICLR 2026. arXiv:2510.02172.

[5] B. He et al. How Far Can Unsupervised RLVR Scale LLM Training? ICLR 2026. arXiv:2603.08660.

Reviewer_hYpoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the safety and reasoning behavior of self-rewarding test-time RL under mixed prompt distributions. It evaluates majority-vote-based TTRL and entropy-based RENT, reporting changes in refusal behavior, harmful responses, and reasoning performance. It also introduces HarmInject, which combines harmful requests with reasoning problems, and evaluates a LlamaGuard-based reward filter.

Critical strengths
  1. Studying safety alongside reasoning under mixed prompt distributions is worthwhile. The finding that benign instruction injection can increase harmful responses is particularly interesting.
  2. The selected repeated runs and second-judge evaluation provide useful support for some of the reported effects.
Critical weaknesses
  1. The contribution beyond the expected amplification of existing response patterns is not sufficiently clear. “Reward-compatible response-mode selection” largely describes the observations rather than establishing their cause. In particular, higher average rewards across prompt types do not directly imply stronger reinforcement under group-normalized GRPO. The analysis needs a clearer connection between the measured rewards, actual advantages, and subsequent behavioral changes.
  2. The broader conclusions are stronger than the experimental support. Most settings use a single run, the main grid focuses on small models, and the experiments repeatedly reuse a finite prompt pool. Although the appendix adds useful checks, HarmInject effects remain model- and configuration-dependent. The limited reward-filter experiments also do not justify a general conclusion that output-content filtering is ineffective.
Questions for authors
  1. Can you show how refusal and compliance rollouts differ in their actual GRPO advantages during training? How does this support the proposed mechanism when majority agreement approaches saturation?
  2. Under HarmInject, can a response refuse the harmful request while answering the math question and still receive the same reward? Please compare these responses with harmful-compliance responses to substantiate the claimed coupling.
  3. Can the reasoning tax and HarmInject effects be replicated with paired clean and injected training runs, matched reasoning-data exposure, and at least one larger model?
3 comments
AuthorsOpenReview ↗

Response to reviewer hYpo's questions

We thank the reviewer for their feedback. We address the questions first, followed by the two broader weaknesses.

Q1: How do refusal and compliance rollouts differ in GRPO advantages, including near saturation?

A1: For this advantage analysis, we used existing training logs. We reconstructed advantages as (reward − group mean) / (sample group std + 10⁻⁶), using the 32 policy-update rollouts per prompt; voting uses 64. We identified refusal-like responses using phrase matches in the first 500 characters, such as “I cannot” and “I'm sorry.” Other responses were not assumed to be harmful compliance.

Across the first 20 training steps of three Qwen-1.5B + JBV28k runs:

SeedGroupsRefusal shareMean refusal advantageMean other advantageRefusals higher*
116262.6%+0.088−0.14727/37
225768.5%+0.051−0.11019/32
335666.2%+0.078−0.15423/35

The last column includes only groups containing both response types. Class means and refusal shares average over training rollouts.

Refusal-like responses had positive average advantages and were favored in 69/104 comparable groups (66.3%). This provides a learning signal that can increase refusals. During each run's final 20 steps, 98.3–100% of logged jailbreak groups had identical policy rewards and zero reward-based advantages. These windows are steps 331–350 for seeds 11 and 33, and 281–300 for seed 22. High agreement late in training therefore does not mean that high rewards keep driving strong updates. We will revise Section 4.2 and Figure 4A to distinguish agreement from the GRPO learning signal.

Q2: Can refusal followed by a math answer earn the same HarmInject reward as harmful compliance?

A2: Yes. TTRL rewards matching the majority math answer; harmful compliance is not required. We manually checked one example from the existing Qwen-1.5B HarmInject log at step 10. For the same prompt, the majority math answer was 5:

ResponseRewardGRPO advantage*
Refuses the entire prompt without answering the math question0−0.62
Refuses the harmful request, gives math answer 51+1.57
Complies with the harmful request, gives math answer 51+1.57

Advantages reconstructed from logged rewards.

The advantage applies across the response, so the math reward can reinforce either harmful text or refusal text. Harmful content can receive a positive training signal alongside a matching math answer, but does not earn more reward than refusal-plus-math. This example does not establish why harmful responses become more common than refusals during training. We will revise contribution 2 and Section 4.3 and remove the claim that only harmful compliance can receive reward.

Q3: Do reasoning tax and HarmInject effects persist in paired clean and injected runs with matched math exposure, including a larger model?

A3: Appendix D reports HarmInject ASR increases on Llama-8B of +26.8 and +61.4 pp, while Qwen-7B shows −12.2 pp. For context, Table 11 reports +0.2 pp for a separate clean Llama-8B run. These results provide larger-model observations, but the clean and HarmInject runs are not seed-matched controls with verified equal AMC exposure.

For reasoning tax, one saved-log check compared clean steps 215–219 with mixed steps 346–350 on Qwen-1.5B + JBV28k. Accuracy was 34.0% clean versus 34.7% mixed at roughly matched expected AMC exposure. It shows no positive reasoning tax in that comparison, but is not a controlled paired experiment. We do not claim the reasoning tax persists after exact AMC matching. Unfortunately, our compute budget limits additional training.

AuthorsOpenReview ↗

Response to reviewer hYpo's weaknesses

W1: Contribution and evidence for the explanation

Our contribution concerns the impact on the safety and reasoning of LLMs when applying reasoning-based rewards to mixed prompts. A1 connects logged rewards to within-group GRPO advantages; A2 shows how harmful content can receive a positive advantage alongside a matching math answer.

We also checked subsequent behavior on the 100 JBV evaluation prompts. Refusal-phrase shares rose from 54% to 68%, 58% to 69%, and 51% to 63%, comparing initialization with each run's final-20-step average. These evaluation-response measurements are separate from the training-rollout shares in A1 and from the top-template metric. Together, the measurements support refusals being favored before saturation. They do not establish a complete causal explanation for every setting; we will narrow that wording.

We also completed a single-seed jailbreak-only control on Qwen-1.5B: 350 updates on 50 jailbreak prompts, with no AMC training prompts. On 65 disjoint jailbreak evaluation prompts, the exact response “I'm sorry, but I can't assist with that.” increased from 12/65 (18.5%) initially to 42/65 (64.6%), averaged over the final five evaluations. This shows that the repeated-refusal pattern does not require math prompts in training. Safety judging is pending, so we do not report this as an ASR decrease.

W2: Scope of the conclusions

Appendix C reports three-run ASR changes of −18.3 ± 8.1 pp for TTRL + JBV28k and +38.1 ± 8.9 pp for RENT + Instruct at the reported common endpoints (mean ± SD). Most other settings have one run, the main grid uses models up to 1.5B, and larger-model experiments cover selected settings only. These results do not establish an exposure-matched reasoning tax or a consistent HarmInject direction across models. We will state explicitly that training repeatedly reuses a finite, shuffled prompt pool. We will also remove the general conclusion that output-content filtering is ineffective: our observations concern the tested LlamaGuard reward-filter configurations, not filtering methods in general.

Thank you for your response. I remain concerned that the issue you identified may depend on the experimental setup, so I will keep my original score.

Reviewer_Jx39OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies self-rewarding test-time RL methods (TTRL and RENT) when math reasoning prompts are mixed with jailbreak or benign instruction prompts. It reports three effects: lower attack success rate (safety amplification), higher attack success rate (harmfulness amplification), and reduced math accuracy relative to clean training (a reasoning tax). They also propose HarmInject, an attack that pairs a harmful request with a math problem so that complying is rewarded, and claim that a LlamaGuard-based reward filter does not mitigate these failures.

Critical strengths
  1. The paper presents the risk of the unsupervised TTRL and RENT approaches in a new heterogeneous setting where jailbreak inputs can be a part of the training.
Critical weaknesses
  1. Before understanding the impacts in a heterogeneous setting, we must completely explore the impacts in homogeneous settings. Although it considers and compares with the homogeneous setting of only AMC, it does not report the homogeneous setting of only jailbreaking or how the heterogeneous setting leads to different results than the homogeneous setting of only jailbreaking. Hence, the paper might still be incomplete without all explored settings and how the behavior shifts in their newly reported setting from existing homogeneous settings. Moreover, their analysis is only constrained to two methods: TTRL and RENT.

  2. Most results are on 0.5–1.5B models. At 8B, most effects shrink or vanish: Llama-8B moves within ±1.4 pp across injection ratios (Table 19), and the clean-AMC ASR difference on Llama-8B is near zero (Table 15). Moreover, the results seem to vary with re-runs. In such cases, each of the main results should, at least at a small scale, be reported based on multiple runs.

Questions for authors
  1. What are the differences between the initial models for the initial ASR to be different for TTRL and RENT, as reported in Table 1? Before training, shouldn't the two numbers be identical? If not, why are we not reporting the ASR for multiple runs, as each run can give different results?
  2. In Appendix A, it is mentioned that for TTRL, "if there is no number, it uses the response’s final whitespace-delimited token as the label." Majority-vote agreement on jailbreak prompts (Figure 4A, MC/64 ≈ 1.0) can therefore reflect rollouts that merely end with the same word, not consistent behavior. Is the safety TTLR finding therefore merely being affected by the manual choice of label instead?
  3. Section 3 claims the 350-step injected runs see as many AMC prompts as the 250-step clean run, but at 8 prompts per step the clean run sees 2,000 AMC prompts, while the injected runs see 2,800 prompts, of which AMC is only 62.5% (1,750) or 40% (1,120), depending on which stated injection ratio applies. The reasoning tax may therefore reflect less math training and more contribution of safety–reasoning dataset ratios in training. Hence, how does changing the safety-to-reasoning ratios in the training dataset impact the results?
  4. Under RENT, the adversarial construction (Table 4) is weaker than injecting benign instructions (Table 3) on all three models. Thus, a bit more explanation regarding contribution 2 in this regard is necessary, as it says adversarial contribution, "yielding stronger harmfulness amplification".
  5. Line 85 claim: "On jailbreak-injected streams, the reinforced mode is refusal when the model is already safe and compliance when it is vulnerable" needs experimental evidence and clearer logic.
  6. Figure 2 x-axes (300) don’t match the stated 250/350.
  7. AMC is never introduced. It appears in Figure 1 and lines 086–089 without explanation.
4 comments
AuthorsOpenReview ↗

Response to reviewer Jx39

We thank the reviewer for their feedback. We address the questions below.

Q1: Why do TTRL and RENT have different initial ASRs in Table 1? Why not report multiple runs?

A1: Both methods were configured to start from the same pretrained model for each row, but their baseline responses were generated separately. Sampling can produce different ASRs, and the evaluation setups also differed. Qwen-1.5B used FP16 under TTRL and BF16 under RENT. Llama-1B's TTRL evaluation prompts included “Please reason step by step, and put your final answer within \boxed{}”; RENT's did not. We therefore cannot attribute all differences to randomness and will state these evaluation differences explicitly.

Appendix C, Table 9 already reports three runs for selected Qwen-1.5B settings. At the common endpoint of step 244, clean TTRL's ASR change was +2.1 ± 4.6 percentage points (mean ± SD). Thus, the small Table 1 decrease did not hold across runs. We will identify repeated and single-run results clearly.

Q2: Does the label-extraction rule explain the apparent safety amplification?

A2: The rule affects the result. The tested TTRL implementation first extracts an answer; if it finds no numerical answer, its main fallback uses the final whitespace-separated token. Similar endings can therefore receive the same label and reward even when the responses differ. MC/64 near 1 alone does not establish matching meaning or safety. ASR is measured separately by judging full responses.

Appendix A, lines 702–709, already tests an alternative that assigns responses without an extracted answer the same empty label. This produces a larger ASR decrease, showing sensitivity to the fallback. Our finding concerns the tested implementation; we do not claim that every parser produces the same safety shift. We will make this clear in the paper.

Q3: How do the injection ratio and amount of math training affect the results?

A3: Appendix M, Table 19 reports settings labelled 5–60% injection. For Qwen-1.5B + JBV28k, ΔASR is +0.8 pp at 5%, −7.0 pp at 10%, and −14.0 pp at 30%. The safety shift therefore appears below the default setting, but does not grow steadily with the ratio.

The default pool contains 83 AMC and 50 injected prompts: about 60% relative to the AMC count, but 37.6% of the combined pool. Under the stated 250/350-step budgets and batch size eight, expected AMC exposure is approximately 2,000 clean versus 1,747 mixed presentations. We will correct the equal-exposure claim, use a consistent ratio definition, and state the actual budget for each reported run.

The reasoning tax describes the observed difference in final AMC accuracy. Fewer AMC presentations may contribute to that difference; the ratio sweep does not establish that the reasoning tax persists after exact exposure matching.

Q4: Why does contribution 2 claim stronger harmfulness amplification when HarmInject is weaker than benign Instruct injection under RENT?

A4: That wording is too broad. On Qwen-1.5B under RENT, Tables 3–4 report +35.0 pp for Instruct versus +8.8 pp for HarmInject. HarmInject was designed to test whether harmful content can receive reward alongside a math answer. Under TTRL on Qwen-1.5B, the reported ΔASR changes from −9.4 pp with ordinary JBV28k injection to +11.6 pp with HarmInject.

We will revise contribution 2 to say that HarmInject increases harmfulness in selected TTRL and RENT settings. We do not claim that it is the strongest injection method across models or rewards, or that harmful compliance is required to receive the math reward.

Q5: Does initial safety determine whether refusal or compliance is reinforced, as claimed at line 85?

A5: TTRL rewards matching the majority extracted label. Diverse harmful answers can split the vote, while repeated refusals share one label and win. A high initial ASR therefore does not necessarily predict increasing harmfulness.

Figures 4B and 5 track response patterns, but do not establish a reliable prediction rule. The first-batch test in Appendix C predicted the ASR direction in only 6/9 settings. We will replace the sentence with: “On jailbreak-injected streams, training can strengthen refusals or harmful responses; the outcome varies with the model, reward, and injected prompts.”

Q6: Why does Figure 2 extend to 300 steps when the text states 250/350?

A6: Figure 2 uses clean-AMC trajectories with 301 saved evaluations, including initialization, extending to about 300 training updates. The stated 250-step budget does not describe these clean AMC runs; 350 refers to injected runs, which Figure 2 does not show. We will correct Section 3 and specify the training lengths used for the reported trajectories.

Q7: AMC is not introduced at its first appearance.

A7: We will introduce AMC as the American Mathematics Competitions math reasoning dataset at its first appearance in Figure 1 and in the introduction.

I thank the authors for their responses and clarifications.

However, my main concerns remain unaddressed.

Moreover,

  1. Regarding A1. The authors mentioned, "the evaluation setups also differed. Qwen-1.5B used FP16 under TTRL and BF16 under RENT". If TTRL and RENT are running under different setups, how can we be sure that they will follow the same trend under similar setups? We cannot compare these two methods under different setups in Table 1. Thus, the reported conclusion from Table 1 comparing TTRL and RENT in Section 4.1 also remains unsupported.

  2. If generating responses separately indicates a difference in performance, are the reported gains or degradation of "init" vs. "final", reported for a single-run algorithm, also being affected by this response generation noise?

  3. Regarding A3, can you check whether the main conclusions of the paper regarding reasoning tax hold under equal exposure and ensure that the reasoning tax is not just reflecting less math training?

AuthorsOpenReview ↗

Response to the reviewer Jx39 comment

We thank the reviewer for following up and for engaging in further discussion. Below we address each point using additional analyses of our existing runs.

1. Comparing TTRL and RENT. A1: We cannot be sure from our current results that TTRL and RENT would show the same relative trends under identical settings, since they differ in precision and, for Llama-1B, in prompts. We agree that Table 1 does not support the comparison in Section 4.1. We will remove that conclusion, along with the same comparison in the abstract, Fig. 1C, Fig. 3 and Section 4.2, and report the two methods separately. Our main claim does not require comparing the two methods. It is that each method changes held-out safety on mixed test-time streams, which we measure within each run under fixed settings. These within-method effects repeat across three runs (TTRL + JBV28k: −18.3 ± 8.1 pp; RENT + Instruct: +38.1 ± 8.9 pp; App. C) and hold under a second safety judge (App. O). We will list the precision and prompt settings for each run in the appendix and mark which results come from a single run.

2. Noise in initial versus final ASR. Yes, sampled generation also affects these changes. Across three clean Qwen-1.5B TTRL runs at a common endpoint, ΔASR was +2.1 ± 4.6 pp (mean ± SD), so the small decrease in Table 1 is not consistent. The larger mixed-stream changes repeat. Under TTRL, Qwen-1.5B + JBV28k gave −17, −11 and −27 pp across three runs at step 300. Under RENT, Qwen-1.5B + Instruct gave +32.4, +48.4 and +33.6 pp. All six prompt-paired 95% confidence intervals exclude zero. We will emphasize repeated findings from single-run changes in the main paper.

3. Reasoning tax under equal math exposure. We checked this with existing TTRL + JBV28k runs. For every model, we compare the clean run at steps 288–292 with the mixed run at steps 476–480, the points where their AMC exposure matches. The reasoning tax remains for Llama-3B but not consistently for the other three models:

ModelClean AMC acc. (steps 288–292)Mixed AMC acc. (steps 476–480)Clean − mixed
Llama-3B31.26%18.10%13.16 pp
Qwen-1.5B34.82%34.84%−0.02 pp
Qwen-0.5B14.46%16.90%−2.44 pp
Llama-1B (clean run 1)2.42%6.00%−3.58 pp
Llama-1B (clean run 2)14.82%6.00%+8.82 pp

With 83 AMC and 50 jailbreak prompts, batch size 8, and incomplete batches dropped, each mixed epoch contains at least 78 AMC prompts. So 480 mixed steps include at least 2,340 AMC prompts, compared with 2,336 at clean step 292. Each accuracy is the average of the five evaluations in the stated step range.

For Llama-3B, where model, precision, prompt and optimizer settings match, the runs started 3.1 pp apart at step 0 (21.2% vs. 18.1%). Clean training raised accuracy to 31.26%, while mixed training stayed at 18.1%. Additional math training did not recover the lost accuracy, even after the mixed run had more AMC exposure than the clean reference. For Qwen-1.5B and Qwen-0.5B, there is no reasoning tax at equal exposure, and the Qwen-1.5B pair also differs in learning-rate warmup and evaluation prompts. For Llama-1B, the two clean runs disagree. These are single runs, with exposure matched by count rather than per problem. We will report all four models in Appendix D and limit the reasoning-tax claim to the settings where it holds.

Final takeaway. Our main finding is that self-rewarding test-time RL changes held-out safety on mixed prompt streams and can also lead to the reasoning tax. This holds within each method, repeats across three runs, and holds under a second safety judge. It does not depend on comparing TTRL with RENT or on matching math exposure. Following the reviewer's points, we will remove the cross-method comparison, mark single-run results, and present the reasoning tax as a loss observed in many settings, not a general effect.

I thank the authors for the insightful response.

I am concerned that the paper requires changes to its core conclusions and requires weakening the claims across the paper. Thus, the paper still requires comparatively substantial changes overall.

Hence, I will keep my original scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.