acceptodds
Under review as a conference paper at ICLR 2027

From Correct Answers to Correct Supervision in Self-Rewarded Test-Time Learning

Abstract

Self-rewarded learning reinforces the answers that win a model's votes. A correct response can therefore be available without becoming useful supervision. We study this selection step through recorded reward groups, controlled oracle-reward mixtures, and diagnostics of adaptation outcomes. Across 74 self-rewarded Llama-3-8B runs on screened GSM8K, a correct answer appears in 91.5% of training groups but is selected in 63.7%. We derive the group-dependent oracle weight at which an available correct response acquires a positive centered reward; its median is 1/3 among wrong-vote groups containing that response. Across 288 reward-intervention runs, oracle anchoring raises mean held-out gain and reduces its variability. Selection correctness also provides information before the trajectory is complete: using the first 19 of 38 recorded groups, it reduces leave-one-seed-out gain-prediction error by 19.2% beyond response accuracy, with a 16.1% reduction in the reward-mixture cohort after controlling reward weight. These results distinguish producing a correct answer from selecting it for reinforcement, and establish reward selection as a measurable link between available responses and adaptation outcomes.

Primary area:
reinforcement learning
Keywords:
test-time reinforcement learning, self-rewarding models, majority-vote pseudo-labels, reward poisoning, certified defenses, evaluation audit

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_64thOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes self-rewarded test-time learning and tests it on GSM8K. Specifically, they combine information on majority-predicted answers as well as oracle-correct answers in order to improve answer selection.

Critical strengths
  1. The experiments include a range of models across 0.6-8B.
Critical weaknesses
  1. The introduction/abstract fails to explain why self-rewarding is well-motivated for readers who’s less familiar with this literature (compared to just using the oracle answer).
  2. Only 1 test-set was used (GSM8K – a very easy math dataset on grade school problems) meaning that the generality of this approach is not determined based on the experiments.
Questions for authors
  1. It’s not clear to me why we want to consider the r_majority at all when we have access to r_oracle (Table 3 also supports this conclusion).
3 comments
AuthorsOpenReview ↗

Response to Reviewer 64th

Thank you for highlighting our multi-model evaluation (0.6B–8B) and constructive questions.

1. W1 & Q1: Why Consider rmajorityr_{\text{majority}} When We Have Access to roracler_{\text{oracle}}?

When ground-truth labels (roracler_{\text{oracle}}) exist in practice, one should train with roracler_{\text{oracle}} directly—Table 3 and our 12-seed paired grid on S1 (λ=1\lambda=1 beats λ=0\lambda=0 by +17.50 pp, [+11.00,+23.25][+11.00, +23.25]) confirm roracler_{\text{oracle}} is far superior on GSM8K.

  1. Why rmajorityr_{\text{majority}} is used in TTRL: TTRL (Zuo et al., 2025; Wang et al., 2026; Yu et al., 2026; Yan et al., 2026) is designed for unlabeled test-time adaptation, where a deployed model adapts to new test questions without ground-truth answers or an external verifier. Because roracler_{\text{oracle}} is unavailable at test time, the learner samples NN responses per question and uses its own plurality vote (rmajorityr_{\text{majority}}) as a self-generated pseudo-label.
  2. Why our study uses a labeled benchmark and rλ=λroracle+(1−λ)rmajorityr_\lambda = \lambda r_{\text{oracle}} + (1-\lambda) r_{\text{majority}}: We do not propose rλr_\lambda for deployment (λ>0\lambda > 0 needs gold labels). Rather, our paper is an empirical audit and diagnostic study of majority-rewarded TTRL: (i) evaluating TTRL on benchmarks where gold answers are known to the experimenter (hidden from the majority learner) lets us separate failing to generate a correct answer (1−H1 - H) from generating a correct answer that loses the vote and receives negative centered reward (H−AH - A); and (ii) varying λ∈[0,1]\lambda \in [0, 1] at fixed policy and optimizer is a controlled experimental intervention to measure how adaptation responds as wrong-vote supervision is corrected. We revised the opening of the Abstract and Section 1 ([Revised PDF: Page 1]) so this motivation is immediately clear.

2. W2: Generality Beyond GSM8K (Competition-Level MATH + Disjoint Splits S1/S2 + Gemma-2-2B-it S1)

We ran (A) 30 trajectories on competition-level MATH (math-exact-v2 rational-value parser, 24 majority + 6 oracle seeds, CoT derivations with a 1,024-token cap, 300 GRPO steps on 200 train / 100 test items screened for 11–33 correct responses out of 44), and (B) 96 new trajectories across two mutually disjoint GSM8K splits (S1 and S2, including 12 new Gemma-2-2B-it seeds on S1):

Benchmark & CohortRunsBase → Post (%)Gain (pp) & 95% CIHH (%)AA (%)Gap H−AH - A (pp)
GSM8K Original (Llama-3-8B, 38g)7443.00→46.1043.00 → 46.10+3.10[+0.10,+6.00]+3.10 [+0.10, +6.00]91.5491.5463.6963.6927.84 (21.0221.02 strict / 6.836.83 tie)
GSM8K Split S1 (Llama-3-8B, 38g)2449.00→56.0449.00 → 56.04+7.04[+2.33,+11.79]+7.04 [+2.33, +11.79]92.0092.0070.7270.7221.27 (16.0116.01 strict / 5.265.26 tie)
GSM8K Split S2 (Llama-3-8B, 38g)2443.00→50.8343.00 → 50.83+7.83[+4.83,+10.88]+7.83 [+4.83, +10.88]89.9189.9168.3168.3121.60 (16.8916.89 strict / 4.714.71 tie)
GSM8K Split S1 (Gemma-2-2B-it, 19g)1237.00→32.5837.00 → 32.58−4.42 [−7.67,−1.17][−7.67, −1.17]——27.63 [22.81,32.89][22.81, 32.89]
Screened MATH (math-exact-v2, Maj, 19g / 38g)2443.00→42.7943.00 → 42.79−0.21[−1.67,+1.13]-0.21 [-1.67, +1.13]90.79 (19g)67.32 (19g)23.46 (12.5012.50 strict / 10.9610.96 tie; 38g: 23.25)
Screened MATH (math-exact-v2, Oracle λ=1\lambda=1)643.00→43.0043.00 → 43.00+0.00[−2.83,+2.67]+0.00 [-2.83, +2.67]———
  1. The Gap (H−A=21.3H - A = 21.3–27.827.8 pp) Replicates on MATH with CoT and Across Splits/Models: On screened MATH (math-exact-v2), a correct answer appears in 90.79% of the first 19 groups (90.24% over 38g) but wins the vote in 67.32% (67.00% over 38g), leaving a 23.46 pp gap (12.5012.50 pp strict minority, 10.9610.96 pp top-tie). On Split S1, Gemma-2-2B-it shows a 27.63 pp gap and degrades by −4.42 pp ([−7.67,−1.17][-7.67, -1.17]).
  2. Selection Gap and the 300-Step Adaptation Budget on Screened MATH: On competition-level MATH with CoT, H=90.79H=90.79\\% versus A=67.32A=67.32\\% leaves a 23.46 pp selection gap. This reproduces the availability–supervision bottleneck beyond grade-school arithmetic and short-answer prompting. Under 300-step QLoRA, mean pass@1 gain is −0.21-0.21 pp under Majority ([−1.67,+1.13][-1.67,+1.13]) and 0.000.00 pp even with 100% Oracle reward ([−2.83,+2.67][-2.83,+2.67]). Thus, correcting reward selection alone is insufficient for an average gain under this protocol; the result motivates testing greater model capacity or a larger update budget on these harder problems ([Revised PDF: Sec 5.1, 5.3, 6]).
Reviewer_64thOpenReview ↗

Reviewer response

Hi thanks for the response and the various experiments conducted:

W1/Q1: I think the combination of framing the paper as an "empirical audit and diagnostic study of majority-rewarded TTRL" and the use of datasets like GSM8K/MATH (where golden labels are available) doesn't convince me of the usefulness of majority-rewarded TTRL in tasks where golden labels are available. The usefulness of the finding of the paper is weak in my opinion.

W2: I appreciate the authors adding a separate MATH dataset in their experiments but I still feel that both GSM8K and MATH datasets both released in 2021 are somewhat irrelevant for measuring math capabilities for models today. In addition, the models used (Llama3, Gemma2, Qwen3) are all not the most recent generation of the same size, which makes me wonder if the more recent models have already solved these benchmarks and therefore if findings here can translate (e.g. Qwen 3.5 / Gemma4 / Llama 3.1).

Overall, I have no opinion on the validity/insightfulness/novelty of this paper's findings (as I am not familiar with TTRL literature) but my opinion is that the findings are not really useful for improving recent models and hence would keep my original recommendation.

AuthorsOpenReview ↗

Clarifying the Role of Benchmark Labels and Scope of the Mechanism for Newer Models

Dear Reviewer 64th,

Thank you for engaging with our response and sharing your perspective. We would like to clarify two core points so that the role of our experiments and the scope of our findings are clear:

  1. Why labeled benchmarks are used to audit an unlabeled algorithm:

    • We do not advocate using majority-rewarded TTRL when ground-truth rewards are available to the learner; at test time, TTRL is designed for unlabeled inputs where gold answers are unavailable.
    • Gold-labeled benchmarks provide the reference needed to determine whether self-generated pseudo-labels are correct. Gold answers are not used to compute rewards in the majority-only condition; they are used to audit reward selection and are deliberately introduced in the oracle-mixture experiments as a controlled intervention (Section 1, lines 41–46). This follows standard TTRL evaluation practice (e.g., Zuo et al., 2025, who define TTRL on unlabeled inputs and report gold-dependent diagnostic metrics in Appendix E).
    • This audit reveals a concrete failure mode: across GSM8K splits and MATH CoT, 21–28%21\text{--}28\% of recorded training groups (Table 2b, p. 6) contain an available correct answer that loses the plurality vote and receives a negative centered reward. Moreover, in a separate synthetic streaming experiment, a simple margin gate reduces conditional wrong-update admission from 13/4613/46 (28.3%28.3\%) to 2/482/48 (4.2%4.2\%) while retaining 47.9%47.9\% of proposed updates (Table 20, p. 25).
  2. Scope of the mechanism and relevance to newer models:

    • We evaluated four policies spanning 0.6B–8B0.6\text{B}\text{--}8\text{B} on GSM8K and added a CoT study on screened MATH, observing a 19.3–43.019.3\text{--}43.0 pp selection gap across the 19-group cross-model cohorts (Table 7, p. 9) and a 23.4623.46 pp gap on MATH CoT (Table 2b, p. 6).
    • Once a model saturates a benchmark such as GSM8K, this failure mode should be evaluated on harder tasks where the model still makes errors and produces mixed-vote groups.
    • Proposition 1 (§3.3) is conditional on the sampled vote composition: whenever an available correct answer loses the selected plurality, Proposition 1 characterizes its adverse centered supervision and the exact threshold λg∗=sg/(1−pg+sg)\lambda_{g}^{*} = s_{g}/(1 - p_{g} + s_{g}), above which the correct response receives positive centered reward. This characterization does not depend on the model's release date; its relevance to newer models depends on whether they exhibit these mixed-vote configurations on the harder tasks where they are still uncertain.

We hope this clarifies the role of gold labels in auditing unlabeled TTRL and the relevance of our characterization: whenever an available correct answer loses the plurality vote, the same reward mechanism applies, irrespective of model generation.

Reviewer_LrzyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether correct answers sampled during majority-vote self-training actually receive positive reinforcement. Across 74 Llama-3-8B runs on GSM8K, correct answers appear in 91.5% of groups but win only 63.7% of votes. The paper derives when oracle reward turns correct minority answers into positive supervision, finds that adding oracle reward improves gains and stability across 288 runs, and shows that early vote correctness helps predict final gains.

Critical strengths
  • The paper shows why being able to produce a correct answer is not enough for self-training to work. Measuring how often correct answers lose the vote gives a useful way to check whether training is reinforcing mistakes.

  • The experiments show how much results vary under self-generated rewards. Across 288 runs, training based only on votes sometimes makes performance worse, while every run using answer-key rewards improves performance. The authors also report cases where their prediction method provides only a small benefit.

Critical weaknesses
  • The experiments show that adding correct rewards helps, but do not directly test the paper’s explanation of when a correct minority answer starts benefiting from training. Use existing records to check whether rewarding these minority answers predicts improvement, and test a few reward mixtures between 0.25 and 0.5 to examine the predicted transition more closely.

  • The main experiments use one model, selected arithmetic questions, short answers without reasoning, and only about 38 groups of answers per run. This leaves it unclear whether the same pattern holds during longer training with reasoning responses. Report how often correct answers are generated but lose the vote in the existing cross-model runs, or narrow the title and abstract to the tested setting.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer Lrzy (Part 1/2)

We thank Reviewer Lrzy for the positive assessment (Rating: 3) and constructive experimental suggestions.

1. Weakness 1: Testing When Correct Minority Answers Benefit (Dense 0.250.25–0.500.50 Grid + Local Checkpoint Branching)

Section 3.3 (lines 164–195) separates two group-level thresholds for a wrong-vote group with plurality share sgs_g and correct share pg>0p_g > 0: (i) the sign threshold λg∗=sg/(1−pg+sg)≤1/2\lambda_g^* = s_g / (1 - p_g + s_g) \le 1/2 (median 1/31/3), above which the correct minority's centered reward bg(λ)b_g(\lambda) flips from −- to ++; and (ii) the ordering threshold λ=1/2\lambda = 1/2, above which the correct response receives strictly larger reward than the wrong plurality.

A. Dense 8-Point Seed-Paired λ\lambda-Grid on GSM8K Split S1 (12 Shared Seeds × 8 Weights = 96 Runs)

Reward Weight λ\lambda0.000.000.250.250.300.301/31/30.400.400.500.500.750.751.001.00
Mean pass@1 Gain (pp, n=12n=12)+6.75+6.75+6.83+6.83+5.33+5.33+6.92+6.92+13.67+16.00+19.17+24.25
Paired Diff vs. λ=0\lambda=0 & 95% CIBase+0.08[−6.75,6.84]+0.08 [-6.75, 6.84]−1.42[−10.34,5.84]-1.42 [-10.34, 5.84]+0.17[−6.92,6.42]+0.17 [-6.92, 6.42]+6.92 [1.00,13.33][1.00, 13.33]+9.25 [2.17,15.50][2.17, 15.50]+12.42 [7.33,17.33][7.33, 17.33]+17.50 [11.00,23.25][11.00, 23.25]

In the historical wrong-vote groups containing a correct response, the median sign threshold is 1/31/3: at λ≤1/3\lambda \le 1/3, at least half assign ≤0\le 0 centered reward to y∗y^*, while the wrong plurality receives the larger raw reward. On the paired grid, gains remain flat in this range (+5.33+5.33 to +6.92+6.92 pp). Once λ\lambda reaches 0.400.40 and 0.500.50 (near the 1/21/2 ordering crossover), mean gain jumps to +13.67 pp and +16.00 pp, reaching +24.25 pp (Δ=+17.50\Delta = +17.50 pp [11.00,23.25][11.00, 23.25]) at λ=1\lambda = 1.

B. Local Checkpoint-Branching Mechanism Probe Across λ∈0,λg∗−0.02,λg∗+0.02,0.5,1\lambda \in \\{0,\\ \lambda_g^*-0.02,\\ \lambda_g^*+0.02,\\ 0.5,\\ 1\\}

From identical restored checkpoints on wrong-vote groups (H=1,A=0H=1, A=0), we ran 1-group (8-step) GRPO updates on a 4-seed batch (29/32 eligible groups) and an 8-seed replication batch (60/64 eligible groups: 42 strict minority, 18 top-tie). Define λ−=λg∗−0.02\lambda_-=\lambda_g^*-0.02 and λ+=λg∗+0.02\lambda_+=\lambda_g^*+0.02.

First, crossing λg∗\lambda_g^* flips the centered advantage from negative to positive in 100% of groups: 60/60 in the replication and 29/29 in the initial batch. The prespecified narrow step λ+−λ−\lambda_+ - \lambda_- also significantly improves the per-token correct-versus-wrong log-probability update in both batches. Relative to majority reward (λ=0\lambda=0), the above-threshold branch significantly mitigates the negative sequence update, including in the strict-minority subset:

Local Contrast8-Seed Batch (60g): Mean & 95% CI4-Seed Batch (29g): Mean & 95% CI
Advantage sign: λ−→λ+\lambda_- → \lambda_+60/60 Neg → 60/60 Pos29/29 Neg → 29/29 Pos
Prespecified narrow step: per-token (secondary)+0.00505[+0.00151,+0.00873]+0.00505 [+0.00151,+0.00873]+0.00915[+0.00623,+0.01192]+0.00915 [+0.00623,+0.01192]
Post-hoc vs. majority: sequence (primary), λ+−0\lambda_+ - 0+0.07811[+0.02669,+0.13994]+0.07811 [+0.02669,+0.13994]+0.12870[+0.02488,+0.28689]+0.12870 [+0.02488,+0.28689]
Post-hoc vs. majority: sequence, strict minority only+0.09616[+0.0163,+0.1928]+0.09616 [+0.0163,+0.1928] (42g)—

These results distinguish the sign threshold, which turns an available correct answer into positive centered supervision, from the ordering threshold at 1/21/2, above which its raw reward exceeds the wrong plurality's. With shared LoRA parameters and multiple optimizer steps, reward signs, relative sequence updates, and absolute sequence-probability growth need not cross zero together. The paired trajectory grid shows gains rising to +13.67 pp at λ=0.40\lambda=0.40 and +16.00 pp at λ=0.50\lambda=0.50, near the ordering crossover. In the 8-seed local probe, a positive mean correct-versus-wrong sequence update is observed at λ=1\lambda=1 (+0.09407). Complete narrow-step sequence contrasts and all branch results are retained in Appendix B.1 ([Revised PDF: Sec 5.2, Tab 3c; App B.1 Tabs 8–9]).

AuthorsOpenReview ↗

Response to Reviewer Lrzy (Part 2/2)

2. Response to Weakness 2a: Cross-Model Availability–Selection Gap Breakdown (Expanded to 42 Runs)

Across all four models in our original 30-run cross-model cohort plus our 12 new Gemma-2-2B-it runs on Split S1 (4242 runs total), between 19.3019.30 pp and 39.4739.47 pp of all recorded training groups contain a correct answer that loses the plurality vote:

Model & Split CohortRuns / Recorded Groups (38g unless stated)Base pass@1 (%)Mean Gain (pp)Availability HH (%)Selected AA (%)Unselected Gap H−AH - A (pp)
Llama-3-8B-Instruct (Orig. split)12 / 45643.0+0.5+0.590.7961.1829.61
Gemma-2-2B-it (Own screened split)6 / 22841.0+20.793.8671.4922.37
Gemma-2-2B-it (New Split S1, 19g)12 / 228 (19g)37.0−4.42 [−7.67,−1.17][−7.67, −1.17]——27.63 [22.81,32.89][22.81, 32.89]
Qwen3-8B (Own screened split)6 / 22865.0+1.2+1.296.9377.6319.30
Qwen3-0.6B (Own screened split)6 / 22822.0+6.3+6.368.8629.3939.47

Comparing the two Gemma-2-2B-it cohorts using 19-group prefixes provides a striking within-model contrast: for the exact same model architecture, shifting from a split where its selection correctness is high (A=64.9A = 64.9\\%, H-A = 26.3 pp) to Split S1 where its base accuracy is lower (37.037.0\\%) and its selection gap widens (H−A=27.63H-A = 27.63 pp) flips majority-rewarded training from +20.7 pp improvement to −4.42 pp degradation (9 of 12 seeds negative). Split, runtime and batch differences prevent isolating their individual causes. Moreover, across all 42 cross-model runs, adding AA to Model ID + P significantly reduces 19-group seed-LOSO RMSE from 7.80 to 6.43 pp (Δ=+1.37\Delta = +1.37 pp, 95% CI [+0.16,+2.58][+0.16, +2.58]).


3. Response to Weakness 2b: Reasoning Responses (CoT), 1,200-Step Horizon, Screened MATH, and Scoped Claims

We ran (A) a 2 × 2 prompt × horizon study on GSM8K Split S1 across 12 shared seeds (terse vs. chain-of-thought derivation under a matched 1,024-token cap, trained to 1,200 steps = 150 groups with intermediate evaluation at 300 steps), and (B) a 30-run study on screened MATH with CoT derivations (24 majority + 6 oracle seeds, 1,024-token cap, 300 steps, math-exact-v2 parser):

Cohort & Prompt Mode (Cap 1024)Shared SeedsBase pass@1 (%)19-Group H−AH - A Gap (pp)300-Step Gain (pp)1,200-Step Gain (pp) & 95% CIPaired 300→1200300 → 1200 Step Delta (pp)
GSM8K S1: Terse (Direct Answer)1249.0049.0029.39−0.08-0.08+0.67[−14.08,+14.75]+0.67 [-14.08, +14.75]+0.75[−6.34,+7.75]+0.75 [-6.34, +7.75]
GSM8K S1: Derivation (CoT)1274.0074.003.51+0.67+0.67+1.08[−0.33,+2.58]+1.08 [-0.33, +2.58]+0.42[−1.67,+2.58]+0.42 [-1.67, +2.58]
Screened MATH: Derivation (Majority)2443.0043.0023.46 (90.7990.79\\% - 67.32\\%)−0.21[−1.67,+1.13]-0.21 [-1.67, +1.13]— (38-group H−AH-A: 23.25 pp)—
  • Takeaways & Scoped Claims: On GSM8K S1 (screened under terse prompting at 4949\\% base accuracy), switching to CoT raises base pass@1 to 74.0074.00\\% and narrows H−AH - A to 3.513.51 pp, whereas on harder in-band screened MATH problems (where CoT base pass@1 is 43.0043.00\\%), the large 23.46 pp selection gap re-emerges under CoT. Extending GSM8K training from 300 to 1,200 steps yields no reliable extra gain (+0.75+0.75 pp and +0.42+0.42 pp). On screened MATH, the 23.46 pp gap persists under CoT, while 300-step QLoRA yields mean pass@1 gains of −0.21-0.21 pp under Majority and 0.000.00 pp under 100% Oracle reward. Correcting the supervision signal alone is therefore insufficient for an average gain under this protocol, motivating tests of greater model capacity or a larger update budget. Following your recommendation, we report the full cross-model H−AH-A table and explicitly scope our Abstract, Introduction, and Discussion to these tested regimes ([Revised PDF: Abstract, Section 5.4, Section 6]).

Thank you very much for your responses. My concerns have been addressed. I keep my score in the accept range

Reviewer_Yf7kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether correct answers generated during self-rewarded learning are actually selected for reinforcement. It derives when mixing in ground-truth rewards gives a correct response a reward above the group average. Experiments on screened GSM8K examine training gains and whether vote correctness helps predict these gains beyond response accuracy and vote concentration.

Critical strengths
  1. The paper makes the gap between answer availability and supervision concrete. Correct answers appear in 91.5% of recorded groups but are selected in only 63.7%. This is useful evidence of how often voting fails to reinforce an available correct answer.

  2. The diagnostic comparisons use the same responses and gold labels, with controls for response accuracy and vote concentration. This makes the additional information provided by selection correctness more convincing.

Critical weaknesses
  1. The added predictive value is modest against the most complete baseline. With response accuracy and all five concentration signals, the 19-group RMSE reductions are 8.0% and 4.6%, and the reported intervals for improvement include zero. The paper acknowledges this uncertainty, but stronger evidence is needed to establish the size and reliability of this benefit.

  2. The main prediction experiments share one screened GSM8K split and contain only 38 generated groups per run. Holding out training seeds does not establish that the result transfers to other problem samples. The additional model experiments report correlations, rather than repeating the main comparison against response accuracy.

Questions for authors
  1. Using the existing logs, does selection correctness still improve prediction when answer availability HH is added to response accuracy and the concentration features?

  2. Among the 783 groups with an available but unselected correct answer, how many involve the correct answer tying for the highest vote count? This would help assess the contribution of tie handling to the reported gap.

  3. How much do the training seeds overlap across the reward weights in Table 3? Where seeds overlap, do paired comparisons show the same trend?

  4. As an additional validation, could the main 19-group prediction comparison be repeated on an independently sampled split using the same screening rule?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Yf7k (Part 1/2)

Thank you for the rigorous critique. In self-rewarded GRPO, generating a correct response (HH) without winning the vote (AA) turns that group into negative supervision against y∗y^*. Against the 5 gold-free concentration signals (Tabs 6 & 18, 38g), +A+A cuts LOSO error by 44.3% (9.84→5.489.84→5.48 pp, 74 Maj) and 33.1% (7.06→4.737.06→4.73 pp, 131 Mix). Combining gold supply (P,HP, H) with all 5 concentration signals (where P+5P+5 gave 8.08.0\\% & 4.64.6\\% in Orig 19g) constrains group votes without algebraically reconstructing run-level AA: 2-variable P+AP+A (6.68 pp) beats all 7 controls without AA (7.40 pp), and +A+A significantly improves on all 7 features across the 122122 pooled majority runs ([+0.11,+1.08][+0.11, +1.08]).

1. Q1 & W1: Controlling for Availability (HH) and Full 7 Controls (19-Group Prefix, Seed-LOSO)

Cohort (19g, Seed-LOSO)Baseline Control SetNo AA+A+ARed.Δ\DeltaRMSE (pp) & 95% Seed-Cluster CI
Orig. Maj (n=74n=74)P+HP+H / P+P2+HP+P^2+H7.867.86 / 7.707.706.52 / 6.5817.0% / 14.6%+1.34[0.35,2.29]+1.34 [0.35, 2.29] / +1.12[0.20,2.04]+1.12 [0.20, 2.04]
Orig. Maj (n=74n=74)Full 7 (P+H+5P+H+5 signals)7.437.436.857.8%+0.58[−0.09,+1.23]+0.58 [-0.09, +1.23]
Orig. Mix (n=131n=131)P+H+λP+H+\lambda / P+P2+H+λP+P^2+H+\lambda7.157.15 / 7.147.146.04 / 6.0815.5% / 14.8%+1.10[0.50,1.70]+1.10 [0.50, 1.70] / +1.06[0.47,1.64]+1.06 [0.47, 1.64]
Orig. Mix (n=131n=131)Full 7 (P+H+5P+H+5 signals +λ+\lambda)6.266.266.043.5%+0.22[−0.03,+0.46]+0.22 [-0.03, +0.46]
Pooled Maj (n=122n=122, 3 splits)P+SplitP+\text{Split} / P+H+SplitP+H+\text{Split}8.008.00 / 8.038.036.68 / 6.7416.5% / 16.1%+1.32[0.60,2.05]+1.32 [0.60, 2.05] / +1.29[0.56,2.03]+1.29 [0.56, 2.03]
Pooled Maj (n=122n=122, 3 splits)P+P2+H+SplitP+P^2+H+\text{Split} / 2-PC of 77.847.84 / 8.978.976.71 / 6.7214.4% / 25.1%+1.13[0.43,1.85]+1.13 [0.43, 1.85] / +2.25[1.32,3.19]+2.25 [1.32, 3.19]
Pooled Maj (n=122n=122, 3 splits)Full 7 + Split (OLS / Ridge)7.407.40 / 7.377.376.80 / 6.818.1% / 7.6%+0.59[0.11,1.08]+0.59 [0.11, 1.08] / +0.56[0.09,1.03]+0.56 [0.09, 1.03]
Pooled Mix (n=167n=167, 2 splits)P+λ+SplitP+\lambda+\text{Split} / P+H+λ+SplitP+H+\lambda+\text{Split}6.916.91 / 6.956.955.93 / 5.9814.2% / 14.0%+0.99[0.39,1.52]+0.99 [0.39, 1.52] / +0.97[0.37,1.51]+0.97 [0.37, 1.51]
Pooled Mix (n=167n=167, 2 splits)P+P2+H+λP+P^2+H+\lambda / 2-PC +λ+\lambda / Full 7 +λ+\lambda6.93/7.51/6.046.93 / 7.51 / 6.046.01 / 5.93 / 5.8913.3 / 21.0 / 2.5%+0.92[0.34,1.43]+0.92 [0.34, 1.43] / +1.58[0.78,2.32]+1.58 [0.78, 2.32] / +0.15[−0.07,0.36]+0.15 [-0.07, 0.36]

All pooled models include split indicators. Controlling for HH yields 13.3%–17.0% reductions with 95% CIs strictly >0>0 across all cohorts (38g: Pooled Maj 5.85→5.135.85→5.13 pp [+0.26,+1.17][+0.26, +1.17]; Mix 5.11→4.595.11→4.59 pp [+0.21,+0.81][+0.21, +0.81]), and adding AA to all 7 controls in Pooled Maj (n=122n=122) is significant under OLS ([+0.11,+1.08][+0.11, +1.08]) and Ridge ([+0.09,+1.03][+0.09, +1.03]) ([Revised PDF: Sec 5.3, Tab 5; App H Tab 23]).

2. Q2: Tie Handling vs. Strict Minority Losses in the 783 Unselected Groups

In the 2,812 majority groups (7474 runs), of 783 unselected groups (100.0%, or 27.84% of all groups), 591 (75.5%, or 21.02%) are strict minority losses (0<c(y∗)<max⁡ac(a)0 < c(y^*) < \max_a c(a)) and 192 (24.5%, or 6.83%) are top-vote ties (c(y∗)=max⁡ac(a)c(y^*) = \max_a c(a)). Exact uniform tie-breaking gives \mathbb{E}[A_{\text{uniform}}] = \mathbf{63.17\\%} (H−A=28.37H-A = 28.37 pp vs. 63.6963.69\\% and 27.8427.84 pp recorded). New runs fix and log the tie rule and PYTHONHASHSEED ([Revised PDF: Sec 3.2, 5.1, Tab 2]).

3. Q3: Seed Overlap in Table 3 and New 96-Run Paired λ\lambda-Grid

In Table 3 (288 runs), 1 seed (44) was shared across all 5 weights (λ=0/0.25/0.5/0.75/1\lambda=0/0.25/0.5/0.75/1 gains: +24,+18,+23,+20,+23+24, +18, +23, +20, +23 pp), and pairwise overlaps had 3–7 seeds: larger steps show strong paired gains (0→0.75,n=60→0.75, n=6: +15.33 pp [4.17,27.83][4.17, 27.83]; 0→1.0,n=40→1.0, n=4: +17.25 pp [4.75,28.00][4.75, 28.00]; 0.25→0.75,n=70.25→0.75, n=7: +14.00 pp [5.85,22.71][5.85, 22.71]; 0.50→1.0,n=50.50→1.0, n=5: +5.20 pp [1.00,9.40][1.00, 9.40]), while small adjacent steps (n=3n=3–55) are noisy (0→0.25,n=4:−3.250→0.25, n=4: -3.25 [−11.50,5.25][-11.50, 5.25]; 0.25→0.50,n=5:+8.200.25→0.50, n=5: +8.20 [−2.80,20.40][-2.80, 20.40]; 0.50→0.75,n=3:−0.670.50→0.75, n=3: -0.67 [−3.00,2.00][-3.00, 2.00]; 0.75→1.0,n=5:+1.400.75→1.0, n=5: +1.40 [−1.40,4.40][-1.40, 4.40]). Our 96-run (12 shared seeds × 8 weights) paired grid on S1 gives mean gains (pp): λ=0:+6.75\lambda=0: +6.75; 0.25:+6.830.25: +6.83 (Δ=+0.08\Delta=+0.08 [−6.75,6.84][-6.75, 6.84]); 0.30:+5.330.30: +5.33 (−1.42-1.42 [−10.34,5.84][-10.34, 5.84]); 1/3:+6.921/3: +6.92 (+0.17+0.17 [−6.92,6.42][-6.92, 6.42]); 0.40: +13.67 (Δ=\Delta= +6.92 [1.00,13.33][1.00, 13.33]); 0.50: +16.00 (Δ=\Delta= +9.25 [2.17,15.50][2.17, 15.50]); 0.75: +19.17 (Δ=\Delta= +12.42 [7.33,17.33][7.33, 17.33]); 1.00: +24.25 (Δ=\Delta= +17.50 [11.00,23.25][11.00, 23.25]) ([Revised PDF: Sec 5.2, Tab 3b; App F Tab 11]).

AuthorsOpenReview ↗

Response to Reviewer Yf7k (Part 2/2)

4. Q4 & W2: Independent GSM8K Splits (S1, S2) and Expanded 42-Run Cross-Model P→P+AP → P+A Validation

A. Independent Problem Splits S1 & S2 (Frozen Out-of-Sample and Leave-One-Split-Out)

On two mutually disjoint 200/100 GSM8K splits (S1: 24 majority seeds, base pass@1 49.049.0\\%, gain +7.04+7.04 pp, H−A=21.27H-A = 21.27 pp; S2: 24 majority seeds, base 43.043.0\\%, gain +7.83+7.83 pp, H−A=21.60H-A = 21.60 pp; S1 Mixture: 36 runs / 12 seeds), we evaluated (i) the Frozen Historical Predictor (fitted only on the original split) and (ii) Leave-One-Split-Out (training on the remaining splits, with training-fitted Base pass@1, to predict the held-out split):

Validation Protocol & Cohort (19 Groups)P→P+AP → P+A RMSE (pp)P+H→P+H+AP+H → P+H+A RMSE (pp)
Frozen →→ Split S1 Maj (n=24n=24)8.23 → 7.29 (−11.4%)—
Frozen →→ Split S2 Maj (n=24n=24)8.93 → 7.15 (−19.9%)—
Frozen →→ Split S1 Mix (n=36n=36)6.25 → 5.51 (−11.8%)—
Leave-One-Split-Out: Maj (n=122n=122)8.76 → 7.44 (Δ=+1.32\Delta = +1.32, 95% CI [0.68,1.95][0.68, 1.95])8.96 → 7.43 (−17.1%) (Δ=+1.54\Delta = +1.54, 95% CI [0.86,2.21][0.86, 2.21])
Leave-One-Split-Out: Mix (n=167n=167)7.10 → 6.00 (Δ=+1.10\Delta = +1.10, 95% CI [0.56,1.64][0.56, 1.64])7.63 → 6.46 (−15.3%) (Δ=+1.18\Delta = +1.18, 95% CI [0.65,1.71][0.65, 1.71])

The Frozen P→P+AP→P+A predictor reduces RMSE on all three new cohorts, and Leave-One-Split-Out yields significant pooled improvements beyond both PP and P+HP+H. Purging overlapping numeric seeds also preserves the majority P+HP+H improvement: 9.07 → 7.46 pp (Δ=+1.61\Delta=+1.61, 95% CI [0.89,2.32][0.89, 2.32]). Across 122 pooled majority runs, adding AA to all 7 supply and concentration controls improves RMSE under both OLS (7.40→6.807.40→6.80 pp, 95% CI [+0.11,+1.08][+0.11,+1.08]) and Ridge (7.37→6.817.37→6.81 pp, [+0.09,+1.03][+0.09,+1.03]), as detailed in Part 1. Full-seven-control frozen transfer improves on S2 (9.33→7.269.33→7.26 pp) but has a slightly higher RMSE on S1 (7.05→7.397.05→7.39 pp); complete variants are reported in Appendix H ([Revised PDF: Sec 5.3; App H Tabs 23–26]).

B. Repeating P→P+AP → P+A on an Expanded 42-Run Cross-Model Cohort (30 Original + 12 New Gemma-2-2B-it on Split S1)

We repeated the 19-group prefix prediction comparison against response accuracy, using model identity as a control, on the original 30-run cohort and the expanded 42-run cohort spanning four models from three families (0.6B–8B):

Cross-Model CohortBaselineNo AA RMSE (pp)+A+A RMSE (pp)ReductionΔ\DeltaRMSE (pp) & 95% Seed-Cluster CI
Original 30 runsModel+P8.967.4317.1%+1.53[−0.22,+3.44]+1.53 [-0.22,+3.44]
Expanded 42 runsModel+P7.806.4317.6%+1.37[+0.16,+2.58]+1.37 [+0.16,+2.58]

Across all 42 runs (30 seed clusters), adding selection correctness AA significantly improves held-out gain prediction beyond model identity and response accuracy. This repeats the main predictive comparison across models, extending the correlations in the original submission ([Revised PDF: Sec 5.4, Tab 7; App H Tabs 27–29]).

Thank you for the detailed response. I appreciate the additional analyses and clarifications addressing the reviewers’ comments. My assessment remains unchanged for now, but I will continue to consider your responses during the reviewer discussion phase before finalizing my recommendation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the Area Chair and Reviewers Yf7k, Lrzy, and 64th for their constructive feedback. Our core thesis is that in self-rewarded test-time reinforcement learning (TTRL), a policy's ability to generate a correct answer (H≈91.5H \approx 91.5\\%) does not guarantee selecting it as positive supervision (A≈63.7A \approx 63.7\\%)—and in the remaining H−AH - A groups, group-centered policy gradients penalize available correct responses while reinforcing wrong pluralities.

We executed 240 new training and mechanism trajectories plus full counterfactual and pooled reanalyses of our 318 historical records, highlighted in blue in the revised PDF with a 10-page main text:

  1. Supervision Selection Bottleneck (H−AH - A) Across Splits, Models, and Benchmarks (Sec 3.2, 5.1, 5.4; Tabs 2, 7):

    • Across 3 mutually disjoint 200/100 GSM8K splits (Original, S1, S2: 122122 majority runs), 4 models from 3 families, 0.6B–8B (4242 runs), and Chain-of-Thought (CoT) reasoning on competition-level MATH (math-exact-v2, 2424 majority runs), correct answers appear in 9090\\%–9797\\% of groups (HH) for in-band policies, yet 19.319.3–39.539.5 pp fail to select them (H−AH - A). In the original 74-run cohort (2,812 groups), 75.575.5\\% (591/783) of unselected groups are strict minority losses (21.0221.02\\% of all groups) and 24.524.5\\% (192/783) are top-vote ties (6.836.83\\%); exact uniform tie-breaking gives E[A]=63.17\mathbb{E}[A]=63.17\\% vs. 63.6963.69\\% recorded.
    • Within-Model Contrast (Gemma-2-2B-it): On Gemma's own split (Base pass@1 =41.0= 41.0\\%, 19g A=64.9A = 64.9\\%, H - A = 26.3 pp), Gemma improves by +20.7 pp (6/6 positive). On Llama-screened Split S1 (Base =37.0= 37.0\\%, 12 new seeds), its 19g gap is H−A=27.63H - A = 27.63 pp ([22.81,32.89][22.81, 32.89]) and adaptation shifts to −4.42 pp degradation ([−7.67,−1.17][-7.67, -1.17], 9/12 negative).
  2. Two-Threshold Mechanism (λg∗\lambda_g^* Sign Flip vs. λ=1/2\lambda = 1/2 Ordering Crossover; Sec 5.2, Tabs 3b–3c; App B.1, F):

    • Proposition 1 separates the sign threshold λg∗=sg/(1−pg+sg)≤1/2\lambda_g^* = s_g/(1-p_g+s_g) \le 1/2 (median 1/31/3), where a correct minority flips from negative to positive centered reward, from the ordering threshold λ=1/2\lambda = 1/2, above which its raw reward exceeds the wrong plurality's (2λ−1>02\lambda - 1 > 0).
    • Local 1-group (8-step) checkpoint branching on 89 wrong-vote groups (60 new + 29 initial) confirms 100100\\% advantage sign flip (89/8989/89) across λg∗\lambda_g^* and significant mitigation of negative sequence updates relative to λ=0\lambda=0 (+0.0781 [+0.0267,+0.1399][+0.0267, +0.1399] on 60g; +0.1287 [+0.0249,+0.2869][+0.0249, +0.2869] on 29g). On a 96-run (12 shared seeds × 8 weights) paired λ\lambda-grid on S1, trajectory gains stay flat at λ≤1/3\lambda \le 1/3 (+5.33+5.33 to +6.92+6.92 pp) and jump at λ=0.40\lambda = 0.40 (+13.67 pp, Δ=+6.92\Delta = +6.92 [1.00,13.33][1.00, 13.33]) and λ=0.50\lambda = 0.50 (+16.00 pp, Δ=+9.25\Delta = +9.25 [2.17,15.50][2.17, 15.50]), reaching +24.25 pp (Δ=+17.50\Delta = +17.50 [11.00,23.25][11.00, 23.25]) at λ=1\lambda = 1.
  3. Prefix Selection Diagnostics Across Splits and Models (Sec 5.3–5.4, Tabs 4–7; App H Tabs 23–29):

    • Over the 5 gold-free concentration signals (all 38 groups), adding AA cuts LOSO RMSE by 44.3% (9.84→5.489.84 → 5.48 pp, 74 Maj) and 33.1% (7.06→4.737.06 → 4.73 pp, 131 Mix). At the 19-group prefix, adding AA cuts RMSE over supply (P+HP+H) by 16.1% (8.03→6.748.03 → 6.74 pp, [+0.56,+2.03][+0.56, +2.03]) across 122 pooled majority runs, 14.0% (6.95→5.986.95 → 5.98 pp, [+0.37,+1.51][+0.37, +1.51]) across 167 mixture runs, 17.1% (8.96→7.438.96 → 7.43 pp, [+0.86,+2.21][+0.86, +2.21]) and 15.3% (7.63→6.467.63 → 6.46 pp, [+0.65,+1.71][+0.65, +1.71]) in Leave-One-Split-Out transfer, and 17.6% (7.80→6.437.80 → 6.43 pp, [+0.16,+2.58][+0.16, +2.58]) over Model+P across 42 cross-model runs.
    • Against all 7 supply + concentration controls (which constrain group-level vote outcomes without algebraically reconstructing run-level AA), 2-variable P+AP+A (6.68 pp) beats all 7 controls without AA (7.40 pp), and adding AA to all 7 controls is statistically significant in pooled majority runs under OLS (+0.59 pp [+0.11,+1.08][+0.11, +1.08]) and Ridge (+0.56 pp [+0.09,+1.03][+0.09, +1.03]).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.