- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes self-rewarded test-time learning and tests it on GSM8K. Specifically, they combine information on majority-predicted answers as well as oracle-correct answers in order to improve answer selection.
Critical strengths
- The experiments include a range of models across 0.6-8B.
Critical weaknesses
- The introduction/abstract fails to explain why self-rewarding is well-motivated for readers who’s less familiar with this literature (compared to just using the oracle answer).
- Only 1 test-set was used (GSM8K – a very easy math dataset on grade school problems) meaning that the generality of this approach is not determined based on the experiments.
Questions for authors
- It’s not clear to me why we want to consider the r_majority at all when we have access to r_oracle (Table 3 also supports this conclusion).
3 comments
Reviewer response
Hi thanks for the response and the various experiments conducted:
W1/Q1: I think the combination of framing the paper as an "empirical audit and diagnostic study of majority-rewarded TTRL" and the use of datasets like GSM8K/MATH (where golden labels are available) doesn't convince me of the usefulness of majority-rewarded TTRL in tasks where golden labels are available. The usefulness of the finding of the paper is weak in my opinion.
W2: I appreciate the authors adding a separate MATH dataset in their experiments but I still feel that both GSM8K and MATH datasets both released in 2021 are somewhat irrelevant for measuring math capabilities for models today. In addition, the models used (Llama3, Gemma2, Qwen3) are all not the most recent generation of the same size, which makes me wonder if the more recent models have already solved these benchmarks and therefore if findings here can translate (e.g. Qwen 3.5 / Gemma4 / Llama 3.1).
Overall, I have no opinion on the validity/insightfulness/novelty of this paper's findings (as I am not familiar with TTRL literature) but my opinion is that the findings are not really useful for improving recent models and hence would keep my original recommendation.
Clarifying the Role of Benchmark Labels and Scope of the Mechanism for Newer Models
Dear Reviewer 64th,
Thank you for engaging with our response and sharing your perspective. We would like to clarify two core points so that the role of our experiments and the scope of our findings are clear:
-
Why labeled benchmarks are used to audit an unlabeled algorithm:
- We do not advocate using majority-rewarded TTRL when ground-truth rewards are available to the learner; at test time, TTRL is designed for unlabeled inputs where gold answers are unavailable.
- Gold-labeled benchmarks provide the reference needed to determine whether self-generated pseudo-labels are correct. Gold answers are not used to compute rewards in the majority-only condition; they are used to audit reward selection and are deliberately introduced in the oracle-mixture experiments as a controlled intervention (Section 1, lines 41–46). This follows standard TTRL evaluation practice (e.g., Zuo et al., 2025, who define TTRL on unlabeled inputs and report gold-dependent diagnostic metrics in Appendix E).
- This audit reveals a concrete failure mode: across GSM8K splits and MATH CoT, of recorded training groups (Table 2b, p. 6) contain an available correct answer that loses the plurality vote and receives a negative centered reward. Moreover, in a separate synthetic streaming experiment, a simple margin gate reduces conditional wrong-update admission from () to () while retaining of proposed updates (Table 20, p. 25).
-
Scope of the mechanism and relevance to newer models:
- We evaluated four policies spanning on GSM8K and added a CoT study on screened MATH, observing a pp selection gap across the 19-group cross-model cohorts (Table 7, p. 9) and a pp gap on MATH CoT (Table 2b, p. 6).
- Once a model saturates a benchmark such as GSM8K, this failure mode should be evaluated on harder tasks where the model still makes errors and produces mixed-vote groups.
- Proposition 1 (§3.3) is conditional on the sampled vote composition: whenever an available correct answer loses the selected plurality, Proposition 1 characterizes its adverse centered supervision and the exact threshold , above which the correct response receives positive centered reward. This characterization does not depend on the model's release date; its relevance to newer models depends on whether they exhibit these mixed-vote configurations on the harder tasks where they are still uncertain.
We hope this clarifies the role of gold labels in auditing unlabeled TTRL and the relevance of our characterization: whenever an available correct answer loses the plurality vote, the same reward mechanism applies, irrespective of model generation.
Response to Reviewer 64th
Thank you for highlighting our multi-model evaluation (0.6B–8B) and constructive questions.
1. W1 & Q1: Why Consider rmajority When We Have Access to roracle?
When ground-truth labels (roracle) exist in practice, one should train with roracle directly—Table 3 and our 12-seed paired grid on S1 (λ=1 beats λ=0 by +17.50 pp, [+11.00,+23.25]) confirm roracle is far superior on GSM8K.
[Revised PDF: Page 1]) so this motivation is immediately clear.2. W2: Generality Beyond GSM8K (Competition-Level MATH + Disjoint Splits S1/S2 + Gemma-2-2B-it S1)
We ran (A) 30 trajectories on competition-level MATH (
math-exact-v2rational-value parser, 24 majority + 6 oracle seeds, CoT derivations with a 1,024-token cap, 300 GRPO steps on 200 train / 100 test items screened for 1–3 correct responses out of 4), and (B) 96 new trajectories across two mutually disjoint GSM8K splits (S1 and S2, including 12 newGemma-2-2B-itseeds on S1):math-exact-v2, Maj, 19g / 38g)math-exact-v2, Oracle λ=1)math-exact-v2), a correct answer appears in 90.79% of the first 19 groups (90.24% over 38g) but wins the vote in 67.32% (67.00% over 38g), leaving a 23.46 pp gap (12.50 pp strict minority, 10.96 pp top-tie). On Split S1,Gemma-2-2B-itshows a 27.63 pp gap and degrades by −4.42 pp ([−7.67,−1.17]).[Revised PDF: Sec 5.1, 5.3, 6]).