- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes C-TTRL, a semi-supervised test-time reinforcement learning framework that alternates policy updates on unlabeled target problems with critic recalibration using verified responses from a separate labeled reference set. It aims to sustain improvement beyond the RLVR plateau by maintaining reward reliability as the policy evolves. Experiments across OLMo3 and Qwen3 cover mathematical and general reasoning, with held-out evaluations and ablations examining transfer and the role of critic recalibration.
Critical strengths
- The paper presents a practical approach to sustaining test-time learning through critic recalibration.
- C-TTRL achieves substantial improvements across mathematical and general reasoning tasks, with evidence of held-out transfer.
- Frozen-critic and supervised-continuation ablations provide meaningful evidence for the proposed training procedure.
Critical weaknesses
-
The comparisons are not matched by total computational budget. Appendix C reports 1.3–1.6× higher per-step cost than TTRL, while Table 6 uses different training durations across methods, leaving their relative performance at equal GPU-hour or rollout-token budgets unclear.
-
The extended-horizon evidence is uneven. Figure 8 follows C-TTRL to approximately 350 steps, whereas Figure 9 shows Self-Harmony only through 128–136 steps, limiting the evidence for C-TTRL’s longer-term advantage over this competitive baseline.
-
Discussion of recent semi-supervised methods is incomplete. GeoMin[1] also uses labeled rollouts to update reliability estimates for unlabeled data as the policy evolves, but is not discussed; given its recent release, a discussion of its geometric sample selection versus C-TTRL’s critic-based reward estimation would be sufficient without requiring an experimental comparison.
-
Sampling-coverage results need clearer reporting. For Qwen3-14B on AIME 2024, C-TTRL’s Pass@8 is 73.3% in Table 2 but 71.3% in Table 10; the paper should clarify whether these values use different checkpoints or evaluation samples.
References
[1] GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modeling.
Questions for authors
See the Critical Weaknesses above.
3 comments
Thanks for your response. My concerns have been addressed, and I continue to support accepting this paper.
Thank you for your feedback and continued support! We are glad that our additional experiments and clarifications have addressed your concerns. We appreciate your thoughtful comments and constructive suggestions, which helped us strengthen the evaluation and improve the clarity of the paper.
Response to Reviewer JXuF
Thank you for your valuable comments and for recognizing C-TTRL as practical and effective, with substantial empirical gains and thorough ablations.
W1: Comparisons are not matched by total computational budget
Thank you for this suggestion. We compare TTRL with C-TTRL at the corresponding budget. The budget is set by TTRL’s best checkpoint.
At matched GPU hours, C-TTRL slightly outperforms TTRL on both benchmarks; with additional training, C-TTRL continues to improve while TTRL plateaus or degrades (more detail in Figure 8).
W2: The extended-horizon evidence is uneven
Thank you for pointing this out. We acknowledge the difference in training horizons. We stopped the baseline runs after observing that their performance plateaued or degraded with further training. In addition, the reported comparison uses a fixed checkpoint at step 224 for C-TTRL, while selecting the best checkpoint for each baseline based on overall performance across the adaptation benchmarks (Appendix B.3).
We illustrate the evidence for stopping with Self-Harmony as an example.
1. Self-Harmony plateaus within the observed training horizon.
To show this trend, we report Self-Harmony's AIME 2024 Mean@8 from online evaluations during training.
Overall, the AIME 2024 training curves show no sustained upward trend over the observed intervals. Appendix E.3 of the revision provides additional evidence from training logs.
2. Later checkpoints within the plateau can perform worse
To examine whether a later checkpoint performs better, we re-evaluated Qwen3-8B at step 176, where its online AIME 2024 score peaks.
The later checkpoint performs similarly on the adaptation sets but worse on the held-out benchmarks. Compared with the selected checkpoint, step 176 is 1.0 point lower on the adaptation-set average and 3.8 points lower on the held-out average. Therefore, when multiple checkpoints within the plateau perform comparably well, we favor the earlier one for evaluation.
W3: Discussion of recent semi-supervised methods is incomplete
Thank you for pointing us to the missing baseline GeoMin [1], we have added a detailed discussion in the revised manuscript (highlighted in Lines 131–133).
GeoMin and C-TTRL both use a labeled reference set, but in different ways:
Role of labeled data. GeoMin uses labeled data to model geometric distributions and select reliable unlabeled problems through hard filtering. C-TTRL uses labeled data to calibrate a critic model.
Reward source. GeoMin uses majority-vote pseudo-labels as reward source, whereas C-TTRL uses critic scores.
Advantage estimation. GeoMin uses GRPO to compute response-level advantages from group-normalized rewards. C-TTRL uses a PPO-like approach, computing token-level advantages.
[1] GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modeling.
W4: Sampling-coverage results need clearer reporting
We thank the reviewer for catching this inconsistency. The two tables report values from different checkpoints:
We have updated the original Table 10 to use the same checkpoints as Table 2:
The conclusion is unchanged: C-TTRL outperforms TTRL at every sampling budget. We have updated the results in the revised manuscript (Table 12, previously Table 10).