- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This article introduces TTRL-CoCoV, which assigns different operations based on the majority confidence of generated answers: it trains a shared verifier with high-confidence samples, uses the verifier to correct low-confidence pseudo labels, and applies length diversity rewards to high-confidence samples, thereby improving both Pass@1 and Pass@k for unlabeled TTRL.
Critical strengths
-
The experiments had pretty wide coverage and used some of the newer models. The main experiments used Qwen3-4B/8B, and the supplementary experiments also looked at Qwen3-0.6B/1.7B, Qwen2.5-Math-7B, and OctoThinker-8B, including tests across different adaptation datasets, architectures, ablation of components, hyperparameter sensitivity, and FLOPs-matched comparisons.
-
The key results were quite significant and also surpassed the pre-training models. On Qwen3-4B, the method improved Pass@1 by 10.42 and Pass@16 by 20.25 compared to TTRL on average; compared to the Base model, it went up from 30.96/64.41 to 42.65/71.65.
Critical weaknesses
-
Lacks innovation; this method doesn't offer any really exciting insights.
-
Length rewards aren't directly equivalent to effective reasoning diversity—they might encourage extreme lengths without necessarily producing different correct strategies.
-
Verifier learning in low-confidence areas has obvious self-referential loops, and the reliability of the mechanism still needs more direct proof. In Region B, whether a candidate enters the trusted set is determined by the verifier's VPR, and the final pseudo-labels are then chosen from these candidates. The same pseudo-labels are subsequently used to define the verifier's TP/TN/FP/FN rewards. This closed loop could reinforce the verifier's existing biases rather than actually correcting errors.
-
The boundary between hyperparameter selection and being 'completely unlabeled' isn’t clear enough. The VPR threshold, number of candidates, and length reward parameters are all compared for sensitivity using test metrics with ground truth, and the default settings tend to be the best on these test sets. The paper doesn’t clarify whether these parameters were predetermined, chosen on an independent development set, or selected based on results from MATH-500/AIME/GPQA.
Questions for authors
-
What is the central methodological insight of TTRL-CoCoV beyond combining existing components such as confidence-based routing, majority-vote pseudo-labeling, Pass@-oriented exploration, and verifier-guided selection?
-
Can the authors provide direct evidence that the length reward improves diversity in correct reasoning strategies, rather than merely encouraging unusually long or short responses?
-
How does the low-confidence verifier–pseudo-label feedback loop avoid reinforcing initial verifier biases, and were the VPR threshold, candidate count, and reward parameters selected without using ground-truth performance on the reported test sets?
6 comments
Author Rebuttal (Part II)
Re-W3&Q3: Evaluating the Impact of Region B Samples on Verifier Updates
We appreciate the reviewer's thoughtful comment. We conduct an additional ablation to examine the role of Region B samples in verifier learning and further discuss the design rationale for including these samples.
-
Ablation of Region B verifier updates. To examine the reviewer’s concern, we remove the verifier loss for low-confidence samples while retaining verifier-guided pseudo-label selection for generator training. Across six benchmarks, the variant achieves comparable but slightly lower average Pass@1 and Pass@16 than the full method. During training, we also observe higher pseudo-label accuracy in this variant. This suggests that Region B verifier updates may affect pseudo-label reliability and motivates further investigation of the feedback between verification and pseudo-label selection. |Method|AIME24|MATH500|AIME25|AMC|GPQA|DAPO-test|Avg| |---|---|---|---|---|---|---|---| |Low-freeze|20.6/44.1|80.0/92.9|18.8/43.3|52.6/82.2|36.1/85.7|43.9/77.2|42.00/70.90| |CoCoV|20.4/44.2|81.4/94.4| 17.9/45.0| 53.5/81.9|38.2/86.4|44.5/78.0|42.65/71.65|
-
Confidence weighting may help explain why the two variants remain close in downstream performance. Equation (1) weights each sample’s verifier loss by the fraction of generator rollouts supporting the majority answer. A low-confidence sample therefore receives a smaller coefficient even if the verifier accepts an incorrect candidate. High-confidence samples receive larger coefficients, and their majority labels are selected without relying on the verifier’s judgments. This weighting may mitigate the effect of incorrect Region B labels on subsequent verifier updates.
-
Our original rationale for including low-confidence samples in verifier training is to expose the verifier to plausible competing answers. When several candidates attract votes for the same problem, the verifier encounters precisely the distinctions it will later need to make during pseudo-label selection. A candidate that appears plausible but is ultimately incorrect can provide an informative negative example when the selected pseudo-label is reliable.
In future work, we will investigate stricter criteria for deciding which Region B samples can supervise the verifier and track how often an initially accepted incorrect answer persists across subsequent updates.
Re-W4&Q3: Clarifying Hyperparameter Design in the Label-Free Setting
We thank the reviewer for raising this question. The main contribution of TTRL-CoCoV is the observation that exploration value and pseudo-label reliability move in opposite directions along majority confidence, and the confidence-conditioned framework built on it. The mechanisms that carry the hyperparameters in question, namely repeated verification with a pass-rate threshold, Top-K candidate extraction, and a bounded exploration bonus, are common in this area and are not the key insight of this work. Their default values are predetermined by following the practice of existing literature and simple cost considerations, without using ground-truth performance on any reported benchmark. The sensitivity analyses in Appendix F.6 are conducted after these defaults are fixed, to examine their effect on performance
Design principles of the default hyperparameters. For the VPR threshold, we apply the majority-voting principle that Self-Consistency[1] and TTRL[2] use to aggregate sampled answers to repeated verification and set the threshold to 0.5: a candidate is accepted when more than half of its verification judgments are positive, which requires at least five positive judgments out of eight under our verification budget. For the number of candidates, we balance candidate coverage and verification cost. Top-5 in the low-confidence region increases the chance that the correct answer is among the candidates, while Top-3 suffices in the high-confidence region, where the pseudo-label is already given by majority voting and the candidates serve only to train the verifier. For the length reward, we follow the design principle of EVOL-RL[3], which combines majority-based selection with a novelty reward while preserving the priority of the selection signal. Accordingly, our length-diversity bonus is given only to trajectories that match the pseudo-label in the high-confidence region, so it never favors a trajectory that disagrees with the pseudo-label, and it is bounded by and to remain an auxiliary exploration incentive
We will state this configuration protocol explicitly in the revised manuscript.
Additional Reference
[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR 2023.
[2] TTRL: Test-time reinforcement learning, NeurIPS-25.
[3] Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation, arXiv:2509.15194, 2025.
Dear Reviewer BvAg,
We sincerely appreciate your thoughtful review and constructive feedback. Your suggestions have helped us further clarify the central methodological insight of TTRL-CoCoV and strengthen the empirical analysis of its main design choices.
In our rebuttal, we provide the following additional clarifications and evidence:
- Central methodological insight (Re-W1&Q1): We clarify how the mismatch between exploration signal strength and pseudo-label reliability motivates confidence-conditioned verification and exploration. Section 6 (Q1) now examines confidence-based routing directly, with detailed ablations in Appendix F.5.
- Reasoning diversity (Re-W2&Q2): We have further evaluated semantic diversity among correct trajectories and conducted an additional ablation of the length-diversity reward. Section 6 (Q4) summarizes the findings, and Appendix F.3 presents the full results.
- Region B verifier updates (Re-W3&Q3): We have examined the effect of excluding low-confidence samples from verifier updates and further discuss the role of confidence weighting in verifier training.
- Hyperparameter design (Re-W4&Q3): We provide further explanations of the heuristic principles and practical considerations underlying our default configuration (Appendices C.1 and F.6).
We have also revised the manuscript PDF accordingly, with all changes highlighted in blue.
Thank you again for your detailed comments and suggestions. We would be grateful if you could consider our response and the revised paper. We remain open to any further comments and suggestions.
Best regards,
The Authors
Author Rebuttal (Part I)
Thank you very much for your detailed and constructive comments and suggestions. We provide feedback on each question and concern in a point-to-point manner below.
We thank the reviewer for the opportunity to clarify our central methodological insight. Our research question is how to use the Pass@k objective to promote exploration when TTRL relies only on the model's own pseudo-labels. Pass@k optimization has been studied in supervised RLVR, where ground-truth labels determine the direction of exploration, but not in label-free TTRL. The question matters because standard TTRL reduces generation coverage (Table 1). A direct transfer, which uses the majority answer as the label for the Pass@k advantage, recovers only part of this loss (Appendix D.1). We attribute the limited effectiveness of this direct transfer to the use of the same update rule across all confidence levels, even though pseudo-labels become more reliable and the Pass@k exploration signal weakens as confidence increases (Fig. 1). Our central methodological insight is that confidence indicates what each sample needs for effective exploration under the Pass@k objective: low-confidence samples carry a strong exploration signal but need more reliable labels to guide it, high-confidence samples have reliable labels but need an additional incentive to keep exploring, and the reliable labels of high-confidence samples can train the verifier that serves low-confidence samples. Every other component of TTRL-CoCoV follows from this conditioning.
As confidence increases, majority labels become more reliable while the Pass@k exploration signal weakens. When the majority answer serves as the pseudo-label, low-confidence groups receive the strongest Pass@k signal. Yet on MATH500, only 33.72% of the base model’s majority labels in these groups are correct, so a strong update can reinforce a wrong answer. In high-confidence groups, majority-label accuracy reaches 95.25%, but the Pass@k advantage can vanish entirely (Fig. 1 and Table 7).
Confidence guides where we verify pseudo-labels and where we provide an additional exploration reward. We use confidence-based routing to separate samples into low-, medium-, and high-confidence groups so that we can address each group’s needs. For low-confidence groups, discarding the samples would sacrifice their strong exploration signal. We therefore use the more reliable majority labels from high-confidence groups to help train the verifier, then use it to select pseudo-labels for low-confidence groups. This choice is supported by Fig. 5(a), where verification yields its largest gain over majority voting in precisely this region. Medium-confidence groups use the majority label directly. For high-confidence groups, the remaining challenge is to sustain exploration. We observe that their response lengths become less diverse during training, which motivates a bounded length-diversity reward for trajectories that match the majority label.
We appreciate the reviewer's insightful observation. Following your suggestion, we further evaluate semantic diversity among trajectories with correct final answers and conduct an additional ablation of the length-diversity reward.
Direct measurement and ablation of semantic diversity. We evaluate embedding-based semantic diversity on MATH500 with Qwen3-4B-Base, sampling 32 trajectories per problem for each method. Each trajectory is encoded with Qwen3-Embedding-4B, and diversity is measured by the mean pairwise cosine distance among embeddings. We report this metric over all trajectories and separately among those with correct final answers. Compared with TTRL, which serves as our primary baseline throughout the paper, CoCoV partially recovers semantic diversity among correct trajectories. Removing the length reward leads to lower semantic diversity among correct trajectories, suggesting that the reward contributes to the observed improvement. | Method | Diversity (all) | Diversity (correct) | Avg. length | |---|---|---|---| | Base | 0.0868 | 0.0665 | 701.60 | | TTRL | 0.0233 | 0.0196 | 534.99 | | CoCoV w/o Rdiv | 0.0248 | 0.0219 | 574.73 | | CoCoV | 0.0373 | 0.0333 | 1208.24 |
Length as a practical proxy for exploration. We apply a length-diversity reward only to pseudo-label-consistent trajectories in high-confidence regions, aiming to encourage variation among likely correct responses. The additional experiments suggest that this reward promotes greater semantic variation among correct trajectories and helps preserve generation coverage. We will include the additional semantic-diversity ablation in Appendix F.3 of the revised manuscript.