acceptodds
Under review as a conference paper at ICLR 2027

Where to Verify, Where to Explore: Test-Time Reinforcement Learning with Confidence-Conditioned Verification

20 views

Abstract

Test-time reinforcement learning (TTRL) has emerged as a promising paradigm for enhancing the complex reasoning abilities of large language models (LLMs) in a completely label-free manner. Despite existing studies focusing on Pass@1 performance, optimizing Pass@k remains under-explored yet critical in label-free settings, which measures generation coverage for sustained exploration. Optimizing Pass@k in label-free setting is highly non-trivial, as directly applying the Pass@k advantage designs effective for RLVR yields unsatisfactory performance. We show that this transfer utilizes samples uniformly across confidence regions, although samples in different regions differ in both exploration value and label reliability. Low-confidence samples carry the strongest exploration signal, but their mostly incorrect majority labels push exploration in the wrong direction; high-confidence samples have reliable labels, but their exploration value is nearly exhausted and their trajectory diversity collapses. To overcome these hurdles, we propose TTRL-CoCoV (Test-Time Reinforcement Learning with Confidence-Conditioned Verification), a novel confidence-adaptive framework that expands Pass@k coverage and improves Pass@1 by conditioning the operation applied to each sample on its confidence. For high-confidence samples, it uses their reliable pseudo-labels to supervise the verifier and applies a length-diversity reward to prevent diversity collapse; for low-confidence samples, it delegates pseudo-label selection to the verifier to filter out incorrect pseudo-labels; and for medium-confidence samples, it bypasses verification entirely. In this way, reliable high-confidence supervision improves the verifier, which in turn corrects low-confidence pseudo-labels, allowing generation and verification to co-evolve within a single model. Extensive experiments show that TTRL-CoCoV outperforms competing label-free methods across six benchmarks, with average absolute gains of 10.42 points in Pass@1 and 20.25 points in Pass@16 over TTRL. Anonymous code repository for TTRL-CoCoV: https://anonymous.4open.science/r/TTRL-CoCoV-C5BC.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-Time Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BvAgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This article introduces TTRL-CoCoV, which assigns different operations based on the majority confidence of generated answers: it trains a shared verifier with high-confidence samples, uses the verifier to correct low-confidence pseudo labels, and applies length diversity rewards to high-confidence samples, thereby improving both Pass@1 and Pass@k for unlabeled TTRL.

Critical strengths
  1. The experiments had pretty wide coverage and used some of the newer models. The main experiments used Qwen3-4B/8B, and the supplementary experiments also looked at Qwen3-0.6B/1.7B, Qwen2.5-Math-7B, and OctoThinker-8B, including tests across different adaptation datasets, architectures, ablation of components, hyperparameter sensitivity, and FLOPs-matched comparisons.

  2. The key results were quite significant and also surpassed the pre-training models. On Qwen3-4B, the method improved Pass@1 by 10.42 and Pass@16 by 20.25 compared to TTRL on average; compared to the Base model, it went up from 30.96/64.41 to 42.65/71.65.

Critical weaknesses
  1. Lacks innovation; this method doesn't offer any really exciting insights.

  2. Length rewards aren't directly equivalent to effective reasoning diversity—they might encourage extreme lengths without necessarily producing different correct strategies.

  3. Verifier learning in low-confidence areas has obvious self-referential loops, and the reliability of the mechanism still needs more direct proof. In Region B, whether a candidate enters the trusted set is determined by the verifier's VPR, and the final pseudo-labels are then chosen from these candidates. The same pseudo-labels are subsequently used to define the verifier's TP/TN/FP/FN rewards. This closed loop could reinforce the verifier's existing biases rather than actually correcting errors.

  4. The boundary between hyperparameter selection and being 'completely unlabeled' isn’t clear enough. The VPR threshold, number of candidates, and length reward parameters are all compared for sensitivity using test metrics with ground truth, and the default settings tend to be the best on these test sets. The paper doesn’t clarify whether these parameters were predetermined, chosen on an independent development set, or selected based on results from MATH-500/AIME/GPQA.

Questions for authors
  1. What is the central methodological insight of TTRL-CoCoV beyond combining existing components such as confidence-based routing, majority-vote pseudo-labeling, Pass@kk-oriented exploration, and verifier-guided selection?

  2. Can the authors provide direct evidence that the length reward improves diversity in correct reasoning strategies, rather than merely encouraging unusually long or short responses?

  3. How does the low-confidence verifier–pseudo-label feedback loop avoid reinforcing initial verifier biases, and were the VPR threshold, candidate count, and reward parameters selected without using ground-truth performance on the reported test sets?

6 comments
AuthorsOpenReview ↗

Author Rebuttal (Part I)

Thank you very much for your detailed and constructive comments and suggestions. We provide feedback on each question and concern in a point-to-point manner below.

Re-W1&Q1: Clarifying the Central Methodological Insight

We thank the reviewer for the opportunity to clarify our central methodological insight. Our research question is how to use the Pass@k objective to promote exploration when TTRL relies only on the model's own pseudo-labels. Pass@k optimization has been studied in supervised RLVR, where ground-truth labels determine the direction of exploration, but not in label-free TTRL. The question matters because standard TTRL reduces generation coverage (Table 1). A direct transfer, which uses the majority answer as the label for the Pass@k advantage, recovers only part of this loss (Appendix D.1). We attribute the limited effectiveness of this direct transfer to the use of the same update rule across all confidence levels, even though pseudo-labels become more reliable and the Pass@k exploration signal weakens as confidence increases (Fig. 1). Our central methodological insight is that confidence indicates what each sample needs for effective exploration under the Pass@k objective: low-confidence samples carry a strong exploration signal but need more reliable labels to guide it, high-confidence samples have reliable labels but need an additional incentive to keep exploring, and the reliable labels of high-confidence samples can train the verifier that serves low-confidence samples. Every other component of TTRL-CoCoV follows from this conditioning.

  • As confidence increases, majority labels become more reliable while the Pass@k exploration signal weakens. When the majority answer serves as the pseudo-label, low-confidence groups receive the strongest Pass@k signal. Yet on MATH500, only 33.72% of the base model’s majority labels in these groups are correct, so a strong update can reinforce a wrong answer. In high-confidence groups, majority-label accuracy reaches 95.25%, but the Pass@k advantage can vanish entirely (Fig. 1 and Table 7).

  • Confidence guides where we verify pseudo-labels and where we provide an additional exploration reward. We use confidence-based routing to separate samples into low-, medium-, and high-confidence groups so that we can address each group’s needs. For low-confidence groups, discarding the samples would sacrifice their strong exploration signal. We therefore use the more reliable majority labels from high-confidence groups to help train the verifier, then use it to select pseudo-labels for low-confidence groups. This choice is supported by Fig. 5(a), where verification yields its largest gain over majority voting in precisely this region. Medium-confidence groups use the majority label directly. For high-confidence groups, the remaining challenge is to sustain exploration. We observe that their response lengths become less diverse during training, which motivates a bounded length-diversity reward for trajectories that match the majority label.

Re-W2&Q2: Additional Evidence on Reasoning Diversity

We appreciate the reviewer's insightful observation. Following your suggestion, we further evaluate semantic diversity among trajectories with correct final answers and conduct an additional ablation of the length-diversity reward.

  • Direct measurement and ablation of semantic diversity. We evaluate embedding-based semantic diversity on MATH500 with Qwen3-4B-Base, sampling 32 trajectories per problem for each method. Each trajectory is encoded with Qwen3-Embedding-4B, and diversity is measured by the mean pairwise cosine distance among embeddings. We report this metric over all trajectories and separately among those with correct final answers. Compared with TTRL, which serves as our primary baseline throughout the paper, CoCoV partially recovers semantic diversity among correct trajectories. Removing the length reward leads to lower semantic diversity among correct trajectories, suggesting that the reward contributes to the observed improvement. | Method | Diversity (all) | Diversity (correct) | Avg. length | |---|---|---|---| | Base | 0.0868 | 0.0665 | 701.60 | | TTRL | 0.0233 | 0.0196 | 534.99 | | CoCoV w/o RdivR_{\text{div}} | 0.0248 | 0.0219 | 574.73 | | CoCoV | 0.0373 | 0.0333 | 1208.24 |

  • Length as a practical proxy for exploration. We apply a length-diversity reward only to pseudo-label-consistent trajectories in high-confidence regions, aiming to encourage variation among likely correct responses. The additional experiments suggest that this reward promotes greater semantic variation among correct trajectories and helps preserve generation coverage. We will include the additional semantic-diversity ablation in Appendix F.3 of the revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal (Part II)

Re-W3&Q3: Evaluating the Impact of Region B Samples on Verifier Updates

We appreciate the reviewer's thoughtful comment. We conduct an additional ablation to examine the role of Region B samples in verifier learning and further discuss the design rationale for including these samples.

  • Ablation of Region B verifier updates. To examine the reviewer’s concern, we remove the verifier loss for low-confidence samples while retaining verifier-guided pseudo-label selection for generator training. Across six benchmarks, the variant achieves comparable but slightly lower average Pass@1 and Pass@16 than the full method. During training, we also observe higher pseudo-label accuracy in this variant. This suggests that Region B verifier updates may affect pseudo-label reliability and motivates further investigation of the feedback between verification and pseudo-label selection. |Method|AIME24|MATH500|AIME25|AMC|GPQA|DAPO-test|Avg| |---|---|---|---|---|---|---|---| |Low-freeze|20.6/44.1|80.0/92.9|18.8/43.3|52.6/82.2|36.1/85.7|43.9/77.2|42.00/70.90| |CoCoV|20.4/44.2|81.4/94.4| 17.9/45.0| 53.5/81.9|38.2/86.4|44.5/78.0|42.65/71.65|

  • Confidence weighting may help explain why the two variants remain close in downstream performance. Equation (1) weights each sample’s verifier loss by the fraction of generator rollouts supporting the majority answer. A low-confidence sample therefore receives a smaller coefficient even if the verifier accepts an incorrect candidate. High-confidence samples receive larger coefficients, and their majority labels are selected without relying on the verifier’s judgments. This weighting may mitigate the effect of incorrect Region B labels on subsequent verifier updates.

  • Our original rationale for including low-confidence samples in verifier training is to expose the verifier to plausible competing answers. When several candidates attract votes for the same problem, the verifier encounters precisely the distinctions it will later need to make during pseudo-label selection. A candidate that appears plausible but is ultimately incorrect can provide an informative negative example when the selected pseudo-label is reliable.

In future work, we will investigate stricter criteria for deciding which Region B samples can supervise the verifier and track how often an initially accepted incorrect answer persists across subsequent updates.

Re-W4&Q3: Clarifying Hyperparameter Design in the Label-Free Setting

We thank the reviewer for raising this question. The main contribution of TTRL-CoCoV is the observation that exploration value and pseudo-label reliability move in opposite directions along majority confidence, and the confidence-conditioned framework built on it. The mechanisms that carry the hyperparameters in question, namely repeated verification with a pass-rate threshold, Top-K candidate extraction, and a bounded exploration bonus, are common in this area and are not the key insight of this work. Their default values are predetermined by following the practice of existing literature and simple cost considerations, without using ground-truth performance on any reported benchmark. The sensitivity analyses in Appendix F.6 are conducted after these defaults are fixed, to examine their effect on performance

Design principles of the default hyperparameters. For the VPR threshold, we apply the majority-voting principle that Self-Consistency[1] and TTRL[2] use to aggregate sampled answers to repeated verification and set the threshold to 0.5: a candidate is accepted when more than half of its verification judgments are positive, which requires at least five positive judgments out of eight under our verification budget. For the number of candidates, we balance candidate coverage and verification cost. Top-5 in the low-confidence region increases the chance that the correct answer is among the candidates, while Top-3 suffices in the high-confidence region, where the pseudo-label is already given by majority voting and the candidates serve only to train the verifier. For the length reward, we follow the design principle of EVOL-RL[3], which combines majority-based selection with a novelty reward while preserving the priority of the selection signal. Accordingly, our length-diversity bonus is given only to trajectories that match the pseudo-label in the high-confidence region, so it never favors a trajectory that disagrees with the pseudo-label, and it is bounded by λ=0.05\lambda=0.05 and Cmax⁡=2C_{\max}=2 to remain an auxiliary exploration incentive

We will state this configuration protocol explicitly in the revised manuscript.

Additional Reference

[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR 2023.

[2] TTRL: Test-time reinforcement learning, NeurIPS-25.

[3] Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation, arXiv:2509.15194, 2025.

Dear Reviewer BvAg,

We sincerely appreciate your thoughtful review and constructive feedback. Your suggestions have helped us further clarify the central methodological insight of TTRL-CoCoV and strengthen the empirical analysis of its main design choices.

In our rebuttal, we provide the following additional clarifications and evidence:

  • Central methodological insight (Re-W1&Q1): We clarify how the mismatch between exploration signal strength and pseudo-label reliability motivates confidence-conditioned verification and exploration. Section 6 (Q1) now examines confidence-based routing directly, with detailed ablations in Appendix F.5.
  • Reasoning diversity (Re-W2&Q2): We have further evaluated semantic diversity among correct trajectories and conducted an additional ablation of the length-diversity reward. Section 6 (Q4) summarizes the findings, and Appendix F.3 presents the full results.
  • Region B verifier updates (Re-W3&Q3): We have examined the effect of excluding low-confidence samples from verifier updates and further discuss the role of confidence weighting in verifier training.
  • Hyperparameter design (Re-W4&Q3): We provide further explanations of the heuristic principles and practical considerations underlying our default configuration (Appendices C.1 and F.6).

We have also revised the manuscript PDF accordingly, with all changes highlighted in blue.

Thank you again for your detailed comments and suggestions. We would be grateful if you could consider our response and the revised paper. We remain open to any further comments and suggestions.

Best regards,

The Authors

Reviewer_qzL5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the contradiction between exploration and label reliability that reinforcement learning encounters when optimizing the multiple sampling success rate in the absence of labels. It proposes a novel framework based on confidence condition verification.

Critical strengths
  1. The author did not blindly follow the existing multiple sampling optimization methods. Instead, he deeply analyzed the reverse dynamic relationship between confidence and exploration value under the unlabeled setting, clearly pointed out the respective difficulties faced by low-confidence and high-confidence samples in the existing framework, and laid a solid theoretical foundation for the subsequent method design.

  2. Using the reliable pseudo-labels of high-confidence samples to train the validator, and then using the trained validator to filter the noise of low-confidence samples, a perfect closed-loop feedback was formed, ingeniously solving the problems of validator cold start and distribution shift in the unlabeled environment.

Critical weaknesses
  1. The current method mainly relies on result-oriented verification of the final candidate answers, while neglecting the logical verification of the complete reasoning process. When dealing with complex mathematical proofs or tasks requiring rigorous logical deduction, this mechanism may miss false trajectories in the reasoning process that have minor flaws but still yield correct answers by chance. In the future, a process-level reward model can be considered to address this deficiency.

  2. The current framework uses a globally fixed confidence threshold to divide the sample regions. However, as the model's ability improves during training, the confidence distribution of different difficulty samples will dynamically change. The fixed threshold may not maintain the optimal resource allocation efficiency in all training stages and lacks an adaptive adjustment mechanism.

  3. The reliance on the basic model's capabilities is high. As the author stated in the limitations section, this collaborative evolution mechanism highly depends on the high-confidence regions to provide high-quality supervisory signals. If the reasoning ability of the basic model is too weak, resulting in extremely sparse high-confidence samples, the training cycle of the validator will face a shortage of signals, thereby limiting the generalization potential of this method on smaller or weaker models.

Questions for authors

Lack of discussion with some works:

  1. How Far Can Unsupervised RLVR Scale LLM Training?

  2. Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning

4 comments
AuthorsOpenReview ↗

Author Rebuttal (Part I)

We sincerely thank the reviewer for your valuable comments and insightful suggestions. Below is our point-by-point response to the concerns raised.

Re-W1:Discussing Answer-Level and Process-Level Verification

We appreciate the reviewer's valuable suggestion regarding process-level verification. We further discuss its potential benefits and the motivation for answer-level verification in our framework.

  • Answer-level verification supports efficient pseudo-label selection in our framework. Our work focuses on improving Pass@k exploration under imperfect pseudo-label supervision in label-free TTRL. For a low-confidence question, the verifier checks candidate answers against the problem conditions to select a pseudo-label for the generator update. Since several trajectories can produce the same answer, we can evaluate each distinct candidate without reviewing every trajectory that produced it.

  • Process-level verification could help identify flawed reasoning and reduce the risk of reinforcing it. Two reasoning trajectories may reach the same conclusion even though one establishes a necessary condition and the other simply assumes it. Both can receive a positive reward if their final answer matches the pseudo-label. Examining the reasoning process could help identify such unsupported assumptions, while incorporating these judgments into reward assignment could reduce the risk of reinforcing flawed trajectories. Lightman et al. [1] provide related evidence, showing that process-supervised reward models outperform outcome-supervised ones in selecting correct solutions on MATH. However, integrating process-level feedback into our confidence-conditioned Pass@k framework requires further study of how to obtain reliable step-level judgments without process annotations and combine them with pseudo-label selection and Pass@k optimization in label-free TTRL. We consider this a valuable direction for future research, particularly for proof-oriented tasks.

Re-W2: Additional Analysis of Confidence Thresholds

We appreciate the reviewer's insightful observation. Following your suggestion, we provide a new analysis regarding how confidence distributions evolve during training and test an adaptive threshold design.

  • Confidence distribution shifts under fixed thresholds. We track the confidence-region proportions on 500 questions sampled from the DAPO-14k-MATH training set at checkpoints during training. The thresholds are 0.4 and 0.6. As shown below, the confidence distribution shifts substantially during training, changing the proportions of samples assigned to different regions. | Model | Mean confidence | Low | Middle | High | |---|---|---|---|---| | Qwen3-4B-Base | 17.8% | 94.0% | 6.0% | 0.0% | | Step 15 | 39.7% | 55.2% | 23.2% | 21.6% | | Step 30 | 48.3% | 44.6% | 22.0% | 33.4% | | Step 45 | 52.4% | 37.4% | 26.0% | 36.6% | | Step 60 | 52.4% | 38.8% | 23.4% | 37.8% |

  • Evaluation of adaptive confidence thresholds. We implement dynamic thresholds with fixed target region proportions. The first 20 steps use the fixed thresholds of 0.4 and 0.6. At step 20, the target region proportions are determined from the preceding five steps, and the thresholds are subsequently updated every five steps using recent confidence quantiles. As shown below, adaptive thresholds improve AIME25 Pass@1 and GPQA Pass@1, while achieving nearly identical average Pass@1 (42.43 vs. 42.65) and somewhat lower average Pass@16 (69.97 vs. 71.65). These results suggest that adaptive confidence routing is worth further investigation. | Method | AIME24 | MATH500 | AIME25 | AMC | GPQA | DAPO-test | Avg | |---|---|---|---|---|---|---|---| | Dynamic thresholds | 19.7/43.0 | 80.9/93.4 | 19.1/42.5 | 53.0/80.7 | 39.0/85.2 | 42.9/75.0 | 42.43/69.97 | | CoCoV (fixed) | 20.4/44.2 | 81.4/94.4 | 17.9/45.0 | 53.5/81.9 | 38.2/86.4 | 44.5/78.0 | 42.65/71.65 |

  • Design rationale for fixed thresholds. We heuristically set the thresholds (0.4, 0.6) around the 0.5 majority-consensus reference point to reduce sensitivity to sampling fluctuations and distinguish regions with different levels of pseudo-label reliability. This provides a consistent criterion for confidence-based routing throughout training.

We appreciate the reviewer's constructive suggestion and will further discuss the potential of adaptive confidence routing in the revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal (Part II)

Re-W3: Additional Evaluation Across Model Sizes

We thank the reviewer for highlighting this important limitation. We conduct additional analyses of confidence distributions and adaptation performance across Qwen3 model scales.

  • Confidence distributions across base models. We analyze the confidence distributions of Qwen3-1.7B-Base and Qwen3-4B-Base before adaptation, using 500 questions sampled from DAPO-14k-MATH with 32 rollouts per question and confidence thresholds (0.4, 0.6). As shown below, neither model has high-confidence samples in this diagnostic set, with most samples concentrated in the low-confidence region. This supports the reviewer's concern that high-confidence supervision can be scarce. | Model | Avg. Confidence | Low | Medium | High | |---|---|---|---|---| | Qwen3-1.7B-Base | 11.5% | 494 | 6 | 0 | | Qwen3-4B-Base | 17.8% | 470 | 30 | 0 |

  • Adaptation performance across model scales. We further compare CoCoV with our primary baseline TTRL across two model scales. The table reports average Pass@1 / Pass@16 over five benchmarks (AIME24, MATH500, AIME25, AMC, and GPQA). | Method | Qwen3-1.7B | Qwen3-4B | |---|---|---| | Base | 24.06/55.32 | 31.37/63.49 | | TTRL | 26.62/54.14 | 32.56/48.62 | | CoCoV | 26.58/53.46 | 42.28/70.38 |

The results reveal two findings: (1) Limited gains on weaker models support the reviewer's concern. On 1.7B, CoCoV achieves comparable performance to TTRL. This result is consistent with the concern that limited base-model capabilities may constrain the effectiveness of generator-verifier co-evolution. (2) The impact of high-confidence supervision scarcity varies with model capability. Although all two base models have no high-confidence samples in the initial diagnostic set, CoCoV achieves substantially larger improvements on 4B than on 1.7B. These results suggest that initial high-confidence scarcity alone does not determine adaptation performance, and its impact may depend on the underlying model's capabilities.

Re-Q1: Further Discussion of Related Work

We thank the reviewer for pointing out these two closely related works. Both provide perspectives that help position TTRL-CoCoV, and we will add them to the Related Work section of the revised manuscript.

  • How Far Can Unsupervised RLVR Scale LLM Training? He et al.[2] provide a theoretical and empirical analysis of unsupervised RLVR, showing that intrinsic reward mechanisms tend to sharpen a model's initial output distribution. Their findings highlight the dependence of self-generated supervision on the alignment between model confidence and correctness. This complements our analysis of unreliable majority labels and the dependence on base-model capability. Their discussion of generation–verification asymmetry also provides broader context for our use of verification in label-free training.

  • Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning. Wu et al.[3] demonstrate that correct solutions can form a logical minority and propose SR-TTRL to improve pseudo-label selection through trajectory summarization and self-reflection. This shares our motivation of recovering reliable supervision beyond majority voting. Their use of summarized reasoning trajectories also suggests a useful direction for improving the reliability and efficiency of trajectory-aware verification, as discussed in W1.

  • Connection to TTRL-CoCoV. He et al.[2] investigate how intrinsic rewards shape the model's output distribution in unsupervised RLVR, highlighting the role of initial confidence and correctness. Wu et al.[3] study pseudo-label selection when correct solutions form a minority, using trajectory summarization and self-reflection. Our work focuses on the interplay between pseudo-label reliability and Pass@k exploration across confidence regions, and develops a confidence-conditioned framework to coordinate verification and exploration accordingly. Together, these studies provide complementary perspectives on self-generated supervision in label-free reinforcement learning.

Additional Reference

[1] Let's Verify Step by Step, ICLR 2024.

[2] How Far Can Unsupervised RLVR Scale LLM Training?, ICLR 2026.

[3] Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning, ICML 2026.

Dear Reviewer qzL5,

We sincerely thank you for your careful review and valuable suggestions. Your feedback has been helpful in clarifying the scope of our verification design, examining confidence dynamics, and positioning our work within recent research on label-free reinforcement learning.

In our rebuttal, we provide the following additional analyses and discussions:

  • Verification design (Re-W1): We clarify the motivation for answer-level verification and discuss the potential benefits of integrating process-level feedback into confidence-conditioned Pass@k optimization.
  • Confidence thresholds (Re-W2): We have analyzed how confidence distributions evolve during training and conducted additional experiments with adaptive thresholds. Appendix G (Limitations and Future Work) now includes an expanded discussion of adaptive confidence routing.
  • Model capabilities (Re-W3): We have examined the initial confidence distributions and adaptation performance of Qwen3-1.7B-Base and Qwen3-4B-Base to better understand the role of base-model capability.
  • Related work (Re-Q1): We discuss the two suggested papers and clarify their connections to TTRL-CoCoV, and incorporate both into the Introduction and Appendix A (Related Work).

We have also revised the manuscript PDF accordingly, with all changes highlighted in blue.

Thank you again for your insightful comments and suggestions. We would be grateful if you could consider our response and the revised paper. We remain open to any further comments and suggestions.

Best regards,

The Authors

Dear Reviewer qzL5,

Thank you again for your insightful suggestions. We would like to briefly follow up on our response.

We especially appreciate your pointing us to "How Far Can Unsupervised RLVR Scale LLM Training?"[1] and "Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning"[2]. Both papers are now cited and discussed in the revised manuscript. He et al.'s analysis of how intrinsic rewards sharpen the initial output distribution, and SR-TTRL's observation that correct solutions can form a minority, provide valuable context for our analysis of when majority-vote pseudo-labels become unreliable.

We are also grateful for your suggestions on process-level verification and adaptive confidence thresholds, both of which we find to be promising directions. Process-level feedback could identify flawed reasoning that reaches a correct answer by chance, and our preliminary experiments with adaptive thresholds suggest that adjusting the confidence partition during training deserves further study. We have expanded the discussion of adaptive confidence routing in Appendix G accordingly.

We would appreciate it if you could take our response and the revised paper into account, and we hope these improvements may merit a reconsideration of your final assessment. We sincerely thank you again for your time and constructive feedback.

Best Regards,

The Authors

Additional Reference

[1] How Far Can Unsupervised RLVR Scale LLM Training?, ICLR 2026.

[2] Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning, ICML 2026.

Reviewer_aDkxOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TTRL-CoCoV for reinforcement learning with unlabeled reasoning problems. A single model performs both generation and verification, and training depends on the consensus among sampled answers. High confidence examples provide supervision for the verifier and receive a diversity reward. Low confidence examples undergo verification before being used for policy updates. The method combines this procedure with Pass@k optimization and reports improvements in both Pass@1 and Pass@16 over existing baselines.

Presentation 3 Informativeness 1 Soundness 2

Critical strengths

The paper addresses an important limitation of label-free reinforcement learning: gains in Pass@1 can come at the expense of Pass@k. The joint consideration of pseudo-label reliability and exploration provides a clear motivation for the proposed framework and is relevant to sustaining improvement under self-generated supervision.

Critical weaknesses

1. Hyperparameter sensitivity and an unclear model selection protocol

The method introduces several hyperparameters, including confidence thresholds, candidate counts, verification budgets, and diversity reward parameters. While Appendix F.6 provides sensitivity analyses, some relatively small changes lead to substantial performance drops. For instance, increasing the verification acceptance threshold from 0.5 to 0.625 reduces AIME24 Pass@1 from 20.4 to 13.0. The paper does not clearly specify the validation protocol or tuning budget used to select these parameters. In particular, it is unclear how model selection is performed under the stated label-free setting. The reported analyses therefore do not sufficiently establish robustness or ease of transfer to new settings.

2. Limited task coverage and insufficient evidence of general applicability

The evaluation is dominated by mathematics, with five mathematical benchmarks and only one non-mathematical benchmark, GPQA. Since GPQA consists of questions with four answer options, Pass@16 can be strongly affected by option coverage and is less informative about reasoning quality or the ability to select a correct answer. Furthermore, these results evaluate transfer after mathematical training, rather than adaptation on a non-mathematical task.

3. Insufficient ablations of the central routing mechanism

The proposed framework combines Pass@k optimization, verification, sample filtering, and a diversity reward. The component ablations demonstrate their utility within the full framework, but do not isolate the contribution of confidence based routing itself. A more direct comparison would retain the same objective and reward formulation while replacing the routing strategy with uniform verification or random allocation under a matched verification budget. This is particularly important given the existing CoVerRL baseline, which already jointly trains generation and verification. Without such controls, the empirical contribution of the proposed routing mechanism remains unclear.

4. The method is largely incremental.

The main components have precedents in prior work, including majority-vote supervision, joint generator and verifier training, Pass@k optimization, and exploration rewards. The principal contribution appears to be a heuristic combination of these components through confidence thresholds. Although the empirical gains are promising, the paper does not convincingly establish a substantial advance over these existing approaches.

Questions for authors

Please refer to weakness.

4 comments
AuthorsOpenReview ↗

Author Rebuttal (Part I)

Thank you for recognizing that our work addresses an important limitation of label-free reinforcement learning and for your overall positive evaluation. We appreciate the time and effort you devoted to the review, particularly your suggestions for evaluating adaptation beyond mathematics and testing the contribution of confidence-based routing. Below, we respond to your comments point by point and present the additional experiments motivated by your suggestions.

Re-W1:Regarding hyperparameter sensitivity and model selection protocol

We thank the reviewer for raising these concerns. We clarify our hyperparameter configuration protocol and provide additional evidence regarding sensitivity and transferability.

  • Hyperparameter configuration in the label-free setting. Consistent with the label-free adaptation scenarios studied in TTRL [1], Co-rewarding [2] and CoVerRL [3], we do not use a labeled validation set for hyperparameter tuning. The hyperparameters in TTRL-CoCoV fall into two groups: those associated with our confidence-conditioned design, and operational settings based on established principles in reasoning and label-free RL. Our central methodological contribution lies in identifying the mismatch between exploration value and pseudo-label reliability and using confidence to coordinate verification and exploration across sample regions. The default values are chosen heuristically based on these design principles and practical cost considerations, rather than tuned against labeled benchmark performance.

    • (1) Confidence-conditioned design parameters. We set the confidence thresholds (0.4, 0.6) around the 0.5 majority-consensus reference point to distinguish regions with different levels of pseudo-label reliability and exploration value. Below 0.4, weaker consensus makes verification particularly valuable; above 0.6, stronger consensus provides more reliable supervision for the verifier; the intermediate region retains majority-vote supervision. The same confidence-conditioned design determines where the length-diversity reward is applied: only to pseudo-label-consistent trajectories in the high-confidence region.

    • (2) Established mechanisms and practical parameter choices. Following the majority-voting principle used in Self-Consistency[4] and TTRL[1], we set the VPR threshold to 0.5, accepting a candidate when more than half of its verification judgments are positive. Under our budget of eight verification trajectories per candidate, chosen to balance judgment stability and computational cost, this requires at least five positive judgments. For candidate selection, Top-5 in the low-confidence region increases the chance of including a correct answer, while Top-3 reduces verification cost in the high-confidence region, where the pseudo-label is already determined by majority voting. For the length-diversity reward, we draw on the design principle of EVOL-RL[5], which combines majority-based supervision with an auxiliary novelty reward, and heuristically set λ=0.05\lambda=0.05 and Cmax⁡=2C_{\max}=2 to keep the reward auxiliary to the exploration objective.

  • Hyperparameter robustness and transferability. We conduct sensitivity analyses on confidence thresholds, candidate sizes, verification budgets, VPR thresholds, and length-reward parameters. Several configurations achieve comparable performance. For example, confidence thresholds (0.3, 0.6) yield an average Pass@1/16 of 42.72/71.04, compared with 42.28/70.38 under the default (0.4, 0.6). The VPR threshold is more sensitive: increasing it from 0.5 to 0.625 reduces AIME24 Pass@1 from 20.4 to 13.0, because stricter acceptance then requires six rather than five positive judgments out of eight, so fewer low-confidence candidates are accepted and more of these samples are skipped. These results suggest robustness across several practical configurations, with greater sensitivity to verification acceptance. We further evaluate CoCoV on ACCReason (Table 6) and conduct direct label-free adaptation on MMLU-Pro STEM (W2), using the same default hyperparameters without retuning. For each setting, we report the average improvement over our primary baseline TTRL across five benchmarks: the five benchmarks in Table 6 for ACCReason, and AIME24, MATH500, AIME25, AMC, and MMLU-Pro STEM for the latter. | Adaptation Corpus | ΔPass@1 | ΔPass@16 | |---|---|---| | ACCReason | +5.31 | +11.42 | | MMLU-Pro STEM | +4.54 | +6.24 |

Both settings show improvements over TTRL in Pass@1 and Pass@16, providing further evidence that the default configuration transfers across adaptation settings without task-specific retuning. Additional results are provided in Appendix E.2 and F.6.

We will clarify the hyperparameter configuration protocol in the revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal (Part II)

Re-W2: Regarding task coverage and further evidence of general applicability

We thank the reviewer for this valuable comment. Following your suggestion, we conduct three additional evaluations on Qwen3-4B-Base using the exact same default hyperparameters as our mathematical experiments without any retuning:

  • GPQA evaluation across different sampling budgets. To better assess performance beyond Pass@16, we report the complete Pass@k progression (k∈{1,2,4,8,16}k \in \{1, 2, 4, 8, 16\}) on GPQA-Diamond. As shown below, CoCoV improves Pass@1 over both the base model and TTRL and achieves higher Pass@k than TTRL across all evaluated sampling budgets. These results provide additional evidence of improved single-sample accuracy and better preservation of generation coverage. | Method | Pass@1 | Pass@2 | Pass@4 | Pass@8 | Pass@16 | |---|---|---|---|---|---| | Base | 34.28 | 49.75 | 65.30 | 78.58 | 89.27 | | TTRL | 35.31 | 47.24 | 58.64 | 68.57 | 76.39 | | CoCoV | 38.37 | 52.21 | 65.75 | 77.87 | 87.50 |

  • Additional evaluation on MMLU-Pro STEM. We evaluate models trained on DAPO-14k-MATH using MMLU-Pro STEM, which contains ten-option questions. We sample 200 questions from each of five non-mathematical STEM subjects (Physics, Chemistry, Biology, Computer Science and Engineering; 1,000 questions in total, seed 42). CoCoV achieves the highest Pass@16 among the evaluated label-free adaptation methods, suggesting better preservation of generation coverage. | Method | Pass@1 | Pass@16 | |---|---|---| | Base | 48.82 | 89.70 | | GRPO (ground-truth labels) | 64.23 | 91.45 | | TTRL | 58.32 | 87.28 | | SCRL | 56.47 | 87.20 | | Co-Rewarding-III | 60.10 | 84.38 | | CoCoV | 58.39 | 89.10 |

  • Direct label-free adaptation on MMLU-Pro STEM. We directly train Qwen3-4B-Base on unlabeled MMLU-Pro STEM questions, keeping all hyperparameters identical to our default setup. CoCoV outperforms TTRL on MMLU-Pro STEM and all four mathematical benchmarks in both Pass@1 and Pass@16. This provides evidence that the framework can support adaptation on non-mathematical tasks without task-specific hyperparameter retuning. | Method | AIME24 | MATH500 | AIME25 | AMC | Math Avg | MMLU-Pro | |---|---|---|---|---|---|---| | Base | 8.0/26.7 | 67.2/91.7 | 6.8/32.5 | 40.1/76.5 | 30.54/56.87 | 48.8/89.7 | | TTRL | 11.6/31.6 | 75.9/89.7 | 8.1/30.0 | 45.4/78.9 | 35.25/57.55 | 61.3/83.4 | | CoCoV | 18.3/43.1 | 79.3/94.2 | 13.8/35.7 | 50.4/82.2 | 40.45/63.80 | 63.2/89.6 |

We will include the additional MMLU-Pro STEM adaptation results in the revised manuscript.

Re-W3: Additional Evidence for Confidence-Based Routing

We appreciate the reviewer's helpful suggestion. We conduct additional routing ablations on Qwen3-4B-Base to examine the contribution of confidence-based routing.

  • Routing ablation settings. We compare two alternative strategies: (1) Uniform verification, where all samples are routed to verification, pseudo-labels are selected by the verifier, and samples with an accepted candidate are used for verifier training; and (2) Random routing, where samples are randomly assigned to the three regions under a matched verification budget, with the corresponding optimization operations applied in each region. These controls allow us to examine whether confidence-based assignment improves the allocation of verification and exploration operations.

  • Routing ablation results. As shown below, CoCoV achieves higher average Pass@1 and Pass@16 than both alternatives across six benchmarks. These results suggest that confidence-conditioned routing contributes to performance by assigning verification and exploration operations according to sample confidence. The additional controls complement our existing comparison with CoVerRL, which also jointly trains generation and verification.

MethodAIME24MATH500AIME25AMCGPQADAPO-testAvg
Random routing16.3/36.677.3/91.512.9/40.847.4/74.137.3/87.037.4/71.838.10/66.97
Uniform verification12.6/25.877.4/91.210.6/30.047.8/78.636.9/86.035.8/68.636.85/63.37
CoCoV20.4/44.281.4/94.417.9/45.053.5/81.938.2/86.444.5/78.042.65/71.65

We will include these evaluations in the revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal (Part III)

Re-W4:Clarifying the Central Methodological Contribution

We thank the reviewer for asking us to clarify our methodological contribution. Prior work on TTRL has explored ways to improve pseudo-label reliability and sustain exploration through reward shaping. Pass@k training in RLVR offers another way to encourage exploration, but it relies on ground-truth labels to identify correct responses. This raises the question we study: How can we use the Pass@k objective to promote exploration in TTRL when training relies solely on the model’s own pseudo-labels? To answer this question, we first examine what happens when majority-vote pseudo-labels replace ground-truth labels in Pass@k training. Our analysis identifies a mismatch specific to this setting and motivates the roles of the components in TTRL-CoCoV.

  • Directly transferring Pass@k training reveals a mismatch between exploration signal strength and pseudo-label reliability. We first replace ground-truth labels with majority-vote pseudo-labels when computing the Pass@k advantage. On AIME25 with Qwen3-4B-Base, this raises Pass@16 from 14.40 under standard TTRL to 24.60, still below the pre-adaptation score of 32.50 (Table 4). Section 3 reveals a mismatch of this observation: Low-confidence groups receive the strongest exploration signal, but their majority labels are often wrong, so a strong update can reinforce an incorrect answer; High-confidence groups provide more reliable labels, but their exploration signal is weak.

  • This mismatch determines how each component contributes to exploration. We define confidence as the fraction of sampled responses that produce the most frequent answer. Low-confidence groups need more reliable labels to guide their strong exploration signal, while high-confidence groups need an additional incentive to explore. With Pass@k as the training objective, we use confidence to determine how each group contributes to learning. The relatively reliable majority labels from high-confidence groups help train the verifier, which in turn helps select pseudo-labels for low-confidence groups. These selected labels guide both generator and verifier updates, so low-confidence groups can contribute their strong exploration signal even when their own majority labels are unreliable. Medium-confidence groups use the majority label directly. Within high-confidence groups themselves, the Pass@k advantage is weak and response lengths become less diverse during training. We therefore apply a bounded length-diversity reward to trajectories that match the majority label to sustain exploration in these groups.

Additional Reference

[1] TTRL: Test-time reinforcement learning, NeurIPS-25.

[2] Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models, ICLR 2026.

[3] CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution, ACL 2026.

[4] Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR 2023.

[5] Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation, arXiv:2509.15194, 2025.

Dear Reviewer aDkx,

We sincerely thank you for your careful review and constructive suggestions. Your feedback has been particularly helpful in strengthening the experimental evidence for our confidence-conditioned design and clarifying its methodological contribution.

In our rebuttal, we provide the following additional analyses and clarifications:

  • Hyperparameter configuration and transferability (Re-W1): We clarify the heuristic principles underlying our default hyperparameters (Appendices C.1 and F.6) and provide further evidence of transferability without retuning. For clarity, ACCReason in Re-W1 refers to AceReason-Math (Table 6).
  • Non-mathematical evaluation (Re-W2): We have conducted additional GPQA Pass@k evaluations, MMLU-Pro STEM transfer evaluations, and adaptation experiments on MMLU-Pro STEM. The adaptation results are included in Appendix E.2.
  • Confidence-conditioned routing (Re-W3): We have introduced random routing and uniform verification controls to directly examine the contribution of confidence-based sample allocation, with a summary in Section 6 (Q1) and detailed results in Appendix F.5.
  • Methodological contribution (Re-W4): We further clarify how the mismatch between pseudo-label reliability and Pass@k exploration signals motivates our confidence-conditioned optimization framework.

We have also revised the manuscript PDF accordingly, with all changes highlighted in blue.

Thank you again for your constructive comments and suggestions, which have helped us improve the paper. We would be grateful if you could consider our response and the revised paper. We remain open to any further comments and suggestions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.