- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies self-reinforcing exploration bias in test-time reinforcement learning (TTRL) with self-generated supervision. In standard TTRL, pseudo-labels are derived from the model’s own generations, so the supervision signal is coupled with the current policy’s exploration distribution. Consequently, initially dominant solution modes can be repeatedly selected as supervision and further amplified through policy optimization, suppressing less frequent but potentially correct solutions. To address this issue, the paper proposes a structured exploration framework with two components. First, it broadens candidate exploration by combining natural-language chain-of-thought reasoning, executable program-of-thought reasoning, and, for multiple-choice tasks, option-order permutations. The resulting candidates are confidence-filtered, aggregated into a shared answer space, and used to construct a fixed target bank before adaptation. Second, it adjusts the rollout budget for each problem according to the observed accessibility of its fixed target, allocating more samples to difficult-to-reach targets and reducing computation for mastered or persistently uninformative examples. Experiments on five mathematical reasoning benchmarks with Qwen3-1.7B-Base and Granite-4.1-3B-Base show consistent improvements over several TTRL baselines. The method also reduces the total number of sampling calls and generated tokens under the evaluated adaptation schedules.
Critical strengths
-
The formulation of self-reinforcing exploration bias clearly captures the feedback loop between policy-generated candidates, endogenous pseudo-supervision, and subsequent policy updates. This perspective complements prior work focusing primarily on reward estimation or optimization stability.
-
Combining complementary reasoning views expands the candidate solution space, while reachability-aware rollout allocation adjusts the amount of exploration devoted to each instance. These two components are conceptually complementary and target different aspects of restricted exploration.
-
Separating target construction from policy adaptation prevents the evolving policy from continuously redefining its own supervision. The supervision-dynamics analysis provides useful evidence that fixed targets maintain a corrective signal for longer than online majority-vote targets.
-
The proposed method outperforms several recent TTRL baselines on both open-ended and multiple-choice benchmarks and across two different model families. The gains are particularly substantial on more difficult benchmarks such as AIME 2025.
Critical weaknesses
-
Freezing the multi-view targets prevents supervision from drifting toward the evolving policy, but it also removes any mechanism for correcting initially incorrect targets. Once an erroneous target is selected, the subsequent adaptation process may consistently reinforce it. The paper does not sufficiently analyze the failure cases caused by incorrect frozen targets or discuss how targets could be safely revised without reintroducing the original feedback loop. More detailed analysis of the statistics of the correctness should be presented.
-
A substantial part of the gain may come from constructing more accurate pseudo-labels using extensive CoT/PoT sampling and program execution, rather than from mitigating self-reinforcing exploration bias itself. In particular, executable PoT provides an additional and potentially stronger computational pathway. A comparison against simpler alternatives, such as standard supervised fine-tuning on the same fixed target bank, or baselines given the same offline samples and execution budget, would help separate improved target quality from the proposed exploration mechanism.
-
Although the mixed-group probability motivates adapting the rollout budget, it does not directly justify the specific rules used in the implementation: doubling the budget after zero target hits, halving it at a 25% hit rate, pausing after three homogeneous visits, and probing again after ten visits. The paper provides limited sensitivity analysis for these thresholds, leaving it unclear how robust the method is across models, tasks, and training configurations.
-
The experiments are restricted to mathematical reasoning benchmarks and two relatively small base models (1.7B and 3B). It remains unclear whether self-reinforcing exploration bias manifests similarly, and whether the proposed method remains effective, for larger or instruction-tuned models and for non-mathematical reasoning tasks where executable program-based views may be unavailable. Besides, the experiments in Table 12 lacks of the baseline results.
-
Constructing the target bank requires approximately 512–576 generations per problem, including program generation and execution. Although the paper includes this cost and reports lower total sampling under its particular adaptation schedule, the advantage may diminish when only a small number of adaptation steps are performed, when the dataset is large, or when executable views are expensive. A break-even analysis would make the efficiency claim more practically informative.
Questions for authors
- How do you determine the training epoch of different sets in Table 10?
- More results on Qwen3-4B or 8B could help.
16 comments
Weakness 1: Reliability of Frozen Multi-View Targets [2/2]
Exp. 2: Correction Dynamics of Initially Incorrect Supervision
We further investigate the ability of online self-consensus to correct initially erroneous supervision during adaptation. Specifically, we track the correctness of the online pseudo-labels for questions with incorrect initial CoT-majority labels throughout vanilla TTRL training.
Table 2: Correction dynamics of initially incorrect CoT-majority labels during vanilla TTRL adaptation on AMC.
| Metric | Result |
|---|---|
| Questions with incorrect initial CoT-majority labels | 47 |
| Proportion of training visits with correct online labels | 3.2% (45 of 1,410) |
| Incorrect-to-correct label transitions | 10 |
| Correct-to-incorrect label transitions | 12 |
| Questions whose online label is incorrect at the first visit | 44 |
| Of these, questions with a correct label at the end of training | 0 |
As shown in Table 2, online self-consensus occasionally corrects initially incorrect labels during adaptation, but none of the 44 questions whose online label is incorrect at the first visit ends with a correct label. The online supervision remains correct for only 3.2% of training visits on these questions. These findings reveal the limited effectiveness of policy-dependent target revision and further characterize the self-reinforcing feedback between evolving policy predictions and online supervision.
These findings further highlight the limitations of policy-dependent target updates. Although vanilla TTRL reconstructs its supervision from the evolving policy at every visit, the resulting corrections are not sustained during adaptation. Since the supervision signal is generated from the same policy undergoing optimization, incorrect predictions can be repeatedly reinforced through subsequent updates. Moreover, policy disagreement alone does not reliably indicate target correctness, as correctly labeled but difficult instances may also exhibit low target reachability. Consequently, updating targets based on the evolving policy's consensus risks suppressing the low-reachability correct solutions that our framework aims to preserve.
Manuscript Revisions. We have added a new Appendix D.8 ("Reliability and failure analysis of frozen multi-view targets") to the revised paper, providing detailed statistics on target correctness across all questions from five benchmarks (Table 27) and the correction dynamics of the online labels of vanilla TTRL (Table 28). It also lists all successful and unsuccessful target construction cases on AMC, namely every question on which the frozen target and the initial CoT label differ in correctness (Table 29), to which the case studies in Appendix E now refer, and expands the discussion of independent verification for safe target revision, to which the Limitations in Section 5 also refer. Section 4.4 summarizes the correction dynamics. These additions provide a more comprehensive characterization of frozen-target reliability, supervision errors, and potential directions for target correction.
Weakness 2: Target Quality vs. Exploration Gains [1/4]
Weakness 2: Disentangling Target Quality and Exploration Gains
We thank the reviewer for this constructive suggestion. Following the reviewer's recommendation, we conduct two additional controlled evaluations to distinguish the contribution of stronger pseudo-label construction from the proposed exploration mechanism.
In Exp. 3, we directly compare our method with standard supervised fine-tuning (SFT) using the same frozen multi-view target bank, on both mathematical and logical reasoning. We evaluate the two methods under three settings to examine performance under shared supervision and its generalization beyond the adaptation instances (Exp. 3, Table 3). We further evaluate SFT using three learning rates under a fixed training-data budget to characterize optimization sensitivity on small adaptation sets and the corresponding generalization performance (Exp. 3, Table 4). Finally, following the reviewer's suggestion on sampling cost, we construct an additional sampling-budget-matched SFT comparison using the same number of base-model generations as the online sampling budget of our method (Exp. 3, Table 5).
In Exp. 4, we directly quantify the self-reinforcing exploration bias motivating our framework. We track initially reachable correct solutions that are underrepresented by the base model's self-consensus and measure their accessibility after vanilla TTRL, fixed-budget GRPO, and our adaptive method (Exp. 4, Table 6).
Exp. 3: Comparison under Shared Supervision across Adaptation and Generalization Settings
We compare SFT, which fine-tunes the base model on up to eight distinct target-consistent CoT samples per question, and our adaptive method under the same frozen multi-view target bank. This design holds the supervision source, the multi-view target construction (CoT and PoT on mathematical reasoning, CoT and formal reasoning on logical reasoning), and program execution fixed across the two methods.
We evaluate the methods under three settings:
- Transductive adaptation. The model is adapted and evaluated on the same unlabeled questions, following the standard TTRL protocol used in the main experiments. We evaluate on AMC and AIME 2025, and on ProofWriter and RuleTaker.
- Train-test split. We divide MATH-500 into two subject-balanced subsets of 250 questions. Each method is adapted on one subset and evaluated on the other, followed by the reverse split. On the logical tasks, the models adapted on ProofWriter and RuleTaker are evaluated on held-out sets of 600 questions built from theories that the adaptation sets do not use (ProofWriter-H and RuleTaker-H). Every evaluation question is therefore excluded from the corresponding adaptation set.
- Out-of-domain transfer. Models adapted on AMC are directly evaluated on MATH-500 and GPQA-Diamond, and models adapted on one logical task are evaluated on the other, without additional adaptation.
Weakness 1: Reliability of Frozen Multi-View Targets [1/2]
Many thanks for acknowledging that our method can complement the previous related work on TTRL, the reasonableness and effectiveness of our method, and the usefulness of our analysis. We provide the response to each raised concern as follows:
We thank the reviewer for this constructive suggestion. Following the reviewer's recommendation, we conduct two additional analyses to characterize the correctness of frozen multi-view targets and investigate the correction of erroneous supervision during adaptation.
In Exp. 1, we systematically analyze target correctness across all 396 questions from five reasoning benchmarks, comparing frozen multi-view targets against the initial CoT-majority supervision of vanilla TTRL (Exp. 1, Table 1).
In Exp. 2, we investigate the correction dynamics of initially incorrect targets during vanilla TTRL adaptation, examining the frequency and persistence of target corrections (Exp. 2, Table 2).
Exp. 1: Correctness Analysis of Frozen Multi-View Targets
To characterize the reliability of frozen supervision, we compare the correctness of the multi-view targets with the initial CoT-majority labels used by vanilla TTRL. We categorize all 396 questions according to the correctness of their initial CoT labels and frozen multi-view targets. Ground truth is used only for this retrospective analysis, never during training.
Table 1: Correctness comparison between initial CoT-majority labels and frozen multi-view targets across five benchmarks.
As shown in Table 1, multi-view target construction corrects 70 initially incorrect CoT-majority labels, compared with 16 initially correct labels that become incorrect. Consequently, the overall target correctness increases from 38.6% under initial CoT-majority supervision to 52.3% under frozen multi-view supervision. In particular, on AIME25 and HMMT26, most of the questions with wrong labels (39 of 51: 17 of 22 on AIME25 and 22 of 29 on HMMT26) have no correct answer among all of the base model's CoT answers. These results characterize the limitations of single-view self-consensus and demonstrate the contribution of complementary reasoning views to improving target correctness.