acceptodds
Under review as a conference paper at ICLR 2027

Beyond Self-Consensus: Mitigating Self-Reinforcing Exploration Bias in Test-Time Reinforcement Learning

Abstract

Test-time reinforcement learning (TTRL) enables language models to improve reasoning under unlabeled test-time settings through self-generated supervision. Yet this supervision is intrinsically coupled to the model’s own exploration distribution, creating an endogenous feedback process distinct from conventional reinforcement learning with externally grounded rewards. We identify this feedback as *self-reinforcing exploration bias*, characterized by the amplification of existing policy preferences through endogenous pseudo-supervision and the progressive concentration of subsequent exploration around a restricted set of solution modes. To mitigate this bias, we introduce a two-dimensional structured exploration framework that expands reasoning-space coverage through multiple model-driven reasoning views and adapts exploration depth to instance-specific search demand. The resulting framework increases access to alternative solution trajectories while allocating test-time computation according to heterogeneous exploration requirements. Across multiple reasoning benchmarks and model scales, our approach consistently improves TTRL performance while reducing sampling cost by more than 30%. Our analysis shows that the resulting limitation is not solely a consequence of noisy pseudo-rewards or unstable optimization, but also reflects insufficient control over the solution space exposed during adaptation. These findings establish exploration as a quantitatively actionable dimension of TTRL, providing a principled route toward mitigating self-reinforcing adaptation under self-generated supervision.

Primary area:
reinforcement learning
Keywords:
test-time reinforcement learning, large language models, reasoning, entropy collapse, self-distillation, self-consistency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yCAAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies self-reinforcing exploration bias in test-time reinforcement learning (TTRL) with self-generated supervision. In standard TTRL, pseudo-labels are derived from the model’s own generations, so the supervision signal is coupled with the current policy’s exploration distribution. Consequently, initially dominant solution modes can be repeatedly selected as supervision and further amplified through policy optimization, suppressing less frequent but potentially correct solutions. To address this issue, the paper proposes a structured exploration framework with two components. First, it broadens candidate exploration by combining natural-language chain-of-thought reasoning, executable program-of-thought reasoning, and, for multiple-choice tasks, option-order permutations. The resulting candidates are confidence-filtered, aggregated into a shared answer space, and used to construct a fixed target bank before adaptation. Second, it adjusts the rollout budget for each problem according to the observed accessibility of its fixed target, allocating more samples to difficult-to-reach targets and reducing computation for mastered or persistently uninformative examples. Experiments on five mathematical reasoning benchmarks with Qwen3-1.7B-Base and Granite-4.1-3B-Base show consistent improvements over several TTRL baselines. The method also reduces the total number of sampling calls and generated tokens under the evaluated adaptation schedules.

Critical strengths
  1. The formulation of self-reinforcing exploration bias clearly captures the feedback loop between policy-generated candidates, endogenous pseudo-supervision, and subsequent policy updates. This perspective complements prior work focusing primarily on reward estimation or optimization stability.

  2. Combining complementary reasoning views expands the candidate solution space, while reachability-aware rollout allocation adjusts the amount of exploration devoted to each instance. These two components are conceptually complementary and target different aspects of restricted exploration.

  3. Separating target construction from policy adaptation prevents the evolving policy from continuously redefining its own supervision. The supervision-dynamics analysis provides useful evidence that fixed targets maintain a corrective signal for longer than online majority-vote targets.

  4. The proposed method outperforms several recent TTRL baselines on both open-ended and multiple-choice benchmarks and across two different model families. The gains are particularly substantial on more difficult benchmarks such as AIME 2025.

Critical weaknesses
  1. Freezing the multi-view targets prevents supervision from drifting toward the evolving policy, but it also removes any mechanism for correcting initially incorrect targets. Once an erroneous target is selected, the subsequent adaptation process may consistently reinforce it. The paper does not sufficiently analyze the failure cases caused by incorrect frozen targets or discuss how targets could be safely revised without reintroducing the original feedback loop. More detailed analysis of the statistics of the correctness should be presented.

  2. A substantial part of the gain may come from constructing more accurate pseudo-labels using extensive CoT/PoT sampling and program execution, rather than from mitigating self-reinforcing exploration bias itself. In particular, executable PoT provides an additional and potentially stronger computational pathway. A comparison against simpler alternatives, such as standard supervised fine-tuning on the same fixed target bank, or baselines given the same offline samples and execution budget, would help separate improved target quality from the proposed exploration mechanism.

  3. Although the mixed-group probability motivates adapting the rollout budget, it does not directly justify the specific rules used in the implementation: doubling the budget after zero target hits, halving it at a 25% hit rate, pausing after three homogeneous visits, and probing again after ten visits. The paper provides limited sensitivity analysis for these thresholds, leaving it unclear how robust the method is across models, tasks, and training configurations.

  4. The experiments are restricted to mathematical reasoning benchmarks and two relatively small base models (1.7B and 3B). It remains unclear whether self-reinforcing exploration bias manifests similarly, and whether the proposed method remains effective, for larger or instruction-tuned models and for non-mathematical reasoning tasks where executable program-based views may be unavailable. Besides, the experiments in Table 12 lacks of the baseline results.

  5. Constructing the target bank requires approximately 512–576 generations per problem, including program generation and execution. Although the paper includes this cost and reports lower total sampling under its particular adaptation schedule, the advantage may diminish when only a small number of adaptation steps are performed, when the dataset is large, or when executable views are expensive. A break-even analysis would make the efficiency claim more practically informative.

Questions for authors
  1. How do you determine the training epoch of different sets in Table 10?
  2. More results on Qwen3-4B or 8B could help.
16 comments
AuthorsOpenReview ↗

Weakness 1: Reliability of Frozen Multi-View Targets [1/2]

Many thanks for acknowledging that our method can complement the previous related work on TTRL, the reasonableness and effectiveness of our method, and the usefulness of our analysis. We provide the response to each raised concern as follows:

Weakness 1. Multi-View Target Reliability

We thank the reviewer for this constructive suggestion. Following the reviewer's recommendation, we conduct two additional analyses to characterize the correctness of frozen multi-view targets and investigate the correction of erroneous supervision during adaptation.

In Exp. 1, we systematically analyze target correctness across all 396 questions from five reasoning benchmarks, comparing frozen multi-view targets against the initial CoT-majority supervision of vanilla TTRL (Exp. 1, Table 1).

In Exp. 2, we investigate the correction dynamics of initially incorrect targets during vanilla TTRL adaptation, examining the frequency and persistence of target corrections (Exp. 2, Table 2).

Exp. 1: Correctness Analysis of Frozen Multi-View Targets

To characterize the reliability of frozen supervision, we compare the correctness of the multi-view targets with the initial CoT-majority labels used by vanilla TTRL. We categorize all 396 questions according to the correctness of their initial CoT labels and frozen multi-view targets. Ground truth is used only for this retrospective analysis, never during training.

Table 1: Correctness comparison between initial CoT-majority labels and frozen multi-view targets across five benchmarks.

Initial CoT label → Frozen multi-view target# Questions
Correct → Correct137
Incorrect → Correct70
Correct → Incorrect16
Incorrect → Incorrect173
Total396

As shown in Table 1, multi-view target construction corrects 70 initially incorrect CoT-majority labels, compared with 16 initially correct labels that become incorrect. Consequently, the overall target correctness increases from 38.6% under initial CoT-majority supervision to 52.3% under frozen multi-view supervision. In particular, on AIME25 and HMMT26, most of the questions with wrong labels (39 of 51: 17 of 22 on AIME25 and 22 of 29 on HMMT26) have no correct answer among all of the base model's CoT answers. These results characterize the limitations of single-view self-consensus and demonstrate the contribution of complementary reasoning views to improving target correctness.

AuthorsOpenReview ↗

Weakness 1: Reliability of Frozen Multi-View Targets [2/2]

Exp. 2: Correction Dynamics of Initially Incorrect Supervision

We further investigate the ability of online self-consensus to correct initially erroneous supervision during adaptation. Specifically, we track the correctness of the online pseudo-labels for questions with incorrect initial CoT-majority labels throughout vanilla TTRL training.

Table 2: Correction dynamics of initially incorrect CoT-majority labels during vanilla TTRL adaptation on AMC.

MetricResult
Questions with incorrect initial CoT-majority labels47
Proportion of training visits with correct online labels3.2% (45 of 1,410)
Incorrect-to-correct label transitions10
Correct-to-incorrect label transitions12
Questions whose online label is incorrect at the first visit44
Of these, questions with a correct label at the end of training0

As shown in Table 2, online self-consensus occasionally corrects initially incorrect labels during adaptation, but none of the 44 questions whose online label is incorrect at the first visit ends with a correct label. The online supervision remains correct for only 3.2% of training visits on these questions. These findings reveal the limited effectiveness of policy-dependent target revision and further characterize the self-reinforcing feedback between evolving policy predictions and online supervision.

These findings further highlight the limitations of policy-dependent target updates. Although vanilla TTRL reconstructs its supervision from the evolving policy at every visit, the resulting corrections are not sustained during adaptation. Since the supervision signal is generated from the same policy undergoing optimization, incorrect predictions can be repeatedly reinforced through subsequent updates. Moreover, policy disagreement alone does not reliably indicate target correctness, as correctly labeled but difficult instances may also exhibit low target reachability. Consequently, updating targets based on the evolving policy's consensus risks suppressing the low-reachability correct solutions that our framework aims to preserve.

Manuscript Revisions. We have added a new Appendix D.8 ("Reliability and failure analysis of frozen multi-view targets") to the revised paper, providing detailed statistics on target correctness across all questions from five benchmarks (Table 27) and the correction dynamics of the online labels of vanilla TTRL (Table 28). It also lists all successful and unsuccessful target construction cases on AMC, namely every question on which the frozen target and the initial CoT label differ in correctness (Table 29), to which the case studies in Appendix E now refer, and expands the discussion of independent verification for safe target revision, to which the Limitations in Section 5 also refer. Section 4.4 summarizes the correction dynamics. These additions provide a more comprehensive characterization of frozen-target reliability, supervision errors, and potential directions for target correction.

AuthorsOpenReview ↗

Weakness 2: Target Quality vs. Exploration Gains [1/4]

Weakness 2: Disentangling Target Quality and Exploration Gains

We thank the reviewer for this constructive suggestion. Following the reviewer's recommendation, we conduct two additional controlled evaluations to distinguish the contribution of stronger pseudo-label construction from the proposed exploration mechanism.

In Exp. 3, we directly compare our method with standard supervised fine-tuning (SFT) using the same frozen multi-view target bank, on both mathematical and logical reasoning. We evaluate the two methods under three settings to examine performance under shared supervision and its generalization beyond the adaptation instances (Exp. 3, Table 3). We further evaluate SFT using three learning rates under a fixed training-data budget to characterize optimization sensitivity on small adaptation sets and the corresponding generalization performance (Exp. 3, Table 4). Finally, following the reviewer's suggestion on sampling cost, we construct an additional sampling-budget-matched SFT comparison using the same number of base-model generations as the online sampling budget of our method (Exp. 3, Table 5).

In Exp. 4, we directly quantify the self-reinforcing exploration bias motivating our framework. We track initially reachable correct solutions that are underrepresented by the base model's self-consensus and measure their accessibility after vanilla TTRL, fixed-budget GRPO, and our adaptive method (Exp. 4, Table 6).

Exp. 3: Comparison under Shared Supervision across Adaptation and Generalization Settings

We compare SFT, which fine-tunes the base model on up to eight distinct target-consistent CoT samples per question, and our adaptive method under the same frozen multi-view target bank. This design holds the supervision source, the multi-view target construction (CoT and PoT on mathematical reasoning, CoT and formal reasoning on logical reasoning), and program execution fixed across the two methods.

We evaluate the methods under three settings:

  1. Transductive adaptation. The model is adapted and evaluated on the same unlabeled questions, following the standard TTRL protocol used in the main experiments. We evaluate on AMC and AIME 2025, and on ProofWriter and RuleTaker.
  2. Train-test split. We divide MATH-500 into two subject-balanced subsets of 250 questions. Each method is adapted on one subset and evaluated on the other, followed by the reverse split. On the logical tasks, the models adapted on ProofWriter and RuleTaker are evaluated on held-out sets of 600 questions built from theories that the adaptation sets do not use (ProofWriter-H and RuleTaker-H). Every evaluation question is therefore excluded from the corresponding adaptation set.
  3. Out-of-domain transfer. Models adapted on AMC are directly evaluated on MATH-500 and GPQA-Diamond, and models adapted on one logical task are evaluated on the other, without additional adaptation.
Reviewer_FYTqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies self-training / TTRL, where the model generates its supervision signal itself and provides RL updates on top on that self-generated target. The paper investigates self-reinforcing exploration bias, where model's own generations can lead to a mode collapse and hurts exploration. To address this, the paper proposes several heuristics to encourage exploration breadth and depth. The method is evaluated on five mathematical reasoning benchmarks using Qwen3-1.7B-Base and Granite-4.1-3B-Base. Experimental results show consistent improvements over several TTRL baselines.

Critical strengths
  1. The proposed heuristics (i.e., multi-view pseudo-label construction, fixed supervision targets, and adaptive rollout allocation) in the paper, according to the ablation, is complementary and jointly brings better performance. However, each individual ideas are relatively intuitive and related phenomena have been studied in prior work.

  2. Empirically, the proposed method achieves consistent performance improvements across five mathematical reasoning benchmarks and two model families, outperforming several recent TTRL baselines.

Critical weaknesses
  1. The scope of the experiment is quite narrow, where it only focuses on math benchmarks.

  2. The proposed methods are still overall very heuristical, and the generalization is quite unclear.

  3. My biggest concern is whether the improvement is only within the scope of the training presented in the paper, instead of some fundamental improvement. As studied in [1], self-trainining eventually will have collapes during the training trajectory, partially due to the self-reinforcing exploration bias (which many paper also discussed already), and I am not convinced that the proposed method can fundamentally bypass this barrier.

[1] Shafayat, Sheikh, et al. "Can large reasoning models self-train?." arXiv preprint arXiv:2505.21444 (2025).

Questions for authors

Again, how well does the proposed method generalize and if the proposed method actually brings some fundamental improvement?

8 comments
AuthorsOpenReview ↗

Weakness 1: Generalization Beyond Mathematical Reasoning

Many thanks for your thoughtful reviews, and for acknowledging the complementarity of our proposed method. We provide responses to the raised concerns as follows:

1. Generalization Beyond Mathematical Reasoning

The scope of the experiment is quite narrow, where it only focuses on math benchmarks.

We thank the reviewer for highlighting this point. To further examine the generalizability of our method, we extend our experiments beyond mathematical reasoning to two logical reasoning benchmarks, ProofWriter and RuleTaker. We introduce a complementary formal reasoning view based on structured logical deduction and compare our method against four established TTRL baselines (Exp. 1, Table 1).

Exp. 1: Generalization to Non-Mathematical Reasoning Tasks

The central contribution of our framework is multi-view exploration through complementary reasoning pathways, with executable PoT serving as one implementation for mathematical reasoning. To evaluate the generalizability of this strategy, we extend our framework to logical reasoning tasks using CoT and a model-generated formal reasoning view. This experiment evaluates the effectiveness of structured exploration on non-mathematical reasoning tasks without relying on executable programs or external solvers.

Table 1: Performance comparison on logical reasoning benchmarks (Qwen3-1.7B-Base, avg@8 accuracy, %).

MethodProofWriterRuleTaker
Base model (CoT, per sample)44.057.5
Vanilla TTRL33.3 (40.2)47.5 (54.8)
ECHO33.4 (41.2)47.5 (55.2)
SCRL33.4 (42.1)47.5 (56.1)
SCOPE33.3 (38.7)47.5 (54.3)
Ours63.3 (64.1)68.0 (68.3)

Each entry reports the final accuracy, with the best accuracy during adaptation in parentheses. The base model row reports per-sample CoT accuracy. Note that the final performance of these baselines is the same, because they collapse early in training: they reach their best accuracy at the first evaluation (step 15), and from step 45–60 (of 190) on, they answer "Unknown" to almost every question, and their accuracy no longer changes.

As shown in Table 1, our method consistently outperforms all competing TTRL baselines across the two logical reasoning benchmarks. The improvements are sustained through the final stage of adaptation, with our method achieving the highest final accuracy on ProofWriter and RuleTaker. These results demonstrate that multi-view exploration remains effective with alternative reasoning pathways, extending the applicability of our framework beyond mathematical reasoning. The consistent improvements across distinct logical reasoning tasks further support the generalizability of structured exploration under self-generated supervision.

Manuscript revisions. We have added the non-mathematical reasoning experiments and baseline comparisons to Section 4.2 of the revised paper (paragraph "Beyond mathematical reasoning", Table 2 and Figure 3), with the datasets in Appendix B (Table 10), and the full setup, the formal-view prompt, the accuracy of the targets built from each view and the training dynamics on both tasks in Appendix C.8 (Table 16 and Figure 5). We have also expanded the discussion of multi-view exploration in the introduction and in Section 3.1 to clarify that executable PoT represents a task-specific reasoning view rather than a requirement of the proposed framework.

AuthorsOpenReview ↗

Weakness 2 & Question: Methodological Design and Generalization [1/3]

Methodological Design and Generalization

Questions For Authors: Again, how well does the proposed method generalize and if the proposed method actually brings some fundamental improvement?

We thank the reviewer for raising these important questions. Our framework introduces structured exploration along two complementary dimensions, combining multi-view reasoning to expand candidate solution coverage and reachability-aware rollout allocation to sustain informative exploration during adaptation. Stable target construction further separates supervision from the evolving policy, mitigating the self-reinforcing feedback inherent in online self-consensus. The theoretical motivation in Eq. (6) and component ablations in Table 4 of the original manuscript (Table 5 of the revised paper) support these design principles.

To further address these concerns, we conduct additional experiments to provide a more comprehensive assessment of our framework from five complementary perspectives. We first examine the evolution of initially reachable correct solutions to investigate the underlying mechanism of self-reinforcing exploration bias and the contribution of our exploration strategy (Exp. 2, Table 2). We then assess the transferability of the improvements to unseen questions and out-of-domain tasks, while controlling for the quality of supervision targets (Exp. 3, Table 3). We also investigate the sensitivity of adaptive rollout allocation to different hyperparameter settings (Exp. 4, Table 4). Finally, we extend our evaluation to larger models (Exp. 5, Table 5) and non-mathematical reasoning tasks (Exp. 1, Table 1) to examine the broader applicability of the proposed framework.

In Exp. 2, we investigate the mitigation of self-reinforcing exploration bias by tracking initially reachable correct solutions under vanilla TTRL, fixed-budget GRPO, and our adaptive method (Exp. 2, Table 2).

In Exp. 3, we evaluate generalization to unseen questions and out-of-domain tasks through controlled comparisons with SFT under the same frozen multi-view supervision, on both mathematical and logical reasoning (Exp. 3, Table 3).

In Exp. 4, we examine robustness to hyperparameter choices through a sensitivity analysis of three adaptive rollout allocation hyperparameters (Exp. 4, Table 4).

In Exp. 5, we evaluate generalization across model scales by extending our experiments to Qwen3-8B-Base across five reasoning benchmarks and five TTRL baselines (Exp. 5, Table 5).

Finally, Exp. 1 in our response to Comment 1 examines generalization to different reasoning domains using ProofWriter and RuleTaker (Exp. 1, Table 1).

Exp. 2: Mitigation of Self-Reinforcing Exploration Bias

To investigate the underlying exploration mechanism, we examine the evolution of initially reachable correct solutions during adaptation. We focus on questions where the initial CoT-majority prediction is incorrect despite the correct answer being accessible to the base model. We compare vanilla TTRL, fixed-budget GRPO with the same frozen targets and 32 rollouts per question (Fixed-32), and our adaptive method to distinguish the contributions of stable supervision and reachability-aware exploration. Ground truth is used only for this retrospective analysis, never during training.

Table 2: Evolution of initially reachable correct solutions on AMC (Qwen3-1.7B-Base).

Step 180 (AMC, Qwen3-1.7B-Base)Base modelVanilla TTRLFixed-32Ours
Probability of a correct answer on these 8 questions (%)10.73.125.443.8
Questions on which the correct answer still appears in 64 samples7 of 83 of 87 of 87 of 8
Accuracy on all 83 questions (avg@64, %)25.039.840.543.0

The results reveal a clear difference in the evolution of initially reachable correct solutions. Under vanilla TTRL, self-consensus supervision progressively suppresses these solutions, even when they are accessible to the initial policy. Frozen multi-view supervision alleviates this effect, while our adaptive exploration strategy further increases their accessibility under identical targets. This distinction demonstrates that the contribution of our framework extends beyond improving pseudo-label quality.

AuthorsOpenReview ↗

Weakness 2 & Question: Methodological Design and Generalization [2/3]

Exp. 3: Generalization to Unseen Questions and Out-of-Domain Tasks

To examine the transferability of our method beyond the adaptation questions, we compare SFT and our method under identical frozen multi-view supervision, on both mathematical and logical reasoning, under two settings:

  • Train-test split setting, where MATH-500 is split into two halves of 250 questions without using the ground truth (balanced by subject). Each method is trained on one half and evaluated on the other, and vice versa, so every question is evaluated by a model that was never trained on it. On the logical tasks, the models adapted on ProofWriter and RuleTaker are evaluated on held-out sets of 600 questions built from theories that the adaptation sets do not use (ProofWriter-H and RuleTaker-H).
  • Out-of-domain setting, where the models adapted on AMC are evaluated on MATH-500 and GPQA-Diamond, on which no method was trained, and the models adapted on one logical task are evaluated on the other.

SFT fine-tunes the base model on up to eight distinct target-consistent CoT samples per question from the target bank. This controlled comparison helps distinguish the contribution of the adaptation strategy from the quality of the supervision targets.

Table 3: Generalization performance under shared frozen multi-view supervision (Qwen3-1.7B-Base, avg@8 accuracy, %).

SettingTrain → EvaluateBaseSFTOurs
Train-test splitMATH-500 half → other half54.861.264.7
Out-of-domainAMC → MATH-50053.861.1 ± 0.664.2
Out-of-domainAMC → GPQA-Diamond17.724.7 ± 0.227.1
Train-test splitProofWriter → ProofWriter-H45.052.1 ± 0.665.0
Train-test splitRuleTaker → RuleTaker-H56.460.8 ± 0.667.3
Out-of-domainProofWriter → RuleTaker57.560.6 ± 0.467.4
Out-of-domainRuleTaker → ProofWriter44.051.3 ± 0.457.1

SFT uses our pre-registered configuration (lr 3e-6), with the mean ± standard deviation over three seeds.

Under the same frozen supervision, our method consistently outperforms SFT on every unseen evaluation, by 2.4–3.5 points on mathematical reasoning and by 5.8–12.9 points on logical reasoning (e.g., 64.7 vs. 61.2 on the MATH-500 split). This suggests that the gains are not solely attributable to the quality of the frozen targets or fitting the adaptation instances. Instead, the training and exploration strategy contributes to improvements that transfer to previously unseen questions, supporting the generalizability of our approach.

Exp. 4: Robustness to Adaptive Rollout Hyperparameters

To assess the dependence of our method on specific implementation choices, we investigate the sensitivity of adaptive rollout allocation to three hyperparameters: the hit-rate threshold for halving the rollout budget, the maximum rollout budget, and the pause duration. We vary one hyperparameter at a time and compare the resulting configurations with fixed-budget GRPO under the same frozen multi-view supervision.

Table 4: Hyperparameter sensitivity on AMC (Qwen3-1.7B-Base, avg@8 accuracy, %).

ConfigurationAMC (avg@8)
Default (threshold 0.25, nmax⁡n_{\max} 256, pause 10 visits)42.7
Halving threshold 0.12541.4
Halving threshold 0.542.2
nmax⁡n_{\max} 12840.0
Pause 5 visits40.6
Frozen MV + fixed-32 GRPO (fixed sampling budget)38.6

The adaptive exploration strategy consistently outperforms fixed-budget GRPO across the tested hyperparameter configurations. Although the magnitude of improvement varies with the available rollout budget and allocation settings, the overall advantage persists. This indicates that the effectiveness of reachability-aware exploration does not depend on a narrowly tuned set of heuristic parameters. The allocation rules serve as a practical implementation of the underlying exploration strategy rather than the sole source of its effectiveness.

Reviewer_t236OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies self-reinforcing exploration bias in test-time reinforcement learning (TTRL) caused by endogenous pseudo-supervision, proposing a two-dimensional structured exploration framework that combines multi-view candidate expansion (CoT and PoT) with offline target freezing, along with reachability-aware adaptive rollout allocation. The method is validated on five mathematical reasoning benchmarks across two base model backbones, demonstrating gains in pass@1 and reductions in policy rollout costs.

Critical strengths

The paper clearly diagnoses the endogenous feedback loop and mode collapse in online self-consistency TTRL, effectively demonstrating that freezing offline multi-view consensus targets mitigates supervision drift while adaptive rollout allocation dynamically reallocates sampling effort to challenging instances.

Critical weaknesses

The proposed setup effectively operates as offline self-distillation from an ensemble of heterogeneous reasoning views rather than standard online test-time reinforcement learning, making the claimed gains primarily a function of offline teacher consensus quality rather than fundamental advancements in test-time exploration. Furthermore, relying on executable PoT limits applicability to coding and formal math domains, making the approach difficult to generalize to non-executable open-ended reasoning tasks.

Questions for authors

Can the authors clarify how this framework extends beyond domains where exact symbolic code execution is readily available, and how much of the final performance gain is strictly attributable to standard offline knowledge distillation from the frozen multi-view teacher rather than the test-time RL adaptation dynamics?

6 comments
AuthorsOpenReview ↗

Offline Distillation vs. TTRL Adaptation [1/4]

Many thanks for acknowledging our diagnosis of the endogenous feedback loop in online self-consistency TTRL, and the effectiveness of frozen multi-view targets and adaptive rollout allocation. We provide the response to each raised concern as follows:

Contributions of Offline Supervision and Test-Time RL Adaptation

how much of the final performance gain is strictly attributable to standard offline knowledge distillation from the frozen multi-view teacher rather than the test-time RL adaptation dynamics?

We thank the reviewer for raising this important question. Offline multi-view target construction is an important component of our framework, but the resulting improvements are not solely attributable to offline teacher quality. Our method separates target construction from policy adaptation, while retaining on-policy reinforcement learning and introducing reachability-aware exploration to improve the learning signal from frozen supervision.

To distinguish these contributions, we conduct additional controlled experiments from three complementary perspectives. We first compare offline SFT and our adaptive method under identical frozen multi-view targets, on both mathematical and logical reasoning, to examine the contribution of RL adaptation (Exp. 1, Table 1, 2). We then examine the contribution of additional offline teacher supervision to the performance gains (Exp. 1, Table 3). Finally, we examine the evolution of initially reachable correct solutions to directly evaluate the contribution of adaptive exploration beyond target construction (Exp. 2, Table 4).

Exp. 1: Disentangling Offline Distillation and RL Adaptation

To distinguish the contribution of the frozen multi-view teacher from that of the adaptation strategy, we compare offline SFT and our method using exactly the same frozen target bank. Offline SFT fine-tunes the base model on up to eight distinct target-consistent CoT samples per question from the target bank, while our method uses the frozen targets as rewards for on-policy adaptation with reachability-aware rollout allocation. We evaluate the methods under three settings, on mathematical reasoning and on the two logical reasoning benchmarks of Exp. 3 (ProofWriter and RuleTaker):

  1. Transductive adaptation. The model is adapted and evaluated on the same unlabeled questions, following the standard TTRL protocol used in the main experiments. We evaluate on AMC and AIME 2025, and on ProofWriter and RuleTaker.
  2. Train-test split. We divide MATH-500 into two subject-balanced subsets of 250 questions. Each method is adapted on one subset and evaluated on the other, followed by the reverse split. On the logical tasks, the models adapted on ProofWriter and RuleTaker are evaluated on held-out sets of 600 questions built from theories that the adaptation sets do not use (ProofWriter-H and RuleTaker-H). Every evaluation question is therefore excluded from the corresponding adaptation set.
  3. Out-of-domain transfer. Models adapted on AMC are directly evaluated on MATH-500 and GPQA-Diamond, and models adapted on one logical task are evaluated on the other, without additional adaptation.
AuthorsOpenReview ↗

Offline Distillation vs. TTRL Adaptation [2/4]

Table 1. Comparison under identical frozen multi-view targets (Qwen3-1.7B-Base, avg@8 accuracy, %).

SettingTrain → EvaluateBaseSFTOurs
TransductiveAMC → AMC25.235.9 ± 1.242.7
TransductiveAIME 2025 → AIME 20253.74.5 ± 0.722.5
Train-test splitMATH-500 half → other half54.861.264.7
Out-of-domainAMC → MATH-50053.861.1 ± 0.664.2
Out-of-domainAMC → GPQA-Diamond17.724.7 ± 0.227.1
TransductiveProofWriter → ProofWriter44.052.3 ± 0.263.3
TransductiveRuleTaker → RuleTaker57.561.6 ± 0.368.0
Train-test splitProofWriter → ProofWriter-H45.052.1 ± 0.665.0
Train-test splitRuleTaker → RuleTaker-H56.460.8 ± 0.667.3
Out-of-domainProofWriter → RuleTaker57.560.6 ± 0.467.4
Out-of-domainRuleTaker → ProofWriter44.051.3 ± 0.457.1

SFT uses the pre-registered learning rate 3×10−63\times10^{-6} (mean ± standard deviation over three seeds).

To further examine SFT under the same fixed training-data budget, we vary its learning rate across 10−610^{-6}, 3×10−63\times10^{-6}, and 10−510^{-5}. This experiment evaluates the sensitivity of SFT to optimization choices on the relatively small adaptation sets and measures the corresponding transfer performance beyond the adaptation questions.

We also report SFT with all three learning rates:

Table 2: SFT sensitivity to learning rate (avg@8 accuracy, %).

SFT learning rate1e-63e-61e-5 †
Transductive, AMC → AMC29.935.944.6
Transductive, AIME 2025 → AIME 20252.64.512.1
Out-of-domain, AMC → MATH-50059.361.158.2
Out-of-domain, AMC → GPQA-Diamond24.024.722.3
Transductive, ProofWriter → ProofWriter51.452.355.1
Transductive, RuleTaker → RuleTaker60.461.664.4
Train-test split, ProofWriter → ProofWriter-H50.952.152.9
Train-test split, RuleTaker → RuleTaker-H59.360.861.1
Out-of-domain, ProofWriter → RuleTaker59.060.660.6
Out-of-domain, RuleTaker → ProofWriter50.451.351.8

† Best of the three tested learning rates, selected with the ground truth (an oracle choice not available at test time).

As shown in Table 1, offline distillation benefits from the improved multi-view supervision, but its performance does not fully account for the gains achieved through RL adaptation. Under the same frozen targets, our method improves over SFT in all eleven evaluations, by 2.4–18.0 points on mathematical reasoning and by 5.8–12.9 points on logical reasoning, including the questions that neither method was adapted on (e.g., 64.7 vs. 61.2 on the MATH-500 split and 65.0 vs. 52.1 on ProofWriter-H), demonstrating the additional value of on-policy optimization. Table 2 further shows substantial sensitivity of SFT to the learning rate under the small mathematical adaptation sets. The configuration favored by transductive adaptation lets SFT exceed our method on AMC (44.6 vs. 42.7), but it does not provide the strongest transfer performance on the unseen out-of-domain datasets (58.2 on MATH-500 and 22.3 on GPQA-Diamond). On the logical tasks, even this configuration stays below our method in all six settings. These comparisons establish an important distinction between the quality of the supervision signal and the effectiveness of the adaptation strategy. Although the frozen target bank is shared, different adaptation approaches produce different outcomes. In particular, the additional gains from on-policy adaptation indicate that the framework's contribution extends beyond offline imitation of the multi-view teacher. Exp. 2 below further isolates the contribution of reachability-aware allocation under the same frozen targets.

AuthorsOpenReview ↗

Offline Distillation vs. TTRL Adaptation [3/4]

To directly control for the additional sampling budget used during adaptation, we further construct a sampling-budget-matched SFT baseline. The frozen multi-view target bank remains unchanged. We expand the base-model CoT sampling pool to match the number of online generations used by our method, corresponding to 1,152 samples per question on AMC and 3,072 samples per question on AIME 2025, and construct SFT datasets from the resulting target-consistent trajectories.

Table 3: Sampling-budget-matched SFT comparison (Qwen3-1.7B-Base, avg@8 accuracy, %).

SettingStandard SFTSampling-budget-matched SFTOurs
AIME 2025 → AIME 202512.117.822.5
AMC → MATH-50061.161.064.2
AMC → GPQA-Diamond24.725.427.1

SFT entries are the best of the tested learning rates and pool settings, selected with the ground truth.

The enlarged sampling pool improves SFT performance on AIME 2025, indicating that additional target-consistent trajectories can strengthen offline distillation. However, the performance gap remains under the matched sampling budget, with our method achieving higher accuracy on AIME 2025 and on both out-of-domain evaluations. These results further distinguish the contribution of increased offline sampling from that of the subsequent on-policy adaptation procedure.

Comments on the paper

2 comments
Area_Chair_ETc8OpenReview ↗

Author- Reviewer Discussion

Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.

AuthorsOpenReview ↗

Summary of New Experiments and Revisions

We sincerely thank the reviewers for their constructive feedback and the Area Chair for facilitating the discussion. During the rebuttal, we expanded our experiments and analyses to address concerns about generalization, the contribution of exploration beyond pseudo-label quality, long-term training stability, and computational efficiency. We summarize the main additions below.

1. Generalization beyond mathematics and to larger models (Section 4.2, Tables 1–2, and Appendix C.8).

Beyond mathematical reasoning, we extended our framework to non-mathematical logical reasoning on two benchmarks, ProofWriter and RuleTaker, with a formal reasoning view that requires no program execution. Our method outperforms four TTRL baselines by 29.9 and 20.5 percentage points, respectively, demonstrating its effectiveness in mitigating self-reinforcing exploration bias across diverse tasks. We further evaluated our framework on the larger Qwen3-8B-Base model across five mathematical benchmarks against five TTRL baselines, consistently achieving the best performance and demonstrating that our improvements extend to larger model scales.

2. Gains beyond improved pseudo-label quality (Section 4.4 and Appendices C.8, D.5 and D.7).

We further demonstrate that our performance gains arise from structured exploration rather than merely improved pseudo-label quality. Under identical frozen targets, our method consistently outperforms SFT across all 11 settings, while adaptive exploration increases the probability of generating initially underrepresented correct answers from 25.4% to 43.8% compared with fixed-budget GRPO. Moreover, even without multi-view reasoning and with comparable initial pseudo-label accuracy, our framework outperforms vanilla TTRL by 15.0 and 10.6 percentage points on ProofWriter and RuleTaker.

3. Long-horizon training stability (Section 4.3 and Appendices D.6–D.7).

We further demonstrate that our framework mitigates performance degradation during prolonged test-time adaptation, enabling more stable and sustained performance improvements. By extending training to 1,216 steps on MATH-500 and doubling the AMC training schedule, we show that vanilla TTRL progressively loses its earlier gains on held-out benchmarks or saturates at a lower level, whereas our method maintains near-peak performance. Furthermore, held-out evaluations show that the pass@16 of vanilla TTRL can degrade below that of the base model, while our framework preserves its improvements.

4. Target reliability, robustness, and computational efficiency (Appendices C.5, D.8 and D.9).

We further demonstrate the robustness and computational efficiency of our structured exploration framework. Our adaptive rollout allocation consistently outperforms fixed-budget GRPO across all tested hyperparameter configurations, demonstrating that its effectiveness is not dependent on a specific allocation strategy. Moreover, the computational savings fully offset the additional multi-view target-construction cost within the original training schedule across all five mathematical benchmarks, demonstrating the practical efficiency of our framework.

Manuscript revisions. Beyond these experiments, we revised the methodology and related-work discussions to clarify the generality of model-driven reasoning views, the distinction between frozen targets and offline distillation, and the relationship to prior work on self-training collapse. Textual revisions are highlighted in blue, with new or updated tables identified in their captions.

We appreciate the reviewers' and Area Chair's consideration of these additional results in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.