acceptodds
Under review as a conference paper at ICLR 2027

MS-TTRL: Breaking the Frequency Conflict in Multiple-Solution Test-Time Reinforcement Learning

Abstract

Scientific, engineering, and mathematical reasoning tasks can admit multiple valid solutions, while exhaustive annotation of the full solution set is often prohibitively costly. Test-time reinforcement learning (TTRL) applies reinforcement learning to unlabeled test instances, offering a label-free approach to multiple-solution reasoning. In this setting, generation frequency serves two opposing objectives: correctness estimation favors frequent outcomes, whereas outcome-level diversity preservation favors less frequent outcomes. We term this tension the Frequency Conflict. To resolve this conflict, we propose MS-TTRL, which obtains frequency-independent correctness estimates through Outcome-Level Self-Verification while reserving generation frequency solely for diversity preservation. Three-State Outcome Cache and the Candidate State support verification reuse across repeated observations and confidence-aware handling of ambiguous outcomes, while Confidence-Gated Inverse Probability Scaling selectively strengthens the learning signals of rare, reliably verified outcomes. Experiments across two benchmarks, four multiple-solution tasks, and three backbone models, each evaluated with five random seeds, show that MS-TTRL improves Recovery Rate by 19.13 percentage points on average over the best-performing label-free baseline and approaches the performance of the supervised IPS-GRPO.

Primary area:
reinforcement learning
Keywords:
Test-Time Reinforcement Learning, Multiple-Solution Problem, Outcome-Level Diversity, Self-Verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_degWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies test-time reinforcement learning for multiple-solution reasoning problems, where each problem may admit several distinct valid outcomes but complete solution annotations are unavailable during training. The paper identifies a “Frequency Conflict” in existing label-free TTRL methods, generation frequency is used both to estimate correctness, which favors frequent outcomes, and to preserve outcome-level diversity, which favors infrequent outcomes. As a result, frequency alone cannot distinguish rare valid solutions from rare erroneous ones. To address this issue, the paper proposes MS-TTRL, which separates correctness estimation from diversity preservation. It uses outcome-level self-verification to estimate correctness, while generation frequency is reserved for diversity-aware reward scaling. The method further introduces a Three-State Outcome Cache that categorizes outcomes as correct, candidate, or wrong; ambiguous candidate outcomes receive soft rewards and are re-verified when revisited. Finally, Confidence-Gated Inverse Probability Scaling amplifies rare outcomes only after they have been verified with sufficiently high confidence. Experiments cover three backbone models and five random seeds. MS-TTRL consistently improves Recovery Rate and Outcome Recovery Rate over TTRL and TTRL-CoCoV, with an average Recovery Rate improvement of 19.13 percentage points over the strongest label-free baseline.

Critical strengths
  1. The distinction between trajectory-level diversity and outcome-level diversity is important in multiple-solution reasoning. Existing TTRL methods based on majority-vote pseudo-labels are naturally biased toward dominant outcomes and can suppress rare but valid alternatives. The paper clearly explains why methods designed for single-answer reasoning do not directly transfer to multiple-solution settings.

  2. The central observation—that generation frequency induces opposing preferences when used for both correctness estimation and diversity preservation—is intuitive and well motivated. The formulation helps explain why simply rewarding rare outcomes is unsafe and why extending majority voting to the top-𝑘 outcomes does not resolve the underlying ambiguity between rare valid and rare invalid outcomes.

  3. The method improves both Recovery Rate and Outcome Recovery Rate across all evaluated HypoSpace model–task combinations and on MACE-Math. The gains are observed across model scales, model families, and five random seeds. Some improvements are substantial, particularly on 3D Structure Reconstruction, where majority-based TTRL exhibits severe outcome-level collapse despite high trajectory accuracy.

Critical weaknesses
  1. MS-TTRL uses the same model as both the policy and the verifier. Consequently, generation and verification errors may be strongly correlated. A model that generates an outcome due to a systematic misunderstanding may also accept that outcome during verification. Repeating verification several times primarily reduces sampling variance, but does not necessarily mitigate systematic errors shared by the generator and verifier. Therefore, the verification pass rate should not automatically be interpreted as a reliable correctness probability.

  2. The policy model is continuously updated, and the same evolving model is also used as the verifier. However, once an outcome enters the Correct or Wrong set, its state is permanently reused and never reconsidered. An early false acceptance can therefore continue receiving positive and inverse-frequency-scaled rewards, while an early false rejection can permanently suppress a valid outcome. This creates a mismatch between the evolving verifier and the persistent cache.

  3. The evaluated tasks have finite solution spaces, highly structured outputs, explicit rules, and relatively straightforward outcome extraction. These properties are particularly favorable for outcome-level self-verification and caching. It is therefore unclear whether the method extends to the broader scientific and engineering reasoning settings emphasized in the introduction, where outcomes may be open-ended, difficult to canonicalize, or impossible to verify reliably through reverse reasoning.

  4. The current comparison does not fully isolate which part of MS-TTRL produces the improvement. In particular, the paper lacks comprehensive comparisons with verifier-only rewards, verifier rewards combined with standard entropy or novelty regularization, periodic re-verification, frozen-verifier training, and inference-time diverse sampling without reinforcement learning. The untrained base model is also an important reference. This is especially relevant because the paper states that MS-TTRL remains below the base model on MACE-Math, while the corresponding base-model result is not shown in the main table.

  5. TTRL-CoCoV primarily promotes trajectory-level diversity, whereas MS-TTRL explicitly targets diversity among final outcomes. Its limited improvement in Recovery Rate therefore does not strongly establish the superiority of MS-TTRL over alternative outcome-level diversity methods; it mainly confirms that diverse reasoning trajectories do not necessarily produce diverse final answers. This comparison is relevant but not sufficient for the paper’s central claim. More related baselines should be included to emphasize the advantage of the proposed method.

Questions for authors

I have a basic question to this study: Does the method actually solve the multiple-solution discovery problem? Its success depends on the policy first sampling a valid outcome and the self-verifier reliably recognizing it. Therefore, the current evidence supports improved preservation of observed valid outcomes rather than general recovery of the full solution space.

Others refer to Weakness. In summary, I think there should be more evidence, such as related baselines, tasks, or verifier effectiveness, to prove the strengths of the method.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank reviewer degW for the thorough review and valuable feedback on shared biases between generation and verification, cache reliability, applicability to open-ended tasks, baseline and component comparisons, and the distinction between solution discovery and diversity preservation. We have revised the manuscript accordingly, with newly added headings highlighted in red in the updated PDF.

W1: Self-verifier reliability and shared biases

We agree that using the same model for generation and verification can introduce shared systematic biases. In our formulation, pt(z)p_t(z) is a verification pass rate, not a calibrated probability of correctness. Its purpose is to provide reasonably reliable outcome rewards in the label-free setting.

Precision, recall, and false-accept rate

To evaluate these rewards, we measured the precision, recall, and false-accept rate of the self-verifier's acceptance decisions on sampled outcomes:

TaskPrecision (%)Recall (%)False-accept rate (%)
Causal92.9285.1728.85
3D99.82100.002.04
Boolean99.8299.990.58

High precision and recall across all three tasks support the reliability of the self-verifier. The false-accept rate is elevated only on Causal (28.85%). Overall, despite shared systematic biases, the self-verifier provides reliable rewards without ground-truth labels.

Comparison with an external verifier

We also recorded the outcomes assessed by MS-TTRL's self-verifier and re-verified these same outcomes offline using Phi4-4B-mini-Instruct:

TaskPrecision (%)Recall (%)False-accept rate (%)
Causal98.393.300.24
3D99.2777.116.53
Boolean99.9461.330.12

Phi achieves a lower false-accept rate on Causal at the cost of a substantial drop in recall. Its overall verification performance on 3D and Boolean is worse or comparable. Reliable rewards for multiple-solution tasks must support both the correctness of accepted outcomes and coverage of valid outcomes. The self-verifier maintains high precision and recall on all three tasks, showing that shared systematic biases do not prevent it from providing an effective correctness signal.

W2: Self-verifier updates and the need for re-verification

Handling uncertain judgments

In addition to the verifier quality discussed in W1, the Candidate Set reduces the risk of permanently caching uncertain judgments too early. Outcomes with intermediate verification pass rates remain in the Candidate Set, receive soft rewards, and are re-verified by the updated model when the corresponding problem is revisited. This keeps outcomes with insufficient evidence from immediately receiving either the inverse-frequency amplification reserved for confirmed correct outcomes or the zero reward that would suppress them.

The Candidate Set ablation in the paper supports this design. On Causal, removing the Candidate Set and using the midpoint threshold yields 74.80% accuracy and 41.79% Recovery Rate (RR), compared with 93.75% and 45.89% for full MS-TTRL. This demonstrates that the Candidate Set helps mitigate permanent caching errors.

Re-verifying cached outcomes

We have also added a Self-Reconsider baseline, which re-verifies selected cached outcomes from the Correct and Wrong Sets during training, to examine how reusing earlier judgments affects coverage of valid solutions. Results for Qwen3-1.7B are as follows:

TaskMethodRR (%)Standard verification samples
CausalSelf-Reconsider48.8798,240
CausalMS-TTRL45.8925,496
3DSelf-Reconsider27.2029,792
3DMS-TTRL28.397,456
BooleanSelf-Reconsider49.3216,520
BooleanMS-TTRL46.003,264

Relative to MS-TTRL, Self-Reconsider changes RR by +2.98, −1.19, and +3.32 percentage points on Causal, 3D, and Boolean, respectively. MS-TTRL uses 74.0%, 75.0%, and 80.2% fewer standard verification samples. Cache reuse thus substantially reduces repeated verification while maintaining comparable coverage of valid solutions.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

W3: Evaluation on an open-ended task

We have added a small-molecule editing task from ChatDrug [1]. Given a source molecule, the model must generate a different molecule with a single connected component, at least one additional hydrogen-bond donor, and a molecular fingerprint similarity of at least 0.4 to the source molecule.

These problems do not provide a predefined list of candidate molecules, making the task more open-ended and closer to a real scientific application. We use Qwen3.5-4B and evaluate all methods on the same 48 test problems with 32 samples per problem. UValid@32 is the average number of distinct valid molecules generated per problem.

MethodAccuracy (%)UValid@32
Base Model26.505.56
TTRL59.050.81
TTRL-CoCoV58.531.23
MS-TTRL77.217.60

MS-TTRL improves accuracy and UValid@32 over TTRL by 18.16 percentage points and 6.79, respectively, and over TTRL-CoCoV by 18.68 percentage points and 6.38. It also improves over the base model by 50.71 percentage points in accuracy and 2.04 in UValid@32. These results demonstrate the effectiveness of MS-TTRL in a more open-ended task which better reflects real-world scientific problems.

W4: Additional baselines and comparison with the base model

We added the requested comparisons using Qwen3-1.7B: verifier-only rewards, verifier rewards with entropy regularization, periodic re-verification, a frozen verifier, and the untrained base model.

Verification and cache reuse

We compare MS-TTRL with Self-Verify, Self-Verify with entropy regularization, and Self-Reconsider, reporting RR and standard verification sample counts.

Recovery Rate (%)Causal3DBoolean
Self-Verify46.9227.1046.28
Self-Verify + entropy regularization36.359.5841.63
Self-Reconsider48.8727.2049.32
MS-TTRL45.8928.3946.00
Standard verification samplesCausal3DBoolean
Self-Verify259,26443,28029,600
Self-Verify + entropy regularization72,41620,60825,760
Self-Reconsider98,24029,79216,520
MS-TTRL25,4967,4563,264

MS-TTRL achieves similar RR to Self-Verify and Self-Reconsider across the three tasks, while using substantially fewer standard verification samples: reductions of 90.2%, 82.8%, and 89.0% relative to Self-Verify, and 74.0%, 75.0%, and 80.2% relative to Self-Reconsider. These results show that cache reuse reduces verification overhead while preserving solution coverage.

Verifier rewards with entropy regularization yield lower coverage than MS-TTRL on all three tasks and do not effectively improve outcome-level diversity.

Frozen verifier

We next compare MS-TTRL with a variant that freezes the verifier:

Recovery Rate (%)Causal3DBooleanMACE-Math
Frozen verifier44.6727.2045.2134.50
MS-TTRL45.8928.3946.0044.26

A frozen verifier also improves outcome coverage, but less than MS-TTRL. These results indicate that the self-verifier improves as model training proceeds.

Inference-time diversity methods

Without training, we evaluate higher-temperature sampling (T=1.2T=1.2) and Diverse Beam Search (DBS), which penalizes repetition to encourage diverse paths.

Recovery Rate (%)Causal3DBoolean
Base Model + higher-temperature sampling (T=1.2T=1.2)41.338.9630.71
Base Model + DBS28.154.2718.11
MS-TTRL45.8928.3946.00
Outcome Recovery Rate (%)Causal3DBoolean
Base Model + higher-temperature sampling (T=1.2T=1.2)19.694.9422.68
Base Model + DBS14.242.2310.84
MS-TTRL23.2616.9336.49

Averaged over the three tasks, MS-TTRL improves RR over higher-temperature sampling and DBS by 13.09 and 23.25 percentage points, respectively, and also achieves substantially higher Outcome Recovery Rate (ORR). These results show that promoting diversity through inference-time sampling alone has limited effectiveness.

Comparison with the base model

Finally, we compare the untrained base model with TTRL and MS-TTRL:

Recovery Rate (%)Causal3DBooleanMACE-Math
Base Model38.976.8732.1249.72
TTRL36.949.0731.0334.39
MS-TTRL45.8928.3946.0044.26
Outcome Recovery Rate (%)Causal3DBooleanMACE-Math
Base Model18.783.9823.7537.78
TTRL17.425.2822.6322.75
MS-TTRL23.2616.9336.4934.74

Relative to the untrained base model, MS-TTRL improves RR by 6.92, 21.52, and 13.88 percentage points on Causal, 3D, and Boolean, respectively, with ORR also improving on each task. Training therefore increases coverage of valid outcomes under the same evaluation sampling budget.

On MACE-Math, MS-TTRL remains below the base model by 5.46 percentage points in RR and 3.04 percentage points in ORR. However, the corresponding drops for TTRL are 15.33 and 15.03 percentage points. MS-TTRL therefore effectively mitigates mode collapse on MACE-Math relative to TTRL.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

W5: Comparison with outcome-level diversity methods

Existing methods for outcome-level diversity, such as IPS-GRPO and DMPO, target settings with ground-truth labels. To the best of our knowledge, prior work in the label-free setting focuses on trajectory-level diversity, and MS-TTRL is the first method to target outcome-level diversity in this setting.

Beyond TTRL and TTRL-CoCoV, Section 5.4 of the paper evaluates a family of baselines that treat the top kk most frequent outcomes as positive pseudo-labels (k=1,2,3,4k=1,2,3,4) to promote outcome-level diversity. MS-TTRL substantially outperforms every variant, confirming that expanding the set of frequency-based pseudo-labels alone does not resolve the frequency conflict.

Q1: Preserving outcome-level diversity during RL post-training

MS-TTRL does not recover the full solution space. Its central contribution is to address outcome-level mode collapse during label-free RL post-training, preserving diversity across distinct valid outcomes.

Prior work has observed that reinforcement learning from verifiable rewards (RLVR) can improve success rates under small sampling budgets, but can hardly enable a model to solve problems that remain beyond its reach even with repeated sampling [2, 3]. MS-TTRL belongs to the same paradigm. Nevertheless, RLVR remains an important approach to post-training that effectively improves model capability and accuracy. In multiple-solution reasoning, these gains can coexist with generation concentrated on a small subset of valid outcomes. Preserving diversity across distinct valid outcomes is therefore a central objective of post-training.

More specifically, methods such as IPS-GRPO address outcome-level mode collapse through inverse-frequency scaling with ground-truth correctness labels. In the label-free setting, however, methods such as TTRL also use frequency to judge correctness, creating the frequency conflict. MS-TTRL resolves this conflict by estimating correctness through outcome-level self-verification while reserving generation frequency for diversity preservation.

[1] Shengchao Liu et al. Conversational Drug Editing Using Retrieval and Domain Feedback. ICLR, 2024.

[2] Yang Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837, 2025.

[3] Xinyu Zhu et al. The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning. arXiv:2506.01347, 2025.

Reviewer_YW9tOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MS-TTRL, a label-free test-time reinforcement learning method for multiple-solution reasoning. It addresses the Frequency Conflict, where frequency-based correctness estimation favors common outcomes while diversity preservation favors rare ones. MS-TTRL decouples these objectives through outcome-level self-verification, caches verification results, and selectively boosts rewards for rare, confidently verified outcomes. Experiments show an average improvement of 19.13 percentage points in Recovery Rate over the strongest label-free baselines.

Critical strengths
  1. Clearly identifies and mitigates the frequency conflict in multiple-solution settings. In multiple-solution TTRL, using generation frequency to estimate correctness conflicts with using it to preserve outcome diversity: rare but valid answers may be penalized alongside rare errors. The paper offers a well-motivated solution by decoupling outcome-level self-verification from frequency-based diversity weighting. The top-k pseudo-label experiments and advantage analysis of rare valid outcomes provide further empirical support for this design, offering a useful principle for reward construction in multiple-solution settings.
  2. Achieves consistent improvements in solution coverage, with ablations supporting the roles of key components. Across the evaluated configurations spanning four tasks and three backbone models, MS-TTRL substantially improves Recovery Rate over the stronger label-free baseline. Ablation experiments show that removing diversity scaling significantly reduces recovery on the 3D task, while removing confidence gating sharply decreases accuracy on the Causal task. These results support the design choices and demonstrate the method’s ability to balance valid-solution coverage and output correctness.
Critical weaknesses
  1. The headline comparison against label-free baselines is potentially confounded by unequal verification budgets. The TTRL baselines estimate pseudo-labels by majority voting over only 8 rollouts, which is thin for majority estimates to stabilize, whereas MS-TTRL performs multiple verification trials per outcome. The reported 19.13-point Recovery Rate gain may therefore partly reflect additional verification compute rather than the resolution of the frequency conflict, and no matched-budget control (e.g., 32–64 rollouts for the baselines) is reported. This is consequential because superiority over the strongest label-free baselines is the paper's central empirical claim.
  2. Verification verdicts are cached permanently and never reassessed. Since test-time training updates the same model responsible for self-verification, early misclassifications in either direction become fixed and are subsequently reinforced or suppressed throughout the remainder of the run. The paper reports no measurement of cache error rates, no sensitivity analysis to initial verifier quality, and no re-verification ablation. This is consequential because the method's stated goal is recovering rare but valid solutions, which is exactly the regime where early verification is least reliable; the mechanism that delivers the gains could also be the one that entrenches its own errors.
Questions for authors
  1. As the model's capabilities improve during test-time training, its correctness estimates may evolve. Outcomes misclassified early in training could therefore be reassessed correctly by the improved verifier. Permanently caching a problem–outcome pair (x,z)(x,z) as correct or incorrect without re-verification risks retaining outdated judgments, potentially reinforcing invalid outcomes or suppressing valid solutions.
  2. The TTRL baseline relies on majority voting to estimate pseudo-labels, whose reliability generally improves with more samples. However, the experiments use only 8 rollouts, which may be insufficient for majority-vote estimates to stabilize and could yield unreliable pseudo-labels early in training. In contrast, MS-TTRL uses multiple verification trials, providing additional opportunities to assess outcome validity. Evaluating larger rollout budgets (e.g., 32 or 64), ideally under matched computational budgets, would help clarify whether MS-TTRL's gains persist when the majority-voting baselines receive more reliable estimates.
3 comments

We thank reviewer YW9t for the careful review. The main concerns are computational fairness, the reliability of cached verification judgments, and re-verification. We have revised the manuscript accordingly, with newly added headings highlighted in red in the updated PDF.

W1 & Q2: Computational budget and comparison with TTRL-32

We increased TTRL's sampling budget to 32 responses per problem (TTRL-32). The tables below report Recovery Rate (RR), Outcome Recovery Rate (ORR), runtime, and peak GPU memory for Qwen3-1.7B on Causal, 3D, Boolean, and MACE-Math.

Recovery Rate (%)Causal3DBooleanMACE-Math
TTRL36.949.0731.0334.39
TTRL-3236.579.0728.9035.08
MS-TTRL45.8928.3946.0044.26
Outcome Recovery Rate (%)Causal3DBooleanMACE-Math
TTRL17.425.2822.6322.75
TTRL-3217.235.2818.4222.92
MS-TTRL23.2616.9336.4934.74
RuntimeCausal3DBooleanMACE-Math
TTRL2h 21m38m28m—
TTRL-327h 12m1h 50m1h 47m7h 56m
MS-TTRL2h 51m46m38m3h 5m
Peak GPU memory (GB)Causal3DBooleanMACE-Math
TTRL129.14117.57102.87—
TTRL-32129.36129.45110.69137.01
MS-TTRL131.83123.29107.54139.57

Increasing TTRL’s sampling budget leaves the frequency conflict unresolved because correctness is still inferred from generation frequency, allowing outcome-level mode collapse to persist and yielding no consistent improvement in RR or ORR over standard TTRL. MS-TTRL outperforms TTRL-32 on all four tasks with much fewer computational resources. The RR gains are 9.32, 19.32, 17.10, and 9.18 percentage points, respectively, accompanied by substantial improvements in ORR. These results show that MS-TTRL's gains come from resolving the frequency conflict, rather than a modest increase in computation.

W2 & Q1: Self-verifier reliability and the need for re-verification

The Self-Reconsider comparison shows that re-verifying outcomes already cached in the Correct or Wrong Sets yields comparable solution coverage, while MS-TTRL uses 74.0%–80.2% fewer standard verification samples. Uncertain outcomes are already re-verified through the Candidate State when the corresponding problem is revisited.

The Candidate State as an uncertainty buffer

The Candidate State reduces the risk of permanently caching uncertain judgments too early. When the self-verifier is uncertain about a problem–outcome pair (x,z)(x,z), the outcome remains in the Candidate Set and is re-verified by the updated model when the problem is revisited. It therefore need not be assigned immediately to the Correct or Wrong Set. Outcomes already in the Correct or Wrong Set are not re-evaluated by this mechanism. Sections 4.2 and 5.3.1 describe the Candidate State and its ablation results, respectively.

Reliability of self-verification decisions

We further assess the quality of the self-verifier's acceptance decisions by reporting precision, recall, and false-accept rate on the outcomes actually sampled during training.

TaskPrecision (%)Recall (%)False-accept rate (%)
Causal92.9285.1728.85
3D99.82100.002.04
Boolean99.8299.990.58

The self-verifier achieves high precision and recall on all three tasks; only Causal has a relatively high false-accept rate (28.85%). These results support its reliability and show that it provides a reliable reward signal in the label-free setting.

Ablation of re-verification

To assess the reliability of the self-verifier and the need for re-verification, we added Self-Reconsider, a control that re-verifies selected cached outcomes in the Correct and Wrong Sets during training. The results for Qwen3-1.7B are as follows.

TaskMethodRR (%)Standard verification samples
CausalSelf-Reconsider48.8798,240
CausalMS-TTRL45.8925,496
3DSelf-Reconsider27.2029,792
3DMS-TTRL28.397,456
BooleanSelf-Reconsider49.3216,520
BooleanMS-TTRL46.003,264

Self-Reconsider changes RR by only +2.98, −1.19, and +3.32 percentage points relative to MS-TTRL on Causal, 3D, and Boolean, respectively. MS-TTRL achieves this comparable coverage with 74.0%, 75.0%, and 80.2% fewer standard verification samples, demonstrating that cache reuse substantially reduces repeated verification.

Thank you for the detailed responses and the additional experiments.

The TTRL-32 comparison convincingly shows that MS-TTRL's gains ars not simply a result of extra compute, and the Self-Reconsider ablation shows that cache reuse preserve most of the coverage at a much lower verification cost. I appreciate these additional results.

But my main concern is whether self-verification can separate rare valid outcomes from rare invalid ones. However, the precision/recall in G.1 is pooled over sampled responses and dominated by frequent outcomes, so it does not directly answer this.

Taking everything into account, I will maintain my evaluation.

We thank reviewer YW9t for clarifying the question. Section 5.5 shows that MS-TTRL assigns positive advantages to rare valid outcomes more often than TTRL or TTRL-CoCoV. To complement this result, we measured acceptance rates for both rare valid and rare invalid outcomes. We hope this analysis can address reviewer YW9t’s concern about whether self-verification can distinguish rare valid outcomes from rare invalid ones.

Learning signals for rare valid outcomes (Section 5.5)

For each Qwen3-1.7B rollout group containing at least two distinct valid outcomes, we identify the least frequent among those valid outcomes, retaining all ties. Figure 4 shows the mean advantage and positive advantage rate of these outcomes at each training epoch. The tables below report both metrics computed over the full training run for each method and task.

Mean Advantage

MethodCausal3DBoolean
TTRL−1.09−1.86−1.33
TTRL-CoCoV−0.65−1.08−0.71
MS-TTRL+0.75+1.67+1.42

Positive Advantage Rate

MethodCausal3DBoolean
TTRL6.80%0.00%5.50%
TTRL-CoCoV8.46%4.76%7.21%
MS-TTRL70.04%95.10%94.11%

We average both metrics across Causal, 3D, and Boolean. The mean advantage for rare valid outcomes is +1.28 under MS-TTRL, −1.43 under TTRL, and −0.81 under TTRL-CoCoV. MS-TTRL reinforces these outcomes much more often, with a positive advantage rate of 86.42%, compared with 4.10% for TTRL and 6.81% for TTRL-CoCoV.

Can the self-verifier distinguish rare valid outcomes from rare invalid outcomes?

For this comparison, we consider rollout groups with at least two distinct parseable outcomes. Within each rollout group, we select all outcomes tied for the lowest frequency, count each distinct outcome once, and use ground-truth labels to classify it as valid or invalid. We count a selected outcome as accepted if MS-TTRL has placed it in the Correct Set by the end of the epoch.

TaskAcceptance rate for rare valid outcomesFalse-accept rate for rare invalid outcomes
Causal1,730/2,250 (76.89%)1,643/4,052 (40.55%)
3D1,935/1,935 (100.00%)29/575 (5.04%)
Boolean1,065/1,068 (99.72%)22/606 (3.63%)

MS-TTRL accepted rare valid outcomes more often than rare invalid outcomes on all three tasks. With each task weighted equally, the average acceptance rate was 92.20% for rare valid outcomes and 16.41% for rare invalid outcomes.

Reviewer_d6hSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks how TTRL should behave when a problem admits many valid answers, and observes that generation frequency is being asked to do two opposing jobs: majority voting reads frequency as correctness, while diversity weighting reads rarity as value. MS-TTRL replaces the frequency-based correctness estimate with outcome-level self-verification (the same policy model verifies each distinct outcome n=8 times), then reuses cached verdicts, keeps ambiguous outcomes alive as candidates with soft rewards, and gates inverse-probability scaling to the verified-correct set. Across three HypoSpace subsets and MACE-Math with three backbones and five seeds, it reports a 19.13-point average gain over the best label-free baseline, reaching or exceeding supervised IPS-GRPO on several 3D cells while remaining below the untrained model on MACE.

Critical strengths

1.Naming the dual role of generation frequency is a cleaner diagnostic than pass@k or entropy: it explains why TTRL can sit at 99.87% accuracy with 9.07% recovery on 3D, a failure mode that standard metrics report as success. 2.The top-k experiment is a well-designed negative result — accuracy on Causal falls from 99.97 to 18.95 to 1.29 as k grows — showing the conflict is not a thresholding artifact that more pseudo-labels can fix.

Critical weaknesses

1.The decoupling may be nominal: an outcome counts as correct only when nearly all verification samples agree, and the verifier is the same model at the same temperature, so its errors are plausibly correlated with generation frequency. No verifier accuracy, false-accept rate, or correlation between pass rate and frequency is reported. 2.The minimal self-verification variant already matches or beats the full model on most tasks, and the headline average is dominated by settings where the baseline collapsed. The main table also has no untrained-model row, so it is unclear whether training helps coverage at all. 3.No label-free competitor from the related work is run, and no compute-matched control spends the extra verification budget on additional rollouts instead. The gain therefore cannot be separated from extra inference compute, and evaluation rests on very small test sets.

Questions for authors
  1. What are the verifier's precision, recall and false-accept rate against the offline ground-truth solution sets, and how strongly does p_t(z) correlate with n_t(z) across outcomes? If the correlation is high, in what sense is correctness estimation frequency-independent?
  2. Why does plain Self-Verify outperform the full MS-TTRL on two of three tasks? A control that gives every candidate a constant soft reward instead of p_t would show whether the gain comes from verification or simply from the exploratory effect of not zeroing uncertain outcomes.
  3. How does MS-TTRL compare with TTRL at G=16 or 32 under matched total generated tokens, and with the untrained model on each HypoSpace cell? Given that MACE already falls below the base model, which settings show a genuine coverage improvement over no training?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We appreciate reviewer d6hS's detailed review and constructive questions about self-verifier reliability, the relationship between verification pass rates and generation frequency, the roles of individual components, and comparisons with larger-budget TTRL and the untrained base model. We have revised the manuscript accordingly, with newly added headings highlighted in red in the updated PDF.

W1 & Q1: Self-verifier reliability and correlation with generation frequency

Precision, recall, and false-accept rate

We evaluated the self-verifier's precision, recall, and false-accept rate on the outcomes actually sampled during training:

TaskPrecision (%)Recall (%)False-accept rate (%)
Causal92.9285.1728.85
3D99.82100.002.04
Boolean99.8299.990.58

Precision and recall are high across all three tasks, supporting the reliability of the self-verifier. The false-accept rate is elevated only on Causal (28.85%). Overall, the self-verifier provides reliable rewards in the label-free setting.

Correlation with generation frequency

We also measured the correlation between the verification pass rate pt(z)p_t(z) and generation frequency nt(z)n_t(z) in MS-TTRL, and compared it with the correlations of ground-truth rewards and TTRL's majority-vote rewards with nt(z)n_t(z). The results for Qwen3-1.7B are:

Pearson correlation with nt(z)n_t(z)Causal (360 steps)3D (120 steps)Boolean (90 steps)
Ground-truth reward0.4510.4910.258
MS-TTRL verification pass rate pt(z)p_t(z)0.2680.4800.250
TTRL majority-vote reward0.8080.5950.552

MS-TTRL has a lower positive correlation than TTRL on all three tasks. Generation frequency and correctness are naturally positively correlated in a pretrained model. The distinction is how correctness is estimated: TTRL directly relies on generation frequency, while MS-TTRL uses self-verification. Here, decoupling means that correctness estimation and diversity preservation no longer use frequency as a shared criterion with opposing preferences for frequent and rare outcomes; it does not mean that correctness and generation frequency become uncorrelated. We have revised potentially misleading wording such as “frequency-independent” in the manuscript.

Learning signals for rare valid outcomes

The central issue is that judging correctness directly by frequency penalizes rare, underrepresented valid outcomes and reduces diversity. Section 5.5 of the original manuscript examines this mechanism by measuring the Mean Advantage and Positive Advantage Rate of rare valid outcomes under different training methods. In the table below, values are ordered as Causal / 3D / Boolean:

MethodMean AdvantagePositive Advantage Rate (%)
TTRL-1.09 / -1.86 / -1.336.80 / 0.00 / 5.50
TTRL-CoCoV-0.65 / -1.08 / -0.718.46 / 4.76 / 7.21
MS-TTRL0.75 / 1.67 / 1.4270.04 / 95.10 / 94.11
IPS-GRPO (supervised reference)1.64 / 2.19 / 1.6686.16 / 100.00 / 92.82

Averaged across the three tasks, MS-TTRL achieves a Mean Advantage of 1.28 and a Positive Advantage Rate of 86.42%, substantially exceeding TTRL (−1.43; 4.10%) and TTRL-CoCoV (−0.81; 6.81%) and approaching supervised IPS-GRPO (1.83; 92.99%). These stronger learning signals for rare valid outcomes explain its advantage in outcome-level diversity.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W2 & Q2: Comparison with Self-Verify and the role of candidate rewards

Verification cost and cache reuse

Plain Self-Verify achieves comparable Recovery Rate (RR) to MS-TTRL on Causal, 3D, and Boolean, but re-verifies the distinct outcomes generated at every training step. MS-TTRL reuses previous verification decisions through its Three-State Outcome Cache, reducing repeated verification. We quantify verification cost by the total number of verification samples generated over the complete training run. The Qwen3-1.7B results are:

Recovery Rate (%)Causal3DBoolean
Self-Verify46.9227.1046.28
MS-TTRL45.8928.3946.00
Verification cost (total verification samples)Causal3DBoolean
Self-Verify259,26443,28029,600
MS-TTRL25,4967,4563,264
Reduction in verification samples90.2%82.8%89.0%

MS-TTRL thus maintains nearly the same RR as plain Self-Verify while reducing verification cost by 90.2%, 82.8%, and 89.0% on Causal, 3D, and Boolean, respectively.

Verification-based candidate rewards

To isolate the role of verification in the Candidate Set, we conducted the suggested experiment by replacing every candidate's reward with a constant value of 0.5. The table reports RR and Outcome Recovery Rate (ORR) for Qwen3-1.7B:

TaskMethodRR (%)ORR (%)
CausalMS-TTRL45.8923.26
CausalMS-TTRL (constant soft reward)42.3721.40
3DMS-TTRL28.3916.93
3DMS-TTRL (constant soft reward)12.396.74
BooleanMS-TTRL46.0036.49
BooleanMS-TTRL (constant soft reward)41.9235.26

Using the verification pass rate pt(z)p_t(z) improves RR by 3.52, 16.00, and 4.08 percentage points over the constant-reward variant on Causal, 3D, and Boolean, respectively. Candidate rewards therefore benefit from the self-verifier's signal, not only from continued exploration of uncertain outcomes.

W3 & Q3: Larger TTRL sampling budgets and comparison with the base model

We increased TTRL's sampling budget to 32 responses per problem (TTRL-32) and added evaluation results for the untrained base model. The following tables report RR, ORR, runtime, and peak GPU memory for Qwen3-1.7B on Causal, 3D, Boolean, and MACE-Math:

Recovery Rate (%)Causal3DBooleanMACE-Math
Base Model38.976.8732.1249.72
TTRL36.949.0731.0334.39
TTRL-3236.579.0728.9035.08
MS-TTRL45.8928.3946.0044.26
Outcome Recovery Rate (%)Causal3DBooleanMACE-Math
Base Model18.783.9823.7537.78
TTRL17.425.2822.6322.75
TTRL-3217.235.2818.4222.92
MS-TTRL23.2616.9336.4934.74
RuntimeCausal3DBooleanMACE-Math
TTRL2h 21m38m28m—
TTRL-327h 12m1h 50m1h 47m7h 56m
MS-TTRL2h 51m46m38m3h 5m
Peak GPU memory (GB)Causal3DBooleanMACE-Math
TTRL129.14117.57102.87—
TTRL-32129.36129.45110.69137.01
MS-TTRL131.83123.29107.54139.57

Increasing TTRL’s sampling budget from 8 to 32 yields no consistent improvement in RR or ORR. The larger budget leaves the frequency conflict unresolved: correctness is still estimated from generation frequency, so outcome-level mode collapse persists. With fewer computational resources, MS-TTRL outperforms TTRL-32 in RR by 9.32, 19.32, 17.10, and 9.18 percentage points on Causal, 3D, Boolean, and MACE-Math, respectively, while also achieving higher ORR on all four tasks.

Relative to the untrained base model, MS-TTRL improves RR by 6.92, 21.52, and 13.88 percentage points on Causal, 3D, and Boolean, respectively, with gains in ORR on all three tasks as well. These results show that training improves solution coverage on these tasks. On MACE-Math, MS-TTRL remains below the base model but exhibits substantially less mode collapse than baselines such as TTRL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.