acceptodds
Under review as a conference paper at ICLR 2027

TTPO: Test-Time Policy Optimization

14 views

Abstract

Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time training, on-policy self-distillation, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VPZKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TTPO adapts language models to unlabeled math problems using majority-vote answers. It distills agreeing rollouts from a thinking-mode teacher and penalizes disagreeing rollouts with a GRPO loss. Token weighting and masking refine the updates. Experiments on three Qwen3 sizes show gains over TTRL and OPSD-TTT, including transfer to other math benchmarks.

Critical strengths

(1) The asymmetric objective is simple and well motivated. Figure 3 supports the choice: distilling positive rollouts and penalizing negative ones works better than reversing the two updates. (2) Improvements hold across three model sizes. The cross-benchmark results in Figure 4 also provide evidence that the updates help beyond the problems used for adaptation.

Critical weaknesses

(1) The main components of TTPO, including majority-vote pseudo-labeling, on-policy self-distillation, sample-level routing between RL and distillation, and token-level selection, have been explored in prior work. In particular, SRPO, CANON, U-OPSD, and SCRL are closely related to the proposed framework. However, direct comparisons with these methods are missing, making it difficult to assess the benefits of the proposed asymmetric objective over existing approaches.

(2) The effectiveness of token-level selection is not sufficiently isolated. Although the ablations show benefits from positive-sample weighting and negative-sample masking, both rely on heuristic combinations of entropy, likelihood, and teacher–student divergence. Low-probability tokens under low-entropy distributions do not necessarily indicate reasoning errors. Comparisons with simpler alternatives, such as entropy-only, likelihood-only, and random selection, are needed to better justify these designs.

(3) TTPO requires 64 rollouts per problem, each up to 16,000 tokens, but lacks comparisons of computational costs. In addition, selecting the best checkpoint based on test-set accuracy may introduce label leakage. Fixed-step evaluation would help address this concern.

Questions for authors

See Weaknesses. If the authors can adequately address my concerns, I would be willing to raise my score.

4 comments
AuthorsOpenReview ↗

Rebuttal [1/3]

Dear Reviewer VPZK,

Thank you for your time and for recognizing that the asymmetric objective is simple and well motivated (Figure 3), with improvements that hold across model sizes and benchmarks. All new experiments use Qwen3-1.7B unless stated otherwise. Our responses are below.

[Weakness 1]: The main components of TTPO, including majority-vote pseudo-labeling, on-policy self-distillation, sample-level routing between RL and distillation, and token-level selection, have been explored in prior work. In particular, SRPO, CANON, U-OPSD, and SCRL are closely related to the proposed framework. However, direct comparisons with these methods are missing, making it difficult to assess the benefits of the proposed asymmetric objective over existing approaches.

Thank you for raising this question. These methods are indeed related, but TTPO differs from each of them in its core design: which samples are distilled and which are penalized, what the teacher is conditioned on, and the token-level rules of each branch.

MethodUpdate on PUpdate on NTeacher conditioned onGT
SRPOGRPO on correct samplesdistillation on incorrect samplesfull text of a correct sibling rolloutYes
CANONdistillationdistillationfull text of one majority rolloutNo
U-OPSDnot updateddistillationfull text of an agreeing rolloutNo
SCRLGRPO under strong consensusentropy-gated negative labels– (no teacher)No
TTPOFKL + token weightsGRPO + token maskagreeing final answer onlyNo

Direct comparison. We reproduce all four methods with their original objectives. Since SRPO requires GT, we compare in the OpenThoughts setting of Table 1 (SRPO†), and for fairness all methods follow the OPSD configuration (training data, LoRA, 100 steps × 32 trajectories, and best-checkpoint evaluation).

MethodAIME26HMMT26BRUMO25Avg
SRPO†38.329.848.338.8
CANON37.529.848.938.7
U-OPSD44.431.349.741.8
SCRL40.030.850.640.5
TTPO46.531.654.744.3

TTPO achieves the best average, including over the GT-based SRPO†. We attribute the gaps to the following differences:

  • SRPO† is designed for a large training budget; under the low-cost budget shared by all methods for a fair comparison, it cannot reach its best performance.
  • CANON was designed for test-time adaptation of non-thinking models: it distills all samples from a frozen non-thinking teacher conditioned on a full consensus solution, which gives little signal for thinking-mode evaluation. Our teacher ablation (Table 4) confirms this: on AIME26, a non-thinking teacher conditioned on a full trajectory reaches 40.8, whereas TTPO's thinking teacher conditioned on the answer reaches 46.5.
  • U-OPSD distills only the disagreeing samples toward a teacher that sees an agreeing solution, so a wrong vote pulls them toward a wrong answer. TTPO instead penalizes them, which stays accurate because disagreeing rollouts are mostly wrong whether or not the vote is correct (Fig. 1); negative-only distillation also trails TTPO in Fig. 3 (43.9 vs. 48.9).
  • SCRL filters TTRL's pseudo-labels by confidence, but both branches remain sequence-level GRPO, without a dense token-level distillation signal on agreeing samples.

We have added this comparison and discussion to Appendix D.6 of the revised paper.

AuthorsOpenReview ↗

Rebuttal [2/3]

[Weakness 2]: The effectiveness of token-level selection is not sufficiently isolated. Although the ablations show benefits from positive-sample weighting and negative-sample masking, both rely on heuristic combinations of entropy, likelihood, and teacher–student divergence. Low-probability tokens under low-entropy distributions do not necessarily indicate reasoning errors. Comparisons with simpler alternatives, such as entropy-only, likelihood-only, and random selection, are needed to better justify these designs.

Thank you for this suggestion. We agree that such a token is not necessarily an error. The score does not need to identify errors exactly: it targets anomalous tokens, where the model confidently produced unlikely content, and spares high-probability tokens, which are typically locally correct steps, so that penalizing a disagreeing trajectory does not damage its correct reasoning. We compare it with the requested alternatives at the same retention rate, and also with update-strength-matched controls (Table 3 setting: OpenThoughts, Qwen3-1.7B; best checkpoint for every variant, as in Table 1). The strength-matched negative control removes the mask and halves λ to 0.05; "uniform × w̄ᵢ" keeps each trajectory's total Soft-OR weight but spreads it evenly over its tokens.

VariantAIME26HMMT26BRUMO25Avg
TTPO (Soft-OR weights; mask −log p·(1−Ĥ), 50% kept)46.531.654.744.3
Pos.: uniform (Table 3)43.330.652.842.2
Pos.: uniform × w̄ᵢ45.027.345.339.2
Pos.: entropy only45.830.350.642.2
Pos.: disagreement only45.629.052.842.5
Neg.: no mask (Table 3)45.429.550.041.6
Neg.: no mask, λ = 0.0544.430.150.841.8
Neg.: random 50%43.930.349.741.3
Neg.: likelihood only46.728.548.641.3
Neg.: entropy only46.431.151.142.9

Every alternative falls below TTPO. Random selection at the same retention rate stays at the no-mask level, and so does the strength-matched control, so the gain does not come from penalizing fewer tokens. Likelihood alone is no better than random and entropy alone helps only partly; the same holds for the single factors on the positive side. The combination is what selects the informative tokens.

We have added this comparison to Appendix D.3 of the revised paper.

AuthorsOpenReview ↗

Rebuttal [3/3]

[Weakness 3]: TTPO requires 64 rollouts per problem, each up to 16,000 tokens, but lacks comparisons of computational costs. In addition, selecting the best checkpoint based on test-set accuracy may introduce label leakage. Fixed-step evaluation would help address this concern.

Thank you for raising both points.

(a) Computational cost. For a fair comparison, all methods follow OPSD's default training configuration, with the same batch size of 32 trajectories per step, LoRA setting and learning rate, so the training cost per step is controlled.

MethodSamples × lengthProblems / stepTrajectories updated / problemTokens updated / trajectoryStepsTime / step
GRPO8 × 16k4816k50077.3 s
OPSD1 × 1,0243211,02410012.7 s
TTPO64 × 16k481,02410051.2 s

OpenThoughts, Qwen3-1.7B, 4×H20.

Compared with GRPO, TTPO is faster per step and needs 5× fewer steps, yet it is more accurate (Table 1: 40.1 vs. 35.7 for GRPO†; 39.8 vs. 33.0 at step 100). Compared with OPSD, it trades compute for labeled data, sampling enough for a reliable, noise-robust vote: it matches the GT-supervised OPSD† (39.7), and its up to 128× more generated tokens per step cost only about 4× the time.

At exactly the same cost (same samples, steps and hyperparameters), TTPO also outperforms majority-vote distillation of all samples (TTT, step 100):

Method (64 × 16k samples, 8 updated / problem, 100 steps)AIME26HMMT26BRUMO25Avg
Majority-vote distillation of all samples45.326.342.838.1
TTPO47.833.653.144.8

Moreover, TTPO scales with its sampling budget, offering a smooth trade-off between compute and accuracy that can be matched to the available resources (AIME26, step 100, Avg@32):

Length × K16k×816k×1616k×3216k×648k×644k×641k×64
AIME2643.645.346.246.546.145.437.8
Answer extractable99.6%99.6%99.8%99.8%99.0%92.3%45.6%
Time / step (s)22.823.234.051.244.540.68.94

More samples make the vote more robust to noise, and lengths of 4k or more keep answers extractable; at 1k only 45.6% are, and accuracy falls to the base (37.8). When compute is limited, a small loss in accuracy buys a large saving: 16k×32 keeps 46.2 at two-thirds of the time, and 16k×16 keeps 45.3 at less than half.

(b) Checkpoint selection. We originally reported each method's peak checkpoint, following the reporting protocol of OPSD: RL-based methods are typically trained for more steps than distillation-based ones (500 vs. 100 here), so comparing peaks shows the best attainable performance of every method under its own training budget. This protocol comes from OPSD's OpenThoughts setting, but in the TTT setting we introduce, selecting checkpoints by test accuracy could be read as using ground-truth labels. We therefore additionally evaluate every method at a fixed step 100 in both settings (mean ± standard error over 4 evaluation runs), which requires no labels. TTPO remains the best method under both protocols, matching or exceeding the GT-supervised OPSD† without any labels:

ModelBaseGRPO†OPSD†TTPO
1.7B34.6 ± 1.233.0 ± 0.939.3 ± 0.839.8 ± 1.6
4B56.0 ± 1.255.2 ± 0.857.7 ± 1.258.1 ± 0.7
8B58.6 ± 1.258.9 ± 1.360.7 ± 0.761.3 ± 1.0

OpenThoughts (Table 1), step 100: Avg over AIME25 / HMMT25 / AIME26 / HMMT26 / BRUMO25.

ModelBaseTTRLOPSD-TTTTTPO
1.7B38.0 ± 1.335.5 ± 2.041.9 ± 1.544.8 ± 1.9
4B57.4 ± 1.357.8 ± 1.957.7 ± 1.960.5 ± 0.9
8B60.7 ± 1.662.1 ± 1.262.5 ± 1.764.8 ± 1.3

TTT (Table 2), step 100: Avg over AIME26 / HMMT26 / BRUMO25.

In the revised paper, we have added the training-cost table and the sampling-budget analysis to Appendix D.4, and fixed step-100 results with standard errors for every benchmark to both main tables (Table 1 and Table 2).

Thank you again for your thorough review and for your openness to reconsidering the score. We hope these results address your concerns, and we would be happy to answer any further questions.

Sincerely, TTPO Authors

AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer VPZK,

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.

If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

TTPO Authors

Reviewer_GmTxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. TTPO adapts language models to unlabeled test problems using majority-vote pseudo-labels. It applies self-distillation to agreeing rollouts and negative RL updates to disagreeing ones, with token weighting and masking. Experiments on Qwen3 models TTPO matches ground-truth-supervised methods, substantially outperforms existing label-free approaches

Critical strengths
  • The paper investigates a useful and important problem: how to adapt reasoning models without ground-truth answers.
  • Results demonstrate consistent improvements across various model sizes.
  • The paper conducts extensive ablation studies to examine various update strategies, token selection, teacher prompts, and rollout choices.
Critical weaknesses
  • I think the evaluation setting is confusing. Why select the best ckpt based on test accuracy? Doesn't this imply that ground-truth labels are being used during the model selection process?
  • TTPO generates 64 rollouts per problem, while sampling and training budgets differ across methods. Please compare performance at matched total computational cost.
  • The robustness argument in Eqs. (10)–(11) needs justification. Agreement with a pseudo-label does not remove its influence on the teacher or guarantee harmless updates.
Questions for authors

None

5 comments
AuthorsOpenReview ↗

Rebuttal [1/2]

Dear Reviewer GmTx,

Thank you for your time and for recognizing the importance of adapting reasoning models without ground truth, the consistent improvements across model sizes, and the breadth of our ablations. All new experiments use Qwen3-1.7B unless stated otherwise. Our responses are below.

[Weakness 1]: I think the evaluation setting is confusing. Why select the best ckpt based on test accuracy? Doesn't this imply that ground-truth labels are being used during the model selection process?

Thank you for your question. We originally reported each method's peak checkpoint, following the reporting protocol of OPSD: RL-based methods are typically trained for more steps than distillation-based ones (500 vs. 100 here), so comparing peaks shows the best attainable performance of every method under its own training budget. This protocol comes from OPSD's OpenThoughts setting, but in the TTT setting we introduce, selecting checkpoints by test accuracy could be read as using ground-truth labels. We therefore additionally evaluate every method at a fixed step 100 in both settings (mean ± standard error over 4 evaluation runs), which requires no labels. TTPO remains the best method under both protocols, matching or exceeding the GT-supervised OPSD† without any labels:

ModelBaseGRPO†OPSD†TTPO
1.7B34.6 ± 1.233.0 ± 0.939.3 ± 0.839.8 ± 1.6
4B56.0 ± 1.255.2 ± 0.857.7 ± 1.258.1 ± 0.7
8B58.6 ± 1.258.9 ± 1.360.7 ± 0.761.3 ± 1.0

OpenThoughts (Table 1), step 100: Avg over AIME25 / HMMT25 / AIME26 / HMMT26 / BRUMO25.

ModelBaseTTRLOPSD-TTTTTPO
1.7B38.0 ± 1.335.5 ± 2.041.9 ± 1.544.8 ± 1.9
4B57.4 ± 1.357.8 ± 1.957.7 ± 1.960.5 ± 0.9
8B60.7 ± 1.662.1 ± 1.262.5 ± 1.764.8 ± 1.3

TTT (Table 2), step 100: Avg over AIME26 / HMMT26 / BRUMO25.

In the revised paper, we have added fixed step-100 results with standard errors for every benchmark to both main tables (Table 1 and Table 2).

[Weakness 2]: TTPO generates 64 rollouts per problem, while sampling and training budgets differ across methods. Please compare performance at matched total computational cost.

Thank you for raising this question. For a fair comparison, all methods follow OPSD's default training configuration, with the same batch size of 32 trajectories per step, LoRA setting and learning rate, so the training cost per step is controlled.

MethodSamples × lengthProblems / stepTrajectories updated / problemTokens updated / trajectoryStepsTime / step
GRPO8 × 16k4816k50077.3 s
OPSD1 × 1,0243211,02410012.7 s
TTPO64 × 16k481,02410051.2 s

OpenThoughts, Qwen3-1.7B, 4×H20.

Vs. GRPO. TTPO is faster per step and needs 5× fewer steps, yet it is more accurate (Table 1: 40.1 vs. 35.7 for GRPO†; 39.8 vs. 33.0 at step 100).

Vs. OPSD. TTPO trades compute for labeled data, sampling enough for a reliable, noise-robust vote. It matches the GT-supervised OPSD† (39.7), and its up to 128× more generated tokens per step cost only about 4× the time.

At the same cost. We compare TTPO with majority-vote distillation of all samples, using exactly the same samples, steps and hyperparameters (TTT, step 100):

Method (64 × 16k samples, 8 updated / problem, 100 steps)AIME26HMMT26BRUMO25Avg
Majority-vote distillation of all samples45.326.342.838.1
TTPO47.833.653.144.8

TTPO is better on every benchmark, so its gain comes from how the samples are used, not from more compute.

Moreover, TTPO scales with its sampling budget, offering a smooth trade-off between compute and accuracy that can be matched to the available resources (AIME26, step 100, Avg@32):

Length × K16k×816k×1616k×3216k×648k×644k×641k×64
AIME2643.645.346.246.546.145.437.8
Answer extractable99.6%99.6%99.8%99.8%99.0%92.3%45.6%
Time / step (s)22.823.234.051.244.540.68.94

More samples make the vote more robust to noise, and lengths of 4k or more keep answers extractable; at 1k only 45.6% are, and accuracy falls to the base (37.8). When compute is limited, a small loss in accuracy buys a large saving: 16k×32 keeps 46.2 at two-thirds of the time, and 16k×16 keeps 45.3 at less than half.

We have added the training-cost table and the sampling-budget analysis to Appendix D.4 of the revised paper.

AuthorsOpenReview ↗

Rebuttal [2/2]

[Weakness 3]: The robustness argument in Eqs. (10)–(11) needs justification. Agreement with a pseudo-label does not remove its influence on the teacher or guarantee harmless updates.

Thank you for this comment. We would like to clarify that the paper does not claim that agreement with a pseudo-label removes its influence on the teacher or guarantees harmless updates. Eqs. (10)–(11) are approximations that describe which term dominates each update: on agreeing samples, a wrong pseudo-label has only a small influence, so the update approximately reduces to thinking → non-thinking distillation. We now verify this directly (TTT, AIME26, step 100).

(a) A wrong pseudo-label barely changes what the teacher teaches. To measure its influence on the teacher, we train with the distillation branch only, on agreeing trajectories (whose answer equals the pseudo-label), with a teacher that either sees this answer or no answer:

Distilled trajectoriesTeacher: no answerTeacher: own answer
all agreeing (Fig. 3)45.746.7
agreeing, wrong pseudo-label only46.446.1
agreeing, correct pseudo-label only43.143.9

On majority-consistent but wrong trajectories (row 2), the wrong-answer-conditioned and answer-free teachers perform nearly identically (46.1 vs. 46.4). Row 3 is lower for two reasons: only about 15% of problems have a correct label, so most batches contribute few positives and starve the distillation branch (as in Fig. 5); and the base already solves these problems (98.3, see (b)), so reinforcing them adds little.

(b) TTPO preserves what the model already solves and still improves where the pseudo-label is wrong. We split the 30 problems by whether the step-0 vote is correct and compare TTPO with the base on each group (Avg@12):

ProblemsBaseTTPO
correct pseudo-label (n = 5)98.398.3
wrong pseudo-label (n = 25)25.737.7

Even with most pseudo-labels wrong, the accuracy on problems the model already solves is fully preserved.

(c) The robustness holds even when most pseudo-labels are wrong. Each step trains on 4 problems, k of which get a wrong pseudo-label (noise = k/4): problems whose vote is already wrong are chosen first and keep their vote; otherwise a correctly voted problem takes its largest wrong-answer cluster. The rest get the GT answer.

Wrong pseudo-labels0% (GT)25%50%75%natural (≈85%)100%
TTPO (AIME26)44.446.746.446.447.845.8
Correct among vote negatives0.74%1.75%1.10%1.25%1.29%1.37%

TTPO is noise-tolerant: from 25% to the natural rate accuracy stays at 46.4–47.8, and even at 100% it keeps 45.8 (+8.0 over the base, 37.8). GT labels (0%) are not better, as in Fig. 5: few rollouts match the GT on hard problems, starving both branches. Among the original vote's negatives (before noise injection), only 0.7–1.8% are correct at every noise level (training average).

In the revised paper, we have stated explicitly in Appendix E.2 that Eqs. (10)–(11) are approximations, and added these results to Appendix D.5.

Thank you again for your thorough review. We hope our rebuttal has addressed your concerns, and we would be happy to answer any further questions.

Sincerely, TTPO Authors

AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer GmTx,

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.

If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

TTPO Authors

Reviewer_JrXeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

TTPO adapts language models to unlabeled test problems by using majority-vote answers as pseudo-labels. It applies self-distillation to trajectories that agree with the majority and selective RL penalties to those that disagree, with token-level weighting on both branches. Experiments on three Qwen3 models show improvements across several math benchmarks.

Critical strengths
  • The method does not treat all pseudo-labeled trajectories in the same way. Separating majority-consistent and inconsistent responses offers a useful way to combine detailed distillation feedback with negative updates, rather than forcing every response toward a potentially wrong answer.

  • The ablations cover several meaningful choices, including branch assignments, token selection, teacher conditioning, and sample composition. Reversing the two branches substantially hurts performance, providing evidence for the proposed assignment. Results across three model sizes and transfer between math datasets also strengthen the empirical findings.

Critical weaknesses
  1. The experiments do not directly establish why TTPO remains reliable when pseudo-labels are wrong. The observation that most majority-inconsistent responses are incorrect does not show how much the method suppresses the remaining correct responses. Likewise, agreement between a trajectory’s final answer and the teacher’s supplied answer does not make wrong-answer conditioning harmless to intermediate reasoning. Controlled noise experiments and separate measurements for correct and incorrect pseudo-labels are needed to support this explanation.
  2. The component ablations leave several alternative explanations open. Removing the negative-token mask increases the amount of penalization, so the performance drop does not establish that the mask identifies better tokens to penalize. The large non-thinking gains could also come mainly from thinking-to-non-thinking distillation rather than the asymmetric objective. Random masks, update-strength-matched controls, and branch ablations evaluated in non-thinking mode are missing from the evidence needed to distinguish these possibilities.
  3. The main comparisons do not fully isolate the training objective from sampling budget and teacher configuration. TTPO uses 64 samples to construct its pseudo-labels, whereas OPSD-TTT uses a single deterministic thinking-mode output. Majority-based distillation controls appear in local ablations but are not evaluated across the full main setting. Reporting peak checkpoint performance without a clearly label-free selection rule or multi-run uncertainty also makes the smaller gains difficult to assess.
Questions for authors
  1. Could the authors evaluate majority-based all-sample distillation, positive-only distillation, and a simple all-sample distillation-plus-GRPO objective across the main models and TTT benchmarks? These comparisons should use the same pseudo-labels, teacher, sampled trajectories, and comparable tuning budgets. A confidence-filtered TTRL baseline or a closely related label-free self-distillation method would also help establish whether asymmetric routing offers more than simpler alternatives.

  2. Could the authors separate problems with correct and incorrect pseudo-labels and measure the effect of each training branch on both groups? Please include controlled pseudo-label noise levels, track whether correct minority responses lose probability, and compare wrong-answer-conditioned versus answer-free teachers on majority-consistent but incorrect trajectories. This would directly test the claim that the two branches handle wrong pseudo-labels relatively safely.

  3. Could the authors compare the negative-token mask with random selection at the same retention rate, probability-only selection, entropy-only selection, and unmasked updates with a reduced RL weight? For positive weighting, entropy-only, teacher–student-disagreement-only, and uniform weighting with matched average weight would help determine whether the gains come from selecting informative tokens rather than simply changing update strength.

7 comments
AuthorsOpenReview ↗

Rebuttal [1/4]

Dear Reviewer JrXe,

Thank you for your time and for recognizing that treating majority-consistent and inconsistent responses differently is a useful way to combine distillation feedback with negative updates. We ran the requested experiments on Qwen3-1.7B; our responses are below.

[Weakness 1 & Question 2]: The experiments do not directly establish why TTPO remains reliable when pseudo-labels are wrong. The observation that most majority-inconsistent responses are incorrect does not show how much the method suppresses the remaining correct responses. Likewise, agreement between a trajectory's final answer and the teacher's supplied answer does not make wrong-answer conditioning harmless to intermediate reasoning. Controlled noise experiments and separate measurements for correct and incorrect pseudo-labels are needed to support this explanation. / Could the authors separate problems with correct and incorrect pseudo-labels and measure the effect of each training branch on both groups? Please include controlled pseudo-label noise levels, track whether correct minority responses lose probability, and compare wrong-answer-conditioned versus answer-free teachers on majority-consistent but incorrect trajectories. This would directly test the claim that the two branches handle wrong pseudo-labels relatively safely.

Thank you for this detailed suggestion. Our original argument was only an aggregate observation; we now measure each part directly (TTT on AIME26, step 100; GT used only for analysis or noise injection).

(i) Effect on correct-label vs. wrong-label problems. Of the 30 problems, the step-0 majority vote is correct on 5 and wrong on 25 (cells: Avg@12).

Variant (P / N)Correct-label (n = 5)Wrong-label (n = 25)
Base98.325.7
Positive-only (FKL / –)96.736.7
All-sample distillation (FKL / FKL)95.035.3
TTPO (FKL / GRPO)98.337.7

On the correct-label problems, TTPO keeps the base accuracy (98.3): the correct trajectories the model has already mastered are not suppressed, whereas positive-only and all-sample distillation drop to 96.7 and 95.0. On the wrong-label problems all three variants improve, and penalizing the disagreeing samples (+12.0) helps more than ignoring (+11.0) or distilling them (+9.6).

(ii) Controlled noise. Each step trains on 4 problems, k of which get a wrong pseudo-label (noise = k/4): problems whose vote is already wrong are chosen first and keep their vote; otherwise a correctly voted problem takes its largest wrong-answer cluster. The rest get the GT answer.

Wrong pseudo-labels0% (GT)25%50%75%natural (≈85%)100%
TTPO (AIME26)44.446.746.446.447.845.8
Correct among vote negatives0.74%1.75%1.10%1.25%1.29%1.37%

TTPO is noise-tolerant: from 25% to the natural rate (≈85%) accuracy stays at 46.4–47.8, and even at 100% it keeps 45.8 (+8.0 over the base, 37.8). GT labels (0%) are not better, as in Fig. 5: few rollouts match the GT on hard problems, starving both branches. Among the original vote's negatives (before noise injection), only 0.7–1.8% are correct at every noise level (training average).

(iii) Suppression of correct minority responses. First, mis-penalization is rare: on wrong-label problems only 1.3% of penalized rollouts are correct at step 0, and 0.6–2.5% during training (the other 21% in Fig. 1 mostly agree with the wrong label; only about 1% are correct). Second, on the fixed problem set (thinking):

CheckpointAvg@12Avg@12, base-solved (20)Problems covered (of 30)
Base37.856.720
2540.660.421
5045.668.319
7548.970.821
10047.870.021

Correct answers are reinforced, not suppressed: base-solved accuracy rises from 56.7 to 70.0 and coverage from 20 to 21 problems.

(iv) Wrong-answer-conditioned vs. answer-free teacher (distillation branch only; each trajectory's answer equals the privileged answer):

Distilled trajectoriesTeacher: no answerTeacher: own answer
all agreeing (Fig. 3)45.746.7
agreeing, wrong pseudo-label only46.446.1
agreeing, correct pseudo-label only43.143.9

Row 2 is the comparison you asked for: on majority-consistent but wrong trajectories, the wrong-answer-conditioned and answer-free teachers perform nearly identically (46.1 vs. 46.4). Row 3 is lower for two reasons: only about 15% of problems have a correct label, so most batches contribute few positives and starve the distillation branch (as in Fig. 5); and these are problems the base already solves (98.3 in (i)), so reinforcing them adds little.

We have added (i), (ii), and (iv) to Appendix D.5 and clarified that the distillation branch tolerates wrong pseudo-labels because their influence is small, not absent.

AuthorsOpenReview ↗

Rebuttal [2/4]

[Weakness 2(a) & Question 3]: The component ablations leave several alternative explanations open. Removing the negative-token mask increases the amount of penalization, so the performance drop does not establish that the mask identifies better tokens to penalize. [...] Random masks, update-strength-matched controls, [...] are missing from the evidence needed to distinguish these possibilities. / Could the authors compare the negative-token mask with random selection at the same retention rate, probability-only selection, entropy-only selection, and unmasked updates with a reduced RL weight? For positive weighting, entropy-only, teacher–student-disagreement-only, and uniform weighting with matched average weight would help determine whether the gains come from selecting informative tokens rather than simply changing update strength.

Thank you for pointing out this alternative explanation. We ran all the requested controls (Table 3 setting: OpenThoughts, Qwen3-1.7B; best checkpoint for every variant, as in Table 1). Keeping 50% of tokens roughly halves the update, so the strength-matched control removes the mask and halves λ to 0.05; "uniform × w̄ᵢ" keeps each trajectory's total Soft-OR weight but spreads it evenly over its tokens.

VariantAIME26HMMT26BRUMO25Avg
TTPO (Soft-OR weights; mask −log p·(1−Ĥ), 50% kept)46.531.654.744.3
Pos.: uniform (Table 3)43.330.652.842.2
Pos.: uniform × w̄ᵢ45.027.345.339.2
Pos.: entropy only45.830.350.642.2
Pos.: disagreement only45.629.052.842.5
Neg.: no mask (Table 3)45.429.550.041.6
Neg.: no mask, λ = 0.0544.430.150.841.8
Neg.: random 50%43.930.349.741.3
Neg.: likelihood only46.728.548.641.3
Neg.: entropy only46.431.151.142.9

Every control falls below TTPO. With the total weight unchanged, uniform × w̄ᵢ drops to 39.2; the strength-matched and random-mask controls stay at the no-mask level (41.3–41.8). The gain therefore comes from which tokens are selected, not from the update strength, and each single criterion falls short of the combination.

We have added this table to Appendix D.3 of the revised paper.

[Weakness 2(b)]: The large non-thinking gains could also come mainly from thinking-to-non-thinking distillation rather than the asymmetric objective. [...] branch ablations evaluated in non-thinking mode are missing from the evidence needed to distinguish these possibilities.

Thank you for raising this possibility. The non-thinking gain in the paper (39.8 on AIME26 for Qwen3-1.7B) is measured in the OpenThoughts setting, so in this same setting, with the same thinking teacher, we compare the update strategies of Fig. 3 in non-thinking mode (AIME26, step 100):

P / NBaseGRPO / GRPOGRPO / FKLGRPO / –– / FKL– / GRPOFKL / –FKL / FKLFKL / GRPO (TTPO)
Non-thinking8.88.67.58.611.70.010.09.739.8

Distillation alone does not explain the gain: the variants that distill from the same thinking teacher without TTPO's routing (FKL / –, FKL / FKL, – / FKL) stay at 9.7–11.7, close to the base, as does pure RL (8.6); only TTPO reaches 39.8. The thinking teacher is necessary but not sufficient: the asymmetric objective is what carries its signal into non-thinking mode.

AuthorsOpenReview ↗

Rebuttal [3/4]

[Weakness 3]: The main comparisons do not fully isolate the training objective from sampling budget and teacher configuration. TTPO uses 64 samples to construct its pseudo-labels, whereas OPSD-TTT uses a single deterministic thinking-mode output. Majority-based distillation controls appear in local ablations but are not evaluated across the full main setting. Reporting peak checkpoint performance without a clearly label-free selection rule or multi-run uncertainty also makes the smaller gains difficult to assess.

Thank you for this comment. We address the sampling budget and the checkpoint selection in turn.

(a) Sampling budget. For a fair comparison, all methods follow OPSD's default training configuration, with the same batch size of 32 trajectories per step, so the training cost per step is controlled.

MethodSamples × lengthProblems / stepTrajectories updated / problemTokens updated / trajectoryStepsTime / step
GRPO8 × 16k4816k50077.3 s
OPSD1 × 1,0243211,02410012.7 s
TTPO64 × 16k481,02410051.2 s

OpenThoughts, Qwen3-1.7B, 4×H20.

Compared with GRPO, TTPO is faster per step and needs 5× fewer steps, yet it is more accurate (Table 1: 40.1 vs. 35.7 for GRPO†; 39.8 vs. 33.0 at step 100). Compared with OPSD, it trades compute for labeled data, sampling enough for a reliable, noise-robust vote: it matches the GT-supervised OPSD† (39.7), and its up to 128× more generated tokens per step cost only about 4× the time.

Moreover, TTPO scales with its sampling budget, offering a smooth trade-off between compute and accuracy that can be matched to the available resources (AIME26, step 100, Avg@32):

Length × K16k×816k×1616k×3216k×648k×644k×641k×64
AIME2643.645.346.246.546.145.437.8
Answer extractable99.6%99.6%99.8%99.8%99.0%92.3%45.6%
Time / step (s)22.823.234.051.244.540.68.94

More samples make the vote more robust to noise, and lengths of 4k or more keep answers extractable; at 1k only 45.6% are, and accuracy falls to the base (37.8). When compute is limited, a small loss in accuracy buys a large saving: 16k×32 keeps 46.2 at two-thirds of the time, and 16k×16 keeps 45.3 at less than half.

OPSD-TTT uses a single greedy output to mirror OPSD, which needs no vote. The sampling budget does not explain TTPO's advantage: the majority-vote distillation controls in our response to Question 1 (a) use exactly TTPO's samples and still trail TTPO by 3.9–6.7 Avg.

(b) Checkpoint selection. We originally reported each method's peak checkpoint, following the reporting protocol of OPSD: RL-based methods are typically trained for more steps than distillation-based ones (500 vs. 100 here), so comparing peaks shows the best attainable performance of every method under its own training budget. This protocol comes from OPSD's OpenThoughts setting, but in the TTT setting we introduce, selecting checkpoints by test accuracy could be read as using ground-truth labels. We therefore additionally evaluate every method at a fixed step 100 in both settings (mean ± standard error over 4 evaluation runs), which requires no labels. TTPO remains the best method under both protocols, matching or exceeding the GT-supervised OPSD† without any labels:

ModelBaseGRPO†OPSD†TTPO
1.7B34.6 ± 1.233.0 ± 0.939.3 ± 0.839.8 ± 1.6
4B56.0 ± 1.255.2 ± 0.857.7 ± 1.258.1 ± 0.7
8B58.6 ± 1.258.9 ± 1.360.7 ± 0.761.3 ± 1.0

OpenThoughts (Table 1), step 100: Avg over AIME25 / HMMT25 / AIME26 / HMMT26 / BRUMO25.

ModelBaseTTRLOPSD-TTTTTPO
1.7B38.0 ± 1.335.5 ± 2.041.9 ± 1.544.8 ± 1.9
4B57.4 ± 1.357.8 ± 1.957.7 ± 1.960.5 ± 0.9
8B60.7 ± 1.662.1 ± 1.262.5 ± 1.764.8 ± 1.3

TTT (Table 2), step 100: Avg over AIME26 / HMMT26 / BRUMO25.

In the revised paper, we have added this cost comparison to Appendix D.4, and fixed step-100 results with standard errors for every benchmark to both main tables (Table 1 and Table 2).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Reviewers, AC, SAC, and PC,

We sincerely thank all reviewers for their insightful and constructive feedback, which has significantly strengthened our paper. Below, we summarize the progress of the rebuttal phase.


We appreciate that the reviewers recognized several key strengths of our work, including:

  • Well-motivated and simple-yet-effective design: TTPO tackles the important challenge of label-free adaptation by distilling majority-consistent responses and penalizing inconsistent ones, rather than forcing every response toward a potentially wrong pseudo-label. (Reviewer JrXe, GmTx, VPZK)
  • Consistent gains across scales and benchmarks: TTPO consistently improves models of three sizes, and its gains transfer to benchmarks beyond the adaptation problems. (Reviewer JrXe, GmTx, VPZK)
  • Comprehensive experiments and analyses: Extensive experiments and ablations on branch assignments, token selection, teacher conditioning and rollout choices validate each design choice; in particular, reversing the two branches substantially hurts performance. (Reviewer JrXe, GmTx, VPZK)

To address the reviewers' concerns, we conducted the following additional experiments and revised the paper accordingly, with all changes highlighted in blue:

  • Fixed-step evaluation: We report every method at a fixed step 100 with standard errors (Tables 1 and 2, Sec. 4.1), confirming that TTPO remains the best label-free method under a label-free checkpoint rule. (Reviewer JrXe, GmTx, VPZK)
  • Computational cost: We compare the training cost of GRPO, OPSD and TTPO and vary TTPO's sampling budget (Appendix D.4), showing that TTPO is cheaper and more accurate than GRPO and offers a flexible compute–accuracy trade-off. (Reviewer JrXe, GmTx, VPZK)
  • Robustness to wrong pseudo-labels: We separate correct- and wrong-label problems, inject controlled pseudo-label noise and compare teacher conditioning (Appendix D.5, with the analysis in Appendix E.2 clarified accordingly), verifying that TTPO stays robust even when most pseudo-labels are wrong. (Reviewer JrXe, GmTx)
  • Token selection controls: We compare our positive weighting and negative masking with single-criterion, random and update-strength-matched alternatives (Appendix D.3), validating that our token selection outperforms all alternatives on both branches. (Reviewer JrXe, VPZK)
  • Comparison with related methods: We compare TTPO with SRPO, CANON, U-OPSD and SCRL under the same training configuration (Appendix D.6), showing that TTPO outperforms all of them. (Reviewer JrXe, VPZK)

We sincerely thank all the reviewers again for their time and valuable feedback. Although some reviewers have not yet had the chance to respond to our rebuttal, we believe our additional experiments and clarifications address their concerns, and we would greatly appreciate the program committee (reviewers, AC, SAC, PC) taking them into consideration.

Sincerely,

TTPO Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.