- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TTPO adapts language models to unlabeled math problems using majority-vote answers. It distills agreeing rollouts from a thinking-mode teacher and penalizes disagreeing rollouts with a GRPO loss. Token weighting and masking refine the updates. Experiments on three Qwen3 sizes show gains over TTRL and OPSD-TTT, including transfer to other math benchmarks.
Critical strengths
(1) The asymmetric objective is simple and well motivated. Figure 3 supports the choice: distilling positive rollouts and penalizing negative ones works better than reversing the two updates. (2) Improvements hold across three model sizes. The cross-benchmark results in Figure 4 also provide evidence that the updates help beyond the problems used for adaptation.
Critical weaknesses
(1) The main components of TTPO, including majority-vote pseudo-labeling, on-policy self-distillation, sample-level routing between RL and distillation, and token-level selection, have been explored in prior work. In particular, SRPO, CANON, U-OPSD, and SCRL are closely related to the proposed framework. However, direct comparisons with these methods are missing, making it difficult to assess the benefits of the proposed asymmetric objective over existing approaches.
(2) The effectiveness of token-level selection is not sufficiently isolated. Although the ablations show benefits from positive-sample weighting and negative-sample masking, both rely on heuristic combinations of entropy, likelihood, and teacher–student divergence. Low-probability tokens under low-entropy distributions do not necessarily indicate reasoning errors. Comparisons with simpler alternatives, such as entropy-only, likelihood-only, and random selection, are needed to better justify these designs.
(3) TTPO requires 64 rollouts per problem, each up to 16,000 tokens, but lacks comparisons of computational costs. In addition, selecting the best checkpoint based on test-set accuracy may introduce label leakage. Fixed-step evaluation would help address this concern.
Questions for authors
See Weaknesses. If the authors can adequately address my concerns, I would be willing to raise my score.
4 comments
Rebuttal [2/3]
[Weakness 2]: The effectiveness of token-level selection is not sufficiently isolated. Although the ablations show benefits from positive-sample weighting and negative-sample masking, both rely on heuristic combinations of entropy, likelihood, and teacher–student divergence. Low-probability tokens under low-entropy distributions do not necessarily indicate reasoning errors. Comparisons with simpler alternatives, such as entropy-only, likelihood-only, and random selection, are needed to better justify these designs.
Thank you for this suggestion. We agree that such a token is not necessarily an error. The score does not need to identify errors exactly: it targets anomalous tokens, where the model confidently produced unlikely content, and spares high-probability tokens, which are typically locally correct steps, so that penalizing a disagreeing trajectory does not damage its correct reasoning. We compare it with the requested alternatives at the same retention rate, and also with update-strength-matched controls (Table 3 setting: OpenThoughts, Qwen3-1.7B; best checkpoint for every variant, as in Table 1). The strength-matched negative control removes the mask and halves λ to 0.05; "uniform × w̄ᵢ" keeps each trajectory's total Soft-OR weight but spreads it evenly over its tokens.
| Variant | AIME26 | HMMT26 | BRUMO25 | Avg |
|---|---|---|---|---|
| TTPO (Soft-OR weights; mask −log p·(1−Ĥ), 50% kept) | 46.5 | 31.6 | 54.7 | 44.3 |
| Pos.: uniform (Table 3) | 43.3 | 30.6 | 52.8 | 42.2 |
| Pos.: uniform × w̄ᵢ | 45.0 | 27.3 | 45.3 | 39.2 |
| Pos.: entropy only | 45.8 | 30.3 | 50.6 | 42.2 |
| Pos.: disagreement only | 45.6 | 29.0 | 52.8 | 42.5 |
| Neg.: no mask (Table 3) | 45.4 | 29.5 | 50.0 | 41.6 |
| Neg.: no mask, λ = 0.05 | 44.4 | 30.1 | 50.8 | 41.8 |
| Neg.: random 50% | 43.9 | 30.3 | 49.7 | 41.3 |
| Neg.: likelihood only | 46.7 | 28.5 | 48.6 | 41.3 |
| Neg.: entropy only | 46.4 | 31.1 | 51.1 | 42.9 |
Every alternative falls below TTPO. Random selection at the same retention rate stays at the no-mask level, and so does the strength-matched control, so the gain does not come from penalizing fewer tokens. Likelihood alone is no better than random and entropy alone helps only partly; the same holds for the single factors on the positive side. The combination is what selects the informative tokens.
We have added this comparison to Appendix D.3 of the revised paper.
Rebuttal [3/3]
[Weakness 3]: TTPO requires 64 rollouts per problem, each up to 16,000 tokens, but lacks comparisons of computational costs. In addition, selecting the best checkpoint based on test-set accuracy may introduce label leakage. Fixed-step evaluation would help address this concern.
Thank you for raising both points.
(a) Computational cost. For a fair comparison, all methods follow OPSD's default training configuration, with the same batch size of 32 trajectories per step, LoRA setting and learning rate, so the training cost per step is controlled.
| Method | Samples × length | Problems / step | Trajectories updated / problem | Tokens updated / trajectory | Steps | Time / step |
|---|---|---|---|---|---|---|
| GRPO | 8 × 16k | 4 | 8 | 16k | 500 | 77.3 s |
| OPSD | 1 × 1,024 | 32 | 1 | 1,024 | 100 | 12.7 s |
| TTPO | 64 × 16k | 4 | 8 | 1,024 | 100 | 51.2 s |
OpenThoughts, Qwen3-1.7B, 4×H20.
Compared with GRPO, TTPO is faster per step and needs 5× fewer steps, yet it is more accurate (Table 1: 40.1 vs. 35.7 for GRPO†; 39.8 vs. 33.0 at step 100). Compared with OPSD, it trades compute for labeled data, sampling enough for a reliable, noise-robust vote: it matches the GT-supervised OPSD† (39.7), and its up to 128× more generated tokens per step cost only about 4× the time.
At exactly the same cost (same samples, steps and hyperparameters), TTPO also outperforms majority-vote distillation of all samples (TTT, step 100):
| Method (64 × 16k samples, 8 updated / problem, 100 steps) | AIME26 | HMMT26 | BRUMO25 | Avg |
|---|---|---|---|---|
| Majority-vote distillation of all samples | 45.3 | 26.3 | 42.8 | 38.1 |
| TTPO | 47.8 | 33.6 | 53.1 | 44.8 |
Moreover, TTPO scales with its sampling budget, offering a smooth trade-off between compute and accuracy that can be matched to the available resources (AIME26, step 100, Avg@32):
| Length × K | 16k×8 | 16k×16 | 16k×32 | 16k×64 | 8k×64 | 4k×64 | 1k×64 |
|---|---|---|---|---|---|---|---|
| AIME26 | 43.6 | 45.3 | 46.2 | 46.5 | 46.1 | 45.4 | 37.8 |
| Answer extractable | 99.6% | 99.6% | 99.8% | 99.8% | 99.0% | 92.3% | 45.6% |
| Time / step (s) | 22.8 | 23.2 | 34.0 | 51.2 | 44.5 | 40.6 | 8.94 |
More samples make the vote more robust to noise, and lengths of 4k or more keep answers extractable; at 1k only 45.6% are, and accuracy falls to the base (37.8). When compute is limited, a small loss in accuracy buys a large saving: 16k×32 keeps 46.2 at two-thirds of the time, and 16k×16 keeps 45.3 at less than half.
(b) Checkpoint selection. We originally reported each method's peak checkpoint, following the reporting protocol of OPSD: RL-based methods are typically trained for more steps than distillation-based ones (500 vs. 100 here), so comparing peaks shows the best attainable performance of every method under its own training budget. This protocol comes from OPSD's OpenThoughts setting, but in the TTT setting we introduce, selecting checkpoints by test accuracy could be read as using ground-truth labels. We therefore additionally evaluate every method at a fixed step 100 in both settings (mean ± standard error over 4 evaluation runs), which requires no labels. TTPO remains the best method under both protocols, matching or exceeding the GT-supervised OPSD† without any labels:
| Model | Base | GRPO† | OPSD† | TTPO |
|---|---|---|---|---|
| 1.7B | 34.6 ± 1.2 | 33.0 ± 0.9 | 39.3 ± 0.8 | 39.8 ± 1.6 |
| 4B | 56.0 ± 1.2 | 55.2 ± 0.8 | 57.7 ± 1.2 | 58.1 ± 0.7 |
| 8B | 58.6 ± 1.2 | 58.9 ± 1.3 | 60.7 ± 0.7 | 61.3 ± 1.0 |
OpenThoughts (Table 1), step 100: Avg over AIME25 / HMMT25 / AIME26 / HMMT26 / BRUMO25.
| Model | Base | TTRL | OPSD-TTT | TTPO |
|---|---|---|---|---|
| 1.7B | 38.0 ± 1.3 | 35.5 ± 2.0 | 41.9 ± 1.5 | 44.8 ± 1.9 |
| 4B | 57.4 ± 1.3 | 57.8 ± 1.9 | 57.7 ± 1.9 | 60.5 ± 0.9 |
| 8B | 60.7 ± 1.6 | 62.1 ± 1.2 | 62.5 ± 1.7 | 64.8 ± 1.3 |
TTT (Table 2), step 100: Avg over AIME26 / HMMT26 / BRUMO25.
In the revised paper, we have added the training-cost table and the sampling-budget analysis to Appendix D.4, and fixed step-100 results with standard errors for every benchmark to both main tables (Table 1 and Table 2).
Thank you again for your thorough review and for your openness to reconsidering the score. We hope these results address your concerns, and we would be happy to answer any further questions.
Sincerely, TTPO Authors
Kindly Reminder
Dear Reviewer VPZK,
Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.
If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.
Sincerely,
TTPO Authors
Rebuttal [1/3]
Dear Reviewer VPZK,
Thank you for your time and for recognizing that the asymmetric objective is simple and well motivated (Figure 3), with improvements that hold across model sizes and benchmarks. All new experiments use Qwen3-1.7B unless stated otherwise. Our responses are below.
Thank you for raising this question. These methods are indeed related, but TTPO differs from each of them in its core design: which samples are distilled and which are penalized, what the teacher is conditioned on, and the token-level rules of each branch.
Direct comparison. We reproduce all four methods with their original objectives. Since SRPO requires GT, we compare in the OpenThoughts setting of Table 1 (SRPO†), and for fairness all methods follow the OPSD configuration (training data, LoRA, 100 steps × 32 trajectories, and best-checkpoint evaluation).
TTPO achieves the best average, including over the GT-based SRPO†. We attribute the gaps to the following differences:
We have added this comparison and discussion to Appendix D.6 of the revised paper.