- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SelectTTRL, a TTRL approach to reduce the computational cost by selecting useful questions before expensive response sampling and training. It introduces COS, a cosine similarity score based on a correctness related direction that is extracted from labeled examples and applies standard TTRL only to questions with intermediate scores. Experiments show that SelectTTRL generally maintains performance close to full data TTRL while reducing generated tokens by 18.38% and training-step time by 6.48% on average. Also, the paper provides a theoretical analysis explaining the relationship between COS and question-level accuracy. The main claim is that experiments across multiple reasoning benchmarks demonstrate that SelectTTRL substantially reduces test-time computation while maintaining or improving the performance of standard TTRL.
Critical strengths
-
Resource-Efficient TTRL Alternative: The method effectively demonstrates that test-time training performance can be maintained close to full-data TTRL while using fewer rollouts, without necessitating changes to the underlying training procedure.
-
Correlated Performance Signal: The extraction of a correctness direction from a separate dataset provides an innovative approach to scoring, demonstrating clear positive correlations with accuracy across several distinct reasoning tasks.
-
Empirical and Theoretical Grounding: The paper provides evidence via theoretical analysis, steering experiments, and statistical evaluations to demonstrate that the Correctness Optimization Score (COS) reliably captures question solvability.
Critical weaknesses
-
Under-Justified Selection Mechanics and Baselines: While COS correlates with question accuracy, the paper does not convincingly demonstrate that it identifies optimal training examples. The specific choice of filtering for the 25th–60th percentile interval lacks a rigorous justification. To isolate the value added by the correctness direction, the evaluation should include direct comparisons against standard intermediate-perplexity selection or alternative intermediate COS intervals under an identical training budget.
-
Unclear End-to-End Computational Efficiencies: The practical reduction in computational costs requires further clarification. Although the method filters out 65% of the data (selecting only 35% of questions), it yields a modest 6.48% average reduction in training step time. The paper does not fully account for the total training budget or the upfront costs associated with calibration, initial response generation, and scoring overhead.
-
Inconsistent Benchmark Trajectories: The downstream accuracy gains are highly inconsistent across different models and benchmarks, with several configurations underperforming compared to full-data TTRL. Furthermore, the reported improvements—particularly on smaller benchmarks like AIME—lack variance or uncertainty estimates across independent training runs, making the statistical reliability of the results difficult to assess.
-
Ambiguous Scoring Implementation and Optimization Rationale: The precise implementation details of the scoring procedure remain ambiguous, specifically whether the main experiments utilize a single rollout response or an average over multiple COS scores. Mechanistically, while the theoretical analysis supports a link between COS and correctness, it does not fully explain why filtering for this specific sub-population leads to superior or more stable reinforcement learning updates.
Questions for authors
Please address the following methodological and empirical clarifications:
-
Difficulty-Based Selection Control: How does your proposed 25th–60th percentile COS selection rule compare against standard data pruning baselines, such as selecting questions based on intermediate token perplexity or mid-tier model confidence, when restricted to the exact same sample and training budget?
-
End-to-End Wall-Clock Analysis: Given that selecting 35% of the questions only saves 6.48% in average training step time, could you provide a comprehensive breakdown of the wall-clock time? Please clarify whether this metric accounts for the pre-processing overhead of dataset calibration, initial response generation, and scoring.
-
Scoring Function Replication: Please clarify the exact execution of the scoring phase in the main experiments. Are the COS values calculated from a single sampled response per question, or are they averaged over multiple generated rollout paths?
-
Statistical Variance: Could you provide error bars or confidence intervals across multiple independent training seeds for the core benchmark evaluations? This would help clarify the stability of the method, particularly on high-variance benchmarks like AIME.
2 comments
Response to Reviewer zxc4 (2/2)
3. Implementation of COS scoring
Does the main experiment use a single response or average COS over multiple rollouts?
We use one greedily decoded response per question to compute COS and rank the target questions in the main experiments.
4. Independent training seeds and the gains on AIME
Can you provide uncertainty estimates across independent training seeds, particularly on AIME?
Our available GPU resources do not allow additional independent training runs during the rebuttal period.
Regarding the gains on AIME, we attribute much of the benefit of our selection rule to reducing erroneous updates. This makes the method particularly useful on difficult benchmarks, where incorrect consensus can be repeatedly reinforced, and explains the more modest gains on MATH-500. TTRL-Guard [1, Sec. 2.2] observes that recovery becomes increasingly unlikely once false consensus forms. This motivates us to choose a more cautious update interval, which we believe may help maintain stability during subsequent unsupervised training.
Thank you again for your careful reading. We will incorporate these clarifications and discussions into the paper.
[1] Lin et al. Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting. 2026. https://arxiv.org/abs/2605.19444v2
Response to Reviewer zxc4 (1/2)
Thank you for reading our paper. We will discuss your four questions candidly.
1. Comparison with difficulty-based selection controls
GPU limits prevent further baseline training during rebuttal. We evaluate unsupervised harmful-update risk and zero-advantage group rates.
We compare greedy COS 25%–60% with the central 35% selected by perplexity (Mid-PPL), mean token probability (Mid-AvgProb [1]), and negative mean token entropy (Mid-NegEntropy [2]). All methods retain equal numbers of questions and are evaluated on the same response pool.
Based on SCRL [3], we define an unsupervised harmful-update risk index R: positive advantage assigned to weak-consensus groups divided by total absolute advantage. Strong consensus requires at least 37.5% of votes and a lead over the runner-up exceeding 12.5 percentage points; other groups are weak-consensus. We report 100R and zero-advantage group percentages below.
Harmful-update risk (100R, lower is better)
Qwen3-8B
Qwen2.5-7B-Instruct
Zero-advantage groups (%)
Qwen3-8B
Qwen2.5-7B-Instruct
COS has lower risk than Mid-PPL, Mid-AvgProb, and Mid-NegEntropy in 9, 10, and 8 of the 12 settings, respectively. The interval aims to reduce repeated reinforcement of erroneous consensus while retaining useful group-relative learning signals.
2. Cost accounting and matched training budgets
Yes. Table 9 includes calibration, initial response generation, representation extraction, and scoring. Unfortunately, I accidentally deleted the detailed timing table while organizing files, so the component-wise breakdown is unavailable.
Full-data TTRL and SelectTTRL match training steps, questions per step, and responses per question. These schedules apply to both Qwen3-8B and Qwen2.5-7B-Instruct.
The 35% retention rate describes the question pool; matched steps keep savings dependent on tokens and per-step cost. DOTS [4], Feng et al. [5], and FTFT [6] also discuss matched-step comparisons.
Data are not always clean or worth training on, especially without supervision. Training on a subset can reduce wasted computation. For uncurated open-world data, cleaning before training makes these computational savings intuitive.
[1] Spiess et al. Calibration and Correctness of Language Models for Code. ICSE 2025. https://doi.org/10.1109/ICSE55347.2025.00040
[2] Malinin and Gales. Uncertainty Estimation in Autoregressive Structured Prediction. ICLR 2021. https://arxiv.org/abs/2002.07650
[3] Yan et al. What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time. 2026. https://arxiv.org/abs/2603.19880v2
[4] Sun et al. Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay. NeurIPS 2025, Sec. 5.1. https://arxiv.org/abs/2506.05316
[5] Feng et al. Automatic Document Selection for Efficient Encoder Pretraining. EMNLP 2022, Sec. 3.2. https://aclanthology.org/2022.emnlp-main.647/
[6] Du et al. FTFT: Efficient and Robust Fine-Tuning by Transferring Training Dynamics. COLING 2025, Sec. 5. https://aclanthology.org/2025.coling-main.86/