acceptodds
Under review as a conference paper at ICLR 2027

SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning

Abstract

Test-Time Reinforcement Learning (TTRL) enables large language models to adapt directly on unlabeled test problems by constructing pseudo-labels from multiple sampled responses. However, existing TTRL methods typically apply the same expensive sampling and optimization procedure to all test problems, despite substantial differences in their training utility. In this paper, we study whether such problems can be identified before costly test-time training, without access to ground-truth answers. We uncover a stable relationship between internal response representations and problem-level empirical accuracy. Specifically, we define COS as the cosine similarity between a model's response representation and a correctness-related direction constructed from an independent question-answering dataset. COS consistently correlates with problem solvability across different models, datasets, and experimental settings, and we further provide a theoretical analysis explaining this relationship. Building on this finding, we propose SelectTTRL, a representation-guided data selection method that ranks test problems using COS and applies TTRL only to a selected percentile interval. SelectTTRL leaves the standard TTRL pipeline unchanged for the selected problems, while avoiding unnecessary multi-response sampling and reinforcement learning updates on the remaining ones. Experiments across multiple reasoning benchmarks demonstrate that SelectTTRL substantially reduces test-time computation while maintaining or improving the performance of standard TTRL.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-time reinforcement learning, data selection, representation engineering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zxc4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SelectTTRL, a TTRL approach to reduce the computational cost by selecting useful questions before expensive response sampling and training. It introduces COS, a cosine similarity score based on a correctness related direction that is extracted from labeled examples and applies standard TTRL only to questions with intermediate scores. Experiments show that SelectTTRL generally maintains performance close to full data TTRL while reducing generated tokens by 18.38% and training-step time by 6.48% on average. Also, the paper provides a theoretical analysis explaining the relationship between COS and question-level accuracy. The main claim is that experiments across multiple reasoning benchmarks demonstrate that SelectTTRL substantially reduces test-time computation while maintaining or improving the performance of standard TTRL.

Critical strengths
  1. Resource-Efficient TTRL Alternative: The method effectively demonstrates that test-time training performance can be maintained close to full-data TTRL while using fewer rollouts, without necessitating changes to the underlying training procedure.

  2. Correlated Performance Signal: The extraction of a correctness direction from a separate dataset provides an innovative approach to scoring, demonstrating clear positive correlations with accuracy across several distinct reasoning tasks.

  3. Empirical and Theoretical Grounding: The paper provides evidence via theoretical analysis, steering experiments, and statistical evaluations to demonstrate that the Correctness Optimization Score (COS) reliably captures question solvability.

Critical weaknesses
  1. Under-Justified Selection Mechanics and Baselines: While COS correlates with question accuracy, the paper does not convincingly demonstrate that it identifies optimal training examples. The specific choice of filtering for the 25th–60th percentile interval lacks a rigorous justification. To isolate the value added by the correctness direction, the evaluation should include direct comparisons against standard intermediate-perplexity selection or alternative intermediate COS intervals under an identical training budget.

  2. Unclear End-to-End Computational Efficiencies: The practical reduction in computational costs requires further clarification. Although the method filters out 65% of the data (selecting only 35% of questions), it yields a modest 6.48% average reduction in training step time. The paper does not fully account for the total training budget or the upfront costs associated with calibration, initial response generation, and scoring overhead.

  3. Inconsistent Benchmark Trajectories: The downstream accuracy gains are highly inconsistent across different models and benchmarks, with several configurations underperforming compared to full-data TTRL. Furthermore, the reported improvements—particularly on smaller benchmarks like AIME—lack variance or uncertainty estimates across independent training runs, making the statistical reliability of the results difficult to assess.

  4. Ambiguous Scoring Implementation and Optimization Rationale: The precise implementation details of the scoring procedure remain ambiguous, specifically whether the main experiments utilize a single rollout response or an average over multiple COS scores. Mechanistically, while the theoretical analysis supports a link between COS and correctness, it does not fully explain why filtering for this specific sub-population leads to superior or more stable reinforcement learning updates.

Questions for authors

Please address the following methodological and empirical clarifications:

  1. Difficulty-Based Selection Control: How does your proposed 25th–60th percentile COS selection rule compare against standard data pruning baselines, such as selecting questions based on intermediate token perplexity or mid-tier model confidence, when restricted to the exact same sample and training budget?

  2. End-to-End Wall-Clock Analysis: Given that selecting 35% of the questions only saves 6.48% in average training step time, could you provide a comprehensive breakdown of the wall-clock time? Please clarify whether this metric accounts for the pre-processing overhead of dataset calibration, initial response generation, and scoring.

  3. Scoring Function Replication: Please clarify the exact execution of the scoring phase in the main experiments. Are the COS values calculated from a single sampled response per question, or are they averaged over multiple generated rollout paths?

  4. Statistical Variance: Could you provide error bars or confidence intervals across multiple independent training seeds for the core benchmark evaluations? This would help clarify the stability of the method, particularly on high-variance benchmarks like AIME.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zxc4 (1/2)

Thank you for reading our paper. We will discuss your four questions candidly.

1. Comparison with difficulty-based selection controls

How does COS compare with intermediate perplexity or confidence under matched budgets?

GPU limits prevent further baseline training during rebuttal. We evaluate unsupervised harmful-update risk and zero-advantage group rates.

We compare greedy COS 25%–60% with the central 35% selected by perplexity (Mid-PPL), mean token probability (Mid-AvgProb [1]), and negative mean token entropy (Mid-NegEntropy [2]). All methods retain equal numbers of questions and are evaluated on the same response pool.

Based on SCRL [3], we define an unsupervised harmful-update risk index RR: positive advantage assigned to weak-consensus groups divided by total absolute advantage. Strong consensus requires at least 37.5% of votes and a lead over the runner-up exceeding 12.5 percentage points; other groups are weak-consensus. We report 100R100R and zero-advantage group percentages below.

Harmful-update risk (100R100R, lower is better)

Qwen3-8B

MethodMATH-500GPQAMMLU-ProAIME24AIME25AMC23
Mid-PPL12.07917.30615.63522.03938.30315.216
Mid-AvgProb10.42917.16815.21216.38935.26415.405
Mid-NegEntropy10.84516.76715.86922.03937.74418.128
COS7.82613.9808.97134.19227.21620.659

Qwen2.5-7B-Instruct

MethodMATH-500GPQAMMLU-ProAIME24AIME25AMC23
Mid-PPL17.64323.88419.84637.88332.13025.326
Mid-AvgProb18.46225.46520.10437.69227.54225.184
Mid-NegEntropy16.21223.40019.70637.69227.54224.260
COS14.14625.44913.10929.65726.31524.743

Zero-advantage groups (%)

Qwen3-8B

MethodMATH-500GPQAMMLU-ProAIME24AIME25AMC23
Mid-PPL74.9%7.2%40.2%9.1%0.0%34.5%
Mid-AvgProb70.9%7.2%40.8%9.1%0.0%34.5%
Mid-NegEntropy71.4%7.2%40.8%9.1%0.0%34.5%
COS74.9%4.3%52.3%0.0%0.0%41.4%

Qwen2.5-7B-Instruct

MethodMATH-500GPQAMMLU-ProAIME24AIME25AMC23
Mid-PPL60.6%20.3%21.8%9.1%0.0%13.8%
Mid-AvgProb60.6%20.3%22.4%9.1%0.0%13.8%
Mid-NegEntropy61.1%21.7%21.3%9.1%0.0%10.3%
COS57.1%14.5%27.6%0.0%0.0%13.8%

COS has lower risk than Mid-PPL, Mid-AvgProb, and Mid-NegEntropy in 9, 10, and 8 of the 12 settings, respectively. The interval aims to reduce repeated reinforcement of erroneous consensus while retaining useful group-relative learning signals.

2. Cost accounting and matched training budgets

Does Table 9 include preprocessing, and why does retaining 35% of the questions yield a 6.48% training-time reduction?

Yes. Table 9 includes calibration, initial response generation, representation extraction, and scoring. Unfortunately, I accidentally deleted the detailed timing table while organizing files, so the component-wise breakdown is unavailable.

Full-data TTRL and SelectTTRL match training steps, questions per step, and responses per question. These schedules apply to both Qwen3-8B and Qwen2.5-7B-Instruct.

DatasetTTRL stepsSelectTTRL stepsQuestions/stepResponses/questionTraining responses per method
MATH-5007575163238,400
GPQA-Diamond5050163225,600
MMLU-Pro STEM12012083230,720
AIME2412012083230,720
AIME2512012083230,720
AMC2315015083238,400
Total635635——194,560

The 35% retention rate describes the question pool; matched steps keep savings dependent on tokens and per-step cost. DOTS [4], Feng et al. [5], and FTFT [6] also discuss matched-step comparisons.

Data are not always clean or worth training on, especially without supervision. Training on a subset can reduce wasted computation. For uncurated open-world data, cleaning before training makes these computational savings intuitive.

[1] Spiess et al. Calibration and Correctness of Language Models for Code. ICSE 2025. https://doi.org/10.1109/ICSE55347.2025.00040

[2] Malinin and Gales. Uncertainty Estimation in Autoregressive Structured Prediction. ICLR 2021. https://arxiv.org/abs/2002.07650

[3] Yan et al. What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time. 2026. https://arxiv.org/abs/2603.19880v2

[4] Sun et al. Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay. NeurIPS 2025, Sec. 5.1. https://arxiv.org/abs/2506.05316

[5] Feng et al. Automatic Document Selection for Efficient Encoder Pretraining. EMNLP 2022, Sec. 3.2. https://aclanthology.org/2022.emnlp-main.647/

[6] Du et al. FTFT: Efficient and Robust Fine-Tuning by Transferring Training Dynamics. COLING 2025, Sec. 5. https://aclanthology.org/2025.coling-main.86/

AuthorsOpenReview ↗

Response to Reviewer zxc4 (2/2)

3. Implementation of COS scoring

Does the main experiment use a single response or average COS over multiple rollouts?

We use one greedily decoded response per question to compute COS and rank the target questions in the main experiments.

4. Independent training seeds and the gains on AIME

Can you provide uncertainty estimates across independent training seeds, particularly on AIME?

Our available GPU resources do not allow additional independent training runs during the rebuttal period.

Regarding the gains on AIME, we attribute much of the benefit of our selection rule to reducing erroneous updates. This makes the method particularly useful on difficult benchmarks, where incorrect consensus can be repeatedly reinforced, and explains the more modest gains on MATH-500. TTRL-Guard [1, Sec. 2.2] observes that recovery becomes increasingly unlikely once false consensus forms. This motivates us to choose a more cautious update interval, which we believe may help maintain stability during subsequent unsupervised training.

Thank you again for your careful reading. We will incorporate these clarifications and discussions into the paper.

[1] Lin et al. Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting. 2026. https://arxiv.org/abs/2605.19444v2

Reviewer_QpmaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a metric called COS to measure the solvability of problems, which is then used to select high-value problems for SelectTTRL; the authors claim that this approach maintains a favorable balance between performance and efficiency.

Critical strengths
  1. A metric, denoted as "cos," is proposed to quantify the solvability of a given problem by the current model; this metric is subsequently used to select high-value problems for TTRL (Task-Transfer Reinforcement Learning).

  2. It is demonstrated, to a certain extent, that the model maintains an internal, cross-task applicable state representation that reflects the solvability of problems.

Critical weaknesses
  1. The core idea lacks rigor: Even if the cosine similarity (cos) metric could reflect the solvability of a problem by a model, it does not necessarily indicate the problem's training value; thus, the paper’s central thesis rests on an unverified hypothesis.

  2. The main experiments are seriously flawed: Performance on several datasets is significantly lower than that of the full-scale TTRL approach. Furthermore, the improvement observed on the AIME dataset amounts to a difference of only two problems, and the results lack any reporting of uncertainty. Additionally, the discussion regarding efficiency is notably inadequate.

  3. Key analyses and ablation studies are missing: There is no experimental evidence justifying the choice of the specific cosine similarity range used in "SelectTTRL."

  4. The writing and overall presentation of the paper are rather sloppy and perfunctory.

Questions for authors
  1. What is the rationale behind selecting the 25%–60% range for the COS metric?

  2. If the same median-range selection strategy were applied to simple metrics like "ppl" (people/users), how would the results compare to those obtained using the COS metric?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Qpma (1/2)

Thank you for these two core questions; we are happy to discuss both. However, your comments focus almost entirely on training, with virtually no discussion of our representations. Regardless of your experience with representation learning, I believe you can understand our findings and discuss with us the aspects you do not understand. As stated in our response to R1, training and representation questions are equally important.

You also describe our paper as difficult to understand and its writing as rough and perfunctory. We welcome specific examples so that we can explain these points and improve the manuscript.

1. Rationale for selecting the 25%–60% COS interval

What motivates the 25%–60% COS interval?

We first conducted preliminary training on MATH-500 and AIME24, exploring both extremes and several intermediate COS intervals, and selected 25%–60% empirically. This interval balances reliable pseudo-supervision with useful learning signals: it reduces repeated reinforcement of erroneous consensus while avoiding uniform-reward groups with zero relative advantages.

We attribute much of the benefit to reducing erroneous updates, explaining the larger gains on difficult benchmarks such as AIME and the more modest gains on MATH-500. TTRL-Guard [1, Sec. 2.2] observes that recovery becomes increasingly unlikely once false consensus forms. This motivates us to choose a more cautious update interval, which we believe may help maintain stability during subsequent unsupervised training.

Based on SCRL [2], we define an unsupervised harmful-update risk index RR: positive advantage assigned to weak-consensus groups divided by total absolute advantage. Weak consensus fails the criterion that the leading answer has at least 37.5% of votes and leads the runner-up by more than 12.5 percentage points. We report 100R100R and zero-advantage group percentages below.

We rank questions by greedy COS and compare five intervals, each retaining 35% of the questions.

Qwen3-8B

MetricCOS intervalMATH-500GPQAMMLU-ProAIME24AIME25AMC23
100R100R0%–35%18.36817.81818.03132.15032.76823.079
20%–55%8.13715.6729.96733.37126.13719.830
25%–60%7.82613.9808.97134.19227.21620.659
30%–65%9.15110.0248.60225.08027.37122.737
65%–100%6.7939.4707.99923.53023.35810.380
ZZ (%)0%–35%56.6%5.8%26.4%0.0%0.0%41.4%
20%–55%74.9%5.8%47.7%0.0%0.0%37.9%
25%–60%74.9%4.3%52.3%0.0%0.0%41.4%
30%–65%74.9%7.2%54.0%0.0%0.0%37.9%
65%–100%70.3%29.0%63.2%27.3%0.0%34.5%

Qwen2.5-7B-Instruct

MetricCOS intervalMATH-500GPQAMMLU-ProAIME24AIME25AMC23
100R100R0%–35%18.24723.12214.71229.41413.85124.884
20%–55%14.19425.95013.96529.21732.79124.840
25%–60%14.14625.44913.10929.65726.31524.743
30%–65%12.99425.60313.84832.96333.15921.690
65%–100%16.31222.74213.47638.49026.99623.913
ZZ (%)0%–35%44.0%26.1%18.4%0.0%0.0%6.9%
20%–55%52.6%17.4%28.7%0.0%0.0%17.2%
25%–60%57.1%14.5%27.6%0.0%0.0%13.8%
30%–65%58.3%14.5%28.2%9.1%0.0%13.8%
65%–100%61.7%37.7%57.5%9.1%0.0%34.5%

Relative to 0%–35%, the chosen interval lowers update risk in 8/12 settings; relative to 65%–100%, it lowers zero-advantage rates in 8/12 settings, with two ties.

[1] Lin et al. Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting. 2026. https://arxiv.org/abs/2605.19444v2

[2] Yan et al. What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time. 2026. https://arxiv.org/abs/2603.19880v2

AuthorsOpenReview ↗

Response to Reviewer Qpma (2/2)

2. Comparison with perplexity and confidence-based selection

How does COS compare with applying a middle-range selection strategy to perplexity?

We compare greedy COS 25%–60% with the central 35% selected by perplexity (Mid-PPL), mean token probability (Mid-AvgProb [1]), and negative mean token entropy (Mid-NegEntropy [2]). All methods retain the same number of questions, and we evaluate their selections on the same sampled-response pool.

Based on SCRL [3], we define an unsupervised harmful-update risk index RR: positive advantage assigned to weak-consensus groups divided by total absolute advantage. Weak consensus fails the criterion that the leading answer has at least 37.5% of votes and leads the runner-up by more than 12.5 percentage points. We report 100R100R and zero-advantage group percentages below.

Harmful-update risk (100R100R, lower is better)

Qwen3-8B

MethodMATH-500GPQAMMLU-ProAIME24AIME25AMC23
Mid-PPL12.07917.30615.63522.03938.30315.216
Mid-AvgProb10.42917.16815.21216.38935.26415.405
Mid-NegEntropy10.84516.76715.86922.03937.74418.128
COS7.82613.9808.97134.19227.21620.659

Qwen2.5-7B-Instruct

MethodMATH-500GPQAMMLU-ProAIME24AIME25AMC23
Mid-PPL17.64323.88419.84637.88332.13025.326
Mid-AvgProb18.46225.46520.10437.69227.54225.184
Mid-NegEntropy16.21223.40019.70637.69227.54224.260
COS14.14625.44913.10929.65726.31524.743

Zero-advantage groups (%)

Qwen3-8B

MethodMATH-500GPQAMMLU-ProAIME24AIME25AMC23
Mid-PPL74.9%7.2%40.2%9.1%0.0%34.5%
Mid-AvgProb70.9%7.2%40.8%9.1%0.0%34.5%
Mid-NegEntropy71.4%7.2%40.8%9.1%0.0%34.5%
COS74.9%4.3%52.3%0.0%0.0%41.4%

Qwen2.5-7B-Instruct

MethodMATH-500GPQAMMLU-ProAIME24AIME25AMC23
Mid-PPL60.6%20.3%21.8%9.1%0.0%13.8%
Mid-AvgProb60.6%20.3%22.4%9.1%0.0%13.8%
Mid-NegEntropy61.1%21.7%21.3%9.1%0.0%10.3%
COS57.1%14.5%27.6%0.0%0.0%13.8%

COS has lower risk than Mid-PPL, Mid-AvgProb, and Mid-NegEntropy in 9, 10, and 8 of the 12 settings, respectively. On Qwen2.5 MATH-500, COS reduces risk by 19.8% versus Mid-PPL while lowering zero-advantage groups from 60.6% to 57.1%. On Qwen3 MATH-500, the zero-advantage rate is identical, while risk decreases by 35.2%. These results connect COS selection to more reliable update signals while retaining learning opportunities.

[1] Spiess et al. Calibration and Correctness of Language Models for Code. ICSE 2025. https://doi.org/10.1109/ICSE55347.2025.00040

[2] Malinin and Gales. Uncertainty Estimation in Autoregressive Structured Prediction. ICLR 2021. https://arxiv.org/abs/2002.07650

[3] Yan et al. What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time. 2026. https://arxiv.org/abs/2603.19880v2

Reviewer_QpmaOpenReview ↗

Thank you for the authors' response.

I appreciate the additional experiments provided by the authors. However, the concerns I raised in the Weaknesses section were not addressed at all, and they were the main reason for my rating. I will therefore keep my score unchanged, and I hope the authors will substantially revise the paper.

Reviewer_P2syOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes selecting promising training problems before performing expensive test-time reinforcement learning. The authors construct a correctness-related direction from an independent labeled dataset, rank target problems by the cosine similarity between their response representations and this direction, and apply TTRL to approximately 35% of the problems. Evaluation still covers the full target set. The paper investigates this COS signal through correlation analyses, theoretical arguments, and activation interventions, and compares performance and cost on two Qwen models across six benchmarks. Some configurations reduce computation while approaching full-data TTRL performance, although the benefits are not consistent across tasks and evaluation metrics.

Critical strengths

The timing of the selection decision is appealing. It occurs before expensive multi-response sampling and parameter updates, while leaving the subsequent TTRL procedure unchanged. This makes the method relatively easy to integrate. Evaluating on the full target set also gives the performance comparison a clear interpretation.

The analysis of COS is reasonably thorough. In particular, computing COS and accuracy from disjoint sets of responses reduces concerns about reusing the same response to construct both quantities. Activation interventions and feature ablations provide additional evidence that the direction is behaviorally relevant, supporting the broader idea of using internal representations to allocate computation.

Critical weaknesses

My main uncertainty is the practical performance–cost trade-off. Retaining only 35% of the problems suggests substantial savings, but the appendix reports an aggregate reduction of 6.48% in training-step time. It is also unclear whether the full cost includes calibration and initial response generation. Meanwhile, accuracy decreases in some settings, and some highlighted gains come from very small test sets. Without repeated training runs and complete cost accounting, the practical advantage remains difficult to assess.

Second, predicting correctness does not directly establish that COS predicts training utility. The theory primarily supports the former, whereas the method depends on the latter. In particular, the evidence does not yet adequately explain why the fixed 25th–60th percentile interval is the right region to train on. This missing connection is my main reservation about the interpretation of the selection rule.

Questions for authors

Does the cost accounting in Table 9 include calibration, initial response generation, and representation extraction? What are the total sampling budgets, training steps, and numbers of independent training runs for full-data TTRL and SelectTTRL?

How were the 25th–60th percentile interval and readout layer chosen? Are there comparisons between different COS intervals with the same number of selected problems that directly demonstrate greater learning gains in the chosen interval? Such evidence would help establish whether COS is useful for selecting training examples, beyond predicting correctness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer P2sy (1/3)

Thank you very much for your questions and for reading our work carefully! We believe that both the TTRL training questions and the representation-related questions are equally important. Below, I will discuss each of the issues you raised in detail.

1. Cost accounting and matched training budgets

Does Table 9 include calibration and initial scoring costs, and how are the training budgets matched?

The cost accounting in Table 9 includes direction calibration, initial response generation, and representation extraction and scoring. Within each dataset, full-data TTRL and the subset-selection methods use the same number of training steps. The schedules below apply to both Qwen3-8B and Qwen2.5-7B-Instruct.

DatasetTTRL stepsSelectTTRL stepsQuestions/stepResponses/questionTraining responses per method
MATH-5007575163238,400
GPQA-Diamond5050163225,600
MMLU-Pro STEM12012083230,720
AIME2412012083230,720
AIME2512012083230,720
AMC2315015083238,400
Total635635——194,560
AuthorsOpenReview ↗

Response to Reviewer P2sy (2/3)

2. Rationale for selecting the 25%–60% COS interval

How was the 25%–60% interval selected, and how does it compare with other COS intervals of the same width?

We first conducted preliminary training experiments on MATH-500 and AIME24, exploring both extreme COS intervals and several intermediate intervals, and selected 25%–60% empirically. This interval balances the reliability of pseudo-supervision with the availability of learning signals: it reduces repeated reinforcement of erroneous consensus while avoiding questions whose sampled responses yield uniform rewards and therefore zero group-relative advantages.

We attribute much of our benefit to reducing erroneous updates, explaining the larger gains on difficult benchmarks such as AIME and the more modest gains on MATH-500. TTRL-Guard [2, Sec. 2.2] observes that once false consensus forms, recovery during subsequent TTRL becomes increasingly unlikely. This motivates us to choose a more cautious update interval, which we believe may help maintain stability during subsequent unsupervised training. SelectTTRL is therefore particularly suited to challenging datasets.

Based on the consensus-reliability criterion in SCRL [1], we designed a weak-consensus reinforcement metric, interpreted as an unsupervised harmful-update risk index. It measures the positive advantage assigned to weak-consensus groups, normalized by total absolute advantage. A group is classified as weak-consensus when it fails the SCRL criterion: the leading answer receives at least 37.5% of votes and exceeds the runner-up by more than 12.5 percentage points.

The tables below compare harmful-update risk (100R100R, lower is better) and zero-advantage groups (ZZ, %) across COS intervals. We rank questions by the COS of greedily generated responses and evaluate the resulting selections.

Qwen3-8B

MetricCOS intervalMATH-500GPQAMMLU-ProAIME24AIME25AMC23
100R100R0%–35%18.36817.81818.03132.15032.76823.079
20%–55%8.13715.6729.96733.37126.13719.830
25%–60%7.82613.9808.97134.19227.21620.659
30%–65%9.15110.0248.60225.08027.37122.737
65%–100%6.7939.4707.99923.53023.35810.380
ZZ (%)0%–35%56.6%5.8%26.4%0.0%0.0%41.4%
20%–55%74.9%5.8%47.7%0.0%0.0%37.9%
25%–60%74.9%4.3%52.3%0.0%0.0%41.4%
30%–65%74.9%7.2%54.0%0.0%0.0%37.9%
65%–100%70.3%29.0%63.2%27.3%0.0%34.5%

Qwen2.5-7B-Instruct

MetricCOS intervalMATH-500GPQAMMLU-ProAIME24AIME25AMC23
100R100R0%–35%18.24723.12214.71229.41413.85124.884
20%–55%14.19425.95013.96529.21732.79124.840
25%–60%14.14625.44913.10929.65726.31524.743
30%–65%12.99425.60313.84832.96333.15921.690
65%–100%16.31222.74213.47638.49026.99623.913
ZZ (%)0%–35%44.0%26.1%18.4%0.0%0.0%6.9%
20%–55%52.6%17.4%28.7%0.0%0.0%17.2%
25%–60%57.1%14.5%27.6%0.0%0.0%13.8%
30%–65%58.3%14.5%28.2%9.1%0.0%13.8%
65%–100%61.7%37.7%57.5%9.1%0.0%34.5%

Across 12 model–dataset settings, 25%–60% lowers risk versus 0%–35% in 8 settings and lowers zero-advantage rates versus 65%–100% in 8 settings, with two ties, supporting a balance between reliability and update coverage.

[1] Yan et al. What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time. 2026. https://arxiv.org/abs/2603.19880v2

[2] Lin et al. Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting. 2026. https://arxiv.org/abs/2605.19444v2

AuthorsOpenReview ↗

Response to Reviewer P2sy (3/3)

3. Selection of the readout layer

How was the readout layer selected?

We construct the CAA direction and read response representations at the same layer to avoid introducing additional factors through cross-layer use. We select this layer by scanning the model's layers and evaluating each direction on held-out examples from the source dataset.

For Qwen3-8B, we hold out questions from TruthfulQA MC1 for validation. At each layer, we extract activations at the answer-letter position and construct the direction as the mean difference between positive and negative examples. We evaluate projection ROC-AUC on the held-out examples. Layer 21 achieves the highest source-validation AUC, 0.9048, and is selected for both direction construction and readout. Notably, although layer 21 has the highest source-validation AUC, its COS–accuracy correlation on the target datasets is not the strongest. Our layer selection remains unsupervised with respect to the target datasets and depends only on the model and source dataset.

LayerAUCLayerAUCLayerAUC
00.5008130.5616260.8576
10.4824140.6040270.8464
20.4748150.5980280.8452
30.4928160.5620290.8380
40.5388170.6976300.8348
50.5332180.7440310.8332
60.5576190.7964320.8264
70.5484200.8596330.8204
80.5436210.9048340.8204
90.5404220.9012350.8380
100.5464230.8812360.8772
110.5472240.8692——
120.5588250.8664——

Thank you again for your careful reading. We will incorporate this discussion into the revised manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Additional Results: Full-Data TTRL with Approximately Three Times the Training Budget

Thank you for your questions about computational cost. We have recovered additional full-data TTRL results with approximately three times the training-step budget of SelectTTRL. We currently have these results available for Qwen2.5-7B-Instruct. Due to limited GPU resources, we cannot provide the corresponding Qwen3-8B results with a threefold TTRL training budget within the rebuttal period, and we sincerely apologize.

Unfortunately, I accidentally deleted the original training-time records when organizing the experiment files earlier, so we are unable to provide precise training-time measurements. We report the actual training steps and evaluation results below.

Training budgets

DatasetSelectTTRL stepsExtended full-data TTRL stepsSelectTTRL / extended steps
MATH-5007523032.61%
GPQA-Diamond5015033.33%
MMLU-Pro STEM12035034.29%
AIME2412036033.33%
AIME2512036033.33%
AMC2315045033.33%
Total6351,90033.42%

Performance comparison: Qwen2.5-7B-Instruct

We conducted these supplementary experiments using the training and evaluation configurations specified in the paper, extending full-data TTRL to approximately three times the training steps. The tables include the original Table 2 results for comparison. All scores are accuracy (%); Maj.@16 denotes Majority@16.

MethodMATH-500GPQA-DiamondMMLU-Pro STEM
GreedyMean1Maj.@16GreedyMean1Maj.@16GreedyMean1Maj.@16
Matched-TTRL78.8078.3482.0035.3535.0435.8664.6665.3169.28
Random79.8077.6880.8036.6736.8140.4064.4663.4567.27
A-PPL78.4078.0482.4032.8335.8637.8864.6662.7966.47
SelectTTRL79.6078.6183.0036.8735.6138.8962.6564.2268.47
Full-data TTRL (~3x steps)81.0080.3881.2036.3638.0137.8866.0665.8867.07
MethodAIME24AIME25AMC23
GreedyMean1Maj.@16GreedyMean1Maj.@16GreedyMean1Maj.@16
Matched-TTRL23.3321.0423.3316.6716.4616.6751.8150.2353.01
Random16.6715.2120.006.678.9613.3345.7846.0851.81
A-PPL6.6712.9220.006.679.1716.6742.1744.9554.22
SelectTTRL16.6718.3323.3313.3312.0813.3348.1948.2753.01
Full-data TTRL (~3x steps)20.0020.0020.0020.0020.0020.0051.8151.7351.81

Additional full-data TTRL training does not consistently improve performance. On AIME24, Greedy, Mean1, and Majority@16 decrease from 23.33%, 21.04%, and 23.33% to 20.00%, 20.00%, and 20.00%, respectively. Majority@16 also decreases on MATH-500 and MMLU-Pro STEM. We believe these observations are consistent with the risk of accumulating erroneous updates during unsupervised training: once incorrect consensus is reinforced, further training can perpetuate it rather than correct it. Reducing exposure to such updates is a central motivation and advantage of our data-selection approach.

For every dataset, SelectTTRL uses less than 35% of the extended baseline's training steps. Our measured costs are already lower than full-data TTRL at matched steps. We therefore consider it highly likely that full-data TTRL with approximately three times our training-step budget takes more than three times our actual training time. These results demonstrate what we mean by efficient test-time reinforcement learning.

We will incorporate these discussions and supplementary experiments into the paper as soon as possible.

In any case, we thank the Area Chair and every reviewer for the time and effort you have devoted to helping improve this work !

AuthorsOpenReview ↗

Summary of Manuscript Revisions

We have revised the manuscript to clarify our methodology, strengthen the theoretical analysis, and incorporate the additional experiments discussed during rebuttal. The revisions are marked in red for ease of reference. The main changes are:

  1. Section 3 (Method): We clarify the two aspects underlying our data-selection approach—its relationship with correctness and its implications for training utility—and introduce the harmful-update index used in our analysis.

  2. Section 4 (Theory): We extend the analysis to connect ideal COS-based correctness ordering with harmful-update risk and zero-advantage groups. The main conclusions appear in the main text, with detailed derivations in the appendix.

  3. Section 5 and the appendix (Experiments): We add full-data TTRL results with approximately three times the training-step budget, expand the discussion of computational efficiency, and report comparisons across COS selection intervals and against perplexity- and confidence-based selection methods.

We have also streamlined the presentation and reorganized the tables to make these findings easier to follow. We sincerely thank the Area Chair and all reviewers for their time and effort in helping us improve this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.