acceptodds
Under review as a conference paper at ICLR 2027

Sequential Consensus Policy Optimization for Test-time Reinforcement Learning

Abstract

Test-time policy optimization improves language models by sampling multiple rollouts per prompt and using their majority-vote answer as a pseudo-label, without any ground-truth supervision. This procedure is expensive since allocating a fixed rollout budget to every prompt wastes compute on prompts whose answer has already settled. Adaptive stopping rules help, but existing methods decide using only the current visit, discarding what earlier visits to the same prompt revealed about its answer distribution. We introduce Sequential Consensus Policy Optimization (SCPO), which maintains a Dirichlet posterior over each prompt's answer distribution, warm-started by an exponential moving average of past answer counts, and stops sampling once the posterior probability that some answer is the mode exceeds a certified threshold. We establish guarantees for adaptive stopping and gap-dependent rollout complexity, and characterize when the EMA-based memory remains accurate and reduces the rollout budget. Across Qwen2.5-Math-1.5B and Llama-3.1-8B-Instruct on MATH-500, AIME 2024, and LiveCodeBench, SCPO cuts rollouts per prompt by up to 71% and generated tokens by up to 66% relative to TTRL, while maintaining competitive accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time training, reinforcement learning, LLM efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dVKtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The manuscript considers the problem of test-time reinforcement learning, which refers to the scenario that the model has a test set, and sufficient compute, but does not have the labels for the test set, and wants to adapt to the specific test set in order for a better performance. The manuscript focuses on a majority-voting-based methodology that uses the majority answer in a group as a pseudo-label to do training, like TTRL. The authors introduce Sequential Consensus Policy Optimization (SCPO), a new algorithm for this problem. The main intuition is, for problems that the model already has a consistent answer in earlier training epochs, there is less meaning to generate more rollouts for this epoch. In this way, some computation can be saved. Technically, this is implemented by maintaining a Dirichlet posterior over each prompt's answer distribution and minimize the rollout count once the answer distribution has a clear mode. The authors provide some theoretical analysis about the difference between the proposed method and TTRL (i.e., the vanilla algorithm without reducing the rollout count). Empirically, the authors were able to show that using Qwen2.5-Math-1.5B, Llama-3.1-8B-Instruct and on math and coding benchmarks, SCPO is able to achieve a similar performance with previous methods like TTRL, but it saves a significant amount of computation measured by generated token count.

Critical strengths
  • The manuscript's writing is excellent. This manuscript did a good job in describing the problem setting, important related work, and the merits of its own work. The presentation is understandable and pretty clear.
  • The results are believable and as far as I can check, sound. I am convinced that the method saves a good amount of computation measured in tokens compared to prior work.
Critical weaknesses
  • Some questions about the evaluation remains unclear from the manuscript.
    • (Major) The main experiments align different methods' budget using the number of epochs for training. It seems unclear why this quantity is important. Naturally, when we consider computation, we might think of GPU time as a direct measure of budget.
    • (Minor) The tables in the manuscript show error bars, but it is unclear where they come from.
    • (Minor) The models used are relatively old.
  • (Major) As a result, I cannot identify the practical value for this work.
    • When we align different methods in GPU time, it does not seem that SCPO brings about a saving. In fact, the results in Table 4 seem to suggest the opposite.
Questions for authors
  • Q1: Feel free to respond to the weaknesses raised.
  • Q2: When answering Q1, can you also provide an example where in practice, SCPO can benefit?
8 comments
AuthorsOpenReview ↗

Response to Reviewer dVKt (Part 1)

We thank the reviewer for the careful review. We respond to each point below. Here we use W/Q for the weaknesses/questions and R for our responses.

W1.1: Why aligning the number of epochs is important?

R: In test-time reinforcement learning (TTRL, Zuo et al., 2025; OptPO, Wang et al., 2025b), methods are typically compared at a matched number of epochs. This ensures that every method revisits each prompt the same number of times and takes the same number of gradient steps. However, to address the reviewer's concern about compute budget, we show below that SCPO also matches or beats TTRL when evaluated under wall-clock GPU time.

W2: The practical value for this work: GPU time of SCPO.

R: We agree that GPU time matters. Our additional experiments show that under the same number of epochs, with a large extra rollout batch size (EBS), SCPO's wall-clock time matches or improves on the baselines. In Table 4 (in the manuscript), we already showed that at EBS=8, SCPO reduces runtime by 9.5% compared with OptPO. Besides Table 4, we ran SCPO with EBS=16 with Qwen2.5-Math-1.5B on MATH-500 and broke down wall-clock time into three main components: rollout generation, Monte Carlo stopping checks, and policy optimization (loss, backpropagation, and parameter updates):

MethodMean@16 ↑R/P ↓Tokens ↓Avg. Generation time (s/step)Avg. Stopping check time (s/step)Avg. Optimization time (s/step)Runtime ↓
TTRL72.564.0216.7M47.20.043.54h30m
SCPO (EBS=8)72.834.6122.6M54.21.443.64h55m
SCPO (EBS=16)72.735.0126.2M46.01.341.84h33m

At EBS=8, SCPO's overhead comes from rollout generation (54.2 s/step, about 1.15× TTRL). This is a batching effect: TTRL generates all its rollouts in one large batch that vLLM handles efficiently, whereas SCPO generates in sequential rounds with a stopping check in between (Section 5.3 in the manuscript). Raising EBS to 16 closes most of the gap: SCPO uses 41.8% fewer tokens than TTRL with nearly the same Mean@16 and almost the same wall-clock time (4h33m vs. 4h30m).

We also conducted experiments with Llama3.1-8B-Instruct on AIME 2024:

MethodMean@16 ↑R/P ↓Tokens ↓Avg. Generation time (s/step)Avg. Optimization time (s/step)Runtime ↓
TTRL8.764.0106.2M41.952.16h31m
SCPO (EBS=16)8.226.552.2M54.339.36h31m

and Qwen3.5-2B on MMLU College Physics:

MethodMean@16 ↑R/P ↓Tokens ↓Avg. Generation time (s/step)Avg. Optimization time (s/step)Runtime ↓
TTRL91.064.022.4M22.816.056m21s
SCPO (EBS=8)91.832.111.5M15.516.349m05s

In both settings, SCPO uses about half the tokens of TTRL and its runtime is equal on Llama and about 13% faster on Qwen3.5-2B. These results suggest that SCPO can save tokens substantially with comparable or lower runtime.

W1.2: The tables in the manuscript show error bars, but it is unclear where they come from.

R: To obtain those error bars, we ran each experiment 3 times and report the mean and standard deviation across runs in the tables.

W1.3: The models used are relatively old.

R: Regarding latest LLMs, we experimented with Qwen3.5-2B on MMLU College Physics using minimum 32 rollouts per prompt with an extra rollout batch size EBS=8, and report the results in the below table (token savings are relative to TTRL). The results show that SCPO saves token by 48.7% and runtime by 12.9%, while Mean@16 and Pass@16 are comparable or higher than baselines:

MethodMean@16 ↑Pass@16 ↑R/P ↓Num. tokens ↓Token saving ↑Time ↓
Base83.098.0----
TTRL91.096.164.022.4M0.0%56m 21s
OptPO (EBS=8)89.897.145.617.4M22.1%74m 53s
SCPO (EBS=8)91.897.132.111.5M48.7%49m 05s
AuthorsOpenReview ↗

Response to Reviewer dVKt (Part 2)

Q2: Practical examples where SCPO can benefit

R: SCPO may not reduce wall-clock inference time, but it generates substantially fewer tokens, which matters when test-time computation is budget-limited.

For example, under a fixed inference-time token budget, the tokens saved on problems with clear consensus can be spent on other problems. It could also lower costs in API-based services that support test-time adaptation and charge for generated tokens.

Because rollouts are typically generated in parallel, using fewer of them can also lower KV-cache memory demand. In memory-constrained settings, where limited GPU memory caps the batch size or forces KV-cache recomputation, SCPO may ease memory pressure and improve generation efficiency.

SCPO can reduce GPU energy consumption even when wall-clock runtime is comparable. For Llama-3.1-8B-Instruct on AIME 2024, SCPO (EBS=16) uses about 8% less GPU energy than TTRL (6.12 vs. 6.65 kWh), estimated from all GPUs' power measurements recorded in WandB. A likely reason is that SCPO spends less time in the policy optimization phase, which consumes more power (average 1,136 W, versus 875 W during rollout generation).

I thank the authors for the detailed response. I have gone through it. I appreciate the additional experiments that align the GPU time, as well as those with newer models like Qwen3.5-2B. I think the response sufficiently addressed my questions about the evaluation (i.e., the first weakness in my review).

When aligning the GPU time, what I am reading from the rebuttal experiments (which understandably don't have error bars), as well as the main paper's ones (where error bars do overlap), SCPO does not seem to improve the performance metric (I think I should be looking at Mean@16) statistically significantly. Thus, the argument for the practical benefit of SCPO is really about cost, and potential of "easing memory pressure and improving generation efficiency" that the authors mentioned in the response to Q2. I think this improvement is real and of value. But I also think that this improvement, within a relatively small scope (test-time reinforcement learning is not really something that people often do when evaluating models in practice), is incremental (no significant performance gain).

Overall, I think this is a well-executed study, and I would be happy to recommend acceptance at another venue, e.g., TMLR. For ICLR, I tend to still recommend weak rejection.

Reviewer_3jhVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Test-time reinforcement learning (TTRL) improves language models without ground-truth labels by sampling multiple responses per prompt and using majority voting to construct pseudo-labels, but a fixed rollout budget wastes computation on prompts whose answers are already clear. This paper proposes Sequential Consensus Policy Optimization (SCPO), which uses a Dirichlet posterior to estimate the probability that an answer is the most likely one and stops sampling when the estimated winner confidence exceeds a statistically corrected threshold. It further introduces a prompt-specific EMA prior to reuse answer statistics across training epochs. The theoretical analysis establishes finite-sample Monte Carlo certification, gap-dependent stopping guarantees, bounds on incorrect early crossings, and tracking guarantees for historical information under policy drift. Experiments on mathematical reasoning and code-generation benchmarks show reduced rollout and token consumption while largely preserving final model performance.

Critical strengths
  1. A principled approach to adaptive rollout allocation. SCPO combines posterior-based stopping with an explicit correction for Monte Carlo estimation error. The theoretical analysis also distinguishes confidence in the posterior winner from the probability of selecting the true mode, which is an important distinction when majority-vote answers serve as pseudo-labels.
  2. Consistent reductions in rollout and token consumption. Experiments across two models and multiple reasoning and coding benchmarks show that SCPO can reduce the number of rollouts and generated tokens while maintaining competitive performance. The use of cross-epoch information is also a reasonable extension of existing adaptive stopping methods, particularly when the answer distribution for a prompt remains stable across visits.
Critical weaknesses
  1. Reduced token consumption does not always lead to faster training. On MATH-500 with Qwen2.5-Math-1.5B, SCPO reduces generated tokens by 43.4% compared with TTRL, but takes 4 hours and 55 minutes instead of 4 hours and 30 minutes. This suggests that the overhead of checking the stopping condition and the impact of adaptive sampling on batch efficiency can outweigh the savings from generating fewer tokens. Therefore, the practical benefit of SCPO remains unclear when wall-clock training time is the main concern.
  2. Winner confidence does not necessarily indicate answer correctness. SCPO estimates how likely an answer is to be the most probable answer under the model's current output distribution. However, a model can be highly confident in an incorrect answer, and TTRL may further reinforce such errors through majority-vote rewards. In addition, the method assumes that answers can be represented as categories in a finite set. This is relatively straightforward for multiple-choice questions or normalized mathematical answers, but less clear for open-ended generation and code, where defining and grouping answer categories can be difficult.
Questions for authors
  1. Wall-clock efficiency: Why does SCPO take longer than TTRL on MATH-500 despite reducing generated tokens by 43.4%? Can the authors provide a breakdown of the time spent on rollout generation, stopping checks, and batch scheduling, and evaluate whether different batch sizes can translate token savings into actual wall-clock speedups?
  2. Winner confidence and pseudo-label correctness: Can the authors use benchmarks with ground-truth answers to measure pseudo-label accuracy at different winner-confidence levels, particularly for high-confidence but incorrect answers? Can they also analyze how these errors affect the final model performance? Since the method relies on a categorical representation of answers, it would also be useful to evaluate how answer normalization and changes in the answer distribution across epochs affect the reliability of the stopping rule and the benefits of EMA.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 3jhV (Part 1)

We thank the reviewer for the constructive review and the detailed questions. We respond to each point below. Here we use W/Q for the weaknesses/questions and R for our responses.

W1 & Q1: Reduced token consumption does not always lead to faster training. Breakdown SCPO wall-clock time, and evaluate with different batch sizes.

R: We agree that fewer tokens do not automatically mean faster training, and that batch size matters here. With a sufficiently large extra rollout batch size (EBS), our additional experiments show that SCPO's wall-clock time matches or improves on the baselines. Besides Table 4 (in the manuscript), we ran SCPO with EBS=16 using Qwen2.5-Math-1.5B on MATH-500 and broke down wall-clock time into three main components: rollout generation, Monte Carlo stopping checks, and policy optimization (loss, backpropagation, and parameter updates):

MethodMean@16 ↑R/P ↓Tokens ↓Avg. Generation time (s/step)Avg. Stopping check time (s/step)Avg. Optimization time (s/step)Runtime ↓
TTRL72.564.0216.7M47.20.043.54h30m
SCPO (EBS=8)72.834.6122.6M54.21.443.64h55m
SCPO (EBS=16)72.735.0126.2M46.01.341.84h33m

At EBS=8, SCPO's overhead comes from rollout generation (54.2 s/step, about 1.15× TTRL), not from the stopping checks (1.3-1.4 s/step) or batch scheduling (about 17s over the whole run). This is mainly a batching effect. TTRL generates all its rollouts in one large batch that inference engines like vLLM handle efficiently, whereas SCPO generates in sequential rounds with a stopping check in between, so it uses fewer rollouts but batches them less efficiently (see Section 5.3 in the manuscript).

A larger EBS means more rollouts between checks, so R/P and token count increase slightly, as noted in Section 5.3. But it also means fewer, larger batches, which reduces generation time. Raising EBS to 16, SCPO still uses 41.8% fewer tokens than TTRL with nearly the same Mean@16, and wall-clock time is almost the same (4h33m vs. 4h30m). So if wall-clock time matters more than token efficiency, we can simply use a larger EBS.

We also conducted experiments with Llama3.1-8B-Instruct on AIME 2024:

MethodMean@16 ↑R/P ↓Tokens ↓Avg. Generation time (s/step)Avg. Optimization time (s/step)Runtime ↓
TTRL8.764.0106.2M41.952.16h31m
SCPO (EBS=16)8.226.552.2M54.339.36h31m

and Qwen3.5-2B on MMLU College Physics:

MethodMean@16 ↑R/P ↓Tokens ↓Avg. Generation time (s/step)Avg. Optimization time (s/step)Runtime ↓
TTRL91.064.022.4M22.816.056m21s
SCPO (EBS=8)91.832.111.5M15.516.349m05s

In both settings, SCPO uses about half the tokens of TTRL and its runtime is equal on Llama and about 13% faster on Qwen3.5-2B. These results suggest that SCPO can save tokens substantially with comparable or lower runtime.

AuthorsOpenReview ↗

Response to Reviewer 3jhV (Part 2)

W2.1 & Q2.1: Pseudo-label accuracy vs. winner confidence, and the effect of wrong labels

R: The reviewer is correct that winner confidence does not imply answer correctness. That is one of TTRL's two main weaknesses. SCPO addresses the remaining: it reduces the number of rollouts and tokens used by TTRL rather than increasing answer correctness, as stated in the Limitations section.

Along with the results in Figure 3a of the manuscript where SCPO shows better pseudo-label accuracy than TTRL and OptPO throughout training, we do additional analyses with SCPO running with Qwen2.5-Math-1.5B on MATH-500.

Firstly, we fixed the winner-confidence threshold (stopping threshold) at 0.60.6 throughout training, recorded the winner confidence q^L\hat q_L at every prompt visit (480 prompts x 10 epochs), split them by q^L\hat q_L intervals and measured pseudo-label accuracy in each group. The table below shows that pseudo-label accuracy increases monotonically with winner confidence q^L\hat q_L:

Winner confidence q^L\hat{q}_L# VisitsPseudo-label accuracy
q^L<0.6\hat{q}_L < 0.625819.8%
0.6≤q^L<0.80.6 \leq \hat{q}_L < 0.854538.7%
0.8≤q^L<0.90.8 \leq \hat{q}_L < 0.917552.0%
0.9≤q^L<0.950.9 \leq \hat{q}_L < 0.9514355.2%
0.95≤q^L≤10.95 \leq \hat{q}_L \leq 13,67987.6%
Total4,80076.1%

On difficult questions where no answers reach the winner confidence threshold (the first row), SCPO samples all 64 rollouts, so its pseudo-label is the same as the 64-vote majority. SCPO stops early only when a consensus has formed, and does not degrade pseudo-label accuracy.

Pseudo-label correctness generally increases with confidence, but some high-confidence leaders are incorrect: 456 (12.4%) of the 3,679 visits with q^L≥0.95\hat q_L \geq 0.95 selected a wrong pseudo-label.

Secondly, to see how wrong pseudo-labels affect the final model, we partitioned the 480 questions visited in the first epoch by their status at their first training visit (initial pseudo-label correctness and whether q^L≥0.9\hat q_L \geq 0.9). We report the number of questions in each group and group-wise Mean@16 before and after training:

GroupNum. of QuestionsBase model Mean@16Final model Mean@16Mean@16 Change
Incorrect, q^L≥0.9\hat{q}_L \geq 0.92136.948.2+11.3 pp
Incorrect, q^L<0.9\hat{q}_L < 0.91059.223.0+13.8 pp
Correct, q^L≥0.9\hat{q}_L \geq 0.923945.496.4+51.0 pp
Correct, q^L<0.9\hat{q}_L < 0.911525.073.5+48.5 pp

Starting from an incorrect pseudo-label slows down improvement on that question. Initially correct questions gained +48.5 to +51.0 pp, whereas initially incorrect questions gained +11 to +14 pp, a gap of roughly 37 pp. The initially incorrect questions still improve due to two contributing factors: First, all 500 prompts update a single shared policy, so what the model learned from the ~75% initially correct prompts may also benefit harder ones. Second, as the policy improves over epochs, its outputs on difficult prompts shift, so correct answers can begin to win the vote and displace the earlier incorrect pseudo-labels. These results show that a high-confidence but wrong pseudo-label slows learning on a prompt but does not lock the model into it.

AuthorsOpenReview ↗

Response to Reviewer 3jhV (Part 3)

W2.2: Concern on answer extraction

R: Answer normalization matters in math, where \frac{3}{2}, 3/2, and 1.5 are the same value. Without normalization, they would be counted as separate groups with noisy counts. The effect on stopping can go either way: the leader's votes get split across variants, which can delay stopping, but the competing answers get split too, which can make stopping earlier.

For code, answers are not compared syntactically. As detailed in Appendix F, on LiveCodeBench-v6 rollouts are executed against the problem's test cases, and an answer category v∈Vv \in \mathcal{V} is defined by the tuple of execution outputs. Programs with identical output profiles fall into the same discrete category, a common approach in code consensus (e.g., AlphaCode). For open-ended tasks, we agree that it is difficult to handle. As noted in Section 6, text without verifiable execution or canonical forms (e.g., essay generation) would require semantic clustering (e.g., embedding similarity) to form discrete equivalence classes.

Besides, our implementation also supports dynamic categorization. The set V\mathcal{V} is maintained as an answer dictionary, and any newly generated answer is added with the initial prior parameter α0\alpha_0. Unseen answers have count 0, so they cannot become the leader and trigger stopping, since stopping requires q^L>0.5\hat{q}_L > 0.5.

Q2.2: Evaluate how changes in the answer distribution across epochs affect the reliability of the stopping rule and the benefits of EMA.

R: Thank you for an insightful question. We provide a heuristic argument for how the stopping rule and EMA behave when the answer distribution shifts across epochs. We consider three cases:

  • (i) Stable consensus (large empirical vote gap Δ^min⁡\hat{\Delta}_{\min}, no shift): When earlier visits consistently favor the mode, the prior accumulates a parameter lead (αj⋆(r)−αj(r)≫0\alpha_{j^{\star}}^{(r)} - \alpha_j^{(r)} \gg 0). This statistical head start allows SCPO to certify stopping with fewer rollouts, which directly leads to the token savings reported in Table 2 (of the manuscript) while maintaining accuracy.

  • (ii) Contested mode (small Δ^min⁡\hat{\Delta}_{\min}): When the policy is transitioning or undecided, past votes are split. As shown in Figure 1a, prompts whose leader flips naturally begin with Δ^min⁡≈0\hat{\Delta}_{\min} \approx 0. The prior assigns balanced pseudo-counts (αj1≈αj2\alpha_{j_1} \approx \alpha_{j_2}), so it adds near-zero directional bias and prevents premature stopping.

  • (iii) Confident history + distribution shift: If earlier epochs favored an answer (even an incorrect one) but policy updates shift the distribution toward a new mode, lock-in is mathematically prevented. In particular, total historical mass is bounded at Mr<Nmin⁡M_r < N_{\min} (line 300 in the PDF), so the mandatory prefix of Nmin⁡N_{\min} fresh rollouts always outweighs past memory. Moreover, when fresh rollouts disagree with the prior, winner confidence falls below the stopping threshold, and SCPO then cannot early stop and samples up to Nmax⁡=64N_{\max} = 64. This allows the new mode to emerge and override the prior.

Empirically, in our first-visit breakdown, the 21 prompts that start with high-confidence incorrect labels are not trapped by the prior and still gained +11.3 pp in Mean@16 after training.

Reviewer_et4xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

SCPO reduces the rollout cost of consensus-based test-time reinforcement learning. It uses a Dirichlet model to estimate the probability that an answer is the policy’s mode, stopping generation when a Monte Carlo confidence estimate crosses a threshold. An EMA of fixed-prefix answer counts carries information across epochs. The paper analyzes stopping, mode identification, and memory under policy drift, and evaluates SCPO against TTRL [1] and OptPO [2] on mathematical reasoning and code generation.

References

[1] Zuo, Yuxin, et al. "Ttrl: Test-time reinforcement learning." Advances in Neural Information Processing Systems 38 (2026): 131459-131483.

[2] Wang, Youkang, et al. "OptPO: Optimal Rollout Allocation for Test-time Policy Optimization." arXiv preprint arXiv:2512.02882 (2025).

Critical strengths
  1. Careful statistical treatment of historical evidence and stopping. Updating the EMA from a mandatory fixed prefix avoids the selection bias associated with reusing the entire adaptively stopped group. The analysis also distinguishes Monte Carlo certification, identification of the policy’s mode, and ground-truth correctness. These are important distinctions for consensus-based training.

  2. Useful token savings in several settings. On Qwen–MATH-500, SCPO obtains Mean@16 of 72.8 versus 72.5 for TTRL while reducing generated tokens from 216.7M to 121.2M (Table 1). The EMA, threshold, minimum-rollout, and runtime ablations provide meaningful evidence about the method’s behavior.

Critical weaknesses
  1. The benefit beyond simpler reductions in voting budget is insufficiently established. Adaptive voting is already studied in Adaptive-Consistency [1] and OptPO [2], making the contribution of cross-epoch memory especially important. On Qwen–MATH-500, SCPO averages 34.2 rollouts, close to its minimum of 32, but there is no fixed-32-vote TTRL comparison. The EMA ablation reduces tokens from 126.2M to 121.2M, approximately 4%, and is not reported for the Llama settings with the largest headline savings (Table 2). OptPO’s stopping settings and a comparison across matched accuracy–cost operating points are also missing. The current results therefore do not establish whether the added machinery improves substantially over simply using fewer votes.

  2. The performance-preservation and practical-efficiency claims need qualification. On Llama–MATH-500, SCPO’s Mean@16 is 57.2 ± 0.4, compared with 60.7 ± 1.0 for TTRL and 59.7 ± 0.8 for OptPO (Table 1). Pass@16 is preserved, but average sampled-answer accuracy decreases. Furthermore, the reported runtime is 4h55m for SCPO versus 4h30m for TTRL, a 9.3% increase despite substantial token savings (Table 4). SCPO is faster than OptPO in this experiment, but the evidence supports a task-dependent accuracy–token–runtime tradeoff rather than a general efficiency improvement with preserved performance.

References

[1] Aggarwal, Pranjal, Aman Madaan, and Yiming Yang. "Let’s sample step by step: Adaptive-consistency for efficient reasoning and coding with llms." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

[2] Wang, Youkang, et al. "OptPO: Optimal Rollout Allocation for Test-time Policy Optimization." arXiv preprint arXiv:2512.02882 (2025).

Questions for authors
  1. How does SCPO compare with fixed-16/32-vote TTRL and retuned OptPO at matched accuracy or token budgets? Please also clarify how the at-most-32 training rollouts are selected from voting groups of different sizes, so the policy-update budgets can be compared.

  2. How is the persistent answer set VV constructed for mathematical answers and code outputs? What happens when a new answer appears, what prior mass does it receive, and which mm enters the Monte Carlo correction? The implementation should explain how these choices relate to the fixed-support analysis.

  3. Appendix F states that LiveCodeBench consensus uses the official test cases. Are these public examples, private evaluation inputs, or a separate set? Are reference outputs or pass/fail labels consulted, and do all baselines receive identical access? Comparing generated outputs can remain label-free, but the protocol needs to be explicit.

8 comments
AuthorsOpenReview ↗

Response to Reviewer et4x (Part 1)

We thank the reviewer for the careful reading and constructive feedback. Below, we use W/Q for the weaknesses/questions and R for our responses.

W1.2: EMA ablation for the Llama settings with the largest headline savings.

R: We ran the EMA ablation with Llama3.1-8B-Instruct on AIME 2024, the setting with the largest token saving relative to TTRL (R/P is the average rollouts per prompt):

MethodMean@16 ↑Pass@16 ↑R/P ↓Num. tokens ↓
w/o EMA8.426.726.551.1M
SCPO9.326.718.335.9M

In this setting, the cross-epoch prior reduces rollouts by 30.9% and tokens by 29.7% relative to the variant without it, while matching Mean@16 and Pass@16. Together with the Table 2 results (in the manuscript), this shows that cross-epoch memory saves substantially more tokens than adaptive stopping alone.

AuthorsOpenReview ↗

Response to Reviewer et4x (Part 2)

W1.1 & Q1.1: How does SCPO compare with fixed-16/32-vote TTRL?

R: Thank you for the suggestion. We ran TTRL with fixed-16/32-vote on MATH-500 with Qwen2.5-Math-1.5B (the setting with average rollouts per prompt (R/P) closest to the pre-defined minimum Nmin⁡N_{\min}), and SCPO with minimum and maximum rollouts per prompt are 16 and 32, respectively. The results are shown in the below table:

MethodR/PTokensMean@16Pass@16
TTRL (fixed 16 votes)16.055.2M70.790.2
SCPO (min 16 votes)24.791.3M71.890.0
TTRL (fixed 32 votes)32.0107.7M72.589.6
SCPO (min 32 votes)34.2121.2M72.8 ±\pm 0.290.5 ±\pm 1.0
TTRL (fixed 64 votes)64.0216.7M72.5 ±\pm 0.490.4 ±\pm 0.3

We followed the original TTRL paper to use 64 rollouts per prompt for voting in our main experiments, however, in this setting, about 32 rollouts appear to be enough: 32-vote TTRL reaches similar Mean@16 to 64-vote TTRL and SCPO with Nmin⁡=32N_{\min}=32. In contrast, 16-vote TTRL is noticeably weaker as it uses too few rollouts for voting, which makes the majority-voted pseudo-labels noisier. This is exactly where SCPO helps: with Nmin⁡=16N_{\min}=16 it gains +1.1 pp Mean@16 over 16-vote TTRL (71.8 vs. 70.7) with Pass@16 nearly unchanged (90.0 vs. 90.2), by spending extra rollouts only on prompts where consensus has not yet formed (24.7 rollouts per prompt on average).

The posterior-based stopping works as intended. In the first 2 epochs, when the model is still weak, SCPO spends 29.6 rollouts per prompt on average. Once accuracy improves and consensus becomes clearer, the number of generated rollouts decreases steadily, and it needs only 21.8 in the last epoch. Compared with 32-vote TTRL, SCPO (Nmin⁡=16N_{\min}=16) uses 22.8% fewer rollouts and 15.2% fewer tokens, and compared with standard 64-vote TTRL, it uses 61.4% fewer rollouts and 57.9% fewer tokens, while staying within 0.7 pp Mean@16 and 0.4 pp Pass@16 of both.

These results show that SCPO gives a strong accuracy-vs-compute trade-off even with a small minimum budget: it outperforms 16-vote TTRL in accuracy, and stays close to 32/64-vote TTRL at a much lower rollout cost. On the other hand, a fixed budget for TTRL has to be tuned for each task, which may be impractical in test time, and even then it forces a trade-off: It wastes votes on easy prompts, or it runs out of votes on contested prompts. SCPO avoids this by deciding per prompt when to stop, using current answer counts and EMA memory from previous epochs, which makes SCPO useful for real-world LLM deployment where question difficulty is unknown and the model's consensus on each question changes over training epochs.

W1.3 & Q1.2: How does SCPO compare with retuned OptPO at matched accuracy or token budgets?

R: To compare OptPO and SCPO at matched accuracy or number of tokens, we retuned OptPO by sweeping its minimum number of rollouts per prompt (Nmin⁡N_{\min}) on Qwen2.5-Math-1.5B / MATH-500. Each OptPO configuration was run twice and we report the average. Token savings are relative to 64-vote TTRL:

MethodNmin⁡N_{\min}Mean@16 ↑Pass@16 ↑R/P ↓Num. tokens ↓Saving ↑
OptPO870.890.534.7142.1M34.4%
OptPO1671.190.530.4123.7M42.9%
OptPO3272.990.538.2139.2M35.8%
SCPO3272.890.534.2121.2M44.1%

At Nmin⁡=8N_{\min}=8, OptPO uses nearly as many rollouts as SCPO (34.7 vs. 34.2) but is worse on both Mean@16 (70.8 vs. 72.8) and tokens (142.1M vs. 121.2M), because it spends most of its additional rollouts on low-consensus prompts, which also have much longer responses. We hypothesize this is partly because OptPO estimates the parameters p0p_0 and q0q_0 of its noise model from only eight rollouts, which makes its stopping threshold sensitive to noise in the early vote counts. At Nmin⁡=16N_{\min}=16, OptPO spends about as many tokens as SCPO (123.7M vs. 121.2M), but its Mean@16 is 1.7 points lower (71.1 vs. 72.8). At Nmin⁡=32N_{\min}=32 (as in Table 1 of the manuscript), OptPO matches the Mean@16 of SCPO (72.9 vs. 72.8) but spends about 15% more tokens (139.2M vs. 121.2M).

AuthorsOpenReview ↗

Response to Reviewer et4x (Part 3)

W2: The performance-preservation and practical-efficiency claims need qualification. On Llama / MATH-500, SCPO decreases average accuracy. SCPO increases runtime despite substantial token savings.

R: Accuracy-token trade-off: We agree that SCPO has a lower Mean@16 than the baselines with Llama3.1-8B-Instruct on MATH-500. However, in all other settings in Table 1 (in the manuscript), SCPO stays comparable to the baselines on Mean@16 and Pass@kk while using far fewer rollouts and tokens.

Token-runtime trade-off: We agree that fewer tokens do not automatically mean faster training. In Table 4 of the manuscript, with Qwen2.5-Math-1.5B on MATH-500, using an extra rollout batch size (EBS) of 8, SCPO's wall-clock time is higher than TTRL. However, our additional experiments show that with a large EBS, SCPO's wall-clock time matches or improves on the baselines.

We ran SCPO with EBS=16 in the same setting as the Table 4 and broke down wall-clock time into three main components: rollout generation, Monte Carlo stopping checks, and policy optimization (loss, backpropagation, and parameter updates):

MethodMean@16 ↑R/P ↓Tokens ↓Avg. Generation time (s/step)Avg. Stopping check time (s/step)Avg. Optimization time (s/step)Runtime ↓
TTRL72.564.0216.7M47.20.043.54h30m
SCPO (EBS=8)72.834.6122.6M54.21.443.64h55m
SCPO (EBS=16)72.735.0126.2M46.01.341.84h33m

At EBS=8, SCPO's overhead comes from rollout generation (54.2 s/step, about 1.15× TTRL). This is a batching effect: TTRL generates all its rollouts in one large batch that vLLM handles efficiently, whereas SCPO generates in sequential rounds with a stopping check in between (Section 5.3 in the manuscript). Raising EBS to 16 closes most of the gap: SCPO uses 41.8% fewer tokens than TTRL with nearly the same Mean@16 and almost the same wall-clock time (4h33m vs. 4h30m).

We also conducted experiments with Llama3.1-8B-Instruct on AIME 2024:

MethodMean@16 ↑R/P ↓Tokens ↓Avg. Generation time (s/step)Avg. Optimization time (s/step)Runtime ↓
TTRL8.764.0106.2M41.952.16h31m
SCPO (EBS=16)8.226.552.2M54.339.36h31m

and Qwen3.5-2B on MMLU College Physics:

MethodMean@16 ↑R/P ↓Tokens ↓Avg. Generation time (s/step)Avg. Optimization time (s/step)Runtime ↓
TTRL91.064.022.4M22.816.056m21s
SCPO (EBS=8)91.832.111.5M15.516.349m05s

In both settings, SCPO uses about half the tokens of TTRL and its runtime is equal on Llama and about 13% faster on Qwen3.5-2B. These results suggest that SCPO can save tokens substantially with comparable or lower runtime.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.