- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The manuscript considers the problem of test-time reinforcement learning, which refers to the scenario that the model has a test set, and sufficient compute, but does not have the labels for the test set, and wants to adapt to the specific test set in order for a better performance. The manuscript focuses on a majority-voting-based methodology that uses the majority answer in a group as a pseudo-label to do training, like TTRL. The authors introduce Sequential Consensus Policy Optimization (SCPO), a new algorithm for this problem. The main intuition is, for problems that the model already has a consistent answer in earlier training epochs, there is less meaning to generate more rollouts for this epoch. In this way, some computation can be saved. Technically, this is implemented by maintaining a Dirichlet posterior over each prompt's answer distribution and minimize the rollout count once the answer distribution has a clear mode. The authors provide some theoretical analysis about the difference between the proposed method and TTRL (i.e., the vanilla algorithm without reducing the rollout count). Empirically, the authors were able to show that using Qwen2.5-Math-1.5B, Llama-3.1-8B-Instruct and on math and coding benchmarks, SCPO is able to achieve a similar performance with previous methods like TTRL, but it saves a significant amount of computation measured by generated token count.
Critical strengths
- The manuscript's writing is excellent. This manuscript did a good job in describing the problem setting, important related work, and the merits of its own work. The presentation is understandable and pretty clear.
- The results are believable and as far as I can check, sound. I am convinced that the method saves a good amount of computation measured in tokens compared to prior work.
Critical weaknesses
- Some questions about the evaluation remains unclear from the manuscript.
- (Major) The main experiments align different methods' budget using the number of epochs for training. It seems unclear why this quantity is important. Naturally, when we consider computation, we might think of GPU time as a direct measure of budget.
- (Minor) The tables in the manuscript show error bars, but it is unclear where they come from.
- (Minor) The models used are relatively old.
- (Major) As a result, I cannot identify the practical value for this work.
- When we align different methods in GPU time, it does not seem that SCPO brings about a saving. In fact, the results in Table 4 seem to suggest the opposite.
Questions for authors
- Q1: Feel free to respond to the weaknesses raised.
- Q2: When answering Q1, can you also provide an example where in practice, SCPO can benefit?
8 comments
Response to Reviewer dVKt (Part 2)
Q2: Practical examples where SCPO can benefit
R: SCPO may not reduce wall-clock inference time, but it generates substantially fewer tokens, which matters when test-time computation is budget-limited.
For example, under a fixed inference-time token budget, the tokens saved on problems with clear consensus can be spent on other problems. It could also lower costs in API-based services that support test-time adaptation and charge for generated tokens.
Because rollouts are typically generated in parallel, using fewer of them can also lower KV-cache memory demand. In memory-constrained settings, where limited GPU memory caps the batch size or forces KV-cache recomputation, SCPO may ease memory pressure and improve generation efficiency.
SCPO can reduce GPU energy consumption even when wall-clock runtime is comparable. For Llama-3.1-8B-Instruct on AIME 2024, SCPO (EBS=16) uses about 8% less GPU energy than TTRL (6.12 vs. 6.65 kWh), estimated from all GPUs' power measurements recorded in WandB. A likely reason is that SCPO spends less time in the policy optimization phase, which consumes more power (average 1,136 W, versus 875 W during rollout generation).
I thank the authors for the detailed response. I have gone through it. I appreciate the additional experiments that align the GPU time, as well as those with newer models like Qwen3.5-2B. I think the response sufficiently addressed my questions about the evaluation (i.e., the first weakness in my review).
When aligning the GPU time, what I am reading from the rebuttal experiments (which understandably don't have error bars), as well as the main paper's ones (where error bars do overlap), SCPO does not seem to improve the performance metric (I think I should be looking at Mean@16) statistically significantly. Thus, the argument for the practical benefit of SCPO is really about cost, and potential of "easing memory pressure and improving generation efficiency" that the authors mentioned in the response to Q2. I think this improvement is real and of value. But I also think that this improvement, within a relatively small scope (test-time reinforcement learning is not really something that people often do when evaluating models in practice), is incremental (no significant performance gain).
Overall, I think this is a well-executed study, and I would be happy to recommend acceptance at another venue, e.g., TMLR. For ICLR, I tend to still recommend weak rejection.
Response to Reviewer dVKt (Part 1)
We thank the reviewer for the careful review. We respond to each point below. Here we use W/Q for the weaknesses/questions and R for our responses.
R: In test-time reinforcement learning (TTRL, Zuo et al., 2025; OptPO, Wang et al., 2025b), methods are typically compared at a matched number of epochs. This ensures that every method revisits each prompt the same number of times and takes the same number of gradient steps. However, to address the reviewer's concern about compute budget, we show below that SCPO also matches or beats TTRL when evaluated under wall-clock GPU time.
R: We agree that GPU time matters. Our additional experiments show that under the same number of epochs, with a large extra rollout batch size (EBS), SCPO's wall-clock time matches or improves on the baselines. In Table 4 (in the manuscript), we already showed that at EBS=8, SCPO reduces runtime by 9.5% compared with OptPO. Besides Table 4, we ran SCPO with EBS=16 with Qwen2.5-Math-1.5B on MATH-500 and broke down wall-clock time into three main components: rollout generation, Monte Carlo stopping checks, and policy optimization (loss, backpropagation, and parameter updates):
At EBS=8, SCPO's overhead comes from rollout generation (54.2 s/step, about 1.15× TTRL). This is a batching effect: TTRL generates all its rollouts in one large batch that vLLM handles efficiently, whereas SCPO generates in sequential rounds with a stopping check in between (Section 5.3 in the manuscript). Raising EBS to 16 closes most of the gap: SCPO uses 41.8% fewer tokens than TTRL with nearly the same Mean@16 and almost the same wall-clock time (4h33m vs. 4h30m).
We also conducted experiments with Llama3.1-8B-Instruct on AIME 2024:
and Qwen3.5-2B on MMLU College Physics:
In both settings, SCPO uses about half the tokens of TTRL and its runtime is equal on Llama and about 13% faster on Qwen3.5-2B. These results suggest that SCPO can save tokens substantially with comparable or lower runtime.
R: To obtain those error bars, we ran each experiment 3 times and report the mean and standard deviation across runs in the tables.
R: Regarding latest LLMs, we experimented with Qwen3.5-2B on MMLU College Physics using minimum 32 rollouts per prompt with an extra rollout batch size EBS=8, and report the results in the below table (token savings are relative to TTRL). The results show that SCPO saves token by 48.7% and runtime by 12.9%, while Mean@16 and Pass@16 are comparable or higher than baselines: