acceptodds
Withdrawn from ICLR 2027

SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) and multimodal LLMs (MLLMs) excel at chain-of-thought reasoning but face distribution shift at test-time and a lack of verifiable supervision. Recent test-time reinforcement learning (TTRL) methods derive label-free pseudo-rewards from self-consistency voting over sampled trajectories, yet they often collapse: the majority-vote reward prevails, responses shorten, and Pass@1 declines. We trace this to uniform sequence updates in which most tokens are low-entropy followers, while a small high-entropy subset determines the reasoning branches. Thus we propose SPINE, a token-selective test-time reinforcement learning framework that (i) performs distribution-aware forking-token selection to update only decision-critical branch points, and (ii) applies a robust entropy-band regularizer at those tokens to prevent premature collapse and suppress noisy drift. SPINE plugs into GRPO-style objectives (optionally with a KL anchor) and requires neither labels nor reward models. Across eight benchmarks spanning multimodal VQA and text-only reasoning, SPINE consistently improves Pass@1 over TTRL while mitigating response-length collapse and stabilizing training dynamics. Experiments on 8B-scale LLM and MLLM backbones further demonstrate that these gains persist with increased model capacity, supporting token-selective updates at chain-of-thought branch points as an effective label-free mechanism for test-time adaptation. Code will be released.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-Time Compute, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bTaLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SPINE, a label-free test-time RL framework that uses majority-vote self-consistency rewards with GRPO. It introduces two components: distribution-aware selection of high-entropy tokens using Otsu’s thresholding, and an entropy-band regularizer constructed from the median and median absolute deviation of the selected tokens’ entropies. Policy optimization and KL regularization are restricted to selected positions. Experiments on text and multimodal reasoning benchmarks show improvements over standard TTRL and fixed-ratio high-entropy token selection, alongside reduced response-length collapse.

Critical strengths

Results across text-only and multimodal tasks support the proposed method's practical value within the evaluated settings.

Critical weaknesses
  1. Heuristic design with limited theoretical justification.

The method combines Otsu-based token selection with median/MAD-based entropy regularization, but the rationale for these particular choices remains largely empirical. The paper does not sufficiently establish why high-entropy positions reliably identify consequential reasoning decisions or why the proposed asymmetric entropy band is appropriate for optimization under noisy pseudo-rewards. The ablations support empirical effectiveness but provide limited insight into the conditions under which these mechanisms should work.

  1. Incremental contribution to the central challenge of label-free TTRL.

A fundamental difficulty in this setting is the mismatch between self-consistency and correctness: reinforcing majority answers can amplify existing errors. SPINE retains this reward mechanism and primarily modifies where and how strongly the policy is updated. While this improves optimization stability, the paper offers limited new mechanisms or understanding for distinguishing correct from incorrect consensus, escaping erroneous consensus, or determining when pseudo-reward optimization improves actual accuracy.

  1. Limited evidence of generalization across model families and scales.

All evaluated backbones belong to the Qwen family, and the main experimental evidence concentrates on 1.5B–3B models. Although the paper includes 8B evaluations and tasks beyond mathematics, the larger-model validation remains limited. Since entropy distributions and pseudo-label reliability may vary substantially across model families and capability levels, broader experiments are needed to establish the general applicability of the proposed selection and regularization mechanisms.

  1. Reported AMC scores require clarification.

Appendix B.7 specifies 83 evaluation questions, and the evaluation protocol uses one deterministic response per question. Accuracy should therefore change in increments of approximately 1.205 percentage points. However, several individual-seed results in Table 10, including 62.77%, 65.42%, and 78.43%, do not correspond to an integer number of correct answers. This discrepancy makes the reported multi-seed results difficult to interpret.

Questions for authors

See above.

Reviewer_RDibOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPINE defines label-free test-time RL based on majority voting pseudo-reward generation, where the Otsu method chooses high-entropy tokens to apply the entropy-band penalty to that location using the median and median absolute deviation. The experiments involve SPINE's application on eight text or multimodal datasets with different Qwen family models. The experiments are conducted comparing SPINE to TTRL and fixed ratio high-entropy token updates. Other experiments include three separate training of AMC with Qwen3-1.7B and Qwen3-8B along with selected evaluation on 8B models.

Critical strengths
  • These experiments span various kinds of tasks and take into account training randomness. Tables 1 and 2 consider text-only and multimodal tasks and provide similar baselines including Top-20% HE-TTRL and SCRL. Three independent runs for AMC are provided in Table 10, which show improvements of 6.87 and 13.33 percentage points depending on the reported means. I appreciate these experiments although some questions about accuracy reporting are raised below.

  • The ablation experiments prove the necessity of both components being used in combination. According to Table 4, selection by Otsu alone gets an average result of 38.58, which increases to 42.17 when entropy-band regularization is applied compared with 35.34 for regular TTRL. Cross-task results also give preliminary evidence that the improvements after adaptation can be transferred to other tasks. These are my primary reasons for taking this approach into account.

Critical weaknesses
  • Some accuracies do not correspond to the claimed evaluation method. Since there is one deterministic answer to each question, the AMCs of 62.77% and 78.43% are not possible for 83 questions. Also, 35.55% is not possible for the claimed number of 198 questions in the GPQA-diamond dataset. The actual number of sample sizes, the number of evaluations, and the aggregation technique need to be explained.

  • There is no isolation of the source of the stability increase. There is no positive lower bound of the adaptive entropy band at all steps, so the formula itself cannot avoid the problem of entropy reduction even though there are empirical results in Figure 3. Moreover, the KL divergence is normalized according to the selected token size, but the policy term is not normalized, so changing the retention rate changes their relationship.

Questions for authors
  • Clarity: Is there a way you could give me the sample size, correct sample size, number of repetitions in evaluations, and aggregation procedure for each AMC and GPQA score, and tell me why these numbers are not consistent with deterministic evaluations? This is what is impacting my understanding most of all.

  • Mechanistic clarification: In case of entropy decreasing at all chosen places, how will the recalculated band work to compensate the loss? It would be nice to separate the things ensured by the theory and the empirical findings presented in Figure 3. Have you made any comparisons of your bands to fixed or historic bands?

  • Additional validation: Could you make a comparison between Otsu, random selection, and fixed-ratio selection having equaled the real retention ratio and policy loss scale? It would also be nice to see the exact adaptation steps and tokens produced per each dataset.

Reviewer_yf3HOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the collapse of TTRL:the majority-vote reward prevails, responses shorten, and Pass@1 declines. The paper attributes this to uniform updates over all tokens, most of which are low-entropy, while only a small set of high-entropy "forking tokens" decide the reasoning direction. To address this, the paper proposes SPINE, which uses distribution-aware forking-token selection to update only decision-critical branch points, and applies a robust entropy-band regularizer at those tokens to prevent premature collapse and noisy drift. The experiments show that SPINE improves over TTRL and other label-free baselines on multimodal VQA and text reasoning benchmarks with several Qwen backbones from 1.5B to 8B, while keeping response length and entropy more stable during training.

Critical strengths

None

Critical weaknesses

The explanation of TTRL collapse is not well-supported and lacks direct evidence. The paper claims that collapse comes from uniform token updates under noisy pseudo-rewards, but it only shows that SPINE trains more stably, not that it works by solving this problem. The gains may instead come from the general benefit of finer-grained credit assignment, which is general effective in RL, rather than from mitigating noisy pseudo-rewards.

Questions for authors

Could the authors provide more direct evidence that SPINE improves TTRL by addressing the noisy pseudo-reward problem, rather than as a general RL improvement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.