- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SPINE, a label-free test-time RL framework that uses majority-vote self-consistency rewards with GRPO. It introduces two components: distribution-aware selection of high-entropy tokens using Otsu’s thresholding, and an entropy-band regularizer constructed from the median and median absolute deviation of the selected tokens’ entropies. Policy optimization and KL regularization are restricted to selected positions. Experiments on text and multimodal reasoning benchmarks show improvements over standard TTRL and fixed-ratio high-entropy token selection, alongside reduced response-length collapse.
Critical strengths
Results across text-only and multimodal tasks support the proposed method's practical value within the evaluated settings.
Critical weaknesses
- Heuristic design with limited theoretical justification.
The method combines Otsu-based token selection with median/MAD-based entropy regularization, but the rationale for these particular choices remains largely empirical. The paper does not sufficiently establish why high-entropy positions reliably identify consequential reasoning decisions or why the proposed asymmetric entropy band is appropriate for optimization under noisy pseudo-rewards. The ablations support empirical effectiveness but provide limited insight into the conditions under which these mechanisms should work.
- Incremental contribution to the central challenge of label-free TTRL.
A fundamental difficulty in this setting is the mismatch between self-consistency and correctness: reinforcing majority answers can amplify existing errors. SPINE retains this reward mechanism and primarily modifies where and how strongly the policy is updated. While this improves optimization stability, the paper offers limited new mechanisms or understanding for distinguishing correct from incorrect consensus, escaping erroneous consensus, or determining when pseudo-reward optimization improves actual accuracy.
- Limited evidence of generalization across model families and scales.
All evaluated backbones belong to the Qwen family, and the main experimental evidence concentrates on 1.5B–3B models. Although the paper includes 8B evaluations and tasks beyond mathematics, the larger-model validation remains limited. Since entropy distributions and pseudo-label reliability may vary substantially across model families and capability levels, broader experiments are needed to establish the general applicability of the proposed selection and regularization mechanisms.
- Reported AMC scores require clarification.
Appendix B.7 specifies 83 evaluation questions, and the evaluation protocol uses one deterministic response per question. Accuracy should therefore change in increments of approximately 1.205 percentage points. However, several individual-seed results in Table 10, including 62.77%, 65.42%, and 78.43%, do not correspond to an integer number of correct answers. This discrepancy makes the reported multi-seed results difficult to interpret.
Questions for authors
See above.