- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RCTS: it uses the differences in a model's behavior under normal reasoning and reflective conditions to generate soft pseudo-labels, and selectively updates tokens in the student trajectory through Cross-Group Token Divergence, achieving test-time reinforcement learning without labels or external validators.
Critical strengths
-
The conceptual motivation of this work is strong—I like how RCTS jointly addresses self-referential pseudo-supervision and coarse token-level credit assignment in label-free test-time RL. Using the same policy snapshot in two contextual roles is a simple but interesting idea, avoiding the need for an external verifier, reward model, or separate teacher network.
-
The experimental evaluation is fairly comprehensive. The paper covers two model families and four reasoning benchmarks, with additional studies on model scale, independent adaptation runs, sampling budgets, hyperparameter sensitivity, and soft versus hard teacher supervision.
Critical weaknesses
-
Whether reflective teachers truly reduce self-referential errors is still lacking the most direct evidence. Teachers and students come from the same model snapshot, and the reflection prompt reads both the original question and all student attempts, so teacher distribution still heavily depends on the original rollout. Length-normalized self-probability and answer concentration are not equivalent to correctness; When a model forms consistent but incorrect answers after reflection, the current mechanism may actually grant it higher confidence. The paper does not use offline truth values to compare pseudo-label accuracy and confidence calibration for student majority, teacher top-1, and teacher-weighted distribution, so the core "less self-referential" claim is currently mainly indirectly supported by final performance.
-
The absolute position alignment assumption of CGTD is weak. Equations (10)–(12) directly compare the empirical distribution of different students and teachers answering on the same token index, but these long reasoning trajectories do not align with reasoning structure; The same position may correspond to problem analysis, calculation, verification, or summarization respectively. Therefore, a larger divergence may reflect misalignment in expression length, writing style, or reasoning stage, rather than decisive content of the answer. The paper also acknowledges that CGTD is not a causal locating mechanism, but currently only analyzes one AMC23 trajectory, which cannot fully prove its systematic selection of more valuable reasoning tokens.
Questions for authors
See the weaknesses above.