acceptodds
Under review as a conference paper at ICLR 2027

Reflection-Conditioned Teacher–Student Test-Time Reinforcement Learning

Abstract

Test-time reinforcement learning aims to adapt large language models on unlabeled queries using only self-generated reasoning trajectories. This setting is attractive when ground-truth labels, executable verifiers, or trusted reward models are unavailable, but it also makes the learning signal fragile. Existing self-consistency-based objectives often use the same query-only rollouts both to define pseudo-rewards and to receive updates, which can turn unreliable agreement into overconfident supervision. Moreover, standard policy-gradient objectives apply a trajectory-level advantage to all tokens, despite the fact that only a subset of tokens is likely to determine the final answer. We propose RCTS, a reflection-conditioned teacher–student framework for label-free test-time reasoning adaptation. RCTS uses a single policy snapshot in two contextual roles: query-only student rollouts reveal the model's initial behavior, while reflection-conditioned teacher rollouts provide a structured reference. From the teacher rollouts, RCTS constructs a teacher-weighted answer distribution that assigns soft consistency scores to student trajectories. It further introduces Cross-Group Token Divergence to select student tokens that most differ from the reflection-conditioned teacher distribution. The final update applies mask-selective GRPO only to selected query-only student tokens, regularized by a KL anchor to the base policy. Experiments across mathematical reasoning and knowledge-intensive QA show that RCTS consistently improves the frozen models and is competitive with representative label-free test-time adaptation baselines.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-Time Reinforcement Learning, Label-Free Adaptation, Large Language Model Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cnoVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RCTS: it uses the differences in a model's behavior under normal reasoning and reflective conditions to generate soft pseudo-labels, and selectively updates tokens in the student trajectory through Cross-Group Token Divergence, achieving test-time reinforcement learning without labels or external validators.

Critical strengths
  1. The conceptual motivation of this work is strong—I like how RCTS jointly addresses self-referential pseudo-supervision and coarse token-level credit assignment in label-free test-time RL. Using the same policy snapshot in two contextual roles is a simple but interesting idea, avoiding the need for an external verifier, reward model, or separate teacher network.

  2. The experimental evaluation is fairly comprehensive. The paper covers two model families and four reasoning benchmarks, with additional studies on model scale, independent adaptation runs, sampling budgets, hyperparameter sensitivity, and soft versus hard teacher supervision.

Critical weaknesses
  1. Whether reflective teachers truly reduce self-referential errors is still lacking the most direct evidence. Teachers and students come from the same model snapshot, and the reflection prompt reads both the original question and all student attempts, so teacher distribution still heavily depends on the original rollout. Length-normalized self-probability and answer concentration are not equivalent to correctness; When a model forms consistent but incorrect answers after reflection, the current mechanism may actually grant it higher confidence. The paper does not use offline truth values to compare pseudo-label accuracy and confidence calibration for student majority, teacher top-1, and teacher-weighted distribution, so the core "less self-referential" claim is currently mainly indirectly supported by final performance.

  2. The absolute position alignment assumption of CGTD is weak. Equations (10)–(12) directly compare the empirical distribution of different students and teachers answering on the same token index, but these long reasoning trajectories do not align with reasoning structure; The same position may correspond to problem analysis, calculation, verification, or summarization respectively. Therefore, a larger divergence may reflect misalignment in expression length, writing style, or reasoning stage, rather than decisive content of the answer. The paper also acknowledges that CGTD is not a causal locating mechanism, but currently only analyzes one AMC23 trajectory, which cannot fully prove its systematic selection of more valuable reasoning tokens.

Questions for authors

See the weaknesses above.

Reviewer_DfUzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes RCTS, which performs test-time RL without labels, using the same model as both student and teacher under different prompts. It first generates student answers, then critiques them and then generates teacher answers conditioned on the critiques. The method combines multiple tricks: soft agreement between answers, a token-selection mechanism, and sparse updates of only selected student tokens using GRPO with KL regularization.

Critical strengths

I think that the core strength of the paper is that it tries to treat an interesting and challenging problem of test time training with no access to rewards or labels.

Critical weaknesses

I have a big problem of clarity with this paper and the methodology to validate the approach:

  • The writing of the paper at some point is purely a concatenation of equations, and the method is clearly over-engineered, with three pages of equations to describe the pipeline and insufficient ablations to justify each component. I do not understand why every design choice was done, and if it is even necessary.

  • For example, we do not even know if the teacher signal is a good pseudo reward: do its pseudo-rewards predict correctness better than majority voting?

  • As the datasets are small (AMC, AIME), the bolded gains are sometimes exactly being correct on an additional datapoint, which can be pure noise. This means we do not know if all the machinery introduced even helps training. In addition, the method proposed has a substantial additional generation cost, which is not taken into account when comparing to TTRL for example.

Questions for authors

I'm inclined for a reject for this paper, but addressing the weaknesses above can make me lean towards a more positive position:

  • What motivates/explains all the knobs of the method? was it empirical results? because the gains are not even interesting for the complexity and the additional compute overhead introduced.

  • Can you at least show that the pseudo-rewards introduced predict response correctness better than majority voting?

  • Please validate the method on larger datasets so that the gains, if they exist, can be at least significant.

  • Can you compare TTRL with your method for fixed time budget?

Reviewer_rxrJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RCTS, a variant of test time RL for LLMs. Works use a student-teacher setup in which student and teacher produces their own rollouts in which student produces both the answer and reflection on the answer that becomes input for the teacher. The paper evaluates on 2 baselines and 4 datasets.

Critical strengths

The paper raises an important question in TTRL literature that is: of unreliable self-consistency rewards models. For this paper produces a form of 2 step system in which the model answers then reflect and then again answers the final answer. Through this self-reflections authors claim to have improved test time learning

Critical weaknesses
  1. The student and the teacher are the same model, so if a model is biased towards a wrong answer the teacher can magnify the bias even more rather than mitigating it. There is absolutely no safeguard in the method to prevent the model collapse. The paper started it claims by critiquing existing method that they cause overconfident supervision (in abstract) but I dont see how this method tackle this issue.

  2. The number of benchmark and baseline the method is evaluated is far from sufficient. The paper evaluates only 4 benchmarks, in which AMC and AIME are less than <50 samples dataset. The main table used on 2 baselines and in ablation further 2 baseline on limited dataset were tested. These experimentations are encouraging but in no way sufficient to claim the method works.

  3. The paper uses hand crafted reflection prompt (appendix B). The paper should show prompt dependence of the method. Whether using shorter, less informative or extremely length prompt has any effects or not.

Questions for authors

The authors should reflect on the comments written in above section

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.