acceptodds
Under review as a conference paper at ICLR 2027

Teach Me After You Learn It: Lookahead Self-Distillation for RLVR

Abstract

On-policy self-distillation (OPSD) has emerged as a popular training paradigm for LLMs, providing dense, token-level supervision by aligning a model’s own distribution with the distribution induced by privileged information. However, existing OPSD methods derive supervision solely from privileged information of the same instance, leaving inter-instance supervision largely unexplored. We propose LookahEAd self-Distillation (LEAD), which introduces inter-instance distillation by allowing the teacher to first learn from other on-policy rollouts before teaching the student. LEAD constructs lookahead teachers through virtual RL updates on disjoint subsets of on-policy rollouts, and uses each teacher to provide cross-group token-level supervision for the other subset. To reconcile the complementary yet potentially conflicting signals from OPSD and RLVR, we introduce harmonious modulation, which adaptively modulates the GRPO advantage based on the agreement between the two signals. Extensive experiments on scientific reasoning and interactive-agent benchmarks show that LEAD achieves the best in-domain average performance across model scales while maintaining out-of-domain generalization comparable to GRPO.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, on-policy self-distillation, large language model reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WUfNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LEAD, a lookahead self-distillation framework that combines GRPO with on-policy self-distillation for RLVR. The method constructs lookahead teachers through virtual RL updates, introduces cross-group scoring to provide additional token-level supervision, and employs harmonious modulation to reconcile self-distillation signals with GRPO advantages. Experiments on scientific reasoning and interactive-agent benchmarks show improvements in average in-domain performance, while out-of-domain performance remains largely comparable to GRPO.

However, the methodological novelty is not sufficiently distinguished from existing GRPO-based self-distillation approaches, and the comparisons with recent related methods are limited. Moreover, the performance improvements over GRPO are relatively modest in several settings, with little consistent improvement in out-of-domain generalization. Given the additional training complexity and the unclear practical advantages over existing approaches, I find the current evidence insufficiently convincing to support the paper's contributions.

Critical strengths
  1. Reasonable motivation and method design. The paper addresses the limitations of sparse reward supervision in RLVR and attempts to incorporate inter-instance information into on-policy self-distillation. The lookahead and cross-scoring mechanisms provide an intuitive extension to existing self-distillation approaches.

  2. Evaluation across different task settings. The experiments cover both scientific reasoning and multi-turn interactive-agent benchmarks, with additional ablation studies examining the proposed components. The results demonstrate some improvements in in-domain performance.

Critical weaknesses
  1. Limited methodological novelty and insufficient comparison with recent work. Combining GRPO with on-policy self-distillation has already been extensively explored in recent studies. Although LEAD introduces lookahead updates and cross-group scoring, its contribution is not sufficiently distinguished from existing RLVR–OPSD approaches. More importantly, the paper lacks experimental comparisons with several closely related contemporary methods, such as SRPO, RLCSD, and SDAR, despite their relevance to the proposed framework. The related-work discussion is also insufficient to establish what fundamentally new capabilities LEAD provides beyond existing methods.

  2. Limited practical improvements relative to the additional complexity. Although LEAD achieves higher average in-domain performance, the improvements over GRPO are relatively modest in several settings, and some individual results are even worse. More importantly, the out-of-domain results show little consistent improvement over GRPO, raising questions about the effectiveness of the proposed inter-instance supervision for generalization. Considering the additional complexity introduced by virtual updates, lookahead teachers, and cross-scoring, the practical benefits are not sufficiently convincing. Additionally, the presentation of experimental results could be improved, as lengthy table captions and complicated evaluation descriptions make the findings difficult to follow.

Questions for authors
  1. Given the extensive recent literature on combining GRPO with on-policy self-distillation, could the authors provide a more comprehensive discussion and comparison with closely related methods, particularly SRPO, RLCSD, and SDAR? What is the distinct methodological contribution of LEAD beyond existing approaches?

  2. Why are the experimental comparisons limited to GRPO, SDPO, and RLSD? Could the authors include additional competitive baselines to better establish the effectiveness of LEAD relative to recent methods?

  3. Although LEAD improves average in-domain performance, several individual results remain below GRPO, while the out-of-domain improvements are limited. Could the authors clarify under what conditions LEAD provides meaningful advantages over GRPO? Do these improvements justify the additional computational overhead, particularly under matched training-compute budgets?

  4. The cross-scoring mechanism partitions rollouts into disjoint groups, but both groups may still contain responses to the same questions. Could the authors clarify whether the improvements arise from genuine cross-query knowledge transfer or merely from cross-rollout supervision within the same task? A comparison using query-disjoint groups would help support the claimed inter-instance contribution.

Reviewer_AdomOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LEAD, which constructs temporary teachers through virtual RL updates on separate subsets of an on-policy batch. Each teacher scores the other subset under privileged conditioning, and the resulting token-level signal modulates GRPO advantages. Experiments cover four scientific-reasoning domains and three interactive-agent benchmarks using Qwen3-4B and Qwen3-8B. The paper reports improved in-domain averages while maintaining out-of-domain performance generally comparable to GRPO.

Critical strengths
  • A practical and well-motivated method. Updating the teacher using other rollouts before supplying self-distillation signals is a useful extension of privileged-context supervision. The bounded modulation in Eqs. (9)–(11) preserves each GRPO advantage’s sign without requiring a larger external teacher.

  • Positive results across complementary settings. LEAD achieves the highest reported in-domain suite average in all four model-size/task-suite combinations (Table 1). Scientific-reasoning averages exceed the strongest baseline by 2.02 points at 4B and 1.78 at 8B. The 8B ALFWorld result is particularly notable: 84.4% versus 75.0% for the strongest reported baseline.

  • Ablations provide useful support for the proposed design.

Critical weaknesses
  • Closely related baselines are missing. RLCSD is explicitly discussed as addressing privileged-context distortions, yet is absent from the comparisons.

  • The interpretation as isolated inter-instance supervision needs stronger validation. Section 3.2.1 partitions rollouts rather than questions, so both groups contain responses to the same questions. This excludes the scored trajectory from the current virtual-update subset but does not isolate transfer across distinct questions.

Questions for authors
  • Can the authors report both in-domain and OOD performance at the same validation-selected checkpoints, across independent seeds?

  • Equation (8) subtracts the batch mean and can reverse the sign of the original teacher–student log–probability gap. How should “agreement” in Eq. (10) be interpreted after this transformation, and how does sign-preserving normalization compare?

Reviewer_1iFEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses lookahead gradient stepped policy as a teacher to distill back through a score with privileged info to the original policy which I find neato. Average performance is higher or similar to RL.

Critical strengths

I like lookahead “teachers” since they are interesting and novel. Various performance improvements through this lookahead method are reported in the paper.

Critical weaknesses

Section 4.1 had different checkpoint rules for in and out domain performance which is suspicious. Maximum score can be biased rather than the mean in OOD tests. A baseline where that keeps the responses to a specific question in the same group was not ran. Lookahead steps have been used in mutli-agent RL gradient corrections but are not discussed https://arxiv.org/abs/1709.04326 and https://arxiv.org/abs/1811.08469. Further there is an omission of privileged-information distillation: https://arxiv.org/abs/2602.04942. In table 6, SDPO results do not seem comparable to GRPO and LEAD since the learning rate is larger and the KL penalty was not used.

Questions for authors

In-domain and out-of-domain metric performance need to correspond. I think comparing the stop-gradient of the virtual update with one without would differentiate the work from other lookahead works. I think this method is novel overall and well situated for ICLR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.