acceptodds
Under review as a conference paper at ICLR 2027

Learning from How Large Language Models Have Changed

Abstract

Supervised fine-tuning (SFT) and on-policy distillation (OPD) improve language models through demonstrations and teacher feedback. However, the learner’s current state alone may not reveal how these signals should be allocated to support further task improvement. We introduce REWIND, which uses the model’s responses to prior training to guide fine-grained allocation of training signals within each sequence while preserving each supervision term’s learning direction. The same approach applies to both SFT and OPD without requiring an auxiliary feedback-generation loop. We provide theoretical justification by showing that REWIND can locally reduce task risk at a population SFT stationary point and by establishing teacher consistency for fixed-prefix OPD. Experiments on six mathematical reasoning benchmarks at two model scales demonstrate consistent performance gains in both SFT and OPD. Matched ablations further support the importance of token-specific signal allocation, the training-origin reference, and information from competing tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, supervised fine-tuning, on-policy distillation, training dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EmFMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new method, REWIND, which constructs a normalized geometric reference distribution based on prediction discrepancies between historical student checkpoints and the current model under the same prefix. This distribution is used to derive positive weights that are clipped and detached from the gradient computation, scaling the original SFT or OPD loss on a per-token basis without altering the learning targets defined by demonstrations or teacher feedback. Experiments are conducted using Qwen3-1.7B and Qwen3-4B to compare REWIND against multiple baselines on mathematical reasoning and coding tasks. The results show that REWIND achieves the highest average scores across both task categories, at both model scales, under both SFT and OPD settings.

Critical strengths
  1. Theoretically,the REWIND paper introduces a unified approach to leveraging historical information in both SFT and OPD and the authors also analyze the role of competing tokens.
  2. Experimentally, the authors evaluate REWIND against corresponding baselines across two model scales, mathematical reasoning and coding tasks, REWIND achieves the highest average scores in all four settings. Ablations using Shuffle and Binary demonstrate that the gains do not arise merely from using non-uniform weights, but also depend on the token-level allocation of historical signals.
Critical weaknesses
  1. The OPD experiments consider only one training configuration, making it difficult to disentangle the contributions of token-level historical signal allocation, the covariance term, and the competition term to the observed performance improvements.

  2. There is a gap between the theoretical analysis and the practical training setup. The teacher-consistency argument in the appendix relies on binary partition weights and fixed prefixes, whereas the actual OPD implementation uses a multi-class partition consisting of top-15 tokens and a tail group. Furthermore, the main experiments use only a single random seed 42, lacking multi-seed results to verify the robustness and stability of the reported improvements.

Questions for authors
  1. Have the authors investigated the effectiveness of both methods on novel tasks or datasets, similar to the evaluation in Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum? In particular, are these methods effective on tasks where the model weak region?

  2. Other static weighting methods, such as GCE, can also improve performance on MATH and coding tasks. Does REWIND provide additional gains over these methods?

1 comment
AuthorsOpenReview ↗

A kind request before the discussion closes

Dear Reviewer EmFM,

We are sorry to write once more, and we understand how busy this period is.

Your comments have been a great help to us. Working through them has made the paper clearer and stronger, and we are grateful for the care you put into the review and for your positive assessment of our work.

The discussion closes today. If you could find a few minutes to read our rebuttal and leave a short reply, we would be truly grateful. Even one line on whether your concerns have been addressed, or on what remains open, would mean a great deal to us. If anything is unclear, we will reply right away.

Thank you sincerely for your time.

Best regards,

The Authors

Reviewer_EF7DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REWIND, a unified token-level reweighting method for LLM post-training, covering both SFT and OPD. REWIND uses changes in token predictions between a historical checkpoint and the current model to assign token-level weights, based on the idea that tokens whose predictions move in a useful direction may provide more informative training signals. Experiments on Qwen3 backbones show that REWIND improves over standard training and several token-weighting baselines on the reported benchmarks.

Critical strengths
  1. The proposed method REWIND is a unified framework applicable to both SFT and OPD, which reweights the student-side residual factor in their target-logit gradients.
  2. The theory gives useful intuition and motivation for REWIND’s weighting rule.
  3. The authors design various model and task settings to provide empirical evidence of REWIND's effectiveness. Sensitivity and ablation experiments further support their conclusion.
Critical weaknesses
  1. The theory gives useful intuition for REWIND’s weighting rule, but it is still local and idealized. It justifies a one-step weighted update under specific assumptions, rather than showing that the historical guide remains aligned with downstream risk throughout long-horizon LLM post-training. The main support for the method therefore remains empirical.

  2. The experimental comparison miss several closely related baselines that use reference or historical models, such as Rho-1, Token Cleaning, and ssToken [1-3]. It is hard to tell whether REWIND’s gains come from its specific historical-residual design or from the broader benefit of reference-based reweighting in current experiment setups.

[1] Not All Tokens Are What You Need for Pretraining

[2] Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning

[3] ssToken: Self-modulated and semantic-aware token selection for LLM fine-tuning

  1. The evaluation is limited to two Qwen3 backbones, with no experiments on other model families or larger models. In REWIND-OPD, the historical signal is introduced after a 0.5-epoch standard OPD warm-up. The paper does not ablate the warm-up length or compare with baselines with matched warm-up. This leaves some uncertainty about how much of the OPD gain comes from the weighting rule itself versus the training schedule.
Questions for authors
  1. How does REWIND perform compared with reference-based reweighting baselines?
  2. Does REWIND generalize to larger backbones and to backbones from other model families?

If these concerns are adequately addressed, I would be willing to reconsider my score.

1 comment
AuthorsOpenReview ↗

A kind request before the discussion closes

Dear Reviewer EF7D,

We are sorry to write once more, and we understand how busy this period is.

Your comments have been a great help to us. Working through them has made the paper clearer and stronger, and we are grateful for the care you put into the review.

The discussion closes today. If you could find a few minutes to read our rebuttal and leave a short reply, we would be truly grateful. In your review you kindly mentioned that you would be willing to reconsider your score if your concerns were addressed, so your view on whether they have been would mean a great deal to us. If anything is unclear, we will reply right away.

Thank you sincerely for your time.

Best regards,

The Authors

Reviewer_GLrSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

REWIND uses a model’s training history to reweight token-level supervision in SFT and on-policy distillation. It compares current predictions with a frozen earlier checkpoint and uses the resulting weights to scale existing losses without changing their targets. The method improves reported math and code averages at two model sizes, with additional SFT gains on logic and instruction following. SFT ablations suggest that assigning historical weights to the corresponding tokens matters.

Critical strengths
  1. The paper presents an interesting way to use the student’s own training history. By comparing current predictions with an earlier checkpoint, REWIND adds information that current-confidence weighting cannot capture. The same weighting rule can be applied to SFT and on-policy distillation without changing their supervision targets.

  2. The empirical gains are fairly broad: REWIND achieves the highest reported math and code averages at both model sizes in SFT and OPD, with further SFT gains on logic and instruction following. The Section 5.2 ablations provide useful support for the mechanism. In particular, shuffling weights among tokens with similar current confidence hurts performance, suggesting that assigning each historical weight to its corresponding token matters.

Critical weaknesses
  1. Section 3 does not fully justify the key assumption behind REWIND. Section 3.1 shows that earlier parameter movement can align with the remaining task correction in a simplified model; Section 3.2 characterizes the proposed token weights; and Section 3.3 shows a benefit if the reweighted gradients align with task-risk reduction. The analysis does not establish that REWIND’s observed prediction changes produce that alignment. This matters because a token can gain probability during SFT without becoming more useful for the downstream task.

  2. The experimental evidence for the mechanism is stronger in SFT than in OPD. The detailed history ablations in Section 5.2 are limited to SFT on two math benchmarks, with no corresponding OPD test that disrupts token-to-weight assignment. The SFT comparison also omits DFT [1], a directly relevant probability-based token-weighting method cited by the paper. These gaps make it harder to attribute the gains specifically to checkpoint history.

Reference

[1] Y. Wu et al., On the Generalization of SFT: A Reinforcement Learning Perspective with Reward, ICLR 2026.

Questions for authors
  1. Theory: Is there an assumption or result connecting REWIND’s observed token-level prediction changes to the positive task-gradient alignment required in Section 3.3? If this connection is not established, what claim should readers take from the theoretical analysis about the effectiveness of the proposed weights?

  2. OPD mechanism: Can the authors provide an OPD analogue of the Section 5.2 weight-shuffling ablation, using the same prompts, teacher, and training budget? This would help determine whether matching historical weights to their tokens explains the OPD gains.

  3. SFT baseline: Why is DFT [1] not included, despite being cited? A matched comparison would help assess whether checkpoint history improves on a simpler current-probability weighting rule.

Reference

[1] Y. Wu et al., On the Generalization of SFT: A Reinforcement Learning Perspective with Reward, ICLR 2026.

2 comments
AuthorsOpenReview ↗

A kind request before the discussion closes

Dear Reviewer GLrS,

We are sorry to write once more, and we understand how busy this period is.

Your comments have been a great help to us. Working through them has made the paper clearer and stronger, and we are grateful for the care you put into the review.

The discussion closes today. If you could find a few minutes to read our rebuttal and leave a short reply, we would be truly grateful. Even one line on whether your concerns have been addressed, or on what remains open, would mean a great deal to us. If anything is unclear, we will reply right away.

Thank you sincerely for your time.

Best regards,

The Authors

Thank you for the response; the OPD ablations and DFT comparison address my main concerns, so I will raise my score to 3. I encourage tuning λ\lambda, κ\kappa and the reference on a validation set.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.