acceptodds
Under review as a conference paper at ICLR 2027

Learning Where and How to Train from Model History: Supervised Fine-Tuning with Historical States

Abstract

Supervised fine-tuning (SFT) adapts large language models to downstream tasks through reference responses. We study how a model’s own training history can improve the use of existing supervision by guiding which examples receive training and how their targets shape model updates. We propose WMSS, a framework that pairs the current model with a trainable model initialized from a weaker historical checkpoint. The framework uses current predictive entropy and historical–current entropy differences to allocate training opportunities, and jointly trains the two models through a mixed-logit objective on the selected examples. This produces historical feedback for sample selection and token-level correction while preserving the supervised data pool and original reference targets. We provide a theoretical characterization by deriving an exact decomposition of the mixed residual into residual-scale modulation and candidate-allocation shifts. Experiments across model families demonstrate consistent gains in mathematical reasoning, code generation, and logical reasoning over SFT baselines with the same nominal epoch budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2JnpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes WMSS, a supervised fine-tuning framework that uses a model's weaker historical state to guide further training under the original reference targets. WD-DS selects examples using current predictive entropy and historical entropy differences, while WD-JT mixes historical and current logits before computing cross-entropy. The paper derives local gradient and margin identities and evaluates the method on mathematical reasoning, code generation, and logical reasoning across four Qwen and Gemma backbones. Results show consistent improvements over SFT and several offline baselines, with additional multi-seed math results, component ablations, and optimization controls. Inference uses only the final strong model.

Critical strengths
  1. The empirical improvement is broad and reasonably well supported. Tables 1–3 show gains across four backbones and three task domains. In Table 2, the three-run math averages improve from 63.3 to 69.9 on Qwen3-4B and from 65.8 to 73.1 on Qwen3-8B. Fixed main-run coefficients and preselected training endpoints strengthen the evidence that the method offers a reusable improvement over SFT. This breadth is the strongest reason for acceptance.

  2. The paper offers a useful mechanism with meaningful controls. Section 5 separates correction scale from redistribution toward historical alternatives, while Section 6.4 tests larger learning rates, label smoothing, a uniform reference, and shuffled weak-logit pairings. The combined method outperforms these controls and both component-only variants. Together, these results support the value of sample-aligned historical predictions and provide a practical way to use them without additional deployment cost.

Critical weaknesses
  1. The efficiency comparison does not fully establish an advantage at matched compute. Section 6.2 and Appendix D.5 acknowledge roughly 1.2× SFT computation despite fewer updates. The longer-SFT diagnostic is not a matched continuation: its near-two-epoch MATH500 score is 59.6, versus 66.2 for the main SFT baseline. Consequently, that trajectory does not settle whether a comparably configured SFT run given the additional budget would close some of the gap. This limits the strength of the efficiency interpretation, although the reported wall-clock overhead is modest.

  2. The strongest mechanism controls have narrower coverage than the main performance claim. Figure 6 tests alternative explanations on one backbone and three math benchmarks, while the repeated-seed comparisons also focus on Qwen3 mathematics. Code, logic, and Gemma results are useful evidence of transfer, but do not establish equally robust component attribution in those settings. The local logit identities also do not guarantee favorable full-network updates, as Appendix E acknowledges. The broader mechanistic interpretation should remain conditional on this scope.

Questions for authors
  1. Clarification: What configuration or seed differences explain the 59.6 versus 66.2 near-two-epoch SFT discrepancy? Is there an existing continuation from the main baseline that permits comparison at WMSS's actual compute or wall-clock budget?

  2. Clarification: Are additional training-seed results or uniform/shuffled-reference controls already available for code, logic, or Gemma? Please distinguish these from the multi-seed math evidence rather than undertaking a new experimental campaign.

  3. Clarification: Is the historical branch frozen throughout each WD-JT stage, and is the factor λ left in the implemented loss gradient or compensated in the optimizer? This would connect the local analysis and the compute accounting more precisely to the training implementation.

1 comment
AuthorsOpenReview ↗

A kind request before the discussion closes

Dear Reviewer 2Jnp,

We are sorry to write once more, and we understand how busy this period is.

Your comments have been a great help to us. Your questions showed us where the paper was not precise enough, and working through them has made it clearer and stronger. We are grateful for the care you put into the review and for your positive assessment of our work.

The discussion closes soon. If you could find a few minutes to read our rebuttal and leave a short reply, we would be truly grateful. Even one line on whether your concerns have been addressed, or on what remains open, would mean a great deal to us. If anything is unclear, we will reply right away.

Thank you sincerely for your time.

Best regards,

The Authors

Reviewer_S2gGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WMSS, a method that leverages a model’s own historical checkpoints to improve supervised fine-tuned (SFT) training. It selects training samples based on the prediction entropy of both the current and historical models, and refines the learning signal by blending the logits from these models while training against the original labels. This approach requires no additional generation of training responses and incurs no extra inference overhead during deployment.

Critical strengths
  1. The paper investigates how to obtain further gains from existing supervised data by reusing historical model checkpoints. The approach requires no additional response generation during training and introduces no inference overhead, making it practically appealing.

  2. WMSS uses historical predictions both to select training examples and to shape token-level learning signals. These components form a consistent approach to deciding where and how to train, and the ablation results support the usefulness of both components.

Critical weaknesses
  1. The mixed-logit objective is closely connected to Product-of-Experts training, particularly prior work on learning from weak-model errors. The data-selection component also relates closely to cross-model entropy-based selection methods such as InstructDiff. The contribution is therefore better characterized as a historical-checkpoint-based SFT framework with additional analysis and empirical validation than as an entirely new learning principle.

  2. The experiments demonstrate that WMSS is effective, but provide less evidence that using the model’s own history is particularly beneficial compared with other weak references. Direct comparisons with closely related PoE objectives and data-selection methods would help isolate the contribution of the proposed design.

  3. Although inference remains unchanged, historical-model evaluation and sample selection introduce additional training computation. Comparisons under the same nominal number of epochs do not fully establish an advantage under matched compute budgets.

Questions for authors

Please refer to weaknesses.

4 comments
AuthorsOpenReview ↗

A kind request before the discussion closes

Dear Reviewer S2gG,

We are sorry to write once more, and we understand how busy this period is.

Your comments have been a great help to us. Your points on the connection to PoE training and InstructDiff made us rethink how the paper positions its contribution, and working through them has made the paper clearer and stronger. We are grateful for the care you put into the review.

The discussion closes soon. If you could find a few minutes to read our rebuttal and leave a short reply, we would be truly grateful. Even one line on whether your concerns have been addressed, or on what remains open, would mean a great deal to us. If anything is unclear, we will reply right away.

Thank you sincerely for your time.

Best regards,

The Authors

The clarification of the contribution and the additional comparisons partially address my concerns. Still, the external-reference comparison changes both the model size and whether the reference is frozen or jointly trained, so it does not isolate the benefit of using the model’s own history. The longer-SFT results help address the compute concern, but an advantage under matched total training compute remains insufficiently established.

Thank you very much for coming back to our response and for reading it so carefully.

About the external-reference comparison, we think our label caused a misunderstanding. "WMSS (w/o WD-DS)" just means WD-JT on the full data, and "Joint-Logit Training" is only the name of the module. In both rows we stop the gradient at the reference branch and update only the main model, so freezing is the same in both. What changes is the reference.

The size and the source of the reference do change together, but we don't see a way around this. An external weak reference has to be a different, smaller model, and someone has to pick it and usually train it first. With the model's own history we get the reference for free, and it works better: 69.6% against 65.4%.

Figure 6 is the experiment where everything else is fixed and only the reference changes. We keep the same model and the same pipeline, and replace the reference with a uniform one or with one from a shuffled example. Both are clearly worse than WMSS, on Qwen3 math and also on Gemma logical reasoning.

About matched compute, the difficulty is that the extra cost of WMSS is forward passes, not SFT updates. Once we turn it into extra SFT steps, the answer depends on how we schedule those steps. So we checked both ends:

  • Same checkpoint and same number of updates: 79.5% against 76.6% at 123 steps, and 78.4% against 76.2% at 218 steps (Figure 4).
  • About 2.5× the compute of two-epoch SFT, which is much more than WMSS uses: the best of 109 SFT checkpoints gets 66.2% on MATH-500, and WMSS gets 71.4% (Figure 5).

A matched run would sit somewhere between these two, and SFT is behind at both.

Thanks again for the discussion.

I will raise my score to weak accept.

Reviewer_38geOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces WMSS, a method that enhances Supervised Fine-Tuning (SFT) by leveraging historical model checkpoints. It selects high-value samples based on entropy discrepancies and optimizes token-level predictions by blending logits from both historical and current model versions. The approach yields performance improvements across tasks involving mathematics, coding, and logical reasoning.

Critical strengths
  • Extensive experiments were conducted to verify the effectiveness of the method.
  • The method’s performance—in terms of both effectiveness and efficiency—was compared and analyzed.
Critical weaknesses
  • There is a lack of experiments using more powerful models—for instance, models of a similar size to the Qwen3.5 series.
  • The presentation and analysis of training dynamics (such as loss curves and entropy curves) are missing.
  • The entire caption for "2 WD-DS" in Figure 2 is obscured.
Questions for authors
  • Regarding Equation 8, have you experimented with setting α\alpha and γ\gamma to different values? If they are intended to be the same, expressing the term as an absolute value might make it easier to understand.
  • Given that current models typically undergo a two-step post-training process—Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL)—is the model trained via the WMSS algorithm better suited for the subsequent RL phase compared to the baseline model?
1 comment

Thank you for your response, and please accept my apologies for the delay in my reply. In light of the significant effort put into the rebuttal, I will increase my score.

Nevertheless, I remain of the opinion that this paper suffers from a serious lack of innovation and bears clear, heavy hallmarks of "auto-research". As these are broader, systemic issues that cannot be resolved during the rebuttal phase, I am not requesting any further revisions. However, I believe the Area Chair should take this into account when making the final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.