acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Supervised Fine-Tuning through Policy Projection: From Uniform Imitation to Selective Adaptation

Abstract

Supervised fine-tuning (SFT) has become a key recipe for adapting large language models (LLMs) to different domains through imitation of task-specific demonstrations. However, its deterministic objective applies indiscriminate learning to each reference token, which may lead to overfitting and ignore whether the supervision is compatible with the model's current predictive state. We first develop a structured policy projection framework that views fine-tuning as controlled movement in policy space and provides a principled design space by characterizing each token-level projection in terms of where the policy is directed with minimum shift and how strongly it contributes to optimization. This formulation reveals standard SFT as a rigid case with a degenerate projection target and uniform strength. Based on this, we further propose Modulated Fine-Tuning (MFT), which combines a compatibility-adaptive relaxed projection with an information-aware projection strength to modulate token-wise supervision according to the current policy, consolidating compatible and informative supervision while applying more conservative updates to strongly conflicting targets. Experiments across diverse domains show that MFT consistently outperforms SFT, improving average performance by +6.96 on mathematical reasoning and +10.8 on general domain tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sft, rl, llm

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b15eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a policy projection framework that separates token-level supervision into a projection target and a projection strength. Its instantiation, Modulated Fine-Tuning (MFT), combines compatibility-adaptive target relaxation with information-aware weighting. Experiments report improvements over SFT and several stabilized SFT baselines, including benefits after subsequent RL.

Critical strengths
  • The paper addresses the important problem of balancing supervised adaptation with preservation of pretrained capabilities.

  • MFT is simple to implement and shows promising results across mathematical reasoning, code generation, and medical tasks.

Critical weaknesses
  • The compatibility assumption needs targeted validation. MFT attenuates supervision for reference tokens that strongly disagree with the current policy. Such disagreement can also indicate correct, unfamiliar knowledge or difficult examples. Controlled experiments separating harmful supervision from correct but conflicting supervision would clarify when this attenuation helps or impedes learning.

  • The algorithmic contribution beyond token weighting is unclear. Appendix A.8 shows that MFT’s update is equivalent to token-weighted SFT with detached coefficient (w_t\widehat Z_t). Thus, target relaxation and projection strength affect the implemented gradient through their product. The paper should clarify what additional insight or design principle the projection framework provides beyond existing adaptive weighting approaches.

  • The minimum-shift motivation lacks direct empirical validation. Cross-domain performance and the token-probability analysis in Figure 3 provide indirect evidence, but do not directly measure full-distribution policy drift. Moreover, locally KL-minimal targets do not by themselves establish minimal drift of the final shared-parameter policy. KL measurements against the initial model on held-out contexts would strengthen this argument.

  • Model diversity and uncertainty reporting are limited. Experiments use Qwen3-8B-Base and Qwen3-1.7B-Base. Additional model families would strengthen the evidence for generality and reliability.

Questions for authors

The empirical results are promising, and the method is practically appealing. My main reservations concern the compatibility assumption, the contribution beyond adaptive token weighting, and the evidence connecting local projection geometry to knowledge preservation. Further details and supporting arguments are provided in the Weaknesses section.

3 comments

We sincerely thank the reviewer for your constructive feedback.

W1: Compatibility Assumption

We agree that low compatibility may indicate either harmful supervision or correct but unfamiliar knowledge. However, reliably distinguishing these cases is itself a challenging data-quality identification problem, potentially requiring additional annotations, verification mechanisms, or external supervision.

Importantly, MFT does not rely on making such distinctions. Instead of explicitly classifying supervision as beneficial or harmful, MFT uses policy compatibility as a continuous signal to adaptively regulate the extent of policy updates. Its relaxed projection conservatively attenuates strongly conflicting supervision rather than categorically discarding it, while information-aware strength further modulates token-wise learning.

This avoids the additional complexity of explicit supervision-quality identification, while still achieving substantial improvements in both in-domain reasoning and out-of-domain generalization. Our results demonstrate that effective selective adaptation can be achieved in an easy way without requiring an explicit distinction between harmful and correct-but-conflicting supervision.

W2: Contribution Beyond Token Weighting

Our main contribution lies not in introducing a different gradient form, but in establishing a principled and unified design space for supervised adaptation from the perspective of KL-minimal policy projection.

Specifically, our framework is motivated by the minimum-shift principle underlying RL's tendency to forget less than SFT: RL implicitly favors task-satisfying policies that minimize KL divergence from the original policy, thereby limiting unnecessary disruption of pretrained knowledge. Inspired by this insight, we bring the same principle into supervised adaptation by first defining a feasible set of distributions satisfying the supervision constraints, and then selecting the KL-minimal projection target qtq_t from this set relative to the current policy. This establishes a principled design space that characterizes supervised adaptation through the projection target qtq_t and its strength ZtZ_t.

Importantly, the framework also offers a unifying perspectiveon existing fine-tuning methods and systematically designing new ones. Standard SFT corresponds to a degenerate projection with uniform strength, while DFT, InfoSFT, EAFT, and so on can be represented through different target-strength formulations within our framework.

W3: KL Measurement and Analysis

To directly validate our minimum-shift motivation, we conducted additional experiments measuring the reverse KL divergence DKL(πtuned∣πbase)D_{\mathrm{KL}}(\pi_{\mathrm{tuned}}|\pi_{\mathrm{base}}) on held-out medical-domain text, with all models fine-tuned on mathematical reasoning data. This metric quantifies how much the fine-tuned policy deviates from the pretrained distribution, particularly when assigning probability to outputs that were unlikely under the base model.

(1) Comparison at matched task performance.

We first trained an MFT model with mathematical reasoning performance comparable to SFT. Under this controlled comparison, MFT exhibits lower reverse KL divergence than SFT, indicating that MFT can achieve similar task adaptation with less disruption to the pretrained output distribution.

MethodKL Divergence
SFT0.9796
MFT0.5798

(2) Comparison with advanced baseline.

We further compared MFT against DFT. MFT achieves lower reverse KL divergence while outperforming both methods on in-domain mathematical reasoning and out-of-domain general tasks.

MethodKL Divergence
DFT1.7699
MFT1.5088

These results provide direct empirical support for our minimum-shift motivation, suggesting that MFT achieves a more favorable balance between acquiring new capabilities and preserving pretrained policy patterns.

W4: Model Family Diversity

To further validate the generality of MFT, we conduct additional experiments on two distinct model families DeepSeekMath-7B and LLaMA-3.1-8B. The results consistently demonstrate substantial improvements with MFT across these architectures, further supporting its effectiveness and generality beyond a specific model family.

Math TasksAIME26AIME25Math500OlympiadMinervaAvg.
DeepSeekMath-7B0.410.6239.8811.3818.2614.11
SFT0.830.2130.667.5812.3210.32
MFT1.240.8346.1515.1619.3416.54
General TasksMMLUARC-CGPQAAvg.
DeepSeekMath-7B25.554.325.735.2
SFT19.247.223.029.8
MFT28.168.926.641.2
Math TasksAIME26AIME25Math500OlympiadMinervaAvg.
LLaMA-3.1-8B0.210.2113.683.144.854.42
SFT0.410.2126.846.017.918.24
MFT0.830.6228.116.399.199.03
General TasksMMLUARC-CGPQAAvg.
LLaMA-3.1-8B30.327.620.926.3
SFT25.959.622.636.0
MFT34.274.923.544.2

Dear Reviewer b15e,

We hope our responses have addressed your concerns. We would greatly appreciate it if you could take a moment to share your thoughts when you have time.

Thank you for your consideration and efforts.

Reviewer_V6aAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper revisits supervised fine-tuning through a policy projection framework. It describes token-level learning in terms of a projection target and an update strength. Based on this framework, the authors propose Modulated Fine-Tuning (MFT). MFT relaxes the supervision target according to its compatibility with the current model and adjusts token weights using a coefficient based on probability and information content. The paper derives the local projection and evaluates the method on mathematical reasoning, general tasks, code generation, and medical question answering. It also evaluates MFT as an initialization for subsequent reinforcement learning.

Critical strengths

The projection framework provides a useful perspective for understanding and designing SFT. The paper separates the local supervision target from its contribution to the shared update. It derives the target that minimizes KL divergence under the specified constraint. This gives the local target construction a clear mathematical basis and helps explain the relationship between MFT and standard SFT.

The experiments cover several settings and support the method’s practical value. The main experiments include relevant baselines such as DFT, ASFT, and PSFT. Additional experiments extend the evaluation to code and medical tasks. MFT improves overall performance in several settings, and the gains also extend to subsequent RL. Component ablations and comparisons of weighting functions provide further support for the main design choices.

Critical weaknesses

The gains over strong baselines are modest on some tasks. For example, MFT improves the average mathematical reasoning score from 29.89 with DFT to 30.80 (Table 1). This is a useful improvement, but the gain over an existing method is limited.

The explanation based on reduced policy change would benefit from direct validation. The paper proves that the constructed soft target minimizes KL divergence to the current policy under the specified constraint. However, it does not directly compare how well the trained models preserve the output distribution before fine-tuning. Such a comparison would strengthen the explanation.

Questions for authors

On the same evaluation inputs and at comparable task performance, does the output distribution after MFT remain closer to the model before fine-tuning than after SFT or DFT? Could the authors provide direct measurements, such as KL divergence, to support the explanation that MFT reduces unnecessary policy change?

4 comments

We sincerely thank the reviewer for your constructive feedback.

W1: Performance Comparison

We acknowledge that MFT achieves a modest improvement over DFT on in-domain mathematical reasoning tasks. However, MFT's key advantage lies not merely in improving in-domain performance, but in preserving broader generalization capabilities during adaptation. Notably, MFT achieves a substantially larger gain on out-of-domain general tasks (49.2 vs. 45.9), outperforming all competing offline fine-tuning methods. These results highlight MFT's ability to balance effective task adaptation with the retention of general-domain capabilities, rather than over-specializing to the fine-tuning domain.

W2: KL Measurement and Analysis

To directly validate our minimum-shift motivation, we conducted additional experiments measuring the reverse KL divergence DKL(πtuned∣πbase)D_{\mathrm{KL}}(\pi_{\mathrm{tuned}}|\pi_{\mathrm{base}}) on held-out medical-domain text, with all models fine-tuned on mathematical reasoning data. This metric quantifies how much the fine-tuned policy deviates from the pretrained distribution, particularly when assigning probability to outputs that were unlikely under the base model.

(1) Comparison at matched task performance.

We first trained an MFT model with mathematical reasoning performance comparable to SFT. Under this controlled comparison, MFT exhibits lower reverse KL divergence than SFT, indicating that MFT can achieve similar task adaptation with less disruption to the pretrained output distribution.

MethodKL Divergence
SFT0.9796
MFT0.5798

(2) Comparison with advanced baseline.

We further compared MFT against DFT. MFT achieves lower reverse KL divergence while outperforming both methods on in-domain mathematical reasoning and out-of-domain general tasks.

MethodKL Divergence
DFT1.7699
MFT1.5088

These results provide direct empirical support for our minimum-shift motivation, suggesting that MFT achieves a more favorable balance between acquiring new capabilities and preserving pretrained policy patterns.

Dear Reviewer V6aA,

We hope our responses have addressed your concerns. We would greatly appreciate it if you could take a moment to share your thoughts when you have time.

Thank you for your consideration and efforts.

Thank you for the additional experiments and clarifications. The reverse-KL analysis addresses my request for direct evidence supporting reduced policy shift and strengthens the paper. I remain weakly positive and maintain my original score of 3

Thank you for your time and effort.

We appreciate the reviewer's positive recognition of our work.

Reviewer_8cTDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Modulated Fine-Tuning (MFT), which adjusts how much the model learns from each teacher token. Standard SFT treats every teacher token indiscriminately, and the paper argues that it may cause overfitting and disturb existing capabilities's the student model. The key idea is to relax the targe when the teacher token conflicts with the student model -- a teacher token close to the model’s preferred token receives stronger supervision, while a strongly conflicting token receives a smaller correction (controlled by w_t). Thus, different tokens are given different learning strength. It can be also interpreted as a variant of DFT with more flexibility one each token, allowing learning at a finer token-level granularity.

The empirical results are promising on both mathematical reasoning and general tasks, with significant improvement over vanilla SFT, and moderate improvement over DFT. Additional experiments are also conducted on RL initialization, code generation, and medical knowledge.

Critical strengths

S1. It gives fine-grained control on what to be learned from the teacher model via MFT, in order to avoid overfitting and for student models with different levels of capabilities. The analytical framework is reasonable and explains the intuition behind the proposed approach MFT.

S2. Preliminary experiments are promising, especially in comparison to vanilla SFT.

Critical weaknesses

W1. Needs an in-depth comparison between DFT and MFT. The experimental improvement of MFT over DFT is marginal. It seems MFT gives more flexibility than DFT with more hyper-parameters. So it would be helpful to analyze what the additional flexibility implies and how to achieve better performance by adjusting them.

W2. Related methods discussed by the paper, such as information-aware weighting (InfoSFT — Sabbaghi et al., 2026) and entropy-adaptive fine-tuning (EAFT — Diao et al., 2026), are closely related to the proposed approach but are missing from the main experiments and comparisons. Needs to discuss why MFT is better or why not compare with these approaches.

W3. Experiments on comparing different fine-tuning approaches as a preceding stage of RL are not informative. When you compare different X + RL approaches, the hyper-parameters for different X must be different, for example, number of samples and learning rate for X. From Table 1, it is unclear how and whether such hyper-parameters are tuned individually for each X.

Questions for authors

Q1. In-depth comparison between DFT and MFT (as in W1).

Q2. Comparison with InfoSFT (Sabbaghi et al., 2026) and EAFT (Diao et al., 2026), at least analytically -- whether they are somehow equivalent or subsumed by MFT (or vice versa).

Q3. More details on how hyper-parameters are selected or tuned for X + RL experiments.

3 comments

We sincerely thank the reviewer for your constructive feedback. In fact, our structured projection framework provides a common representation for these related methods, allowing us to clarify both their connections and their distinctions (see the table below).

MethodqqZZ
SFTδyt∗\delta_{y_t^*}11
DFTδyt∗\delta_{y_t^*}ptp_t
EAFTδyt∗\delta_{y_t^*}H~t=HtTop−Klog⁡K\widetilde{H}_t = \frac{H_t^{\mathrm{Top-}K}}{\log K}
InfoSFTδyt∗\delta_{y_t^*}pt[logit⁡(pˉ)−logit⁡(pt)]+p_t[\operatorname{logit}(\bar{p})-\operatorname{logit}(p_t)]_+
MFT(1−wt)πθ(⋅∣st)+wtδyt∗(1-w_t)\pi_\theta(\cdot\mid s_t)+w_t\delta_{y_t^*}ptlog⁡(e+1/pt)p_t\log(e+1/p_t)

Notation: We denote pt=πθ(yt∗∣st)p_t = \pi_\theta(y_t^*\mid s_t) in short, and H~t\widetilde H_t here denotes the normalized Top-K predictive entropy.

W1 & Q1: Comparison between DFT and MFT

Although DFT and MFT both adapt token-level supervision according to the current policy, they originate from different motivations and address different aspects of supervised adaptation.

(1) Our Core Advantage

MFT aims to acquire new capabilities while preserving the pretrained model's broader generalization ability. Compared to DFT, MFT achieves a substantially larger gain on out-of-domain general tasks (49.2 vs. 45.9 in general tasks), which is consistent with our design aimed at acquiring new knowledge while preserving more pretrained capabilities.

(2) Motivation Difference

DFT revisits SFT from an RL-inspired policy-gradient perspective, aiming to rectify problematic implicit reward weighting and stabilize optimization. In contrast, MFT is motivated by the minimum-shift principle underlying RL's tendency to forget less, introducing KL-minimal policy projection as a principled foundation for a unified design framework that systematically characterizes and guides supervised adaptation.

(3) Comparison Under Our Framework.

  • DFT retains the hard reference target (qt=δyt∗q_t=\delta_{y_t^*}), regardless of its compatibility with the current policy, leaving the projection destination unmodulated. In contrast, MFT introduces a KL-minimal relaxed target with wt=(pt/pmax⁡,t)λw_t=(p_t/p_{\max,t})^\lambda, adapting the projection to the reference token's relative alignment with the policy mode. This enables more conservative target construction for strongly conflicting supervision while preserving the relative distribution of non-target tokens.
  • DFT adopts Zt=ptZ_t=p_t to rectify the implicit reward weighting of SFT and also interpret it as a confidence-based weighting method. In contrast, MFT introduces Zt=ptlog⁡(e+1/pt)Z_t=p_t\log(e+1/p_t) from an information-aware perspective. This design couples target confidence with its information distance from the current policy, while maintaining non-vanishing supervision strength for highly probable targets.

(4) Additional Flexibility.

Our ablation study (Sec. 5.4) has already systematically validated these choices. Table 4 demonstrates the complementary benefits of target relaxation and strength modulation. Table 5 shows that an intermediate λ=0.5\lambda=0.5 achieves overall better performance, balancing sufficient adaptation against excessive policy shifts. Table 6 further shows that our information-aware strength design outperforms alternative confidence-based and entropy-based schedules. Together, these results demonstrate that MFT's additional flexibility is structured and empirically justified.

W3 & Q3: More details for X + RL experiments

To ensure a fair comparison, all methods use the same 5K training samples with the same batch size 256 for 1 epoch during the short warm-start stage. For SFT, DFT, ASFT, and MFT, we adopt the same learning rate of 1×10−51\times10^{-5}, while PSFT uses 1×10−61\times10^{-6}, following the configuration recommended in its original paper. For baselines with method-specific hyperparameters, such as ASFT and PSFT, we directly adopt the recommended settings from their original papers.

W2 & Q2: Comparison with InfoSFT and EAFT

Beyond specific token-weighting formulations, MFT establishes a higher-level design space grounded in the KL-minimal projection principle, which aims to acquire new knowledge while minimizing unnecessary policy shifts and preserving pretrained capabilities. Within our unified framework, InfoSFT and EAFT can be characterized as degenerate cases with hard projection targets (qt=δyt∗q_t=\delta_{y_t^*}) and method-specific projection strengths (ZtZ_t) as shown in the table above, whereas MFT further explores compatibility-aware target relaxation alongside information-awarestrength modulation.

(1) InfoSFT derives information-aware weights to improve learning from expert demonstrations, emphasizing informative tokens while suppressing less useful supervision. However, its weighting primarily depends on target probability without explicitly considering relative policy compatibility. MFT additionally introduces a compatibility-aware relaxed projection target, enabling more conservative adaptation to strongly conflicting targets while retaining informative learning signals.

(2) EAFT uses predictive entropy to suppress confident conflicts and mitigate forgetting. However, entropy captures overall predictive uncertainty rather than the alignment of a specific reference token with the policy mode. MFT directly models this alignment through relative confidence, while its information-aware strength further regulates supervision. This enables more target-specific adaptation across heterogeneous predictive states.

(3) Performance.

We provide additional experiments on EAFT and InfoSFT under the same settings as our main experiment. The results below show that MFT outperforms EAFT and InfoSFT on both mathematical reasoning and general-domain benchmarks.

Math TasksAIME26AIME25Math500OlympiadMinervaAvg.
EAFT2.282.9148.7916.6615.6817.26
InfoSFT3.753.5361.5527.1322.7623.74
MFT(Ours)9.189.7971.0335.1728.8130.80
General TasksMMLUARC-CGPQAAvg.
EAFT37.648.729.738.7
InfoSFT45.845.934.141.9
MFT(Ours)58.449.140.249.2

Dear Reviewer 8cTD,

We hope our responses have addressed your concerns. We would greatly appreciate it if you could take a moment to share your thoughts when you have time.

Thank you for your consideration and efforts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.