- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a policy projection framework that separates token-level supervision into a projection target and a projection strength. Its instantiation, Modulated Fine-Tuning (MFT), combines compatibility-adaptive target relaxation with information-aware weighting. Experiments report improvements over SFT and several stabilized SFT baselines, including benefits after subsequent RL.
Critical strengths
-
The paper addresses the important problem of balancing supervised adaptation with preservation of pretrained capabilities.
-
MFT is simple to implement and shows promising results across mathematical reasoning, code generation, and medical tasks.
Critical weaknesses
-
The compatibility assumption needs targeted validation. MFT attenuates supervision for reference tokens that strongly disagree with the current policy. Such disagreement can also indicate correct, unfamiliar knowledge or difficult examples. Controlled experiments separating harmful supervision from correct but conflicting supervision would clarify when this attenuation helps or impedes learning.
-
The algorithmic contribution beyond token weighting is unclear. Appendix A.8 shows that MFT’s update is equivalent to token-weighted SFT with detached coefficient (w_t\widehat Z_t). Thus, target relaxation and projection strength affect the implemented gradient through their product. The paper should clarify what additional insight or design principle the projection framework provides beyond existing adaptive weighting approaches.
-
The minimum-shift motivation lacks direct empirical validation. Cross-domain performance and the token-probability analysis in Figure 3 provide indirect evidence, but do not directly measure full-distribution policy drift. Moreover, locally KL-minimal targets do not by themselves establish minimal drift of the final shared-parameter policy. KL measurements against the initial model on held-out contexts would strengthen this argument.
-
Model diversity and uncertainty reporting are limited. Experiments use Qwen3-8B-Base and Qwen3-1.7B-Base. Additional model families would strengthen the evidence for generality and reliability.
Questions for authors
The empirical results are promising, and the method is practically appealing. My main reservations concern the compatibility assumption, the contribution beyond adaptive token weighting, and the evidence connecting local projection geometry to knowledge preservation. Further details and supporting arguments are provided in the Weaknesses section.
3 comments
W4: Model Family Diversity
To further validate the generality of MFT, we conduct additional experiments on two distinct model families DeepSeekMath-7B and LLaMA-3.1-8B. The results consistently demonstrate substantial improvements with MFT across these architectures, further supporting its effectiveness and generality beyond a specific model family.
| Math Tasks | AIME26 | AIME25 | Math500 | Olympiad | Minerva | Avg. |
|---|---|---|---|---|---|---|
| DeepSeekMath-7B | 0.41 | 0.62 | 39.88 | 11.38 | 18.26 | 14.11 |
| SFT | 0.83 | 0.21 | 30.66 | 7.58 | 12.32 | 10.32 |
| MFT | 1.24 | 0.83 | 46.15 | 15.16 | 19.34 | 16.54 |
| General Tasks | MMLU | ARC-C | GPQA | Avg. |
|---|---|---|---|---|
| DeepSeekMath-7B | 25.5 | 54.3 | 25.7 | 35.2 |
| SFT | 19.2 | 47.2 | 23.0 | 29.8 |
| MFT | 28.1 | 68.9 | 26.6 | 41.2 |
| Math Tasks | AIME26 | AIME25 | Math500 | Olympiad | Minerva | Avg. |
|---|---|---|---|---|---|---|
| LLaMA-3.1-8B | 0.21 | 0.21 | 13.68 | 3.14 | 4.85 | 4.42 |
| SFT | 0.41 | 0.21 | 26.84 | 6.01 | 7.91 | 8.24 |
| MFT | 0.83 | 0.62 | 28.11 | 6.39 | 9.19 | 9.03 |
| General Tasks | MMLU | ARC-C | GPQA | Avg. |
|---|---|---|---|---|
| LLaMA-3.1-8B | 30.3 | 27.6 | 20.9 | 26.3 |
| SFT | 25.9 | 59.6 | 22.6 | 36.0 |
| MFT | 34.2 | 74.9 | 23.5 | 44.2 |
Dear Reviewer b15e,
We hope our responses have addressed your concerns. We would greatly appreciate it if you could take a moment to share your thoughts when you have time.
Thank you for your consideration and efforts.
We sincerely thank the reviewer for your constructive feedback.
W1: Compatibility Assumption
We agree that low compatibility may indicate either harmful supervision or correct but unfamiliar knowledge. However, reliably distinguishing these cases is itself a challenging data-quality identification problem, potentially requiring additional annotations, verification mechanisms, or external supervision.
Importantly, MFT does not rely on making such distinctions. Instead of explicitly classifying supervision as beneficial or harmful, MFT uses policy compatibility as a continuous signal to adaptively regulate the extent of policy updates. Its relaxed projection conservatively attenuates strongly conflicting supervision rather than categorically discarding it, while information-aware strength further modulates token-wise learning.
This avoids the additional complexity of explicit supervision-quality identification, while still achieving substantial improvements in both in-domain reasoning and out-of-domain generalization. Our results demonstrate that effective selective adaptation can be achieved in an easy way without requiring an explicit distinction between harmful and correct-but-conflicting supervision.
W2: Contribution Beyond Token Weighting
Our main contribution lies not in introducing a different gradient form, but in establishing a principled and unified design space for supervised adaptation from the perspective of KL-minimal policy projection.
Specifically, our framework is motivated by the minimum-shift principle underlying RL's tendency to forget less than SFT: RL implicitly favors task-satisfying policies that minimize KL divergence from the original policy, thereby limiting unnecessary disruption of pretrained knowledge. Inspired by this insight, we bring the same principle into supervised adaptation by first defining a feasible set of distributions satisfying the supervision constraints, and then selecting the KL-minimal projection target qt from this set relative to the current policy. This establishes a principled design space that characterizes supervised adaptation through the projection target qt and its strength Zt.
Importantly, the framework also offers a unifying perspectiveon existing fine-tuning methods and systematically designing new ones. Standard SFT corresponds to a degenerate projection with uniform strength, while DFT, InfoSFT, EAFT, and so on can be represented through different target-strength formulations within our framework.
W3: KL Measurement and Analysis
To directly validate our minimum-shift motivation, we conducted additional experiments measuring the reverse KL divergence DKL(πtuned∣πbase) on held-out medical-domain text, with all models fine-tuned on mathematical reasoning data. This metric quantifies how much the fine-tuned policy deviates from the pretrained distribution, particularly when assigning probability to outputs that were unlikely under the base model.
(1) Comparison at matched task performance.
We first trained an MFT model with mathematical reasoning performance comparable to SFT. Under this controlled comparison, MFT exhibits lower reverse KL divergence than SFT, indicating that MFT can achieve similar task adaptation with less disruption to the pretrained output distribution.
(2) Comparison with advanced baseline.
We further compared MFT against DFT. MFT achieves lower reverse KL divergence while outperforming both methods on in-domain mathematical reasoning and out-of-domain general tasks.
These results provide direct empirical support for our minimum-shift motivation, suggesting that MFT achieves a more favorable balance between acquiring new capabilities and preserving pretrained policy patterns.