acceptodds
Under review as a conference paper at ICLR 2027

A Unifying Lens on Fine-Tuning Through Target Distribution Design

Abstract

Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory. However, an observed token can be non-unique, noisy, or misaligned with the model prior. Strictly fitting toward this one-hot target may be suboptimal, especially when the pretrained model encodes a rich knowledge prior. In this work, we reinterpret SFT as target distribution design: instead of studying only the loss objective, we analyze the token-level target that a loss drives the model to match. We introduce the Target framework, which decomposes SFT supervision into two explicit choices: (1) how strongly to rely on the observed token, and (2) how to allocate the remaining probability mass over alternatives. This perspective unifies many existing SFT variants as implicit choices of the target distribution. Building on this view, we propose Target-SFT which directly constructs the desired target by adaptively balancing label imitation with a teacher-guided residual distribution. Across three datasets and seven models spanning math and medical reasoning, this method consistently outperforms all baselines, showing the effectiveness of this target-based approach. Overall, our formulation reveals a fundamental design perspective and opens a broader search space for training objectives.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-training, SFT

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ahoMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TARGET, a framework that reinterprets supervised fine-tuning as target distribution design. It decomposes token-level supervision into label imitation strength and alternative probability allocation. Based on this formulation, the authors introduce TARGET-SFT, which adaptively combines hard-label supervision with a teacher-guided soft distribution. The method demonstrates improvements across several mathematical and medical reasoning benchmarks.

Critical strengths
  1. The paper provides an intuitive perspective for understanding the relationships among different SFT objectives.
  2. The experiments are generally comprehensive.
Critical weaknesses
  1. This work is essentially an adaptive knowledge distillation method framed as a novel SFT framework. Its central objective combines hard-label supervision with teacher-guided soft targets, a well-established direction in the KD literature. In particular, the teacher–student geometric interpolation is mathematically equivalent to the logit interpolation employed in [1], raising concerns about the novelty of the proposed formulation. Furthermore, recent works [2–5] have explored closely related ideas in soft-target construction and adaptive token-level distillation, yet they are neither adequately discussed in the related work nor included as experimental baselines. These omissions make it difficult to assess the paper's novelty and empirical contributions relative to existing KD methods.

  2. The proposed decomposition of supervision into hard-label and residual distributions follows directly from the linearity of cross-entropy. Its conceptual novelty is therefore limited. I also find that the claim in Section 5.2 that any differentiable token-level loss can be represented by a target distribution through Eq. (6) is mathematically incorrect in general. The resulting vector is not guaranteed to be nonnegative or normalized.

[1] TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models. ICLR 2025

[2] Logits-Based Finetuning. EMNLP 2025

[3] Revisiting Knowledge Distillation for Autoregressive Language Models. ACL 2024

[4] LLM-Oriented Token-Adaptive Knowledge Distillation. AAAI 2026

[5] Entropy-Aware On-Policy Distillation of Language Models. ICML 2026

Questions for authors

What hyperparameter search was conducted for the baselines? Could the reported improvements be attributed to insufficient hyperparameter tuning of the competing methods rather than the proposed method itself?

2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer ahoM (1/2)

We sincerely thank the reviewer for their valuable feedback. We hope the following responses address the questions.


W1: Distillation Baselines & Novelty Clarification

We address this in three parts: (a) Distinction from TAID, (b) Comparison with distillation methods, and (c) Novelty of our framework, each supported by new experiments.

(a) The teacher–student geometric interpolation is mathematically equivalent to the logit interpolation employed in TAID

We thank the reviewer for pointing out the connection. The interpolation is similar, but our method differs in motivation and the role of this distribution. TAID addresses the teacher-student gap by using a temporally adaptive distribution as the full training target. In contrast, our work addresses the rigid label imitation in SFT, by explicitly allocating supervision and the distribution serves only as residual. We compare with TAID and other distillation methods directly (Table 1).

Importantly, our central contribution is the Target framework, rather than the particular interpolation in Target-SFT, which is one instantiation to show the design value of our framework. To clarify this, we further show that Target can systematically improve existing approaches beyond our own example instantiation (Table 2).

(b) Limited comparison with existing KD methods makes it difficult to assess the paper's novelty and empirical contributions.

We compare with 8 new distillation methods, including the suggested TAID, LFT, ATKD, AdaKD, and 4 additional objectives Veto, HPD, CSD, and AMiD. All methods followed their official implementations. Methods such as EOPD require on-policy student responses, which fall outside our setting and will be discussed in the revised related work.

Table 1: Comparison with distillation methods

MethodMinervaOlympiadAIME24AMC23Math500Avg.
Base8.2315.203.7518.1231.5214.92
Distillation
Distill19.8920.315.0126.7252.6525.32
LFT17.4217.023.1127.3445.5922.10
CSD16.9518.744.3725.6247.9822.73
HPD18.0221.256.6729.0651.2225.24
ATKD23.6623.975.8336.2559.7329.89
Veto23.5625.817.5135.7862.1030.95
TAID27.1827.258.1342.5064.4933.91
AdaKD29.9929.729.3844.0667.3036.09
AMiD31.4130.858.5545.0069.1036.98
Ours
Target-SFT33.0934.8413.1351.7272.3841.24

Target-SFT outperforms all new baselines, with 4.26% gains over the strongest baseline AMiD and 7.33% over TAID. These results establish our empirical strengths against recent distillation methods.

(c) [W2] The proposed decomposition follows directly from the linearity of cross-entropy.

We hope to clarify our contribution in the supervision-design principle: separating how strongly to imitate the label γt\gamma_t from how to allocate supervision among alternatives π~t\tilde{\pi}_t, and explicitly coordinating these two choices. This provides concrete guidance for future fine-tuning designs.

To show the value of this framework, we apply Target to existing distillation methods. This considers LFT, Veto, TAID, and AMiD, which are token-level distributions that can be directly used as the residual π~t\tilde{\pi}_t. It allows us to preserve their distribution-design choices under our supervision structure. Each original and Target-augmented pair uses identical hyperparameters reported in the original paper, to isolate the effect of our Target formulation.

Table 2: Improving existing methods with Target framework

MethodMinervaOlympiadAIME24AMC23Math500Avg.
Distill19.8920.315.0126.7252.6525.32
Distill + TARGET30.3230.738.5446.7268.7937.02 (+11.70)
LFT17.4217.023.1127.3445.5922.10
LFT + TARGET22.4823.674.7931.7257.2727.99 (+5.89)
Veto23.5625.817.5135.7862.1030.95
Veto + TARGET29.7631.019.5945.0068.9136.86 (+5.91)
TAID27.1827.258.1342.5064.4933.91
TAID + TARGET33.4032.7010.8344.0671.0938.42 (+4.51)
AMiD31.4130.858.5545.0069.1036.98
AMiD + TARGET34.5434.039.5748.4471.6939.65 (+2.67)

Target framework improves all existing methods with an average gain of 6.14%. Notably, TAID within Target shows substantial additional gains. This also extends to stronger methods such as AMiD, showing that the benefits are not restricted to weaker baselines or our specific residual design.

These results highlight an actionable framework for supervision design beyond algebraic decomposition. While linearity permits the decomposition, Target makes these choices and intuition explicit, providing a reusable structure for objective designs.

We appreciate the reviewer's constructive feedback, and this opportunity to present our contribution in a clearer and more extensive way.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer ahoM (2/2)

W2: The resulting vector in Section 5.2 is not guaranteed to be nonnegative or normalized.

We thank the reviewer for raising this detail. We agree that the loss-to-target intuition in Section 5.2 requires further specification. We provide a natural extension of our target view that resolves both issues, providing a cleaner interpretation than the current formulation.

Intuitively, when does a negative target entry arise: Assigning Qk=0Q_k=0 already gives the strongest suppression of token kk that cross-entropy towards a distribution can express. Therefore, Qk<0Q_k < 0 occurs when the loss applies stronger downward pressure on kk beyond ordinary probability matching.

Therefore, remaining centered on our target view, we express Q through two valid probability distributions, where an attractive target q+q^+ indicates token probabilities to encourage, and a repulsive target q−q^- indicates excess suppression. This repulsive branch thus collects the negative entries under Q. Formally, let m≥0m \ge 0 denote the total magnitude of negative entries in QQ, we can define Q=(1+m)q+−mq−,Q=(1+m)q^+-mq^-,

The gradient gg under original loss LL can therefore be represented exactly as ∇zL=∇z[(1+m)CE(q+,p)−mCE(q−,p)]\nabla_z L = \nabla_z \left[(1+m)\mathrm{CE}(q^+,p)-m\mathrm{CE}(q^-,p)\right]

=(1+m)(p−q+)−m(p−q−)= (1+m)(p-q^+) - m(p-q^-)

=p−Q=g.= p-Q = g.

This gives a more complete interpretation of the supervision target encoded by a loss, even when certain tokens are pushed harder than standard probability matching permits. When m=0m=0, it reduces to the current formulation.

To make our claims more precise, we will include in revision (1) a zero-sum gradient condition, which holds naturally for losses operating through softmax probabilities, and (2) this attractive-repulsive extension, which makes explicit the role of arbitrary token gradient in target designs. This change strengthens the general loss-to-target intuition in Section 5.2; our proposed Target framework and Target-SFT remain unaffected since they construct valid distributions by design. We appreciate the reviewer for highlighting this point!


Q1: Could the improvements be attributed to insufficient hyperparameter tuning of baselines rather than the proposed method itself?

The distillation baseline mixes label and teacher supervision with hyperameter weight cc, we report results on the tuning of c∈{0.2,0.4,0.6,0.8,1.0}c \in \{0.2, 0.4, 0.6, 0.8, 1.0 \}.

Datasetc=0.20.40.60.81.0
NuminaMath-CoT18.6522.9326.0631.2122.8
OpenR1-15k19.1320.4123.1025.5425.27

The best performance occurs at c=0.8c=0.8, the value already used in the main experiments. And Target-SFT remains well above it, with +7.75 on NuminaMath and +14.87 on OpenR1.

The other training configurations (learning rate, batch size, optimizer, etc) all followed the identical settings in prior works [1, 2], without tuning specifically for our method. We additionally sweep the learning rates over {5e-6, 1e-5, 2e-5, 5e-5, 1e-4, 2e-4, 5e-4} for all methods:

Method5e-61e-52e-55e-5*1e-42e-45e-4
NuminaMath-CoT
SFT9.9912.4815.1516.9117.1616.4311.70
SFT (p)22.9022.8826.7733.3530.4326.3510.11
Distill17.2423.9127.7631.2131.4131.2323.11
Target-SFT26.9831.8136.5738.9637.9836.0425.63
OpenR1-15k
SFT12.148.5710.5819.8422.9324.4123.05
SFT (p)15.2419.4234.0939.4418.5135.7310.87
Distill16.6718.6220.5725.5427.3829.2224.76
Target-SFT18.4026.5436.3640.4141.2439.5224.64

Target-SFT stays above all baselines at every learning rate, until at 5e-4 every method degrades.

References

[1] Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum
[2] On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification


We will include these discussions in the revised draft, and we thank the reviewer again for their helpful advice to improve our work.

Reviewer_gYnROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TARGET, a framework that interprets SFT through label-imitation strength and residual probability allocation. TARGET-SFT combines confidence-based label weighting with a teacher-guided residual distribution. Experiments span seven Qwen and LLaMA models across mathematical and medical reasoning, reporting improvements over SFT, probability-weighted SFT, and distillation baselines.

Critical strengths
  1. The framework offers an intuitive perspective on the supervision induced by different SFT objectives.

  2. Teacher guidance supplies corrective signals that confidence-based weighting alone may suppress.

  3. Experiments cover multiple model families and include useful component ablations and alternative distillation objectives.

Critical weaknesses
  1. The loss-to-target claim requires qualification. In Section 5.2, (Q=p-\nabla_z L) need not be a probability distribution. For example, (L=-2\log p_y) gives (Q=2\delta_y-p), with negative entries for non-reference tokens. Appendix B.3 discusses signed residuals, but does not establish the universal probability-target claim. The paper should state validity conditions and distinguish probability targets from detached, gradient-equivalent signed pseudo-targets.

  2. The Beta formulation does not establish the validity of the trust proxy. Setting (\alpha=p_y) and (\beta=1-p_y) yields (\gamma=p_y) by construction, without establishing its relationship to label reliability. Section 6 acknowledges the confidence–unfamiliarity confound, and Appendices F–G provide analyses and examples. Controlled experiments with known label corruption and correct but unfamiliar supervision would better validate the proxy and teacher fallback.

  3. The adopted residual needs a matching theoretical justification. Section 6 derives a residual proportional to (\pi_\theta(a)\pi_T(a)^\eta), then explicitly adopts the related form (\pi_\theta(a)^{1-\eta}\pi_T(a)^\eta). The latter is generally not the solution to the preceding objective. Providing an objective that yields the adopted form and comparing it with the originally derived solution would clarify this design choice.

  4. Comparisons and statistical validation remain incomplete. Appendix E adds SRKL, JS, and TVD distillation comparisons. However, comparisons with adaptive SFT methods discussed in the related work, such as ASFT or EAFT, would better establish the contribution. Variability across independent training seeds is also unreported, leaving the reliability of smaller improvements uncertain.

Questions for authors

While the empirical contribution is promising, stronger theoretical clarification and more comprehensive comparisons are needed to establish the significance of the proposed framework. Further details and supporting arguments are provided in the Weaknesses section.

2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer gYnR (1/2)

We sincerely thank the reviewer for their valuable feedback. We hope the following responses address the questions.


W1: The loss-to-target claim requires qualification. Appendix B.3 discusses signed residuals, but does not establish the universal probability-target claim.

We thank the reviewer for raising this detail. We agree that the loss-to-target intuition in Section 5.2 requires further specification. Following a similar idea outlined in Appendix B.3, we present a natural extension that makes the interpretation precise for general loss objectives.

Intuitively, when does a negative target entry arise: Assigning Qk=0Q_k=0 already gives the strongest suppression of token kk that cross-entropy towards a distribution can express. Therefore, Qk<0Q_k < 0 occurs when the loss applies stronger downward pressure on kk beyond ordinary probability matching.

Therefore, remaining centered on our target view, we express Q through two valid probability distributions, where an attractive target q+q^+ indicates token probabilities to encourage, and a repulsive target q−q^- indicates excess suppression. This repulsive branch thus collects the negative entries under Q. Formally, let m≥0m \ge 0 denote the total magnitude of negative entries in QQ, we can define Q=(1+m)q+−mq−,Q=(1+m)q^+-mq^-,

The gradient gg under original loss LL can therefore be represented exactly as ∇zL=∇z[(1+m)CE(q+,p)−mCE(q−,p)]\nabla_z L = \nabla_z \left[(1+m)\mathrm{CE}(q^+,p)-m\mathrm{CE}(q^-,p)\right]

=(1+m)(p−q+)−m(p−q−)= (1+m)(p-q^+) - m(p-q^-)

=p−Q=g.= p-Q = g.

This gives a complete interpretation of the supervision target encoded by a loss, even when certain tokens are pushed harder than standard probability matching permits. When m=0m=0, it reduces to the current formulation.

To make our claims more precise, we will include in revision (1) the zero-sum gradient condition, which holds naturally for losses operating through softmax probabilities, and (2) this attractive-repulsive extension, which covers losses whose gradients no single probability target can express. This change strengthens the general loss-to-target intuition in Section 5.2; our proposed Target framework and Target-SFT remain unaffected since they construct valid distributions by design. We appreciate the reviewer for highlighting this point!

W2: The Beta formulation does not establish the validity of the trust proxy. Controlled experiments with known label corruption and correct but unfamiliar supervision would better validate the proxy and teacher fallback.

Thank you for the question. We hope to clarify that the Beta formulation motivates a student-dependent proxy, rather than claiming that it measures label correctness. In fact, the teacher-guided residual in Target-SFT is designed to address this ambiguity by providing supervision when label imitation is weakened due to student unfamiliarity.

Following the reviewer's suggestion, we conduct controlled experiments under a dataset with label corruption. Specifically, we used Qwen3.5-9B to rewrite all training responses in OpenR1-15k to contain at least one genuine mathematical flaw while preserving the correct final answer.

MethodMinervaOlympiadAIME24AMC23Math500Avg.(Corrupt)Avg.(Clean)
Base9.5915.033.7518.4431.2515.61-
SFT8.978.511.8613.1228.0612.1119.98
SFT (p)29.4329.026.8746.2566.8035.6739.44
Distill19.9619.684.7832.1952.8925.9025.54
Target-SFT31.1131.028.9643.9169.1436.8340.41

Under this noisy supervision, Target-SFT still achieves the highest accuracy. Notably, standard SFT has a substantial 7.87% degradation from overfitting noisy reasoning, while Target-SFT drops only 3.58%. The advantage over SFT (p) also reinforces the effectiveness of our teacher fallback beyond simply weakening label imitation. These results provide controlled evidence for the robustness of our target-design approach, supporting the motivation for jointly designing label trust and residual supervision.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer gYnR (2/2)

W3: The adopted residual needs a matching theoretical justification. Providing an objective that yields the adopted form and comparing it with the originally derived solution would clarify this design choice.

Thank you for raising the question. We provide the principled derivation for the adopted residual below, which follows from our motivation in Section 6 to use a KL-regularized objective that preserves student prior and teacher fallback:

(1-\eta)\mathrm{KL}(q|\pi_\theta) +\eta\mathrm{KL}(q|\pi_T),$$ whose closed-form solution is our residual term $$\tilde{\pi}(a)\propto\pi_\theta(a)^{1-\eta}\pi_T(a)^\eta.$$ We will include this discussion to clarify our design choice beyond practical convenience in the revised draft. >**W4: Comparisons with adaptive SFT methods and variability across independent training seeds would better establish the contribution.** Following the reviewer's suggestions, we compare with 7 additional adaptive SFT methods (DFT, ASFT, iw-SFT, ProFit, EAFT, SFT+KL, Proximal SFT). All methods followed the official implementation of the works, and the identical training and evaluation configuration in our setting. |Method|Minerva|Olympiad|AIME24|AMC23|Math500|Avg.| |:-|-:|-:|-:|-:|-:|-:| |Base|8.23|15.20|3.75|18.12|31.52|14.92| |**SFT**||||||| |SFT|14.45|15.35|1.87|24.22|43.99|19.71| |SFT (*p*)|31.18|33.56|11.45|47.34|70.75|39.00| |*SFT + KL*|12.53|12.86|0.21|17.34|39.70|16.53| |*EAFT*|13.39|14.54|1.24|19.69|42.94|18.36| |*iw-SFT*|14.39|15.07|1.87|20.47|44.76|19.31| |*Proximal SFT*|16.69|17.53|2.28|26.72|51.69|22.98| |*ASFT*|26.77|30.23|11.26|44.06|64.96|35.46| |*ProFit*|30.24|31.04|11.25|49.06|69.05|38.13| |*DFT*|31.98|33.32|12.91|48.12|70.85|39.44| |**Ours**||||||| |**Target-SFT**|**33.09**|**34.84**|**13.13**|**51.72**|**72.38**|**41.24**| Target-SFT achieves the highest average accuracy of 41.24, outperforming the best new SFT baselines (DFT at 39.44). To measure variability across training seeds, we repeat the Qwen2.5-Math-1.5B runs of Table 1 with five seeds. Every performance improvement is statistically significant and consistent with the paper. |Method|0|1|2|3|4|Δ|*p*| |:-|-:|-:|-:|-:|-:|-:|-:| |**NuminaMath-CoT**|||||||| |SFT|16.91|18.16|17.18|17.57|16.72|+20.55 ± 0.41|6e-11| |SFT (*p*)|33.35|33.11|32.16|32.31|32.94|+5.08 ± 0.39|3e-6| |Distill|31.21|30.51|31.32|31.12|30.83|+6.86 ± 0.35|2e-6| |Target-SFT|**38.96**|**37.12**|**37.41**|**37.75**|**38.05**|-|-| |**OpenR1-15k**|||||||| |SFT|19.84|20.15|19.87|19.96|19.71|+20.44 ± 0.27|2e-8| |SFT (*p*)|39.44|39.87|39.09|39.56|39.00|+0.95 ± 0.30|0.017| |Distill|25.54|24.95|25.96|25.23|25.32|+14.94 ± 0.31|5e-10| |Target-SFT|**40.41**|**40.39**|**39.84**|**39.84**|**41.24**|-|-| --- We will include these discussions in the revised draft, and we thank the reviewer again for their helpful advice to improve our work.
Reviewer_9LapOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the TARGET framework, which interprets SFT as a process of designing a target distribution, thereby unifying existing methods under a single framework. Building on this, the paper introduces the TARGET-SFT method and demonstrates consistent performance improvements through experiments.

Critical strengths
  1. This method offers a useful perspective for understanding the objectives of SFT. I believe this unified, systematic analytical framework can provide guidance for the design of methods in this field.
  2. The paper proposes an effective method and demonstrates empirical improvements through experiments.
Critical weaknesses
  1. The universality claimed for this theoretical framework appears to exceed what is supported by the current arguments. Section 5.2 asserts that any differentiable token-level SFT loss function can be represented via an induced target distribution (defined as Qk=pk−∂L/∂zkQ_k = p_k - \partial L / \partial z_k). However, this construction does not necessarily yield a valid probability distribution, as there is no guarantee of non-negativity. As currently formulated, the approach seems applicable only to a restricted class of objective functions. Please correct me if I am mistaken.
  2. The empirical evidence fails to sufficiently establish the advantages of TARGET-SFT. The primary experiments compare TARGET-SFT only against relatively weak baselines. Several more advanced SFT objective functions, such as those mentioned in the Related Work section, are not included in the evaluation. For now, it is difficult to determine what specific advantages TARGET-SFT offers over these alternatives.
  3. The decoding settings used during the training process appear to be inconsistent. Could you provide further clarification or experiments regarding this?
  4. The experiments were conducted on two different types of GPUs (A6000 and H200), which to my knowledge have non-negligible impacts on performance. Could you specify which experiments were run on the A6000 and which were run on the H200?
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer 9Lap (1/2)

We sincerely thank the reviewer for their valuable feedback. We hope the following responses address the questions.


W1: Section 5.2 construction does not necessarily yield a valid probability distribution, as there is no guarantee of non-negativity.

We thank the reviewer for raising this detail. We agree that the induced target Q=p−gQ=p-g in Section 5.2 is not necessarily nonnegative for general loss objectives. This observation leads to a natural extension of our target-design view that resolves the issue.

Intuitively, when does a negative target entry arise: Assigning Qk=0Q_k=0 already gives the strongest suppression of token kk that cross-entropy towards a distribution can express. Therefore, Qk<0Q_k < 0 occurs when the loss applies stronger downward pressure on kk beyond ordinary probability matching.

Therefore, remaining centered on our target view, we express Q through two valid probability distributions, where an attractive target q+q^+ indicates token probabilities to encourage, and a repulsive target q−q^- indicates excess suppression. This repulsive branch thus collects the negative entries under Q. Formally, let m≥0m \ge 0 denote the total magnitude of negative entries in QQ, we can define Q=(1+m)q+−mq−,Q=(1+m)q^+-mq^-,

The gradient gg under original loss LL can therefore be represented exactly as ∇zL=∇z[(1+m)CE(q+,p)−mCE(q−,p)]\nabla_z L = \nabla_z \left[(1+m)\mathrm{CE}(q^+,p)-m\mathrm{CE}(q^-,p)\right]

=(1+m)(p−q+)−m(p−q−)= (1+m)(p-q^+) - m(p-q^-)

=p−Q=g.= p-Q = g.

This gives a complete interpretation of the supervision target encoded by a loss, even when certain tokens are pushed harder than standard probability matching permits. When m=0m=0, it reduces to the current formulation.

To make our claims more precise, we will include in revision (1) the zero-sum gradient condition, which holds naturally for losses operating through softmax probabilities, and (2) this attractive-repulsive extension, which captures the role of arbitrary token gradient in target designs.

This change strengthens the general loss-to-target intuition in Section 5.2; our proposed Target framework and Target-SFT remain unaffected since they construct valid distributions by design.Thank you again for highlighting this point; we appreciate the opportunity to improve our work!

AuthorsOpenReview ↗

Rebuttal Response to Reviewer 9Lap (2/2)

W2: The primary experiments compare TARGET-SFT only against relatively weak baselines.

To strengthen the evaluation of Target-SFT, we include 7 additional adaptive SFT baselines (DFT, ASFT, iw-SFT, ProFit, EAFT, SFT+KL, Proximal SFT) and 8 distillation baselines (LFT, CSD, HPD, ATKD, Veto, TAID, AdaKD, AMiD).

All methods followed the official implementation of the works. The experiments are conducted with Qwen2.5-Math-1.5B on the OpenR1-15k dataset, under the identical training and evaluation configurations in the main paper.

MethodMinervaOlympiadAIME24AMC23Math500Avg.
Base8.2315.203.7518.1231.5214.92
SFT
SFT14.4515.351.8724.2243.9919.71
SFT (p)31.1833.5611.4547.3470.7539.00
SFT + KL12.5312.860.2117.3439.7016.53
EAFT13.3914.541.2419.6942.9418.36
iw-SFT14.3915.071.8720.4744.7619.31
Proximal SFT16.6917.532.2826.7251.6922.98
ASFT26.7730.2311.2644.0664.9635.46
ProFit30.2431.0411.2549.0669.0538.13
DFT31.9833.3212.9148.1270.8539.44
Distillation
Distill19.8920.315.0126.7252.6525.32
LFT17.4217.023.1127.3445.5922.10
CSD16.9518.744.3725.6247.9822.73
HPD18.0221.256.6729.0651.2225.24
ATKD23.6623.975.8336.2559.7329.89
Veto23.5625.817.5135.7862.1030.95
TAID27.1827.258.1342.5064.4933.91
AdaKD29.9929.729.3844.0667.3036.09
AMiD31.4130.858.5545.0069.1036.98
Ours
Target-SFT33.0934.8413.1351.7272.3841.24

Target-SFT achieves the highest average accuracy of 41.24, outperforming the strongest SFT and distillation baselines (DFT at 39.44 and AMiD at 36.98, respectively).

W3: Could you provide further clarification on decoding settings?

All settings decode at temperature 1.0 except Qwen2.5-Math-7B trained on OpenR1-15k. In this exception, we observed occasional nonsensical generations from both the distillation baselines and Target-SFT. The community has reported similar failure, which may be related to the flatter output distribution of the Qwen2.5-Math-7B-Instruct teacher. We thus followed Qwen's official recommendations to use a temperature of T=0.7T=0.7. This choice improved all baseline performance, thus the evaluations for this 7B model on OpenR1 remain fair. We will clarify this in Appendix A.

W4: Could you specify which experiments were run on the A6000 and which were run on the H200?

All experiments with 1.5B models (Qwen2.5-Math-1.5B, Qwen2.5-1.5B) were conducted on A6000, and all experiments with larger models (1.7B, 3B, 7B, 8B) on H200. Thus all methods within a model-dataset setting ran on the same device, to ensure fair comparison and prevent differences from the hardware-side. We will include this detail in Appendix A.


We will include these discussions in the revised draft, and we thank the reviewer again for their helpful advice to improve our work.

Thank you for the additional experiments, which address some of my concerns. However, I still lean towards a weak reject at this point, and I encourage the authors to further strengthen the paper in future revisions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank the Area Chair and all reviewers for your time and thoughtful feedback. To help with the evaluation of our work, we summarize the key updates from the rebuttal.


1 - Summary of Contribution

Our work views SFT through its target distribution: the probabilities that training encourages the model to match. This perspective makes two designs in SFT supervision explicit: how strongly to imitate the observed label, and how to allocate the remaining probability mass. The Target framework uses these two choices to explain existing objectives and guide new designs.

Target-SFT is a concrete instantiation of this principle, by using the model's confidence to adjust label imitation and a teacher-guided distribution to assign remaining mass. It outperforms every baseline in all evaluated settings, showing the practical value of jointly designing the two parts of supervision. New experiment (b) further shows that our Target structure improves existing methods systematically, extending its value beyond Target-SFT.


2 - New Experiments

  • (a) We compare Target-SFT with stronger baselines (9Lap, gYnR, ahoM), including 7 recent SFT and 8 distillation methods [1-7, 8-15]. Target-SFT achieves the highest accuracy, outperforming the best new SFT and distillation baselines (41.24 vs. 39.44 and 36.98, respectively).

  • (b) To show the value of Target framework (ahoM), we apply Target's supervision structure to 5 existing distillation methods. This improves all methods (average gains of 6.14), suggesting that the Target structure can systematically improve existing approaches. This reinforces our key contribution in exposing the explicit structure in label imitation strength and residual probability allocation.

All new experiments align with findings in paper. This reinforces our contribution and the evaluation against existing works, including related work TAID [13] raised by reviewer ahoM.


3 - Key Clarification

In response to concerns about the “Loss-to-Target” intuition in Section 5.2 (9Lap, gYnR, ahoM), we will state the zero-sum gradient condition, which holds for losses operating through softmax probabilities. And we use an attractive-repulsive decomposition for induced targets with negative entries, giving a clean interpretation while preserving our target-based view.

This makes our universality claim precise; the Target framework and Target-SFT are unchanged since they construct valid probability distributions by design.


We greatly appreciate the opportunity to strengthen our work. We hope this is useful in summarizing all the resolved concerns, and key additions during rebuttal.

AuthorsOpenReview ↗

New Baseline References

References

[1] SFT+KL RL's Razor: Why Online Reinforcement Learning Forgets Less
[2] EAFT Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
[3] iw-SFT Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)
[4] Proximal SFT Proximal Supervised Fine-Tuning
[5] ASFT Anchored Supervised Fine-Tuning
[6] ProFit ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection
[7] DFT On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

[8] LFT Logits-Based Finetuning
[9] CSD Distillation of Large Language Models via Concrete Score Matching
[10] HPD Hybrid Policy Distillation for LLMs
[11] ATKD Revisiting Knowledge Distillation for Autoregressive Language Models
[12] Veto Stable On-Policy Distillation through Adaptive Target Reformulation
[13] TAID TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
[14] AdaKD LLM-Oriented Token-Adaptive Knowledge Distillation
[15] AMiD AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distribution

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.