acceptodds
Under review as a conference paper at ICLR 2027

Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectives

Abstract

Standard negative log-likelihood (NLL) for Supervised Fine-Tuning (SFT) applies uniform token-level weighting. This rigidity creates a two-fold failure mode: (i) overemphasizing low-probability targets can amplify gradients on noisy supervision and disrupt robust priors, and (ii) uniform weighting provides weak sharpening when the model is already confident. Existing methods fail to resolve the resulting plasticity–stability dilemma, often suppressing necessary learning signals alongside harmful ones. To address this issue, we unify token-level SFT objectives within a generalized deformed-log family and show that they all share a gate x error gradient structure, where the gate controls how much the model trusts its current prediction. We further propose two gate designs according to the predictive information they consume–probability-level and distribution-level. At the probability level, by employing the Cayley transform, we map the model's continuously evolving uncertainty onto a continuous focus trajectory, enabling a seamless interpolation between uncertain novel concepts and well-established knowledge. At the distribution level, we propose Dynamic Entropy Fine-Tuning (DEFT), which gates on the concentration of the predictive distribution to distinguish diffuse uncertainty from concentrated disagreement. % Extensive experiments and analyses demonstrate that both Cayley-Trans and DEFT achieve a better balance between exploration and exploitation, leading to improved overall performance, with DEFT—guided by distribution-level information—consistently delivering the strongest gains. Across seven backbones and three capability regimes, both methods improve over the standard NLL objective without per-task tuning. DEFT delivers the strongest gains, raising the regime-averaged Overall score from 30.58 to 40.59 (+10.01), suggesting that distribution-level information is a substantive and underexplored dimension of token-level SFT.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Supervised Fine-Tuning, Generalized Entropic Objectives

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_rPKvOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates token-level gradient allocation in supervised fine-tuning through a generalized cross-entropy framework. The authors characterize the gradient of probability-based SFT objectives as a product of a confidence-dependent gate and the standard prediction error. Building on this interpretation, they introduce Cayley-Trans, which adaptively adjusts the loss exponent using the ground-truth token probability, and DEFT, which incorporates the Rényi-2 concentration of the full predictive distribution. Experiments across mathematical reasoning, medical reasoning, and synthetic weak-prior tasks demonstrate encouraging improvements over several existing SFT objectives.

Critical strengths
  1. The paper studies a relevant problem in SFT objective design and provides a useful perspective on confidence-dependent gradient allocation.
  2. The proposed methods are computationally lightweight and do not require additional trainable parameters or reference models.
  3. The experimental settings and results are generally comprehensive.
Critical weaknesses
  1. The paper argues that predictive concentration helps distinguish harmful knowledge conflicts from valuable new supervision. However, a low-probability, low-entropy prediction does not necessarily indicate noisy or undesirable supervision. It may equally reflect an incorrect but highly confident model prediction that should be corrected. Maybe designing synthetic experiments could help.
  2. The novelty is a little limited. Previous methods such as EAFT and RankTuner have already explored distribution-aware token weighting to distinguish uncertain predictions from confident conflicts. What is the fundamental advantage of the proposed approach?
  3. Section J.4 seems to be AI-generated. The authors are encouraged to revise and provide fully reproducible details.
Questions for authors

I see in Table 2 the results for Qwen2.5-Math-7B; the results are nearly identical. Is there a particular interpretation?

3 comments

W1. Correcting confident but wrong predictions

Thank you for pointing out this potential trade-off. When the probability of a correct supervised target is low and the predictive distribution is highly concentrated, DEFT does reduce the gradient at that position relative to NLL. To quantify how this mechanism affects actual correction, we have added a diagnostic experiment based on verified correct supervision labels.

To further examine whether DEFT can correct confident but wrong priors, we run a forward pass with the fixed base model over all 67,601 mathematical training examples. We treat the label tokens inside the final answer's \boxed{...} span as fully correct ground-truth tokens; this excludes tokens in the reasoning process. Before fine-tuning, we identify tokens with reference probability p≤0.1p\le0.1 and Shannon entropy H≤0.3H\le0.3, as well as those satisfying p≤0.3p\le0.3 and H≤0.5H\le0.5, as tokens reflecting confident but wrong priors (CBW tokens). Using NLL as the standard comparison, we measure the proportion of these CBW tokens whose reference token becomes the top-1 prediction after training with NLL or DEFT.

Metric definition. The correction rate is the proportion of tokens identified as CBW before fine-tuning whose reference token becomes the top-1 prediction after training. The NLL and DEFT correction rates below are computed on the same CBW set. The relative correction rate is

RelativeCorrectionDEFT=CorrectionRateDEFTCorrectionRateNLL×100%RelativeCorrection_{DEFT} = \frac{CorrectionRate_{DEFT}}{CorrectionRate_{NLL}} \times 100\%

ModelSettingCBW countOverall proportionNLL correction rateDEFT correction rateRelative correction rate
LLaMA-3.1-8Bp ≤ 0.10, H ≤ 0.384,7750.3088%14.02%11.10%79.17%
LLaMA-3.1-8Bp ≤ 0.30, H ≤ 0.5175,4560.6391%15.15%12.34%81.43%
Qwen2.5-Math-7Bp ≤ 0.10, H ≤ 0.3177,4030.6236%17.05%14.12%82.79%
Qwen2.5-Math-7Bp ≤ 0.30, H ≤ 0.5309,9291.0895%17.05%14.46%84.79%

The results show that DEFT achieves approximately 79%–85% of NLL's correction rate, demonstrating that DEFT retains the ability to correct the model's confident but wrong priors.

The training trajectories below further show that tokens at initially confident but wrong positions can be corrected during DEFT training. Thus, initially suppressed positions remain learnable. This observation is consistent with the dynamic nature of the gate: DEFT recomputes the weights from the current predictive state, and being initially classified as a conflict does not permanently mask a position. Table 5 in the paper provides complementary evidence: restoring NLL weights in the dynamically identified conflict region reduces overall task performance. Taken together, these results show that DEFT improves overall adaptation performance while retaining the ability to correct initially conflicting positions.

The proportion (%) of CBW tokens whose reference token becomes the top-1 prediction during training under the Strict setting (p≤0.1p\le0.1, H≤0.3H\le0.3 bits) is shown below:

StepLLaMA NLLLLaMA DEFTQwen NLLQwen DEFT
00.00%0.00%0.00%0.00%
338.91%6.30%12.99%11.20%
6611.13%9.29%14.92%12.50%
9912.86%9.60%16.31%13.57%
13213.71%10.06%16.87%14.10%
16513.93%10.02%17.00%14.17%
19814.14%10.83%17.06%14.32%
23114.36%10.94%17.22%14.37%
26414.02%11.10%17.05%14.46%

W2. Fundamental advantages over EAFT and RankTuner

Thank you for emphasizing this comparison. EAFT and RankTuner already use distributional information.

Below, we explain the rationale for this choice, its mechanistic differences from existing methods, and the empirical results.

For the generalized loss La(p)=(1−pa)/aL_a(p)=(1-p^a)/a, let rr denote the supervision-label distribution for a fixed context, fix 0<a<10<a<1, and write β=1/(1−a)\beta=1/(1-a) and Zβ=∑iriβZ_\beta=\sum_i r_i^\beta. Theorem 1/2 gives the optimal risk as

=\frac{1-Z_\beta^{1/\beta}}a =\frac{1-e^{-aH_\beta(r)}}a =L_a\!\left(e^{-H_\beta(r)}\right).$$ Thus, Rényi entropy characterizes the optimal supervised risk of the original loss, while $e^{-H_\beta(r)}$ is the equivalent probability that produces the same scalar loss. Rényi-2 corresponds exactly to the distributional statistic in the gradient of the original loss. Let $C_t=\sum_vP_t(v)^2$. Then $\lVert\nabla_{z_t}L_a\rVert_2^2=p_t^{2a}\lVert P_t-e_{y_t}\rVert_2^2=p_t^{2a}(1-2p_t+C_t)$ Therefore, $(p_t,C_t)$ exactly recovers the squared Euclidean length of the NLL correction vector. Within the power-sum candidate family $T_q=\sum_vP_t(v)^q$ for finite orders $q>1$, when the vocabulary size is at least $4$, only $q=2$ provides this exact scalar representation for every interior predictive distribution. For other orders, there exist predictions with the same $p_t,T_q$ but different $C_t$, so the length cannot be recovered from these two inputs.

W2 (continue)

DEFT directly uses the second-order concentration as the exponent and stops its gradient according to the convention in Section 4.3:

G_t^{\mathrm{DEFT}}=p_t^{C_t},\qquad p_t\le G_t^{\mathrm{DEFT}}\le1.$$ The exponent can change with the model state at each step but is held fixed during differentiation, so it does not change the NLL gradient direction for an individual token. At fixed $p_t$, $\partial\ln G_t^{\mathrm{DEFT}}/\partial C_t=\ln p_t<0$: the more concentrated the prediction, the smaller the token's gradient weight. This provides an explicit rationale for both the second-order statistic and the direction of its weighting effect. DEFT therefore considers two factors simultaneously: how concentrated the model's prediction is, and whether the model supports the supervised answer. Specifically, $$\nabla_{z_t} L_{DEFT} = p_t^{C_t} (P_t - e_{y_t})$$ We next compare how each method determines this token weight. To maintain the same gradient interpretation, all weights below are discussed under the convention that they are held fixed during differentiation. **Difference from EAFT: distinguishing agreement from conflict with the supervised answer at low entropy.** In its normalized Top-$K$ Shannon-entropy weighting form, EAFT uses $$L_t^{\mathrm{EAFT}}=\widetilde H_t(-\log p_t),\qquad G_t^{\mathrm{EAFT}}=\widetilde H_t.$$ As the entropy used by EAFT approaches zero, its entropy gate approaches zero for both low-probability conflicting tokens and high-probability mastered tokens. DEFT also attenuates gradients for both types of tokens, since $0<p_t^{C_t}<1$ when $0<p_t<1$. The difference is that it also distinguishes the degree of attenuation using the probability of the supervised answer. At the same concentration, a low-probability conflicting label receives a smaller weight and stronger attenuation, whereas a high-probability agreeing label receives a weight close to one and weaker attenuation. Thus, for the same low-entropy predictive distribution, EAFT uses the same entropy multiplier for different supervised labels, while DEFT further adjusts the weight according to the label probability. **Difference from RankTuner: the basis for weighting and consistency across tasks.** RankTuner's relative scaling factor combines the supervised label's rank $R_t$ with the expected rank: $$S_t=2^{f(\mathbb E[R_t])-f(R_t)},\qquad f(x)=\frac1{\log_2(x+1)},\qquad G_t^{\mathrm{RankTuner}}=b_tS_t.$$ Here, $b_t$ is the base weight and $S_t$ is the rank adjustment. When $R_t>\mathbb E[R_t]$, the strictly decreasing function $f$ gives $S_t>1$. RankTuner therefore increases the token's contribution relative to its base weight, using an actual rank worse than expected as the basis for assigning greater corrective weight. The two methods differ particularly in how they allocate learning signals in the low-$p_t$, low-entropy region. If the model strongly favors a non-target label and the supervised label's rank is substantially worse than expected, i.e., $R_t\gg\mathbb E[R_t]$, RankTuner increases its Relative Scale. It interprets this as a critical error in which the model underperforms at a position that should be relatively clear. From the perspective of prior preservation, DEFT instead treats this state as a potential confident conflict: higher concentration moves the gate toward the linear endpoint $p_t$, reducing the token-level gradient weight of that supervision signal to limit how strongly external supervision rewrites an existing concentrated prediction. Thus, the two methods differ in both their statistics and functional forms, emphasizing corrective compensation relative to predictive difficulty and conflict suppression controlled by predictive concentration, respectively. **Another difference is whether the base weight changes across tasks.** In our comparison configurations, RankTuner uses RankTuner-p for mathematical training, with $b_t=p_t$ and total weight $p_tS_t$, and RankTuner-one for medical and FigFont training, with $b_t=1$ and total weight $S_t$. RankTuner therefore uses different base-weighting rules depending on the training task in these experiments. DEFT uses $C_t=\sum_vP_t(v)^2$ and $G_t=p_t^{C_t}$ uniformly across these tasks, without switching between $p_t$ weighting and unit weighting by task. Table 1 in the main text directly compares these methods. The average mathematical scores of DEFT and RankTuner are 11.04 and 10.78 on LLaMA, and 36.73 and 36.01 on Qwen, respectively. We further report per-benchmark results after training Qwen2.5-Math-7B-Instruct on mathematical data, covering both in-domain (IND) mathematics and out-of-domain (OOD) medical performance.

W2 (continue)

In-domain mathematics (IND)

MethodMATH500MinervaOlympiadBenchAIME24AMC23Avg
Initial Instruct55.9523.7121.742.2936.7228.08
NLL49.91 ± 0.4713.92 ± 0.4016.54 ± 0.271.62 ± 0.3120.59 ± 1.7420.52 ± 0.32
EAFT50.17 ± 0.4015.23 ± 0.4416.63 ± 0.311.37 ± 0.6021.94 ± 0.9221.07 ± 0.29
InfoSFT62.31 ± 0.1924.55 ± 0.1826.26 ± 0.393.36 ± 1.0032.94 ± 0.9929.89 ± 0.31
RankTuner-p69.33 ± 0.4129.30 ± 0.7531.99 ± 0.235.16 ± 1.0341.35 ± 0.2835.42 ± 0.35
DEFT70.35 ± 0.0829.54 ± 0.5532.31 ± 0.084.58 ± 1.1342.07 ± 1.2235.77 ± 0.10

Out-of-domain medicine (OOD)

MethodMedMCMedQAPubMedMMLU-PGPQALancetMedB(4)MedB(5)MedXNEJMAvg
Initial Instruct33.6429.9370.9034.5335.3832.7728.2524.0311.1123.8832.44
NLL31.82 ± 0.4029.73 ± 0.7169.32 ± 0.5032.12 ± 0.4634.67 ± 3.0232.77 ± 0.9628.76 ± 2.3222.66 ± 1.4110.05 ± 0.3024.77 ± 1.5931.67 ± 0.36
EAFT31.54 ± 0.3730.76 ± 0.7368.66 ± 0.2631.65 ± 0.7431.64 ± 3.0331.02 ± 0.9327.01 ± 1.9124.67 ± 1.1310.56 ± 0.7325.97 ± 0.5531.35 ± 0.29
InfoSFT32.69 ± 0.3629.70 ± 1.2367.92 ± 0.9933.24 ± 0.2632.15 ± 1.5931.94 ± 0.8826.49 ± 1.5023.90 ± 1.399.73 ± 0.3825.40 ± 1.2231.32 ± 0.29
RankTuner-p33.60 ± 0.2032.24 ± 0.6667.46 ± 0.8332.70 ± 0.3736.05 ± 2.7732.62 ± 1.2229.48 ± 1.2020.52 ± 1.2010.44 ± 0.5625.67 ± 1.2332.08 ± 0.38
DEFT32.34 ± 0.2432.18 ± 0.7067.20 ± 0.8531.53 ± 0.5236.67 ± 0.9331.96 ± 0.5630.84 ± 0.5723.27 ± 0.4910.17 ± 1.0525.20 ± 2.5732.14 ± 0.11

The results show that DEFT achieves greater improvements in IND performance while better preserving the model's existing capabilities, achieving a plasticity–stability balance.

W3. Presentation and reproducibility details in Section J.4

Thank you for pointing this out. In the revised manuscript, we have added detailed experimental settings to the sections “Experimental Setup” and “Experimental Records and Reproducibility” to ensure that the experiments are reproducible.

Q1. Similar medical results for Qwen2.5-Math-7B in Table 2

Thank you for highlighting this observation. Table 2 concerns medical reasoning, which belongs to the Model-Intermediate (MI) regime in our setting. Your observation is consistent with the findings reported in Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum: in the intermediate regime, where the model has partial task priors, performance gaps between objectives narrow, and neither strategies that emphasize existing priors nor those that strengthen new-knowledge learning show a consistently dominant advantage.

Specifically, the average scores of the methods on Qwen2.5-Math-7B in Table 2 range from 34.27 to 35.05, a span of 0.78 percentage points. DEFT achieves an average score of 35.05, improving on NLL's 34.29 by 0.76 percentage points. This is consistent with the interpretation that an objective's effectiveness depends on the alignment between model priors and the task; it does not imply that improvements should be equally large in every setting.

Reviewer_H5VgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces two new token level loss weighting schemes to improve the training dynamics during SFT of pretrained language models. Starting with a qualitative identification of that standard NLL loss will e.g. often forget predictions that are initially high confidence after pretraining, the manuscript parametrises a family of weighting schemes through a deformed log. Two new weightings, one using target class probability (Cayley transform) and one using the full predictive distribution (Reyni entropy), are introduced and evaluated against other baseline loss weighting strategies. In particular their DEFT weighting (exponential transform of the Reyni entropy) shows good performance across all evaluation domains.

Critical strengths

The proposed method is practically viable and shows robust performance increase without introducing additional hyperparameters.

The manuscript evaluates and ablates both important architecture choices and target datasets across spectrum of difficulties and domains.

It targets a relevant and pressing questions regarding the training dynamics of LLMs.

Critical weaknesses

Although the paper does a number of ablations, there are some critical directions left open in terms of isolating the contribution of the new weighting. In particular I was looking for an ablation using the same gating mechanism but using the Shannon entropy instead. Given EAFTs somewhat positive results in comparison, it seems relevant to keep the standard Shannon entropy as a baseline for any further transforms.

The theory, although appreciated, seems to have some gaps: For example I can’t find a proof of Theorem 1 in the referenced appendix. Rather, there is a theorem 2 with some overlap. I must admit that I haven’t completely grasped if Theorem 2 implies Theorem 1, and if so I encourage the authors to clarify the connection.

I was also surprised that there are no further justification or ablations of the Renyi degree, why is order 2 the right choice? It doesn’t seem to follow directly from Theorem 1.

Although most of the paper is easy to follow, there are also multiple point where the reader is worried about the human supervision of the text and statements. Some examples include uses of unnecessary terms like “load-bearing” (line 413), “triptych” for describing three ablation directions without any other mention of this term (line 417). I would encourage simpler, more direct language in these statements.

“All code and evaluation scripts will be made publicly available in an anonymous repository to facilitate replication and verification.” Where can I find this? I don’t see any links or supplemental in open review.

Questions for authors

“The parameter α thus acts as a focus index” (“16266_Gradients_Must_Earn_Thei”, p. 4) This is the first definition of focus index? The reader would benefit from a clearer presentation when used earlier in the manuscript.

“Probability-level trust gating: the gate depends only on the supervised-target probability p” (“16266_Gradients_Must_Earn_Thei”, p. 5) Am I understanding this correctly that the probability level trust gating is using the explicit target class probability which under normal training would simply be 1 since the target token is one-hot encoded?

“With natural logarithms, H2(P ) = − log P v Pv2.” (“16266_Gradients_Must_Earn_Thei”, p. 5) Why not write explicitly that this is the Renyi entropy here.

“test of plasticity” (“16266_Gradients_Must_Earn_Thei”, p. 8) Have you evaluated plasticity compared to second order optimizers?

“This supports freezing concentration” (“16266_Gradients_Must_Earn_Thei”, p. 8) Would be great to remind the reader on where in the manuscript they can read on the distinction between frozen/unfrozen concentration in terms of the stop gradients I assume?

“This indicates that the gate on these predictive states is load-bearing rather than incidental;” (“16266_Gradients_Must_Earn_Thei”, p. 8) I encourage the authors to rephrase statements like this to ease the burden of the reader wondering if the text has had human supervision.

“Analysis of the ablation triptych” (“16266_Gradients_Must_Earn_Thei”, p. 8) The reader is again wondering if this has had human supervision. Triptych is, i assume referring to three ablations directions, but this does not help the reader in any meaningful way.

“All code and evaluation scripts will be made publicly available in an anonymous repository to facilitate replication and verification.” (“16266_Gradients_Must_Earn_Thei”, p. 16) Where can I find this? I dont see any links or supplemental in open review.

“Proposition 7” (“16266_Gradients_Must_Earn_Thei”, p. 25) Why is this a proposition? Isnt this just the definition of \alpha_DEFT as given on line 262.

5 comments

W1. Shannon entropy ablation

Thank you for this constructive suggestion. We have added a control experiment that directly uses Shannon entropy as the gating exponent. The results are as follows:

ModelMethodMinervaOlympiadAIME24AMC23MATH500Avg
LLaMA-3.1-8BShannon entropy9.145.590.219.0623.149.43
LLaMA-3.1-8BDEFT9.886.270.6912.2426.1511.04
Qwen2.5-Math-7BShannon entropy29.8129.045.6236.4163.6932.91
Qwen2.5-Math-7BDEFT32.9532.227.8741.8868.7236.73

W2. Relationship between Theorems 1 and 2

Thank you for pointing this out. Theorem 2 in Appendix G.2 restates Theorem 1 and provides its proof, including the boundary cases; it is not a separate result that independently derives DEFT. We will clarify this numbering issue and the respective roles of the different distributions and exponents.

W3. Justification and ablation of the Rényi order

Thank you for raising this question. Our motivation for considering Rényi entropy first comes from the generalized loss itself: its optimal supervised risk can be expressed as a function of Rényi entropy.

The generalized loss is

\qquad \ell_0(p)=-\ln p.$$ Its gradients are $$\frac{\partial\ell_a}{\partial p}=-p^{a-1},\qquad \nabla_zp=p(e_y-P),\qquad \nabla_z\ell_a=p^a(P-e_y).$$ Thus, the exponent controls a scalar multiplier of the NLL gradient. For $0\le a\le1$, the multiplier $p^a$ lies in $[p,1]$: $a=0$ corresponds to NLL, and $a=1$ corresponds to the linear probability loss. At a fixed predictive state, increasing the exponent strengthens attenuation without changing the token's gradient direction. Stop-gradient means holding the exponent fixed during the current differentiation; its value can still change across training steps. Let $r$ denote the supervision distribution for a fixed context, and use a common, fixed exponent $0<a<1$ for all candidate labels. Theorems 1 and 2 characterize the minimum of the following risk over the closed probability simplex $\Delta_K$: $$R_a(P;r)=\sum_i r_i\ell_a(P_i) =\frac{1-\sum_i r_iP_i^a}{a},\qquad P\in\Delta_K.$$ Minimizing this risk is equivalent to maximizing $\sum_i r_iP_i^a$. An optimum assigns no mass outside the positive-probability support of $r$, because moving such mass to a supervised coordinate increases this sum. On this support, the objective is strictly concave, and its derivative diverges as a coordinate approaches zero; consequently, all coordinates of the optimum on this support are positive. The stationarity conditions under the normalization constraint give $$r_i(P_i^*)^{a-1}=\lambda,\qquad P_i^*=\frac{r_i^\beta}{Z_\beta},\qquad \beta=\frac1{1-a},\qquad Z_\beta=\sum_jr_j^\beta,$$ with zero coordinates outside the support. Strict concavity guarantees that this is the unique optimum. Since $1+a\beta=\beta$, substituting it into the risk yields $$\sum_i r_i(P_i^*)^a =Z_\beta^{-a}\sum_i r_i^{1+a\beta} =Z_\beta^{1-a}=Z_\beta^{1/\beta}, \qquad B_a(r):=\min_P R_a(P;r) =\frac{1-Z_\beta^{1/\beta}}a.$$ Now consider the Rényi entropy $H_\beta(r)=\ln Z_\beta/(1-\beta)$. Because $(1-\beta)/\beta=-a$, the same optimal value can be written as $$B_a(r)=\frac{1-e^{-aH_\beta(r)}}a =\ell_a\!\left(e^{-H_\beta(r)}\right).$$ In the NLL limit $a\to0$, the order satisfies $\beta\to1$, and the minimum risk becomes Shannon entropy, namely $B_0(r)=H_1(r)$. The Rényi expression extends this familiar loss–entropy relationship to nonzero power exponents. The scalar loss $\ell_a$ is strictly decreasing in its probability argument. Therefore, there is a unique probability that produces the same loss as the optimal supervised risk: $$\ell_a^{-1}\!\left(B_a(r)\right) =\left(1-aB_a(r)\right)^{1/a} =Z_\beta^{1/(\beta-1)} =e^{-H_\beta(r)}.$$ The same loss family therefore explains both the Rényi quantity and its exponential mapping: they describe the optimal supervised risk and the probability matching that risk, respectively. This equivalent probability summarizes the scalar risk; it is not a coordinate of the optimal prediction. Other monotonic entropy representations can also express the risk, while the Rényi form provides the direct exponential identity above. This Bayes-risk derivation explains why we consider Rényi entropy and its exponential mapping. When constructing the adaptive method, we apply this concentration form to the model's current predictive distribution $P$ and consider $c_q(P)=e^{-H_q(P)}$ ($q>1$) as a candidate focus index. It can be interpreted as the reciprocal of the effective support size and takes values in $[1/K,1]$, consistent with the exponent range of the generalized loss. For the logit gradient under standard Euclidean geometry, any frozen $a$ satisfies $$\|\nabla_z\ell_a\|_2^2 =p^{2a}E(P,y),\qquad E(P,y):=\|P-e_y\|_2^2 =1-2p+C(P),\qquad C(P)=\sum_iP_i^2.$$

W3.(continue)

Given pp, the single statistic C(P)C(P) exactly characterizes the squared norm of the correction vector; specifying the exponent then determines the squared gradient norm.

This provides a concrete criterion for comparing entropy orders: whether the corresponding summary retains the magnitude information already present in the signal being scaled. This criterion arises from a representation question about the original loss, rather than introducing a new objective claimed to optimize downstream performance.

We formalize this criterion as an exact scalar representation property. Let KK be the vocabulary size and Tq(P)=∑iPiqT_q(P)=\sum_iP_i^q. The vocabularies of the LLMs considered here are much larger than 44, so K≥4K\ge4. Under this condition, for finite q>1q>1, there exists a decoder satisfying

E(P,y)=D ⁣(p,Tq(P))E(P,y)=\mathcal D\!\left(p,T_q(P)\right)

for every interior predictive distribution (with all components strictly positive) if and only if q=2q=2. Sufficiency follows directly from D(p,t)=1−2p+t\mathcal D(p,t)=1-2p+t. The proposition allows an arbitrary decoder, without requiring continuity or differentiability.

Proof of necessity. Fix pp and all coordinates except three positive tail coordinates, denoted by x,u,t=s−x−ux,u,t=s-x-u, with fixed total mass ss. The corresponding Jacobian determinant is

=2q(x-t)(u-t) \left[ \frac{x^{q-1}-t^{q-1}}{x-t} -\frac{u^{q-1}-t^{q-1}}{u-t} \right].$$ Choose $0<t<u<x$. The function $v^{q-1}$ is strictly convex for $q>2$ and strictly concave for $1<q<2$. The two secant slopes therefore differ, so the determinant is nonzero. By the inverse function theorem, the local image of $(T_q,C)$ contains an open set. Hence, there exist two predictive distributions with the same $p,T_q$ but different $C$, and therefore different $E=1-2p+C$. No decoder can map the same input to these two different values. Since $c_q=T_q^{1/(q-1)}$ is an invertible transformation, the conclusion also applies to the candidate concentration $c_q$. **From the second-order statistic to the DEFT exponent.** At order two, $c_2(P)=e^{-H_2(P)}=C(P)$ is also the probability that two independent samples from the model coincide. DEFT directly uses $a_{\mathrm D}=C(P)$ as its exponent. The proof above establishes its role in exactly representing Euclidean gradient magnitude information within the candidate family considered. We also compared different orders experimentally (In the order-ablation table, the label $\beta$ denotes the Rényi order $q$ used for $c_q=e^{-H_q}$) : |Method|Minerva|Olympiad|AIME24|AMC23|MATH500|Avg| |---|---:|---:|---:|---:|---:|---:| |NLL|6.35|3.45|0.00|6.56|17.30|6.73| |β=1|9.80|6.12|0.00|10.31|26.09|10.46| |β=1.5|8.56|6.73|1.03|10.62|26.11|10.61| |β=2|9.88|6.27|0.69|12.24|26.15|11.05| |β=inf|10.27|5.79|0.21|10.16|25.36|10.36| ### W4. Clarity of the writing Thank you for these careful comments. We have corrected these inappropriate expressions in the revised manuscript. ### W5 / Q8. Code availability Thank you for pointing out this omission. As stated in the Experimental Setup and Experimental Records and Reproducibility sections, we use the code from *Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum* as our reference. The only difference is the objective implementation, which can be added directly using the definitions in our paper. We will organize our code and release the corresponding link as soon as possible. ### Q1. First definition of the focus index Thank you for pointing out this presentation issue. The original manuscript used “focus index” in the introduction but did not explain it until §3.1, which made the discussion harder to follow. We have added this explanation at the first occurrence of the term in the revised manuscript. The additions are marked in blue. ### Q2. Meaning of the supervised-target probability Thank you for this question. Here, $p$ is the probability that the model assigns to the supervised target token, namely $p=p_\theta(\tilde y\mid c)$; it is not the value of the target class in the one-hot label. The one-hot label specifies which token is the supervised target, whereas the probability that the model assigns to that token need not equal 1. We have explicitly distinguished the label from the model output in the notation definitions in §2 of the revised manuscript. The changes are marked in blue. ### Q3. Naming $H_2(P)$ Thank you for this suggestion. The quantity $H_2(P)=-\ln\sum_vP_v^2$ is indeed the order-2 Rényi entropy. We have stated this explicitly alongside the formula in §4.2 of the revised manuscript.

Q4. Plasticity comparisons with second-order optimizers

Thank you for this suggestion. We have not conducted comparisons with second-order optimizers. Second-order and approximate second-order methods may introduce additional curvature-estimation, computational, or memory costs. In our experimental setting, we focused on comparing different objective functions under fixed AdamW settings.

Our plasticity results reflect the adaptation performance of different objectives on weak-prior tasks; they do not establish an advantage over second-order optimizers.

Q5. Meaning of frozen concentration

Thank you for pointing out the need for clarification. Your understanding is correct: “freezing concentration” means applying stop-gradient to C(P)C(P) during backpropagation. Concentration is still recomputed from the current model predictions on every forward pass; it is treated as a constant only in the corresponding backward pass, rather than remaining fixed throughout training.

We have clarified this in the revised manuscript. The changes are marked in blue.

Q6. Rephrasing the rhetorical expression

Thank you for pointing out this wording issue. We have removed this rhetorical expression in the revised manuscript. The changes are marked in blue.

Q7. Replacing “triptych”

Thank you for this suggestion. “Triptych” simply referred to the three main ablation panels in Figure 2 and had no additional technical meaning. The term did not help readers understand the experiment. We have changed the heading to “Analysis of gating ablations” in the revised manuscript. The changes are marked in blue.

Q9. Presentation of Proposition 7

Thank you for pointing this out. Your understanding is correct: the relationship in Proposition 7,

αDEFT=e−H2(P)\alpha_{\mathrm{DEFT}}=e^{-H_2(P)}

follows directly from the definition of order-2 Rényi entropy and Eq. (12). The statement that αDEFT\alpha_{\mathrm{DEFT}} is strictly decreasing in H2(P)H_2(P) is also a direct consequence of this identity. The purpose of that passage was to explain the connection between concentration and Rényi entropy, so it did not need to be presented as a separate proposition.

In the revised manuscript, we have replaced Proposition 7 with a brief explanatory remark, removed the redundant standalone proof, and explicitly stated this relationship at its definition in §4.2. The changes are marked in blue.

Reviewer_G7rcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies token-level learning signal allocation in supervised fine-tuning of large language models. It organizes probability-based objectives within a generalized cross-entropy/deformed-log family, whose gradients factor into a scalar gate and the prediction error. The authors propose two adaptive methods: Cayley-Trans adjusts the focus index according to the target-token probability, while Dynamic Entropy Fine-Tuning (DEFT) uses the collision probability of the full predictive distribution as a detached exponent. The theoretical analysis characterizes Bayes-optimal predictions for constant focus indices and properties of the proposed gates. Experiments cover mathematics, medical reasoning, and synthetic FigFont tasks.

Critical strengths
  • Clear theoretical explanations and reasonably comprehensive experiments. The paper clearly explains the proposed gating mechanisms and the role of stop-gradient. The experiments cover three task settings with five random seeds in the primary comparisons, supported by several ablation studies and diagnostic analyses.

  • Clear update rules and an informative analytical framework. The gate × error decomposition provides a common basis for comparing different objectives. Equation (13) explicitly characterizes how DEFT incorporates information beyond the target-token probability. Section 4.3 and Appendix H.4 clearly explain the role of stop-gradient and acknowledge the equivalence to detached weighted-cross-entropy updates.

Critical weaknesses
  • DEFT may suppress beneficial corrections to confidently wrong predictions. When the model confidently favors an incorrect token, the correct supervised target has low probability pp, while the predictive concentration α\alpha is high. The resulting gate pαp^\alpha therefore attenuates the very update needed to correct the model.

  • The experiments do not directly establish the claimed balance between prior preservation and new knowledge acquisition. The motivation emphasizes forgetting under NLL, but downstream task scores alone do not separate retained knowledge from newly acquired capabilities. The cross-regime evaluations provide useful transfer evidence, yet lack the corresponding base-model references needed to quantify retention. Likewise, the answer-corruption experiment does not measure this trade-off.

Questions for authors

Please address the concerns raised in the weaknesses above. In addition, please clarify the following points:

  • The distinct roles of Cayley-Trans and DEFT. The paper proposes two fine-tuning strategies under the gate ×\times error framework, using different mechanisms to compute their gates. However, their respective advantages and intended use cases remain insufficiently discussed. Under what conditions should Cayley-Trans be preferred over DEFT, and what independent value does it provide?

  • With the focus index detached, Cayley-Trans closely resembles NLL: its gate approaches one at both probability extremes and only modestly downweights intermediate-probability tokens. It is therefore unclear how this adjustment addresses the motivating concerns about forgetting and the coverage–sharpening trade-off, despite the substantial gains reported on some benchmarks. Please explain the connection between this gate behavior and the observed improvements. Comparisons with simple alternative probability-based gates and controls for effective update scale would help establish whether the gains specifically depend on the Cayley construction.

6 comments

We thank the reviewer for recognizing the gate × error framework and our analysis of stop-gradient.

W1. DEFT's ability to correct confidently wrong priors

Thank you for highlighting this potential trade-off. When the correct supervised target has low probability and the predictive distribution has high concentration, DEFT does reduce the gradient at that position relative to NLL. To quantify how this mechanism affects actual correction, we added a diagnostic experiment using verified correct supervision labels.

Specifically, we run a forward pass with the fixed base model over all 67,601 mathematical training examples. We regard the label tokens in the dataset's final answers, within \boxed{...}, as definitively correct; these are the Ground Truth tokens and exclude reasoning tokens. Using the model before fine-tuning, we identify tokens with reference probability ≤0.1 and Shannon entropy ≤0.3, as well as those with probability ≤0.3 and Shannon entropy ≤0.5, as tokens reflecting confidently wrong priors (CBW tokens). Taking NLL as the standard comparison, we measure the proportion of these CBW tokens whose reference token becomes the Top-1 probability token after training with NLL or DEFT.

Metric definition

The correction rate is the proportion of tokens identified as CBW before fine-tuning whose reference token becomes the Top-1 probability token after training. NLL and DEFT are evaluated on the same CBW set in the table below. The relative correction rate is

RelativeCorrection_DEFT = (CorrectionRate_DEFT / CorrectionRate_NLL) * 100%

ModelSettingCBW countOverall proportionNLL correction rateDEFT correction rateRelative correction rate
LLaMA-3.1-8Bp ≤ 0.10, H ≤ 0.384,7750.3088%14.02%11.10%79.17%
LLaMA-3.1-8Bp ≤ 0.30, H ≤ 0.5175,4560.6391%15.15%12.34%81.43%
Qwen2.5-Math-7Bp ≤ 0.10, H ≤ 0.3177,4030.6236%17.05%14.12%82.79%
Qwen2.5-Math-7Bp ≤ 0.30, H ≤ 0.5309,9291.0895%17.05%14.46%84.79%
The results show that DEFT achieves approximately 79%–85% of NLL's correction rate, demonstrating that DEFT retains the ability to correct confidently wrong model priors.

The training trajectories below further show that tokens at initially confidently wrong positions can be corrected during DEFT training. This demonstrates that initially attenuated positions can still be learned. This observation is consistent with the dynamic nature of the gate: DEFT recomputes its weights from the current predictive state, and an initially conflicting status does not constitute permanent masking. Table 5 in the paper provides complementary evidence: restoring NLL weights in the dynamically identified conflict region reduces overall task performance. Together, these results indicate that DEFT improves overall adaptation while retaining the ability to correct initially conflicting positions.

Under the Strict setting (p ≤ 0.1, H ≤ 0.3 bits), the proportion of CBW tokens that become the Top-1 probability token during training (%) is as follows:

StepLLaMA NLLLLaMA DEFTQwen NLLQwen DEFT
00.00%0.00%0.00%0.00%
338.91%6.30%12.99%11.20%
6611.13%9.29%14.92%12.50%
9912.86%9.60%16.31%13.57%
13213.71%10.06%16.87%14.10%
16513.93%10.02%17.00%14.17%
19814.14%10.83%17.06%14.32%
23114.36%10.94%17.22%14.37%
26414.02%11.10%17.05%14.46%

W2. General-capability retention and the plasticity–stability balance

Thank you for raising the question of capability retention. Tables 12–13 in the paper evaluate changes in cross-domain performance under different SFT losses. In the revised version, we added the models' performance before training to make capability retention easier to assess.

MethodTable 12: Math training → Medical evaluationTable 13: Medical training → Math evaluationTable 14: FigFont training → Medical evaluation
Base22.750.9122.75
NLL43.145.6035.60
−p-p44.068.8022.10
Hellinger (α=0.5\alpha=0.5)43.905.6027.60
GCE (α=0.7\alpha=0.7)44.709.5030.30
Cayley-Trans44.868.0040.10
DEFT45.3510.2038.70

The results show that DEFT achieves better IND performance across the three regimes and better overall OOD performance. Because these experiments start from base models, however, improvements in OOD performance are difficult to attribute separately to retention of existing capabilities or cross-domain transfer.

To better verify DEFT's ability to preserve general capabilities and achieve a plasticity–stability balance, we added cross-domain evaluations starting from an Instruct model. Specifically, we train Qwen2.5-Math-7B-Instruct on the MATH task and evaluate its performance on the IND MATH tasks and OOD Medical tasks.

W2 (continue)

MATH task

MethodMATH500MinervaOlympiadBenchAIME24AMC23Avg
Base55.9523.7121.742.2936.7228.08
−log⁡p-\log p49.91 ± 0.4713.92 ± 0.4016.54 ± 0.271.62 ± 0.3120.59 ± 1.7420.52 ± 0.32
−p-p71.78 ± 0.3429.81 ± 0.9132.49 ± 0.285.34 ± 0.5743.09 ± 1.2736.50 ± 0.35
Hellinger68.14 ± 0.3128.11 ± 0.5130.85 ± 0.165.05 ± 0.6840.00 ± 1.3834.43 ± 0.39
GCE69.87 ± 0.3629.23 ± 0.3831.96 ± 0.355.79 ± 0.6041.45 ± 0.8735.66 ± 0.15
EAFT50.17 ± 0.4015.23 ± 0.4416.63 ± 0.311.37 ± 0.6021.94 ± 0.9221.07 ± 0.29
InfoSFT62.31 ± 0.1924.55 ± 0.1826.26 ± 0.393.36 ± 1.0032.94 ± 0.9929.89 ± 0.31
RankTuner-p69.33 ± 0.4129.30 ± 0.7531.99 ± 0.235.16 ± 1.0341.35 ± 0.2835.42 ± 0.35

Medical task

MethodMedMCMedQAPubMedMMLU-PGPQALancetMedB(4)MedB(5)MedXNEJMAvg
Base33.6429.9370.9034.5335.3832.7728.2524.0311.1123.8832.44
−log⁡p-\log p31.82 ± 0.4029.73 ± 0.7169.32 ± 0.5032.12 ± 0.4634.67 ± 3.0232.77 ± 0.9628.76 ± 2.3222.66 ± 1.4110.05 ± 0.3024.77 ± 1.5931.67 ± 0.36
−p-p33.82 ± 0.3630.67 ± 1.2968.38 ± 0.3131.57 ± 0.8432.82 ± 3.0832.57 ± 1.7928.44 ± 1.8722.47 ± 2.1510.38 ± 0.8525.04 ± 0.8931.62 ± 0.61
Hellinger32.58 ± 0.2832.13 ± 1.0167.82 ± 0.7532.82 ± 0.4334.77 ± 0.9433.30 ± 1.0129.67 ± 1.0720.58 ± 0.9310.31 ± 0.3025.21 ± 0.8731.92 ± 0.18
GCE33.38 ± 0.1632.54 ± 0.5167.64 ± 0.3232.85 ± 0.4134.10 ± 1.0733.69 ± 1.6429.94 ± 0.7022.47 ± 1.2110.01 ± 0.7223.85 ± 0.9432.04 ± 0.20
EAFT31.54 ± 0.3730.76 ± 0.7368.66 ± 0.2631.65 ± 0.7431.64 ± 3.0331.02 ± 0.9327.01 ± 1.9124.67 ± 1.1310.56 ± 0.7325.97 ± 0.5531.35 ± 0.29
InfoSFT32.69 ± 0.3629.70 ± 1.2367.92 ± 0.9933.24 ± 0.2632.15 ± 1.5931.94 ± 0.8826.49 ± 1.5023.90 ± 1.399.73 ± 0.3825.40 ± 1.2231.32 ± 0.29
RankTuner-p33.60 ± 0.2032.24 ± 0.6667.46 ± 0.8332.70 ± 0.3736.05 ± 2.7732.62 ± 1.2229.48 ± 1.2020.52 ± 1.2010.44 ± 0.5625.67 ± 1.2332.08 ± 0.38
Cayley-Trans32.45 ± 0.1430.74 ± 1.1568.70 ± 1.8131.83 ± 0.6731.79 ± 1.2932.77 ± 1.4830.19 ± 1.4223.49 ± 0.5010.03 ± 0.6827.25 ± 1.2231.92 ± 0.29
DEFT32.34 ± 0.2432.18 ± 0.7067.20 ± 0.8531.53 ± 0.5236.67 ± 0.9331.96 ± 0.5630.84 ± 0.5723.27 ± 0.4910.17 ± 1.0525.20 ± 2.5732.14 ± 0.11

These results show that DEFT achieves better IND performance while exhibiting the smallest decline in the model's original general capabilities, demonstrating its ability to preserve general capabilities and maintain a plasticity–stability balance.

Q1.

Thank you for this valuable question. Cayley and DEFT share the same generalized loss. They differ in the information used to set the focus index aa and the resulting allocation of learning signals.

Specifically, let PP denote the model's prediction, p=Pyp=P_y the probability of the supervised label, and rr the supervision distribution at the same context. For the generalized loss ℓa(p)=(1−pa)/a\ell_a(p)=(1-p^a)/a, fixing a common exponent 0<a<10<a<1 across labels makes risk minimization equivalent to maximizing ∑iriPia\sum_i r_iP_i^a. This objective is strictly concave on the positive-probability support of rr, and its stationarity condition under the normalization constraint is ri(Pi∗)a−1=λr_i(P_i^*)^{a-1}=\lambda. Theorems 1/2 therefore give

\beta=\frac1{1-a},\quad Z_\beta=\sum_jr_j^\beta,$$ $$B_a(r)=\min_P\sum_i r_i\ell_a(P_i) =\frac{1-Z_\beta^{1/\beta}}a =\frac{1-e^{-aH_\beta(r)}}a =\ell_a\!\left(e^{-H_\beta(r)}\right).$$ Here, $H_\beta(r)=\ln Z_\beta/(1-\beta)$, and the optimal prediction is zero outside the supervision support. Thus, the connection between Rényi entropy and the generalized loss comes from the optimal risk itself: because the scalar loss is strictly decreasing, $e^{-H_\beta(r)}$ is the unique equivalent probability that produces the same optimal risk value. For adaptive training and finite orders $q>1$, we consider the same form of concentration on the model's predictions, $c_q(P)=e^{-H_q(P)}=(\sum_iP_i^q)^{1/(q-1)}$. Higher entropy yields a smaller exponent; at a fixed $0<p<1$, the gate $p^{c_q(P)}$ moves closer to the NLL gate of $1$. Here, the current model prediction determines the exponent, while the predictive order $q$ and the $\beta$ in the fixed-exponent risk play different roles. The rationale for order two comes from the actual gradient with the exponent frozen: $$\begin{aligned} \nabla_z\ell_a&=p^a(P-e_y),\\ \|\nabla_z\ell_a\|_2^2&=p^{2a}E(P,y),\\ E(P,y)&=\|P-e_y\|_2^2=1-2p+C(P),\quad C(P)=\sum_iP_i^2. \end{aligned}$$

Q1(continue)

Given pp, C(P)C(P) exactly characterizes the squared Euclidean length of the correction vector. For K≥4K\ge4 and finite q>1q>1, within the power-sum family Tq=∑iPiqT_q=\sum_iP_i^q, this length can be recovered exactly from (p,Tq)(p,T_q) for every interior prediction if and only if q=2q=2. This provides a structural rationale for order two as an exact representation of the original gradient magnitude. DEFT further chooses a=Ca=C as its control rule; uniqueness of the statistical representation does not imply task optimality of this control rule.

For Cayley, we instead use the surprisal of the non-target event: the total probability assigned to the non-target set is 1−p=∑i≠yPi1-p=\sum_{i\ne y}P_i, and the surprisal of this event is Ierr=−ln⁡(1−p)I_{\mathrm{err}}=-\ln(1-p). As the non-target event becomes less likely, IerrI_{\mathrm{err}} increases, indicating that the model increasingly favors the supervised label. Under the Cayley rule, the corresponding focus index also approaches 11.

To connect the coverage anchor at low probability with the sharpening anchor at high probability, we use the intrinsic Fisher–Rao radius s≜1−p∈[0,1]s\triangleq\sqrt{1-p}\in[0,1] from the paper to parameterize the model's deviation from the supervised label. This radius is a monotone reparameterization of the Fisher–Rao distance to the supervised vertex and satisfies s=e−Ierr/2s=e^{-I_{\mathrm{err}}/2}, directly linking the geometric coordinate to the event surprisal above. Applying the Cayley transform gives the focus index:

A(p)=T\!\left(\sqrt{1-p}\right) =\frac{1-\sqrt{1-p}}{1+\sqrt{1-p}}.$$ When the target probability is low, the non-target event is relatively likely, $I_{\mathrm{err}}$ is small, $s$ is close to $1$, and the exponent approaches $0$. As the model becomes confident in the supervised label, the non-target event becomes unlikely, $I_{\mathrm{err}}$ increases, $s$ approaches $0$, and the exponent approaches $1$. This construction uses only the allocation of probability between the target and non-target sets. For both Cayley and DEFT, the exponent can change with the model state at each step but is frozen during differentiation, so reweighting does not change an individual token's NLL gradient direction. The two methods implement different rules through this common interface: $$\begin{aligned} a_{\mathrm D}&=e^{-H_2(P)}=C(P),\\ a_{\mathrm C}=A(p)&=\frac{1-\sqrt{1-p}}{1+\sqrt{1-p}}\\ &=\frac{1-e^{-I_{\mathrm{err}}/2}}{1+e^{-I_{\mathrm{err}}/2}} =\tanh\!\left(\frac{I_{\mathrm{err}}}{4}\right),\\ I_{\mathrm{err}}&=-\ln(1-p). \end{aligned}$$ The corresponding gates are $w_{\mathrm D}=p^{C(P)}$ and $w_{\mathrm C}=p^{A(p)}$. Cayley's exponent reads only the supervised-label probability and its complementary event probability; the surprisal here is not the entropy of the full distribution. DEFT additionally reads how the tail probability is distributed. Specifically, for $i\ne y$, let $R_i=P_i/(1-p)$. Then $$C(P)=p^2+(1-p)^2C(R),\qquad \frac{\partial\ln w_{\mathrm D}}{\partial C(R)} =(1-p)^2\ln p<0.$$ Thus, with the label probability fixed, a more concentrated tail produces stronger scalar attenuation under DEFT, whereas Cayley's weight does not change. This distinction concerns only the exponent and scalar weight: the full gradients of both methods still use the entire predictive distribution through $P-e_y$. This also explains Cayley's independent value. If low-probability supervision is correct, Cayley does not further attenuate its correction signal merely because a competing label has become more concentrated, and it recovers the NLL gate as $p\downarrow0$. DEFT, by contrast, attenuates gradients under concentrated disagreement. As the experiments in W1 show, although DEFT retains the ability to correct confidently wrong predictions, its correction rate may be lower than NLL's. To further compare the correction abilities of the two gates, we evaluate NLL, DEFT, and Cayley-Trans on **a fixed set of CBW tokens selected using the initial base model**. These statistics are likewise restricted to label tokens in the final answers within `\boxed{...}`. Here, **CBW $\tau=0.7$ means that the initial model assigns a probability of at least $0.7$ to a non-target token that is also its Top-1 prediction**. Specifically, let $P_t^{(0)}$ be the model's predictive distribution at position $t$ before training, and $y_t$ the supervised label. The selection criterion is $$\hat y_t^{(0)}=\arg\max_i P_t^{(0)}(i),\qquad \hat y_t^{(0)}\ne y_t,\qquad P_t^{(0)}\!\left(\hat y_t^{(0)}\right)\ge\tau=0.7.$$ Thus, $\tau$ constrains **the initial model's confidence in the incorrect competing token**, not the probability of the supervised label. We also report results for the criterion in W1: target probability $p_y^{(0)}\le0.1$ and Shannon entropy $H(P^{(0)})\le0.3$ bits.
Reviewer_igptOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the plasticity–stability dilemma in SFT and unifies token-level objectives through a gate × error gradient structure. It proposes Cayley-Trans, which uses target-token probability, and DEFT, which uses Rényi-2 distribution concentration to dynamically adjust the focus index. Experiments across seven backbones and three model capability regimes show improvements over standard NLL, with DEFT achieving strong overall performance without task-specific tuning.

Critical strengths
  1. Unified framework for SFT objectives. The paper provides a unified gate × error gradient structure for token-level SFT objectives and distinguishes probability-level from distribution-level trust gating. This offers a clear perspective on how different objectives allocate learning signals.

  2. Extensive empirical evaluation. The paper evaluates Cayley-Trans and DEFT across seven backbones and three model capability regimes, with comparisons against several existing SFT objectives. DEFT achieves strong overall performance without task-specific tuning, demonstrating its practical value across different settings.

Critical weaknesses
  1. Knowledge preservation and correction. The paper does not directly evaluate whether DEFT preserves general capabilities or corrects confidently wrong priors, as acknowledged in Appendix Q. This makes it difficult to assess whether the proposed method achieves the intended plasticity–stability balance.

  2. Effect of gradient scaling. Although the paper includes gate intervention experiments, it remains unclear whether the improvements come from distribution-level information or changes in update magnitude. The absence of update-scale-matched controls makes it difficult to isolate the contribution of the proposed gate.

Questions for authors
  1. Could the authors provide direct evidence that DEFT preserves general capabilities and corrects confidently wrong priors? Could update-scale-matched controls also help clarify whether the improvements come from distribution-level information or changes in update magnitude?

  2. How does the choice of Rényi-2 distribution concentration as the focus index in DEFT relate to the theoretical analysis of Cayley-Trans? Why is this specific mapping chosen over other concentration-based focus rules?

5 comments

W1 / Q1. Preservation of general capabilities and correction of confidently wrong priors

Thank you for these constructive comments. We have conducted the following additional analyses and experiments to address capability preservation, the plasticity–stability balance, and the correction of confidently wrong priors.

Tables 12–14 evaluate changes in cross-domain performance after training with different SFT losses. In the revised manuscript, we have added the corresponding model performance before fine-tuning to make capability preservation easier to assess. The cross-domain results are summarized below.

MethodTable 12: Math training → Medical evaluationTable 13: Medical training → Math evaluationTable 14: FigFont training → Medical evaluation
Base22.750.9122.75
NLL43.145.6035.60
−p-p44.068.8022.10
Hellinger43.905.6027.60
GCE44.709.5030.30
Cayley-Trans44.868.0040.10
DEFT45.3510.2038.70

These results show that DEFT achieves better IND performance across the three regimes and better overall OOD performance. Because training starts from base models, however, the OOD improvements alone do not readily distinguish preservation of existing capabilities from cross-domain capability transfer.

To examine DEFT's preservation of general capabilities and its plasticity–stability balance more directly, we additionally evaluate cross-domain performance from an Instruct initialization. Specifically, we fine-tune Qwen2.5-Math-7B-Instruct on the MATH task and evaluate it on both the IND mathematics benchmarks and the OOD medical benchmarks.

Mathematics task

MethodMATH500MinervaOlympiadBenchAIME24AMC23Avg
Base55.9523.7121.742.2936.7228.08
−log⁡p-\log p49.91 ± 0.4713.92 ± 0.4016.54 ± 0.271.62 ± 0.3120.59 ± 1.7420.52 ± 0.32
−p-p71.78 ± 0.3429.81 ± 0.9132.49 ± 0.285.34 ± 0.5743.09 ± 1.2736.50 ± 0.35
Hellinger68.14 ± 0.3128.11 ± 0.5130.85 ± 0.165.05 ± 0.6840.00 ± 1.3834.43 ± 0.39
GCE69.87 ± 0.3629.23 ± 0.3831.96 ± 0.355.79 ± 0.6041.45 ± 0.8735.66 ± 0.15
EAFT50.17 ± 0.4015.23 ± 0.4416.63 ± 0.311.37 ± 0.6021.94 ± 0.9221.07 ± 0.29
InfoSFT62.31 ± 0.1924.55 ± 0.1826.26 ± 0.393.36 ± 1.0032.94 ± 0.9929.89 ± 0.31
RankTuner-p69.33 ± 0.4129.30 ± 0.7531.99 ± 0.235.16 ± 1.0341.35 ± 0.2835.42 ± 0.35
Cayley-Trans66.73 ± 0.2327.55 ± 0.4831.32 ± 0.364.54 ± 0.6838.69 ± 1.4133.76 ± 0.25
DEFT70.35 ± 0.0829.54 ± 0.5532.31 ± 0.084.58 ± 1.1342.07 ± 1.2235.77 ± 0.10

Medical task

MethodMedMCMedQAPubMedMMLU-PGPQALancetMedB(4)MedB(5)MedXNEJMAvg
Base33.6429.9370.9034.5335.3832.7728.2524.0311.1123.8832.44
−log⁡p-\log p31.82 ± 0.4029.73 ± 0.7169.32 ± 0.5032.12 ± 0.4634.67 ± 3.0232.77 ± 0.9628.76 ± 2.3222.66 ± 1.4110.05 ± 0.3024.77 ± 1.5931.67 ± 0.36
−p-p33.82 ± 0.3630.67 ± 1.2968.38 ± 0.3131.57 ± 0.8432.82 ± 3.0832.57 ± 1.7928.44 ± 1.8722.47 ± 2.1510.38 ± 0.8525.04 ± 0.8931.62 ± 0.61
Hellinger32.58 ± 0.2832.13 ± 1.0167.82 ± 0.7532.82 ± 0.4334.77 ± 0.9433.30 ± 1.0129.67 ± 1.0720.58 ± 0.9310.31 ± 0.3025.21 ± 0.8731.92 ± 0.18
GCE33.38 ± 0.1632.54 ± 0.5167.64 ± 0.3232.85 ± 0.4134.10 ± 1.0733.69 ± 1.6429.94 ± 0.7022.47 ± 1.2110.01 ± 0.7223.85 ± 0.9432.04 ± 0.20
EAFT31.54 ± 0.3730.76 ± 0.7368.66 ± 0.2631.65 ± 0.7431.64 ± 3.0331.02 ± 0.9327.01 ± 1.9124.67 ± 1.1310.56 ± 0.7325.97 ± 0.5531.35 ± 0.29
InfoSFT32.69 ± 0.3629.70 ± 1.2367.92 ± 0.9933.24 ± 0.2632.15 ± 1.5931.94 ± 0.8826.49 ± 1.5023.90 ± 1.399.73 ± 0.3825.40 ± 1.2231.32 ± 0.29
RankTuner-p33.60 ± 0.2032.24 ± 0.6667.46 ± 0.8332.70 ± 0.3736.05 ± 2.7732.62 ± 1.2229.48 ± 1.2020.52 ± 1.2010.44 ± 0.5625.67 ± 1.2332.08 ± 0.38
Cayley-Trans32.45 ± 0.1430.74 ± 1.1568.70 ± 1.8131.83 ± 0.6731.79 ± 1.2932.77 ± 1.4830.19 ± 1.4223.49 ± 0.5010.03 ± 0.6827.25 ± 1.2231.92 ± 0.29
DEFT32.34 ± 0.2432.18 ± 0.7067.20 ± 0.8531.53 ± 0.5236.67 ± 0.9331.96 ± 0.5630.84 ± 0.5723.27 ± 0.4910.17 ± 1.0525.20 ± 2.5732.14 ± 0.11

The results show that DEFT improves IND performance while exhibiting the smallest decline in the model's original general capabilities, demonstrating its ability to preserve general capabilities and maintain a plasticity–stability balance.

W1 / Q1 (continue)

Correction of confidently wrong priors

Thank you for highlighting this potential trade-off. When the correct supervised target has low probability and the predictive distribution is highly concentrated, DEFT does reduce the gradient at that position relative to NLL. To quantify the effect of this mechanism on actual correction, we have added a diagnostic experiment using verified correct supervision labels.

Specifically, we perform a forward pass with the fixed base model on all 67,601 mathematics training examples. Here, ground truth refers to the label tokens in the final answer's \boxed{...} expression, excluding the reasoning tokens. We treat these ground-truth answer tokens as correct. Before fine-tuning, we identify confident-but-wrong (CBW) tokens using a reference-token probability of p≤0.1p\le0.1 and Shannon entropy of H≤0.3H\le0.3, and additionally consider the setting p≤0.3p\le0.3, H≤0.5H\le0.5. Using NLL as the standard baseline, we compare the proportion of these CBW positions at which the reference token becomes the top-1 prediction after NLL or DEFT training.

Metric definition. The correction rate is the proportion of tokens identified as CBW before fine-tuning whose reference token becomes the top-1 prediction after training. NLL and DEFT are evaluated on the same CBW set in each setting. The relative correction rate is

RelativeCorrectionDEFT=CorrectionRateDEFTCorrectionRateNLL×100%RelativeCorrection_{DEFT} = \frac{CorrectionRate_{DEFT}}{CorrectionRate_{NLL}} \times 100\%

ModelSettingCBW countOverall proportionNLL correction rateDEFT correction rateRelative correction rate
LLaMA-3.1-8Bp ≤ 0.10, H ≤ 0.384,7750.3088%14.02%11.10%79.17%
LLaMA-3.1-8Bp ≤ 0.30, H ≤ 0.5175,4560.6391%15.15%12.34%81.43%
Qwen2.5-Math-7Bp ≤ 0.10, H ≤ 0.3177,4030.6236%17.05%14.12%82.79%
Qwen2.5-Math-7Bp ≤ 0.30, H ≤ 0.5309,9291.0895%17.05%14.46%84.79%

DEFT achieves approximately 79%–85% of NLL's correction rate, demonstrating that it retains the ability to correct confidently wrong priors.

The training trajectories below further show that tokens at initially confidently wrong positions can be corrected during DEFT training. Thus, initially attenuated positions can still be learned. This observation is consistent with the dynamic nature of the gate: DEFT recomputes weights from the current predictive state, so an initial conflict designation does not permanently mask a token. Table 5 in the paper provides complementary evidence: restoring NLL weights in dynamically identified conflict regions reduces overall task performance. Together, these results show that DEFT improves overall adaptation while retaining the ability to correct initially conflicting positions.

The following table reports the percentage of CBW tokens whose reference token becomes the top-1 prediction during training under the strict setting (p≤0.1p\le0.1, H≤0.3H\le0.3 bits).

StepLLaMA NLLLLaMA DEFTQwen NLLQwen DEFT
00.00%0.00%0.00%0.00%
338.91%6.30%12.99%11.20%
6611.13%9.29%14.92%12.50%
9912.86%9.60%16.31%13.57%
13213.71%10.06%16.87%14.10%
16513.93%10.02%17.00%14.17%
19814.14%10.83%17.06%14.32%
23114.36%10.94%17.22%14.37%
26414.02%11.10%17.05%14.46%

W2 / Q1. Controls for the effect of update scale

Thank you for this valuable suggestion. To better understand the source of DEFT's gains, we globally shuffle its gates and examine whether the gains come from the additional distribution-level information introduced by DEFT while keeping the overall update magnitude unchanged.

MATHDEFTGlobal shuffleChange after shuffling
LLaMA-3.1-8B11.046.53−4.51
Qwen2.5-Math-7B36.7324.17−12.56

With the update magnitude unchanged, globally shuffling DEFT's gates substantially reduces performance, bringing it close to NLL. This indicates that indiscriminately changing update magnitudes cannot reproduce the performance gains.

Q2. Why Rényi-2, how does it relate to Cayley-Trans, and why this concentration mapping?

Thank you for this insightful question. We explain the connection to the generalized loss, the choice of order two, and the relationship between the two focusing rules below.

Q2.1. Why use the Rényi mapping?

The motivation for considering Rényi entropy first comes from the generalized loss itself: its optimal supervised risk can be expressed as a function of Rényi entropy.

The generalized loss is

\qquad \ell_0(p)=-\ln p.$$ Its gradients are $$\frac{\partial\ell_a}{\partial p}=-p^{a-1},\qquad \nabla_zp=p(e_y-P),\qquad \nabla_z\ell_a=p^a(P-e_y).$$ The exponent therefore controls a scalar multiplier of the NLL gradient. For $0\le a\le1$, the multiplier $p^a$ lies in $[p,1]$: $a=0$ corresponds to NLL, and $a=1$ to the linear probability loss. At a fixed predictive state, increasing the exponent strengthens attenuation without changing the token's gradient direction. Stop-gradient holds the exponent fixed during the current differentiation, while allowing its value to vary across training steps. Let $r$ be the supervision distribution for a fixed context, with a common fixed exponent $0<a<1$ across candidate labels. Theorems 1 and 2 characterize the minimum of the following risk over the closed probability simplex $\Delta_K$: $$R_a(P;r)=\sum_i r_i\ell_a(P_i) =\frac{1-\sum_i r_iP_i^a}{a},\qquad P\in\Delta_K.$$ Minimizing this risk is equivalent to maximizing $\sum_i r_iP_i^a$. The optimum allocates no mass outside the positive support of $r$, because moving such mass to a supervised coordinate increases this sum. On that support, the objective is strictly concave, and its derivative diverges as a coordinate approaches zero, so every supported coordinate is positive at the optimum. The stationarity condition under the normalization constraint gives $$r_i(P_i^{\ast})^{a-1}=\lambda,\qquad P_i^{\ast}=\frac{r_i^{\beta}}{Z_{\beta}},\qquad \beta=\frac{1}{1-a},\qquad Z_{\beta}=\sum_j r_j^{\beta}.$$ with zero coordinates outside the support. Strict concavity guarantees that this optimum is unique. Since $1+a\beta=\beta$, substitution into the risk yields $$\sum_i r_i(P_i^*)^a =Z_\beta^{-a}\sum_i r_i^{1+a\beta} =Z_\beta^{1-a}=Z_\beta^{1/\beta}, \qquad B_a(r):=\min_P R_a(P;r) =\frac{1-Z_\beta^{1/\beta}}a.$$ Now use Rényi entropy, $H_\beta(r)=\ln Z_\beta/(1-\beta)$. Because $(1-\beta)/\beta=-a$, the same optimal value can be written as $$B_a(r)=\frac{1-e^{-aH_\beta(r)}}a =\ell_a\!\left(e^{-H_\beta(r)}\right).$$ In the NLL limit $a\to0$, the order satisfies $\beta\to1$, and the minimum risk becomes Shannon entropy: $B_0(r)=H_1(r)$. The Rényi expression extends this familiar loss–entropy connection to nonzero power exponents. Because the scalar loss $\ell_a$ is strictly decreasing in its probability argument, there is a unique probability that produces the same loss as the optimal supervised risk: $$\ell_a^{-1}\!\left(B_a(r)\right) =\left(1-aB_a(r)\right)^{1/a} =Z_\beta^{1/(\beta-1)} =e^{-H_\beta(r)}.$$ Thus, the same loss family explains both the Rényi quantity and its exponential mapping: they describe the optimal supervised risk and the probability corresponding to that risk, respectively. This equivalent probability summarizes a scalar risk; it is not a coordinate of the optimal predictor. Other monotone entropy representations can also express the risk, while the Rényi form provides the direct exponential identity above. #### Q2.2. Why choose order two? The Bayes-risk derivation explains why we consider Rényi entropy and its exponential mapping. To construct an adaptive method, we apply this concentration form to the model's current predictive distribution $P$ and consider $c_q(P)=e^{-H_q(P)}$, with $q>1$, as candidate focus indices. This quantity can be interpreted as the inverse effective support size and lies in $[1/K,1]$, consistent with the exponent range of the generalized loss. For the logit gradient under standard Euclidean geometry, any frozen $a$ satisfies $$\|\nabla_z\ell_a\|_2^2 =p^{2a}E(P,y),\qquad E(P,y):=\|P-e_y\|_2^2 =1-2p+C(P),\qquad C(P)=\sum_iP_i^2.$$ Given $p$, the single statistic $C(P)$ exactly characterizes the squared norm of the correction vector. Specifying the exponent then determines the squared gradient norm. This provides a concrete criterion for comparing entropy orders: whether the corresponding summary retains the magnitude information already present in the signal being scaled. The criterion arises from a representation question about the original loss, rather than a separate objective claimed to optimize downstream performance. We formalize this criterion as an exact scalar representation property. Let $K$ denote the vocabulary size and $T_q(P)=\sum_iP_i^q$. The LLM vocabularies considered here are much larger than $4$, satisfying $K\ge4$.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.