- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates token-level gradient allocation in supervised fine-tuning through a generalized cross-entropy framework. The authors characterize the gradient of probability-based SFT objectives as a product of a confidence-dependent gate and the standard prediction error. Building on this interpretation, they introduce Cayley-Trans, which adaptively adjusts the loss exponent using the ground-truth token probability, and DEFT, which incorporates the Rényi-2 concentration of the full predictive distribution. Experiments across mathematical reasoning, medical reasoning, and synthetic weak-prior tasks demonstrate encouraging improvements over several existing SFT objectives.
Critical strengths
- The paper studies a relevant problem in SFT objective design and provides a useful perspective on confidence-dependent gradient allocation.
- The proposed methods are computationally lightweight and do not require additional trainable parameters or reference models.
- The experimental settings and results are generally comprehensive.
Critical weaknesses
- The paper argues that predictive concentration helps distinguish harmful knowledge conflicts from valuable new supervision. However, a low-probability, low-entropy prediction does not necessarily indicate noisy or undesirable supervision. It may equally reflect an incorrect but highly confident model prediction that should be corrected. Maybe designing synthetic experiments could help.
- The novelty is a little limited. Previous methods such as EAFT and RankTuner have already explored distribution-aware token weighting to distinguish uncertain predictions from confident conflicts. What is the fundamental advantage of the proposed approach?
- Section J.4 seems to be AI-generated. The authors are encouraged to revise and provide fully reproducible details.
Questions for authors
I see in Table 2 the results for Qwen2.5-Math-7B; the results are nearly identical. Is there a particular interpretation?
3 comments
W2 (continue)
DEFT directly uses the second-order concentration as the exponent and stops its gradient according to the convention in Section 4.3:
G_t^{\mathrm{DEFT}}=p_t^{C_t},\qquad p_t\le G_t^{\mathrm{DEFT}}\le1.$$ The exponent can change with the model state at each step but is held fixed during differentiation, so it does not change the NLL gradient direction for an individual token. At fixed $p_t$, $\partial\ln G_t^{\mathrm{DEFT}}/\partial C_t=\ln p_t<0$: the more concentrated the prediction, the smaller the token's gradient weight. This provides an explicit rationale for both the second-order statistic and the direction of its weighting effect. DEFT therefore considers two factors simultaneously: how concentrated the model's prediction is, and whether the model supports the supervised answer. Specifically, $$\nabla_{z_t} L_{DEFT} = p_t^{C_t} (P_t - e_{y_t})$$ We next compare how each method determines this token weight. To maintain the same gradient interpretation, all weights below are discussed under the convention that they are held fixed during differentiation. **Difference from EAFT: distinguishing agreement from conflict with the supervised answer at low entropy.** In its normalized Top-$K$ Shannon-entropy weighting form, EAFT uses $$L_t^{\mathrm{EAFT}}=\widetilde H_t(-\log p_t),\qquad G_t^{\mathrm{EAFT}}=\widetilde H_t.$$ As the entropy used by EAFT approaches zero, its entropy gate approaches zero for both low-probability conflicting tokens and high-probability mastered tokens. DEFT also attenuates gradients for both types of tokens, since $0<p_t^{C_t}<1$ when $0<p_t<1$. The difference is that it also distinguishes the degree of attenuation using the probability of the supervised answer. At the same concentration, a low-probability conflicting label receives a smaller weight and stronger attenuation, whereas a high-probability agreeing label receives a weight close to one and weaker attenuation. Thus, for the same low-entropy predictive distribution, EAFT uses the same entropy multiplier for different supervised labels, while DEFT further adjusts the weight according to the label probability. **Difference from RankTuner: the basis for weighting and consistency across tasks.** RankTuner's relative scaling factor combines the supervised label's rank $R_t$ with the expected rank: $$S_t=2^{f(\mathbb E[R_t])-f(R_t)},\qquad f(x)=\frac1{\log_2(x+1)},\qquad G_t^{\mathrm{RankTuner}}=b_tS_t.$$ Here, $b_t$ is the base weight and $S_t$ is the rank adjustment. When $R_t>\mathbb E[R_t]$, the strictly decreasing function $f$ gives $S_t>1$. RankTuner therefore increases the token's contribution relative to its base weight, using an actual rank worse than expected as the basis for assigning greater corrective weight. The two methods differ particularly in how they allocate learning signals in the low-$p_t$, low-entropy region. If the model strongly favors a non-target label and the supervised label's rank is substantially worse than expected, i.e., $R_t\gg\mathbb E[R_t]$, RankTuner increases its Relative Scale. It interprets this as a critical error in which the model underperforms at a position that should be relatively clear. From the perspective of prior preservation, DEFT instead treats this state as a potential confident conflict: higher concentration moves the gate toward the linear endpoint $p_t$, reducing the token-level gradient weight of that supervision signal to limit how strongly external supervision rewrites an existing concentrated prediction. Thus, the two methods differ in both their statistics and functional forms, emphasizing corrective compensation relative to predictive difficulty and conflict suppression controlled by predictive concentration, respectively. **Another difference is whether the base weight changes across tasks.** In our comparison configurations, RankTuner uses RankTuner-p for mathematical training, with $b_t=p_t$ and total weight $p_tS_t$, and RankTuner-one for medical and FigFont training, with $b_t=1$ and total weight $S_t$. RankTuner therefore uses different base-weighting rules depending on the training task in these experiments. DEFT uses $C_t=\sum_vP_t(v)^2$ and $G_t=p_t^{C_t}$ uniformly across these tasks, without switching between $p_t$ weighting and unit weighting by task. Table 1 in the main text directly compares these methods. The average mathematical scores of DEFT and RankTuner are 11.04 and 10.78 on LLaMA, and 36.73 and 36.01 on Qwen, respectively. We further report per-benchmark results after training Qwen2.5-Math-7B-Instruct on mathematical data, covering both in-domain (IND) mathematics and out-of-domain (OOD) medical performance.W2 (continue)
In-domain mathematics (IND)
| Method | MATH500 | Minerva | OlympiadBench | AIME24 | AMC23 | Avg |
|---|---|---|---|---|---|---|
| Initial Instruct | 55.95 | 23.71 | 21.74 | 2.29 | 36.72 | 28.08 |
| NLL | 49.91 ± 0.47 | 13.92 ± 0.40 | 16.54 ± 0.27 | 1.62 ± 0.31 | 20.59 ± 1.74 | 20.52 ± 0.32 |
| EAFT | 50.17 ± 0.40 | 15.23 ± 0.44 | 16.63 ± 0.31 | 1.37 ± 0.60 | 21.94 ± 0.92 | 21.07 ± 0.29 |
| InfoSFT | 62.31 ± 0.19 | 24.55 ± 0.18 | 26.26 ± 0.39 | 3.36 ± 1.00 | 32.94 ± 0.99 | 29.89 ± 0.31 |
| RankTuner-p | 69.33 ± 0.41 | 29.30 ± 0.75 | 31.99 ± 0.23 | 5.16 ± 1.03 | 41.35 ± 0.28 | 35.42 ± 0.35 |
| DEFT | 70.35 ± 0.08 | 29.54 ± 0.55 | 32.31 ± 0.08 | 4.58 ± 1.13 | 42.07 ± 1.22 | 35.77 ± 0.10 |
Out-of-domain medicine (OOD)
| Method | MedMC | MedQA | PubMed | MMLU-P | GPQA | Lancet | MedB(4) | MedB(5) | MedX | NEJM | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Initial Instruct | 33.64 | 29.93 | 70.90 | 34.53 | 35.38 | 32.77 | 28.25 | 24.03 | 11.11 | 23.88 | 32.44 |
| NLL | 31.82 ± 0.40 | 29.73 ± 0.71 | 69.32 ± 0.50 | 32.12 ± 0.46 | 34.67 ± 3.02 | 32.77 ± 0.96 | 28.76 ± 2.32 | 22.66 ± 1.41 | 10.05 ± 0.30 | 24.77 ± 1.59 | 31.67 ± 0.36 |
| EAFT | 31.54 ± 0.37 | 30.76 ± 0.73 | 68.66 ± 0.26 | 31.65 ± 0.74 | 31.64 ± 3.03 | 31.02 ± 0.93 | 27.01 ± 1.91 | 24.67 ± 1.13 | 10.56 ± 0.73 | 25.97 ± 0.55 | 31.35 ± 0.29 |
| InfoSFT | 32.69 ± 0.36 | 29.70 ± 1.23 | 67.92 ± 0.99 | 33.24 ± 0.26 | 32.15 ± 1.59 | 31.94 ± 0.88 | 26.49 ± 1.50 | 23.90 ± 1.39 | 9.73 ± 0.38 | 25.40 ± 1.22 | 31.32 ± 0.29 |
| RankTuner-p | 33.60 ± 0.20 | 32.24 ± 0.66 | 67.46 ± 0.83 | 32.70 ± 0.37 | 36.05 ± 2.77 | 32.62 ± 1.22 | 29.48 ± 1.20 | 20.52 ± 1.20 | 10.44 ± 0.56 | 25.67 ± 1.23 | 32.08 ± 0.38 |
| DEFT | 32.34 ± 0.24 | 32.18 ± 0.70 | 67.20 ± 0.85 | 31.53 ± 0.52 | 36.67 ± 0.93 | 31.96 ± 0.56 | 30.84 ± 0.57 | 23.27 ± 0.49 | 10.17 ± 1.05 | 25.20 ± 2.57 | 32.14 ± 0.11 |
The results show that DEFT achieves greater improvements in IND performance while better preserving the model's existing capabilities, achieving a plasticity–stability balance.
W3. Presentation and reproducibility details in Section J.4
Thank you for pointing this out. In the revised manuscript, we have added detailed experimental settings to the sections “Experimental Setup” and “Experimental Records and Reproducibility” to ensure that the experiments are reproducible.
Q1. Similar medical results for Qwen2.5-Math-7B in Table 2
Thank you for highlighting this observation. Table 2 concerns medical reasoning, which belongs to the Model-Intermediate (MI) regime in our setting. Your observation is consistent with the findings reported in Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum: in the intermediate regime, where the model has partial task priors, performance gaps between objectives narrow, and neither strategies that emphasize existing priors nor those that strengthen new-knowledge learning show a consistently dominant advantage.
Specifically, the average scores of the methods on Qwen2.5-Math-7B in Table 2 range from 34.27 to 35.05, a span of 0.78 percentage points. DEFT achieves an average score of 35.05, improving on NLL's 34.29 by 0.76 percentage points. This is consistent with the interpretation that an objective's effectiveness depends on the alignment between model priors and the task; it does not imply that improvements should be equally large in every setting.
W1. Correcting confident but wrong predictions
Thank you for pointing out this potential trade-off. When the probability of a correct supervised target is low and the predictive distribution is highly concentrated, DEFT does reduce the gradient at that position relative to NLL. To quantify how this mechanism affects actual correction, we have added a diagnostic experiment based on verified correct supervision labels.
To further examine whether DEFT can correct confident but wrong priors, we run a forward pass with the fixed base model over all 67,601 mathematical training examples. We treat the label tokens inside the final answer's
\boxed{...}span as fully correct ground-truth tokens; this excludes tokens in the reasoning process. Before fine-tuning, we identify tokens with reference probability p≤0.1 and Shannon entropy H≤0.3, as well as those satisfying p≤0.3 and H≤0.5, as tokens reflecting confident but wrong priors (CBW tokens). Using NLL as the standard comparison, we measure the proportion of these CBW tokens whose reference token becomes the top-1 prediction after training with NLL or DEFT.Metric definition. The correction rate is the proportion of tokens identified as CBW before fine-tuning whose reference token becomes the top-1 prediction after training. The NLL and DEFT correction rates below are computed on the same CBW set. The relative correction rate is
RelativeCorrectionDEFT=CorrectionRateNLLCorrectionRateDEFT×100%
The results show that DEFT achieves approximately 79%–85% of NLL's correction rate, demonstrating that DEFT retains the ability to correct the model's confident but wrong priors.
The training trajectories below further show that tokens at initially confident but wrong positions can be corrected during DEFT training. Thus, initially suppressed positions remain learnable. This observation is consistent with the dynamic nature of the gate: DEFT recomputes the weights from the current predictive state, and being initially classified as a conflict does not permanently mask a position. Table 5 in the paper provides complementary evidence: restoring NLL weights in the dynamically identified conflict region reduces overall task performance. Taken together, these results show that DEFT improves overall adaptation performance while retaining the ability to correct initially conflicting positions.
The proportion (%) of CBW tokens whose reference token becomes the top-1 prediction during training under the Strict setting (p≤0.1, H≤0.3 bits) is shown below:
W2. Fundamental advantages over EAFT and RankTuner
Thank you for emphasizing this comparison. EAFT and RankTuner already use distributional information.
Below, we explain the rationale for this choice, its mechanistic differences from existing methods, and the empirical results.
For the generalized loss La(p)=(1−pa)/a, let r denote the supervision-label distribution for a fixed context, fix 0<a<1, and write β=1/(1−a) and Zβ=∑iriβ. Theorem 1/2 gives the optimal risk as
=\frac{1-Z_\beta^{1/\beta}}a =\frac{1-e^{-aH_\beta(r)}}a =L_a\!\left(e^{-H_\beta(r)}\right).$$ Thus, Rényi entropy characterizes the optimal supervised risk of the original loss, while $e^{-H_\beta(r)}$ is the equivalent probability that produces the same scalar loss. Rényi-2 corresponds exactly to the distributional statistic in the gradient of the original loss. Let $C_t=\sum_vP_t(v)^2$. Then $\lVert\nabla_{z_t}L_a\rVert_2^2=p_t^{2a}\lVert P_t-e_{y_t}\rVert_2^2=p_t^{2a}(1-2p_t+C_t)$ Therefore, $(p_t,C_t)$ exactly recovers the squared Euclidean length of the NLL correction vector. Within the power-sum candidate family $T_q=\sum_vP_t(v)^q$ for finite orders $q>1$, when the vocabulary size is at least $4$, only $q=2$ provides this exact scalar representation for every interior predictive distribution. For other orders, there exist predictions with the same $p_t,T_q$ but different $C_t$, so the length cannot be recovered from these two inputs.