acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features

Abstract

Supervised fine-tuning (SFT) learns most aggressively from tokens that the model deems least likely. This helps acquire new behaviors, but also amplifies noisy or conflicting supervision and can overwrite useful pretrained knowledge. Through a unified policy-loss view, we revisit existing token-reweighting methods and show that they assign nonnegative coefficients to demonstrated tokens. Consequently, they can suppress or amplify supervised updates, but cannot reverse harmful features once learned. Moreover, larger training weights do not amount to feature extrapolation, since they change the optimization trajectory rather than scale a fixed SFT direction. We argue that reversal and extrapolation require a stable reference frame defined by a fixed SFT delta. Motivated by this, we propose **SCALE** (**S**elective **C**ontrol of **A**daptation via **L**ocal **E**ntropy), an entropy-guided adaptation-strength-control method that freezes the pretrained model and the SFT delta and learns bounded token- and module-specific gates by minimizing predictive entropy alone. These gates suppress, reverse, or extrapolate frozen SFT features according to their alignment with entropy reduction. Across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B-Base, SCALE achieves mathematical-reasoning averages of 37.84, 43.60, and 36.57, exceeding the strongest corresponding baselines while remaining competitive on general-retention benchmarks. It also attains the best average code-generation performance across HumanEval, HumanEval+, and MBPP for all three models. These results suggest that effective SFT correction can benefit from controlling how already learned residuals are used, rather than only modifying how they are learned.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post Training, SFT

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ab2bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper identifies sensitivity to noisy reference and overcorrection as major limitations of SFT and claims that existing token reweighting methods are limited in addressing these issues. The authors introduce SCALE, a method that uses module-specific gating to adjust the strength of learned adaptations. The gates are trained by minimizing the model’s predictive entropy on the training data. Experiments QWEN model variants show improvements on mathematical reasoning and code generation.

Critical strengths
  1. The author’s claims on the limitations of SFT are well-grounded.

  2. The author’s claims on the limitations of existing token reweighting methods are convincing.

  3. The authors evaluate on a variety of math and coding benchmarks.

Critical weaknesses
  1. The motivation for using predictive entropy in adaptation-strength-control is unclear. The assumptions on entropy-task consistency are broad and experimentally unsupported.

  2. The main experiments favor SCALE by design. Table1 and Table2 compare baselines (stage 1 only) and SCALE (stage 1 AND stage 2). Table 6 in the appendix addresses this concern only partially, but shows results on a single QWEN model.

  3. The claimed benefits of SCALE are insufficiently demonstrated. It is unclear whether the method is enabling the model to actively “reverse” the learned feature as a result of SCALE training.

Questions for authors
  1. How does SCALE differ from existing entropy-based test-time-training methods?

  2. Can you provide experiment results conducted based on equal compute and data?

  3. Can you provide results for other model families?

3 comments

Q1. How does SCALE differ from entropy-based test-time training?

Thank you for raising this distinction. SCALE trains its gates on demonstration prefixes before deployment. At inference, one forward pass gives the gate for each token. TENT updates its parameters on test inputs.

Q2. What does matching the data control, and how much compute does SCALE use?

With the same 20K additional math examples, SCALE [−4,4] gained 21.57 points over LoRA +20k in Table 6.

With the same Stage-I adapter, 20K prefixes, entropy objective and 157 updates, LoRA +20k w/ Entropy scored 28.48 and SCALE [−4,4] scored 37.84. With equal training time on one H200 (about 22.7 minutes), SCALE [−3,3] scored 34.80 versus 29.04 for continued LoRA.

Table 6 · Qwen2.5-Math-1.5B · Math Avg@16 (%)

MethodMath500MinervaOlympiadAIME24AMC23Avg
LoRA39.609.4910.800.4117.0315.47
LoRA +20k40.129.7211.581.0418.9116.27
SCALE w/ CE41.9710.3111.360.8216.4116.17
SCALE w/ Entropy71.6031.4431.348.5546.2537.84

Figure 4 · single-H200 training time (minutes)

Method or stageMath Qwen2.5-Math-1.5BMath Qwen2.5-Math-7BCode Qwen2.5-Math-1.5BCode Qwen2.5-Math-7B
LoRA8818553124
DFT8718654123
DoRA130305109275
TALR9520166136
SCALE gates (Stage II)23462036
LoRA + one gate run11123173160

Q3. Does the result transfer to another model family?

On Gemma-3-4B-PT, SCALE raised the five-benchmark average from 4.74% with LoRA to 7.98%.

Gemma-3-4B-PT · Math Avg@16 (%)

MethodMath500MinervaOlympiadAIME24AMC23Avg
Base4.451.811.470.211.721.93
LoRA13.153.662.470.214.224.74
SCALE20.676.214.420.008.597.98

I appreciate the response by the authors. However, the concerns raised in the critical weaknesses section still remain unanswered. In particular, the motivational experiment in Figure 1(left), which summarizes the central claims of the paper, requires more explanation and deeper analysis to be more convincing. I will keep my current score.

Reviewer_Qn7EOpenReview ↗

Thanks.

I would like to thank the reviewers for answering some of my concerns. I'll maintain my score.

Reviewer_Qn7EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SCALE, which learns how strongly to apply a previously trained LoRA adapter by learning token- and module-specific gates through entropy minimization. Both the base model and the adapter remain frozen during gate training. The gates can suppress, reverse, or extrapolate the adaptation. Experiments on three models (all Qwen, up to 7B) show improvements esp. in mathematical reasoning and improvements in avg. code-gen. performance across models.

Critical strengths
  • Overall, the method is rather simple to apply.
  • Separating learning a LoRA adaptation and learning how strongly to apply this adaptation seems sensible.
  • Only introduces a few additional trainable parameters.
  • Interesting results, especially the improvements in mathematical reasoning.
  • The ablations show more detail, esp. by removing reversal or extrapolation and comparing against cross-entropy calibration. The paper also looks at both task perf. and more general perf. of the model.
Critical weaknesses
  • The headline experimental results use the best oracle setting chosen on the evaluation benchmarks, out of 42 different gate ranges. This is pointed out by the authors as well, but it makes it difficult to judge how much of the improvement would hold with validation-based model selection.
  • The continued-LoRA baseline uses cross-entropy. There is no comparison to a version that continues training the LoRA adapter through entropy minimization (e.g., an EM-FT-style baseline). Also, maybe compare to learning simpler gates, e.g., a single coefficient to scale the LoRA adaptation or a constant coefficient for each module, rather than token-dependent coefficients. These controls should also allow reversal and extrapolation.
  • Minimizing entropy can make the model more confident even when it is wrong. The theory in the paper provides an improvement guarantee under the assumption that gate updates that reduce entropy also tend to improve task performance. However, the paper does not show whether this assumption holds in their experiments.

References: EM-FT paper. https://arxiv.org/abs/2505.15134

Questions for authors
  1. How much of the improvement still holds when choosing the hyperparams (i.e., gate range) on a separate validation set and then evaluating the chosen configuration without further selection on the benchmarks?
  2. How does SCALE perform compared to: (1) continuing to train the LoRA adapter via entropy minimization; and (2) learning a single global coefficient or a constant coefficient per module via entropy minimization? I.e., use the same calibration data and objective, but change which parameters are learned.
  3. Can the authors provide examples where their entropy calibration does not help or hurts perf.? Maybe especially cases where the initial model already makes confidently wrong predictions?
1 comment

Q1. How much of the gain survives independent range selection?

Thank you for suggesting independent selection. We removed MATH training problems overlapping NuminaMath, the five benchmarks or the full MATH test split, then drew two disjoint sets of 128. On the first set, we sampled two answers per problem and chose the gate range with the highest average accuracy. We then evaluated that range on the second set. With matched decoding, SCALE beat DFT by 2.35, 4.52 and 0.69 points.

Five-benchmark Math Avg@16 (%) · validation-selected ranges · DFT from Table 1

BackboneMethodGate rangeMath Avg@16
Qwen2.5-Math-1.5BDFT—32.81
Qwen2.5-Math-1.5BSCALE[−6,2]35.16
Qwen2.5-Math-7BDFT—37.52
Qwen2.5-Math-7BSCALE[−3,3]42.04
Qwen3-4B-BaseDFT—34.66
Qwen3-4B-BaseSCALE[−3,1]35.35

We tested 42 ranges per Qwen2.5 model and five for Qwen3-4B-Base. Confirmation accuracy was 76.17%, 80.47% and 72.66%, respectively.

Q2. How do continued LoRA, static gates and SCALE compare?

SCALE gained 3.06 points over per-projection gates under the same entropy objective.

Qwen2.5-Math-1.5B · gate range [−4,4] · Math Avg@16 (%)

MethodMath500MinervaOlympiadAIME24AMC23Avg
LoRA +20k w/ Entropy62.2420.0923.114.7932.1928.48
Global gate36.6110.0110.181.2416.2514.86
Per-projection gate68.8521.7528.957.9346.4134.78
SCALE71.6031.4431.348.5546.2537.84

Continued LoRA updated the adapter. The three gate variants froze it and used the same calibration data.

Q3. When does entropy calibration reinforce a wrong answer?

MATH500 · example 1

User message
Given $f(x) = \frac{\sqrt{x-1}}{x-2}$, what is the smallest possible integer value for $x$ such that $f(x)$ has a real number value?

The square root requires x ≥ 1, and the denominator excludes only x = 2. At x = 1 the expression equals zero, so 1 is the smallest admissible integer. The Stage-I model already favored 3. After calibration, that wrong answer rose from 10/16 to 14/16 draws, while the correct answer fell from 4/16 to 2/16.

Across all 500 MATH500 problems, mean accuracy rose from 39.60% with Stage-I LoRA to 69.71% with validation-selected SCALE [−6,2]. With 16 draws per problem from each Qwen2.5-Math-1.5B model, the number of correct answers rose on 414 problems, stayed the same on 53 and fell on 33 (6.6%).

MATH500 · overall accuracy and two examples of regression

MethodMATH500 Avg@16Wrong 3 on domain itemCorrect 6 on log₂(64)
LoRA39.60%10/1615/16
SCALE [−6,2]69.71%14/167/16

MATH500 · example 2

User message
Evaluate $\log_264$.

The answer is 6 because 2⁶ = 64. Correct answers fell from 15/16 to 7/16 draws, and SCALE returned 5 in eight draws.

SCALE keeps the base model and the LoRA adapter frozen. A bounded gate lets it reduce the adapter’s contribution at one token without weakening it at every token. The gate still learns from entropy alone. On the domain problem, it reinforced the answer 3, consistent with treating the denominator as positive when it only needs to be nonzero.

Reviewer_NimnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCALE freezes a LoRA model after SFT and uses entropy minimization to train token-wise gates that suppress, reverse, or amplify the learned delta. The paper reports gains in math and code on three backbones while largely retaining general capabilities. Range sweeps and clipping ablations examine the role of negative coefficients and coefficients above one.

Critical strengths
  1. The method is lightweight and easy to add to an existing LoRA model. Stage II needs no correctness labels and reportedly takes 20–46 minutes on one GPU (Appendix G).

  2. The math gains are substantial. Table 7 reports 37.83±0.20 over three seeds for 1.5B with a fixed range, versus 32.31±0.92 for DFT. Table 5 also connects the entropy signal to downstream accuracy.

Critical weaknesses
  1. How much of the gain comes from the gate, and how much from entropy training? In Table 6, the same gate scores 16.17 with cross-entropy and 37.84 with entropy, but there is no control that continues training LoRA from the same Stage-I checkpoint with entropy. EM-FT already uses entropy training for reasoning, and AdaMerging/WEMoE use entropy to learn coefficients of frozen deltas. A matched LoRA control and one constant coefficient per module would show what the token-wise signed gate adds. The claim about correcting harmful features from noisy supervision also goes beyond the evidence: Table 5 does not connect those features to noisy demonstrations, and removing some reversals improves HumanEval by 2.85 points in Table 10.

  2. The main tables report the best results from a search, so the practical gain is unclear. Section 6.1 selects among 42 ranges using the corresponding evaluation results; Figure 3 explicitly calls this an oracle sweep. The fixed-range result in Table 7 helps, but for 1.5B, [-6,3] gives about 37.8 in Tables 6–7, about 37.0 by visual reading of Figure 3, and 37.05 in Table 9. I also could not tell whether the code metrics and capability-retention scores use the same checkpoint. Please list the range and selection rule for each row, and report fixed-range or independent-validation selection. That would make the size of the lead much easier to judge.

Questions for authors
  1. With the same checkpoint, data, and entropy objective, how do continued LoRA training and one constant coefficient per module compare with SCALE? A clear advantage in accuracy, stability, or retention would strengthen the method contribution.

  2. Which range and checkpoint does each row of Tables 1–2 use? Are models shared across code and retention metrics? Please explain the [-6,3] discrepancy and give the decoding and answer-extraction settings.

  3. Is there controlled noisy-data evidence for correcting harmful features? If not, please narrow that claim. Do all token-reweighting baselines use the same LoRA configuration?

2 comments

Q1. What does token-wise gating add?

Thank you for suggesting these controls. SCALE gained 9.36 points over continued LoRA and 3.06 points over a gate with one constant coefficient per module.

Qwen2.5-Math-1.5B · gate range [−4,4] · Math Avg@16 (%)

MethodMath500MinervaOlympiadAIME24AMC23Avg
LoRA +20k w/ Entropy62.2420.0923.114.7932.1928.48
Per-projection gate68.8521.7528.957.9346.4134.78
SCALE71.6031.4431.348.5546.2537.84

All three methods used the same Stage-I adapter, 20K prefixes and top-100 entropy objective. We tuned the LoRA learning rate separately.

Q2. How were ranges selected?

For each backbone, the Table 1 reports retention on the checkpoint with the highest test Math Avg@16. Table 2 selects by HumanEval pass@1.

Test-selected checkpoints (%)

BackboneMath rangeMath Avg@16RetentionCode rangeHumanEval pass@1
Qwen2.5-Math-1.5B[−4,4]37.8433.34[0,6]41.67
Qwen2.5-Math-7B[−5,5]43.6045.40[0,6]57.11
Qwen3-4B-Base[−4,1]36.5748.34[0,5]53.25

We held out two sets of 128 problems from the MATH training split. On the first set, we sampled two answers per problem and chose the gate range with the highest average accuracy. We then evaluated that range on the second set.

Validation-selected math checkpoints (%)

BackboneSelected rangeConfirmation accuracyMath Avg@16
Qwen2.5-Math-1.5B[−6,2]76.1736.59
Qwen2.5-Math-7B[−3,3]80.4742.10
Qwen3-4B-Base[−3,1]72.6635.35

With the gate range fixed at [0,6] across all three models, HumanEval pass@1 was 41.7% for Qwen2.5-Math-1.5B, 57.1% for Qwen2.5-Math-7B and 52.8% for Qwen3-4B-Base (Figure 3).

Q3. Which ranges and evaluation settings were used?

The correct range for the 37.84 score in Table 6 is [−4,4].

Qwen2.5-Math-1.5B math results

SCALE configurationRangeMath Avg@16 (%)
Table 6, entropy[−4,4]37.84
Figure 3 sweep[−6,3]36.99
Table 7[−6,3]37.83 ± 0.20
Table 9, gate intact[−6,3]37.05

We evaluated the same [−6,3] checkpoint twice and obtained 36.99 in Figure 3 and 37.05 in Table 9. Table 7 reports the mean and standard deviation across three training seeds at that range.

Shared prompt for all three Qwen models

System instruction
Please reason step by step, put your final answer within \boxed{}, and stop immediately after the boxed answer.
User message
{benchmark question, verbatim}

Math used 16 draws, seed 0, temperature 1, top-p 1 and a 4,096-token limit. The grader normalized the last boxed answer, marked answer or final number, depending on the benchmark.

For Qwen2.5-Math-1.5B and Qwen2.5-Math-7B, we extracted an answer from over 96% of outputs.

Code used function completion with a 1,024-token limit. HumanEval used three draws (temperature 0.2, top-p 0.95) for pass@1. HumanEval+ and MBPP used greedy decoding.

Q4. What do reversals show?

On 30 AIME24 problems, SFT raised next-token entropy by over 0.05 nats at 5,912 of 33,879 positions (Figure 1, upper left).

SCALE lowered entropy on these same prefixes in every base-entropy quintile.

Entropy change (nats)

Method comparisonQ1Q2Q3Q4Q5
Stage-I LoRA − base+0.138+0.212+0.272+0.255+0.292
SCALE − Stage-I LoRA−0.144−0.315−0.533−0.713−1.168

In Figure 1 (lower left), reversal lowered mean entropy relative to SFT in four of five bins. In layers 16–27, reversal was 1.25 points more frequent at “harmful” than “helpful” positions.

Inference-time clipping (%)

Gate at inferenceQwen2.5-Math-1.5B math Avg@16Qwen2.5-Math-7B math Avg@16Qwen2.5-Math-7B code HumanEval pass@1
Learned gate intact37.0542.5249.80
Negative coefficients set to 020.1033.3752.64

Qwen2.5-Math was pretrained for mathematics. Reversal can recover a useful math prediction or undo a useful code update. Removing reversal hurt math on both models but improved Qwen2.5-Math-7B HumanEval. Capping the [0,6] gate at one also cut HumanEval from 57.11 to 47.15.

Reviewer_NimnOpenReview ↗

Reply to author's rebuttal

Thank you for the clarification. Your response helped me better understand the paper. After considering your explanation, I have decided to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.