acceptodds
Under review as a conference paper at ICLR 2027

Equal Gain per Unit Statistical Distance: A Principle for Token Reweighting

Abstract

Supervised fine-tuning (SFT) offers a simple way to learn from fixed demonstrations, but improving its generalization remains a challenge. Recent methods assign token-level weights to the cross-entropy loss and report empirical gains, yet benchmark results alone do not explain how these weights value learning progress. In this work, we compare token weights in a common unit: the objective gain each weight assigns per unit of Fisher–Rao (FR) distance toward the demonstrated token. We find that Dynamic Fine-Tuning (DFT) and InfoSFT each assign more uniform gains than SFT over different ranges of target-token probability, offering a possible explanation for their empirical advantages. Building on this insight, we derive Vocab Relative Advantage (VRA), the token weight that gives uniform gain per unit of FR distance across the probability range and is unique up to scale. VRA achieves higher average accuracy than both SFT and DFT on each of six backbones. On Qwen3-8B, it reaches 39.23% across five mathematics benchmarks, compared with 29.30% for SFT, and also surpasses DFT, InfoSFT and ProFit. Matched one-step comparisons further show that, relative to SFT, VRA redistributes progress from low-confidence targets to middle- and high-confidence targets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dGtcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an information-geometric framework to compare token-level reweighting schemes in supervised fine-tuning (SFT). By projecting the categorical next-token distribution onto a binary target-versus-rest Bernoulli distribution, the authors measure the local objective gain per unit of Fisher-Rao (FR) arc length. Under this metric, standard SFT assigns diverging gain as target probability p approaches zero, whereas probability-weighted methods such as DFT and InfoSFT assign vanishing gain at both endpoints. The authors propose an equal-gain principle that requires constant marginal gain per unit FR distance across all confidence levels, which uniquely yields the Vocab Relative Advantage (VRA) weight w(p) = sqrt(p / (1 - p)) up to scale. They also show that this weight matches the analytical expectation of a GRPO-style standardized target-match reward over the vocabulary. Using a stabilized weight sqrt(p / (1 - p + eps)), VRA outperforms SFT and DFT in average mean@8 across six language model backbones on five mathematics benchmarks, transfers to on-policy verifier-accepted rollouts, and shifts one-step parameter updates from low-confidence tokens toward middle- and high-confidence tokens.

Critical strengths
  1. The unifying framework is clean and illuminating. Expressing detached token weights through marginal utility per unit Bernoulli Fisher-Rao distance (Table 1 and Figure 1) provides a parameter-invariant lens to contrast SFT, DFT, InfoSFT, and ProFit, while the alternative derivation in Appendix C neatly connects VRA to vocabulary-level reward standardization without token sampling.
  2. The mechanistic analysis goes beyond nominal gradient inspection. In Section 6.2, the authors restore actual SFT checkpoints and AdamW states to run matched one-step updates under global gradient clipping, verifying that VRA's directional redistribution from p < 0.1 to p >= 0.1 survives adaptive per-parameter normalization.
  3. The paper is written with unusual precision regarding what its theory and diagnostics do and do not establish. The authors explicitly distinguish nominal objective profiles from realized multi-parameter updates, report negative or null structural differences honestly in Appendix E.3, and validate the method across six backbones as well as on-policy rollout and KL-regularized regimes in Appendix D.
Critical weaknesses
  1. The paper does not isolate the empirical contribution of the high-confidence factor (1 - p + eps)^(-1/2) from the low-confidence factor p^(1/2), nor does it ablate the stabilization constant eps. Relative to DFT (w(p) = p), VRA changes the low-confidence scaling to sqrt(p) (which coincides with the square-root member of SAFT) and simultaneously introduces (1 - p + eps)^(-1/2), which strongly amplifies weights as p approaches 1. Because 45.57% of the gold census tokens have p > 0.996 and 35.19% have p >= 0.999 (Figure 3a), the choice of eps = 10^-3 directly determines the weight cap (approx. 31.6) on over a third of all training tokens. Without ablating w(p) = sqrt(p) and varying eps in the gold-data setting, it is unclear whether VRA's advantage over DFT stems from uniform FR gain across the full interval or primarily from softer low-probability suppression vs. high-probability amplification.
  2. The theoretical justification equates constant objective derivative dU_w / d_ell with balanced learning incentives, but actual progress under gradient flow is not uniform across p. As shown in Section 4.2, under Euclidean logit gradient ascent z_dot = w(p)(1 - p), the induced velocity in Fisher-Rao arc length is ell_dot = w(p) sqrt(p) (1 - p)^(3/2), which for VRA equals p(1 - p) and still vanishes at both endpoints. Furthermore, the Bernoulli coarsening discards how the negative gradient -w(p) pi_theta(a | s) redistributes mass among the V - 1 non-target tokens.
  3. The main gold-data evaluation in Section 5 is limited to a single domain (mathematical reasoning on NuminaMath-CoT 30K) and relies on an unweighted macro-average (AVG) where AIME24 (30 problems) and AMC23 (40 problems) carry the same weight as MATH-500 (500 problems) and OlympiadBench (675 problems). On Qwen2.5-7B (Table 3), VRA exceeds DFT by only 0.15 points in AVG (31.99 vs. 31.84), and on Qwen3-4B (Table 2), VRA underperforms DFT, InfoSFT, and ProFit on Minerva Math (24.65 vs. 29.48 for DFT) while winning the macro-average largely through gains on the 30 AIME24 and 40 AMC23 problems. In addition, InfoSFT and ProFit are omitted from the four additional backbones in Table 3.
Questions for authors
  1. How does VRA compare on gold-data fine-tuning (Qwen3-8B and Qwen3-4B) against the unnormalized square-root weight w(p) = sqrt(p) (which matches VRA's low-confidence asymptotics without blowing up near p = 1) and across different values of the stabilizer eps (e.g., 10^-1, 10^-2, 10^-4)?
  2. In Appendix E.2 (Figure 7), VRA and ProFit achieve nearly identical gold-target acquisition in both the middle-probability group (0.8855 vs. 0.8854) and high-probability group (0.1415 vs. 0.1402) at step 300, yet differ by 2.77 points in benchmark AVG. Have you examined matched one-step updates or non-target entropy changes comparing VRA directly against DFT and ProFit (rather than only against unit-weight SFT) to explain what drives their downstream performance gap?
  3. Could you report the problem-count-weighted (micro-averaged) accuracy and per-benchmark breakdowns for the four additional models in Table 3, as well as gold-data SFT results on a non-math task (such as code generation or instruction following) to verify that the equal-gain weight generalizes beyond NuminaMath-CoT in the standard teacher-forced setting?
8 comments
AuthorsOpenReview ↗

Response to Reviewer dGtc (1/6)

We are sincerely grateful for your thorough and detailed review of our paper.
Below we detail our response to the weaknesses (i.e., W1$\sim$W3) and questions (i.e., Q1$\sim$Q3) point by point. Please kindly let us know whether our response resolves the issues you raised, or whether there are any remaining concerns. Please note that the section, figure, and table numbers mentioned below are based on the original submission unless we explicitly refer to the revised manuscript; their relative positions may change in the revision.

(W1) The paper does not isolate the empirical contribution of the high-confidence factor (1−p+ϵ)−1/2(1-p+\epsilon)^{-1/2} from the low-confidence factor p1/2p^{1/2}, nor does it ablate the stabilization constant ϵ\epsilon. Without ablating w(p)=pw(p)=\sqrt{p} and varying ϵ\epsilon in the gold-data setting, it is unclear whether VRA's advantage over DFT stems from uniform FR gain across the full interval or primarily from softer low-probability suppression vs. high-probability amplification.

Thank you for this precise analysis of the two factors in the VRA weight. Separating them makes the source of the improvement much clearer, and we have added the ablations in Appendices G.1 and G.2 of the revised manuscript. The two factors can be separated, and neither one alone accounts for the advantage: it follows uniform gain over the full interval. The tables are in our response to (Q1); throughout, AVG is the unweighted mean of mean@8 over the five benchmarks, as in Table 2.

  • Softer low-probability suppression alone does not explain it. Replacing pp with p\sqrt{p} changes AVG from 36.93 to 37.54 on Qwen3-8B and from 28.02 to 27.64 on Qwen3-4B (Table R1).
  • Nor does high-probability amplification as such. VRA adds the factor (1−p+ϵ)−1/2(1-p+\epsilon)^{-1/2} to p\sqrt{p} and reaches 39.23 on Qwen3-8B, but amplifying high confidence further (exponent 0.750.75) lowers AVG to 37.75 (Table R1).
  • The stabilizer ϵ\epsilon does not drive the result. It caps the weight on a third of the targets, but these targets carry only 1.24% of the target-logit signal, and AVG stays within 37.43–39.23 over four orders of magnitude of ϵ\epsilon (Table R2).
AuthorsOpenReview ↗

Response to Reviewer dGtc (2/6)

(W2) The theoretical justification equates constant objective derivative dUw/dℓdU_w/d\ell with balanced learning incentives, but actual progress under gradient flow is not uniform across pp. Furthermore, the Bernoulli coarsening discards how the negative gradient −w(p)πθ(a∣s)-w(p)\pi_\theta(a\mid s) redistributes mass among the V−1V-1 non-target tokens.

Thank you for this careful theoretical analysis, which helped us state more precisely what the principle claims; we have added the clarification as Appendix B.6 of the revised manuscript. We would like to respectfully clarify that equal gain is a statement about how the objective values progress, not about optimization speed, and that the Bernoulli projection loses no progress toward the demonstration.

  • Equal gain specifies the objective's valuation of progress, not a uniform optimization speed.
    • We agree with your calculation. Under the single-logit Euclidean gradient flow in Section 4.2, ℓ˙=(dℓdz)2dUwdℓ=p(1−p) Cw(p)\dot\ell=\left(\frac{d\ell}{dz}\right)^2\frac{dU_w}{d\ell}=p(1-p)\,C_w(p).
    • The factor p(1−p)p(1-p) comes from the logit parameterization and is the same for every weighting rule, so the speed ratio of any two rules at the same pp equals the ratio of their profiles; the profile is the only part a token weight controls.
    • The ideal VRA rule sets Cw(p)=1C_w(p)=1, so its FR velocity is p(1−p)p(1-p) and vanishes at both endpoints, consistent with our acquisition-time analysis.
    • A weight that instead equalized ℓ˙\dot\ell under this flow would be w∝p−1/2(1−p)−3/2w\propto p^{-1/2}(1-p)^{-3/2}, i.e., Cw∝1/[p(1−p)]C_w\propto1/[p(1-p)]: it rewards low-confidence progress more steeply than SFT and changes with the parameterization, whereas CwC_w does not.
    • By balanced incentives, we mean that an equal target-event FR displacement receives equal marginal objective gain, independently of confidence. This does not require equal displacement per training step, which is why we separately analyze the objective profiles, scalar acquisition dynamics, and realized optimizer updates.
  • The Bernoulli projection loses no progress toward the demonstration.
    • All rules in our comparison use weights that are functions of the target probability pp, with other rule parameters fixed, so their local utilities depend on the vocabulary distribution only through pp.
    • The same holds for the distance: on the full vocabulary simplex, the Fisher–Rao distance from πθ(⋅∣s)\pi_\theta(\cdot\mid s) to a distribution qq is 2arccos⁡∑aπθ(a∣s) q(a)2\arccos\sum_a\sqrt{\pi_\theta(a\mid s)\,q(a)}, which for the one-hot demonstration equals 2arccos⁡p=ℓ(1)−ℓ(p)2\arccos\sqrt p=\ell(1)-\ell(p), the remaining Bernoulli arc length. CwC_w is therefore exactly the gain per unit reduction in full-vocabulary distance to the demonstrated token.
  • What the projection omits is rearrangement among non-target tokens at fixed pp.
    • For any detached weight the per-token logit gradient is w(p)(ea⋆−πθ)w(p)(e_{a^\star}-\pi_\theta), so the term −w(p) πθ(a∣s)-w(p)\,\pi_\theta(a\mid s) has the same direction for every rule and differs only by the scalar w(p)w(p); token by token, it cannot separate the rules.
    • The vocabulary-level derivation in Appendix C is consistent with this: it sums all non-target terms exactly and assumes nothing about their probabilities. The reward there does not distinguish among non-target alternatives, but their individual probabilities remain in the full softmax gradient.
    • The scalar profile therefore does not claim to determine the redistribution among non-target alternatives. Rule-dependent differences in it arise only through aggregation across tokens in shared parameters, where realized updates are an empirical question; Section 6.2 measures them on target probabilities. A weight that additionally depended on non-target entropy or relative probabilities would extend the class of rules beyond w(p)w(p).
AuthorsOpenReview ↗

Response to Reviewer dGtc (3/6)

(W3) The main gold-data evaluation in Section 5 is limited to a single domain (mathematical reasoning on NuminaMath-CoT 30K) and relies on an unweighted macro-average (AVG) where AIME24 (30 problems) and AMC23 (40 problems) carry the same weight as MATH-500 (500 problems) and OlympiadBench (675 problems). In addition, InfoSFT and ProFit are omitted from the four additional backbones in Table 3.

Thank you for these helpful comments on the breadth of our evaluation, which led us to extend it in Appendices G.5 and G.6 of the revised manuscript. VRA remains the best fine-tuned method under micro-averaging on five of the six backbones and on a second domain, medical question answering. We would also like to respectfully clarify the scope of Table 3. The tables are in our response to (Q3).

  • The advantage extends to a second domain. We fine-tuned Qwen3-8B on gold medical question-answering data, where VRA has the highest macro and micro accuracy: 53.59 and 57.41, vs. 52.24 and 55.79 for Base (Table R8).
  • VRA's lead holds under micro-averaging on five of the six backbones. When each benchmark is weighted by its number of problems, VRA remains the best fine-tuned method on all but Qwen2.5-7B (Tables R6 and R7), where DFT is higher (39.35 vs. 38.22).
  • InfoSFT and ProFit are compared on the two main models; Table 3 has a narrower scope. On Qwen3-8B and Qwen3-4B (Table 2), VRA has the highest AVG against both baselines. We do not have InfoSFT and ProFit runs on the four additional backbones; Table 3 examines whether the SFT/DFT/VRA comparison extends to them.
Reviewer_hXpzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares token weights through objective gain per unit Fisher–Rao distance and derives VRA as the weight making that gain constant. The stabilised rule improves average maths accuracy over SFT and DFT on six backbones, with diagnostics showing how it redistributes learning across token-confidence levels.

Critical strengths

I liked the clean connection between the geometric argument and a simple, cheap training change. The paper carefully distinguishes objective incentives from actual optimiser behaviour.

The gains extend across six backbones, with useful comparisons against recent weighting methods. The matched one-step experiments add substance beyond benchmark tables.

Critical weaknesses

The main results lack training-seed variation and uncertainty estimates. This matters particularly for the 0.15-point gain over DFT on Qwen2.5-7B and the small AIME/AMC test sets.

Equal gain is an appealing design choice, but its necessity remains untested. The experiments do not isolate it from nearby weighting shapes or method-specific learning-rate effects; the scale controls in Appendix D concern the separate on-policy setting.

Questions for authors

Do the main gains survive repeated training runs and comparable learning-rate searches for each method?

How sensitive are results to the stabiliser and nearby weight exponents? This would help establish whether the exact equal-gain shape matters.

4 comments
AuthorsOpenReview ↗

Response to Reviewer hXpz (1/3)

Thank you very much for your careful review and insightful questions.
Below we detail our response to the weaknesses (i.e., W1$\sim$W2) and questions (i.e., Q1$\sim$Q2) point by point. If any concern remains after reading our response, please kindly let us know, and we will be glad to address it. Please note that the section, figure, and table numbers mentioned below are based on the original submission unless we explicitly refer to the revised manuscript; their relative positions may change in the revision.

(W1) The main results lack training-seed variation and uncertainty estimates. This matters particularly for the 0.15-point gain over DFT on Qwen2.5-7B and the small AIME/AMC test sets.

Thank you for pointing this out. Seed variation and uncertainty estimates were missing from the submission; we have added them in Appendix G.3 of the revised manuscript, which makes our main comparison much more reliable. With three training seeds, VRA's AVG is above DFT's on every seed of all three backbones, including Qwen2.5-7B. The tables are in our response to (Q1); throughout, AVG is the unweighted mean of mean@8 over the five benchmarks, as in Table 2.

  • The gains hold across training seeds.
    • We added two training seeds (43 and 44) for SFT, DFT, and VRA on Qwen3-8B, Qwen3-4B, and Qwen2.5-7B, and report the mean ± standard deviation on every benchmark (Table R1).
    • VRA's three-seed AVG is 38.98 ± 0.26 vs. 37.08 ± 0.13 for DFT on Qwen3-8B, and 29.53 ± 0.09 vs. 27.81 ± 0.71 on Qwen3-4B (Table R1).
  • On Qwen2.5-7B, the margin over DFT is larger on both new seeds.
    • We agree that a 0.15-point gap from a single run cannot be separated from seed variation.
    • On the two new seeds, VRA is above DFT by 1.57 and 1.15 points (Table R2), and the three-seed means are 32.90 ± 0.87 for VRA and 31.94 ± 0.18 for DFT (Table R1).
  • On the two Qwen3 models, the advantage does not depend on the small AIME/AMC sets.
    • To check that the comparison does not rest on these two sets, we also weight each benchmark by its number of problems (Micro column of Table R2).
    • Under this weighting, VRA remains above DFT on the two Qwen3 models (45.56 vs. 44.63 on Qwen3-8B; 37.03 vs. 35.31 on Qwen3-4B). On Qwen2.5-7B the two methods are level (39.37 vs. 39.48; Table R2).

(W2) Equal gain is an appealing design choice, but its necessity remains untested. The experiments do not isolate it from nearby weighting shapes or method-specific learning-rate effects; the scale controls in Appendix D concern the separate on-policy setting.

Thank you for this insightful comment. Testing whether equal gain is needed, and not only appealing, makes the case for the principle much stronger; we have added both controls in Appendices G.1 and G.4 of the revised manuscript. We agree that the scale controls in Appendix D belong to the on-policy setting and do not address the main experiments. In the main gold-data setting, VRA remains the best against nearby weighting shapes and under a separate learning-rate search for each method. The tables are in our responses to (Q1) and (Q2).

  • Every nearby weighting shape falls below VRA.
    • Both neighboring exponents of the VRA-shaped weight fall below VRA's 39.23 AVG: 35.65 at β=0.25\beta=0.25 and 37.75 at β=0.75\beta=0.75 (Table R4).
    • So do all the simpler power weights pγp^{\gamma}: the best of them, p\sqrt{p}, reaches 37.54 (Table R5).
  • The advantage survives a separate learning-rate search for each method.
    • With four learning rates for each method and the best one chosen separately for each, VRA reaches 40.92 AVG, above DFT's 37.79 and SFT's 30.15 (Table R3).
AuthorsOpenReview ↗

Response to Reviewer hXpz (2/3)

(Q1) Do the main gains survive repeated training runs and comparable learning-rate searches for each method?

Thank you for this important question. Repeated runs and comparable learning-rate searches make our main comparison much more reliable, and we have added both in Appendices G.3 and G.4 of the revised manuscript. The main gains survive both checks: VRA stays above DFT on every training seed, and when the learning rate is chosen separately for each method.

  • The gains survive repeated training runs.
    • We added two training seeds (43 and 44) for SFT, DFT, and VRA on Qwen3-8B, Qwen3-4B, and Qwen2.5-7B. VRA's AVG is above DFT's on every seed of all three backbones (Table R2); on Qwen2.5-7B, where the original margin was 0.15 points, the margins on the two new seeds are 1.57 and 1.15 points.
    • VRA also has the highest three-seed mean on each backbone (Table R1): 38.98 vs. 37.08 for DFT on Qwen3-8B, 29.53 vs. 27.81 on Qwen3-4B, and 32.90 vs. 31.94 on Qwen2.5-7B.
  • The gains survive a separate learning-rate search for each method.
    • We trained SFT, DFT, and VRA on Qwen3-8B with three additional learning rates (Table R3). With the best of the four chosen separately for each method, VRA reaches 40.92 AVG, above DFT's 37.79 and SFT's 30.15.

Table R1. Repeated runs in the main gold-data setting (same configuration, checkpoint selection, and evaluation as Tables 2 and 3). Mean ± standard deviation of mean@8 over training seeds 42, 43, and 44; seed 42 is the run in Table 2, or in Table 3 for Qwen2.5-7B. AVG is the unweighted mean of mean@8 over the five benchmarks:

ModelMethodAIME24MATH-500MinervaOlympiadAMC23AVG
Qwen3-8BSFT5.00 ± 0.4261.91 ± 1.4620.55 ± 1.4323.47 ± 1.0437.50 ± 0.3129.69 ± 0.40
Qwen3-8BDFT6.81 ± 0.8769.55 ± 0.7330.85 ± 2.4933.39 ± 0.7344.79 ± 3.1637.08 ± 0.13
Qwen3-8BVRA6.81 ± 2.1070.31 ± 1.5530.27 ± 4.3934.68 ± 1.0052.81 ± 2.1938.98 ± 0.26
Qwen3-4BSFT3.46 ± 1.6851.77 ± 2.8913.88 ± 0.8517.77 ± 1.3625.73 ± 2.9022.52 ± 1.13
Qwen3-4BDFT2.90 ± 0.4357.71 ± 1.7023.17 ± 5.6125.38 ± 2.9429.90 ± 1.6027.81 ± 0.71
Qwen3-4BVRA5.84 ± 1.2659.05 ± 0.7623.51 ± 1.1627.90 ± 2.5731.35 ± 2.1929.53 ± 0.09
Qwen2.5-7BSFT3.75 ± 0.7254.28 ± 0.8114.80 ± 1.0919.18 ± 0.0429.79 ± 1.8324.36 ± 0.16
Qwen2.5-7BDFT7.23 ± 1.2765.58 ± 1.0122.88 ± 0.9528.51 ± 0.3835.52 ± 1.0031.94 ± 0.18
Qwen2.5-7BVRA9.04 ± 1.2065.61 ± 2.0821.03 ± 0.6128.62 ± 0.7640.21 ± 2.0132.90 ± 0.87

Table R2. AVG of each run of Table R1, by training seed. Micro is the three-seed mean when each benchmark is weighted by its number of problems (30 / 500 / 272 / 675 / 40) instead of equally. Bold marks the best method for each model:

ModelMethodSeed 42Seed 43Seed 44Micro
Qwen3-8BSFT29.3030.1029.6635.62
Qwen3-8BDFT36.9337.1437.1644.63
Qwen3-8BVRA39.2338.7238.9845.56
Qwen3-4BSFT21.3222.7023.5528.21
Qwen3-4BDFT28.0228.3927.0335.31
Qwen3-4BVRA29.4429.6329.5337.03
Qwen2.5-7BSFT24.5224.2024.3729.94
Qwen2.5-7BDFT31.8432.1531.8539.48
Qwen2.5-7BVRA31.9933.7233.0039.37

Table R3. Learning rates on Qwen3-8B (same setting as Table 2, changing only the learning rate; the 10−510^{-5} rows are from Table 2). Bold marks each method's best AVG:

MethodLearning rateAIME24MATH-500MinervaOlympiadAMC23AVG
SFT5×10−65\times10^{-6}6.2559.8519.2920.9034.3828.13
SFT10−510^{-5} (paper)4.5860.2322.1822.3637.1929.30
SFT3×10−53\times10^{-5}3.7463.0325.0526.7831.8830.09
SFT5×10−55\times10^{-5}6.2660.4023.2125.8635.0030.15
DFT5×10−65\times10^{-6}4.1570.8031.9334.2647.8137.79
DFT10−510^{-5} (paper)5.8469.0027.9833.4148.4436.93
DFT3×10−53\times10^{-5}7.1062.9029.3027.8340.9433.61
DFT5×10−55\times10^{-5}6.6655.5823.8123.9634.0628.82
VRA5×10−65\times10^{-6}10.4371.9534.2036.4851.5640.92
VRA10−510^{-5} (paper)8.7568.5331.6033.5453.7539.23
VRA3×10−53\times10^{-5}7.5363.4823.5931.7940.0033.28
VRA5×10−55\times10^{-5}5.0057.5025.4327.0138.4430.68
AuthorsOpenReview ↗

Response to Reviewer hXpz (3/3)

(Q2) How sensitive are results to the stabiliser and nearby weight exponents? This would help establish whether the exact equal-gain shape matters.

Thank you for this constructive question. It goes directly to whether the exact equal-gain shape matters, and we have added both studies in Appendices G.1 and G.2 of the revised manuscript. The results are sensitive to the shape of the weight and not to the stabilizer.

  • The shape matters: every nearby alternative falls below equal gain.
    • Within the VRA-shaped family (p1−p+ϵ)β\big(\frac{p}{1-p+\epsilon}\big)^{\beta}, both neighbors of the equal-gain exponent β=0.5\beta=0.5 are lower than VRA's 39.23 AVG: 35.65 at β=0.25\beta=0.25 and 37.75 at β=0.75\beta=0.75 (Table R4).
    • Outside this family, all the power weights pγp^{\gamma} are lower as well (Table R5). The best of them, p\sqrt{p} at 37.54, shares VRA's low-confidence scaling but not its high-confidence factor.
  • The stabilizer matters little.
    • AVG shows no clear collapse anywhere in the range of ϵ\epsilon: it stays within 37.43–39.23 over four orders of magnitude (Table R6).
    • Moving closer to the ideal weight does not improve over ϵ=10−3\epsilon=10^{-3}: AVG is 38.32 at 10−410^{-4} and 38.86 at 10−510^{-5}, against 39.23 (Table R6).
  • The two findings are consistent, because the two changes act on very different parts of the training signal.
    • The exponent tilts the gain profile over the whole confidence range, whereas ϵ\epsilon changes it only for targets with p≥0.999p\ge0.999, whose gradient w(p)(1−p)w(p)(1-p) is small.
    • These targets supply 1.24% of the target-logit signal at ϵ=10−3\epsilon=10^{-3} and 2.60% with no stabilizer at all (last column of Table R6).

Table R4. VRA-shaped weights w(p)=(p1−p+ϵ)βw(p)=\big(\frac{p}{1-p+\epsilon}\big)^{\beta} with ϵ=10−3\epsilon=10^{-3} on Qwen3-8B (main gold-data setting, same configuration, checkpoint selection, and evaluation as Table 2; the SFT and VRA rows are from Table 2). β=0.5\beta=0.5 is VRA and β=0\beta=0 is SFT; Figure 10 of the revised manuscript plots the weights:

β\betaAIME24MATH-500MinervaOlympiadAMC23AVG
00 (SFT)4.5860.2322.1822.3637.1929.30
0.250.259.5868.8523.9532.1443.7535.65
0.50.5 (VRA, paper)8.7568.5331.6033.5453.7539.23
0.750.759.1868.5531.9934.3544.6937.75

Table R5. Power weights w(p)=pγw(p)=p^{\gamma} on Qwen3-8B (same setting as Table R4; the SFT and DFT rows are from Table 2). They interpolate between SFT (γ=0\gamma=0) and DFT (γ=1\gamma=1):

γ\gammaAIME24MATH-500MinervaOlympiadAMC23AVG
00 (SFT)4.5860.2322.1822.3637.1929.30
0.250.257.9167.0025.9828.2439.0633.64
0.50.510.4069.5528.2033.9045.6337.54
0.750.7510.4369.9022.8534.5444.0636.36
11 (DFT)5.8469.0027.9833.4148.4436.93

Table R6. Stabilizer ϵ\epsilon of VRA on Qwen3-8B (same setting as Table R4; the ϵ=10−3\epsilon=10^{-3} row is from Table 2). The last column is the share of the total target-logit signal ∑twϵ(pt)(1−pt)\sum_t w_\epsilon(p_t)(1-p_t) that comes from targets with p≥0.999p\ge0.999 (768 training examples scored by the base model, where they are 34.6% of the targets). The last row is the ideal weight without a stabilizer, for which only this share is computed:

ϵ\epsilonAIME24MATH-500MinervaOlympiadAMC23AVGSignal from p≥0.999p\ge0.999 (%)
10−110^{-1}7.9468.8528.3034.8547.1937.430.19
10−210^{-2}7.9370.0331.0634.0647.5038.120.50
10−310^{-3} (paper)8.7568.5331.6033.5453.7539.231.24
10−410^{-4}9.9969.3333.9833.6144.6938.322.11
10−510^{-5}7.9471.2527.5931.2956.2538.862.50
00 (ideal weight)––––––2.60

Thank you again for the careful review and the insightful questions. If any concern remains, please let us know; we would be glad to discuss it further.

AuthorsOpenReview ↗

We would like to hear from Reviewer hXpz

Hi Reviewer hXpz,

We sincerely appreciate your careful review and insightful questions on our work. We would like to kindly follow up on the response we submitted to your comments, and we would be very grateful if you could let us know whether it has addressed your concerns. If you have any additional concerns, please don't hesitate to let us know; we would really appreciate the opportunity to discuss them further.

Thank you again!

Submission#45572 Authors

Reviewer_cL7vOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a concise and general geometric perspective for improving token-level weighting mechanisms in SFT, based on the Fisher–Rao distance. The authors first introduce a metric to quantify the gain in the objective function per unit of FR distance. Further, they derive a concept termed Vocab Relative Advantage (VRA), which aims to ensure a consistent gain.

Critical strengths
  1. The motivation is clear: by leveraging the concept of geometric distance, the paper provides a systematic analysis and comparison of various existing methods.
  2. A concise and general equal-gain principle is proposed, with a mathematically rigorous derivation.
  3. This principle demonstrates robust empirical improvements across a variety of models and settings.
Critical weaknesses
  1. Although the method proposed in the paper is derived clearly and logically based on the equal-gain principle, this principle itself appears to be supported primarily through empirical validation. It remains unclear why balancing target gains based on unit FR distance leads to better generalization. Could the authors provide a more direct analysis or more direct experimental evidence?
  2. Could the authors conduct comparisons using other simpler weighting functions to verify whether the performance gains stem specifically from the proposed equal-gain principle, or from the adoption of a probabilistic intermediate weighting scheme?
  3. Could the authors provide further clarification regarding the experimental setup, for instance, whether multiple independent random seeds were used for training, and how the specific checkpoints were selected?
  4. The VRA method actually employed in the paper utilizes the following stabilization weighting function: wϵ(p)=p1−p+ϵ,ϵ=10−3.w_\epsilon(p)=\sqrt{\frac{p}{1-p+\epsilon}},\qquad \epsilon=10^{-3}. This deviates from the theoretical principle, especially when considering the actual percentage of tokens affected. Could the authors provide experimental results using a smaller ϵ\epsilon value (e.g., ϵ=10−5\epsilon=10^{-5}) to determine whether a configuration closer to the theoretical formulation yields performance improvements?
Questions for authors

See Weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer cL7v (1/5)

We sincerely appreciate your constructive comments on our paper.
Below we detail our response to the weaknesses (i.e., W1$\sim$W4) point by point. Please kindly let us know whether our response addresses the issues you raised, and whether any concerns remain. Please note that the section, figure, and table numbers mentioned below are based on the original submission unless we explicitly refer to the revised manuscript; their relative positions may change in the revision.

(W1) Although the method proposed in the paper is derived clearly and logically based on the equal-gain principle, this principle itself appears to be supported primarily through empirical validation. It remains unclear why balancing target gains based on unit FR distance leads to better generalization. Could the authors provide a more direct analysis or more direct experimental evidence?

Thank you for this insightful question. It led us to close the rationale gap between the principle itself and the established generalization improvement, which makes our work more solid. Through our more direct analyses below, we conclude that equal gain generalizes better because it fully trains task-general tokens (on which all correct solutions agree) without over-training demonstration-specific tokens (on which correct solutions differ).

  • Generalization requires learning what all correct solutions share.
    • A task-general token is fixed by a rule that every correct solution follows, such as the 11 after "(x+1)2=x2+2x+(x+1)^2 = x^2 + 2x +"; predicting it means learning the expansion rule, which applies to new problems.
    • A demonstration-specific token is one of several equally valid choices, such as "So" or "Thus" at the start of a step; predicting it means learning only the demonstration's wording.
  • A task-general token is fixed by a rule, so the model usually predicts it with high pp; a demonstration-specific token is one of several valid choices, so the model spreads probability across them and usually predicts it with low pp.
    • The results in Table R2 (and Figure 9 in the revised manuscript) support this: 64% of targets with p≥0.9p \ge 0.9 are math symbols, LaTeX and numbers, while 63% of targets with p<0.1p < 0.1 are words. Connectives such as "So" and "Thus" make up 11% of these low-confidence targets, versus 2.4% of all targets.
AuthorsOpenReview ↗

Response to Reviewer cL7v (2/5)

(W1, continued)

  • A gain works against what generalization requires if it either (1) diverges as p→0p \to 0, over-training demonstration-specific tokens, or (2) vanishes as p→1p \to 1, under-training task-general tokens.
    • A weight w(p)w(p) gives a target with confidence pp the gain Cw(p)=w(p)(1−p)/pC_w(p) = w(p)\sqrt{(1-p)/p}, the objective gain per unit FR distance. Because CwC_w depends only on pp, its behavior as p→0p \to 0 sets how much demonstration-specific tokens are trained, and its behavior as p→1p \to 1 sets how much task-general tokens are trained.
    • As shown in Table R1, three weights, including the SFT weight w=1w = 1, have gains that diverge as p→0p \to 0, so they over-train demonstration-specific tokens and hurt generalization.
      • Our results in Table R3 (and Figure 8 in the revised manuscript) show that in one step, 91.91% of SFT's objective gain comes from targets with p<0.1p < 0.1.
      • SFT generates 25.9 tokens with p<0.1p < 0.1 per solution (Table R3), versus 0.61–1.31 for the other methods. To separate this effect from problem difficulty, we compare solutions to the same problem: among SFT's solutions, those with more low-confidence tokens than the problem's median are correct 39.6% of the time, versus 51.8% for the rest.
      • Keeping the gain from diverging as p→0p \to 0 recovers most of the gap. The ProFit weight differs from the SFT weight only by masking targets with p<0.1p < 0.1; this raises AVG from 29.30 to 36.46 (Table R3), recovering 72% of the gap between SFT and VRA.
    • Six of them, including the SFT and DFT weights w=1w = 1 and w=pw = p, have gains that vanish as p→1p \to 1, as does the ProFit weight w=1[p>0.1]w = \mathbb{1}[p > 0.1], so they under-train task-general tokens and limit generalization.
      • Table R3 (and Figure 8 in the revised manuscript) shows that targets with p≥0.9p \ge 0.9 contribute only 29.62% of DFT's one-step gain and −0.90% of SFT's, far smaller than 53.65% of the proposed VRA's.
      • Across methods, AVG rises in the same order as the share of one-step gain from targets with p≥0.9p \ge 0.9, as shown in Table R3.
  • Equal gain serves generalization better than a gain tilted toward either limit.
    • Since confidence indicates token type only imperfectly (24% of targets with p<0.1p < 0.1 in Table R2 are also math symbols, LaTeX and numbers), equal gain trains every target by the same amount per unit FR distance. As a result, it fully trains task-general tokens wherever their confidence falls, without over-training demonstration-specific ones.
    • Among the eight weights in Table R1, only VRA's keeps its gain constant at both limits, and it has the highest AVG.

Table R1. Gain profiles (written without the stabilizer) and AVG on Qwen3-8B in the main gold-data setting (same configuration and evaluation as Table 2; per-benchmark scores are in Tables R4 and R5). AVG is the unweighted mean of mean@8 over the five benchmarks:

Weight w(p)w(p)Gain per unit FR distance Cw(p)C_w(p)As p→0p\to0As p→1p\to1AVG
11 (SFT)(1−p)/p\sqrt{(1-p)/p}divergesvanishes29.30
(p1−p+ϵ)0.25\big(\frac{p}{1-p+\epsilon}\big)^{0.25}(1−pp)1/4\big(\frac{1-p}{p}\big)^{1/4}divergesvanishes35.65
p/(1−p+ϵ)\sqrt{p/(1-p+\epsilon)} (VRA, paper)11constantconstant39.23
(p1−p+ϵ)0.75\big(\frac{p}{1-p+\epsilon}\big)^{0.75}(p1−p)1/4\big(\frac{p}{1-p}\big)^{1/4}vanishesgrows37.75
p0.25p^{0.25}p−1/41−pp^{-1/4}\sqrt{1-p}divergesvanishes33.64
p\sqrt{p}1−p\sqrt{1-p}constantvanishes37.54
p0.75p^{0.75}p1/41−pp^{1/4}\sqrt{1-p}vanishesvanishes36.36
pp (DFT)p(1−p)\sqrt{p(1-p)}vanishesvanishes36.93

Table R2. Gold training targets by token type, grouped by their probability under the base model (768 training examples; %):

Token typeAll targetsp<0.1p<0.1p≥0.9p\ge0.9
Connectives and step markers (since, thus, let, first, ...)2.410.70.7
Content words (nouns, verbs, adjectives)12.932.79.0
Function words (articles, prepositions, ...)12.620.08.5
Math symbols and LaTeX39.619.144.3
Numbers14.84.819.2
Punctuation and formatting14.39.614.6
Other3.43.23.5

Table R3. Training signal (one gradient step from the base model, Figure 8 and Appendix E.4 of the revised manuscript) and solutions generated on the five benchmarks. AVG is from Table 2:

MethodShare of the one-step gain from targets with p<0.1p<0.1 (%)Share of the one-step gain from targets with p≥0.9p\ge0.9 (%)Tokens with p<0.1p<0.1 per generated solutionAVG
SFT91.91−0.90-0.9025.929.30
ProFit0.0018.351.3136.46
DFT−4.68-4.6829.620.7136.93
VRA−4.67-4.6753.650.6139.23
AuthorsOpenReview ↗

Response to Reviewer cL7v (3/5)

(W2) Could the authors conduct comparisons using other simpler weighting functions to verify whether the performance gains stem specifically from the proposed equal-gain principle, or from the adoption of a probabilistic intermediate weighting scheme?

Following the reviewer's valuable ablation suggestion, which makes the proposed equal-gain principle more sound and solid, we trained two families of simpler weights (see Figure 10 of the revised manuscript for the weight plot) on Qwen3-8B in the main gold-data setting (same configuration and evaluation as Table 2): (1) power weights pγp^{\gamma}, which interpolate between SFT (γ=0\gamma=0) and DFT (γ=1\gamma=1), and (2) VRA-shaped weights (p1−p+ϵ)β\big(\frac{p}{1-p+\epsilon}\big)^{\beta}, where β=0.5\beta=0.5 is VRA. Checkpoints are selected on a held-out validation set (256 problems, disjoint from the training subset); the SFT, DFT, and VRA rows are from Table 2. We would like to humbly argue that the gain comes from the equal-gain principle, not from intermediate probability weighting.

  • Intermediate weighting alone is not enough. Every power weight lies between SFT and DFT, yet the best reaches 37.54 AVG, below VRA's 39.23.
  • Keeping the gain constant as p→0p \to 0 raises AVG from 29.30 to 37.54. The SFT weight w=1w = 1 and the p\sqrt{p} weight both have gains that vanish as p→1p \to 1 (cf. Table R1), but the gain of the SFT weight diverges as p→0p \to 0 while that of the p\sqrt{p} weight stays constant.
  • Keeping the gain constant as p→1p \to 1 raises AVG from 37.54 to 39.23. The p\sqrt{p} weight and VRA both have constant gains as p→0p \to 0, but the gain of the p\sqrt{p} weight vanishes as p→1p \to 1 while that of the proposed VRA stays constant.
  • AVG peaks where the gain is equal. Within the VRA-shaped family, β=0.5\beta = 0.5 is the only exponent that keeps the gain constant; tilting it toward p→0p \to 0 (β=0.25\beta=0.25) or p→1p \to 1 (β=0.75\beta=0.75) lowers AVG to 35.65 or 37.75.

Table R4. Power weights w(p)=pγw(p)=p^{\gamma}:

γ\gammaAIME24MATH-500MinervaOlympiadAMC23AVG
00 (SFT)4.5860.2322.1822.3637.1929.30
0.250.257.9167.0025.9828.2439.0633.64
0.50.510.4069.5528.2033.9045.6337.54
0.750.7510.4369.9022.8534.5444.0636.36
11 (DFT)5.8469.0027.9833.4148.4436.93

Table R5. VRA-shaped weights w(p)=(p1−p+ϵ)βw(p)=\big(\frac{p}{1-p+\epsilon}\big)^{\beta}:

β\betaAIME24MATH-500MinervaOlympiadAMC23AVG
00 (SFT)4.5860.2322.1822.3637.1929.30
0.250.259.5868.8523.9532.1443.7535.65
0.50.5 (VRA, paper)8.7568.5331.6033.5453.7539.23
0.750.759.1868.5531.9934.3544.6937.75

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief summary of the discussion-phase interactions

Dear Area Chairs,

We sincerely thank you for your time and effort in overseeing the review process, especially given this year's surge in ICLR submissions. We truly understand that with this much heavier reviewing workload, ensuring fairness and quality places great demands on your role.

To help facilitate your final assessment, we summarize our paper and the outcomes of the discussion phase below.

Our paper

  • introduces a geometric perspective for comparing token weights in supervised fine-tuning (i.e., the objective gain that each weight assigns per unit of Fisher-Rao distance toward the demonstrated token), and
  • derives a simple and principled approach, VRA, the unique (up to scale) token weight with uniform gain per unit of Fisher-Rao distance, which improves average accuracy of state-of-the-art SFT methods by up to 3.24% on all six backbones.

The positive acknowledgments from reviewers include

  • (Motivation and Framework) The paper has a clear motivation and offers a clean and illuminating unifying framework that systematically analyzes and compares existing token-weighting methods (Reviewer cL7v and dGtc);
  • (Principle) The equal-gain principle is concise and general, with a mathematically rigorous derivation (Reviewer cL7v), and connects cleanly to a simple, cheap training change (Reviewer hXpz);
  • (Empirical Results) The method shows robust empirical improvements (Reviewer cL7v) that extend across six backbones (Reviewer hXpz and dGtc), with useful comparisons against recent weighting methods (Reviewer hXpz);
  • (Analysis and Rigor) The mechanistic analysis with matched one-step experiments goes beyond nominal gradient inspection and benchmark tables (Reviewer hXpz and dGtc), and the paper is written with unusual precision about what its theory and diagnostics do and do not establish (Reviewer dGtc).

In the discussion period, we responded to all weaknesses and questions raised by the three reviewers and updated the PDF accordingly, with all changes highlighted in blue.

We are encouraged that two of the three reviewers have acknowledged that our responses addressed their concerns, resulting in a unanimous score of 3, 3, 3.

  • Reviewer cL7v noted that our additional experiments and explanations largely address the concerns and raised the score from 2 to 3.
  • Reviewer dGtc confirmed that our responses have satisfactorily addressed the primary concerns and maintained the positive score.

Unfortunately, we have not yet received feedback from Reviewer hXpz after our follow-up reminder. Nevertheless, we believe that our responses have effectively addressed all major concerns of Reviewer hXpz, with all the experiments and clarifications that the reviewer requested added to the revised paper, as summarized below.

  • Concern 1 by Reviewer hXpz (training-seed variation and uncertainty estimates): We add repeated runs with three training seeds and report standard deviations; VRA stays above DFT on every seed, including on Qwen2.5-7B, where the original margin was small (Appendix G.3).
  • Concern 2 by Reviewer hXpz (whether equal gain is necessary, given nearby weighting shapes and method-specific learning rates): We add nearby weight exponents and simpler power weights, all of which fall below VRA (Appendix G.1), and a separate learning-rate search for each method, under which VRA remains the best (Appendix G.4).
  • Concern 3 by Reviewer hXpz (sensitivity to the stabilizer): We vary the stabilizer ϵ\epsilon and find that it matters little, whereas the equal-gain shape matters (Appendix G.2).

Again, we sincerely appreciate your time in handling our submission and look forward to the possibility of our work contributing meaningfully to the field.

Best Regards,

Submission#45572 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.