- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an information-geometric framework to compare token-level reweighting schemes in supervised fine-tuning (SFT). By projecting the categorical next-token distribution onto a binary target-versus-rest Bernoulli distribution, the authors measure the local objective gain per unit of Fisher-Rao (FR) arc length. Under this metric, standard SFT assigns diverging gain as target probability p approaches zero, whereas probability-weighted methods such as DFT and InfoSFT assign vanishing gain at both endpoints. The authors propose an equal-gain principle that requires constant marginal gain per unit FR distance across all confidence levels, which uniquely yields the Vocab Relative Advantage (VRA) weight w(p) = sqrt(p / (1 - p)) up to scale. They also show that this weight matches the analytical expectation of a GRPO-style standardized target-match reward over the vocabulary. Using a stabilized weight sqrt(p / (1 - p + eps)), VRA outperforms SFT and DFT in average mean@8 across six language model backbones on five mathematics benchmarks, transfers to on-policy verifier-accepted rollouts, and shifts one-step parameter updates from low-confidence tokens toward middle- and high-confidence tokens.
Critical strengths
- The unifying framework is clean and illuminating. Expressing detached token weights through marginal utility per unit Bernoulli Fisher-Rao distance (Table 1 and Figure 1) provides a parameter-invariant lens to contrast SFT, DFT, InfoSFT, and ProFit, while the alternative derivation in Appendix C neatly connects VRA to vocabulary-level reward standardization without token sampling.
- The mechanistic analysis goes beyond nominal gradient inspection. In Section 6.2, the authors restore actual SFT checkpoints and AdamW states to run matched one-step updates under global gradient clipping, verifying that VRA's directional redistribution from p < 0.1 to p >= 0.1 survives adaptive per-parameter normalization.
- The paper is written with unusual precision regarding what its theory and diagnostics do and do not establish. The authors explicitly distinguish nominal objective profiles from realized multi-parameter updates, report negative or null structural differences honestly in Appendix E.3, and validate the method across six backbones as well as on-policy rollout and KL-regularized regimes in Appendix D.
Critical weaknesses
- The paper does not isolate the empirical contribution of the high-confidence factor (1 - p + eps)^(-1/2) from the low-confidence factor p^(1/2), nor does it ablate the stabilization constant eps. Relative to DFT (w(p) = p), VRA changes the low-confidence scaling to sqrt(p) (which coincides with the square-root member of SAFT) and simultaneously introduces (1 - p + eps)^(-1/2), which strongly amplifies weights as p approaches 1. Because 45.57% of the gold census tokens have p > 0.996 and 35.19% have p >= 0.999 (Figure 3a), the choice of eps = 10^-3 directly determines the weight cap (approx. 31.6) on over a third of all training tokens. Without ablating w(p) = sqrt(p) and varying eps in the gold-data setting, it is unclear whether VRA's advantage over DFT stems from uniform FR gain across the full interval or primarily from softer low-probability suppression vs. high-probability amplification.
- The theoretical justification equates constant objective derivative dU_w / d_ell with balanced learning incentives, but actual progress under gradient flow is not uniform across p. As shown in Section 4.2, under Euclidean logit gradient ascent z_dot = w(p)(1 - p), the induced velocity in Fisher-Rao arc length is ell_dot = w(p) sqrt(p) (1 - p)^(3/2), which for VRA equals p(1 - p) and still vanishes at both endpoints. Furthermore, the Bernoulli coarsening discards how the negative gradient -w(p) pi_theta(a | s) redistributes mass among the V - 1 non-target tokens.
- The main gold-data evaluation in Section 5 is limited to a single domain (mathematical reasoning on NuminaMath-CoT 30K) and relies on an unweighted macro-average (AVG) where AIME24 (30 problems) and AMC23 (40 problems) carry the same weight as MATH-500 (500 problems) and OlympiadBench (675 problems). On Qwen2.5-7B (Table 3), VRA exceeds DFT by only 0.15 points in AVG (31.99 vs. 31.84), and on Qwen3-4B (Table 2), VRA underperforms DFT, InfoSFT, and ProFit on Minerva Math (24.65 vs. 29.48 for DFT) while winning the macro-average largely through gains on the 30 AIME24 and 40 AMC23 problems. In addition, InfoSFT and ProFit are omitted from the four additional backbones in Table 3.
Questions for authors
- How does VRA compare on gold-data fine-tuning (Qwen3-8B and Qwen3-4B) against the unnormalized square-root weight w(p) = sqrt(p) (which matches VRA's low-confidence asymptotics without blowing up near p = 1) and across different values of the stabilizer eps (e.g., 10^-1, 10^-2, 10^-4)?
- In Appendix E.2 (Figure 7), VRA and ProFit achieve nearly identical gold-target acquisition in both the middle-probability group (0.8855 vs. 0.8854) and high-probability group (0.1415 vs. 0.1402) at step 300, yet differ by 2.77 points in benchmark AVG. Have you examined matched one-step updates or non-target entropy changes comparing VRA directly against DFT and ProFit (rather than only against unit-weight SFT) to explain what drives their downstream performance gap?
- Could you report the problem-count-weighted (micro-averaged) accuracy and per-benchmark breakdowns for the four additional models in Table 3, as well as gold-data SFT results on a non-math task (such as code generation or instruction following) to verify that the equal-gain weight generalizes beyond NuminaMath-CoT in the standard teacher-forced setting?
8 comments
Response to Reviewer dGtc (2/6)
(W2) The theoretical justification equates constant objective derivative with balanced learning incentives, but actual progress under gradient flow is not uniform across . Furthermore, the Bernoulli coarsening discards how the negative gradient redistributes mass among the non-target tokens.
Thank you for this careful theoretical analysis, which helped us state more precisely what the principle claims; we have added the clarification as Appendix B.6 of the revised manuscript. We would like to respectfully clarify that equal gain is a statement about how the objective values progress, not about optimization speed, and that the Bernoulli projection loses no progress toward the demonstration.
- Equal gain specifies the objective's valuation of progress, not a uniform optimization speed.
- We agree with your calculation. Under the single-logit Euclidean gradient flow in Section 4.2, .
- The factor comes from the logit parameterization and is the same for every weighting rule, so the speed ratio of any two rules at the same equals the ratio of their profiles; the profile is the only part a token weight controls.
- The ideal VRA rule sets , so its FR velocity is and vanishes at both endpoints, consistent with our acquisition-time analysis.
- A weight that instead equalized under this flow would be , i.e., : it rewards low-confidence progress more steeply than SFT and changes with the parameterization, whereas does not.
- By balanced incentives, we mean that an equal target-event FR displacement receives equal marginal objective gain, independently of confidence. This does not require equal displacement per training step, which is why we separately analyze the objective profiles, scalar acquisition dynamics, and realized optimizer updates.
- The Bernoulli projection loses no progress toward the demonstration.
- All rules in our comparison use weights that are functions of the target probability , with other rule parameters fixed, so their local utilities depend on the vocabulary distribution only through .
- The same holds for the distance: on the full vocabulary simplex, the Fisher–Rao distance from to a distribution is , which for the one-hot demonstration equals , the remaining Bernoulli arc length. is therefore exactly the gain per unit reduction in full-vocabulary distance to the demonstrated token.
- What the projection omits is rearrangement among non-target tokens at fixed .
- For any detached weight the per-token logit gradient is , so the term has the same direction for every rule and differs only by the scalar ; token by token, it cannot separate the rules.
- The vocabulary-level derivation in Appendix C is consistent with this: it sums all non-target terms exactly and assumes nothing about their probabilities. The reward there does not distinguish among non-target alternatives, but their individual probabilities remain in the full softmax gradient.
- The scalar profile therefore does not claim to determine the redistribution among non-target alternatives. Rule-dependent differences in it arise only through aggregation across tokens in shared parameters, where realized updates are an empirical question; Section 6.2 measures them on target probabilities. A weight that additionally depended on non-target entropy or relative probabilities would extend the class of rules beyond .
Response to Reviewer dGtc (3/6)
(W3) The main gold-data evaluation in Section 5 is limited to a single domain (mathematical reasoning on NuminaMath-CoT 30K) and relies on an unweighted macro-average (AVG) where AIME24 (30 problems) and AMC23 (40 problems) carry the same weight as MATH-500 (500 problems) and OlympiadBench (675 problems). In addition, InfoSFT and ProFit are omitted from the four additional backbones in Table 3.
Thank you for these helpful comments on the breadth of our evaluation, which led us to extend it in Appendices G.5 and G.6 of the revised manuscript. VRA remains the best fine-tuned method under micro-averaging on five of the six backbones and on a second domain, medical question answering. We would also like to respectfully clarify the scope of Table 3. The tables are in our response to (Q3).
- The advantage extends to a second domain. We fine-tuned Qwen3-8B on gold medical question-answering data, where VRA has the highest macro and micro accuracy: 53.59 and 57.41, vs. 52.24 and 55.79 for Base (Table R8).
- VRA's lead holds under micro-averaging on five of the six backbones. When each benchmark is weighted by its number of problems, VRA remains the best fine-tuned method on all but Qwen2.5-7B (Tables R6 and R7), where DFT is higher (39.35 vs. 38.22).
- InfoSFT and ProFit are compared on the two main models; Table 3 has a narrower scope. On Qwen3-8B and Qwen3-4B (Table 2), VRA has the highest AVG against both baselines. We do not have InfoSFT and ProFit runs on the four additional backbones; Table 3 examines whether the SFT/DFT/VRA comparison extends to them.
Response to Reviewer dGtc (1/6)
We are sincerely grateful for your thorough and detailed review of our paper.
Below we detail our response to the weaknesses (i.e., W1$\sim$W3) and questions (i.e., Q1$\sim$Q3) point by point. Please kindly let us know whether our response resolves the issues you raised, or whether there are any remaining concerns. Please note that the section, figure, and table numbers mentioned below are based on the original submission unless we explicitly refer to the revised manuscript; their relative positions may change in the revision.
(W1) The paper does not isolate the empirical contribution of the high-confidence factor (1−p+ϵ)−1/2 from the low-confidence factor p1/2, nor does it ablate the stabilization constant ϵ. Without ablating w(p)=p and varying ϵ in the gold-data setting, it is unclear whether VRA's advantage over DFT stems from uniform FR gain across the full interval or primarily from softer low-probability suppression vs. high-probability amplification.