- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
It is known that during LLM post-training, such as RL or distillation, only a small fraction of parameters change significantly. This paper tries to explain why, using the diagonal Fisher information, which measures how sensitive the model's output distribution is to each parameter. The authors show with a simple inequality that parameters with little effect on the output also get small gradients during training. Empirically, they pick the parameters with the highest Fisher before training, train only those, and get performance close to full-parameter training. They also propose a mechanism for why Fisher is small for most parameters, namely that the next tokens the model considers likely tend to react to a parameter change in similar ways.
Critical strengths
The question is a reasonable one. Update sparsity has been reported in several recent papers, and it makes sense to look for a single quantity that does not depend on the reward or the teacher. The bound is simple, and it is nice that SFT, RL, and two forms of distillation all fit the same expression. Each step of the argument comes with its own experiment (overlap with gradients, fixed-mask training, stability over training, token-level decomposition), and the Fisher masks beat random masks by a large margin throughout.
Critical weaknesses
- I am not sure what is new here. That fine-tuning can be done with a small subset of parameters has been shown many times (lottery tickets, intrinsic dimension, DARE, skill localization). Training a fixed mask of the highest-Fisher parameters is essentially the FISH mask of Sung et al. (2021), and the authors note in a footnote that a similar Fisher bound already appears in Luo et al. (2026). The theory itself is one application of Cauchy–Schwarz plus a standard decomposition.
- The link between Fisher and gradients is close to circular. Both are built from the same score function, so it is expected that coordinates with large scores rank high in both. The more interesting question is why Fisher is concentrated on a few coordinates in the first place, and the paper does not really answer it.
- The central claim is not tested directly. The paper is motivated by why on-policy training in particular leads to sparse updates, but there is no comparison with off-policy SFT, where prior work reports dense updates. It also does not separate its explanation from the two other factors. Adam normalizes each coordinate, so a small expected gradient does not have to mean a small update, and in bf16, small updates can be rounded away entirely. Either could produce the observed sparsity without the Fisher argument.
- The experiments are small. The main results use one 1.7B model on math only, and the OPD evaluation uses 30 AIME problems, which is noisy (some recovery values exceed 150%). The introduction is also hard to follow. Figure 1 and Equation 1 use symbols that are only defined in later sections, and the order of the argument in Figure 1 is the reverse of the section order.
Questions for authors
- What happens to the Fisher distribution and to update sparsity under off-policy SFT? If updates are dense there, is the difference explained by Fisher or by the signal term?
- Do the results still hold when training with SGD instead of Adam, or in FP32 so that rounding does not remove small updates?
- Compared with Sung et al. (2021) and Luo et al. (2026), what exactly does this paper show that is new?
6 comments
3. On-policy versus off-policy training, Adam, and numerical precision (W3, Q1, Q2)
On-policy versus off-policy SFT. The observation that off-policy SFT typically produces denser updates, while near-policy training can produce sparse updates, was already established by Mukherjee et al. (2025). Our goal is to explain this previously observed phenomenon rather than rediscover it.
Our theoretical framework provides a useful distinction here. At a fixed policy and prompt distribution, model Fisher does not depend on the demonstration responses. Off-policy SFT changes the gradient's distribution-mismatch term, which can weaken the implication from small Fisher to small gradients. Thus, our theory does not predict that off-policy SFT must exhibit the same degree of gradient concentration as on-policy training. A direct comparison of how Fisher and gradients evolve under off-policy SFT would further clarify this distinction.
We have additionally conducted on-policy rejection-sampling SFT experiments with Qwen3-1.7B-Base. High-Fisher coordinates capture approximately 46–50% of the largest-gradient coordinates across the evaluated checkpoints and selection budgets. With only 1% of parameters trainable, the initial Fisher mask achieves 53.10% MATH-500 correctness, compared with 54.70% for full training and 30.55% for an equally sized random mask. These results extend our empirical findings beyond RL and OPD.
Optimizer dynamics and numerical precision. We acknowledge that our bounds concern expected gradients at a fixed policy and do not establish a guarantee for accumulated updates under AdamW or SGD. Adam's coordinate-wise normalization, gradient noise, and optimization dynamics can all affect the eventual parameter changes. We do not claim that Fisher is the sole explanation for observed update sparsity.
As also discussed in Mukherjee et al., numerically observed sparsity should be distinguished from mathematically exact zero updates: sufficiently small updates can become indistinguishable under finite precision. Our analysis aims to explain why many gradients can have small magnitudes in the first place, rather than attributing exact zeros solely to Fisher.
Importantly, our new RL and SFT experiments use FP32 parameters, gradients, and Adam optimizer states, with BF16 used for forward computation. Hence, their fixed-mask results do not rely on BF16 parameter-storage rounding to eliminate small updates. We have not evaluated SGD and do not claim optimizer-independent guarantees. Extending the analysis to different optimizers remains an important direction for future work.
4. Additional experiments and presentation (W4)
To address the concerns about model scale and experimental generality, we additionally repeat Section 3.1 and 3.2 on Qwen3-4B-Base with RL, and on-policy rejection-sampling SFT experiments on Qwen3-1.7B-Base.
4.1. Qwen3-4B-Base with RL
Fisher identifies large-gradient coordinates.
| Policy | q=0.5% | q=1% | q=5% |
|---|---|---|---|
| Initial | 46.29% | 43.46% | 40.90% |
| After 250 updates | 44.96% | 42.14% | 41.70% |
| Random baseline | 0.50% | 1.00% | 5.00% |
Initial Fisher masks sustain learning.
| Trainable parameters | Fisher mask | Random mask |
|---|---|---|
| 0.5% | 46.25% | 28.95% |
| 1% | 53.60% | 27.75% |
| 5% | 59.40% | 32.25% |
4.2. Qwen3-1.7B-Base with on-policy SFT
We conduct on-policy rejection-sampling SFT for 1,000 rollout batches. At each batch, the current policy generates responses, and only those verified as correct contribute to the supervised loss.
Fisher identifies large-gradient coordinates.
| Checkpoint | q=0.5% | q=1% | q=5% |
|---|---|---|---|
| Initial | 48.03% | 46.18% | 46.40% |
| Step 250 | 49.64% | 47.43% | 48.44% |
| Step 500 | 50.30% | 48.04% | 49.02% |
| Step 750 | 49.36% | 47.26% | 48.36% |
| Step 1000 | 50.02% | 47.87% | 49.31% |
| Random baseline | 0.50% | 1.00% | 5.00% |
Initial Fisher masks sustain learning.
| Trainable parameters | Fisher mask | Random mask |
|---|---|---|
| 0.5% | 51.45% | 26.30% |
| 1% | 53.10% | 30.55% |
| 5% | 52.85% | 46.25% |
Overall, these additional experiments support both of our main empirical findings: high-Fisher coordinates capture a disproportionate fraction of large gradients, and fixed initial Fisher masks sustain substantial learning. The Qwen3-4B results extend our evidence to a larger model, while the rejection-sampling SFT experiments extend it beyond RL and OPD.
References
- Amari, S. (1998). Natural Gradient Works Efficiently in Learning. Neural Computation, 10(2), 251–276.
- Martens, J. (2020). New Insights and Perspectives on the Natural Gradient Method. Journal of Machine Learning Research, 21(146), 1–76.
Thank you for the response. The new experiments on Qwen3-4B and on-policy SFT, and the error bars, are helpful. However, my main concerns are not resolved, so I keep my score.
- Calling the paper an explanation rather than a method does not resolve the novelty issue. If the paper is an explanation, it should show that Fisher explains the sparsity better than the other factors it cites, such as the optimizer, numerical precision, and near-policy data. The response leaves all of these to future work.
- According to the response, Fisher at a fixed policy is the same for on-policy and off-policy SFT, and only the mismatch term changes. Then Fisher cannot explain why on-policy training leads to sparse updates and off-policy training does not. The new SFT experiment is also on-policy, so this question is still open.
- The paper never measures update sparsity in its own runs. Recall and fixed-mask results show that high-Fisher coordinates are useful, but not that sparse updates actually happen in this FP32 setup, or that Fisher is the reason.
- Random masks are the only baseline. I would like to see at least a layer-matched random mask and a simple magnitude or gradient-based mask. Otherwise it is unclear whether Fisher tells us more than which layers or modules have large scores.
- The response now says the low-Fisher mechanism is a mix of probability concentration and similar sensitivities. But probability concentration alone would already make the Real/Uniform ratio small, so the experiment does not support the similarity claim. The introduction still makes this claim, and it is not clear if it will be changed.
We thank the reviewer for the detailed feedback. We would like to clarify the scientific motivation and contributions of our work, and provide additional experimental results addressing the concerns about generality.
1. Novelty and scientific contribution (W1, Q3)
We agree that sparse fine-tuning, Fisher-based parameter selection, and Fisher-based gradient bounds have been studied previously. Our contribution is not a new sparse training algorithm or the Cauchy–Schwarz inequality itself, but an explanation for why learning becomes concentrated in a small subset of parameters across different post-training objectives.
Specifically, our work makes three contributions:
This differs from Sung et al. (2021), who use Fisher masks for efficient sparse training, and Luo et al. (2026), who study Fisher-based sensitivity and parameter movement in the context of continual learning and catastrophic forgetting, including under a locally stationary SGD approximation. Our focus is instead on explaining the concentration of learning across post-training objectives and investigating its relationship with on-policy generation.
2. Fisher–gradient relationship and the origin of low Fisher (W2)
We agree that Fisher and gradients share the same score function, but their relationship is not circular. Fisher measures the second moment of the score, whereas expected gradients depend on its correlation with the objective-specific learning signal. In particular, writing (hi=∂ilogπθ(y∣x)), we have (Fi=E[hi2]) and (gi=E[ahi]), where (a) denotes the learning signal. Thus, large Fisher does not necessarily imply large gradients, since positive and negative contributions can cancel. The distinction between Fisher information and objective gradients is well established in the natural-gradient literature (Amari, 1998; Martens, 2020). Our theoretical result establishes a one-sided implication: sufficiently small Fisher constrains expected gradients when the objective-dependent scale is controlled. Whether this relationship is informative in practice is precisely what our gradient-support experiments test, using independently estimated Fisher and gradients.
More importantly, the second half of our paper directly investigates why Fisher itself can be small under on-policy generation. We express token Fisher as the variance of logit sensitivities under the model's next-token distribution. This reveals how probability concentration and similarity in the sensitivities of high-probability tokens can jointly reduce Fisher. Our Real-versus-Uniform experiments provide evidence for this combined effect, although they do not isolate the contribution of each factor. We will clarify this distinction in the revision.