acceptodds
Under review as a conference paper at ICLR 2027

Understanding LLM Parameter Update Sparsity through the Lens of Fisher

Abstract

Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training paradigms suggests shared structure in training dynamics. In this paper, we examine this pattern through the diagonal model Fisher, which measures the sensitivity of the model's output distribution to individual parameters and is independent of any particular reward or teacher signal. Theoretically, we show that small diagonal Fisher leads to small expected gradients across a range of training objectives, providing a common explanation for sparse gradient updates. Empirically, we test this connection in RL and OPD. We find that Fisher identifies where gradients are concentrated, and fixed sparse masks selected from the initial Fisher retain a large proportion of the improvement from full training. Finally, we investigate the mechanisms underlying low Fisher in on-policy training. Our results show that high-probability next tokens tend to have similar parameter sensitivities, contributing to low Fisher. Together, these results establish the diagonal model Fisher as a unifying perspective linking update sparsity to on-policy training dynamics in LLM post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM post-training, reinforcement learning, on-policy distillation, parameter update sparsity, Fisher information

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_mL1pOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

It is known that during LLM post-training, such as RL or distillation, only a small fraction of parameters change significantly. This paper tries to explain why, using the diagonal Fisher information, which measures how sensitive the model's output distribution is to each parameter. The authors show with a simple inequality that parameters with little effect on the output also get small gradients during training. Empirically, they pick the parameters with the highest Fisher before training, train only those, and get performance close to full-parameter training. They also propose a mechanism for why Fisher is small for most parameters, namely that the next tokens the model considers likely tend to react to a parameter change in similar ways.

Critical strengths

The question is a reasonable one. Update sparsity has been reported in several recent papers, and it makes sense to look for a single quantity that does not depend on the reward or the teacher. The bound is simple, and it is nice that SFT, RL, and two forms of distillation all fit the same expression. Each step of the argument comes with its own experiment (overlap with gradients, fixed-mask training, stability over training, token-level decomposition), and the Fisher masks beat random masks by a large margin throughout.

Critical weaknesses
  1. I am not sure what is new here. That fine-tuning can be done with a small subset of parameters has been shown many times (lottery tickets, intrinsic dimension, DARE, skill localization). Training a fixed mask of the highest-Fisher parameters is essentially the FISH mask of Sung et al. (2021), and the authors note in a footnote that a similar Fisher bound already appears in Luo et al. (2026). The theory itself is one application of Cauchy–Schwarz plus a standard decomposition.
  2. The link between Fisher and gradients is close to circular. Both are built from the same score function, so it is expected that coordinates with large scores rank high in both. The more interesting question is why Fisher is concentrated on a few coordinates in the first place, and the paper does not really answer it.
  3. The central claim is not tested directly. The paper is motivated by why on-policy training in particular leads to sparse updates, but there is no comparison with off-policy SFT, where prior work reports dense updates. It also does not separate its explanation from the two other factors. Adam normalizes each coordinate, so a small expected gradient does not have to mean a small update, and in bf16, small updates can be rounded away entirely. Either could produce the observed sparsity without the Fisher argument.
  4. The experiments are small. The main results use one 1.7B model on math only, and the OPD evaluation uses 30 AIME problems, which is noisy (some recovery values exceed 150%). The introduction is also hard to follow. Figure 1 and Equation 1 use symbols that are only defined in later sections, and the order of the argument in Figure 1 is the reverse of the section order.
Questions for authors
  1. What happens to the Fisher distribution and to update sparsity under off-policy SFT? If updates are dense there, is the difference explained by Fisher or by the signal term?
  2. Do the results still hold when training with SGD instead of Adam, or in FP32 so that rounding does not remove small updates?
  3. Compared with Sung et al. (2021) and Luo et al. (2026), what exactly does this paper show that is new?
6 comments

We thank the reviewer for the detailed feedback. We would like to clarify the scientific motivation and contributions of our work, and provide additional experimental results addressing the concerns about generality.

1. Novelty and scientific contribution (W1, Q3)

We agree that sparse fine-tuning, Fisher-based parameter selection, and Fisher-based gradient bounds have been studied previously. Our contribution is not a new sparse training algorithm or the Cauchy–Schwarz inequality itself, but an explanation for why learning becomes concentrated in a small subset of parameters across different post-training objectives.

Specifically, our work makes three contributions:

  1. A unified Fisher perspective: We show that expected coordinate gradients in SFT, RL, and OPD admit bounds with the same dependence on the diagonal model Fisher, despite their different learning signals.
  2. Empirical validation: We show that Fisher identifies coordinates carrying large gradients and that sparse subsets selected using the initial Fisher can sustain much of the improvement from full training. The fixed-mask experiments serve as evidence for our explanation, rather than as a proposed parameter-efficient training method.
  3. An explanation for low Fisher under on-policy generation: We connect model Fisher to the probability-weighted variation in next-token logit sensitivities and empirically investigate why this variation can be small.

This differs from Sung et al. (2021), who use Fisher masks for efficient sparse training, and Luo et al. (2026), who study Fisher-based sensitivity and parameter movement in the context of continual learning and catastrophic forgetting, including under a locally stationary SGD approximation. Our focus is instead on explaining the concentration of learning across post-training objectives and investigating its relationship with on-policy generation.

2. Fisher–gradient relationship and the origin of low Fisher (W2)

We agree that Fisher and gradients share the same score function, but their relationship is not circular. Fisher measures the second moment of the score, whereas expected gradients depend on its correlation with the objective-specific learning signal. In particular, writing (hi=∂ilog⁡πθ(y∣x))(h_i=\partial_i\log\pi_\theta(y\mid x)), we have (Fi=E[hi2])(F_i=\mathbb{E}[h_i^2]) and (gi=E[ahi])(g_i=\mathbb{E}[a h_i]), where (a) denotes the learning signal. Thus, large Fisher does not necessarily imply large gradients, since positive and negative contributions can cancel. The distinction between Fisher information and objective gradients is well established in the natural-gradient literature (Amari, 1998; Martens, 2020). Our theoretical result establishes a one-sided implication: sufficiently small Fisher constrains expected gradients when the objective-dependent scale is controlled. Whether this relationship is informative in practice is precisely what our gradient-support experiments test, using independently estimated Fisher and gradients.

More importantly, the second half of our paper directly investigates why Fisher itself can be small under on-policy generation. We express token Fisher as the variance of logit sensitivities under the model's next-token distribution. This reveals how probability concentration and similarity in the sensitivities of high-probability tokens can jointly reduce Fisher. Our Real-versus-Uniform experiments provide evidence for this combined effect, although they do not isolate the contribution of each factor. We will clarify this distinction in the revision.

3. On-policy versus off-policy training, Adam, and numerical precision (W3, Q1, Q2)

On-policy versus off-policy SFT. The observation that off-policy SFT typically produces denser updates, while near-policy training can produce sparse updates, was already established by Mukherjee et al. (2025). Our goal is to explain this previously observed phenomenon rather than rediscover it.

Our theoretical framework provides a useful distinction here. At a fixed policy and prompt distribution, model Fisher does not depend on the demonstration responses. Off-policy SFT changes the gradient's distribution-mismatch term, which can weaken the implication from small Fisher to small gradients. Thus, our theory does not predict that off-policy SFT must exhibit the same degree of gradient concentration as on-policy training. A direct comparison of how Fisher and gradients evolve under off-policy SFT would further clarify this distinction.

We have additionally conducted on-policy rejection-sampling SFT experiments with Qwen3-1.7B-Base. High-Fisher coordinates capture approximately 46–50% of the largest-gradient coordinates across the evaluated checkpoints and selection budgets. With only 1% of parameters trainable, the initial Fisher mask achieves 53.10% MATH-500 correctness, compared with 54.70% for full training and 30.55% for an equally sized random mask. These results extend our empirical findings beyond RL and OPD.

Optimizer dynamics and numerical precision. We acknowledge that our bounds concern expected gradients at a fixed policy and do not establish a guarantee for accumulated updates under AdamW or SGD. Adam's coordinate-wise normalization, gradient noise, and optimization dynamics can all affect the eventual parameter changes. We do not claim that Fisher is the sole explanation for observed update sparsity.

As also discussed in Mukherjee et al., numerically observed sparsity should be distinguished from mathematically exact zero updates: sufficiently small updates can become indistinguishable under finite precision. Our analysis aims to explain why many gradients can have small magnitudes in the first place, rather than attributing exact zeros solely to Fisher.

Importantly, our new RL and SFT experiments use FP32 parameters, gradients, and Adam optimizer states, with BF16 used for forward computation. Hence, their fixed-mask results do not rely on BF16 parameter-storage rounding to eliminate small updates. We have not evaluated SGD and do not claim optimizer-independent guarantees. Extending the analysis to different optimizers remains an important direction for future work.

4. Additional experiments and presentation (W4)

To address the concerns about model scale and experimental generality, we additionally repeat Section 3.1 and 3.2 on Qwen3-4B-Base with RL, and on-policy rejection-sampling SFT experiments on Qwen3-1.7B-Base.

4.1. Qwen3-4B-Base with RL

Fisher identifies large-gradient coordinates.

Policyq=0.5%q=1%q=5%
Initial46.29%43.46%40.90%
After 250 updates44.96%42.14%41.70%
Random baseline0.50%1.00%5.00%

Initial Fisher masks sustain learning.

Trainable parametersFisher maskRandom mask
0.5%46.25%28.95%
1%53.60%27.75%
5%59.40%32.25%

4.2. Qwen3-1.7B-Base with on-policy SFT

We conduct on-policy rejection-sampling SFT for 1,000 rollout batches. At each batch, the current policy generates responses, and only those verified as correct contribute to the supervised loss.

Fisher identifies large-gradient coordinates.

Checkpointq=0.5%q=1%q=5%
Initial48.03%46.18%46.40%
Step 25049.64%47.43%48.44%
Step 50050.30%48.04%49.02%
Step 75049.36%47.26%48.36%
Step 100050.02%47.87%49.31%
Random baseline0.50%1.00%5.00%

Initial Fisher masks sustain learning.

Trainable parametersFisher maskRandom mask
0.5%51.45%26.30%
1%53.10%30.55%
5%52.85%46.25%

Overall, these additional experiments support both of our main empirical findings: high-Fisher coordinates capture a disproportionate fraction of large gradients, and fixed initial Fisher masks sustain substantial learning. The Qwen3-4B results extend our evidence to a larger model, while the rejection-sampling SFT experiments extend it beyond RL and OPD.

References

  • Amari, S. (1998). Natural Gradient Works Efficiently in Learning. Neural Computation, 10(2), 251–276.
  • Martens, J. (2020). New Insights and Perspectives on the Natural Gradient Method. Journal of Machine Learning Research, 21(146), 1–76.

Thank you for the response. The new experiments on Qwen3-4B and on-policy SFT, and the error bars, are helpful. However, my main concerns are not resolved, so I keep my score.

  • Calling the paper an explanation rather than a method does not resolve the novelty issue. If the paper is an explanation, it should show that Fisher explains the sparsity better than the other factors it cites, such as the optimizer, numerical precision, and near-policy data. The response leaves all of these to future work.
  • According to the response, Fisher at a fixed policy is the same for on-policy and off-policy SFT, and only the mismatch term changes. Then Fisher cannot explain why on-policy training leads to sparse updates and off-policy training does not. The new SFT experiment is also on-policy, so this question is still open.
  • The paper never measures update sparsity in its own runs. Recall and fixed-mask results show that high-Fisher coordinates are useful, but not that sparse updates actually happen in this FP32 setup, or that Fisher is the reason.
  • Random masks are the only baseline. I would like to see at least a layer-matched random mask and a simple magnitude or gradient-based mask. Otherwise it is unclear whether Fisher tells us more than which layers or modules have large scores.
  • The response now says the low-Fisher mechanism is a mix of probability concentration and similar sensitivities. But probability concentration alone would already make the Real/Uniform ratio small, so the experiment does not support the similarity claim. The introduction still makes this claim, and it is not clear if it will be changed.
Reviewer_rSjGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses diagonal model Fisher as analysis tool to study the concentration of learning in a small subset of parameter coordinates during LLM post-training. Its main contributions include deriving Fisher-based bounds on expected coordinate gradients, and showing that small amount of Fisher-selected trainable parameters can recover over 95% full-training improvement (using Qwen3-1.7B-Base model for MATH500 dataset). Moreover, to explain why this low-Fisher structure occurred, this paper derives a token-level Fisher, discovering the relationship between lower Fisher structure and the similar sensitivities among high-probability tokens. Finally, the authors empirically examine the stationarity of diagonal Fisher during the post-training stage, indicating that the initial selection is sufficiently informative.

Critical strengths
  1. The initial Fisher-mask experiments are convincing. Masks are selected before training and remain fixed. Updating only 0.1% of parameters recovers 85.21% of the full-training improvement.
  2. The theoretical analysis is clear and provides a mechanistic explanation for how low Fisher values arise. Proposition 1 establishes a Fisher-based bound on expected gradients. Proposition 2 connects diagonal Fisher to token-level sensitivity variation.
  3. The validation experiments and indicators are carefully designed.
Critical weaknesses
  1. It would also be informative to compare Fisher-based selection with the other methods, such as the magnitude-based PaF (Liao et al., ACL 2023), and gradient-based static masks (Liu et al., ACL 2025). These comparisons could help clarify whether the additional cost of Fisher estimation yields benefits over a simpler selection criterion.
  2. Additional model evaluations would strengthen the generality of the findings. The current RL and OPD experiments focus on Qwen3-1.7B. It would be helpful to examine whether the effectiveness also hold for larger models or other model families.
  3. The motivation is attractive, but the writing and language are sometimes ambiguous. For example, the expression “Fisher is small” can be small set of parameters or the value of diagonal matrix is small. A careful language revision would further improve clarity and readability.
Questions for authors

see weaknesses

1 comment

We thank the reviewer for these comments.

1. Paper comparison

We thank the reviewer for suggesting these comparisons. We agree that understanding the motivation, applicability, and computational trade-offs of different selection criteria is important. The key distinction is that PaFi and Liu et al. (2025) primarily aim to develop effective parameter-efficient fine-tuning methods, whereas we use Fisher-based selection to investigate why learning is concentrated in a small subset of parameters.

MethodMotivationSelection criterionApplicability and benefits
PaFi (Liao et al., 2023)Efficient, task-agnostic fine-tuningSelect parameters with the smallest pretrained weight magnitudesNo gradient estimation required; the same mask can be reused across downstream tasks, reducing selection and storage costs
Liu et al. (2025)Effective sparse fine-tuningSelect parameters using gradient-based salience metrics computed from downstream training dataTask-specific selection; requires gradient computation but provides a simple, computationally efficient static-mask method with strong downstream performance
Ours (Fisher)Explain concentrated learning across post-training objectivesSelect parameters with the largest diagonal model Fisher, estimated from model-generated responsesIndependent of the particular reward or teacher signal; theoretically motivated across SFT, RL, and OPD, but requires additional Fisher estimation

Our Fisher-based selection follows directly from our theoretical result: small diagonal model Fisher constrains expected coordinate gradients across different post-training objectives when the objective-dependent scale is controlled. We therefore use fixed Fisher masks to test whether the theoretically identified coordinates can sustain learning throughout training, with equally sized random masks as a baseline. The success of these experiments provides empirical support for our Fisher-based explanation of update sparsity.

We acknowledge that Fisher estimation incurs additional computational cost compared with simpler criteria such as weight magnitude, and we do not claim superior parameter-selection efficiency or training speedups. Our objective is to explain the observed concentration of learning, rather than to optimize sparse fine-tuning performance. A direct comparison with these methods would be useful for evaluating practical selection efficiency, but would address a different question from the central scientific objective of this paper.

2. Additional model

To evaluate whether our findings generalize to a larger model, we repeat Section 3.2 and 3.3 on Qwen3-4B-Base for q=0.5%,1%,5%q=0.5\%,1\%,5\%. We train for 250 updates and evaluate on MATH-500.

Fisher identifies large-gradient coordinates.

PolicyTop 0.5%Top 1%Top 5%
Initial46.29%43.46%40.90%
After 250 updates44.96%42.14%41.70%
Random baseline0.50%1.00%5.00%

Initial Fisher masks sustain learning.

Trainable parametersFisher maskRandom mask
0.5%46.25%28.95%
1%53.60%27.75%
5%59.40%32.25%
Full (100%)64.65%—
Initial (0%)26.55%—

We can see that, high-Fisher coordinates consistently capture a substantial fraction of the largest-gradient coordinates, and Fisher-selected masks substantially outperform random masks.

3. Wording

We thank the reviewer for the helpful suggestions. For example, we will clarify that “small Fisher” refers to small values of individual diagonal Fisher entries, rather than a small subset of parameters. We will incorporate this feedback and improve the overall clarity and readability of the paper in the revision.

Reviewer_zUSUOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why gradient updates during LLM post-training are often concentrated to a small subset of parameters across different objectives when using near on-policy data. The authors study it from a lens of diagonal model Fisher (i.e., an objective agnostic view), tying the phenomenon to small diagonal model FIsher. Theoretically, the authors show that (1) for SFT, on-policy RL and two forms of OPD, the expected gradient of each parameter is bounded by the square root of its Fisher times an objective-specific signal or mismatch scale. So low-Fisher parameters receive small expected gradients. (2) Each token’s Fisher equals the variance of logit sensitivities across candidate next tokens, weighted by the model’s probabilities, implying that the Fisher is small when the model puts its probability on tokens that respond similarly to a parameter change. Empirically, the authors show that in the case of Qwen3-1.7B-Base on SimpleRLZoo, the top 1% Fisher coordinates contain 47.4% of top-1% gradient coordinates. Next when trained with Dr. GRPO, training only a fixed mask of top initial-Fisher parameters recovers about 85% improvement at 0.1% of parameters. The authors further show that token Fisher under the model’s own distribution is 6–25× smaller than under a uniform distribution, supporting the proposed mechanism. Finally, the initial high-Fisher set remains largely stable during training (~71–78% retained), and similar results hold for OPD.

Critical strengths
  • The proposed explanation for sparse gradient updates is objective agnostic and thus ties observations across near-policy SFT, on-policy RL and on-policy distillation
  • The paper is well organized and easy to follow. The paper first substantiates the claims mathematically and then empirically using multiple interesting experiments
Critical weaknesses
  • The experimental setup in the main paper is quite narrow. While the claims and theoretical analyses cover three on-/near-policy objectives, the main paper experiments study only on-policy RL, with only one model x training dataset combination. Empirical study of SFT is missing from the paper altogether
  • Although the authors mention that the experiments were ran for multiple seeds, no error bars are reported anywhere
Questions for authors
  • Why did the authors not include SFT in the experimental setup?
  • How much of the Fisher subset identified in Figure 2(a) is retained as the top Fisher subset over the course of training? How does it change over training steps and is the correlation between the Firsher subet and high gradient coordinates maintained?
3 comments

We thank the reviewer for these questions.

1. SFT

We thank the reviewer for suggesting additional SFT experiments. Following this suggestion, we have conducted new experiments on on-policy rejection-sampling SFT. As shown by Mukherjee et al. (2025), SFT can also exhibit parameter update sparsity when training data are near the policy distribution. We therefore extend both of our empirical tests to this setting.

Specifically, we train Qwen3-1.7B-Base on mathematical reasoning for 1,000 rollout batches. At each batch, the current model generates responses, and only those verified as correct contribute to the SFT loss. We repeat Section 3.2 and 3.3 at q=0.5%,1%,5%q=0.5\%,1\%,5\%.

Fisher identifies where gradients are concentrated.

Checkpointq=0.5%q=1%q=5%
Initial48.03%46.18%46.40%
Step 25049.64%47.43%48.44%
Step 50050.30%48.04%49.02%
Step 75049.36%47.26%48.36%
Step 100050.02%47.87%49.31%
Random baseline0.50%1.00%5.00%

Initial Fisher masks sustain learning.

Trainable parametersFisher maskRandom mask
0.5%51.45%26.30%
1%53.10%30.55%
5%52.85%46.25%

We can see that, high-Fisher coordinates consistently capture a substantial fraction of the largest-gradient coordinates, and Fisher-selected masks substantially outperform random masks. These additional results show that our two main empirical findings also hold for on-policy rejection-sampling SFT, extending our experimental evidence beyond RL and OPD.

2. Fisher subset

Both aspects are examined in our paper, in Figure 3(a) and Table 4, respectively.

Fisher–gradient relationship throughout training. Figure 3(a) shows that high-Fisher coordinates consistently capture a substantially larger fraction of the largest-gradient coordinates than random subsets, both at initialization and at later checkpoints. Thus, the Fisher–gradient relationship remains informative as training progresses.

Persistence of the initial Fisher subset. Section 4.2 and Table 4 measure the overlap between the initial top-Fisher subset and the corresponding subset at later checkpoints. For example, at (q=1%), the initial subset retains 76.70%, 74.37%, 73.68%, and 73.22% of its coordinates at steps 250, 500, 750, and 1000, respectively, far above the 1% random baseline.

We will make these connections more explicit in the revised paper.

3. Error bars

We report mean ± sample standard deviation across training seeds below.

RL Test 1: Gradient-support recall (%)

Checkpointq = 0.05%q = 0.1%q = 0.5%q = 1%q = 5%
Initial53.1053.0749.6047.4046.69
Step 25053.90 ± 0.1153.93 ± 0.1650.22 ± 0.2047.87 ± 0.1848.76 ± 0.13
Step 50053.64 ± 0.0553.59 ± 0.0449.85 ± 0.2647.58 ± 0.2348.87 ± 0.09
Step 75053.83 ± 0.1253.63 ± 0.1449.70 ± 0.1747.42 ± 0.1648.86 ± 0.13
Step 100053.95 ± 0.1353.69 ± 0.1049.54 ± 0.2547.30 ± 0.2848.80 ± 0.24
Random0.050.100.501.005.00

RL Test 2: Fixed-mask training on MATH-500

The shared initial score is 22.65%.

q (%)FullFisherR0R1RandomRecoveryΔ
0.0556.41 ± 0.4549.65 ± 0.3822.7523.6623.21 ± 0.6779.98 ± 1.3626.44 ± 0.77
0.156.41 ± 0.4551.41 ± 1.2024.3623.4423.90 ± 0.3185.21 ± 4.0127.51 ± 1.12
0.556.41 ± 0.4554.00 ± 0.4726.3926.4526.42 ± 0.7092.87 ± 2.3227.58 ± 1.01
156.41 ± 0.4555.03 ± 0.3030.9331.2131.07 ± 0.1895.90 ± 1.4323.96 ± 0.38
556.41 ± 0.4556.01 ± 0.3047.5847.6047.59 ± 0.3798.82 ± 1.108.43 ± 0.61

OPD Test 1: Fisher–gradient support recall (%)

Checkpointq = 0.5%q = 1%q = 5%
Initial (step 0)37.1235.6637.84
Step 543.26 ± 0.3741.05 ± 0.3641.90 ± 0.20
Step 1047.64 ± 0.0945.63 ± 0.1147.64 ± 0.06
Step 1547.75 ± 0.0545.80 ± 0.0448.25 ± 0.06
Step 2049.98 ± 0.1147.86 ± 0.0950.11 ± 0.17
Random0.501.005.00

OPD Test 2: Fixed-mask training on AIME24

The shared initial scores are 16.25% Avg@8 and 36.67% pass@8.

q (%)MetricFullFisherR0R1RandomRecoveryΔ
0.5Avg@826.67 ± 2.5725.10 ± 1.4214.6915.4215.05 ± 0.7989.49 ± 27.1310.05 ± 1.73
0.5pass@847.50 ± 3.1952.50 ± 4.1933.3330.8332.08 ± 2.85156.25 ± 65.7520.42 ± 1.60
1Avg@826.67 ± 2.5726.15 ± 1.1515.0016.4615.73 ± 0.5299.47 ± 26.1610.42 ± 1.61
1pass@847.50 ± 3.1955.00 ± 1.9229.1738.3333.75 ± 2.85185.42 ± 78.2821.25 ± 3.15
5Avg@826.67 ± 2.5726.25 ± 2.2824.2723.3323.80 ± 2.35106.59 ± 59.142.45 ± 0.43
5pass@847.50 ± 3.1950.83 ± 3.1949.1745.8347.50 ± 2.89147.92 ± 78.283.33 ± 3.60

I would like to thank the authors for their detailed rebuttal. The additional on-policy SFT results make the experiments section of the paper more complete and the error bars show minimal variance.

However, I still hold reservations about the breadth of the experimental setup. I believe having results on one more task, model pair would substantially strengthen the validity of the results, especially since Qwen models on high school math data exhibit several non-trivial (yet non-generalizable) results.

I would thus like to maintain my positive score.

Reviewer_zwpSOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates parameter update sparsity in LLM post-training through the diagonal model Fisher information. The authors theoretically establish a relationship between Fisher information and expected gradients across different training objectives, including SFT, RL, and on-policy distillation (OPD). Experiments demonstrate that high-Fisher parameters capture a substantial proportion of large-gradient coordinates, and updating only a small subset of Fisher-selected parameters can retain much of the performance achieved by full-parameter training.

Critical strengths
  1. This paper provides an interesting theoretical perspective on parameter update sparsity in LLM post-training. By connecting diagonal Fisher information with expected gradients, the authors attempt to provide a unified explanation for the sparse updates observed across different training objectives.

  2. The experimental results are promising. The Fisher-selected parameters demonstrate substantially better gradient-support recall than random selection. More importantly, updating small fraction of parameters achieves performance close to full-parameter training, suggesting that Fisher information can identify important parameters for effective learning.

Critical weaknesses
  1. The theoretical contribution is somewhat limited. The main relationship between Fisher information and expected gradients is derived from the Cauchy–Schwarz inequality, and similar Fisher-based bounds have been studied previously. More importantly, the theoretical results concern expected gradients at a fixed policy, rather than actual parameter updates throughout training. Therefore, it remains unclear how the proposed analysis sufficiently explains the observed parameter update sparsity under practical optimization algorithms.

  2. The proposed explanation for low Fisher is not fully supported by the experiments. This work compares token Fisher under the model's probability distribution against a uniform distribution. However, the differences may arise simply from the concentration of probability mass on a small number of tokens, rather than the claimed similarity of their parameter sensitivities. The current analysis does not sufficiently distinguish these two effects.

Questions for authors

The authors should clarify how the proposed Fisher-based analysis explains actual parameter update sparsity rather than only small expected gradients. In particular, the theoretical bound does not account for gradient variance, optimizer dynamics, or accumulated parameter changes. Since the experiments use AdamW, it is unclear whether the same relationship would hold under different optimization algorithms. The authors should also further distinguish the theoretical contribution from existing Fisher-based gradient bounds.

2 comments

We thank the reviewer for these comments. We agree that the theoretical scope and the interpretation of our low-Fisher experiments should be clarified.

1. Theoretical contribution and optimizer dynamics.

We agree that the Cauchy–Schwarz inequality itself is standard. However, our theoretical contribution lies in identifying a common explanation for concentrated learning across different post-training objectives, rather than deriving a new mathematical inequality. Specifically, we show that the expected coordinate gradients of SFT, RL, and two forms of OPD can all be expressed as a model score weighted by an objective-dependent signal. Despite their different learning signals, the resulting bounds share the same dependence on the diagonal model Fisher, which is independent of the particular reward, teacher, or demonstration data at a fixed policy. This establishes that, when the objective-dependent scale is controlled, coordinates with small Fisher cannot carry large expected gradients across these different objectives.

Importantly, we also investigate why Fisher itself can be small under on-policy generation through our token-level analysis. Together, these results provide a coherent explanation: the model's own probability distribution can give rise to low Fisher on many coordinates, which in turn constrains where gradients can concentrate across post-training objectives. This helps explain why similar learning concentration emerges despite differences in training objectives.

We acknowledge that this theoretical explanation concerns expected gradients rather than accumulated parameter updates under AdamW. Our experiments bridge this gap empirically: high-Fisher coordinates capture a disproportionate fraction of large gradients, while fixed subsets selected from the initial Fisher sustain much of the improvement from full training, substantially outperforming random masks. These results support the relevance of our theoretical perspective over actual training trajectories, although they do not establish a formal guarantee for update sparsity under AdamW. Extending the analysis to gradient variance, optimizer dynamics, and accumulated parameter changes remains an important direction for future work.

2. Interpretation of the low-Fisher mechanism.

Our Real-versus-Uniform comparison is directly motivated by the token-level Fisher decomposition, which expresses Fisher as the variance of logit sensitivities under the model's next-token distribution. By keeping the model, prefixes, and logit sensitivities fixed while changing only the token probabilities, this experiment directly tests whether the model's own probability weighting reduces sensitivity variation relative to uniform weighting. The consistently low Real-to-Uniform ratios provide strong empirical evidence for this effect.

Importantly, probability concentration and similarity of sensitivities among high-probability tokens are not competing explanations, but two related ways in which the probability-weighted variance can become small. Our proposed mechanism concerns their combined effect: the model assigns probability mass in a way that produces low variation in logit sensitivities. The Real-versus-Uniform experiment is designed to test precisely this combined effect, rather than separately quantify the contribution of each factor.

We agree that distinguishing the individual contributions would provide further insight. We explored this separation in preliminary experiments but did not observe sufficiently consistent results to draw separate conclusions. We will clarify that our findings support the combined probability-weighting mechanism, without claiming that similarity of sensitivities alone explains low Fisher.

Many thanks to the answers! I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.