acceptodds
Under review as a conference paper at ICLR 2027

A FEW GOOD WEIGHTS: SPARSE SUBNETWORKS IN PREFERENCE-OPTIMIZED LLMS

Abstract

Reinforcement learning fine-tuning of Large Language Models (LLMs) induces sparse, full-rank weight updates concentrated subnetworks containing fewer than of parameters. We show the structure of these subnetworks is determined primarily by the learning objective, rather than the training data. Subnetworks produced by DPO and GRPO differ substantially in parameter selection and internal representation, even on similar data, while subnetworks produced by the same objective on different datasets remain closely aligned. To establish this result rigorously, we develop a theoretical framework for identifying the sparse subnetwork by scoring each parameter's importance. We derive an optimal Oracle scoring function with guarantees on loss retention, prove the existence of an optimal scoring function at any sparsity level, establish a certifiability condition for evaluating practical scoring functions, and develop a scalable threshold estimation algorithm for constructing subnetworks in multi-billion-parameter LLMs. We propose warm-start magnitude scoring, which recovers dense DPO and GRPO training dynamics at or beyond sparsity across mathematics, coding, syntax, and semantics. We characterize objective-specific subnetwork divergence through Jaccard similarity, Centered Kernel Alignment, and linear probing. Finally, we introduce a sparsity-aware AdamW optimizer kernel implemented in Triton that reduces optimizer step complexity by the factor , achieving and speedups over PyTorch and HuggingFace 8-bit AdamW respectively at sparsity.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Sparse Reinforcement Learning, Sparse Optimization, Preference Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qwUrOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether preference optimization of LLMs can be supported by extremely sparse, high-rank weight updates. It combines per-parameter saliency scoring with hybrid local-global mask construction, presents analyses of loss retention, optimal scoring, and certifiability under stated assumptions, and uses early dense-training weight displacement as a practical approximation to full-run Oracle scoring. Experiments cover DPO and GRPO, multiple datasets and model families, and subnetwork comparisons using Jaccard similarity, CKA, and linear probes. The systems component implements a sparsity-aware AdamW optimizer and reports optimizer-step speedups.

Critical strengths

1.The work connects theoretical analysis, subnetwork diagnostics, and a systems implementation. On Llama-3.1-8B-Instruct, warm-start magnitude masks track dense DPO training dynamics while updating about 2.5% of parameters during the sparse phase (Figure 1). Complementary cross-model training experiments mainly use Oracle masks (Appendix A.1). Sparsity sweeps, cross-task transfer, and representation diagnostics provide insight beyond final accuracy alone. These empirical contributions offer a useful framework for further study, even though the theoretical claims need the clarification requested below. 2.The paper examines parameter overlap, internal representations, functional probes, and hardware timings, and explicitly distinguishes full-run Oracle scoring from the more practical warm-start alternative. The model-family extensions and sparse optimizer implementation mean that the main observations do not rely entirely on one model or one diagnostic. This breadth increases the potential impact of the work.

Critical weaknesses

1.The relative influence of learning objective and training data would benefit from a more controlled comparison. DPO masks from Light-R1 and Tülu3 have high overlap, whereas Figure 2 reports lower cross-dataset overlap and CKA for GRPO. Figure 14 also shows useful GRPO cross-dataset transfer, which is distinct from structural similarity. The objective comparisons use different training recipes, including learning rates, schedules, and batch configurations (Tables 7-8). A focused comparison on the same base model with comparable data exposure and training budgets, together with data-construction and sampling details, would strengthen the attribution to the objective. Otherwise, the conclusion should be limited to the tested configurations. 2.End-to-end training efficiency remains insufficiently quantified. At 97.5% sparsity, Table 2 reports a 1.69x AdamW optimizer-step speedup over PyTorch AdamW, not a full-training speedup. Appendix B.4 states that the deployed implementation retains dense first- and second-moment buffers and adds mask/index storage, so reduced optimizer memory traffic does not imply reduced resident VRAM. Although LoRA quality comparisons are already provided, a matched-hardware comparison of total time, peak VRAM, and corresponding model quality for the practical warm-start method, dense AdamW, and LoRA would clarify the full-pipeline benefit. The accounting should include dense warm-start, scoring/mask construction, and subsequent training. 3.The fixed-point argument in Appendix D.2 needs correction or stronger assumptions. A finite-valued non-increasing map need not have a fixed point, and repeated application need not converge; it can alternate between two values. These properties therefore do not justify the existence and convergence claims as written. Section 3.2.2 already states existence and uniqueness under magnitude-curvature alignment, but Theorem 3 attaches this condition only to uniqueness and Appendix D.2 gives more general existence and convergence arguments. If alignment fixes the ranking with consistent tie-breaking, the map is constant, not strictly decreasing as stated in Appendix D.2. Please make the assumptions and proof consistent across these claims. This concern is about the theory and does not by itself invalidate the observed warm-start training results.

Questions for authors

1.Building on the existing comparisons, could you provide a focused DPO/GRPO comparison across two data sources using the same base model, comparable data exposure, and comparable training budgets, and report mask Jaccard similarities? Please clarify how the training inputs and sampling are constructed for each combination. This would help separate objective and data effects; if such matching is not feasible, please clarify the resulting limits on the objective-dominates-data claim. 2.Could you compare the practical warm-start method, dense AdamW, and LoRA on the same hardware under comparable training settings, reporting total training time, peak VRAM, and corresponding model quality? Please include dense warm-start, scoring/mask construction, and subsequent training in the timing, and identify the optimizer and state-storage path used. The existing optimizer-step timings and LoRA quality results can serve as context for this full-pipeline comparison.

2 comments
AuthorsOpenReview ↗

Response to Reviewer qwUr

We thank the reviewer for the thoughtful and constructive review. We are pleased that the reviewer found that the work connects theoretical analysis, subnetwork diagnostics, and a systems implementation, and that warm-start magnitude masks track dense DPO training while updating about 2.52.5% of parameters. We also appreciate the recognition that the breadth of diagnostics and model families means the main observations do not rely on a single model or diagnostic. Below, we respond to each of the reviewer's comments.

[Weakness 1, Question 1]: Controlled Comparison of Objective and Data

We agree that the two objectives used different recipes (Tables 10–11), and ran the requested comparison: DPO and GRPO on two data sources each, at two matched learning rates, on the same base model and with identical mask construction (Appendix A.5, Table 5):

Oracle-mask Jaccardlr 5×10−75\times10^{-7}lr 5×10−65\times10^{-6}
data only: DPO Light-R1 ↔\leftrightarrow DPO Tülu30.800.31
data only: GRPO Open-R1 ↔\leftrightarrow GRPO Tülu3-RLVR0.920.31
objective only: DPO Tülu3 ↔\leftrightarrow GRPO Tülu3-RLVR0.900.30
seed ceiling0.9570.66

At matched learning rates, changing the objective and changing the data shift overlap by similar amounts, so we limit the claim to the standard recipe of each objective, as the reviewer suggests. DPO sees 128 offline preference pairs per step; GRPO sees one prompt per step with 8 on-policy completions. As the reviewer notes, transfer is distinct from structural similarity: GRPO masks transfer across datasets (Figure 14) despite low overlap.

[Weakness 2, Question 2]: End-to-End Training Efficiency

On one B200 at matched settings (gradient checkpointing on), with held-out accuracy at lr 5×10−65\times10^{-6}:

armoptimizer / states/steppeak VRAM (GB)held-out acc.
dense8-bit AdamW25.681.30.568–0.590
sparse warm-startSparseAdamW, dense bf16 moments + mask25.994.20.570–0.596
LoRA r=64r = 648-bit AdamW28.731.30.556–0.566

Including the warm-start steps and mask construction (444 s), the full sparse pipeline takes about 1.45×1.45\times the dense wall-clock. We agree with the reviewer's reading of Appendix B.4. The deployed implementation keeps dense first- and second-moment buffers and adds mask and index storage, so the sparse optimizer reduces optimizer memory traffic rather than resident VRAM. The reported 1.69×1.69\times figure is an optimizer-step speedup, not an end-to-end training speedup, and the abstract and Section 4.5 state this.

[Weakness 3]: Consistency of the Fixed-Point Argument

We thank the reviewer for this precise analysis, and agree with each point. A finite-valued non-increasing map need not have a fixed point, and repeated application can alternate between two values, so the original existence and convergence claims did not follow. The reviewer is also correct that the assumptions were inconsistent: Section 3.2.2 stated existence and uniqueness under the alignment condition, whereas Theorem 3 attached it only to uniqueness. Furthermore, if the alignment condition fixes the ranking under consistent tie-breaking, Φ\Phi is constant rather than strictly decreasing.

We have replaced Theorem 3 with a corrected result whose assumptions are stated once and used consistently (Appendix D.2):

  1. Φ\Phi is non-increasing.
  2. Any self-consistent mask minimizes the bound of Theorem 1.
  3. When no self-consistent mask exists, bisection on BB returns a mask within an explicitly certified gap, CS2(B(M)−B∘)2\tfrac{C_S}{2}\big(B(\mathcal{M}) - B^{\circ}\big)^2, of the optimum.

The alignment condition is no longer used for existence or uniqueness. It now appears only in Corollary 1 (Oracle Optimality), which states when the Oracle mask minimizes the bound, and requires the condition only for pairs of parameters on either side of the Oracle threshold. Section 3.2.2 has been revised to match. We agree with the reviewer that this concern does not affect the observed warm-start training results.

Thank you for the detailed response. The matched-learning-rate comparisons clarify how the objective and data affect mask overlap, and the end-to-end benchmark clarifies the practical cost of sparse training. The revised Theorem 3 addresses my concern about the fixed-point argument.

Reviewer_oyqHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies sparse update in RL, e.g., for DPO and GRPO, selected using parameter displacement during dense training. It reports that selected masks can recover dense training behavior while updating only 2.5% of parameters, and argues that mask structure depends more on the objective than the dataset. Then it devises a sparse adamW kernel to get a 1.69× optimizer-step speedup over PyTorch AdamW.

Critical strengths
  1. Selected masks closely follow dense DPO training while random masks with the same budget learn much less.

  2. The sparse optimizer provides a measurable implementation benefit beyond reducing the number of trainable parameters only.

Critical weaknesses
  1. The objective-specific interpretation is not sufficiently controlled. DPO and GRPO differ in learning rate by 10×, alongside differences in clipping, batch size, and schedule. These factors can affect displacement-based masks. Could the author explains more on this? As i believe some papers are claiming they are not related to learning objetive as well [1][2]

  2. Some theory questions: it seems the theoretical guarantees need correction. Appendix D.2 assumes that a finite-valued, non-increasing map has a fixed point and convergent iteration; such maps can instead have two-cycles. Also, the theoretical optimal score depends on curvature, while the empirical Oracle uses displacement magnitude alone. Figure 5’s margin distributions do not establish the pointwise certification condition in Theorem 2. Could the authors address it?

Questions for authors
  1. Are the mask-overlap findings robust to learning rate and numerical precision?

  2. How can the fixed-point argument be corrected, and under what verified assumptions does the magnitude Oracle match the theoretical optimal score?

  3. Still i think the objective varaint claim is conflicted with [1][2] and some recent studies. Could you explain more?

[1] Zhu, Hanqing, Zhenyu Zhang, Hanxian Huang, DiJia Su, Zechun Liu, Jiawei Zhao, Igor Fedorov et al. "The path not taken: Rlvr provably learns off the principals." arXiv preprint arXiv:2511.08567 (2025). [2] Shenfeld, I., Pari, J., & Agrawal, P. (2026, April). Rl's razor: Why online reinforcement learning forgets less. In International Conference on Learning Representations (Vol. 2026, pp. 59839-59864).

3 comments
AuthorsOpenReview ↗

Response to Reviewer oyqH - Part 1

We thank the reviewer for the careful review and the constructive questions. We are pleased that the reviewer found that selected masks closely follow dense DPO training while random masks with the same budget learn much less, and that the sparse optimizer provides a measurable implementation benefit beyond reducing the number of trainable parameters. The reviewer's comments on the theoretical framework led us to substantially revise Theorems 2 and 3. Below, we respond to each of the reviewer's comments.

[Weakness 1, Questions 1 and 3]: Objective-Specific Interpretation, Robustness to Learning Rate and Precision, and Related Work

We thank the reviewer for raising this point, and agree that the original comparison did not control for the optimization configuration. The DPO and GRPO recipes differed in learning rate by 10×10\times, as well as in gradient clipping, batch size, and schedule, and displacement-based masks are sensitive to these settings. To isolate the effect of the objective, we repeated the Oracle mask comparisons for DPO and GRPO at matched learning rates of 5×10−75\times10^{-7} and 5×10−65\times10^{-6} (Appendix A.5, Tables 4–5, Figures 18–19).

At matched learning rates, the separation between DPO and GRPO masks largely disappears. At 5×10−65\times10^{-6}, all Oracle pairs, both within and across objectives, share a Jaccard similarity of approximately 0.3. At 5×10−75\times10^{-7}, the DPO and GRPO masks trained on Tülu3 overlap (0.90) nearly as strongly as the two GRPO masks (0.92). The divergence between DPO and GRPO reported in the original manuscript therefore reflects primarily the different learning rates of the two recipes, rather than the objective itself. We have revised the abstract, introduction, and Section 4.2 accordingly: the separation holds under the standard training recipe of each objective, and largely disappears at matched learning rates.

These results are consistent with the work highlighted by the reviewer. Zhu et al. (2025) find that, for a fixed pretrained model, RLVR updates localize to preferred parameter regions that are highly consistent across runs and largely invariant to datasets and RL recipes. Our matched-learning-rate results extend this observation to preference optimization. At a fixed learning rate, DPO and GRPO masks overlap far above the chance level of approximately 0.013 at ρ=97.5\rho = 97.5%, regardless of the objective or the dataset. Shenfeld et al. (2026) attribute the lower forgetting of RL to on-policy sampling, which biases updates toward solutions close in KL to the base model. Since DPO is trained offline whereas GRPO is on-policy, this account could predict some difference between their updates. Within the configurations we test, however, no such difference appears in mask structure at matched learning rates.

Regarding numerical precision, Zhu et al. (2025) also show that bf16 precision hides small updates, which contributes to the apparent sparsity of RL fine-tuning. We observe the same effect in our setting. At a learning rate of 5×10−75\times10^{-7}, many per-step updates fall below bf16 resolution, so the magnitude scores take few distinct values and tie at the selection threshold. Ties are broken by small random noise before thresholding (Algorithm 5), so masks keep exactly 2.52.5% of parameters, but more than half of each mask at this learning rate is chosen by the tie-break. With fp32 master weights at 5×10−75\times10^{-7}, 94.294.2% of coordinates move, and the resulting mask has Jaccard 0.017 with the bf16 Oracle.

[Weakness 2, Question 2]: Correcting the Fixed-Point Argument

The reviewer is correct: a finite-valued, non-increasing map need not have a fixed point, and repeated application can alternate between two values, so the original existence and convergence arguments in Appendix D.2 did not hold. We have replaced Theorem 3 with a corrected result that does not rely on the existence of a fixed point. Writing Φ(B)\Phi(B) for the withheld norm of the mask selected at BB, we show the following:

  1. Φ\Phi is non-increasing.
  2. Any self-consistent mask, one with Φ(B∗)=B∗\Phi(B^{\ast}) = B^{\ast}, minimizes the bound of Theorem 1. This follows from the convexity of the quadratic term.
  3. When no self-consistent mask exists, which is exactly the case the reviewer describes, bisection on BB locates the point B∘B^{\circ} where Φ\Phi crosses the diagonal. The two masks on either side of the jump both minimize the linearized objective at B∘B^{\circ}, and the better of them is within an explicitly certified gap, CS2(B(M)−B∘)2\tfrac{C_S}{2}\big(B(\mathcal{M}) - B^{\circ}\big)^2, of the optimal bound.

Bisection replaces the original fixed-point iteration, so the algorithm no longer depends on convergence of repeated application of Φ\Phi.

AuthorsOpenReview ↗

Response to Reviewer oyqH - Part 2

[Weakness 2, Question 2]: When the Magnitude Oracle Matches the Optimal Score

We agree that the optimal score depends on curvature while the Oracle uses displacement magnitude alone, and that the conditions under which the two agree should be stated explicitly. We now give these conditions as Corollary 1 (Oracle Optimality). If s(θi)≤s(θj)s(\theta_i) \le s(\theta_j) for every withheld parameter ii and retained parameter jj of the Oracle mask, then the Oracle mask minimizes the bound of Theorem 1. The condition only concerns pairs on either side of the Oracle threshold. Since ∣Δθi∣≤∣Δθj∣\lvert\Delta\theta_i\rvert \le \lvert\Delta\theta_j\rvert for such pairs, it fails only when a withheld parameter's curvature exceeds a retained parameters by more than a factor (Δθj/Δθi)2(\Delta\theta_j/\Delta\theta_i)^2. We state it as an explicit assumption of the Oracle comparison rather than as a verified property.

[Weakness 2]: Figure 5 and the Pointwise Certification Condition

The reviewer is correct that the margin distributions in Figure 5 do not establish the pointwise condition of Theorem 2, since the condition must hold jointly for each parameter. We strengthened Theorem 2 so that it remains informative when the condition fails for some parameters. If the condition is violated on a set VV, the warm-start and Oracle masks differ in at most ∣V∣\lvert V\rvert parameters, so their Jaccard similarity is at least N−∣V∣N+∣V∣\frac{N - \lvert V\rvert}{N + \lvert V\rvert} with N=(1−ρ)∣θ∣N = (1-\rho)\lvert\theta\rvert. At ρ=99.5\rho = 99.5%, the condition holds for 99.8399.83–99.8899.88% of coordinates for k=50k = 50–200200, giving a certified bound of about 0.5–0.6. At ρ=97.5\rho = 97.5%, the margins in Figure 5 sit at the tie-break noise floor (Appendix C.3); parameters whose margin falls below the tie-break noise are counted as violations, so exact ties from bf16 updates cannot inflate the certificate.

I appreciate the authors' thorough response a lot and their willingness to directly acknowledge and address the errors and incorrect claims in the original submission. I will take these revisions into account in my final assessment :)

Reviewer_rNH7OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies sparse subnetworks in preference-optimized LLMs. Given a dense fine-tuning update Δθ\Delta\theta, a binary mask MM at sparsity ρ\rho is built from a per-parameter score. The main practical score is warm-start magnitude, s(θi)=∣θik−θi0∣s(\theta_i) = |\theta^k_i - \theta^0_i|. The "Oracle" is the same score at k=Tk = T.

The paper makes five claims:

  1. Recovery of dense training. At ρ=97.5%\rho = 97.5\%, Oracle and warm-start masks recover dense DPO/GRPO training loss on Llama-3.1-8B-Instruct, with supporting curves for Qwen3-8B/32B and OLMo-3-7B. Random masks do not.
  2. Objective over data. Subnetwork structure is determined mainly by the learning objective rather than the data. DPO masks on Light-R1 and Tülu3 have Jaccard ≈0.85\approx 0.85, while DPO and GRPO masks diverge in Jaccard, CKA, and linear probes.
  3. Theory. A loss-retention bound (Theorem 1), an "optimal" scoring function defined through a fixed point (Theorem 3), and a per-parameter certifiability condition (Theorem 2).
  4. Certifiability check. An empirical check of the certifiability margins.
  5. Kernel. A Triton sparse AdamW kernel giving ∼1.7×\sim 1.7\times optimizer-step speedups over PyTorch AdamW at ρ=97.5%\rho = 97.5\%.
Critical strengths
  1. Asking whether subnetwork structure is set by the objective or by the data is interesting, and Jaccard, CKA, and probing are reasonable tools for it.
  2. The loss-curve experiments cover several model families (Llama, Qwen3 at two sizes, OLMo-3). The random-mask baseline makes the "which parameters, not how many" point clearly.
  3. The paper derives closed-form random-mask baselines for Jaccard (Appendix A.4.4), which gives a useful chance-level reference.
  4. The authors release an engineering artifact, the sparse AdamW Triton kernel, and are candid about its limits, including dense moment storage and coalescing inefficiency (Appendix B.4).
Critical weaknesses
  1. The probing and CKA analyses evaluate a different model from the one studied. Appendix A.4.3 states that for probing, and also for CKA, which shares the extraction stage, every masked tensor is replaced by W~=W⊙M\tilde W = W \odot M, so masked entries are set to zero. At ρ=97.5%\rho = 97.5\%, this zeroes 97.5% of the covered pretrained weights. The subnetwork defined in Section 3.1 is instead θM=θ+M⊙Δθ\theta_M = \theta + M \odot \Delta\theta, where frozen weights keep their pretrained values.

    • The probes and CKA therefore measure activations of an almost entirely ablated network, not of the fine-tuned subnetwork.
    • This likely explains why syntax probe accuracy for Oracle, warm-start, and random masks falls to about 0.30–0.40 in Figures 17–18, below the 0.5 chance level of a balanced binary task.
    • It means Figure 2 (CKA), Figure 4, and Section 4.3 do not support the claims that DPO and GRPO subnetworks have different internal representations or that "selected parameters encode these capabilities."
    • Section 3.3.3 also describes a pairwise Bradley–Terry probe on h+−h−h^+ - h^-, while Appendix A.4.3 describes per-text logistic regression. Please clarify which was used.
  2. The objective-versus-data claim is confounded and partly contradicted by the paper's own figure.

    • Hyperparameters differ. The DPO and GRPO runs differ in learning rate (5×10−75\times10^{-7} vs. 5×10−65\times10^{-6}), gradient clipping (1.0 vs. 0.1), effective batch size (128 vs. 8), warmup, and GPU count (Tables 7–8). Magnitude-based masks under Adam are strongly shaped by these settings, so the DPO/GRPO divergence may reflect optimization configuration rather than the objective. The two DPO runs that agree share identical hyperparameters.
    • No seed-to-seed baseline. Mask overlap is not reported for the same objective and data with different seeds, so "0.85" has no within-condition reference.
    • GRPO masks also diverge across data. Figure 2 appears to show GRPO masks trained on different datasets (Open-R1 vs. Tülu3, Open-R1 vs. ReasonMed) at Jaccard of only about 0.3, and the text says they are "dissimilar by Jaccard and CKA." If so, the claim holds for DPO but not GRPO, and the abstract overstates it.
    • DPO–GRPO overlap is not near zero. The DPO–GRPO Jaccard in Figure 2 looks closer to 0.1–0.15 than near zero, roughly ten times the E[J]≈0.0127\mathbb{E}[J] \approx 0.0127 chance level.
  3. The theory does not apply to the method or experiments as stated.

    • Theorem 1 bounds a different quantity. It bounds L(θ+M⊙Δθ)−L(θ+Δθ)L(\theta + M\odot\Delta\theta) - L(\theta + \Delta\theta), the effect of masking a fixed dense update after the fact. The experiments instead train the subnetwork sparsely, which produces a different update, so the bound says nothing about the training curves.
    • Theorem 1 has internal inconsistencies. The main text writes the diagonal term as Hii2θi2\tfrac{H_{ii}}{2}\theta_i^2, while the proof uses κi2Δθi2\tfrac{\kappa_i}{2}\Delta\theta_i^2. The statement uses ∥∇L(θ)∥∞\|\nabla L(\theta)\|_\infty, while the proof uses ∇L(θ+Δθ)\nabla L(\theta + \Delta\theta). CC is defined from HH at θ\theta, but the residual involves H(ξ)H(\xi) along the segment, so a supremum over the segment is needed. β\beta-smoothness is assumed and never used.
    • Theorem 1 is likely vacuous. The bound depends on C2∥δ∥12\tfrac{C}{2}\|\delta\|_1^2 with C=max⁡i∑j≠i∣Hij∣C = \max_i \sum_{j\ne i}|H_{ij}| and ∥δ∥1\|\delta\|_1 summed over roughly 8×1098\times10^9 parameters. The claim that "CC is small" is supported only by citing diagonal preconditioners. No value of CC or of the bound is reported, so its tightness is unknown.
    • Theorem 3 existence argument. The argument is that Φ\Phi is non-increasing and takes finitely many values, "so a fixed point exists." A non-increasing step function on a finite set can jump over the diagonal, so this does not follow.
    • Theorem 3 uniqueness argument. Under the alignment condition the ranking induced by s∗(θi,B)s^*(\theta_i, B) does not depend on BB, so Φ\Phi is constant, not "strictly decreasing" as claimed. In that case s∗s^* produces the same mask as plain ∣Δθi∣|\Delta\theta_i| ranking, which reduces the "optimal scoring function" to the Oracle.
    • Theorem 2 is close to trivial and compares quantities on incompatible scales. ss and s∗s^* have different units, and the margin is measured against ss's own threshold τρ(s)\tau_\rho(s). The statement "at a local minimum, Fi=HiiF_i = H_{ii} exactly" is false in general: the Fisher equals the Gauss–Newton term, which matches the Hessian only under additional conditions.
    • The empirical check tests something different. Section 4.4 measures ∣swarm−sOracle∣|s_{\text{warm}} - s_{\text{Oracle}}|, not the deviation from s∗s^*. The reported margins of 10−1710^{-17} to 10−1310^{-13} are far below bf16 resolution for these parameters, which raises the question of whether they reflect real differences or numerical artifacts.
Questions for authors
  1. Can you recompute the CKA and probing results on the actual subnetwork θ+M⊙Δθ\theta + M \odot \Delta\theta after sparse training, rather than on W⊙MW \odot M with frozen weights zeroed? Why does syntax probe accuracy fall below 0.5 for a balanced binary task?
  2. What is the mask Jaccard for the same objective and dataset with different seeds? What happens to the DPO–GRPO comparison when learning rate, batch size, clipping, and optimizer are matched?
  3. How do you reconcile "structure is determined by the objective, not the data" with the low Jaccard between GRPO masks trained on different datasets in Figure 2?
  4. Can you report numerical values of CC, ϵ\epsilon, and ∥δ∥1\|\delta\|_1, and the resulting value of the Theorem 1 bound, for at least one model?
  5. Please correct the existence and uniqueness arguments in Theorem 3. Under the alignment condition, does s∗s^* not reduce to ranking by ∣Δθi∣|\Delta\theta_i|?
  6. What are the held-out preference accuracy and reward-model or win-rate evaluations for dense, Oracle, warm-start, random, and LoRA, over multiple seeds?
  7. How do warm-start masks perform for small kk (for example k≤25k \le 25, i.e. ≤5%\le 5\% of training)? How do they compare with other one-shot scores such as SNIP, Fisher, or Wanda-style scores?
  8. What are the end-to-end training wall-clock time and peak memory for sparse vs. dense training?
  9. Are the reported margins of 10−1710^{-17} to 10−1310^{-13} meaningful given bf16 weights?
5 comments
AuthorsOpenReview ↗

Response to Reviewer rNH7 - Part 1

We thank the reviewer for the exceptionally careful and detailed review. We are pleased that the reviewer found the question of whether subnetwork structure is determined by the learning objective or by the training data to be interesting, and considered Jaccard similarity, CKA, and linear probing to be appropriate tools for studying it. We also appreciate the recognition of our evaluation across several model families, of the random-mask baseline as a clear demonstration that performance depends on which parameters are updated, of the closed-form random-mask baselines for Jaccard similarity, and of the sparse AdamW kernel and its candid discussion of limitations. The reviewer's comments on the theoretical framework were especially valuable, and led us to substantially revise and strengthen Theorems 1–3. Below, we respond to each of the reviewer's comments.

[Weakness 1, Question 1]: Probing and CKA Evaluate an Ablated Network

We thank the reviewer for raising this point, and agree that the original manuscript did not make the purpose of this design clear. Setting the withheld weights to zero was a deliberate choice. It is an ablation that isolates the parameters selected by each mask and measures which capabilities those parameters carry on their own. Following the reviewer's suggestion, we also ran the probing and CKA analyses on the subnetwork θ+M⊙Δθ\theta + \mathcal{M}\odot\Delta\theta defined in Section 3.1, in which withheld weights retain their pretrained values. In this setting, CKA is approximately 1.0 at every layer for every pair of masks, including random masks, and linear-probe accuracy matches the base model for random, DPO Oracle, and GRPO Oracle masks across all four tasks and both learning rates (Figure 23; CKA in Figure 20, right). This is expected: each subnetwork changes only 2.52.5% of the parameters, by small amounts, so its representations are dominated by the shared pretrained weights, and these analyses cannot distinguish any mask from a random one. The ablation isolates the contribution of the selected parameters, and therefore provides the informative comparison.

We have revised Appendix A.5.2 and the caption of Figure 22 to distinguish the two objects explicitly, and to state its conclusions as properties of the selected parameters rather than of the representations of the fine-tuned subnetwork. The below-chance syntax accuracies occur only for random masks and for masks built at lr 5×10−75\times10^{-7}, where only about 11% of coordinates move in bf16, so most of each mask is chosen by tie-breaking and behaves almost like a random mask; at 5×10−65\times10^{-6}, both Oracle masks are above chance from the first layer (Figure 24). We also resolved the inconsistency between Section 3.3.3 and Appendix A.5.2: all results use the pairwise Bradley–Terry probe on h+−h−h^{+} - h^{-}, and both sections now describe it.

[Weakness 2, Questions 2–3]: Objective Versus Data

We agree that the original comparison confounded the objective with its recipe, whose learning rates differ by 10×10\times. We repeated the comparison at matched learning rates (Appendix A.5, Tables 4–5):

Oracle-mask Jaccardlr 5×10−75\times10^{-7}lr 5×10−65\times10^{-6}
seed ↔\leftrightarrow seed0.9570.66–0.68
DPO Light-R1 ↔\leftrightarrow DPO Tülu30.800.31
GRPO Open-R1 ↔\leftrightarrow GRPO Tülu3-RLVR0.920.31
DPO Tülu3 ↔\leftrightarrow GRPO Tülu3-RLVR0.900.30
DPO Light-R1 ↔\leftrightarrow GRPO Open-R10.790.32
random ↔\leftrightarrow random0.0130.013

At matched learning rates, the DPO–GRPO separation largely disappears, and GRPO and DPO masks diverge across datasets by the same amount. The apparent contradiction in the original Figure 2 arose because the DPO pair was measured at 5×10−75\times10^{-7} and the GRPO pair at 5×10−65\times10^{-6}. The original DPO–GRPO overlap (0.09–0.13) is about 10×10\times chance, not near zero. We have scoped the claim in the abstract, introduction, Section 4.2, and conclusion to the standard training recipe of each objective.

AuthorsOpenReview ↗

Response to Reviewer rNH7 - Part 2

[Weakness 3]: Theorem 1 Bounds Post-Hoc Masking Rather Than Sparse Training

We thank the reviewer for this important point. Theorem 1 bounds L(θM)−L(θ+Δθ)\mathcal{L}(\theta_{\mathcal{M}}) - \mathcal{L}(\theta+\Delta\theta), where θM\theta_{\mathcal{M}} is the pretrained model plus the masked dense update, whereas our experiments train the subnetwork directly. We now connect the two explicitly in Section 3.2.1. Sparse training under M\mathcal{M} only updates the parameters selected by M\mathcal{M}, and θM\theta_{\mathcal{M}} changes only those parameters, so θM\theta_{\mathcal{M}} is one of the models that sparse training can reach. Therefore, if sparse training attains a loss no higher than L(θM)\mathcal{L}(\theta_{\mathcal{M}}), then L(θsparse)−L(θ+Δθ)≤L(θM)−L(θ+Δθ)\mathcal{L}(\theta_{\text{sparse}}) - \mathcal{L}(\theta+\Delta\theta) \le \mathcal{L}(\theta_{\mathcal{M}}) - \mathcal{L}(\theta+\Delta\theta), and Theorem 1 also bounds the loss gap of the sparse-trained subnetwork.

We also added a per-step descent-retention result (Proposition 1, Appendix D.5). It shows that a masked step retains the fraction ⟨M⊙g,u⟩/⟨g,u⟩\langle \mathcal{M}\odot g, u\rangle / \langle g, u\rangle of the first-order decrease of the dense step, which connects the theory to the gradient energy capture analysis in Appendix D.6.

[Weakness 3]: Internal Inconsistencies in Theorem 1

We thank the reviewer for identifying these, and have corrected all of them. The diagonal term and the scoring function are now written as Hii2Δθi2\tfrac{H_{ii}}{2}\Delta\theta_i^2 and s(θi)=κ(θi)2Δθi2s(\theta_i) = \tfrac{\kappa(\theta_i)}{2}\Delta\theta_i^2. The gradient term uses ϵ=∥∇L(θ+Δθ)∥∞\epsilon = \lVert\nabla\mathcal{L}(\theta+\Delta\theta)\rVert_\infty, consistent with the proof. The curvature constant is now defined as a supremum over the box Q={θ+t⊙Δθ:t∈[0,1]∣θ∣}\mathcal{Q} = \lbrace \theta + t\odot\Delta\theta : t\in[0,1]^{\lvert\theta\rvert}\rbrace. This box contains the segment used in the Taylor expansion for every mask, so the constant does not depend on the mask. Finally, we removed the unused β\beta-smoothness assumption: twice continuous differentiability suffices for the Lagrange form of the remainder, and also guarantees that the supremum is finite.

We also corrected a further issue in the statement. The original constant contained only off-diagonal curvature, which holds only when κ(θi)=Hii\kappa(\theta_i) = H_{ii}. Theorem 1 now uses CS=sup⁡ξ∈Q∥H(ξ)−DS∥max⁡C_S = \sup_{\xi\in\mathcal{Q}}\lVert H(\xi) - D_S\rVert_{\max}, so the bound holds for an arbitrary scoring function. CSC_S separates into the off-diagonal curvature, which no scoring function can remove, and the mismatch between κ\kappa and the true diagonal.

[Weakness 3, Question 4]: Tightness of Theorem 1

We agree that the constant in the original bound, which summed ∣θ∣−1\lvert\theta\rvert - 1 off-diagonal entries per row, made the bound very loose at LLM scale. The revised Theorem 1 replaces this row sum with the largest absolute entry of H−DSH - D_S, which is substantially tighter. Prior work provides evidence that the off-diagonal coupling captured by CSC_S is small relative to the diagonal blocks in Transformers. Zhang et al. (2024) show that Hessian spectra differ sharply across parameter blocks in Transformers, and Dong et al. (2025) show that a near-block-diagonal Hessian structure emerges as the number of output classes grows, as in LLMs with vocabularies exceeding 10410^4. We nevertheless acknowledge that, because of the ∥δ∥12\lVert\delta\rVert_1^2 term, the bound remains numerically loose at the scale of our experiments. The measured terms (DPO on Light-R1, 256 pairs) confirm this:

lrmaskwithheld ∥δ∥1\lVert\delta\rVert_1measured loss gap (held-out)ϵ∥δ∥1\epsilon\lVert\delta\rVert_1
5×10−75\times10^{-7}Oracle / warm-start / random2.5×10−62.5\times10^{-6} / 2.80 / 27860.00060.0006 / −0.0017-0.0017 / 0.0060.0062×10−52\times10^{-5} / 24 / 2.4×1042.4\times10^{4}
5×10−65\times10^{-6}Oracle / warm-start / random1.40×1031.40\times10^{3} / 1.41×1031.41\times10^{3} / 1.18×1041.18\times10^{4}0.0030.003 / 0.0030.003 / 0.0390.0396.8×1036.8\times10^{3} / 6.9×1036.9\times10^{3} / 5.8×1045.8\times10^{4}

Its role in our framework is structural: it identifies the score-dependent quantity that a mask should minimize. This yields the optimal scoring function of Theorem 3, the Oracle optimality condition of Corollary 1, and the certifiability condition of Theorem 2. We have revised Appendix D.1 to state this scope explicitly.

AuthorsOpenReview ↗

Response to Reviewer rNH7 - Part 3

[Weakness 3, Question 5]: Existence and Uniqueness in Theorem 3

The reviewer is correct: a non-increasing step function can jump over the diagonal, so the original argument did not establish that a fixed point exists, and the uniqueness argument did not follow either. We have replaced Theorem 3 with a corrected result (Appendix D.2). It shows that Φ\Phi is non-increasing, and that any self-consistent mask (one with Φ(B∗)=B∗\Phi(B^{\ast}) = B^{\ast}) minimizes the bound of Theorem 1, by a convexity argument. When no self-consistent mask exists, bisection on BB returns a mask whose bound is within an explicitly certified gap, CS2(B(M)−B∘)2\tfrac{C_S}{2}\big(B(\mathcal{M}) - B^{\circ}\big)^2, of the optimum.

We also agree with the reviewer's observation that, under the alignment condition, the optimal scoring function reduces to the Oracle ranking. We now state this explicitly as Corollary 1 (Oracle Optimality). If s(θi)≤s(θj)s(\theta_i) \le s(\theta_j) for every withheld parameter ii and retained parameter jj of the Oracle mask, then the Oracle mask minimizes the bound of Theorem 1. This condition only concerns pairs on either side of the Oracle threshold. Since ∣Δθi∣≤∣Δθj∣\lvert\Delta\theta_i\rvert \le \lvert\Delta\theta_j\rvert for such pairs, it fails only when a withheld parameter's curvature exceeds a retained parameter's by more than a factor (Δθj/Δθi)2(\Delta\theta_j/\Delta\theta_i)^2.

[Weakness 3]: Theorem 2

We thank the reviewer for these observations, and revised Theorem 2 to address each of them. First, the condition now compares ss with h(s∗)h(s^{\ast}) for a strictly increasing scaling function hh, so that the two scores are on the same scale. A mask depends only on the ranking a scoring function induces, so h(s∗)h(s^{\ast}) selects the same mask as s∗s^{\ast}. Second, the margin is now measured from the selection threshold of s∗s^{\ast}. We also show in Appendix D.3 that the theorem holds equally with the margin measured from the threshold of ss. Third, we corrected a further issue: with the threshold defined as the score of the last retained parameter, that parameter has zero margin, so the original condition could never hold. The threshold is now taken strictly between the lowest retained and highest withheld values.

The revised theorem also gives a quantitative guarantee. If the condition is violated on a set VV of weights, the two masks differ in at most ∣V∣\lvert V\rvert parameters, so their Jaccard similarity is at least N−∣V∣N+∣V∣\frac{N - \lvert V\rvert}{N + \lvert V\rvert} with N=(1−ρ)∣θ∣N = (1-\rho)\lvert\theta\rvert. For random scoring, this implies E∣V∣≥ρN\mathbb{E}\lvert V\rvert \ge \rho N, so the certificate correctly fails (Appendix D.4).

Finally, the reviewer is correct that the Fisher equals the Gauss–Newton term, and matches the Hessian only under additional conditions. We removed the claim that Fi=HiiF_i = H_{ii} exactly at a local minimum, and now describe the diagonal Fisher as a Gauss–Newton approximation.

[Weakness 3, Question 9]: Empirical Certifiability Check and bf16 Resolution

We agree that Section 4.4 measured the deviation from the Oracle rather than testing the certifiability condition itself. At ρ=99.5\rho = 99.5%, the condition holds for 99.8399.83–99.8899.88% of coordinates for warm-start k=50k = 50–200200, giving a certified Jaccard lower bound of about 0.5–0.6. The reviewer is also correct that margins of 10−1510^{-15} to 10−1710^{-17} lie far below bf16 resolution. They reflect exact ties broken by the tie-break noise of Algorithm 5: at ρ=97.5\rho = 97.5%, the selection threshold is only about 2×2\times the noise amplitude, whereas at ρ=99.5\rho = 99.5% it is about 109×10^{9}\times (Appendix C.3, referenced from Section 4.4 and Figure 5). Parameters whose ordering is decided by tie-breaking have negligible margin and fall in VV, so ties cannot inflate the certificate (Appendix D.3).

[Question 6]: Held-Out Preference Accuracy and Multiple Seeds

Held-out Tülu3 preference accuracy per seed (500 pairs, SE ≈0.022\approx 0.022; Figure 26):

lrdenseOraclewarm-startrandomLoRA
5×10−75\times10^{-7}0.516 / 0.5320.516 / 0.5320.506 / 0.5180.458 / 0.4780.510 / 0.540
5×10−65\times10^{-6}0.570 / 0.590 / 0.5680.566 / 0.580 / 0.5660.596 / 0.5700.524 / 0.5260.556 / 0.566 / 0.566

At 5×10−65\times10^{-6}, warm-start vs random gives McNemar p=0.0065p = 0.0065 and warm-start vs dense p=1.0p = 1.0. We did not run reward-model or win-rate evaluations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.