- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether preference optimization of LLMs can be supported by extremely sparse, high-rank weight updates. It combines per-parameter saliency scoring with hybrid local-global mask construction, presents analyses of loss retention, optimal scoring, and certifiability under stated assumptions, and uses early dense-training weight displacement as a practical approximation to full-run Oracle scoring. Experiments cover DPO and GRPO, multiple datasets and model families, and subnetwork comparisons using Jaccard similarity, CKA, and linear probes. The systems component implements a sparsity-aware AdamW optimizer and reports optimizer-step speedups.
Critical strengths
1.The work connects theoretical analysis, subnetwork diagnostics, and a systems implementation. On Llama-3.1-8B-Instruct, warm-start magnitude masks track dense DPO training dynamics while updating about 2.5% of parameters during the sparse phase (Figure 1). Complementary cross-model training experiments mainly use Oracle masks (Appendix A.1). Sparsity sweeps, cross-task transfer, and representation diagnostics provide insight beyond final accuracy alone. These empirical contributions offer a useful framework for further study, even though the theoretical claims need the clarification requested below. 2.The paper examines parameter overlap, internal representations, functional probes, and hardware timings, and explicitly distinguishes full-run Oracle scoring from the more practical warm-start alternative. The model-family extensions and sparse optimizer implementation mean that the main observations do not rely entirely on one model or one diagnostic. This breadth increases the potential impact of the work.
Critical weaknesses
1.The relative influence of learning objective and training data would benefit from a more controlled comparison. DPO masks from Light-R1 and Tülu3 have high overlap, whereas Figure 2 reports lower cross-dataset overlap and CKA for GRPO. Figure 14 also shows useful GRPO cross-dataset transfer, which is distinct from structural similarity. The objective comparisons use different training recipes, including learning rates, schedules, and batch configurations (Tables 7-8). A focused comparison on the same base model with comparable data exposure and training budgets, together with data-construction and sampling details, would strengthen the attribution to the objective. Otherwise, the conclusion should be limited to the tested configurations. 2.End-to-end training efficiency remains insufficiently quantified. At 97.5% sparsity, Table 2 reports a 1.69x AdamW optimizer-step speedup over PyTorch AdamW, not a full-training speedup. Appendix B.4 states that the deployed implementation retains dense first- and second-moment buffers and adds mask/index storage, so reduced optimizer memory traffic does not imply reduced resident VRAM. Although LoRA quality comparisons are already provided, a matched-hardware comparison of total time, peak VRAM, and corresponding model quality for the practical warm-start method, dense AdamW, and LoRA would clarify the full-pipeline benefit. The accounting should include dense warm-start, scoring/mask construction, and subsequent training. 3.The fixed-point argument in Appendix D.2 needs correction or stronger assumptions. A finite-valued non-increasing map need not have a fixed point, and repeated application need not converge; it can alternate between two values. These properties therefore do not justify the existence and convergence claims as written. Section 3.2.2 already states existence and uniqueness under magnitude-curvature alignment, but Theorem 3 attaches this condition only to uniqueness and Appendix D.2 gives more general existence and convergence arguments. If alignment fixes the ranking with consistent tie-breaking, the map is constant, not strictly decreasing as stated in Appendix D.2. Please make the assumptions and proof consistent across these claims. This concern is about the theory and does not by itself invalidate the observed warm-start training results.
Questions for authors
1.Building on the existing comparisons, could you provide a focused DPO/GRPO comparison across two data sources using the same base model, comparable data exposure, and comparable training budgets, and report mask Jaccard similarities? Please clarify how the training inputs and sampling are constructed for each combination. This would help separate objective and data effects; if such matching is not feasible, please clarify the resulting limits on the objective-dominates-data claim. 2.Could you compare the practical warm-start method, dense AdamW, and LoRA on the same hardware under comparable training settings, reporting total training time, peak VRAM, and corresponding model quality? Please include dense warm-start, scoring/mask construction, and subsequent training in the timing, and identify the optimizer and state-storage path used. The existing optimizer-step timings and LoRA quality results can serve as context for this full-pipeline comparison.
2 comments
Thank you for the detailed response. The matched-learning-rate comparisons clarify how the objective and data affect mask overlap, and the end-to-end benchmark clarifies the practical cost of sparse training. The revised Theorem 3 addresses my concern about the fixed-point argument.
Response to Reviewer qwUr
We thank the reviewer for the thoughtful and constructive review. We are pleased that the reviewer found that the work connects theoretical analysis, subnetwork diagnostics, and a systems implementation, and that warm-start magnitude masks track dense DPO training while updating about 2.5% of parameters. We also appreciate the recognition that the breadth of diagnostics and model families means the main observations do not rely on a single model or diagnostic. Below, we respond to each of the reviewer's comments.
[Weakness 1, Question 1]: Controlled Comparison of Objective and Data
We agree that the two objectives used different recipes (Tables 10–11), and ran the requested comparison: DPO and GRPO on two data sources each, at two matched learning rates, on the same base model and with identical mask construction (Appendix A.5, Table 5):
At matched learning rates, changing the objective and changing the data shift overlap by similar amounts, so we limit the claim to the standard recipe of each objective, as the reviewer suggests. DPO sees 128 offline preference pairs per step; GRPO sees one prompt per step with 8 on-policy completions. As the reviewer notes, transfer is distinct from structural similarity: GRPO masks transfer across datasets (Figure 14) despite low overlap.
[Weakness 2, Question 2]: End-to-End Training Efficiency
On one B200 at matched settings (gradient checkpointing on), with held-out accuracy at lr 5×10−6:
Including the warm-start steps and mask construction (444 s), the full sparse pipeline takes about 1.45× the dense wall-clock. We agree with the reviewer's reading of Appendix B.4. The deployed implementation keeps dense first- and second-moment buffers and adds mask and index storage, so the sparse optimizer reduces optimizer memory traffic rather than resident VRAM. The reported 1.69× figure is an optimizer-step speedup, not an end-to-end training speedup, and the abstract and Section 4.5 state this.
[Weakness 3]: Consistency of the Fixed-Point Argument
We thank the reviewer for this precise analysis, and agree with each point. A finite-valued non-increasing map need not have a fixed point, and repeated application can alternate between two values, so the original existence and convergence claims did not follow. The reviewer is also correct that the assumptions were inconsistent: Section 3.2.2 stated existence and uniqueness under the alignment condition, whereas Theorem 3 attached it only to uniqueness. Furthermore, if the alignment condition fixes the ranking under consistent tie-breaking, Φ is constant rather than strictly decreasing.
We have replaced Theorem 3 with a corrected result whose assumptions are stated once and used consistently (Appendix D.2):
The alignment condition is no longer used for existence or uniqueness. It now appears only in Corollary 1 (Oracle Optimality), which states when the Oracle mask minimizes the bound, and requires the condition only for pairs of parameters on either side of the Oracle threshold. Section 3.2.2 has been revised to match. We agree with the reviewer that this concern does not affect the observed warm-start training results.