- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies when inactive directions in gated low-rank adapters can be safely removed during training without changing subsequent optimization decisions. The authors propose a framework combining future-inactivity certification with compact summaries of the discarded optimizer state. They provide theoretical guarantees for preserving the original learning dynamics under specified assumptions and evaluate the approach on Letter Recognition, CIFAR features, and synthetic settings.
Critical strengths
- The paper studies an interesting problem: preserving not only model predictions but also subsequent optimization behavior when removing inactive parameters.
- The proposed framework provides mathematical guarantees for safe state removal under the specified optimization dynamics.
- The experiments examine both the correctness of the preserved training decisions and the storage/computational costs of certification.
Critical weaknesses
-
Poor readability and unclear presentation. The paper is difficult to follow due to extensive specialized terminology and insufficient intuitive explanations. Important concepts, assumptions, and their relationships are not clearly motivated. A simple end-to-end example illustrating the actual problem, proposed algorithm, and practical benefits would substantially improve accessibility.
-
Limited practical applicability. The theoretical framework relies on a highly specific gated adapter and optimization procedure. It is unclear how the approach can be applied to commonly used training setups, such as LoRA fine-tuning with AdamW.
-
Insufficient experimental validation. The experiments focus on relatively small classification tasks with fixed features and synthetic settings. Evaluations on realistic large-scale models and fine-tuning tasks would help demonstrate the practical relevance of the method.
-
Unclear efficiency benefits. Despite the motivation of reducing training-state storage, the reported savings are relatively limited when the complete storage cost is considered, while certification introduces additional computational overhead. For example, Table 4 reports a 9.49% storage reduction on CIFAR with a 1.43× runtime cost, and negative storage savings on Letter. The practical trade-off is therefore not sufficiently convincing.
Questions for authors
- Could the authors provide a simple example or algorithmic overview explaining how the different components work together and what is actually removed during training?
- How could the proposed framework be extended to standard LoRA training with commonly used optimizers such as AdamW?
- Under what model sizes, training durations, or sparsity levels would the storage savings outweigh the certification overhead?
- How does the proposed approach compare with simpler alternatives, such as retaining inactive directions without updating them or using conventional checkpoint compression?
End-to-end example, AdamW extension, break-even conditions, and comparison with lazy retention and checkpoint compression
We thank the reviewer for the careful reading and concrete questions. We agree that readability is the main weakness, and we will restructure the paper around the example below. Q1. End-to-end example: what is removed and why it is hard. The adapter is W = W_0 + s * B * Diag(γ) * A; atom j is the rank-one pair (b_j, a_j) with gate γ_j in [0, 1]. An L1-type gate cost drives some gates to exactly zero during training. The atom then no longer affects predictions, so one would like to delete b_j and a_j from the training state. Two things can go wrong: