- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks when a low-rank component can be deleted after its gate becomes zero. Although the component no longer affects predictions, its factors can still affect optimization through a shared constraint and regularization penalty. For the particular optimizer studied here, the authors replace those factors with one scalar and certify that their gates will remain zero for a fixed number of updates. They prove that this preserves the training decisions and demonstrate storage savings on regression and small classification tasks.
Critical strengths
- The authors check rejected trials and acceptance decisions, distinguish floating-point agreement from real-arithmetic guarantees, and report negative runtime results.
Critical weaknesses
- The novelty is insufficiently established. Safe screening, dynamic variable elimination, and exact state reduction are substantial existing directions. The brief closing paragraph does not explain precisely how the proposed certificates and scalar reduction advance beyond them. Relevant connections include dynamic screening, Gap Safe screening, and formal neural-network compression. These methods preserve different objects, so a substantive comparison of assumptions and guarantees is essential. Without it, I find the claimed advance difficult to assess.
- The demonstrated benefit is limited relative to the restrictive setup. The scalar reduction relies heavily on the isotropic penalty and shared-ball geometry. Validation remains on small problems, and the main eight-adapter experiment saves only about 14.6 KiB of live state while increasing completion time from 14.883 to 15.326 seconds. No reduction in process memory or increase in concurrent capacity is established. This leaves the paper short of demonstrating why the specialized guarantee matters in a practical training workload.
Questions for authors
Which part of the result goes beyond existing screening and exact-reduction results, and why does it require a new argument?