acceptodds
Withdrawn from ICLR 2027

When Can a Zero Be Deleted? Exact State Compression for Gated Low-Rank Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A low-rank component can leave the predictor while still determining the optimizer's next accepted update. We solve the resulting state-elimination problem for anchored gated low-rank training with one shared Euclidean ball. A current-checkpoint certificate keeps selected gates at zero over a declared horizon; one evolving radial statistic then replaces their factor directions in every projection, complete objective, and line-search decision, including rejected trials. An accepted-update family establishes why this radial information cannot all be discarded. The same construction separates when memory is reclaimed from which learning rule is executed: immediate, staged, and delayed release under one license preserve the same real training observations. A unified loss budget supplies computable certificates; a sharp cross-entropy bound admits 35 rather than 28 of 118 zero candidates in nonlinear confirmation. In a quality-first shared-backbone study, eight independently initialized adapters all admit release, removing 18 of 77 learned zero slots and 10.43% of their aggregate live state. All three release schedules match the full reference's trial signatures over 200 accepted blocks per adapter. Regression continuations retain 49.9–71.1% live-state savings, and directed rounding validates a complete nonlinear continuation. These results turn structural inactivity into a storage decision without adding a new real update rule.

Primary area:
optimization
Keywords:
Sparse LoRA, Gated-LoRA, Bregman Kernel, Proximal Operators, Active-Face Identification, Finite-Time Guarantees, Model Compression, Parameter-Efficient Fine-Tuning (PEFT)

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_QHY7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks when a low-rank component can be deleted after its gate becomes zero. Although the component no longer affects predictions, its factors can still affect optimization through a shared constraint and regularization penalty. For the particular optimizer studied here, the authors replace those factors with one scalar and certify that their gates will remain zero for a fixed number of updates. They prove that this preserves the training decisions and demonstrate storage savings on regression and small classification tasks.

Critical strengths
  • The authors check rejected trials and acceptance decisions, distinguish floating-point agreement from real-arithmetic guarantees, and report negative runtime results.
Critical weaknesses
  • The novelty is insufficiently established. Safe screening, dynamic variable elimination, and exact state reduction are substantial existing directions. The brief closing paragraph does not explain precisely how the proposed certificates and scalar reduction advance beyond them. Relevant connections include dynamic screening, Gap Safe screening, and formal neural-network compression. These methods preserve different objects, so a substantive comparison of assumptions and guarantees is essential. Without it, I find the claimed advance difficult to assess.
  • The demonstrated benefit is limited relative to the restrictive setup. The scalar reduction relies heavily on the isotropic penalty and shared-ball geometry. Validation remains on small problems, and the main eight-adapter experiment saves only about 14.6 KiB of live state while increasing completion time from 14.883 to 15.326 seconds. No reduction in process memory or increase in concurrent capacity is established. This leaves the paper short of demonstrating why the specialized guarantee matters in a practical training workload.
Questions for authors

Which part of the result goes beyond existing screening and exact-reduction results, and why does it require a new argument?

Reviewer_sq9hOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a way to reduce redundant information kept during gated low-rank adaptation. The key observation is that if a component has been zeroed by the gate, its impact to subsequent optimization steps can be encoded by a few numbers, under the assumption of an isotropic quadratic optimizer, a Euclidean ball constraint, and a projected-GD type optimizer, and that the component will not reactivate in the future. The authors derive sufficient conditions for the no-reactivation guarantee from objective-value bounds and gate-gradient estimates.

Critical strengths

I didn't find a particular strength of the paper.

Critical weaknesses
  1. The paper is extremely difficult to read. I was struck by the very first sentence "a low-rank component can leave the predictor while still determining the optimizer’s next accepted update". What does that even mean? A low-rank component of what? Predictor for what?What does it mean to leave a predictor? What's an "accepted update" of an optimizer? The paper is full of such undefined non-standard terms ("licenses", "certificate portfolios", etc.) that take superhuman effort to decipher.
  2. The algorithm relies critically on the presence of an isotropic quadratic anchor, one joint Euclidean-ball constraint, and a specific projected-gradient/Bregman-prox update scheme. These are very remote to practically relevant algorithms like the Adam family, which undermines the significance of the paper as it has no strong theoretical contribution either.
  3. The main shared-backbone experiment uses a frozen 64->32->10 tanh network for rotated digit classification. Eight adapters are independently initialized, but they use the same task, data, and backbone. This is far removed from practical large-model fine-tuning where LoRA is actually used. The measured benefits are also small (Table 3).
Questions for authors

I have no particular questions to ask.

Reviewer_Xj7pOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors define a training setup in which a frozen weight matrix receives a low-rank update made of rank-one terms, each scaled by a gate that a penalty pushes toward zero, and all terms are kept within a fixed distance of the values they reach after a warm-up. Within this setup they study when the terms whose gate has reached zero can be deleted without changing training at all. This is hard because those terms still share the distance limit with the others, so deleting them changes the next update, and because a gate at zero could later become nonzero. They show that one number is enough to replace the deleted terms and give a check that the gates stay at zero for a fixed number of steps, supported by small experiments.

Critical strengths
  • The problem the authors approach is hard since an inactive part of the model can still change how training proceeds. Solving it exactly may be useful beyond this setting, for example in fine-tuning with adapters, although the paper does not make this case.
  • The theory appears generally correct with non-trivial results from the setup.
  • The paper shows that a single number is both enough and necessary to replace the deleted factors, which is a tight answer to the question the main research qquestion
  • The authors design a modified entropy for the gates so that they can reach exactly zero
  • Results are reported for every seed, so the experiments do not appear cherry-picked
Critical weaknesses
  • The paper does not explain why the problem matters, in practice or in theory, or where it sits in the literature. Related work is a few sentences that list papers without saying how this work differs from them, including the closest prior idea of screening coordinates that will stay zero. As a reader I had to guess at the value of the work.
  • The training setup the theory depends on is never motivated, for instance the shared ball ∣Θ−Θw∣≤R|\Theta - \Theta_w| \le R around the anchor Θw\Theta_w, the anchor penalty μ2∣Θ−Θw∣2\frac{\mu}{2}|\Theta - \Theta_w|^2, a projected gradient optimizer with line search, and full-batch gradients. These choices are what make the problem hard and what make the solution possible, yet the paper doesn't say where they come from or which realistic training settings use them.
  • The central claim is that deletion is exact but the paper does not show what exactness gains over simply deleting the inactive factors. In their experiments (sec 6.3) simple deletion changes training only slightly, and its effect on final loss or accuracy is never reported.
  • The certificate for nonlinear models relies on worst-case bounds over the whole feasible region.It's already conservative for a two-layer network, where it releases only part of the inactive gates, and it would likely be far too loose for deeper networks. So also it is unclear how the method could scale.
  • The experiments use very small models and task, and seeds are reported in full, but scale and architecture barely vary, which is where the usefulness of the certificate is most uncertain to me. Also, the memory saved is small and there is no speedup.
  • I find the paper very hard to read as many terms are used without being defined, so even the setting is difficult to follow.
Questions for authors
  • Is there a reason for choosing this optimizer, and would the results hold with Adam, where each deleted coordinate would be rescaled differently and ζ\zeta alone might no longer be enough?
  • Without the shared ball and the anchor penalty, as in standard LoRA training, would deleting a slot with γj=0\gamma_j = 0 already be exact?
  • Does the loss budget behind the certificate extend to minibatch training, where the objective can increase?
  • The method does not allow the step-size logic to use the gradient of a deleted gate. How does this interact with common practices such as clipping gradients by their global norm?
  • What happens if a deleted gate would need to become active again after the certified horizon HH? how would recovery work, given that only ζ\zeta is kept?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.