acceptodds
Under review as a conference paper at ICLR 2027

When Learning Can Safely Forget: Transition-Faithful Certificates for Adapter State

Abstract

An inactive direction can disappear from predictions while still determining the next learning update. We construct state release that preserves an unfinished learner's update rule. Future-inactivity permission fixes omitted gate outputs; summaries closed under the remaining optimizer reads preserve their residual effects. Common radial dynamics retain one energy, complete linear-channel queries are characterized by a Gram matrix, and distinguishable quadratic modes retain separate energies. For the radial gated reference, a future cross-entropy budget yields an optimal adverse-probability relaxation; transporting a reliable dual cost reduces repeated checking. The reduction preserves declared learning events through adaptive release, rejection, rollback and corresponding-point restoration. On fixed Letter and CIFAR executions, compilation removes over 96% of dual evaluations while retaining over 90% of fresh-check release integrals. Paired trials, same-candidate interventions and fixed-mode dimension tests validate the information that must be retained. Same-checkpoint permission comparisons identify the additional positive authorizations from loss information, while complete-record measurements quantify storage gains and their acquisition costs.

Primary area:
optimization
Keywords:
parameter-efficient learning, sparse adaptation, checkpoint compression, gating networks, release certificates, resumable learning state, optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6zsfOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when inactive directions in gated low-rank adapters can be safely removed during training without changing subsequent optimization decisions. The authors propose a framework combining future-inactivity certification with compact summaries of the discarded optimizer state. They provide theoretical guarantees for preserving the original learning dynamics under specified assumptions and evaluate the approach on Letter Recognition, CIFAR features, and synthetic settings.

Critical strengths
  1. The paper studies an interesting problem: preserving not only model predictions but also subsequent optimization behavior when removing inactive parameters.
  2. The proposed framework provides mathematical guarantees for safe state removal under the specified optimization dynamics.
  3. The experiments examine both the correctness of the preserved training decisions and the storage/computational costs of certification.
Critical weaknesses
  1. Poor readability and unclear presentation. The paper is difficult to follow due to extensive specialized terminology and insufficient intuitive explanations. Important concepts, assumptions, and their relationships are not clearly motivated. A simple end-to-end example illustrating the actual problem, proposed algorithm, and practical benefits would substantially improve accessibility.

  2. Limited practical applicability. The theoretical framework relies on a highly specific gated adapter and optimization procedure. It is unclear how the approach can be applied to commonly used training setups, such as LoRA fine-tuning with AdamW.

  3. Insufficient experimental validation. The experiments focus on relatively small classification tasks with fixed features and synthetic settings. Evaluations on realistic large-scale models and fine-tuning tasks would help demonstrate the practical relevance of the method.

  4. Unclear efficiency benefits. Despite the motivation of reducing training-state storage, the reported savings are relatively limited when the complete storage cost is considered, while certification introduces additional computational overhead. For example, Table 4 reports a 9.49% storage reduction on CIFAR with a 1.43× runtime cost, and negative storage savings on Letter. The practical trade-off is therefore not sufficiently convincing.

Questions for authors
  1. Could the authors provide a simple example or algorithmic overview explaining how the different components work together and what is actually removed during training?
  2. How could the proposed framework be extended to standard LoRA training with commonly used optimizers such as AdamW?
  3. Under what model sizes, training durations, or sparsity levels would the storage savings outweigh the certification overhead?
  4. How does the proposed approach compare with simpler alternatives, such as retaining inactive directions without updating them or using conventional checkpoint compression?
1 comment
AuthorsOpenReview ↗

End-to-end example, AdamW extension, break-even conditions, and comparison with lazy retention and checkpoint compression

We thank the reviewer for the careful reading and concrete questions. We agree that readability is the main weakness, and we will restructure the paper around the example below. Q1. End-to-end example: what is removed and why it is hard. The adapter is W = W_0 + s * B * Diag(γ) * A; atom j is the rank-one pair (b_j, a_j) with gate γ_j in [0, 1]. An L1-type gate cost drives some gates to exactly zero during training. The atom then no longer affects predictions, so one would like to delete b_j and a_j from the training state. Two things can go wrong:

  1. The gate may come back. Its next update depends on the direction of (b_j, a_j), so after deletion the next gate value is unknown. Fix: permission, a certified lower bound showing that the gate's forcing stays positive, so the gate stays exactly 0, for the whole remaining run. We derive it from a computable bound E on future training cross-entropy, which prevents all samples from being adverse at once (Thm. 3).
  2. A dead direction still affects other parameters. The optimizer's joint norm-ball projection, ridge term and accept/reject test all read ||d_D||^2. Naive deletion flips 42 of 1,864 accept/reject decisions. Fix: summary. We keep one scalar v = ||d_D||^2, updated by v+ = ψ^2 * v, which reproduces every such read exactly (Table 1). The algorithm runs every 10 sweeps. For each zero-gate atom it computes the bound (cached and rescaled between checks, so 38 instead of 1,637 evaluations on CIFAR); if the bound is certified positive, it deletes (b_j, a_j) and adds their energy to v. Theorem 1 states that the run then makes exactly the same decisions as the full run, including after rollback and checkpoint restore. Empirically, all 48,539 decisions match. Applicability and Q2 (LoRA + AdamW). The framework needs exact zeros (e.g. SoRA-style proximal gates) and a characterization of which hidden quantities are still read. Our general result is that summary size equals the number of distinguishable hidden dynamics: one energy for radial updates, a Gram matrix for linear channels, K energies for K modes. For AdamW, a released coordinate has zero task gradient after permission, so m_i <- β1 * m_i and s_i <- β2 * s_i. With ε = 0, every ratio m_i / sqrt(s_i) scales by the same factor, and the 2x2 Gram matrix of [d_D, m_D / sqrt(s_D)] is exactly sufficient. With ε > 0, coordinates must be grouped by second-moment value (per coordinate in the worst case, i.e. no compression). If no global norm control reads the parameters, released coordinates are not read at all and only permission remains; our global-ball certificate (App. D) covers any dynamics inside a norm ball. We will add this derivation as an appendix, labeled as not yet evaluated. Experiments and Q3 (when savings pay off). We agree that the tasks are small; they were chosen to test an exactness claim at the level of individual decisions. Savings depend on atom size, the number of released atoms and pause residence time, while certification cost depends on the number of checks. Letter is negative because its atoms are small (1,432 bytes) and only 6 are released, so fixed control bytes dominate; larger CIFAR atoms (3,160 bytes) yield a 9.49% saving. The fixed-mode test isolates size: a 100-fold increase in coordinates per mode raises the full record from 14,115 to 346,737 bytes, while the summary stays at about 10.3 KB (Table 3). Our break-even rule (Sec. 6) says release lowers total cost if and only if a_mem * h * ΔB > b_time * ΔT. In transformer LoRA an atom has d_in + d_out entries per layer, so we expect the favorable regime, but we have not measured it and will state this as a limitation. Compiled CE is 33% faster than the geometric certificate on CIFAR while releasing 39% more. We do not claim faster training or lower peak memory than unchecked full (1.43x time on CIFAR). Q4. Simpler alternatives. (a) Keeping inactive directions without updating them changes the run by construction, because the full learner contracts them and their norm enters projection, ridge and acceptance. The exact version of this idea is our "lazy" baseline (store each direction plus a scalar multiplier). It matches all decisions but still stores the directions: 344,538 vs 10,362 bytes at K = 4 and 1,800 coordinates per mode (Table 3). (b) Checkpoint compression is complementary. Every byte count we report is already after zlib compression of all representations (raw/zlib-0/6/9 in Table S5), and lossy compressors such as ExCP do not preserve the trajectory. Release removes information that is provably unneeded; compression can then be applied on top. We will also add an overview figure, a glossary and section roadmaps. We hope these answers clarify the contribution.
Reviewer_v6V1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper aims to remove inactive adapter parameters from memory while preserving the subsequent training process. It combines two requirements: proving that the removed components will remain inactive throughout the remaining training run, and preserving their effects on the optimizer through compact summaries. Experiments on Letter and CIFAR classification tasks, together with synthetic controls, show agreement with the original training decisions and substantially fewer certificate evaluations through reuse of previous calculations. However, complete checkpoint-storage savings vary across workloads.

Critical strengths
  1. The paper addresses an important problem in memory-efficient training: safely discarding inactive adapter state while preserving the optimizer’s subsequent decisions, rather than only maintaining the model’s current predictions.
Critical weaknesses
  1. The experiments use small, fixed-feature classification tasks, providing limited evidence of benefits at practical training scales. Even on CIFAR, the proposed method incurs approximately 43% more continuation time and 7% higher peak process memory than continuing with the full state without certification. Although it reduces checkpoint size in this setting, these results do not demonstrate faster training or lower peak training memory, limiting the practical case for reducing training resource costs.

  2. The main scalar-summary construction relies on restrictive training dynamics. In particular, all released coordinates undergo the same scalar transformation, dD+=ψdDd_D^+=\psi d_D . This structure allows their effects to be represented by a single squared norm. Although the paper considers additional structured dynamics, it does not establish comparable compression and certification for optimizers that apply different scaling factors to different coordinates.

Questions for authors

Questions for the authors

  1. Can the method achieve lower total runtime or peak memory on larger adapter-training workloads after accounting for certification and checkpointing overhead?
  2. Can the inactivity certificate and compact-state construction extend to coordinate-adaptive optimizers such as AdamW, and what additional information would need to be retained?
1 comment
AuthorsOpenReview ↗

Scope is the persistent resumable state; scaling regime for savings and exact summary conditions for AdamW

We thank the reviewer for the accurate summary and for recognizing the soundness of the guarantee. We address both weaknesses and both questions directly. W1/Q1. Scale, runtime and peak memory. The reviewer is right that our measurements do not show faster training or lower peak training memory, and we do not claim them. The resource we target is the persistent resumable state, i.e. what must be stored at a pause to continue the exact same run, not the transient working set. Relative to unchecked full, compiled certification costs 1.27x (Letter) and 1.43x (CIFAR) continuation time and 1.15x/1.07x full-process peak RSS; CIFAR saves 9.49% of the complete record and Letter costs 0.66% (Table 4, App. G). We will state these trade-offs in the abstract and introduction rather than only in Section 5. What the current data indicate about scaling:

  1. Certification cost depends on the number of certified checks, not on the number of sweeps. Caching cuts bound evaluations from 1,637 to 38 on CIFAR and permission time from 50.8 s to 2.3 s. Compiled CE is also 33% faster than the geometric baseline while releasing 39% more.
  2. The saving per released atom grows with atom size, while the summary stays fixed. In the fixed-mode test, a 100-fold increase in coordinates per mode raises the full record from 14,115 to 346,737 bytes, while the summary stays at about 10.3 KB (Table 3). Letter is negative because its atoms are small (1,432 bytes) and only 6 are released across jobs, so fixed control and cache bytes dominate. In transformer LoRA, a rank-one atom has d_in + d_out entries per adapted layer, while the added state is a few scalars per zero episode. Our break-even rule (Sec. 6) says release lowers total cost if and only if a_mem * h * ΔB > b_time * ΔT, i.e. when saved bytes times residence time exceed the extra certification time. We therefore expect gains with large atoms, many released atoms and long pause residence. We have not measured this regime, and we will state it as an open question, not as a result. An A800 three-layer microbenchmark in our artifact shows the same pattern: persistent bytes fall (19,072 to 16,692), peak GPU memory is unchanged, decisions match, and naive deletion mismatches 12/12 and 10/12 trials. W2/Q2. Scalar summary and coordinate-adaptive optimizers (AdamW). We agree that the one-scalar result relies on all released coordinates sharing one scalar map. Our general result is that summary size equals the number of distinguishable hidden dynamics, not the number of coordinates: linear channels require exactly a Gram matrix (Thm. 2, necessary and sufficient), and K distinct spectral modes require K energies (App. D). AdamW fits this picture once we use one key fact: after permission, a released coordinate has zero task gradient. With decoupled weight decay λ and g_i = 0: m_i+ = β1 * m_i, s_i+ = β2 * s_i, d_i+ = (1 - ηλ) * d_i - η * c_t * m_i+ / (sqrt(s_i+) + ε), where c_t is the common bias correction.
  • If ε = 0, r_i = m_i / sqrt(s_i) is multiplied by the common factor β1 / sqrt(β2), so Z = [d_D, r_D] evolves as Z+ = Z * A_t. The 2x2 Gram matrix (3 scalars) is then exactly closed, as for heavy ball.
  • If ε > 0, coordinates with different s_i follow different maps. They must be grouped by second-moment value, with one Gram matrix per group. In the worst case the grouping is per coordinate and release brings no compression. This is a real limit of exact decision preservation, not of the certificate.
  • Adam with coupled L2 is not closed, because s+ contains g_i^2 = μ^2 * d_i^2. Moreover, plain AdamW has no line search or global projection. If no global norm control reads the parameters, released coordinates are never read by a retained update and the summary is empty; the remaining requirement is permission. Our CE certificate uses the radial contraction of the reference learner, whereas for momentum-type dynamics we already give a separate global-ball certificate (App. D) that holds for any dynamics staying inside a norm ball. Standard LoRA additionally needs a mechanism that produces exact zeros, such as SoRA-style proximal gates. We will add this derivation as an appendix and label AdamW clearly as a derived extension that we have not yet evaluated. We hope this clarifies the scope: exact continuation with a smaller persistent state, together with a precise statement of what other optimizers must retain.
Reviewer_2SQSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to characterize when information can be dropped from states for unfinished adapter learning trajectories. The proposed certification method aims to keep the learner's set of candidates, acceptance and rollback, including effects after directional inactivity, while releasing unneeded information. The authors tested the method on CIFAR and Additional Letter benchmarks.

Critical strengths

This is an interesting area of research; a better understanding of what can and cannot be released from state during training runs is a promising area. This paper is aimed at LoRA style adapters, but ideas may be extendable to other sparse training.

Critical weaknesses
  1. This paper is one of the most difficult to read that I have experienced in a while. Please define terms before you use them (eg CE, permission, etc), keep terminology constant, and give a roadmap of where you are going (intro = overall roadmap, within each section, give a section roadmap). A good paper has a nice takeaway, like method X solves problem Y, which is shown by evidence Z. This is not particularly clear in this paper.
  2. The method needs more empirical support. I would like to see better characterization of benchmarks (I think that the authors are using CIFAR-10, but that is unclear), and interactions with additional models (please define base LLM) and adapters. Currently, it is really hard for me to determine what this paper is trying to show with its existing numerical work.
  3. I do not think that I could implement Algorithm 1 as most of the pseudocode lines are unclear and do not reference previously defined terms (eg "outward margin" in "Evaluate a fresh outward margin" literally only appears in Algorithm 1)
Questions for authors
  1. Could you give me a one sentence summary on what problem this method solves and what evidence you have that it is better than existing tools.
1 comment
AuthorsOpenReview ↗

One-sentence problem statement, evidence over existing tools, and concrete fixes for terminology, benchmarks and Algorithm 1

We thank the reviewer for the candid feedback. We accept that the presentation failed to communicate the contribution, and we treat this as the main issue to fix. We answer the question first, then the three weaknesses. Q1. One-sentence summary and evidence. When a gate of a gated LoRA-style adapter reaches exactly zero in the middle of training, we certify that it will stay zero for the rest of the run and replace the deleted rank-one direction by one stored scalar, so that the resumed run makes exactly the same optimizer decisions as the full run. Naive deletion does not achieve this (it flips 42 of 1,864 accept/reject decisions), whereas our compact state matches all 48,539 reference decisions across 12 jobs, including 20,297 trials that depend on released energy. Why existing tools do not solve this: SoRA/AdaLoRA remove zero-rank blocks for inference or reallocate rank; safe screening certifies exclusion from the final optimum; checkpoint compression (e.g., ExCP) is lossy. None of them guarantees that the remaining training trajectory (accept/reject, backtracking, rollback, stopping) is unchanged. Against the natural alternative certificate (a geometric/simplex worst-case bound), at the same 20 recorded release points our cross-entropy (CE) bound authorizes release at 18 points where the simplex bound cannot. Over the run it releases 55% (Letter) and 39% (CIFAR) more direction storage, and caching reduces bound evaluations from 430 to 14 (Letter) and from 1,637 to 38 (CIFAR). W1. Terminology, roadmap, takeaway. We agree. In the revision we will:

  1. Add a glossary and define every term at first use, e.g. CE = cross-entropy training loss; atom = the j-th rank-one component (b_j, a_j) with gate γ_j in [0, 1]; permission = a certificate that a currently zero gate remains exactly zero for the entire remaining run; release = physically deleting b_j and a_j from the training state; summary = the scalar v = ||d_D||^2 that replaces them.
  2. Use one term per concept (no synonyms such as "licensing" or "authorization" for permission).
  3. End the introduction with an explicit roadmap and open each section with a two-line roadmap.
  4. State the takeaway up front. Problem: deleting inactive adapter directions mid-training can silently change later optimizer decisions. Method: a future-inactivity certificate plus a one-scalar summary. Evidence: exact decision agreement, 18/20 additional authorizations over the simplex bound, and more than 96% fewer bound evaluations through caching. W2. Benchmarks and models. We apologize for the ambiguity. There is no LLM in our experiments. "Base" denotes the frozen base weight W_0 of a linear classifier on fixed features, and the adapter is W = W_0 + s * B * Diag(γ) * A with rank 8. The natural benchmarks are:
  • CIFAR-10, class-incremental from 5 to 10 classes on frozen DINOv2 features; three 400/100 fit/validation resplits of a 500-image pool.
  • UCI Letter Recognition, 13 to 26 classes with replay, 153 quadratic features; three 1,300/520 splits, plus an additional disjoint Letter cohort. Each split runs with two gate kernels (12 jobs, 1,200 sweeps each). Synthetic controls vary the number of hidden modes and the coordinates per mode (18 to 1,800). We will summarize this in a table at the start of Section 5. We agree that the scale is small. These experiments test an exactness claim (does every decision match, and which retained information is necessary), so we report decision counts and dimension tests rather than accuracy gains. Extending the evaluation to transformer LoRA is the main open step, and we will state this explicitly as a limitation. W3. Algorithm 1. We agree that it cannot be implemented as written. We will rewrite it so that every line refers to a defined quantity and equation. For example, "Evaluate a fresh outward margin" becomes: compute, with outward-rounded interval arithmetic, a guaranteed lower bound m_j of w_j - C_θ(a_j, E) (Eq. 17); release atom j if and only if m_j > 0. A "floating hint" will be defined as the same margin computed in ordinary floating point, used only to decide whether the certified check is worth running. We will also add a worked end-to-end example (one atom's gate hits zero, the bound is computed, the direction is deleted, its energy is merged into v, and v is then updated by v+ = ψ^2 * v) and link each line to the released code. We hope these clarifications make the problem and the evidence clear, and we would be grateful if the reviewer could reconsider the assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.