acceptodds
Under review as a conference paper at ICLR 2027

Durable Repair under Stored-Coordinate Budgets: A Ridge-Trained Checkpoint Separation

Abstract

We study one-shot factor edits followed by compulsory ridge-regularized minibatch SGD. Capability is the best probability of keeping both task risks safe throughout the continuation window, optimized over the complete legal action class. In a rank- linear learner at batch sizes near , the same stored-coordinate row-write budget gives certain -step survival at a young checkpoint but at most survival after old training, on a history event of probability at least . A task subset of prior mass is certified repairable at editing time; its contribution to the young-minus-old capability gap exceeds . More mobility restores reliable survival. The separation holds on an explicit joint neighborhood in cap, ridge, step size, and risk threshold, including caps on both sides of the exact-representation energy. This neighborhood is small and establishes local openness. The proof combines a full-factor lower bound, feasible witnesses, and stopped finite-batch estimates, with executable arithmetic certificates. Seven-history policy experiments support the ridge mechanism; nonlinear age effects are mixed. The result separates age-generated checkpoint states, without matching their predictors or risks or establishing a representation-invariant limitation.

Primary area:
learning theory
Keywords:
learning theory, continual learning, model editing, stochastic gradient descent, linear neural networks, ridge regularization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WaNvOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the theoretical analysis examines how long the recovery effect persists when model parameters are edited only once and training is then resumed using standard SGD. Focusing on a ridge-regularized two-layer linear network, this study shows that, even with the same editing budget, recovery is maintained for 840 steps at checkpoints in the early stages of training, whereas at checkpoints where training has progressed sufficiently, the probability of recovery being maintained becomes extremely low.

Critical strengths
  • Theoretical rigor.: The discussion of impossibility focuses on editing strategies that satisfy certain constraints, rather than on specific editing algorithms.
  • Clarity of problem formulation.: The paper distinguishes between success immediately after editing and sustained success following continuous learning.
  • Connection between mathematical proofs and experiments.: The relationship among finite-batch SGD, Ridge regularization, and parameter-shifting constraints is analyzed from both theoretical and simulation perspectives.
Critical weaknesses
  • The effect of training age is not independently isolated.: It remains unclear whether the observed difference in optimal survival capability is attributable to training age itself or simply to differences in checkpoint states. Although the authors explicitly acknowledge this limitation, it may restrict the interpretation of the results.

  • The experimental validation is limited.: While the experiments with the two-layer linear network provide convincing results, the findings are largely limited to a theoretically constructed model. Moreover, the experiments with the tanh network do not validate the paper's central claim that durable repair can be difficult even when optimizing over the complete legal action class, rather than evaluating a single editing policy. It would be helpful to readers if these limitations were more explicitly discussed in the main text.

  • The impossibility result depends on the choice of editing budget.: The impossibility result critically depends on a stored-coordinate editing budget, which is not invariant under function-preserving reparameterizations. Although this restriction is explicitly acknowledged, it remains unclear whether the resulting separation reflects a fundamental limitation of durable model repair or a property of the chosen parameterization and intervention metric.

Questions for authors

See Weaknesses.

Reviewer_HJY1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the durability of a one-shot, budget-constrained parameter edit under subsequent training. It asks whether an edit that initially brings both task risks below their thresholds can remain successful after training resumes, and whether checkpoints obtained at different training ages have different optimal durable-repair capabilities under the same editing budget. In a ridge-regularized two-layer linear model, the authors establish a separation that applies to the complete admissible edit class.

Critical strengths

The paper clearly distinguishes immediate repairability, the durability of a particular repair policy, and the best durability achievable over all admissible edits. The controlled experiments provide useful evidence for the proposed mechanism, particularly through paired interventions on the future ridge coefficient. The presentation also distinguishes the scope of the theoretical results from that of the policy-level experiments and explicitly discusses the limitations of the nonlinear evidence.

Critical weaknesses
  1. The intended application scenario is unclear, and the paper does not establish how well its approach generalizes, for example, to larger models.
  2. If the goal of editing is to make a model perform well on both old and new tasks, what advantages does it offer over training a model from scratch on data from both tasks? Alternatively, what advantages does it offer over fine-tuning the model on the new task?
  3. If the experimental model is very small, why is it necessary to study whether its capabilities are preserved during continued training, rather than applying another edit after training is complete? Retraining a small model is also relatively inexpensive. If the model is large, is the approach presented in this paper still applicable, and what advantages would it offer?
Questions for authors

Same as critical weaknesses.

Reviewer_kSHpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the durability of one-shot model repairs under constrained parameter updates and subsequent stochastic gradient descent (SGD). The authors formulate an optimal survival capability that considers all admissible repair strategies and theoretically demonstrate that older checkpoints can have significantly lower repair durability than younger checkpoints under the same editing budget. The theoretical findings are supported by numerical experiments on two-layer linear networks, with additional experiments on a small nonlinear model.

Critical strengths
  1. The paper investigates an important problem related to model editing and continual learning, particularly whether parameter modifications can remain effective during subsequent training.

  2. The paper provides detailed theoretical results, including bounds on optimal survival probabilities and robustness guarantees under small parameter perturbations.

Critical weaknesses
  1. The main theoretical results rely on a highly specific two-layer linear network with carefully chosen parameter constraints and ridge regularization. It remains unclear whether the conclusions extend to more general neural networks or practical model editing scenarios.

  2. Most experiments are conducted on simplified linear models. Although a small nonlinear experiment is included, its results do not consistently support the claimed deterioration with training age. Experiments on more realistic neural networks would strengthen the paper.

  3. Figures 1 and 2 are overly stylized and difficult to follow. Figure 1 does not clearly illustrate the underlying mechanism, while Figure 2 presents multiple experimental factors in a complicated layout. Simplifying the figures and improving their readability would make the paper more accessible.

  4. The young and old checkpoints are not matched in terms of their predictors or risks. Therefore, it is difficult to determine whether the observed differences arise specifically from training age or simply from differences in the underlying checkpoint states.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.