- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In this paper, the theoretical analysis examines how long the recovery effect persists when model parameters are edited only once and training is then resumed using standard SGD. Focusing on a ridge-regularized two-layer linear network, this study shows that, even with the same editing budget, recovery is maintained for 840 steps at checkpoints in the early stages of training, whereas at checkpoints where training has progressed sufficiently, the probability of recovery being maintained becomes extremely low.
Critical strengths
- Theoretical rigor.: The discussion of impossibility focuses on editing strategies that satisfy certain constraints, rather than on specific editing algorithms.
- Clarity of problem formulation.: The paper distinguishes between success immediately after editing and sustained success following continuous learning.
- Connection between mathematical proofs and experiments.: The relationship among finite-batch SGD, Ridge regularization, and parameter-shifting constraints is analyzed from both theoretical and simulation perspectives.
Critical weaknesses
-
The effect of training age is not independently isolated.: It remains unclear whether the observed difference in optimal survival capability is attributable to training age itself or simply to differences in checkpoint states. Although the authors explicitly acknowledge this limitation, it may restrict the interpretation of the results.
-
The experimental validation is limited.: While the experiments with the two-layer linear network provide convincing results, the findings are largely limited to a theoretically constructed model. Moreover, the experiments with the tanh network do not validate the paper's central claim that durable repair can be difficult even when optimizing over the complete legal action class, rather than evaluating a single editing policy. It would be helpful to readers if these limitations were more explicitly discussed in the main text.
-
The impossibility result depends on the choice of editing budget.: The impossibility result critically depends on a stored-coordinate editing budget, which is not invariant under function-preserving reparameterizations. Although this restriction is explicitly acknowledged, it remains unclear whether the resulting separation reflects a fundamental limitation of durable model repair or a property of the chosen parameterization and intervention metric.
Questions for authors
See Weaknesses.