- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper considers the setting where samples from the training set are deleted and a new parameter is trained. Under this setting, the paper asks the question of what's the rank of the difference between the previous parameter and the new parameter. The paper assumes Gaussian data in a bilinear model, considers a regularized MSE, and derives asymptotic bound when . The paper accompanies the result with experiments on Gaussian data and some transformer settings.
Critical strengths
- It seems that the problem considered by the paper is fairly new.
- Based on the theoretical result, the author provided a prescription to choose the rank.
- The paper has results on transformer setup.
Critical weaknesses
- The theoretical setup is fairly simple. It assumes Gaussian inputs, linear model, and response with roughly no structure that depend on the input.
- Bounds are derived in asymptotic style, leaving out a functional form that is hard to interpret.
- From a significance perspective, the problem of "how to correct the parameter under deletion" would be more complicated than an expressivity perspective. Current analysis only reviews a bound that says "larger rank recovers better". However, in practice more questions such as optimization difficulty should be considered. Therefore, I think the prescription provided has limited significance, especially given that there is no experimental results supporting the effectiveness of the prescription.
- Concerns about the paper's soundness is present. In Section 5. the definitions of and are unclear, with the only info "We now compare the correction’s component along U with its total squared norm. Their limits will be denoted by and ." This is insufficient since it is unclear what "their" is referring to. Given this ambiguous definition, the paper immediately claimed "These quantities are expressed through the mean-one Marchenko–Pastur law" with no justification. The same issue lies in the rest of Section 5 from Eq. (9) to Eq. (12). In Section 5.1, the paper claims The theorem below describes the normalized squared singular values of the remaining correction block through the aspect-one Marchenko–Pastur law ", but no theorem is given in Section 5.1 and is never defined explicitly. The following two sentences "In contrast, describes eigenvalues of the feature covariance. The law has support [0,4]." are also unclear and not justified sufficiently. There are more locations in the paper that has similar issues. I urge the authors to double check that claims are supported with sufficient evidence. Without a satisfying response addressing all the soundness issues (not limited to the examples I pointed out above), I will likely not raise my score.
- It seems unclear how the derived result predicts the transformer's experiment.
- Writings are unclear in many locations, for instance: a). and need to be explicitly defined. Right now they are only associated with ambiguous notions of "full and retained objectives". b). Statement of Proposition 1, "the right-hand side is ...". It is unclear what is "the right-hand side" from the equation above, which contains two eqaulties. c). Eq. (5), the notations is not defined.
- I also suggest the author to explicitly say which theorem is proved in which appendix section.
Questions for authors
Section 4 relies heavily on the separability of the Hessian. What does it mean for the objective considered?
From a high-level perspective, how does the Hessian-based analysis help the derived asymptotic bound?
2 comments
I have read the rebuttal. However, I don't think the rebuttal clearly respond to the weaknesses pointed out in my review. ICLR allows an updated version of the paper to be submitted, but I don't see revisions of the paper that addresses my concern. Therefore, I choose to maintain the score I give.
Response to Reviewer zrhX
W1–W2. What assumptions does the theorem use, and what does it give for the error at each rank?
Assumption 1 uses pairs of independent standard Gaussian feature vectors and a deletion chosen independently of the sample. It also requires d2/n to tend to a finite positive constant, so the number of examples and the number of parameters grow in proportion. With positive regularization held fixed, Theorem 2 gives the limiting error curve χτ(r/d) uniformly over all positive ranks. Equations (13)–(14) give τ explicitly. For example, when d2/n tends to 1 and regularization is 0.5, any positive rank that grows more slowly than d leaves 25% of the squared prediction change in the limit. We can then use the same curve to choose a rank for a smaller error. For a target of 5%, the required rank is about 39% of full rank.
The theorem allows responses that depend on the features. Indeed, for a single deletion, the response enters the correction through a scalar residual, which cancels from the relative error whenever the correction is nonzero.
W3. Is there experimental evidence for rank selection?
Section 8.1 and Appendix K.4 compare the rank suggested by the limiting formula with the smallest rank that meets the tolerance. The rank suggested by the formula missed 22 of 1,350 targets, while the numerical bounds identified the smallest sufficient rank for every target. Thus, the formula supplies a starting rank and the bounds can then check it for a particular fit.
W4. What do m, m2 and the two spectral distributions describe?
Let M be the result of applying the inverse Hessian to the unit gradient direction U. The quantity m is the limit of ⟨M,U⟩, the component along U. The quantity m2 is the limit of ∥MF∥2, the squared size of the whole matrix. Appendix G.1 derives both limits from the covariance eigenvalue distribution νγ and concentration in the independent direction U (Eqs. (G.5)–(G.6)).
Together, these limits give m2−m2 as the limiting squared size of the component orthogonal to U. To obtain the error curve, we then need to know how that size is spread across singular values. Appendix G shows that the normalized squared singular values of the remaining block follow μ1 in the limit, with density (4−t)/t/(2π) on (0,4). Hence, we integrate t against μ1 below the cutoff to find the fraction of the block's squared mass left at a given rank.
W6–W7. Can we clarify the notation and give more direct references to the proofs?
We will define WD and WR as the minimizers of Eq. (2) over the full and retained index sets, respectively. We will also write out the grouped-deletion form of Eq. (4) and use rank(B) in Eq. (5). We will add the density next to Eq. (13) and point from Theorem 2 directly to its proof in Appendix G (Theorem S3).
Q1–Q2. What does Hessian separability mean here, and how does the Hessian enter the proof?
Separability means that the full retained Hessian, including regularization, acts by multiplying the update on the left and right. If HR[A]=PAQ, then its inverse sends xz⊤ to P−1xz⊤Q−1, which still has rank one. In the Gaussian model, however, the Hessian can mix these directions. The proof therefore studies what happens when its inverse acts on the deleted example's gradient. This gives the part of the correction that a low-rank update misses.
W5. How does the theorem relate to the experiments with transformer features?
In the text experiments, we keep the transformer features fixed and refit the head after deletion. In §N.6, we compare Frobenius errors with the Gaussian curve to see how the weight approximation changes with rank on these features. The formula itself applies under Assumption 1.
To test whether a similar effect appears when the features can change, we also trained a small nonlinear network on handwritten digits. We then refit all weights after each deletion, starting from the original model. Across four settings with 40 random deletions each, the mean error left by the best rank-one update ranged from 32% to 50% of the squared change in the hidden-layer weights. At rank eight, it fell to between 2% and 8%. The comparison is therefore with the weight change from a local refit.
We hope these clarifications address your concerns about the theorem's regime, the explicit error formula, and the evidence for rank selection. In light of the results already in the submitted paper, we ask you to reconsider your score of 1.