W1. The theoretical novelty relative to established kernel methods is insufficiently clarified.
The central contribution relies on expressing parameter updates in the span of training gradients and replacing a parameter-space linear system with a sample-space formulation. These ideas are closely related to established results, including the representer theorem, the Woodbury identity, and kernel-based influence estimation. Although the reduced system and the treatment of the forget-set components are potentially useful, the paper does not clearly distinguish which aspects constitute new theoretical contributions beyond these established relationships. Furthermore, the related work does not sufficiently discuss closely connected research on NTK-based unlearning, gradient-kernel attribution, and efficient influence estimation. A clearer comparison with existing mathematical formulations and relevant methods is needed to establish the novelty and significance of the proposed approach.
W2. The computational comparison excludes the cost of constructing the NTK matrix.
Section 4.1 assumes that the empirical NTK matrix is precomputed and stored. However, constructing this matrix requires computing per-example parameter gradients, and its storage cost scales quadratically with the product of the dataset size and output dimension. Consequently, the runtime improvements reported in Figure 1 primarily demonstrate the efficiency of solving the reduced linear system rather than the end-to-end computational advantage of the proposed method. This distinction is particularly important because the paper is motivated by improving the scalability of influence functions for large models. In such settings, NTK construction may represent a substantial computational bottleneck. Although the authors discuss amortizing the NTK construction cost across multiple influence queries, this benefit is not quantitatively evaluated. The paper should report NTK construction time, peak memory consumption, and end-to-end runtime, including the number of influence queries required to amortize the preprocessing cost.
W3. The absence of exact dual-space retraining leaves the practical advantage of influence approximation unclear.
Once the NTK matrix is available, retraining the linearized model on the retained dataset can also be formulated directly in the dual space. For squared-error objectives, this corresponds to solving a kernel-based quadratic optimization problem, while other convex losses can be optimized in the same sample-dependent space. Therefore, the proposed influence approximation should be compared with exact retraining in the dual space, using the same precomputed kernel. This comparison is especially important because the proposed method aims to approximate the effect of data removal efficiently. Without evaluating the runtime and accuracy relative to direct dual-space retraining, it remains unclear whether influence approximation provides a meaningful practical advantage once the NTK has already been constructed. A comparison between dual-space influence estimation and exact dual-space retraining would substantially strengthen the motivation for the proposed method.
W4. Experimental validation is limited, and the applicability to nonlinear models remains uncertain.
The experiments mainly involve relatively small linearized networks trained on subsets of MNIST and CIFAR-10. These controlled settings are appropriate for verifying the theoretical formulation but do not sufficiently support the broader motivation of scalable influence estimation for large neural networks. In particular, the paper does not evaluate pretrained models or realistic fine-tuning tasks, despite identifying such settings as promising applications. Furthermore, Section 4.3 and Figure 2 reveal an important trade-off: stronger regularization improves agreement between nonlinear models and their linearizations, but also reduces predictive accuracy. This raises concerns about whether the proposed method remains reliable in practically useful nonlinear regimes, where models may move substantially away from their initialization.
Figure 3(a) also shows a substantial difference between parameter-space and dual-space influence estimates under weak regularization. This observation deserves further analysis, particularly regarding the different curvature approximations used by the two methods. Experiments on realistic pretrained models, together with a more systematic analysis of regularization, predictive performance, and influence estimation error, would strengthen the practical relevance of the paper.
Weakness 1
Our proof of Theorem 1 does not involve the representer theorem. However, representer theorem can be used to prove the Theorem 1 after reparameterizing the empirical risk as described in Section 3.4.
We skimed through the three papers you mentioned. The first two papers you mentioned seems to be studies of alternatives to influence functions for data attribution. The first paper does data attribution by using the representer theorem, while the second uses model linearization, random projection, and one-step Newton approximation to estimate attribution scores. Neither paper focuses on influence functions themselves. Our work, on the other hand, is a study of the dual representation of influence functions and the conditions under which such representation exists. For the third paper, if we understand it correctly, their method first trains a nonlinear model on data points that will not require forgetting, then trains a linearized model on forgettable data points using regularized squared error loss. The unlearning is focused on the linearized model, and is done by using influence function. However, instead of evaluting the inverse Hessain vector product (IHVP) required by the influence function, the IHVP is approximated by solving a quadratic optimization problem using SGD and Jacobian vector product. This is again different from our study of dual representation of influence functions. As a result, we don't think these papers study the same dual space formulation and equivalence results presented in our work. However, we may have overlooked some details, so please let us know specifically which aspects of our work you believe have already been well-studied in these papers.
Weakness 2
We did not measure the number of requests needed to amortize the cost of forming $ \mathbf{K} $, as we think this analysis depends on too many factors and could be a separate study by itself. For the empirical NTK, construction of $ \mathbf{K} $ requires computing
∇θf(x(i),θ′)⊤∇θf(x(i),θ′)
for all data points $ \mathbf{x}^{(i)} $, the break-even number thus depends on the model, dataset, GPU hardware, number of GPUs, and the method used to parallelize kernel construction and influence function evaluation. For the analytical NTKs derived from infinitely wide neurual networks, entries of $ \mathbf{K} $ can be evaluated analytically without constructing gradients or Jacobians. In that setting, even an expensive construction of $ \mathbf{K} $ is worthwhile, since it allows influence function to be used for models with infinite number of parameters in $ \boldsymbol{\theta} $-space.
Weakness 3
We do compare our method with the original $ \boldsymbol{\theta} $-space representation of influence functions by Koh and Liang as shown in Figure 1. Our claim concerns the equivalence and computational efficiency of the dual representation relative to this original representation, which is why the $ \boldsymbol{\theta} $-space representation is a directly relevant baseline. Comparing the two representations under matched settings directly supports our claim.