acceptodds
Under review as a conference paper at ICLR 2027

A Dual Representation of Influence Functions for Linearizable Models

Abstract

In this paper, we present a dual representation of influence functions, whose computational complexity scales with dataset size rather than model size. Both analytically and experimentally, we show that this representation can be an efficient alternative to the original influence functions for estimating changes in parameters, model outputs and loss due to data point removal, when model size is large relative to dataset size, or when evaluating the original influence functions in parameter space is infeasible. The dual representation, however, is limited to linearizable models, which are models whose behavior can be approximated by their linearizations throughout training, and requires materializing a matrix, whose size grows with the product of model output dimension and dataset size.

Primary area:
interpretability and explainable AI
Keywords:
Influence functions, neural tangent kernel, dual representation, kernel trick

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_njQ4OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper looks at how to make influence functions cheaper when the model is much larger than the training set. The main idea is to work in a "dual" space instead of the parameter space. For linearized models, the solution can be written as a combination of the training points' gradient features, so the linear system grows with dataset size instead of model size. The part of the solution that corresponds to the removed points is known in closed form, so only the retained points need to be solved for. Using the analytical NTK, the method can also estimate how removing data changes the outputs and loss of infinitely wide networks. In data-removal experiments on small subsets of MNIST and CIFAR-10, the method matches the standard influence function and retraining on linearized models while running faster. For nonlinear networks, both methods only work well when a large λ keeps the network close to its linearization.

Critical strengths

Strengths

  1. The dual formulation is clearly presented, and the derivations are easy to follow.
  2. The reduced system makes the cost independent of model size, and the cost shrinks as more data is removed.
Critical weaknesses
  1. limited novelty The core theorem (3.1) seems to be the representer theorem applied to linear networks. Also, the influence functions in the resulting kernel form is also well-studied [1,2,3]. This should be clearly discussed in related work or Method.

  2. The computational efficiency is not clear In section 4.1, the K is precomputed and cached. But the computing time of K is not reported in the paper. Therefore, it's unclear how many calls are needed to amortize this one-time cost

  3. Lack of baselines The paper does not compare with popular or related influence functions.

[1] Representer Point Selection for Explaining Deep Neural Networks [2] TRAK: Attributing Model Behavior at Scale [3] Mixed-Privacy Forgetting in Deep Networks

Questions for authors

See weaknesses.

2 comments

Weakness 1

Our proof of Theorem 1 does not involve the representer theorem. However, representer theorem can be used to prove the Theorem 1 after reparameterizing the empirical risk as described in Section 3.4.

We skimed through the three papers you mentioned. The first two papers you mentioned seems to be studies of alternatives to influence functions for data attribution. The first paper does data attribution by using the representer theorem, while the second uses model linearization, random projection, and one-step Newton approximation to estimate attribution scores. Neither paper focuses on influence functions themselves. Our work, on the other hand, is a study of the dual representation of influence functions and the conditions under which such representation exists. For the third paper, if we understand it correctly, their method first trains a nonlinear model on data points that will not require forgetting, then trains a linearized model on forgettable data points using regularized squared error loss. The unlearning is focused on the linearized model, and is done by using influence function. However, instead of evaluting the inverse Hessain vector product (IHVP) required by the influence function, the IHVP is approximated by solving a quadratic optimization problem using SGD and Jacobian vector product. This is again different from our study of dual representation of influence functions. As a result, we don't think these papers study the same dual space formulation and equivalence results presented in our work. However, we may have overlooked some details, so please let us know specifically which aspects of our work you believe have already been well-studied in these papers.

Weakness 2

We did not measure the number of requests needed to amortize the cost of forming $ \mathbf{K} $, as we think this analysis depends on too many factors and could be a separate study by itself. For the empirical NTK, construction of $ \mathbf{K} $ requires computing

∇θf(x(i),θ′)⊤∇θf(x(i),θ′)\nabla_{\boldsymbol{\theta}} f(\mathbf{x}^{(i)}, \boldsymbol{\theta}')^\top \nabla_{\boldsymbol{\theta}} f(\mathbf{x}^{(i)}, \boldsymbol{\theta}')

for all data points $ \mathbf{x}^{(i)} $, the break-even number thus depends on the model, dataset, GPU hardware, number of GPUs, and the method used to parallelize kernel construction and influence function evaluation. For the analytical NTKs derived from infinitely wide neurual networks, entries of $ \mathbf{K} $ can be evaluated analytically without constructing gradients or Jacobians. In that setting, even an expensive construction of $ \mathbf{K} $ is worthwhile, since it allows influence function to be used for models with infinite number of parameters in $ \boldsymbol{\theta} $-space.

Weakness 3

We do compare our method with the original $ \boldsymbol{\theta} $-space representation of influence functions by Koh and Liang as shown in Figure 1. Our claim concerns the equivalence and computational efficiency of the dual representation relative to this original representation, which is why the $ \boldsymbol{\theta} $-space representation is a directly relevant baseline. Comparing the two representations under matched settings directly supports our claim.

Thank you for your response. I will maintain my score.

Reviewer_1vK2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dual representation of influence functions for linearizable models, aiming to reduce the computational cost of estimating the effects of training data removal. Instead of computing inverse Hessian-vector products (IHVPs) in the parameter space, the authors reformulate influence estimation in a sample-dependent dual space using the empirical Neural Tangent Kernel (NTK). The authors establish a correspondence between the original and dual formulations under convexity and rank assumptions. They further derive a reduced linear system whose dimension scales with the number of retained training examples and output dimensions, rather than the number of model parameters. An additional property is that the reduced system becomes smaller as more training examples are removed. The formulation also extends to infinitely wide neural networks through analytical NTKs. Experiments on linearized fully connected and convolutional networks, nonlinear networks under different regularization strengths, and infinitely wide networks demonstrate computational advantages and comparable influence estimation accuracy in the examined settings.

Overall, the paper presents an interesting theoretical perspective on influence functions. However, the novelty relative to established kernel formulations, the end-to-end computational benefits, and the practical advantages over exact dual-space retraining remain insufficiently established.

Critical strengths

S1. A theoretically motivated alternative to parameter-space influence estimation.

The paper provides a clear mathematical formulation that connects influence functions with kernel-based representations. The theoretical derivations establish a relationship between parameter-space and dual-space influence estimation under explicit assumptions.

S2. An efficient reduced linear system.

The reduced system in Eq. (13) has a dimension proportional to the number of retained examples rather than the number of model parameters. The observation that its dimension decreases as more examples are removed is an appealing structural property.

S3. Extension to infinitely wide neural networks.

The kernel formulation naturally extends influence estimation to infinitely wide neural networks when analytical NTKs are available, offering a potentially useful perspective for theoretical analysis.

Critical weaknesses

W1. The theoretical novelty relative to established kernel methods is insufficiently clarified.

The central contribution relies on expressing parameter updates in the span of training gradients and replacing a parameter-space linear system with a sample-space formulation. These ideas are closely related to established results, including the representer theorem, the Woodbury identity, and kernel-based influence estimation. Although the reduced system and the treatment of the forget-set components are potentially useful, the paper does not clearly distinguish which aspects constitute new theoretical contributions beyond these established relationships. Furthermore, the related work does not sufficiently discuss closely connected research on NTK-based unlearning, gradient-kernel attribution, and efficient influence estimation. A clearer comparison with existing mathematical formulations and relevant methods is needed to establish the novelty and significance of the proposed approach.

W2. The computational comparison excludes the cost of constructing the NTK matrix.

Section 4.1 assumes that the empirical NTK matrix is precomputed and stored. However, constructing this matrix requires computing per-example parameter gradients, and its storage cost scales quadratically with the product of the dataset size and output dimension. Consequently, the runtime improvements reported in Figure 1 primarily demonstrate the efficiency of solving the reduced linear system rather than the end-to-end computational advantage of the proposed method. This distinction is particularly important because the paper is motivated by improving the scalability of influence functions for large models. In such settings, NTK construction may represent a substantial computational bottleneck. Although the authors discuss amortizing the NTK construction cost across multiple influence queries, this benefit is not quantitatively evaluated. The paper should report NTK construction time, peak memory consumption, and end-to-end runtime, including the number of influence queries required to amortize the preprocessing cost.

W3. The absence of exact dual-space retraining leaves the practical advantage of influence approximation unclear.

Once the NTK matrix is available, retraining the linearized model on the retained dataset can also be formulated directly in the dual space. For squared-error objectives, this corresponds to solving a kernel-based quadratic optimization problem, while other convex losses can be optimized in the same sample-dependent space. Therefore, the proposed influence approximation should be compared with exact retraining in the dual space, using the same precomputed kernel. This comparison is especially important because the proposed method aims to approximate the effect of data removal efficiently. Without evaluating the runtime and accuracy relative to direct dual-space retraining, it remains unclear whether influence approximation provides a meaningful practical advantage once the NTK has already been constructed. A comparison between dual-space influence estimation and exact dual-space retraining would substantially strengthen the motivation for the proposed method.

W4. Experimental validation is limited, and the applicability to nonlinear models remains uncertain.

The experiments mainly involve relatively small linearized networks trained on subsets of MNIST and CIFAR-10. These controlled settings are appropriate for verifying the theoretical formulation but do not sufficiently support the broader motivation of scalable influence estimation for large neural networks. In particular, the paper does not evaluate pretrained models or realistic fine-tuning tasks, despite identifying such settings as promising applications. Furthermore, Section 4.3 and Figure 2 reveal an important trade-off: stronger regularization improves agreement between nonlinear models and their linearizations, but also reduces predictive accuracy. This raises concerns about whether the proposed method remains reliable in practically useful nonlinear regimes, where models may move substantially away from their initialization.

Figure 3(a) also shows a substantial difference between parameter-space and dual-space influence estimates under weak regularization. This observation deserves further analysis, particularly regarding the different curvature approximations used by the two methods. Experiments on realistic pretrained models, together with a more systematic analysis of regularization, predictive performance, and influence estimation error, would strengthen the practical relevance of the paper.

Questions for authors

Q1. Theoretical novelty.

Could the authors clarify the main theoretical contribution beyond established results such as the representer theorem, the Woodbury identity, and existing kernel-based influence formulations? In particular, which aspects of the reduced system or forget-set treatment are genuinely new?

Q2. End-to-end computational efficiency.

Does the cold-start runtime in Figure 1 include NTK construction? Could the authors report the preprocessing time, memory cost, and end-to-end runtime across different model and dataset sizes, as well as the number of deletion queries required to amortize NTK construction?

Q3. Exact dual retraining baseline.

How does the proposed dual-space influence method compare with directly retraining the linearized model in dual space using the same precomputed NTK? What are the respective runtime and approximation accuracy, particularly for squared-error objectives?

Q4. Applicability to nonlinear models.

Given the trade-off between linearizability and predictive accuracy shown in Figure 2, can the authors demonstrate that the proposed method remains effective for pretrained or fine-tuned nonlinear models without relying on strong regularization? How do the authors explain the substantial difference between the two influence representations under weak regularization in Figure 3(a)?

2 comments

Question 1

Our proof of Theorem 1 does not involve the representer theorem. However, representer theorem can be used to prove the Theorem 1 after reparameterizing the empirical risk as described in Section 3.4.

We do not have any results that involve the use of the Woodbury identity, could you clarify which result you believe follows from this identity?

To the best of our knowledge, the work most closely related to our kernel based influence function formulation is “Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations” by Golatkar et al. In this work, they use tools from NTK theory to estimate the retrained model paraemters $ \boldsymbol{\theta}_r $ from the original model parameters $ \boldsymbol{\theta} $ without retraining from scratch. Their result follows directly from NTK theory rather than through an influence function argument.

Our work established a relationship between their update and the (dual) influence function based update. Specifically, their update (equation (10) in their paper) can be recovered by specializing our formulation to squared error and taking the limit as $ \lambda \to 0^+ $. Don't have enough space to show the proof here, please let us know if you want to see a proof. This why our results can be considered as a genrealization of their results. We generalizes their unregularized squared error result in two ways: it accommodates loss functions beyond squared error and it incorporates regularization.

In addition, what's new is that we establish equivalence between evluating influence function in original parameter space ($ \boldsymbol{\theta} −space)andindualparameterspace(-space) and in dual parameter space ( \Delta \boldsymbol{\alpha} $-space), together with the conditions under which this equivalence holds. This equivalence leads to reduced linear system to solve and the application of kernel trick to influence function, which allows influence function to be applied to model with infinite number of parameters in $ \boldsymbol{\theta} $-space.

Question 2

No. In all experiments, the NTK matrix $ \mathbf{K} $ is precomputed and stored. For experiments that involve the MNIST subset (200 data points from each of the 10 classes), $ |\mathcal D|=2{,}000 $ and $ d_{\text{out}}=10 $. Storing the NTK matrix in single precision (float32) requires $ ( 10 \times 2{,}000)^2 \times 4=1.60 $ GB. For the binary CIFAR-10 subset (1,000 data points from each of classes 0 and 1), $ |\mathcal D|=2{,}000 $ and $ d_{\text{out}}=1 $, so storing the matrix requires $ ( 1 \times 2{,}000)^2 \times 4=16 $ MB.

Given these precomputed matrices, the reported runtimes are already end-to-end. For the dual representation, the reported runtime includes constructing the dual-space linear system, solving it using CG, and mapping the resulting parameter back to the original parameter space, i.e., $ \Delta \boldsymbol{\alpha}_r^\star \to \boldsymbol{\theta}_r^\star $, using the mapping $ \phi(\cdot) $. For the original representation, the reported runtime includes constructing and solving the linear system in $ \boldsymbol{\theta} $-space.

We did not measure the number of requests needed to amortize the cost of forming $ \mathbf{K} $, as we think this analysis depends on too many factors and could be a separate study by itself. For the empirical NTK, construction of $ \mathbf{K} $ requires computing

∇θf(x(i),θ′)⊤∇θf(x(i),θ′)\nabla_{\boldsymbol{\theta}} f(\mathbf{x}^{(i)}, \boldsymbol{\theta}')^\top \nabla_{\boldsymbol{\theta}} f(\mathbf{x}^{(i)}, \boldsymbol{\theta}')

for all data points $ \mathbf{x}^{(i)} $, the break-even number thus depends on the model, dataset, GPU hardware, number of GPUs, and the method used to parallelize kernel construction and influence function evaluation. For the analytical NTKs derived from infinitely wide neurual networks, entries of $ \mathbf{K} $ can be evaluated analytically without constructing gradients or Jacobians. In that setting, even an expensive construction of $ \mathbf{K} $ is worthwhile, since it allows influence function to be used for models with infinite number of parameters in $ \boldsymbol{\theta} $-space.

Question 3

Our Theorem 1 established training equivalence in the original parameter space ($ \boldsymbol{\theta} −space)andthedualparameterspace(-space) and the dual parameter space ( \Delta \boldsymbol{\alpha} $-space). Specifically, training in $ \boldsymbol{\theta} $-space till convergence is equivalent to training in $ \Delta \boldsymbol{\alpha} $-space till convergence in the sense that they lead to the same optimal parameters. Therefore, the $ \boldsymbol{\theta} $-space retraining baseline already reported in our experiments also serves as the $ \Delta \boldsymbol{\alpha} $-space retraining baseline.

Question 4

As studied in ''Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent" by Jaehoon, et al, sufficiently wide neural networks can exhibit training dynamics similar to that of linear models without explicit regularization. Thus, strong regularization is not the only way to satisfy our linearization assumption.

The substantial difference between the two influence representations under weak regularization is because weak regularization may violates the assumptions made by both representations of influence functions.

The dual representation of influence function is developed for linearizable models, which are models whose behavior can be approximated by their linearizations throughout training. For nonlinear models trained with weak regularization, this assumption may no longer hold, for the reason discussed in the third paragraph of Section 3.5.

The original representation of influence functions also benefits from strong regularization. This is because, with strong regularization, the nonlinear model can be accurately approximated by its linearization throughout training. For a linear model with a convex loss, the regularized empirical risk will be strongly convex in $ \boldsymbol{\theta} $. This ensures that the $ \boldsymbol{\theta} $-space Hessian is invertible, which is required by the original influence function formulation. Under weak regularization, the Hessian may become singular or ill-conditioned, making influence estimates undefined or numerically unstable. This observation is also consistent with the findings in "Influence functions in deep learning are fragile" by Basu, et al, where they report that the accuracy of influence estimates is sensitive to the strength of regularization.

Reviewer_xfKxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper develops a dual representation of an endpoint Newton approximation for training-data deletion in affine and linearized models. It represents updates using the training Jacobian Gram matrix and exploits known forget coefficients to solve only for retained output dimensions. Under smooth convex output losses, positive quadratic regularization, convergence, and a full-row-rank Jacobian, the full and reduced dual systems recover the same parameter-space Newton update without additional algebraic approximation. Experiments study runtime and deletion accuracy in linearized networks, the regularization-dependent behavior of a nonlinear FCNN, and output/loss influence with infinite-width kernels.

Critical strengths
  1. A clean and useful theoretical construction. The stationary-point representation and the reduction to retained coefficients fit together coherently. In particular, the known forget block is compatible with the full Newton system; the reduction is more than a heuristic substitution. I did not identify a substantive error in the central affine-model algebra. This offers a useful computational perspective when retained output dimension is much smaller than parameter dimension, and I consider that theoretical insight sufficient to merit serious consideration.

  2. A meaningful structural benefit for group deletion. The reduced system becomes smaller as more data are removed, and a cached kernel can support multiple deletion queries. The paper also makes its linearizability and storage limitations visible and shows the accuracy cost of stronger regularization. These observations help explain both the potential and the limits of the approach rather than presenting only favorable configurations.

Critical weaknesses
  1. The practical boundary in pretrained nonlinear models remains unclear. Section 4.3 and Figures 2/3a already show that stronger regularization improves tracking and deletion estimates at a cost in task accuracy, but this evidence is confined to a small MNIST FCNN, and maybe even worse in bigger and practical models. The concluding suggestion about pretrained-model fine-tuning is not tested. Output or first-order training tracking alone does not control Jacobian drift or the residual-curvature term in a true nonlinear Hessian. This limits the demonstrated practical reach; it does not invalidate the affine theorem.
  2. The computational and novelty claims need sharper qualification. Section 4.1 assumes a precomputed stored Gram matrix, so the cold-start results do not clearly establish savings for a request starting without a cache. Kernel construction/storage, parameter mapping, and CG accuracy should be reported. Smaller dual dimension alone does not guarantee better conditioning. In addition, a precise comparison with the regularized NTK influence analysis of Zhang and Zhang (AAAI 2022) and the kernel-based group forgetting of Golatkar et al. (ECCV 2020) would clarify the increment in general losses, vector outputs, and the reduced Newton system. I regard these as addressable limitations of the current presentation and evidence.
Questions for authors
  1. Clarification of the computational claim. Does Figure 1 cold-start include building K from scratch? Please separate construction, transfer/storage, solving, and parameter mapping, and report CG tolerance/iterations and mapped residuals at comparable accuracy. If K is amortized over requests, what is the break-even number of requests?

  2. Clarification of the theoretical increment and scope. What exactly is new relative to the two kernel-based works above, especially in their shared squared-loss setting? Please also distinguish the endpoint retain-objective Newton update used here from the classical infinitesimal influence function using the full-data Hessian. Algebraic equivalence of representations should remain separate from exact retained-data retraining.

  3. Suggested bounded validation of the nonlinear extension. Could the authors add one pretrained Transformer fine-tuning case, with a manageable dataset and a few regularization strengths and deletion fractions? Start full-data and retained-data fine-tuning from the same pretrained checkpoint, with matched objectives and training procedures. Compare dual and primal IF on the same affine surrogate; IF versus converged retraining within that surrogate; and actual nonlinear outputs versus linear predictions at the same proposed update. Then compare the updated nonlinear model with nonlinear retained-data retraining. Report retain/test utility, output disagreement or KL, and end-to-end cost. A directional Hessian-vector diagnostic would help determine whether training tracking also predicts update accuracy. This would clarify the gap at useful task performance; a clear failure boundary would itself be informative. Full pretraining or a broad new benchmark suite is not requested.

1 comment

Question 1

No. In all experiments, the NTK matrix $ \mathbf{K} $ is precomputed and stored. We did separate the construction and storage of $ \mathbf{K} $ from solving and parameters mapping. The reported runtime for the dual representation of influence functions includes both solving the dual space linear system using CG and mapping the resulting parameter back to the original parameter space, i.e., $ \Delta \boldsymbol{\alpha}_r^\star \to \boldsymbol{\theta}_r^\star $, using the mapping $ \phi(\cdot) $. For the original representation of influence functions, the reported runtime includes only solving the linear system in $ \boldsymbol{\theta} $-space. Both linear systems are solved using CG with a tolerance of $ 10^{-8} $ and a maximum of $ 500 $ iterations.

We did not measure the break-even number of requests needed to amortize the cost of forming $ \mathbf{K} $, as we think this analysis depends on too many factors and could be a separate study by itself. For the empirical NTK, construction of $ \mathbf{K} $ requires computing

∇θf(x(i),θ′)⊤∇θf(x(i),θ′)\nabla_{\boldsymbol{\theta}} f(\mathbf{x}^{(i)}, \boldsymbol{\theta}')^\top \nabla_{\boldsymbol{\theta}} f(\mathbf{x}^{(i)}, \boldsymbol{\theta}')

for all data points $ \mathbf{x}^{(i)} $, the break-even number thus depends on the model, dataset, GPU hardware, number of GPUs, and the method used to parallelize kernel construction and influence function evaluation. For the analytical NTKs derived from infinitely wide neurual networks, entries of $ \mathbf{K} $ can be evaluated analytically without constructing gradients or Jacobians. In that setting, even an expensive construction of $ \mathbf{K} $ is worthwhile, since it allows influence function to be used for models with infinite number of parameters in $ \boldsymbol{\theta} $-space.

Question 2

Golatkar et al. (ECCV 2020). This work uses tools from NTK theory to estimate the retrained model paraemters $ \boldsymbol{\theta}_r $ from the original model parameters $ \boldsymbol{\theta} $ without retraining from scratch, where they assume both models are trained using unregularized squared error loss. The basic idea behind their proposed method is straightforward. When using the linearized model and squared error loss, both $ \boldsymbol{\theta}_r $ and $ \boldsymbol{\theta} $ have closed form expressions, therefore their difference $ \boldsymbol{\theta}_r - \boldsymbol{\theta} $ also has closed form expression. To estimate $ \boldsymbol{\theta}_r $, one only needs to evaluate the closed form expression for $ \boldsymbol{\theta}_r-\boldsymbol{\theta} $ and add it back to $ \boldsymbol{\theta} $. This result follows directly from NTK theory rather than through an influence function argument.

Our work established a relationship between their update and the (dual) influence function based update. Specifically, their update (equation (10) in their paper) can be recovered by specializing our formulation to squared error and taking the limit as $ \lambda \to 0^+ $. Don't have enough space to show the proof here, please let us know if you want to see a proof. This why our results can be considered as a genrealization of their results. We generalizes their unregularized squared error result in two ways: it accommodates loss functions beyond squared error and it incorporates regularization.

Zhang and Zhang (AAAI 2022). This work and our work study two different things. This work studies how accurately influence functions estimate the influence of removing a single training point from an overparameterized neural network trained with regularized squared error loss. Their analysis is conducted in the original parameter space ($ \boldsymbol{\theta} $-space).

Our work, on the other hand, studies how to re-represent the influence function in a dual parameter space ($ \Delta \boldsymbol{\alpha} $-space) so that the influence of removing a group of data points can be computed more efficiently. In addition, our work accommadates loss functions beyond squared error, provided that they satisfy our assumptions.

Question 3

Thank you for this concrete and constructive suggestion. We agree that evaluating our method on a pretrained transformer would further demonstrate its practical applicability and limitations. However, given our lack of experience on training and using transformer, the implementation effort and computation resources required, it is unlikely that we can complete these experiments during the current review period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.