acceptodds
Under review as a conference paper at ICLR 2027

Data Attribution via Sketched Metadifferentiation

Abstract

Data attribution seeks to quantify how individual training examples shape a model’s predictions and underpins problems including data valuation, machine unlearning, and model interpretability. Despite having a long line of work, computationally scalable methods often struggle to predict the effect of removing training data in neural networks due to their non-convex nature. To overcome this challenge, metagradient-based methods such as MAGIC (Ilyas and Engstrom, 2025) differentiate each prediction through the entire training run and compute its exact influence with respect to the training data, but require a separate run for every prediction. To reduce this cost, we cast budgeted attribution as estimating a large influence matrix from a small number of measurements. We show that the measurements most appropriate for recovering this matrix differ from those best suited for attribution itself. We then present two algorithms, MAGE and SPELL, suited for reconstruction and attribution respectively, that run on existing meta-gradient machinery at no extra cost. Empirical studies demonstrate strong performance over existing baselines across training scales and measurement budgets.

Primary area:
interpretability and explainable AI
Keywords:
Data attribution, metagradients, influence functions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wfTPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies budgeted predictive data attribution across many test queries by representing per-query metagradients as the rows of an influence matrix. It introduces MAGE for influence-matrix reconstruction and SPELL for a row-normalized objective motivated by the linear datamodeling score, both using a limited number of metagradient measurements formed from linear combinations of queries. The methods are evaluated on ResNet-9/CIFAR-10 and a 92M-parameter GPT-2-style model fine-tuned on TinyStories. MAGE achieves the lowest reconstruction error, while SPELL performs best on the reported attribution metric among the evaluated computable baselines.

Critical strengths
  1. The paper presents a clear and useful formulation of many-query predictive data attribution as influence-matrix estimation under a limited metagradient budget. The observation that a single replay can measure arbitrary linear combinations of query influences is simple but practically important, as it allows the cost of metadifferentiation to be shared across many queries.
  2. The paper also makes a clear distinction between global matrix reconstruction and per-query attribution quality. The different behavior of MAGE and SPELL, together with the appendix analysis of highly imbalanced influence-row norms, helps explain why these two objectives can favor different measurement strategies.
Critical weaknesses
  1. The efficiency claim is not fully supported by an equal-compute comparison. MAGE and SPELL require an additional forward-mode differentiation for each replay, while the experimental budget counts only reverse REPLAY calls. Without end-to-end wall-clock or FLOP measurements that include this extra computation, it is difficult to determine whether the proposed methods are more efficient than the baselines under the same computational budget.
  2. Several methodological claims also depend on approximations that are not directly validated. MAGE uses V⊤VV^\top V as a proxy for the true influence geometry V⊤JJ⊤VV^\top J J^\top V, while SPELL optimizes a surrogate objective rather than the reported mean Spearman LDS itself. Although the experiments show that these choices work in the two evaluated settings, the paper does not directly test the geometric assumptions behind them or examine more challenging regimes where the approximations may break down.
Questions for authors
  1. Could the authors provide an equal-compute comparison that includes the additional forward-mode differentiation required by MAGE and SPELL, for example using wall-clock time, FLOPs, or GPU-hours?
  2. How closely does V⊤VV^\top V approximate the true influence geometry V⊤JJ⊤VV^\top J J^\top V, beyond the reported correlation between query-gradient norms and influence-row norms?
  3. SPELL is derived through several approximations and ultimately optimizes a row-normalized squared surrogate, whereas the reported attribution metric is mean Spearman LDS. Could the authors clarify the relationship between these objectives and provide evidence that improvements in the surrogate translate to improvements in the reported LDS?
  4. In the CIFAR-10 experiments, are examples assigned zero training weight still included in the forward pass and BatchNorm statistics? If so, how is this intervention equivalent to retraining after removing those examples from the training set?
  5. Were the MAGE and SPELL measurements and projection coefficients obtained by running the stated reverse- and forward-mode differentiation procedures, or were they simulated from the fully computed influence matrix YY?
1 comment

We thank the reviewer for their careful reading and for recognizing both our formulation of many-query attribution and the distinction between reconstruction and attribution! We address each of the questions raised by the reviewer below.

  1. Equal compute.\textbf{Equal compute.} We thank the reviewer for pointing this out! As discussed in our paper, we only count replays, since they are the only\textbf{only} operation that recovers MAGIC (lines 133-134). All methods are therefore compared under the same replay budget BB (lines 300-304).

  2. Accuracy of V⊤V.\textbf{Accuracy of } V^\top V. In considering V⊤VV^\top V as a proxy, we have relied only on its empirical performance, with MAGE nearly matching the oracle rank-BB SVD of YY in both settings (Figures 1 and 3). However, we argue that it is the natural proxy available without spending additional replays.

  3. Surrogate and LDS.\textbf{Surrogate and LDS.} Similar to TRAK (https://arxiv.org/abs/2303.14186), our surrogate is validated empirically rather than theoretically. At B=50B = 50, SPELL attains an LDS of 0.34 against 0.25 for the best computable baseline on CIFAR-10, and 0.39 against 0.32 on TinyStories (Tables 1 and 2).

  4. Zero-weight examples.\textbf{Zero-weight examples.} Yes! We retain zero-weight examples in the forward pass and BatchNorm statistics, and our ground-truth retrains apply the identical intervention (lines 318-319), following the single-model setting of MAGIC. We will state this convention explicitly in the revision.

  5. Measurements.\textbf{Measurements.} To ablate various methods and to save computational cost, we compute the measurements offline from the exact YY, and our supplementary code includes a test showing that this is equivalent to running the replays directly. This choice has no effect on any of our results.

Once again, we sincerely appreciate the reviewer's thoughtful questions and hope that our responses have addressed their concerns. We would be delighted if the reviewer would consider raising their score 🌠✨!

Reviewer_iE9uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a promising extension to MAGIC to save computational cost by replacing one replay per query with a small budget of probes, with each measuring a linear combination of the rows of the influence matrix through REPLAY. The authors then note that reconstruction and attribution are optimized by different measurements, and introduce MAGE for recovering the matrix and SPELL for data attribution respectively. Experiments on a ResNet-9 on CIFAR-10 and a 92M GPT-2 on TinyStories validate the efficacy of each method on its intended objective.

Critical strengths
  1. The core framing is simple and useful. The observation that reverse-mode differentiation through training is linear motivates the proposed method. Converting per-query attribution into low-rank matrix estimation under a fixed budget overcomes the bottleneck that makes MAGIC impractical for many held-out examples.

  2. Separating reconstruction from attribution. The paper derives that matrix reconstruction and attribution need two different estimators. And the experiment results validate this.

Critical weaknesses
  1. The motivation and the type of contribution are unclear. The paper is framed as reducing the cost of MAGIC, but the savings come with a large loss in attribution quality that is not discussed. At k=500 and B=50, SPELL reaches an LDS of 0.335 against MAGIC's 0.909 on CIFAR-10 and 0.393 against 0.995 on TinyStories, so a tenfold reduction in replays recovers only about 40% of the attribution signal. The paper should state where it sits between "yet another compute-accuracy tradeoff" and "a genuinely better Pareto front," and support by plotting LDS against actual compute for its methods and for baselines at matched budgets.

  2. The baselines are too weak. All five computable comparisons are variants of the same measurement-selection idea, namely random probes, first-B queries, and PCA, plus two SVD oracles. Established data attribution methods such as TRAK[1], EKFAC influence function[2], and LOGRA[3] are absent, even though they are far cheaper than replays. Unrolling-based methods would be even more informative, since they approximate the same training-trajectory quantity that replay computes exactly. SOURCE [4] is one of the natural candidates. Without them it is difficult to tell whether a reader who needs attribution on a fixed budget should prefer SPELL or run an approximate influence function.

[1] TRAK: Attributing Model Behavior at Scale

[2] Studying Large Language Model Generalization with Influence Functions

[3] What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions

[4] Training Data Attribution via Approximate Unrolled Differentiation

Questions for authors
  1. comparisons with baseline methods measuring computation cost (in FLOPS/peak memory) and LDS score would substantially strengthen the paper.
Reviewer_ASh1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors address the problem of data attribution where we try to quantify which individual examples are responsible for model predictions. They base their work on the recent meta-gradient based method MAGIC (Ilyas & Engstrom, 2025) which is an accurate but computationally expensive method, since we need to differentiate through the entire training procedure for each prediction. The central idea of the proposed work is to reduce the cost of computing the influence matrix (kk predictions ×\times nn training examples) using the idea of sketching with a small number of B<kB < k queries. The key insight is that the REPLAY method used by MAGIC allows for measuring a linear combination of rows of the influence matrix YY. This along with an extra forward pass can be used to come up with an estimate Y^\hat{Y}

The authors come up with two different procedures - (a) MAGE that optimizes ∣∣Y−Y^∣∣F2||Y - \hat{Y}||_F^2 and (b) SPELL that optimizes for a proxy of the linear data modeling scpre (LDS). This boils down to maximizing ratio of norms of Y^\hat{Y} and YY. They evaluate MAGE and SPELL across two settings, with RESNET-9 trained on CIFAR-10 and GPT-2 trained on TinyStories. They compare for relative frobenius error (reconstruction), and LDS (attribution) across different ratios of B and k (k fixed to 500). They include different baselines and oracles for MAGE and SPELL. They show that MAGE and SPELL excel among other methods in coming close to oracles (which assume knowledge of YY).

Critical strengths
  • The development is clear and easy to follow, the key insight I pointed to in the summary is a good discovery, and the authors have developed the methods with good grounding in theory. This solid development based on theory is one of the key strengths of the paper.

  • The experiments are clear as presented which is a strength but I do have other concerns that I list later.

Critical weaknesses
  • I did not see a good proxy for computation time and memory or resources used except for BB. It was not clear what computational overhead is added by the MAGE and the SPELL sketching methods.

  • The experiments have missed a few key aspects. Is M=300M=300 enough? Looking at the oracle performance (particularly in GPT2) compared to MAGIC, I also wonder if both BB and kk or at least BB should be upped so that the oracles come close to the MAGIC performance and then we can see how SPELL catches up to it.

Questions for authors
  1. I wonder why MAGE is needed. Attribution is a clear use case that motivates SPELL, but where would we need to reconstruct the attribution matrix exactly. The authors list data valuation as that use case and imply that MAGE can be used to identify training data that drives model behavior. How is this different from re-training on modified data (the counterfactual attribution)? Mathematically they are indeed different as the authors show, but is good reconstruction even necessary?

  2. How are PCA and MAGE related? Could the authors expand on what they discuss in lines 305-306? They seem close in performance.

  3. How general is the observation in lines 241-242 (does the empirical observation hold generally)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.