- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies budgeted predictive data attribution across many test queries by representing per-query metagradients as the rows of an influence matrix. It introduces MAGE for influence-matrix reconstruction and SPELL for a row-normalized objective motivated by the linear datamodeling score, both using a limited number of metagradient measurements formed from linear combinations of queries. The methods are evaluated on ResNet-9/CIFAR-10 and a 92M-parameter GPT-2-style model fine-tuned on TinyStories. MAGE achieves the lowest reconstruction error, while SPELL performs best on the reported attribution metric among the evaluated computable baselines.
Critical strengths
- The paper presents a clear and useful formulation of many-query predictive data attribution as influence-matrix estimation under a limited metagradient budget. The observation that a single replay can measure arbitrary linear combinations of query influences is simple but practically important, as it allows the cost of metadifferentiation to be shared across many queries.
- The paper also makes a clear distinction between global matrix reconstruction and per-query attribution quality. The different behavior of MAGE and SPELL, together with the appendix analysis of highly imbalanced influence-row norms, helps explain why these two objectives can favor different measurement strategies.
Critical weaknesses
- The efficiency claim is not fully supported by an equal-compute comparison. MAGE and SPELL require an additional forward-mode differentiation for each replay, while the experimental budget counts only reverse REPLAY calls. Without end-to-end wall-clock or FLOP measurements that include this extra computation, it is difficult to determine whether the proposed methods are more efficient than the baselines under the same computational budget.
- Several methodological claims also depend on approximations that are not directly validated. MAGE uses as a proxy for the true influence geometry , while SPELL optimizes a surrogate objective rather than the reported mean Spearman LDS itself. Although the experiments show that these choices work in the two evaluated settings, the paper does not directly test the geometric assumptions behind them or examine more challenging regimes where the approximations may break down.
Questions for authors
- Could the authors provide an equal-compute comparison that includes the additional forward-mode differentiation required by MAGE and SPELL, for example using wall-clock time, FLOPs, or GPU-hours?
- How closely does approximate the true influence geometry , beyond the reported correlation between query-gradient norms and influence-row norms?
- SPELL is derived through several approximations and ultimately optimizes a row-normalized squared surrogate, whereas the reported attribution metric is mean Spearman LDS. Could the authors clarify the relationship between these objectives and provide evidence that improvements in the surrogate translate to improvements in the reported LDS?
- In the CIFAR-10 experiments, are examples assigned zero training weight still included in the forward pass and BatchNorm statistics? If so, how is this intervention equivalent to retraining after removing those examples from the training set?
- Were the MAGE and SPELL measurements and projection coefficients obtained by running the stated reverse- and forward-mode differentiation procedures, or were they simulated from the fully computed influence matrix ?
We thank the reviewer for their careful reading and for recognizing both our formulation of many-query attribution and the distinction between reconstruction and attribution! We address each of the questions raised by the reviewer below.
Equal compute. We thank the reviewer for pointing this out! As discussed in our paper, we only count replays, since they are the only operation that recovers MAGIC (lines 133-134). All methods are therefore compared under the same replay budget B (lines 300-304).
Accuracy of V⊤V. In considering V⊤V as a proxy, we have relied only on its empirical performance, with MAGE nearly matching the oracle rank-B SVD of Y in both settings (Figures 1 and 3). However, we argue that it is the natural proxy available without spending additional replays.
Surrogate and LDS. Similar to TRAK (https://arxiv.org/abs/2303.14186), our surrogate is validated empirically rather than theoretically. At B=50, SPELL attains an LDS of 0.34 against 0.25 for the best computable baseline on CIFAR-10, and 0.39 against 0.32 on TinyStories (Tables 1 and 2).
Zero-weight examples. Yes! We retain zero-weight examples in the forward pass and BatchNorm statistics, and our ground-truth retrains apply the identical intervention (lines 318-319), following the single-model setting of MAGIC. We will state this convention explicitly in the revision.
Measurements. To ablate various methods and to save computational cost, we compute the measurements offline from the exact Y, and our supplementary code includes a test showing that this is equivalent to running the replays directly. This choice has no effect on any of our results.
Once again, we sincerely appreciate the reviewer's thoughtful questions and hope that our responses have addressed their concerns. We would be delighted if the reviewer would consider raising their score 🌠✨!