acceptodds
Under review as a conference paper at ICLR 2027

Distilling Diffusion Score Discrepancy for Efficient Training Data Attribution

Abstract

Training data attribution for diffusion models aims to identify the training samples that influence a generated instance, but existing methods either require costly per-sample gradient computation or query-specific model optimization. Moreover, most methods attribute changes in a proxy loss rather than changes in the actual model's generative behavior. We address these limitations by formulating attribution directly with a local score discrepancy measure, which applies to any diffusion variant (including DDPM, EDM, and flow matching), and by showing that such a measure can be estimated without retraining, as a preconditioned gradient similarity. We instantiate this estimator as Training-data Influence via score Discrepancy (TID), which uses Kronecker-factored curvature to avoid random projections and per-sample gradient storage. We then distill TID into TIDE, a forward-only student trained online to reproduce the teacher's rankings from the diffusion model's internal activations. Under counterfactual evaluation on CIFAR-10, ArtBench-10, and MS-COCO, TID matches or outperforms state-of-the-art approaches, while TIDE retains most of TID's accuracy at four to five orders of magnitude lower per-query cost, attributing generated samples in milliseconds and faster than the generation itself.

Primary area:
interpretability and explainable AI
Keywords:
Training data attribution, Diffusion models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bpxmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates which training images helped shape a diffusion model’s generated image. It introduces a gradient-based influence estimator TID, and a lightweight student that turns attribution into fast vector search (TIDE). The main contributions are combining diffusion-model features, online teacher supervision, and a ranking loss. Experiments report competitive attribution quality and substantial query-time speedups with cached embeddings. TIDE closely matches its teacher’s rankings with Spearman correlations around 0.86–0.89. Removing highly ranked training images and retraining also changes generated outputs, suggesting the selected examples matter.

Critical strengths
  1. TIDE turns expensive attribution calculations into fast searches over cached embeddings. Its strong agreement with the teacher suggests that much of the attribution signal survives this compression, making repeated analysis more practical.

  2. The authors retrain models after deleting selected training images, testing whether those selections actually affect generated outputs. Combined with comparisons against several attribution baselines, this gives concrete evidence that the method identifies consequential training data.

Critical weaknesses
  1. The deletion tests miss query-specific accuracy. The authors remove the union of training images selected for multiple queries. Shuffling the query-to-selection assignments leaves that union unchanged, so the experiment cannot tell whether the method linked each generated image to the right training examples.

  2. The gains over prior work aren’t convincing. Missing comparisons with Fast Data Attribution’s student and the same diffusion features without distillation make it hard to identify what TIDE adds. Misdescribing the prior method’s unlearning cost further weakens the argument for its claimed advantages.

Questions for authors

Is it reasonable to replace the squared norm of output changes with their squared sum (line 144~145)? Could you directly validate whether this approximation preserves attribution rankings on a small model?

3 comments

Thank you for the comments. Based on your feedback, we provide additional control-query results, a diffusion-feature baseline comparison, and an empirical scalarization analysis.

1. Query targeting in the deletion evaluation (W1).

We agree that grouped removal may obscure the query-to-selection matching. To examine whether the selected training data could affect unrelated queries, we evaluate an additional control group. Each run involves 40 query samples. The first 20 are the normal targeted queries samples that are attributed and used to determine the removal set; the remaining 20 are randomly selected and do not relate to the selections. We regenerate both groups using the same retrained model and evaluate them against the same random-removal reference. Mean counterfactual AUCs over six seeds are:

Method and query groupCIFAR-10ArtBench-10MS-COCO
TIDE, attributed queries0.8580.8500.748
TIDE, control queries0.6010.5070.539
TID, attributed queries0.8350.8890.783
TID, control queries0.5870.5280.588
MUCS, attributed queries0.8700.8500.738
MUCS, control queries0.5590.5300.551

For every method and dataset, the attributed queries show substantially larger AUCs than the controls, and the AUCs of the controlled group are almost around random level (0.5). This demonstrates that the selected training data preferentially affects the queries used to construct it, rather than affecting unrelated queries, or producing degradation on the diffusion model.

We also examine how much the per-query top-2% selections overlap. Mean pairwise overlap across the 20 queries and six seeds is:

MethodCIFAR-10ArtBench-10MS-COCO
Random expectation2.0%2.0%2.0%
TIDE2.6%3.7%2.8%
TID3.0%4.3%2.6%
MUCS3.0%4.4%4.1%

These statistics show that the methods select largely different examples for different queries. They complement the control-query experiment by establishing diversity of the selections. We also want to clarify that we follow the MUCS protocol to evaluate this to achieve a feasible cost (Serrà et al., 2026). Separately retraining for all 20 queries would multiply the retraining count by 20.

2. What distillation adds beyond diffusion features (W2).

We evaluate the requested frozen-feature baseline on CIFAR-10. It uses the same source activations as TIDE (the residual streams of all 12 DiT blocks, mean pooling, M=32) and is directly scored with cosine similarity without postprocessing. We compare its recovery of the teacher's top-kk training examples with TIDE and external encoders. Mean overlap over 20 queries and six seeds is:

Top-kk overlap with TIDk=100k=100k=500k=500k=1,000k=1{,}000k=2,000k=2{,}000
TIDE44.5%54.2%58.5%62.7%
Frozen diffusion features21.0%29.6%34.4%40.2%
CLIP5.2%9.8%12.7%17.1%
DINO5.7%9.8%12.6%16.6%

Diffusion features do recover more of the teacher's rankings than the external encoders like CLIP, but distillation further improves overlap. This directly establishes that the learned embedding captures substantially more of TID's attribution ranking than the underlying frozen features alone.

3. Comparison with Fast Data Attribution (W2).

Our main reason for not reproducing FDA (Wang et al., 2025) is the cost of collecting its supervision. On MS-COCO, FDA uses AbU+ (Wang et al., 2024) to rank 10,000 candidates for each of 5,000 generated queries, with 4,900 queries used for training and 100 for validation. It reports 1,250 GPU-hours for this curation, before student training. Its Stable Diffusion curation costs 15,150 GPU-hours.

To estimate the expense of an analogous unlearning-supervised pipeline on our models, we use our measured MUCS runtime and consider two different training set budgets, full training set (50k samples) and the 10k candidates. The resulting estimates are:

QuantityCIFAR-10ArtBench-10MS-COCO
MUCS per query, full training set1,374 s1,452 s3,404 s
MUCS per query, 10k candidates298 s323 s321 s
Supervision for 4,900 queries, full training set1,870 h1,976 h4,633 h
supervision for 4,900 queries, 10k candidates406 h440 h437 h
TIDE's complete one-off stage3.8 h4.2 h10.0 h
Estimated supervision cost (10k) / TIDE one-off cost107×105×44×

The estimations show that collecting a similarly sized supervision set would exceed our complete distillation budget by a large margin. Even reducing the supervision set to 2,000 queries would leave a significant difference under the same assumptions. TIDE instead computes teacher scores online for within-batch pairs, sharing a forward–backward pass across each sample's pairwise supervision.

We use MUCS as the representative unlearning baseline because it is the strongest unlearning method in our comparisons; the original AbU performs at random level in our experiments, consistent with the MUCS paper. FDA uses the improved AbU+ teacher, but their Table 1 reports only modest gains over AbU. The same table also shows that the student generally underperforms or only matches its corresponding unlearning-based teacher. Given the substantial computational cost and these performance limitations, we do not include FDA in our empirical comparison.

4. Correction to the description of AbU (W2).

We identified and corrected a mischaracterization in Section 2.2: AbU performs unlearning for each query, rather than for each training sample. We thank the reviewer for pointing this out.

5. Validity of summed-output scalarization (Q1).

We agree that the squared sum of output changes is not generally equal to their squared norm. We therefore directly evaluate the ranking fidelity of this approximation using a controlled setting on CIFAR-10. We reuse approximately 40 subset-retrained models per seed from our CIFAR-10 benchmark and ablation experiments, across six seeds. For each subset-retrained model, we compute Δθm=θ∖Um−θ\Delta\theta_m=\theta^{\setminus\mathcal U_m}-\theta. We rescale all directions by a common factor to obtain a specified median relative parameter shift ρ:=median⁡m∥cΔθm∥2/∥θ∥2\rho := \operatorname{median}_m \|c \Delta\theta_m\|_2 / \|\theta\|_2, then compute the Spearman correlation between exact score discrepancies and those from the linearized norm in Eq. 2 or the summed-output scalarization in Eq. 3. Averaging over 20 queries and four noise levels gives:

Relative shift ρ\rhoOutput linearization, Eq. 2Summed-output scalarization, Eq. 3
10−410^{-4}1.000.76
10−310^{-3}1.000.76
10−210^{-2}1.000.76
10−110^{-1}0.970.74

These results show that scalarization introduces a measurable approximation error but retains substantial ranking agreement over the tested range. Separately, we evaluate the final TID scores using the same set of models. For each of 20 generated queries, we again measure the score discrepancy between the original model and each retrained model at four noise levels sampled from the Karras generation schedule (Karras et al., 2022). We compare these measurements with attribution predictions aggregated over the corresponding removed subset.

For each query and noise level, we compute Spearman correlation across the retrained models, then average these correlations over queries and seeds:

Methodσ=6.46\sigma=6.46σ=3.83\sigma=3.83σ=2.17\sigma=2.17σ=1.17\sigma=1.17
TID0.540.600.650.60
MUCS0.400.520.630.61
DAS0.300.440.420.44
CLIP0.200.310.390.43
Random scores0.240.320.400.41

The random baseline assigns uniformly random scores to training samples and aggregates them over the same removed subsets. TID consistently exceeds most baseline methods and achieves the highest correlation at three of the four noise levels. These experiments provide empirical evidence for both the scalarization step and the fidelity of the final estimator.

References:

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

Joan Serrà, Dipam Goswami, Fabio Morreale, Wei-Hsiang Liao, and Yuki Mitsufuji. Training data attribution in diffusion models via mirrored unlearning and noise-consistent skew. In Advances in Neural Information Processing Systems (NeurIPS), 2026.

Sheng-Yu Wang, Aaron Hertzmann, Alexei A. Efros, Jun-Yan Zhu, and Richard Zhang. Data attribution for text-to-image models by unlearning synthesized images. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

Sheng-Yu Wang, Aaron Hertzmann, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu. Fast data attribution for text-to-image models. In Advances in Neural Information Processing Systems (NeurIPS), 2025.

Reviewer_CpLQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TID and TIDE for training data attribution in diffusion models. TID defines influence through the local score discrepancy induced by removing a training example and approximates it using gradient-based similarity. TIDE distills TID’s influence rankings into low-dimensional embeddings derived from the diffusion model’s internal activations, enabling efficient retrieval without backward passes at query time. Experiments on three datasets evaluate attribution quality by removing selected training examples, retraining the models, and measuring changes in generated outputs. The results demonstrate a favorable trade-off between attribution accuracy and per-query computational cost.

Critical strengths
  • TIDE distills TID’s influence rankings using the diffusion model’s internal activations. By caching training-image embeddings and eliminating backward passes at query time, it offers a favorable trade-off between attribution accuracy and computational cost.

  • The paper evaluates attribution by removing selected training examples and retraining models across three datasets. Results across multiple similarity metrics and random seeds, together with ablations of key design choices, support the method’s practical utility.

Critical weaknesses
  • Insufficient validation of approximation fidelity: TID relies on first-order output linearization, summed-output scalarization, and approximations to parameter changes and curvature, alongside normalization and the removal of certain weights and correction terms. Although the empirical effects of these choices are examined, it remains unclear how well the final scores preserve the rankings induced by the original leave-one-out score discrepancy. The high ranking agreement between TIDE and TID does not validate the teacher’s approximation itself. A small-scale comparison between actual leave-one-out score discrepancies and TID scores would strengthen the connection between the proposed target and the final algorithm.
Questions for authors
  • Validation on a toy dataset or a small model, where actual leave-one-out retraining is feasible, would be particularly useful. Comparing the resulting score-discrepancy rankings with TID’s rankings, and examining how they change after each approximation step, would help establish the fidelity of the approximations.

  • Figure 2 varies the computational budget for TID and TIDE, but does not provide corresponding budget sweeps for the baselines. Could DAS, D-TRAK, or MUCS match or exceed TID’s attribution accuracy when given a comparable per-query computational budget? Comparing their accuracy–cost curves would help clarify whether TID’s advantage persists under matched computational budgets.

3 comments

Thank you for your constructive feedback on the validation of approximation fidelity. In response, we provide a direct evaluation of how well our estimator predicts measured retraining discrepancies. We also clarify below how baseline methods, including MUCS, DAS and D-TRAK, are expected to perform under budgets comparable to those of TID or TIDE.

1. Fidelity of TID to measured retraining discrepancies (W1; Q1).

We reuse approximately 40 subset-retrained models per seed from our CIFAR-10 benchmark and ablation experiments, across six seeds. For each of 20 generated queries, we measure the score discrepancy between the original model and each retrained model at four noise levels sampled from the Karras generation schedule (Karras et al., 2022). We compare these measurements with attribution predictions aggregated over the corresponding removed subset.

For each query and noise level, we compute Spearman correlation across the retrained models, then average these correlations over queries and seeds:

Methodσ=6.46\sigma=6.46σ=3.83\sigma=3.83σ=2.17\sigma=2.17σ=1.17\sigma=1.17
TID0.540.600.650.60
MUCS0.400.520.630.61
DAS0.300.440.420.44
CLIP0.200.310.390.43
Random scores0.240.320.400.41

The random baseline assigns uniformly random scores to training samples and aggregates them over the same removed subsets. TID consistently exceeds most baseline methods and achieves the highest correlation at three of the four noise levels. This experiment evaluates the final estimator against actual model changes. Although the computational cost makes it infeasible to provide the direct LOO results requested by the reviewer, we hope this experiment provides alternative evidence of TID’s fidelity.

2. Baseline computational budgets (Q2).

We use baseline configurations reported in MUCS (Serrà et al., 2026). Moreover, the baseline budgets are already larger than those of TID and TIDE: MUCS uses 100 noise levels on all three datasets, and DAS and D-TRAK use M=100 timesteps on CIFAR-10 and M=50 on ArtBench and COCO, while the main TID and TIDE configurations use M=32. At these configurations, TID and TIDE exceed D-TRAK and DAS in mean counterfactual AUC on all three datasets and exceed MUCS on ArtBench and MS-COCO. The budgets are therefore already comparable, and in favor of the baselines.

References:

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

Joan Serrà, Dipam Goswami, Fabio Morreale, Wei-Hsiang Liao, and Yuki Mitsufuji. Training data attribution in diffusion models via mirrored unlearning and noise-consistent skew. In Advances in Neural Information Processing Systems (NeurIPS), 2026.

The additional experiments partially address my concern. However, while the individual approximations are motivated by prior work or supported by ablations, it remains unclear how faithfully their combination preserves the original score-discrepancy target. In particular, output scalarization, the approximations to parameter changes and curvature, and the subsequent normalization and omission of weighting terms can all alter the resulting rankings. Ablations demonstrating their practical effectiveness do not directly establish approximation fidelity. The comparison with actual subset-retraining discrepancies provides useful supporting evidence, but does not fully resolve the gap between the original leave-one-out formulation and the final estimator. My concern about this connection therefore remains.

Thank you for the follow-up. We would like to clarify the purpose of the experiment in our previous response. It is not an ablation of downstream effectiveness, but a direct evaluation of attribution fidelity through subset retraining, following the principle of the linear datamodeling score (LDS), a standard measure of attribution fidelity in the literature (Park et al., 2023; Zheng et al., 2024; Lin et al., 2025). As additional evidence, and to specifically address the leave-one-out formulation, we also conducted exact leave-one-out retraining using a small model trained on the full CIFAR-10 training set.

Setup. We train a denoiser (MLP, 68k parameters) on the full CIFAR-10 training set, with images downsampled to 8×8, using the EDM objective and 4,000 training steps. For each of the 50k training samples, we retrain the model from scratch with only that sample removed while keeping the same initialization and training randomness. Each leave-one-out model tells us how much each query changes when one training sample is removed. For each of 64 queries, we measure the output discrepancy of Eq. 1 for every removed sample at four noise levels of the generation schedule, same as before. We compare this measurement with the prediction at each stage of the derivation. Eqs. 2 and 3 use the true parameter changes obtained from leave-one-out retraining, while Eqs. 4 and 5 use the exact G\mathbf{G} computed over the full training set. We also evaluate two intermediate stages to separate the effects of individual approximations. The first removes the leverage factor 1/(1−hi)1/(1-h_i) from Eq. 4 while retaining the residual rir_i, isolating the effect of omitting leverage. The second replaces G\mathbf{G} in Eq. 5 with its block-diagonal approximation across layers, retaining each diagonal block exactly. This separates the effect of ignoring cross-layer curvature from that of the within-layer Kronecker factorization introduced in Eq. 6. Finally, Eq. 7 evaluates the complete TID estimator, including normalization. Each stage of the derivation is scored using Spearman correlation with the measured leave-one-out discrepancy across the 50,000 training samples, averaged over queries.

Stageσ=6.46\sigma=6.46σ=3.83\sigma=3.83σ=2.17\sigma=2.17σ=1.17\sigma=1.17Mean
Eq. 2 (output linearization)1.001.001.001.001.00
Eq. 3 (output scalarization)0.660.700.700.690.69
Eq. 4 (Newton step with exact G\mathbf{G}, residual rir_i and leverage hih_i)0.410.460.480.470.46
Eq. 4, without the leverage term0.410.460.480.470.46
Eq. 5 (residual and leverage dropped)0.430.490.520.520.49
Eq. 5, block-diagonal G\mathbf{G} per layer0.440.500.540.570.51
Eq. 6 (K-FAC)0.440.500.520.530.50
Eq. 7 (TID)0.440.490.510.490.48

We hope this exact leave-one-out experiment provides a clearer picture of the connection between the original target and the final estimator. Output linearization closely preserves the discrepancy rankings, while scalarization reduces the mean correlation to 0.69. The remaining approximations together introduce a further reduction to 0.48. Thus, the approximations do incur a loss of fidelity, but these losses do not accumulate monotonically. The final TID estimator retains meaningful ranking agreement with the original leave-one-out score-discrepancy target across all four noise levels. This experiment directly evaluates the combined effect of the approximations against that target, complementing the previous subset-retraining evidence.

References

Jinxu Lin, Linwei Tao, Minjing Dong, and Chang Xu. Diffusion attribution score: Evaluating training data influence in diffusion models. In International Conference on Learning Representations (ICLR), 2025.

Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. TRAK: Attributing model behavior at scale. In International Conference on Machine Learning (ICML), 2023.

Xiaosen Zheng, Tianyu Pang, Chao Du, Jing Jiang, and Min Lin. Intriguing properties of data attribution on diffusion models. In International Conference on Learning Representations (ICLR), 2024.

Reviewer_uB9HOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TID, which aims to compute training data attribution for generative models. There are mentioned two main gaps in the literature: focus on proxies, such as loss gradients, rather than data samples themselves, and the computational cost of prior work. TID builds on recent work from various research fields to derive a new formulation for attribution methods. Although the initial formulation focuses on training data, it reduces to loss estimations. However, the empirical results are promising and the overall cost seems to be reduced compared to at least one of the baselines used. Moreover, this paper introduces TIDE, an efficient alternative that reduces online computation by distilling knowledge from TID. However, it is not clearly reported the computational cost of the training-phase TIDE, which must consider the computational cost (time and memory) of TID as well. To this end, both gaps seem to be partially addressed. Further evidence, and clearer reporting of evidence on these two axes would enhance the significance of this work.

Critical strengths
  • It seems that there is a strong theoretical analysis. Although some assumption may have gone unreported, and estimation errors may have not been sufficiently reported, the overall picture seems sufficient for a work that mainly leans to empirical contribution.
  • The results are promising. The proposed method outperforms state-of-the-works and the online (per-query) cost of TIDE (not TID) is impressive (Fig 2).
  • The ablation analysis on M, along with Table 2 mitigate concerns about the effect of relaxations and estimations used at a sufficient extend.
Critical weaknesses
  • In the introduction you mention two gaps in prior work, yet it is not clear whether TID(E) addresses them.
    • 1. Your mention that prior work focuses on gradients, while ''a shift in loss does not necessarily reflect a shift in generated output". However, in Eq (2) gradients are introduced in TID as well.
    • 2. The cost of gradient based attribution. The main paper does not elaborate on this axis of the contribution. TID requires computing high-dimensional matrices and matrix inversion (eg, Eq. 6). This arises concerns about the computation and complexity of TID, and TIDE. Moreover, (at least) the one-off computation of TIDE must consider the runtime of TID, since TIDE cannot be computed without TID (i refer to Table 10). Moreover, Table 10 does not include the D-TRAK baseline, while Table 9 suggests that D-TRAK may be more efficient than TID+TIDE, while MUCS does not require training.
  • Taylor Expansion & Newton Step & Unreported assumptions: In line 136, you use first-order Taylor expansion, while prior work in unlearning typically considers second-order expansion, which requires the inversion of Hessian (whatever this means for computational cost and applicability). In fact, the influence function is determined by this third term [1]. How rough is this approximation, and how these concerns could be mitigated? Moreover, prior unlearning work, e.g. [1], that leverages linearization and Newton-step approaches adopts (strong) convexity assumptions about the loss landscape, which, in general, are violated, in high-dimensional generative models. Does not introduce limitations? Moreover, why G\mathbf{G}, Al\mathbf{A}_l, Bl\mathbf{B}_l are assumed invertible? Please, explicitly report that this is an assumption (if this is the case) and provide evidence (if exist) that this is not a strong assumption or it does not affect the subsequent formulations. Clearly mentioning the assumptions and the limitations (if there are such), it would be appreciated.
  • Batch Influence vs Sample Influence requires additional requirements: Prior unlearning work [2,3] that leverages newton-step updates and estimates data influences requires that the samples are i.i.d. according to the training set distribution, and that these samples are a significant minority of the training set. I think that is assumption should be clearly stated. Moreover did your batch selection approach aligns with this requirement?
  • Excessive use of estimations and relaxations: The mathematical formulation of the paper is well-presented. However, some assumptions (that seem to be required) seem unreported. The concerns arised by estimations and relaxations could be further mitigated by additional ablation analyses regarding the seed sensitivity, the damping factor (for K-FAC), the effect of batch size in both performance and computational efficiency.

[1] Guo, Chuan, et al. "Certified data removal from machine learning models." ICML (2020).

[2] Zhang, Binchi, et al. "Towards certified unlearning for deep neural networks." arXiv preprint arXiv:2408.00920 (2024).

[3] Guo, Jinduo, and Yinzhi Cao. "A Robust Certified Machine Unlearning Method Under Distribution Shift." arXiv preprint arXiv:2601.06967 (2026).

Questions for authors

Please, see the Weaknesses section. I think that the major concern is about the level that the reported literature gaps are genuinely addressed. Please, elaborate on how the relaxed objective at Eq. 2 does not fall on the pitfalls of gradient-based approaches that focus on a proxies rather than the training data themselves (perhaps at lines 200-221). Moreover, I think that it should be stated more clearly (and accordingly reported on the tables e.g. Table 10) that TIDE necessitates TID, and its one-time computational cost adds to that of TID. In the same line of thought, it should be clearly stated that efficiency merits refer only to online (per query) computations, given that D-Track with unreported time and memory cost (at Table 10) seems to be more efficient than TID+TIDE, and that MUCS doesn't require training-side preprocessing. Addressing such questions or explicitly reporting the limitations could increase the overall evaluation.

4 comments

Thank you for the detailed comments. We clarify below the distinction between the attribution objective and its gradient-based approximation, account explicitly for TIDE's teacher-related computation, and explain the assumptions underlying the local approximation.

1. Connection between the motivation and the objective (W1, formulation; Questions for Authors).

The motivation behind our formulation is what the gradients measure, rather than the use of gradients themselves. Some existing approaches measure changes in denoising loss, but this loss is not a direct measure of the model's generation behavior. For example, Song et al. (2021) show that the usual weighted score-matching objective does not directly optimize likelihood and that reducing it does not necessarily improve likelihood. Our proposed target instead measures changes in the model's score field, which governs the reverse-time generation dynamics. Section 2.1 also discusses the relationship to existing output-based objectives.

The gradients in TID arise from the approximations used to estimate the score discrepancy target, and serve only as proxies for this target. Since gradient computation can be computationally expensive, we further propose TIDE, which addresses the separate efficiency concern: it distills the teacher's rankings into embeddings that require only forward passes at attribution time. Thus, the new formulation we introduce in TID and the distillation stage in TIDE address different aspects of the motivation.

2. Matrix computation, TIDE's total cost, and baseline comparisons (W1, efficiency; Questions for Authors).

The matrices inverted in Eq. 6 are layerwise Kronecker factors, rather than a matrix whose dimension is the total number of parameters (Bae, 2024). For example, a linear layer with 512 inputs and 1,024 outputs only requires inversions of two Kronecker factors of sizes 512 × 512 and 1,024 × 1,024. Without Kronecker factorization, the inversion would involve a 524,288 × 524,288 matrix. Table 9 explicitly includes their eigendecomposition cost, denoted by D3D_3.

TIDE does depend on full TID for supervision, and all teacher-related computation is already included in its reported one-off cost (3.8 h for CIFAR, 4.2h for ArtBench, 10.0h for COCO). Specifically, we estimate the factors before distillation, compute teacher scores online for within-batch pairs during distillation, train the embedding model, and construct the training embedding bank. This does not require a separate TID attribution pass over every training example for every generated query. We have noticed this ambiguity and have made factor preparation explicit in Table 9 and clarified its inclusion in Table 10's caption.

We have also added D-TRAK to Table 10. The measured one-off runtimes on a single H100 are:

MethodCIFAR-10ArtBench-10MS-COCO
D-TRAK19.6 h11.6 h37.5 h
DAS18.1 h12.3 h38.1 h
TIDE, including teacher computation3.8 h4.2 h10.0 h

Regarding the efficiency comparison with MUCS, Figure 2 reports warm-cache per-query costs, which we use as the primary setting for efficiency comparisons to reflect realistic usage. MUCS requires no training-side preprocessing, but each query incurs unlearning and training-set scoring. Including preprocessing, the total times for the 20 queries in our protocol are approximately 3.8, 4.2, and 10.0 hours for TIDE, versus 7.6, 8.0, and 18.9 hours for MUCS. TIDE therefore reaches the break-even point after approximately 10–11 queries in these settings.

3. First-order expansion, Newton approximation, convexity, and invertibility (W2).

The first-order expansion in Eq. 2 and the second-order approximation in Newton-based unlearning have different purposes. A second-order approximation of an objective is used to estimate the parameter perturbation Δθ\Delta\theta. Our first-order expansion instead approximates the resulting change in the model output:

Fθ+Δθ−Fθ≈∇θFθΔθ.F_{\theta+\Delta\theta}-F_\theta\approx \nabla_\theta F_\theta\Delta\theta.

This expansion is not the step used to estimate Δθ\Delta\theta. For that step, we follow the local-surrogate approach used in TRAK and employ a one-step Newton approximation for a scalar regression surrogate (Park et al., 2023). Squared regression on the linearized model is quadratic in the surrogate parameters, and its curvature is captured using the second-order matrix defined in Eq. 4 . We then approximate this curvature using layerwise K-FAC factors to make preconditioning computationally feasible. Therefore, this step does involve a second-order matrix, similar to the Hessian matrix in the unlearning methods.

This construction does not require the original diffusion-model objective to be globally convex, as the convexity assumption applies only to the local squared-regression surrogate. The resulting estimator remains an approximation to retraining, and we evaluate its fidelity empirically. In an additional controlled analysis on CIFAR, we compare our linearization approximation with ground-truth score discrepancies across different parameter-shift magnitudes. The approximation achieves Spearman correlations of 1.00, 1.00, 1.00, and 0.97 with exact output discrepancies at relative parameter-shift magnitudes of 10−410^{-4}, 10−310^{-3}, 10−210^{-2}, and 10−110^{-1}, respectively. These results support the validity of the first-order approximation over the tested range.

We do not assume that the empirical K-FAC factors are invertible without regularization. Before inversion, we add a diagonal damping term equal to αdamp=0.1\alpha_{\mathrm{damp}}=0.1 times the mean eigenvalue of each factor. This implementation detail is now stated immediately after Eq. 6 and in Appendix E.

4. Individual attribution and grouped removal (W3).

Our derivation estimates individual training-sample scores and does not invoke the certified-unlearning guarantees of the cited papers. The K-FAC factors are empirical averages over the entire training set, as now stated explicitly in Section 2.1 and Appendix E. Their computation does not require selecting an i.i.d. subset of deletion candidates.

We interpret “batch selection approach” as referring to the batched removal used in our evaluation protocol. TID estimates individual training-sample influence, and we do not claim that its scores fully capture the grouped data influence. Grouped removal only serves as a counterfactual evaluation: it tests whether removing the examples selected by an attribution method changes the targeted generations.

We adopt this evaluation protocol from MUCS (Serrà et al., 2026). Related subset-retraining evaluations are also widely used in training data attribution, including the linear datamodeling score (LDS) used in TRAK (Park et al., 2023), which evaluates how well aggregated individual attribution scores predict model behavior under changes to the training subset. Such evaluations empirically assess the usefulness of individual scores without assuming that data effects are exactly additive. If “batch selection” refers to a different aspect of our method, we would appreciate clarification.

5. Fidelity under successive approximations (W4).

We understand the reviewer’s main concern of whether the successive approximations preserve attribution fidelity, so we address this concern directly by evaluating the TID estimator against measured score discrepancy it targets.

We reuse approximately 40 subset-retrained models per seed from our CIFAR-10 benchmark and ablation experiments, across six seeds. For each of 20 generated queries, we measure the score discrepancy between the original model and each retrained model at four noise levels sampled from the Karras generation schedule (Karras et al., 2022). We compare these measurements with attribution predictions aggregated over the corresponding removed subset.

For each query and noise level, we compute Spearman correlation across the retrained models, then average these correlations over queries and seeds:

Methodσ=6.46\sigma=6.46σ=3.83\sigma=3.83σ=2.17\sigma=2.17σ=1.17\sigma=1.17
TID0.540.600.650.60
MUCS0.400.520.630.61
DAS0.300.440.420.44
CLIP0.200.310.390.43
Random scores0.240.320.400.41

The random baseline assigns uniformly random scores to training samples and aggregates them over the same removed subsets. TID consistently exceeds most baseline methods and achieves the highest correlation at three of the four noise levels.

Regarding the suggested sensitivity analyses, our main results already report means and standard errors over six random seeds. We use the same relative-damping coefficient, αdamp=0.1\alpha_{\mathrm{damp}}=0.1, and distillation batch size, 128, across all three datasets, without dataset-specific tuning. These hyperparameters are commonly used in data attribution or modern deep learning. For example, relative damping is also used in existing curvature-based attribution implementations, such as Kronfluence. Given the cost of additional counterfactual evaluations, we prioritized the direct fidelity analyses above over further hyperparameter sweeps. Together, these results provide empirical evidence that the approximations retain useful attribution information under our reported settings.

References:

Juhan Bae. Kronfluence: Influence functions with (eigenvalue-corrected) Kronecker-factored approximate curvature. GitHub repository, 2024. URL https://github.com/pomonam/kronfluence.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. TRAK: Attributing model behavior at scale. In International Conference on Machine Learning (ICML), 2023.

Joan Serrà, Dipam Goswami, Fabio Morreale, Wei-Hsiang Liao, and Yuki Mitsufuji. Training data attribution in diffusion models via mirrored unlearning and noise-consistent skew. In Advances in Neural Information Processing Systems (NeurIPS), 2026.

Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2021.

Reviewer_uB9HOpenReview ↗

Thank you, I increase my initial rating to weak accept.

Thank you for your detailed response and additional experiments, clarification on the assumption and genuine report of what it is empirically evaluated. Explicit report of the limitations is also appreciated. My initial concerns about potential over-statements and assumptions used are mitigated. I am still concerned about the first-order Taylor expansion. Specifically, wether the Spearman test is sufficient to support the validity of the first-order expansion, and the heuristic αdamp\alpha_\text{damp}. I increase my initial rating to "weak accept".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.