- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper revisits the evaluation methodology for training data attribution (TDA) in image generation models such as Flow Matching and diffusion models. It points out that the standard metric, the Linear Datamodeling Score (LDS), can change its value solely through reweighting operations during aggregation, such as excluding low-influence samples or applying squaring and power transformations, without altering the underlying attribution scores at all. This sensitivity arises from the long-tailed distribution of attribution scores and the noise introduced by finite sampling. From this perspective, the paper shows that the reported advantage of DAS may stem not from genuine attribution accuracy but from the suppression of low-influence samples caused by its squaring operation. As a remedy, the paper proposes SNR-LDS, which models measurement noise as zero-mean Gaussian and selects scores using an SNR threshold (), and controlled-source retrieval, which directly evaluates attribution rankings by recovering injected samples of foreign concepts. The authors also formulate FMAS, an influence-function-based method for Flow Matching.
Critical strengths
-
The paper clearly demonstrates, with concrete examples, a structural flaw in LDS: its value can shift through aggregation-side reweighting alone, such as truncation of low-influence samples or power and squaring transformations, without any change to the underlying attribution scores. LDS has been used as the de facto standard metric in much prior work, and this flaw directly concerns the reliability of method comparisons themselves. For this reason, the observation is a valuable contribution to the evaluation methodology of the field as a whole.
-
The paper carefully verifies the claim that the reported advantage of DAS arises from the suppression of low-influence samples through squaring rather than from a genuine improvement in attribution accuracy. It does so through controlled comparisons, including applying squaring to other methods and combining it with truncation. In particular, the analysis that compares linear and squared readouts on identical absolute-score heads and shows that the advantage of squaring diminishes as aggregation is restricted to high-influence samples (Fig. 2(c)) provides concrete support for the reweighting explanation. This is a convincing contribution.
Critical weaknesses
-
Although the paper takes "the reliability of evaluation" as its central theme, the proposed evaluator SNR-LDS itself carries a reliability concern. SNR-LDS assumes that the measurement noise follows a zero-mean Gaussian distribution with a single shared scale across all samples within each method and query, but the validity of this assumption is not established. As the authors themselves show in Fig. 1(b), sampling variation differs systematically with the magnitude of influence, so the possibility that the noise is sample-dependent cannot be ruled out.
-
A more serious issue is that the paper provides no justification for the choice of the threshold . In Fig. 3(a), the authors show that SNR-LDS is robust to reweighting, with changes within 0.39 points. At the same time, the sensitivity analysis in Table 12 shows that the ranking of methods flips as varies. The central claim of the paper is that LDS is unreliable because it depends on aggregation-side reweighting. However, in exchange for robustness to reweighting, SNR-LDS introduces a dependence along a different axis, namely that its conclusions depend on the choice of the threshold . This can be read as reproducing, in a different form, the very problem the paper criticizes.
Questions for authors
The following questions all bear on my assessment of the paper.
-
Regarding the isotropic-variance Gaussian assumption in SNR-LDS, is there any evidence that the score residuals in the central region actually follow a zero-mean Gaussian distribution with homogeneous variance? Is the magnitude-dependent variation shown in Fig. 1(b) consistent with this assumption? If the noise were in fact sample-dependent, to what extent would the validity of the SNR-based selection be undermined?
-
On what basis was the threshold chosen? Table 12 shows that the ranking of methods reverses depending on , so is it possible to present a robust conclusion that does not depend on the threshold? Furthermore, how should the robustness to reweighting shown in Fig. 3(a) and the sensitivity to shown in Table 12 be understood together, from the standpoint of the evaluator's reliability?
4 comments
W2&Q2: Threshold
In Table 12 of the appendix, our goal in examining benchmark performance across is not to suggest that method rankings arbitrarily flip with , but to show that when the threshold is set too low (relaxed), unreliable background noise re-enters aggregation and makes the evaluation untrustworthy.
Specifically, in vanilla LDS, post-hoc reweighting transformations such as squaring suppress the weights of noisy low-influence samples during aggregation, granting artificial performance gains across methods. SNR-LDS addresses this issue by restricting subset aggregation to training samples with reliable attribution predictions. Consequently, serves as a pre-specified signal-to-noise ratio threshold fixed in the evaluation protocol and applied uniformly across all methods to define what constitutes a reliable prediction.
To determine what signal-to-noise ratio qualifies as reliable, we follow the standard convention in statistics and signal detection, under which pure background noise rarely crosses the threshold. When is lowered to or , a large fraction of the retained set consists of background-noise exceedances, and at , SNR-LDS degenerates completely into vanilla full-sum LDS. Thus, the robustness to reweighting in Figure 3(a) and the sensitivity to small in Table 12 are two sides of the same mechanism: score reweighting inflates vanilla LDS by suppressing background noise in the regime; once this unreliable background is filtered out at , reweighting no longer yields artificial gains (Figure 3(a)); conversely, lowering the threshold to in Table 12 re-admits background noise into aggregation, re-exposing the evaluator to reweighting bias and causing the ranking to flip.
To verify this directly, we evaluate how well SNR-LDS at different thresholds correlates with the counterfactual deletion effect on CIFAR-2 FM. For each query and method, we compare the per-query evaluator score against the true query-loss increase after deleting the method's top-300 or top-1000 training samples and retraining the model:
| SNR Threshold | Expected Noise Share in Retained Set | Correlation with Deletion Effect (top-300) | Correlation with Deletion Effect (top-1000) |
|---|---|---|---|
| (vanilla LDS) | — | ||
| (default threshold) | |||
As shown in the table, when is too small (), the evaluator fails to filter out background noise and correlates poorly with true deletion effects; the ranking flip at in Table 12 is a direct result of this noise contamination. Once is set high enough to filter out background noise, the correlation with true deletion effects rises substantially and remains stable. Thus, follows the standard convention while lying squarely within the reliable regime where evaluation aligns with true counterfactual effects. We have incorporated these experiments and analyses into Appendix E.2 (Figure 9) of the revised paper.
I thank the authors for their detailed responses and the additional experiments. My two main concerns have been addressed.
On the Gaussian background assumption (W1/Q1), the new results in Appendix E.7 (Tables 18 and 19) showing near-homogeneous central noise and a stable ranking under sample-wise scales resolve my concern.
On the threshold (W2/Q2), the reframing of the Table 12 behavior as background noise re-entering aggregation, together with Figure 9 showing that follows the convention while aligning with counterfactual deletion effects, answers my concern that SNR-LDS might reproduce the reweighting dependence of LDS.
I am raising my rating from 2 to 3.
Dear Reviewer, Thank you very much for engaging with our response and raising your score.
Thank you for your valuable review and suggestions. Below we try to respond to the comments in Weaknesses (W) and Questions (Q).
W1&Q1: Shared noised scale
To clarify, SNR-LDS does not assume that measurement errors across all samples strictly follow a homoscedastic Gaussian distribution; rather, it uses the zero-mean homoscedastic Gaussian as an approximate central background working model for the near-zero region to locate the boundary where high-influence samples stand out from background error.
First, the central region of the score distribution is dominated by background measurement error. On diffusion models, if the near-zero central region contained Gaussian-distributed true signals, full-sum linear aggregation (vanilla LDS) would be mathematically optimal, and truncating or downweighting the central region via squaring would lose information and lower correlation. However, across all evaluated platforms, excluding or downweighting the near-zero central region substantially improves subset-prediction correlation (Figure 2), confirming that the central bulk contains negligible true attribution signal and is instead dominated by measurement error.
Second, we model this central background error as approximately homoscedastic Gaussian because, for background samples unrelated to a given query, the observed score sums many nearly independent zero-mean fluctuations, which naturally converges to a zero-mean Gaussian in the central region by the central limit theorem (matching the bell-shaped central peak in Figure 1(a)). Moreover, because these unrelated samples have no special gradient alignment with the query, they experience the query's stochastic fluctuations homogeneously and share an approximately uniform scale σ within the central region.
Furthermore, to test whether noise varies across samples, we independently repeat the four core methods 8 times on CIFAR-2 FM and measure each sample's noise standard deviation si across influence quantiles (normalized by the central-window noise level sc):
0–50%50–80%80–90%90–95%95–98%98–99%99–100%As shown in the table, the sampling noise in the central region is nearly flat across all four methods, supporting our use of a single background scale within the central window.
Finally, although noise increases in the high-influence tail outside the central region, this heteroscedasticity does not undermine SNR-LDS. First, σ^q is fitted only within the central window, where the noise profile is flat. Second, because tail samples have larger noise, accounting for tail heteroscedasticity simply raises their individual thresholds slightly and removes a few borderline samples near the threshold. When we replace the shared sampling variance in σ^q2 with each sample's measured variance si2 to form a sample-wise noise scale σi at ζ=3, both the SNR-LDS scores and the method ranking remain virtually unchanged:
We have added this verification to Appendix E.7 (Tables 18 and 19) of the revised paper.