- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper discusses gradient-based data selection methods such as CRAIG and GradMatch. It aims to show that these methods do not actually identify informative samples needed for learning the function, but are more biased towards the model's parameters. To prove this, the paper applies function-preserving reparameterizations (for example, scaling RELU) to models. So while these are computationally invisible (the models still return the same result), the paper shows that the subsets selected using these models and gradient-based data selection varied. The accuracies of these models trained on these subsets remain similar, however. So the paper argues that data "identifiability" and "downstream utility" are fundamentally disconnected.
Critical strengths
The paper is easy to follow and uses mathematical theorems to prove points where needed. It discusses the difference between picking specific data points and actually training a successful model. Discussing this difference is important because data efficiency is a major topic in machine learning right now.
Critical weaknesses
The paper has a few key weaknesses to consider. For example, the paper assumes that because the algorithms select completely different data when the weights change, the methods must be flawed. But in redundant datasets, it could be possible that different data subsets form the same decision boundary. This point needs to be explored and discussed. Along the same lines, the paper studies gradient-based selection methodologies, which are approximations. So, while discussing the difference between data points and training a model, there should be a discussion with regard to true data estimation methods such as true influence functions that require the Hessian inverse at optimal weights. Additionally, many of the experimental results include simple datasets like MNIST or Fashion-MNIST, which are very simple and don't reflect complex real-world situations.
Questions for authors
- Since different data subsets can easily form the same decision boundary without hurting the final accuracy, is the selection method actually flawed, or is it simply picking equally useful examples from redundant datasets?
- Isn't it already obvious that data selection changes when the model's weights change, considering that these gradient methods are mathematically built entirely on those weights?
- Because these gradient methods are only approximations, how do they actually compare to true data estimation methods, such as true influence functions that require the Hessian inverse at optimal weights?
- Does this problem actually affect the massive generative language models widely used today? Could there be some empirical evidence added to the paper?
4 comments
Response to Reviewer NXSC (2/2)
W2 / Q3. Gradient methods are approximations; how do they compare with true influence functions that use the inverse Hessian at the optimum?
Short answer. Exact influence is on the invariant side of our characterization, with two caveats, and we have now tested it directly: exact-influence selections do not change, while damped influence and gradient similarity do.
Details. For a loss invariant under a linear reparameterization with Jacobian , , so exact influence is invariant, whereas raw gradient selectors use the identity in place of . The two caveats: at a minimum with a nonzero symmetry direction, is singular; and a damped inverse is invariant for all gradient sets only if is orthogonal.
The experiment (App. L.11) uses the classifier layer of the five MNIST and five Fashion-MNIST acquisition models (650 parameters; exact float64 Hessian at the trained weights). We selected the 50 candidates with the largest score , where is the held-out validation-loss gradient and is , or , before and after cross-layer equalization and unit rescalings. Replacement (%):
| Score | MNIST, CLE | MNIST, σ=0.5 | Fashion, CLE | Fashion, σ=0.5 |
|---|---|---|---|---|
| Exact influence, | 0 | 0 | 0 | 0 |
| Damped, , three | 2.4–3.2 | 9.6–13.2 | 2.8–4.0 | 10.7–14.6 |
| Gradient similarity, | 0.4 | 6.3 | 2.4 | 10.2 |
Exact-influence selections are unchanged (exact score invariance also needs gradients in the range of ). The Hessian is singular in every model (numerical nullity 101–173 of 650, including the 65 softmax-shift directions), hence the pseudo-inverse. On the original models, exact influence and gradient similarity share 18–42% of their selections.
Changes in the paper. New paragraph "Exact influence versus gradient similarity" in Sec. 7 (ll. 486–496; proofs in App. L.9; experiment in App. L.11, Table 31). It ends:
On our MNIST and Fashion-MNIST classifier layers, exact (pseudo-inverse) influence selects the same examples under CLE and rescalings, while damped influence and gradient similarity change up to 15% on average (Appendix L.11).
W3 / Q4. Many results use simple datasets such as MNIST and Fashion-MNIST; does the problem affect large generative language models?
Short answer. The revision adds cross-layer equalization on the CIFAR-10 ConvNet and a decoder-only generative language model (Qwen2.5-0.5B with LoRA); both show the same pattern. We do not extrapolate to much larger models.
Details. Beyond the vision benchmarks, the paper fine-tunes LoRA end to end in MiniLM (three ranks) and ModernBERT-base on SNLI (23 trained instances), now with a standard rebalancing operation and float32 write-back. The new decoder has rank-8 LoRA on every attention query and value projection, with SNLI posed as next-token prediction of label words (five instances). Under LoRA rebalancing (Table 2, % replaced):
| Model | BADGE, inverse-CDF | BADGE, keyed | Deterministic | Balanced / merged |
|---|---|---|---|---|
| MiniLM, r = 4 / 8 / 16 | 43.9 / 31.4 / 37.0 | 4.2 / 3.2 / 4.2 | 13.2 / 9.2 / 10.0 | 0 |
| ModernBERT-base, r = 8 | 56.8 | 1.8 | 3.8 | 0 |
| Qwen2.5-0.5B (decoder), r = 8 | 50.2 | 7.9 | 15.2 | 0 |
In the decoder, as in the encoders, orthogonal basis changes leave raw selection unchanged. On the CIFAR-10 ConvNet, cross-layer equalization replaces 57.8% (inverse-CDF), 2.6% (keyed), 12.0% (CRAIG) and 23.6% (GradMatch), and 0% with QuotientGrad. The LoRA factor symmetry and the bound in Proposition 3 do not depend on model or pool size.
Changes in the paper. Sec. 6.1 (ll. 417–422) and Table 2; App. L.12; CIFAR-10 column of Table 1 and App. L.13; scope in Limitations (ll. 515–517). Sec. 6.1 ends (ll. 455–456):
The decoder shows the same pattern in a small generative model; we do not extrapolate to larger ones.
If these answers address your concerns, we would be grateful if you would consider raising your score. If any concern remains that affects your rating, please let us know which, and we will address it during the discussion period.
I thank the authors for providing this information. This answers my questions. I will revise my scores accordingly.
Thank you
Thank you for reading our responses and for letting us know. We are glad they answer your questions.
Response to Reviewer NXSC (1/2)
Thank you for your review. You found the paper "easy to follow", using "mathematical theorems to prove points where needed", and considered its distinction between picking specific data points and training a successful model important "because data efficiency is a major topic in machine learning right now". Your questions on influence functions (Q3) and on generative language models (Q4) led to two new experiments (App. L.11, L.12).
W1 / Q1. Different subsets of redundant data can form the same decision boundary; is the selection method flawed, or is it picking equally useful examples?
Short answer. Our evidence is consistent with the latter, and the revision says so directly; we do not claim that the selectors are flawed.
Details.
So, as you suggest, equivalent representations pick different examples without a consistent loss of accuracy. The paper's claim is narrower: which examples are picked is not determined by the predictor. This matters when the identities themselves are used, e.g. to order annotations, to audit a selection, or to reproduce a query from a stored checkpoint; we did not measure those costs (Limitations, ll. 518–519).
Changes in the paper. The introduction (ll. 73–74) now says that "different subsets of redundant data can be equally useful". Sec. 6.2 concludes (ll. 476–478):
Q2. Isn't it already obvious that data selection changes when the model's weights change?
Short answer. That the gradients change is standard, and we label it as background (Remark 1). Whether the selection changes, and by how much, does not follow from it.
Details. The reparameterizations preserve the predictor analytically; in the vision rescaling audits, the maximum logit difference is 1.3×10−13 (Figure 2). Three findings are not implied by the gradient change:
Changes in the paper. The introduction calls the gradient change "a familiar fact in invariant optimization" and then separates the steps (ll. 71–73):