- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates the common use of last-layer representations as proxies for gradient-based coreset selection and proposes three alternatives: input-level selection (CRAIG-First), validation-guided intermediate-layer selection (CRAIG-FT), and distributed neuron selection based on parameter updates (CRAIG-NI). Its main contribution is a systematic comparison of these proxies within the CRAIG framework, challenging the assumption that the last layer is always the most informative. The authors evaluate their methods on 663 tabular classification and regression datasets, with additional experiments on CIFAR-10, MNIST, and Gemma3-1B instruction tuning on Dolly-15k. Results show that CRAIG-First offers the highest efficiency on tabular tasks, while CRAIG-NI provides a better balance between efficiency and performance on vision tasks and competitive results on instruction tuning
Critical strengths
-
Extensive empirical evaluation: The study evaluates five coreset selection strategies across 663 tabular datasets, providing strong empirical evidence that last-layer proxies are not consistently optimal. This broad evaluation makes the findings more convincing than conclusions drawn from a small number of benchmarks.
-
Practical insights into proxy selection: The paper demonstrates a meaningful trade-off between efficiency and predictive quality, with CRAIG-First performing well on tabular tasks and CRAIG-NI showing promising results on vision and language tasks. These findings challenge a common design assumption and offer useful guidance for developing more effective data selection methods.
Critical weaknesses
-
Limited methodological novelty: The proposed methods are largely heuristic adaptations of existing coreset selection and layer-selection techniques. The paper provides limited theoretical justification for why parameter velocity identifies informative neurons or why the resulting proxy better approximates full-model gradients. This weakens the methodological contribution beyond the empirical comparison.
-
Insufficient validation of practical efficiency and generalizability: Although the tabular evaluation is extensive, the vision and LLM experiments cover only a small number of datasets, architectures. Furthermore, efficiency is measured by sample consumption rather than wall-clock time, overlooking the overhead of proxy construction and selection. These limitations make it difficult to establish whether the proposed methods offer reliable computational advantages in practise.
Questions for authors
-
The paper measures efficiency by cumulative sample consumption rather than wall-clock time. Can the authors provide end-to-end runtime comparisons, including proxy computation and coreset selection overhead, to demonstrate that the proposed methods deliver actual computational savings over random selection and CRAIG-Last?
-
For CRAIG-NI, parameter velocity is used to identify important neurons. Can the authors provide evidence that velocity-based neuron selection consistently improves gradient approximation or coreset quality, compared with random neuron selection or other importance criteria?
2 comments
Reviewer: please respond to the authors, and let them know if your concerns have been addressed.
Response to Reviewer gS5K
We thank the reviewer for the detailed review and the constructive feedback.
W1 (novelty and theory). We view the paper as an analysis of a design choice that most gradient-based selectors make by default (Table 1), tested on 663 datasets. The probes are deliberately simple and matched in budget so that only the location of the proxy changes. We agree that CRAIG-NI has no formal guarantee; it is a heuristic. Its motivation is that parameter velocity is roughly proportional to gradient magnitude (Eq. 3), so the neurons attached to the most-updated weights are expected to contribute most to the CRAIG bound (Eqs. 2 and 4). The ranking can miss neurons whose per-example gradients are large but cancel, which we will note in the Limitations. Its support is empirical, and Q2 below compares it with the other selection rules in the paper.
W2 (vision and LLM). We agree that these are pilot studies, as stated in the Limitations, and will highlight this in the abstract. Within them, NI has a higher E than Last in all 8 vision columns (Table 6), and the high E of Last in the LLM experiment comes from stopping early at a perplexity about 6% above Random (Q = 0.943 vs. 0.983 for NI).
Q1 (runtime). We report the wall-clock time per complete run, including warmup, proxy computation, selection and training (mean over 50 tabular datasets):
Warmup takes 0.28 s for all variants, and the alternatives take 2.2–2.5 times as long as CRAIG-Last, within the same order of magnitude. None of the extra steps (representation extraction, facility location on a proxy of one layer's width, and NI's velocity computation) grows faster with N than facility location itself.
The paper is about data efficiency, that is, how many training samples are needed to reach peak performance, which C measures because the selection routine is shared across variants. In practice, CRAIG and similar methods are also used for one-shot selection, or run on a small proxy model whose subset then trains a larger model (Coleman et al., 2019; Yang et al., 2024). In these, the selection cost is paid once, while the reduction in training samples carries over. We do not claim that CRAIG is faster than Random per run; our claim is that the last layer gives the least informative subsets among the proxies tested. We will add these timings and rename C to "sample cost".
Q2 (velocity vs. other criteria). We evaluate coreset quality directly through the quality, sample cost, and efficiency of models trained on the selected subsets, which is what gradient approximation is meant to serve; measuring the full-model gradient error itself would require the per-example full gradients that proxies are designed to avoid. On these measures, the paper compares velocity-based selection (NI) with four other rules: a randomly chosen layer at each epoch (RL), which serves as the uninformed control, the fixed last or input layer (Last, First), and a validation-chosen layer (FT) (Tables 2 and 5–7). NI has a higher E than both RL and Last in all 16 tabular and 8 vision columns, and it is the only rule that does so while keeping pooled Q ≥ 0.976 in every modality, including the LLM. First and FT reach higher E on tabular data, but First has the lowest quality in regression and falls below Random in three vision columns, and FT stops improving early in instruction tuning. Velocity therefore gives the most consistent results across modalities, at the cost of only two parameter snapshots per epoch.
We hope these responses address the reviewer's concerns and would be grateful if they were taken into account in the final assessment.