acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Last Layer: Rethinking Gradient Proxies for Coreset Selection

Abstract

Gradient-based coreset selection offers a theoretically rigorous framework for data-efficient learning; however, its computational cost necessitates practical approximations. To scale these methods, existing heuristics, such as in CRAIG, approximate full gradient similarity by restricting gradient computation to the final layer, implicitly assuming that the penultimate representation suffices for data selection; a phenomenon we refer to as the Last-Layer Bias. In this work, we critically examine this assumption and investigate how the informative gradient signal is distributed across network depth. We propose a family of alternative gradient proxies that exploit information beyond the final layer, including input-level gradients (CRAIG-First), data-driven intermediate layer selection via FlexTuning (CRAIG-FT), and a distributed proxy based on Global Node Importance (CRAIG-NI), which selects sparse, non-contiguous subsets of high-activity neurons. We conduct one of the largest empirical studies in the coreset literature, evaluating 663 tabular classification and regression datasets, and extend our analysis to image classification and instruction tuning in large language models. Our results show that last-layer proxies often exhibit diminishing returns and barely outperform random selection (CRAIG-RL). In contrast, input-level proxies consistently accelerate convergence on tabular datasets, while distributed proxies such as CRAIG-NI generalize robustly across vision and language tasks, with particularly strong gains on visual benchmarks, effectively mitigating the Last-Layer Bias.

Primary area:
optimization
Keywords:
Coresets, CRAIG, Data Efficient ML, Data Selection, Efficient Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gS5KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the common use of last-layer representations as proxies for gradient-based coreset selection and proposes three alternatives: input-level selection (CRAIG-First), validation-guided intermediate-layer selection (CRAIG-FT), and distributed neuron selection based on parameter updates (CRAIG-NI). Its main contribution is a systematic comparison of these proxies within the CRAIG framework, challenging the assumption that the last layer is always the most informative. The authors evaluate their methods on 663 tabular classification and regression datasets, with additional experiments on CIFAR-10, MNIST, and Gemma3-1B instruction tuning on Dolly-15k. Results show that CRAIG-First offers the highest efficiency on tabular tasks, while CRAIG-NI provides a better balance between efficiency and performance on vision tasks and competitive results on instruction tuning

Critical strengths
  1. Extensive empirical evaluation: The study evaluates five coreset selection strategies across 663 tabular datasets, providing strong empirical evidence that last-layer proxies are not consistently optimal. This broad evaluation makes the findings more convincing than conclusions drawn from a small number of benchmarks.

  2. Practical insights into proxy selection: The paper demonstrates a meaningful trade-off between efficiency and predictive quality, with CRAIG-First performing well on tabular tasks and CRAIG-NI showing promising results on vision and language tasks. These findings challenge a common design assumption and offer useful guidance for developing more effective data selection methods.

Critical weaknesses
  1. Limited methodological novelty: The proposed methods are largely heuristic adaptations of existing coreset selection and layer-selection techniques. The paper provides limited theoretical justification for why parameter velocity identifies informative neurons or why the resulting proxy better approximates full-model gradients. This weakens the methodological contribution beyond the empirical comparison.

  2. Insufficient validation of practical efficiency and generalizability: Although the tabular evaluation is extensive, the vision and LLM experiments cover only a small number of datasets, architectures. Furthermore, efficiency is measured by sample consumption rather than wall-clock time, overlooking the overhead of proxy construction and selection. These limitations make it difficult to establish whether the proposed methods offer reliable computational advantages in practise.

Questions for authors
  1. The paper measures efficiency by cumulative sample consumption rather than wall-clock time. Can the authors provide end-to-end runtime comparisons, including proxy computation and coreset selection overhead, to demonstrate that the proposed methods deliver actual computational savings over random selection and CRAIG-Last?

  2. For CRAIG-NI, parameter velocity is used to identify important neurons. Can the authors provide evidence that velocity-based neuron selection consistently improves gradient approximation or coreset quality, compared with random neuron selection or other importance criteria?

2 comments
AuthorsOpenReview ↗

Response to Reviewer gS5K

We thank the reviewer for the detailed review and the constructive feedback.

W1 (novelty and theory). We view the paper as an analysis of a design choice that most gradient-based selectors make by default (Table 1), tested on 663 datasets. The probes are deliberately simple and matched in budget so that only the location of the proxy changes. We agree that CRAIG-NI has no formal guarantee; it is a heuristic. Its motivation is that parameter velocity is roughly proportional to gradient magnitude (Eq. 3), so the neurons attached to the most-updated weights are expected to contribute most to the CRAIG bound (Eqs. 2 and 4). The ranking can miss neurons whose per-example gradients are large but cancel, which we will note in the Limitations. Its support is empirical, and Q2 below compares it with the other selection rules in the paper.

W2 (vision and LLM). We agree that these are pilot studies, as stated in the Limitations, and will highlight this in the abstract. Within them, NI has a higher E than Last in all 8 vision columns (Table 6), and the high E of Last in the LLM experiment comes from stopping early at a perplexity about 6% above Random (Q = 0.943 vs. 0.983 for NI).

Q1 (runtime). We report the wall-clock time per complete run, including warmup, proxy computation, selection and training (mean over 50 tabular datasets):

MethodTotal (s)
CRAIG-Last10.29
CRAIG-RL22.36
CRAIG-First23.47
CRAIG-NI25.53
CRAIG-FT25.82

Warmup takes 0.28 s for all variants, and the alternatives take 2.2–2.5 times as long as CRAIG-Last, within the same order of magnitude. None of the extra steps (representation extraction, facility location on a proxy of one layer's width, and NI's velocity computation) grows faster with N than facility location itself.

The paper is about data efficiency, that is, how many training samples are needed to reach peak performance, which C measures because the selection routine is shared across variants. In practice, CRAIG and similar methods are also used for one-shot selection, or run on a small proxy model whose subset then trains a larger model (Coleman et al., 2019; Yang et al., 2024). In these, the selection cost is paid once, while the reduction in training samples carries over. We do not claim that CRAIG is faster than Random per run; our claim is that the last layer gives the least informative subsets among the proxies tested. We will add these timings and rename C to "sample cost".

Q2 (velocity vs. other criteria). We evaluate coreset quality directly through the quality, sample cost, and efficiency of models trained on the selected subsets, which is what gradient approximation is meant to serve; measuring the full-model gradient error itself would require the per-example full gradients that proxies are designed to avoid. On these measures, the paper compares velocity-based selection (NI) with four other rules: a randomly chosen layer at each epoch (RL), which serves as the uninformed control, the fixed last or input layer (Last, First), and a validation-chosen layer (FT) (Tables 2 and 5–7). NI has a higher E than both RL and Last in all 16 tabular and 8 vision columns, and it is the only rule that does so while keeping pooled Q ≥ 0.976 in every modality, including the LLM. First and FT reach higher E on tabular data, but First has the lowest quality in regression and falls below Random in three vision columns, and FT stops improving early in instruction tuning. Velocity therefore gives the most consistent results across modalities, at the cost of only two parameter snapshots per epoch.

We hope these responses address the reviewer's concerns and would be grateful if they were taken into account in the final assessment.

Reviewer: please respond to the authors, and let them know if your concerns have been addressed.

Reviewer_pax9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the Last-Layer Bias, a term introduced by the authors to question whether the commonly used last-layer representation is a suitable proxy for selecting informative training examples in gradient-based coreset selection. Using CRAIG as a common framework, the authors develop several strategies—CRAIG-First, CRAIG-RL, CRAIG-FT, and CRAIG-NI—to systematically compare the roles of representations from individual layers and across layers. The study examines whether useful selection signals are concentrated in particular network layers or distributed throughout the network. In particular, CRAIG-NI constructs a distributed proxy using neurons associated with large parameter updates across multiple layers.

The authors conduct an extensive evaluation on 663 tabular datasets and extend their analysis to image classification and LLM instruction tuning. The findings suggest that relying exclusively on the last-layer proxy may not always be the best choice, and that different tasks may benefit from different proxy representations. As an additional methodological exploration, CRAIG-First achieves high sample efficiency on tabular tasks, while CRAIG-NI offers a favorable quality–efficiency trade-off on vision tasks and shows some promise in language tasks.

Critical strengths
  1. Clear presentation and organization. The paper is well written and logically structured. Its motivation, methodological design, and experimental results are presented clearly, making the work easy to follow.
  2. Extensive empirical evaluation. The authors evaluate their approaches on 663 tabular datasets covering both classification and regression tasks. Additional evaluations on vision and language tasks further broaden the scope of the study. The substantial experimental effort provides valuable empirical evidence.
  3. Meaningful problem identification and practical solutions. The paper questions the commonly used last-layer approximation in CRAIG and systematically investigates alternative proxy selection strategies. The results show that some alternatives achieve higher sample efficiency while maintaining competitive predictive performance.
  4. Potential broader implications. The investigation of layer selection and distributed gradient proxies provides useful insights beyond the CRAIG framework. Although the evaluation primarily focuses on CRAIG, the underlying ideas may inspire future research on gradient-based data selection and related optimization methods.
Critical weaknesses
  1. Some limitations in methodological novelty. Several strategies extend existing ideas, such as using early-layer representations, selecting intermediate layers, or aggregating information across layers. Similar directions have been explored in data influence estimation and related areas. Nevertheless, a key contribution of this paper is that it brings these layer-selection strategies together within a unified framework and systematically examines their differences through a large-scale empirical study. This comparative analysis is itself a valuable research contribution.
  2. The methodological justification could be strengthened. Although the empirical results support the effectiveness of alternative proxies, the relationship between the selected representations and the original gradient-matching objective is not fully established. In particular, the rationale for using parameter-update magnitude to identify informative neurons would benefit from further explanation.
Questions for authors

Q1. Guidance on proxy selection for new tasks (Sections 6–7).

The results suggest that different proxy strategies work best for different types of tasks. Could the authors offer more concrete guidance, based on these findings, on when to favor CRAIG-First, CRAIG-FT, or CRAIG-NI for a new dataset or model?

Q2. Rationale for parameter-update magnitude in CRAIG-NI (Section 4.3, Equations 3–4).

Why are neurons associated with larger parameter updates expected to provide more informative proxies for coreset selection? Could this criterion overlook useful information from parameters with small net updates but substantial differences in per-example gradients?

Q3. Fixed layer selection in CRAIG-FT (Section 4.2 and Appendix E).

CRAIG-FT uses FlexTuning to select a layer after warmup and then keeps that layer fixed throughout the subsequent training process. Since layer representations may evolve during training, could the authors explain the rationale for selecting a layer only once and discuss when this fixed strategy might become less effective?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pax9

We thank the reviewer for the positive and constructive review.

W1 (novelty). We agree with the reviewer's assessment. What we consider new is the comparison of proxy locations under a fixed selection rule, a proxy that is not tied to a single layer (CRAIG-NI), the evaluation on 663 datasets, and the finding that the best proxy depends on the modality.

W2 and Q2 (parameter-update magnitude). Parameter velocity is roughly proportional to gradient magnitude (Eq. 3), so the neurons attached to the most-updated weights are expected to account for most of the gradient-matching error in the CRAIG bound (Eqs. 2 and 4). The coreset objective (Eq. 1) matches the aggregate gradient over the training set, and parameters whose per-example gradients largely cancel contribute little to that aggregate. Ranking by net update therefore focuses the proxy on the directions along which training actually moves the model, and it needs only two parameter snapshots and no per-example gradients. Empirically, at the same budget, NI has a higher E than both Last and RL in all 16 tabular and 8 vision columns. Ranking neurons by the variance of their per-example gradients would complement velocity for parameters with large but cancelling gradients, and we see it as a natural extension for future work.

Q1 (guidance for new tasks). Our results suggest choosing the proxy according to the data and the depth of the model. For shallow models on tabular or other low-dimensional features, CRAIG-First gives the largest sample savings, while CRAIG-FT or CRAIG-NI are preferable when final quality matters more than sample savings (Tables 2 and 5). For deep or pretrained models, a distributed proxy such as CRAIG-NI gives the best balance of quality and cost (Table 6 for vision and Table 7 for instruction tuning). On tabular and vision tasks, the last layer is never the most efficient choice. We will add this guidance to Section 7.

Q3 (fixed layer in CRAIG-FT). CRAIG-FT tests whether one fixed layer is sufficient; dynamic selection is covered by NI, which re-ranks neurons every epoch, and by RL. Re-running the probe every epoch would add a full-data epoch and L validation evaluations each time. Our results agree with the reviewer's intuition. On tabular MLPs, the fixed FT layer outperforms NI (E of 2.19 vs. 1.82), whereas on fine-tuned deep models NI performs better (1.196 vs. 1.106 in vision), and FT stops improving early in instruction tuning (Q of 0.952 vs. 0.983). A fixed layer is likely to be less effective with longer training and with deeper or pretrained models, where later checkpoints differ more from the warmup model. Periodic re-probing is a possible remedy, which we will discuss in Section 7.

Reviewer: while your initial assessment leans positive, please do still respond to the authors and let them know if they have addressed your concerns.

Reviewer_EhgwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • This paper studies whether the last-layer representation is the best choice for coreset selection algorithms that use gradients as a proxy. The authors term it as “Last-Layer Bias”.
  • The paper explores how effective gradient signals are from layers at various depths.
  • The paper uses CRAIG as the coreset selection algorithm and compares against first-layer activations (CRAIG-First), randomly sampled layer representation (CRAIG-RL), validation-guided FlexTuning-based selection of a fixed layer (CRAIG-FT), and construction of a node-importance strategy (CRAIG-NI).
  • The authors report that first-layer selection offers the best sample-efficiency trade-off for tabular tasks, while CRAIG-NI performs competitively on vision and language tasks.
Critical strengths
  1. The problem statement being considered is a reasonable one. Gradients from the penultimate layer are commonly used in gradient-based coreset selection algorithms. Investing representations from which layer (what type of) representation should be used for gradient-based coreset selection is meaningful.
Critical weaknesses
  1. There are quite a few gradient-based coreset selection algorithms published over the years. While CRAIG (published in 2019) is a well-established method, the coreset selection literature has evolved considerably over the years. Quite a few gradient-based methods have been reported to perform much better than CRAIG (GradMatch (2021), LESS (2024), In2Core (2024), G2IS (2025), GIST (2026)). Although the authors include supplementary experiments with a GLISTER-inspired method, these experiments are limited to tabular datasets and do not reproduce the original gradient-based GLISTER formulation. Conducting experiments with only one algorithm severely limits the study's applicability.
  2. MLPs are quite simple architectures. The field has moved on to transformers and foundation models. The applicability of the inferences reported in the paper (based on networks with only 5 layers) is severely limited without experimentation on these modern architectures.
  3. Similarly, MNIST and CIFAR-10 are quite simple datasets. Evaluation should include Tiny ImageNet and ImageNet datasets to ensure the observations scale to larger datasets.
  4. How are the hyperparameters (batch size, accuracy threshold, warmup epochs K, etc.) selected? An ablation study on hyperparameter sensitivity would provide a clearer idea of how these hyperparameters affect downstream task metrics.
  5. The Table 2 caption should include what each of the headings of the experiments represents. It would provide greater clarity in understanding the results.
  6. No theoretical justification is provided for why CRAIG-NI is more impactful.
  7. The efficiency metric does not measure actual computational efficiency. While the cost metric compares the cumulative number of unique samples consumed to reach peak performance between a given method and random selection, a true comparison should have been wall-clock runtime, FLOPs, GPU-hours, or total computational work. Greedy facility-location selection is quite computationally heavy and should be accounted for in the metrics for a fair comparison.
  8. The proposed CRAIG methods include an initial warmup phase on the full dataset before the coreset selection. But as described in Appendix C, no warm-up is involved in random baseline selection. A fair evaluation should include identical steps between the various methods being considered.
  9. Can you specify how multi-class F1 averaging is computed?

Minor

  1. As per the ICLR 2027 guidelines, all citations should be in parentheses, except for cases when the authors or publications are included in the sentence. Throughout the paper, the citations are provided without any parentheses.
Questions for authors
  • Please refer to the weaknesses section of the review.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Ehgw

We thank the reviewer for the detailed review. The paper studies one design choice, namely which part of the network the gradient proxy is taken from, with the selection rule held fixed. We do not propose a new state-of-the-art selector. Our claim is that the last-layer proxy is not a reliable default.

W1 (one algorithm). We use CRAIG because its facility-location step is deterministic once the proxy is fixed, so differences between variants come from the proxy. GradMatch uses the same last-layer approximation, so our findings apply to it directly. In2Core, LESS, G2IS, and GIST already use gradients beyond the last layer (first-k layers or LoRA parameters), which agrees with our results, but none of them compares proxy locations under a fixed selection rule. Appendix M uses a GLISTER-style rule with an activation proxy, as stated there; it tests a different selection rule, and GLISTER-First and GLISTER-FT rank first and second in all 8 columns of Table 8, with Last never performing the best. We will discuss what these results imply for other coreset approaches and add G2IS and GIST to the related work.

W2 (architectures). Besides the MLPs, the paper includes Gemma-3-1B (QLoRA instruction tuning) and an ImageNet-pretrained ResNet-18. MLPs are a standard baseline for tabular data (Gorishniy et al., 2021) and made 663 datasets feasible. For Gemma, the high E of CRAIG-Last comes from stopping early at a best perplexity about 6% above Random (Q = 0.943 vs. 0.983 for NI; Table 7).

W3 (larger vision datasets). We have not performed ImageNet-scale runs due to computational constraints, and our choice of MNIST and CIFAR-10 was due to these datasets being the benchmarks of the original CRAIG paper. NI has a higher E than Last in all 8 columns of Table 6, with Q ≥ 1.005, and we will describe the vision and LLM results as pilot studies in the abstract.

W4 (hyperparameters). All hyperparameters were fixed in advance and shared by all methods (AdamW, learning rate 10⁻³, batch size 32, up to 30 selection epochs after a warmup of at most 3 epochs). NI's budget is set to one layer's width and is not tuned. Coreset size (five values) and architecture (two) are varied, and in all 16 groups of Tables 2 and 5, Last has the lowest E and First the highest. We did not vary the learning rate or warmup length; we will highlight this in the Limitations and add a hyperparameter table to the appendix.

W5 (Table 2 caption). The revised caption will define the column groups (All; Coreset x% = datasets with m = x% of the training set; Arch; n = number of datasets) and the metrics: Q, best test F1 relative to Random (↑); C, unique samples used to reach peak, relative to Random (↓); σ_C, the standard deviation of C; E = Q/C (↑).

W6 (theory). We agree that CRAIG-NI has no formal guarantee, and we present it as a heuristic. Its motivation is the one given in Section 4.3: parameter velocity is roughly proportional to gradient magnitude (Eq. 3), so the neurons attached to the most-updated weights are expected to contribute most to the CRAIG bound (Eq. 2), and NI restricts the proxy to them (Eq. 4). We do not claim that this ranking is optimal; for example, it can miss neurons whose per-example gradients are large but cancel. We support this probe empirically: with the same budget as one layer, NI has a higher E than both Last and RL in all 16 tabular and 8 vision columns.

W7 (compute). We agree that C counts samples rather than compute, as Section 5.2 states. The paper is about data efficiency, that is, how many training samples are needed to reach peak performance. In practice, CRAIG and similar methods are also used for one-shot selection, or run on a small proxy model whose subset then trains a larger model (Coleman et al., 2019; Yang et al., 2024), so the selection cost is paid once while the reduction in training samples carries over. Facility location is identical across our variants, so it does not explain the differences in C. The measured wall-clock time per run (mean over 50 tabular datasets) is 10.3 s for CRAIG-Last and 22.4–25.8 s for the other variants. We do not claim that any CRAIG variant is faster than Random; our claim concerns which proxy gives the most informative subsets. We will rename C to sample cost and add these timings.

W8 (warmup for Random). All methods use the same selection phase (up to 30 epochs with the same m); only the CRAIG variants have the short warmup. Since all CRAIG variants share it, and on each dataset E_a/E_b = (F1_a/S_a)/(F1_b/S_b) does not involve Random, the ranking of proxies is unaffected. The warmup can only help Last relative to Random, and Last still stays near E = 1. We will highlight this in Section 5.1 and Appendix C.

W9 (F1). F1 is computed on the test split with macro averaging, and Q compares all methods under the same averaging.

Minor. We will use \citep throughout.

Reviewer_EhgwOpenReview ↗

Thanks for the rebuttal, but some issues persist

I thank the authors for their detailed rebuttal. While I appreciate the effort to isolate the effect of proxy location while keeping the selection mechanism fixed, there are significant concerns with the paper in its current form, and I will retain my rating.

Alongside my original review, the major bottlenecks of the paper in its current form (including the rebuttal) are:

  1. The argument that GradMatch uses a similar last-layer approximation does not establish that the empirical observations transfer directly to GradMatch. Similarly, without experimental validation on other recent methods, the paper’s specific conclusion remains limited to only one method, CRAIG. That is insufficient to prove the hypothesis proposed in the paper.

  2. The large number of datasets does not compensate for limited architectural diversity and complexity. The vision experiments are restricted to MNIST and CIFAR-10, very primitive datasets by current standards. Therefore, the question of scalability to larger, more challenging visual datasets and to complex architectures remains unanswered.

  3. Hyperparameters influence the representation learned at different network depths. This is specifically relevant to a study whose central argument concerns which layers contain informative signals during training. Fixing hyperparameters in advance limits the scope of the study.

  4. The computational overhead of the proposed method remains substantial. The lack of computational analysis for a data-efficient deep learning study is quite limiting.

Comments on the paper

1 comment

Reviewers: please respond to the authors, and let them know if there are any outstanding concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.