- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates an efficient way for filtering a LLM post-training dataset using a test-task-aware query set. It extends the exiting method, LESS, to use the gradients at the final layer only, which can be more efficiently computed than those at the whole model. The efficiency is evaluated in FLOPs and wall clock settings, and the performance is demonstrated in SFT and RL.
Critical strengths
- While some performance decrease exists, the method seems a good choice for tuning a tradeoff between performance and efficiency for LESS-like gradient-based data selection of LLM post-training.
Critical weaknesses
- This paper lacks AI USE STATEMENT, which may be a violation of the conference policy.
- The technical novelty is limited. Using the gradients only at the last layer, instead of all the layer, for data analysis is not novel as the related work subsection cites. Transferring such successes to the LESS method for LLM post-training might not be a large contribution.
Questions for authors
How about AI USE STATEMENT?
Another caveat is that LESSER uses only the base model, while LESS uses Adam-preconditioned LoRA (Hu et al., 2021) gradients at four warmup checkpoints for computational tractability and empirical improvements.
It would be nice to add apple-to-apple comparison between LESSER and LESS for clarifying the claimed difference and the improvement.
The effectiveness or ineffectiveness of LESSER seems different depending on tasks. This makes me to worry about the generality of the effectiveness. How do you think why such a large variance exists?
5 comments
Q1. Please see W1.
Q2. Thank you for suggesting this; it is an interesting experiment to run. We first note that the Adam-preconditioned LoRA gradients at four warmup checkpoints are exactly the machinery that LESSER removes, contributing to our cost savings.
As a direct apples-to-apples comparison, we consider LESS with its LoRA gradients computed at the base model at initialization, without warmup checkpoints and without Adam. We call this LESS-init. We report results at k=1,000 and k=5,000 on two models and five benchmarks, averaged over three seeds:
| Model, task | k=1,000: LESSER | LESS-init | LESS | k=5,000: LESSER | LESS-init | LESS |
|---|---|---|---|---|---|---|
| Llama-3.2-3B, TyDiQA | 63.3 | 61.6 | 66.4 | 64.3 | 63.8 | 66.3 |
| Llama-3.2-3B, MMLU-Pro | 22.1 | 12.8 | 22.8 | 26.9 | 27.2 | 26.1 |
| Llama-3.2-3B, GSM8K | 32.3 | 32.1 | 34.2 | 34.6 | 34.9 | 34.5 |
| Llama-3.2-3B, Codex | 61.3 | 56.3 | 56.3 | 57.4 | 55.9 | 58.1 |
| Llama-3.2-3B, BBH | 48.0 | 47.5 | 47.6 | 48.5 | 49.2 | 49.6 |
| Llama-2-7B, TyDiQA | 50.2 | 51.6 | 54.6 | 51.7 | 51.2 | 52.9 |
| Llama-2-7B, MMLU-Pro | 19.7 | 19.7 | 19.2 | 20.2 | 20.6 | 20.2 |
| Llama-2-7B, GSM8K | 19.2 | 20.6 | 19.2 | 21.7 | 20.7 | 21.3 |
| Llama-2-7B, Codex | 29.5 | 26.6 | 25.0 | 29.1 | 28.4 | 24.5 |
| Llama-2-7B, BBH | 38.7 | 39.8 | 40.5 | 41.0 | 41.3 | 42.9 |
We observe that at the same checkpoint, LESSER performs on par with LESS-init. At both budgets, LESSER tracks the performance of LESS-init, while LESS-init scores slightly worse than LESS in some cases. We also find that on MMLU-Pro at k=1,000, LESS-init has a big performance drop on Llama-3.2-3B. This is because the fine-tuned model can’t generate coherent answers. Furthermore, LESS’s lead on TyDiQA drops after we remove the checkpoints.
On cost, LESS-init is faster than LESS since it avoids warmup training and 3 extra backward passes, but it still needs a backward pass per example. The wallclock of LESS-init remains about 4.45x slower than LESSER.
Q3. We wish to point out that task-dependence is a well-known phenomenon of selection methods [1], and is not unique to LESSER. For instance, even established works like LESS and RDS+ also display similar variance (see Tables 9, 10, 11). The broad consensus in the literature attributes this to two main factors: (1) task-headroom and (2) seed variance.
(1) Task headroom: Selection can only matter where fine-tuning on the pool changes the result. Nayak et al. [1] restrict their conclusion about LESS to "target tasks that benefit from additional training (i.e., zero-shot performance is not saturated)" and report that the gains of targeted selection "diminish at larger budgets". Our results show similar findings and reinforce our claims that LESSER is a computationally efficient alternative to LESS. The tasks where LESSER appears less effective are coincidentally also the tasks where every selector also degrades.
(2) Seed variance: Prior work has shown that fine-tuning results vary with the random seed [6] [7]. While we already run multiple seeds for our results, outcomes can still be noisier in certain cases. For example, Codex uses the 164 HumanEval problems scored by sampled pass@10, which gives evaluation numbers that fluctuate (standard deviation 1.5 pp) for LESS and LESSER.
We appreciate the reviewer for surfacing this insightful question, and we will include further discussion of this in the revised manuscript.
References
[1] Nayak et al. "A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)" (ICML 2026)
[2] Xia et al. "LESS: Selecting Influential Data for Targeted Instruction Tuning" (ICML 2024)
[3] Yang et al. "GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning" (arXiv 2026)
[4] Panigrahi et al. "In Good GRACEs: Principled Teacher Selection for Knowledge Distillation" (ICLR 2026)
[5] Ivison et al. "Large-Scale Data Selection for Instruction Tuning" (arXiv 2025)
[6] Madaan et al. "Quantifying Variance in Evaluation Benchmarks" (arXiv 2024)
[7] Dodge et al. "Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping" (arXiv 2020)
[8] Deng et al. "Efficient Forward-Only Data Valuation for Pretrained LLMs and VLMs" (ACL 2026)
[9] Ran et al. "Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation" (NeurIPS 2026)
Dear Reviewer iP4E, as the discussion period ends tomorrow, we would like to check whether our responses have addressed your concerns. We are happy to answer any further questions before the deadline.
Thank you for the thoughtful comments and questions which we address below.
W1. We wish to clarify that we do have an AI usage statement, please see line 1341. We will add a pointer to it from the main text so that it is easier to find.
W2. We wish to clarify our contributions, which extend beyond the surface-level similarity to other works. Specifically, we make novel contributions in our problem application, our technical methodology, and our scientific understanding:
Applications: To our knowledge, we are the first to investigate and demonstrate the effectiveness of output-layer gradients across modern LLM post-training settings. While our related work highlights the use of gradient-based selection for SFT [2], RL [3], and teacher distillation [4], those works are not focused on computationally efficient alternatives. While the closest in methodology are [8][9], they focus on data attribution or valuation, i.e., explaining an already-trained model or filtering out junk data, which is a different focus from targeted selection, which aims to improve downstream performance. As a result, they may omit design choices that selection needs: for example, For-Value [8] does not normalize its features, and normalization is known to be necessary for selection [2].
Methodology: Our methodological contribution is a gradient-based selection feature at the cost of a forward pass. In targeted data selection, gradient-based selectors such as LESS [2] require backpropagation at several warmup checkpoints, and the area has long sought efficient alternatives (e.g., the analysis of Nayak et al. [1]). The efficient alternative so far, RDS+ [5], replaces gradients with hidden states, and it does not behave like LESS: across all 60 settings in our tables, its gains over Random are only weakly related to those of LESS (Spearman = 0.15). LESSER has the same cost as RDS+ and follows LESS much more closely (Spearman = 0.61), while being 17.8x cheaper than LESS. That is, efficiency does not have to come at the cost of the gradient signal.
Understanding: We also explain why output-layer gradients are sufficient (Section 5). Prior work [8][9] judges a feature by how well it scores individual examples, and evaluates it with per-example detection metrics such as AUROC. We find that this is not required. In SFT, LESSER and LESS select largely different examples (Jaccard similarity 0.037) and rank individual examples differently (median Spearman 0.52). Yet their training batches move the model in similar directions: the gradient alignment between LESSER and LESS batches is 0.29, compared with 0.03 for RDS+, and about zero for Random.
References
[1] Nayak et al. "A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)" (ICML 2026)
[2] Xia et al. "LESS: Selecting Influential Data for Targeted Instruction Tuning" (ICML 2024)
[3] Yang et al. "GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning" (arXiv 2026)
[4] Panigrahi et al. "In Good GRACEs: Principled Teacher Selection for Knowledge Distillation" (ICLR 2026)
[5] Ivison et al. "Large-Scale Data Selection for Instruction Tuning" (arXiv 2025)
[6] Madaan et al. "Quantifying Variance in Evaluation Benchmarks" (arXiv 2024)
[7] Dodge et al. "Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping" (arXiv 2020)
[8] Deng et al. "Efficient Forward-Only Data Valuation for Pretrained LLMs and VLMs" (ACL 2026)
[9] Ran et al. "Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation" (NeurIPS 2026)