- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GRAIN: Gradient-Referenced Alignment of Informative Tokens for LLM Supervised Fine-Tuning is a task aware token-selection method for supervised finetuning. It is motivated by the well studied idea that tokens within a response contribute differently to downstream task performance. It scores tokens by comparing their last hidden state loss gradients against task directions estimated from a small benchmark-specific reference set, that is separate from what they evaluate on.
GRAIN measures how closely each token’s hidden state loss gradient aligns with benchmark reference gradients (from a set of reference tasks). This alignment is gated/attuned by hidden-state similarity, and is eventually combined with a metric where they measure loss decrease relative to the pretrained base model on the same reference set of tasks. After an initial full base SFT warmup (15%), training retains the highest-score tokens within each response, with a default selection rate of 60% (the other 85% is GRAIN). The gradient signal is computed from forward-pass outputs without additional backbone backward passes (a major contribution in my opinion), while reference directions are refreshed periodically and base-model losses are cached offline for easier/cheaper later computation.
The paper provides a theoretical connection of hidden-state alignment to first-order parameter gradient influence. Experiments across three model backbones and seven math, reasoning, and knowledge benchmarks show the highest reported average performance for GRAIN on each backbone, compared with full-token SFT, random token selection, and existing token selector methods from literature. The ablations examine gradient alignment and reference excess loss separately and together, showing the effect of each component. They also cover sensitivity to token selection rates and reference task set size, and computational comparisons.
Critical strengths
-
A practical approach to task-aware token selection. The paper clearly connects gradient similarity to token-level supervision. Computing the hidden state gradient from forward pass outputs avoids expensive full token calculations, and is a neat contribution. The theoretical discussion provides a sound justification of the methods, and I appreciate its inclusion.
-
Promising initial results across the three model backbones. GRAIN achieves the highest reported average performance for each backbone, and the component ablation supports combining gradient alignment with reference excess loss. Its computational benefit needs an appropriate comparison: GRAIN takes 107 minutes, versus 83 for full-token SFT, 104 for Rho-1, and 92 for ssToken. The contribution is making task-aware gradient scoring practical, rather than making SFT faster overall. But it is in line with Rho-1 and ssToken, and this is practically good.
Critical weaknesses
-
The gains need stronger evidence of repeatability and a clearer explanation of when the method works. Appendix D.3 reports a single training run with seed 42. GRAIN’s average improvement over the strongest other SFT method is 1.24 percentage points on LLaMA-3.2-3B, 0.41 on LLaMA-3.1-8B, and 0.55 on Qwen2.5-7B. Without multiple training seeds, it is difficult to assess the reliability of the smaller differences. The ablations also leave important design choices under-explained in my opinion. Appendix A suggests that coherent gradient directions make structured reasoning tasks more suitable, but does not empirically establish this boundary. On ASDiv, alignment-only scores 10.25, versus 55.08 for REL-only and 58.02 for GRAIN; explaining this gap would help me understand the reason for this transfer behavior. GRAIN also uses full-token SFT for the first 15% of training, but neither removes this warmup nor varies its duration, leaving its importance to the final results unclear.
-
The reference-set construction leaves a cross-benchmark overlap concern unresolved. The authors describe separate splits and explicit exclusion measures, which I acknowledge and appreciate. However, GRAIN uses GSM-Plus examples to construct a reference direction while evaluating on the full GSM8K test set. GSM-Plus contains perturbations of GSM8K test questions, so excluding identical question strings does not establish separation from the evaluation task/questions. Leakage seems to be very possible here (https://huggingface.co/datasets/qintongli/GSM-Plus/blob/main/README.md). Using development examples for targeted task selection is reasonable. The concern is specifically whether variants of evaluation questions influence training-time selection of tokens. The paper does not establish whether its exclusions address this relationship or what the overlap in the reference and evaluation sets might be.
Questions for authors
The questions below are listed in order of importance to my assessment. I recognize that 2 and 3 may be completable in a faster fashion.
-
Reliability across seeds. Can you report results across at least three independent training seeds for GRAIN and the strongest baselines re: the mean and standard deviation? This is particularly relevant to the average gains of 0.41 percentage points on LLaMA-3.1-8B and 0.55 on Qwen2.5-7B.
-
Reference-set overlap. Does your exclusion procedure check originating GSM8K question IDs across benchmarks, rather than only exact question matches? Since GSM-Plus contains variants of GSM8K test questions, please clarify the overlap and whether the main results and affected ablations hold when the GSM-Plus reference direction is more deeply explored in reference to overlap with the evaluation set.
-
Scope and transfer behavior. What evidence supports gradient coherence as the explanation for where GRAIN works well? In particular, what causes alignment-only ASDiv performance of 10.25, compared with 55.08 for REL-only and 58.02 for GRAIN? Does this reflect reasoning performance, answer formatting, or another effect of token selection? Could you dive deeper into this?
-
Dependence on warmup. How sensitive are the results to the initial 15% full-token SFT warmup? Have you tested shorter warmup periods or immediate token selection, and do the gains persist? I would like an ablation on this if possible.
4 comments
Response 2/2
Q3: Align-only on ASDiv and gradient coherence
Response: Thank you for this question. Align-only nearly collapses on ASDiv alone while staying close to GRAIN on GSM8K and GSM-Plus, which points to an evaluation-level failure rather than weaker reasoning, so we inspected its generations.
Diagnosis. After answering, Align-only often keeps generating invented "Given the following problem..." blocks, and the ASDiv scorer takes the last number, so a correct first answer is overwritten. Scoring only the first answer brings Align-only to GRAIN's level while REL-only and GRAIN barely change, so the gap reflects termination rather than reasoning (revised Appendix D.7).
Table: ASDiv accuracy on LLaMA-3.2-3B with the default scoring and with only the first answer scored.
| Variant | Default scoring (last match) | First answer only |
|---|---|---|
| Align-only | 10.25 | 57.58 |
| REL-only | 55.08 | 55.04 |
| GRAIN | 58.02 | 58.06 |
Role of REL and gradient coherence. We hypothesize that REL keeps end-of-answer tokens to which alignment gives little credit. Gradient coherence remains a hypothesized favorable condition rather than an established boundary, and our new measurement is consistent with it: alignment agrees with parameter-gradient similarity on reasoning tasks and hardly at all on MMLU (revised Appendix D.11). We will state this in Appendix A.
Q4: Sensitivity to the full-token warmup
Response: Thank you for the question. Like the reference-model training in Rho-1 and Token Cleaning, GRAIN starts with a short full-token warmup so that REL becomes informative. To test how much the results depend on it, we varied the warmup from 0% to 15% on LLaMA-3.2-3B. With 0% warmup, REL is zero at the start, so selection initially reduces to Align-only and REL takes effect as the model moves away from the base model. Shorter warmups cause only small drops, and even without warmup GRAIN still outperforms Full Tokens and Rho-1 (revised Appendix D.8). Meanwhile, a moderate warmup gives REL a better starting point and yields the best results, which is why we use 15% by default.
Table: Effect of the full-token warmup. LLaMA-3.2-3B, seed 42.
| Method | MMLU | BBH | GSM8K | MathQA | GSM-Plus | ASDiv | Minerva | AVG |
|---|---|---|---|---|---|---|---|---|
| GRAIN, 0% warmup | 56.74 | 48.12 | 30.98 | 37.03 | 18.95 | 55.28 | 7.35 | 36.35 |
| GRAIN, 5% warmup | 56.82 | 48.23 | 31.18 | 37.25 | 19.34 | 57.64 | 7.21 | 36.81 |
| GRAIN, 10% warmup | 56.91 | 48.35 | 31.21 | 37.22 | 19.47 | 57.87 | 7.55 | 36.94 |
| GRAIN, 15% warmup (default) | 56.86 | 48.30 | 31.32 | 37.31 | 19.65 | 58.02 | 7.68 | 37.02 |
| Full Tokens | 56.72 | 47.96 | 29.33 | 36.06 | 16.67 | 55.00 | 5.48 | 35.32 |
| Rho-1 | 56.74 | 48.15 | 29.94 | 36.64 | 17.55 | 57.27 | 4.19 | 35.78 |
Thank you again for your valuable feedback, which has helped us significantly improve the paper. We hope these responses address your concerns, and we would be happy to clarify any remaining questions during the discussion phase.
Thank you for the detailed response and additional experiments. These address the main concerns behind my original recommendation.
-
Repeatability and overlap: The three-seed comparisons strengthen the results. I particularly appreciate the acknowledgment of source-level overlap, its quantification, and the retraining without GSM-Plus references. This directly addresses my central concern. Please keep the distinction between these corrected main results and ablations using the original reference configuration clear.
-
ASDiv and mechanism: The generation analysis explains the alignment-only collapse much more convincingly and clarifies the practical contribution of REL. I agree with retaining gradient coherence as a hypothesis rather than an established explanation of where GRAIN works.
-
Warm-up and cost: The zero-warm-up and matched-time experiments address the concern that the gains might mainly reflect initial full-token training or additional compute. The contribution is clearer as making task-aware token scoring practical.
These additions sufficiently resolve my central objections, and I am raising my recommendation to a 3. The remaining limitations concern generalization beyond the tested setting and the strength of the mechanistic interpretation. Thank you for engaging carefully with the review. I appreciate your time and effort.
Thank you for taking the time to carefully consider our responses and for your thoughtful suggestions. We are grateful for your positive impression and support for our work. Your encouraging comments mean a lot to us, and we deeply appreciate your recognition of our efforts.
In the final version, we will clearly separate the corrected main results from the ablations using the original reference configuration, keep gradient coherence framed as a hypothesis, and discuss generalization and the scope of the mechanistic interpretation in the limitations.
Response 1/2
We sincerely thank Reviewer mpgK for the constructive review. We address each concern below.
W1: Repeatability and when the method works
Response: Thank you for raising these points. The concern has three parts: whether the gains are repeatable, why Align-only fails on ASDiv, and how much the full-token warmup matters. We answer them with new experiments in Q1, Q3 and Q4, respectively.
W2 & Q2: Overlap between GSM-Plus references and GSM8K
Response: Thank you for raising this concern and for the question. The key question is whether variants of GSM8K test questions in our reference set steer token selection toward the evaluation, so we first quantify the overlap and then remove it. Overlap. Our original exclusion matched question text only, so the "stable-id exclusion" in criterion (D1) was inaccurate. Tracing source questions, the GSM-Plus reference items come from 96 GSM8K test questions (7.35% of those evaluated); no other reference family overlaps with an evaluation set, and the Table 1 baselines never use the reference set.
Removing the overlap. We removed the GSM-Plus family from the reference set and retrained GRAIN with three seeds (Table 1). GRAIN still has the best GSM8K score and the best average on every model, so its gains do not come from the overlapping questions. We have corrected (D1) and added the audit to revised Appendix D.6.
Table: GRAIN retrained without GSM-Plus references. Three-seed means from Table 1; best baseline per column.
Re-scoring the original checkpoints. Excluding the 96 questions from the original evaluation gives the same picture: GRAIN stays ahead of all baselines on the overlap-free questions, its average barely changes, the affected ablations keep their ranking, and on the overlapping questions Rho-1 and Token Cleaning even score higher than GRAIN.
Table: GSM8K re-scoring with the original checkpoints. LLaMA-3.2-3B, seed 42; accuracy (%) on the 96 overlapping and the 1,210 overlap-free questions, and the original seven-benchmark average before and after replacing GSM8K by its overlap-free score.
Q1: Multiple training seeds
Response: Thank you for the question. Single-seed margins of about 0.4–0.6 points on the larger models could fall within seed variance, so we retrained GRAIN and all baselines with three seeds (42, 123, 3407); GRAIN uses the reference set without GSM-Plus (see W2). GRAIN has the highest mean AVG on all three models, and its margin over the strongest baseline exceeds the seed-to-seed standard deviation on every model. We have updated Table 1 to report three-seed means (revised Appendix D.4).
Table: AVG over three training seeds (mean ± std).