acceptodds
Under review as a conference paper at ICLR 2027

GRAIN: Gradient-Referenced Alignment of Informative Tokens for LLM Supervised Fine-Tuning

Abstract

Token-level selection has emerged as an effective approach to improving data efficiency in supervised fine-tuning (SFT) of large language models (LLMs). Its benefit hinges on the scoring signal: a useful score should reflect how much a token contributes to the target task, yet remain cheap enough to compute during training. Existing selectors score tokens by loss comparisons against reference or historical model states, which capture learning dynamics but do not explicitly encode downstream task objectives. In contrast, gradient-based influence offers a task-aware alternative, yet incurs substantial computational overhead at the token level. To fill this gap, we introduce Grain, an efficient task-aware token selector that approximates gradient-based influence using last-hidden-state gradients computed directly from the forward pass. Specifically, each token is scored by the gated alignment of this gradient with task directions estimated from a small held-out set, complemented by a lightweight excess-loss term computed from cached base-model losses. Together, these components enable online selection without additional backward passes or a separately trained reference model. Theoretical analysis shows that, under stated conditions, the ungated alignment term approximates first-order token-level influence. Extensive experiments demonstrate the superiority of our proposed method over existing token selectors.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Supervised Fine-Tuning, LLM, Token Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mpgKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

GRAIN: Gradient-Referenced Alignment of Informative Tokens for LLM Supervised Fine-Tuning is a task aware token-selection method for supervised finetuning. It is motivated by the well studied idea that tokens within a response contribute differently to downstream task performance. It scores tokens by comparing their last hidden state loss gradients against task directions estimated from a small benchmark-specific reference set, that is separate from what they evaluate on.

GRAIN measures how closely each token’s hidden state loss gradient aligns with benchmark reference gradients (from a set of reference tasks). This alignment is gated/attuned by hidden-state similarity, and is eventually combined with a metric where they measure loss decrease relative to the pretrained base model on the same reference set of tasks. After an initial full base SFT warmup (15%), training retains the highest-score tokens within each response, with a default selection rate of 60% (the other 85% is GRAIN). The gradient signal is computed from forward-pass outputs without additional backbone backward passes (a major contribution in my opinion), while reference directions are refreshed periodically and base-model losses are cached offline for easier/cheaper later computation.

The paper provides a theoretical connection of hidden-state alignment to first-order parameter gradient influence. Experiments across three model backbones and seven math, reasoning, and knowledge benchmarks show the highest reported average performance for GRAIN on each backbone, compared with full-token SFT, random token selection, and existing token selector methods from literature. The ablations examine gradient alignment and reference excess loss separately and together, showing the effect of each component. They also cover sensitivity to token selection rates and reference task set size, and computational comparisons.

Critical strengths
  1. A practical approach to task-aware token selection. The paper clearly connects gradient similarity to token-level supervision. Computing the hidden state gradient from forward pass outputs avoids expensive full token calculations, and is a neat contribution. The theoretical discussion provides a sound justification of the methods, and I appreciate its inclusion.

  2. Promising initial results across the three model backbones. GRAIN achieves the highest reported average performance for each backbone, and the component ablation supports combining gradient alignment with reference excess loss. Its computational benefit needs an appropriate comparison: GRAIN takes 107 minutes, versus 83 for full-token SFT, 104 for Rho-1, and 92 for ssToken. The contribution is making task-aware gradient scoring practical, rather than making SFT faster overall. But it is in line with Rho-1 and ssToken, and this is practically good.

Critical weaknesses
  1. The gains need stronger evidence of repeatability and a clearer explanation of when the method works. Appendix D.3 reports a single training run with seed 42. GRAIN’s average improvement over the strongest other SFT method is 1.24 percentage points on LLaMA-3.2-3B, 0.41 on LLaMA-3.1-8B, and 0.55 on Qwen2.5-7B. Without multiple training seeds, it is difficult to assess the reliability of the smaller differences. The ablations also leave important design choices under-explained in my opinion. Appendix A suggests that coherent gradient directions make structured reasoning tasks more suitable, but does not empirically establish this boundary. On ASDiv, alignment-only scores 10.25, versus 55.08 for REL-only and 58.02 for GRAIN; explaining this gap would help me understand the reason for this transfer behavior. GRAIN also uses full-token SFT for the first 15% of training, but neither removes this warmup nor varies its duration, leaving its importance to the final results unclear.

  2. The reference-set construction leaves a cross-benchmark overlap concern unresolved. The authors describe separate splits and explicit exclusion measures, which I acknowledge and appreciate. However, GRAIN uses GSM-Plus examples to construct a reference direction while evaluating on the full GSM8K test set. GSM-Plus contains perturbations of GSM8K test questions, so excluding identical question strings does not establish separation from the evaluation task/questions. Leakage seems to be very possible here (https://huggingface.co/datasets/qintongli/GSM-Plus/blob/main/README.md). Using development examples for targeted task selection is reasonable. The concern is specifically whether variants of evaluation questions influence training-time selection of tokens. The paper does not establish whether its exclusions address this relationship or what the overlap in the reference and evaluation sets might be.

Questions for authors

The questions below are listed in order of importance to my assessment. I recognize that 2 and 3 may be completable in a faster fashion.

  1. Reliability across seeds. Can you report results across at least three independent training seeds for GRAIN and the strongest baselines re: the mean and standard deviation? This is particularly relevant to the average gains of 0.41 percentage points on LLaMA-3.1-8B and 0.55 on Qwen2.5-7B.

  2. Reference-set overlap. Does your exclusion procedure check originating GSM8K question IDs across benchmarks, rather than only exact question matches? Since GSM-Plus contains variants of GSM8K test questions, please clarify the overlap and whether the main results and affected ablations hold when the GSM-Plus reference direction is more deeply explored in reference to overlap with the evaluation set.

  3. Scope and transfer behavior. What evidence supports gradient coherence as the explanation for where GRAIN works well? In particular, what causes alignment-only ASDiv performance of 10.25, compared with 55.08 for REL-only and 58.02 for GRAIN? Does this reflect reasoning performance, answer formatting, or another effect of token selection? Could you dive deeper into this?

  4. Dependence on warmup. How sensitive are the results to the initial 15% full-token SFT warmup? Have you tested shorter warmup periods or immediate token selection, and do the gains persist? I would like an ablation on this if possible.

4 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank Reviewer mpgK for the constructive review. We address each concern below.


W1: Repeatability and when the method works

Response: Thank you for raising these points. The concern has three parts: whether the gains are repeatable, why Align-only fails on ASDiv, and how much the full-token warmup matters. We answer them with new experiments in Q1, Q3 and Q4, respectively.


W2 & Q2: Overlap between GSM-Plus references and GSM8K

Response: Thank you for raising this concern and for the question. The key question is whether variants of GSM8K test questions in our reference set steer token selection toward the evaluation, so we first quantify the overlap and then remove it. Overlap. Our original exclusion matched question text only, so the "stable-id exclusion" in criterion (D1) was inaccurate. Tracing source questions, the GSM-Plus reference items come from 96 GSM8K test questions (7.35% of those evaluated); no other reference family overlaps with an evaluation set, and the Table 1 baselines never use the reference set.

Removing the overlap. We removed the GSM-Plus family from the reference set and retrained GRAIN with three seeds (Table 1). GRAIN still has the best GSM8K score and the best average on every model, so its gains do not come from the overlapping questions. We have corrected (D1) and added the audit to revised Appendix D.6.

Table: GRAIN retrained without GSM-Plus references. Three-seed means from Table 1; best baseline per column.

ModelGSM8K: best baselineGSM8K: GRAINAVG: best baselineAVG: GRAIN
LLaMA-3.2-3B30.02 (Rho-1)31.8235.88 (Rho-1)37.37
LLaMA-3.1-8B57.71 (Rho-1)59.0751.13 (Rho-1)51.86
Qwen2.5-7B78.61 (Full Tokens)79.8863.27 (Full Tokens)64.09

Re-scoring the original checkpoints. Excluding the 96 questions from the original evaluation gives the same picture: GRAIN stays ahead of all baselines on the overlap-free questions, its average barely changes, the affected ablations keep their ranking, and on the overlapping questions Rho-1 and Token Cleaning even score higher than GRAIN.

Table: GSM8K re-scoring with the original checkpoints. LLaMA-3.2-3B, seed 42; accuracy (%) on the 96 overlapping and the 1,210 overlap-free questions, and the original seven-benchmark average before and after replacing GSM8K by its overlap-free score.

MethodOverlap (96)Overlap-free (1,210)AVG → AVG (overlap-free)
Full Tokens32.2928.9335.32 → 35.29
Random29.1727.6035.03 → 35.01
Rho-139.5829.6735.78 → 35.68
Token Cleaning39.5827.1934.08 → 33.95
ssToken26.0424.7933.28 → 33.27
GRAIN37.5030.8337.02 → 36.95
Align-only38.5430.3329.44 → 29.35
REL-only36.4627.0234.29 → 34.19
GRAIN, 150 held-out35.4231.8236.26 → 36.22
GRAIN, 300 held-out34.3832.4036.95 → 36.93

Q1: Multiple training seeds

Response: Thank you for the question. Single-seed margins of about 0.4–0.6 points on the larger models could fall within seed variance, so we retrained GRAIN and all baselines with three seeds (42, 123, 3407); GRAIN uses the reference set without GSM-Plus (see W2). GRAIN has the highest mean AVG on all three models, and its margin over the strongest baseline exceeds the seed-to-seed standard deviation on every model. We have updated Table 1 to report three-seed means (revised Appendix D.4).

Table: AVG over three training seeds (mean ± std).

ModelFull TokensRandomRho-1Token CleaningssTokenGRAIN
LLaMA-3.2-3B35.49 ± 0.2734.89 ± 0.3035.88 ± 0.2734.31 ± 0.2833.09 ± 0.2137.37 ± 0.32
LLaMA-3.1-8B50.27 ± 0.3047.97 ± 0.3051.13 ± 0.2650.51 ± 0.3149.98 ± 0.2651.86 ± 0.45
Qwen2.5-7B63.27 ± 0.2562.35 ± 0.2261.38 ± 0.1760.86 ± 0.1861.57 ± 0.2864.09 ± 0.38
AuthorsOpenReview ↗

Response 2/2

Q3: Align-only on ASDiv and gradient coherence

Response: Thank you for this question. Align-only nearly collapses on ASDiv alone while staying close to GRAIN on GSM8K and GSM-Plus, which points to an evaluation-level failure rather than weaker reasoning, so we inspected its generations.

Diagnosis. After answering, Align-only often keeps generating invented "Given the following problem..." blocks, and the ASDiv scorer takes the last number, so a correct first answer is overwritten. Scoring only the first answer brings Align-only to GRAIN's level while REL-only and GRAIN barely change, so the gap reflects termination rather than reasoning (revised Appendix D.7).

Table: ASDiv accuracy on LLaMA-3.2-3B with the default scoring and with only the first answer scored.

VariantDefault scoring (last match)First answer only
Align-only10.2557.58
REL-only55.0855.04
GRAIN58.0258.06

Role of REL and gradient coherence. We hypothesize that REL keeps end-of-answer tokens to which alignment gives little credit. Gradient coherence remains a hypothesized favorable condition rather than an established boundary, and our new measurement is consistent with it: alignment agrees with parameter-gradient similarity on reasoning tasks and hardly at all on MMLU (revised Appendix D.11). We will state this in Appendix A.


Q4: Sensitivity to the full-token warmup

Response: Thank you for the question. Like the reference-model training in Rho-1 and Token Cleaning, GRAIN starts with a short full-token warmup so that REL becomes informative. To test how much the results depend on it, we varied the warmup from 0% to 15% on LLaMA-3.2-3B. With 0% warmup, REL is zero at the start, so selection initially reduces to Align-only and REL takes effect as the model moves away from the base model. Shorter warmups cause only small drops, and even without warmup GRAIN still outperforms Full Tokens and Rho-1 (revised Appendix D.8). Meanwhile, a moderate warmup gives REL a better starting point and yields the best results, which is why we use 15% by default.

Table: Effect of the full-token warmup. LLaMA-3.2-3B, seed 42.

MethodMMLUBBHGSM8KMathQAGSM-PlusASDivMinervaAVG
GRAIN, 0% warmup56.7448.1230.9837.0318.9555.287.3536.35
GRAIN, 5% warmup56.8248.2331.1837.2519.3457.647.2136.81
GRAIN, 10% warmup56.9148.3531.2137.2219.4757.877.5536.94
GRAIN, 15% warmup (default)56.8648.3031.3237.3119.6558.027.6837.02
Full Tokens56.7247.9629.3336.0616.6755.005.4835.32
Rho-156.7448.1529.9436.6417.5557.274.1935.78

Thank you again for your valuable feedback, which has helped us significantly improve the paper. We hope these responses address your concerns, and we would be happy to clarify any remaining questions during the discussion phase.

Thank you for the detailed response and additional experiments. These address the main concerns behind my original recommendation.

  1. Repeatability and overlap: The three-seed comparisons strengthen the results. I particularly appreciate the acknowledgment of source-level overlap, its quantification, and the retraining without GSM-Plus references. This directly addresses my central concern. Please keep the distinction between these corrected main results and ablations using the original reference configuration clear.

  2. ASDiv and mechanism: The generation analysis explains the alignment-only collapse much more convincingly and clarifies the practical contribution of REL. I agree with retaining gradient coherence as a hypothesis rather than an established explanation of where GRAIN works.

  3. Warm-up and cost: The zero-warm-up and matched-time experiments address the concern that the gains might mainly reflect initial full-token training or additional compute. The contribution is clearer as making task-aware token scoring practical.

These additions sufficiently resolve my central objections, and I am raising my recommendation to a 3. The remaining limitations concern generalization beyond the tested setting and the strength of the mechanistic interpretation. Thank you for engaging carefully with the review. I appreciate your time and effort.

Thank you for taking the time to carefully consider our responses and for your thoughtful suggestions. We are grateful for your positive impression and support for our work. Your encouraging comments mean a lot to us, and we deeply appreciate your recognition of our efforts.

In the final version, we will clearly separate the corrected main results from the ablations using the original reference configuration, keep gradient coherence framed as a hypothesis, and discuss generalization and the scope of the mechanistic interpretation in the limitations.

Reviewer_pDNROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed GRAIN, an influence-based token selection method for LLM SFT. Instead of computing full parameter gradients to estimate token level influence, GRAIN uses gradients with respect to the final hidden states and measure their alignment with benchmark specific task directions that are constructed from a small held out set. This alignment score is gated by hidden state similarity and further combined with a reference excess loss term to select tokens for SFT. The experiments compare GRAIN with existing token selection methods as well as full SFT, and the ablation studies examine the contribution of the alignment and reference excess loss terms.

Critical strengths

The paper proposes a practical way to address the computational bottleneck in estimating token level influence. Specifically, they use gradients with respect to the final hidden states, which can be computed from the forward pass without an additional backward pass. They also provide a conditional theoretical justification for this approximation and clearly explains the assumptions and limitations. The paper is also well written and easy to follow.

Critical weaknesses

The performance gains over full token SFT are modest on most benchmarks, while the total end to end runtime increases by about 29% ((107-83)/83). GRAIN also uses full token SFT for the first 15% of training steps to have nonzero base to current differences. However, the alignment score could guide token selection from the start, and loss differences could emerge as training proceeds. Without an ablation that removes this warmup, it is unclear how much the final performance depends on the initial full token training and how much the gain comes from the proposed selection method.

Questions for authors
  1. Is the full-token SFT warmup necessary? Could you try selecting tokens from the start, using alignment alone initially and adding reference excess loss as training proceeds?
  2. How should the selected token proportion k be chosen for a new model or dataset? The results in Figure 3 only examines this choice on LLaMA-3.2-3B. Does the default choice, 60%, work similarly well for the other two models?
  3. How well does GRAIN’s last-layer alignment approximate the full parameter gradient-similarity influence in practice? Could you compare their token rankings on a small scale experiment?
  4. Do the baselines also use the held out set that is available to GRAIN? If not, could you include a comparison where the baselines also use this data, to clarify how much of the gain comes from access to additional task-specific data?
  5. Could you report peak GPU memory usage?
  6. Do you think the proposed gradient approximation could also be used for influence based sample selection? What challenges might arise when extending to this setting? I would be interested to hear the authors’ thoughts on this broader application.
3 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank Reviewer pDNR for the constructive review. We address each concern below.


Weakness: "The performance gains ... are modest ..., while the total end to end runtime increases by about 29%."

Response: Thank you for raising this concern. To verify that the gains are consistent, we compare GRAIN and Full Tokens over three training seeds. GRAIN consistently outperforms Full Tokens on all three models. GRAIN's end-to-end cost is also comparable to other reference-based selectors such as Rho-1 (Figure 3), and we will state this accuracy–cost trade-off explicitly in the revision.

Table: Seven-benchmark average over three training seeds (mean ± std).

ModelFull TokensGRAINGain
LLaMA-3.2-3B35.49 ± 0.2737.37 ± 0.32+1.88
LLaMA-3.1-8B50.27 ± 0.3051.86 ± 0.45+1.59
Qwen2.5-7B63.27 ± 0.2564.09 ± 0.38+0.82

Weakness & Q1: "Is the full-token SFT warmup necessary?"

Response: Thank you for raising this concern and for the question. We would like to clarify that the warmup simply follows the common practice of prior reference-based selection methods, including Rho-1 (Lin et al., 2024) and Token Cleaning (Pang et al., 2025), which first train a reference model before scoring by excess loss. For GRAIN, it mainly serves to make REL, which is zero at initialization, informative once selection starts. To further address your concern, we varied the warmup from 0% to 15% on LLaMA-3.2-3B. With 0% warmup, GRAIN selects tokens from the very first training step. At initialization, the current model is identical to the pretrained base model, so REL is zero and selection relies on gradient alignment alone; as training progresses, REL becomes nonzero and naturally contributes to the combined score. This setting thus realizes your suggested schedule of using alignment first and adding REL as training proceeds. Shorter warmups cause only small drops, and GRAIN still outperforms Full Tokens and Rho-1 without any warmup, so its gain does not depend on the initial full-token training (revised Appendix D.8). Meanwhile, a moderate warmup gives REL a better starting point and yields the best results, which is why we use 15% by default.

Table: Effect of the full-token warmup. LLaMA-3.2-3B, seed 42.

MethodMMLUBBHGSM8KMathQAGSM-PlusASDivMinervaAVG
GRAIN, 0% warmup56.7448.1230.9837.0318.9555.287.3536.35
GRAIN, 5% warmup56.8248.2331.1837.2519.3457.647.2136.81
GRAIN, 10% warmup56.9148.3531.2137.2219.4757.877.5536.94
GRAIN, 15% warmup (default)56.8648.3031.3237.3119.6558.027.6837.02
Full Tokens56.7247.9629.3336.0616.6755.005.4835.32
Rho-156.7448.1529.9436.6417.5557.274.1935.78

Q2: Choosing the selected-token proportion k

Response: Thank you for the question. We use k = 60% following Rho-1, Token Cleaning, and ssToken, and keep it fixed across models. To check whether this choice transfers, we repeated the sweep on LLaMA-3.1-8B and Qwen2.5-7B. The trend matches LLaMA-3.2-3B, and 60% is the best tested setting on all three models (revised Appendix D.5).

Table: Seven-benchmark average vs. selected-token proportion k (k = 100% is full-token SFT).

Model40%50%60%70%80%90%100%
LLaMA-3.2-3B35.9036.5037.0236.6536.5436.4735.32
LLaMA-3.1-8B49.5550.7251.3450.9650.4750.2550.00
Qwen2.5-7B62.6763.0463.6663.4363.4963.2363.11
AuthorsOpenReview ↗

Response 2/2

Q3: Last-layer alignment vs. parameter-gradient similarity

Response: Thank you for the question. δ = W_lmᵀ(p − e) is the exact gradient with respect to the final hidden state; what it omits is the backbone Jacobian, and Proposition 1 gives conditions under which its alignment approximates first-order parameter-gradient influence. To check this in practice, we compare token rankings by last-layer alignment with rankings by exact gradient similarity over all parameters of four layers, on 1,000 training sequences with the GRAIN model on LLaMA-3.2-3B. Alignment is consistently positively correlated with gradient similarity and recovers 69% of the most gradient-similar tokens versus 60% for random selection; gating leaves this agreement essentially unchanged, and agreement is highest on reasoning tasks such as BBH and MathQA. REL captures a different signal, how much a token's loss has dropped, and complements alignment rather than approximating gradient similarity (revised Appendix D.11).

Table: Agreement with parameter-gradient similarity. LLaMA-3.2-3B, 1,000 sequences, all parameters of four layers.

ScoreSpearman ρTop-60% recall
Alignment (ungated)0.380.69
Alignment (gated)0.380.69
Random00.60

Q4: Baselines with access to the held-out set

Response: Thank you for the question. The baselines in Table 1 do not use the held-out set: Rho-1 and Token Cleaning build their references from the training pool, and ssToken uses the base model. To make a fair comparison, we retrained every baseline with GRAIN's held-out examples added to its training data. Their averages barely change, so GRAIN's advantage does not come from the additional data (revised Appendix D.9). We will make the difference in data access explicit in Section 5.1.

Table: Baselines with access to GRAIN's held-out set. Seven-benchmark average on LLaMA-3.2-3B (seed 42).

Full TokensRandomRho-1Token CleaningssTokenGRAIN
Original35.3235.0335.7834.0833.28–
With held-out set35.3534.9235.9433.9533.2437.02
Change+0.03−0.11+0.16−0.13−0.04–

Q5: Peak GPU memory

Response: Thank you for the question. We measured peak memory for GRAIN and Full Tokens under the same setup. GRAIN needs about 3.4 GiB more peak allocated memory per GPU and fits on a 24 GB A5000 (revised Appendix D.10).

Table: Peak GPU memory per device. LLaMA-3.2-3B, 4×A5000, LoRA, batch size 4, sequence length 2,048.

MethodPeak allocated (GiB)Peak reserved (GiB)
Full Tokens14.3915.99
GRAIN17.8019.78

Q6: Extension to sample selection

Response: Thank you for this interesting question. In principle, the approximation could serve as a low-cost proxy for influence-based sample selection, such as the targeted instruction tuning of LESS (Xia et al., 2024). Since a response's parameter gradient is the weighted sum of its token gradients, aggregating token alignments with the same weights would approximate the response's first-order effect on the target loss, using forward passes only. LESS instead computes a projected, Adam-preconditioned LoRA gradient for every candidate at several checkpoints, so such a proxy could substantially reduce the selection cost. However, sample selection differs from token selection in several ways, so its effectiveness would require careful validation. First, a whole response also contains unhelpful tokens whose effects may offset the useful ones, which is precisely what token selection avoids; the aggregation would also need to handle length and norm effects, which LESS addresses with cosine similarity. Second, the omitted backbone Jacobian and the optimizer geometry vary across samples and may distort cross-sample rankings. Third, a selected subset must also handle redundancy and task coverage, and sample usefulness changes during training. One possible design is to use the forward-only proxy to shortlist candidates and then apply LESS-style gradient matching to the shortlist. We consider this an interesting direction for future work and will mention it in the revision.


Thank you again for your valuable feedback, which has helped us significantly improve the paper. We hope these responses address your concerns, and we would be happy to clarify any remaining questions during the discussion phase.

AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Dear Reviewer pDNR,

As the discussion period is coming to an end, we would greatly appreciate it if you could take a look at our responses. If you have any remaining questions or concerns, we would be happy to provide further clarification and discussion.

Thank you again for your time and consideration.

Best,

The Authors

Reviewer_QNYYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Introduces GRAIN, an online SFT token selector using last-hidden-state loss gradients aligned with held-out benchmark directions. Combines gated task alignment with reference excess loss, avoiding per-token parameter-gradient backward passes and a separate trained reference model. Reports the highest mean score among compared SFT methods on seven math/reasoning benchmarks for three base models.

Critical strengths
  1. The closed-form last-layer gradient is a simple, implementable approximation to expensive task-aware gradient influence.
  2. Testing across LLaMA-3.2-3B, LLaMA-3.1-8B, and Qwen2.5-7B provides useful model diversity (Table 1).
  3. The alignment/REL ablation and measured runtime overhead make the empirical trade-offs more transparent (Table 2; Figure 3).
Critical weaknesses
  1. The theory only motivates ungated, per-benchmark alignment under restrictive Jacobian assumptions, not the actual gated-plus-REL ranking (Proposition 1).
  2. Held-out directions are built from six of the evaluated benchmark families, leaving transfer to unseen task families unestablished (Section 5.1).
  3. The mean-score gains are sometimes modest and no multi-seed uncertainty is provided; measured total time is 107 min versus 83 min for full-token SFT in the timing study (Figure 3).
Questions for authors
  1. How much improvement remains when held-out directions come from different task families than those used for evaluation?
  2. Can the authors report multi-seed confidence intervals and compare against baselines at matched wall-clock time or token-compute budgets?
  3. Can the authors quantify when last-layer alignment agrees with actual parameter-gradient influence, especially after the gating and REL terms are applied?
3 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank Reviewer QNYY for the constructive review. We address each concern below.


W1: Theory scope: gating and REL

Response: Thank you for raising this point. We agree that Proposition 1 itself covers only the ungated, per-benchmark alignment term, and this scope is stated right after Proposition 1 in Sec. 4. The proposition justifies GRAIN's key approximation: under its assumptions, the alignment δ·m_b, computed from a forward pass, approximates first-order parameter-gradient influence. We further extend this analysis to the gate: under the same assumptions and an approximately common scale across benches and tokens, the gated, aggregated alignment is a proxy for each token's predicted first-order reduction of a token-adaptive weighted objective that emphasizes the benches relevant to the token (Corollary 1, revised Appendix B). REL admits a complementary first-order interpretation in the spirit of TracIn (Pruthi et al., 2020): the net decrease in the token's own loss approximates the accumulated first-order effect of past updates along the actual training trajectory, i.e., learning progress rather than target-task influence (revised Appendix B).

The gate rescales alignment by the hidden-state similarity between a token and the target task, suppressing spurious matches from unrelated tokens. Gating leaves the theoretically motivated part intact: in our new measurement, gated and ungated alignment agree equally well with exact parameter-gradient similarity (Spearman 0.38 and top-60% recall 0.69, versus 0.60 for random selection; revised Appendix D.11), which supports alignment as a practical proxy for gradient similarity beyond the idealized assumptions. The role of REL is also visible empirically: on ASDiv, Align-only often fails to stop after answering, whereas REL-only and GRAIN do not (revised Appendix D.7). The component ablation shows that combining alignment and REL performs best (see Q3).


W2 & Q1: Held-out directions from other task families

Response: Thank you for raising this concern and for the question. To test transfer to task families absent from the held-out set, we replaced it on LLaMA-3.2-3B with Math-only references (GSM8K and MathQA) or Non-math-only references (non-mathematical MMLU subjects and BBH tasks) of the same size, and evaluated on five math benchmarks.

With only non-mathematical references, GRAIN retains more than half of its average gain over Full Tokens and still outperforms Rho-1; math references help further, so task-related references help but are not required. GRAIN also transfers to GSM-Plus-Mini, which has no examples in the clean reference set (best three-seed mean on all three models, Table 1). This tests transfer from non-mathematical tasks to mathematics rather than to arbitrary task families; we will state this scope in Section 5.1 and have added the experiment to revised Appendix D.12.

Table: Cross-family reference transfer. LLaMA-3.2-3B, seed 42; AVG over the five math benchmarks.

MethodGSM8KMathQAGSM-PlusASDivMinervaAVG
Full Tokens29.3336.0616.6755.005.4828.51
Rho-129.9436.6417.5557.274.1929.12
GRAIN (original)31.3237.3119.6558.027.6830.80
GRAIN (Math-only)30.8037.1019.1557.107.3530.30
GRAIN (Non-math-only)30.3036.8518.5056.107.1529.78
AuthorsOpenReview ↗

Response 2/2

W3 & Q2: Multi-seed uncertainty and matched compute

Response: Thank you for raising this concern and for the suggestion. We address uncertainty with three training seeds and compute with a matched wall-clock comparison. Multiple seeds. We retrained GRAIN and all baselines with three seeds (42, 123, 3407). GRAIN has the highest mean AVG on all three models, and its margin over the strongest baseline exceeds the seed-to-seed standard deviation (Table 1).

Table: AVG over three training seeds (mean ± std).

ModelFull TokensRandomRho-1Token CleaningssTokenGRAIN
LLaMA-3.2-3B35.49 ± 0.2734.89 ± 0.3035.88 ± 0.2734.31 ± 0.2833.09 ± 0.2137.37 ± 0.32
LLaMA-3.1-8B50.27 ± 0.3047.97 ± 0.3051.13 ± 0.2650.51 ± 0.3149.98 ± 0.2651.86 ± 0.45
Qwen2.5-7B63.27 ± 0.2562.35 ± 0.2261.38 ± 0.1760.86 ± 0.1861.57 ± 0.2864.09 ± 0.38

Matched compute. GRAIN's end-to-end time is close to Rho-1's and 29% above Full Tokens (Figure 3). Given the same 107 minutes, Full Tokens barely improves and stays well below GRAIN (revised Appendix D.10).

Table: LLaMA-3.2-3B at matched wall-clock time (seed 42).

BenchmarkFull Tokens (83 min)Full Tokens (107 min)GRAIN (107 min)
MMLU56.7256.8256.86
BBH47.9648.0448.30
GSM8K29.3329.5131.32
MathQA36.0636.1137.31
GSM-Plus16.6716.5219.65
ASDiv55.0055.3258.02
Minerva Math5.485.517.68
AVG35.3235.4037.02

Q3: When last-layer alignment agrees with gradient similarity

Response: Thank you for the question. To quantify this, we compare token rankings by last-layer alignment with rankings by exact gradient similarity over all parameters of four layers, on 1,000 training sequences with the GRAIN model on LLaMA-3.2-3B. Overall, alignment recovers 69% of the most gradient-similar tokens versus 60% for random selection (Spearman 0.38), and gating leaves this agreement essentially unchanged. To see when the agreement holds, we break it down by reference task. Agreement is highest on BBH and MathQA and near zero on knowledge-heavy MMLU, consistent with the gradient-coherence hypothesis in Appendix A (revised Appendix D.11).

Table: Agreement by reference task. Spearman ρ between gated alignment and parameter-gradient similarity (four layers), LLaMA-3.2-3B; a random ranking gives ρ ≈ 0.

Reference taskOverallMMLUGSM8KGSM-PlusMATHBBHMathQA
Spearman ρ0.380.030.190.220.270.490.56

GRAIN builds on REL, which measures learning progress (see W1), and adds alignment to inject target-gradient information; the final ranking therefore combines both signals rather than tracking gradient similarity alone. To check whether combining these signals improves performance, we compare GRAIN with each component alone. Combining alignment with REL performs best.

Table: Component ablation. LLaMA-3.2-3B average.

MethodLLaMA-3.2-3B average
REL-only34.29
Align-only29.44
GRAIN37.02

Thank you again for your valuable feedback, which has helped us significantly improve the paper. We hope these responses address your concerns, and we would be happy to clarify any remaining questions during the discussion phase.

AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Dear Reviewer QNYY,

As the discussion period is coming to an end, we would greatly appreciate it if you could take a look at our responses. If you have any remaining questions or concerns, we would be happy to provide further clarification and discussion.

Thank you again for your time and consideration.

Best,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

Dear Area Chairs and Reviewers,

We sincerely appreciate all your time and effort in reviewing our submission. We have uploaded our revised manuscript with all changes highlighted in blue. Below is a summary of the revisions:

1. Robustness of the gains (Reviewers mpgK, pDNR, QNYY): Table 1 and Appendices D.4, D.5 and D.8. GRAIN has the highest three-seed average on all three models, the default k = 60% is the best tested setting, and GRAIN outperforms the baselines even without the full-token warmup.

2. Reference-set overlap with GSM8K (Reviewer mpgK): Appendices C.1 and D.6. GRAIN retrained without GSM-Plus references keeps the highest three-seed average on all three models.

3. Held-out data, compute and reference transfer (Reviewers pDNR, QNYY): Appendices D.9, D.10 and D.12. Baselines trained with GRAIN's held-out examples, and full-token SFT trained for GRAIN's wall-clock time, still trail GRAIN; with only non-mathematical references, GRAIN still outperforms Full Tokens and Rho-1 on the five-benchmark math average.

4. Theory and mechanism (Reviewers mpgK, pDNR, QNYY): Appendix B (Corollary 1) and Appendices D.7 and D.11. We extend Proposition 1 to the gated, aggregated alignment, show that last-layer alignment recovers clearly more gradient-similar tokens than random selection, and show that REL resolves Align-only's termination failure on ASDiv.

Please note that all table, figure and appendix references, here and in our responses, correspond to the revised manuscript. Thank you for bringing these points to our attention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.