acceptodds
Under review as a conference paper at ICLR 2027

Loss Alone Is Not Enough: Stability and Geometry in Dynamic Sample Reweighting

Abstract

Dynamic sample reweighting can emphasize informative examples during training, and several recent methods use the per-sample loss to determine these weights. Per-sample loss indicates which samples are difficult, but it does not reveal how emphasizing those samples changes the geometry of the optimization update. We show that this distinction can matter even on simple strongly convex problems. For a broad class of loss-only reweighting schemes, upweighting higher-loss examples can lead to an arbitrarily worse local convergence rate than gradient descent with uniform sample weights. Moreover, for exponential loss-based reweighting, increasing the step size past a threshold produces a stable two-cycle (a persistent oscillation) even when uniform-weight gradient descent remains stable. To address this limitation, we propose **Geometry-Aware Reweighting (GAR)**, which chooses weights using both loss and gradient information. For minibatch SGD, GAR satisfies the standard upper bound of uniform-weight SGD minus an explicit nonnegative term, and it avoids the instability in our hard example. We also derive practical extensions to momentum and AdamW. Experiments on 120M–300M language models show that GAR achieves lower evaluation log-perplexity than both uniform-weight AdamW and the loss-based LinUpper reweighting (Sow et al., 2025).

Primary area:
optimization
Keywords:
dynamic sample reweighting, stochastic gradient methods, convergence analysis, optimization geometry, adaptive optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eC9yOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how dynamic loss-based sample reweighting affects optimization stability. It shows that component-gradient disagreement can worsen local conditioning and produce attracting two-cycles even on a well-conditioned quadratic objective. It proposes Geometry-Aware Reweighting (GAR), proves convergence and stability guarantees for an idealized version, and evaluates an AdamW adaptation on 120M–300M language models.

Critical strengths

The quadratic construction isolates the effect of reweighting: the average objective remains well conditioned and its optimum remains a fixed point, yet loss-only weighting can converge arbitrarily slowly even with the optimal scalar step size. The comparison between explicit loss weighting and GAR's implicit update provides a useful explanation of the instability and a principled motivation for incorporating gradient geometry.

Critical weaknesses
  1. The guarantees do not justify the implemented algorithm. Theorem 4 and Proposition 3 rely on unavailable oracle gaps, fi(x)−fi(x⋆)f_i(x)-f_i(x^\star). Replacing these gaps with observable losses can change the fixed point when the component losses at the ERM optimum differ. The implementation additionally uses last-layer gradients and only three undamped fixed-point iterations, without approximation-error or solver-convergence guarantees. The practical solver can lose the central stability benefit even before introducing AdamW.

  2. The experiment is too limited to establish a practical benefit. Results consist of final-step log-perplexities in Table 1 on one training corpus. At τ=3\tau=3, average gains over LinUpper are just 0.012–0.033; at τ=1\tau=1, GAR is worse than both baselines on average at every model size. The paper reports three temperatures and inherits training settings from LinUpper, but does not explain the selection criterion or comparative tuning budgets. There are also no ablations separating the geometry correction from look-ahead scoring, or runtime measurements accounting for GAR's additional computation. These omissions make the robustness, source, and cost of the gains difficult to assess.

  3. The theoretical contribution needs clearer discussions. Section 1.1 and Appendix A discuss relevant methods, and Section 4.1 acknowledges that TERM already identifies the covariance contribution to curvature. However, the discussion does not sufficiently separate what is new in the general weighting-map analysis and GAR's stability mechanism from existing analyses. This limits my assessment of the theoretical novelty beyond the useful counterexample.

Questions for authors

See critical weaknesses.

3 comments

Thank you for the careful review. We appreciate your recognition that the quadratic example isolates slowdown at a preserved ERM fixed point and that the explicit and implicit updates help explain the role of gradient geometry.

W1: Oracle gaps, gradient approximation, and fixed-point iterations

The theoretical guarantees use oracle gaps and an exact weight solve. Please see the General Comment for ablations of all three implementation choices. Using estimated loss gaps substantially changes the weights and gives final log-perplexity comparable to or higher than uniform AdamW. The gradient and fixed-point comparisons show only small changes in the checkpoint weights, and the 20k run with more iterations finishes within the range of the default GAR runs.

W2: Empirical gains, temperature, ablations, and runtime

Empirical gains.

Table 1 compares models trained on the same SlimPajama subset for 20,000 steps. At 120M, 210M, and 300M, LinUpper lowers average log-perplexity relative to uniform AdamW by 0.021, 0.023, and 0.012, respectively. GAR at τ=3\tau=3 improves on LinUpper by a further 0.012, 0.025, and 0.033. The improvement is comparable at 120M and larger at both 210M and 300M. Relative to LinUpper, this is a 1.2–3.2% reduction in the geometric mean of perplexity across the seven evaluation domains. In the General Comment, we also provide a three-seed comparison at 120M, where GAR improves over uniform AdamW on each seed.

Experimental setup and temperature.

We used the model configurations, training hyperparameters, and evaluation pipeline from the LinUpper implementation of Sow et al. (2025), as described in Section 7 and Appendix F. The temperature sweep was specific to GAR. We tried τ=1,3,10\tau=1,3,10 at each model size, and the middle value, τ=3\tau=3, consistently gave the lowest average log-perplexity (Table 1).

Quadratic term and look-ahead losses.

Please see the General Comment for these two ablations. Setting αt=0\alpha_t=0 isolates the effect of the quadratic term. The weights change only slightly at matched checkpoints, but final log-perplexity is higher than default GAR on all three matched seeds and close to the LinUpper result in Table 1. Scoring at the current iterate also changes the weights only slightly. Its seed-42 run gives slightly higher final log-perplexity than default GAR on the same seed.

Computational overhead.

The language model experiments are a proof of concept for GAR. On one A100, our current 120M implementation takes about 22–23 hours for 20k steps, compared with 7.4 hours for uniform AdamW. The overhead includes the look-ahead forward pass and the gradient products used to compute the weights. Please see our response to YiH9 (W3 + Q2) for the measurement setup and a possible way to reduce this overhead.

W3: What extends beyond TERM

TERM replaces the average training loss with a log-sum-exp of the individual losses. With positive tilt, a gradient step on that objective uses a weighted average of the individual gradients, with larger losses receiving larger weights. Li et al. (2023) show that disagreement between the individual gradients contributes an extra covariance term to the Hessian (Lemma 2, Eq. (13) in Li et al. (2023)).

We show that the same covariance appears for a much broader class of weighting rules. At the minimizer of the average loss, every example's loss gap is zero. Proposition 2 describes what happens when the iterate moves slightly away from that point. For any differentiable rule that treats examples symmetrically and uses only differences between their loss gaps, the update Jacobian contains the same covariance matrix multiplied by a scalar. This scalar measures how the weight of an example responds to a small change in its loss gap relative to the others. A positive scalar adds curvature and can force a smaller stable step size. In our example, this can make convergence arbitrarily slow even though the average loss is well conditioned and its minimizer remains a fixed point.

We then propose a method that mitigates this issue. GAR uses the squared norm of the weighted gradient when choosing the weights. The resulting implicit update uses the covariance to damp the step in directions where the sample gradients disagree. On our quadratic example, this keeps GAR stable throughout the step-size range where uniform gradient descent is stable. In minibatch SGD, the quadratic term also allows a direct comparison with uniform weights, giving a convergence bound for the original average loss with an explicit nonnegative improvement term relative to the uniform SGD bound.


We will clarify how our analysis and GAR extend beyond TERM in the next revision. We hope these clarifications address your concerns. We would be happy to discuss any remaining questions.

Thank you for the response and the additional ablations. The clarification relative to TERM addresses my positioning concern, and the ablation supports the role of the geometry term. Two gaps remain:

  1. Theory vs. implementation. The guarantees cover oracle gaps, but the gains come from raw losses. With estimated gaps, the weights become nearly uniform and the gain disappears. So the theory does not explain the method that works. The ten-iteration ablation suggests three iterations are enough here, but no condition rules out the divergence in the Eq. (7) example.
  2. Practical cost. GAR runs about 3$\times$ slower than uniform AdamW for small gains over LinUpper, and it is not compared against a compute-matched baseline. Current-iterate scoring could remove one forward pass, but its runtime is not reported. GAR was also the only method that received a temperature sweep.

I will raise my score to 3, but these gaps should be addressed for publication.

Thank you for the follow-up and for recognizing the clarification relative to TERM and the ablation of the quadratic term. We will address the remaining concerns in the revision as follows.

1. Theory versus implementation

Oracle gaps and raw losses

The oracle-gap convergence guarantee does not directly cover the raw-loss AdamW experiments. We will expand the discussion of this distinction and its implications.

Our experiments with 20k-step reference models illustrate the difficulty of estimating the oracle gaps. These references introduce example-dependent errors, and the estimated gaps become less dispersed late in training. At the temperature selected for raw losses, the weights then approach uniform and the final performance is close to uniform AdamW. We will test whether a longer-trained reference alleviates this performance drop.

In our experiments, raw-loss weighting achieved better validation performance than weighting based on estimated loss gaps, although it need not preserve the ERM minimizer. Our convergence analysis aims to minimize the ERM training objective, but preserving its minimizer need not yield the best validation performance. The altered weighting from raw losses may therefore contribute to their better validation performance in our experiments, without contradicting the training-loss guarantees. We will add a discussion of this distinction in the revision.

We will also include a convex minibatch analysis that bounds the ERM objective shift for exact raw-loss GAR with a fixed preconditioner in terms of the component losses at the optimum, temperature, and gradient variation; please see point 2 of our follow-up response to Reviewer XV43 for the bound and its assumptions.

Fixed-point iterations

One clarification is that the GAR stability result on the example in Eq. (7) also applies to raw-loss scores: the component losses at the ERM optimum are equal in that example, so raw losses and oracle gaps differ only by a common additive constant. However, this result assumes an exact solution of the weight-selection problem. Our theory does not guarantee convergence of the fixed-point iteration used to approximate that solution.

In our checkpoint experiments, the spectral radius of the fixed-point iteration Jacobian at the computed solutions was at most 0.360.36 across all examined minibatches and checkpoints, and the three- and ten-iteration weights were very close. We did not observe convergence problems in these experiments. We will clarify the lack of convergence guarantees for the fixed-point iteration and also discuss the numerical results.

2. Practical cost and tuning

Runtime and comparison budget

We will report the measured runtime explicitly and make clear that the performance comparisons use matched training steps. Our current 120M implementation is about three times slower than uniform AdamW, and these results do not establish an advantage at matched compute. The LM experiments were intended as a proof of concept for the AdamW adaptation, and developing an efficient implementation is important future work.

In the 120M experiments on one A100, scoring at the current iterate instead of the look-ahead point reduced median step time by approximately 7%. Increasing the number of fixed-point iterations from three to ten increased median step time by approximately 95%, suggesting that the fixed-point weight computation is a major bottleneck. Further savings may be possible by optimizing the Jacobian–vector product computations within these iterations. We will include these measurements and the total runtimes of the ablation runs alongside their final performance in the revision.

Temperature tuning and the LinUpper baseline

We used the public LinUpper implementation of Sow et al. (2025) and followed their experimental setup. Our reproduced LinUpper results are comparable to those reported in Table 1 of Sow et al. (2025): the average evaluation log-perplexities are 3.2843.284, 3.1153.115, and 3.1143.114 at 120M, 210M, and 300M, respectively, compared with their reported 3.303.30, 3.133.13, and 3.103.10. In particular, our 120M and 210M results improve upon the published values. This provides evidence that our comparison uses a competitive reproduction of LinUpper. We will include the comparison with the published LinUpper results in the revision.


We will incorporate these discussions and measurements into the next revision, with explicit statements of the theoretical scope, empirical comparison budget, and tuning procedure. We hope these additions address the remaining concerns. Thank you for the constructive feedback.

Reviewer_XV43OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that re-weighting samples by loss alone can hurt optimization, because two hard samples can have gradients that reinforce or cancel each other, and the loss is not enough to tell them apart. The authors build a simple convex example where loss-based weighting converges much more slowly or keeps oscillating, and they propose GAR, which picks the weights by trading off the loss against the norm of the combined gradient. On GPT-2-style models, GAR shows lower performance that uniform-weighted AdamW or the LinUpper method from related work.

Critical strengths

I really appreciate the authors for conducting a well motivated and, from my perspective, theoretically sound, study. The following aspects stand out:

  • The paper closes a gap re-weighting literature has repeatedly mentioned as a limitation and that had the potential to reduce the pre-training effectiveness. The clear formulation in proposition 2 is helpful to understand this.
  • The bounds derived from the full weight-dependent part clearly show how the GAR approach cannot do worse than than uniformly weighted SGD. This is helpful for the reader and makes it easy to follow along.
Critical weaknesses
  • Similar to related work, GAR assumes a non-observable loss gap dj=fj(x)−fj(x∗)d_j = f_j(x) − f_j(x*). In practice this determines the effect of the re-weighting mechanism and should be outlined as a clear limitation. Have you run any ablation experiments that would allow some empirical insights how this gap could look like in pre-training data?
  • Related to the previous point, GAR on raw losses might converge to the wrong point, especially with the geometric components. This potentially shifts the training objective and could produce models that do not follow the original intent. Is there a mechanism to contain or quantify this shift? To what degree are the experiments accounting for this?
  • The paper could benefit from an ablation focused on the geometric term to isolate its effects. Similarly, I am wondering how the dynamics change in larger models and whether the re-weighting stabilizes training or leads to noisier results. I do understand that pre-training typically goes beyond what can be done in an academic setting, so I would be interested in some considerations on that that could be derived from the theoretical insights.
Questions for authors

Please see weaknesses.

3 comments

Thank you for the thoughtful review. We appreciate your recognition that Proposition 2 explains how loss-only reweighting changes the update and that Theorem 4 improves on the standard uniform-weight SGD upper bound.

W1: Using a reference model to estimate loss gaps

Please see the General Comment for the comparison of raw losses with gaps estimated from 20k-step reference models. At the temperature selected for raw losses, the estimated gaps yield nearly uniform weights at late checkpoints. In the 20k training runs on seed 42, the GAR reference finishes close to uniform AdamW, and the AdamW reference gives higher final log-perplexity.

W2: Effect of raw losses on the ERM fixed point

The loss gaps are all zero at the ERM stationary point x⋆\mathbf{x}^\star, so exact full-batch GAR gives uniform weights there. In contrast, raw losses can change the fixed point when the component losses at x⋆\mathbf{x}^\star differ.

Consider full-batch GAR with raw losses, a fixed preconditioner, τ>0\tau>0, and an exact weight solver. At a fixed point the weighted gradient Gw\mathbf{G}\mathbf{w} is zero, so the gradient-based correction in Eq. (13) vanishes and the weights reduce to softmax on the raw losses. Conversely, if those softmax weights give a zero weighted gradient, they satisfy Eq. (13) and the iterate is a fixed point. Thus GAR and exponential weighting with the same raw losses share their fixed points in this setting.

For convex component losses, a fixed point xτ\mathbf{x}_\tau minimizes the tilted risk

Fτ(x)=τlog⁡(1n∑iexp⁡(fi(x)/τ)).F_\tau(\mathbf{x})=\tau\log\left(\frac{1}{n}\sum_i\exp(f_i(\mathbf{x})/\tau)\right).

Write f=1n∑ifif=\frac{1}{n}\sum_i f_i. Jensen's inequality gives f≤Fτf\le F_\tau, and optimality gives Fτ(xτ)≤Fτ(x⋆)F_\tau(\mathbf{x}_\tau)\le F_\tau(\mathbf{x}^\star). For a uniformly sampled index II, let Z=fI(x⋆)Z=f_I(\mathbf{x}^\star), a=min⁡ifi(x⋆)a=\min_i f_i(\mathbf{x}^\star), and b=max⁡ifi(x⋆)b=\max_i f_i(\mathbf{x}^\star). Hoeffding's lemma gives log⁡Ees(Z−EZ)≤s2(b−a)2/8\log\mathbb{E}e^{s(Z-\mathbb{E}Z)}\le s^2(b-a)^2/8 for Z∈[a,b]Z\in[a,b]. Taking s=1/τs=1/\tau yields

0\le f(\mathbf{x}_\tau)-f(\mathbf{x}^\star) &\le F_\tau(\mathbf{x}^\star)-f(\mathbf{x}^\star)\\\\ &=\tau\log\mathbb{E}\exp \big((Z-\mathbb{E}Z)/\tau\big) \le \frac{(b-a)^2}{8\tau}. \end{aligned}$$ Increasing $\tau$ reduces this upper bound on the original ERM loss. On our quadratic example (Eq. (7)), the component losses at $\mathbf{x}^\star$ are equal, so the bound is zero. ## W3: Ablation of the quadratic term and training dynamics Ablation (d) in the General Comment isolates the effect of the quadratic term by setting $\alpha_t=0$ and keeping the raw look-ahead losses and temperature fixed. This changes the weights only slightly at matched checkpoints, but increases final log-perplexity on all three matched seeds. The ablation's mean log-perplexity is close to LinUpper's 120M result in Table 1. On the theoretical side, we can compare the variance of the gradient estimates at the ERM stationary point $\mathbf{x}^\star$ in our SGD setting with a fixed preconditioner. With oracle gaps, the scores are all zero at $\mathbf{x}^\star$, so a permutation-equivariant loss-only map returns uniform weights $\mathbf{u}$. At the same point, GAR chooses $\mathbf{w}$ to minimize $$\frac{\eta}{2}\left\lVert \mathbf{G}_t\mathbf{w}\right\rVert_{\mathbf{P}^{-1}}^2+\tau D_{\mathrm{KL}}(\mathbf{w}\\|\mathbf{u})$$ over the simplex. Since $\mathbf{u}$ is feasible, an exact solution $\mathbf{w}_{\mathrm{GAR}}$ gives $\left\lVert \mathbf{G}_t\mathbf{w}_{\mathrm{GAR}}\right\rVert_{\mathbf{P}^{-1}}^2\le\left\lVert \mathbf{G}_t\mathbf{u}\right\rVert_{\mathbf{P}^{-1}}^2$ on every minibatch. Since $\nabla f(\mathbf{x}^\star)=0$, this is a comparison of the gradient deviation $V_t$ in Theorem 1. With unbiased minibatch sampling, GAR's expected squared deviation and its centered variance in the $\mathbf{P}^{-1}$ norm are both bounded by the variance of uniform SGD at that point. The size of the improvement depends on the per-example gradients at the ERM point. --- We will add the discussions of the ERM fixed point and the gradient variance comparison to the next revision. Thank you again for raising these questions. We would be happy to discuss them further.

Thank you for the detailed rebuttal. It resolves my main concerns.

As I went through the rebuttal and the paper, I found some minor remarks that, I think, would be good to address in a revised version of the paper:

(1) The oracle gap discussed in the paper currently carries an implicit limitation where an estimated gap likely leads to uniform weights in later training stages. This may reduce the effect of reweighting.

(2) Re. the tilted-risk fixed-point result: How informative is the result at τ=3\tau=3 when using mini-batch normalization compared to a full batch experiment in respect to the loss range across the entire dataset?

(3) I think, it could be informative to report the uniform token-weighted losses next to the reweighted ones. This would allow the reader to understand how much the objective drifts over time.

I maintain my positive score.

Small edit to my original review paper summary: "...GAR shows lower performance ..." should obviously be "...GAR shows higher performance..."

Thank you for your follow-up and for confirming that our response resolves your main concerns. We appreciate your suggestions and will address them in the revision as follows.

1. Nearly uniform weights with estimated loss gaps

If the goal is to recover the ERM minimizer while using reweighting to improve optimization, uniform weights at that minimizer are natural for preserving the original target. GAR satisfies this property in the exact full-batch formulation with oracle loss gaps: its weights are uniform at the ERM minimizer, which remains a fixed point.

The near-uniform weights in our reference-gap ablation have a different cause. Subtracting the losses of the 20k-step reference models made the estimated gaps very small and less dispersed late in training. At the temperature selected for raw losses, these scores give nearly uniform weights. This behavior follows from the finite reference models and the resulting score scale; it does not indicate that the ERM minimizer has been reached. We will test whether a longer-trained reference alleviates this performance drop.

In our experiments, raw-loss weighting achieved better validation performance than weighting based on estimated loss gaps, although it need not preserve the ERM minimizer. Our convergence analysis aims to minimize the ERM training objective, but preserving its minimizer need not yield the best validation performance. The altered weighting from raw losses may therefore contribute to their better validation performance in our experiments, without contradicting the training-loss guarantees. We will add a discussion of this distinction in the revision.

2. Objective-shift bound in the minibatch setting

The tilted-risk characterization and the bound in our earlier response concern exact full-batch GAR. At a full-batch fixed point, the weighted gradient vanishes, so the gradient-based correction in the weight equation vanishes as well. A zero expected minibatch update does not require the weighted gradient to vanish in each minibatch, so that characterization does not directly extend.

Nevertheless, an objective-shift bound can be derived directly for minibatch GAR. Under the paper's convexity and gradient-variation assumptions, consider GAR using current raw losses, full gradients, an exact weight solve, and a fixed preconditioner. For τ>0\tau>0 and independently and uniformly sampled minibatches of size bb, any point x^\widehat x satisfying EB[GB(x^)wB(x^)]=0\mathbb E_B[G_B(\widehat x)w_B(\widehat x)]=0 obeys

\le \frac{\mathbb E_B[(\Delta_B^\star)^2]}{8\tau} +\frac{\eta\sigma^2}{4b},$$ where $\Delta_B^\star=\max_{i\in B}f_i(x^\star)-\min_{i\in B}f_i(x^\star)$ and $\sigma^2$ is the component-gradient variation bound in Assumption 2. This controls the possible ERM objective shift through the loss variation at the optimum, the temperature, and minibatch gradient variance. At $\tau=3$, the first term is $\mathbb E_B[(\Delta_B^\star)^2]/24$. In LLM pretraining, however, the ERM minimizer and its per-example losses are unavailable, so we cannot evaluate this bound numerically or claim that it is tight for our experiments. The point of this analysis is to characterize these dependencies under the stated assumptions. We will include the bound and clarify its assumptions and scope in the revision. ## 3. Uniform and reweighted losses All evaluation log-perplexities reported in the paper use the standard token-averaged loss without GAR reweighting. We will state this explicitly. The training weights depend on the current model and minibatch, so even the same example can receive different weights in different batches; a reweighted evaluation loss would therefore also depend on how the evaluation batches are constructed. We therefore do not see a natural, batch-independent reweighted loss to report alongside the values in Table 1. Instead, to illustrate the effect of reweighting, we will add controlled checkpoint comparisons on matched minibatches. These will report the reweighted loss alongside the uniform token-averaged loss and compare the per-example loss changes after one update under uniform weighting and GAR. --- We will incorporate these discussions, the minibatch bound, and the additional checkpoint and reference-model experiments into the next revision. Thank you again for the constructive feedback.
Reviewer_YiH9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how sample reweighting affects optimization. It shows that weighting samples based only on their losses can lead to poor conditioning and unstable updates. The authors propose GAR, which also considers gradient information, and provide theoretical analysis and language-model experiments.

Critical strengths
  1. The counterexamples clearly show why loss-based reweighting can cause optimization problems.

  2. The proposed method has theoretical support under the stated assumptions

Critical weaknesses
  1. The practical implementation uses several approximations that are not covered by the main theoretical guarantees. Their effects are not sufficiently evaluated.

  2. The empirical results do not include multiple-seed runs, making it difficult to assess whether the reported gains are robust to training randomness. This is a relatively minor concern, given the relatively consistent improvements reported across the evaluated model sizes and datasets.

  3. The empirical results lack measurements of the additional computational cost.

Questions for authors
  1. How sensitive are the results to the approximations used in the practical implementation?

  2. Do the improvements remain consistent under the same training-time budget?

Suggestions for future work:

I encourage the authors to broaden the empirical evaluation by considering additional baselines and applications beyond language modeling, such as computer vision.

It would also be interesting to study how GAR compares with or complements second-order optimization methods.

1 comment

Thank you for the helpful review. We appreciate your recognition that the counterexamples illustrate the optimization issues with loss-based reweighting and that GAR has theoretical support under the stated assumptions.

W1 + Q1: Sensitivity to the implementation choices

Indeed, our practical implementation uses raw look-ahead losses in place of oracle loss gaps, last-layer gradients in place of full-model gradients, and three fixed-point iterations to compute the weights. Using more iterations or full-model gradients changes the checkpoint weights only slightly. Using estimated loss gaps changes the weights substantially, and the two 20k runs with these scores match or underperform uniform AdamW. Please see the General Comment for the detailed checkpoint and training results, as well as the discussions.

W2: Variation across training seeds

We repeated the 120M comparison of uniform AdamW and GAR with three seeds per method. GAR achieved lower final average log-perplexity in all three seeds (3.271±0.0063.271\pm0.006 versus 3.311±0.0033.311\pm0.003). Please see the General Comment for the per-seed results.

W3 + Q2: Computational cost

The main contribution of this paper is theoretical, and the language model experiments serve as a proof of concept for GAR in pretraining. We measured wall-clock time for 20k steps on the 120M model, using one A100 and batches of 32 sequences of 512 tokens for both methods. The current implementation of GAR is substantially slower than uniform AdamW (about 22–23 hours versus 7.4 hours).

After burn-in, the overhead comes from an extra forward pass at the Adam look-ahead point and three fixed-point iterations to compute the weights. Each iteration requires a vector–Jacobian product, an application of the inverse Adam preconditioner, and a Jacobian–vector product (Appendix F).

Our log-perplexity comparisons use matched training steps. We leave a more efficient implementation and a comparison at matched wall-clock time to future work. One possible way to reduce the overhead is to score the examples at the current iterate and reuse the training forward pass. In ablation (e) of the General Comment, this change has little effect on the weights, and the 20k run finishes about 0.0040.004 above default GAR on the same seed.

Suggestions for future work

Thank you for suggesting a broader evaluation. Our work was motivated by Sow et al. (2025), who proposed a convergence analysis of loss-based sample reweighting and applied their method to language model pretraining. We also explored applying GAR to computer vision, where many widely used architectures include BatchNorm. Because BatchNorm uses shared minibatch statistics, computing per-example gradients independently can give different results from differentiating each example's loss through the shared minibatch forward pass. This complicates the computation of GAR's weights in these architectures, so we focused our experiments on the language model pretraining setting studied by Sow et al.

Thank you also for suggesting second-order methods. Extending GAR to these methods and to optimizers such as Muon raises the question of how to account for their update rules when choosing the sample weights. We see this as an important direction for future work.


We will add the timing comparison and explain the main sources of computational overhead in the next revision. Thank you again for raising these question, and we would be happy to discuss them further.

Comments on the paper

2 comments

We thank the reviewers for their feedback. We would like to first note that the paper's main contributions are theoretical. We show how loss-based reweighting can slow optimization and establish stability and convergence guarantees for GAR under the assumptions in the paper. The language model pretraining experiments serve as a proof of concept for applying GAR in practice. Much of the feedback focuses on these experiments, and we address those concerns with the additional results below. We hope the reviewers and area chair will give particular consideration to our theoretical results in their overall assessment.

Below, we report additional experiments on the 120M model, comparing the weights at fixed checkpoints and final evaluation log-perplexity after 20k training steps.

Weight and training ablations

For the weight comparisons, we used all 20 checkpoints (1k, 2k, …\ldots, 20k) from one GAR run with τ=3\tau=3 (seed 42). At each checkpoint we recomputed weights on the same 64 fresh minibatches of 32 sequences, holding the model and optimizer state fixed and disabling dropout. The table reports mean total variation (TV) distance to the default GAR weights, TV(p,q)=12∥p−q∥1\mathrm{TV}(\mathbf{p},\mathbf{q})=\frac{1}{2}\left\lVert \mathbf{p}-\mathbf{q}\right\rVert_1, at selected checkpoints. Default GAR uses three fixed-point iterations, last-layer gradients, and raw look-ahead losses.

The last column reports final evaluation log-perplexity from separate training runs, averaged over the seven domains in Table 1. Uniform AdamW, default GAR, and the ablation without the quadratic term (d) are reported as mean ±\pm SD over three seeds. The other training ablations use a single seed 42. Each ablation uses τ=3\tau=3 and changes one setting from default GAR. Dashes mark comparisons without a value for the corresponding experiment.

Method or ablation1k2k5k10k15k20kFinal log-perp. (20k steps)
Uniform AdamW——————3.311 ± 0.003
GAR (τ=3\tau=3)——————3.271 ± 0.006
Approximations in the implementation
(a) Ten fixed-point iterations2.7×10⁻⁵6.4×10⁻⁶1.8×10⁻⁶2.7×10⁻⁷1.1×10⁻⁷2.4×10⁻⁸3.270
(b) Full-model gradients0.01060.00580.00620.00320.00220.0011—
(c1) Loss gaps (AdamW-42)0.1710.1740.1740.1740.1760.1763.330
(c2) Loss gaps (GAR-42)0.1610.1640.1640.1650.1660.1663.314
Effect of the quadratic term
(d) No quadratic term (αt=0\alpha_t=0)0.00840.00530.00330.00190.00120.00073.285 ± 0.002
Potential saving of a forward pass
(e) Score at the current iterate0.00110.00110.00120.00030.00020.00013.282

Fixed-point iterations and last-layer gradients.

In (a), ten iterations yield weights close to a fixed point on the checkpoints we examined. The mean TV between the three- and ten-iteration weights is below 3×10−53\times10^{-5} at every checkpoint. The 20k run with ten iterations reaches 3.2703.270, about 0.0080.008 lower than default GAR on the same seed and within the range of the three default GAR seeds.

In (b), we use full-model gradients in the weight calculation in place of the last-layer approximation. Mean TV is 0.01060.0106 at 1k and 0.00110.0011 at 20k, and the Spearman correlation with the weights based on last-layer gradients is at least 0.990.99 throughout. Because using full-model gradients for weight selection is computationally expensive, we evaluated (b) only at fixed checkpoints.

As (d) shows, small weight differences at fixed checkpoints can still affect final performance. We would expect comparable performance from removing the last-layer approximation, given the high agreement in (b) and the slight improvement from refining the weight calculation on one seed in (a).

Raw losses and reference loss gaps.

Our default implementation uses raw losses as observable scores, following Sow et al. (2025). Computing oracle gaps would require knowing the per-example losses at an ERM minimizer x⋆\mathbf{x}^\star, which is unavailable in pretraining. In (c1)–(c2), we subtracted each example's loss at a fixed 20k-step reference model from its look-ahead loss. The references are uniform AdamW seed 42 and GAR seed 42 (the endpoint of the trajectory being analyzed). At a given evaluation point, replacing x⋆\mathbf{x}^\star with xref\mathbf{x}_{\mathrm{ref}} adds a sample-dependent offset fi(x⋆)−fi(xref)f_i(\mathbf{x}^\star)-f_i(\mathbf{x}_{\mathrm{ref}}) to the oracle gap fi(xt)−fi(x⋆)f_i(\mathbf{x}_t)-f_i(\mathbf{x}^\star).

Across the two references and all checkpoints, mean TV to default GAR ranges from about 0.1610.161 to 0.1770.177. The table below shows the within-minibatch spread of the scores and how far the resulting weights are from uniform, evaluated on the default GAR checkpoints. As training progresses, the reference gaps vary less across examples in a minibatch. At τ=3\tau=3, which was selected using raw losses, this narrower spread gives nearly uniform weights late in training. The mean TV to uniform falls from 0.0790.079 at 1k to 0.0110.011 at 20k with the AdamW-42 reference, and from 0.0750.075 to 0.00050.0005 with the GAR-42 reference. Default GAR stays near 0.160.16.

Score or weight statistic1k5k10k20k
Within-minibatch standard deviation of scores
Raw look-ahead loss1.261.251.281.26
Loss gaps, AdamW-42 reference0.640.280.110.08
Loss gaps, GAR-42 reference0.610.250.090.001
Mean TV from uniform weights
Default GAR0.1540.1630.1680.166
Loss gaps, AdamW-42 reference0.0790.0330.0130.011
Loss gaps, GAR-42 reference0.0750.0310.0110.0005

The GAR-42 gaps nearly vanish at 20k because this reference is the endpoint of the run used for the checkpoint comparisons. The score spread also shrinks with the AdamW reference. In the separate full 20k-step training runs, the GAR-42 reference gives 3.3143.314, close to uniform AdamW on the same seed. The AdamW-42 reference gives 3.3303.330, about 0.0160.016 higher than uniform AdamW.

The convergence guarantee in Theorem 4 applies to minibatch SGD with oracle loss gaps computed at an ERM minimizer. Our ablation estimates these gaps using 20k-step checkpoints, with the temperature selected for raw losses. At this temperature, the estimated gaps yield nearly uniform weights at late checkpoints. Due to the short rebuttal period, we did not have time to test how references from larger models or longer training runs would affect the estimated gaps and the resulting weights.

Effect of the quadratic term.

In (d), we isolate the effect of the quadratic term by setting αt=0\alpha_t=0 and keeping the raw look-ahead losses and temperature fixed. From the perspective of Proposition 4, this ablation corresponds to comparing an explicit exponential reweighting step with GAR's implicit step. Removing the quadratic term changes the weights only slightly at matched checkpoints, with mean TV 0.00840.0084 at 1k and 0.00070.0007 at 20k. In the full 20k-step runs, removing the term increases final log-perplexity on all three matched seeds by 0.0060.006, 0.0200.020, and 0.0140.014. The three-seed result of 3.285±0.0023.285\pm0.002 is close to the LinUpper result of 3.2843.284 reported for the 120M model in Table 1.

Potential saving of a forward pass.

In (e), replacing look-ahead losses with current-iterate losses gave mean TV at most 0.00130.0013 across the 20 checkpoints. The 20k run reaches 3.2823.282, about 0.0040.004 higher than default GAR on the same seed. The current-iterate scores can be obtained from the training forward pass, so this variant can remove the extra forward pass at the look-ahead point.

Variation across training seeds

We ran uniform AdamW, GAR (τ=3\tau=3), and the ablation without the quadratic term with seeds 42, 43, and 44 on the 120M model. The table reports average evaluation log-perplexity across the same seven domains as Table 1, at 20,000 steps. Lower is better.

MethodSeed 42Seed 43Seed 44Mean ± SD
Uniform AdamW3.3143.3083.3113.311 ± 0.003
GAR (τ=3\tau=3)3.2783.2673.2693.271 ± 0.006
GAR with αt=0\alpha_t=03.2843.2873.2833.285 ± 0.002

GAR improves on uniform AdamW in all three seeds, by 0.0360.036–0.0420.042 in average log-perplexity. The average improvement is 0.0400.040, compared with 0.0330.033 in Table 1.


The reviewers' feedback helped us strengthen the empirical evaluation. We will incorporate these additional results and their interpretation into the next revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.