- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how dynamic loss-based sample reweighting affects optimization stability. It shows that component-gradient disagreement can worsen local conditioning and produce attracting two-cycles even on a well-conditioned quadratic objective. It proposes Geometry-Aware Reweighting (GAR), proves convergence and stability guarantees for an idealized version, and evaluates an AdamW adaptation on 120M–300M language models.
Critical strengths
The quadratic construction isolates the effect of reweighting: the average objective remains well conditioned and its optimum remains a fixed point, yet loss-only weighting can converge arbitrarily slowly even with the optimal scalar step size. The comparison between explicit loss weighting and GAR's implicit update provides a useful explanation of the instability and a principled motivation for incorporating gradient geometry.
Critical weaknesses
-
The guarantees do not justify the implemented algorithm. Theorem 4 and Proposition 3 rely on unavailable oracle gaps, . Replacing these gaps with observable losses can change the fixed point when the component losses at the ERM optimum differ. The implementation additionally uses last-layer gradients and only three undamped fixed-point iterations, without approximation-error or solver-convergence guarantees. The practical solver can lose the central stability benefit even before introducing AdamW.
-
The experiment is too limited to establish a practical benefit. Results consist of final-step log-perplexities in Table 1 on one training corpus. At , average gains over LinUpper are just 0.012–0.033; at , GAR is worse than both baselines on average at every model size. The paper reports three temperatures and inherits training settings from LinUpper, but does not explain the selection criterion or comparative tuning budgets. There are also no ablations separating the geometry correction from look-ahead scoring, or runtime measurements accounting for GAR's additional computation. These omissions make the robustness, source, and cost of the gains difficult to assess.
-
The theoretical contribution needs clearer discussions. Section 1.1 and Appendix A discuss relevant methods, and Section 4.1 acknowledges that TERM already identifies the covariance contribution to curvature. However, the discussion does not sufficiently separate what is new in the general weighting-map analysis and GAR's stability mechanism from existing analyses. This limits my assessment of the theoretical novelty beyond the useful counterexample.
Questions for authors
See critical weaknesses.
3 comments
Thank you for the response and the additional ablations. The clarification relative to TERM addresses my positioning concern, and the ablation supports the role of the geometry term. Two gaps remain:
- Theory vs. implementation. The guarantees cover oracle gaps, but the gains come from raw losses. With estimated gaps, the weights become nearly uniform and the gain disappears. So the theory does not explain the method that works. The ten-iteration ablation suggests three iterations are enough here, but no condition rules out the divergence in the Eq. (7) example.
- Practical cost. GAR runs about 3$\times$ slower than uniform AdamW for small gains over LinUpper, and it is not compared against a compute-matched baseline. Current-iterate scoring could remove one forward pass, but its runtime is not reported. GAR was also the only method that received a temperature sweep.
I will raise my score to 3, but these gaps should be addressed for publication.
Thank you for the follow-up and for recognizing the clarification relative to TERM and the ablation of the quadratic term. We will address the remaining concerns in the revision as follows.
1. Theory versus implementation
Oracle gaps and raw losses
The oracle-gap convergence guarantee does not directly cover the raw-loss AdamW experiments. We will expand the discussion of this distinction and its implications.
Our experiments with 20k-step reference models illustrate the difficulty of estimating the oracle gaps. These references introduce example-dependent errors, and the estimated gaps become less dispersed late in training. At the temperature selected for raw losses, the weights then approach uniform and the final performance is close to uniform AdamW. We will test whether a longer-trained reference alleviates this performance drop.
In our experiments, raw-loss weighting achieved better validation performance than weighting based on estimated loss gaps, although it need not preserve the ERM minimizer. Our convergence analysis aims to minimize the ERM training objective, but preserving its minimizer need not yield the best validation performance. The altered weighting from raw losses may therefore contribute to their better validation performance in our experiments, without contradicting the training-loss guarantees. We will add a discussion of this distinction in the revision.
We will also include a convex minibatch analysis that bounds the ERM objective shift for exact raw-loss GAR with a fixed preconditioner in terms of the component losses at the optimum, temperature, and gradient variation; please see point 2 of our follow-up response to Reviewer XV43 for the bound and its assumptions.
Fixed-point iterations
One clarification is that the GAR stability result on the example in Eq. (7) also applies to raw-loss scores: the component losses at the ERM optimum are equal in that example, so raw losses and oracle gaps differ only by a common additive constant. However, this result assumes an exact solution of the weight-selection problem. Our theory does not guarantee convergence of the fixed-point iteration used to approximate that solution.
In our checkpoint experiments, the spectral radius of the fixed-point iteration Jacobian at the computed solutions was at most across all examined minibatches and checkpoints, and the three- and ten-iteration weights were very close. We did not observe convergence problems in these experiments. We will clarify the lack of convergence guarantees for the fixed-point iteration and also discuss the numerical results.
2. Practical cost and tuning
Runtime and comparison budget
We will report the measured runtime explicitly and make clear that the performance comparisons use matched training steps. Our current 120M implementation is about three times slower than uniform AdamW, and these results do not establish an advantage at matched compute. The LM experiments were intended as a proof of concept for the AdamW adaptation, and developing an efficient implementation is important future work.
In the 120M experiments on one A100, scoring at the current iterate instead of the look-ahead point reduced median step time by approximately 7%. Increasing the number of fixed-point iterations from three to ten increased median step time by approximately 95%, suggesting that the fixed-point weight computation is a major bottleneck. Further savings may be possible by optimizing the Jacobian–vector product computations within these iterations. We will include these measurements and the total runtimes of the ablation runs alongside their final performance in the revision.
Temperature tuning and the LinUpper baseline
We used the public LinUpper implementation of Sow et al. (2025) and followed their experimental setup. Our reproduced LinUpper results are comparable to those reported in Table 1 of Sow et al. (2025): the average evaluation log-perplexities are , , and at 120M, 210M, and 300M, respectively, compared with their reported , , and . In particular, our 120M and 210M results improve upon the published values. This provides evidence that our comparison uses a competitive reproduction of LinUpper. We will include the comparison with the published LinUpper results in the revision.
We will incorporate these discussions and measurements into the next revision, with explicit statements of the theoretical scope, empirical comparison budget, and tuning procedure. We hope these additions address the remaining concerns. Thank you for the constructive feedback.
Thank you for the careful review. We appreciate your recognition that the quadratic example isolates slowdown at a preserved ERM fixed point and that the explicit and implicit updates help explain the role of gradient geometry.
W1: Oracle gaps, gradient approximation, and fixed-point iterations
The theoretical guarantees use oracle gaps and an exact weight solve. Please see the General Comment for ablations of all three implementation choices. Using estimated loss gaps substantially changes the weights and gives final log-perplexity comparable to or higher than uniform AdamW. The gradient and fixed-point comparisons show only small changes in the checkpoint weights, and the 20k run with more iterations finishes within the range of the default GAR runs.
W2: Empirical gains, temperature, ablations, and runtime
Empirical gains.
Table 1 compares models trained on the same SlimPajama subset for 20,000 steps. At 120M, 210M, and 300M, LinUpper lowers average log-perplexity relative to uniform AdamW by 0.021, 0.023, and 0.012, respectively. GAR at τ=3 improves on LinUpper by a further 0.012, 0.025, and 0.033. The improvement is comparable at 120M and larger at both 210M and 300M. Relative to LinUpper, this is a 1.2–3.2% reduction in the geometric mean of perplexity across the seven evaluation domains. In the General Comment, we also provide a three-seed comparison at 120M, where GAR improves over uniform AdamW on each seed.
Experimental setup and temperature.
We used the model configurations, training hyperparameters, and evaluation pipeline from the LinUpper implementation of Sow et al. (2025), as described in Section 7 and Appendix F. The temperature sweep was specific to GAR. We tried τ=1,3,10 at each model size, and the middle value, τ=3, consistently gave the lowest average log-perplexity (Table 1).
Quadratic term and look-ahead losses.
Please see the General Comment for these two ablations. Setting αt=0 isolates the effect of the quadratic term. The weights change only slightly at matched checkpoints, but final log-perplexity is higher than default GAR on all three matched seeds and close to the LinUpper result in Table 1. Scoring at the current iterate also changes the weights only slightly. Its seed-42 run gives slightly higher final log-perplexity than default GAR on the same seed.
Computational overhead.
The language model experiments are a proof of concept for GAR. On one A100, our current 120M implementation takes about 22–23 hours for 20k steps, compared with 7.4 hours for uniform AdamW. The overhead includes the look-ahead forward pass and the gradient products used to compute the weights. Please see our response to YiH9 (W3 + Q2) for the measurement setup and a possible way to reduce this overhead.
W3: What extends beyond TERM
TERM replaces the average training loss with a log-sum-exp of the individual losses. With positive tilt, a gradient step on that objective uses a weighted average of the individual gradients, with larger losses receiving larger weights. Li et al. (2023) show that disagreement between the individual gradients contributes an extra covariance term to the Hessian (Lemma 2, Eq. (13) in Li et al. (2023)).
We show that the same covariance appears for a much broader class of weighting rules. At the minimizer of the average loss, every example's loss gap is zero. Proposition 2 describes what happens when the iterate moves slightly away from that point. For any differentiable rule that treats examples symmetrically and uses only differences between their loss gaps, the update Jacobian contains the same covariance matrix multiplied by a scalar. This scalar measures how the weight of an example responds to a small change in its loss gap relative to the others. A positive scalar adds curvature and can force a smaller stable step size. In our example, this can make convergence arbitrarily slow even though the average loss is well conditioned and its minimizer remains a fixed point.
We then propose a method that mitigates this issue. GAR uses the squared norm of the weighted gradient when choosing the weights. The resulting implicit update uses the covariance to damp the step in directions where the sample gradients disagree. On our quadratic example, this keeps GAR stable throughout the step-size range where uniform gradient descent is stable. In minibatch SGD, the quadratic term also allows a direct comparison with uniform weights, giving a convergence bound for the original average loss with an explicit nonnegative improvement term relative to the uniform SGD bound.
We will clarify how our analysis and GAR extend beyond TERM in the next revision. We hope these clarifications address your concerns. We would be happy to discuss any remaining questions.