Loss Alone Is Not Enough: Stability and Geometry in Dynamic Sample Reweighting

Dynamic sample reweighting can emphasize informative examples during training, and several recent methods use the per-sample loss to determine these weights. Per-sample loss indicates which samples are difficult, but it does not reveal how emphasizing those samples changes the geometry of the optimization update. We show that this distinction can matter even on simple strongly convex problems. For a broad class of loss-only reweighting schemes, upweighting higher-loss examples can lead to an arbitrarily worse local convergence rate than gradient descent with uniform sample weights. Moreover, for exponential loss-based reweighting, increasing the step size past a threshold produces a stable two-cycle (a persistent oscillation) even when uniform-weight gradient descent remains stable. To address this limitation, we propose **Geometry-Aware Reweighting (GAR)**, which chooses weights using both loss and gradient information. For minibatch SGD, GAR satisfies the standard upper bound of uniform-weight SGD minus an explicit nonnegative term, and it avoids the instability in our hard example. We also derive practical extensions to momentum and AdamW. Experiments on 120M–300M language models show that GAR achieves lower evaluation log-perplexity than both uniform-weight AdamW and the loss-based LinUpper reweighting (Sow et al., 2025).