acceptodds
Under review as a conference paper at ICLR 2027

Distance-Aware Adaptive Scaling for Stochastic Optimization

Abstract

The ideal learning rate for each model parameter depends on both the magnitude of its stochastic gradients and the distance from its initial value to an optimal value. Given comparable gradient histories, a parameter that starts farther from its optimal value should take larger steps. However, standard adaptive methods such as AdaGrad and Adam adapt to stochastic gradient magnitudes without explicitly accounting for these coordinatewise distances. To address this limitation, we introduce bf distance-aware adaptive scaling, which uses the largest observed parameter displacement from a reference point as a surrogate for unavailable distance information. Coordinates with larger displacement receive larger relative updates, while those with small displacement retain approximately the original scaling. We integrate this mechanism into AdaGrad and AdamW, yielding DA-AdaGrad and DA-AdamW, both of which recover their base optimizers when distance-aware scaling is disabled. For DA-AdaGrad, we derive an AdaGrad-style regret bound and identify conditions under which distance-aware scaling improves the regret-bound expression on a fixed trajectory. Experiments on synthetic and real-world datasets demonstrate the effectiveness of the proposed approach.

Primary area:
optimization
Keywords:
Optimization, Adagrad, Adam, AdamW, Adaptive Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bypHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes distance-aware adaptive scaling, which incorporates coordinatewise parameter displacement into adaptive optimization methods. The authors introduce DA-AdaGrad and DA-AdamW, establish a regret bound for DA-AdaGrad, and evaluate the proposed methods on synthetic optimization problems, nonconvex classification, image classification.

Critical strengths

The proposed method is intuitive and relatively simple to incorporate into existing adaptive optimizers. I think this is a great attempt to exploit distance information in adaptive optimization.

Critical weaknesses

1. The empirical improvements appear limited and dependent on the experimental setting.

My main concern is whether the proposed method can consistently improve upon standard adaptive optimizers, particularly AdamW.

In Figure 1, DA-AdaGrad achieves almost identical performance to AdaGrad in several settings. The AdamW baseline is not tuned carefully. Although there are encouraging gains, I am not fully convinced that the current experimental evidence demonstrates a substantial and generalizable improvement over AdamW. More extensive experiments and stronger baseline comparisons would help establish the practical significance of the proposed method.

2. Concerns about the evaluation protocol in the CIFAR experiments.

For both CIFAR-10 and CIFAR-100, Table 1 reports the highest test accuracy achieved during each training run, rather than the test accuracy at a checkpoint selected using an independent validation set.

Questions for authors

See weakness.

1 comment

Thank you for your feedback. We address your comments below.

Q1: In Figure 1, DA-AdaGrad achieves almost identical performance to AdaGrad in several settings.

A: That is actually the purpose of this experiment. The experimental results in Figure 1 illustrate how the benefit depends on distance geometry. The worst case in Fig 1(a) corresponds to equal initial coordinate distances. This observation is consistent with the motivating derivation in Section 4.1, which shows that equal initial coordinate distances contribute only a common factor, resulting in the same relative coordinate scaling as Adagrad. As shown in Fig 1(b,c), the benefit grows when the same total squared initial distance is distributed more unevenly. Similarly, the distance magnitude from initial to the optimal solutions also matters. When it is small, there is little benefit of DA-AdaGrad as shown in Fig 1(e), and the benefit increases when distance magnitude increases at fixed relative dispersion, with curvature controlled and each method's learning rate tuned independently.

Q2: The AdamW baseline is not tuned carefully... For both CIFAR-10 and CIFAR-100, Table 1 reports the highest test accuracy achieved during each training run, rather than the test accuracy at a checkpoint selected using an independent validation set.

A: For AdamW, we tuned the learning rate while keeping (β₁, β₂) = (0.9, 0.999) fixed for both methods, following established practice in training deep learning vision models [1, 2]. We have also expanded the tuning grid for AdamW in our CIFAR-100 experiment to ensure the selected learning rate is on the interior. Our method retains the same 50-epoch, zero-weight-decay recipe, with γ additionally selected for DA-AdamW.

For CIFAR-10 and CIFAR-100, we reran the comparisons using epoch-50 validation accuracy over three tuning seeds on a separate 45,000/5,000 training/validation split. Final models are retrained on all training examples using ten new seeds, and test accuracy is measured once at epoch 50 and reported below:

DatasetAdamW accuracy (%)DA-AdamW accuracy (%)
CIFAR-1093.05 ± 0.1093.40 ± 0.20
CIFAR-10073.28 ± 0.2473.60 ± 0.58

Results are mean ± SD. DA-AdamW retains higher mean CIFAR accuracy after correcting both baseline selection and test reporting.

Q3: stronger evidence is needed for practical improvement beyond particular experimental settings.

A: The submitted IWSLT14 experiment extends the comparison to encoder–decoder Transformer translation: DA-AdamW improves both test BLEU and NLL in all six paired runs, with checkpoints selected by validation NLL (Figure 4).

[1] Chen et al. Symbolic Discovery of Optimization Algorithms. NeurIPS 2023, Appendix M.

[2] Liu et al. On the Variance of the Adaptive Learning Rate and Beyond. ICLR 2020, Appendix B.2.

Reviewer_pzcCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes distance-aware adaptive scaling, which uses historical parameter displacement as a proxy for the unknown distance to the optimum, integrates it into AdaGrad and AdamW, and evaluates its effectiveness through theoretical regret analysis, controlled synthetic experiments, and real-world learning tasks.

Critical strengths
  1. A theoretically motivated and analyzable extension of adaptive optimization.

The paper derives its motivation from the coordinatewise regret-bound tradeoff between gradient history and distance to the optimum. It further establishes an AdaGrad-style regret bound for DA-AdaGrad and identifies conditions under which the bound expression can be improved on a fixed trajectory. These results provide a meaningful theoretical foundation for the proposed method, rather than relying solely on empirical observations.

  1. Well-structured controlled experiments and encouraging empirical results.

The controlled experiments separately investigate distance heterogeneity, distance magnitude, and curvature while holding relevant factors fixed. This design provides more targeted evidence for the proposed motivation than standard benchmark comparisons alone. The additional improvements reported across nonconvex optimization, image classification, and machine translation further demonstrate the practical potential of the approach.

Critical weaknesses

My main concern lies at the insufficient connection between the theoretical motivation and the proposed optimization mechanism.

The paper motivates its method through the dependence of optimal coordinate-wise scaling on the distance to the optimum, but replaces this unavailable distance with maximum historical displacement without sufficiently establishing that the substitution preserves the intended decision-making principle.

While the theoretical analysis identifies conditions under which displacement-aware scaling improves a regret-bound expression, these conditions rely on properties of the optimization trajectory and do not establish that historical displacement reliably provides the distance information motivating the method. The empirical results demonstrate the effectiveness of the overall optimizer, but do not sufficiently isolate whether the improvements arise from the proposed distance-aware allocation mechanism or from more general displacement-dependent step-size adjustments.

This gap is consequential to the paper's central contribution. If distance awareness is intended as the key mechanism, stronger evidence connecting displacement to beneficial coordinate-wise decisions is necessary. If the method is instead presented primarily as an empirical optimization heuristic, the distance-based theoretical motivation provides weaker support for its claimed novelty, placing greater weight on comparative empirical evidence. The current work does not fully resolve this distinction.

Questions for authors
  1. Can more evidence for the proposed distance-aware mechanism be provided?

The paper motivates its method using the theoretical importance of coordinate-wise distances to the optimum, but the proposed displacement-based correction also changes the effective learning rates. What evidence establishes that the observed improvements arise specifically from the proposed coordinate-wise allocation of updates, rather than from more general displacement-dependent step-size amplification? In particular, could the authors provide comparisons with a global time-varying scaling factor or shuffled coordinate-wise displacement factors, under comparable hyperparameter tuning? Such evidence would materially affect my assessment of whether the theoretical motivation is adequately connected to the proposed mechanism.

  1. Independence of the CIFAR-100 evaluation

Appendix F.5 explicitly states that the official CIFAR-100 test set was used for both hyperparameter selection and final performance reporting. Could the authors provide results using a separate validation set for hyperparameter selection and an independent test evaluation? This is important to the reviewer because the reported improvement of DA-AdamW over AdamW is relatively modest (74.02% vs. 73.58%), and selecting hyperparameters on the test set may introduce selection bias. An independent evaluation would materially affect my confidence in the reported improvement.

3 comments

Thank you for your feedback. We address your comments below.

Q1: the improvements may arise from general displacement-dependent step-size amplification rather than the proposed coordinatewise allocation. In particular, could the authors provide comparisons with a global time-varying scaling factor or shuffled coordinate-wise displacement factors, under comparable hyperparameter tuning?

A: Following your suggestion, we added two ablations to the synthetic experiment: a global time-scaling method and a method with shuffled displacement multipliers. More precisely, letting:

A_{t,i} &:= \sum_{\tau=1}^{t} g_{\tau,i}^{2}, \\\\ \delta_{t,i}^{\max} &:= \max_{\tau \leq t} |w_{\tau,i}-w_{1,i}|, \\\\ \bar{\delta}_t &:= \frac{1}{m}\sum_{i=1}^{m}\delta_{t,i}^{\max}, \end{aligned}$$ we set the "Global" and "Shuffled" updates, respectively, as: $$\begin{aligned} w_{t,i} &= w_{t-1,i}-\eta\left(\epsilon+\frac{A_{t,i}}{1+\gamma\bar{\delta}_t^2}\right)^{-\frac{1}{2}}g_{t,i}, \\\\[6pt] w_{t,i} &= w_{t-1,i}-\eta\left(\epsilon+\frac{A_{t,i}}{1+\gamma(\delta_{t,\pi(i)}^{\max})^2}\right)^{-\frac{1}{2}}g_{t,i}, \end{aligned}$$ where $\pi$ is a fixed random permutation of the indices. Using the same settings and score as reported in the first row of Figure 1, with learning rates independently tuned for each method, the score differentials between Adagrad and each method were as follows (positive values mean better performance over Adagrad, negative values mean worse performance): | CV of squared initial distances | DA-AdaGrad | Global | Shuffled displacement | |---|---:|---:|---:| | 0.000 | −0.174 | −0.174 | −0.174 | | 1.436 | +0.082 | −0.126 | −0.156 | | 2.616 | +0.357 | −0.117 | −0.240 | While DA-AdaGrad improves over AdaGrad at larger CV, both the Global and Shuffled methods showed no improvement over AdaGrad in any of these settings. These results are presented for a single $\gamma = 10$, but DA-AdaGrad remains better than both ablation methods in each of the three highest-CV settings for $\gamma \in \{0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000\}$ (Appendix F.8, Tables 11–12). **Q2:** CIFAR-100 used the official test set for both hyperparameter selection and final reporting. **A:** We reran the comparison without using test accuracy for hyperparameter selection. Hyperparameters are selected by epoch-50 validation accuracy over three seeds using a fixed 45,000/5,000 training/validation split. The selected configurations are retrained on all training examples with ten new seeds and evaluated once on the test set at epoch 50. The training horizon and zero-weight-decay recipe are unchanged. The full tuning and evaluation protocol is given in Appendix F.5. AdamW obtains 73.28 ± 0.24% and DA-AdamW 73.60 ± 0.58% test accuracy (mean ± SD). The higher mean therefore remains when test accuracy is excluded from both hyperparameter and checkpoint selection in the rerun.
Reviewer_pzcCOpenReview ↗

Further clarification

Thank the authors for the rebuttal, which resolves some of my concerns.  

However, it is still unclear to me that, to what extent the conditions used in the theoretical analysis are actually met in the experiment? More specifically, could the authors provide some empirical evidence on 1) the relationship between historical displacement and actual distance, and 2) if the alignment conditions mentioned in the theory can be indeed met in the experiments? I hope to see more clarifications from the authors. Thanks.

We thank the reviewer for their rebuttal.

For our synthetic experiment, we have added a figure and analysis to Appendix F.8 (Figure 10 and Table 13) examining the relationship between historical displacement and the initial distance to the optimum. The results show that for coordinates that are initialized far from the optimal, the tracked displacement of the algorithm is very accurate, close to zeros. For coordinates that are initialized close to the optimal point, the estimation error are well controlled and stablized as the algorithm converges.

In Section 5, we denote the upper bound for a fixed trajectory and a given γ\gamma by Bγ∗B^{*}_{\gamma} and establish the following expansion (Appendix C.2):

Bγ∗B0∗=1−γ4(MP−MQ)+O(γ2).\frac{B^{\ast}_{\gamma}}{B^{\ast}_{0}} = 1 - \frac{\gamma}{4}(M_P - M_Q) + O(\gamma^2).

where B0∗B^{\ast}_{0} denotes the regret bound of AdaGrad.

Here, MPM_P and MQM_Q are two weighted means of the squared historical displacement, averaged over iterations tt and coordinates ii. Thus, for sufficiently small positive γ\gamma, MP>MQM_P > M_Q implies Bγ∗B0∗<1\frac{B^{\ast}_{\gamma}}{B^{\ast}_{0}} < 1.

In the the synthetic experiments, larger values of MP−MQM_P-M_Q correspond to greater improvements for DA-AdaGrad over AdaGrad. The table below shows the results for the same γ=10\gamma=10 setting used in our earlier rebuttal, with MP−MQM_P-M_Q evaluated on the AdaGrad trajectories:

Initial CV of di2d_i^2MP−MQM_P-M_QScore Δ\Delta (Positive favors DA-Adagrad)
0.000−0.042421−0.174220
0.378−0.032557−0.089480
1.436+0.261768+0.081860
2.343+0.737577+0.237084
2.616+0.988105+0.356787

These results have been added to Appendix F.8.

Reviewer_2YqDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors propose a distance-aware modification of adaptive gradient methods. The central motivation is that optimal coordinatewise learning rates depend not only on accumulated gradient magnitudes, as in AdaGrad, but also on the distances between the initial parameters and an optimal solution. Since these distances are unknown, the authors propose using the maximum observed displacement of each coordinate from a reference point as a surrogate.

The resulting method, DA-AdaGrad, modifies the usual AdaGrad denominator by a coordinatewise factor depending on the squared maximum displacement. The construction is also extended to AdamW and AMSGrad, using an exponential moving-average reference point.

On the theoretical side, the authors establish an AdaGrad-type regret bound for DA-AdaGrad in the convex setting. They then study conditions under which the resulting upper-bound expression can improve upon that of AdaGrad, particularly when the coordinatewise distances to the optimum are heterogeneous.

The experimental evaluation includes controlled quadratic problems, nonconvex classification with L1 regularization, image classification with ResNet and Vision Transformer architectures, and machine translation. The reported results generally favor the proposed distance-aware variants.

Critical strengths

I liked the mathematical analysis of the paper.

Critical weaknesses
  1. The theoretical improvement is established only for a fixed trajectory. The comparison between DA-AdaGrad and AdaGrad is performed by evaluating two bound expressions using the same iterates and gradients. However, changing the displacement coefficient changes the optimization trajectory. Consequently, the result does not establish that DA-AdaGrad has a smaller regret bound than AdaGrad when each method follows its own trajectory.

  2. The displacement surrogate is not necessarily a reliable estimate of the distance to an optimum. The maximum observed displacement can reflect stochastic fluctuations, overshooting, or movement in an unfavorable direction. A large displacement does not necessarily mean that a coordinate started far from the optimum. The proposed scaling may therefore increase the learning rate precisely when this is undesirable.

  3. The method introduces a displacement coefficient that must be selected in addition to the learning rate. Since the displacement factor directly increases the effective learning rate, its sensitivity to parameter scaling and initialization deserves further investigation. vestigation.

  4. The coordinatewise correction is a meaningful distinction, but the theoretical contribution remains closely related to classical variable-metric AdaGrad analysis.

Questions for authors

Can the authors comment about the weaknesses of their paper ?

3 comments

Thank you for your feedback. We address your comments below.

Q1: “Consequently, the result does not establish that DA-AdaGrad has a smaller regret bound than AdaGrad when each method follows its own trajectory.”

A: Agree. We have clarified this distinction in Section 5. We would like to note that: 1) Comparison of bounds on the same gradient sequence is also used in the original AdaGrad paper (Duchi et al., 2011, Section 1.2); 2) any improvement on the order of rate is impossible as T\sqrt{T} is the provable lower bound of worst-case regret under our considered setting.

However, the fixed-trajectory assumption makes the comparison tractable. The analysis also suggests that the benefit may depend on the dispersion and magnitude of initial coordinate distances, motivating the controlled experiments in Figure 1.

Q2: “The maximum observed displacement can reflect stochastic fluctuations, overshooting, or movement in an unfavorable direction. A large displacement does not necessarily mean that a coordinate started far from the optimum. The proposed scaling may therefore increase the learning rate precisely when this is undesirable.”

A: We agree that the maximum observed displacement only serves as a surrogate. It may not be accurate to capture the exact distance. However, the key is to leverage the relative magnitude of displacement of different coordinates so that we can scale different coordinates differently. Substantial updates with consistent signs accumulate displacement, whereas small or opposing updates can produce less movement.

Large early gradients can leave AdaGrad's subsequent effective learning rates small (Zeiler, 2012, Section 2.2.2). After an overshoot, the accumulated squared gradients remain even when the gradient reverses, which can slow the return toward the optimum. The displacement correction can help counteract this suppression during the return. Please notice that we still have a global learning rate parameter η\eta to control the overall magnitude of the learning rate and reduce the risk of undesirable behavior such as divergence.

Q3: the additional γ parameter warrants investigation of sensitivity to parameter scaling and initialization.

A: Sensitivity to γ\gamma: We have expanded the synthetic experiments to study the sensitivity of γ\gamma. For each γ∈{0.1,0.3,1,3,10,30,100,300,1000}\gamma \in \{0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000\}, we tuned our method’s learning rate independently and measured performance using the same metrics as in Figure 1. We find that the optimal gamma value correlates positively with dispersion in initial distances. The results are shown in Figure 8 and Table 9 in Appendix F.7 of the revision.

Sensitivity to initialization: Figure 1(e–h) in the paper has already examined sensitivity to initial distance magnitude for a fixed γ\gamma. To evaluate the relationship between optimal γ\gamma and average distance to the optimum, we have included evaluations of different γ\gamma values at a wide range of initial distances. Holding the dispersion constant, we varied the initial distances by factors a∈{0.1,0.3,1,3,10}a\in\{0.1,0.3,1,3,10\} at each tested γ\gamma, keeping the objective fixed. Table 10 and Figure 9 in Appendix F.7 show that smaller initial distances require larger γ\gamma for a substantial advantage over AdaGrad.

Q4: "The coordinatewise correction is a meaningful distinction, but the theoretical contribution remains closely related to classical variable-metric AdaGrad analysis."

A: We agree that our analysis builds on classical variable-metric AdaGrad. Our aim is to identify conditions under which estimating the distance term can improve optimization performance.

Duchi, Hazan and Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. JMLR, 2011, Section 1.2, pp. 2124–2125.

Zeiler, M. D. ADADELTA: An Adaptive Learning Rate Method. arXiv:1212.5701, 2012.

I thank the authors for their detailed answer. I had already noticed the global learning rate eta that can can balance for too large initial displacements. But if there is a aparameter to tune, than one loses the interest of distance aware adaptive scaling. I therefore keep my score for this paper.

AuthorsOpenReview ↗

Thank you!

We thank the reviewer for reading our rebuttal.

First, the presence of a tunable hyperparameter does not, by itself, diminish the paper’s contribution. Our main contribution is a method for correcting the coordinate-wise scaling of AdaGrad and other adaptive methods by accounting for each coordinate’s displacement from its initial value. Widely used adaptive algorithms also require learning-rate tuning.

Second, while parameter-free algorithms are an interesting research direction, developing such algorithms is not the focus of this paper. We believe the contribution should be assessed in terms of the proposed scaling correction and its theoretical and empirical benefits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.