- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes distance-aware adaptive scaling, which incorporates coordinatewise parameter displacement into adaptive optimization methods. The authors introduce DA-AdaGrad and DA-AdamW, establish a regret bound for DA-AdaGrad, and evaluate the proposed methods on synthetic optimization problems, nonconvex classification, image classification.
Critical strengths
The proposed method is intuitive and relatively simple to incorporate into existing adaptive optimizers. I think this is a great attempt to exploit distance information in adaptive optimization.
Critical weaknesses
1. The empirical improvements appear limited and dependent on the experimental setting.
My main concern is whether the proposed method can consistently improve upon standard adaptive optimizers, particularly AdamW.
In Figure 1, DA-AdaGrad achieves almost identical performance to AdaGrad in several settings. The AdamW baseline is not tuned carefully. Although there are encouraging gains, I am not fully convinced that the current experimental evidence demonstrates a substantial and generalizable improvement over AdamW. More extensive experiments and stronger baseline comparisons would help establish the practical significance of the proposed method.
2. Concerns about the evaluation protocol in the CIFAR experiments.
For both CIFAR-10 and CIFAR-100, Table 1 reports the highest test accuracy achieved during each training run, rather than the test accuracy at a checkpoint selected using an independent validation set.
Questions for authors
See weakness.
Thank you for your feedback. We address your comments below.
Q1: In Figure 1, DA-AdaGrad achieves almost identical performance to AdaGrad in several settings.
A: That is actually the purpose of this experiment. The experimental results in Figure 1 illustrate how the benefit depends on distance geometry. The worst case in Fig 1(a) corresponds to equal initial coordinate distances. This observation is consistent with the motivating derivation in Section 4.1, which shows that equal initial coordinate distances contribute only a common factor, resulting in the same relative coordinate scaling as Adagrad. As shown in Fig 1(b,c), the benefit grows when the same total squared initial distance is distributed more unevenly. Similarly, the distance magnitude from initial to the optimal solutions also matters. When it is small, there is little benefit of DA-AdaGrad as shown in Fig 1(e), and the benefit increases when distance magnitude increases at fixed relative dispersion, with curvature controlled and each method's learning rate tuned independently.
Q2: The AdamW baseline is not tuned carefully... For both CIFAR-10 and CIFAR-100, Table 1 reports the highest test accuracy achieved during each training run, rather than the test accuracy at a checkpoint selected using an independent validation set.
A: For AdamW, we tuned the learning rate while keeping (β₁, β₂) = (0.9, 0.999) fixed for both methods, following established practice in training deep learning vision models [1, 2]. We have also expanded the tuning grid for AdamW in our CIFAR-100 experiment to ensure the selected learning rate is on the interior. Our method retains the same 50-epoch, zero-weight-decay recipe, with γ additionally selected for DA-AdamW.
For CIFAR-10 and CIFAR-100, we reran the comparisons using epoch-50 validation accuracy over three tuning seeds on a separate 45,000/5,000 training/validation split. Final models are retrained on all training examples using ten new seeds, and test accuracy is measured once at epoch 50 and reported below:
Results are mean ± SD. DA-AdamW retains higher mean CIFAR accuracy after correcting both baseline selection and test reporting.
Q3: stronger evidence is needed for practical improvement beyond particular experimental settings.
A: The submitted IWSLT14 experiment extends the comparison to encoder–decoder Transformer translation: DA-AdamW improves both test BLEU and NLL in all six paired runs, with checkpoints selected by validation NLL (Figure 4).
[1] Chen et al. Symbolic Discovery of Optimization Algorithms. NeurIPS 2023, Appendix M.
[2] Liu et al. On the Variance of the Adaptive Learning Rate and Beyond. ICLR 2020, Appendix B.2.