- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The work considers steepest-descent and mirror-descent updates as noncommuting operators, then claims that composing them can retain controlled update magnitudes and induce an max-margin bias. It proposes HORST, an AdamW update with a multiplicative correction. Experiments show improvements in various settings.
Critical strengths
The proposing observation on optimizers are noncommutative operators is interesting. The performance of the proposing methods looks promising, and the authors do provide code for reproducibility.
Critical weaknesses
Theorem 4.7 is incorrect. I'm not sure it is repairable and do not read the proof but there is a simple try-and-fail counterexample as follows. Consider , , and . Entropy-after-sign descent gives , so the direction is proportional to . However, the unique solution of subject to is . This means the sign descent endpoint does not inherit the max-margin bias. The infinite-time and cannot be interchanged. Moreover, the theorem excludes the sparse limits, only covers normalized limits in , excluding vectors with zero coordinates.
This errors make me hesitate going through the rest of theoretical part of the paper.
Talking about the experiments, since this paper proposes a new optimizers, there are two direct and important factors the readers want to see, which are the performance and the runtime. While performance is a must have, I do not see any report on runtime. There is not report wall-clock training time, GPU-hours, measured optimizer overhead, or inference speedups. The hardware report is also very limited, only 4xA100 with 40gb (for vision experiments), which is considered small at current state of optimizer-proposing paper.
Questions for authors
-
I might be wrong at the Theorem 4.7. If I am, I hope the authors can explain. Otherwise, the authors should revisit the theorem and revise.
-
The efficiency aspect is importance. I hope the author can report the runtime, compared it to traditional AdamW to see the tradeoff between performance and efficiency of the proposing method.
3 comments
I thank the authors for their responses.
I noticed an issue with how HAM hyperparameters are chosen. In Appendix F, HAM gets its best validation accuracy with , but Table 2 uses because of better sparsity. Since the comparison is about accuracy at the same sparsity level, HAM should be tuned for accuracy, using a similar tuning process to HORST, and tested with the same seeds. HAM best reported result is still < HORST, but that does not fully address the comparison.
The reported 6.9% increase in runtime answers my original question for this setting. Table 1 also shows that HORST holds up better after pruning (though each optimizer was only run once). These results are useful evidence of better sparsity, but they do not establish the claimed bias.
I appreciate the improvements, but the gap of the baseline tuning issue still leaves me unconvinced. I keep my score at 2.
We thank the reviewer for their follow up and appreciate their acknowledgment that the runtime measurements address their original question and that the additional experiments provide useful evidence.
We agree that validation accuracy at the prescribed sparsity should determine hyperparameter selection. HAM’s highest single-run validation accuracy in the expanded sweep is %, obtained with and , compared with HORST’s mean accuracy of % at % sparsity. Following your suggestion, we evaluate this accuracy-selected HAM configuration at all three sparsity levels using the same three seeds as in Table 2. At sparsity, HAM achieves $59.46\pm1.40$%, compared with $69.41\pm2.75$% for HORST, leaving a substantial gap of percentage points. The results are reported below.
HAM with alpha = 20 and beta = 0. Mean top-1 accuracy and approximate 95% confidence intervals across three seeds.
| Sparsity (%) | Top-1 accuracy (%) |
|---|---|
| 70 | |
| 80 | |
| 90 |
Furthermore, we extend the sweep and cover now and . The evaluated configurations include the entire hyperparameter grid used for HORST, together with additional values of . We report the final accuracy top 1% accuracy values below:
| HAM alpha | beta = 0 | beta = 0.001 | beta = 0.01 |
|---|---|---|---|
| 5 | 58.46 | 59.59 | 57.37 |
| 10 | 58.15 | 58.89 | 58.68 |
| 20 | 60.69 | 58.74 | 58.00 |
| 50 | 56.61 | 55.72 | 56.94 |
| 200 | 58.72 | 58.92 | 59.31 |
| 500 | 58.52 | 59.28 | 58.54 |
| 2000 | 59.16 | 57.02 | 59.46 |
Moreover, at % and % sparsity, we additionally swept HAM over the same hyperparameter grid used for HORST, as reported in the Tables 6 and 7 in the appendix. None of these configurations improves on HAM’s mean accuracies with and in Table 2, and HORST retains a substantial accuracy advantage at both sparsity levels.
Regarding the sparsity promotion effect, we would also like to highlight the complementary weight distribution evidence. Figure 3 shows stronger concentration near zero among the unmasked weights with AC/DC training, while Figure 10 shows a similar effect in a densely trained language model. Together with the pruning robustness in Table 1 and the performance improvements across vision and language tasks, particularly at higher sparsity, these observations provide empirical support for improved sparsifiability.
We hope these additional results address your remaining concerns and would appreciate it if you could reconsider your score in light of this evidence.
We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your concerns in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.
W1. The statement in Theorem 4.7 assumes p<∞ and therefore excludes the counter-example. However, thank you for sharing it to revise our conjecture about the possibility of the limit extension. We have updated the manuscript and include the provided example in the appendix at line 1102. We replace the conjecture in the main text with this statement at line 377:
This characterization concerns strictly positive limit directions at p<∞ and can fail at the exact sign descent endpoint p=∞ (Example D.4). The order of composition still affects the update at this endpoint. By Lemma 4.3, applying sign descent after a separable mirror operator erases the coordinatewise rescaling and recovers ordinary sign descent with a L∞ max-margin bias in separable linear classification. Applying the mirror operator last retains its rescaling in the update. This distinction motivates placing the mirror correction after the AdamW step in HORST. It is true that Theorem 4.7 characterizes strictly positive limit directions, and we have clarified this scope using two arguments:
Why do we target an (L1) bias?
The (L1) norm gives a convex way to favor small coordinates, and the resulting solution can remain dense. There is also a statistical motivation. In sparse regression, Hastie et al. (2020) find that Lasso generalizes better than best subset selection when the signal is weak relative to the noise.
The purpose of HORST.
The entropy correction is mainly designed to temper the (L∞)-type bias associated with sign descent type updates. To encourage more weights to have smaller magnitudes, we use other algorithms to do the pruning in the experimental validation section. We would like to highlight the ability of improving sparsifiability through dense training. In Table 1, pruning 40% of the weights in dense-trained DeiT-Small reduces top-1 accuracy by 2.71 percentage points with HORST and 9.31 with AdamW. These results demonstrate that HORST improves sparsifiability even when the model is trained densely. Note that the operator characterization, noncommutativity result, and gradient-scale boundedness of the composition do not rely on Theorem 4.7.
Hastie, Trevor J. et al. “Best Subset, Forward Stepwise or Lasso? Analysis and Recommendations Based on Extensive Comparisons.” Statistical Science (2020): n. pag.
W2. We agree that computational cost is an important practical consideration. As discussed in Section 5, HORST reuses the Adam moments already computed during the optimizer step, so its multiplicative correction requires no additional forward or backward passes. In addition, we now report the average wall-clock time per epoch for a DeiT-small trained on four NVIDIA GH200 GPUs, excluding warm-up epochs: 72 seconds for AdamW, 75 seconds for HAM, and 77 seconds for HORST. HORST therefore incurs a modest overhead of 6.9% relative to AdamW and 2.7% relative to HAM, compared to the substantial performance gain in accuracy. In the main text, we will qualify this in Section 5 by adding the following sentence: On DeiT-Small, HORST incurs a modest wall-clock overhead per epoch of 6.9% relative to AdamW and 2.7% relative to HAM, with timing details provided in Appendix F.
Questions
See weakness