acceptodds
Under review as a conference paper at ICLR 2027

HORST: Composing Optimizer Geometries for Sparse Transformers

Abstract

Sparsifying transformers remains a fundamental challenge, as standard optimizers fail to simultaneously encourage sparsity and maintain training stability. Effective adaptive optimizers exhibit an implicit bias favoring stability, yet, sparsity requires an bias. To integrate sparsity, we propose a composition of optimizer steps, which we cast as non-commutative operators to analyze and combine their optimization geometries in a principled way. This yields HORST (Hyperbolic Operator for Robust Sparse Training), a modular optimizer that inherits stability from adaptive methods while inducing sparsity bias through a hyperbolic mirror map. Our experiments demonstrate its utility for sparse training of transformers on both vision and language tasks. HORST consistently improves over AdamW baselines with the largest gains at higher sparsity and also outperforms the closely related HAM optimizer in this regime.

Primary area:
optimization
Keywords:
Sparsity, AdamW, sign descent, implicit regularization, steepest descent, mirror descent, operators, composition

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pYrxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The work considers steepest-descent and mirror-descent updates as noncommuting operators, then claims that composing them can retain controlled update magnitudes and induce an ℓ1\ell_1 max-margin bias. It proposes HORST, an AdamW update with a multiplicative correction. Experiments show improvements in various settings.

Critical strengths

The proposing observation on optimizers are noncommutative operators is interesting. The performance of the proposing methods looks promising, and the authors do provide code for reproducibility.

Critical weaknesses

Theorem 4.7 is incorrect. I'm not sure it is repairable and do not read the proof but there is a simple try-and-fail counterexample as follows. Consider x=(2,1)x=(2,1), y=1y=1, and θ0=(1,1)\theta_0=(1,1). Entropy-after-sign descent gives θ˙=(θ1,θ2)\dot\theta=(\theta_1,\theta_2), so the direction is proportional to (1,1)(1,1). However, the unique solution of min⁡θ∣θ∣1\min_\theta |\theta|_1 subject to 2θ1+θ2≥12\theta_1+\theta_2\geq1 is (1/2,0)(1/2,0). This means the sign descent endpoint does not inherit the ℓ1\ell_1 max-margin bias. The infinite-time and p→∞p\to\infty cannot be interchanged. Moreover, the theorem excludes the sparse limits, only covers normalized limits in R>0n\mathbb{R}^n_{>0}, excluding vectors with zero coordinates.

This errors make me hesitate going through the rest of theoretical part of the paper.

Talking about the experiments, since this paper proposes a new optimizers, there are two direct and important factors the readers want to see, which are the performance and the runtime. While performance is a must have, I do not see any report on runtime. There is not report wall-clock training time, GPU-hours, measured optimizer overhead, or inference speedups. The hardware report is also very limited, only 4xA100 with 40gb (for vision experiments), which is considered small at current state of optimizer-proposing paper.

Questions for authors
  • I might be wrong at the Theorem 4.7. If I am, I hope the authors can explain. Otherwise, the authors should revisit the theorem and revise.

  • The efficiency aspect is importance. I hope the author can report the runtime, compared it to traditional AdamW to see the tradeoff between performance and efficiency of the proposing method.

3 comments

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your concerns in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.

W1. The statement in Theorem 4.7 assumes p<∞p < \infty and therefore excludes the counter-example. However, thank you for sharing it to revise our conjecture about the possibility of the limit extension. We have updated the manuscript and include the provided example in the appendix at line 1102. We replace the conjecture in the main text with this statement at line 377:

This characterization concerns strictly positive limit directions at p<∞p < \infty and can fail at the exact sign descent endpoint p=∞p =\infty (Example D.4). The order of composition still affects the update at this endpoint. By Lemma 4.3, applying sign descent after a separable mirror operator erases the coordinatewise rescaling and recovers ordinary sign descent with a L∞L_\infty max-margin bias in separable linear classification. Applying the mirror operator last retains its rescaling in the update. This distinction motivates placing the mirror correction after the AdamW step in HORST. It is true that Theorem 4.7 characterizes strictly positive limit directions, and we have clarified this scope using two arguments:

Why do we target an (L1L_1) bias?

The (L1L_1) norm gives a convex way to favor small coordinates, and the resulting solution can remain dense. There is also a statistical motivation. In sparse regression, Hastie et al. (2020) find that Lasso generalizes better than best subset selection when the signal is weak relative to the noise.

The purpose of HORST.

The entropy correction is mainly designed to temper the (L∞L_{\infty})-type bias associated with sign descent type updates. To encourage more weights to have smaller magnitudes, we use other algorithms to do the pruning in the experimental validation section. We would like to highlight the ability of improving sparsifiability through dense training. In Table 1, pruning 40%40\% of the weights in dense-trained DeiT-Small reduces top-1 accuracy by 2.71 percentage points with HORST and 9.31 with AdamW. These results demonstrate that HORST improves sparsifiability even when the model is trained densely. Note that the operator characterization, noncommutativity result, and gradient-scale boundedness of the composition do not rely on Theorem 4.7.

Hastie, Trevor J. et al. “Best Subset, Forward Stepwise or Lasso? Analysis and Recommendations Based on Extensive Comparisons.” Statistical Science (2020): n. pag.

W2. We agree that computational cost is an important practical consideration. As discussed in Section 5, HORST reuses the Adam moments already computed during the optimizer step, so its multiplicative correction requires no additional forward or backward passes. In addition, we now report the average wall-clock time per epoch for a DeiT-small trained on four NVIDIA GH200 GPUs, excluding warm-up epochs: 72 seconds for AdamW, 75 seconds for HAM, and 77 seconds for HORST. HORST therefore incurs a modest overhead of 6.9% relative to AdamW and 2.7% relative to HAM, compared to the substantial performance gain in accuracy. In the main text, we will qualify this in Section 5 by adding the following sentence: On DeiT-Small, HORST incurs a modest wall-clock overhead per epoch of 6.9%6.9\% relative to AdamW and 2.7%2.7\% relative to HAM, with timing details provided in Appendix F.

Questions

See weakness

I thank the authors for their responses.

I noticed an issue with how HAM hyperparameters are chosen. In Appendix F, HAM gets its best validation accuracy with α=20\alpha=20, but Table 2 uses α=2000\alpha=2000 because of better sparsity. Since the comparison is about accuracy at the same sparsity level, HAM should be tuned for accuracy, using a similar tuning process to HORST, and tested with the same seeds. HAM best reported result is still < HORST, but that does not fully address the comparison.

The reported 6.9% increase in runtime answers my original question for this setting. Table 1 also shows that HORST holds up better after pruning (though each optimizer was only run once). These results are useful evidence of better sparsity, but they do not establish the claimed bias.

I appreciate the improvements, but the gap of the baseline tuning issue still leaves me unconvinced. I keep my score at 2.

We thank the reviewer for their follow up and appreciate their acknowledgment that the runtime measurements address their original question and that the additional experiments provide useful evidence.

We agree that validation accuracy at the prescribed sparsity should determine hyperparameter selection. HAM’s highest single-run validation accuracy in the expanded sweep is 60.6960.69%, obtained with α=20\alpha=20 and β=0\beta=0, compared with HORST’s mean accuracy of 69.4169.41% at 9090% sparsity. Following your suggestion, we evaluate this accuracy-selected HAM configuration at all three sparsity levels using the same three seeds as in Table 2. At 90%90\% sparsity, HAM achieves $59.46\pm1.40$%, compared with $69.41\pm2.75$% for HORST, leaving a substantial gap of 9.959.95 percentage points. The results are reported below.

HAM with alpha = 20 and beta = 0. Mean top-1 accuracy and approximate 95% confidence intervals across three seeds.

Sparsity (%)Top-1 accuracy (%)
7071.41±0.3571.41 \pm 0.35
8067.99±0.4067.99 \pm 0.40
9059.46±1.4059.46 \pm 1.40

Furthermore, we extend the sweep and cover now α∈[5,10,20,50,200,500,2000]\alpha \in [ 5 , 10, 20 ,50 ,200, 500, 2000] and β∈[0,1e−2,1e−3]\beta \in [ 0, 1e-2, 1e-3]. The evaluated configurations include the entire hyperparameter grid used for HORST, together with additional values of α\alpha. We report the final accuracy top 1% accuracy values below:

HAM alphabeta = 0beta = 0.001beta = 0.01
558.4659.5957.37
1058.1558.8958.68
2060.6958.7458.00
5056.6155.7256.94
20058.7258.9259.31
50058.5259.2858.54
200059.1657.0259.46

Moreover, at 7070% and 8080% sparsity, we additionally swept HAM over the same hyperparameter grid used for HORST, as reported in the Tables 6 and 7 in the appendix. None of these configurations improves on HAM’s mean accuracies with α=2000\alpha=2000 and β=0\beta=0 in Table 2, and HORST retains a substantial accuracy advantage at both sparsity levels.

Regarding the sparsity promotion effect, we would also like to highlight the complementary weight distribution evidence. Figure 3 shows stronger concentration near zero among the unmasked weights with AC/DC training, while Figure 10 shows a similar effect in a densely trained language model. Together with the pruning robustness in Table 1 and the performance improvements across vision and language tasks, particularly at higher sparsity, these observations provide empirical support for improved sparsifiability.

We hope these additional results address your remaining concerns and would appreciate it if you could reconsider your score in light of this evidence.

Reviewer_jbdtOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how optimizer geometry affects transformer sparsifiability. HORST takes an AdamW step and then applies an elementwise exponential correction using the same Adam-preconditioned direction, whereas HAM uses the raw gradient in its correction. An operator-composition analysis, including a claimed L1 max-margin characterization for a positive-entropy, finite-p continuous flow, motivates this design. Experiments evaluate sparse training of DeiT with AC/DC and RigL, and post-hoc pruning of GPT-2 and a 494M-parameter LLaMA-style model. The main empirical finding is better preservation of performance at higher target sparsity.

Critical strengths
  1. The visual results show substantial gains across two pruning pipelines. With AC/DC at 90% target sparsity, DeiT-Base improves from 68.86 to 76.83 top-1 accuracy and DeiT-Small from 59.64 to 69.41 (Table 2, p.9; three seeds with reported 95% confidence intervals). The quoted DeiT-Small result uses beta=1e-2, as indicated by the table's star, while the DeiT-Base result uses the default beta=0. HORST also improves the reported RigL results at all three tested sparsities (Table 6, p.23), supporting usefulness beyond AC/DC.

  2. The experiments examine complementary aspects of sparsifiability. Pruning separately trained dense DeiT checkpoints tests the optimizer's effect independently of sparse training (Table 1, p.8). The language experiments extend the evidence to magnitude and Wanda pruning and include downstream evaluations (Tables 8, 10-15, pp.24-27). These language results use one training seed per optimizer and model configuration, so they establish a promising empirical trend within the tested settings. In the LLaMA experiment, only MLP weights outside the first and last blocks are pruned, so its target sparsity is not whole-model sparsity.

Critical weaknesses
  1. Theorem 4.7/D.3 needs explicit domain and realizability conditions. The stated assumptions allow one-dimensional data (x,y)=(-1,+1),(+1,-1), which are linearly separable by theta=-1. With positive initialization theta(0)=1, L(theta)=2 exp(theta), and any finite integer p>=2, Eq. (1) gives theta_dot=-theta and theta(t)=exp(-t). Its normalized limit is the strictly positive direction +1, but the stated L1 margin problem has the unique optimum theta=-1. The time reparameterization in D.3 does not remove this example because its clock factor remains bounded above and below by positive constants. Thus the theorem is not valid as stated (pp.6-7, 17-19). A corrected statement should give the complete domain/trajectory conditions and justify the claimed KKT problem; this counterexample alone does not establish which additional assumptions suffice.

  2. The comparison with HAM does not fully isolate direction alignment from hyperparameter selection. Appendix F searches HORST alpha in {5,10,200} and beta in {0,1e-3,1e-2}, while fixing HAM alpha=200 and searching beta (pp.21-22). The language experiments likewise use HAM's default alpha=200 and transfer HORST's selected alpha=5 (pp.24-25). The direct DeiT-Small HAM comparison in Table 4 also uses one seed. Shared base learning rates are reported, but comparable per-method alpha/learning-rate search budgets are not. Since Appendix B emphasizes that correction scale matters, this limits the attribution of the HAM gap specifically to direction alignment. It does not invalidate the gains over AdamW or establish that HAM would match HORST after tuning.

Questions for authors
  1. What complete domain, separability, and trajectory assumptions make Theorem 4.7/D.3 valid and exclude the counterexample above? Please state the conditions needed to invoke the cited Theorem 3.4, and clarify whether the resulting KKT claim is for the stated full-space L1 margin problem or for an orthant-constrained problem. I appreciate that the paper already presents the finite-p flow as motivation for practical HORST and leaves the limiting regime open; I am not requesting a new general Adam convergence theorem.

  2. Can you report existing sensitivity results, or a modest comparable-budget HAM alpha/learning-rate check on one representative high-sparsity DeiT setting, together with the selected HAM/HORST settings? This would help determine how much of the reported advantage of HORST over HAM supports the direction-alignment explanation without requiring a full experimental rerun.

1 comment

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your concerns in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.

W1 Theory.

We agree that the domain and realizability conditions should be stated explicitly. Our proof applies the steepest-flow result of Tsilivis et al. (ICLR 2025, Corollary 3.8.1) to the balanced factorization. Their realizability assumption (A3) requires the summed exponential loss to be below 11 at some finite time. In the counterexample, the loss stays above 22. Theorems 4.7 and D.3 now state (A3) and require positive initialization (\theta_0>0) for the entropy mirror map. We also updated the proof in Appendix D.3 to show how the factor-space KKT conditions yield the L1L_1-margin characterization for strictly positive limit directions.

W2: HAM hyperparameter tuning.

We agree that tuning HAM's hyperparameters strengthens the empirical comparison. We have therefore conducted an additional sweep over α∈{5,20,50,200,500,2000}\alpha \in \{5,20,50,200,500,2000\} on DeiT-Small with AC/DC at 90%90\% sparsity, following the hyperparameter choices investigated in [1].

The highest validation accuracy in the sweep is 60.69%60.69\% at α=20\alpha=20. We select α=2000\alpha=2000 for the three-seed comparison across sparsity levels in Table 2 to emphasize HAM's intended sparsification mechanism, while also reporting the highest-accuracy configuration found in the sweep. Importantly, even this configuration remains 8.72 percentage points below HORST's mean accuracy of 69.41%69.41\% at 90%90\% sparsity. Moreover, with α=2000\alpha=2000, HAM underperforms compared to HORST by at least 6 percentage points across all tested sparsity levels. We include the sweep results in Appendix F and the three-seed HAM comparison in the revised Table 2.

For the existing sweep on the AC/DC sparse training task, for HORST α=10\alpha =10 has 78.17%78.17\% validation accuracy performance on 70%70\% sparsity. This is still significantly better than the performance of AdamW which has 76.15%76.15\% validation accuracy as reported in Table 2 in the manuscript.

[1] Jacobs, Tom et al. “Hyperbolic Aware Minimization: Implicit Bias for Sparsity.” (ICLR 2026).

Questions

See weaknesses.

Reviewer_4GjxOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper discusses how we can train sparse transformers without losing performance or making training unstable. The authors motivate this by highlighting the inability of commonly used optimization methods, particularly Adam and its variants, to achieve sparsity. They explain that Adam can essentially be viewed as a mirror descent operator with an L∞L_{\infty} norm; this effect makes weight magnitudes more stable and closer to each other, which works against sparsity.To address this, they propose a specific way of using mirror descent (hyperbolic entropy) which they claim induces an L1L_1 regularization bias, thereby promoting sparsity. However, performing standard mirror descent is computationally expensive because it must be coordinate-based. Therefore, they introduce a modular way of composing descent operators (mirror and steepest) to combine their strengths, achieving both sparsity and training stability simultaneously. They also highlight that the order in which these operators are applied to the parameters is crucial. The authors support their contributions and claims both theoretically and experimentally.

If my concerns and questions are answered I would be happy to increase my score.

Critical strengths
  • Very clear and informative motivation, paper structure and presentation. I personally found the paper very infromative and I learned from it. Although a few of the central intuitions could be explained more explicitly.
  • The have a very simple algorithmic modification with a solid theoretical foundation behind to support their modifications and proposed algorithm.
  • They show significant experimental improvements over existing methods
Critical weaknesses

I don't have much to say about the theory, however for experimental side:

  1. The comparison with HAM in some results is missing, for example, I would like to see how HAM shapes weights as in figure one. Also HAM is missing from Table 2 as well.
  2. I find hyperparameter searches for HAM is somewhat limited (App. F) and I don't believe the best hyperparameter setting for HAM has been found.
  3. There's already a experimental evidence of the order of descent operator in Fig. 3, but I would like to see the performance of Adam-after-Exp for real datasets and transformer benchmarks as well. So for ablation: Could a representative transformer experiment further separate the effects of adaptive-direction choice, multiplicative-update magnitude, and composition order? HAM changes the direction inside the exponential, so the HAM comparison alone does not isolate ordering.
Questions for authors

Questions

Section 3 composes direction operators while holding θ\theta fixed, whereas Algorithm 1 first updates the parameters and then applies a multiplicative correction at the intermediate point. Could the authors explicitly derive the relationship between these constructions and state which theoretical conclusions apply to the implemented update?


I believe an eigenvalue interpretation of inverse coercivity would be useful for the presentation purposes. Effectively, the μ\mu bounds the smallest eigenvalue of the inverse Hessian from below, or equivalently the largest eigenvalue of the Hessian from above.

Could the authors include this interpretation and use it to explain the three examples 3.9-3.11? This explanation would make the role of the scaling clearer. The inverse coercivity assumption prevents the mirror metric from suppressing gradient directions arbitrarily strongly. It is distinct from an upper bound on update amplification, which is addressed by the growth condition in Corollary 4.5.


Can you provide the fractions of pruned weights in each layer? If the fraction of pruned weights is so much different between layers Also how does it matter to prune from earlier layer than later layers?


Scope of the practical stability claim. Gradient-scale boundedness controls a property of the update direction, but does not by itself establish convergence or stable finite-step transformer training. Could the authors distinguish these meanings of stability and provide a small learning-rate or mirror-strength sensitivity experiment supporting the practical claim?


Suggestions

A short intuitive explanation of the relationship between Adam and sign descent would improve the introduction. Similarly, explaining what an “(L_1) implicit bias” means would make the motivation more accessible to readers unfamiliar with the cited literature.


Around lines 200–203, it would help to distinguish the generic update using an arbitrary operator (A) from the update obtained by substituting the composed operator (AB). Definition 3.8 should also clarify the distinction between a mirror-flow direction and an exact discrete mirror-descent step.


Some empirical descriptions should be qualified. The language results support particularly strong improvements at higher sparsity, but HORST does not obtain the best perplexity at every sparsity level or the best score on every downstream task.


The paper provides useful background on mirror geometry and clearly explains that both stages of HORST reuse the same Adam direction. An additional connection to Bregman proximal methods could help clarify the practical update. I believe making a connection here would strengthen the conceptual presentation.

4 comments

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your concerns in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.

W1, W2: HAM hyperparameter sweep. Our original hyperparameter choice was motivated by the HAM paper [1], which used the same hyperparameters across several datasets and architectures. We agree that including HAM in Table 2 and adding a hyperparameter sweep strengthens the manuscript. We therefore evaluate α∈{5,20,50,200,500,2000}\alpha \in \{5,20,50,200,500,2000\} on DeiT-Small with AC/DC at 90%90\% sparsity. The highest validation accuracy of HAM in the sweep is 60.69%60.69\% at α=20\alpha=20. We select α=2000\alpha=2000 for the three-seed comparison across sparsity levels in Table 2 to emphasize HAM's intended sparsification mechanism, while also reporting the highest-accuracy configuration found in the sweep. Importantly, this configuration remains 8.72 percentage points below HORST's mean accuracy of 69.41%69.41\% at 90%90\% sparsity. With α=2000\alpha=2000, HAM underperforms compared to HORST by at least 6 percentage points across all tested sparsity levels. HAM nevertheless improves on AdamW by 0.41 and 1.62 percentage points at 70%70\% and 80%80\% sparsity, respectively. At 9090% sparsity, the confidence intervals of HAM and AdamW overlap. In this setting, we also observe training instability for HAM, accompanied by a substantial drop in validation accuracy. Our updated Figure 3 now includes HAM's weight distribution for a DeiT-Base run achieving 76.12%76.12\% validation accuracy, remaining substantially below HORST. Like HORST, HAM produces a stronger concentration of weights near zero than AdamW.

W3. Thank you for suggesting this interesting ablation. Accordingly, we apply the mirror rescaling before the Adam update. Let dk=Adam⁡(gk)d_k=\operatorname{Adam}(g_k) and d~k=Adam⁡(∣θk∣⊙gk)\tilde d_k=\operatorname{Adam}(|\theta_k|\odot g_k) denote Adam’s directions before learning-rate scaling, each computed with its own moment estimates. The swapped update is

\qquad \theta_{k+1}=e^{-\eta\beta}\theta_{k+\frac12}-\eta\alpha\,\tilde d_k.$$ We apply the correction only to nonzero coordinates of the weight tensors after the AdamW step. We use $\alpha=5$ and $\beta=0$ and evaluate the swapped variant in the DeiT-Small AC/DC setup on ImageNet at $90\%$ sparsity. The generalization performance is $60.46\%$ which is above the mean performance of AdamW but still inside its confidence interval $95\%$. In contrast, HORST achieves a substantially higher mean accuracy of $69.41%$ in the same setting. Furthermore, the swapped version additionally requires separate first- and second-moment estimates for the correction, introducing additional memory and computational overhead. Questions 1. We are happy to make the connection between the two constructions explicit. In Section 5, line 394, where our algorithm HORST is introduced, let $d_k$ be the Adam direction before learning rate scaling. The AdamW step gives $u_k=(1-\eta\lambda)\theta_k-\eta d_k$, and the exponential correction gives $\theta_{k+1}=u_k\odot\exp[-\eta(\alpha \operatorname{sign}(u_k)\odot d_k+\beta)]$. Using $u_k=\theta_k+O(\eta)$ and $u_k\odot\operatorname{sign}(u_k)=|u_k|$, we obtain, at fixed $\alpha$, $$\theta_{k+1} =\theta_k-\eta\bigl[(1+\alpha|\theta_k|)\odot d_k +(\lambda+\beta)\theta_k\bigr]+O(\eta^2).$$ Thus, the first-order mirror term is $\alpha|\theta_k|\odot d_k$, the entropy mirror direction evaluated at the original iterate. For large $\alpha$, it dominates the additive Adam term on coordinates where $\alpha|\theta_{k,i}|\gg1$. Theorem 4.7 provides the $L_1$ margin characterization for the corresponding mirror-only flow with a finite-(p) steepest direction. We added this derivation in the revised manuscript in Section 5. 2. We agree that this interpretation of the inverse coercivity condition is useful and have included it in the revised manuscript in the definition at line 259 in the following way: Or equivalently denote the matrix $H_{\theta} : = \nabla^2 R(\theta)$ then all eigenvalues of the inverse $H_{\theta}^{-1}$ are being lower bounded by $\mu$: \begin{equation*} \lambda_{\text{min}}(H_{\theta}^{-1}) \geq \mu \qquad \Leftrightarrow \qquad \lambda_{\text{max}}(H_{\theta}) \leq \mu^{-1}, \end{equation*} where $\lambda_{\text{min}}$ and $\lambda_{\text{max}}$ are the smallest and largest eigen values.
  1. In the language task setting, we use Wanda and layerwise magnitude pruning, which fix the sparsity budget per layer explicitly and prune the same percentage of the weights in each layer. However, for the DeIT-base and DeIT-small trained on Imagenet with the sparse training method AC/DC, the sparsity can vary per layer in the model. Therefore, we report the layerwise allocation in Appendix F at line 1309 for both AdamW, HAM (α=2000\alpha =2000), and HORST on DeIT-base trained with AC/DC and 70%70\% overall sparsity. We find that HORST sparsifies earlier layers of the model more than later ones. Moreover, the layerwise densities for HAM are similar to that of AdamW. This is an indication that layer allocation can be an important factor in the performance improvement for this task.

  2. We indeed mean with stability the stability with respect to gradient noise or perturbations. For language tasks, which can have heavy tailed data, this is an important property for training. We have clarified this in the revised manuscript at line 223, by adding this after the bounded operator definition:

This is important when stochastic gradients have heavy tailed distributions (as is usually the case in language tasks). Gradient-scale boundedness limits how large a single update can become in response to a gradient spike at a fixed optimizer state and learning rate.

Moreover, we agree that stability over multiple iterations and convergence require a separate analysis of the training dynamics. To illustrate the impact of stability, we analyze the classification example with noise spikes in Figure 8 in the Appendix. We find that exp after GD is more sensitive to the spikes than exp after Adam.

Suggestions

  1. We agree and have included an explicit description of the connection between Adam and sign descen at line 71: To see this, the Adam step divides its first-moment estimate by a coordinatewise RMS estimate. With both moment decay rates (β1=β2=0\beta_1 = \beta_2 =0) set to zero and the numerical stability constant (ϵ→0+\epsilon \to 0^+) taken to zero, its normalized direction is the gradient sign.

Moreover, for the implicit bias explanation, we have added the following clarification at line 49 where it was first mentioned in the introduction in context of gradient descent: In other words, without explicit regularization, the optimizer can still favor low-norm solutions or directions with a large normalized margin.

  1. We have made the composition with respect to the gradient gg and parameter θ\theta explicit in the revised manuscript at line 204. Moreover, since we have defined the general update step at the start of Section 3, we propose to remove the update step parts from Definition 3.4 and 3.8. This delineates better the update geometry from a particular descent step.

  2. We have added the following qualification sentence in Section 6 in the language paragraph: At lower sparsity, AdamW or HAM achieve slightly lower or comparable perplexity than HORST in some settings.

  3. We agree that the proximal connection helps improve the exposition for the optimizer introduction. We mention the following at the start of Section 5 together with construction connection from Q1 at line 394: The multiplicative update implements an entropic Bregman proximal step [2], which approximates the corresponding mirror flow as η→0\eta \rightarrow 0.

[1] Jacobs, Tom et al. “Hyperbolic Aware Minimization: Implicit Bias for Sparsity.” (ICLR 2026).

[2] Marc Teboulle, Entropic Proximal Mappings with Applications to Nonlinear Programming, Mathematics of Operational Research 1992.

I thank the authors for their detailed answer. Almost all of my concerns are satisfied, therefore I'm happily increasing my initial score.

I have yet one more comment about the fraction of pruned weights per layer. I understand that's out of scope of the paper but I'm curious and interested to see the effect of pruning larger fraction earlier than later in layers. A nice additional ablation would be a random weight pruning with fraction matched with each method per layer. This can be a nice comparison of the effect of pruning magnitude per layer.

We thank the reviewer for their positive reassessment and thoughtful suggestions. We agree that the proposed ablation would be interesting in future work that focuses on the role of layerwise sparsity ratios.

Comments on the paper

1 comment

Dear Area Chair,

Our paper identifies a dichotomy between steepest descent and mirror descent geometries and introduces optimizer-operator composition as a design principle. We show that the entropy mirror map can overwrite the steepest-descent implicit bias in the analyzed finite-p setting. This motivates the sparsity-aware optimizer HORST (Hyperbolic Operator for Robust Sparse Training), which we evaluate in sparse training and pruning settings across vision and language tasks.

The initial reviews highlighted the interesting operator-composition perspective (pYrx), clear presentation and solid theoretical motivation (4Gjx), and substantial improvements across pruning methods (4Gjx, jbdt).

During rebuttal, we addressed the main concerns:

HAM tuning: We expanded the sweep to include the full HORST grid and additional values, evaluated the accuracy-selected HAM configuration with the same three seeds, and added matched sweeps at 70% and 80% sparsity. HORST retains a substantial advantage.

Theory: We added the positivity and realizability assumptions, clarified the finite p and strictly positive limit directions, and replaced the endpoint conjecture with the reviewer’s example. We also made the connection to the implemented update explicit.

Experiments and runtime: We added a composition-order ablation on transformers, HAM weight distributions, and layerwise sparsity comparisons. Measured overhead is 6.9% over AdamW and 2.7% over HAM.

Reviewer 4Gjx increased their score to 4, stating that almost all concerns were satisfied. Reviewer pYrx confirmed that the runtime question was answered. Our latest experiments directly address their remaining request on HAM tuning and evaluation with the same three seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.