acceptodds
Withdrawn from ICLR 2027

Which Geometry on Which Layer? Measured Sign–Spectral Assignmentin Mixed-Optimizer Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern training of large neural networks now commonly uses mixed recipes, assigning AdamW-like or sign-based updates to embedding and output layers and spectral updates to hidden matrices. These assignments are fixed by layer type for the entire run and are largely inherited heuristics that must be re-tuned across architectures and training regimes, without a clear criterion for selecting the appropriate geometry, that is, the norm under which the steepest descent step is taken. We turn the sign-versus-spectral assignment into a measurable, layerwise decision problem. For ideal memoryless steps, we derive the finite-batch geometry ratio , where and are conservative descent-lemma lower bounds for the specified spectral and sign steps. Its closed form separates signal efficiency from a geometry-dependent global curvature factor. In the positive-signal regime, the same smoothness assumption gives two-sided bounds that determine which step has the larger conditional expected decrease when the ratio crosses explicit noise-dependent thresholds. We then derive a local directional quadratic score from gradient alignments and Hessian–vector products and use it in Generalized-Scion (G-Scion) to control layerwise routing. In a controlled input-rank experimental setup, including directional curvature matches the measured same-iterate ordering in all seven aggregate settings, whereas the curvature-free proxy selects the opposite geometry. On M GPT-2, the output score routes the output layer from spectral warmup to sign, and G-Scion improves validation loss over fixed Scion in memoryless and matched-momentum comparisons. ViT-B/16 results further show that a spectral edge assignment can outperform fixed Scion outside language modelling. These results suggest that optimizer geometry need not be hard-coded by layer type, and they are a first step toward understanding how it should be assigned.

Primary area:
optimization
Keywords:
Optimization, Mixed-Optimizer Training, Layerwise Optimizer Selection, Optimization Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7PuCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks whether the sign-vs-spectral choice can instead be turned into a measurable, per-layer, possibly stage-dependent decision. For this interesting question, the authors derive a finite-batch ratio RLB(W;B):=Bspec(W)/Bsign(W)R_{LB}(W;B) := B_{\mathrm{spec}}(W)/B_{\mathrm{sign}}(W) of two conservative descent-lemma lower bounds and then propose a measurement-driven assignment method (G-Scion). For experiments, on 31M GPT-2/OWT, the output-head score routes from spectral to sign and G-Scion (embedding, output) lowers validation loss vs. fixed Scion; on ViT-B/16/CIFAR-100, a spectral-edge assignment beats fixed Scion by 7.9–11.9 accuracy points. In conclusion, notably, the paper is unusually candid that its evidence "supports a diagnostic framework … rather than a general claim of optimal routing or practical optimizer superiority."

Critical strengths
  1. Well-posed question. With Muon and mixed sign/spectral recipes now standard, "which geometry on which layer, decided by measurement rather than convention" is a genuinely important and underexplored optimization question.
  2. The distinction between a curvature-free proxy (which selects the wrong geometry) and a directional-curvature-aware score is the paper's best idea, and the controlled input-rank experiment (opposite selections in all seven settings) is a convincing isolation of that point.
Critical weaknesses
  1. The proposed method seems to lose to a trivial baseline. By the authors' own reporting, G-Scion's online gate does not beat the best static assignment and AdamW+Muon "remains a stronger practical reference." All reported gains are over fixed Scion only. Moreover, the comparison of experiments is also insufficient, and many key hyperparameter selections are not explained. For a paper whose headline contribution is a measurement-driven assignment method, being dominated by a plain always-spectral run is close to fatal.
  2. The theory does not bind to the method. The analytical ratio RLBR_{\mathrm{LB}} "does not certify the implemented gate at a threshold of one," its decisive regions (RLB>3R_{\mathrm{LB}}>3 or <1/3<1/3) are never estimated in the LM experiments, and the routing score RdirR_{\mathrm{dir}} is explicitly a different quantity from RLBR_{\mathrm{LB}}. So the elegant signal-efficiency ×\times curvature factorization is essentially decorative with respect to the decisions G-Scion actually makes.
Questions for authors

Please see weaknesses.

  1. Could you report G-Scion against always-spectral (and always-sign) on the edge groups, at matched well-tuned setting? Since static spectral already beats the gate, what concrete setting (model, stage, layer) does the online gate win in?
  2. In the 31M LM runs, what are the measured RLBR_{\mathrm{LB}} values and how often do layers fall in the certification regions (>3>3 or <1/3<1/3) vs. the indeterminate band [1/3,3][1/3,3]? Without this, how should a reader interpret the threshold-of-one gate?
  3. Could you provide at least one larger-scale run with complete directional records?
Reviewer_4QV8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to assign sign or spectral updates to parameter groups. It derives a ratio of descent lower bounds for ideal memoryless steps, then proposes a separate local score based on gradient alignment and Hessian–vector products. G-Scion uses an EMA of this score to switch eligible embedding and output groups once, after a spectral warmup; hidden matrices remain spectral.

On 31M GPT-2, G-Scion improves validation loss over the paper’s fixed sign-edge Scion baseline. However, the additional static-assignment experiment finds that keeping both edge groups spectral performs better than the gate in all five seeds. The scheduled-update study also finds that the directional score ranks candidates less accurately than always choosing spectral. The paper acknowledges these failures and presents the framework primarily as a diagnostic study.

Critical strengths
  1. Comparing candidate updates from the same checkpoint is a useful design. The paper distinguishes immediate loss decrease from the outcome of an entire training trajectory and includes controls that expose the difference. The static-assignment and replay experiments are particularly informative, even though their results are unfavorable to the proposed controller.

  2. The paper is careful about what its theory proves. A ratio of lower bounds is not automatically a ratio of realized improvements, and the stated two-sided bounds make the unresolved region explicit. The manuscript also acknowledges that its directional score does not estimate the global-bound ratio and that the larger-scale and ViT records do not validate the selector.

Critical weaknesses
  1. The actual selector is validated at too small a scale and training budget. The central evidence uses 31M GPT-2 for 2,000 steps at effective batch 128: 256,000 sequences, with sequence length and total tokens unclear. The 1B runs use a different proxy and do not validate the directional score; intermediate records are incomplete, and ViT scores are frequently nonfinite. Since geometry preferences may change with model size and training stage, these experiments do not establish reliable routing beyond the tiny LM setting. The conservative-step theory does not supply that missing validation.

  2. The selector loses to simpler fixed choices even in that setting. Table 10 reports loss 5.45 for the gate versus 4.92 for spectral/spectral, which wins in all five seeds. The warm-up gate and frozen-route replay both reach rounded loss 5.23, providing no demonstrated benefit from continued measurement. Candidate-ranking accuracy is also below always choosing spectral: 219/288 for the full-group diagnostic and 205/288 for the implemented score, versus 264/288. The score compares locally line-optimal decreases rather than actual scheduled steps. Improvement over fixed sign-edge Scion therefore does not establish the proposed selector’s value.

Questions for authors
  1. Can the actual directional score and routing policy be validated on substantially larger models and longer, explicit token budgets, against all four static assignments? The existing 1B proxy runs do not answer this question.

  2. What evidence shows an advantage over fixed spectral assignment or one-shot selection? Please evaluate rankings of actual scheduled updates on independent data and provide per-seed results, complete configurations, and score/route traces.

Reviewer_mNBcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This authors study how to choose update methods for different network layers during training using hybrid optimizers.The authors attempt to utilize local gradient geometric to choose between different optimizers. The authors use evaluation metrics based on gradient alignment and directional curvature, and they also evaluate their effectiveness both theoretically and experimentally.

Critical strengths
  • When training with the hybrid optimizers, it is worth exploring what kind of update methods should be used for different layers, and whether these preferences will change during the training stage.

  • The authors use gradient alignment and directional curvature to guide the allocation of update methods across layers.

  • The authors distinguish the theoretical descent bound from the local score and the gains observed over a full training run. They also find that static allocation outperforms online gating. They report cases where the score fails, highlighting the limits of relying on local geometry.

Critical weaknesses
  • The score does not fully reflect the actual updates, and its ranking performance is limited. Formula (11) compares the improvement obtained with an optimal line-search step size in each direction. In training, however, the step size is preset, so the theory does not guarantee the same ranking.In Table 9, the full set of parameters correctly identifies 219 out of 288 updates, fewer than the 264 identified by always choosing the spectral method. The curvature score performs even worse.In all seven controlled input-ranking settings, the spectral method outperforms the alternatives. These results therefore do not fully establish that the scoring system can identify the strengths of both methods.

  • Some key evidence is difficult to reproduce or reconcile.The main text does not compare the verified configuration with historical runs or analyze the resulting differences in routing and performance. Yet the appendix discusses such routing and performance differences on this basis.Given the paper establishes R_0 <= 1, the appendix should explain how the proxy values for the spectral method are normalized and what decision threshold is used. The relevant records should therefore be completed and corrected, or the appendix claims should be removed.

  • The benefit of online measurement has not been verified. In Table 10, the static spectral-spectral allocation outperforms G-Scion across all seeds.Thus, relative to the fixed symbol–symbol baseline, this improvement does not show that continuous online measurement is better than simple static allocation with a preset switching strategy.

Questions for authors

Please refer to the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.