acceptodds
Under review as a conference paper at ICLR 2027

Beyond Scalar Effective Learning Rates: Update Geometry under Parameter Symmetries

Abstract

Parameter symmetries preserve a model's function but can change the speed and direction of its updates. We describe these changes through *effective learning geometry*, using a *quotient update map* that recovers angular effective learning rates (ELRs) for scale-invariant models as a special case. For strictly rectangular value–output (VO) factors of full rank, we prove that Euclidean factor responses admit one common scalar on the entire product tangent space exactly for orthogonal basis changes. Probes with zero optimizer history in a transformer with 123.6M parameters reveal distinct compatibility structures: AdamW responds to both basis orientation and nonuniform scaling, whereas Muon is nearly compatible with orthogonal changes but remains sensitive to nonorthogonal scaling. Directional mismatch dominates the residual, with little contribution from variation in optimal rates across batches. Starting from identical functions with corresponding non-VO updates, fitting a scalar directly to the full output response reduces AdamW's residual from 0.192 to 0.102 and Muon's from 0.082 to 0.064. For both optimizers, the fitted residuals remain larger than those under exact update transport. Optimizer history attenuates the VO mismatch for AdamW and has a small, mixed effect for Muon. Paired pretraining trajectories show how the tested scalar compensations alter loss and prediction dynamics. Symmetry, update direction, and optimizer history thus determine what a change in learning rate can absorb.

Primary area:
optimization
Keywords:
Effective Learning Geometry, Parameter Symmetries

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u4JbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how symmetries in neural network parameters can result in different update dynamics despite two parametrisations encoding the same function. Specifically, they seek to go beyond symmetries which only alter the update dynamics by a scalar effective learning rate change, and look at symmetries which can result in updates with different directions, using the term effective learning geometry analogously to effective learning rate. They explore this via a diagnostic which quantifies how much of this difference can be mitigated through a scalar learning rate change, versus actual update geometry changes, and present exploratory experiments with modern architectures and optimisers.

Critical strengths
  • The idea of studying effective learning geometry vs effective learning rate is genuinely interesting, as is understanding their mechanisms with optimisers like Adam and Muon which have qualitatively different update dynamics to SGD. If exposed and explored properly, this would advance our understanding of how neural networks learn, and why certain optimisation strategies work better than others.

  • The mathematical sections are dense but generally well-written, carefully building up notation and definitions and presenting one proposition and one theorem, with Theorem 1 in particular stating a relevant and interesting result for the subject at hand (when do parametrisation changes lead to qualitatively different update directions).

Note:

  • I did not thorough check the correctness of the mathematics, since it is outside my direct experience and therefore would take an impractical amount of time.
Critical weaknesses
  • The experimental setup and results sections are difficult to follow. It is not clear to me having read through the paper slowly what purpose the experiments serve, and the writing style makes it difficult to understand many of the details, though it feels like they could be explained more simply. Written sections about the results mostly describe what is already in the plots, and could benefit from more analysis / takeaways.

  • Linked with the previous point about the experiments, whilst the motivation and mathematical formulation at the start of the paper are promising, the rest of the paper feels it lacks direction, and it's not clear what its main message is. Though only discussed briefly in the theoretical section from lines 188-193, the comparison of the aforementioned effects between optimisers with different update geometries seems to be a big focus of the experiments, and I wonder if this should have been given more careful treatment in the theoretical section.

  • My initial "Weak Reject" score reflects my comments above, primarily the lack of focus in the second half of the paper. If the authors can convince me that the experiments are actually relevant and interesting (including revising the paper to include clearer explanations) I would be inclined to raise my score to "Weak Accept". It is unlikely that I will raise my score to "Clear Accept" as I believe there would need to be more substantial contributions / discoveries in the paper.

Questions for authors
  • At a high level, what are the experiments telling us about effective learning geometries vs effective learning rates? Why are these findings interesting? How is this different to what we knew before this paper?

  • Was AI used extensively in the writing, particularly in section 4 and beyond? I have no objection to this in principle, but I think these sections in particular could benefit from rewriting to make the main message and details clearer (it is currently quite dense with technical phrases and terminology which I feel the reader has not been prepared for).

Non-critical questions

  • In Theorem 1, what is meant by epsilon_{S | \mathcal T}?
  • Why do the curves in figure 3c,d feature large spikes near the start?
Reviewer_6NUnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether a scalar learning-rate correction is enough to compensate for reparameterizing a model without changing its function. The known case is scaling under normalization (the f(w)=f(cw)f(w)=f(cw) weight symmetry, fixed by multiplying the LR by c2c^2), which is the "effective learning rate" (ELR) idea. The authors test a richer symmetry in attention: WOW_O and WVW_V only enter through their product, so (WOS,S−1WV)(W_O S , S^{-1} W_V) is the same model for any invertible SS.

They prove that for plain gradient descent on such a bottlenecked product (the bottleneck is the per-head dimension dh=dmodel/nheads=64d_h = d_{\text{model}}/n_{\text{heads}} = 64, smaller than dmodel=768d_{\text{model}} = 768), one LR multiplier reproduces the original update if and only if SS is orthogonal (a rotation); any stretching changes the update's direction, not just its size. They then measure this for AdamW and Muon on a 124M-param Llama-style model trained to 20 tokens/param (roughly the Chinchilla-optimal ratio), by fitting the best scalar between the canonical update and the one computed at the transformed weights and reporting the leftover residual. Muon is unaffected by rotations but not by stretching; AdamW is affected by both; the residual is almost entirely directional (>99.99%), so per-layer or per-batch scalars don't help; Adam's accumulated moments reduce it, Muon's momentum doesn't. In paired training runs, val losses stay close while predictions diverge, and per-step LR rescaling doesn't close the gap; a control that applies the canonical model's update, mapped through SS, to the transformed copy keeps the two runs identical to numerical precision, confirming the setup itself is sound. The takeaway is that scalar ELR is a special case of a direction-dependent response, and that matching loss curves doesn't imply a shared training trajectory.

Critical strengths
  • The method and measurements look sound. They run a control where the transformed copy gets the canonical model's update mapped through SS instead of its own, and the two runs stay identical to  1e−8~1e-8 in loss, so the setup itself isn't adding noise. Numerical precision was picked before looking at any results, and the error split in Eq. 7 cleanly separates "the direction is wrong" from "the right LR varies between batches". My soundness concerns are minor, the method overall is correct.

  • The AdamW vs Muon comparison is a real result. Muon is immune to rotations and is not affected by which axes get stretched, only how much; AdamW is sensitive to both; and AdamW's accumulated moments bring its mismatch down by  0.22~0.22 while Muon's momentum does nothing. This is the one part of the paper I learned something from.

Critical weaknesses
  • After checking the cited ELR papers and adjacent work, I think the main negative result is a weak contribution. The paper shows that a single LR multiplier can't compensate a dense or diagonal basis change in the VO product. But the ELR work it positions itself against (Liu et al. 2026, Xiao et al. 2026) is about the LR / weight-norm ratio under scaling and Liu et al. only claim that loss curves collapse; it does not claim a scalar LR fixes a κ=4\kappa = 4 stretch, and Theorem 1 itself says plain GD can not. Theorem 1 also follows from a known result that isn't cited: the balancedness invariant of gradient flow on factored matrices (O⊤O−VV⊤O^\top O - V V^\top is conserved; Du, Hu and Lee 2018, Arora, Cohen and Hazan 2018). The one basis change that actually happens in practice, S=cIS = cI from choosing different init scales for WOW_O and WVW_V, should be fixable with per-tensor LR multipliers (c2c^2 on OO, c−2c^{-2} on VV), but this is not tested in this paper, because every scalar in the paper applies one multiplier to OO and VV together. There's no study of any basis change that occurs naturally (init scale, weight-decay drift, model averaging, checkpoint conversion) and no stated consequence for how anyone should train, therefore the paper doesn't show its practical value, if there's any. This is the main reason for my rating and for the informativeness score.

  • The training runs don't fully support the "same loss, different model" claim. The ~0.4 gap in projected logits between the two representatives has no reference point: there's no number for the gap between two canonical runs with different seeds, so there's no way to tell whether 0.4 is above normal seed-to-seed variance. If two seeds also differ by ~0.4, the headline is just normal training sensitivity. On top of that, the scalar-compensation branches in Section 5.4 are confounded: the fitted multipliers drop to ~0.03 (Muon) and ~0.1 (AdamW) and are fit against a canonical replica that's already at a different function, so these branches are basically runs with the VO learning rate cut 10-30x. That's how I'd read Muon's lower val loss there.

The seed baseline is one extra run and could change my score if the gap comes out well above seed noise. For Section 5.4 I'd want either the symmetry claims dropped, or a control that trains the transformed copy with the VO learning rate cut by a fixed factor (~0.03 for Muon) and no per-step fitting, to show whether the compensated branches differ from plain LR reduction.

Questions for authors

Clarification requests:

  • How far apart do two normal (untransformed) runs end up in the same projected-logit metric (Table 4) if you only change the training seed or the data order? Without that number I can't tell whether the ~0.4 gap between representatives is large.
  • Why is the same multiplier always applied to both OO and VV? For the simple case S=cIS = cI (just a different init scale for WOW_O vs WVW_V), wouldn't separate multipliers on OO and VV (c2c^2 and c−2c^{-2}) reproduce the original update exactly? That seems to follow from the theorem's own intuition.
  • In Section 5.4, once the two runs are no longer the same model, what is the per-step scalar being compared against? And since the fitted multipliers go down to ~0.03 for Muon, isn't the "compensated" branch basically just a run with a 30x smaller VO learning rate? Would a run with a fixed 0.03x VO learning rate and no fitting look the same?
Reviewer_QqYdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Neural networks contain symmetries where they can compute the same function with different weights. One such symmetry exists in attention heads, where the value V and output O weights affect the output only through their product VO. The head behavior is the same if you transform V by some matrix and O by that matrix inverse.

The "effective learning rate" literature shows that, in some cases, you can absorb different representations within the same equivalence class into the scalar learning rate. In particular, for weights followed by a normalization (i.e. scale-invariant layers), making the weights larger slows learning down and you can increase the learning rate to compensate exactly.

Their theory shows that, for gradient descent, any transformation that is not a pure rotation cannot be absorbed into a single LR rescaling.

They use pretrained 124M language models and change the weights to an equivalent representation, then take one optimizer step for Adam and Muon from each setting and find the single LR rescaling that makes the changes agree as closely as possible in terms of the network output (not the raw weights). Without momentum (so there is no memory in the optimizer updates): the step from Muon doesn't change if you rotate the head coordinates, but it does if you rescale the coordinates unevenly. Adam is very sensitive to rotations but also to rescaling. Doing a per-layer LR adjustment doesn't absorb these transformations either. If you add momentum history back in, Adam becomes less sensitive to the transformations but it doesn't help Muon much.

The paper frames this idea as "effective learning geometry" where an optimizer response to equivalent weights depends on the direction and a single notion of "effective learning rate" is too coarse.

Critical strengths

The core idea in this paper is nice: the effective learning rate idea has become popular recently (adjust the LR to compensate a change of parameterization) and this paper asks whether it holds beyond the simple case of scale-invariant weights.

I like the probe setup where you compare what one optimizer step does to the function from different equivalent weights. This setup is general and could be used for any symmetry and optimizer. The paper acts as a nice bridge between the literature on effective learning rate and weight decay, and the recent literature on parameter symmetries and geometry. The contrast between optimizers (Adam and Muon) matches the theory. It's interesting that the optimizer memory helps Adam (makes it much less sensitive to the transformation) but not Muon (little change).

Critical weaknesses
  1. The paper introduces a general idea but then focuses on a specific case (VO symmetry). If the paper title is going to assert the general idea, it would be better if the paper included multiple symmetries, e.g. apply the same ideas to QK, MLP + activation and norm scale symmetries. I would either suggest changing the title to specify the paper is about VO symmetries, or do a more general study of multiple symmetries in Transformers.

  2. The presentation structure could be improved. I would suggest elevating the probe technique as a contribution, since it is a general method that can be applied to any symmetries, and then explaining why you consider VO symmetries as a special case. The current presentation is confusing because you start with a general idea about symmetries and then Section 3 jumps straight into details about VO matrices without any connection.

  3. The paper asks whether different VO representations can be absorbed into a single LR rescaling and concludes the answer is no. But there are intermediate rescalings like separate scalars for V and O, per-channel scalars, and per-head scalars. It would be more interesting to know which rescalings are required for each type of symmetry.

  4. Overall the writing could be improved significantly. A lot of sentences use terms that haven't been defined yet or use jargon, e.g. "descent proposal" -> "update direction", "scalar-conformal" -> "proportional", "compatibility structures" -> "equivariant transformations". The sentence structures are often confusing. Here is an example sentence that I found confusing but this is an issue throughout the paper: "Full output fitting at identical functions, controlled optimizer histories, and scalar and transport interventions during pretraining connect local responses to history, outputs, and training dynamics. They show how memory modifies mismatch and scalar compensation affects loss and prediction paths."

Questions for authors
  1. I was confused about how you handled weight decay. The theory doesn't include weight decay but the experiments do use weight decay. How did you or should you account for weight decay in relation to the symmetries and LR rescaling?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.