- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how symmetries in neural network parameters can result in different update dynamics despite two parametrisations encoding the same function. Specifically, they seek to go beyond symmetries which only alter the update dynamics by a scalar effective learning rate change, and look at symmetries which can result in updates with different directions, using the term effective learning geometry analogously to effective learning rate. They explore this via a diagnostic which quantifies how much of this difference can be mitigated through a scalar learning rate change, versus actual update geometry changes, and present exploratory experiments with modern architectures and optimisers.
Critical strengths
-
The idea of studying effective learning geometry vs effective learning rate is genuinely interesting, as is understanding their mechanisms with optimisers like Adam and Muon which have qualitatively different update dynamics to SGD. If exposed and explored properly, this would advance our understanding of how neural networks learn, and why certain optimisation strategies work better than others.
-
The mathematical sections are dense but generally well-written, carefully building up notation and definitions and presenting one proposition and one theorem, with Theorem 1 in particular stating a relevant and interesting result for the subject at hand (when do parametrisation changes lead to qualitatively different update directions).
Note:
- I did not thorough check the correctness of the mathematics, since it is outside my direct experience and therefore would take an impractical amount of time.
Critical weaknesses
-
The experimental setup and results sections are difficult to follow. It is not clear to me having read through the paper slowly what purpose the experiments serve, and the writing style makes it difficult to understand many of the details, though it feels like they could be explained more simply. Written sections about the results mostly describe what is already in the plots, and could benefit from more analysis / takeaways.
-
Linked with the previous point about the experiments, whilst the motivation and mathematical formulation at the start of the paper are promising, the rest of the paper feels it lacks direction, and it's not clear what its main message is. Though only discussed briefly in the theoretical section from lines 188-193, the comparison of the aforementioned effects between optimisers with different update geometries seems to be a big focus of the experiments, and I wonder if this should have been given more careful treatment in the theoretical section.
-
My initial "Weak Reject" score reflects my comments above, primarily the lack of focus in the second half of the paper. If the authors can convince me that the experiments are actually relevant and interesting (including revising the paper to include clearer explanations) I would be inclined to raise my score to "Weak Accept". It is unlikely that I will raise my score to "Clear Accept" as I believe there would need to be more substantial contributions / discoveries in the paper.
Questions for authors
-
At a high level, what are the experiments telling us about effective learning geometries vs effective learning rates? Why are these findings interesting? How is this different to what we knew before this paper?
-
Was AI used extensively in the writing, particularly in section 4 and beyond? I have no objection to this in principle, but I think these sections in particular could benefit from rewriting to make the main message and details clearer (it is currently quite dense with technical phrases and terminology which I feel the reader has not been prepared for).
Non-critical questions
- In Theorem 1, what is meant by epsilon_{S | \mathcal T}?
- Why do the curves in figure 3c,d feature large spikes near the start?