- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This review was entirely human-written, GPT-6 Astra was used after the fact to adversarially contradict my review. I manually corrected some of my statements that I judged to have been misrepresented based on this.
The authors look at how we can analyze pre-training trajectories through the lens of Finite-Time Lyapunov Exponent theory from dynamical systems. They use the AdamW state to get a proxy for the FTLE. They claim through experiments that when the proxy FTLE is high (high divergence in the dynamical system), the pre-training trajectory varies significantly with small perturbations. This divergence does not show up in the training loss. They use this insight to design an FTLE-aware optimizer. They show additional experiments which I am not sure I understand due to the extremely dense and jargon-heavy text that simply describes results rather than interpreting them. I cannot evaluate the paper properly when it comes to the quality of the results.
Critical strengths
- I really like the main idea of looking at pre-training through the lens of dynamical systems theory: this leads to being able to reuse ideas that have been iterated on for a long time in the literature
- The core finding that different pathways in optimizer trajectory can still lead to the same loss is interesting, and deserves further studying.
Critical weaknesses
Nevertheless, the presentation of the paper is poor which makes it below the acceptance threshold. The authors would need to rewrite the paper for it to pass what I would expect from a paper at ICLR. I encourage the authors to resubmit at a later date since the core of the paper is very exciting and interesting!
- Presentation
- Much of the paper is extremely hard to read. Section 3.1 is the only section that I was able to read easily. The rest of the text is filled with jargon and complicated sentence structures.
- Un-introduced datasets, baselines, and metrics (e.g. Table 3), L200 lists metrics and datasets, but do not define them, give relevant citations for the datasets. The authors need to 1. fix the sentence L200 which is grammatically incorrect 2. add the relevant citations 3. explain why these datasets and metrics were chosen in the context of FTLE for pre-training.
- Opaque methodology that is not justified
- The normalized Adam-state FTLE is not clearly explained, it is not clear how this derives from the general form of the FTLE: the authors should either link an appendix to show how these are equivalent mathematically or give a qualitative explanation.
- The Lyapunov-aware AdamW control equations are un-justified: the authors need to justify the choices they made for these.
- Missing reference (L201, ?? reference), typos in notation (L159, is supposed to be ? )
- Experiment clarity: the choices for each experiment are poorly justified which leads to confusing the reader
- Figure 1 is difficult to parse, the caption does not give any details and the main text simply describes the results.
- Figure 2 does not show that high and low-FTLE respond differently: it simply implies that locations (in weight space) of high FTLE lead to high separation and this persists over time: the training time in Figure 2 is 200 steps but the checkpoints were taken 2000 apart. The experiment needs to be more carefully designed.
Questions for authors
- Muon is more common for pre-training LLM's do you expect the findings to be transferrable?
- I rationalize the findings by thinking of e.g. a 2d quadratic function (convex bowl) with noise. At certain points where noise is high, you would have high FTLE as the trajectory can diverge either side of the noise "bump", but the loss would then still go down similarly. In this frame, are the results that trajectories diverge while loss decreases similarly surprising?