- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper suggests Mode Dynamics framework, which has three steps:
step one is effectively SVD on the differences between model checkpoints ;
step 2 is taking checkpoint-dependent amplitudes () from step 1 as weights and multiply it with the Jacobian vector products to associate the modes in parameter space with the model input-output functions;
step 3 (section 3.4 and not 3.3) shows how much each detected mode influences the input-output functions
The paper tests 4 settings - scalar regression, controlled next-token prediction, CIFAR-100 autoencoding, and a public Pythia-70M pretraining trajectory to test if leading trajectory-derived modes reconstruct observed output change
Critical strengths
- The paper attemps to address an important problem in the interpretability and safety domain. Indeed as stated in the introduction - the object of interest is the function learnt by the neural network and understanding degree of freedom of this function and how this function is shaped could be crucial for interpretability and safety domains.
- The paper is relatively easy to follow and understand on a high level
- Control experiments and error bars are great - initialization frozen baseline and random subspace baselines are good controls to contextualize the proposed methods behaviour. (but better explnataions that signed-controls preserve length but not angles would be helpful for a naive reader)
Critical weaknesses
1.The biggest and most crucial criticism - scientific impact of the method: While the methodology seems to be simple and elegant, the results seems to be raw. Yes, we can perform SVD on the weight changes across different model types and see that we find most important directions - but what new information does it tell us? What can we learn about neural networks from these type of analysis?
To my impression, the claims currently done in the paper are known before:
- we already know that learning dynamics are path-dependent and could not be approximated with a single tagent plane due to non-linear complex loss landscape - see references 3-4
- we already know that more complex tasks require more capacity / circuits / principal components / modes in the same network (see references 1, 2).
The paper mentions some potential directions in future work but given the simplicity of methodology I would argue that one of the examples of such analysis should be provided in a paper, especially something involving causal interventions might be particularly useful, to prove that modifying during training changes model behaviour - otherwise the method is still close to the post-hoc analysis like the ones described in "mechanistic interpretability" section of related work.
2.Clarity / Notation strictness:
* Sometimes new letters in the mathematical notations are introduces without explictly saying what they mean, the meaning of the indexes are also not specified everywhere (like equations 10 and 11 - what are there? the also have different meanings in both formulas).
* In general, the text seems to be overly condensed, for example, 1st paragraph of the related works is unreadable for a reader, who is not accustomed for the jargons of the field (e.g. collective coordinates ) - hence related work paragraph becomes unhelpful / looses its purpose
* Neural tangent kernel (NTK) is the only baseline and in section 4.1 and could have been introduced at least a bit in the text (what is does / what are the limitations)
- Lack of baselines - what are the other methods which could compare things across checkpoints? why there are not compared against?
References:
- Wang, K., Variengien, A., Conmy, A., Shlegeris, B., & Steinhardt, J. (2022). Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. arXiv preprint arXiv:2211.00593.
- Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., ... & Olah, C. (2022). In-context learning and induction heads. arXiv preprint arXiv:2209.11895.
- Chizat, L., Oyallon, E., & Bach, F. (2019). On lazy training in differentiable programming. Advances in neural information processing systems, 32.
- Song, M., Montanari, A., & Nguyen, P. (2018). A mean field view of the landscape of two-layers neural networks. Proceedings of the National Academy of Sciences, 115(33), E7665-E7671.
Questions for authors
- Please reply for criticism 1 from the previous fields
- How much does the modes depend on the initial initializations? Does their amount (e.g. rank from page 2) depends on the initializations?
- Why have you used only 100 test images for CIFAR and not the whole test dataset? How these 100 images were selected?
- Do you think that Pythia-70M results are valid, given that checkpoints are probably quite far apart from each other? How close the checkpoints should be to each other to make Mode Dynamics conclusions trustworthy and reliable?
- How many modes were considered here for the participation ratio for fig 3?
- Do I get it right that
concentrationin the context of the paper is roughly the effective rank / spectral sparsity andreconstruction fidelityis the quiality of the low-rank approximation ?
6 comments
Follow up on main insight and causal intervention
Thanks a lot for a brief, clear reply and integrating my suggestions.
Main insight
I still struggle to distill the main insight. Given the other reviews, I am clearly no alone in this struggle. What does the insight of "how the velocities of many parameters are coordinated" actually mean? E.g. does it give some better understanding of the learning process (beyond what was know before), some ideas on how to manipulate the learning procedure or as reviewer WNUJ mentioned - do we get some a stronger link between output change and model capabilities?
Causal intervention
I find it very surprising that on a relatively simple task of induction removing one mode basically destroys the performance? How exactly do you suppress the modes? What is the by-chance performance? And why do you think the model does not adjust to relearn the solution in some other direction?
Clarifying the main insight and causal intervention
Thank you for the follow-up. We clarify the main insight and causal test below.
Main insight
What coordinated velocities mean. A mode is a trajectory-derived direction in parameter space that specifies a shared signed pattern of parameter velocities. It determines which parameters move together or oppositely and their relative amounts. Its amplitude specifies how strongly that pattern is expressed at each step. A tensor-matched control randomly permutes and sign-flips the mode coordinates within each tensor. This preserves each mode's per-tensor norm but changes which parameters move together. Suppressing these controls does not produce the same impairment, showing that the specific coordination matters.
What this reveals about learning. Prior work showed that learning can exhibit low-dimensional parameter structure (Gur-Ari et al., 2019; Li et al., 2018; Brokman et al., 2024). Low dimensionality alone does not identify which coordinated movements change predictions or support capability acquisition. We find that large parameter movement does not necessarily imply large output change: smaller coordinated movements can affect predictions more, and these effects evolve during training. In controlled induction, suppressing the dominant output-changing modes delays held-out induction, whereas suppressing their tensor-matched controls does not. These results identify a compact, coordinate-specific route used by standard training to acquire induction.
Manipulation and capabilities. Modes provide concrete intervention targets. Projecting selected modes out of every gradient delays acquisition of held-out induction. This suggests using modes to intervene on learning dynamics, although we have tested only suppression using modes from a completed reference run.
Here, output change is the numerical movement of the logits, whereas a capability is successful behavior on unseen inputs. Held-out induction therefore provides a controlled capability measure. Reconstruction connects modes to output change. The causal experiment goes further: suppressing modes selected by their output contribution on training contexts impairs induction on unseen contexts. Testing whether this extends to broader capabilities, such as mathematical reasoning or code generation, is an important next step.
Causal intervention
How we suppress modes. For each seed, we train a reference model, extract fixed orthonormal velocity modes , and restart from the same initialization on the same training contexts. At every SGD step, we replace the gradient with
Thus, at every step, we remove the component of the update along the selected reference modes while leaving all other directions available. The modes are not recomputed, and neither parameters nor forward computations are removed.
Why a single mode can have a large effect. The causal result is consistent with the reconstruction evidence: retaining only the leading mode reproduces the trained model's induction decisions on the training contexts with 100% Top-1 agreement (revised Figure 4B), although additional modes improve the full logits (revised Figure 4A). We interpret suppression as strongly impairing acquisition, not completely destroying induction: mean held-out accuracy remains above chance, and some runs reroute with longer training. Larger sequence lengths or vocabularies are also associated with broader mode recruitment (revised Figure 3), so the single-mode effect should not be generalized to all capabilities.
Chance. The task-level chance baseline is 16.7% under uniform guessing among the six possible target tokens; uniform guessing among all ten output symbols would give 10%. At 10,000 steps, mean held-out accuracy under Top-1/3/5 suppression is 34.6%, 35.6%, and 48.3%, compared with 99.4% for the baseline. These conditions remain above chance on average but far below the baseline.
Adjustment and rerouting. We appreciate this question because it distinguishes delayed learning from permanent failure. Every baseline run exceeds 90% held-out accuracy by step 7,000. In contrast, none of the 15 Top-1/3/5 runs reaches 90% by step 10,000, whereas every tensor-matched control does.
We extended the same fixed-projection protocol to 100,000 steps. All five Top-5 runs reach 90% held-out accuracy by step 18,000. Two of five Top-1 runs reach it by step 13,000, and two of five Top-3 runs reach it by step 22,000. At step 100,000, the remaining six runs have held-out accuracy below 39%. Because the selected modes remain suppressed, recovery uses alternative directions. Thus, in this task, these modes provide an efficient learning route, while suppression forces slower and less reliable rerouting within the 100,000-step budget.
Main insight, the causal intervention you suggested, and answers to each question
Thank you for the detailed review.
Main insight
Our main insight is that the functional change produced by learning is organized in collective modes of parameter-space velocity. This organization lies not in individual parameters or their separate velocities, but in how the velocities of many parameters are coordinated. Treating checkpoint updates as discrete velocities, we define a mode as a direction along which parameters across many neurons and tensors move in fixed proportion. SVD extracts these directions, but not which ones matter: the directions with the most motion need not change the function most. In a new comparison on held-out Pythia contexts, the five modes with the largest output effect give R2=0.728, versus 0.134 for the five with the most motion. The causal test below shows the same during training.
Path dependence and task capacity
We agree that path dependence (Chizat et al., 2019, Mei et al., 2018) and task-dependent capacity (Wang et al., 2022, Olsson et al., 2022) are known. Mode Dynamics recovers both, as consistency checks rather than new claims. Evaluated on the tangent plane at initialization, all 50 Pythia modes give held-out R2=−0.018, versus 0.9998 along the path. More complex tasks also recruit more modes: PR is 3.8 for copying, versus 11.0 for induction and 13.0 for random labels.
New causal intervention
As you suggested, we modified the modes during training, fixing selected amplitudes at zero. In a one-layer transformer trained with full-batch SGD on an induction task, we found the modes in a reference run, then retrained from the same initialization with every gradient projected away from the selected directions. Separate runs suppress the Top-1/3/5 dominant modes, modes with matched parameter motion but less output change, or tensor-matched controls that shuffle the dominant modes within each tensor. Across five seeds and 216 held-out contexts, step-7,000 accuracy falls from 95.7% to 29.8%, 21.5%, and 38.6% under dominant-mode suppression, versus 86–89% for energy-matched modes and about 96% for tensor-matched controls (revised Figure 4). No dominant-mode run reaches 90% by step 10,000, unlike 13 of 15 energy-matched runs.
Initialization
In a new experiment, five runs differ only in initialization. Across them, PR is 1.99±0.52, Top-5 R2 is 0.974±0.015, and Keff, the number of modes holding 90% of the contribution energy, ranges from 2 to 5. Numerical rank varies little, from 47 to 52. The algebraic rank r is bounded by the number of recorded updates, so it is not our mode count. We do not claim that individual directions align across initializations.
CIFAR test images
For each seed, we drew one test image per class with a prespecified seed and froze the 100 indices before scoring, excluding them from training, ranking, and mode selection. One image per class guarantees class coverage at a cost of 7,840 full-RGB JVP evaluations per trajectory.
Checkpoint spacing
The reconstruction conclusions are valid at Pythia's spacing, but the mode counts grow with checkpoint density. Going from 51 to 101 checkpoints raises held-out R2 at a matched 40% of modes from 0.9668 to 0.9877, while Keff rises from 13 to 21. In a new analysis of five controlled trajectories, refining spacing from 1,000 to 100 steps keeps Top-5 R2 above 0.96, while Keff averages 2.0–2.2 down to 200-step spacing and 3.0 at 100 steps. Every tested spacing sufficed for reconstruction, but none gave converged counts.
Baselines
Section 4.1 uses three baselines. Dimension-matched random subspaces test whether trajectory-derived directions matter, and the initialization-frozen reconstruction tests path dependence. The NTK trains the model linearized at initialization, so it cannot capture a changing Jacobian. Methods that compare checkpoints credit change to training examples (TracIn, Pruthi et al., 2020) or to single parameters (loss change allocation, Lan et al., 2019), not to coordinated directions, so they give no comparable quantity.
Concentration and reconstruction
For Question 5, Figure 3 computes PR over all 1,000 modes, one per recorded update. For Question 6, roughly yes, with one difference each. PR is a standard effective-dimension formula, but we apply it to the energies of the output contributions gk,t(x), not to singular values. Top-K reconstruction is not a fixed low-rank approximation of the outputs: it projects the parameter updates onto K modes and re-evaluates their effects along the path.
Manuscript revisions (marked in blue)