- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how the effective dimensionality of neural network parameter trajectories changes as more independently trained runs are considered. The authors propose a framework based on symmetry-aware alignment, temporal centering, and spectral analysis to measure the joint geometry of multiple training trajectories. Across several datasets and model scales, they find that effective dimension grows rapidly with the first few runs but exhibits diminishing marginal growth as more runs are added.
Critical strengths
-
The paper studies a clear and interesting question: how the effective dimensionality of parameter trajectories changes as additional independent training runs are included. The run-count perspective provides a useful way to characterize the geometry across multiple training trajectories.
-
The proposed analysis pipeline is systematic and well motivated. In particular, the authors explicitly account for known parameter symmetries and static trajectory offsets through symmetry-aware alignment and temporal centering.
-
The main empirical observation is consistent across the settings evaluated in the paper. The authors also provide several useful robustness checks.
Critical weaknesses
-
The paper primarily provides an empirical characterization of the phenomenon, while offering limited explanation of why the shared spectral structure emerges. It remains unclear whether the observed concentration is driven by the pretrained initialization, optimization dynamics, or other factors.
-
The practical or functional significance of the observed spectral concentration remains unclear. For example, the paper does not investigate whether runs that contribute little additional effective dimension are also functionally similar, make similar predictions, or can be well approximated using the shared trajectory structure. This makes it difficult to assess what the measured geometric redundancy implies beyond the spectral observation.
Questions for authors
See weaknesses above.
-
Can the authors directly measure the similarity of the leading subspaces across trajectory subsets? Would the leading subspace estimated from a small number of runs remain stable as more runs are added?
-
What do the authors believe is the main source of the observed shared spectral structure?
-
Could the authors extend the main run-count and matched-null experiments beyond ?