acceptodds
Withdrawn from ICLR 2027

How Much New Geometry Does Another Training Run Add? Effective-Dimension Scaling Across Aligned Parameter Trajectories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural network training evolves in a high-dimensional parameter space, yet independent runs often exhibit shared geometric structure. Prior work has identified such structure in both converged solutions and training trajectories. How the joint spectrum of complete parameter trajectories changes with the number of independent runs remains less well understood. We organize checkpoints from multiple runs into a joint trajectory tensor and measure the effective dimension of its singular-value spectrum after symmetry-aware alignment and temporal centering. We compare the observed spectra with the weak and strong matched-rotation nulls and repeat the analysis across multiple nested trajectory orderings. Across the evaluated architectures and tasks, effective dimension grows rapidly over the first few runs, followed by diminishing marginal growth over the measured range of up to 40 runs. The observed dimension remains far below the strong matched-rotation null and the algebraic rank bound. These results show that aligned parameter trajectories have a strongly concentrated, effectively low-rank joint spectrum over the observed run-count range.

Primary area:
optimization
Keywords:
training trajectories, effective dimensionality, spectral analysis, symmetry alignment, training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Dq9ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how the effective dimensionality of neural network parameter trajectories changes as more independently trained runs are considered. The authors propose a framework based on symmetry-aware alignment, temporal centering, and spectral analysis to measure the joint geometry of multiple training trajectories. Across several datasets and model scales, they find that effective dimension grows rapidly with the first few runs but exhibits diminishing marginal growth as more runs are added.

Critical strengths
  • The paper studies a clear and interesting question: how the effective dimensionality of parameter trajectories changes as additional independent training runs are included. The run-count perspective provides a useful way to characterize the geometry across multiple training trajectories.

  • The proposed analysis pipeline is systematic and well motivated. In particular, the authors explicitly account for known parameter symmetries and static trajectory offsets through symmetry-aware alignment and temporal centering.

  • The main empirical observation is consistent across the settings evaluated in the paper. The authors also provide several useful robustness checks.

Critical weaknesses
  • The paper primarily provides an empirical characterization of the phenomenon, while offering limited explanation of why the shared spectral structure emerges. It remains unclear whether the observed concentration is driven by the pretrained initialization, optimization dynamics, or other factors.

  • The practical or functional significance of the observed spectral concentration remains unclear. For example, the paper does not investigate whether runs that contribute little additional effective dimension are also functionally similar, make similar predictions, or can be well approximated using the shared trajectory structure. This makes it difficult to assess what the measured geometric redundancy implies beyond the spectral observation.

Questions for authors

See weaknesses above.

  • Can the authors directly measure the similarity of the leading subspaces across trajectory subsets? Would the leading subspace estimated from a small number of runs remain stable as more runs are added?

  • What do the authors believe is the main source of the observed shared spectral structure?

  • Could the authors extend the main run-count and matched-null experiments beyond N=40N=40?

Reviewer_N5nqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how the joint spectrum of parameter trajectories from independent training runs of the same architecture changes as more runs are added. The authors stack symmetry-aligned, temporally centered checkpoints from multiple runs into a joint trajectory tensor, analyze it with two effective-dimension metrics, and show that it has a strongly concentrated, low-rank spectrum. The effective dimension grows quickly over the first few runs and only slowly afterward.

Critical strengths
  • The authors introduce a controlled measurement framework that accounts for various phenomena, such as symmetry alignment and temporal centering, when constructing the joint trajectory tensor.
  • The paper stays focused on the central empirical question (the spectral analysis of the trajectories). The experiments are reasonably designed and include null models and several ablations that help put the observed spectra in context.
  • The paper is reasonably well written and, for the most part, easy to follow.
Critical weaknesses
  • Minor: Some parts of the paper could be organized or written better. For instance, the sentence in lines 35-38 should be split. I am also not a huge fan of the many \paragraph{}s, especially in the introduction, as they artificially break the flow of the text and make it harder to parse. I also found Sections 4.2 and 4.3 hard to follow; please consider rewriting them.
  • Minor: The paper does not report the task performance (e.g., accuracy) of the fine-tuned DeBERTa-v3 runs, nor the fine-tuning hyperparameters. Section 4.1 only states that they follow Jin et al. (2023), with the learning rate and batch size adjusted by model size.
  • Critical: I do not understand the potential impact of this analysis. The authors state in their conclusion that they "characterize finite-range spectral concentration", but I do not see a broader insight about training dynamics or potential applications. In my opinion, this is the main and most fundamental weakness of the paper. The authors have done a careful job with the measurement itself, but I believe they should motivate their work better. They could do so by answering questions such as "What are the main implications of the observed results?", "Do we gain any important insight into training dynamics?", or "Are any of the observed behaviors exploitable in certain empirical contexts?" Regarding the last question, I can imagine that certain techniques, such as model merging, could benefit from the authors' observations. Studying exactly how this could be the case would make the paper much more compelling and easier to accept.

As stated above, I believe the analysis is conducted in a reasonable manner, but the current manuscript lacks a clear motivation for it. I acknowledge that, since I do not work directly in the (sub)field of the paper, I might be missing some crucial background. However, even if that is the case and the paper does offer a very important result, the authors should clearly state why it is important, what its implications are, and what its potential applications could be, ideally with empirical results that highlight the importance of the analysis. I recommend rejection at this point, but I encourage the authors to further develop and expand their paper.

Questions for authors
  • What are the implications (theoretical or empirical) of the main observations?
  • In the DeBERTa-v3 experiments, all runs start from the same pretrained checkpoint, so I would expect the alignment transformations to be close to the identity. How far from the identity are they in practice, and how do the main results change without alignment?
Reviewer_b4K5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper empirically analyzes collections of independent training trajectories to study how their joint effective dimension changes as more runs are included. It uses two tensor flattenings, and two rotation-based null controls.

Under Mode 1 of flattening, effective dimension increases substantially when the first few runs are added, followed by decreasing marginal growth as more runs are included. This is consistent with additional runs contributing increasingly redundant spectral variation.

Under Mode 2, the normalized spectral energy profiles remain relatively stable across run counts.

Critical strengths
  1. The paper is clearly presented, with well-defined measurements and the conclusions are fair.

  2. The empirical findings looks solid, supported by the null comparisons and robustness analyses; with code base available.

Critical weaknesses
  1. Although the analysis includes checkpoints along training trajectory, it makes limited use of the temporal nature of training trajectories beyond including them in the unfolded matrices. Analyses that explicitly examine temporal relationships or exploit the tensor structure, e.g. tensor decomposition might better establish what is learned beyond endpoint-based spectral analysis.

  2. The novelty appears somewhat incremental, relative to prior work on low-dimensional training trajectories and shared weight subspaces. The paper could more clearly establish what new understanding emerges from the run-count analysis.

Questions for authors

What additional insight comes from including intermediate checkpoints compared with applying the same spectral analysis only to final checkpoints? e.g. If I organize end point by stacking them, or by appending them as columns -- will the spectral results and effective dimension result be different from trajectory?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.