acceptodds
Withdrawn from ICLR 2027

The Information Geometry of Representation Formation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Most analyses of neural representations focus on trained endpoints, leaving the process by which internal structure forms comparatively unmeasured. We instead treat representation formation itself as an object of study, asking whether the path a representation takes through training reveals information that its endpoint does not. We formalize this idea by viewing representation learning as a geometric trajectory, using Bures geometry to measure movement through representation space and quantum Fisher information to characterize local sensitivity to input factors. We find that learning concentrates local sensitivity on task-relevant factors, and that changing only the prediction target reorganizes this geometry around the corresponding visual factor. More strikingly, representation trajectories become systematically less direct as the learned computation requires higher-order interactions among input factors. This effect persists under matched-performance and training-duration controls and transfers from MLPs on vector inputs to a residual CNN on rendered and natural images. Importantly, formation history contains predictive information about the stability of the learned representation: controlling for interaction order, path tortuosity predicts its subsequent drift under continued training and sensitivity to parameter perturbations, whereas endpoint displacement does not. Finally, representation-aware optimization lets us actively shape these trajectories, showing that formation geometry is not only measurable but actionable. Together, these results establish representation formation as a new source of information about learning: its geometry reveals properties of the computation being constructed, predicts subsequent representational stability, and provides a target for intervention during optimization.

Primary area:
interpretability and explainable AI
Keywords:
representation learning, interpretability, information geometry, representation dynamics, quantum Fisher information, optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3KMSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to study how a hidden layer representation forms during training. At each checkpoint, the representation on a fixed evaluation set is summarized by the centered, trace normalized feature covariance, treated as a density matrix like state ρt\rho_t. Movement between checkpoints is measured with the Bures distance, and tortuosity τ=LB/DB\tau = L_B / D_B (path length divided by initial to final displacement) summarizes how direct the training trajectory is. The quantum Fisher information (QFI), which is the infinitesimal Bures metric, quantifies the local sensitivity of the state to controlled perturbations of known input factors. Pulling this metric back to parameter space yields a representation natural gradient (RepNG).

Empirically, the paper reports four findings. (i) After training, QFI concentrates on task relevant factors, and retargeting the task on dSprites reorganizes the sensitivity accordingly. (ii) Tortuosity increases with the interaction order kk of the target computation. This holds for sign pattern and parity tasks with MLPs on Gaussian inputs, under matched accuracy and matched duration controls, checkpoint subsampling, and alternative distances, and it transfers to a small residual CNN on dSprites and to XOR tasks over animal/vehicle labels on CIFAR-10 composites (5 seeds). (iii) In a fresh 30 seed sweep with interaction order as a fixed effect, tortuosity predicts representational drift over 30 further epochs (β=0.87\beta = 0.87) and sensitivity to random weight perturbations (β=0.51\beta = 0.51), whereas initial to final displacement predicts neither. (iv) RepNG yields lower tortuosity than Adam and SGD at matched accuracy in a 24 dimensional random subspace of a 6-16-8-4 MLP, while its difference from output natural gradient is not statistically resolved at matched performance (p=0.32p = 0.32).

Critical strengths
  1. A clean, reusable measurement framework, applied with unusual care. Defining the state as a trace normalized covariance gives a bounded object with a principled local counterpart (QFI is the infinitesimal Bures metric), so local sensitivity and global movement share one geometry. The controls are strong for this genre: fixed evaluation sets, within seed pairing, hyperparameters selected on disjoint pilot seeds, permutation tests, checkpoint subsampling (Table 5), alternative distances (App. E.6), and a function preserving reparameterization audit (Table 4). The main ordering reproduces across MLPs, a small residual CNN, and CIFAR-10 composites. This makes the descriptive finding credible and the tool reusable, including potentially on longitudinal neural recordings, where covariance states of population activity are directly measurable.

  2. A potentially new prognostic use of training histories (Sec. 4.3, App. G). Within a fixed task, a path statistic predicts later drift (β=0.87\beta = 0.87) and perturbation sensitivity (β=0.51\beta = 0.51), using an appropriate design: fresh seeds, task fixed effects, seed clustered errors, and permutation inference. If this survives stronger baselines (see Weakness 1b), it would justify treating formation history as an object of study in its own right, which is the paper's core aspiration.

Critical weaknesses
  1. The two central interpretations are not separated from simpler explanations. (a) Interaction order is confounded with optimization difficulty. Parity targets provide no low order signal and are known to induce long plateaus with hidden progress (e.g., Barak et al., 2022). Higher kk also means far more time at high loss: Table 7 reports 1.9 versus 175.3 epochs to threshold for k=1k=1 versus k=4k=4. Matched accuracy leaves duration unmatched, and matched duration leaves progress unmatched; neither matches difficulty itself. Some comparisons are near degenerate: k=1k=1 parity stops after about 2 epochs with τ=1.014\tau = 1.014; in Table 2 (CIFAR, seed 4, k=1k=1), LB=DB=1.277L_B = D_B = 1.277 and τ=1.000\tau = 1.000, apparently a single segment trajectory; checkpoint counts per trajectory are not reported, and CIFAR uses only 5 seeds. Because DB≤2D_B \le \sqrt{2} and is near saturated in the CNN and CIFAR runs (about 1.1 to 1.3), τ\tau there is essentially path length, so the finding partly reduces to "harder tasks move the representation more overall," which is less surprising. Intermediate orders are weakly separated in the matched duration control (Table 6: 2.2942.294 vs. 2.2842.284 at 60 updates, 1.9401.940 vs. 1.9561.956 at 40 updates, both inverted). No trajectory level analysis supports the "intermediate construction and reorganization" account; Figs. 1 and 5 are schematics. (b) The claim that the path carries information the endpoint does not is tested only against initial to final displacement, a quantity with no a priori reason to predict drift. Natural competitors are not examined: path length alone, recent step size or velocity near the end of training, final loss, gradient norm, a sharpness proxy, and spectral properties of the final state ρT\rho_T. Continued motion of a not yet converged representation would produce the observed associations without any role for path shape. This weakness is consequential because, if these simpler explanations hold, both headline claims reduce to known phenomena.

  2. The intervention, theory, and fit to the chosen area are limited. RepNG is tested only in a 24 to 48 dimensional random subspace of a roughly 284 parameter MLP stopped at 72% accuracy. At matched performance it is indistinguishable from output natural gradient (p=0.32p = 0.32), and the headline 29% reduction is relative to Adam, whose per coordinate normalization is arbitrary in a randomly rotated subspace. Since RepNG explicitly minimizes per step representational movement, shorter paths are largely built into the method, and its smaller endpoint displacement (DB=0.636D_B = 0.636 vs. 0.7360.736 for Adam, Table 8) is consistent with simply moving less overall. The paper never shows that lower tortuosity buys anything, not even the stability that Sec. 4.3 associates with it. Theorems 1 and 2 and Proposition 3 are standard results from the cited literature. The claimed local to global unification remains formal: the QFI analysis and the path analysis are never connected empirically. Finally, for the selected primary area (neuroscience and cognitive science), the paper contains no neural or behavioral data and does not engage with directly relevant work such as representational drift, mathematical theories of learning dynamics (e.g., Saxe et al., 2019), or classic category learning paradigms that manipulate interaction structure (Shepard, Hovland & Jenkins, 1961).

Questions for authors
  1. Difficulty matched control. Can a lower order task be made as hard to learn as a higher order one, for example k=1k=1 with tuned label or input noise so that loss curves match k=3k=3, and can same kk tasks be run at different difficulties? Does τ\tau then track interaction order or difficulty?

  2. Stronger baselines for Sec. 4.3. Does τ\tau remain predictive after adding LBL_B, recent step size, final loss, gradient norm, a sharpness proxy, and spectral properties of ρT\rho_T as covariates? Please report coefficients with confidence intervals and per kk scatter plots.

  3. Trajectory composition. How many checkpoints enter each CIFAR and dSprites CNN trajectory? Is Table 2, seed 4, k=1k=1 a single segment? Do the CIFAR results hold at matched duration, at sub epoch resolution, and with more seeds?

  4. Mechanism. Can you plot distance to the final state and step size over training, to show whether the extra path length comes from plateau wandering or from genuine reorganization events?

  5. Intervention. Do RepNG trained networks show lower subsequent drift or perturbation sensitivity, as Sec. 4.3 would predict? How many runs were excluded as not "successful matched runs" (App. A.6)? Given that the optimizer alone shifts τ\tau by about 0.6 to 0.8 (Table 8), comparable to the whole kk range in Table 5, does the kk ordering hold across optimizers, learning rates, and batch sizes?

  6. Which layer defines the state in each MLP experiment, and what sample sizes are used for the tasks in App. A.4?

  7. On the unit trace slice, the geodesic length is the Bures angle rather than the chord dBd_B, so a geodesic trajectory at the DBD_B values in Tables 1 and 2 already has τ≈1.06\tau \approx 1.06 to 1.081.08. Does any conclusion change if the angle is used instead of the chord?

If Q1 and Q2 are addressed and the claims survive, I would raise it to 3. Suggestions for future work: consider stimulus indexed representations (Gram or shape metrics), since ρ\rho is invariant to which stimulus maps to which feature direction; consider the Shepard, Hovland, and Jenkins category types, which dissociate interaction order from the number of relevant dimensions and come with a known human difficulty ordering; and apply the method to neural population recordings collected during learning, where representational drift is directly observable.

Reviewer_D3X6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how representation geometry forms over the course of training and what factors influence these representational learning dynamics, such as difficulty of the task. The authors look at whether nonlinear learning dynamics predict future representational instability in training and also conduct ablations with an optimizer that encourages linearity. The results are intuitive. Experiments are simple and small-scale.

Critical strengths

I think the motivation itself is quite interesting and the results are novel. I have not seen prior work that shows that representations develop nonlinearly for harder tasks. In fact, I have mainly seen results that refute this, showing that learning dynamics are primarily linear (e.g. https://labs.baseten.co/articles/the-unreasonable-efficiency-of-a-single-gradient-step, [1]).

[1] Wang, T., Liu, J., Wu, Z., Jin, S., Chen, W., Xu, H. and Miao, N., 2026. Linear Dynamics in the RLVR Training of Large Language Models. arXiv preprint arXiv:2601.04537.

Critical weaknesses

This paper does not engage strongly with relevant prior work on linear feature learning dynamics nor representation dynamics, such as [1-3]. The utility of the proposed optimizer is completely unclear to me. Figures 1 and 5 are generated by AI

[1] Shai, Adam S., Sarah E. Marzen, Lucas Teixeira, Alexander G. Oldenziel, and Paul M. Riechers. "Transformers represent belief state geometry in their residual stream." Advances in Neural Information Processing Systems 37 (2024): 75012-75034. [2] Park, Core Francisco, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. "Iclr: In-context learning of representations." In International Conference on Learning Representations, vol. 2025, pp. 53258-53284. 2025. [3] Park, Core F., Maya Okawa, Andrew Lee, Hidenori Tanaka, and Ekdeep S. Lubana. "Emergence of hidden capabilities: Exploring learning dynamics in concept space." Advances in Neural Information Processing Systems 37 (2024): 84698-84729.

Questions for authors

What is the proposed downstream utility of these types of analyses and more importantly of the optimizer? Why did you not consider transformer architectures or non-synthetic data? Do you see similar dynamics for different types of training (classification, regression, generation, RL, etc)?

Reviewer_zBTZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines some geometric quantities for studying the representational structure of neural networks during and after training, including a QFI-based local sensitivity, and tortuosity which compares the traveled representational distance to the straight-line distance. Next the paper shows a few experimental results for these quantities: the QFI is shown to become mainly aligned to task-relevant feature directions after training, the tortuosity increasing with the number of interactions in the task, tortuosity is predictive of future drift and sensitivity to weight perturbations, and a newly introduced natural gradient "RepNG" is shown to make trajectories more direct.

Critical strengths

Paper introduces and clearly explains some potentially useful definitions for characterizing representation geometry. They all feel natural and as far as I know have not been explicitly written down before in this context.

Critical weaknesses

Significance of new definitions has not been fully established in my eyes. E.g. the paper does mention "Formation history can matter even when endpoints appear equivalent.", but in the experiments the only way this has been shown to matter is that tortuosity predicts continued-training drift and parameter sensitivity. I can see how both of these could be related to stability, but a more general sense of stability that matters in practice (e.g. convergence, robustness, generalization) has not been shown. The paper could benefit from an experiment testing a connection to previously more established quantities, such as testing if models that fail to converge have higher overall tortuosity, showing that training with RepNG provides practical benefits, showing that models similar endpoints but different tortuosity are somehow different in behavior, or anything of this nature.

As far as I can tell, the QFI matching the task-relevant directions after training seems not very novel as a result as neural networks are already known to compress task-irrelevant directions (e.g. Saxe 2014, Achille 2018, Flesch 2022), although it validates the use of QFI here and is a different and perhaps more direct test.

Questions for authors

Is tortuosity correlated to any other property, besides continued-training drift and parameter sensitivity?

Do RepNG-trained models have lower continued-training drift or perturbation sensitivity?

Could it be the case that trajectories with high tortuosity are simply experiencing more noise (i.e. are more jittery as a random walk), or does tortuosity capture something beyond this? If so, did you do any experiments distinguishing a high-tortuosity trajectory from a noisy one?

What should I take away from figure 3, i.e. what does the observation that the tortuosity correlates to the interaction order tell me?

How is the layer chosen from which to take the representations, and which layer is used for the MLP experiments and for RepNG?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.