Intrinsic Dimension in Frozen Weight Space: Distortion at Solution, Collapse at Origin

A neural network can be trained through a few coordinates mapped into its weights by a fixed low-dimensional map, a chart. The smallest such chart that solves a task is reported as the task's intrinsic dimension, and the same construction now generates weights outright. A basic question has gone unasked: is a chart that is well behaved in weight space enough to give a well-behaved optimization problem in the trained coordinates? We study it where the chart is fixed, random and known exactly, and find that it is not. The architecture reshapes the mapped directions in two ways. It distorts them: at trained solutions a handful of directions carry almost all of the pullback Fisher mass, and a better-oriented chart helps only slightly, at intermediate budgets. And it can collapse them: an operation that multiplies two generated matrices, as attention multiplies its query and key, responds to the chart only at higher order near a zero base point, so attention barely moves while the rest of the network trains around it. We characterize collapse by a functional order: the power of latent scale in its gradient. The computation graph predicts this integer for charts without leading-term cancellation. Measured slopes follow it on graph and image hosts, approaching it more slowly on a vision transformer; changing which matrices are generated moves the slopes as predicted without adding trainable parameters. Measurements with zero-centred charts, as in current weight generators, therefore depend on the host and the base point as well as the task; the original construction, centred at initialization, avoids collapse but not distortion.