acceptodds
Desk rejected at ICLR 2027

When Geometry Flips Architecture Rankings: An Exact Theory of Alignment and Complexity

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

A growing family of model-selection heuristics scores an architecture by how well its inductive bias matches the geometry of the data. We ask when such a score can reverse: given two architectures and a group acting on input coordinates, when does scrambling the geometry flip which one generalises better? Working in a Gaussian sequence idealization in which an architecture acts as diagonal shrinkage in the Fourier basis of the input geometry, we give four results. First, averaging the data over a permutation group makes the covariance distance-independent, so the population fit of any exponential-decay locality score collapses to its floor, and when the mean off-diagonal covariance vanishes the score becomes unidentifiable rather than small, a failure mode we observe in of scrambled fits on natural image patches. Second, the risk gap between two architectures splits exactly into a geometry-free complexity gap and an alignment gain equal to times the covariance between the architectures' squared-bias profiles and the data spectrum; preference inverts under scrambling precisely when opposes and dominates . Third, in a two-band model the inversion boundary admits an exact closed form whose minimum is at noise level , so a geometry-matched architecture pays off only inside a bounded window of sample sizes. Fourth, along any chain of subgroups the alignment gain is controlled by a quantity that contracts monotonically, giving a lock-in theorem: past a certain coarseness of scrambling, the ranking can no longer change. Experiments on natural image patches, a synthetic Gaussian field and handwritten digits confirm each prediction, and parameter-matched convolutional and fully connected networks invert where the theory says they should.

Primary area:
learning theory
Keywords:
Geometric inductive bias, architecture ranking, permutation groups, geometry scrambling, spectral bias, model selection, Reynolds operators, alignment gain, convolutional networks, equivariance, generalization theory, architecture selection

Related papers

Discussion (0)

Sign in to comment.