acceptodds
Withdrawn from ICLR 2027

Intrinsic Dimension in Frozen Weight Space: Distortion at Solution, Collapse at Origin

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A neural network can be trained through a few coordinates mapped into its weights by a fixed low-dimensional map, a chart. The smallest such chart that solves a task is reported as the task's intrinsic dimension, and the same construction now generates weights outright. A basic question has gone unasked: is a chart that is well behaved in weight space enough to give a well-behaved optimization problem in the trained coordinates? We study it where the chart is fixed, random and known exactly, and find that it is not. The architecture reshapes the mapped directions in two ways. It distorts them: at trained solutions a handful of directions carry almost all of the pullback Fisher mass, and a better-oriented chart helps only slightly, at intermediate budgets. And it can collapse them: an operation that multiplies two generated matrices, as attention multiplies its query and key, responds to the chart only at higher order near a zero base point, so attention barely moves while the rest of the network trains around it. We characterize collapse by a functional order: the power of latent scale in its gradient. The computation graph predicts this integer for charts without leading-term cancellation. Measured slopes follow it on graph and image hosts, approaching it more slowly on a vision transformer; changing which matrices are generated moves the slopes as predicted without adding trainable parameters. Measurements with zero-centred charts, as in current weight generators, therefore depend on the host and the base point as well as the task; the original construction, centred at initialization, avoids collapse but not distortion.

Primary area:
optimization
Keywords:
intrinsic dimension, reparameterization, optimization geometry, Fisher information, hypernetworks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1m8XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The primary goal of this paper is to introduce new theoretical findings concerning training neural networks through a lower-dimensional weight representation mapped onto the actual weights through a chart. This work proves that starting at weights set to zero (common in recently introduced methods) actually paralyzes trainability of important operations in the model and that this comes from the architecture and not task complexity. The authors also show in limited experiments that for randomly initialized weights, the chart-training avoids total collapse but the weights remain distorted.

Critical strengths

The biggest strength of the paper is that it theoretically exposes a blind spot in how we measure intrinsic dimension. The authors prove that using a zero-centered frozen chart to generate weights practically paralyzes multiplicative operations, like attention, because gradients vanish. So instead of measuring the actual difficulty of a task, past work has often just been measuring an architecture's inability to train around its own dead pathways.

Critical weaknesses

The main weakness is a quite limited experimental section. The main issue is that the authors only evaluate their theory on very small, older image and graph tasks rather than large modern models, especially on language tasks. This is a critical weakness because in a small network, breaking one operation naturally forces a visible drop in performance or forces the network to train around it. However, large networks contain way more redundancy, meaning they might easily route around these blocked operations without any real accuracy loss. By avoiding experiments on large-scale models, the authors do not prove that this zero-centered gradient problem actually harms the performance of the large models the ML community cares about.

Furthermore, every test they run builds the model entirely from scratch starting from zero-initialized weights or randomly-initialized state. From the practical perspective, the insights from the paper would be particularly interesting in the fine tuning (PEFT) setting which the experimental section completely ignores.

The second big weakness is poor presentation, specifically:

  • The paper jumps right into the topic without proper introduction and presentation of core concepts. The very first sentence of the introduction immediately starts talking about how a "frozen random chart maps a unit-norm latent vector" and drops the weights below their "usual initialization". It completely skips explaining why we should care about training in a small space or what the core problem is that the paper tries to solve. Furthermore, defining 'intrinsic dimension' and 'weight generation' at a high level in paragraph 1 would make the paper significantly more accessible.
  • Theorems read more like data dumps. I don’t see much explanation what they mean, what understanding they bring
  • In the experiments section, the questions of interest are presented in one order and then answered in another order.
  • Figure 5 would be more readable if it had explicit annotations for layer types (e.g., WQ, WK, conv1).
  • In Figure 7b, multiple markers are plotted for the same legend keys, making it difficult to tell which specific operations correspond to which points across the four tested architectures.
  • This may be a personal opinion, but figures are usually way easier to read if the description contains the conclusions from the figure and not barely states what it presents (as in this paper).
Questions for authors
  1. How does functional order interact with Pre-LN LayerNorm / RMSNorm architectures and deep residual streams in modern LLMs compared to the compact models tested?
  2. Are the limitations identified by the theory still an issue for larger models?
  3. How would the fine tuning setting change the experimental results for questions Q1-Q4?
  4. Given the fine tuning setting should avoid multiplicative collapse, do your distortion bounds (Theorem 1) significantly impact parameter-efficient fine-tuning (PEFT) in practice?
Reviewer_LymbOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies training a network only through a low-dimensional latent zz. A frozen affine chart g(z)=θ0+C⊤zg(z)=\theta_0+C^\top z with orthonormal rows maps zz into weight space. This is the setting of intrinsic-dimension (ID) measurements, where the base point is the random initialization (Li et al., 2018), and of recent zero-centred weight generators, where θ0=0\theta_0=0. The paper separates two effects:

  • Distortion. At trained solutions the pulled-back Fisher is strongly anisotropic, and a retained-decrease ratio γ(C)\gamma(C) quantifies how much local decrease a chart keeps.
  • Collapse. With θ0=0\theta_0=0, a block's latent gradient vanishes like εord\varepsilon^{\mathrm{ord}} along z=εuz=\varepsilon u. A power-counting rule on the computation graph (Thm. 2, Eq. 7) predicts this functional order before training, e.g. 4 for query/key in a graph transformer (GT). Under a non-cancellation condition the order does not depend on the chart (Thm. 3). Freezing or merging a factor, or moving θ0\theta_0, lowers it.

Experiments use small GIN/GT/ViT/CNN hosts. Fitted gradient slopes match the predicted integers (Table 9). Under Adam, attention escape times are ordered by parameterization (Table 14). Charts aimed at curvature directions give small gains (Table 7). An ID-style threshold moves from d=64d=64 to d=2048d=2048 when the base point moves from initialization to zero (Table 16).

Critical strengths
  1. A clean separation of two mechanisms with different interventions. Geometry depends on the chart's span; the order depends on the parameterization and base point (Thm. 3, Cor. 1). The predictions are concrete and falsifiable, including the order −1-1 with the classifier outside the chart (Table 17).
  2. Careful checks of the predictions. Fitted slopes lie within 0.01 of the predicted integers on GT and on the CNN without normalization (Table 9). They survive projection onto the chart across ten chart draws (Table 1) and barely change under a shared latent. Where the theory falls short, e.g. on ViT, the paper says so.
  3. Unusually transparent reporting. Each claim gets a verdict (Table 4), withdrawn artifacts are listed (App. F), and confounds and the missing d=0d=0 control are acknowledged (l. 357–358, 1505–1506). The exactly solvable scalar model (§3.6, Prop. 2) is a helpful illustration.
Critical weaknesses
  1. The link from functional order to training outcomes is weak.
    • The optimizer measures degree, not order. All runs use Adam/AdamW. By the paper's own Thm. 4(a) (l. 302–304), growth under bounded-step optimizers is governed by the degree qq of the block (2 for the factored score), not by its order (4). Consistently, the factored escape time barely changes when ε\varepsilon shrinks tenfold: 283 vs. 265 steps on NCI1 (Table 14). Meanwhile, the key-frozen and merged arms share order 3 yet escape 1.25–4× apart.
    • Confounds. The arms differ in initial score scale (Table 11), so the comparisons conflate order with scale. The factored arm also shares one QKV latent (l. 1401–1402), unlike the setting of Thm. 4(b).
    • The opening example. The base-point arm, of order 0, also barely moves attention: entropy ratio 0.9911 → 0.9866 (Table 12). The 50× and base-point rows are single-seed.
    • Accuracy. Accuracy does not follow the order on GT (Table 11). On ViT, the differences are at 14–17% accuracy (Tables 13, 15).
    • Contribution 2 is not shown. The claim that low ID need not imply an easy task (l. 87–89) has only Table 16 as evidence. By the paper's own account, that crossing "chiefly records when the chart first beats a near-trivial predictor" (l. 1504–1505), and no d=0d=0 control was run.
    • Requests:
      • comparisons under a common regime that includes plain SGD, where order should matter (Prop. 2);
      • arms matched in initial scale;
      • multi-seed versions of Table 12;
      • a d=0d=0 control.
  2. The effects are small, at near-trivial accuracy, without the statistics needed to interpret them.
    • PROTEINS: the threshold is 1.5 points above the majority rate of 0.596 (l. 469).
    • NCI1: GT with a random chart stays at 0.635–0.641 for d=8d=8 to 1024 (Table 6), and no arm reaches 90% of free accuracy by d=8192d=8192.
    • ViT on CIFAR-100: it reaches 0.14–0.17.
    • Aiming: the gain is +0.21 ± 0.09 points (Table 7).
    • Statistics: results use 3 seeds and fixed splits, report the last-epoch test score, and have no validation fold or model selection (App. E). Cross-validation is the usual protocol for the TU datasets.
    • Chart contribution: GT trains 36,096 parameters with its degree embedding outside the chart (l. 1704), so the chart may contribute little.
    • Requests: more seeds with error bars, cross-validation, d=0d=0 baselines, and budgets at which the chart reaches non-trivial accuracy.
  3. The theory and scope are modest relative to the framing, and the presentation is hard to follow.
    • Theory. The core of Thm. 2 is power counting, which the paper itself calls elementary (l. 126). Equality is assumed via non-cancellation and only checked numerically (Table 1). The gradient-flow case of Prop. 2 appears closely related to classical analyses of small-initialization factorized linear models (e.g. Saxe et al., 2014). Initialization scale for generated weights has been studied for hypernetworks (Chang et al., ICLR 2020). Neither is cited. Fisher anisotropy at solutions is known (Karakida et al., 2019; Papyan, 2019, both cited).
    • Scope. The standard ID constructions use nonzero base points: Li et al. (2018), Aghajanyan et al. (2021), Gressmann et al. (2020). There every block has order 0 (Cor. 1), so collapse concerns only zero-centred weight generators. The title and abstract suggest broader relevance to ID. The paper should also state that "intrinsic dimension" here is the weight-space notion of Li et al., not the intrinsic dimension of data.
    • Presentation. The paper is very dense and introduces many coined terms (chart, valuation, collapse, aiming band, functional order). Key caveats sit in captions and appendices: single-seed rows, the missing d=0d=0 control, confounds. Despite considerable effort, I found it hard to follow as a reader from an adjacent area. I suggest restructuring, moving the caveats into the main text, and introducing the scalar model of §3.6 early as a worked example.
Questions for authors
  1. Does full-parameter training from the same small weights reproduce the attention immobility? If so, what is specific to charts or ID?
  2. In Table 12, the order-0 base-point arm moves about as little as the 50× arm. How is the opening example evidence for order? Could you report multi-seed entropies and an arm with matched initial score s.d. but a different order?
  3. Under Adam, which sets the escape time: order or degree? Did you run plain SGD to test the log⁡(1/ε)\log(1/\varepsilon) growth of Prop. 2?
  4. What does GT reach at d=0d=0, i.e. training only the parameters outside the chart, on NCI1 and PROTEINS?
  5. Can non-cancellation be proved for almost every chart, e.g. via analyticity in CC plus one witness, rather than checked on ten draws?
  6. In Thm. 1, what is the constant cc, and can the aiming band be computed for your Fisher estimates?
  7. In the merged arm, what are the shape and rank of WQKW_{QK} per head, and does merging change the function class?
  8. Does your replication of Sen & Mukherjee (2026) include their training regularizers? Does any published zero-centred weight generator use attention?

(9. Please relate your notion of intrinsic dimension (Li et al.) to the intrinsic dimension of data, to avoid confusion for readers from that area.)

Reviewer_vyQvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies intrinsic-dimension analysis with zero-centred weight subspaces, and shows that multiplicative operations such as attention can be largely untrained in this setting. The paper develops a theoretical tool that predicts the leading-order accessibility of training signals from the computation graph. The prediction is verified with measured gradient slopes on various architectures in graph (GIN, graph transformer) and image (CNN, ViT) domain, suggesting that intrinsic dimension measurements can be confounded by architectural accessibility.

Critical strengths

The experiments are conducted across diverse domains (graph and image) and several architectures, which is reasonably thorough for a theory paper. The experiments are carefully controlled to test the specific claims made. The authors transparently report the scope and limitations of their predictions and findings.

Critical weaknesses

[Concern 1] The zero-centered setting is not motivated. I could not find a paper that uses zero-centered weights in intrinsic dimension literatures. The suggested architectural accessibility problem due to multiplicative operations only holds in zero-centered setting. In fact, Li et al., 2018 already explicitly avoids this for similar reason:

This convention serves an important purpose for neural network training: it allows the network to benefit from beginning in a region of parameter space designed by any number of good initialization schemes (Glorot & Bengio, 2010; He et al., 2015) to be well-conditioned, such that gradient descent via commonly used optimizers will tend to work well.

The problem this paper addresses appears to be just one of many pathologies that could occur with improper and ill-conditioned weight initalization.

[Concern 2] The actionable conclusion is limited. A trivial solution to this problem appears to be just using randomly initialized weights in the null-space of the chart. Regardless of the correctness of the theoretical and experimental results, it is therefore unclear whether this accessibility problem is a real one.

[Concern 3] The paper invents non-standard terminologies for standard objects ("host" for the target network, "chart" for a random subspace parametrization, "valuation", "aiming band"). The writing is also unnecessarily dense and hard to follow. A substantial rewrite with standard terminology and a clearer narrative is needed.

Questions for authors

Is there any intrinsic dimension research that uses zero-centering?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.