- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The primary goal of this paper is to introduce new theoretical findings concerning training neural networks through a lower-dimensional weight representation mapped onto the actual weights through a chart. This work proves that starting at weights set to zero (common in recently introduced methods) actually paralyzes trainability of important operations in the model and that this comes from the architecture and not task complexity. The authors also show in limited experiments that for randomly initialized weights, the chart-training avoids total collapse but the weights remain distorted.
Critical strengths
The biggest strength of the paper is that it theoretically exposes a blind spot in how we measure intrinsic dimension. The authors prove that using a zero-centered frozen chart to generate weights practically paralyzes multiplicative operations, like attention, because gradients vanish. So instead of measuring the actual difficulty of a task, past work has often just been measuring an architecture's inability to train around its own dead pathways.
Critical weaknesses
The main weakness is a quite limited experimental section. The main issue is that the authors only evaluate their theory on very small, older image and graph tasks rather than large modern models, especially on language tasks. This is a critical weakness because in a small network, breaking one operation naturally forces a visible drop in performance or forces the network to train around it. However, large networks contain way more redundancy, meaning they might easily route around these blocked operations without any real accuracy loss. By avoiding experiments on large-scale models, the authors do not prove that this zero-centered gradient problem actually harms the performance of the large models the ML community cares about.
Furthermore, every test they run builds the model entirely from scratch starting from zero-initialized weights or randomly-initialized state. From the practical perspective, the insights from the paper would be particularly interesting in the fine tuning (PEFT) setting which the experimental section completely ignores.
The second big weakness is poor presentation, specifically:
- The paper jumps right into the topic without proper introduction and presentation of core concepts. The very first sentence of the introduction immediately starts talking about how a "frozen random chart maps a unit-norm latent vector" and drops the weights below their "usual initialization". It completely skips explaining why we should care about training in a small space or what the core problem is that the paper tries to solve. Furthermore, defining 'intrinsic dimension' and 'weight generation' at a high level in paragraph 1 would make the paper significantly more accessible.
- Theorems read more like data dumps. I don’t see much explanation what they mean, what understanding they bring
- In the experiments section, the questions of interest are presented in one order and then answered in another order.
- Figure 5 would be more readable if it had explicit annotations for layer types (e.g., WQ, WK, conv1).
- In Figure 7b, multiple markers are plotted for the same legend keys, making it difficult to tell which specific operations correspond to which points across the four tested architectures.
- This may be a personal opinion, but figures are usually way easier to read if the description contains the conclusions from the figure and not barely states what it presents (as in this paper).
Questions for authors
- How does functional order interact with Pre-LN LayerNorm / RMSNorm architectures and deep residual streams in modern LLMs compared to the compact models tested?
- Are the limitations identified by the theory still an issue for larger models?
- How would the fine tuning setting change the experimental results for questions Q1-Q4?
- Given the fine tuning setting should avoid multiplicative collapse, do your distortion bounds (Theorem 1) significantly impact parameter-efficient fine-tuning (PEFT) in practice?