- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper tries to provide a theoretical proof with a single assumption that data possess a hidden hierarchy that is shared at the top (semantic content) and separated at the bottom (structure). The paper experiments with synthetic data modeled with a multilingual random hierarchy model (RHM) and compares a GPT-style decoder-only transformer with a theoretical belief propagation (BP) network. Results showed that geometric similarity was highest on the middle layer, with the emergence of surface morphosyntax layers in the early and later stages of the layers. The same findings were observed in both the transformers and BP network, whether trained on both synthetic languages A & B or independently. The paper also proves, with the proposed theory, that there is a better representation to present the shared semantic content by subtracting the predictable component from preceding layers, which is called layerwise novelty. Also, separates coordinate alignment from geometric similarity affected by the code-switching rate and training size via latent cross-probing.
Critical strengths
- The layerwise novelty part is a novelty and a significant contribution in terms of LLM theory and interpretability. This highlights the applicability of the theory proposed by the paper, and the paper itself is able to prove it empirically.
- Able to distinguish the geometric similarity in multilingual representations and coordinate alignment for cross-lingual transfer, which is I believe critical in terms of understanding LLM dynamics around this area.
Critical weaknesses
- Incremental extension from the monolingual RHM and BP kinda limited the contributions of the work
- The way the experiment is conducted does not explain further on why the training dynamics on transformers, mainly on gradient descent, map monotonically to layer depth. What I mean by it is the multilingual RHM data and the usage of BP network kinda induce the middle-layer similarity peak directly into the network construction. Knowing how BP message passing work, it is kinda intended and expected to behave this way. Maybe I should say this is more towards algorithmic proxy rather than theory of how transformer learns multilingual data?
- I think modelling cross-lingual divergence solely as disjoint symbol substitutions on a fixed-depth oversimplified actual linguistic structure.
Questions for authors
- If the languages differ in a more complex way, for example like order (one is right-to-left, other is left-to-right), or any other complex differences, do most findings still hold?
- can the authors clarify why the standard transformer is expected to follow the same approach as BP?
2 comments
Thank you for the very insightful response. I think now my concerns has been address, I will adjust my review accordingly.
Response to Reviewer QHDj
We thank the reviewer for their thoughtful review.
Contribution beyond RHM: The technical contributions of our paper going past previous works include studying next-token prediction on the Random Hierarchy Model (RHM), and constructing theoretical networks that solve this task using encoded belief propagation (BP) messages. Most importantly, our paper introduces a framework to model different languages/modalities, based on the notion that they share the high level rules of their generative hierarchy. This key addition allowed us to capture many phenomena observed but unexplained in LLMs, and also make new predictions which we verified.
Training dynamics vs algorithmic proxy: A full-fledged theory of gradient descent in deep architectures (let alone transformers) is a long-standing question in the field, which has not been resolved. However, the RHM is one of the rare cases where a rigorous theory of dynamics approximating GD was made in MLPs, see Ren et al (2026). It corresponds to the supervised setting where the root is predicted, and where the networks learn a representation similar to the encoding part of BP.
This is consistent with our results, in particular Fig.10 that provides a direct check of layerwise agreement between the theoretical BP network and the trained transformer. We show that the accuracy of linearly predicting layer-l BP representation hBP(l) from layer-l′ transformer representation htransformer(l′) follows the same trend as predicting hBP(l) from hBP(l′).
We will discuss the existing connections between our theoretical network and GD, and cite Ren et al.
Complex linguistic structures: Any tractable theory of natural languages must obviously make simplifying assumptions. Our goal is to use the simplest setup that captures various observations in LLMs. The comparison with natural languages indeed support that our conclusions are robust to the specific choices we made. Robustness is already tested with respect to our choice of fixed topology trees: in Fig.17 of the paper, we show that our conclusions also hold for trees with variable topology. In response to the specific comment of the referee on shared v.s. non-shared vocabulary, we show in https://anonymous.4open.science/r/rebuttal_shared_vs_disjoint_vocabulary-04FD/ that having partially shared, instead of disjoint, vocabulary does not qualitatively change our conclusions.
Languages with Left-to-right vs right-to-left order: The transformer's next-token prediction objective follows the order in which the text is read, regardless of whether the script is written left-to-right or right-to-left. We thus expect our findings not to depend on script direction.
References:
Yunwei Ren, Yatin Dandi, Florent Krzakala, Jason D. Lee, Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chaining. Colt 2026.