- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates how hybrid attention architectures affect multilingual representations. Most importantly, the authors find that cross-lingual alignment emerges predominantly in full-attention layers: Across a range of architectures that alternate attention types, configurations placing full attention first or as the only attention method consistently exhibit the highest degree of multilinguality in their representations, which has direct implications for model design. The authors show, using alternative architectures and distillation experiments, that models that start out with full attention in the first layers outperform their counterparts in terms of multilingual alignment in every setting.
I enjoyed reading this paper a lot. The weaknesses I see are rather minor and don't have any critical criticism at this point. I recommend weak acceptance, but I'm happy to increase my score if the authors provide additional input to the questions below.
Critical strengths
-
In general: The paper identifies an interesting and relevant gap and investigates it thoroughly. An interesting hypothesis is derived from representational analyses (Findings 3, 4, 6) and then tested empricially, which gives the paper a very nice flow. Findings are highly actionable and could directly influence architectural choices in future multilingual models:
- To my knowledge, this is the first study of how hybrid attention affects multilinguality, and it is timely given that many current open-weight model families adopt hybrid architectures as a standard
- The fix that the authors propose to improve the degree of multilinguality is very simple and easy to adopt in principle, by a simple reordering of layers.
-
The paper is, for the most part, very clearly and concisely written:
- The authors are refreshingly transparent about imperfect comparisons and argue convincingly that these comparisons are sufficient to support their claims, which substantially increased my confidence in the results.
- Section 5.2 is the strongest part of the paper: it is clearly written and ties the individual findings together extremely well.
- The main findings are clearly fleshed out (also just visually), which makes following the paper very easy
Critical weaknesses
- The choice of SoftCKA is insufficiently motivated. The authors state "We find this metric for cross-lingual alignment much less noisy than alternatives we explore" without providing evidence. Which alternatives were considered, and what are their measurable disadvantages?
- Section 6 did not fully convince me, mainly because the evaluation is thin: The students are only compared via KL to the teacher and cross-entropy on held-out text - I would have liked to see actual downstream results.
- Writing/clarity: the placement and purpose of Section 4 were initially unclear to me. I assume it serves as a motivating evaluation for the remainder of the paper; this should be stated explicitly. In addition, OneRuler should be introduced briefly. A single sentence would spare readers from consulting the original paper.
Questions for authors
- Which other mtrics for cross-lingual alignment did you test? Why did they perform more noisy?
- Can you report downstream results for the distilled students, e.g. on the benchmarks from Appendix B, and OneRuler at longer contexts?
Response to Reviewer YRFs
We are grateful for the reviewer’s positive feedback and hope to address some of their weaknesses/questions raised.
W1 & Q1
We acknowledge that L38-44 is a bit vague about why we liked SoftCKA more than alternatives. In large part, it was because of the amount of space the description of the metric already required ! We’ll expand on the intuition behind the metric and remove the claim that it was less noisy because that’s hard to quantify (though this is how we selected it originally, based on visualizations).
Obviously, representations happen at the token level (hidden states). CKA is a natural starting point because it measures shared structure relative to the variation within each representation, giving us a common scale for comparing alignment across layers. It also makes few assumptions about the “shape” of the representation space. The main problem with mean-pooling embeddings is that directions cancel out and not all tokens are equally informative. In contrast, SoftCKA compares every token state in one sentence with every token state in its translation, then measures the variation in those similarities after centering, relative to the corresponding within-language variation. This lets us examine token-level structure even when translations differ in length or word order. We interpret agreement with prior layerwise findings as a qualitative consistency check.
Alternatives we looked into were more basic: either mean-pooling or using the last hidden state and then using CKA or cosine similarity as measures of similarity. We also tried using logit lens and calculating KL-div. All seemed to suffer from significant information loss.
W2 & Q2
For further convincing, we also showed in response to reviewer SGZG that English-only training resulted in far smaller difference between standard-periodic and reverse-periodic (ours) layer ordering, which is new evidence.
But wrto evaluation: we stuck to training signals (on the validation set) primarily because we have not trained the student models to completion (would take significant compute). We believe the most important finding is what the training curves show: In all experiments, not only has our experimental ordering learned much faster (lower loss or KL) but it is still learning faster at the end of our trainings (steeper slope), implying that it would continue having lower loss or KL for significantly longer in training.
Upon your request, we did however try benchmarks that work with simple next-token completion setups, Belebele and Global-PIQA, and found that they did show statistics between the layer orderings and provide further evidence for our conclusions. We evaluated the standard-periodic and reverse-periodic students zero-shot on Belebele (covers 23 out of 26 languages) and Global PIQA (covers 22). Reverse periodic is more accurate on Belebele for both models (+3.0 points for Qwen3-4B, +2.4 for Granite) and on Global PIQA for Qwen3-4B (+2.3). On Global PIQA for Granite the difference (+1.1) is positive but not statistically significant. N.B. statistical significance is computed by bootstrapping questions, not with more random seeds. As with KL, the gain is larger for non-Latin scripts (Belebele +3.5 / +3.0 vs. +2.0 / +1.2 for Latin), and reverse periodic is ahead in 19 of 23 Belebele languages for each model. We were surprised such downstream evaluations would reflect the difference so early in training, so we thank the reviewer for this suggestion.
W3
We acknowledge that Section 4 does not fit super cleanly into the rest of the paper. However, we simply found that any discussion of the hybrid attention and multilinguality was incomplete without a discussion of token-inflated sequences in multilingual settings. Through our experience with multilingual LLMs, one of the biggest nuisances has been lower-resource languages (with longer sequences) constantly throwing OOMs and forcing lower batches. Given the principle purpose of linear attention (long sequences), we wanted to study the drawbacks (for multilinguality) more directly. We will use some of the extra space allotted to add a better intro & conclusion sentence to Section 4 for better integration. We will also certainly introduce OneRuler more thoroughly, this is valuable feedback.