acceptodds
Under review as a conference paper at ICLR 2027

Multilinguality in Hybrid Attention LLMs

31 views

Abstract

In response to the growing demand for long sequences in agentic and reasoning use cases, many state-of-the-art LLMs combine multiple variants of attention to mitigate the quadratic complexity of traditional softmax attention. These hybrid attention LLMs aim to balance the strengths and limitations of full attention and alternatives based on recurrence. This work presents a first study of how hybrid attention impacts the multilinguality of LLMs. Beyond the impact on long sequences in poorly tokenized languages, our study is motivated by the possibility that the inductive biases of the recurrent state alter linguistic processing. Our interpretability analysis confirms this, showing that cross-lingual representations in hybrid models develop in patterns tied to the ordering of recurrent and full-attention layers. Across diverse models, we notably observe a pronounced spike in cross-lingual alignment around the first full-attention layer. These findings lead us to question the conventional ordering of attention layers. In distillation experiments on multilingual data, all alternative layer orderings outperform the standard throughout training, learning up to 2.5X faster. These stark, replicable results prompt our theory that multilingual models would benefit from starting with a full-attention layer rather than recurrent layers.

Primary area:
interpretability and explainable AI
Keywords:
linear attention, multilingual, cross-lingual transfer, recurrent attention, hybrid attention, representation learning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_YRFsOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates how hybrid attention architectures affect multilingual representations. Most importantly, the authors find that cross-lingual alignment emerges predominantly in full-attention layers: Across a range of architectures that alternate attention types, configurations placing full attention first or as the only attention method consistently exhibit the highest degree of multilinguality in their representations, which has direct implications for model design. The authors show, using alternative architectures and distillation experiments, that models that start out with full attention in the first layers outperform their counterparts in terms of multilingual alignment in every setting.

I enjoyed reading this paper a lot. The weaknesses I see are rather minor and don't have any critical criticism at this point. I recommend weak acceptance, but I'm happy to increase my score if the authors provide additional input to the questions below.

Critical strengths
  • In general: The paper identifies an interesting and relevant gap and investigates it thoroughly. An interesting hypothesis is derived from representational analyses (Findings 3, 4, 6) and then tested empricially, which gives the paper a very nice flow. Findings are highly actionable and could directly influence architectural choices in future multilingual models:

    • To my knowledge, this is the first study of how hybrid attention affects multilinguality, and it is timely given that many current open-weight model families adopt hybrid architectures as a standard
    • The fix that the authors propose to improve the degree of multilinguality is very simple and easy to adopt in principle, by a simple reordering of layers.
  • The paper is, for the most part, very clearly and concisely written:

    • The authors are refreshingly transparent about imperfect comparisons and argue convincingly that these comparisons are sufficient to support their claims, which substantially increased my confidence in the results.
    • Section 5.2 is the strongest part of the paper: it is clearly written and ties the individual findings together extremely well.
    • The main findings are clearly fleshed out (also just visually), which makes following the paper very easy
Critical weaknesses
  • The choice of SoftCKA is insufficiently motivated. The authors state "We find this metric for cross-lingual alignment much less noisy than alternatives we explore" without providing evidence. Which alternatives were considered, and what are their measurable disadvantages?
  • Section 6 did not fully convince me, mainly because the evaluation is thin: The students are only compared via KL to the teacher and cross-entropy on held-out text - I would have liked to see actual downstream results.
  • Writing/clarity: the placement and purpose of Section 4 were initially unclear to me. I assume it serves as a motivating evaluation for the remainder of the paper; this should be stated explicitly. In addition, OneRuler should be introduced briefly. A single sentence would spare readers from consulting the original paper.
Questions for authors
  • Which other mtrics for cross-lingual alignment did you test? Why did they perform more noisy?
  • Can you report downstream results for the distilled students, e.g. on the benchmarks from Appendix B, and OneRuler at longer contexts?
1 comment
AuthorsOpenReview ↗

Response to Reviewer YRFs

We are grateful for the reviewer’s positive feedback and hope to address some of their weaknesses/questions raised.

W1 & Q1

We acknowledge that L38-44 is a bit vague about why we liked SoftCKA more than alternatives. In large part, it was because of the amount of space the description of the metric already required ! We’ll expand on the intuition behind the metric and remove the claim that it was less noisy because that’s hard to quantify (though this is how we selected it originally, based on visualizations).

Obviously, representations happen at the token level (hidden states). CKA is a natural starting point because it measures shared structure relative to the variation within each representation, giving us a common scale for comparing alignment across layers. It also makes few assumptions about the “shape” of the representation space. The main problem with mean-pooling embeddings is that directions cancel out and not all tokens are equally informative. In contrast, SoftCKA compares every token state in one sentence with every token state in its translation, then measures the variation in those similarities after centering, relative to the corresponding within-language variation. This lets us examine token-level structure even when translations differ in length or word order. We interpret agreement with prior layerwise findings as a qualitative consistency check.

Alternatives we looked into were more basic: either mean-pooling or using the last hidden state and then using CKA or cosine similarity as measures of similarity. We also tried using logit lens and calculating KL-div. All seemed to suffer from significant information loss.

W2 & Q2

For further convincing, we also showed in response to reviewer SGZG that English-only training resulted in far smaller difference between standard-periodic and reverse-periodic (ours) layer ordering, which is new evidence.

But wrto evaluation: we stuck to training signals (on the validation set) primarily because we have not trained the student models to completion (would take significant compute). We believe the most important finding is what the training curves show: In all experiments, not only has our experimental ordering learned much faster (lower loss or KL) but it is still learning faster at the end of our trainings (steeper slope), implying that it would continue having lower loss or KL for significantly longer in training.

Upon your request, we did however try benchmarks that work with simple next-token completion setups, Belebele and Global-PIQA, and found that they did show statistics between the layer orderings and provide further evidence for our conclusions. We evaluated the standard-periodic and reverse-periodic students zero-shot on Belebele (covers 23 out of 26 languages) and Global PIQA (covers 22). Reverse periodic is more accurate on Belebele for both models (+3.0 points for Qwen3-4B, +2.4 for Granite) and on Global PIQA for Qwen3-4B (+2.3). On Global PIQA for Granite the difference (+1.1) is positive but not statistically significant. N.B. statistical significance is computed by bootstrapping questions, not with more random seeds. As with KL, the gain is larger for non-Latin scripts (Belebele +3.5 / +3.0 vs. +2.0 / +1.2 for Latin), and reverse periodic is ahead in 19 of 23 Belebele languages for each model. We were surprised such downstream evaluations would reflect the difference so early in training, so we thank the reviewer for this suggestion.

W3

We acknowledge that Section 4 does not fit super cleanly into the rest of the paper. However, we simply found that any discussion of the hybrid attention and multilinguality was incomplete without a discussion of token-inflated sequences in multilingual settings. Through our experience with multilingual LLMs, one of the biggest nuisances has been lower-resource languages (with longer sequences) constantly throwing OOMs and forcing lower batches. Given the principle purpose of linear attention (long sequences), we wanted to study the drawbacks (for multilinguality) more directly. We will use some of the extra space allotted to add a better intro & conclusion sentence to Section 4 for better integration. We will also certainly introduce OneRuler more thoroughly, this is valuable feedback.

Reviewer_CbjKOpenReview ↗
Rating
4(+3)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+2)
Read review
Paper summary

The paper compares hybrid attention LLMs and their full attention counterparts (to the extent such counterparts exist) in multilingual settings to investigate how hybrid attention models represent cross-lingual information. The comparison covers long-context retrieval performance, multilingual token alignment, and the effects of different attention layer arrangements in distillation experiments.

Critical strengths
  • Different layer arrangements in hybrid distillation experiments show interesting patterns, suggesting that languages with non-Latin scripts benefit from starting with a full attention layer rather than a recursive layer.
Critical weaknesses
  • For some experiments and findings, the paper provides only general descriptions of the results, occasionally supplemented by an example graph (e.g. showing results for a single model or language), but omits the full numerical results. For example, I could not find results supporting Finding 6, and similar issues occur throughout the paper to some extent. The lack of supporting results makes it difficult to verify the validity of the experiments and findings. If these discussions are intended as motivation only, their outcomes should not be presented as findings, and if they represent actual experimental findings, the paper should provide results to support them.

  • Given the AI Use Statement, it is somewhat unclear what the authors’ actual contribution to the work is.

Questions for authors

None

6 comments
AuthorsOpenReview ↗

Ensuring Reviewer can see Responses

Dear Reviewer CbjK, thanks for your message saying you couldn't see the above responses. The AC kindly notified us 13 Nov 2026 that our responses were accidentally not made public to you, which we thought we changed 14 Nov 2026 (visibility says Everyone on our end).

If you still cannot see them (https://openreview.net/forum?id=xT5agIyCVG&noteId=dEwHAHObTa and https://openreview.net/forum?id=xT5agIyCVG&noteId=KGa4nqeMvx), we will repost them quickly !

Thanks, 21932 authors

AuthorsOpenReview ↗

Response to Reviewer CbjK

(we understood why the visibility to the reviewer was not working and fixed it)

We hope to address some of the concerns raised below:

W1

This is very constructive feedback. The paper choices we made in this regard were trying to prioritize readability and digestibility over extensive detail into the specific mechanics of how we came to these findings. We provided concise summaries of findings in the very limited main body, while providing reproducibility via details in the long appendix and the anonymized code uploaded (which we discuss in Reproducibility Statement). However, we will work to do this better and take advantage of the extra page for the next versions:

  1. Finding 6 was our most abridged, as you note. Our initial thought was that, since this fit the least well with the other findings yet was the most complex to explain, we decided to keep it very high-level. But we appreciate the feedback that it needs more substantiation, so we created an Appendix H to discuss more details and provide a markdown version in the next comment.
  2. For Findings 1-5,7, we argue that sufficient evidence is provided:
  • Finding 1 has the Table 1, which shows the surprising result that the gap in long-context retrieval actually gets reduced in long contexts for hybrid models. The results are aggregated for simplicity, and because there is noise at the individual-language level. However, we will dump in the (new) Appendix G the full per-language, per-length results.
  • Finding 2 is directly supported by Figure 2, but also indirectly by Figures 3 (for Qwen3.5 especially).
  • Finding 3, perhaps the most important, is supported by the large Figure 3 (and indirectly Figure 4 and E.1). We will point to the these additional visualizations.
  • Finding 4 is supported by Figures 3, 4, and E.1. We will point to these other figures more clearly.
  • Finding 5 is supported by Figures 3 and F.1. We looked at two SWA models (Tiny Aya and Olmo-3), both where no such pattern emerges.
  • Finding 7 is the most well-supported, given that we committed an entire section to it.

In general, with your feedback, we have worked on many small edits to Section 5 and a few to Section 4, that will be visible in the Rebuttal Revision (that we will add soon with improvements based on feedback from the other reviewers). We have also added two Appendices G and H. We are happy to discuss further.

W2

There seems to have been a grave misunderstanding about how we filled out the AI Use Statement, which was meant to be fully truthful and transparent. Yes, AI tools were used during many different parts of the project. However, all the experimental, intellectual, writing, and even most of the coding contributions are the authors’. The original project idea was from the authors, with motivation based on personal intuition and experience in the field. Much of the code was AI written, but with heavy involvement by authors. The writing was fully human written, with AI assistance for low-level language, grammar, and edits as well as high-level feedback provided along the way.

AuthorsOpenReview ↗

Additional numbers for Finding 6 in markdown

Finding 6 details

ModelFull attention layers0-toks average(min, max)2-toks average(min, max)5-toks average(min, max)10-toks average(min, max)
Qwen3480.467(0.122, 0.673)0.526(0.191, 0.743)0.583(0.203, 0.773)0.649(0.231, 0.831)
Qwen3.5100.634(0.549, 0.767)0.662(0.570, 0.792)0.695(0.600, 0.817)0.745(0.648, 0.861)
Olmo-3320.302(0.198, 0.419)0.358(0.256, 0.464)0.412(0.297, 0.574)0.479(0.354, 0.642)
Olmo-Hybrid80.448(0.298, 0.633)0.467(0.315, 0.649)0.493(0.334, 0.672)0.540(0.364, 0.714)

Each value represents the proportion of the normalized attention matrix A that is devoted to tokens within {0,2,5,10} tokens from the current. In the A matrix, this is top {1,3,6,11} diagonal(s) values in the lower-trianguler (masked) matrix. This is averaged across all full attention layers (which is why there are fewer in hybrid models) and 13 languages using FLoRes data. We provide the min and max here, even though the range changes across languages (and there are some outliers). Typically non-English languages devote a smaller proportion to closer tokens. For this reason, we are not including these values in the Appendix table (also doesn’t fit).

What we see here is that the full attention layers in hybrid models consistently weigh local tokens more. This is regardless of what we define as “local” (the current token or <10 tokens away).

The implementation is in calculate_attention_distance.py and attn_dist_analysis.ipynb in the anonymized code repository.

Reviewer_SGZGOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how hybrid attention impacts the multilinguality of LLMs. The analysis shows that cross-lingual representations in hybrid models develop in patterns tied to the ordering of recurrent and full attention layers. Hybrid models interleave recurrent layers with full softmax attention layers to handle long contexts needed for agentic and reasoning use cases. The authors compare four hybrid models (Qwen3.5-35B, OLMo-Hybrid, Ring-mini-linear-2.0, Granite-4.0-H-Micro) with their closest non-hybrid counterparts. The authors also introduce SoftCKA, a cross-lingual alignment metric that compares translated sentences without needing one-to-one token matching. They distill Qwen3-4B into Gated DeltaNet hybrids with 25% full attention, training for 1B tokens on multilingual FineWeb2 and varying only where the full-attention layers go. The baseline used puts full attention at the end of each block. They test starting with a full-attention layer and that performs better, reaching the baseline's final loss with about 35–40% of the tokens. The largest gains are in non-Latin-script languages such as Telugu, Tamil, Burmese and Khmer. This prompts their theory that multilingual models would benefit from starting with a full-attention layer rather than recurrent layers.

Critical strengths
  1. Hybrid attention is now standard in leading open models (Qwen3.5, Nemotron, MiniMax, Kimi), and this is the first work to ask how it affects multilinguality. The layer-ordering experiment shows useful results on multilingual datasets. The share of full attention is fixed at 25% and only its placement changes. The result is replicated with resampled data and a second teacher model (Granite-4.1-3B), and it holds in 23–25 of 26 languages. The paper shows large gains in non-Latin-script languages.

  2. The proposed SoftCKA metric is well-designed and useful to compare different architectures. SoftCKA compares every token in one language with every token in the other using kernel similarity, which acts as a soft matching. It then normalizes by the within-language similarity and aggregates over the corpus, reducing noise from individual sentences. Most metrics use mean pooling or last token, that can cause information loss.

Critical weaknesses

In the experiment, the student model is built by converting the teacher's layers into recurrent layers. These are partly reinitialized and must be relearned, while the layers kept as full attention are copied directly from the teacher and work from the start. The winning configuration keep layer 1 as the teacher's original full-attention layer, whereas the standard layout converts it. Since every subsequent layer depends on layer 1, it's possible that relearning the first layer is more difficult. It's difficult to separate the effects of multi-linguality from distillation. An English-only control run would help separate the two effects.

Questions for authors
  1. Do the gains hold with longer training, larger teachers, or a recurrent type other than Gated DeltaNet (e.g. Mamba-2)?
  2. Do the distilled students show differences on downstream multilingual tasks (e.g. Belebele, Global PIQA)?
2 comments
AuthorsOpenReview ↗

Response to Reviewer SGZG

We would like to thank the reviewers for their review and their positive feedback on our work. We apologize for taking so long to respond, as we were running 2 requested experiments which each took 99 hours on L40S GPUs. We believe this has strengthened our work and we’re working on a Rebuttal Revision incorporating this new evidence:

W1

As requested, we additionally ran an English-only control to disentangle the effects of distillation versus multilinguality. We believe these experiments provide a 3rd source of evidence to believe that our results are in large part related to multilinguality:

  1. As mentioned L454-456, Li et al., 2026 did very comparable distillation training and layer selection using the same metrics and never concluded that a first-layer full attention was a good idea (finding, if anything, the opposite).
  2. As mentioned very briefly L452-454, English and other high-resource, Latin-script languages were the ones where the gap between “standard-periodic” and our alternatives were the smallest. Our alternatives had a much bigger impact on other languages, especially non-Latin scripts; reverse periodic lowers KL by 12% on Latin-script languages vs. 41% on non-Latin scripts for Qwen3-4B.
  3. (new) We ran two training runs on English-only training data and English valid+test data drawn from our multilingual training valid+test sets. We distilled Qwen3-4B with standard and reverse periodic on 1B English-only tokens, with all other settings unchanged and one seed per ordering. Full results on the test set are displayed in the markdown table below. Yes, the “reverse periodic” (ours, RP) does learn faster than “standard periodic” (SP) and ends with a lower English KL and CE, which means some of the difference between reverse-periodic and standard-periodic in our multilingual experiments may be explained by a preference of distillation to this layer ordering. However, this effect is much smaller than on other languages: English-only training lowers KL by 6%, vs. 26–28% on 25 non-English languages of our multilingual runs. A pure distillation preference should give English the same gain either way, yet English gains more in the multilingual runs (11–12%; one English-only seed, so only suggestive), though far less than other languages (#2 above). In clearest terms, RP’s advantage on multilingual data far exceeds that on Eng data, no matter the metric. As a result, most of this gap cannot be explained as a distillation artifact.
Training runSplit of test setSP KLRP KLΔKL%SP CE-gapRP CE-gapΔCE gap%
Eng-only s0Eng-only0.1200.1130.0076.0%0.0840.0780.0078.1%
Multiling s0Eng-only0.1600.1400.02012.2%0.1190.1060.01411.3%
Multiling s1Eng-only0.1600.1420.01811.3%0.1200.1080.0129.9%
Multiling s0non-Eng0.1410.1010.04028.2%0.0690.0420.02739.0%
Multiling s1non-Eng0.1370.1010.03626.2%0.0660.0420.02436.0%

We plan on adding this further evidence and be more clear about reasons #1 and #2 above, though this table and the Eng-only training curves may go in the appendix for space.

Q1

These are unfortunately not questions we can answer with our compute budget, even if we did experiment with a very different 2nd teacher, Granite-4.1-3B.

Q2

We stuck to training signals (on the valid set) primarily because we have not trained the student models to completion (would take significant compute). We believe the most important finding is what the training curves show: In all experiments, not only has our experimental ordering learned much faster (lower loss or KL) but its lead holds to the end of training (stable RP/SP ratio), implying that it would continue having lower loss or KL for significantly longer in training.

Upon your request, we did however try benchmarks that work with simple next-token completion setups, Belebele and Global-PIQA, and found that they did show statistical difference between the layer orderings and provide further evidence for our conclusions. We evaluated the standard-periodic and reverse-periodic students zero-shot on Belebele (covers 23 out of 26 languages) and Global PIQA (covers 22). Reverse periodic is more accurate on Belebele for both models (+3.0 acc for Qwen, +2.4 for Granite) and on Global PIQA for Qwen3-4B (+2.3). On Global PIQA for Granite the difference (+1.1) is positive but not stat sig. N.B. significance is computed by bootstrapping questions, not more random seeds. As with KL, the gain is larger for non-Latin scripts (Belebele +3.5 / +3.0 vs. +2.0 / +1.2 for Latin), and reverse periodic is ahead in 19 of 23 Belebele languages for each model. We did not expect such downstream evals would reflect the difference so early in training, so we thank the reviewer for this suggestion. We will add this to the paper.

Thank you for the follow up. I have edited the score based on the response above.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to Reviewer CbjK

We hope to address some of the concerns raised below:

W1

This is very constructive feedback. The paper choices we made in this regard were trying to prioritize readability and digestibility over extensive detail into the specific mechanics of how we came to these findings. We provided concise summaries of findings in the very limited main body, while providing reproducibility via details in the long appendix and the anonymized code uploaded (which we discuss in Reproducibility Statement). However, we will work to do this better and take advantage of the extra page for the next versions:

  1. Finding 6 was our most abridged, as you note. Our initial thought was that, since this fit the least well with the other findings yet was the most complex to explain, we decided to keep it very high-level. But we appreciate the feedback that it needs more substantiation, so we created an Appendix H to discuss more details and provide a markdown version in the next comment.
  2. For Findings 1-5,7, we argue that sufficient evidence is provided:
  • Finding 1 has the Table 1, which shows the surprising result that the gap in long-context retrieval actually gets reduced in long contexts for hybrid models. The results are aggregated for simplicity, and because there is noise at the individual-language level. However, we will dump in the (new) Appendix G the full per-language, per-length results.
  • Finding 2 is directly supported by Figure 2, but also indirectly by Figures 3 (for Qwen3.5 especially).
  • Finding 3, perhaps the most important, is supported by the large Figure 3 (and indirectly Figure 4 and E.1). We will point to the these additional visualizations.
  • Finding 4 is supported by Figures 3, 4, and E.1. We will point to these other figures more clearly.
  • Finding 5 is supported by Figures 3 and F.1. We looked at two SWA models (Tiny Aya and Olmo-3), both where no such pattern emerges.
  • Finding 7 is the most well-supported, given that we committed an entire section to it.

In general, with your feedback, we have worked on many small edits to Section 5 and a few to Section 4, that will be visible in the Rebuttal Revision (that we will add soon with improvements based on feedback from the other reviewers). We have also added two Appendices G and H. We are happy to discuss further.

W2

There seems to have been a grave misunderstanding about how we filled out the AI Use Statement, which was meant to be fully truthful and transparent. Yes, AI tools were used during many different parts of the project. However, all the experimental, intellectual, writing, and even most of the coding contributions are the authors’. The original project idea was from the authors, with motivation based on personal intuition and experience in the field. Much of the code was AI written, but with heavy involvement by authors. The writing was fully human written, with AI assistance for low-level language, grammar, and edits as well as high-level feedback provided along the way.

AuthorsOpenReview ↗

Additional numbers for Finding 6 in markdown

Finding 6 details

ModelFull attention layers0-toks average(min, max)2-toks average(min, max)5-toks average(min, max)10-toks average(min, max)
Qwen3480.467(0.122, 0.673)0.526(0.191, 0.743)0.583(0.203, 0.773)0.649(0.231, 0.831)
Qwen3.5100.634(0.549, 0.767)0.662(0.570, 0.792)0.695(0.600, 0.817)0.745(0.648, 0.861)
Olmo-3320.302(0.198, 0.419)0.358(0.256, 0.464)0.412(0.297, 0.574)0.479(0.354, 0.642)
Olmo-Hybrid80.448(0.298, 0.633)0.467(0.315, 0.649)0.493(0.334, 0.672)0.540(0.364, 0.714)

Each value represents the proportion of the normalized attention matrix A that is devoted to tokens within {0,2,5,10} tokens from the current. In the A matrix, this is top {1,3,6,11} diagonal(s) values in the lower-trianguler (masked) matrix. This is averaged across all full attention layers (which is why there are fewer in hybrid models) and 13 languages using FLoRes data. We provide the min and max here, even though the range changes across languages (and there are some outliers). Typically non-English languages devote a smaller proportion to closer tokens. For this reason, we are not including these values in the Appendix table (also doesn’t fit).

What we see here is that the full attention layers in hybrid models consistently weigh local tokens more. This is regardless of what we define as “local” (the current token or <10 tokens away).

The implementation is in calculate_attention_distance.py and attn_dist_analysis.ipynb in the anonymized code repository.

Dear reviewers,

As we only have a few more days to go before the discussion window closes, please go through the author responses and acknowledge them. You may highlight aspects of your initial review comments that have been satisfactorily addressed by the responses, as well as those which require further clarification.

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.