acceptodds
Under review as a conference paper at ICLR 2027

Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations

Abstract

Softmax attention is the cornerstone of modern large language models. However, its memory requirements scale linearly with sequence length, and its compute requirements scale quadratically. Linear recurrent models, such as linear attention and state space models, have become widely studied as alternatives to softmax attention due to their linear compute and constant memory requirements. While these sub-quadratic token mixing methods (mixers) achieve promising efficiency gains and competitive results on a wide range of benchmarks, current linear recurrent models still lag behind on tasks that require long-context retrieval or in-context learning. A growing body of work studies hybrid architectures that attempt to mitigate these trade-offs by statically interleaving or merging attention and recurrent blocks. In this work, we explore a new axis of developing hybrid models: across the token sequence. We propose \antelope, a hybrid model that can flexibly switch between different mixers throughout a sequence, e.g., quadratic attention for rich context utilization and linear recurrences to reduce quadratic computation. Oryx ties at least 90% of its parameters across mixers, enabling attention and recurrent modes to operate over shared internal representations. We validate our design with Mamba-2 and Gated DeltaNet variants, up to 1.4B models. Under fixed token budgets and a mixed-training strategy, Oryx achieves comparable or better performance than its single-mixer baselines. At the 1.4B scale, all instances of Oryx outperform their respective baselines by at least 0.7 percentage points on averaged language modeling tasks. On retrieval tasks, Oryx achieves performance comparable to the Transformer baseline even when processing only a tiny fraction (<10%) of the tokens in the attention mode. These results suggest that attention and linear recurrent models can share internal representations, and motivate sequence-axis hybridization as a promising direction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid architecture, attention, linear attention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_khdbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

ORYX enables switching between softmax attention and recurrent mixers within a sequence through shared key-value projections and chunk-level mixed-mode training. Experiments with Mamba-2 and Gated DeltaNet demonstrate competitive standalone performance and cross-mode retrieval.

Critical strengths

Sequence-axis hybridization is an interesting direction. Validation with two recurrent architectures and the chunk-level training ablation provide useful evidence that switching compatibility can be learned.

Critical weaknesses

The comparisons mainly involve single-mixer baselines. Although parameters and training tokens are matched, the results do not establish whether sequence-axis switching offers a better performance–cost trade-off than a fixed hybrid architecture.

The practical efficiency benefit remains unclear. Appendix C analyzes simplified mixer FLOPs, while switching requires maintaining both the full KV cache and recurrent state. End-to-end latency, throughput, and memory measurements are needed to assess the actual savings.

Retrieval evaluation is limited to 2K contexts, and compatibility is uneven across tasks. On NIAH-3, ORYX-TM drops from 81.6 in attention-only mode to 58.0 with linear context processing and attention-based querying. This leaves an important limitation of cross-mode retrieval insufficiently explained

I lean toward weak rejection. The switching mechanism is interesting, but the current evidence does not yet establish its practical advantage over existing hybrid designs or adequately characterize its retrieval limitations.

Questions for authors

Could you compare with a fixed inter-layer hybrid using the same data and a matched training-compute budget? Since ORYX uses a higher peak learning rate, were the baselines given a comparable tuning budget?

At comparable retrieval accuracy, what are the measured prefill latency, decoding throughput, and peak memory relative to optimized baselines? Processing fewer tokens with attention does not by itself quantify the end-to-end benefit.

What explains the NIAH-3 drop from 81.6 to 58.0 for ORYX-TM, whereas ORYX-TG drops only from 99.4 to 95.8? Does the failure depend on the needle position or switching location?

Do the retrieval results remain stable beyond 2K contexts and when varying the fixed 97.5% context–prompt split used for real-world tasks? This would clarify how sensitive the gains are to the manually chosen switching schedule.

2 comments
AuthorsOpenReview ↗

Response to Reviewer khdb 1/

We thank the reviewer for finding our exploration into sequence-axis hybridization interesting and finding our empirical validation useful. The joint response presents a comparison with fixed inter-layer hybrid models, discusses the NIAH-3 drop for Oryx-TM, and efficiency results. We summarize them here and also address the additional questions below.

compare with a fixed inter-layer hybrid

We trained Mamba-2 and GDN 1.4B hybrid models at a 3:1 recurrent to attention ratio using the original baseline setup, matching Oryx’s parameter count and token budget. On our language modeling and commonsense benchmarks, both Oryx models outperform their respective hybrid counterparts on average performance when using only the linear mode and perform comparably when using only the attention mode.

On real-world retrieval tasks, the cross-mode (linear-to-attention switch) Oryx performs similarly to the hybrid counterparts except on the FDA task; we expand on this discrepancy in the joint response. When averaging across the other five tasks (SWDE, SQuAD, TQA, NQ, and DROP), Oryx-TM achieves a 41.4 average accuracy to 41.1 for the Mamba-2 hybrid, and Oryx-TG achieves a 40.1 average accuracy to 40.8 for the GDN hybrid. On synthetic retrieval tasks, Oryx performs comparably to the hybrid baselines except for Oryx-TM on NIAH-3, which we address below.

Beyond the strong performance of mode-switching, we would like to emphasize that the shared representation design and the mixer flexibility it entails do not have to replace, but can be complementary to, current hybrid design. We are excited to explore how we can interleave Oryx-like layers with linear layers to enable architectures that are similar to conventional attention-linear hybrids but have flexible mixer switching.

Please see our joint response (2/) for the complete results and discussion.

were the baselines given a comparable tuning budget?

The hyperparameters used to train the original Oryx models were not exhaustively tuned. The 2×2\times learning rate was a heuristic motivated by its disjoint query parameters. The hyperparameters for the baselines were established from prior literature [1, 2]. We trained additional Transformer and Mamba-2 baselines with the same learning rate as our Oryx setup. We find that Oryx-TM in attention-mode still outperforms the higher-lr Transformer baseline on average accuracy 56.3 to 55.8. The higher-lr Mamba-2 baseline slightly outperforms the recurrent-mode of Oryx-TM 56.8 to 56.6 on average accuracy, but Oryx-TM retains a better perplexity on Lambada at 10.5 compared to 10.9. We highlight that Oryx-TM's recurrent mode is still competitive despite utilizing only half the standalone Mamba-2 baseline’s recurrent state size.

ModelLAMB. ppl ↓LAMB. accHellaS. accₙPIQAARC-EARC-C accₙWinoGr.OBQAAvg. ↑
Transformer (2×2\times lr)10.950.161.173.674.739.160.631.655.8
Mamba-2 (2×2\times lr)10.949.162.575.376.141.859.533.456.8
ORYX-TM, attention11.150.261.375.075.742.058.031.656.3
ORYX-TM, Mamba-210.550.462.175.275.343.658.531.256.6

practical efficiency benefit remains unclear.

We extend our FLOPs analysis to our cross-mode real-world retrieval setup and find that mode switching reduces the mixer FLOPs by approximately 70.5% relative to full attention and 32.6% relative to a 3:1 Mamba-2 hybrid under the assumptions detailed in joint response (3/3). Oryx’s additional linear recurrent state introduces a small increase in memory requirements, equivalent to 3.125% of the full KV cache at 2K context length. As latency and throughput heavily depend on kernel implementations and optimizations, we leave such benchmarks for future work.

What explains the NIAH-3 drop from 81.6 to 58.0 for ORYX-TM

We hypothesize that Oryx-TM's NIAH-3 degradation stems from Mamba-2’s weakness in processing the uuid needle. Oryx-TM performs well on NIAH-2, which retains the same setup as NIAH-3 except needle type, and regular Mamba-2 models suffer on the same task, as shown in the original Gated DeltaNet’s table 2 [4]. Thus, weaker representations from prior layers could affect the representations used by attention during retrieval.

AuthorsOpenReview ↗

Response to Reviewer khdb 2/2

Do the retrieval results remain stable beyond 2K contexts

Our retrieval evaluations use the 2K context length at which all models were pretrained with, a setting also commonly used in architecture-based papers: the Mamba series use 2K context length and Gated DeltaNet evaluates real-world retrieval on 2K as well [1, 2, 3, 4]. Our setting already exposes substantial differences between the retrieval capabilities of pure linear models, hybrid models, Transformers, and the cross-mode vs recurrent-only modes of Oryx. Evaluation beyond 2K would additionally test length extrapolation, where our RoPE positional encoding used for our Transformer baselines and attention component of Oryx would affect performance, even without mode-switching [5, 6].

We hope our new results and clarifications have addressed the reviewer’s concerns and are happy to answer any follow-up questions.

[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces, https://arxiv.org/abs/2312.00752

[2] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, https://arxiv.org/abs/2405.21060

[3] Mamba-3: Improved Sequence Modeling using State Space Principles, https://arxiv.org/abs/2603.15569

[4] Gated Delta Networks: Improving Mamba2 with Delta Rule, https://arxiv.org/abs/2412.06464

[5] Extending Context Window of Large Language Models via Positional Interpolation, https://arxiv.org/abs/2306.15595

[6] Rope to Nope and Back Again: A New Hybrid Attention Strategy, https://arxiv.org/abs/2501.18795

Reviewer_JksnOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes hybrid-mixer models that can switch between linear RNNs and softmax attention at the token level to reduce the quadratic computational complexity of attention. The authors propose sharing only the key and value projection weights between mixers and training the model by randomly assigning chunks to different mixers, either linear RNNs or softmax attention.

On short LM benchmarks and retrieval tasks, Oryx achieves competitive performance in single-mixer mode compared to models from the same family. In multi-mixer mode, where the context is processed using a linear mixer and answers are generated using attention, Oryx with Gated DeltaNet outperforms other baselines on synthetic retrieval tasks and performs on par with transformer baselines on real-world retrieval tasks.

Critical strengths

The paper provides potentially useful insights into hybrid-mixer models and their training. The authors introduce random chunk assignment for training such hybrids and show that it makes the models robust to mode switching. They also demonstrate that sharing query projections between linear RNNs and attention mixers can negatively affect performance.

Critical weaknesses
  • The paper claims to introduce a new direction in hybrid architectures that operate across the sequence dimension. However, similar approaches already exist, such as TransMamba and NAtS-L. TransMamba (Li et al., 2026) is mentioned in the related work, but NAtS-L (Deng et al., 2026) is not discussed.

  • One of the main benefits of such hybrid architectures is the ability to switch between mixers throughout generation. However, the paper does not extensively evaluate the models in this setting. In the main experiments, switching between mixers occurs only once. Multiple switches are evaluated only in terms of perplexity (without generation) in the appendix. Moreover, the paper does not explore learned routing, which has already been introduced in related approaches such as NAtS-L.

  • The proposed method is similar to the existing NAtS-L approach, but the authors do not provide any comparison with it. Several important questions remain unanswered: (1) How does random chunk assignment compare to learned routing? (2) How does NAtS-L perform compared to Oryx when NAtS-L has access to the full KV cache? (3) Could learned routing improve Oryx? Furthermore, NAtS-L also addresses the problem of growing memory usage, while Oryx focuses only on reducing computational cost. This makes Oryx less appealing from an overall efficiency perspective.

  • In Section 4.1.1, the authors write: “Notably, despite these tasks being evaluated ‘out-of-distribution’ — the mode switching training does not explicitly train the entire model with all chunks assigned to the same mixer — the models are still able to generalize to this edge case.” However, the reported results do not provide strong evidence for this claim, as the evaluated LM benchmarks contain mostly short sequences (fewer than 128 tokens using the GPT-2 tokenizer employed in the paper).

Questions for authors

NA

7 comments
AuthorsOpenReview ↗

Response to Reviewer Jksn 1/

We thank the reviewer for appreciating the insights from our experiments and ablations. As discussed in the joint response, our core contribution is the exploration and understanding of design choices that enable shared representations and support flexible switching between mixers. We address the reviewer-specific questions below.

similar approaches already exist, such as TransMamba and NAtS-L.

We thank the reviewer for pointing us to NAtS-L. We were not aware of this concurrent work during our related work research and will add it to our paper.

Our submission briefly discusses the differences between TransMamba and Oryx, which we will expand on. TransMamba introduces a memory converter to enable the switch between attention and recurrence; however, the switch occurs only once per layer from attention to Mamba-2. Oryx supports multiple switching in both directions by maintaining both an attention cache and recurrent state. Our separate-query and chunk-training ablations identify the design components which enable this functionality for both Mamba-2 and Gated DeltaNet.

NAtS-L is a sequence-level hybrid model that deploys a learnable router which assigns chunks of key-values to either attention or recurrence, and its output combines the readouts of the sparse attention cache and sparse linear recurrent state. Oryx’s output instead depends only on the selected mixer mechanism. Oryx also retains the full KV cache and continuously updates the recurrent state, enabling subsequent attention modes to utilize the full sequence history, including tokens processed by the recurrent mode. Furthermore, due to the differences in objective, Oryx is able to switch on a token-level during inference, while NAtS-L operates on a chunk basis and requires the rerolling of the non-selected mechanism’s memory.

How does random chunk assignment compare to learned routing?

How does NAtS-L perform compared to Oryx when NAtS-L has access to the full KV cache?

Could learned routing improve Oryx?

Our mixed-chunk training trains the shared representations across various mixer selection patterns without constraining future inference to certain mode assignments. Our experiments establish that both mixers can operate in a standalone manner and can operate jointly within a sequence. Learned routing is a natural extension of our framework, and could be used to dynamically allocate compute on a per-layer or per-model granularity. We discuss routing in our submission’s discussion and conclusion section.

A NAtS-L variant which retains the full KV history would change the underlying sparsity of the attention mode while retaining the combined readout. We believe these additional routing ablations fall outside the scope of our submission, which focuses on enabling compatible representations for standalone and multi-mixer operations. Our current model design establishes that attention and recurrent mechanisms can share compatible representations on which routing can operate upon.

while Oryx focuses only on reducing computational cost. This makes Oryx less appealing from an overall efficiency perspective.

Oryx was designed to enable flexible computation allocation. While Oryx does require maintaining both the full sequence’s KV cache and the recurrent hidden state (we discuss the memory and compute tradeoffs in our shared response), the current design provides a basis for further applications. For instance, a learned router could be introduced during pretraining or through subsequent finetuning, or a model trained with both mixers can be deployed in a standalone mode.

Our ablations provide the basis for this second application. While chunked mixed-mode training is important for mode-switching, sequence-level mixing already enables effective standalone performance. In particular, the attention-mode validation perplexity is nearly identical at 15.09 vs 15.08 for sequence-level and chunk-level assignment respectively. This suggests using the recurrent mode to reduce pretraining costs while deploying the resulting model in attention-only mode.

In additional preliminary experiments, we trained 350M Oryx-TM models with QK normalization, no gate and pre-output projection normalization, and no convolution (replaced with Mamba-3 style BC bias [1]) with sequence-level assignment. Compared to a Transformer baseline with QKNorm, which achieved a validation perplexity of 15.58, our Oryx variant achieved an attention-mode perplexity of 15.17. Our results suggest this to be an interesting area for further exploration.

AuthorsOpenReview ↗

Response to Reviewer Jksn 2/2

One of the main benefits of such hybrid architectures is the ability to switch between mixers throughout generation. However, the paper does not extensively evaluate the models in this setting.

Our submission evaluates different switch positions and switch counts in both directions through per-token perplexity, including switch locations that do not align with training chunk boundaries (Figures 8-10). Across our tested switches, we observe little to no perplexity degradation relative to the no-switch baselines, supporting compatibility across mixer transitions. By evaluating the perplexity under various switching settings, we isolate the effects of the mixer transition on language modeling in a controlled setting.

We will move these plots from the appendix into the main paper to make them more prominent. These experiments complement our retrieval evaluations, which assess switch capabilities under generation.

However, the reported results do not provide strong evidence for this claim, as the evaluated LM benchmarks contain mostly short sequences

Our single-mode retrieval results and token-level perplexity measurements for no-switch also evaluate standalone performance at 2K context length. These results extend beyond the short language-modeling and commonsense benchmarks highlighted by the reviewer. We will revise the placements and clarify the original wording to distinguish our longer-context evaluations from shorter tasks. We hope our new results and clarifications have addressed the reviewer’s concerns and are happy to answer any follow-up questions.

[1] Mamba-3: Improved Sequence Modeling using State Space Principles, https://arxiv.org/abs/2603.15569

Thank you for your response. As I mentioned, the main benefit of such a model is the ability to switch between mixers multiple times during generation. In practice, switching could happen much more frequently than in your experiments. Moreover, perplexity is usually not enough to understand how models perform on downstream tasks, and it does not address my concern about generation performance.

Since you do not have a mechanism to decide when to switch, you could at least run evaluations, on the benchmarks used in the paper, with Oryx switching between RNNs and attention modes every K tokens, where K is small (e.g., 1, 2, 4, 16, 32, etc.), and show how performance changes as the switching frequency increases.

For now, I will keep my score unchanged.

Reviewer_AgqDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ORYX, an architecture that switches between softmax attention and recurrent mixers within a sequence using largely shared parameters. Shared key/value projections, separate queries, and chunked mixed-mode training enable switching between attention and either Mamba-2 or Gated DeltaNet. Experiments show competitive performance against single-mixer baselines.

Critical strengths
  1. The shared-representation design is interesting, and the experiments demonstrate that both mixers can remain effective within the same model.

  2. Supporting repeated, bidirectional switching with both Mamba-2 and Gated DeltaNet strengthens the contribution. The ablations on separate queries and chunked training are useful.

Critical weaknesses
  1. The most important missing comparison is a conventional attention–linear hybrid. The paper motivates ORYX relative to inter-layer and intra-layer hybrids, but evaluates only against pure Transformer, Mamba-2, and GDN baselines. A simple inter-layer hybrid is a necessary baseline here: without it, the experiments do not establish the benefit of sequence-axis switching over a fixed mixture.

  2. The practical efficiency trade-off is also unclear. ORYX retains full KV caches and continuously updates recurrent states, whereas a conventional inter-layer hybrid requires KV caches only in its attention layers. The fraction of tokens using attention therefore provides insufficient evidence of an efficiency advantage. Measured prefill/decode latency and memory usage are missing.

Questions for authors
  1. How does ORYX compare with a simple inter-layer attention–GDN or attention–Mamba hybrid under comparable training budgets?

  2. What are the measured latency and memory costs of mixed execution, including maintenance of both states?

  3. What explains the ORYX-TM NIAH-3 degradation, and how does it change with the switch location or attention budget?

2 comments
AuthorsOpenReview ↗

Response to Reviewer AgqD

We thank the reviewer for recognizing the shared-representation design as an interesting direction and the experiments and ablations as useful. Our joint response addresses the conventional attention-linear hybrid comparisons, Oryx-TM NIAH-3 degradation, and practical efficiency, and we summarize them below.

missing comparison is a conventional attention–linear hybrid.

We trained Mamba-2 and GDN 1.4B hybrid models at a 3:1 recurrent to attention ratio using the original baseline setup, matching Oryx’s parameter count and token budget. On our language modeling and commonsense benchmarks, both Oryx models outperform their respective hybrid counterparts on average performance when using only the linear mode and perform comparably when using only the attention mode.

On real-world retrieval tasks, the cross-mode (linear-to-attention switch) Oryx performs similarly to the hybrid counterparts except on the FDA task; we expand on this discrepancy in the joint response. When averaging across the other five tasks (SWDE, SQuAD, TQA, NQ, and DROP), Oryx-TM achieves a 41.4 average accuracy to 41.1 for the Mamba-2 hybrid, and Oryx-TG achieves a 40.1 average accuracy to 40.8 for the GDN hybrid. On synthetic retrieval tasks, Oryx performs comparably to the hybrid baselines except for Oryx-TM on NIAH-3, which we address below.

Beyond the strong performance of mode-switching, we would like to emphasize that the shared representation design and the mixer flexibility it entails do not have to replace, but can be complementary to, current hybrid design. We are excited to explore how we can interleave Oryx-like layers with linear layers to enable architectures that are similar to conventional attention-linear hybrids but have flexible mixer switching.

Please see our joint response (2/) for the complete results and discussion.

practical efficiency trade-off is also unclear

We extend our FLOPs analysis to our cross-mode real-world retrieval setup and find that mode switching reduces the mixer FLOPs by approximately 70.5% relative to full attention and 32.6% relative to a 3:1 Mamba-2 hybrid under the assumptions detailed in joint response (3/3). Oryx’s additional linear recurrent state introduces a small increase in memory requirements, equivalent to 3.125% of the full KV cache at 2K context length. As latency and throughput heavily depend on kernel implementations and optimizations, we leave such benchmarks for future work.

What explains the ORYX-TM NIAH-3 degradation

We hypothesize that Oryx-TM's NIAH-3 degradation stems from Mamba-2’s weakness in processing the uuid needle. Oryx-TM performs well on NIAH-2, which retains the same setup as NIAH-3 except needle type, and regular Mamba-2 models suffer on the same task, as shown in the original Gated DeltaNet’s table 2 [1]. Thus, weaker representations from prior layers could affect the representations used by attention during retrieval.

Separately, on our real-world retrieval tasks, we find that ranging the switch position from 75% to 97.5% (our reported setting) does not significantly change overall accuracy. Average accuracy changes by at most 0.3 percentage points for Oryx-TM and 0.3 percentage points for Oryx-TG.

ModelLinear input prefixSWDESQuADFDATQANQDROPAvg.
Oryx-TM75%50.741.656.064.127.822.343.7
-80%51.141.855.664.028.222.043.8
-90%51.041.756.063.728.422.643.9
-95%50.741.456.664.328.322.544.0
-97.5%50.940.956.464.928.222.443.9
Oryx-TG75%44.041.447.564.827.721.441.1
-80%43.841.447.265.128.121.141.1
-90%44.840.947.164.527.721.541.1
-95%45.241.048.065.227.821.541.4
-97.5%46.141.147.964.128.020.941.4

We hope our new results and clarifications have addressed the reviewer’s concerns and are happy to answer any follow-up questions.

[1] Gated Delta Networks: Improving Mamba2 with Delta Rule, https://arxiv.org/abs/2412.06464

To further explore the switching capabilities of our shared-representation Oryx models and Oryx-TM's NIAH-3 performance, we evaluate our 1.4B models under the setting where attention and recurrence are alternated every KK tokens on our original downstream tasks.

For our language modeling task, we find that performance is robust and stable across various KK values. Across K=1...128K=1...128, the average accuracy on our language modeling tasks remains within 0.2 percentage points across KK values.

ModelModeKAvg. accuracy
Oryx-TMAttention only-56.3
Alternating156.5
256.5
456.5
856.6
1656.5
3256.4
6456.5
12856.4
Oryx-TGAttention only-56.5
Alternating156.7
256.8
456.8
856.7
1656.7
3256.7
6456.7
12856.8

For our retrieval task, we process the context with alternating attention and recurrent mechanisms while retaining the switch to attention for prompt and generation. On real-world retrieval, the average accuracy across all tasks is within 0.1 percentage points of the full-recurrent context processing for Oryx-TG and 0.3 percentage points for Oryx-TM, and context alternating actually improves synthetic retrieval performance over full-recurrent context processing by more than 4 percentage points for Oryx-TM and enables comparable performance to full attention processing for Oryx-TG.

We highlight that this setting actually increases Oryx-TM's accuracy on NIAH-3, despite substantially increasing the number of mixer transitions compared to prior experiments. These results support our hypothesis that the drop was due to Mamba-2’s processing and demonstrates that frequent mode switching can actually lead to substantially stronger retrieval capabilities.

ModelModeKReal-world Avg.Synthetic Avg.
Oryx-TMAttention only-44.093.6
Alternating143.890.6
243.990.3
443.890.0
843.990.3
1643.790.7
3243.890.1
6443.790.5
12843.790.4
Oryx-TGAttention only-41.199.1
Alternating141.399.2
241.399.1
441.499.1
841.499.0
1641.399.1
3241.399.0
6441.499.1
12841.499.0

All alternating results average over attention-first and recurrent-first schedules.

To summarize, in our joint and reviewer specific response, we have addressed your hybrid baseline concern. We find that our shared-representation sequence-axis hybrid models maintain competitive performance on language modeling and retrieval tasks compared to strong interleaved hybrid baselines. Furthermore, we find that multiple mixer switches within the context does not degrade retrieval accuracy, but can actually boost performance, suggesting Oryx-TM's NIAH-3 degradation is due to Mamba-2’s inductive biases. In addition, our analysis demonstrates that our Oryx-style sequence-hybridization can result in around a 33% decrease in mixer FLOPs compared to a 3:1 Mamba-2 hybrid model. Taken together, our results demonstrate the potential that shared representations can enable for a separate axis of hybridization.

As the end of the discussion phase is approaching, we would appreciate hearing whether our additional results and experiments address your questions or concerns. We look forward to hearing back from you, and we are more than happy to answer any lingering questions or concerns.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Joint Response 1/

We thank the reviewers for their time and effort in reviewing our paper, and for their thoughtful comments and feedback. We are glad that they found our exploration into sequence-axis hybridization through shared representations interesting (AgqD, khdb) and appreciated the insights from our separate-query and chunk-training ablations (AgqD, Jksn, khdb). We address the shared concerns about interleaved hybrid comparisons, Oryx-TM's NIAH-3 retrieval behavior, and practical efficiency below; reviewer-specific questions are addressed in the individual responses.

AuthorsOpenReview ↗

Joint Response 2/

Core Contribution

We would like to underscore that the core contribution of our paper is exploring a separate axis of hybridization enabled by shared representations. While many works have focused on improving interleaved-layer hybrids, there has been less emphasis on developing models which can flexibly change the sequence-mixing mechanism across the input.

Our ablations on separating queries, linear-model specific components (e.g., short conv, gate, and norm), and chunked mixed-mode training identify crucial design choices that enable both mechanisms to operate effectively over shared representations. Across Mamba-2 and GDN, our Oryx models are competitive with strong baselines on language modeling, can switch multiple times between mixers with little to no perplexity degradation, and exhibit cross-mode retrieval capabilities.

We believe that sequence-axis hybridization can supplement existing inter-layer hybrid design instead of superseding it. For example, replacing the attention layers in hybrid models with Oryx layers would preserve the efficiency from the existing linear layers while enabling flexible modeling.

Interleaved Hybrid Comparisons

In this section, we compare our Oryx models with Mamba-2 and GDN-based hybrid models at the 1.4B model scale. Both hybrid models interleave recurrent and attention layers at a 3:1 ratio and follow the original baseline training setup, matching Oryx training tokens and parameter count.

ModelLAMB. pplLAMB. accHellaS. acc_nPIQAARC-EARC-C acc_nWinoGr.OBQAAvg.
3:1 Mamba2 Hybrid10.250.761.474.574.341.260.531.856.3
3:1 GDN Hybrid10.650.561.774.374.142.061.330.656.4
ORYX-TM: attention mode11.150.261.375.075.742.058.031.656.3
ORYX-TM: Mamba-2 mode10.550.462.175.275.343.658.531.256.6
ORYX-TG: attention mode10.949.961.874.975.441.759.831.856.5
ORYX-TG: GDN mode10.650.062.175.176.243.162.032.257.3

On commonsense reasoning and language modeling tasks, both Oryx models outperform or are comparable to their hybrid counterparts. In linear mode, Oryx-TM outperforms its Mamba-2 hybrid counterpart 56.6 to 56.3 on average accuracy, and Oryx-TG also outperforms the GDN hybrid 57.3 to 56.4 . In attention mode, Oryx-TM performs similarly at 56.3 to 56.3, and Oryx-TG as well at 56.5 to 56.4 compared to its respective hybrid baseline.

ModelSWDESQuADFDATQANQDROPReal-world avg.Real-world Avg.*NIAH-1NIAH-2NIAH-3Synthetic avg.
Transformer42.342.855.265.926.722.942.640.199.090.495.094.8
3:1 Mamba2 Hybrid45.944.366.263.728.922.645.341.1100.099.899.499.7
3:1 GDN Hybrid44.442.069.964.330.822.645.740.898.6100.099.099.2
ORYX-TM50.940.956.464.928.222.443.941.499.897.658.085.1
ORYX-TG46.141.147.964.128.020.941.440.199.097.695.897.5

where Avg includes all six real-world tasks, and Avg* excludes the FDA.

We also compare the retrieval capabilities of the hybrid models against our Oryx models using the linear to attention switch. On most real-world retrieval tasks, Oryx performs close to the hybrid baseline. When excluding the FDA benchmark, the Mamba-2 hybrid achieves an average accuracy of 41.1, GDN hybrid 40.8, Oryx-TM 41.4, and Oryx-TG 40.1; the Transformer baseline achieves an average accuracy of 40.1. Our results demonstrate the strong retrieval abilities of Oryx while mode-switching. On synthetic retrieval, Oryx-TG maintains comparable performance with the GDN hybrid. Oryx-TM also performs similarly to its hybrid counterpart except on NIAH-3, which we discuss in the next section.

It is unclear what aspect of the FDA dataset results in this large discrepancy. Interestingly, the hybrid models improve upon FDA from the Transformer baseline’s 55.2 accuracy by more than 10 percentage points, a gain reported for Mamba-2 hybrids but not GDN hybrids [1, 2]. One explanation might be the interaction between tokenization (e.g., GPT-2 vs Llama) and linear model (e.g., GDN vs Mamba-2). We leave this exploration as future work.

AuthorsOpenReview ↗

Joint Response 3/3

Oryx-TM's NIAH-3 retrieval

Oryx-TM maintains strong performance on NIAH-1 and -2 in both attention-only mode as well as cross-mode retrieval, outperforming the Transformer baseline on both tasks. However, this strong performance is not exhibited on NIAH-3 despite cross-mode retrieval improving NIAH-3 performance from the standalone accuracy of 33.2 to 58.0. We hypothesize that this drop in performance is due to the weakness of the Mamba-2 linear layer on processing the uuid needle of NIAH-3 rather than an issue stemming from mode switching, as the sole difference between NIAH-2 and NIAH-3 is the needle type (number vs uuid).

This interpretation is consistent with the results from the original Gated DeltaNet paper [2], which finds that Mamba-2 suffers on (S-) NIAH-3 at all three context lengths tested, which the authors attribute to a lack of sufficiently expressive state-transition, e.g., delta rule. Our Oryx-TG results do not see the same degradation as Oryx-TM and perform strongly on all three synthetic retrieval tasks.

During the linear-to-attention mode switch, although Oryx retains the KV cache for the entire sequence, the keys and values of deeper layers depend on the hidden states produced by the earlier linear layers. Thus, Mamba-2’s dynamics affect the representations subsequently queried by attention. Mamba-2’s difficulty on NIAH-3 may be further compounded by Oryx-TM's reduction in state-size, which decreases the recurrent state size per layer by 50% compared to the pure Mamba-2 and Mamba-2 hybrid baselines. As the key and value projections are trained jointly across both mixers, this modeling bias may also explain Oryx-TM's attention-only performance gap on NIAH-3.

Practical Inference Efficiency

By extending our Appendix C’s Mamba-2 FLOP analysis to our mixed-mode real-world retrieval task setup (first 97.5% tokens processed by Mamba-2 and subsequent by attention), we find that our sequence-hybridization setup reduces the mixer-specific FLOPs by approximately 70.5% compared to full attention at 2K context length. With chunk size and head dimensions C=Dk=Dv=128C=D_k=D_v=128 and equivalent head and depth counts, the cross-mode retrieval Oryx model requires approximately 32.6% fewer mixer FLOPs than a 3:1 Mamba-2-to-attention interleaved hybrid model. We note that these calculations are specific to mixer operations, e.g., state updates and read outs, and do not measure the projections, gating, and other model components.

On the memory side, Oryx retains the full KV cache and recurrent state so future tokens can utilize the full context. Under the shared head structure, each Oryx layer requires 2TDmodel2TD_{\mathrm{model}} for the KV cache and DkDmodelD_kD_{\mathrm{model}} for the recurrent state. At T=2048T=2048 and Dk=128D_k=128, the recurrent state is only 3.125%3.125\% the size of the KV cache assuming equal precision. This fraction further decreases at longer context lengths.

While latency measurements are valuable, they are heavily dependent on kernel implementations and the level of optimization. Our current compute analysis quantifies the FLOPs reduction; We leave implementation-specific latency and throughput benchmarking to future work.

[1] Mamba-3: Improved Sequence Modeling using State Space Principles, https://arxiv.org/abs/2603.15569

[2] Gated Delta Networks: Improving Mamba2 with Delta Rule, https://arxiv.org/abs/2412.06464

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.