- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
ORYX enables switching between softmax attention and recurrent mixers within a sequence through shared key-value projections and chunk-level mixed-mode training. Experiments with Mamba-2 and Gated DeltaNet demonstrate competitive standalone performance and cross-mode retrieval.
Critical strengths
Sequence-axis hybridization is an interesting direction. Validation with two recurrent architectures and the chunk-level training ablation provide useful evidence that switching compatibility can be learned.
Critical weaknesses
The comparisons mainly involve single-mixer baselines. Although parameters and training tokens are matched, the results do not establish whether sequence-axis switching offers a better performance–cost trade-off than a fixed hybrid architecture.
The practical efficiency benefit remains unclear. Appendix C analyzes simplified mixer FLOPs, while switching requires maintaining both the full KV cache and recurrent state. End-to-end latency, throughput, and memory measurements are needed to assess the actual savings.
Retrieval evaluation is limited to 2K contexts, and compatibility is uneven across tasks. On NIAH-3, ORYX-TM drops from 81.6 in attention-only mode to 58.0 with linear context processing and attention-based querying. This leaves an important limitation of cross-mode retrieval insufficiently explained
I lean toward weak rejection. The switching mechanism is interesting, but the current evidence does not yet establish its practical advantage over existing hybrid designs or adequately characterize its retrieval limitations.
Questions for authors
Could you compare with a fixed inter-layer hybrid using the same data and a matched training-compute budget? Since ORYX uses a higher peak learning rate, were the baselines given a comparable tuning budget?
At comparable retrieval accuracy, what are the measured prefill latency, decoding throughput, and peak memory relative to optimized baselines? Processing fewer tokens with attention does not by itself quantify the end-to-end benefit.
What explains the NIAH-3 drop from 81.6 to 58.0 for ORYX-TM, whereas ORYX-TG drops only from 99.4 to 95.8? Does the failure depend on the needle position or switching location?
Do the retrieval results remain stable beyond 2K contexts and when varying the fixed 97.5% context–prompt split used for real-world tasks? This would clarify how sensitive the gains are to the manually chosen switching schedule.
2 comments
Response to Reviewer khdb 2/2
Do the retrieval results remain stable beyond 2K contexts
Our retrieval evaluations use the 2K context length at which all models were pretrained with, a setting also commonly used in architecture-based papers: the Mamba series use 2K context length and Gated DeltaNet evaluates real-world retrieval on 2K as well [1, 2, 3, 4]. Our setting already exposes substantial differences between the retrieval capabilities of pure linear models, hybrid models, Transformers, and the cross-mode vs recurrent-only modes of Oryx. Evaluation beyond 2K would additionally test length extrapolation, where our RoPE positional encoding used for our Transformer baselines and attention component of Oryx would affect performance, even without mode-switching [5, 6].
We hope our new results and clarifications have addressed the reviewer’s concerns and are happy to answer any follow-up questions.
[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces, https://arxiv.org/abs/2312.00752
[2] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, https://arxiv.org/abs/2405.21060
[3] Mamba-3: Improved Sequence Modeling using State Space Principles, https://arxiv.org/abs/2603.15569
[4] Gated Delta Networks: Improving Mamba2 with Delta Rule, https://arxiv.org/abs/2412.06464
[5] Extending Context Window of Large Language Models via Positional Interpolation, https://arxiv.org/abs/2306.15595
[6] Rope to Nope and Back Again: A New Hybrid Attention Strategy, https://arxiv.org/abs/2501.18795
Response to Reviewer khdb 1/
We thank the reviewer for finding our exploration into sequence-axis hybridization interesting and finding our empirical validation useful. The joint response presents a comparison with fixed inter-layer hybrid models, discusses the NIAH-3 drop for Oryx-TM, and efficiency results. We summarize them here and also address the additional questions below.
We trained Mamba-2 and GDN 1.4B hybrid models at a 3:1 recurrent to attention ratio using the original baseline setup, matching Oryx’s parameter count and token budget. On our language modeling and commonsense benchmarks, both Oryx models outperform their respective hybrid counterparts on average performance when using only the linear mode and perform comparably when using only the attention mode.
On real-world retrieval tasks, the cross-mode (linear-to-attention switch) Oryx performs similarly to the hybrid counterparts except on the FDA task; we expand on this discrepancy in the joint response. When averaging across the other five tasks (SWDE, SQuAD, TQA, NQ, and DROP), Oryx-TM achieves a 41.4 average accuracy to 41.1 for the Mamba-2 hybrid, and Oryx-TG achieves a 40.1 average accuracy to 40.8 for the GDN hybrid. On synthetic retrieval tasks, Oryx performs comparably to the hybrid baselines except for Oryx-TM on NIAH-3, which we address below.
Beyond the strong performance of mode-switching, we would like to emphasize that the shared representation design and the mixer flexibility it entails do not have to replace, but can be complementary to, current hybrid design. We are excited to explore how we can interleave Oryx-like layers with linear layers to enable architectures that are similar to conventional attention-linear hybrids but have flexible mixer switching.
Please see our joint response (2/) for the complete results and discussion.
The hyperparameters used to train the original Oryx models were not exhaustively tuned. The 2× learning rate was a heuristic motivated by its disjoint query parameters. The hyperparameters for the baselines were established from prior literature [1, 2]. We trained additional Transformer and Mamba-2 baselines with the same learning rate as our Oryx setup. We find that Oryx-TM in attention-mode still outperforms the higher-lr Transformer baseline on average accuracy 56.3 to 55.8. The higher-lr Mamba-2 baseline slightly outperforms the recurrent-mode of Oryx-TM 56.8 to 56.6 on average accuracy, but Oryx-TM retains a better perplexity on Lambada at 10.5 compared to 10.9. We highlight that Oryx-TM's recurrent mode is still competitive despite utilizing only half the standalone Mamba-2 baseline’s recurrent state size.
We extend our FLOPs analysis to our cross-mode real-world retrieval setup and find that mode switching reduces the mixer FLOPs by approximately 70.5% relative to full attention and 32.6% relative to a 3:1 Mamba-2 hybrid under the assumptions detailed in joint response (3/3). Oryx’s additional linear recurrent state introduces a small increase in memory requirements, equivalent to 3.125% of the full KV cache at 2K context length. As latency and throughput heavily depend on kernel implementations and optimizations, we leave such benchmarks for future work.
We hypothesize that Oryx-TM's NIAH-3 degradation stems from Mamba-2’s weakness in processing the uuid needle. Oryx-TM performs well on NIAH-2, which retains the same setup as NIAH-3 except needle type, and regular Mamba-2 models suffer on the same task, as shown in the original Gated DeltaNet’s table 2 [4]. Thus, weaker representations from prior layers could affect the representations used by attention during retrieval.