- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Softmax attention supports precise KV retrieval through dot products but requires a full KV cache, whereas linear attention maintains a fixed-size state cache without full Q -> K retrieval over the history. To address this trade-off, the paper's Interdomain Attention (IA) separately compresses keys and values through interdomain projections into fixed-size caches at a fixed temporal resolution, while retaining explicit, query-driven multiplicative retrieval along the temporal dimension of the keys within a single attention operation. The implementation uses S4D to maintain and update the KV projections, incorporates modern linear attention components such as QK Norm and short convolutions, and supports chunkwise Triton execution through the FLA library. Downstream evaluations of models trained from scratch show that the full IA model achieves better loss curves than softmax attention and S4D-only at matched parameter counts in both general language modeling and genomic modeling. In genomic modeling, it also matches or outperforms the original Mamba-based Caduceus. Ablations further demonstrate the benefits of IA's KV maintenance and query-conditioned readout.
Critical strengths
- IA's motivation is clear and promising. It sits between softmax attention and traditional linear attention with a state: KV information is projected onto a fixed number of orthogonal HiPPO basis functions to keep cache storage complexity constant, while queries explicitly retrieve information from interdomain key states.
- The evaluation is fairly comprehensive for the selected baselines, covering language modeling scaling curves, long-context recall benchmarks, and genomic modeling tasks that require long contexts.
- The paper provides detailed ablations of IA's core design and supporting components, including memory decay, gating, and short convolutions, clarifying each component's contribution to the final model. It also reports measured inference speedup curves and training throughput.
- The paper clearly describes IA's forward computation and the different component combinations (Fig. 6).
Critical weaknesses
- IA has a sensible motivation and design, with fairly thorough experiments and ablations. However, a major omission in its evaluation as a linear attention component is the lack of direct comparisons with other linear attention approaches, including recent methods such as DeltaNet/Gated DeltaNet, and models with relatively simple state management, such as Mamba2. Establishing IA's advantages over traditional linear attention requires showing that it outperforms other methods using a unified projected state cache in both scaling performance and downstream recall under matched budgets. This is especially important given its poor performance on the recall tasks in the appendix.
- Relatedly, IA uses an S4D-style state cache update policy. Since interdomain projection and QK retrieval are orthogonal to the state cache management mechanisms of other linear attention approaches, further evidence of compatibility with modern, more complex state decay, writing, and update rules may be needed to demonstrate IA's potential in practical models.
- Although the authors incorporate several modern linear attention techniques, the ablations raise some concerns. Table 3 primarily establishes the benefits of query-conditioned readout, the gains from kernel processing over simple linear projection are limited. These experiments do not adequately separate the contributions of IA's core design choices: independent interdomain projections for keys and values, query dependence, and multiplicative combination.
I would be happy to raise my score if the authors provide additional experiments that directly focus on these issues.
Questions for authors
- The number of HiPPO basis functions, , is a key hyperparameter that directly affects state capacity and the resolution at which KV information is maintained, yet the experiments lack an ablation of . In principle, varying it should offer a more flexible quality-efficiency trade-off.
- The inference speed results show overall performance trends, but IA remains significantly slower than softmax attention during training despite using FLA. I agree that a straightforward FLA Triton adaptation may be suboptimal and that linear attention can incur greater overhead on short sequences. Nevertheless, these results raise further doubts about whether IA's interdomain projection with S4D state updates can use the GPU efficiently. I recommend a more detailed analysis, such as a latency breakdown of the operations within the IA kernel, with each class mapping to the corresponding terms in the IA's equations.
- A substantial portion of the main experimental section focuses on the specialized domain of genomic modeling, with a very lightweight Caduceus backbone. I believe the paper would be more convincing if the main text mainly focused on comparisons against multiple baselines in general language modeling, along with the corresponding recall and long-context tasks.
6 comments
Response to Reviewer 5Xs1 (2/2)
I would be happy to raise my score if the authors provide additional experiments that directly focus on these issues.
Many thanks for the useful suggestions. We have conducted them and please see #1, #2, and #3 of the general response.
The number of HiPPO basis functions, M, is a key hyperparameter that directly affects state capacity and the resolution at which KV information is maintained, yet the experiments lack an ablation of M. In principle, varying it should offer a more flexible quality-efficiency trade-off.
Thank you for the suggestion. Increasing alone makes the recurrent state larger, so an improvement is expected, but a comparison between different state sizes is not meaningful. So we change at a fixed state size (Appendix I, Table 28). In this case, a larger means fewer key/value heads or smaller feature/value widths. With 4 heads, with half the key/value heads (NLL 2.7873) is as good as (2.7881). with feature/value width 64 (2.8201) is worse. So at a fixed state size, there should be a best combination of , the number of key/value heads, and the feature/value widths.
The inference speed results show overall performance trends, but IA remains significantly slower than softmax attention during training despite using FLA. I agree that a straightforward FLA Triton adaptation may be suboptimal and that linear attention can incur greater overhead on short sequences. Nevertheless, these results raise further doubts about whether IA's interdomain projection with S4D state updates can use the GPU efficiently. I recommend a more detailed analysis, such as a latency breakdown of the operations within the IA kernel, with each class mapping to the corresponding terms in the IA's equations.
Thank you for the suggestion. As we wrote in #4 of the general response, we added a single-GPU training breakdown to Appendix G. For training speed, IA requires the training time of softmax attention with PyTorch Flash SDPA, as shown in Appendix G ( GH200 GPUs), and on a single GPU. A breakdown of the IA operations is newly added: the SSM state update and the query-conditioned readout take 47.5% of the forward pass (Table 24). However, as we mention in the paper, though we leverage FLA [3], our implementation is not yet fully optimised, which we leave for future work.
[3] Yang and Zhang, FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism, 2024.
A substantial portion of the main experimental section focuses on the specialized domain of genomic modeling, with a very lightweight Caduceus backbone. I believe the paper would be more convincing if the main text mainly focused on comparisons against multiple baselines in general language modeling, along with the corresponding recall and long-context tasks.
We propose IA as a general purpose architecture for sequence, rather than that specialised for Language modelling. To show this, we replace the Mamba layers of Caduceus with IA and keep the rest of the backbone. Caduceus-IA outperforms the corresponding Caduceus baselines on 14 (PS) and 12 (Ph) of the 18 Nucleotide Transformer tasks (Figure 3, Table 18), and Caduceus-IA-Ph has the highest or tied-highest accuracy on five of the eight GenomicBenchmarks tasks (Table 17). Genomics also requires long contexts (up to 131K tokens) in our variant-effect prediction experiment, where a fixed-size memory is useful.
Comment by 5Xs1
Thank you to the authors for their detailed response. The new exploration of state read/write mechanisms and extra ablations of IA's internal design clarify how its components work together. The ablation of M at a fixed state size budget, and the analysis of IA kernel bottlenecks also clarify the boundaries of the design and its implementation for now. Based on these additions, I consider the paper is already in weak accept.
However, I still remain a critical concerned about one of the author's response:
Regarding the recall tasks, as we wrote in #2 of the general response, our memory is fixed-size, like other modern recurrent models, so IA shares their difficulty with exact recall.
In my view, this claim does not appear to hold. The RULER S-NIAH-1 results reported in the Gated DeltaNet and Gated DeltaNet-2 papers show the following:
| Method | 1K | 4K |
|---|---|---|
| IA | 65.5 | 9.5 |
| GDN | 98.4 | 91.4 |
| Mamba-2 | 99.2 | 65.4 |
Although the evaluation protocols and pretraining budgets may differ across these papers, the large gap raises a question about IA's capabilities:
Can IA, proposed in 2026, at least outperform Mamba-2 from 2024 on general long-context recall under matched conditions?
In genomic modeling, IA also performs roughly on par with or slightly better than the Mamba-1-based Caduceus. I acknowledge the substantial progress IA makes over the vanilla S4D baseline. However, IA changes the same part of the core state update rule as Mamba-2 and GDN, so these mechanisms cannot be directly combined. Nevertheless, the authors have yet to demonstrate a theoretical or practical advantage of IA over these methods, which leaves me questioning the initial motivation for IA: Given the current results, exploring Mamba-like state policies may be more promising than developing IA from S4D, and this is also the direction the community is pursuing now.
If feasible, I encourage the authors to provide theoretical evidence or a small-scale case study demonstrating the potential advantage of developing IA from S4D over a Mamba-like state policy. Such evidence would fully address my core concern about IA.
Response to Reviewer 5Xs1 (1/2)
Thank you for your feedback. IA changes how read/write is performed, and keeps the state update rule of S4D. As you note in your next comment, the interdomain projection and the query-conditioned readout are orthogonal to how the state is updated, so we measure their contribution with the update rule fixed. Using the same recurrence and state size as S4D-only, IA achieves lower perplexity at all four scales (7.98 vs. 9.13 at 1.3B), higher accuracy on RULER (0% → 65.5/28.0/9.5% at 1K/2K/4K), and higher answer log-likelihood in all 14 recall settings. The query-conditioned readout is also better than the linear readout when we add Mamba-style selective writing and selective memory decay (Table 27 in Appendix H). Mamba and Gated DeltaNet use different recurrences, so a comparison with them would not isolate the contribution of the IA principles. Regarding the recall tasks, as we wrote in #2 of the general response, our memory is fixed-size, like other modern recurrent models, so IA shares their difficulty with exact recall.
Thank you for the suggestion. IA already works with input-dependent writing and decay: Section 3.3 adds Mamba-style selective writing and selective memory decay to the S4D recurrence. In Figure 4, adding selective memory decay lowers the perplexity of language modeling from 16.095 to 16.008. With both selective updates, replacing the linear readout with the query-conditioned readout lowers NLL in both domains (3.000 -> 2.787 in language modeling and 0.9939 -> 0.9892 in genomics, Table 27 in Appendix H).
Thank you for your feedback. In the updated submission, we separate these design choices. For query dependence, we replace the query with a learned input-independent vector and keep everything else in Base IA (Table 3). NLL increases from 2.786 to 2.938 in language modeling and from 0.9904 to 1.0083 in genomics, over three seeds. For the independent projections of keys and values, IA stores the keys and the values in separate memories, so its readout also combines keys and values of different tokens. We added a variant that combines each key only with the value of the same token (same-token pairing, Table 26 in Appendix H). Its NLL (2.7314) is lower than that of Base IA (2.7909), but a recurrent memory that stores these pairs would be about 96 times larger than the state of IA (Rd instead of R+d per basis function). We agree that the gain from kernel processing over the linear projection is small. When we change the input processing, the NLL in language modeling stays within 0.020 of Base IA (Table 3).