acceptodds
Under review as a conference paper at ICLR 2027

Interdomain Attention: Beyond Token-Level Key-Value Memory

Abstract

Softmax attention-based transformers and recurrent memory-based models (e.g., state space models (SSMs)) form the foundation of modern sequence models. Hybrid architectures aim to combine the expressivity from attention and the computational efficiency from recurrent memory, yet so far common practices simply stack both attention and recurrent layers. We propose Interdomain Attention (IA) as a novel hybrid attention architecture that mixes attention and recurrent memory in the same layer. Motivated by a kernel regression view of softmax attention, IA compresses values and kernel features of keys into a recurrent memory parameterized by an SSM. Then IA approximates dot-product attention by implicitly reconstructing the attention matrix and values from memory states; importantly this construction maintains SSM’s computational efficiency advantage. We demonstrate IA’s efficacy by designing and pre-training IA network architectures, where ablation studies identify attention-like readout as the primary source of gain over SSMs. Across 125M-1.3B models pre trained on text data, IA improves over an S4D baseline at matched memory size. It also outperforms softmax attention-based Transformer in perplexity and accuracy on common-sense reasoning tasks. Based on Caduceus backbone, IA matches or outperforms the original SSM based architecture in both perplexity and downstream tasks on genomic data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
attention, transformers, deep state space models, RNNs

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_5Xs1OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Softmax attention supports precise KV retrieval through dot products but requires a full KV cache, whereas linear attention maintains a fixed-size state cache without full Q -> K retrieval over the history. To address this trade-off, the paper's Interdomain Attention (IA) separately compresses keys and values through interdomain projections into fixed-size caches at a fixed temporal resolution, while retaining explicit, query-driven multiplicative retrieval along the temporal dimension of the keys within a single attention operation. The implementation uses S4D to maintain and update the KV projections, incorporates modern linear attention components such as QK Norm and short convolutions, and supports chunkwise Triton execution through the FLA library. Downstream evaluations of models trained from scratch show that the full IA model achieves better loss curves than softmax attention and S4D-only at matched parameter counts in both general language modeling and genomic modeling. In genomic modeling, it also matches or outperforms the original Mamba-based Caduceus. Ablations further demonstrate the benefits of IA's KV maintenance and query-conditioned readout.

Critical strengths
  1. IA's motivation is clear and promising. It sits between softmax attention and traditional linear attention with a D×DD\times D state: KV information is projected onto a fixed number of orthogonal HiPPO basis functions to keep cache storage complexity constant, while queries explicitly retrieve information from MM interdomain key states.
  2. The evaluation is fairly comprehensive for the selected baselines, covering language modeling scaling curves, long-context recall benchmarks, and genomic modeling tasks that require long contexts.
  3. The paper provides detailed ablations of IA's core design and supporting components, including memory decay, gating, and short convolutions, clarifying each component's contribution to the final model. It also reports measured inference speedup curves and training throughput.
  4. The paper clearly describes IA's forward computation and the different component combinations (Fig. 6).
Critical weaknesses
  1. IA has a sensible motivation and design, with fairly thorough experiments and ablations. However, a major omission in its evaluation as a linear attention component is the lack of direct comparisons with other linear attention approaches, including recent methods such as DeltaNet/Gated DeltaNet, and models with relatively simple state management, such as Mamba2. Establishing IA's advantages over traditional linear attention requires showing that it outperforms other methods using a unified projected state cache in both scaling performance and downstream recall under matched budgets. This is especially important given its poor performance on the recall tasks in the appendix.
  2. Relatedly, IA uses an S4D-style state cache update policy. Since interdomain projection and QK retrieval are orthogonal to the state cache management mechanisms of other linear attention approaches, further evidence of compatibility with modern, more complex state decay, writing, and update rules may be needed to demonstrate IA's potential in practical models.
  3. Although the authors incorporate several modern linear attention techniques, the ablations raise some concerns. Table 3 primarily establishes the benefits of query-conditioned readout, the gains from kernel processing over simple linear projection are limited. These experiments do not adequately separate the contributions of IA's core design choices: independent interdomain projections for keys and values, query dependence, and multiplicative combination.

I would be happy to raise my score if the authors provide additional experiments that directly focus on these issues.

Questions for authors
  1. The number of HiPPO basis functions, MM, is a key hyperparameter that directly affects state capacity and the resolution at which KV information is maintained, yet the experiments lack an ablation of MM. In principle, varying it should offer a more flexible quality-efficiency trade-off.
  2. The inference speed results show overall performance trends, but IA remains significantly slower than softmax attention during training despite using FLA. I agree that a straightforward FLA Triton adaptation may be suboptimal and that linear attention can incur greater overhead on short sequences. Nevertheless, these results raise further doubts about whether IA's interdomain projection with S4D state updates can use the GPU efficiently. I recommend a more detailed analysis, such as a latency breakdown of the operations within the IA kernel, with each class mapping to the corresponding terms in the IA's equations.
  3. A substantial portion of the main experimental section focuses on the specialized domain of genomic modeling, with a very lightweight Caduceus backbone. I believe the paper would be more convincing if the main text mainly focused on comparisons against multiple baselines in general language modeling, along with the corresponding recall and long-context tasks.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 5Xs1 (1/2)

IA has a sensible motivation and design, with fairly thorough experiments and ablations. However, a major omission in its evaluation as a linear attention component is the lack of direct comparisons with other linear attention approaches, including recent methods such as DeltaNet/Gated DeltaNet, and models with relatively simple state management, such as Mamba2. Establishing IA's advantages over traditional linear attention requires showing that it outperforms other methods using a unified projected state cache in both scaling performance and downstream recall under matched budgets. This is especially important given its poor performance on the recall tasks in the appendix.

Thank you for your feedback. IA changes how read/write is performed, and keeps the state update rule of S4D. As you note in your next comment, the interdomain projection and the query-conditioned readout are orthogonal to how the state is updated, so we measure their contribution with the update rule fixed. Using the same recurrence and state size as S4D-only, IA achieves lower perplexity at all four scales (7.98 vs. 9.13 at 1.3B), higher accuracy on RULER (0% → 65.5/28.0/9.5% at 1K/2K/4K), and higher answer log-likelihood in all 14 recall settings. The query-conditioned readout is also better than the linear readout when we add Mamba-style selective writing and selective memory decay (Table 27 in Appendix H). Mamba and Gated DeltaNet use different recurrences, so a comparison with them would not isolate the contribution of the IA principles. Regarding the recall tasks, as we wrote in #2 of the general response, our memory is fixed-size, like other modern recurrent models, so IA shares their difficulty with exact recall.

Relatedly, IA uses an S4D-style state cache update policy. Since interdomain projection and QK retrieval are orthogonal to the state cache management mechanisms of other linear attention approaches, further evidence of compatibility with modern, more complex state decay, writing, and update rules may be needed to demonstrate IA's potential in practical models.

Thank you for the suggestion. IA already works with input-dependent writing and decay: Section 3.3 adds Mamba-style selective writing and selective memory decay to the S4D recurrence. In Figure 4, adding selective memory decay lowers the perplexity of language modeling from 16.095 to 16.008. With both selective updates, replacing the linear readout with the query-conditioned readout lowers NLL in both domains (3.000 -> 2.787 in language modeling and 0.9939 -> 0.9892 in genomics, Table 27 in Appendix H).

Although the authors incorporate several modern linear attention techniques, the ablations raise some concerns. Table 3 primarily establishes the benefits of query-conditioned readout, the gains from kernel processing over simple linear projection are limited. These experiments do not adequately separate the contributions of IA's core design choices: independent interdomain projections for keys and values, query dependence, and multiplicative combination.

Thank you for your feedback. In the updated submission, we separate these design choices. For query dependence, we replace the query with a learned input-independent vector and keep everything else in Base IA (Table 3). NLL increases from 2.786 to 2.938 in language modeling and from 0.9904 to 1.0083 in genomics, over three seeds. For the independent projections of keys and values, IA stores the keys and the values in separate memories, so its readout also combines keys and values of different tokens. We added a variant that combines each key only with the value of the same token (same-token pairing, Table 26 in Appendix H). Its NLL (2.7314) is lower than that of Base IA (2.7909), but a recurrent memory that stores these pairs would be about 96 times larger than the state of IA (RdRd instead of R+dR+d per basis function). We agree that the gain from kernel processing over the linear projection is small. When we change the input processing, the NLL in language modeling stays within 0.020 of Base IA (Table 3).

AuthorsOpenReview ↗

Response to Reviewer 5Xs1 (2/2)

I would be happy to raise my score if the authors provide additional experiments that directly focus on these issues.

Many thanks for the useful suggestions. We have conducted them and please see #1, #2, and #3 of the general response.

The number of HiPPO basis functions, M, is a key hyperparameter that directly affects state capacity and the resolution at which KV information is maintained, yet the experiments lack an ablation of M. In principle, varying it should offer a more flexible quality-efficiency trade-off.

Thank you for the suggestion. Increasing MM alone makes the recurrent state larger, so an improvement is expected, but a comparison between different state sizes is not meaningful. So we change MM at a fixed state size (Appendix I, Table 28). In this case, a larger MM means fewer key/value heads or smaller feature/value widths. With 4 heads, M=128M=128 with half the key/value heads (NLL 2.7873) is as good as M=64M=64 (2.7881). M=128M=128 with feature/value width 64 (2.8201) is worse. So at a fixed state size, there should be a best combination of MM, the number of key/value heads, and the feature/value widths.

The inference speed results show overall performance trends, but IA remains significantly slower than softmax attention during training despite using FLA. I agree that a straightforward FLA Triton adaptation may be suboptimal and that linear attention can incur greater overhead on short sequences. Nevertheless, these results raise further doubts about whether IA's interdomain projection with S4D state updates can use the GPU efficiently. I recommend a more detailed analysis, such as a latency breakdown of the operations within the IA kernel, with each class mapping to the corresponding terms in the IA's equations.

Thank you for the suggestion. As we wrote in #4 of the general response, we added a single-GPU training breakdown to Appendix G. For training speed, IA requires 2.82×2.82\times the training time of softmax attention with PyTorch Flash SDPA, as shown in Appendix G (4×4\times GH200 GPUs), and 1.68×1.68\times on a single GPU. A breakdown of the IA operations is newly added: the SSM state update and the query-conditioned readout take 47.5% of the forward pass (Table 24). However, as we mention in the paper, though we leverage FLA [3], our implementation is not yet fully optimised, which we leave for future work.

[3] Yang and Zhang, FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism, 2024.

A substantial portion of the main experimental section focuses on the specialized domain of genomic modeling, with a very lightweight Caduceus backbone. I believe the paper would be more convincing if the main text mainly focused on comparisons against multiple baselines in general language modeling, along with the corresponding recall and long-context tasks.

We propose IA as a general purpose architecture for sequence, rather than that specialised for Language modelling. To show this, we replace the Mamba layers of Caduceus with IA and keep the rest of the backbone. Caduceus-IA outperforms the corresponding Caduceus baselines on 14 (PS) and 12 (Ph) of the 18 Nucleotide Transformer tasks (Figure 3, Table 18), and Caduceus-IA-Ph has the highest or tied-highest accuracy on five of the eight GenomicBenchmarks tasks (Table 17). Genomics also requires long contexts (up to 131K tokens) in our variant-effect prediction experiment, where a fixed-size memory is useful.

Reviewer_5Xs1OpenReview ↗

Comment by 5Xs1

Thank you to the authors for their detailed response. The new exploration of state read/write mechanisms and extra ablations of IA's internal design clarify how its components work together. The ablation of M at a fixed state size budget, and the analysis of IA kernel bottlenecks also clarify the boundaries of the design and its implementation for now. Based on these additions, I consider the paper is already in weak accept.

However, I still remain a critical concerned about one of the author's response:

Regarding the recall tasks, as we wrote in #2 of the general response, our memory is fixed-size, like other modern recurrent models, so IA shares their difficulty with exact recall.

In my view, this claim does not appear to hold. The RULER S-NIAH-1 results reported in the Gated DeltaNet and Gated DeltaNet-2 papers show the following:

Method1K4K
IA65.59.5
GDN98.491.4
Mamba-299.265.4

Although the evaluation protocols and pretraining budgets may differ across these papers, the large gap raises a question about IA's capabilities:

Can IA, proposed in 2026, at least outperform Mamba-2 from 2024 on general long-context recall under matched conditions?

In genomic modeling, IA also performs roughly on par with or slightly better than the Mamba-1-based Caduceus. I acknowledge the substantial progress IA makes over the vanilla S4D baseline. However, IA changes the same part of the core state update rule as Mamba-2 and GDN, so these mechanisms cannot be directly combined. Nevertheless, the authors have yet to demonstrate a theoretical or practical advantage of IA over these methods, which leaves me questioning the initial motivation for IA: Given the current results, exploring Mamba-like state policies may be more promising than developing IA from S4D, and this is also the direction the community is pursuing now.

If feasible, I encourage the authors to provide theoretical evidence or a small-scale case study demonstrating the potential advantage of developing IA from S4D over a Mamba-like state policy. Such evidence would fully address my core concern about IA.

Reviewer_GV49OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Interdomain Attention (IA), a hybrid sequence-modeling architecture that integrates query-dependent attention with fixed-size recurrent memory. IA projects key features and values onto SSM-based temporal bases and performs attention-like readout over compressed memory states. Experiments on language modeling and genomic sequence modeling demonstrate improved perplexity and competitive downstream performance. Ablations identify query-conditioned readout as the primary contributor to performance improvements.

Critical strengths
  1. IA integrates recurrent memory and query-dependent retrieval within a single layer, offering a conceptually meaningful alternative to conventional hybrid architectures.
  2. The paper evaluates multiple model scales and two distinct domains. At 1.3B parameters, IA improves validation perplexity from 8.62 to 7.98 over Softmax. Component ablations provide useful evidence supporting the importance of attention-like readout.
Critical weaknesses
  1. Insufficient competitive baselines and practical efficiency. The main comparisons focus on S4D and Softmax, without matched experiments against modern architectures such as Mamba-2/3 or Gated DeltaNet.
  2. Appendix Table 15 reveals substantial performance degradation on recall-intensive tasks. At 4K context, IA achieves only 9.5% accuracy on RULER compared with 89% for Softmax, and obtains 0% on all Phonebook settings. These results undermine the claimed attention-like retrieval capability and suggest that stable long-context perplexity does not necessarily imply effective information retention.
Questions for authors
  1. What explains the substantial retrieval degradation observed in Appendix Table 15 despite improved perplexity? Do existing memory-capacity analyses help explain this behavior?
  2. Since the practical implementation drops the normalization denominator in Equation (5), to what extent does the resulting operator approximate normalized softmax attention?
  3. Can the authors clarify the absence of matched modern SSM/linear-attention baselines and whether relevant comparative results are available?
  4. How much of the reported inference advantage remains under identical CUDA-graph and prefill evaluation conditions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer GV49 (1/2)

Insufficient competitive baselines and practical efficiency. The main comparisons focus on S4D and Softmax, without matched experiments against modern architectures such as Mamba-2/3 or Gated DeltaNet.

Thank you for your feedback. As we mention in detail in #1 of the general response, we intentionally chose our baselines (S4D-only and softmax attention) to ensure the major improvement comes from IA principles. S4D-only vs IA compares how read/write is performed, and softmax attention vs IA compares how KV cache is stored. Mamba and Gated DeltaNet use different recurrences, so a comparison with them would not isolate the contribution of the IA principles.

For the practical efficiency, as we wrote in #4 of the general response, we have rerun the softmax baseline with CUDA graph using vLLM, and updated corresponding sections: inference speed paragraph in Section 5.1 and Appendix F. Regarding the decode (Table 20 in the updated submission), for smaller batch sizes and shorter sequence lengths, softmax attention in vLLM is faster than IA, e.g., at batch size 1 and 4k tokens, 4.3 ms vs 12.3 ms per step for softmax attention and IA respectively, but softmax attention’s latency grows as the batch size and sequence length get larger, e.g., at batch size 8 and 16k tokens, IA is 2.2 times faster than softmax attention. Regarding the prefill (Table 21 in the updated submission), both models now compute the vocabulary logits of the last token only, and softmax attention uses PyTorch SDPA. At batch size 8 and 16k tokens, IA needs 6.4 GiB of additional memory while softmax attention needs 34.8 GiB, but IA's prefill latency is 1.4 times longer than that of softmax attention.

Appendix Table 15 reveals substantial performance degradation on recall-intensive tasks. At 4K context, IA achieves only 9.5% accuracy on RULER compared with 89% for Softmax, and obtains 0% on all Phonebook settings. These results undermine the claimed attention-like retrieval capability and suggest that stable long-context perplexity does not necessarily imply effective information retention.

Thank you for pointing this out. As we wrote in #2 of the general response, our memory is fixed-size, like other modern recurrent models, so IA shares their difficulty with exact recall. At the same time, IA improves the RULER accuracy over S4D-only (0% → 65.5/28.0/9.5% at 1K/2K/4K), so we conclude that the IA principle improves retrieval performance over the same recurrence. As discussed in [1,2], fixed-state models typically perform poorly on exact recall tasks, and IA inherits this limitation. Phonebook is a typical example: both IA and S4D-only have 0% accuracy, but IA still has a higher answer log-likelihood than S4D-only, as in all 14 recall settings. Though we have an attention-like readout structure, it does not mean that we do not suffer from this limitation.

[1] Arora et al., Zoology: Measuring and Improving Recall in Efficient Language Models, ICLR 2024.

[2] Jelassi et al., Repeat After Me: Transformers are Better than State Space Models at Copying, ICML 2024.

AuthorsOpenReview ↗

Response to Reviewer GV49 (2/2)

What explains the substantial retrieval degradation observed in Appendix Table 15 despite improved perplexity? Do existing memory-capacity analyses help explain this behavior?

As we answered in #2 of the general response, IA shares the difficulty of fixed-state models with exact recall [1,2].

While perplexity takes the local likelihood into account, recall requires to match tokens across longer sequences. As the context grows, the same fixed-size memory has to memorize a longer token sequence, so the accuracy decreases with respect to the length.

[1] Arora et al., Zoology: Measuring and Improving Recall in Efficient Language Models, ICLR 2024.

[2] Jelassi et al., Repeat After Me: Transformers are Better than State Space Models at Copying, ICML 2024.

Since the practical implementation drops the normalization denominator in Equation (5), to what extent does the resulting operator approximate normalized softmax attention?

Thank you for the question. Without the denominator, IA approximates only the numerator of the Nadaraya–Watson kernel regression estimator in Equation (5), so the sum of the attention weights is not normalized. Our work follows recent linear attention models [4], which drop the denominator and use other normalization instead. We also trained IA with the denominator of Equation (5), and its training diverged (Table 26 in Appendix H). Instead of the denominator, IA uses L2-normalized key features and a normalization before the recurrence (Equation (6)) to stabilize the training.

[4] Yang et al., Parallelizing Linear Transformers with the Delta Rule over Sequence Length, NeurIPS 2024.

Can the authors clarify the absence of matched modern SSM/linear-attention baselines and whether relevant comparative results are available?

As we wrote in #1 of the general response, we intentionally chose our baselines (S4D-only and softmax attention) to ensure the major improvement comes from IA principles, so the comparison with modern SSM/linear-attention architectures is not included.

How much of the reported inference advantage remains under identical CUDA-graph and prefill evaluation conditions?

As we wrote in #4 of the general response, under identical CUDA-graph and prefill conditions, IA is faster than softmax attention for larger batch sizes and longer sequences (2.2 times at batch size 8 and 16k tokens), and needs less additional memory in prefill (6.4 GiB vs. 34.8 GiB).

Dear Reviewer GV49:

As the discussion period is approaching its end, we would like to kindly follow up and ask whether our rebuttal has addressed your concerns. We would greatly appreciate any feedback you may have and would be happy to address any remaining questions or concerns before the discussion ends.

Best,

Authors of Submission 31946

Reviewer_BW6aOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to achieve content-related attention readout in fixed-size circular memory, proposing Interdomain Attention (IA). The method starts from the perspective of kernel regression, projecting key features and values onto a shared temporal basis, maintaining two sets of compressed states through complex S4D recurrence, and then calculating outputs using query-conditioned readout. The actual implementation uses learned feature maps, normalization, and chunkwise kernels, exploring enhancements such as query MLP, output gates, and selective memory updates. Across four language model scales, IA outperforms matched-state S4D control; the validation PPL for the 1.3B model is 7.98, lower than Softmax's 8.62, and the average accuracy on eight common sense tasks increases by 3.03 percentage points. In genomic tasks, Caduceus-IA's pre-training PPL is close to that of the original Caduceus, with improvements in some downstream tasks. The authors believe that combining recurrent compression and attention-like readout at the same layer is an effective hybrid architecture, attributing the main benefits to query-dependent readout.

Critical strengths
  1. The method design is relatively clear: key features and values are compressed separately, and then combined through query-conditioned readout, achieving content-related reading in a fixed-size recurrent state.

  2. The language modeling results are somewhat convincing. The main training setup and state size control are good, and IA outperforms S4D control across four model sizes, with a 1.3B improvement also reflected in multiple downstream tasks.

  3. Table 3 provides a direct ablation of the readout, indicating that query-conditioned readout can indeed improve language modeling, and the benefits do not rely on selective updates.

Critical weaknesses
  1. There is a lack of direct comparison with the closest methods. The current language models mainly compare Softmax and the S4D control constructed by the authors, and have not yet compared with methods like H3 and Gated DeltaNet, making it difficult to determine the incremental value of the new design relative to existing recurrent memory methods.

  2. The explanation of the core mechanism is not sufficient. The input ablation simultaneously altered convolution and normalization, and in Tables 3 and 7, the complete IA cell on the genomic task even performs worse than the control with linear input and linear readout; the existing results support that query-conditioned readout is useful, but have not sufficiently demonstrated what unique benefits the interdomain design itself brings.

  3. The settings for efficiency comparison are inconsistent. The inference tests only enable CUDA graph for IA, and in the prefill GPU memory comparison, only chunked IA does not calculate vocabulary logits; considering that the training time is still about 2.8 times that of Softmax, the actual efficiency advantage needs to be validated under consistent settings.

Questions for authors
  1. The input ablation of Table 3 simultaneously changed convolution and normalization. Are there consistent control results that maintain these preprocessing steps? How do you explain the phenomenon where the complete IA cell is less effective than the linear input and linear readout control in genomic tasks?

  2. Are there already control results with H3 or Gated DeltaNet under a similar budget? If not, what current evidence supports the advantage of "compressing key/value separately and then combining the readout" compared to existing methods?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BW6a (1/2)

There is a lack of direct comparison with the closest methods. The current language models mainly compare Softmax and the S4D control constructed by the authors, and have not yet compared with methods like H3 and Gated DeltaNet, making it difficult to determine the incremental value of the new design relative to existing recurrent memory methods.

Thank you for your feedback. As we mention in detail in #1 of the general response, we intentionally chose our baselines (S4D-only and softmax attention) to ensure the major improvement comes from IA principles. S4D-only vs IA compares how read/write is performed, and softmax attention vs IA compares how KV cache is stored. Mamba and Gated DeltaNet use different recurrences, so a comparison with them would not isolate the contribution of the IA principles. H3 stores the products of keys and values in one recurrent memory. To test this way of storing, we added a variant of IA that keeps only the key–value pairs from the same token (same-token pairing, Table 26 in Appendix H). Its NLL (2.7314) is lower than that of Base IA (2.7909), but a recurrent memory that stores these products would be about 96 times larger than the state of IA (RdRd instead of R+dR+d per basis function).

The explanation of the core mechanism is not sufficient. The input ablation simultaneously altered convolution and normalization, and in Tables 3 and 7, the complete IA cell on the genomic task even performs worse than the control with linear input and linear readout; the existing results support that query-conditioned readout is useful, but have not sufficiently demonstrated what unique benefits the interdomain design itself brings.

Thank you for your feedback. We now ablate convolution and normalization separately (Table 3). Each of these changes is small compared with the IA readout. In language modeling, the NLL stays within 0.020 of Base IA, and changing the IA readout increases NLL by at least 0.152.

In genomics, linear-projection input + linear readout is better than Base IA because its input applies a convolution to the values, and IA is better when it uses the same input.

AuthorsOpenReview ↗

Response to Reviewer BW6a (2/2)

The settings for efficiency comparison are inconsistent. The inference tests only enable CUDA graph for IA, and in the prefill GPU memory comparison, only chunked IA does not calculate vocabulary logits; considering that the training time is still about 2.8 times that of Softmax, the actual efficiency advantage needs to be validated under consistent settings.

Thank you for pointing out this issue. As we wrote in #4 of the general response, we have rerun the softmax baseline with CUDA graph using vLLM, and updated corresponding sections: inference speed paragraph in Section 5.1, Appendix F, and Appendix G. Regarding the decode (Table 20 in the updated submission), for smaller batch sizes and shorter sequence lengths, softmax attention in vLLM is faster than IA, e.g., at batch size 1 and 4k tokens, 4.3 ms vs 12.3 ms per step for softmax attention and IA respectively, but softmax attention’s latency grows as the batch size and sequence length get larger, e.g., at batch size 8 and 16k tokens, IA is 2.2 times faster than softmax attention. Regarding the prefill (Table 21 in the updated submission), both models now compute the vocabulary logits of the last token only, and softmax attention uses PyTorch SDPA. At batch size 8 and 16k tokens, IA needs 6.4 GiB of additional memory while softmax attention needs 34.8 GiB, but IA's prefill latency is 1.4 times longer than that of softmax attention. For training speed, IA requires 2.82×2.82\times the training time of softmax attention with PyTorch Flash SDPA, as shown in Appendix G (4×4\times GH200 GPUs), and IA takes 1.68 times as long on a single GPU. As we mention in the paper, though we leverage FLA [3], our implementation is not yet fully optimised, which we leave for future work.

[3] Yang and Zhang, FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism, 2024.

The input ablation of Table 3 simultaneously changed convolution and normalization. Are there consistent control results that maintain these preprocessing steps? How do you explain the phenomenon where the complete IA cell is less effective than the linear input and linear readout control in genomic tasks?

Thank you for the questions. Yes, we now compare the two readouts with the same input (Table 25).

In genomics, linear-projection input + linear readout (NLL 0.9822) is better than Base IA (0.9904). The reason is the input. The linear-projection input applies a convolution to the values, and Base IA does not. If we give IA the same input, IA (0.9791) is better than linear-projection input + linear readout (0.9822). So with the same input, the query-conditioned readout is better than the linear readout.

We use this value convolution in our full genomics model (Section 3.3, Figure 4).

Are there already control results with H3 or Gated DeltaNet under a similar budget? If not, what current evidence supports the advantage of "compressing key/value separately and then combining the readout" compared to existing methods?

As we wrote in #1 of the general response, we intentionally chose our baselines (S4D-only and softmax attention) to ensure the major improvement comes from IA principles, while Gated DeltaNet uses a different recurrence. For H3, storing the key–value products as in same-token pairing has a lower NLL than IA, but needs a recurrent memory about 96 times larger. The current evidence is the comparison with S4D-only, which uses the same recurrence and state size. IA achieves lower perplexity than S4D-only at all four scales (7.98 vs. 9.13 at 1.3B). In the ablation, replacing the query with a learned input-independent vector increases NLL from 2.786 to 2.938 in language modeling, and removing the query and using a linear readout increases it to 3.000 (Table 3).

Dear Reviewer BW6a:

As the discussion period is approaching its end, we would like to kindly follow up and ask whether our rebuttal has addressed your concerns. We would greatly appreciate any feedback you may have and would be happy to address any remaining questions or concerns before the discussion ends.

Best,

Authors of Submission 31946

Reviewer_pRd8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Interdomain Attention (IA) as an efficient (sota) alternative to State Space Models (SSMs) and traditional Softmax Attention. This might be especially relevant and is evaluated for expensive inputs such as genomic data.

Critical strengths

In general, I really like the paper, some highlights would include:

  • Great motivation and problem definition. After reading the Abstract and the Introduction, I was clearly able to see the contribution of the paper.
  • Evaluations look strong at first sight
  • Figures are clean and well-scaled, especially Figure 1 helped a lot for understanding the problem in a fast way
Critical weaknesses

While the paper in general is well-written and the topic is important as well, I'm not convinced that IA does really help and/or is better than already existing attention / state space model types.

  • I did not find any information about statistical evaluations in the paper, e.g., cross-validations or multi-seed evaluations as well as confidence intervals denoting standard error or standard deviation in the tables.
  • I understand that these models are huge and the data is not small as well. However, it seems like there is not hyperparameter tuning at all (not even a smaller grid-search). This would enhance the trust in this method a lot especially compared to the baselines! Would it maybe be possible to extensively tune a very naive baseline which is computationally feasible? A lot of work recent work shows that often simple baselines are enough.
  • It's great to see similar results for the ablation study in Figure 4 on two datasets. However, since these ablations are not really hyperparameter tuned, I'm not really convinced that they reflect the real contributions of the different features.
  • Im wondering whether the IA architecture could be compared to more baselines. So, e.g., the attention baseline seems to be pre-LN-normed already (which is a fair comparison to IA), but what about a SiLU softmax attention baseline? Even a convolution-hybrid softmax attention baseline would be feasible since IA does use L2Norm(SiLU(SC(.))). Such comparisons would help distinguish the advantages of the proposed IA mechanism from those of the additional architectural components.
  • mini mini nitpick for Figure 4 because it extends beyond the right margin
Questions for authors

see Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer pRd8 (1/2)

While the paper in general is well-written and the topic is important as well, I'm not convinced that IA does really help and/or is better than already existing attention / state space model types.

Thank you for your feedback. As we mention in detail in #1 of the general response, we intentionally chose our baselines (S4D-only and softmax attention) to ensure the major improvement comes from IA principles. S4D-only vs IA compares how read/write is performed, and softmax attention vs IA compares how KV cache is stored. Mamba and Gated DeltaNet use different recurrences, so a comparison with them would not isolate the contribution of the IA principles.

I did not find any information about statistical evaluations in the paper, e.g., cross-validations or multi-seed evaluations as well as confidence intervals denoting standard error or standard deviation in the tables.

Thank you for your advice that makes our evaluation stronger. As we mention in detail in #3 of the general response, in Table 3 of the updated submission (details in Table 25), we added a more detailed mechanism ablation using three seeds and show mean and standard deviation of them, and we see that the improvements from IA design are much larger than seed variation. Since each seed means a full pretraining run, we ran three full pretraining runs for each 125M language-model ablation reported in Table 3, and left a single seed for the larger models.

I understand that these models are huge and the data is not small as well. However, it seems like there is not hyperparameter tuning at all (not even a smaller grid-search). This would enhance the trust in this method a lot especially compared to the baselines! Would it maybe be possible to extensively tune a very naive baseline which is computationally feasible? A lot of work recent work shows that often simple baselines are enough.

We agree that tuning the hyperparameter could further strengthen the comparison. In preliminary study, we have tried comparing multiple learning rates (1.5e-3, 3e-3 and 6e-3 for language modeling at 125M, and 4e-3, 8e-3 and 1.6e-2 for genomics) and we observe that choosing a larger learning rate can have better perplexity in the small scale experiment, but at the same time we observe that it is more likely to have the spikes in loss curve. Therefore, we chose the same peak learning rate to Mamba paper’s (3e-3 at 125M and 1e-3 at 1.3B), which Mamba uses also for Transformer++ baseline for language modeling, and Caduceus paper’s (8e-3) for genomics pretraining, and used them for all compared models. These settings were tuned for the baselines in their original papers, and IA uses the same settings, so the comparison is fair to the baselines.

AuthorsOpenReview ↗

Response to Reviewer pRd8 (2/2)

It's great to see similar results for the ablation study in Figure 4 on two datasets. However, since these ablations are not really hyperparameter tuned, I'm not really convinced that they reflect the real contributions of the different features.

Thank you for sharing your concern on the incremental ablations. In Figure 4, each step keeps the seed and all other settings of the previous step (see our answer above on how we chose the learning rate) and changes only the component being added to ablate. If we tune the hyperparameter in each incremental ablation step, then there would be another concern that the contribution from the change and the tuning is mixed. Compared to the difference on NLL by random seed in the updated Table 3, Q/K convolution, residual query MLP, and output gate are effective on both domains, and selective memory decay and value convolution are effective for language modeling and genomics modeling respectively. However, the improvements in Fig. 4 is optional enhancement and ablation on the main structure from IA’s principle is backed by the updated Table 3, with multiple seeds. (please see #3 of the general response)

Im wondering whether the IA architecture could be compared to more baselines. So, e.g., the attention baseline seems to be pre-LN-normed already (which is a fair comparison to IA), but what about a SiLU softmax attention baseline? Even a convolution-hybrid softmax attention baseline would be feasible since IA does use L2Norm(SiLU(SC(.))). Such comparisons would help distinguish the advantages of the proposed IA mechanism from those of the additional architectural components.

Thank you for the suggestion. We added two Softmax baselines with the components of IA on the queries and keys (Table 26 in Appendix H). Adding the short convolution and SiLU gives an NLL of 2.8485, still higher than both the Softmax baseline (2.8381) and Base IA (2.7909) with the same seed. With the L2 normalization as well, the NLL is much higher (3.3691), because the attention weights become almost uniform. So the advantage of IA does not come from these components.

mini mini nitpick for Figure 4 because it extends beyond the right margin

Thank you for finding the issue. We fixed it in the new submission.

I still like the paper in general very much and I really appreciate the author’s efforts for the rebuttal. However, my cross-validation and hyperparameter tuning concerns mostly remain (a relevant paper might be [1]). For the tuning of ablations, I believe that adding a feature or leaving it out can completely change the optimization space. Simply using the same hyperparameters is in my opinion not the reliable way. I know that this is very uncommon, especially for genomic datasets, but I think it is highly needed here. Thank you for adding the new baseline results. I think these make the paper stronger!

I know this is a very specific concern, however, I guess my current score also reflects that this paper should be accepted in general.

[1] K. Vishniakov et al., “Tokenization to Transfer: Do Genomic Foundation Models Learn Good Representations?,” in International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, Eds., 2026, pp. 25386–25406.

Thank you for your reply and for your positive assessment of the paper and the new baseline results.

We agree that adding/removing a component can change the best hyperparameters, so using the same hyperparameters for all variants is the limitation of our ablations. Due to the limited amount of time/resource, we cannot tune each variant during the discussion period.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1/2)

We thank all reviewers for their constructive feedback. We have uploaded a revised manuscript. The changes are shown in red. The changes are:

  • Inference speed (Section 5.1, Appendix F): Softmax attention is now measured with CUDA graphs in vLLM, and prefill computes the last-token logits for both models (BW6a, GV49, 5Xs1).
  • Mechanism ablation (Section 5.3, Table 3, Appendix H): Table 3 is replaced by a three-seed ablation that changes the readout and the input processing separately (pRd8, BW6a, 5Xs1). The original Table 3 is now Table 27.
  • Further variants and baselines (Table 26 in Appendix H): an input-independent query, same-token pairing, Softmax baselines with the short convolution and SiLU of IA, and IA with the normalization denominator (pRd8, BW6a, GV49, 5Xs1).
  • Training time (Appendix G): the step time on a single GPU and a breakdown of the IA forward and backward passes by operation (Table 24) (5Xs1).
  • Number of basis functions (Appendix I): MM is varied at a fixed state size (Table 28) (5Xs1).
  • Figure 4: the figure no longer extends beyond the margin (pRd8).

We also edited the wording of the appendix for readability. These edits do not change the content, so they are not marked in red. For the camera-ready version, we will reorganise the appendix so that the new results are placed more naturally.

We are encouraged that the reviewers found the problem important, the motivation clear (pRd8, 5Xs1), and our novel single-layer hybrid design of IA meaningful (BW6a, GV49, 5Xs1). The main concerns are about the evaluation (baselines, recall, ablations, and speed measurement), which we address with new experiments below.

Summary of contributions

We propose Interdomain Attention (IA), a novel hybrid attention architecture that mixes attention and recurrent memory in the same layer. IA compresses values and kernel features of keys into a recurrent memory parameterized by an SSM. Then IA approximates attention by implicitly reconstructing the attention matrix and values from memory states.

Reviewers found the motivation and problem definition clear (pRd8, 5Xs1) and the presentation clear (pRd8, 5Xs1). They also mentioned that our IA design is meaningful single-layer hybrid design (BW6a, GV49, 5Xs1), the evaluation covers multiple model scales and both language modeling and genomics (BW6a, GV49, 5Xs1), and that ablations support the importance of our attention-like readout (BW6a, GV49, 5Xs1).

Commonly mentioned weakness/questions

1. Missing comparison with modern baselines (BW6a, GV49, 5Xs1, pRd8)

Our baselines (S4D-only and softmax attention) were chosen intentionally to ensure the major improvement of the network comes from IA principles. Softmax and IA differ only in how the memory is maintained: softmax has growing KV cache while IA has the fixed size recurrent state, which corresponds to the coefficients of key features and values projected onto basis functions. S4D-only and IA differ only in how read/write is performed: both use the same S4D recurrence, but S4D-only feeds it linear projections of the input and reads it with linear projection, while IA feeds it key features and values and reads it out with the query in the form of approximate attention. All three share the backbone, data and training recipe, and S4D-only and IA share the same recurrence and state size. Mamba and Gated DeltaNet use different recurrences, so a comparison with them would not isolate the contribution of the IA principles.

2. Poor performance on recall-intensive tasks (GV49, 5Xs1)

Our memory is fixed-size, like other modern recurrent models, so IA shares their difficulty with exact recall. At the same time, IA improves the RULER accuracy over S4D-only (0% → 65.5/28.0/9.5% at 1K/2K/4K), so we conclude that the IA principle improves retrieval performance over the same recurrence. Since IA changes how read/write is performed and not the recurrence itself, we expect that a recurrence with better recall would also improve the recall of IA. As discussed in [1,2], fixed-state models typically perform poorly on exact recall tasks, and IA inherits this limitation. Phonebook is a typical example: both IA and S4D-only have 0% accuracy, but IA still has a higher answer log-likelihood than S4D-only, as in all 14 recall settings. Though we have an attention-like readout structure, it does not mean that we do not suffer from this limitation.

[1] Arora et al., Zoology: Measuring and Improving Recall in Efficient Language Models, ICLR 2024.

[2] Jelassi et al., Repeat After Me: Transformers are Better than State Space Models at Copying, ICML 2024.

AuthorsOpenReview ↗

General response (2/2)

Commonly mentioned weakness/questions (continued)

3. Ablations do not isolate the contribution of the core interdomain design (BW6a, 5Xs1, pRd8)

Thank you for suggesting clearer isolation for ablation study. We have conducted a more detailed ablation corresponding to each suggestion. In the new submission, Table 3 is replaced by this ablation with three seeds (pRd8), and Appendix H gives the details (Table 25); the original Table 3 is now Table 27 in Appendix H. We obtained the following conclusions, which strengthen the effectiveness of the IA principle.

  • Decoupling L2 normalization and convolution (BW6a):
    • Each engineering change from Base IA (removing the key L2 norm, adding value SiLU, and adding value conv.) is small compared with the IA query and readout. In language modeling, the NLL stays within 0.020 of Base IA (Table 3), which is much less than changing the IA readout (at least +0.152, see next bullet point). In genomics, removing the key L2 norm and adding value SiLU change NLL by at most 0.0012, which is much less than changing the IA readout (at least +0.014, see next bullet point). Adding value conv. lowers NLL to 0.011 below Base IA, consistent with Figure 4, where we use this only for genomics.
  • Comparison between query-dependent and query-independent (5Xs1)
    • We added an input-independent query variant in Table 3 that replaces the query with a learned input-independent vector and keeps everything else in Base IA unchanged so the readout can no longer depend on the current token. Over three seeds, this increases NLL by 0.152 in language modeling (2.938 ± 0.005 vs. 2.786 ± 0.012) and by 0.018 in genomics (1.0083 ± 0.0009 vs. 0.9904 ± 0.0012). For comparison, the seed standard deviation of these two variants is at most 0.012 in language modeling and 0.0012 in genomics, so seed variation cannot explain the gaps. When we remove the query entirely and use a linear readout instead, the NLL increases by 0.214 and 0.014 respectively. Therefore, the query-dependent readout has the largest effect in both domains. In language modeling, the input-independent variant is also worse than Softmax attention when trained with the same seed (2.935 vs. 2.838, Table 26, seed 101), whereas Base IA is better (2.791).

4. Inconsistent speed comparison (BW6a, GV49, 5Xs1)

Thank you for identifying this issue. We reevaluated the throughput of softmax attention using vLLM with CUDA graphs, and for prefill we now use the same last-token vocabulary logits for both models. We updated the Inference speed paragraph in Section 5.1, modified Appendix F, and added a single-GPU training breakdown to Appendix G. For smaller batch sizes and shorter sequence lengths, softmax attention is faster than IA, e.g., 4.3 ms per step for softmax attention and 12.3 ms for IA at batch size 1 and 4k tokens, but as we observed with scaled dot-product attention (SDPA), its latency grows as the batch size and sequence length get larger, e.g., at batch size 8 and 16k tokens, IA is 2.2 times faster than softmax attention. For training speed, IA requires 2.82×2.82\times the training time of softmax attention with PyTorch Flash SDPA, as shown in Appendix G (4×4\times GH200 GPUs), and 1.68×1.68\times on a single GPU. A breakdown of the IA operations (Table 24) is newly added. However, as we mention in the paper, though we leverage FLA [3], our implementation is not yet fully optimised, which we leave for future work.

[3] Yang and Zhang, FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism, 2024.

Dear Reviewers,

Please carefully review the authors’ rebuttal and engage in further discussion with the authors to clarify any remaining concerns and questions. Your active participation during the discussion period is greatly appreciated.

Best, AC

AuthorsOpenReview ↗

Summary of the discussion

Dear Area Chair,

Thank you for handling our submission. We briefly summarize the discussion.

Revision

We added

  • a three-seed mechanism ablation (Table 3)
  • further variants and baselines (Table 26)
  • a new speed measurement of softmax attention with vLLM and CUDA graphs (Section 5.1, Appendix F)
  • a breakdown of the training time (Table 24)
  • an ablation of M at a fixed state size (Table 28)

Remaining points

  • Comparison with other fixed-state models such as Mamba-2 and Gated DeltaNet (5Xs1). We chose S4D intentionally, since it is the simplest SSM that IA can be applied to. IA and S4D-only share the same recurrence and state size, so the improvement comes from IA alone (#1 of the general response). A comparison with these models under matched conditions is not included. We will work on the suggestions of Reviewer 5Xs1 for the camera-ready version.
  • Hyperparameter tuning of the ablation variants (pRd8). All variants use the same hyperparameters, and the main effects over three seeds are much larger than the seed variation (Table 3). We agree that tuning each variant would make the ablation more reliable.

Reviewers

  • 5Xs1 (rating 2 → 3) wrote:

    as my other concerns have largely been addressed, I am willing to raise my score to 3

  • pRd8 (rating 3) wrote:

    my current score also reflects that this paper should be accepted in general

  • BW6a (rating 3) and GV49 (rating 2) have not replied to our rebuttal. Their main concerns (baselines, ablations, recall, and the speed measurement) are answered in #1–#4 of the general response.

Best regards,

Authors of Submission 31946

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.