acceptodds
Under review as a conference paper at ICLR 2027

Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning

Abstract

Sequence modeling tasks can involve intrinsic higher-order dependencies, while standard self-attention assigns scores to token pairs and does not explicitly parameterize such interactions. We introduce Higher-Order Modular Attention (HOMA), which fuses pairwise attention with an explicit triadic attention pathway made tractable through overlapping blocks, local windows, and a low-rank projection. We compare HOMA with matched pairwise and purely triadic baselines on controlled PARITY and MATCH3 tasks, as well as TAPE benchmarks. HOMA is competitive with or outperforms the baselines, with its clearest advantages when the underlying dependencies extend beyond the explicitly modeled triadic order. The learned nonlinear fusion of these two representations further improves convergence speed and parameter efficiency. Overall, our results provide empirical evidence that HOMA is an effective attention design when task structure extends beyond pairwise interactions.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
higher-order attention, self-attention, transformers, sequence modeling, triadic interactions, parameter efficiency, protein sequence modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Bm4KOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper is about higher-order attention. In contrast to standard attention that compares a query with each key, higher-order attention compare any query with multiple keys simultaneously. Conceptually, instead of computing the vector of attention scores (pre-softmax) between query q with c channles and L keys K with c channels by q^T@K (omitting any scale normalization), higher-order attention with three factors (triadic) uses keys K1 and K2 and compute the outer tensor products q_c * K1_{j,c} * K2_{z,c} where c runs over the number of channels. Put differently, this triadic attention procedure is equivalent to calculating the L-dimensional vectors p1=\sqrt{q_c}*K1_{j,c} and p2=\sqrt{q_c}*K2_{z,c} and then computing the outer vectors product between the p1 and p2, p1 \otimes p2 \in \setR{L \times L}.

To reduce the complexity of the resulting L \times L \ times L attention score tensor, the authors propose two clever approximations:

  • They compose the input sequence into overlapping blocks of length l with some stride. Final attention results are obtained by some averaging procedure.
  • The expensive triadic attention within each block is calculated only within a shorter window with size w. Thereby, the computational complexity reduces from O(l^3 d_head) to O(l w^2 d_head).
  • Lastly, queries, keys1 and keys2 in the triadic pathway are not computed with the full nunber of channels d_head but a much smaller dimension to further reduce complexity.

The proposed algorithm is evaluated with synthetic experiments (parity-k, majority-k, match2 and match3) plus a short evaluation on tasks from a protein benchmark.

Critical strengths
  • The proposed triadic attention is sound (I checked the formulas carefully).
  • Intuitively, this much stronger attention might create more powerful transformer layers.
  • I like the ideas proposed here to reduce memory and computational complexities.

Overall, I think this is a very interesting paper. The strong results on parity-k compared with standard pairwise attention are very interesting (see below).

Critical weaknesses
  • A standard argument brought forth against higher-order attention could be: With ever deeper transformers, at some point these very deep transformers achieve the same performance. I suppose this could be correct. Could the authors run a simple test: Train a 1-layer transformer with their triadic attention and compare that with a standard transformer (2-attention) with k={1,2,3,..} layers with some toy data. How many layers are necessary to reach the same performance? It could be interesting to compare the computational complexity of the 1-layer triadic attention model vs the model with k-blocks with standard attention. When does pairwise attention achieve break-even?
  • fig 3a, secondary structure: Why does Blockwise-2D catch up with Blockwise-3D and HOMA for d_model=256 and 512 but not pairwise-2D? Intuitively, the block partitioning has a similar mechanics as a convolution. Could pairwise attention combined with a suitable convolution achieve a similar performance improvement?
  • It would be interesting to see some numbers on the computational complexities (in TFLOPs) of each model.
Questions for authors
  • please add equation numbers to simplify discussion
  • line 162 "\odot denotes the element-wise product": This seems wrong. Given two matrices A, B \in \setR^{m \times n}, the element-wise product C = A \odot B is also a m \times n matrix. Here, matrix \tilde{V}{jz} must be L \times L \times L \times d_head, thus the authors need something akin to the outer vector product to calculate \tilde{V}{jz}.
  • line 196 "overlap averaging": What is that? Explain that (preferably with pseudo-code) in the appendix.
  • line 199, windowed triadic attention within blocks: "What is meant by "we restrict (j,z) to a local neighborhood of size w<<l"? I suspect that the authors mean that they modify the computation of the unnumbered equation in lines 151-156 - is that correct? If so, define the changed formula.
  • low-rank projection in line 203: If I am not mistaken, this has been proposed by Arora et al. "Simple linear attention language models balance the recall-throughput tradeoff". Please add this reference.
  • The parity-k results are very interesting. Are the authors aware that standard transformers cannot solve this task (Hahn, "Theoretical Limitations of Self-Attention in Neural Sequence Models") and require auto-regression (Perez et al. "Attention is Turing-Complete")? Could the limitation of self-attention be the standard pairwise attention?
  • fig 3a, contact prediction, flourescence: Why is d_model not evaluated for 512?
  • ablations: Is there any explanation why the performance of the model with the triadic pathway drops when the interaction exceeds the window size but the model with HOMA attention still works? Is this because the pairwise attention in HMA is not limited to the window size?
1 comment

Thank you for your feedback. We address each point below.

Weaknesses

  1. We expanded our PARITY depth study (Table 1; Appendix C.1, Tables C.4–C.5), comparing one HOMA layer with 1–12 Pairwise-2D layers at d_model=64 and 32 over 400 epochs. Here are some results:
  • Accuracy break-even: Pairwise-2D needs 2 layers on PARITY-4 and 3 layers on PARITY-5 to reach 0.90 in all seeds, whereas one HOMA layer reaches 1.000 on both.

  • Compute break-even: None within the tested 12 layers. The smallest successful Pairwise-2D models require 17× and 27× the training TFLOPs, 23× and 25× the epochs, and 1.4× and 2.0× the parameters of one HOMA layer on PARITY-4 and PARITY-5, respectively.

  • Increasing depth does not always help: At 12 layers, Pairwise-2D fails to reach 0.90 on either task within 400 epochs, as shown in Tables C.4–C.5 and the learning curves in Figures 2 and C.1.

  1. Yes, overlapping blockwise attention introduces locality, much like convolution. This may explain why Blockwise-2D improves with width on Secondary Structure, where local residue interactions are important. Combining pairwise attention with convolution could offer similar benefits, but we have not tested this.

  2. We added training TFLOPs for PARITY (Table 1; Appendix C), MATCH3 (Appendix D, Table D.3), and TAPE (Appendix F, Table F.2).

Questions

Q1. We numbered all equations and added references throughout the paper.

Q2. We should clarify that \tilde{V}_{jz} = V_j \odot V_z is defined for each pair (j,z). Since V_j, V_z \in \setR^{d_head}, their element-wise product is also in \setR^{d_head}. Across all pairs, \tilde{V} \in \setR^{L \times L \times d_head}.

There is no third sequence index because \tilde{V}{jz} does not depend on the query i, which enters through A^{(3)}{ijz} in O^{(3)}i = \sum{j,z} A^{(3)}{ijz} \tilde{V}{jz}. We have clarified the shape of \tilde{V} after Eq. (8).

Q3. Due to striding, some positions appear in multiple blocks. "Overlap averaging" means that the final output at each position is the average of its outputs from all blocks containing it. We clarified this in the methods and added pseudocode in Appendix I.

Q4. Correct. The window restricts both j and z to within R_w = (w−1)/2 of the query, giving at most w² pairs. We clarified Section 3.2 and added the modified equations in Appendix J.

Q5. We now cite Arora et al. (2024). While our low-rank factorization of W^{(U)} in Eq. (12) is related to BASED, the purpose differs. BASED reduces the query and key dimensions to bound the recurrent state size, whereas our factorization reduces the number of parameters while projecting back to the full dimension, leaving the triadic computation unchanged.

Q6. We cite Hahn (2020) in the introduction. However, Hahn's limitation concerns asymptotic behavior as sequence length grows, while Pérez et al.'s result relies on autoregressive computation. Our PARITY-k task instead involves local k-bit parity at fixed length, with L=16 and k ≤ 5, solved by non-autoregressive encoders in a single forward pass. In this setting, the distinction is one of learnability rather than impossibility. We extended PARITY-k training to 400 epochs and found that Pairwise-2D can learn PARITY-4/5 with 2–6 layers in at least two of three seeds, but a single layer reaches only 0.629/0.538, compared with HOMA's 1.000. On MATCH3, Pairwise-2D never reaches 0.90 at widths up to 256, whereas several triadic variants do.

These results suggest that HOMA provides an inductive bias that makes higher-order interactions easier to learn, without implying that pairwise attention cannot learn them or that HOMA removes Hahn's asymptotic limitation.

Q7. Figure 3 compares the width required for pairwise attention to match HOMA's performance. We extended the width range where needed and computationally feasible. On Secondary Structure, this required d_model=512, as Blockwise-2D was still improving substantially at 256, from 0.552 at 128 to 0.622 at 256. On Fluorescence, all models plateau around d_model=128–256, and both pairwise baselines nearly match HOMA at 256, with Pairwise-2D reaching 0.655, Blockwise-2D 0.656, and HOMA 0.657. On Contact Prediction, HOMA reaches 0.134 at d_model=128 and maintains this performance at 256, while Pairwise-2D reaches only 0.088 at 256. Contact Prediction was also our most computationally expensive task, so extending it to d_model=512 was beyond our available compute budget.

Q8. Yes. In these experiments, the pairwise pathway attends to the full sequence and is therefore not restricted by the triadic window. When interactions extend beyond this window, Blockwise-3D drops to chance, approximately 0.50, while HOMA can still rely on its pairwise pathway and performs at least as well as Pairwise-2D, as shown in Table G.1. HOMA-add shows the same behavior despite having no fusion MLP, suggesting that the pairwise pathway prevents this collapse.

We would be happy to address any more questions or comments.

Reviewer_65K4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Higher-Order Modular Attention (HOMA), which arguments the pairwise attention with an explicit triadic pathway. To address the cubic complexity, the authors proposed overlapping sequence blocks with local attention window and low-rank factorization.

Experiments were conducted on three benchmarks, including PARITY, MATCH3 and three TAPE tasks.

Critical strengths
  1. The idea of combining rather than replacing pairwise attention is a clear and well-motivated design choice

  2. The synthetic experiments on PARITY and MATCH are well designed and the experimental results support the claim that explicit triadic interactions are easier to learn under limited width, depth, and optimization budgets.

  3. The experimental setup and hyper-parameters are provided, making the results reproducible.

Critical weaknesses
  1. The protein experiments did not ablate higher-order interaction from local attention patterns. On secondary structure, global Pairwise-2D performs very poorly, while simply switching to overlapping local blocks produces a large improvement. Thus, much of the gain over standard attention is attributable to the blockwise/local inductive bias. The incremental HOMA gain over the strongest relevant controls is considerably smaller.

  2. The efficiency claims are not fully supported. HOMA makes triadic attention tractable, but it is not generally efficient relative to pairwise attention.

  3. The evaluation uses small models trained on three TAPE tasks. Experiments on other natural data, such as languages and images, are highly recommended.

Questions for authors

NA

1 comment

Thank you for your time and consideration. Below, we address the weaknesses you raised.

W1:

Blockwise-2D uses the same overlapping blocks as HOMA, allowing us to compare the mechanisms with locality held fixed. We do mention that, with increased width, Blockwise-2D catches up with HOMA and Blockwise-3D. The clearest evidence of this trend is on the Secondary Structure dataset. HOMA improves accuracy by 0.09–0.10 at smaller widths, but the gap decreases to 0.017 at d_model=512. However, note that Blockwise-2D requires up to 52× more parameters to match HOMA's accuracy.

W2:

HOMA is not cheaper per training step. At matched widths, it requires 1.2–3.3× more training TFLOPs than Blockwise-2D and uses more memory.

Our efficiency claims concern three aspects:

  • Scalability: Local triadic attention scales linearly with sequence length for fixed window sizes.
  • Parameter efficiency: On Secondary Structure, Blockwise-2D requires up to 52× more parameters to match HOMA.
  • Convergence: On PARITY-4, HOMA uses 0.08 TFLOPs to reach 0.90 accuracy, compared with 1.30 for Pairwise-2D.

These advantages are not universal. Blockwise-3D is cheaper at greater depths on PARITY, while F&S is more computationally efficient on MATCH3.

W3:

Our goal is to test whether explicit triadic interactions help with higher-order dependencies under controlled conditions. Testing on protein datasets was well motivated because they contain multi-residue interactions and TAPE provides standard benchmarks.

We evaluate three protein tasks across multiple widths and three seeds. Larger-scale experiments in language or vision are beyond our current compute budget. We acknowledge this limitation and do not claim that the findings generalize beyond the evaluated settings.

We would be more than happy to answer any questions you may have.

Reviewer_xMpDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Higher-Order Modular Attention (HOMA), which combines pairwise and explicit triadic attention through learned nonlinear fusion. Overlapping blocks, local triadic windows, and a low-rank projection make the higher-order pathway tractable. Comparisons with pairwise and purely triadic variants on synthetic PARITY and MATCH tasks show improved learning and parameter efficiency in several higher-order settings. Evaluations on three TAPE protein-modeling tasks show task-dependent gains. The study also includes capacity sweeps, component ablations, and throughput and memory measurements.

Critical strengths
  • The paper poses a focused, testable research question and keeps its claims appropriately measured.
  • Synthetic experiments demonstrate clear learning gains on higher-order tasks, supported by lower-order controls, capacity sweeps, and convergence curves.
  • Protein-modeling evaluations extend the evidence beyond synthetic tasks, with variability reported across three seeds.
  • Parameter, throughput, and memory reporting makes the computational trade-offs transparent.
Critical weaknesses
  • W1: The comparisons do not fully separate triadic interactions from added parameters and nonlinear processing. This matters especially in the small diagnostic models, where pairwise baselines omit the standard MLP but HOMA retains nonlinear fusion. HOMA-ADD does not test whether a pairwise model with comparable nonlinear capacity could achieve similar gains.
  • W2: The practical significance of the protein-task gains remains uncertain. The TAPE experiments provide useful controlled evidence, but do not establish whether the benefits extend to workflows using pretrained protein representations, limiting the demonstrated application impact.
  • W3: The empirical study is coherent, but the broader insight and architectural advance over existing pairwise–triadic hybrids need clearer articulation. The comparisons demonstrate benefits over individual pathways, but leave unclear what is learned beyond the effectiveness of this particular combination.
Questions for authors
  1. How does HOMA compare with a pairwise baseline matched in total parameters and equipped with comparable nonlinear processing?
  2. Do diagnostic baselines catch up with additional training or reasonable hyperparameter tuning, or do the gaps persist after convergence?
  3. What evidence supports the relevance of the TAPE gains to practical protein-modeling workflows, particularly those using pretrained representations?
  4. How were the training hyperparameters selected, using which data and search budget? Did each attention mechanism receive comparable tuning?
  5. What is HOMA’s specific advance over earlier pairwise–triadic hybrids, particularly the 2-simplicial Transformer? Please distinguish the new interaction, approximation, and fusion components and explain their added value.

Minor comment:

  • Summarize the TAPE architecture in the main setup. The appendix provides details, but stating the depth, FFN, and task heads earlier would improve readability.
1 comment

Thank you for your detailed feedback and questions. We address each point below.

Q1 (W1).

HOMA-add ablates nonlinear fusion by adding the pairwise and triadic outputs without the fusion MLP. With fewer parameters than HOMA (19,202 vs. 25,490), HOMA-add still solves PARITY-4 and PARITY-5 with one layer (accuracy 1.000 and 0.990).

This alone does not establish that a matched pairwise model would perform worse. We added this control in Appendix C.1, Tables C.4–C.5. We test Pairwise-2D + MLP, which adds a residual MLP sublayer after each attention layer, trained under the same 400-epoch protocol as Table 1. Exact parameter matching is not possible, so we compare against larger baselines, favouring them.

With one layer, Pairwise-2D + MLP has twice HOMA's parameters (51,394). Even so, it reaches 0.90 in only one of three seeds on PARITY-4 and in none on PARITY-5.

With two layers, Pairwise-2D + MLP has four times HOMA's parameters. It reaches 0.90 in all seeds, but needs 25–39× the epochs and 52–79× the training TFLOPs of one HOMA layer.

This suggests that extra parameters and nonlinear processing let a pairwise model learn the task eventually, but they do not replace the triadic pathway.

Q2 (W1).

We extended PARITY training from 40 to 400 epochs (Table 1; Appendix C.1, Tables C.4–C.5). A single Pairwise-2D layer still struggles on PARITY-4 and PARITY-5, reaching only 0.629 and 0.538 accuracy. Deeper pairwise models can reach 0.90, but the smallest models that succeed in all seeds require 17× and 27× the training TFLOPs of one HOMA layer on PARITY-4 and PARITY-5, respectively. We have clarified the distinction between learnability and computational efficiency in the revised manuscript.

All mechanisms share the same hyperparameters within each task, including the learning rate, without mechanism-specific tuning. While additional tuning could narrow the gap, our experiments compare the attention mechanisms under identical training settings.

Q3 (W2).

We agree that our TAPE experiments do not establish gains in pipelines using pretrained protein representations. Our goal was to isolate the effect of the attention mechanism by changing only the attention operator while keeping everything else fixed. The TAPE results therefore provide evidence of HOMA's benefits on protein prediction tasks under controlled settings, rather than in pretrained workflows. Since HOMA replaces only the self-attention sublayer, it could also be incorporated into protein language models and pretrained at scale. However, evaluating this remains outside the scope of our work. We clarify this scope in Section 3.2 and identify evaluation with pretrained models as future work in the Conclusion.

Q4 (W1).

Our goal was to compare attention mechanisms under a common training pipeline. Within each task, all models share the same training hyperparameters (Appendix A, Tables A.1–A.3). We tried a few learning rates and configurations and selected settings that gave stable training and reasonable convergence, without using test data for selection. We did not perform an exhaustive grid search or tune each mechanism separately.

For TAPE, block size, stride, and triadic window were chosen based on local protein structure and computational constraints. Learning-rate schedules were used for Fluorescence and Contact Prediction to improve stability.

While mechanism-specific tuning could improve individual baselines, our approach allows a controlled comparison under consistent training conditions.

Q5.

We have expanded our synthetic experiments to include comparisons with the 2-Simplicial Transformer and Fast and Simplex (F&S) on PARITY and MATCH3. HOMA differs from earlier pairwise–triadic hybrids in three ways:

HOMA uses trilinear scores and Hadamard-product values, avoiding the costly third-order value tensor of 2-Simplicial. Unlike F&S, both interacting partners are restricted to local windows.

Overlapping blocks and local windows reduce triadic complexity from O(L³d) to O(Lw²d), while a rank-8 projection limits parameter growth. The pairwise pathway helps capture dependencies beyond the local window.

A learned nonlinear MLP combines pairwise and triadic outputs. On PARITY-4/5, HOMA reaches 0.90 accuracy in 5.3/7.7 epochs versus 12.7/24.3 for additive fusion.

HOMA's clearest gains appear on higher-order PARITY tasks, achieving perfect accuracy on PARITY-4/5 with 25k parameters, compared with 0.644/0.563 for 2-Simplicial with 29k. However, these gains are task-dependent, as F&S outperforms HOMA on MATCH3 at N=8.

Regarding minor comment: Done. We have added a short summary of the TAPE architecture to Section 3.2. We would be happy to address any further questions or comments you may have.

Comments on the paper

1 comment

We noticed a masking issue in the MATCH3 experiments that affected Blockwise-3D, HOMA-add, and HOMA. We fixed it and reran the affected experiments using the same settings and seeds. All updated results are now included in Figure 3 and Table D.1, and we have revised the discussion accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.