Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning

Sequence modeling tasks can involve intrinsic higher-order dependencies, while standard self-attention assigns scores to token pairs and does not explicitly parameterize such interactions. We introduce Higher-Order Modular Attention (HOMA), which fuses pairwise attention with an explicit triadic attention pathway made tractable through overlapping blocks, local windows, and a low-rank projection. We compare HOMA with matched pairwise and purely triadic baselines on controlled PARITY and MATCH3 tasks, as well as TAPE benchmarks. HOMA is competitive with or outperforms the baselines, with its clearest advantages when the underlying dependencies extend beyond the explicitly modeled triadic order. The learned nonlinear fusion of these two representations further improves convergence speed and parameter efficiency. Overall, our results provide empirical evidence that HOMA is an effective attention design when task structure extends beyond pairwise interactions.