acceptodds
Withdrawn from ICLR 2027

Attention Is Not All You Need Without Structure: Neuro-Symbolic Multi-Head Attention

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformer architectures rely on Multi-Head Attention (MHA) as the central mechanism for modelling interactions between tokens in sequence data. Although this is effective for capturing statistical dependencies, the standard MHA lacks explicit structural inductive bias for relational reasoning, which limits systematic generalisation for tasks requiring multi-step inference. Prior neuro-symbolic methods attempt to fix this limitation through external reasoning modules or discrete symbolic pipelines, which can complicate integration with modern transformer architectures and break end-to-end differentiability. In this work, we introduce a role-conditioned extension of MHA that injects structured relational inductive bias directly into the attention computation while preserving full end-to-end differentiability along with transformer compatibility as a drop-in replacement. The proposed mechanism augments standard attention with a parallel role-conditioned projection pathway based on learned role embeddings, combined through a learnable fusion gating mechanism. Additionally, a role-relational attention bias enables symbolic attention weights to depend on interactions between inferred token roles, allowing relational structure to influence attention scores directly. We evaluate the proposed architecture on language modelling, relational reasoning, and compositional generalisation benchmarks, including WikiText-2, LAMBADA, CLUTRR, and SCAN, using multi-seed experiments with paired statistical testing. Results demonstrate consistent reductions in perplexity on language modelling benchmarks, substantial improvements on relational reasoning tasks, and improved probabilistic modelling on compositional tasks. Ablation studies confirm that the observed gains arise from role-conditioned structural bias rather than architectural duplication alone. These findings demonstrate that structured inductive bias can be embedded directly within attention mechanisms, enhancing Multi-Head Attention while preserving end-to-end optimisation and transformer-native design.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Multi-Head Attention, Neuro-symbolic AI, Transformer Architectures

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_chdaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes "Neurosymbolic Multi-head attention",. A "role embedding" is computed as R=sofmtax(XW)PR = \mathrm{sofmtax}(X W) P, where WW and PP are learned. A "role-conditioned" representation is computed as Xsym=X⊙RX_{sym} = X \odot R, where ⊙\odot is the elementwise product. Then, "neurosymbolic multi-head attention" is computed as a softmax of of inner products between XsymX_{sym} added to inner products of RR. This is framed as capturing a "role-conditioned structural bias". The proposed architecture is evaluated on a number of benchmarks and tassks.

Critical strengths
  • The paper seeks to contribute to the literature on structural inductive biases in neural networks and attention mechanisms, which is an area that is of interest to the ICLR community.
Critical weaknesses

There are several significant conceptual and experimental issues.

  • The paper presents an incomplete and inaccurate view of the prior literature. It frames prior work as being limited to “external modules” or “fixed structural signals”. This is inaccurate and ignores a large literature on learned, in-attention structural biases. See the list below, generated with the help of AI, for a selection of relevant work that is not discussed or address. Of relevance, [1,2] propose a mechanism which is essentially the role embedding of Eqn 2. These works are neither discussed nor compared against in the experiments.
  • The experimental methodology and results suggest significant issues regarding soundness. The proposed architecture appears to have over twice the parameter count and compute requirements of the standard attention baseline. Many claimed improvements are very small (e.g., in SCAN 30.930.9 vs 30.730.7, and in LAMBDADA 20.2020.20 vs 20.2820.28). Additionally values reported are not comparable to the literature, and suggest methodological issues. For example, the perplexity reported for WikiText-2 is on the order of 500 whereas typical values in the literature are 20-100. This suggests severe implementation or methodological issues.
  • Several choices in the architecture design are generally not very well motivated or justified, and it is unclear in what formal sense the architecture is "neuro-symbolic". In several places, an architectural component is described then given a label that implies a neurosymbolic function (e.g, "role", etc.), but there isn't a clear argument for why this should be interpretted as functionally analogous to neurosymbolic computation, and there isn't mechanistic interpretability evidence that it learns such a functional role.

References

[Generated with the help of AI when prompted to find the most relevant work on relational and symbolic inductive biases in transformers. Listed most-recent first.]

  1. Altabaa, A., & Lafferty, J. (2025). Disentangling and Integrating Relational and Sensory Information in Transformer Architectures. ICML 2025.
  2. Altabaa, A., Webb, T. W., Cohen, J. D., & Lafferty, J. (2024). Abstractors and Relational Cross-Attention: An Inductive Bias for Explicit Relational Reasoning in Transformers. ICLR 2024. arXiv:2304.00195.
  3. Mittal, S., Raparthy, S. C., Rish, I., Bengio, Y., & Lajoie, G. (2022). Compositional Attention: Disentangling Search and Retrieval. ICLR 2022. arXiv:2110.09419.
  4. Lamb, A., He, D., Goyal, A., Ke, G., Liao, C.-F., Ravanelli, M., & Bengio, Y. (2021). Transformers with Competitive Ensembles of Independent Mechanisms. arXiv:2103.00336.
Questions for authors
  • Please compare your experimental results to results appearing in the literature and explain the discrepency.
  • Please discuss and compare your work to prior work on structural inductive biases in transformers.
3 comments

Thank you for the review. We answer the reviewer’s questions and then the three weaknesses. Two of the points, the parameter count and the perplexity, rest on numbers that we can clarify.

Note: It must also be noted that during the revision of the paper, we identified an error in the code for the CLUTRR experiments that affected both the MHA and NS-MHA results. Results for CLUTRR have now been updated accordingly in the revised paper.

Question 1: results compared with the literature. Our perplexities are not comparable with published WikiText-2 numbers, and the revised paper says so (Section 4.1). Ours are per token over GPT-2 subword tokens, from a single layer model with no positional encoding, trained on separate lines cut at 128 tokens. Published values of 20 to 100 come from much stronger models, longer context and usually word level perplexity. We also report the final epoch, when validation loss has already started to rise (Figure 2), so our number is higher than the best epoch. These differences explain the gap. Both standard MHA and proposed NS-MHA models use the same evaluation code and settings, so the comparison between them, which is what the paper is about, is not affected. We would also like to clarify that we do not claim a competitive language model.

Parameters and compute: The attention layer of NS-MHA has about 1.8 times the parameters of the standard layer, but the full model has only 1.5% more parameters on WikiText-2, LAMBADA and SCAN and 2.4% more on CLUTRR, because the embeddings and the output layer make up most of it (Table 2). The extra compute is a constant factor of about two on the attention term only, with the same asymptotic cost (Section 5.5). As a result, the model does not have twice the parameters or twice the compute overall.

Question 2 and Prior work: We agree with the reviewer that the mechanism in Eq. 2 has close precedent, most of all the learned library of symbols in Abstractors and the parallel pathways of the Dual Attention Transformer. Our initial version did discuss several methods that put structure inside attention (Relational Attention, the TP Transformer, the Structure Aware Transformer, Slot Attention), but it missed the papers listed by the reviewer and its summary sentence was too broad. The revised Section 2 now covers Abstractors, the Dual Attention Transformer, Compositional Attention and Transformers with Competitive Ensembles of Independent Mechanisms. The paper does not claim a learned role bank as new. Its claim is the combination of components inside one differentiable layer, and an experimental comparison with these methods is left as future work, as explained in the revised limitations section (Section 5.6).

Small improvements: We agree with the reviewer and the revised paper no longer calls them improvements. On WikiText-2 and LAMBADA it states the differences as NS-MHA matching MHA. On CLUTRR the accuracy is similar for both models (14.69 and 14.73 percent), and per token accuracy on SCAN is about 31 percent for both.

The neuro symbolic label: We agree that the paper gave no evidence for it, and the revised version uses the term as motivation only. It is now stated that we have no evidence that the learned roles correspond to symbolic or linguistic categories, and that an equally accurate name is role conditioned structured attention. Analysing what the roles capture is listed as a future work.

With the claims now limited to what the experiments show, we hope the soundness concerns are addressed.

Thank you for the authors for their rebuttal. Please seem my responses to each point below.


Question 1: results compared with the literature

Our perplexities are not comparable with published WikiText-2 numbers, and the revised paper says so (Section 4.1). Ours are per token over GPT-2 subword tokens, from a single layer model with no positional encoding, trained on separate lines cut at 128 tokens. Published values of 20 to 100 come from much stronger models, longer context and usually word level perplexity. We also report the final epoch, when validation loss has already started to rise (Figure 2), so our number is higher than the best epoch. These differences explain the gap. Both standard MHA and proposed NS-MHA models use the same evaluation code and settings, so the comparison between them, which is what the paper is about, is not affected. We would also like to clarify that we do not claim a competitive language model.

This seems like a significant issue with the experimental methodology. You should use an experimental design that aligns with the literature so that it can be compared meaningfully. Single-layer models without positional encoding is poor experimental design for the goals of this experiment.


Parameters and compute:

The attention layer of NS-MHA has about 1.8 times the parameters of the standard layer, but the full model has only 1.5% more parameters on WikiText-2, LAMBADA and SCAN and 2.4% more on CLUTRR, because the embeddings and the output layer make up most of it (Table 2). The extra compute is a constant factor of about two on the attention term only, with the same asymptotic cost (Section 5.5). As a result, the model does not have twice the parameters or twice the compute overall.

Again, this highlights a significant issue in experimental design. The embedding layer should not account for >90% of the model's parameters.


** Question 2 and Prior work**

Thank you for revising the paper to improve the discussion of prior work.


Overall, this paper has major experimental design issues, confirmed by the rebuttals. These issues seem fundamental and do not seem fixable during the rebuttal stage. I encourage the authors to redesign and re-run their suite of experiments to test their ideas properly. Additionally, the paper should describe these experimental details so that the reader can evaluate its soundness and interpret the reported results in the proper context.

We would like to sincerely thank the reviewer for their time and effort put in providing us with constructive feedback for our work. We acknowledge the limitations/concerns raised. Given the lack of time to address all of them in this rebuttal, we will use them to improve our work in the future.

Reviewer_gFg6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Neuro-Symbolic Multi-Head Attention (NS-MHA), which extends conventional attention with a parallel role-conditioned pathway. It uses learned role embeddings to modulate token representations, introduces a pairwise role-relational attention bias, and combines the resulting attention outputs through a learned gate. The authors evaluate the architecture on WikiText-2, LAMBADA, CLUTRR, and SCAN. However, improvements on language modeling are relatively small (except maybe for CLUTRR), and the method does not improve SCAN accuracy. Although the results suggest that additional role-conditioned capacity may help relational tasks, it is unclear whether the improvement comes from a meaningful symbolic inductive bias, additional parameters, or simply a more expressive attention mechanism.

Critical strengths
  1. The architecture is simple and easy to integrate. Keeping the standard attention pathway while adding a role-conditioned one is a practical design. The entire mechanism remains differentiable and can theoretically replace existing attention layers without requiring an external symbolic reasoning system.

  2. The authors attempt to isolate the contribution of role embeddings. Including a dual-attention baseline without role embeddings is a step in the right direction.

Critical weaknesses
  1. The idea of incorporating relational or symbolic representations directly into attention has already been explored in several papers, including TP-attention (which the authors cite), Relational Attention [1], and Abstractors [2]. It is unclear to me what the proposed architecture brings above and beyond recombining existing ideas. I do not clearly see what new capabilities the proposed architecture introduces or how it improves over these methods. Moreover, that are several studies that are highly related to this work and are not included, even to position better the paper with respect to the state-of-the-art, see [2,3,4]

  2. The empirical observations are somewhat underwhelming, and hide behind statistically significant results which only compare against standard attention. Moreover, the only strong result is on CLUTRR, and it’s unclear whether these are done on sequence length that exceed those of training (i.e., a true test of compositional generalization).

  3. Unclear whether the gains come from symbolic reasoning or simply more parameters. The architecture adds a substantial number of parameters, but the authors don't compare against a parameter-matched standard Transformer. More importantly, there's no evidence that the learned roles actually represent symbolic relationships or that these representations are responsible for the improvements. Without this, the symbolic pathway could just be another way of increasing model capacity. In fact, table 3 shows that with or without RE essentially changes nothing to their stronger empirical argument.

[1] Diao, C., & Loynd, R. (2023). Relational Attention: Generalizing Transformers for Graph-Structured Tasks. International Conference on Learning Representations (ICLR 2023).

[2] Altabaa, Awni, and John Lafferty. "Disentangling and integrating relational and sensory information in transformer architectures." arXiv preprint arXiv:2405.16727 (2024).

[3] Altabaa, Awni, et al. "Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in transformers." International Conference on Learning Representations. Vol. 2024. 2024.

[4] Chakravarthy, Ayush K., Jacob Labe Russin, and Randall O’Reilly. "Systematicity emerges in transformers when abstract grammatical roles guide attention." Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop. 2022.

Questions for authors
  1. What can NS-MHA represent that existing relational-attention architectures cannot? In particular, how does it compare theoretically and empirically with TP-attention, Relational Attention, and Abstractors? Without these comparisons, it is difficult to assess whether the proposed formulation advances the state of the art or simply combines existing architectural ideas.

  2. Does NS-MHA actually improve systematic generalization to unseen relational structures? For example, if trained on CLUTRR chains of 2–3 hops, how does performance evolve when evaluated on 4–10 hops? Likewise, how does it perform on SCAN's compositional and length-generalization splits? These experiments seem essential to support the central motivation.

3 comments

Thank you for the review. Most of the reviewer’s concerns ask whether our claims go beyond our evidence. We revised the paper taking this comment into consideration so that our claims don’t go beyond our evidence.

Note: It must also be noted that during the revision of the paper, we identified an error in the code for the CLUTRR experiments that affected both the MHA and NS-MHA results. Results for CLUTRR have now been updated accordingly in the revised paper.

Question 1 and Contribution: The revised paper does not claim greater expressivity, and it no longer presents the work as a performance gain. Its claim is now narrower and supported by the experiments: role conditioned structure can sit inside the attention layer as one differentiable system, trained with the model's ordinary loss, with no external module and no predefined rules, and it replaces standard attention without losing accuracy, at a cost of 1.5 to 2.4 percent more parameters in the full model (Table 2). Relational Attention, Abstractors, the Dual Attention Transformer and Chakravarthy et al. are now discussed in Section 2 as the closest precedents, and the paper does not claim any single component as new. A direct comparison with them is left as future work, as discussed in the revised limitations section (Section 5.6).

Question 2 (Systematic generalisation): We conducted the described experiment: CLUTRR/v1, trained on chains of 2 to 3 hops and tested on chains of 2 to 10 hops, with ten seeds (Section 5.2, Table 3, Figure 5). Accuracy is 14.69% for MHA and 14.73% for NS-MHA, and the configuration trained on 2 to 4 hops gives the same pattern (Appendix A). NS-MHA reaches a lower training loss and a higher validation loss in all ten seeds. We report this as: on this split NS-MHA matches MHA, and neither model generalises to chains longer than those seen in training, so the paper makes no claim about that. The rerun replaces our earlier CLUTRR numbers (see comment above about the error in the code), so the accuracy gap from the first version no longer exists. For SCAN we use the length split only, as a per token benchmark.

Results and significance tests: We agree with the reviewer that the differences are small. The revised paper now states that NS-MHA matches MHA, instead of performing better than it, and the p values are provided as a check on consistency across seeds.

Parameters and roles: NS-MHA has about 1.8 times the attention parameters of MHA (1,854,480 against 1,050,624), which is 1.5 to 2.4 percent of the whole model depending on the benchmark. The two models share the same setup, with only the attention layer swapped, but they are not parameter matched, which is now clearly stated in the revised paper. The paper does not claim that the role structure, rather than extra capacity, explains any difference. Regarding the ablation, we agree with the reviewer that removing the role embeddings leads to a minor decrease in performance. In addition, we have no evidence that the learned roles are symbolic, and the revised paper now states that the neuro symbolic label is motivation only.

Because the revised claims now match what the experiments show, we hope this resolves the soundness concerns.

Thank you for your rebutall. In light of the replies, all my main concern remain. The authors cannot show a clear advantage of NS-MHA over MHA, and do not compare their results verus other proposals. Furthermore, OOD evaluation in the CLUTRR task bearly shows an improvement for NS-MHA over MHA and is again not compared to other proposals. In consequence, I will maintain my score, and I invite the authors to incorporate other proposals, which will allow them to better position their work and show the potential value of NS-MHA more convincingly.

We would like to sincerely thank the reviewer for their time and effort put in providing us with constructive feedback for our work. We acknowledge the limitations/concerns raised. Given the lack of time to address all of them in this rebuttal, we will use them to improve our work in the future.

Reviewer_AbALOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed to add symbolic reasoning structures, including symbolic role embeddings and corresponding attention MHSA modules, to form a dedicated forward pathway in language models and enhance their symbolic reasoning abilities. Their modules are differentiable (and thus gradient optimizable), compared with external auxiliary symbolic modules. They've done experiments comparing with vanilla Transformer structure in a series of tasks, verifying their module's effectiveness.

Critical strengths

A trainable structural variant naturally stands out since nowadays training-free methods are way too predominant. This motivates me to lean more towards acceptance. The paper is also of good clarity, clear to understand.

Critical weaknesses
  1. Shall distinguish the contributions of the added neuro-symbolic module from effects of scaling. Their proposed module likely adds noticeable additional number of parameters. I cannot infer whether the baseline MHSA model uses the exact same total number of parameters compared with their proposed one (and it is likely not the same). I didn't see number of layers specifications, and by the per-layer specifications it seems that NS-MHA has more parameters. The ablation also seems not enough because it only includes an NS-MHA without role embeddings and does not include a matched vanilla transformer. How does the performance of the model compare with an ordinary transformer model (without symbolic attention expansions), but with the same parameter magnitude of the augmented transformer?

  2. Since the authors mentioned their method several times as a differentiable symbolic-reasoning module and stated its advantages over discrete logical operators or external reasoning modules, in the experiments section, a brief comparison with these discrete/external methods is necessary. This is especially important because their statement can be explained two ways: on the one hand, a differentiable module seems like a more seamless integration; on the other hand, it also means harder integration with pre-trained open-source/proprietary models.

Both of them, including the following questions, make me think the paper is below the requirements of an accepted work. But considering the topic, I give an overall "weak reject" assessment.

Questions for authors
  1. Following W2, since the proposed method acts as a parallel forward pathway, could the authors do experiments training with their proposed modules fused in a pre-trained LLM (just like training a LoRA/adapter on the LLM; that only introduces a limited number of tunable parametersc (compared with full-parameter tuning), while the main memory load comes from the momentum and RMS, so restricted computational resources are less likely an excuse)?

  2. I don't want to blame the work for marginal PPL reduction/acc improvement on WikiText-2, LAMBADA, and SCAN, because that sounds too result-oriented, but could the authors provide one or more tasks on which NS-MHA offers substantial performance gains other than CLUTRR? (Also the paper mentioned SCAN as "nearly saturated" but it still is about 31%, could the author explain it?)

3 comments

Thank you for the review, and for saying that a trainable structural variant stands out. That is the claim the revised paper makes: role structure can be built into attention itself, trained end to end with the model's ordinary loss, with no external module, without losing accuracy. We take your questions first.

Note: It must also be noted that during the revision of the paper, we identified an error in the code for the CLUTRR experiments that affected both the MHA and NS-MHA results. Results for CLUTRR have now been updated accordingly in the revised paper.

Question 1 (a pretrained model with an adapter style setup): We agree this is the most useful test of practical value. It is a separate study and outside the scope of this feasibility paper, and it is listed in the revised limitations section (Section 5.6) as future work. The reviewer’s point about integration is fair: the differentiable design avoids an external module, but how best to attach it to a pretrained model needs its own experiments.

Question 2 (tasks other than CLUTRR with substantial gains, and the SCAN remark): The aim of the paper has been reframed as feasibility, not a gain, and the revised paper does not claim one. On CLUTRR we reran the experiment with ten seeds, training on chains of 2 to 3 hops and testing on 2 to 10 (Section 5.2). Both models reach the same accuracy, 14.69% and 14.73% percent, so the paper reports parity, and the higher NS-MHA number in the first version is gone (see note above about issue with code). On SCAN, the reviewer is right to question the phrase "nearly saturated". Per token accuracy is only about 31 percent for both models, so SCAN is not saturated, and we removed that claim. Section 5.4 now describes SCAN as a per token benchmark, not as a test of exact match compositional generalisation.

Weakness 1 (scaling versus the module): We agree with the reviewer that the baseline does not have the same total number of parameters, and the revised paper now states it. The attention layer of NS-MHA has 1,854,480 parameters against 1,050,624 for MHA, and across the whole model the difference is 1.5 to 2.4 percent depending on the benchmark (Table 2). The two models share the same setup, with only the attention layer swapped, and the paper describes the model: a single transformer block with d 512, 8 heads and a feed forward width of 2048. Regarding the ablation, we agree with the reviewer that removing the role embeddings leads to a minor decrease in performance. We no longer claim that the role structure, rather than extra capacity, is responsible for any difference.

Weakness 2 (discrete and external methods): We did not compare against them, and the revised limitations section (Section 5.6) now states so. A comparison would help a reader judge the trade-off between seamless integration and ease of use with pretrained models, and is left for future work.

Reviewer_AbALOpenReview ↗

Response to Author's Response

I thank the authors for their response, while most of my proposed weaknesses/questions are treated as "agreed, noted in the paper" by the authors, so most of them are still unsolved. Also the response gives me a stronger sense (at least stronger than the original paper) that it is AI-generated. Anyway I choose to retain my "Weak Reject" rating.

We would like to sincerely thank the reviewer for their time and effort put in providing us with constructive feedback for our work. We acknowledge the limitations/concerns raised. Given the lack of time to address all of them in this rebuttal, we will use them to improve our work in the future.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.