acceptodds
Under review as a conference paper at ICLR 2027

Making Transformers Logical

Abstract

Recent work has shown that transformers operating on logical tasks will often fit a simple function that interpolates between observed examples. This means that they often fail to generalize to unseen instances of logical boolean or arithmetic problems. Generalization on the Unseen (GOTU) has therefore become a stress test for understanding the inductive bias of transformers. In this paper, we demonstrate that poor logical extrapolation is not necessarily an inherent property of transformers, and that the desired behavior can be obtained by directly modifying this inductive bias in the network architecture. We show that including a simple attention mask is often sufficient to enable near-perfect extrapolation on boolean tasks and improves sample efficiency on arithmetic tasks. We show that although a similar bias may be encoded by initializing query and key parameters, a mask-based encoding is preserved during training. We additionally explore how a similar mechanism may be applied in vision. These results show the potential for using mask-based priors to encode logical inductive biases in transformers.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Transformer, Inductive Bias, Mask Transformer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_c4GWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how attention maskes can enabled "Generalization on the Unseen (GOTU)" in transformers learning logical functions. Prior theoretical work on GOTU (Abbe et al. 2024) showed that transformers learn a minimum-degree interpolator, a specific inductive bias that determines whether transformers can generalize to unseen inputs when learning logical functions. This paper studies whether incorporating an additive mask, QKT+log⁡σ(M)Q K^T + \log \sigma(M), with MM initialized by hand with knowledge of the interaction structure of the task, can modify the inductive bias and lead to generalization to unseen examples in logical functions. The paper shows success in generalization across several logical functions.

Critical strengths
  • The question this paper studies is very clear, specific, and is well-posed. I personally like the Abbe et al. 2024 paper, so I'm glad to see follow-up work on the topic, including empirical work.
  • The observation that the way the inductive bias is encoded (e.g., via the query/key params vs additive mask) can be critical to the learned solution at the end of learning, even if it has similar behavior at initialization.
  • The paper's results are clear and clean. GOTU goes from 0% to 100% on severl boolean tasks.
Critical weaknesses

Overall, while the paper seems sound, it is very modest in scope, and the contributions are limited.

  • The masks are chosen by hand, so they're not really "learned". They explicitly constrain the task structure into the model. This makes successful generalization a somewhat unsurprising finding, and makes the significance limited.
  • The experiments are quite limited in scope.
  • A key point emphasized in the paper is the comparison between encoding the inductive bias through the query/key matrices vs a mask, showing that the mask's effects persist throughout training whereas the query/key matrices get "washed out". However, the fact that the mask persists is almost guaranteed by construction, since with the choice of initialization (App G.1) the mask is essentially a hard mask with near-zero gradient.
  • The "local gradient-level" analysis (contribution 3) is modest and shallow. It's just computing a derivative and a norm bound.
Questions for authors
  • How does a hard mask compare to the results you report? Given that the mask is initialized to be nearly hard with near zero gradients, what is the significance of keeping it differentiable and learnable at all?
  • What happens when the mask is misspecified? for example extra or missing allowed interactions?
  • Do you have an empirical or theoretical explanation for how the mask avoids the minimum-degree interpolating solution (e.g., by building on Abbe et al's analysis with the masks), and whether you can characterize the modified inductive bias with masking?
3 comments

We thank the reviewer and are pleased that they found the question studied clear, specific, and well-posed. The Abbe et al. (2024) paper is a key inspiration for our work and we are glad they have encouraged follow up work in this direction.

Q: How does a hard mask compare to the results you report?

The second contribution in our paper is to highlight the difference between encoding inductive priors through an attention mask and through initialization. In prior literature (see: Zheng et al. 2025, Structured Initialization for Vision Transformers) it was shown that incorporating a convolutional prior by initializing Q and K matrices in attention can improve on sample efficiency in vision problems.

We show that (because this prior is non-persistent) it can be washed out during training, particularly in the GOTU setting where a key extrapolation feature is not observed during training (see Figure 7). This makes encoding a prior more applicable with a logical mask than encoding the same attention prior through initialization.

To show how this relates to prior literature and a less structured domain, we demonstrate it on a vision problem (Section 5.3 Image Classification). As this is less structured than boolean problems (with explicit token interactions) explored earlier in the paper, we expect that a 'soft' mask will have improved performance over a hard mask. We test this in Appendix (B2) and show the learned logical mask has an improvement over a strict hard logical mask.

DatasetDefaultMimetic (Init)Impulse (Init)Hard Logical MaskLearned Logical Mask
CIFAR1093.9394.98 (+1.05)95.03 (+1.10)94.60 (+0.67)95.95 (+2.02)
CIFAR10077.2679.01 (+1.75)79.62 (+2.36)79.84 (+2.58)82.06 (+4.80)
ImageNet10083.4784.55 (+1.08)83.51 (+0.04)83.87 (+0.40)85.83 (+2.36)

What happens when the mask is misspecified? For example extra or missing interactions?

We thank the reviewer for this suggestion. To test this we look at the 2-parity experiment.

In this paper we define the 2-parity logical mask to contain the desired f(x)=x1x2f(x)=x_1 x_2 interaction.

[0 1 0]
[1 0 0]
[0 0 1]

We try two misspecifications. 1) Which blocks the interaction term x1x2x_1 x_2, and 2) which includes an additional term. We can see that blocking the desired interaction causes performance on the unseen domain drop to 0%, while an additional interaction causes a significant drop in performance.

  1. Blocked interaction mask
[1 0 0]
[0 1 0]
[0 0 1]
  1. Extra interaction mask
[1 1 0]
[1 0 0]
[0 0 1]
MethodTrain Acc. (%)Unseen Acc. (%)
2-Parity Mask100.00 ± 0.00100.00 ± 0.00
Blocked Interaction Mask100.00 ± 0.000.00 ± 0.00
Extra Interaction Mask100.00 ± 0.0040.00 ± 48.99

Q: Do you have an empirical or theoretical explanation for how the mask avoids the minimum-degree interpolating solution (e.g., by building on Abbe et al's analysis with the masks)?

The intuitive empirical explanation is perhaps best shown through Figure 7 in the paper which visually shows the decision boundaries for the 2-Parity problem. Under the GOTU setting a key part of the domain is unobserved during training. As a result the model will fit the observe parts of the domain, and converges to a simple decision boundary (an interpolative MDI solution). The use of a persistent logical mask enforces the desired token interaction and blocks convergence to the interpolative MDI solution.

In the Appendix we have included a new section (in red) which provides a theoretical explanation for how the mask avoids the MDI solution. By necessity, this is a fairly involved argument directly built on Abbe et al. (2024)'s construction. The intuition behind this approach is that the use of an idealised mask can block lower-order interactions from the pre-activation attention while retaining the desired higher-order interactions (e.g. under two parity, the desired x1x2x_1 x_2 term). When viewed in the spectrum the masked model can directly remove the lower-order Fourier components which characterizes the MDI solution under the Fourier-Walsh transform of the interpolator. Please let us know if you would like any more details on this approach.

Thank you to the authors for their rebuttal. Please see my responses below to each section of their rebuttal.

Q: How does a hard mask compare to the results you report?

It seems the authors may have misunderstood my question, since I do not see a direct answer in their response. My question is simply: the experiments in the paper use an additive "soft mask", but the it is initialized at large enough values that it is nearly a hard mask and it's gradient is very small. In that case, why use a learnable soft mask at all rather than simply encoding it as a hard mask?

What happens when the mask is misspecified? For example extra or missing interactions?

Thank you for these results. They are interesting supplemental results that I think would be interesting to incorporate into the paper. The results are intuitive: if you remove an interaction that is needed, the model fails, if you allow an interaction that is not needed, the model could learn to ignore it but may fail to generalize.

If the authors find this interesting, they could consider varying the "level of misspecification" and seeing whether results improve as the level of misspecification decreases. For example, for a higher-order target function, you could add more and more unneeded interaction terms in the mask, and see whether the model can learn to ignore them. You could try to investigate when the model succeeds vs fails in such settings.

Q: Do you have an empirical or theoretical explanation for how the mask avoids the minimum-degree interpolating solution (e.g., by building on Abbe et al's analysis with the masks)?

Thank you for this response. The intuitive explanation makes sense. I will leave the evaluation of the new theory section in the appendix to reviewer eHSe since they seem to have spent more time on it.


Overall: my impression of this work is that it is an interesting direction to explore, as I mentioned in my review. However, I view the level of contribution to be too modest for a top publication venue like ICLR. While the soundness and presentation seem good, the paper's contributions are limited to simple synthetic experiments in a narrow domain. In its current form, the paper feels more like a strong course project, or perhaps a workshop paper. I encourage the authors to continue to develop this work.

We thank the reviewer for their constructive comments and are pleased that they found this research direction interesting.

How does a hard mask compare to the results you report?

In our experiments, we found that for the Boolean problems, there is indeed no difference between using a hard mask and using large initialization values for an additive mask. However, as the Boolean problem is not the only one that has structure in the dataset, we would like to investigate whether we can inject the desired structure while allowing certain flexibility. We take the vision task as an example to demonstrate that a soft additive mask can have a broader application scenario even if they show similar behaviour on Boolean problems.

What happens when the mask is misspecified? For example extra or missing interactions?

We thank the reviewer for their question. We also find this question interesting and would like to put it into one section in Appendix J, highlighted in red.

Do you have an empirical or theoretical explanation for how the mask avoids the minimum-degree interpolating solution

We extend our previous proof to an idealized multilayer random-feature model in Section 6 and Appendix I, highlighted in red. For each target monomial within the supported degree range, we can construct a target-dependent, layerwise mask that restricts each masked feature’s Fourier support to the desired monomial. The resulting minimum-norm interpolator therefore recovers the target on the interpolator solution space, including both the seen and unseen domains.

Finally, we thank the reviewer for their continuous engagement and constructive feedback, which have helped us improve the paper.

Reviewer_b2ebOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose adding a logical mask to the Transformer’s attention mechanism by introducing the term log⁡σ(M)\log \sigma(M) into the attention logits, (XWQ)(XWK)⊤/d(XW_Q)(XW_K)^\top/\sqrt{d}, to improve logical reasoning performance. They report experimental results supporting the approach. The method offers a simple way to incorporate task-specific logical inductive biases into neural networks.

Critical strengths

The proposed method was simple and easy to understand and also easy to implement. If such logic mask can be identified, this method can improve the reasoning perfomance of transformer-based neural networks. Authors also argue the method can be applied for computer vision tasks.

Critical weaknesses

The main limitation is that the approach assumes prior knowledge of which logical mask to use. The authors use XOR as an example to demonstrate the benefits of introducing log⁡σ(M)\log \sigma(M), but how should the mask change if the task instead involves OR? This raises the need for an additional mechanism to identify an appropriate mask. My first concern is whether such a mechanism exists and can operate without prior knowledge of the target rule.

My second concern concerns the scope of the approach. Although logical masks encode task-specific interaction structures in Transformers, it remains unclear whether they can support systematic reasoning comparable to symbolic systems. Addressing this question requires evaluation beyond aggregate accuracy on sampled datasets, including tests of logical validity, compositional generalization, and robustness to changes in rules and problem structure.

Questions for authors

The motivation of this work seems that neural networks often fail to generalize to unseen instances of logical boolean or arithmetic problems. My question is: who saw the unseen instances before the Transformer started self-attention?

If we have this logic mask, why do we still need self-attention to learn the Q, K, V?

1 comment

We thank the review and are pleased they found our approach simple to understand and implement.

Q: The main limitation is that the approach assumes prior knowledge of which logical mask to use. The authors use XOR as an example to demonstrate the benefits of introducing (log⁡σ(M)\log \sigma(M)), but how should the mask change if the task instead involves OR?

The OR function is f(x)=x1+x2−1f(x) = x_1 + x_2 - 1 which contains no interactions between the first two tokens (x1x2x_1x_2). Therefore, the relevant mask would be:

1000
0100
0010
0001

In contrast, the XOR function is f(x)=x1x2f(x)=x_1 x_2 contains interactions within the first two tokens. The relevant mask is therefore:

0100
1000
0010
0001

Our work focuses on the inductive bias of transformers. In particular, it seeks to answer a question raised in Abbe et al. (2024) "Generalization on the Unseen, Logic Reasoning and Degree Curriculum", namely whether the difficulty of extrapolating logical rules is an inherent property of transformers. We use handcrafted masks to show a counter-example, showing that the "Minimum Degree Interpolator" convergence can be avoided with a structured attention mask. This is the key contribution in our paper.

While the key focus of our paper does rely on prior knowledge of the desired logical interaction, for less structured tasks (e.g. vision) we do include experiments (Section 5.3 Image Classification) to demonstrate how a structured mask can improve upon structured initialization (both encoding a convolutional inductive prior in attention). Zheng et al. (2025), "Structured Initialization for Vision Transformers" provides some context for this direction of research.

Q: Although logical masks encode task-specific interaction structures in Transformers, it remains unclear whether they can support systematic reasoning comparable to symbolic systems.

Prior work (Abbe et al. 2024) had proposed that transformers cannot extrapolate to an unseen domain for certain boolean problems. Our work provides a counter-example to this through the use of structured attention masks. We do not make a claim on supporting systemic reasoning comparable to symbolic systems, though we note that our work provides a pathway for these questions to be explored further in the future.

Q: If we have this logic mask, why do we still need self-attention to learn the Q, K, V? Our mask encodes a prior desired token interaction for a logical task (e.g. addition), however it does not provide the full weighted interaction. This still needs to be learned through gradient descent on the Q, K, V matrices (see: Figure 5, which demonstrates that incorporating the inductive prior for arithmetic tasks through a logical mask improves sample efficiency relative to approaches that encode this prior through structured initialization).

Reviewer_eHSeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses the shortcoming of transformers in generalizing logical rules to unseen inputs. It proposes an attention mask prior that is added to the standard query-key outer product in logit space before softmax. According to the authors, this prior shall encode interaction strength between tokens for fixed, input-independent boolean formulae that are known ab initio. In essence, each logit entry zijz_{ij} in the outer product QKTd\frac{QK^T}{\sqrt{d}} is multiplied by a sigmoid gate σ(mij)\sigma(m_{ij}) after exponentiation, where mij∈Rm_{ij}\in\mathbb{R} is a learnable parameter. The softmax attention matrix AA hence now holds entries Aij=exp⁡(zijσ(mij))∑kexp⁡(zikσ(mik)).A_{ij}=\frac{\exp(z_{ij}\sigma(m_{ij}))}{\sum_k \exp(z_{ik}\sigma(m_{ik}))}.

The authors show that the partial derivatives w.r.t. mijm_{ij} are not affected by the dimension rescaling 1d\frac{1}{\sqrt{d}} and hence retain larger gradients norms than the gradients that pass through the query and key embeddings qi,kjq_{i},k_{j}.
Empirically, the authors compare their prior against a default initialization baseline and two initialization schemes that were proposed in a recent paper for Vision Transformers in particular. They evaluate these initialization schemes on three types of datasets: 1. logical formulae with two or three inputs, 2. integer arithmetic (addition and multiplication), and 3. image classification datasets CIFAR-10, CIFAR-100 and ImageNet100. Their method substantially outperforms the other initializations on logic and arithmetic tasks. Also, they achieve slight but consistent improvements on the image classification benchmarks.

Critical strengths
  1. The paper is clearly written and easy to follow. The idea is simple and several visualizations underscore its intuition.
  2. The initialization outperforms a few prior methods across logic, arithmetic and vision tasks, partially with substantial generalization benefits.
Critical weaknesses
  1. Their initialization technique does not seem robust and scalable.
  • It is data-independent and assumes a fixed token order, and fully relies on manual engineering or belief encoding. While query and key embeddings would depend on the actual data, the mask inductive bias does not consider the data at all. Swapping the order of tokens would hence immediately break any asymmetric mask prior. Encoding position-specific interactions in the mask hence seems doomed for failure if we want to generalize to input permutations.
  1. There are neither convincing empirical nor theoretical arguments why this mask initialization would scale well to higher-dimensional and more complicated representations.
  • Empirically, evaluation on logic functions is limited to boolean formulae with conjunctive terms of at most three inputs. For higher-dimensional logic, it is unclear how the mask prior could distinguish the presence or absence of higher-order conjunctive terms, since the attention matrix can only encode binary interactions. That way, I don't see how the method lives up to the claim of avoiding the convergence to the minimum-degree interpolator (MDI) that previous methods exhibit.
  • Moreover, their method requires a number of parameters that scales quadratically(!) in the sequence length TT. O(T2)O(T^2) parameters seems prohibitively expensive compared to previous initialization techniques that work with existing O(T)O(T) storage for query-/key-embeddings.
Questions for authors

1a. Requiring hand-designed mask priors does not scale well to difficult and less-structured tasks. Where do the authors see evidence that a significant fraction of challenging modeling tasks exhibit fixed logical patterns? 1b. How shall this initialization technique generalize to permutations of inputs? 2a. Parametrizing each entry in the attention mask with a prior requires a number of parameters that scales quadratically(!) in the sequence length. How shall this scale to larger architectures in contrast to query-/key-initializations, given the memory constraints of even advanced computing hardware? 2b. How would this mask prior effectively encode higher-order interactions of boolean variables or digits? It seems to me that such higher-order interactions can only be realized via careful initializations across multiple layers. Can the authors show empirical evidence that their initialization retains a significantly better test accuracy when fitting higher-order boolean or arithmetic formulae?

7 comments

We thank the reviewer and are pleased that the found our paper clear and easy to follow.

Q: Requiring hand-designed mask priors does not scale well to difficult and less-structured tasks. Where do the authors see evidence that a significant fraction of challenging modeling tasks exhibit fixed logical patterns?

Our work focuses on the inductive bias of transformers. In particular, it seeks to answer a question raised in Abbe et al. (2024) "Generalization on the Unseen, Logic Reasoning and Degree Curriculum", namely whether the difficulty of extrapolating logical rules is an inherent property of transformers. In this paper, we show a counter-example by demonstrating that including a structured attention mask is able to avoid the "Minimum Degree Interpolator" issue. This is the key contribution of our paper.

Additionally, we show that alternative ways of encoding this inductive prior (e.g. through structured initialization of Q and K values) defaults to the MDI solution (see: Figure 7) showing that the attention mask encodes a more persistent prior.

While the focus of the paper is narrowly on exploring the inductive bias of transformers (showing how logical priors can be encoded), we do include an experiment on less structured vision tasks (Section 5.3 Image Classification) to demonstrate how a structured mask can improve upon structured initialization (both encoding a convolutional inductive prior in attention). Zheng et al. (2025), "Structured Initialization for Vision Transformers" provides some context for this direction of research.

Q: How shall this initialization technique generalize to permutations of inputs?

For the problems we are interested in the token order matters. For example, in the 2-party problem, we have a Boolean sequence of 15 values. The goal is to learn whether the first two values is the same or different (the XOR problem). The training set only contains examples with only three of the leading sequences, e.g. 00…, 01…, 10… Examples with the final case of 11… are unseen.

Permuting these tokens would change the expected output (e.g. 00101 = 1 is different from 10001 = 0)

For arithmetic problems such as addition and multiplication token order also matters (e.g. [1][2]x[8][3] is different from [3][1]x[2][8])

Parameterizing each entry in the attention mask with a prior requires a number of parameters that scales quadratically in the sequence length

The key contribution for our paper is to show that the convergence to a minimum degree interpolator (MDI) for Generalization on the Unseen (GOTU) can be avoided by encoding a logical bias in a persistent mask. While an efficient implementation is out of scope for this paper, the efficiency of this implementation could be improved in two ways: 1) Generating masks algorithmically, 2) By using an approximate mask through a MLP. In Appendix D we do show that using a MLP as a mask parameterization can extend to problems of additional length without additional parameters.

Q: How would this mask prior effectively encode higher-order interactions of boolean variables or digits?

For higher-order logical interactions this could be encoded through a mask enabling additional token interaction. For example, in Cyclic-5 f(x)=x0x1x2x3x4+x1x2x3x4x5+… )f(x) = x_0x_1x_2x_3x_4 + x_1x_2x_3x_4x_5 + \dots) we can encode a cyclic 5 token interaction. This extends the cyclic-3 interaction we include in the paper.

This mask would look like:

11111000
01111100
00111110
00011111
10001111
11000111
11100011
11110001

We test this cyclic-5 experiment. Consistent with the cyclic-3 experiments, including the persistent logical mask improves extrapolation performance on the unseen component.

MethodTrain Acc. (%)Unseen Acc. (%)
No Mask99.05 ± 0.80.88 ± 1.04
Cyclic-5 Mask100.00 ± 0.0199.90 ± 0.00

Thank you for your clarifications, concessions about the scope of the paper and the additional experimental results.

Concessions about the scope

I now understand that this paper focuses on providing evidence against the suggestion of Abbe et al. (2024) that Transformers inherently struggle to extrapolate logical rules and bias towards a Minimum-Degree-Interpolators of the training data. Also, I now understand that this paper does not(!) claim to present a method that achieves computational efficiency and also only consider problems where well-ordered inputs are provided, which considerably limits the transferability of the presented methodology.

Sufficient insights against the MDI hypothesis?

The remainder of my reply will now focus on the more narrow scope of the paper to show evidence against the MDI Hypothesis.

The additional experiments on Cyclic-5 seem like a strong case for the ability of the attention mask prior to encode interactions beyond the degree of 2. This surprises me as the attention mask only weighs how the value components vjv_j are added. Representing multiplication of several terms via addition should only be possible with a log-space transform, but this has to be learned by the network in earlier layers. But why would the network only be able to achieve this with this strong prior?

The author's finding is surprising and interesting to me. I encourage the authors to include these higher-order experiments in the main paper.

Now is this finding alone sufficiently relevant for the research field to justify publication at this conference? I am still uncertain about that, since the paper does not provide any revealing theoretical argument or intuition for why this higher-order extrapolation on unseen inputs becomes possible. The authors only raise an argument about larger gradients, but this argument feels even misleading considering that their strong mask initialization (see Table 13) will likely render the mask gradients exponentially small, and in particular not considerably larger than the QKV-gradients.

Also, the results from Abbe do not assume knowledge of a strong prior such as this hard-coded mask, so the insights from this paper will hardly shake the theoretically and empirically strongly grounded MDI bias from Abbe et al..

I will follow the threads with the other reviewers, but so far I am not convinced to recommend this paper for publication.

We thank the reviewer for their continued engagement with our submission and are glad that they found the additional Cyclic-5 experiment surprising and interesting. We thank the reviewer for this suggestion and will include this experiment in the Appendix.

Insights on the MDI hypothesis

In response to reviewer c4GW, we have included a more detailed theoretical explanation for how the use of a mask avoids the MDI solution (see the end of the Appendix in red). By necessity, this is a fairly involved argument directly built on Abbe et al. (2024)’s construction. But the main intuition behind this approach is that the use of an idealised mask can block lower-order interactions from the pre-activation attention while retaining the desired higher-order interactions (e.g. under two parity, the desired x1x2x_1x_2 term). When viewed in the spectrum the masked model can directly remove the lower-order Fourier components which characterizes the MDI solution under the Fourier-Walsh transform of the interpolator. This analysis is consistent with Abbe et al. (2024)'s approach and in essence shows that the MDI can be avoided with additional constraints. Please let us know if you would like any more details on this approach.

Wider Significance

In terms of the wider significance our work helps to provide understanding on how inductive biases can be encoded in transformers. There is significant community interest in inductive biases as they can help design more efficient optimization strategies and improve understanding in generalization.

In particular there is current interest in where to encode inductive priors with recent work focused on encoding attention priors through initialization of the Q and K matrices. Prior literature (see: Zheng et al. 2025, Structured Initialization for Vision Transformers) had shown that incorporating a convolutional inductive prior by initializing Q and K matrices in attention could improve on sample efficiency in vision problems.

The second contribution in our paper is to highlight an issue with encoding these inductive priors at initialization. We show that (because an initialized prior is non-persistent) it can be washed out during training, particularly in the GOTU setting where a key extrapolation feature is not observed during training (see Figure 7). This makes encoding a prior more practical with a logical attention mask than encoding the same attention prior through initialization.

Beyond logical problems we demonstrate this on a vision problem (Section 5.3 Image Classification), where we encode a convolutional prior through the attention mask. We demonstrate that this leads improved performance relative to encoding the same prior through initialization.

DatasetDefaultMimetic (Init)Impulse (Init)Logical Mask
CIFAR1093.9394.98 (+1.05)95.03 (+1.10)95.95 (+2.02)
CIFAR10077.2679.01 (+1.75)79.62 (+2.36)82.06 (+4.80)
ImageNet10083.4784.55 (+1.08)83.51 (+0.04)85.83 (+2.36)

We again thank the reviewer for their constructive engagement and are happy to any further questions.

Comments on the paper

1 comment

We thank the Area Chair and reviewers for their constructive suggestions and continuous engagement, and we will revise the paper to sharpen its positioning, provide the theoretical explanation for the logical mask, and incorporate the new experiments.

Key Strengths:

Clear and well-posed research question. The paper investigates whether the minimum-degree extrapolation bias of Transformers can be altered and whether the way a structural prior is encoded affects its persistence during training.

Simple and interpretable approach. The proposed additive attention mask provides a straightforward way to introduce structured inductive priors into Transformers without modifying the standard QKV computation.

Strong empirical evidence. The method achieves near-perfect extrapolation on several Boolean GOTU tasks, substantial improvements in arithmetic problems, and consistent gains in image classification. The additional Cyclic-5 experiment further demonstrates its ability to support higher-order interactions.

Importance of prior parameterization. The paper highlights that encoding the same prior through QK initialization or an additive attention mask can lead to substantially different learned solutions, even when their initial attention patterns are similar.

Key Concerns and Our Responses:

Key ConcernRaised ByOur Response
The method relies on manually designed priors, limiting its practical applicability.All threeWe clarified that our goal is to study how an available prior changes inductive bias, rather than how to discover the prior automatically. Vision experiments demonstrate benefits from approximate locality priors without knowing object locations.
The method may not support higher-order interactions beyond pairwise relationships.eHSeWe added Cyclic-5 experiments, improving unseen accuracy from 0.88% to 99.90%, demonstrating that attention masks can facilitate higher-order logical extrapolation.
The theoretical analysis does not sufficiently explain how masking avoids the MDI solution.eHSe, c4GWWe extended the analysis to an idealized multilayer random-feature model. We show that suitable layerwise masks can restrict Fourier support to a target monomial, allowing minimum-norm interpolation to recover the target on seen and unseen inputs within the model's capability.
Strong initialization makes the soft mask effectively hard, questioning the need for learnability.eHSe, c4GWWe clarified that hard and soft masks behave similarly on Boolean tasks with exact priors. However, learned masks outperform hard masks on all three image-classification benchmarks, where the prior is approximate.
What will happen when the prior contains incorrect or missing interactions.c4GWWe added mask-misspecification experiments involving blocked and extra connections. These identify the importance of prior correctness and clarify the conditions under which extrapolation can fail.
The method assumes a known token ordering and does not generalize to arbitrary permutations.eHSeWe clarified that the Boolean and arithmetic tasks considered have meaningful token orderings. Our vision experiments further show that useful priors need not specify exact foreground positions. General permutation invariance remains outside our scope.
Full attention masks require quadratic parameter storage and may not scale to long sequences.eHSeWe introduced an implicit neural representation parameterization supporting variable sequence lengths without increasing the number of mask parameters. This addresses parameter storage.
The experiments are narrow and do not establish general systematic reasoning capabilities.eHSe, b2eb, c4GWWe clarified the scope of our claims and expanded the empirical evidence with Cyclic-5 and vision experiments.

Core Contribution and Points of Disagreement

We consider our core contribution to be a controlled demonstration that the Transformer’s minimum-degree extrapolation bias is not fixed. A task-aligned interaction prior can redirect the learned extrapolation solution and, under the evaluated settings, is preserved more effectively through an additive-logit bias than through QK initialization.

The remaining concerns include (i) theoretical explanation: our extended analysis establishes how masking can modify Fourier support and the resulting interpolator in an idealized multilayer model. (ii) automatic prior discovery and permutation invariance: we consider these directions beyond the scope of this work, as our primary contribution is not to discover a universally applicable mask, but to investigate how a given structural prior can alter the inductive bias of transformers and how its parameterization affects persistence during training. We would like to leave this for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.