acceptodds
Under review as a conference paper at ICLR 2027

Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata

Abstract

Induction heads provide a mechanistic account of in-context learning in sequential data, but existing theory largely assumes that the context relevant to a prediction forms a contiguous block. In multidimensional data, serialization breaks this assumption by scattering spatial neighbors across distant positions in the token sequence. We study how transformers overcome this routing problem in multi-dimensional stochastic and deterministic cellular automata, where each trajectory is generated by an unknown local rule and presented as a flattened sequence without an explicit coordinate-based spatial inductive bias. We introduce spatial induction heads, two-layer gather-and-match circuits in which the first layer reconstructs the relevant spatial neighborhood and the second matches the resulting configuration against earlier occurrences. We give two explicit realizations of the gather and show that the positional dimension required for spatial routing depends only on the local neighborhood and spatial dimension, not on grid volume or trajectory horizon. We further construct a matching layer which implements Bayesian counting. The end-to-end circuit can approximate the Bayesian posterior arbitrarily closely for stochastic rules and can predict exactly for deterministic rules. Empirically, trained two-layer transformers generalize to unseen rules in one and two dimensional settings, achieving near-perfect deterministic rollouts and less than 0.005 nats KL from the Bayes-optimal predictor on stochastic rules. Attention patterns and layerwise probes align with the predicted gather-and-match computation, providing mechanistic evidence for spatial induction in trained transformers.

Primary area:
interpretability and explainable AI
Keywords:
in-context learning, induction heads, cellular automata, transformers, mechanistic interpretability, Bayes-optimal prediction

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_3b5qOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studied how Transformers perform in-context learning on multidimensional cellular automata. The authors observed that flattening a spatial grid into a one-dimensional token sequence breaks the contiguity assumption behind standard induction heads: spatial neighbors end up scattered across distant positions. To address this, they proposed spatial induction heads, a two-layer gather-and-match circuit. The first layer gathers the relevant spatial neighborhood, and the second layer matches the resulting configuration against earlier occurrences to predict the next state. They provided two explicit gather constructions and show that the positional width needed for spatial routing depends only on the local neighborhood and spatial dimension, not on grid size or trajectory length. They further showed that the matching layer implements Bayesian counting, approximating the Bayes-optimal predictor for stochastic rules and predicting exactly for deterministic rules. Experiments on 1D and 2D cellular automata showed generalization to unseen rules, with attention patterns, ablations, and linear probes supporting the proposed circuit.

Critical strengths
  1. The paper identified an important gap that existing induction head method assumes a contiguous context, which fails for flattened spatial data.
  2. The paper presented a clean and promising theoretical framework with the proposed two-layer gather-and-match circuit.
Critical weaknesses
  1. The presentation is only friendly to a narrow range of readers who happen to work on the same task. The significance of the work is not conceivable for general machine learning researchers.
  2. The paper does not directly analyze whether the learned positional embeddings reflect the theoretical constructions. For stochastic rules, the reported KL is against the k-gram estimate rather than the Bayes posterior.
Questions for authors
  1. Do the learned positional embeddings show any structure resembling the constructions used in the proofs, such as Fourier-like patterns or colorings?
  2. Can the authors report the divergence between the model's predictions and the Bayes posterior directly, rather than only against the k-gram estimate?
  3. How much does performance drop if the coverage filter is removed, so that some scored queries involve unseen configurations?
  4. Are there path patching or causal tracing experiments that directly show information flowing from the first-layer routing cells to the second-layer matching head?
  5. How does the model compare with a Transformer that uses explicit 2D positional encodings or a convolutional inductive bias, under similar parameter counts?
9 comments

We thank the reviewer for recognizing the gap we address in existing induction-head theory and the clarity and promise of our gather-and-match framework. We appreciate their thoughtful feedback and address their concerns below.


W1: Accessibility and broader significance

Our work extends an established line of theory on in-context learning of Markov chains [1, 2, 4, 6, 11], published at ICLR and NeurIPS, including a NeurIPS 2025 spotlight [2]. These works advance our understanding of induction heads, one of the best-understood mechanisms of in-context learning in language models. Transformers also routinely process spatial data, such as images and video, as flattened sequences, yet how they recover spatial structure remains poorly understood. Our goal is to extend this mechanistic understanding to multidimensional data through the proposed gather-and-match circuit. The importance of learning spatial rules from limited examples is further reflected in benchmarks such as ARC-AGI [8]. We agree that the paper should make these broader connections clearer and be more accessible to general machine learning researchers, and will improve the exposition accordingly.


W2 & Q1: Structure of the learned positional embeddings

We ran additional experiments at the positional widths given by Theorem 7 (5 in 1D and 7 in 2D), using both random initialization and initialization with the construction's Fourier positional features. At these widths, models initialized with the Fourier construction achieve near-perfect sequence accuracy, matching the free-table models that use 128 and 1024 dimensions (Table B1). This confirms that the theoretical widths suffice in the trained architecture. In contrast, random initialization at the same widths does not reach comparable accuracy under the training configurations tested, indicating that gradient descent does not reliably discover the compact solution. The free-table models have substantially higher effective ranks (102.6 in 1D and 195.1 in 2D), suggesting that gradient descent uses a higher-dimensional representation when given the flexibility, whereas the Fourier construction reaches the same accuracy with only 5 and 7 dimensions (effective rank 4.9 and 6.1).

Table B1. Positional width, effective rank, and accuracy.

SettingPositional widthInitializationTF cell acc.AR cell acc.TF seq. acc.Effective rank
1D, bottleneck (construction width)5Random0.81180.73280.11385.0
1D, bottleneck (construction width)5Fourier construction (Theorem 7)1.00001.00000.99984.9
1D, free table (main text)128Random1.00001.00000.9994102.6
2D, bottleneck (construction width)7Random0.73400.65040.03066.4
2D, bottleneck (construction width)7Fourier construction (Theorem 7)1.00000.99990.99916.1
2D, free table (main text)1024Random1.00000.99990.9989195.1

Although the free tables do not recover the construction's parameters, they share its structure. We group Layer-1 attention logits according to the query–key grid displacement (temporal difference and spatial offset on the torus) and measure the explained variance R2R^2, compared with grouping by sequence displacement and random labels. Across both 1D and 2D settings, with either freely learned positional tables or the theory-prescribed positional dimension dpd_p, spatial displacement explains more variance than sequence displacement, suggesting that learned scores reflect spatial structure more strongly than sequential structure (Table B2).

Table B2. Explained variance of Layer-1 positional scores. Grid (keys at t−1t-1) considers only previous-step keys. Sequence groups pairs by token-stream distance. Random uses random groups as a control.

SettingGrid R2R^2Grid R2R^2 (keys at t−1t-1)Sequence R2R^2Random R2R^2
1D W=16W=16, free table (main text)0.520.760.510.01
1D W=16W=16, dp=5d_p=5, Theorem 7 initialization0.890.860.830.01
2D 6×66\times6, free table (main text)0.670.650.530.005
2D 6×66\times6, dp=7d_p=7, Theorem 7 initialization0.650.470.600.005

W2 & Q1 (continue)

Fourier structure. The routing also relies on a few spatial frequencies, as in the construction. Removing the 4 strongest frequency groups of the learned table drops accuracy to at most 0.62, far more than removing 4 random groups (Table B3).

Table B3. TF cell accuracy after Fourier ablation of the learned positional table (main-text models).

SettingArchitectureChanceKeep top 4Remove top 4Remove 4 random
1D V=2V=2Shared-head0.50000.99010.58940.6460
1D V=2V=2Dedicated-heads0.50000.98180.61650.9401
1D V=3V=3Shared-head0.33330.59390.40860.7688
1D V=3V=3Dedicated-heads0.33330.79020.46890.9195
2D VNShared-head0.50000.58630.57300.7611
2D VNDedicated-heads0.50000.99660.57350.8122

W2 & Q2: KL to the Bayes posterior

We clarify that for the stochastic reference, the add-one-smoothed kk-gram predictor is exactly the Bayes posterior for α=1\alpha=1, the experimental prior (Theorem 3; Appendix I.3). The reported KL is therefore already against that posterior.


Q3: Removing the coverage filter

Unseen configurations (nc=0n_c = 0) are a limit of induction itself. For unseen configurations, without prior occurrences, the model has no evidence about their rule-specific behavior. Under our prior, the Bayes-optimal predictor therefore defaults to the prior mean 1/V1/V (Lemma D.2), which our construction implements exactly (Proposition 8).

Our experiments evaluate only resolvable queries (nc≥1n_c\geq1), since predictions at nc=0n_c=0 measure the model's ability to recover the prior rather than learn the rule in context.

Removing the filter barely changes performance: deterministic accuracy drops by at most 0.0003, and the stochastic KL is unchanged (Table B4). On the unseen configurations themselves, models trained on stochastic rules predict a nearly uniform distribution (H(q)/log⁡V≥0.98H(q)/\log V \geq 0.98), matching the Bayes-optimal answer (Table B5).

Table B4. Filtered vs. unfiltered test sets. Deterministic: TF cell accuracy. Stochastic: TF KL(P^1∥q)\mathrm{KL}(\hat P_1\|q). Filtered values are from paper Tables 1 and 14.

SettingRulesArch.FilteredUnfiltered
1D V=2V=2Det.Shared-head1.00000.9999
1D V=2V=2Det.Dedicated-heads1.00000.9999
1D V=3V=3Det.Shared-head1.00000.9998
1D V=3V=3Det.Dedicated-heads1.00000.9998
2D VNDet.Shared-head1.00000.9997
2D VNDet.Dedicated-heads1.00000.9997
1D V=2V=2Stoch.Shared-head0.00200.0020
1D V=2V=2Stoch.Dedicated-heads0.00460.0046
1D V=3V=3Stoch.Shared-head0.00260.0026
1D V=3V=3Stoch.Dedicated-heads0.00040.0004
2D VNStoch.Shared-head0.00140.0014
2D VNStoch.Dedicated-heads0.00330.0033

Table B5. Stochastic rules: unseen configurations (nc=0n_c=0), unfiltered test sets. H(q)/log⁡V=1H(q)/\log V = 1 means the prediction is uniform.

SettingArch.KL$(U|q)$H(q)/log⁡VH(q)/\log V
1D V=2V=2Shared-head0.01160.9841
1D V=2V=2Dedicated-heads0.00410.9946
1D V=3V=3Shared-head0.00170.9984
1D V=3V=3Dedicated-heads0.00020.9998
2D VNShared-head0.00100.9986
2D VNDedicated-heads0.00070.9990

Thank you for the detailed rebuttal and the additional experiments. I find the new analyses of positional embeddings particularly informative. The clarifications regarding the Bayesian posterior and the coverage filter satisfactorily address my corresponding questions.

My main remaining concern is the accessibility and broader significance of the work. While I appreciate the connections to previous studies of induction heads and spatial Transformers, I would still encourage the authors to explain more clearly what general insights this framework provides beyond the specific cellular automata setting, as well as its limitations.

Overall, the rebuttal has strengthened my confidence in the technical contributions. I am inclined to maintain my positive assessment.

Reviewer_oWJ3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores the mechanisms and theoretical framework of Multidimensional Cellular Automata in Transformers, introducing 'Spatial Induction Heads.' In this setup, the first layer of attention gathers the spatial neighbors needed for the current prediction, and the second layer looks into past experience to find instances where the same neighbor combinations appeared, then uses the subsequent states of those examples to make predictions.

Critical strengths
  1. The perspective of the problem is interesting. Previous induction head theories typically assumed that the required context existed in a continuous segment of tokens preceding the current position, whereas this paper points out the key issue of "neighbors being scattered" after serializing 2D/multi-dimensional data, which makes some previous theories inapplicable.
  2. The design of "Spatial Induction Heads," with the first layer finding neighbors and the second layer finding matches, is quite reasonable, and it provides a verifiable computational mechanism rather than a post-hoc interpretation of attention maps.
  3. The authors support their core argument "the first layer forms domain representations, and the second layer generates conditional distributions" from multiple angles, including analyzing attention landing points, observing performance changes after masking theoretically predicted positions, and using layer-wise linear probes.
Critical weaknesses
  1. The claim in the text that the 'positional dimension does not depend on grid size or trajectory length' has a gap with the absolute position table setting in the actual model. The paper proves that the positional representation dimension required for spatial routing does not increase with grid size or trajectory length, but the experimental model uses a learnable absolute position embedding table, which is still bound by the maximum token position. Therefore, the paper has not clearly verified whether the model can extrapolate to larger grids, different row widths, or longer trajectories.
  2. This model requires the guarantee during testing that 'the same domain combinations must have appeared in the historical data,' which means the model does not need to handle rare configurations, unseen combinations, or situations with insufficient samples, and these cases are often the hardest. This is a limitation of this task setting in practical applications.
  3. The experimental phenomenon that 'masking the first-layer routing positions or the second-layer matching precedents leads to performance degradation' does support the core mechanism of the model, but these positions naturally contain the information required by the task, so damaging them is bound to be harmful, and it cannot completely rule out that the model also uses other parallel paths. Linear probes can only prove that the information is readable, and they do not indicate that the model relies on this information when making decisions.
  4. The authors’ claim of 'no spatial inductive bias' needs to be more rigorous. The model indeed does not receive 2D coordinates as input, but it sees a fixed row-major serialization, row separators, time separators, and a learnable absolute position table. As long as the grid size is fixed, spatial neighborhoods correspond to very stable sequential offsets. Strictly speaking, it is not explicitly provided with geometric coordinates, but the training data still provides very strong, learnable, and repeatable serialized geometry.
Questions for authors

Please refer the the weaknesses

9 comments

We thank the reviewer for their positive assessment of our problem perspective, the gather-and-match mechanism, and the supporting mechanistic experiments. We appreciate their careful reading and thoughtful feedback on the connection between our theory and experiments. We address each weakness below:


W1: Positional width vs. a learned absolute table

The claim in the text that the 'positional dimension does not depend on grid size or trajectory length' has a gap with the absolute position table setting in the actual model. The paper proves that the positional representation dimension required for spatial routing does not increase with grid size or trajectory length, but the experimental model uses a learnable absolute position embedding table, which is still bound by the maximum token position.

We clarify that our claim concerns the positional width dpd_p, not the number of positions NN. The learned absolute positional table P∈Rdp×NP\in\mathbb{R}^{d_p\times N} indeed has one column per token position and grows with context length. Our theoretical result establishes that the width dpd_p required for spatial routing remains independent of grid size and trajectory length (Lemma 6 and Theorem 7). It does not claim that the number of positional embeddings is independent of NN.

To validate this distinction empirically, we trained models with the construction widths dp=5d_p=5 in 1D and dp=7d_p=7 in 2D, initialized using the positional features from Theorem 7. These models achieve near-perfect accuracy, matching the substantially wider learned positional tables used in our main experiments (Table R2).

Table R2. Accuracy at theoretical versus unconstrained positional widths.

SettingPositional widthInitializationTF cell acc.AR cell acc.TF seq. acc.
1D5Theorem 71.00001.00000.9998
1D128Random (main text)1.00001.00000.9994
2D7Theorem 71.00000.99990.9991
2D1024Random (main text)1.00000.99990.9989

These results confirm that the theoretically sufficient positional widths also work in the trained architecture, without requiring positional width to grow with the grid size or trajectory horizon.

the paper has not clearly verified whether the model can extrapolate to larger grids, different row widths, or longer trajectories.

Our results focus on generalization to unseen rules at fixed geometry, rather than extrapolation across geometries within a single trained model. Our theorem establishes that, for any grid and horizon, a two-layer transformer can implement spatial routing with positional width dpd_p independent of grid size and trajectory length. This is an expressivity guarantee, not a claim that the same learned parameters generalize across geometries. We agree that cross-geometry extrapolation is an interesting direction for future work.


W2: Rare and unseen configurations

We clarify that rare configurations are covered both in theory and practice. In theory, Theorem 9 holds at any nc≥1n_c \ge 1, and our test datasets are randomly generated and as a result our near-perfect accuracy and low KL reports do not exclude rare occurrences. Furthermore, we ran additional experiments providing fine-grained accuracy per occurrences:

Table O3. Performance by configuration frequency (unfiltered test sets). Deterministic: teacher-forced cell accuracy (higher is better). Stochastic: KL(P^1∥q)\mathrm{KL}(\hat P_1\|q) to the Bayes posterior (lower is better).

SettingRulesArch.nc=1n_c=1nc=2n_c=2nc=3n_c=3–44nc=5n_c=5–88nc>8n_c>8
1D V=2V=2Det.Shared-head0.99170.99930.99981.00001.0000
1D V=2V=2Det.Dedicated-heads0.99610.99960.99991.00001.0000
1D V=3V=3Det.Shared-head0.98660.99890.99991.00001.0000
1D V=3V=3Det.Dedicated-heads0.99790.99990.99991.00001.0000
2D VNDet.Shared-head0.99971.00001.00001.00001.0000
2D VNDet.Dedicated-heads0.99740.99970.99991.00001.0000
1D V=2V=2Stoch.Shared-head0.01080.00940.00750.00490.0019
1D V=2V=2Stoch.Dedicated-heads0.01280.01360.01010.00670.0045
1D V=3V=3Stoch.Shared-head0.00240.00290.00300.00270.0024
1D V=3V=3Stoch.Dedicated-heads0.00060.00050.00040.00040.0004
2D VNStoch.Shared-head0.00220.00220.00180.00120.0014
2D VNStoch.Dedicated-heads0.00810.00890.00820.00590.0031

W2 (continue)

Even with just one prior occurrence (nc=1n_c=1), deterministic accuracy is at least 98.66%, and stochastic KL divergence from the Bayes posterior is below 0.014 nats. Thus, strong performance on rare configurations is demonstrated empirically, not only theoretically.

Unseen configurations (nc=0n_c = 0) are a limit of induction itself. For unseen configurations, without prior occurrences, the model has no evidence about their rule-specific behavior. Under our prior, the Bayes-optimal predictor therefore defaults to the prior mean 1/V1/V (Lemma D.2), which our construction implements exactly (Proposition 8).

Our experiments evaluate only resolvable queries (nc≥1n_c\geq1), since predictions at nc=0n_c=0 measure the model's ability to recover the prior rather than learn the rule in context.

For completeness, we also examine the unseen configurations in these unfiltered test sets. On them, models trained on stochastic rules predict a nearly uniform distribution (H(q)/log⁡V≥0.98H(q)/\log V \geq 0.98), matching the Bayes-optimal answer (Table O4).

Table O4. Stochastic rules: unseen configurations (nc=0n_c=0), unfiltered test sets. H(q)/log⁡V=1H(q)/\log V = 1 means the prediction is uniform.

SettingArch.KL$(U|q)$H(q)/log⁡VH(q)/\log V
1D V=2V=2Shared-head0.01160.9841
1D V=2V=2Dedicated-heads0.00410.9946
1D V=3V=3Shared-head0.00170.9984
1D V=3V=3Dedicated-heads0.00020.9998
2D VNShared-head0.00100.9986
2D VNDedicated-heads0.00070.9990

W3: Causal evidence for the predicted circuit

We appreciate this concern and agree that ablations establish necessity rather than sufficiency, and that probes establish decodability rather than use. We conducted three complementary causal interventions that test both whether the predicted pathways are sufficient for performance and whether changing information along those pathways changes the model's prediction.

1. Causal scrubbing: Are alternative positions needed?

We replace activations outside the theoretically predicted positions (Layer-1 routing cells and Layer-2 matching precedents) with activations from another trajectory, while preserving the predicted positions. As a control, we instead replace the predicted positions.

Table O6. Causal scrubbing. Fraction of performance recovered relative to fully scrubbed activations (1 = clean performance; 0 = fully scrubbed performance).

SettingArchitectureDet.: outsideDet.: insideStoch.: outsideStoch.: inside
1D V=2V=2Shared-head0.99920.07370.99960.2590
1D V=2V=2Dedicated-heads1.00000.21540.98450.2610
1D V=3V=3Shared-head1.00000.10330.99990.0497
1D V=3V=3Dedicated-heads0.99990.19030.99860.0513
2D VNShared-head1.00000.14710.99980.0688
2D VNDedicated-heads1.00000.16540.99990.0676

Scrubbing all positions outside the predicted pathways preserves at least 98% of performance, whereas scrubbing the predicted positions reduces recovery to at most 26%. Thus, the predicted pathways are sufficient to maintain performance under this intervention, with no substantial contribution from alternative positions detected.

2. Path patching: Does information flow through Layer 1 as predicted?

We splice the cell states of a second trajectory, whose configuration c′c' differs from the query's, into the query's routing cells, and let the change reach the query only along one path. Entries: fraction of queries whose prediction becomes the in-context answer for c′c'. Control: splice the same number of non-routing cells.

Table O7. Path patching, deterministic rules. Fraction of predictions switching to the in-context answer for c′c'.

SettingArchitectureLayer 1 onlyLayer 2 onlyBothNon-routing control
1D V=2V=2Shared-head0.99960.00070.99890.0003
1D V=2V=2Dedicated-heads0.99820.00040.99700.0002
1D V=3V=3Shared-head0.99910.00010.99880.0000
1D V=3V=3Dedicated-heads0.99750.00010.99710.0001
2D VNShared-head0.99920.00030.99870.0002
2D VNDedicated-heads0.99960.00030.99910.0002

when the new routing-cell states reach the query only through its Layer-1 gather, at least 99.75% of predictions switch to the answer the context gives for the new configuration; through Layer 2 directly, at most 0.07% switch; changing non-routing cells has no effect. Information therefore flows from the routing cells, through the query's Layer-1 gather, to the prediction. For stochastic rules, patching the Layer-1 path moves the model's KL to the new configuration's posterior from 0.38–0.45 to 0.001–0.006 nats.

3. Interchange interventions: Is the Layer-1 representation actually used?

Our existing probes show that the configuration becomes decodable at the stages predicted by our constructions (Figure 4). To go beyond decodability, we replace the query's Layer-1 output with that of a query having a different configuration and test whether the prediction follows the substituted representation.

Table O8. Interchange interventions, deterministic rules. Fraction of predictions following the substituted configuration. Control: the source is taken from a random position.

SettingArchitectureInterchange accuracyRandom-position control
1D V=2V=2Shared-head0.99950.4849
1D V=2V=2Dedicated-heads0.99960.4865
1D V=3V=3Shared-head0.99940.3644
1D V=3V=3Dedicated-heads0.99040.3634
2D VNShared-head0.99940.3983
2D VNDedicated-heads0.99960.3992

At least 99% of predictions follow the substituted configuration, while a source from a random position gives near-chance agreement (0.36–0.49). This demonstrates causal reliance on the Layer-1 representation, rather than merely showing that configuration information is decodable from it.

Together, these interventions provide substantially stronger evidence: the predicted pathways preserve performance when alternative positions are disrupted, Layer-1 routing causally determines which configuration reaches the query, and the resulting representation causally controls the prediction. We will incorporate these results into the paper.

Reviewer_SL3nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors present a spatial in-context learning in transformers through multidimensional cellular automata. The papers has a very long mathematical demonstration, but the content of the paper presetns the main contribution as the illustration of a two-layer “spatial induction head” based on a gather-and-match mechanism: The in-paper evaluation is very short but it shows the way and how the approach works, on the other hadn, thre is a weak match on how a computer will perform it

Critical strengths

The authors presen a clear and novel conceptual extension of induction-head theory from contiguous sequential dependencies to spatial dependencies scattered by serialization. The empirical results support well the internal mechanism. In particular, attention analysis, masking/ablation experiments, and layerwise probes. Well deteailed and organized paper.

Critical weaknesses

My main concern is the almost entirely theoretical work (a huge support theorecial is coming after the paper!!!) and is validated only on small, highly controlled synthetic cellular-automata problems. Todays, ML is not only thoery, the best model are the one that are able to be run on MACHINE, so a discussion in how this will run and perfom on a computer architecutre in my opinion is mandatory at ICRL, so lfeel this paper is out of scope here. In fact, the experiments demonstrate that the proposed mechanism can be learned in these settings, but they do not establish whether it provides a useful or scalable mechanism for realistic multidimensional problems.

Questions for authors

How does the actual computational cost scale with grid size, sequence length, and neighborhood size? In particular, what are the FLOPs and memory requirements of the proposed constructions?

Can the authors provide wall-clock/runtime or memory comparisons against simpler baselines or architectures with explicit spatial inductive biases?

Does the mechanism still emerge when training on significantly larger spatial domains and substantially longer sequences?

Can the authors demonstrate the proposed mechanism on at least one task outside discrete cellular automata, ideally where the spatial dependencies are not exact finite-state lookup patterns?

What happens when the context does not provide complete coverage of all relevant configurations?

5 comments

We appreciate the reviewer’s positive assessment of our extension of induction-head theory to spatial dependencies and their appreciation of the experiments supporting the proposed mechanism. We are glad they found the paper detailed and well organized. We also thank them for their honest feedback and practical questions. Below we address them:


W1: Limited evidence of practical utility and scalability

My main concern is the almost entirely theoretical work (a huge support theorecial is coming after the paper!!!) and is validated only on synthetic problems. Todays, ML is not only thoery, the best model are the one that are able to be run on MACHINE, so lfeel this paper is out of scope here.

We appreciate the reviewer’s candid assessment. The ICLR 2027 call for papers lists learning theory and the interpretation of learned representations as topics. Mechanistic studies of transformers on synthetic tasks also have a strong record at ICLR, including Liu et al. [12] (oral) on automata, Akyürek et al. [9] on in-context linear regression, and Nanda et al. [10] (oral) on modular arithmetic.

and is validated only on small, highly controlled synthetic cellular-automata problems.

Our work builds on an established line of theory on in-context learning of Markov chains [1, 2, 4, 6, 11], including recent papers at ICLR 2025 [18] and ICLR 2026 [6], and a NeurIPS 2025 spotlight [2]. Under the standard established by these works and the ICLR papers discussed above, we use controlled settings to make mechanistic claims testable. Cellular automata generalize Markov chains while providing a closed-form Bayes-optimal predictor and a known ground-truth neighborhood, allowing us to isolate and study the underlying mechanisms.

Synthetic grids are far from a niche concern: ARC-AGI [15] consists of small synthetic grids, yet is widely regarded as an important benchmark for abstraction. Many of its tasks require inferring spatial rules from a few examples, presenting a challenge closely related to the in-context learning problem we study.

Our experiments also operate at sequence lengths comparable to vision transformers. A ViT-B/16 at 224px uses a 14×1414\times14 patch grid (197 tokens), while our 2D streams contain 504 and 840 tokens. Although our spatial grids are smaller, their side lengths differ by only a factor of 14/6≈2.314/6\approx2.3.

Moreover, small grids do not imply a trivial learning problem: a 6×66\times6 binary grid admits 236≈692^{36}\approx 69 billion possible states, and even the binary von Neumann setting has 232≈4.32^{32}\approx4.3 billion possible deterministic rules. Prior transformer work on automata either trains a separate model for each automaton [12], provides the rule in the prompt [13], or, when the rule must be inferred in context, finds multi-step prediction challenging and resorts to greater depth or auxiliary rule-prediction losses [14]. In contrast, our models learn spatial routing from data and infer unseen transition rules in context, with near-perfect autoregressive rollouts, using just two layers.

My main concern is the almost entirely theoretical work, the experiments demonstrate that the proposed mechanism can be learned in these settings, but they do not establish whether it provides a useful or scalable mechanism for realistic multidimensional problems.

The paper is not only theoretical. We train standard transformers, which reach stellar autoregressive rollouts on held-out rules, including the Turing-complete rules such as rule 110, and come within 0.005 nats of the Bayes-optimal predictor on stochastic rules. Attention alignment, causal ablations, and probes confirm the predicted mechanism. By using only standard learned embeddings, attention and MLPs, our work gives theoretical grounding for observations in vision such as the ViT, where learned 1D positions recover 2D structure.

Given these precedents, we understand the reviewer's concern not as one of scope, but as a call for more discussion of practical relevance, which we will add.

Q1: Computational cost and scaling

The required widths and head counts are given in Theorem 7 and Appendix G. Thus, the computational costs follow directly by substituting these bounds into the standard complexity of two-layer causal attention.

Let N=T(∣G∣+∣G∣/Wd)N=T(|G|+|G|/W_d). Construction I uses h=∣U∣h=|\mathcal U| first-layer heads, while Construction II uses h=1h=1; both use one second-layer head. Substituting the widths from Theorem 7 and Appendix G gives the conservative forward FLOP bound

+4Nd_{\mathrm{model}}d_{\mathrm{MLP}}.$$ Thus, at fixed widths, the dominant cost is quadratic in $T|G|$, with an additional head-count factor of $|\mathcal U|$ for Construction I. Neighborhood size also affects the widths specified by the constructions. Standard attention requires $$O\left((h+1)N^2+(h+1)Nd_{\mathrm{model}}+Nd_{\mathrm{MLP}}\right)$$ activation memory, plus model parameters and the $O(Nd_p)$ positional table. Memory-efficient attention avoids storing the quadratic attention matrices. For practical context, Table S1 reports costs of our trained models, which use wider representations than the constructions. **Table S1. Trained-model costs.** Peak memory is measured on one A100 (80GB) at each model's training batch size. | Setting | Architecture | Input tokens | GFLOPs/seq | Peak memory (GiB) | |---|---|---:|---:|---:| | 1D $W=16$, $T=10$ | Shared | 168 | 0.16 | 1.6 | | | Dedicated | 168 | 0.12 | 1.1 | | 1D $W=32$, $T=12$ | Shared | 394 | 1.56 | 14.6 | | | Dedicated | 394 | 1.15 | 2.5 | | 1D $W=64$, $T=64$ | Dedicated | 4,158 | 105.71 | 55.1 | | 2D $6\times6$, $T=12$ | Shared | 502 | 27.34 | 18.8 | | | Dedicated | 502 | 18.91 | 13.5 | --- ## **Q2: Runtime and memory comparisons with simpler and spatially biased baselines** The positional baselines keep the same two-layer architecture and change only how position enters attention, so **their asymptotic cost** is unchanged. Further, we emphasize that our goal is to explain how transformers learn spatial data in-context, without encoding an inductive bias, rather than to claim superior accuracy or efficiency over spatially informed architectures. For completeness, we report them below: **Table S2. Baseline accuracy and computational cost.** Memory and training throughput are measured on one RTX A6000 at matched batch sizes (1D: 1024; 2D: 512). Full-run training times use the hardware shown. | Setting | Model | Params | Fwd GFLOPs/seq | Peak memory (GiB) | Tokens/s | Train time | TF cell acc. | Seq. acc. | |---|---|---:|---:|---:|---:|---|---:|---:| | 1D $W=16$ | Learned absolute, Shared-head (main text) | 0.42M | 0.16 | 1.6 | 5.14M | 0.18 h, 1×B200 | 1.0000 | 0.9990 | | 1D $W=16$ | Learned absolute, Dedicated-heads (main text) | 0.29M | 0.12 | 1.2 | 5.97M | 0.74 h, 1×A6000 | 1.0000 | 0.9994 | | 1D $W=16$ | Sequence-relative bias, Shared-head | 0.40M | 0.16 | 2.1 | 4.27M | 0.92 h, 1×6000 Ada | 0.9966 | 0.8631 | | 1D $W=16$ | Sequence-relative bias, Dedicated-heads | 0.27M | 0.12 | 1.8 | 4.86M | 1.35 h, 1×A6000 | 0.9996 | 0.9704 | | 1D $W=16$ | One layer, $d_{\text{model}}=128$ | 0.22M | 0.08 | 1.2 | 8.26M | 0.54 h, 1×6000 Ada | 0.9114 | 0.4483 | | 1D $W=16$ | One layer, $d_{\text{model}}=256$ | 0.83M | 0.29 | 2.1 | 5.46M | 0.94 h, 1×6000 Ada | 0.9236 | 0.4973 | | 2D $6\times6$ | Learned absolute, Shared-head (main text) | 25.71M | 27.34 | 20.1 | 0.40M | 6.16 h, 2×B300 | 1.0000 | 1.0000 | | 2D $6\times6$ | Learned absolute, Dedicated-heads (main text) | 17.32M | 18.91 | 15.2 | 0.57M | 4.14 h, 2×B300 | 1.0000 | 0.9989 | | 2D $6\times6$ | Sequence-relative bias, Shared-head | 25.20M | 27.34 | 20.3 | 0.35M | 7.37 h, 2×B300 | 0.6899 | 0.0160 | | 2D $6\times6$ | Axial RoPE, Shared-head | 25.20M | 27.34 | 21.0 | 0.41M | 9.31 h, 1×B300 | 0.8798 | 0.0408 | | 2D $6\times6$ | Axial RoPE, Dedicated-heads | 16.80M | 18.91 | 17.1 | 0.60M | 3.70 h, 2×B300 | 1.0000 | 0.9992 |

Q3: Larger spatial domains and longer sequences

Our construction holds for any grid size, horizon, and spatial dimension. Empirically, trained models reach near-perfect held-out accuracy across 1D and 2D grids of 16–64 cells and sequences of 170–4,160 tokens (Table S3a).

The mechanism still emerges at the longest sequence. On the 1D W=64W=64, T=64T=64 model (4,160 tokens), we restrict each layer's attention to the positions our mechanism predicts, or mask them (Table S3b). Keeping only the predicted positions preserves accuracy (0.9997–1.0000), masking them drops it to 0.41–0.56, and masking random positions has no effect.

Table S3a. Performance across grid sizes and horizons. NN: serialized stream length.

TaskRules∣G∣\lvert G\rvertTTNNMetricShared-headDedicated-heads
1D V=2V=2, k=3k=3Det.1610170TF / AR cell acc.1.0000 / 0.99991.0000 / 1.0000
1D V=2V=2, k=3k=3Stoch.1620340AR KL0.00190.0044
1D V=3V=3, k=3k=3Det.3212396TF / AR cell acc.1.0000 / 1.00001.0000 / 1.0000
1D V=3V=3, k=3k=3Stoch.1620340AR KL0.00260.0004
2D VN V=2V=2, k=5k=5Det.3612504TF / AR cell acc.1.0000 / 1.00001.0000 / 0.9999
2D VN V=2V=2, k=5k=5Stoch.3620840AR KL0.00130.0038
1D V=2V=2, k=3k=3 (Fig. 1)Det.64644,160TF / AR cell acc.--1.0000 / 1.0000

Table S3b. Attention interventions at 4,160 tokens (1D W=64W=64, Dedicated-heads). TF cell accuracy.

InterventionLayer 1Layer 2Both
None1.00001.00001.0000
Keep only predicted positions0.99971.00000.9999
Keep only random positions0.54570.41000.4234
Mask predicted positions0.55700.40890.4876
Mask random positions1.00001.00001.0000

Q4: Tasks beyond discrete cellular automata

We appreciate the question. We focus on cellular automata because they are the natural multidimensional analogue of the kk-th order Markov chains that underpin the theory of in-context learning [1, 2, 4, 6, 11, 17, 18]. That literature likewise exclusively studies finite-state Markov chains, which admit a closed-form Bayes-optimal predictor against which a mechanism can be verified.

Within this framing, the class we cover is broad. Our theory holds for any number of states VV, any spatial dimension dd, and any neighborhood of kk offsets. The rule space is vast: VVkV^{V^k} unique deterministic rules and distribution dimension Vk(V−1)V^k(V-1) for probabilistic rules.

Moreover, stochastic CA are not deterministic lookup tasks: the same neighborhood can produce different next states, requiring the model to infer a conditional distribution. Even deterministic CA include turing-complete rules, such as Rule 110.


Q5: Incomplete context coverage

Induction means predicting the future based on the past (using data), so unseen configurations (nc=0n_c = 0) are a limit of induction itself. In this case, the Bayes-optimal prediction is just the prior mean 1/V1/V (Lemma D.2), which our construction will return (Proposition 8).

In experiments, we score only resolvable queries because unresolvable ones have the same optimal answer under every rule, so they carry no information about in-context learning.

For completeness, we also evaluate on unfiltered test sets. On the unseen configurations they contain, models trained on stochastic rules predict a nearly uniform distribution (H(q)/log⁡V≥0.98H(q)/\log V \geq 0.98), matching the Bayes-optimal answer (Table S4).

Table S4. unseen configurations (nc=0n_c=0), unfiltered test sets. H(q)/log⁡V=1H(q)/\log V = 1 means the prediction is uniform.

SettingArch.KL$(U|q)$H(q)/log⁡VH(q)/\log V
1D V=2V=2Shared-head0.01160.9841
1D V=2V=2Dedicated-heads0.00410.9946
1D V=3V=3Shared-head0.00170.9984
1D V=3V=3Dedicated-heads0.00020.9998
2D VNShared-head0.00100.9986
2D VNDedicated-heads0.00070.9990
Reviewer_apUKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies in-context learning by two-layer transformers when inputs structued on a d-dimensional grid are flattened into a token stream with row and time separators. The example task domain is to predict next states of cellular automata evolving in >1 dimension though local spatial kernels over k neighbors in the spatial domain. The model must infer the rule, and its spatially local kernels, from earlier transitions in the same ICL stream. The authors introduce spatial induction heads as a two-layer circuit. The first layer gathers, into separate residual blocks, the previous-step states of the cells. The second layer matches the gathered configuration against earlier configurations and averages the states that followed them. The authors give two constructions of the first layer: one head per offset, or one head for all offsets followed by an MLP that separates the offsets using a graph coloring of the torus. The main theoretical result and proof show that the positional table needed for the first layer has width O(|N| d log |N|), where |N| is the number of offsets to be gathered, independent of grid volume and of the number of time steps.The authors also derive the network’s end-to-end KL divergence from the Bayes predictor for stochastic rules. The trained models reach near-perfect accuracy on held-out deterministic rules and a very small KL divergence from the k-gram estimator on stochastic rules. They show that attention mass concentrates on the predicted routing cells and matching precedents, and that masking those cells reduces accuracy whereas masking random cells does not. They also show that the neighborhood configuration can be recoved via linear probes after the first layer and the predictive distribution can be recovered after the second layer.

Critical strengths
  1. The problem is well posed and the theory is constructive: a Bayes-optimal reference, explicit weights for the two layers with stated widths and logit scales, and an end-to-end bound (Theorem 9, Appendix H). The bound makes explicit the tradeoff between separating matching from non-matching precedents and keeping the attention scores of the matching precedents equal. The grid-independent addressing bound (Theorem E.4) is a clean result that I have not seen before. The coloring construction for the shared head keeps the identities of the gathered cells separate within one attention head.

  2. The experiments match the theory. It’s lovely how the Bayes predictor gives an absolute reference for stochastic rules; that the ablation of attention to theory-predicted cells has a random-cell control; and that the layer at which the configuration becomes linearly decodable differs between the two constructions as predicted.

Critical weaknesses
  1. The empirical settings are very mild illustrations of the routing problem that motivates the paper. In the 1D settings, the parents of the target cell at step t − 1 occupy a contiguous block of the stream (displacements W − 1, W and W + 1 from the current token), so spatial routing is needed only at the two wraparound positions. The 2D von Neumann setting scatters the neighbors across the stream, and its grid is 6 × 6 with 504 tokens. The abstract and introduction motivate the work with vision transformers on serialized images, but the evidence that trained transformers solve spatial routing comes from this one small grid.

  2. The headline theoretical claim, that the positional width needed for routing is independent of grid volume and of the number of time steps, isn’t tested. The trained models use d_model between 128 and 1024 and a free positional table with at most 504 columns, so it seems the model can learn an arbitrary embedding for each position and need not use the compact Fourier addressing of the construction. The paper doesn’t analyze the learned table: it reports neither the effective rank of the table, nor whether the positional part of the query–key score depends only on grid displacement. The grid volume |G| or the number of time steps T are not, as far as I could find, varied in the experiments. The attention-mass, ablation and probe results are consistent with the gather-and-match decomposition, but they are also consistent with other addressing schemes. The "mechanistic evidence" therefore supports the two-stage functional decomposition rather than the specific constructions of Theorem 7.

  3. The paper does not include architectural controls. Section 3.3 states that Appendix C.3 compares alternative positional embedding schemes, but that appendix contains only the impossibility theorem for sequence-relative biases. I do not find experiments with a one-layer model, with a sequence-relative bias, or with explicit 2D relative or rotary embeddings. Theorem C.6 also depends on toroidal wraparound. On an open grid with a fixed row length, a fixed spatial offset corresponds to a fixed sequence displacement, so the obstruction is a consequence of periodic boundary conditions rather than of serialization as such. The main text should state this condition.

Questions for authors
  1. Clarification: does the dedicated-heads model specialize one head per offset? Figure 3 averages attention mass over heads; please report, for each head, the attention mass on each routing offset.

  2. Clarification: in the stochastic setting, is the second-layer attention approximately uniform across matching precedents, as counting requires, or weighted toward recent precedents? A comparison of the model's predictive distribution with the posterior for several values of α and with a recency-weighted estimator would test the counting claim directly.

  3. Clarification: please report the effective rank of the learned positional table and the dependence of the positional score on grid displacement. Please also train models with d_p fixed to the width given by Theorem 7, for example 7 for the 2D von Neumann setting. Do these models reach the same accuracy?

  4. Clarification: please add a one-layer baseline and a sequence-relative-bias baseline in the 2D setting.

  5. Suggestion: can the 2D experiments be scaled to a grid on which the routing set is far from contiguous in the stream, for example a 16 × 16 torus with a Moore neighborhood, even with fewer time steps?

  6. Minor: Section 3.3 refers to Appendix C.3 for an empirical comparison of embedding schemes; please correct the reference or add the experiment.

9 comments

We thank the reviewer for recognizing our constructive theory and the close agreement between our theoretical predictions and experimental findings. We appreciate the reviewer’s thoughtful feedback and in-depth engagement with our paper, and found the suggested additional experiments quite insightful. Below we address your concerns:


W1: Limited evidence of spatial routing beyond wraparound

The parents of the target cell at step t − 1 occupy a contiguous block of the stream, so spatial routing is needed only at the two wraparound positions.

We clarify that the contiguity assumption in our abstract refers to prior theoretical constructions, not to an architectural advantage of transformers. Previous Markov-chain constructions [1, 2, 4] assume that relevant tokens immediately precede the query. In contrast, even 1D cellular automata have an underlying 2D space–time geometry: although the parents are adjacent to one another, they are not adjacent to the query in the serialized sequence. Our proofs assume neither adjacency to the query nor contiguity among the relevant tokens. Moreover, self-attention has no intrinsic preference for nearby tokens, and our learned absolute positional embeddings provide no prior favoring a contiguous block (W) positions back over scattered dependencies.

To illustrate that attention does not favor contiguous over scattered dependencies, we ran additional experiments replacing the 1D neighborhood ({-1,0,1}) with ({-3,0,3}), placing the relevant cells in three separate groups in the sequence. Both architectures retain near-perfect accuracy on held-out rules (AR: 0.9991 for Shared-head and 1.0000 for Dedicated-heads; Table R1).

Furthermore, wraparound targets constitute a significant portion of the data, breaking both contiguity and fixed sequence offsets. At these positions, a parent crosses the periodic boundary, so the same spatial displacement corresponds to a different sequence offset. Across all tested 1D settings, our models achieve at least 0.9999 TF and 0.9992 AR accuracy on wraparound targets (Table R1). In 2D, wraparound affects 20 of 36 cells (56%), yet both architectures achieve near-perfect overall TF and AR accuracy on held-out rules (paper Table 1). Thus, the models achieve near-perfect prediction even where a contiguous-block shortcut fails.

Table R1. Cell accuracy on held-out deterministic 1D rules ((V=2,k=3)), rounded to four decimal places. The (W=64) model is the Figure 1 checkpoint ((T=64), 4,160 tokens).

NeighborhoodWArchitectureTF cell acc.AR cell acc.TF on wraparound targetsAR on wraparound targets
{-1,0,1} (main text)16Shared-head1.00000.99991.00000.9998
{-1,0,1} (main text)16Dedicated-heads1.00001.00001.00001.0000
{-1,0,1}64Dedicated-heads1.00001.00001.00001.0000
{-3,0,3}16Shared-head0.99990.99910.99990.9992
{-3,0,3}16Dedicated-heads1.00001.00001.00000.9999

For the spread neighborhood ({-3,0,3}), these comprise 6 of 16 cells per step. The (W=64) experiment further demonstrates accurate routing when the parents are four times farther back in the sequence.

The 2D von Neumann setting scatters the neighbors across the stream, and its grid is 6 × 6 with 504 tokens. The abstract and introduction motivate the work with vision transformers on serialized images, but the evidence that trained transformers solve spatial routing comes from this one small grid.

Regarding scale, a ViT-B/16 at 224px operates on a 14×1414\times14 patch grid (197 tokens); our 2D streams are longer, with 504 and 840 tokens. The grid side length differs by only a factor of 14/6≈2.314/6\approx2.3, and is of the same order of magnitude.

For perspective, a 6×66\times6 binary grid admits 236≈692^{36}\approx 69 billion possible subsets of cells, and the model must learn from data which cells to route into the local context used for prediction.

Moreover, controlled settings are standard for isolating mechanisms: prior work [1, 2, 4] similarly motivates its study through language-model ICL while investigating simplified Markov-chain settings.

W2 & Q3: Limited empirical validation of the theoretical positional construction

The headline theoretical claim, that the positional width needed for routing is independent of grid volume and of the number of time steps, isn’t tested. The trained models use dmodeld_{\text{model}} between 128 and 1024 and a free positional table with at most 504 columns, so it seems the model can learn an arbitrary embedding for each position and need not use the compact Fourier addressing of the construction.

We clarify that Theorem 7 is an existence result: for any grid and horizon, it provides a construction whose positional width is independent of both. The primary goal of our experiments is to test whether gradient descent, at standard widths, discovers the theoretically predicted gather-and-match mechanism. Our empirical results support this by showing that trained models learn the predicted attention patterns, routing and matching behavior.

Our theoretical construction specifies one valid assignment of positional features, whereas the experiments allow gradient descent to learn its own. While our primary focus is on the existence of such constructions, we agree that understanding the positional features learned by gradient descent is an interesting direction. We explore this question further through additional experiments discussed below.

This follows the Markov-chain ICL literature [1, 2, 4]; Edelman et al., for example, validate their construction by showing that trained attention patterns match the proposed mechanism, rather than training at its width or recovering its parameters.

Please report the effective rank of the learned positional table. Please also train models with dpd_p fixed to the width given by Theorem 7, for example 7 for the 2D von Neumann setting. Do these models reach the same accuracy?

Thank you for the suggestion. We ran additional experiments at the positional widths given by Theorem 7 (5 in 1D and 7 in 2D), using both random initialization and initialization with the construction's positional features.

At the construction widths, models initialized with Theorem 7's features achieve near-perfect sequence accuracy, matching the free-table of positional embeddings models that use 128 and 1024 dimensions (Table R2). This confirms that the theoretical widths suffice in the trained architecture. In contrast, random initialization at the same widths does not reach comparable accuracy under the training configurations tested, indicating that gradient descent does not reliably discover the compact solution.

We also report the effective rank of the learned positional tables. The free-table models have substantially higher effective ranks (102.6 in 1D and 195.1 in 2D) than the construction requires, suggesting that gradient descent uses a higher-dimensional representation when given the flexibility.

Table R2. Positional width, effective rank, and accuracy.

SettingPositional widthInitializationTF cell acc.AR cell acc.TF seq. acc.Effective rank
1D, bottleneck (construction width)5Random0.81180.73280.11385.0
1D, bottleneck (construction width)5Theorem 7 construction1.00001.00000.99984.9
1D, free table (main text)128Random1.00001.00000.9994102.6
2D, bottleneck (construction width)7Random0.73400.65040.03066.4
2D, bottleneck (construction width)7Theorem 7 construction1.00000.99990.99916.1
2D, free table (main text)1024Random1.00000.99990.9989195.1

please report the dependence of the positional score on grid displacement

We quantify this by grouping Layer-1 attention logits according to the query–key grid displacement (temporal difference and spatial offset on the torus) and measuring the explained variance (R2R^2). Table R3 compares this against grouping by sequence displacement and random labels. Across both 1D and 2D settings, with either freely learned positional tables or the theory-prescribed positional dimension dpd_p, spatial displacement explains more variance than sequence displacement, suggesting that learned scores reflect spatial structure more strongly than sequential structure.

Table R3. Explained variance of Layer-1 positional scores. Grid (keys at t−1t-1) considers only previous-step keys. Sequence groups pairs by token-stream distance. Random uses random groups as a control.

SettingGrid R2R^2Grid R2R^2 (keys at t−1t-1)Sequence R2R^2Random R2R^2
1D W=16W=16, free table (main text)0.520.760.510.01
1D W=16W=16, dp=5d_p=5, Theorem 7 initialization0.890.860.830.01
2D 6×66\times6, free table (main text)0.670.650.530.005
2D 6×66\times6, dp=7d_p=7, Theorem 7 initialization0.650.470.600.005

W2 & Q3 (continue)

The attention-mass, ablation and probe results are consistent with the gather-and-match decomposition, but they are also consistent with other addressing schemes. The "mechanistic evidence" therefore supports the two-stage functional decomposition rather than the specific constructions of Theorem 7.

We agree that the mechanistic evidence does not uniquely identify the specific Fourier parameterization of Theorem 7. Our primary empirical claim is that trained models discover the proposed gather-and-match mechanism, rather than recover the exact parameters of one theoretical construction.

We hope our additional experiments addressing this concern provide further evidence for the proposed addressing mechanism.

The grid volume ∣G∣|G| or the number of time steps TT are not, as far as I could find, varied in the experiments.

Our paper evaluates a range of grid sizes and time horizons, spanning 16–64 cells, 10–64 steps, and serialized streams of 170–4,160 tokens. We summarize these results in Table R5 for convenience. Performance remains near-perfect across these settings: deterministic TF cell accuracy rounds to 1.0000, AR cell accuracy is at least 0.9999, and stochastic AR KL is at most 0.0044.

Each row corresponds to a separately trained model evaluated on held-out rules, rather than a controlled scaling sweep. NN denotes the serialized stream length, including separators. The final row uses the Figure 1 checkpoint configuration (Table 15), with additional results from Table R1.

Table R5. Performance across grid sizes and time horizons.

TaskRules∣G∣\lvert G\rvertTTNNMetricShared-headDedicated-heads
1D V=2V=2, k=3k=3Det.1610170TF / AR cell acc.1.0000 / 0.99991.0000 / 1.0000
1D V=2V=2, k=3k=3Stoch.1620340AR KL0.00190.0044
1D V=3V=3, k=3k=3Det.3212396TF / AR cell acc.1.0000 / 1.00001.0000 / 1.0000
1D V=3V=3, k=3k=3Stoch.1620340AR KL0.00260.0004
2D VN V=2V=2, k=5k=5Det.3612504TF / AR cell acc.1.0000 / 1.00001.0000 / 0.9999
2D VN V=2V=2, k=5k=5Stoch.3620840AR KL0.00130.0038
1D V=2V=2, k=3k=3 (Fig. 1)Det.64644,160TF / AR cell acc.--1.0000 / 1.0000

...so it seems the model can learn an arbitrary embedding for each position and need not use the compact Fourier addressing of the construction.

The routing also relies on a few spatial frequencies, as in the construction. Removing the 4 strongest frequency groups of the learned table drops accuracy to at most 0.62, far more than removing 4 random groups (Table R10).

Table R10. TF cell accuracy after Fourier ablation of the learned positional table (main-text models).

SettingArchitectureChanceKeep top 4Remove top 4Remove 4 random
1D V=2V=2Shared-head0.50000.99010.58940.6460
1D V=2V=2Dedicated-heads0.50000.98180.61650.9401
1D V=3V=3Shared-head0.33330.59390.40860.7688
1D V=3V=3Dedicated-heads0.33330.79020.46890.9195
2D VNShared-head0.50000.58630.57300.7611
2D VNDedicated-heads0.50000.99660.57350.8122

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.