acceptodds
Withdrawn from ICLR 2027

Can we train a computer? Two mechanisms for content addressing in transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformers can be programmed by hand: with the right weights, one executes the instructions of a computer. Does training find such a solution? We ask this where the answer is checkable. For SUBLEQ, a one-instruction language that is Turing-complete given unbounded memory, we write a correct -layer transformer circuit by hand, then train a second transformer of the same architecture from scratch. Training succeeds: it executes an instruction exactly, and runs steps on its own output on almost every program that keeps running. It does not rediscover our circuit. Both networks need the same primitive, attending to the memory cell whose address is stored elsewhere, and both build it, but they encode it differently. The hand-built circuit gives every position the same key direction and separates positions by distance along it, which forces attention scores spanning about logits (Gaussian addressing); the trained network gives positions their own directions and spikes at the target with scores spanning tens (orthogonal-key addressing). Across trained networks, over seeds, one-instruction variants, depths, and widths, none reaches the hand-built regime. The difference is invisible to the tool one reaches for first. Weight-space mode connectivity reports that the two networks are unrelated, but it reports the same for two seeds of one recipe, so it cannot separate a different algorithm from a different seed; a probe read against the known circuit separates the two mechanisms cleanly. A construction is therefore an existence proof, not a prediction of what training builds, and a claim that two networks compute alike is worth no more than what the instrument behind it can tell apart.

Primary area:
interpretability and explainable AI
Keywords:
Representation learning, interpretability, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qMaPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks the question whether a transformer model converges towards the exact same algorithm as a hand-coded ideal network, using the SUBLEQ language as training data. Two ~3.2M param models are used, one of them learned while the other's weights are chosen by hand. The learned network does not converge towards the hardcoded one. The authors align the two weight vectors and linearly interpolate between them, finding that no connectivity in weight space exists between the two. Attention head probes show that the attention mechanism resembles pointer dereferencing operations, and experiments with weight decay do not result in meaningful changes to the model's behavior.

Critical strengths
  1. The question of whether gradient descent can find optimally constructed hardcoded solutions is somewhat relevant
Critical weaknesses
  1. The paper essentially runs one experiment, training 3M param transformers on a single instruction language, and comparing them to a single hardcoded solution they claim is the theoretical optimum the model should reach. First of all, it is hard to understand why one single hardcoded solution should be any kind of optimum. A slightly more scientific approach would have been to find an exhaustive set of all possible optimal solutions that can be hardcoded using a general rule system for choosing the weights by hand. Otherwise, the non-connectivity between the claimed optimum and the learned solution could just as well be a random disconnect between these two particular solutions. Secondly, running a single set of experiments on a tiny model with a single source of data does not give enough evidence for any of the claims. The generality of the claims should be proven by at least multiple different instruction languages and on multiple larger model sizes, as well as with insightful ablations.

  2. There is ample research on this exact question. For example Xu et al. (https://arxiv.org/abs/2604.25166) train transformers on MicroPy outputs, Liu et al. let models execute complex python programs (https://aclanthology.org/2023.findings-acl.308/), as well as work on cellular automata (https://arxiv.org/abs/2412.01417), and the work is even closely connected to the literature of neural turing machines, just to name a few:

Questions for authors
  1. How do you train your model on SUBLEQ? I understand that you use a BERT-style transformer, but then what are examples for inputs, and what are the targets?
  2. Why did you use an encoder-only transformer instead of a decoder-only one?
  3. Why did you only use a single hardcoded solution instead of proving the set of optimal solutions that can be hardcoded?
  4. Why did you not test your claims on other instruction languages as well as other model types or sizes?
Reviewer_1wzhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether transformers trained to execute SUBLEQ instructions learn the same internal mechanisms as a hand-coded transformer. The authors construct a four-layer transformer using Gaussian addressing and train another transformer of the same architecture on single-step state transitions. Both achieve high execution accuracy, but the trained model uses less-aligned keys and spike-shaped attention scores rather than the quadratic scores of the hand-coded construction.

The paper further compares the two solutions using weight-space mode connectivity and mechanistic probing. The authors find that the tested alignment methods fail to connect even independently trained networks, while attention-head probes identify shared pointer-dereferencing operations. These observations are examined across multiple seeds, instruction variants, and architectures, together with sparsification experiments.

Critical strengths
  1. The comparison between a hand-coded and a trained transformer provides a controlled setting for studying internal computation. The authors identify concrete differences in attention geometry despite similar execution behavior.

  2. The empirical analysis is reasonably thorough. The observations are consistent across seeds, instruction variants, and architectures. The mode-connectivity experiments also provide useful evidence that failure to find a low-loss path does not necessarily imply different addressing mechanisms.

Critical weaknesses
  1. Limited insight into the learned mechanism. The main finding is that the trained transformer uses a different addressing mechanism from the hand-coded construction. However, Gaussian addressing is explicitly chosen in the construction, and Appendix D shows that orthogonal-key addressing can also be constructed by hand. The observed difference therefore depends heavily on the choice of reference. More importantly, the paper does not explain why training consistently produces the observed mechanism. The analysis of score ranges suggests a possible explanation, but does not establish what drives this preference during optimization. As a result, the main contribution remains largely descriptive.

  2. Insufficient evidence of algorithm learning. The paper does not clearly specify a disjoint train/test split or examine overlap between training and evaluation states. Evaluation appears to rely primarily on samples from the training distribution. Given the fixed 32-cell machine and finite address space, it remains unclear to what extent the model has learned the SUBLEQ execution algorithm rather than finite input-output mappings or address-specific rules. The attention probes provide some evidence of pointer dereferencing, but do not establish that the learned computation generalizes across unseen operand configurations. A more rigorous evaluation is needed to support the claim of algorithm learning.

  3. Incomplete characterization of the learned computation. The mechanistic analysis identifies a few attention heads that perform canonical addressing operations, but these explain only part of the computation. Keeping the four identified heads yields only 44.8% accuracy on changed positions, while removing one non-canonical head reduces accuracy to 50.1%. The learned algorithm therefore remains largely uncharacterized beyond the observed attention geometry. This limits the conclusions that can be drawn about shared computational primitives and differences between the two implementations.

  4. Limited conclusions from the mode-connectivity analysis. The failure of the tested alignment methods to connect the hand-coded and trained networks is not particularly informative given their substantial parameter-scale differences. The additional finding that independently trained networks also exhibit loss barriers is more interesting, but does not establish that these networks implement the same complete algorithm. Moreover, the paper does not evaluate the stronger transformer alignment methods discussed in the related work. The results demonstrate limitations of the tested methods, rather than a general limitation of mode connectivity for identifying shared algorithms.

Questions for authors
  1. Why does training consistently produce orthogonal-key addressing? The difference in attention score ranges offers a possible explanation, but what evidence supports this as the cause? An analysis of how key alignment and attention scores evolve during training, or controlled experiments on the factors influencing the learned mechanism, would substantially strengthen the contribution.

  2. How are training and evaluation states separated? Is there any overlap between them? How does the model perform on explicitly held-out PC values, operand configurations, or combinations of memory addresses? What evidence distinguishes learning the SUBLEQ execution algorithm from learning finite input-output mappings for the fixed machine?

  3. How much of the main observation depends on the hand-coded reference? Appendix D already demonstrates that orthogonal-key addressing can be constructed analytically. Would a complete hand-coded executor using this mechanism provide a more meaningful comparison with the trained models? What conclusions about training would remain independent of the chosen reference?

  4. Can the authors further characterize the computation performed by the trained network? In particular, what is the role of the non-canonical head L2H0, whose removal substantially reduces accuracy? Given that the identified canonical heads explain only part of the computation, how confidently can the authors characterize the similarities and differences between the learned and hand-coded algorithms?

  5. Can the mode-connectivity analysis be extended beyond the tested alignment methods? In particular, could the stronger transformer alignment methods of Theus et al. (2025), already discussed in the paper, reduce the barriers between independently trained networks? What additional evidence supports treating these networks as implementing the same algorithm, rather than merely sharing similar addressing geometry?

Reviewer_EHLKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies hand-crafted versus learned transformers for SUBLEQ, a Turing-complete programming language. The key to the implementation is content addressing (looking up the contents of a given memory location), and the hand-crafted solution uses "Gaussian addressing" (which uses the integer i itself for position i) and "orthogonal-key addressing" (which uses random vectors for different positions).

The first experiment tries to distinguish the two solutions using weight-space mode connectivity (interpolating between the two solutions' parameters to see if the solution breaks). Although it does distinguish them, it also distinguishes between two models with the same addressing method but different settings. So weight-space mode connectivity is judged a failure.

Critical strengths

The question posed by this paper is generally interesting, and the particular issue of Gaussian vs orthogonal addressing is a very important one, I think maybe the most important divide between constructions in theory papers versus transformers in practice.

The finding that transformers automatically learn orthogonal addressing and not Gaussian addressing is interesting (though perhaps not surprising).

Critical weaknesses

The paper exhibits writing problems that are typical of language models: unmotivated statements, undefined terminology, unexplained metaphors, incomplete sentences.

It's not very clear to me how the findings 1-3 (lines 074-084) answer the question, "When a transformer is trained from random initialization to execute an algorithm that admits a clean hand-designed solution, does gradient descent find that solution?" (line 043-045). Finding 1 is interesting but doesn't seem to me to answer the question. Finding 2 seems relevant to the question but I could not understand what the answer was. Finding 3 is just a recitation of some facts and numbers?

The paper is missing a great deal of information about how the two transformers are constructed/trained, and the task on which they are tested.

I would say that more background is needed on the probing methodology; the paper seems to be assuming that the reader already knows what it is.

Questions for authors
  • Could you spell out more how findings 1-3 answer the question posed by the introduction?
  • Could you provide some more explanation of the probing methodology in sections 6-8?
  • Can you say something about length generalization? Is it the case that Gaussian addressing solution has perfect length generalization, but the learned orthogonal addressing transformers have none?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.