acceptodds
Withdrawn from ICLR 2027

Probing transformer reasoning transfer through alphabet equivalence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A reasoner is abstract if it applies a general procedure to a problem's structure, indifferent to how that structure is concretely instantiated; it is concrete if what it learned is tied to the particular instantiation trained on. This gives a testable marker: an abstract reasoner should transfer mastery of one problem instance to another. We study equivalence via alphabet renaming, and test transfer using a weight-transplant control – copying trained per-symbol weights onto the new instance without retraining – distinguishing genuine reasoning failure from symbols never trained on. We instantiate this with Sudoku, where an equivalent puzzle is any relabeling of its digits. A transformer trained on one such puzzle transfers to an equivalent one no better than an untrained network – yet the transplant control recovers its original performance exactly. Reasoning, in that narrow sense, is abstract; recognizing that a new instance calls for it is not. We probe this recognition gap three ways. Freezing the shared network and adapting only a new alphabet's parameters fares worse than training an unfrozen model from scratch on the same data: prior exposure to other alphabets does not ease learning a new one. Recombining already-known symbols into unfamiliar combinations yields graded, partial success – above chance, short of the seamless transfer an abstract reasoner should show, degrading as combinations diverge from training. Replacing the model's fixed, per-alphabet output vocabulary with a mechanism that selects among a puzzle's own visible symbols helps substantially with recombination, but only marginally with transfer to a truly unseen alphabet. These results isolate a specific failure to recognize alphabet-equivalent instances as such, distinct from a general failure to reason, and show several natural interventions narrow, but do not close, this gap.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
systematic generalization, compositional generalization, transformers, combinatorial reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rfsVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The submission evaluates whether transformer-based models trained on Sudoku puzzles with one alphabet can generalize to Sudoku puzzles with a different alphabet. It finds that models fail to generalize to alphabets whose corresponding tokens have never been encountered during training. When the model is trained on diverse alphabets that mix a set of known symbols, they can they generalize to novel alphabet combinations.

Critical strengths

None

Critical weaknesses
  • The main finding that models do not generalize to tokens not encountered during training is not surprising. There is a rich body of literature on compositional generalization and the role of symbol substitutivity that has long established similar findings but is hardly mentioned
  • The paper has severe presentation issues, it uses very jargon-heavy language, making it difficult to follow.
Questions for authors

The classifier head’s disjoint per-alphabet output rows mean it cannot use a row it never trained; that much is architectural. What is not settled is how total the resulting failure is, since a shared encoder processing both alphabets could plausibly leak some above-chance signal even through untrained output rows.

Can you explain what reason there is to expect anything other than chance performance? What is meant with "a shared encoder processing both alphabets"? Has the encoder been trained on the held-out alphabet while the head hasn't?

Reviewer_tWaYOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tests whether a transformer trained on Sudoku in one symbol alphabet transfers to the same puzzles in another. Naive transfer is at chance; a weight-transplant control (copying the trained alphabet’s embedding and output rows into the new alphabet’s slots) recovers full accuracy, which the paper reads as the solving procedure being reusable and only symbol recognition failing. It then probes this gap with few-shot adaptation (4.2), recombination of known alphabets (4.2–4.3), a pointer head (4.3), and mixed-alphabet training (4.4–4.5). Mixing during training solves recombination; zero-shot transfer to a never-trained alphabet stays at 0% exact match throughout, which the paper takes as evidence that an architectural change is needed.

Critical strengths

The paper’s framing of equivalence under relabeling is a reusable way to pose the question. The setup is tightly controlled, all alphabets share identical underlying puzzles, and 4.5 correctly separates pool size from mixing exposure. The writing is clear.

Critical weaknesses
  1. The held-out alphabet E does not test what the paper claims. E’s embedding row is untrained, so zero-shot E asks whether the model can handle an arbitrary random vector as a symbol. The paper’s motivating question is whether a model recognizes a known problem under a different surface form, where in any realistic setting the alternative tokens already have trained representations. The experiment that would test this, training B’s tokens on a non-Sudoku task and then evaluating B-Sudoku zero-shot, is absent. The 4.3–4.5 zero-shot conclusions and the architectural recommendation should be reframed around what E actually measures.

  2. The transplant control (4.1) is uninformative by construction. Copying A’s embedding and output rows into B’s slots and evaluating B(P) with the argmax restricted to B’s logits computes exactly the same function as evaluating A(P): identical embedding vectors, identical encoder, identical restricted logits. Exact recovery is therefore guaranteed, and so is naive failure, since B’s rows are random. 4.1 cannot distinguish a “representational gap” from untrained parameters because there is nothing else in this setup for a gap to consist of, so the headline claim that reasoning is abstract and only recognition fails is not an empirical finding.

  3. “Architecture is necessary” is premature. Growing the pool from 6 to 25 alphabets leaves 225 fixed, memorizable symbol embeddings, so there is still no pressure to treat an unseen vector generically. The natural endpoint of this axis, sampling fresh random embeddings for every puzzle so memorization is impossible, is untested and costs one pretraining run. Until it is run, the paper has shown that a small increase in pool size doesn’t help, not that data-side interventions can’t.

  4. The few-shot result (4.2) is read as entanglement when it is probably an optimization result. By 4.1’s transplant, copying any trained alphabet’s rows into E’s slots should give ~99%, so a near-perfect solution exists within the exact parameters finetune trains. Finetune reaching only 39.5% therefore shows that gradient descent on 1000 examples does not find a solution that exists, not that the frozen body is “entangled with specific alphabets.” The from-scratch comparison doesn’t change this, since from-scratch has every parameter free.

Questions for authors

See weaknesses.

Reviewer_DDgKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work tests if the reasoning learned by small Transformers is dependent on symbols it's trained on. They used Sudoku as an example, when changing the symbols of the puzzle (puzzle structure & solution unchanged), model accuracy drop from % to roughly random guessing. Copying input and outputparameters corresponding to the old symbols, however, revover the performance. This shows that the reasoning can be found back when the LLM know the corespondance.

Critical strengths
  1. The problem is well-defined and the experiment design is convincing. They isolates LLM's sensitivity to symbol identity by just relabeling the symbols is clean.
  2. The authors find that when the LLM have ever seen part of the symbols swapping in, it will perform better than under the situation when the LLM never seen the symbols. This opens a door in resolving the identified issue.
Critical weaknesses
  1. The performance recovery by parameter transplantation does not establish abstract reasoning. It, by construction, restores the original internal representations, so I think recoverying the performance is actually expected.
  2. The adaptation comparison confounds the benefit of pretraining with restrictions on trainable parameters. Spacifically, the pretrained model is largely frozen, whereas the scratch model is fully trainable. I think a fully fine-tuned pretrained baseline is needed to assess whether prior learning helps.
Questions for authors
  1. Beyond parameter transplanation, is there any evidence that we can distinguish a failure of symbol recognition from a failure of reasoning generalization?
  2. Does a pretrained model learn the new symbols faster than a model trained from scratch when all parameters are allowed to update?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.