acceptodds
Under review as a conference paper at ICLR 2027

Can a One-Layer Transformer Solve Hard Sudoku? Test-Time Compute, Generalization, and Mechanistic Analysis

Abstract

Modern AI agents pair a language model with a harness that executes actions, maintains state, and returns feedback. We study where the boundary between the two lies in its simplest form: a one-layer, one-head Transformer paired with a simple harness, on Sudoku. At each step, the Transformer observes the grid and the candidates of every cell and selects either a cell–digit placement or ; the harness applies the action, propagates constraints, and keeps the search history needed to backtrack. We train the Transformer by imitating a minimum-remaining-values (MRV) solver on easy puzzles only. We show that the Transformer generalizes from easy to hard puzzles: applied repeatedly, it solves all held-out easy, medium, and hard puzzles and of extreme puzzles, whose searches run to thousands of actions, and harder puzzles need only more test-time compute. Thanks to its simple structure, we can trace how the model computes each action: attention selects the cell with the fewest candidates, the MLP favors the teacher's digit at that cell, and a dead-end cell shifts the output to . With only M parameters, the Transformer solves hard Sudoku almost perfectly, while prior single-model solvers use M to M parameters.

Primary area:
interpretability and explainable AI
Keywords:
Looped transformers, mechanistic interpretability, out of distribution generalization, memorization-to-generalization transition.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_84WgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether a one-layer, one-head Transformer can solve Sudoku when paired with an external harness that handles candidate tracking, constraint propagation, and backtracking. Trained by imitating a minimum-remaining-values (MRV) solver on easy puzzles, the 0.87M-parameter model reportedly solves all evaluated easy, medium, and hard puzzles.

Critical strengths

The small architecture enables a detailed mechanistic analysis, supported by interventions on inputs and activations. The problem of Sudoku is a very well explored problem.

Critical weaknesses

The presentation would benefit from clearer explanations of key experimental choices in the main text. Difficulty is defined through preprocessing-solver backtrack counts, but the thresholds and training details are largely deferred to the appendix. Blue emphasis within the prose is visually distracting, although primarily a stylistic issue. Experiments cover only Sudoku, limiting evidence for broader applicability, and parameter-count comparisons require care because the harness performs substantial computation. The related work also omits "Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error (NeurIPS 2025)", which studies Transformer-based Sudoku solving with guessing and backtracking.

Questions for authors
  1. Could you clarify the motivation for the solver-based difficulty thresholds in Appendix B.1 (easy: 1–9 backtracks; medium: 10–99; hard: 100–999)? How sensitive are the results to the preprocessing solver and these thresholds?

  2. Appendix B.5 reports 1,000 training epochs on a fixed set of teacher traces with online symmetry augmentation. What evidence rules out memorization of this training pool, and are symmetry-equivalent puzzles excluded across training and evaluation? Have the authors considered comparing this setup with streaming training on freshly generated puzzles to assess generalization further?

  3. How well does your approach generalize beyond Sudoku-Extreme? Could you evaluate the trained model on independently generated puzzles from SudokuPy and on David G. Radcliffe’s 3 Million Sudoku Puzzles with Ratings (2020)? Such experiments would help assess robustness across data sources and difficulty distributions.

  4. Could you clarify the relationship between your approach, Shah et al., Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzles (NeurIPS 2024), and the work of Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error (NeurIPS 2025)? A direct experimental comparison would help establish the advantages of your model and harness.

3 comments

We thank the reviewer for recognizing that our small model allows a detailed mechanistic analysis and for suggesting that we evaluate it on independently generated Sudoku puzzles. Please check the revised PDF.

W1. Critical weaknesses

The presentation would benefit from clearer explanations of key experimental choices in the main text. Difficulty is defined through preprocessing-solver backtrack counts, but the thresholds and training details are largely deferred to the appendix. Blue emphasis within the prose is visually distracting, although primarily a stylistic issue. Experiments cover only Sudoku, limiting evidence for broader applicability, and parameter-count comparisons require care because the harness performs substantial computation. The related work also omits "Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error (NeurIPS 2025)", which studies Transformer-based Sudoku solving with guessing and backtracking.

Response.

  • Presentation and experimental choices. We now state the difficulty thresholds and key training settings explicitly in the main text and reduce the blue emphasis. Q1 explains the order-of-magnitude grouping and shows increasing search effort under both our teacher and tdoku.

  • Broader applicability. Sudoku allows us to study reliable composition over long searches while keeping the rules fixed. The additional 318,000-puzzle external evaluation establishes transfer across Sudoku sources (Q3). Extensions to other constraint satisfaction problems remain future work; our response to reviewer zfww’s Q1 describes the formulation and required adaptations.

  • Environment computation. We have measured the environment’s work and total solving time, including initialization and failed branches, as reported in our response to reviewer j3mn’s Q2. We will present these measurements alongside the parameter count, which describes the learned policy.

  • Related work. The omission of Giannoulis et al. (2025) was an oversight, and we have added it. Q4 explains the differences in search representation, model–environment interaction, and mechanistic analysis. The comparison in Q3 additionally places our external evaluation alongside the published results of Shah et al. and Giannoulis et al.

We have incorporated these clarifications and additional results into the revised manuscript, including the difficulty analysis, overlap checks, external evaluations, and comparison with Shah et al. and Giannoulis et al. We also clarify the respective roles of the model and harness.

Q1. Difficulty criterion

Could you clarify the motivation for the solver-based difficulty thresholds in Appendix B.1 (easy: 1–9 backtracks; medium: 10–99; hard: 100–999)? How sensitive are the results to the preprocessing solver and these thresholds?

Response.

We define difficulty through preprocessing search effort: each failed candidate assignment counts as one backtrack. The thresholds—1–9, 10–99, 100–999, and at least 1,000—separate orders of magnitude of search cost. Reasoning length captures an important aspect of difficulty in sequential tasks, including multi-step mathematical reasoning and long-horizon agent tasks: longer solutions require reliable decisions over more steps. Our Sudoku setting studies this aspect while keeping the rules fixed. Appendix B.3 reports median teacher forward-decision counts of 7, 26, 231, and 791 across these bands.

The learning experiments provide complementary evidence. Under a fixed training budget, Figure 3(b,c) shows that the transition from memorization to generalization occurs at larger training-set sizes for hard and extreme puzzles.

To assess whether our difficulty grouping is meaningful under another solver, we compared the same evaluation puzzles using tdoku, a C++ Sudoku solver with stronger constraint propagation. Its backtrack-based ratings are provided with the source dataset. Median tdoku backtrack counts increase from 1 to 4, 28, and 34 across our easy, medium, hard, and extreme groups. This supports the overall increase in search difficulty under an independent grader.

Q2. Training-evaluation overlap

Appendix B.5 reports 1,000 training epochs on a fixed set of teacher traces with online symmetry augmentation. What evidence rules out memorization of this training pool, and are symmetry-equivalent puzzles excluded across training and evaluation? Have the authors considered comparing this setup with streaming training on freshly generated puzzles to assess generalization further?

Response.

Our overlap audit supports generalization beyond the fixed teacher traces. Across the paper’s 2,000 evaluation puzzles and 4,500 additional extreme puzzles, we found zero matches with the 5,000 training puzzles, including under all rotations, reflections, and digit relabelings used in training augmentation.

We also checked 10,566,302 intermediate states along teacher trajectories, up to 8,192 actions per puzzle. Comparing both the grid and candidate state against the entire 158,215-state teacher pool, including validation states, yielded zero exact or symmetry-equivalent matches. On these unmatched states, the model achieves 99.9992% agreement with the teacher’s action, requiring both cell and digit to match for forward actions and REVERT at dead ends. These decisions therefore occur on states that are neither training examples nor their augmented copies.

These findings complement the held-out solve rates, causal mechanism analyses, and memorization-to-generalization transition in Figure 3(b,c): fitting small training sets alone does not produce successful solving, whereas larger datasets yield reliable held-out performance.

We train on offline teacher traces from only 5,000 easy puzzles, with online symmetry augmentation, to test whether a small, fixed training pool suffices for generalization. Streaming fresh puzzles could provide broader coverage and less repetition, potentially making generalization easier. We therefore focus on demonstrating generalization from this limited, fixed pool and do not include streaming training in this study.

Q3. Generalization beyond Sudoku-Extreme

How well does your approach generalize beyond Sudoku-Extreme? Could you evaluate the trained model on independently generated puzzles from SudokuPy and on David G. Radcliffe’s 3 Million Sudoku Puzzles with Ratings (2020)? Such experiments would help assess robustness across data sources and difficulty distributions.

Response.

Our model learns a general, MRV-guided search procedure for Sudoku through interaction with the harness. The model selects cell–digit assignments or REVERT, while the harness updates the state and maintains the backtracking stack. This allows the same learned procedure to be applied across puzzle sources through a common state representation and evaluation protocol. We therefore evaluate the unchanged model and harness on independently sourced puzzles to test whether this procedure generalizes beyond the training distribution, without fine-tuning.

We evaluated the paper’s 1,000-epoch checkpoint on 100,000 independently generated SudokuPy puzzles, 100,000 puzzles uniformly sampled from Radcliffe’s collection, the 100,000-puzzle Kaggle-filtered test split, and the 18,000-puzzle RRN test set, using the same environment and unmasked greedy decoding.

Complete-puzzle solve rates (%) are:

MethodParametersRandom / SudokuPyKaggle-unfilteredKaggle-filteredRRN
Shah et al. (2024), greedy42M67.16†67.1687.1894.76
Giannoulis et al. (2025), DFS42M98.9098.9099.5099.40
Ours, greedy, 49,152 actions0.87M99.998100.0099.99999.16
Ours, greedy, 8,192 actions0.87M99.99699.99799.99997.06

† Giannoulis et al. (2025) estimate Shah’s Random accuracy from its Kaggle-unfiltered performance. Prior-work values are published results; ours use the stated action budgets. We use the released Kaggle-filtered and RRN test sets, while SudokuPy and Kaggle-unfiltered are fresh samples. Parameter counts describe the learned models; our policy operates alongside the environment.

Across all 318,000 external puzzles, we found zero training-puzzle matches, including under the rotations, reflections, and digit relabelings used in training augmentation. Excluding puzzles solved by initial singleton fills, solve rates at 49,152 actions remain 99.998% on SudokuPy and 100% on the unfiltered Radcliffe sample.

Q4. Relationship to Shah et al. (2024) and Giannoulis et al. (2025)

Could you clarify the relationship between your approach, Shah et al., Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzles (NeurIPS 2024), and the work of Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error (NeurIPS 2025)? A direct experimental comparison would help establish the advantages of your model and harness.

Response.

We agree that the relationship should be stated explicitly. The omission of Giannoulis et al. (2025) was an oversight, and we will add it to the related work. The key distinction across these studies is how search is represented and how computation is divided between the model and its environment.

Shah et al. (2024) train an autoregressive Transformer on solver-ordered, mistake-free solution sequences and probe its candidate representations. Giannoulis et al. (2025) extend transcript-based solving to guessing and backtracking, and further study efficient depth-1 guessing. Both use approximately 42M-parameter Transformers.

Our harness has two roles: executing state transitions and maintaining the information needed for backtracking. The one-layer, one-head, 0.87M-parameter policy selects each cell–digit assignment or REVERT. The harness performs candidate updates, singleton fills, and state restoration using its backtracking stack.

This separation mirrors how models and harnesses work together in practice. For example, in coding agents, the model chooses edits and tool calls, while the file system stores current files and Git maintains version history. Our setting makes this interaction explicit, allowing us to investigate how a small learned policy works with a stateful harness to carry out reliable search.

The benefit becomes important over long searches. A full transcript of explored branches can grow beyond a fixed context window. Our harness maintains the active backtracking stack, while the policy receives only the current grid and candidate state. This allows the same small model to make thousands of decisions without processing the entire trajectory and makes its decision mechanisms directly analyzable.

We train on only 5,000 easy puzzles, with trajectories capped at 200 forward actions and online symmetry augmentation. The resulting policy generalizes to much longer searches and, without fine-tuning, to the 318,000 external puzzles reported in Q3. These evaluations include the released Kaggle-filtered and RRN test sets, plus independently generated SudokuPy puzzles and a fresh Radcliffe sample.

The comparison in Q3 provides evidence about solving accuracy across these sources. Training data, inference procedures, and the allocation of computation differ across methods, so the parameter counts describe the learned components rather than a matched comparison of total computational cost. Our contribution is reliable generalization from short, easy traces together with an explicit, causally tested explanation of the learned search decisions.

We hope the additional results and manuscript revisions address your concerns, and we would appreciate your reconsideration of the rating in light of this new evidence.

Reviewer_j3mnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that a one-layer, one-head Transformer can imitate an MRV policy within an environment that maintains candidates, propagates constraints, and backtracks, then traces how the learned policy remains reliable over longer Sudoku searches.

Critical strengths
  1. The policy's reliability over long rollouts is worth studying. Trained on short easy traces, it solves all 500 test puzzles in each of the easy, medium, and hard categories, and 99.8% of extreme puzzles. Actions are unmasked, and a single illegal choice stalls greedy execution, making the low error rate consequential.

  2. The mechanistic analysis is the strongest part of the paper. QK decomposition, enumeration of 512 candidate sets, MLP ablations, and candidate-projection interventions connect candidate count and digit priority to action selection. The appendix retains attention counterexamples and finite-precision errors, which helps establish the explanation's actual scope.

Critical weaknesses
  1. The model-harness boundary needs a direct baseline comparison. Appendix B.3 explains that completeness follows from the environment as long as the policy chooses legal forward actions and reverts at dead ends. Only about 3.1 to 5.4 forward decisions survive on the solution path on average; propagation fills the remaining cells. The MRV teacher comparison confirms imitation, but random legal choices, fixed-order choices, or a small candidate-feature network would show what the Transformer adds to search efficiency. The parameter-count comparison with 5M to 27M single-model solvers also needs to account for this external computation.

  2. The scope of easy-to-hard generalization is narrower than the headline may suggest. In Appendix F.2.5, 99.898% of teacher forward decisions involve two candidates. Harder puzzles mostly require more applications of the same local rule. This supports length generalization, with limited change in local decision complexity. The main solve rates and mechanistic account also concern one trained model; multiple seeds in the data-size sweep do not establish replication of that result.

Questions for authors

How do random-legal and fixed-order policies perform under the same harness and action budgets? Please report propagation work and total runtime alongside policy actions. Does a second independently trained model reproduce the solve curve and the count, tie-breaking, and revert mechanisms? Accuracy stratified by minimum candidate count would help delimit the generalization claim.

2 comments

We thank the reviewer for recognizing the reliability of the policy over long searches and the value of the mechanistic analysis. We address the model–environment boundary, computational cost, replication, and the scope of generalization below. Please check the updated PDF.

W1. Contribution of the learned policy and environment computation

The model-harness boundary needs a direct baseline comparison. Appendix B.3 explains that completeness follows from the environment as long as the policy chooses legal forward actions and reverts at dead ends. Only about 3.1 to 5.4 forward decisions survive on the solution path on average; propagation fills the remaining cells. The MRV teacher comparison confirms imitation, but random legal choices, fixed-order choices, or a small candidate-feature network would show what the Transformer adds to search efficiency. The parameter-count comparison with 5M to 27M single-model solvers also needs to account for this external computation.

Response.

We show how a small learned policy and a stateful harness jointly support reliable search. The policy selects a cell–digit assignment or REVERT, while the harness performs candidate updates, singleton fills, and state restoration. This division enables a one-layer Transformer to make decisions over long searches and allows us to identify and causally test the computations behind those decisions.

The 3.1–5.4 surviving decisions count assignments retained on the final solution path. Complete searches also explore failed branches, with the teacher averaging 1,222 forward decisions on extreme puzzles.

To quantify the policy’s contribution, we compare policies under the same harness and action budgets. At 8,192 actions, random-legal, fixed-order, and Transformer policies solve 5.4%, 84.8%, and 97.4% of extreme puzzles, respectively. These results show how the learned decisions improve search efficiency. Q1 reports all four difficulties. Q2 reports the environment’s work and total solving time, including initialization and failed branches, alongside the 0.87M learned-policy parameter count.

W2. Scope of generalization and independent replication

The scope of easy-to-hard generalization is narrower than the headline may suggest. In Appendix F.2.5, 99.898% of teacher forward decisions involve two candidates. Harder puzzles mostly require more applications of the same local rule. This supports length generalization, with limited change in local decision complexity. The main solve rates and mechanistic account also concern one trained model; multiple seeds in the data-size sweep do not establish replication of that result.

Response.

Our controlled benchmark investigates whether decisions learned from short trajectories remain reliable when composed over much longer searches. We will make this scope explicit. Q4 now reports full-policy agreement by minimum candidate count, including the less frequent three-candidate states, while identifying their presence in training.

We also trained three additional models for 1,000 epochs. They reproduce the solve curves closely, achieving 99.4–100% on extreme puzzles within 49,152 actions, and recover consistent candidate-count attention, positional tie-breaking, and REVERT behavior. Q3 reports these independent replication results.

We have added the baseline comparisons, computational cost measurements, independent training runs, and full-policy agreement results to the revised manuscript. These additions clarify the learned policy’s contribution and strengthen the evidence for reliable generalization over longer searches.

Q1. Random-legal and fixed-order baselines

How do random-legal and fixed-order policies perform under the same harness and action budgets?

Response.

We evaluated all policies on the same 500 held-out puzzles per difficulty, using the same environment and action budgets. Random-legal results are averaged over five seeds. Each entry below gives the solve rate at 8,192 / 49,152 actions:

DifficultyRandom legalFixed orderOur Transformer
Easy72.2% / 84.6%99.8% / 100%100% / 100%
Medium52.2% / 70.4%99.2% / 100%100% / 100%
Hard12.6% / 38.3%98.6% / 100%100% / 100%
Extreme5.4% / 21.8%84.8% / 98.8%97.4% / 99.8%

All policies share candidate updates, singleton fills, and state restoration. The hand-coded baselines backtrack automatically at dead ends, while our model predicts REVERT itself. The results show improved search efficiency, especially on extreme puzzles at the smaller budget.

Q2. Propagation work and total runtime

Please report propagation work and total runtime alongside policy actions.

Response.

We measured mean per-puzzle cost on 500 held-out puzzles per difficulty, using batch size one, an NVIDIA A40 for policy inference, and CPU environment operations, with a 65,536-action cap.

DifficultyPolicy actionsCandidate updatesSingleton fillsPolicy / environment time (s)Total time (s)
Easy69.23,0638430.081 / 0.0570.146
Medium189.08,6912,3780.221 / 0.1530.397
Hard602.528,5087,8750.696 / 0.5241.290
Extreme2,427.4102,45128,1532.848 / 1.7434.872

Candidate updates count candidate-bit removals, and singleton fills are automatic assignments performed by the environment. Counts include initial propagation and repeated work on failed branches. Total time includes initialization, state encoding, policy inference, environment operations, and backtracking. Every policy action counts, including REVERT and invalid actions, and averages include puzzles that exhaust the budget.

Q3. Independently trained models

Does a second independently trained model reproduce the solve curve and the count, tie-breaking, and revert mechanisms?

Response.

Our results are reproducible and consistent across independent training runs. We trained three additional models for 1,000 epochs with different random seeds, keeping the teacher data, training–validation split, architecture, and optimization settings fixed. All three reproduce the original solve curves on easy, medium, and hard puzzles, reaching 100% within 8,192 actions. On extreme puzzles, solve rates are 97.4–97.6% at 8,192 actions and 99.4–100% at 49,152 actions, compared with 97.4% and 99.8% for the original model.

The mechanism analyses also recover consistent candidate-count attention, positional tie-breaking, and REVERT responses. Each model selects the MRV cell on all 2,154 tested forward states, and attention follows the modified candidate counts in all 443 interventions. Each model also predicts REVERT on all 5,344 sampled dead-end states, with no false REVERT on 6,068 clean states.

Q4. Accuracy by minimum candidate count

Accuracy stratified by minimum candidate count would help delimit the generalization claim.

Response.

Our easy-to-hard setting studies whether a learned local decision rule remains reliable when composed over much longer searches. To complement the candidate-count and attention analyses in Appendix F.2.5, we report full-policy agreement with the MRV teacher, requiring both cell and digit to match for forward actions and REVERT at dead ends.

We evaluate teacher-trajectory states up to 8,192 actions per puzzle. Each entry gives correct actions / total and accuracy:

Minimum candidate countPaper’s 2,000 puzzles—500 per difficultyExpanded 5,000 extreme puzzles
2746,262 / 746,263 (99.9999%)5,076,243 / 5,076,252 (99.9998%)
3953 / 953 (100%)5,303 / 5,382 (98.53%)
≥4No observed statesNo observed states
0: REVERT required738,309 / 738,309 (100%)5,054,297 / 5,054,297 (100%)

Three-candidate states are present in training but constitute only 0.197% of training forward states (154/78,263). The expanded extreme results quantify reliability on this less frequent category, while our solve curves establish reliable composition over longer searches.

We observed no states with minimum candidate count at least four in the training set or in the evaluated teacher trajectories, which were capped at 8,192 actions per puzzle. Complementary evaluations on 318,000 external Sudoku puzzles, reported in our response to reviewer 84Wg’s Q3, demonstrate transfer across puzzle sources without fine-tuning.

We hope the additional results and manuscript revisions address your concerns.

Reviewer_zfwwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is quite interesting study of transformer and its interpretability.

Critical strengths

The paper proposes quite interpretable and explainable description how transformer works in simple, but non-trivial task.

Critical weaknesses

It is not so clear how it can be generalised to more complex systems than Sudoku

Questions for authors

Can the approach be generalised to other cases then Sudoku ?

Is similar interpretability is expected ?

Can one estimate training resources for transformers compering to say MLP and tradeoff with potential gain in performance ?

1 comment

We thank the reviewer for recognizing the value of our mechanistic analysis and for the questions about broader applicability and computational cost. Please check the revised PDF.

W1. Broader applicability

It is not so clear how it can be generalised to more complex systems than Sudoku

Response.

We chose Sudoku as a controlled testbed for investigating how a learned policy and a stateful harness jointly support reliable search. The formulation provides a natural starting point for other constraint satisfaction and search problems, including graph coloring, maze solving, and N-queens. Q1 describes a graph-coloring extension and a possible additional experiment beyond Sudoku.

We have expanded the discussion of extensions beyond Sudoku and how similar mechanistic analyses could apply to other tasks. We also report the training cost and clarify why we chose a small Transformer for this study.

Q1. Extension to other constraint satisfaction problems

Can the approach be generalised to other cases then Sudoku ?

Response.

We focus on Sudoku as a controlled testbed in which explicit constraints, candidate sets, and search states allow us to investigate model–environment interaction and explain the learned decisions.

The formulation provides a natural starting point for other constraint satisfaction and search problems, such as graph coloring, maze solving, and N-queens. For example, in graph coloring, the policy could select a vertex–color assignment or request backtracking, while the environment maintains available colors, propagates constraints, and restores previous states. The minimum-remaining-values heuristic used by our teacher also applies to this setting.

Such an extension would require adapting the state representation, actions, and environment, generating teacher trajectories, and retraining the model. We are considering an additional experiment on one of these tasks to evaluate the formulation beyond Sudoku.

Q2. Interpretability in other domains

Is similar interpretability is expected ?

Response.

Our small architecture and explicit state representation allow us to decompose attention scores, trace the information routed to the output token, and causally test action selection.

The detailed mechanisms need not match across tasks. However, the core computations of selecting a relevant variable, routing its information, and computing an action may recur in other constraint satisfaction problems. We therefore expect similar analyses to be useful for small policies in these settings. An extension experiment would also allow us to test which computations are shared and which depend on the task.

Q3. Training resources and the choice of a Transformer

Can one estimate training resources for transformers compering to say MLP and tradeoff with potential gain in performance ?

Response.

We use a one-layer, one-head Transformer with 0.87M parameters. For context, HRM uses 27M parameters (Wang et al., 2025a). Training for 1,000 epochs took approximately 4.8 hours on a single NVIDIA A40, including periodic evaluation and checkpointing but excluding teacher-data generation.

Attention provides an inductive bias for comparing cells and routing the selected cell’s information. Our analysis shows that it favors cells with fewer candidates, suppresses filled cells, and uses position to resolve ties. It then routes the selected cell’s position and candidates to the MLP for action computation. A generic MLP would need to learn these computations without this explicit attention structure, which may require more parameters to achieve comparable performance. This structure also makes selection, routing, and action computation directly accessible to causal analysis, motivating our choice of a compact Transformer.

We hope the additional results and manuscript revisions address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.