- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a new architecture, STRAT, which is based on the idea of separating data and types in their embedding subspaces orthogonally as an architecturally imposed inductive bias. The authors motivate the work by investigating failure modes with standard transformers and identify three failure modes: the Linear Trap, Gradient Wall, and Open Gate Trap. The authors show benefits of STRAT compared to the transformer on different algorithmic reasoning tasks, with particularly strong improvements for out-of-distribution evaluation .
Critical strengths
-
Thoughtful motivation and experimental design: The authors introduce the paper and the motivation for the work from a very clear angle: how architectural inductive biases can influence the capability of a model to learn logical reasoning. The author's design for STRAT is carefully thought out: first implementing an arithmetic algorithm with a standard transformer in Section 3 to show this is possible and then investigate why this algorithm could be difficult to learn in gradient descent in Sectino 4. This motivates the whole work and the need for the STRAT architecture.
-
Empirical Results: Table 6 shows clear improvements over the standard architecture on a wide suite of algorithmic tasks, showing that STRAT performs better in-distribution and out-of-distribution. The out-of-distribution performance is particularly strong, as it shows that the architectural inductive bias is providing generalisation beyond what the evaluated standard transformer achieves through gradient descent.
Critical weaknesses
- Predefined Types: Section 7's experiments use predefined task-specific typing codings. Even though Section 5.4 motivates and demonstrates that strat can learn meaningful representations for types. Evaluating learned types in Section 7's experiments would strengthen the claims about the generalisation capabilities of STRAT. Also, it is unclear if the geometric structures observed in the paper are consistent across seeds. Do they always appear or are they specific to the seeds that the authors used?
- Support for claims of limitations of standard transformers: The paper is motivated by inherented limitations of standard transformers, but the empirical evidence is only based on six small architectures and a small training budget. Are these failures truly general to the transformer family (i.e., would it persist with different transformer architectures or different-sized models, or alternative training settings like changing the optimizer)?
Questions for authors
- Is it possible for the authors to evaluate learned type representations against the predefined ones on the tasks in Section 7?
- In regard to the limitation of standard transformers, could the authors discuss if the failures in Section 4 persist across different settings? What evidence then supports this claim that it is inherent to the standard transformer architecture and not just the six models evaluated?
6 comments
Response to Reviewer Ceqx (2/3)
W1 - Predefined Types: Section 7's experiments use predefined task-specific typing codings. Even though Section 5.4 motivates and demonstrates that strat can learn meaningful representations for types. Evaluating learned types in Section 7's experiments would strengthen the claims about the generalisation capabilities of STRAT.
Also, it is unclear if the geometric structures observed in the paper are consistent across seeds. Do they always appear or are they specific to the seeds that the authors used?
Q1 - Is it possible for the authors to evaluate learned type representations against the predefined ones on the tasks in Section 7?
Thank you for these helpful suggestions and for giving us an opportunity to further strengthen the claims about the generalisation capabilities of STRAT. Following your suggestions, we evaluated learned Type representations against the predefined ones on all 11 datasets of Section 7, and examined whether the learned geometric structures are consistent across seeds.
For experiments config: all data splits, ID and OOD test sets, model architectures, training settings, hyperparameters, and seeds 41–50 are the same as in Section 7. The predefined Type is replaced by a learnable embedding layer with one vector per token kind. The embedding layer is initialized from N(0, 0.02), the same as in the arithmetic experiments. We set the embedding dimension to 5 for all datasets for consistency, except for Sudoku, where we keep the dimension of its original predefined Type (37) because Sudoku is the most complex task and has the largest vocabulary (each cell as a token).
Table a: Accuracy (%) of STRAT with the predefined Type and with the learned Type on the 11 datasets of Section 7 (mean ± SEM over 10 seeds).
| Dataset | Predefined Type (ID) | Predefined Type (OOD) | Learned Type (ID) | Learned Type (OOD) |
|---|---|---|---|---|
| SATNet Sudoku | 86.58 ± 1.08 | 83.80 ± 1.30 | 89.87 ± 1.08 | 86.70 ± 1.31 |
| SATBench | 92.33 ± 1.36 | 90.50 ± 2.15 | 92.11 ± 1.45 | 90.66 ± 1.57 |
| SATLIB | 99.98 ± 0.02 | 99.96 ± 0.04 | 100.00 ± 0.00 | 100.00 ± 0.00 |
| Tic-Tac-Toe Endgame | 98.19 ± 0.00 | 94.64 ± 0.00 | 97.96 ± 0.23 | 94.46 ± 0.18 |
| Game of Life | 88.71 ± 2.72 | 79.26 ± 12.99 | 87.57 ± 3.08 | 77.16 ± 12.67 |
| PMLB Parity5 | 96.94 ± 0.06 | 97.12 ± 0.25 | 97.25 ± 0.31 | 97.44 ± 0.38 |
| PMLB M-of-N | 100.00 ± 0.00 | 100.00 ± 0.00 | 100.00 ± 0.00 | 100.00 ± 0.00 |
| PMLB MONK-2 | 90.86 ± 3.04 | 90.92 ± 3.07 | 98.28 ± 1.16 | 96.13 ± 2.29 |
| PMLB LED24 | 100.00 ± 0.00 | 100.00 ± 0.00 | 95.10 ± 4.90 | 95.36 ± 4.64 |
| PMLB ThreeOf9 | 80.18 ± 0.57 | 73.12 ± 0.88 | 100.00 ± 0.00 | 100.00 ± 0.00 |
| SPECT Heart | 94.91 ± 1.05 | 93.05 ± 0.15 | 100.00 ± 0.00 | 100.00 ± 0.00 |
| Average | 93.51 | 91.13 | 96.19 | 94.36 |
As shown in Table a, STRAT with the learned Type performs on par with STRAT with the predefined Type: the average accuracy is 96.19 / 94.36 (ID / OOD), compared with 93.51 / 91.13, and no dataset decreases by more than two SEM. This shows that the Type embeddings do not need to be predefined: they can be learned directly from the task.
To further examine whether the learned geometric structures are consistent across seeds, we analyzed the learned Type embeddings on Sudoku with 10 seeds, where the type of a cell is its row, column, and box, and the model is never told any of them. Figure 4 in the revised paper (Appendix G.1) shows the cosine similarity between the learned Type vector of each cell and that of a reference cell, averaged over the 10 seeds. Every board shows a red cross centered on the reference cell: the cells in its row, column, and box have similar Type vectors (red), while all other cells do not (blue). Specifically, the mean cosine similarity to the reference cell is +0.30 for cells in its row, +0.31 in its column, and +0.19 in its box, compared with −0.11 for all other cells (all about 0 before training). Importantly, the same pattern appears in every one of the 10 seeds, as shown in Figure 5 in the revised paper (Appendix G.1). Thus, this structure is not specific to particular seeds: in every seed, the learned Type learns by itself the row, column, and box structure encoded in the predefined Type.
Thank you again for these helpful suggestions. We have included these results and the analysis in the revised version (Section 7 and Appendix G).
Response to Reviewer Ceqx (3/3)
W2 - Support for claims of limitations of standard transformers: The paper is motivated by inherented limitations of standard transformers, but the empirical evidence is only based on six small architectures and a small training budget. Are these failures truly general to the transformer family (i.e., would it persist with different transformer architectures or different-sized models, or alternative training settings like changing the optimizer)?
Q2 - In regard to the limitation of standard transformers, could the authors discuss if the failures in Section 4 persist across different settings? What evidence then supports this claim that it is inherent to the standard transformer architecture and not just the six models evaluated?
Thank you for raising this important concern and giving us an opportunity to further clarify the scale of our experiments and to strengthen the evidence with additional experiments.
In Section 4, we trained 12 configurations of the Transformer block, each with 50 seeds (600 runs in total; Section 4.1). These configurations change the input scaling, normalization, curriculum, regularization, and initialization, and none of the 600 runs succeeds. In addition, the Transformer baselines in Section 7 have 1 to 16 layers and are trained with Adam or AdamW (Appendix D). Across the 11 datasets, their accuracy drops by 11.75 points under distribution shift, compared with 2.39 for STRAT (Table 6).
To further support that the failure comes from the architecture, we conducted two additional experiments. First, we measured how each trained model fails. In 10 of the 12 configurations the failure has the same form: the models use the operators correctly at training magnitudes but ignore them at OOD magnitudes. Second, we conducted a full factorial ablation: with the same data, optimizer, and training budget, we change only the connections of the standard block. When the gate and attention read only the Type, 167 of 200 runs succeed (84%), compared with 40 of 1,400 (2.9%) for all other combinations. Thus, the failure follows the connections that let operand values reach the control computation, rather than a particular model or seed. We have included these experiments in the revised paper (see Section 4.3, Table 2, Section 5.4, and Appendix B.5).
Thank you again for this valuable comment.
Response to Reviewer Ceqx (1/3)
Dear Reviewer Ceqx:
Thank you very much for appreciating the thoughtful motivation and experimental design of our work. We are particularly encouraged by your recognition of STRAT's clear improvements over the standard architecture, especially its strong out-of-distribution performance. Below, we address your specific comments and questions point by point (the corresponding revisions have been included in the uploaded revised version).