- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks the question whether a transformer model converges towards the exact same algorithm as a hand-coded ideal network, using the SUBLEQ language as training data. Two ~3.2M param models are used, one of them learned while the other's weights are chosen by hand. The learned network does not converge towards the hardcoded one. The authors align the two weight vectors and linearly interpolate between them, finding that no connectivity in weight space exists between the two. Attention head probes show that the attention mechanism resembles pointer dereferencing operations, and experiments with weight decay do not result in meaningful changes to the model's behavior.
Critical strengths
- The question of whether gradient descent can find optimally constructed hardcoded solutions is somewhat relevant
Critical weaknesses
-
The paper essentially runs one experiment, training 3M param transformers on a single instruction language, and comparing them to a single hardcoded solution they claim is the theoretical optimum the model should reach. First of all, it is hard to understand why one single hardcoded solution should be any kind of optimum. A slightly more scientific approach would have been to find an exhaustive set of all possible optimal solutions that can be hardcoded using a general rule system for choosing the weights by hand. Otherwise, the non-connectivity between the claimed optimum and the learned solution could just as well be a random disconnect between these two particular solutions. Secondly, running a single set of experiments on a tiny model with a single source of data does not give enough evidence for any of the claims. The generality of the claims should be proven by at least multiple different instruction languages and on multiple larger model sizes, as well as with insightful ablations.
-
There is ample research on this exact question. For example Xu et al. (https://arxiv.org/abs/2604.25166) train transformers on MicroPy outputs, Liu et al. let models execute complex python programs (https://aclanthology.org/2023.findings-acl.308/), as well as work on cellular automata (https://arxiv.org/abs/2412.01417), and the work is even closely connected to the literature of neural turing machines, just to name a few:
- https://arxiv.org/abs/1410.5401
- https://ideas.repec.org/a/nat/nature/v538y2016i7626d10.1038_nature20101.html
- https://arxiv.org/abs/1511.06279 In the current form, I don't see any additional insight from the current work into the learning capacity of transformer models.
Questions for authors
- How do you train your model on SUBLEQ? I understand that you use a BERT-style transformer, but then what are examples for inputs, and what are the targets?
- Why did you use an encoder-only transformer instead of a decoder-only one?
- Why did you only use a single hardcoded solution instead of proving the set of optimal solutions that can be hardcoded?
- Why did you not test your claims on other instruction languages as well as other model types or sizes?