- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how Transformers do stack updates when giving a sequence of push/pop operations. This paper starts from providing a positive result that Transformers are expressive enough to perform this task. Next, on small Transformers, this paper shows several results on small Transformers: (1) only train on specific parts while keeping other parts same as construction; (2) intervene a real model by filtering out the dimensions that did not used by oracle construction; (3) train from scratch. Experiments on Qwen3 shows how increasing context length will affect the performance on this task.
Critical strengths
None.
Critical weaknesses
-
As a submission selecting learning theory as its primary area, this paper is insufficient in theoretical analysis. The results regarding difficulty of acquisition, reliable updates, etc. are all empirical without any formal theorems and proofs. The only theoretical results demonstrated in the submission is about expressivity. However, this proposition also requires more formal descriptions and proofs; and the full expressiveness result is insufficient as well.
-
The problem setting requires further justification. In the setup, the current stack is provided after each operation, which substantially reduces the state maintenance requirement.
-
Sections and experimental results are weakly connected. Although all experiments shares the same task on working on stacks, the experimental setup are significantly different from subsections to subsections. This make it difficult to determine whether the findings are general or not.
Questions for authors
Please refer to the weaknesses part.
-
Could the authors provide further details regarding the expressiveness result? Specifically, could the authors provide detailed constructions (in appendix) and state the theorems more clearly (for instance, the requirement of model depth and width w.r.t. input size)? The current construction in Appendix A.2 (read specific positions, use specific attention scores) is insufficient for formal theoretical results.
-
Could the authors explain why they give current stack status in the context after each operation?