- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper demonstrates that a small attention-only Transformer can implement a soft version of Q-learning in its forward pass. The Transformer is trained on random MDPs with supervision only on the output, and the predicted parameter structure emerges after training. The authors also give a convergence analysis of the training process and show how the temperature affects the training difficulty. Experiments on random tabular MDPs support these claims.
Critical strengths
-
The paper extends the mechanistic view of in-context RL from policy evaluation and on-policy learning to off-policy Q-learning. The construction is clean, and the experiments confirm that the expected structure emerges with output-only supervision.
-
The paper provides a theoretical analysis of the training dynamics, which explains why the Q-learning structure can emerge and how the temperature affects the convergence speed.
Critical weaknesses
-
The experiments are only on small random MDPs.
-
The architecture is hand-designed to make Q-learning possible, it is unclear whether the result holds for a standard Transformer. The theory is only local
Questions for authors
-
Does the Q-learning structure still emerge in a standard Transformer without the hand-design?
-
The convergence result is local. Is there any evidence that training also converges from a random initialization?
3 comments
Thank you for the additional experiments. I would recommend being more explicit about the limitations w.r.t. global convergence and the standard Transformer. I will keep my positive score.
Response to Reviewer zYkh
Thank you for your positive evaluation and helpful feedback. We will make the scope of our theoretical results more explicit in the revised manuscript, particularly with respect to local convergence and applicability to standard Transformers.
Response to Reviewer zYkh
Thank you for your careful review and constructive comments. Please find our response below.
(1) Beyond small random MDPs: new CartPole experiments (Weaknesses 1)
In the revised manuscript, we have added CartPole experiments in Section 5.2 and Appendix C.3. The learned parameters recover the theoretically predicted structure, and the Transformer achieves performance close to Q-learning on an unseen balancing task with its network weights frozen. These results provide further empirical support for our theoretical analysis.
(2) Standard Transformers (Weaknesses 2 & Q1).
Our goal is to identify an explicit Q-learning mechanism and explain how its parameter relations emerge through training. Standard Transformers incorporate multiple layers, multiple attention heads, and MLP sublayers, providing greater representational capacity. These additional components, however, make an explicit analytical characterization of parameter realizations substantially more complicated, and training need not recover the same parameter mechanism. We therefore study a simplified Transformer to make the mechanism and its training dynamics analytically tractable. Using simplified Transformer architectures is common in theoretical studies of in-context learning, including von Oswald et al. [1], Ahn et al. [2], Zhang et al. [3], and Wang et al. [4]. Our contribution extends this mechanistic program to off-policy control and its parameter-learning geometry.
(3) Random initialization and local convergence (Q2)
Our experiments provide empirical evidence that training can converge to the theoretically predicted parameter structures from random initialization. All sixteen valid parameter blocks are initialized using Xavier-normal initialization.
While convergence to global optima is an important theoretical goal, establishing it is a well-recognized challenge in nonconvex optimization [5], and many mathematical studies establish local convergence guarantees [6, 7]. In Transformer training, global analysis is further complicated by the need to control coupled matrix products and softmax nonlinearities beyond a neighborhood of the solution manifold. Nevertheless, local convergence analysis is meaningful: it establishes stability near the solution set and quantifies how quickly optimization returns to it after sufficiently small parameter perturbations. Establishing global convergence remains a challenging direction for future work.
Reference
[1] von Oswald et al. Transformers Learn In-Context by Gradient Descent. ICML, 2023.
[2] Ahn et al. Transformers Learn to Implement Preconditioned Gradient Descent for In-Context Learning. NeurIPS, 2023.
[3] Zhang, Frei, and Bartlett. Trained Transformers Learn Linear Models In-Context. JMLR, 2024.
[4] Wang et al. Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning. ICLR, 2025.
[5] Carmon et al. Lower Bounds for Finding Stationary Points I. Mathematical Programming, 2020.
[6] Uschmajew. Local Convergence of the Alternating Least Squares Algorithm for Canonical Tensor Approximation. SIAM Journal on Matrix Analysis and Applications, 2012.
[7] Rebjock and Boumal. Fast Convergence to Non-Isolated Minima: Four Equivalent Conditions for C2 Functions. Mathematical Programming, 2025.