acceptodds
Under review as a conference paper at ICLR 2027

From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning

Abstract

Transformers can implement reinforcement-learning algorithms in context, yet how off-policy control emerges within Transformers remains poorly understood. We study Q-learning, a foundational model-free control algorithm that combines off-policy bootstrapping with greedy policy improvement. We prove that a two-layer attention-only Transformer can implement a finite-temperature form of Q-learning, where the temperature controls the approximation to greedy action selection, recovering window Q-learning as . We train the structured Transformer on randomly generated MDP tasks using output-only supervision. Experiments show that the theoretically predicted parameter structure emerges during training, including the internal greedy action-selection mechanism, and that the trained Transformer achieves performance comparable to Q-learning on unseen tasks. Our theoretical analysis further shows that the equivalent realizations form a solution manifold and that, under suitable task-diversity conditions, the training iterates converge locally to this manifold at an exponential rate. For the random-task model analyzed here, the local worst-case number of gradient steps needed to reduce the distance to the manifold by a factor is , quantifying the increasing optimization cost as the learned update approaches window Q-learning.

Primary area:
learning theory
Keywords:
In-Context Q-Learning, Transformers, Algorithmic Learning, Parameter Emergence, Optimization Geometry.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zYkhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper demonstrates that a small attention-only Transformer can implement a soft version of Q-learning in its forward pass. The Transformer is trained on random MDPs with supervision only on the output, and the predicted parameter structure emerges after training. The authors also give a convergence analysis of the training process and show how the temperature affects the training difficulty. Experiments on random tabular MDPs support these claims.

Critical strengths
  1. The paper extends the mechanistic view of in-context RL from policy evaluation and on-policy learning to off-policy Q-learning. The construction is clean, and the experiments confirm that the expected structure emerges with output-only supervision.

  2. The paper provides a theoretical analysis of the training dynamics, which explains why the Q-learning structure can emerge and how the temperature affects the convergence speed.

Critical weaknesses
  1. The experiments are only on small random MDPs.

  2. The architecture is hand-designed to make Q-learning possible, it is unclear whether the result holds for a standard Transformer. The theory is only local

Questions for authors
  1. Does the Q-learning structure still emerge in a standard Transformer without the hand-design?

  2. The convergence result is local. Is there any evidence that training also converges from a random initialization?

3 comments
AuthorsOpenReview ↗

Response to Reviewer zYkh

Thank you for your careful review and constructive comments. Please find our response below.

(1) Beyond small random MDPs: new CartPole experiments (Weaknesses 1)

In the revised manuscript, we have added CartPole experiments in Section 5.2 and Appendix C.3. The learned parameters recover the theoretically predicted structure, and the Transformer achieves performance close to Q-learning on an unseen balancing task with its network weights frozen. These results provide further empirical support for our theoretical analysis.

(2) Standard Transformers (Weaknesses 2 & Q1).

Our goal is to identify an explicit Q-learning mechanism and explain how its parameter relations emerge through training. Standard Transformers incorporate multiple layers, multiple attention heads, and MLP sublayers, providing greater representational capacity. These additional components, however, make an explicit analytical characterization of parameter realizations substantially more complicated, and training need not recover the same parameter mechanism. We therefore study a simplified Transformer to make the mechanism and its training dynamics analytically tractable. Using simplified Transformer architectures is common in theoretical studies of in-context learning, including von Oswald et al. [1], Ahn et al. [2], Zhang et al. [3], and Wang et al. [4]. Our contribution extends this mechanistic program to off-policy control and its parameter-learning geometry.

(3) Random initialization and local convergence (Q2)

Our experiments provide empirical evidence that training can converge to the theoretically predicted parameter structures from random initialization. All sixteen valid parameter blocks are initialized using Xavier-normal initialization.

While convergence to global optima is an important theoretical goal, establishing it is a well-recognized challenge in nonconvex optimization [5], and many mathematical studies establish local convergence guarantees [6, 7]. In Transformer training, global analysis is further complicated by the need to control coupled matrix products and softmax nonlinearities beyond a neighborhood of the solution manifold. Nevertheless, local convergence analysis is meaningful: it establishes stability near the solution set and quantifies how quickly optimization returns to it after sufficiently small parameter perturbations. Establishing global convergence remains a challenging direction for future work.

Reference

[1] von Oswald et al. Transformers Learn In-Context by Gradient Descent. ICML, 2023.

[2] Ahn et al. Transformers Learn to Implement Preconditioned Gradient Descent for In-Context Learning. NeurIPS, 2023.

[3] Zhang, Frei, and Bartlett. Trained Transformers Learn Linear Models In-Context. JMLR, 2024.

[4] Wang et al. Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning. ICLR, 2025.

[5] Carmon et al. Lower Bounds for Finding Stationary Points I. Mathematical Programming, 2020.

[6] Uschmajew. Local Convergence of the Alternating Least Squares Algorithm for Canonical Tensor Approximation. SIAM Journal on Matrix Analysis and Applications, 2012.

[7] Rebjock and Boumal. Fast Convergence to Non-Isolated Minima: Four Equivalent Conditions for C2C^2 Functions. Mathematical Programming, 2025.

Thank you for the additional experiments. I would recommend being more explicit about the limitations w.r.t. global convergence and the standard Transformer. I will keep my positive score.

AuthorsOpenReview ↗

Response to Reviewer zYkh

Thank you for your positive evaluation and helpful feedback. We will make the scope of our theoretical results more explicit in the revised manuscript, particularly with respect to local convergence and applicability to standard Transformers.

Reviewer_oAnXOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper studies the implementation and validity of in-context off-policy Q-learning with transformer-based forward pass. The authors provide the input prompt matrix, construct a two-layer transformer, where the first layer is a softmax attention and the second layer is a linear attention. The authors provide theorems to show that the constructed network can perform soft Q-learning with forward pass and show the convergence of pretraining. Experimental analysis are based on a 9-state, 4-action synthetic MDP to show the convergence and error of the purposed model.

Critical strengths

The paper provides a new construction of in-context Q-learning architecture. Compared to the linear-attention based construction in the literature, this paper attempts adding a softmax attention layer for explict weighted representation learning, followed by the linear attention layer that carry out TD-learning. This design can enhance the expressive power of the network.

Critical weaknesses
  1. The position of the paper should be reconsidered: in-context learning for off-policy control has been boardly explored by the literature (DT, QT, QDT, ICQL, S-ICQL, etc.) Some of the related work are actually discussed in the related work, but the discussion and comparison between the purposed method and those related work are not comprehensive (neither theoretically nor empirically). Although the paper provides a lot of mathematical analysis on their method, the experiment setup of this paper itself cannot support an "off-policy control" claim. The experiment is conducted only on one synthetic MDP setting, typical off-policy control scenarios (e.g. MuJoCo) are never evaluated.
  2. The validity of soft Q-learning update through Transformer forward pass is not equivalent to RL iterations that leads to optimized action-value function. The purposed two-layer architecture misses the iterative weight update procedure inside each forward pass, instead absorbing this iteration into pretraining, which may greatly lose expressive power in complex environments.
Questions for authors
  1. Given the extensive literature on Transformer-based off-policy and offline RL, how does this work distinguish itself theoretically and how is the performance competed empirically?

  2. How does implementing a single soft Q-learning update in a two-layer Transformer establish the ability to perform iterative in-context Q-network update and converge to an optimal action-value function?

6 comments
AuthorsOpenReview ↗

Response to Reviewer oAnX (Part 1/3): Weakness 1& Q1 - Theoretical and empirical comparisons with relevant literature

Thank you for your careful review and constructive comments. We address your concerns below.

1. Theoretical and empirical comparisons with relevant literature

We would first like to clarify that the cited works and ours differ fundamentally in their research objective. These works (DT, QDT, QT, ICQL, and S-ICQL) primarily focus on developing more effective Transformer-based control methods and empirically evaluating their performance. However, improved control performance alone does not explain the internal computational mechanisms through which Transformers achieve it. Our work investigates what control algorithms Transformers can implement and learn through training. Specifically, we establish that Transformers can realize Q-learning: we identify the corresponding parameter structures of the transformer and prove convergence of the training dynamics toward the realization set. These results extend the mechanistic analysis of ICRL from policy evaluation (TD algorithm, Wang et al. [1]) to off-policy control, providing a theoretical foundation for understanding how Transformers “learn to learn”.

We have added a comparison with related work in Section 2 of the revised manuscript. Detailed comparisons are provided below.

(1) Comparison with DT, QDT, QT.

DT [2], QDT [3], and QT [4] all use Transformers to predict actions via (ht,gt)→TFθa^t(h_t,\mathbf{g}_t)\xrightarrow{\mathrm{TF}_\theta}\hat a_t, where hth_t is the state–action history including the current state sts_t, gt\mathbf g_t denotes the return-to-go conditions, a^t\hat a_t is the predicted action, and TFθ\mathrm{TF}_\theta is the Transformer operation with parameters θ\theta. They use Q-learning to obtain value estimates for return relabeling or critic-guided policy optimization, helping Transformers learn more effective control policies.

Our Transformer instead implements the Q-learning update rule itself: (wk,Dk)→TFθwk+1(\mathbf{w}_k,\mathcal D_k)\xrightarrow{\mathrm{TF}_\theta}\mathbf{w}_{k+1}, where wk\mathbf{w}_k is the Q-function parameter vector at iteration kk, Dk=(si,ai,ri,si′)i=1n\mathcal D_k=\\{(s_i,a_i,r_i,s_i')\\}_{i=1}^{n} is collected under a behavior policy. At deployment, the Transformer weight θ\theta remains fixed, while each predicted wk+1\mathbf{w}_{k+1} is fed back with the next transition window. Our theoretical analysis therefore characterizes the transformer parameter structures that implement Q-learning updates and explains how training converges toward them.

(2) Comparison with ICQL and S-ICQL.

ICQL [5] and S-ICQL [6] use context-conditioned Q-value estimation to learn effective policies. Both methods use Transformers to estimate Q-values via (C,s,a)→TFθQ^(s,a∣C)(\mathcal{C}, s, a) \xrightarrow{\mathrm{TF}_\theta} \widehat{Q}(s, a \mid \mathcal{C}), where (s,a)(s, a) is the query state-action pair and C\mathcal{C} represents transition information obtained through retrieval or history encoding. S-ICQL trains context-conditioned value functions and policies using Bellman, expectile, and advantage-weighted objectives. These objectives specify how the transformer is trained, but do not establish what update rule its forward computation implements. ICQL is more related to our work: it provides a linear-attention construction for in-context TD updates in local value estimation. Its construction adapts the TD mechanism of Wang et al. [1] from state features to state-action features. However, this adaptation still bootstraps from Q(s′,a′)Q(s',a'), where the successor action a′a' is supplied in the context rather than max⁡a∈AQ(s′,a)\max_{a \in \mathcal{A}} Q(s',a) computed internally from the current Q-function. Thus, this construction does not resolve the essential additional comparison and maximization operation required for Q-learning. Furthermore, ICQL does not analyze how training gives rise to the parameter structures underlying its TD realization, leaving a central question in mechanistic research unanswered.

Our work addresses a qualitatively different mechanistic question compared with Wang et al. [1]: We construct Transformer parameter structures under which attention itself compares Q-values across successor actions, without external assistance, and incorporates the resulting bootstrap target into the value update. More importantly, we characterize the parameter realization manifold and prove convergence of the training dynamics to this manifold. Compared with the linear-attention setting, this requires overcoming substantial technical challenges posed by the non-polynomial objective induced by softmax attention.

(Continued in Part 2/3 below...)

AuthorsOpenReview ↗

Response to Reviewer oAnX (Part 2/3): Weakness 1& Q1 - Theoretical and empirical comparisons with relevant literature

(Continued from Part 1/3)

(3) Off-policy setting and experimental comparison.

Our setting separates the behavior policy used to collect transitions from the target policy implicit in the Q-learning backup. This separation is preserved in our experiments, which therefore retain the essential off-policy character of the learning problem. To broaden our empirical evaluation beyond synthetic MDPs, we have added CartPole experiments to the revised manuscript in Section 5.2 and Appendix C.3: the learned parameters exhibit the predicted structure, and the frozen Transformer achieves returns close to the corresponding soft Q-learning baselines on a new balancing task.

DT, QDT, QT, ICQL, and S-ICQL primarily focus on developing more effective Transformer-based control methods. Our motivation is different: we study whether Transformers can execute Q-learning updates internally and how this mechanism emerges through training. Control-performance comparisons with these methods cannot establish whether the identified mechanism emerges. We should use soft Q-learning as the reference baseline to assess whether the Transformer executes this algorithm internally. Standard continuous-action MuJoCo benchmarks would additionally require extending our finite-action setting, beyond the mechanism analyzed in this work.

AuthorsOpenReview ↗

Response to Reviewer oAnX (Part 3/3): Weakness 2& Q2 - Algorithm iteration and convergence analysis

(Continued from Part 2/3)

2. Algorithm iteration and convergence analysis

Soft Q-learning iterations are not absorbed into pretraining: Pretraining learns the update rule by optimizing the Transformer parameter θ\theta, while deployment repeatedly applies the frozen Transformer TFθ\mathrm{TF}_\theta to update Q-function parameters wk\mathbf{w}_k using new transitions. In the revised manuscript, Figure 1 clarifies the deployment procedure, and Corollary 2 establishes its convergence guarantee.

(1) Deployment procedure of soft Q-learning iterations.

As illustrated in Figure 1, starting from w0\mathbf{w}_0, the agent samples actions under its behavior policy and interacts with the environment to collect a window Dk=(si,ai,ri,si′)i=1n\mathcal D_k=\\{(s_i,a_i,r_i,s_i')\\}_{i=1}^{n} of nn state–action–reward–next-state transitions. It feeds this window together with wk\mathbf{w}_k into the frozen Transformer to obtain wk+1\mathbf{w}_{k+1}. It is then fed back together with the next collected window Dk+1\mathcal D_{k+1} to obtain wk+2\mathbf{w}_{k+2}. Repeating this loop performs successive soft Q-learning updates using the same frozen Transformer. For the exact parameter construction in Theorem 1, this recursion reproduces soft window-Q learning from the same initialization and transition data.

Thus, fixed Transformer weights do not imply a fixed Q-function or policy, and the two-layer architecture imposes no limit on the number of deployment iterations.

(2) Convergence guarantee and optimal policy in Corollary 2.

For an exact frozen Transformer in a finite tabular discounted MDP, the Q-function QkQ_k iterates converge almost surely to QτQ^\tau under the stated assumptions, where QτQ^\tau is the unique fixed point of the Boltzmann Bellman operator. The greedy policy induced by QkQ_k is almost surely optimal for all sufficiently large kk, while the behavior policy may continue exploring. Moreover, the limit QτQ^\tau approaches Q⋆Q^\star, the optimal Q-function targeted by standard Q-learning, exponentially as τ→0\tau\to0.

References

[1] J. Wang, E. Blaser, H. Daneshmand, and S. Zhang. Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning. ICLR, 2025.

[2] L. Chen et al. Decision Transformer: Reinforcement Learning via Sequence Modeling. NeurIPS, 2021.

[3] T. Yamagata, A. Khalil, and R. Santos-Rodriguez. Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL. ICML, 2023.

[4] S. Hu et al. Q-value Regularized Transformer for Offline Reinforcement Learning. ICML, 2024.

[5] Q. Xu et al. In-Context Compositional Q-Learning for Offline Reinforcement Learning. ICLR, 2026. arXiv:2509.24067v2.

[6] J. Liu et al. Scalable In-Context Q-Learning. ICLR, 2026. arXiv:2506.01299v3.

Reviewer_hKstOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates whether transformers can be trained to predict the soft Q-learning updates. The authors show that a two-layer Transformer can exactly implement soft window Q-learning (with a temperature parameter τ\tau). This is then verified by experiments which show that theoretically predicted parameter structure emerges during training. In addition, the authors also characterize the set of weight solutions (that compute the update exactly) and show that the local convergence of gradient descent has an iteration complexity that scales with τ−5\tau^{-5} (convergence is slower as the soft Q-learning update approaches the max Q update).

Critical strengths
  • The paper studies an algorithmic interpretation of in-context reinforcement learning (ICRL), where a Transformer is able to carry out an reinforcement learning algorithm within its forward pass. Existing work (e.g., Liang & Lai, 2026) focuses mostly on the on-policy case, where the data that the agent learns from are from the current policy. The paper makes non-trivial extension to the off-policy case, showing that a Transformer (2-layer, attention only) can perform one step of the Q-learning update (more specifically soft Q-learning update with temperature →0\rightarrow 0 recovering argmax update).
  • Such extension seems non-trivial as the original analysis in the on-policy case (polynomial in loss) cannot be directly translated to the soft Q-learning case (non-polynomial in loss since it involves a softmax selection of actions).
  • Empirical validation: Figure 2 left shows that empirically the transformer can be trained to closely track the ground truth soft Q learning update.
Critical weaknesses

None

Questions for authors

N/A

1 comment

Thank you for your careful reading and positive assessment. We particularly appreciate your recognition of the nontrivial extension to off-policy Q-learning and the analytical challenges of the softmax-induced non-polynomial objective.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.