acceptodds
Under review as a conference paper at ICLR 2027

Provable Transformer Context Compression for In-Context Reinforcement Learning

Abstract

Transformers can perform in-context reinforcement learning without parameter updates. However, existing theoretical constructions often require context sizes that grow with the number of sampled transitions, leading to increasing memory costs in large-scale problems. This raises a fundamental scalability question: how much context is intrinsically necessary to carry out such learning? We answer this question for fixed-policy TD evaluation. We show that, with a shared linear decoder, the minimum context dimension required to exactly reproduce the first TD iterates is the Krylov dimension induced by the sampled transitions. When the available context dimension falls below this threshold, exact reproduction is no longer possible; in this regime, we characterize the optimal approximation error and show that it decays as where is the discounting factor. Moving from representation complexity to actual memory cost, we characterize the optimal retained memory for exact continuous representations as here denotes Krylov dimension induced by the sampled transitions and show experimentally that on a sampled task whose exact rank is full but whose numerical rank is small, a context accurate to is smaller than the sparse sufficient statistics of TD. Finally, we establish the learnability of the shared executor through pretraining by proving global convergence of an auxiliary objective over effective attention parameters under suitable task diversity. Experiments across diverse TD evaluation problems validate parameter recovery and the predicted context–accuracy tradeoff.

Primary area:
learning theory
Keywords:
In-Context Reinforcement Learning, Context Compression, Temporal-Difference Learning, Transformers, Reinforcement Learning Theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aPFWOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper focus on fixed-policy TD evaluation to study the context sizes in in-context reinforcement learning. They found that the context size should be at least the dimension of Krylov space spanned by the transition kernal. Also, they characterize the error under the limited context size. Their experiments supports their theoretical prediction on the limited context window error. At last, the conduct experiments to illustrate the storage efficieny of their method.

Critical strengths

The experiments to support their theorems are clear.

Critical weaknesses

The paper is not well presented. The contribution of this paper is not clear to me.

Questions for authors
  1. The paper presentation is not clear to me: what contribution is made by this paper uniquely? And what techniques are used in this paper to distinguish from previous works? Section 4 provides a series of results, but it is a general storage-efficient method for me. I believe that other storage-efficient mthods have been adopted in practice, while there is no fair compaison. Therefore, I can only get the conclusion that: "this paper analyzes a storage-efficient method", rather than "a supirior method is proposed in this paper".

  2. The paper compared storage to other works in Table 1. However, those works do not focus on storage efficiency. Is this comparison fair?

3 comments

We thank reviewer aPFW for the valuable comments and suggestions. We have revised the paper to better highlight our theoretical contributions. Please find our response below.

- Main contribution: Characterizing context complexity

Our primary motivation is to understand how much Transformer context is needed for in-context reinforcement learning. To this end, we theoretically characterize the context complexity of in-context RL through a natural algebraic structure, the Krylov subspace. Specifically, we establish that the Krylov dimension exactly characterizes the minimum context dimension required for a Transformer to reproduce TD learning. To the best of our knowledge, no prior work has provided such a theoretical characterization of the context complexity required for in-context RL. We have made this contribution more explicit in the abstract, introduction, and corresponding theoretical sections.

- Unique technical novelty

Our technical novelty consists of two main parts: (1) an exact characterization of the minimum context dimension required for TD learning (2) an approximation analysis that quantifies the error when the available context dimension is smaller than the minimum required for exact TD learning.

  • Unlike prior Transformer constructions that implement TD learning while retaining the transition history [1,2,3,4], we compresses raw transitions into a Krylov representation, and prove its optimality. This characterizes the context complexity of in-context reinforcement learning. We realize the compression and the TD-learning updating within the forward pass of a fixed-parameter Transformer. Moreover, we bridge theoretical context complexity and practical memory costs by establishing tight upper and lower bounds on retained task memory, achieving an optimal complexity of Θ(nkL)\Theta(nk_L) for exact representations.

  • We further analyze the setting when the context dimension below the dimension for exact TD learning. We establish matching upper and lower bounds for the optimal dimension-mm approximation error, i.e.,

    \Theta\left(\gamma^m \max\left\\{1,\frac{1}{m(1-\gamma)}\right\\}\right),

    Compared with classical Arnoldi [4,5] and Krylov projection methods [6], our approach achieves a strictly tighter upper bound. Moreover, we prove that these traditional methods are provably suboptimal and cannot attain the minimax-optimal rate.

Additionally, we believe that our approximation framework has the potential to further strengthen classical Krylov-based approximation methods and extend their applicability to broader trajectory-approximation problems.

- Purpose of Table 1

We would like to clarify that Table 1 is intended to compare theoretical complexity characterizations and guarantees, rather than compression performance. The reported memory values serve only to provide a more intuitive interpretation of the theoretical complexity measures. Such comparisons of theoretical guarantees are also common in compression complexity literature. For example, Tables 1 and 2 of [8] compare compressed and uncompressed methods in terms of theoretical complexity and numerical memory consumption.

For compression performance, we compare against seven compression methods and sparse TD under the same TD-learning setting (Table 3 and Appendix H.13, Table 17). At an error tolerance of 10−210^{-2}, our Krylov+POD context achieves significantly lower retained memory than existing compression methods across all eight tasks: 2.52.5–63.2×63.2\times over truncated SVD, 3.53.5–53.0×53.0\times over random projection, and 1.061.06–2.22×2.22\times over POD. It also outperforms sparse TD in retained memory on seven of the eight tasks. POD–Galerkin remains 1.021.02–1.31×1.31\times more compact, partly because our fixed attention executor additionally stores position tags and a query token.

In summary, our central contribution is a theoretical characterization of Transformer context complexity for in-context RL, together with an approximation analysis when the available context is insufficient. The compression experiments provide additional empirical validation of our theoretical insights.

Due to space limitations, the references are provided in the following comment.

- References

[1] Wang, Jiuqi, Ethan Blaser, Hadi Daneshmand, and Shangtong Zhang. "Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning." The Thirteenth International Conference on Learning Representations (ICLR). 2025.

[2] Wang, Jiuqi, Rohan Chandra, and Shangtong Zhang. "Towards Provable Emergence of In-Context Reinforcement Learning." Advances in Neural Information Processing Systems 38 (NeurIPS). 2025.

[3] Xie, Zixuan, Xinyu Liu, Rohan Chandra, and Shangtong Zhang. "Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought." arXiv preprint arXiv:2605.07123. 2026.

[4] Xie, Zixuan, Xinyu Liu, Claire Chen, Shuze Daniel Liu, Rohan Chandra, and Shangtong Zhang. "Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning." arXiv preprint arXiv:2605.07333. 2026.

[5] Arnoldi, Walter E. "The Principle of Minimized Iterations in the Solution of the Matrix Eigenvalue Problem." Quarterly of Applied Mathematics 9, no. 1: 17–29. 1951.

[6] Saad, Yousef. "Analysis of Some Krylov Subspace Approximations to the Matrix Exponential Operator." SIAM Journal on Numerical Analysis 29, no. 1: 209–228. 1992.

[7] Freund, Roland W. "Model Reduction Methods Based on Krylov Subspaces." Acta Numerica 12: 267–319. 2003.

[8] Likhosherstov, Valerii, Krzysztof M. Choromanski, Jared Quincy Davis, Xingyou Song, and Adrian Weller. "Sub-Linear Memory: How to Make Performers SLiM." Advances in Neural Information Processing Systems 34 (NeurIPS), 2021.

Dear Reviewer aPFW,

Thank you again for your valuable feedback. As the discussion period is approaching its end, we would greatly appreciate it if you could review our rebuttal and let us know if you have any remaining concerns.

Thank you for your time and consideration.

Reviewer_SxquOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies context compression for in-context TD evaluation. It characterizes exact context requirements through Krylov dimension, establishes memory and approximation bounds, and constructs attention-based encoders and executors. Experiments validate the predicted context–accuracy tradeoff and demonstrate retained-memory savings on selected tasks.

Critical strengths
  • The problem is clear and is a painpoint in ICRL. This work provides a framework for ICRL representation and memory complexity research.
  • It complements lower bounds with explicit attention-based encoders and executors, including a sample-and-hold construction attaining the optimal approximation rate.
  • The paper connects its results to ICRL, sufficient statistics, Krylov methods, model reduction, and context compression. Appendix G explicitly acknowledges the classical origins of several components and clarifies their relationship to the proposed framework.
  • The experiments examine the exact dimension threshold, context-accuracy tradeoff, parameter recovery, and memory requirements. They align with theoretical claims well.
Critical weaknesses
  • The context-dimension and approximation results assume a shared linear decoder and therefore do not establish corresponding limits for arbitrary Transformers or nonlinear compression schemes. It limits their broader practical implications.
  • The main analysis concerns fixed-policy, visitation-normalized batch TD on an empirical model. Extensions to online stochastic TD, exploration, and policy improvement remain unresolved.
  • Exact constructions rely on real arithmetic, specialized reciprocal operations, rank selection, and token release. Under sampling noise, practical benefits depend more on numerical low rank than exact low rank. Therefore, I think the computational model is too ideal.
  • The experiment suite is still limited. They establish implementation correctness but do not show that trained Transformers naturally discover the proposed compression mechanism.
  • Learned encoders are primarily tested on held-out tasks from the training distribution, with little systematic evaluation on unseen task families, state-space sizes, or horizons. The distribution-shift evaluation is insufficient.

Minor issue:

  • Although the related-work coverage is strong, some important distinctions between classical ingredients and new results are primarily explained in the appendix.
Questions for authors

Could the authors clarify the specific novel contributions of the joint approximation rate and the memory lower bound for continuous encoders relative to existing results in model reduction, minimal realization, and approximation-width theory?

5 comments

We thank reviewer Sxqu for the valuable comments and suggestions. We especially appreciate the reviewer's thoughtful consideration of our work across diverse settings and technical approaches. We address the concerns below.

- Q1:Specific contributions

We would like to clarify our technical contributions and how they differ from classical approaches.

  • Context complexity: Unlike prior Transformer constructions that implement TD learning while retaining the transition history [1,3,4,5], we compresses raw transitions into a Krylov subspace, and prove its optimality. Krylov subspace was used in TD-learning previously, and we extend it into in-context RL to describe the context complexity. Moreover, we bridge theoretical context complexity and practical memory costs by establishing tight upper and lower bounds on retained task memory, achieving an optimal complexity of Θ(nkL)\Theta(nk_L) for exact representations.

  • Approximation: We establish matching upper and lower bounds for the optimal dimension-mm approximation error,

    \Theta\left(\gamma^m \max\left\\{1,\frac{1}{m(1-\gamma)}\right\\}\right),

    compared with classical Arnoldi [6,7] and Krylov projection methods [8], our approach achieves a strictly tighter upper bound. Moreover, we prove that these traditional methods are provably suboptimal and cannot attain the minimax-optimal rate.

Additionally, we believe that our approximation framework has the potential to further strengthen classical Krylov-based approximation methods and extend their applicability to broader trajectory-approximation problems.

- W1: Shared linear decoder

To establish a meaningful context complexity lower bound, some regularity assumptions are necessary, since without any restrictions, a single real number could encode all task information (Remark C.5).

Importantly, our lower bound does not rely on the linearity of the decoder. Theorem 4.3(c) allows arbitrary nonlinear decoders and only requires the encoder to be continuous. We further establish a newly added Corollary C.6, which extends this result to a token lower bound for any fixed nonlinear reader, including Transformers:

m≥k(n+1)dmodel,m \geq \frac{k(n+1)}{d_{\rm model}},

where mm denotes the context length.

Nevertheless, we retain the linear decoder in our definition of context dimension for two reasons: it naturally preserves the algebraic structure of Krylov subspaces, and it is consistent with the standard readout used in prior in-context TD constructions [1,3].

- W2: Online stochastic TD, exploration, and policy improvement.

We discuss online stochastic TD, exploration, and policy improvement respectively.

  • Online TD.

    We interpret the following setting as online stochastic TD. It updates after each transition. After observing NN transitions, we require the retained context to recover vt(s)v_t(s) exactly for any state ss and any historical time 1≤t≤N1\le t\le N, rather than only the final estimate vN(s)v_N(s). Fixing the state sequence and a constant step size α∈(0,1]\alpha\in(0,1], while allowing rewards to vary over an open set, newly added Proposition C.10(a) shows that any continuous encoder supporting these retrospective queries requires Ω(N)\Omega(N) real registers. Thus, exact compression to o(N)o(N) memory is impossible when preserving the entire online TD history.

  • Exploration and policy improvement.

    In theoretical in-context RL, policy improvement [1], exploration, and fixed-policy evaluation [2] involve distinct theoretical challenges and are rarely studied within a unified framework. To our knowledge, existing theoretical in-context RL studies have not characterized the fundamental context and memory complexity of fixed-policy TD learning. Therefore, our work focuses on establishing these complexity limits for fixed-policy evaluation, while extending our framework to policy improvement and exploration remains an important direction for future research.

- W3: Idealized computational model.

We agree that our analysis involves certain idealized computational assumptions, and we address each of these assumptions in turn below.

  • Real arithmetic: Real arithmetic is an idealized assumption for establishing our theoretical guarantees, and our float32 experiments in Appendix H.5 achieve numerical errors on the order of 10−710^{-7}, demonstrating the numerical feasibility of our approach in the tested settings.

  • Reciprocal operations: As shown in Appendix A.10, reciprocal operations can be approximated arbitrarily accurately using ReLU networks under a positive separation margin. This provides a practical approximation mechanism without requiring exact specialized reciprocal operations.

  • Rank selection: For numerical implementations, Appendix D.9(Proposition D.11) provides a tolerance-based alternative that stops when ρj=∣bj+1∣w∣bj∣w≤τ\rho_j=\frac{\\|b_{j+1}\\|_w}{\\|b_j\\|_w}\leq\tau. Under the contraction assumption, this yields a certified trajectory error bound:

    \\|\widehat v_\ell-\widetilde v_\ell^{G}\\|_w \leq \frac{\tau}{(1-\gamma)^2}\\|\widehat r\\|_w.$$ Thus, numerical rank selection can replace exact rank selection with a controlled approximation error.
  • Token release: We believe that token release is a reasonable assumption for Transformers, as tokens whose information has been fully compressed into the retained representation are no longer needed for subsequent computations and can be removed through token pruning or KV-cache eviction. Such memory management mechanisms are feasible in practical Transformer implementations.

- W4: Do trained Transformers discover the mechanism?

In our theoretical framework, the encoder is designed to have fixed parameters. It compresses raw transitions directly within the Transformer, allowing compression and TD learning to be performed together in a single forward computation. Importantly, although the encoder parameters are fixed, the compressed context is generated from each input task and therefore adapts to different tasks.

Our learning question concerns the shared executor operating on these contexts. Theorem 4.8 proves global convergence of its effective attention parameters to those implementing the TD update. Thus, our results combine a constructive compression mechanism with a provably learnable executor. We think learnability of the compression mechanism itself is a separate important problem, and we focus more on understanding the minimum context required for a Transformer to implement reinforcement learning algorithms.

- W5: Distribution shift.

Our work does provide theoretical guarantees under distribution shifts. Specifically, Theorem 4.8 establishes that, for any pretraining distribution satisfying κ>0\kappa>0, the learned Transformer parameters (C,W)(C,W) converge to the solution (−I,I)(-I,I). This implies that the learned executor is not tied to a particular pretraining distribution and can perform the same TD-learning updates under distribution shifts without retraining.

To empirically validate this result, we conduct experiments across 12 pretraining distributions and 12 shifted test distributions. The learned executors consistently maintain errors below 10−810^{-8} under these distribution shifts (Appendix H.11, Tables 13--14), further supporting our theoretical findings.

- References

[1] Wang, Jiuqi, Ethan Blaser, Hadi Daneshmand, and Shangtong Zhang. "Transformers can learn temporal difference methods for in-context reinforcement learning." The Thirteenth International Conference on Learning Representations (ICLR). 2025.

[2] Lin, Licong, Yu Bai, and Song Mei. "Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining." The Twelfth International Conference on Learning Representations (ICLR). 2024.

[3] Wang, Jiuqi, Rohan Chandra, and Shangtong Zhang. "Towards Provable Emergence of In-Context Reinforcement Learning." Advances in Neural Information Processing Systems 38 (NeurIPS). 2025.

[4] Xie, Zixuan, Xinyu Liu, Rohan Chandra, and Shangtong Zhang. "Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought." arXiv preprint arXiv:2605.07123. 2026.

[5] Xie, Zixuan, Xinyu Liu, Claire Chen, Shuze Daniel Liu, Rohan Chandra, and Shangtong Zhang. "Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning." arXiv preprint arXiv:2605.07333. 2026.

[6] Arnoldi, Walter E. "The Principle of Minimized Iterations in the Solution of the Matrix Eigenvalue Problem." Quarterly of Applied Mathematics 9, no. 1: 17–29. 1951.

[7] Saad, Yousef. "Analysis of Some Krylov Subspace Approximations to the Matrix Exponential Operator." SIAM Journal on Numerical Analysis 29, no. 1: 209–228. 1992.

[8] Freund, Roland W. "Model Reduction Methods Based on Krylov Subspaces." Acta Numerica 12: 267–319. 2003.

Dear Reviewer Sxqu,

Thank you again for your valuable feedback. As the discussion period is approaching its end, we would greatly appreciate it if you could review our rebuttal and let us know if you have any remaining concerns.

Thank you for your time and consideration.

Reviewer_MfdyOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at context compression for in-context RL through the lens of what is required to support TD policy evaluation. The authors explore what context dimension or memory is required for exact TD updates and how this also scales as a function of approximation error. The authors attempt to develop a notion of context complexity in terms of the minimum dimension required to reproduce a specified number of empirical TD iterations, which is related to the Krylov dimension. The authors also develop a notion of memory complexity, which they distinguish from the context dimension. It is demonstrated that with suitable task diversity, an auxiliary objective can provide provable global convergence to the target TD update. Experiments are also provided to validate the predicted context-accuracy tradeoff and memory savings in Transformer models.

Critical strengths
  • This paper takes on a very interesting and important topic right now.
  • Attempting to characterize the context and memory requirements for preserving in-context RL computation feels like a significant contribution, providing a foundation for other papers to build off.
  • The paper takes on a rigorous theoretical perspective while also providing relevant experiments.
  • The experiments validate the approach can be used to achieve very strong levels of compression.
Critical weaknesses
  • In Table 1, the favorable comparison of the current approach relies on the fact that the number of states is less than the number of transitions. In the most realistic settings, states are never revisited, which collapses the benefit of this bound.
  • Given the motivation established for the paper, it was a bit disappointing to see no experiments utilizing LLMs.
  • The fact that the learned representations are specific to the current policy potentially limits the usefulness of this approach in settings with more dynamic policies and continual learning.
Questions for authors
  • Q1: Can you discuss how the bound comparisons in Table 1 relates to the Big World Hypothesis [1] in which states are never revisited? It seems like the benefits of the proposed approach may be related to the extent to which generalization is possible in the environment?
  • Q2: What challenges do you foresee in extending the approach in the paper to the setting of in-context RL with LLMs?
  • Q3: How limiting do you believe the dependence on a fixed policy is for continual learning scenarios? Are potential extensions obvious or are there serious challenges that the authors see?

[1] Javed, Khurram, and Richard S. Sutton. "The big world hypothesis and its ramifications for artificial intelligence." Finding the Frame: An RLC Workshop for Examining Conceptual Frameworks. 2024.

7 comments

We thank reviewer Mfdy for the valuable comments and suggestions. We especially appreciate the reviewer's insightful perspective on the connection between the Big World Hypothesis and our work. We address the concerns below.

- W1 / Q1: States that are never revisited in the real world.

We agree that the first stage of our compression (N→nN \to n) relies on revisiting states. However, we would like to clarify that our compression consists of two stages (N→n→kLN \to n \to k_L), and only the first stage depends on state revisitation. The second stage exploits the low dimensional Krylov subspace, and allow us to use orthogonal basis of Krylov subspace to represent all states (only kLk_L dimensions).

To directly investigate the setting described by the Big World Hypothesis, we added three real-world sensor datasets (Table 2, and Appendix H.8): hourly bike rentals, appliance energy use, and CO concentration. Over 99.6% of the raw records are distinct (n≈Nn\approx N), and our method achieves 7.3--15$\times$ lower memory usage than sparse TD and 31--54$\times$ lower memory usage than LSTD statistics. These results demonstrate that our compression remains effective even when state revisitation is extremely limited.

Additionally, Javed and Sutton (2024) argued that function approximation should be used to represent the large number of states in real-world problems, which is what our Theorem 4.4 achieves.

- W2 / Q2: Challenges in extending our framework to LLMs.

Our work primarily focuses on understanding the theoretical mechanisms of in-context reinforcement learning, particularly whether Transformers can implement RL algorithms with fixed parameters and whether these parameters can be learned through pretraining.

Extending our framework to pretrained LLMs would essentially involve fine-tuning them to attain the ability to implement RL algorithms. However, fine-tuning pretrained LLMs differs substantially from training Transformers in controlled theoretical settings. Moreover, from the theoretical perspective, the enormous number of parameters in LLMs makes it challenging to theoretically characterize the role of each parameter and establish rigorous guarantees. Therefore, we focus primarily on understanding the theoretical mechanisms of in-context RL rather than conducting empirical studies of LLM fine-tuning.

Previous theoretical studies on in-context RL [1,2,3,4] and the broader in-context learning literature [5,6,7,8] have also primarily focused on controlled Transformer settings without conducting experiments on pretrained LLMs. We therefore view extending these theoretical guarantees to pretrained LLMs and fine-tuning as an important direction for future research.

- W3 / Q3: Continual learning and dynamic policy.

We discuss the continual learning and dynamic policy respectively. We set continual learning as environmental drift. For slow environmental drift, the policy remains fixed while transition dynamics and rewards may change. Data arrive in a series of transitions, and each transition receives weight λat\lambda^{a_t}, where λ∈(0,1)\lambda\in(0,1) and ata_t is its age in the series of transitions. Under state coverage, the newly added Lemma A.9 extends Theorem 4.3 to the resulting weighted empirical model. Specifically, Lemma A.9 gives

\le \Delta_{\mathrm{drift}} := \frac{\delta_r}{1-\gamma} + \frac{\gamma\delta_P\\|\widehat r\\|_\infty}{(1-\gamma)^2}.$$ Thus, an exact context can be reused at its supported depths as long as $\Delta_{\mathrm{drift}}$ remains within the error tolerance. For dynamic policies, we use a fixed dataset in which every state--action pair used by the policies has been observed. Policies may be chosen adaptively, but each is held fixed during its TD evaluation. Define $$\eta_t=\max_i\\|\pi_t(\cdot\mid i)-\pi_{t-1}(\cdot\mid i)\\|_1,$$ Corollary A.10 in Appendix A.15 shows that, if the context was last constructed exactly for $\pi_\tau$, $$\max_{0\le\ell\le L} \\|\widehat v_\ell^{\pi_t}-\widehat v_\ell^{\pi_\tau}\\|_\infty \le \frac{\bar R}{(1-\gamma)^2} \sum_{j=\tau+1}^{t}\eta_j.$$ Here $\bar R$ bounds the absolute empirical rewards. Thus, the reuse error is controlled by cumulative policy variation. This provides an accuracy guarantee for context reuse along adaptive policy sequences. Due to space limitations, the references are provided in the following comment.

- References

[1] Wang, Jiuqi, Ethan Blaser, Hadi Daneshmand, and Shangtong Zhang. "Transformers can learn temporal difference methods for in-context reinforcement learning." The Thirteenth International Conference on Learning Representations (ICLR). 2025.

[2] Wang, Jiuqi, Rohan Chandra, and Shangtong Zhang. "Towards Provable Emergence of In-Context Reinforcement Learning." Advances in Neural Information Processing Systems 38 (NeurIPS). 2025.

[3] Xie, Zixuan, Xinyu Liu, Rohan Chandra, and Shangtong Zhang. "Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought." arXiv preprint arXiv:2605.07123. 2026.

[4] Xie, Zixuan, Xinyu Liu, Claire Chen, Shuze Daniel Liu, Rohan Chandra, and Shangtong Zhang. "Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning." arXiv preprint arXiv:2605.07333. 2026.

[5] Akyürek, Ekin, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. "What Learning Algorithm Is In-Context Learning? Investigations with Linear Models." The Eleventh International Conference on Learning Representations (ICLR). 2023.

[6] von Oswald, Johannes, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. "Transformers Learn In-Context by Gradient Descent." Proceedings of the 40th International Conference on Machine Learning (ICML). 2023.

[7] Bai, Yu, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. "Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection." Advances in Neural Information Processing Systems 36 (NeurIPS). 2023.

[8] Zhang, Ruiqi, Spencer Frei, and Peter L. Bartlett. "Trained Transformers Learn Linear Models In-Context." Journal of Machine Learning Research 25(49): 1–55. 2024.

Reviewer_MfdyOpenReview ↗

Re: Official Comment by Authors

Thank you for engaging with my review. I especially appreciate the clarification with respect to W1 / Q1. I found the explanation for W3 / Q3 a bit confusing though. First of all, the equations you posted seem broken in the OpenReview markdown. Can you summarize what you can show with this theoretical work for the drifting environment and drifting policy case and how this relates back to the contributions of the paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.