- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed Recurrent Algorithm Distillation to perform in-context reinforcement learning and gather long-term learning progress information. RAD uses a double-component architecture: a Compression Transformer to compress history information into few compact latent tokens, and an AD Transformer to learn the policy with a combined context of compressed memories and recent experience. Experiments demonstrate that RAD performs at eh samelvel with AD, with much less compute cost.
Critical strengths
While memory encoders are learnt a lot in LLM, I find this idea of using compressed memory in In-Context Reinforcement Learning very interesting. Theoretical development are solid, using GRU style update, recursively compress the memory, and use bi-level working memory and long-term memory to make predictions. Current frontier LLM, like Deepseek V4, use hybriad attention, like DSA , to solve long context attention compute problem. I wonder if this could be an extension to be used in AD Transformer.
Critical weaknesses
The experiment can include more baselines. As menitoned in the Figure 3, Dark Key-to-Door, DPT and IPT performs better than RAD. While they require domain knowledge, what if we give empty or random information to DPT and IPT? Is it possible to compare with other more recent baselines like [1] or [2] ? Why Meta-world only compares RAD with AD, not include baselines like DPT and IPT? Is this because of compute cost?
[1] Wang, Fan, et al. "Towards large-scale in-context reinforcement learning by meta-training in randomized worlds." Advances in Neural Information Processing Systems 38 (2026): 171669-171704.
[2] Wu, Wenhao, et al. "Mixture-of-experts meets in-context reinforcement learning." Advances in Neural Information Processing Systems 38 (2026): 24751-24785.
Questions for authors
- RNN is known to be very hard to train [1] [2], RL algorithm can also be unstable especially with large-parameter neural nets [3]. I wonder if adding the compression transformer makes the training hard to converge or need more efforts on hyperparameter searching.
- Why you treat observation, action and reward as distinct tokens? Is there any reasoning that you choose to be different from previous methods' tokenisation?
- I wonder what do you mean by 'scalable' here? Usually, it means we can scale up model parameters/data size to get better performance. Your experiment looks more like 'efficient' than 'scalable' to me.
- minor: Should In-Context Reinforcement Learning be ICRL instead of ICLR in line 31?
[1] Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. "On the difficulty of training recurrent neural networks." International conference on machine learning. Pmlr, 2013.
[2] Bengio, Yoshua, Patrice Simard, and Paolo Frasconi. "Learning long-term dependencies with gradient descent is difficult." IEEE transactions on neural networks 5.2 (1994): 157-166.
[3] Wang, Kevin, et al. "1000 layer networks for self-supervised rl: Scaling depth can enable new goal-reaching capabilities." Advances in Neural Information Processing Systems 38 (2026): 157643-157670.
2 comments
Response to Reviewer Erwt (Part 2)
Q2: Why you treat observation, action and reward as distinct tokens? Is there any reasoning that you choose to be different from previous methods' tokenisation?
We treat them as distinct tokens to make the method scalable. With this tokenizer, the model's context is appending-only, similar to LLM agents. Thus for a single sampled sequence , we can forward it into the Transformer in a single pass, and we can calculate a loss for each action prediction at each observation token, similar to training LLMs by forwarding an entire agent trajectory and calculate the loss on all the model generated tokens. While for the DPT-style tokenizer which treats one timestep as a single token, one tokenized input sequence can only be used to predict one single action , which is less efficient. The original AD paper did not state which tokenization strategy it used, but we believe that the former tokenization method is more scalable and efficient for training, though the two tokenizers have similar performance on the environments, thus we would like to discuss this in our work.
Q3: I wonder what do you mean by "scalable" here? Usually, it means we can scale up model parameters/data size to get better performance. Your experiment looks more like "efficient" than "scalable" to me.
We agree that scalability often refers to improved performance as model size or training data increases. In this paper, we use the term more specifically to mean scalability with respect to accumulated interaction-history length. Standard AD must expand or truncate its explicit context to incorporate longer histories, whereas RAD maintains a fixed-capacity working memory and recurrently compresses older interactions into a fixed-size latent state. Consequently, the amount of context processed for each action does not grow with the total history length, although compression and memory updates still incur a cost. Our results support this narrower computational notion of scalability.
Q4: Should In-Context Reinforcement Learning be ICRL instead of ICLR in line 31?
Thank you for pointing out this typo. We have corrected it to ICRL and proofread the manuscript for similar errors.
Thank you again for the constructive feedback and the opportunity to clarify these points.
References
- Huang, S., Hu, J., Chen, H., Sun, L., & Yang, B. (2024). In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought. arXiv preprint arXiv:2405.20692. https://arxiv.org/abs/2405.20692
- Lee, J., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., & Brunskill, E. (2023). Supervised Pretraining Can Learn In-Context Reinforcement Learning. arXiv preprint arXiv:2306.14892. https://arxiv.org/abs/2306.14892
- Wang, F., et al. (2025). Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds. arXiv preprint arXiv:2502.02869. https://arxiv.org/abs/2502.02869
- Wang, K., Javali, I., Bortkiewicz, M., Trzciński, T., & Eysenbach, B. (2025). 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities. arXiv preprint arXiv:2503.14858. https://arxiv.org/abs/2503.14858
- Wu, W., Liu, F., Li, H., Hu, Z., Dong, D., Chen, C., & Wang, Z. (2025). Mixture-of-Experts Meets In-Context Reinforcement Learning. arXiv preprint arXiv:2506.05426. https://arxiv.org/abs/2506.05426
Response to Reviewer Erwt (Part 1)
Thank you for the detailed suggestions and questions. We address the baseline comparisons, training considerations, and terminology below.
It is important to distinguish the supervision used during pretraining from the context supplied at inference. DPT is pretrained to predict optimal actions conditioned on an interaction dataset and a query state (Lee et al., 2023); this requires optimal-action labels during training. However, DPT can begin evaluation with an empty interaction context and then accumulate information through exploration, so it would be incorrect to claim that an empty test-time context necessarily makes DPT fail. Replacing DPT's training labels with random actions would instead change its supervised objective and is not equivalent to providing an empty inference context. Lee et al. (2023) also conducted experiments on replacing optimal actions with actions sampled from algorithm-generated policies, and the results show that it degrades DPT's performance. IDT, by contrast, conditions on a desied return for inference (Huang et al., 2024), where random or zero desired-return prompts change its behavior. RAD differs in that its training does not require optimal-action labels or a desired-return target at test time; it distills action predictions from source RL learning histories.
We have added T2MIR (Wu et al., 2025) to the revised experiments. The methods generally matches each others' performance, with RAD performing better on Darkroom, whereas T2MIR performs better on Dark Key-to-Door; their overall results are broadly comparable. This suggests that architectural improvements such as mixture-of-experts layers and recurrent memory compression may be complementary, an interesting direction for future work to create computation-efficient and strong-performing ICRL methods.
We have not included AnyMDP (F. Wang et al., 2025) because reproducing its large-scale meta-training protocol is beyond our available computational budget. In addition, AnyMDP is formulated using procedurally generated tabular MDPs with discrete actions. Therefore, the published formulation does not transfer directly to the continuous-action Meta-World tasks.
The initial omission was due to computational and time constraints. We have since evaluated DPT and IDT on Meta-World and added their results to the revised PDF. For DPT, we generated expert-action labels using Meta-World's scripted expert policies. We also aligned the observation and action representations across methods and reran the experiments to improve comparability.
The revised results show that RAD generally matches AD while using a much shorter working-memory context. DPT achieves the strongest overall performance in these experiments, benefiting from its expert-action supervision, while IDT and T2MIR achieve results broadly comparable to RAD and AD. We will make these differences in training supervision explicit so that the comparison is interpreted fairly.
Yes. Jointly optimizing the AD Tranformer and the Compression Transformer is harder to converge than training the original AD model, since the recurrent memory requires back-propation through time, which is generally recognized as costly and unstable. To tackle this and stablize training, we introduce the memory update gating and truncating distant gradient signals. We have also have a more detailed explanation in Appendix C regarding the convergence speed of AD and RAD. RAD generally converges slower than AD, yet it is more efficient during inference.
The work of K. Wang et al. (2025) studies scaling network depth for self-supervised, goal-conditioned RL policies. Our setting is different: ICRL trains a Transformer by supervised distillation of source algorithms' learning histories, without directly applying RL to update the network after supervised pretraining. Thus, the paper raises a related but distinct optimization question; its findings do not directly establish the convergence behavior of RAD.