acceptodds
Under review as a conference paper at ICLR 2027

Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation

Abstract

Algorithm Distillation (AD) has demonstrated the remarkable ability of Transformers to perform in-context reinforcement learning without explicit weight updates. However, capturing long-term learning progress necessitates expansive context windows, which incur prohibitive memory costs and limit scalability in complex, long-horizon tasks. To address this bottleneck, we propose *Recurrent Algorithm Distillation (RAD)*. RAD employs a dual-component architecture: a *Compression Transformer* that distills extended interaction histories into compact latent tokens, and an *AD Transformer* that auto-regressively generates actions using a hybrid context of these compressed memories and recent transitions. By maintaining a fixed-size latent buffer, RAD decouples the effective history length from computational complexity, functionally providing the model with a long-horizon memory. Empirical evaluations across diverse environments demonstrate that RAD matches the asymptotic performance of standard AD with significantly reduced context window sizes, offering a scalable solution for efficient in-context decision-making.

Primary area:
reinforcement learning
Keywords:
in-context reinforcement learning, algorithm distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ErwtOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed Recurrent Algorithm Distillation to perform in-context reinforcement learning and gather long-term learning progress information. RAD uses a double-component architecture: a Compression Transformer to compress history information into few compact latent tokens, and an AD Transformer to learn the policy with a combined context of compressed memories and recent experience. Experiments demonstrate that RAD performs at eh samelvel with AD, with much less compute cost.

Critical strengths

While memory encoders are learnt a lot in LLM, I find this idea of using compressed memory in In-Context Reinforcement Learning very interesting. Theoretical development are solid, using GRU style update, recursively compress the memory, and use bi-level working memory and long-term memory to make predictions. Current frontier LLM, like Deepseek V4, use hybriad attention, like DSA , to solve long context attention compute problem. I wonder if this could be an extension to be used in AD Transformer.

Critical weaknesses

The experiment can include more baselines. As menitoned in the Figure 3, Dark Key-to-Door, DPT and IPT performs better than RAD. While they require domain knowledge, what if we give empty or random information to DPT and IPT? Is it possible to compare with other more recent baselines like [1] or [2] ? Why Meta-world only compares RAD with AD, not include baselines like DPT and IPT? Is this because of compute cost?

[1] Wang, Fan, et al. "Towards large-scale in-context reinforcement learning by meta-training in randomized worlds." Advances in Neural Information Processing Systems 38 (2026): 171669-171704.

[2] Wu, Wenhao, et al. "Mixture-of-experts meets in-context reinforcement learning." Advances in Neural Information Processing Systems 38 (2026): 24751-24785.

Questions for authors
  1. RNN is known to be very hard to train [1] [2], RL algorithm can also be unstable especially with large-parameter neural nets [3]. I wonder if adding the compression transformer makes the training hard to converge or need more efforts on hyperparameter searching.
  2. Why you treat observation, action and reward as distinct tokens? Is there any reasoning that you choose to be different from previous methods' tokenisation?
  3. I wonder what do you mean by 'scalable' here? Usually, it means we can scale up model parameters/data size to get better performance. Your experiment looks more like 'efficient' than 'scalable' to me.
  4. minor: Should In-Context Reinforcement Learning be ICRL instead of ICLR in line 31?

[1] Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. "On the difficulty of training recurrent neural networks." International conference on machine learning. Pmlr, 2013.

[2] Bengio, Yoshua, Patrice Simard, and Paolo Frasconi. "Learning long-term dependencies with gradient descent is difficult." IEEE transactions on neural networks 5.2 (1994): 157-166.

[3] Wang, Kevin, et al. "1000 layer networks for self-supervised rl: Scaling depth can enable new goal-reaching capabilities." Advances in Neural Information Processing Systems 38 (2026): 157643-157670.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Erwt (Part 1)

Thank you for the detailed suggestions and questions. We address the baseline comparisons, training considerations, and terminology below.

W1: While they require domain knowledge, what if we give empty or random information to DPT and IDT?

It is important to distinguish the supervision used during pretraining from the context supplied at inference. DPT is pretrained to predict optimal actions conditioned on an interaction dataset and a query state (Lee et al., 2023); this requires optimal-action labels during training. However, DPT can begin evaluation with an empty interaction context and then accumulate information through exploration, so it would be incorrect to claim that an empty test-time context necessarily makes DPT fail. Replacing DPT's training labels with random actions would instead change its supervised objective and is not equivalent to providing an empty inference context. Lee et al. (2023) also conducted experiments on replacing optimal actions with actions sampled from algorithm-generated policies, and the results show that it degrades DPT's performance. IDT, by contrast, conditions on a desied return for inference (Huang et al., 2024), where random or zero desired-return prompts change its behavior. RAD differs in that its training does not require optimal-action labels or a desired-return target at test time; it distills action predictions from source RL learning histories.

W2: Is it possible to compare with other more recent baselines?

We have added T2MIR (Wu et al., 2025) to the revised experiments. The methods generally matches each others' performance, with RAD performing better on Darkroom, whereas T2MIR performs better on Dark Key-to-Door; their overall results are broadly comparable. This suggests that architectural improvements such as mixture-of-experts layers and recurrent memory compression may be complementary, an interesting direction for future work to create computation-efficient and strong-performing ICRL methods.

We have not included AnyMDP (F. Wang et al., 2025) because reproducing its large-scale meta-training protocol is beyond our available computational budget. In addition, AnyMDP is formulated using procedurally generated tabular MDPs with discrete actions. Therefore, the published formulation does not transfer directly to the continuous-action Meta-World tasks.

W3: Why Meta-world only compares RAD with AD, not include baselines like DPT and IPT? Is this because of compute cost?

The initial omission was due to computational and time constraints. We have since evaluated DPT and IDT on Meta-World and added their results to the revised PDF. For DPT, we generated expert-action labels using Meta-World's scripted expert policies. We also aligned the observation and action representations across methods and reran the experiments to improve comparability.

The revised results show that RAD generally matches AD while using a much shorter working-memory context. DPT achieves the strongest overall performance in these experiments, benefiting from its expert-action supervision, while IDT and T2MIR achieve results broadly comparable to RAD and AD. We will make these differences in training supervision explicit so that the comparison is interpreted fairly.

Q1: I wonder if adding the compression transformer makes the training hard to converge or need more efforts on hyperparameter searching.

Yes. Jointly optimizing the AD Tranformer and the Compression Transformer is harder to converge than training the original AD model, since the recurrent memory requires back-propation through time, which is generally recognized as costly and unstable. To tackle this and stablize training, we introduce the memory update gating and truncating distant gradient signals. We have also have a more detailed explanation in Appendix C regarding the convergence speed of AD and RAD. RAD generally converges slower than AD, yet it is more efficient during inference.

The work of K. Wang et al. (2025) studies scaling network depth for self-supervised, goal-conditioned RL policies. Our setting is different: ICRL trains a Transformer by supervised distillation of source algorithms' learning histories, without directly applying RL to update the network after supervised pretraining. Thus, the paper raises a related but distinct optimization question; its findings do not directly establish the convergence behavior of RAD.

AuthorsOpenReview ↗

Response to Reviewer Erwt (Part 2)

Q2: Why you treat observation, action and reward as distinct tokens? Is there any reasoning that you choose to be different from previous methods' tokenisation?

We treat them as distinct tokens to make the method scalable. With this tokenizer, the model's context is appending-only, similar to LLM agents. Thus for a single sampled sequence (o1,a1,r1,o2,...,oT,aT,rT)(o_1, a_1, r_1, o_2, ..., o_T, a_T, r_T), we can forward it into the Transformer in a single pass, and we can calculate a loss for each action prediction at each observation token, similar to training LLMs by forwarding an entire agent trajectory and calculate the loss on all the model generated tokens. While for the DPT-style tokenizer which treats one timestep as a single token, one tokenized input sequence (x1,x2,...,xT−1,oT)(x_1, x_2, ..., x_{T-1}, o_T) can only be used to predict one single action aTa_T, which is less efficient. The original AD paper did not state which tokenization strategy it used, but we believe that the former tokenization method is more scalable and efficient for training, though the two tokenizers have similar performance on the environments, thus we would like to discuss this in our work.

Q3: I wonder what do you mean by "scalable" here? Usually, it means we can scale up model parameters/data size to get better performance. Your experiment looks more like "efficient" than "scalable" to me.

We agree that scalability often refers to improved performance as model size or training data increases. In this paper, we use the term more specifically to mean scalability with respect to accumulated interaction-history length. Standard AD must expand or truncate its explicit context to incorporate longer histories, whereas RAD maintains a fixed-capacity working memory and recurrently compresses older interactions into a fixed-size latent state. Consequently, the amount of context processed for each action does not grow with the total history length, although compression and memory updates still incur a cost. Our results support this narrower computational notion of scalability.

Q4: Should In-Context Reinforcement Learning be ICRL instead of ICLR in line 31?

Thank you for pointing out this typo. We have corrected it to ICRL and proofread the manuscript for similar errors.


Thank you again for the constructive feedback and the opportunity to clarify these points.

References

  • Huang, S., Hu, J., Chen, H., Sun, L., & Yang, B. (2024). In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought. arXiv preprint arXiv:2405.20692. https://arxiv.org/abs/2405.20692
  • Lee, J., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., & Brunskill, E. (2023). Supervised Pretraining Can Learn In-Context Reinforcement Learning. arXiv preprint arXiv:2306.14892. https://arxiv.org/abs/2306.14892
  • Wang, F., et al. (2025). Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds. arXiv preprint arXiv:2502.02869. https://arxiv.org/abs/2502.02869
  • Wang, K., Javali, I., Bortkiewicz, M., Trzciński, T., & Eysenbach, B. (2025). 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities. arXiv preprint arXiv:2503.14858. https://arxiv.org/abs/2503.14858
  • Wu, W., Liu, F., Li, H., Hu, Z., Dong, D., Chen, C., & Wang, Z. (2025). Mixture-of-Experts Meets In-Context Reinforcement Learning. arXiv preprint arXiv:2506.05426. https://arxiv.org/abs/2506.05426
Reviewer_6LLCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the scalability bottleneck of Algorithm Distillation (AD) for in-context reinforcement learning: capturing long-horizon learning progress requires large context windows, which are computationally expensive due to Transformer quadratic attention. The authors propose Recurrent Algorithm Distillation (RAD), a dual-component architecture combining a Compression Transformer that recursively distills extended interaction histories into a fixed-size set of latent tokens, and an AD Transformer that generates actions from a hybrid context of compressed long-term memory and recent transitions. A GRU-style gated update maintains the latent memory under a fixed budget. The paper also provides a theoretical interpretation of AD as implicit posterior sampling, motivating the need for long-term memory. RAD is validated on Delayed-Bandit, Darkroom, Dark Key-to-Door, and Meta-World, matching or exceeding AD with shorter contexts and substantially lower inference FLOPs, with ablations on memory updates, tokenizers, compressors, and memory size.

Critical strengths

1.The theoretical interpretation of AD as implicit posterior sampling gives the paper a principled foundation rather than a purely heuristic motivation. The derivation shows that the optimal AD policy decomposes into task posterior inference and source algorithm conditioned action prediction. This directly explains why sliding window context truncation damages in context reinforcement learning and why preserving long term information through compressed memory is a meaningful objective. This strength is consequential because it connects RAD’s architectural design to a clear theoretical failure mode of AD. 2.The RAD method and its empirical validation provide strong evidence for scalable in context reinforcement learning. The dual component architecture uses a compression Transformer to consolidate older interaction history into a fixed size latent memory and an AD Transformer to act from compressed memory plus recent transitions. The gated recurrent update keeps memory bounded, and the results on Delayed Bandit, Darkroom, Dark Key to Door, and Meta World show that RAD matches or exceeds AD with shorter contexts while reducing inference FLOPs substantially. The ablations on memory updates, tokenizers, compressors, and memory size further support the design choices. This strength matters because it demonstrates practical scalability without sacrificing in context adaptation.

Critical weaknesses

The central scalability claim is not adequately supported. RAD is evaluated on relatively small benchmarks with short horizons, limited task diversity, and modest model sizes. Delayed-Bandit, Darkroom, Dark Key-to-Door, and Meta-World ML1 do not stress the long-horizon, large-scale regime that motivates the method. Training budgets and context lengths remain modest, and no scaling study across model size, memory size, task count, or horizon is provided. Consequently, it remains unclear whether RAD scales to genuinely complex long-horizon in-context RL problems. This weakens the paper's main contribution.

Questions for authors

How do the authors formally define a long-horizon problem in in-context reinforcement learning, and which of the evaluated environments qualify as long horizon under that definition?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6LLC

Thank you for your thoughtful review and your positive assessment of the paper's motivation and architecture. We address your concern about the definition and evaluation of long-horizon ICRL below.

W1: The central scalability claim is not adequately supported. RAD is evaluated on relatively small benchmarks with short horizons, limited task diversity, and modest model sizes. No scaling study across model size, memory size, task count, or horizon is provided.

Q1: How do the authors formally define a long-horizon problem in in-context reinforcement learning, and which of the evaluated environments qualify as long-horizon under that definition?

Our understanding is that "long-horizon" is a relative term without a universally accepted threshold in reinforcement learning. In the context of ICRL, we believe it is important to distinguish between two notions of horizon: (1) episode horizon, which measures the number of environment steps within a single episode, and (2) in-context learning horizon, which measures the span of interaction history over which an agent must retain and utilize information to adapt its behavior. The latter is particularly relevant to ICRL, as adaptation can occur across multiple episodes without parameter updates (Laskin et al., 2022). From this perspective, we consider a task to involve long-horizon dependencies when successful adaptation requires utilizing information beyond the ICRL agent's short-term context window. Among our evaluated environments, Meta-World features a relatively longer episode horizon (100 steps) compared with the bandit and gridworld tasks. However, we acknowledge that a 100-step episode alone does not establish a genuinely long-horizon benchmark. Our Delayed Bandit experiments more directly investigate long-term dependencies by requiring the agent to retain information and associate delayed rewards with earlier actions, and it can be considered as having a long in-context learning horizon. The results on Delayed Bandit show that RAD can effectively learn useful latent representations for the distant history and perform long-term credit assignment.

The reason we did not include a genuine long-horizon benchmark (e.g., tasks with thousand-step episodes) is that there are very few such benchmarks that are publicly available and satisfy the requirements of an In-Context RL benchmark. According to Laskin et al. (2022), the evaluation environments for ICRL methods should generally satisfy the following requirements:

  • Supports many tasks: The environments should be multi-task to prevent ICRL agents from overfitting to a single task.
  • Task mush be hard to infer: Downstream tasks require exploration to converge.

For existing long-horizon benchmarks like Memory Maze (Pasukonis et al., 2022) which features thoudsand-step episodes, the number of tasks is relatively limited.

Additionally, if we consider AD and RAD as offline RL methods and directly evaluate them on the existing benchmarks like Memory Maze, we will face the challenge of generating the training datasets, which requires running a source online RL algorithm to convergence. However, training a simple online RL algorithm to solve such long-horizon tasks is very challenging, often requiring training additional networks like world-models and are very computation and memory demanding. Unfortunately, due to resource constraints, we are unable to provide these large-scale experiments during the rebuttal period.


Thank you again for highlighting this important limitation.

References

Reviewer_tUCvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Recurrent Algorithm Distillation (RAD) to address the context-length bottleneck in in-context reinforcement learning. RAD combines a policy Transformer with a compression Transformer that recursively summarizes interaction histories into a fixed-size latent memory, while retaining recent transitions at full resolution. The memory is updated through a GRU-style gate, and the model is trained on source RL learning histories through compression pretraining followed by action distillation. Experiments on delayed bandits, grid-world environments, and Meta-World tasks show that RAD generally matches or improves upon standard Algorithm Distillation while using shorter contexts and reducing reported inference FLOPs.

Critical strengths
  1. The method presents a coherent design that decouples the accumulated history length from the directly processed context length, while retaining uncompressed recent interactions to balance local detail with long-term information.

  2. The experiments cover bandits, discrete navigation, and continuous control. Delayed-Bandit specifically evaluates history retention across distractor intervals and demonstrates generalization to unseen distractor lengths.

  3. The paper reports substantial reductions in inference computation and provides ablations of several design choices.

Critical weaknesses
  1. The paper lacks comparisons with the most directly relevant baselines, making it difficult to establish its incremental contribution. Although it discusses related methods such as RMT, structured state-space models, RATE, and RA-DT, the main experiments primarily compare against standard AD, DPT, and IDT. The results support the usefulness of incorporating memory into AD, but do not sufficiently establish RAD’s advantages over existing memory mechanisms.

  2. According to Algorithm 2, the history accessed by the final compression extends to t0+nD+p,t_0+nD+p, whereas action supervision begins at (t0+nD)(t_0+nD). The first pp transitions in the supervised interval have already been incorporated into the compressed memory, potentially allowing the model to access the actions it is being trained to predict. Is this a typographical error or an implementation issue? Could the authors clarify whether the training procedure prevents this potential target leakage?

  3. Table 1 reports average inference FLOPs per action, but the paper does not clearly specify whether these figures include the amortized cost of recurrent compression and gated memory updates, or whether AD and RAD use comparable caching strategies. Moreover, FLOPs alone do not establish practical efficiency: measurements of inference latency and peak memory usage are missing. Since scalability is a central motivation, it would be valuable to show how these costs and task performance change as the interaction history grows, under fixed working-memory and latent-memory capacities.

  4. The existing ablations show that certain gating and compression configurations perform better, but do not directly establish what information the memory retains to support continued adaptation, or whether the performance gains genuinely depend on distant history. Interventions that clear or replace the stored memory, together with evaluations across different history lengths, would provide stronger evidence for the core mechanism than additional ablations of individual architectural choices.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tUCv (Part 1)

Thank you for your thoughtful review and constructive feedback. We address each concern below.

W1: The paper lacks comparisons with the most directly relevant baselines, making it difficult to establish its incremental contribution. Although it discusses related methods such as RMT, structured state-space models, RATE, and RA-DT, the main experiments primarily compare against standard AD, DPT, and IDT.

The listed methods, including RMT, structured state-space models, RATE, and RA-DT, are generally considered as offline RL algorithms. While RAD could also be considered as an offline RL algorithm in some sense, it is often considered as an In-Context RL algorithm, a meta-RL method that focuses on learning the improvement signal of regular RL algorithms and in-context explore and exploit in unseen tasks. In contrast, methods like RMT are not designed to be generalizable. Therefore, we focus our comparison on ICRL methods. According to Laskin et al. (2022), the evaluation environments for ICRL methods should generally satisfy the following requirements:

  • Supports many tasks: The environments should be multi-task to prevent ICRL agents from overfitting to a single task.
  • Task mush be hard to infer: Downstream tasks require exploration to converge.

Many of the existing long-horizon benchmarks do not satisfy these requirements. For instance, RATE is evaluated on Memory Maze (Pasukonis et al., 2022), which do not support configurable multiple tasks. After each episode ends, the environment resets to a different maze. Therefore, we do not consider it as a long-horizon ICRL environment.

Moreover, the main challenge for comparing RAD with RMT-like methods on long-horizon tasks is that RAD has more constraints on the training data, requiring running a source RL algorithm on the tasks until convergence, while RMT-like offline RL methods generally accepts regular offline datasets, possibly generated by human demonstration or scripted policies. However, training a simple online RL algorithm to solve such long-horizon tasks is very challenging, often requiring training additional networks like world-models and are very computation and memory demanding. Unfortunately, due to resource constraints, we are unable to provide these large-scale experiments during the rebuttal period.

Although we have not included genuine long episodic horizon benchmarks, we have conducted experiments on the Delayed Bandit tasks, which feature long-term dependencies and have long in-context learning horizons. The results show that RAD can effectively learn useful latent representations for the distant history and perform long-term credit assignment.

W2: The first pp transitions in the supervised interval have already been incorporated into the compressed memory, potentially allowing the model to access the actions it is being trained to predict. Is this a typographical error or an implementation issue?

We apologize for the incorrect notations and unclear explanation. In our code implementation, RAD is trained on the generated actions in the final working memory, simulating the actual inference process. You can refer to code in the supplementary materials for more details. The corrected objective is:

LRAD(θ,ω,ϕ)=−Ext0:t0+ℓ∼D[1D∑t=t0+nD+pt0+ℓ−1log⁡πθ(at∣zn,xt0+nD:t,ot)]\mathcal{L}_{\mathrm{RAD}}(\theta,\omega,\phi)=-\mathbb{E}_{x_{t_0:t_0+\ell}\sim\mathcal{D}}\left[\frac{1}{D}\sum_{t=t_0+nD+p}^{t_0+\ell-1}\log\pi_\theta\left(a_t\mid z_n,x_{t_0+nD:t},o_t\right)\right]

After the final compression, the context visible to the AD Transformer is the long-term latent memory tokens plus the retained pp recent transitions, and it is trained to continue auto-regressively predicting future action tokens. The retained recent transitions are not used for training the AD Transformer, but only for providing additional context to the AD Transformer during inference. Therefore there is no target leakage in our method. We have updated Section 4.4 to clarify this point.

AuthorsOpenReview ↗

Response to Reviewer tUCv (Part 2)

W3: Table 1 reports average inference FLOPs per action, but the paper does not clearly specify whether these figures include the amortized cost of recurrent compression and gated memory updates, or whether AD and RAD use comparable caching strategies. Moreover, FLOPs alone do not establish practical efficiency: measurements of inference latency and peak memory usage are missing.

The reported FLOPs per action include the computational costs of both the Compression Transformer and the gated long-term-memory update. Both AD and RAD did not implement a KV cache, thus the caching strategies are comparable. We have also added inference latency and peak memory measurements to Table 1 in the revised PDF. The profiling results are reproduced below; all latency and memory figures correspond to the batch sizes shown.

EnvironmentBatchAD FLOPs/actionRAD FLOPs/actionRAD/AD ↓AD latency (ms/action)RAD latency (ms/action)AD peak memory (MiB)RAD peak memory (MiB)
Bandit (short AD)159.8 M43.0 M0.72×1.10981.209710.3213.19
Bandit (long AD)1263.4 M43.0 M0.16×1.15071.209716.0713.19
Darkroom8144.8 M27.8 M0.19×0.12970.180617.8522.69
Dark Key-to-Door33203.4 M85.9 M0.42×0.04420.043544.0649.99
Meta-World501.84 G130.6 M0.07×0.29600.0509208.1086.23

RAD does not improve every resource metric in every environment. For Bandit and Darkroom, its additional Compression Transformer and memory operations introduce overhead that outweighs the latency or memory benefits of a shorter context. In contrast, on Meta-World, RAD reduces measured latency from 0.2960 to 0.0509 ms/action and peak memory from 208.10 to 86.23 MiB. These results illustrate the trade-off: recurrent compression adds a fixed overhead but can become advantageous when the uncompressed AD context is sufficiently costly. We have revised our discussion to distinguish reductions in FLOPs from improvements in wall-clock latency and peak memory.

W4: The existing ablations show that certain gating and compression configurations perform better, but do not directly establish what information the memory retains to support continued adaptation, or whether the performance gains genuinely depend on distant history.

We performed this intervention on the Delayed Bandit benchmark and report the results in Appendix C. We evaluated the first arm prediction after the distractor interval under four conditions: (i) intact long-term memory, (ii) memory cleared to zero, (iii) memory replaced with that obtained from an independent history of the same task, and (iv) memory replaced with that from a different task.

First-action expected regret (lower is better)

DelayIntactClearedSame-task swapDifferent-task swap
00.042 ± 0.0020.042 ± 0.0020.042 ± 0.0020.042 ± 0.002
250.081 ± 0.0110.312 ± 0.0210.087 ± 0.0110.365 ± 0.008
500.058 ± 0.0020.347 ± 0.0060.063 ± 0.0020.390 ± 0.006
1000.075 ± 0.0100.354 ± 0.0080.081 ± 0.0090.391 ± 0.005
2000.106 ± 0.0200.323 ± 0.0100.111 ± 0.0190.370 ± 0.008
4000.196 ± 0.0280.358 ± 0.0030.207 ± 0.0280.377 ± 0.009

Probability assigned to the different-task donor's optimal arm (%)

DelayIntactClearedSame-task swapDifferent-task swap
06.9 ± 1.06.9 ± 1.06.9 ± 1.06.9 ± 1.0
256.9 ± 0.916.0 ± 1.16.7 ± 0.960.4 ± 1.5
507.2 ± 0.619.3 ± 0.56.9 ± 0.571.2 ± 2.2
1008.1 ± 0.719.2 ± 0.57.8 ± 0.666.9 ± 3.0
2008.6 ± 1.418.6 ± 0.69.2 ± 1.058.3 ± 3.9
40014.2 ± 1.320.6 ± 1.614.6 ± 1.546.8 ± 5.7

Values are means ± approximate 95% confidence intervals across five training seeds, evaluated on 100 matched held-out tasks with one evaluation seed.

At a delay of 100, clearing the memory raises expected regret from 0.075 to 0.354, while substituting another history of the same task changes it only slightly (to 0.081). Replacing the memory with that of a different task raises regret to 0.391 and increases the probability assigned to the donor task's optimal arm from 8.1% to 66.9%. This pattern is consistent across all tested delays.Though we cannot determine what exactly the model has memorized due to the "latent" nature of the long-term memory, these results strongly suggest that the model is indeed leveraging distant history to make better predictions.


Thank you again for your constructive suggestions. We hope these revisions and additional analyses address your concerns.

AuthorsOpenReview ↗

Response to Reviewer tUCv (Part 3)

References

Thank you for your detailed response, which has addressed most of my concerns. Therefore, I have decided to increase my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.