acceptodds
Withdrawn from ICLR 2027

Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) achieve strong performance when all task-relevant information is available upfront, as in static prediction and instruction-following problems. However, many real-world decision-making tasks are inherently online: crucial information must be acquired through interaction, feedback is delayed, and effective behavior requires balancing information collection and exploitation over time. While in-context learning enables adaptation without weight updates, existing LLMs often struggle to reliably leverage in-context interaction experience in such settings. In this work, we show that this limitation can be addressed through training. We introduce ORBIT, a multi-task, multi-episode meta–reinforcement learning framework that trains LLMs to learn from interaction in context. After meta-training, Qwen3-8B, a relatively small open-source model, demonstrates substantially improved in-context online learning on entirely unseen environments: it matches or slightly exceeds GPT-4o while outperforming other baselines by a large margin on average. Beyond performance, we find that ORBIT exhibits adaptive exploration, revising its actions after failure without explicit reflection prompts. Scaling experiments further reveal consistent gains with model size, suggesting significant headroom for learn-at-inference-time decision-making agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Meta-reinforcement learning, In-context online learning, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MP5FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose a multi-task, multi-episode meta-reinforcement learning framework. The LLM (e.g. Qwen3-8B) is trained on multiple environments (e.g. Minesweeper and Hangman) to optimize a (undiscounted) cross-episode return. The cross-episode return is simply accumulated binary reward obatained at the end of each trial. After training, the model undergoes evaluataion on unseen environments (e.g. ALFWorld, WebShop), which tests transferability of the general meta-learned skill, unlike prior work (e.g. LaMer) that trains and tests on the single task. The method is compared with single-episode standard RL training, prompt-based methods and prior meta-RL method.

Critical strengths
  • While Meta-RL has been shown to be an effective for language agents, training on multiple environments together and evaluate on unseen environments seems underexplored.
Critical weaknesses
  • Limited novelty: The method is very similar to prior work LaMer, with the main difference being the prompt. For example, there is no reflection prompt. However, I think this design choice is also not very well supported in the paper.

  • Weak ablation study: Given the similarity with prior work, the attribution of gains over LaMer (which is significant as shown in table 2) is not clear. I think the key points might either be to include past full trajectories (including reasoning tokens), or the objective itself (on using an undiscounted cross-episode return or optimize total success count), though the latter is less likely. However, this crucial ablation study is not inlcuded in the paper.

Questions for authors
  • L68-69, the author mentioned multi-episode in-context online learning as their setting. Can the author clarify if there is any difference from prior work's multi-episode setting such as Reflexion?

  • Can the author provide some intuition on why explicit reflection prompts is not needed? Since the model is trained on multiple tasks together, and evaluate on completely unseen environment, which is harder, I find it surprising that no reflection prompt is needed compared to LaMer.

  • Why there is no comparison with LaMer in Table 3?

  • Compare to LaMer, the success count over all T episodes is optimized. Why is that useful to optimize the total successful episode?

  • Is the standard RL baseline also multi-task?

  • LaMer uses non-thinking mode for efficiency as mentioned in their paper. Is the LaMer in Table 2 using non-thinking mode too but Orbit using thinking mode? If this is the case, is there a capability difference between the base models as well as on the training memory and compute footprint?

Reviewer_12SROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ORBIT, a cross-episode meta-RL framework that trains LLMs to adapt to unfamiliar tasks through in-context interaction. ORBIT preserves interaction histories across episodes and optimizes cumulative task success using GRPO. Experiments on five unseen task families demonstrate improved in-context adaptation, particularly on Maze and Mastermind, with additional analyses of exploration behavior and model scaling.

Critical strengths
  1. Evaluation on unseen task families. Unlike prior meta-RL approaches that primarily evaluate within the same environment family, ORBIT evaluates on disjoint task families, providing stronger evidence of transferable in-context adaptation.

  2. Promising empirical improvements. ORBIT substantially outperforms single-episode RL on Maze and Mastermind in Episode 3. The exploration analysis further suggests improved adaptation following unsuccessful attempts.

Critical weaknesses
  1. Limited methodological novelty and insufficient isolation of contributions. ORBIT shares substantial methodological similarities with LaMer and the concurrent work MAGE. Its main design choices, including cross-episode history preservation, cumulative success rewards, and GRPO, are relatively straightforward. The experiments do not sufficiently isolate which components drive the improvements over prior methods.

  2. Insufficient evidence for general-purpose in-context online learning. The evaluation is limited to three short episodes, with the largest gains concentrated on Maze and Mastermind. Improvements over the strongest baselines are modest on ALFWorld and ScienceWorld, and ORBIT does not outperform the strongest baseline on WebShop. Moreover, the experiments do not isolate the contribution of task-specific interaction histories from other learned behavioral changes.

Questions for authors
  1. Could the authors provide controlled comparisons that independently vary context preservation, reward design, and episode termination while keeping the optimization algorithm and training budget fixed? This would clarify the methodological contribution beyond LaMer and related approaches.

  2. What happens if previous-episode histories are removed or replaced with histories from different task instances? Such interventions would help establish whether the gains genuinely depend on relevant accumulated experience.

  3. The success-count objective assigns identical rewards to success patterns (1,0,0) and (0,0,1), with the same advantage applied to all tokens in a trajectory. How do the authors establish that this objective promotes adaptive exploration and learning from failure, rather than merely maximizing cumulative success?

  4. Can the authors provide evidence beyond three episodes or evaluate on more challenging long-horizon interactive tasks? How does performance change as the number of episodes increases, particularly given the growing context length?

  5. Appendix G does not report variance across independent training seeds. Could the authors provide training-seed variability and clarify the claim that Episode-1 performance remains comparable to the base model, given the noticeable difference on ScienceWorld?

Reviewer_HCDjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies improving in-context learning ability of LLM agents across repeated attempts at unfamiliar tasks. ORBIT retains the full interaction history across three episodes, optimizes the total number of successful episodes using GRPO, and trains on five partially observable game environments. Evaluation uses five task families excluded from post-training. With Qwen3-8B, ORBIT shows the clearest gains over single-episode RL on Maze and Mastermind, with smaller or comparable gains on the other environments. The paper also examines exploration behavior, model-size scaling, and a second model family. Its main contribution is empirical evidence of cross-family transfer under a simple meta-RL setup.

Critical strengths
  1. Transfer to unseen task families. Training and evaluation use different environment families, providing a more demanding test than holding out instances within a benchmark. The later-episode gains on Maze and Mastermind offer useful evidence that post-training can improve feedback-based adaptation.

  2. Evidence across model sizes and families. Across Qwen3-4B, 8B, and 14B, the strongest benefits of increasing model size appear in later episodes on Maze and Mastermind, while first-episode performance changes relatively little. This pattern is consistent with larger models making better use of accumulated interaction experience. The additional improvements over prompting and single-episode RL baselines with GLM-4-9B extend the evidence beyond the main Qwen3-8B setting, providing initial support for the applicability of the training approach across model capacities and families.

Critical weaknesses
  1. Insufficient isolation of the meta-RL contribution. ORBIT and the single-episode RL baseline differ in both reward horizon and training context, leaving unclear whether the gains require cross-episode rewards or mainly arise from training on feedback histories. Similar first-episode performance does not resolve this distinction. A baseline using the same multi-episode context but episode-local rewards, together with test-time history interventions, is needed. Table 3's failure-conditioned state coverage also lacks uncertainty estimates and does not directly establish more useful exploration.

  2. Unclear practical and comparative advantage. ORBIT is evaluated through repeated resets of the same hidden instance, leaving its applicability to workflows without exact resets unclear. Token usage and latency are also unreported, so its practical advantage over experience-based prompting remains uncertain. Comparisons with related training methods are limited: LaMer's reproduction is unstable, while MAGE, PAPRIKA, and Iterative RMFT are discussed but not evaluated in the same setting. The connection to MRT's regret-based meta-RL formulation [1] is also missing. A concrete application, cost-matched comparisons with suitable prior methods, and clearer distinctions in training objectives would better establish ORBIT's added value.

[1] Qu et al., Optimizing test-time compute via meta reinforcement finetuning, ICML 2025.

Questions for authors
  1. Contribution of cross-episode rewards. Could you compare with training on the same multi-episode histories using episode-local rewards, and test whether removing or replacing prior histories eliminates the gains?

  2. Comparison with related methods. Could you supplement the experiments with selected related methods, such as MAGE, PAPRIKA, or Iterative RMFT, using the same task split and budget?

  3. Exploration and variability. Could you report Table 3's sample sizes and uncertainty on a common failed-instance set, distinguish adaptation after failure from reuse after success, and provide main-result means and error bars across multiple runs?

  4. Practical deployment setting. Which application best matches the fixed-instance reset protocol, and how would the method handle workflows where actions change the state and exact resets are unavailable? Could you report the performance–cost trade-off against experience-based prompting under matched interaction and inference budgets?

  5. Cost analysis. ICRL requires collecting multiple trajectories as context, resulting in long training contexts. Could you provide a cost analysis covering trajectory collection, training compute, and inference token usage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.