- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper propose a multi-task, multi-episode meta-reinforcement learning framework. The LLM (e.g. Qwen3-8B) is trained on multiple environments (e.g. Minesweeper and Hangman) to optimize a (undiscounted) cross-episode return. The cross-episode return is simply accumulated binary reward obatained at the end of each trial. After training, the model undergoes evaluataion on unseen environments (e.g. ALFWorld, WebShop), which tests transferability of the general meta-learned skill, unlike prior work (e.g. LaMer) that trains and tests on the single task. The method is compared with single-episode standard RL training, prompt-based methods and prior meta-RL method.
Critical strengths
- While Meta-RL has been shown to be an effective for language agents, training on multiple environments together and evaluate on unseen environments seems underexplored.
Critical weaknesses
-
Limited novelty: The method is very similar to prior work LaMer, with the main difference being the prompt. For example, there is no reflection prompt. However, I think this design choice is also not very well supported in the paper.
-
Weak ablation study: Given the similarity with prior work, the attribution of gains over LaMer (which is significant as shown in table 2) is not clear. I think the key points might either be to include past full trajectories (including reasoning tokens), or the objective itself (on using an undiscounted cross-episode return or optimize total success count), though the latter is less likely. However, this crucial ablation study is not inlcuded in the paper.
Questions for authors
-
L68-69, the author mentioned multi-episode in-context online learning as their setting. Can the author clarify if there is any difference from prior work's multi-episode setting such as Reflexion?
-
Can the author provide some intuition on why explicit reflection prompts is not needed? Since the model is trained on multiple tasks together, and evaluate on completely unseen environment, which is harder, I find it surprising that no reflection prompt is needed compared to LaMer.
-
Why there is no comparison with LaMer in Table 3?
-
Compare to LaMer, the success count over all T episodes is optimized. Why is that useful to optimize the total successful episode?
-
Is the standard RL baseline also multi-task?
-
LaMer uses non-thinking mode for efficiency as mentioned in their paper. Is the LaMer in Table 2 using non-thinking mode too but Orbit using thinking mode? If this is the case, is there a capability difference between the base models as well as on the training memory and compute footprint?