- Rating
- 4(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes TEMPA, a training-free framework that helps frozen LLM agents improve through repeated interaction. It stores action sequences between reward events as reusable memory segments and uses empirical continuation values to select them. A controller decides when to execute remembered behavior and when to explore using the LLM. The method is evaluated on Jericho and ScienceWorld.
Critical strengths
-
The approach is intuitive and practically relevant. Storing and replaying successful multi-step behavior reduces repeated LLM inference and avoids estimating separate values for individual actions.
-
The empirical results are strong across both benchmarks. TEMPA outperforms several training-free baselines while consuming fewer tokens. The ablations also provide useful evidence for the contributions of reward-based segmentation, continuation values, adaptive routing, and persistent execution.
Critical weaknesses
-
The effectiveness of reward-based segmentation under sparse rewards is not fully established. TEMPA uses intermediate reward events to create shorter, reusable segments. When rewards become very sparse, segments may span most of an episode, potentially reducing the benefits of temporal abstraction. Although the method is motivated by sparse and delayed rewards, the experiments do not directly examine its sensitivity to reward frequency.
-
The evaluation focuses on repeated interaction with a fixed ScienceWorld task variation. This setting is appropriate for studying adaptation, but it also favors the reuse of stored action sequences. It remains unclear how useful these segments are when the task variation changes between episodes.
Minor weakness. The related-work discussion omits relevant memory-based approaches such as MIRA and AdaMEM. MIRA uses trajectory memory to guide RL training with fewer LLM queries, while AdaMEM uses trajectory and strategy memory for test-time adaptation. Discussing how TEMPA differs from these methods would help clarify its novelty.
References:
Nourzad, N., & Joe-Wong, C. (2026). MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidance. ICLR 2026.
Zhang, Y., Li, Y., Payani, A., & Wang, L. (2026). AdaMEM: Test-Time Adaptive Memory for Language Agents. ICML 2026.
Questions for authors
-
How sensitive is TEMPA to reward sparsity? In particular, how do its performance gains and token savings change as intermediate reward events become less frequent?
-
How does TEMPA perform when ScienceWorld task variations change between episodes while accumulated memory is retained?
Hi Reviewer JYKe,
Could you take a look at authors’ responses and let them know if your concerns are addressed? Thanks!
Best regards, AC