acceptodds
Under review as a conference paper at ICLR 2027

TEMPA: Temporally Extended Memory for Policy Adaptation

Abstract

Large language model (LLM) agents must learn from online experience to improve their behavior, even though their parameters remain frozen after deployment. Reusing complete episode trajectories as prompt context can incur additional token costs, while estimating individual action values from experience is challenging under sparse and delayed rewards. These challenges motivate policy adaptation through planning over action segments at an intermediate temporal granularity. We introduce TEMPA (Temporally Extended Memory for Policy Adaptation), a gradient-free framework that adapts an agent's policy over actor actions and executable memory segments. It partitions interaction at reward events and stores the resulting reward-to-reward segments as executable options. These segments are selected using empirical advantages. A router uses evidence from exploratory trials to choose between executing the selected segment and invoking the LLM actor. TEMPA reuses multi-step behaviors without separate credit estimates for constituent actions and reduces repeated actor inference during memory execution.We formulate memory exploitation as finite-horizon semi-Markov planning and establish a conditional sublinear bound on exploration loss relative to the incumbent policy. Experiments on Jericho and ScienceWorld show that TEMPA achieves higher average scores and stronger late-episode performance. Its total inference token consumption is roughly 56% lower than the mean across the seven baselines. Ablations support the contributions of reward-based segmentation, continuation values, adaptive routing, and persistent execution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Test-Time Adaptation, Gradient-Free Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_wJXdOpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TEMPA, a training-free framework that helps frozen LLM agents improve through repeated interaction. It stores action sequences between reward events as reusable memory segments and uses empirical continuation values to select them. A controller decides when to execute remembered behavior and when to explore using the LLM. The method is evaluated on Jericho and ScienceWorld.

Critical strengths
  • The approach is intuitive and practically relevant. Storing and replaying successful multi-step behavior reduces repeated LLM inference and avoids estimating separate values for individual actions.

  • The empirical results are strong across both benchmarks. TEMPA outperforms several training-free baselines while consuming fewer tokens. The ablations also provide useful evidence for the contributions of reward-based segmentation, continuation values, adaptive routing, and persistent execution.

Critical weaknesses
  • The effectiveness of reward-based segmentation under sparse rewards is not fully established. TEMPA uses intermediate reward events to create shorter, reusable segments. When rewards become very sparse, segments may span most of an episode, potentially reducing the benefits of temporal abstraction. Although the method is motivated by sparse and delayed rewards, the experiments do not directly examine its sensitivity to reward frequency.

  • The evaluation focuses on repeated interaction with a fixed ScienceWorld task variation. This setting is appropriate for studying adaptation, but it also favors the reuse of stored action sequences. It remains unclear how useful these segments are when the task variation changes between episodes.


Minor weakness. The related-work discussion omits relevant memory-based approaches such as MIRA and AdaMEM. MIRA uses trajectory memory to guide RL training with fewer LLM queries, while AdaMEM uses trajectory and strategy memory for test-time adaptation. Discussing how TEMPA differs from these methods would help clarify its novelty.

References:

Nourzad, N., & Joe-Wong, C. (2026). MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidance. ICLR 2026.

Zhang, Y., Li, Y., Payani, A., & Wang, L. (2026). AdaMEM: Test-Time Adaptive Memory for Language Agents. ICML 2026.

Questions for authors
  1. How sensitive is TEMPA to reward sparsity? In particular, how do its performance gains and token savings change as intermediate reward events become less frequent?

  2. How does TEMPA perform when ScienceWorld task variations change between episodes while accumulated memory is retained?

Reviewer_xDRPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes TEMPA, a training-free framework that adapts frozen LLM agents by storing and reusing executable action segments bounded by reward events. It selects segments using their observed returns and estimated continuation values, while an adaptive router allocates control between memory execution and LLM exploration. Persistent segment execution reduces repeated model inference. Experiments on Jericho and ScienceWorld report improved performance across repeated episodes and reduced token consumption. Ablations examine reward-based segmentation, continuation planning, adaptive routing, and persistent execution, complemented by a conditional analysis of the local router’s exploration cost.

Critical strengths
  • Storing experience as executable action segments allows the agent to reuse successful behavior directly, improving performance while reducing repeated LLM inference.

  • Comparisons across two environments and ablations of segmentation, continuation values, routing, and persistent execution provide evidence that the components contribute to adaptation across repeated episodes.

Critical weaknesses
  1. Clarity of presentation. I found the method difficult to follow. The extensive mathematical formalism used to describe this training-free approach obscures the underlying procedure. The framework diagrams provide limited concrete information; a worked example using an actual trajectory would make the method easier to understand.

  2. Limited evaluation across models. The experiments primarily use Qwen3.6-27B. I would like to see whether the method also works with proprietary models, which would provide stronger evidence of its applicability across different model families.

Questions for authors
  1. How are segments constructed in practice? Is the segmentation rule directly applicable across environments, or does it require dataset-specific design or adaptation?

  2. How does TEMPA differ from related approaches such as AgentKB [1] and ReasoningBank [2]? Why were these methods not included in the experimental comparisons?

  3. How does TEMPA perform with other model families, such as GPT, Claude, and DeepSeek? If the method is not applicable to these models, please explain the limitations.

[1] Tang, X., Qin, T., Peng, T., Zhou, Z., Shao, D., Du, T., ... & Zhou, W. (2025). Agent kb: Leveraging cross-domain experience for agentic problem solving. arXiv preprint arXiv:2507.06229.

[2] Ouyang, S., Yan, J., Hsu, I., Chen, Y., Jiang, K., Wang, Z., ... & Pfister, T. (2026, April). Reasoningbank: Scaling agent self-evolving with reasoning memory. In International Conference on Learning Representations (Vol. 2026, pp. 94327-94354).

Reviewer_pNnLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TEMPA, a gradient-free framework for adapting frozen LLM agents through executable memory segments. It partitions trajectories at reward events and evaluate segments using empirical continuation values, and adaptively switches between memory execution and LLM exploration. Experiments demonstrate consistent performance improvements and reduced inference token consumption.

Critical strengths
  1. TEMPA integrates executable memory, segment-level planning, and adaptive exploration into a unified framework without requiring model post-training.

  2. The method achieves consistent improvements across two benchmarks and two LLM backbones. The ablations provide useful evidence for reward-based segmentation, continuation planning, adaptive routing, and persistent execution.

  3. Reusing executable action sequences avoids repeated LLM generation, providing meaningful token savings alongside improved task performance.

Critical weaknesses
  1. The main components are closely related to established temporal abstraction and executable memory methods. The contribution appears to lie primarily in their integration.

  2. The evaluation primarily measures repeated interactions within the same tasks, with memory retrieval relying on exact observable-state matching. It remains unclear how well the approach transfers across unseen task variations or changes in environment dynamics.

  3. The analysis focuses on aggregate LLM tokens without reporting external memory growth, graph-planning overhead, end-to-end latency, or prefix/KV-cache reuse. These factors may affect the practical efficiency gains.

Questions for authors
  1. How frequently do states match previously stored segments across episodes, and what is the empirical replay success rate? Could the authors evaluate transfer to unseen task variations?

  2. Could the authors report latency and memory overhead information, including the effects of prefix caching?

  3. How robust is reward-boundary segmentation under sparse or delayed rewards? Would alternative progress-based segmentation criteria be useful?

Reviewer_JYKeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a new framework named TEMPA to adapt the agent's policy over actual actions and executable memory segments.

The targeted problem is that when training an LLM agent, it can be very hard to estimate the value of a given action when rewards are sparse. Furthermore, when certain options or experiences have already been visited by the agent, they should be reused to save computation costs.

The core intuition of the paper is to store experiences into memory and use temporal segments to perform empirical advantage estimation. This advantage estimation is not limited to a single action, but extends to executable options as well, making the estimation both more efficient and more accurate.

Critical strengths

The targeted policy adaptation problems make sense, and the underlying idea to reuse multi-policy behaviors without a separate credit estimation is also reasonable. Besides, this paper also provides some theoretical analysis which strengthens the foundation of the proposed methods.

Critical weaknesses

I think the major improvement of this paper is that it uses a more nuanced and advanced way to estimate the advantage of some segments, and this kind of estimation helps solve the sparse reward problem.

However, this estimation can be extended beyond pure text-based methods; the idea behind it seems useful for general RL methods. Because the focus of this paper is gradient-free, it makes the paper's improvement a bit narrow, feeling like incremental work on top of previous gradient-free RL methods.

The experimental results should also be strengthened a lot:

  1. The tested domain is limited to games. Extension to some real-world problems like search would be very helpful.
  2. For games like ScienceWorld, we should see some variance, because ScienceWorld is largely influenced by different random seed results.
Questions for authors

Can this method be used for gradient-based RL methods? Or can the idea be used for some non-RL methods?

What's the computation cost compared with JitRL?

1 comment

Hi Reviewer JYKe,

Could you take a look at authors’ responses and let them know if your concerns are addressed? Thanks!

Best regards, AC

Comments on the paper

2 comments

Thank you for your detailed responses and the additional experiments. I will increase my score.

Thank you for your positive reassessment. We will incorporate the worked example and ReasoningBank results into the revised manuscript and expand the discussion of Agent kb and ReasoningBank. We appreciate your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.