acceptodds
Under review as a conference paper at ICLR 2027

Fast and Slow Online Learning for Agents

Abstract

Large language model (LLM) agents are typically deployed as static models, yet their deployment conditions can differ from training and continue to evolve over time, requiring agents to continually adapt after deployment. Unlike offline post-training, however, every intermediate policy during deployment serves subsequent queries, making test-time learning essentially an online learning problem, where the learner is evaluated by its cumulative performance over the interaction stream rather than only its final performance. This cumulative objective requires both fast adaptation and strong long-run improvement. We therefore introduce Fast and Slow Online Learning (FSOL), a framework that combines memory for fast adaptation with RL for progressively internalizing experience into model weights. We evaluate FSOL on writing-style adaptation, search-augmented QA, WebShop, and ALFWorld across stationary streams, distribution shifts, mixed tasks, and sequential tasks. Across these settings, FSOL consistently achieves higher cumulative performance than memory or RL alone, rapidly adapts to distribution shifts, effectively learns from heterogeneous interleaved tasks, and retains previously acquired capabilities while adapting to new ones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, LLM agents, online learning, reinforcement learning, memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eVJCOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Fast and Slow Online Learning (FSOL), a framework for agents that formulates test-time learning as an online learning problem. The paper first proposes the online learning paradigm by evaluates agents by cumulative performance over the interaction. Specifically, for each incoming query, the agent retrieves successful past experiences, generates one trajectory, receives binary feedback, and saves successful experience to memory. After a batch of queries, it updates the model using PPO or REINFORCE++ with the memory-augmented policy. Empirically, the paper evaluates FSOL on four benchmarks across four online settings: stationary streams, distribution shifts, sequential tasks, and mixed-task streams, covering deployment conditions that the authors describe as largely unexplored in prior work.

Critical strengths
  1. The online learning framework of test-time learning and the proposed method is clear and sensible.
  2. The paper evaluates the FSOL framework in multiple benchmarks and stationary and non-stationary settings and show promising improvements compared to the baselines.
Critical weaknesses
  1. I think the main issue is that the positioning relative to prior work is a bit overstated. StreamBench [1] already studies online adaptation through cumulative performance over an interaction stream as the evaluation setting, while prior work [2] explores combining fast textual adaptation with slow parameter updates. The paper should distinguish this contribution more clearly from the existing test-time learning paradigm.
  2. For presentation, the paper should expand and explain clearly what the MemoryUpdate and RL Update is (the concrete formulation), e.g., how the memory update uses the environment feedback.
Questions for authors
  1. This might be minor but in the appendix, why does the LLM judge for writing only see the response?
  2. Why using different hyperparameters, batch size especially, for the RL (PPO, Reinforce ++ with FSOL, PPO with FSOL)? In this setting, the PPO will get less gradient updates than FSOL based RL training.
  3. Any intuition why raw memory is better than AWM?
  4. It might be interesting to see the same FSOL checkpoint without memory to support that memory accelerates learning in the model’s parameters.
4 comments
AuthorsOpenReview ↗

Reply to Weaknesses 1 and 2

Thank you so much for recognizing (1) the clarity and sound motivation of our online learning framework and FSOL design, and (2) the promising improvements across multiple benchmarks and both stationary and non-stationary settings. Now we address each weakness and question below.

Weakness 1

I think the main issue is that the positioning relative to prior work is a bit overstated. StreamBench [1] already studies online adaptation through cumulative performance over an interaction stream as the evaluation setting, while prior work [2] explores combining fast textual adaptation with slow parameter updates. The paper should distinguish this contribution more clearly from the existing test-time learning paradigm.

Thank you for the comment. We noticed that reference [2] is missing. Could you please clarify which work you are referring to?

We acknowledge that online learning and cumulative performance have been studied previously. Our contribution is to address the trade-off between fast but limited memory adaptation and slow but more capable RL, leveraging their complementary time scales to improve cumulative performance.

Importantly, existing memory + RL methods are difficult to apply to strict online learning for two reasons:

  • Offline preparation: Many require pre-collected memory, SFT warm-start, or trained memory curators, whereas FSOL learns entirely online without prior knowledge of future task distributions, which matches the real-world deployment, where future tasks cannot be foreseen.
  • Multiple rollouts: Many require multiple rollouts per query to estimate memory utility or optimize learning objectives. However, in real-world deployment, agents cannot repeatedly execute the same task (e.g., making online purchases), nor can users be expected to evaluate multiple responses solely for training. FSOL instead uses PPO or REINFORCE++ to learn from a single rollout per query.

We summarize the limitations of existing memory-augmented RL methods under online learning in the following table.

MethodLimitations for Online Learning
SkillRLRequires pre-collected memory and an SFT warm-start before RL training.
EvolveRRequires an SFT warm-start before RL training.
D2Skill and RetroAgentRequire multiple rollouts of the same query, both with and without memory.
Skill-R1Requires pre-collected memory and multiple rollouts to compute its inter-generation objective.
SkillOSRequires offline training of the skill generator before deployment.

We will clarify our novelty in the updated manuscript.

Weakness 2

For presentation, the paper should expand and explain clearly what the MemoryUpdate and RL Update is (the concrete formulation), e.g., how the memory update uses the environment feedback.

Thanks for the suggestions. We will include them for different RL algorithms and different memory methods in the Appendix A of the updated manuscript.

AuthorsOpenReview ↗

Reply to Questions 1-4

Question 1

This might be minor but in the appendix, why does the LLM judge for writing only see the response?

Thank you for pointing this out. Our original judge primarily focuses on evaluating whether the generated response follows the target writing style, which can largely be assessed from the response itself. We agree that including the original user query would allow a more comprehensive evaluation. We will add a prompt-aware evaluation and report the results later, if time permits.

Question 2

Why using different hyperparameters, batch size especially, for the RL (PPO, Reinforce ++ with FSOL, PPO with FSOL)? In this setting, the PPO will get less gradient updates than FSOL based RL training.

We did not conduct an extensive hyperparameter search or tune hyperparameters separately for each benchmark. We started with a default configuration (lr=1e-6, KL coefficient=0.01, batch size=16) and made a few adjustments on WebShop to make training stable (i.e., avoid accuracy crash). For REINFORCE++, FSOL shows accuracy crash with KL=0.01, so we increased it to 0.05. We also tested KL=0.05 for REINFORCE++, but it caused accuracy crash, so we retained KL=0.01 for REINFORCE++ only. For PPO, we increased the batch size from 16 to 32 for FSOL and to 64 for PPO-only to avoid accuracy collapse. We additionally tested both PPO and FSOL with batch size 64, and FSOL still outperformed PPO. See the table below. ALFWorld is still running and we will report the final result later.

MethodWebShopALFWorldSearchWriting
PPO (bs=64)47.740.635.849.7
FSOL (bs=64)55.948.7 (on-going, 60% finished)37.395.3
FSOL (bs=32)56.350.937.596.3

Importantly, these configurations were obtained on WebShop, and then worked well on other benchmarks. Thus, our hyperparameter choices were primarily motivated by training stability and not individually tuned. The results suggest robustness across different environments.

Question 3

Any intuition why raw memory is better than AWM?

Our hypothesis is that smaller models may struggle with extracting generalizable workflows from trajectories or/and translating abstract workflows into concrete actions. In contrast, raw trajectories provide explicit, step-by-step examples that are easier to follow and adapt to the current task. This may explain why raw memory outperforms AWM in our experiments.

Question 4

It might be interesting to see the same FSOL checkpoint without memory to support that memory accelerates learning in the model’s parameters.

Our intuition for the reason why memory can accelerate RL is that it transforms learning the task itself into learning adapting the solution of similar tasks to the current query. We will show the evaluation results of FSOL checkpoints without memory if time permits. However, FSOL is designed to learn a memory-conditioned policy, where retrieval and parametric learning jointly contribute to performance. Evaluating the policy without memory introduces a train-test mismatch, and therefore may not necessarily reflect the effectiveness of the learned policy.

Sorry for missing the reference and [2] should refer to: Tiwari, Rishabh, et al. "Learning, fast and slow: Towards LLMs that adapt continually." arXiv preprint arXiv:2605.12484 (2026).

Thank you for the reply. After reading the rebuttal, I think I still have some concerns on the paper regarding:

  1. I think the overall contribution is somewhat limited by adapting the framework to single-rollout agentic setting. Given the single-rollout agentic setting has also been established by prior work, e.g., [1], I feel like the novelty is limited.
  2. what do you mean by "made a few adjustments on WebShop to make training stable"?
  3. Also, I feel like there's some uncertainty of the baseline results, e.g., raw memory & AWM comparisons. Please double check the implementation.

Given these concerns, I lean toward a weak rejection for now.

[1] Hou, Zhenyu, et al. "Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning." arXiv preprint arXiv:2607.07508 (2026).

AuthorsOpenReview ↗

Reply to followup questions

Thank you for the follow-up comments. We address each point below.

Sorry for missing the reference and [2] should refer to: Tiwari, Rishabh, et al. "Learning, fast and slow: Towards LLMs that adapt continually." arXiv preprint arXiv:2605.12484 (2026).

Thank you for clarifying the reference. We agree that this work shares a similar high-level motivation of combining fast and slow adaptation. However, the key distinction lies in the learning setting and objective.

First, their implementation relies on GEPA for context optimization, which uses validation sets for candidate selection and is not directly applicable to online setting, where each interaction permits only one rollout and future queries are unknown.

Second, FSOL explicitly targets cumulative performance over the interaction stream, rather than primarily improving model adaptation through joint context and parameter optimization. This objective motivates our design of using memory for immediate improvement and RL for higher long-term performance. Our experiments further demonstrate that this principle generalizes across different memory and RL implementations.

I think the overall contribution is somewhat limited by adapting the framework to single-rollout agentic setting. Given the single-rollout agentic setting has also been established by prior work, e.g., [1], I feel like the novelty is limited.

[1] Hou, Zhenyu, et al. "Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning." arXiv preprint arXiv:2607.07508 (2026).

SAO is concurrent with our work (released in July 2026), and we have already cited it in our introduction to motivate the research gap.

Our contribution is not the single-rollout setting itself, but addressing the trade-off between learning speed and performance ceiling under the cumulative learning objective. Existing memory-based methods adapt quickly but have limited performance ceilings, while RL-based methods such as SAO can achieve stronger performance but require many interactions to improve. Thus, SAO is exactly one side of the trade-off we want to address: learning speed versus ceiling performance.

FSOL combines these complementary mechanisms to achieve both fast adaptation and strong long-term performance, thereby improving cumulative reward.

what do you mean by "made a few adjustments on WebShop to make training stable"?

We did not independently tune hyperparameters to maximize each method's performance. The adjustments were made to avoid training collapse.

Specifically, we started with the default configuration (learning rate = 1e-6, KL coefficient = 0.01, batch size = 16):

  • REINFORCE++: Memory-augmented REINFORCE++ suffered from accuracy collapse at KL = 0.01, so we increased it to 0.05. We also tested REINFORCE++ without memory at KL = 0.05, which exhibited collapse. So we keep KL = 0.01 for REINFORCE++ alone
  • PPO: PPO with memory required batch size 32 for stable training, while PPO without memory required batch size 64. We additionally compared both methods at batch size 64 to control for this difference, as reported in our previous response. FSOL still outperformed PPO-only under matched batch sizes, supporting the effectiveness of memory augmentation beyond differences in hyperparameter settings.

Also, I feel like there's some uncertainty of the baseline results, e.g., raw memory & AWM comparisons. Please double check the implementation.

We have carefully rechecked our implementations and found no bugs.

Regarding the uncertainty of memory baselines, we have updated the SFT baselines and SFT + raw memory with 4 random seeds.

MethodsWebShopALFWorldSearchWriting
SFT21.5 ± 5.033.2 ± 2.827.2 ± 1.279.2 ± 1.2
FSOL (SFT + raw memory)58.4 ± 3.940.5 ± 1.635.8± 0.795.7 ± 1.2

As for memory baselines, we acknowledge that the reported results are based on a single run. Nevertheless, FSOL consistently outperforms multiple memory-only baselines by substantial margins across benchmarks, suggesting that the improvement is unlikely to be explained by randomness.

More importantly, FSOL is a general framework that can incorporate different memory and RL methods. Its core contribution is demonstrating how fast non-parametric adaptation and slow parametric learning complement each other to improve cumulative performance, rather than relying on the superiority of any particular memory method.

Reviewer_Yr2iOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the scenario in which agents continually adapt after receiving sequence feedback. The framework combines memory as fast feedback and experience for updating model weights as slow feedback to propose FSOL. Finally, FSOL beats only memory or only RL and perform higher cumulative performance.

Critical strengths

(1) The scenario matches the deployment environment. The design of the online environment is practical, i.e., a distribution shift.

(2) Transparent for the case studies of PPO+AWM on WebShop. Great discussion and report on the complementarity of the mechanisms (memory and RL).

Critical weaknesses

(1) The novelty is limited. For architecture, StreamBench discusses evaluating the agent in a streaming feedback environment; MemRL reports a cumulative success rate; Evo-Memory studies streaming task evaluation by test-time learning. For components, memory and RL are well-known and popular. The novelty of this paper is its focus on LLM agents in a single-rollout online setting. In my opinion, the contribution is limited.

(2) The baselines are weak. The chosen baselines are general methods, i.e., memory and PPO. Given that the communities of memory, RL, and continual learning are very popular, I think reporting the SOTA methods will help the paper demonstrate the empirical contribution (see Question 1).

Questions for authors

(1) The experiment doesn't involve SOTA memory, RL, and continual learning methods. Can the authors compare with training-free memory methods, e.g., ReasoningBank (ICLR 2026), ACE (ICLR 2026), Dynamic Cheatsheet (EACL 2026), and ExpeL (AAAI 2024)? For the slow part, can the authors compare to sampling SFT or replay?

(2) How are hyperparameters (KL coefficient, batch size, retrieval threshold, memory window, reward normalization) selected? Are they tuned on the same streams used to report cumulative accuracy? If so, how is this different from tuning on the test set?

(3) Why do the distribution shift, mixed task stream, and sequential task stream experiments apply different combinations of memory and RL? Please specify the configuration in Figures 3, 4, and 5 and report the other combinations.

(4) Can the authors verify that the AWM implementation is functioning, so the comparison does not rest on a degraded baseline?

5 comments
AuthorsOpenReview ↗

Reply to Weakness 1

Thank you so much for recognizing (1) the practical relevance of our online learning setting, including distribution shifts, and (2) our analysis of the complementarity between memory and RL, including transparent reporting and discussion of cases where their combination does not improve performance. We now address each weakness and question below.

Weakness 1

The novelty is limited. For architecture, StreamBench discusses evaluating the agent in a streaming feedback environment; MemRL reports a cumulative success rate; Evo-Memory studies streaming task evaluation by test-time learning. For components, memory and RL are well-known and popular. The novelty of this paper is its focus on LLM agents in a single-rollout online setting. In my opinion, the contribution is limited.

We acknowledge that online settings and cumulative performance have been explored in prior memory works. The novelty of our work is to solve the trade-off between current methods of online learning: memory adapts fast but has limited capability, while RL can have high ceiling performance, but learn slowly. Our method FSOL utilizes the complementary time scales of memory and RL, to achieve good cumulative performance.

More importantly, previous methods of memory + RL can hardly be applied to online setting.

  • Prior works need to pre-collected memory from larger teacher models, perform SFT for warm start, or train memory curators before RL, while our work assumes all memory generation and learning happens online during deployment. For online learning, future tasks cannot be foreseen and may come from changing task distributions. Therefore, an online learner does not know task distribution and should not rely on task-specific offline preparation.
  • Prior works require generating multiple rollouts per query, while generating multiple rollouts per query is not feasible in online learning. An agent cannot repeatedly execute real-world tasks (e.g., making online purchases) or ask users to evaluate multiple responses solely for training. Several existing methods require multiple rollouts of the same query to estimate memory utility or optimize memory-related objectives. In our method FSOL, we use PPO or REINFORCE++ so that the advantage estimation can be done with single rollout per query.

We summarize the limitations of existing memory-augmented RL methods under online learning in the following table.

MethodLimitations for Online Learning
SkillRLRequires pre-collected memory and an SFT warm-start before RL training.
EvolveRRequires an SFT warm-start before RL training.
D2Skill and RetroAgentRequire multiple rollouts of the same query, both with and without memory.
Skill-R1Requires pre-collected memory and multiple rollouts to compute its inter-generation objective.
SkillOSRequires offline training of the skill generator before deployment.

We will clarify our novelty in the updated manuscript.

AuthorsOpenReview ↗

Reply to Weakness 2 and Question 1

Weakness 2 and Question 1

The baselines are weak. The chosen baselines are general methods, i.e., memory and PPO. Given that the communities of memory, RL, and continual learning are very popular, I think reporting the SOTA methods will help the paper demonstrate the empirical contribution (see Question 1).

The experiment doesn't involve SOTA memory, RL, and continual learning methods. Can the authors compare with training-free memory methods, e.g., ReasoningBank (ICLR 2026), ACE (ICLR 2026), Dynamic Cheatsheet (EACL 2026), and ExpeL (AAAI 2024)? For the slow part, can the authors compare to sampling SFT or replay?

Could you clarify what you mean by replay-based RL in this context? We assume you are referring to RL methods that reuse past trajectories for policy updates, potentially with importance sampling corrections, as in [1]. We are currently exploring this approach and will report the experimental results once they are available.

We have included additional baselines in the table below. Among the memory-based methods, ExpeL and action sequence consistently rank as the top two. Notably, ExpeL combines summarized rules with raw trajectories, suggesting that trajectories play an important role in its effectiveness. Nevertheless, FSOL continues to achieve the best performance across the evaluated benchmarks. We will report the Writing results once they are available.

MethodsWebShopALFWorldSearch
ACE0.20.714.9
AWM12.222.126.8
ReasoningBank0.211.323.0
ExpeL (modified to online version)34.934.037.8
Raw memory35.633.233.1
Zero-shot4.013.015.0
FSOL (PPO + AWM)46.748.837.8
FSOL (PPO + Raw)56.350.937.5
FSOL (REINFORCE++ + Raw)52.552.138.8

We have also added experiments with SFT (rejection sampling) and SFT + raw memory. The results show that SFT + raw memory consistently outperforms SFT alone across all four benchmarks. Moreover, we observe a similar fast–slow learning pattern, providing further evidence that the fast–slow decomposition extends beyond PPO and REINFORCE++. See the figure in Appendix D of the updated manuscript.

MethodsWebshopALFWorldSearchWriting
SFT21.533.224.077.1
FSOL (SFT + raw memory)58.440.537.794.7

[1] Ma et al. Freshness-aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning.

AuthorsOpenReview ↗

Reply to Question 2-4

Question 2

How are hyperparameters (KL coefficient, batch size, retrieval threshold, memory window, reward normalization) selected? Are they tuned on the same streams used to report cumulative accuracy? If so, how is this different from tuning on the test set?

We did not conduct an extensive hyperparameter search or tune hyperparameters separately for each benchmark. Instead, we started with a default configuration (learning rate = 1e-6, KL coefficient = 0.01, batch size = 16) and made only a few adjustments on WebShop to ensure training stability (i.e., to avoid accuracy collapse).

For REINFORCE++, FSOL show accuracy collapse with a KL coefficient of 0.01, so we increased it to 0.05. We also tested a KL coefficient of 0.05 for REINFORCE++ alone, but observed accuracy collapse, so we retained 0.01 for this baseline. For PPO, we increased the batch size from 16 to 32 for FSOL and to 64 for PPO-only to avoid accuracy collapse. To ensure a fair comparison, we additionally evaluated both PPO and FSOL (PPO + raw memory) with a batch size of 64, and FSOL continued to outperform PPO, as shown in the table below.

MethodWebShopALFWorldSearchWriting
PPO (bs=64)47.740.635.849.7
FSOL (bs=64)55.948.7 (on going, 60% finished)37.395.3
FSOL (bs=32)56.350.937.596.3

Importantly, these configurations were obtained on WebShop, and then worked well on other benchmarks. Thus, our hyperparameter choices were primarily motivated by training stability and not individually tuned. The results suggest robustness across different environments.

Question 3

Why do the distribution shift, mixed task stream, and sequential task stream experiments apply different combinations of memory and RL? Please specify the configuration in Figures 3, 4, and 5 and report the other combinations.

Yes we will report more combination for each setting later.

Question 4

Can the authors verify that the AWM implementation is functioning, so the comparison does not rest on a degraded baseline?

Yes we find a bug in AWM implementation. The original prompt for workflow extraction does not provide few-shot examples. The updated results of exact same prompt as the official code is in the the reply to Weakness 2. Thank you for pointing this out.

Reviewer_Y8i7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper frames deployment-time adaptation of LLM agents as an online learning problem, emphasizing cumulative accuracy across the interaction stream rather than final performance. It proposes Fast-Slow Online Learning (FSOL), which combines retrieval-based memory for immediate adaptation with reinforcement learning for gradual policy improvement.

FSOL integrates two complementary mechanisms: fast adaptation, which retrieves successful past trajectories or workflows as prompting context, and slow learning, which applies single-rollout REINFORCE++ or PPO updates using memory-conditioned trajectories. This design aims to combine rapid experience reuse with persistent improvements in model behavior.

Experiments with Qwen3-4B-Instruct-2507 across four tasks show that FSOL achieves 60.2% average accuracy on stationary streams, outperforming standalone RL and memory baselines. On non-stationary streams, the reported results indicate faster recovery after distribution shifts, adaptation to mixed-task streams, and retention of previously learned skills in sequential settings.

Critical strengths
  1. A well-posed problem formulation. Treating test-time adaptation as online learning, with cumulative accuracy and a single rollout per query, matches how deployed agents are actually used. It also explains why neither fast memory nor slow RL alone is sufficient. The framing is simple and likely to be adopted.

  2. Consistent, sizable gains across conditions, reported honestly. Results are averaged over four seeds with SEM.

    • Stationary streams (Table 1). FSOL with REINFORCE++ and raw memory improves on both of its components. The REINFORCE++ comparison uses the same batch size (16) for FSOL and RL alone (Table 2).
      • WebShop: 52.5 vs 38.4 for REINFORCE++ alone and 34.0 for raw memory alone
      • ALFWorld: 52.1 vs 33.7 and 25.0
    • Other stream types. The gains hold across distribution-shift, mixed-task and sequential-task streams.
    • A case where it does not help. The paper reports that AWM memory does not help PPO on WebShop (46.7 vs 47.7 for PPO alone), which makes the positive results more credible.
Critical weaknesses
  1. Limited methodological novelty. FSOL primarily combines existing retrieval-based memory (raw trajectories or AWM-style workflows) with standard REINFORCE++ or PPO updates, conditioning both the policy and reference distributions on retrieved context. Similar memory-plus-RL approaches have been explored in prior work (e.g., SkillRL and Skill-R1), while the broader fast-memory/slow-learning paradigm is well established. Thus, the contribution lies mainly in the online learning formulation and empirical evaluation rather than a fundamentally new method.

  2. Insufficient experimental controls and unclear sources of improvement. The PPO baseline uses a larger batch size than FSOL (64 vs. 32), resulting in half as many updates over the same stream and potentially confounding the comparison. The experiments also lack replay-based RL or existing memory-plus-RL baselines to isolate the benefits of retrieval. Moreover, retrieved memories contain successful action sequences, raising concerns about near-duplicate lookup rather than genuine learning. Without held-out evaluation or testing the learned weights independently of memory, it remains unclear whether FSOL produces transferable policy improvements.

Questions for authors
  1. Beyond computing the RL update under the same retrieved memory (Eq. 6), what distinguishes FSOL from the memory-plus-RL methods cited in Sec 2 when they are run online with one rollout per query? Can any of them be run as a baseline?
  2. What fraction of queries in each stream have a near-duplicate earlier in the stream (same target product, or same task template and objects)? How often do the top-3 retrieved items share the current goal?
  3. Can you evaluate the final FSOL and RL-only weights on held-out tasks, with and without memory?
3 comments
AuthorsOpenReview ↗

Reply to Weakness 1

Thank you so much for recognizing (1) our well-motivated online learning formulation with realistic single-rollout constraints, and (2) FSOL's consistent improvements across different types of streams, supported by rigorous experiments. Now we address each weakness and questions below.

Weakness 1

Limited methodological novelty. FSOL primarily combines existing retrieval-based memory (raw trajectories or AWM-style workflows) with standard REINFORCE++ or PPO updates, conditioning both the policy and reference distributions on retrieved context. Similar memory-plus-RL approaches have been explored in prior work (e.g., SkillRL and Skill-R1), while the broader fast-memory/slow-learning paradigm is well established. Thus, the contribution lies mainly in the online learning formulation and empirical evaluation rather than a fundamentally new method.

The novelty of our work lies in tackling online learning problems using both memory and RL, while prior works on memory-plus-RL cannot be applied to online learning.

  • Prior works need to pre-collected memory from larger teacher models, perform SFT for warm start, or train memory curators before RL, while our work assumes all memory generation and learning happens online during deployment. For online learning, future tasks cannot be foreseen and may come from changing task distributions. Therefore, an online learner does not know task distribution and should not rely on task-specific offline preparation.
  • Prior works require generating multiple rollouts per query, while generating multiple rollouts per query is not feasible in online learning. An agent cannot repeatedly execute real-world tasks (e.g., making online purchases) or ask users to evaluate multiple responses solely for training. Several existing methods require multiple rollouts of the same query to estimate memory utility or optimize memory-related objectives. In our method FSOL, we use PPO or REINFORCE++ so that the advantage estimation can be done with single rollout per query.

We summarize the limitations of existing memory-augmented RL methods under our online learning protocol in the following table.

MethodLimitations for Online Learning
SkillRLRequires pre-collected memory and an SFT warm-start before RL training.
EvolveRRequires an SFT warm-start before RL training.
D2Skill and RetroAgentRequire multiple rollouts of the same query, both with and without memory.
Skill-R1Requires pre-collected memory and multiple rollouts to compute its inter-generation objective.
SkillOSRequires offline training of the memory curator before deployment.

We will clarify our novelty in the updated manuscript.

AuthorsOpenReview ↗

Reply to Weakness 2

Weakness 2

We address each point below.

Insufficient experimental controls and unclear sources of improvement. The PPO baseline uses a larger batch size than FSOL (64 vs. 32), resulting in half as many updates over the same stream and potentially confounding the comparison.

We observed that PPO suffers from a performance collapse with a batch size of 32 (accuracy drops to 0), so we increased the batch size to 64 for stable training. To ensure a fair comparison, we also conducted additional experiments with FSOL (PPO + raw) using the same batch size of 64. The results show that FSOL continues to outperform PPO under this setting.

MethodWebShopALFWorldSearchWriting
PPO (bs=64)47.740.635.849.7
FSOL (bs=64)55.948.7 (still ongoing, 60% finished)37.395.3
FSOL (bs=32)56.350.937.596.3

The experiments also lack replay-based RL or existing memory-plus-RL baselines to isolate the benefits of retrieval.

Could you clarify what you mean by replay-based RL in this context? We assume you are referring to RL methods that reuse past trajectories for policy updates, potentially with importance sampling corrections, as in [1]. We are currently exploring this approach and will report the results when available.

We would also like to clarify that replay-based RL address different aspects from FSOL. It is orthogonal to our current method: replay-based RL could naturally be integrated into and serve as the parametric learning component of FSOL.

Regarding other memory-plus-RL methods, we have addressed their applicability to our online learning setting in our response to Weakness 1.

[1] Ma et al. Freshness-aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning.

Moreover, retrieved memories contain successful action sequences, raising concerns about near-duplicate lookup rather than genuine learning.

Our paper clearly demonstrates that FSOL substantially outperforms action sequence alone, indicating that its improvements go beyond simple memory lookup. Furthermore, SearchQA and Writing feature diverse queries without any duplicates. In WebShop some target products are similar but with different color or size; in ALFWorld, some instructions are same but the environment layouts are different. The strong performance of FSOL on these two benchmarks provides further evidence that its effectiveness does not depend on repeated or duplicate queries.

Without held-out evaluation or testing the learned weights independently of memory, it remains unclear whether FSOL produces transferable policy improvements.

Evaluating FSOL without memory introduces a train-test mismatch. FSOL is designed to learn a memory-conditioned policy, where retrieval and parametric learning jointly contribute to performance. Removing memory during evaluation changes the input distribution from that encountered during training, and therefore may not accurately reflect the effectiveness of the learned policy.

AuthorsOpenReview ↗

Reply to Questions 1-3

Question 1

Beyond computing the RL update under the same retrieved memory (Eq. 6), what distinguishes FSOL from the memory-plus-RL methods cited in Sec 2 when they are run online with one rollout per query? Can any of them be run as a baseline?

We have discussed this issue in our reply to Weakness 1. Since these existing algorithms include designs requiring offline training / SFT / memory curation, or multiple rollouts per query, they can hardly be applied to online learning as baselines.

Question 2

What fraction of queries in each stream have a near-duplicate earlier in the stream (same target product, or same task template and objects)? How often do the top-3 retrieved items share the current goal?

  • Near-duplicate queries. WebShop is an online shopping benchmark in which no two queries are exactly identical, although some share the same target product with different requirements, such as size or color. Similarly, in ALFWorld, some tasks share the same instructions but differ in environment configurations (e.g., object locations), requiring different action sequences. SearchQA and Writing contain diverse queries without near-duplicates.
  • Evaluating generalization beyond similar queries. Defining near-duplicates consistently across these heterogeneous benchmarks is challenging. To address the underlying concern of whether FSOL relies on repeated or similar queries, we will additionally report performance on held-out test sets for WebShop and ALFWorld, while SearchQA and Writing already contain diverse queries without duplicates. These experiments will help assess whether FSOL's improvements generalize beyond queries encountered during online training.

Question 3

Can you evaluate the final FSOL and RL-only weights on held-out tasks, with and without memory?

We will provide the requested held-out evaluation results with and without memory. As we mentioned before, FSOL is designed to learn a memory-conditioned policy, so memory-free evaluation may not capture the effectiveness of FSOL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.