- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents Fast and Slow Online Learning (FSOL), a framework for agents that formulates test-time learning as an online learning problem. The paper first proposes the online learning paradigm by evaluates agents by cumulative performance over the interaction. Specifically, for each incoming query, the agent retrieves successful past experiences, generates one trajectory, receives binary feedback, and saves successful experience to memory. After a batch of queries, it updates the model using PPO or REINFORCE++ with the memory-augmented policy. Empirically, the paper evaluates FSOL on four benchmarks across four online settings: stationary streams, distribution shifts, sequential tasks, and mixed-task streams, covering deployment conditions that the authors describe as largely unexplored in prior work.
Critical strengths
- The online learning framework of test-time learning and the proposed method is clear and sensible.
- The paper evaluates the FSOL framework in multiple benchmarks and stationary and non-stationary settings and show promising improvements compared to the baselines.
Critical weaknesses
- I think the main issue is that the positioning relative to prior work is a bit overstated. StreamBench [1] already studies online adaptation through cumulative performance over an interaction stream as the evaluation setting, while prior work [2] explores combining fast textual adaptation with slow parameter updates. The paper should distinguish this contribution more clearly from the existing test-time learning paradigm.
- For presentation, the paper should expand and explain clearly what the MemoryUpdate and RL Update is (the concrete formulation), e.g., how the memory update uses the environment feedback.
Questions for authors
- This might be minor but in the appendix, why does the LLM judge for writing only see the response?
- Why using different hyperparameters, batch size especially, for the RL (PPO, Reinforce ++ with FSOL, PPO with FSOL)? In this setting, the PPO will get less gradient updates than FSOL based RL training.
- Any intuition why raw memory is better than AWM?
- It might be interesting to see the same FSOL checkpoint without memory to support that memory accelerates learning in the model’s parameters.
4 comments
Reply to Questions 1-4
Question 1
This might be minor but in the appendix, why does the LLM judge for writing only see the response?
Thank you for pointing this out. Our original judge primarily focuses on evaluating whether the generated response follows the target writing style, which can largely be assessed from the response itself. We agree that including the original user query would allow a more comprehensive evaluation. We will add a prompt-aware evaluation and report the results later, if time permits.
Question 2
Why using different hyperparameters, batch size especially, for the RL (PPO, Reinforce ++ with FSOL, PPO with FSOL)? In this setting, the PPO will get less gradient updates than FSOL based RL training.
We did not conduct an extensive hyperparameter search or tune hyperparameters separately for each benchmark. We started with a default configuration (lr=1e-6, KL coefficient=0.01, batch size=16) and made a few adjustments on WebShop to make training stable (i.e., avoid accuracy crash). For REINFORCE++, FSOL shows accuracy crash with KL=0.01, so we increased it to 0.05. We also tested KL=0.05 for REINFORCE++, but it caused accuracy crash, so we retained KL=0.01 for REINFORCE++ only. For PPO, we increased the batch size from 16 to 32 for FSOL and to 64 for PPO-only to avoid accuracy collapse. We additionally tested both PPO and FSOL with batch size 64, and FSOL still outperformed PPO. See the table below. ALFWorld is still running and we will report the final result later.
| Method | WebShop | ALFWorld | Search | Writing |
|---|---|---|---|---|
| PPO (bs=64) | 47.7 | 40.6 | 35.8 | 49.7 |
| FSOL (bs=64) | 55.9 | 48.7 (on-going, 60% finished) | 37.3 | 95.3 |
| FSOL (bs=32) | 56.3 | 50.9 | 37.5 | 96.3 |
Importantly, these configurations were obtained on WebShop, and then worked well on other benchmarks. Thus, our hyperparameter choices were primarily motivated by training stability and not individually tuned. The results suggest robustness across different environments.
Question 3
Any intuition why raw memory is better than AWM?
Our hypothesis is that smaller models may struggle with extracting generalizable workflows from trajectories or/and translating abstract workflows into concrete actions. In contrast, raw trajectories provide explicit, step-by-step examples that are easier to follow and adapt to the current task. This may explain why raw memory outperforms AWM in our experiments.
Question 4
It might be interesting to see the same FSOL checkpoint without memory to support that memory accelerates learning in the model’s parameters.
Our intuition for the reason why memory can accelerate RL is that it transforms learning the task itself into learning adapting the solution of similar tasks to the current query. We will show the evaluation results of FSOL checkpoints without memory if time permits. However, FSOL is designed to learn a memory-conditioned policy, where retrieval and parametric learning jointly contribute to performance. Evaluating the policy without memory introduces a train-test mismatch, and therefore may not necessarily reflect the effectiveness of the learned policy.
Sorry for missing the reference and [2] should refer to: Tiwari, Rishabh, et al. "Learning, fast and slow: Towards LLMs that adapt continually." arXiv preprint arXiv:2605.12484 (2026).
Thank you for the reply. After reading the rebuttal, I think I still have some concerns on the paper regarding:
- I think the overall contribution is somewhat limited by adapting the framework to single-rollout agentic setting. Given the single-rollout agentic setting has also been established by prior work, e.g., [1], I feel like the novelty is limited.
- what do you mean by "made a few adjustments on WebShop to make training stable"?
- Also, I feel like there's some uncertainty of the baseline results, e.g., raw memory & AWM comparisons. Please double check the implementation.
Given these concerns, I lean toward a weak rejection for now.
[1] Hou, Zhenyu, et al. "Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning." arXiv preprint arXiv:2607.07508 (2026).
Reply to followup questions
Thank you for the follow-up comments. We address each point below.
Sorry for missing the reference and [2] should refer to: Tiwari, Rishabh, et al. "Learning, fast and slow: Towards LLMs that adapt continually." arXiv preprint arXiv:2605.12484 (2026).
Thank you for clarifying the reference. We agree that this work shares a similar high-level motivation of combining fast and slow adaptation. However, the key distinction lies in the learning setting and objective.
First, their implementation relies on GEPA for context optimization, which uses validation sets for candidate selection and is not directly applicable to online setting, where each interaction permits only one rollout and future queries are unknown.
Second, FSOL explicitly targets cumulative performance over the interaction stream, rather than primarily improving model adaptation through joint context and parameter optimization. This objective motivates our design of using memory for immediate improvement and RL for higher long-term performance. Our experiments further demonstrate that this principle generalizes across different memory and RL implementations.
I think the overall contribution is somewhat limited by adapting the framework to single-rollout agentic setting. Given the single-rollout agentic setting has also been established by prior work, e.g., [1], I feel like the novelty is limited.
[1] Hou, Zhenyu, et al. "Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning." arXiv preprint arXiv:2607.07508 (2026).
SAO is concurrent with our work (released in July 2026), and we have already cited it in our introduction to motivate the research gap.
Our contribution is not the single-rollout setting itself, but addressing the trade-off between learning speed and performance ceiling under the cumulative learning objective. Existing memory-based methods adapt quickly but have limited performance ceilings, while RL-based methods such as SAO can achieve stronger performance but require many interactions to improve. Thus, SAO is exactly one side of the trade-off we want to address: learning speed versus ceiling performance.
FSOL combines these complementary mechanisms to achieve both fast adaptation and strong long-term performance, thereby improving cumulative reward.
what do you mean by "made a few adjustments on WebShop to make training stable"?
We did not independently tune hyperparameters to maximize each method's performance. The adjustments were made to avoid training collapse.
Specifically, we started with the default configuration (learning rate = 1e-6, KL coefficient = 0.01, batch size = 16):
- REINFORCE++: Memory-augmented REINFORCE++ suffered from accuracy collapse at KL = 0.01, so we increased it to 0.05. We also tested REINFORCE++ without memory at KL = 0.05, which exhibited collapse. So we keep KL = 0.01 for REINFORCE++ alone
- PPO: PPO with memory required batch size 32 for stable training, while PPO without memory required batch size 64. We additionally compared both methods at batch size 64 to control for this difference, as reported in our previous response. FSOL still outperformed PPO-only under matched batch sizes, supporting the effectiveness of memory augmentation beyond differences in hyperparameter settings.
Also, I feel like there's some uncertainty of the baseline results, e.g., raw memory & AWM comparisons. Please double check the implementation.
We have carefully rechecked our implementations and found no bugs.
Regarding the uncertainty of memory baselines, we have updated the SFT baselines and SFT + raw memory with 4 random seeds.
| Methods | WebShop | ALFWorld | Search | Writing |
|---|---|---|---|---|
| SFT | 21.5 ± 5.0 | 33.2 ± 2.8 | 27.2 ± 1.2 | 79.2 ± 1.2 |
| FSOL (SFT + raw memory) | 58.4 ± 3.9 | 40.5 ± 1.6 | 35.8± 0.7 | 95.7 ± 1.2 |
As for memory baselines, we acknowledge that the reported results are based on a single run. Nevertheless, FSOL consistently outperforms multiple memory-only baselines by substantial margins across benchmarks, suggesting that the improvement is unlikely to be explained by randomness.
More importantly, FSOL is a general framework that can incorporate different memory and RL methods. Its core contribution is demonstrating how fast non-parametric adaptation and slow parametric learning complement each other to improve cumulative performance, rather than relying on the superiority of any particular memory method.
Reply to Weaknesses 1 and 2
Thank you so much for recognizing (1) the clarity and sound motivation of our online learning framework and FSOL design, and (2) the promising improvements across multiple benchmarks and both stationary and non-stationary settings. Now we address each weakness and question below.
Weakness 1
Thank you for the comment. We noticed that reference [2] is missing. Could you please clarify which work you are referring to?
We acknowledge that online learning and cumulative performance have been studied previously. Our contribution is to address the trade-off between fast but limited memory adaptation and slow but more capable RL, leveraging their complementary time scales to improve cumulative performance.
Importantly, existing memory + RL methods are difficult to apply to strict online learning for two reasons:
We summarize the limitations of existing memory-augmented RL methods under online learning in the following table.
We will clarify our novelty in the updated manuscript.
Weakness 2
Thanks for the suggestions. We will include them for different RL algorithms and different memory methods in the Appendix A of the updated manuscript.