acceptodds
Under review as a conference paper at ICLR 2027

Learning to Learn Experiential Knowledge

Abstract

Language agents can solve individual tasks effectively, yet they remain limited at accumulating experiential knowledge: converting prior task trajectories into reusable knowledge in free-form text that improves performance on future tasks. Prior work prompts strong models to write rules, skills, and strategies into an in-context notebook, but few approaches train models to improve this capability over long task streams. Training is challenging because the utility of each update is only revealed through later tasks. We study experiential knowledge accumulation through a persistent in-context notebook that is read by a task model and rewritten after each task by a notebook model. We propose NoteTaker, which trains the notebook model in two stages: SFT on teacher-generated notebook updates, followed by RL on short task streams, where each update receives credit only from subsequent task rewards that it could influence. Across three benchmarks, separately trained notebook models improve task models when deployed in online 40-task streams. These improvements transfer to held-out tasks, including AppWorld Test-Challenge tasks that involve applications and APIs absent from notebook model training. On AppWorld Test-Challenge, our finetuned Qwen3.5-9B notebook model increases task success from 0.0% to 25.8%, compared with 23.0% when Fable-5.1 serves as the notebook model. Analysis shows that SFT establishes basic experience abstraction and knowledge organization, while RL aligns notebook updates with downstream task performance, consistently improving over SFT across benchmarks and model sizes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continual Learning, Memory, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bSusOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a notebook model to extract reusable knowledge from task trajectories while keeping the task model frozen. It combines teacher-based SFT with RL using rewards from subsequent tasks. Experiments on three benchmarks show improvements over SFT and transfer to longer task streams and unseen task models.

Critical strengths

Freezing the task model cleanly isolates the contribution of the notebook system. RL consistently improves over SFT across three benchmarks and two model sizes, and transfer to unseen task models makes the results more compelling.

Critical weaknesses
  1. I am not sure how much the long-stream gains depend on continued knowledge accumulation. Figures 3--4 show cumulative success, which can keep increasing even if the notebook stops improving after the first few tasks. A comparison with a learned notebook frozen after eight updates would help establish whether continued updating is actually useful.

  2. The evidence for the proposed credit assignment is less convincing than the overall benefit of RL. Figure 5 shows gains of 1.9 and 4.3 points over the outcome-reward baseline, but does not show uncertainty or variation across training runs. It is hard to tell whether these gains are robust enough to support the specific method.

Questions for authors
  1. How does continued updating compare with freezing the notebook after eight updates on the same task streams? This would directly test whether updates beyond the training horizon remain useful.

  2. Are the credit-assignment gains in Figure 5 consistent across training seeds? Could you report uncertainty for this comparison?

  3. What exactly does the notebook model receive as post-task feedback on each benchmark? Does held-out evaluation provide only success/failure, or also detailed test diagnostics or reference solutions?

5 comments

We thank the reviewer for their time and for recognizing the consistent improvements from RL and the generalization to unseen task models. We conduct the requested freeze-after-8 control and a second training seed. We also discuss feedback with an example for each benchmark.

W1/Q1: A comparison with a learned notebook frozen after eight updates would help establish whether continued updating is actually useful.

Thank you for the suggestion. On the same held-out ALFWorld streams, the trained notebook model updates the notebook for tasks 1-8, and the notebook is then either kept updating or frozen for tasks 9-40 (success %):

All 40 tasksLast 8 tasks
4B, frozen after 882.484.6
4B, continued updating86.0 (+3.6)94.1 (+9.5)
9B, frozen after 884.488.2
9B, continued updating86.6 (+2.2)96.3 (+8.1)

Continued updating improves success, mostly late in the stream. We will add this control for all benchmarks to the appendix, together with non-cumulative per-position success curves.

W2/Q2: Are the credit-assignment gains in Figure 5 consistent across training seeds?

We trained an independent second run of both methods with different random seeds. The table reports the held-out AppWorld success rate after 160 training steps. The causal accumulated reward stays ahead of the trajectory-level outcome reward in both runs at 4B and 9B. We will add these results in the revision.

EstimatorModelSeed 1Seed 2Average
Trajectory-level outcome reward4B12.811.812.3
Causal accumulated reward (ours)4B14.714.314.5 (+2.2 / +17.9%)
Trajectory-level outcome reward9B19.821.820.8
Causal accumulated reward (ours)9B24.125.124.6 (+3.8 / +18.3%)

Q3: What exactly does the notebook model receive as post-task feedback on each benchmark? Does held-out evaluation provide only success/failure, or also detailed test diagnostics or reference solutions?

After a task is scored, the notebook model receives the task model's trajectory and a feedback string, which can affect only later tasks. Held-out evaluation provides the diagnostics of each benchmark's own evaluator, and never reference solutions. We provide a description and a short example of the feedback for each benchmark below:

BenchmarkFeedbackHeld-out example
AppWorldEvaluator report, assertion message, termination reason.Num Passed Tests: 1 Num Failed Tests: 6 ↵ >> Failed Requirement: assert model changes match venmo.Notification, venmo.PaymentRequest ↵ AssertionError: {'venmo.Transaction', 'venmo.Notification', 'venmo.User'} == {'venmo.Notification', 'venmo.PaymentRequest'}
ALFWorldBinary outcome, number of actions, cumulative reward, final observation, termination reason.ALFWorld binary outcome: FAIL. Executed 45 admissible actions and received cumulative native reward 0. ↵ Final environment feedback: You arrive at countertop 2. … ↵ Episode outcome: FAIL. The agent reached the 50-turn evaluation limit.
MiniGridEpisode status, reward, step and invalid-action counts, outcome, termination reason.GridGames episode terminated; reward=-1.000000; primitive_steps=1; invalid_actions=0 ↵ Episode outcome: FAIL. The environment ended after 1 model turns.

We would be very happy to continue the discussion and address any remaining concerns. If our responses have resolved your concerns, we would kindly ask you to consider reassessing your evaluation of our paper.

Thank you again for taking the time to review our paper. As the discussion period is closing soon, we would greatly appreciate your feedback on our responses and the additional results. If you have any remaining concerns or would like further clarification, we would be happy to address them!

Thank you for the additional experiments and for clarifying the feedback available to the notebook model. It remains unclear how much of the improvement would carry over to settings where the model receives only success or failure feedback, so I am keeping my original rating.

Reviewer_4MkLOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the accumulation of experimental knowledge in the token space using a learnable note-taker model and a frozen task model. The note-taker model generates/rewrites a notebook with input prompts to the task model. The note-taker model is trained on segments of task trajectories by SFT + RL: after each task, a powerful teacher model (Kimi-K2.6) maps the previous notebook and completed trajectory to a revised notebook, providing supervised examples for the note-taker model to learn; and then continues RL training. The experimental results show that the proposed approach performs better than using the frozen task model alone and with non-learnable notebook models.

Critical strengths
  1. The idea of learning a notebook to retain the task knowledge and benefit future task learning is interesting and distinct from previous works.

  2. The experiments show clear improvement compared to the frozen task model with hand-designed contexts as the notebook.

Critical weaknesses
  1. The motivation of this work is not clear to me.

    • The goal of this work is to accumulate knowledge from past tasks and perform better on future tasks. However, it is unclear why learning a notebook achieves this goal better than learning the task model. Specifically, learning the notebook requires high computational cost as well: The note-taker model has the same size as the task model. And to learn the note-taker model, it requires SFT from samples generated by a powerful teacher (Kimi-K2.6) and then RL. It could be beneficial to see the performance difference between learning the task model and learning the note-taker model.
  2. Some experimental settings/results are not well discussed.

    • The influence of the teacher model. The teacher model used in the paper is Kimi-K2.6, which has 1T size. It’s unclear how necessary it is to use such a large teacher model. How does the teacher model influence performance?
    • Considering the SFT and RL time, what is the computational overhead? The current notebook size is limited to 4096. Will it influence the model capacity for long-horizon task learning?
    • The legends in Figure 4(b), Figure 5(a), and (b) do not match the plots, making the plots hard to interpret. What do the dots in Figure 5 mean?
Questions for authors

Please see the weakness.

2 comments

We thank the reviewer for their time and for finding the idea of a learned notebook interesting and distinct, with clear experimental improvements. Below, we clarify why we train a separate notebook model, add an experiment that removes the teacher (RL without SFT), and report the compute cost and how often notebooks reach the size limit.

W1: why learn a notebook rather than the task model?

There are two reasons: (1) the notebook models can be used with task trajectories generated by any models, and (2) we aim to isolate the knowledge abstraction from task solving for analysis purposes.

First, we can use a notebook model with trajectories generated by other models. Fig. 4b shows that a notebook model trained with one task model not only improves performance with that task model but also with trajectories generated by other task models.

Second, this study focuses on one isolated capability: abstracting reusable knowledge from task trajectories. Training a unified model, or jointly training both models, cannot distinguish whether an improvement comes from better task solving or from better knowledge abstraction.

We agree that the notebook model and the task model can be the same model. Training a unified model could improve both capabilities (task solving and knowledge abstraction) simultaneously and may yield higher benchmark scores. We leave this to future work.

W1: Is it necessary to use such a large teacher model? How does the teacher model influence performance?

To test how much the teacher SFT matters, we remove it entirely and run RL directly from the base model without SFT (held-out AppWorld success rate):

Zero-gradient groups at the first rolloutStep 0Step 32Step 64Step 96Step 128Step 160
4B, RL from base81%0.20.51.21.43.53.4
4B, RL from SFT2%5.16.49.410.511.514.3
9B, RL from base25%1.22.64.09.413.215.2
9B, RL from SFT2%11.010.815.318.421.125.1

The first column is the share of rollout groups in which every rollout gets the same reward, so the group gives no learning signal. SFT greatly reduces this share and thus makes RL training more efficient. The teacher is used only offline to produce SFT targets, and the trained notebook model surpasses it on AppWorld. The choice of teacher model is a hyperparameter in our experiment, and it can be replaced by other models.

W2: Considering the SFT and RL time, what is the computational overhead?

SFT takes 4–5 hours on 8 H200 GPUs. A complete RL run on one benchmark (e.g., AppWorld) takes about 70 hours on 8 H200 GPUs. We will add these details to the appendix.

W2; The current notebook size is limited to 4096. Will it influence the model capacity for long-horizon task learning?

On a 40-task stream of AppWorld, 7-10% of notebook updates reach the limit for the 4B notebook model and 26-34% for the 9B notebook model. We chose this budget so that the notebook fills up as knowledge accumulates over a long stream, which tests whether the notebook model can compress useful information into a limited space. The budget is a hyperparameter and can be increased for tasks that require retaining more information. We will add this discussion to the revision. As a side note for clarity, the notebook size limit is 16K bytes (Line 907), while 4K tokens is the task model's per-response token limit (Line 906).

W2: The legends in Figure 4(b), Figure 5(a), and (b) do not match the plots, making the plots hard to interpret. What do the dots in Figure 5 mean?

The legends match the plots, but we thank the reviewer for pointing out that they are confusing. In the revision, Fig. 5(a) labels the two estimators "causal accumulated reward" and "trajectory-level outcome reward" to match the caption and shows 4B and 9B held-out success in separate panels. Fig. 4(b) replaces the "Seen / Unseen" outline keys with a shaded band that marks the seen task model. Each panel also gets a title stating its takeaway.

In Fig. 5, each dot is one measurement and the line is a smoothed trend through the dots. In the left panel of Fig. 5(a), a dot is the mean training reward of one RL rollout. In the right panel of Fig. 5(a) and in Fig. 5(b), a dot is the held-out success rate of one saved checkpoint. We will state this in the captions.


We would be very happy to continue the discussion and address any remaining concerns. If our responses have resolved your concerns, we would kindly ask you to consider reassessing your evaluation of our paper.

Thank you again for taking the time to review our paper. As the discussion period is closing soon, we would greatly appreciate your feedback on our responses and the additional results. If you have any remaining concerns or would like further clarification, we would be happy to address them!

Reviewer_PyZQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Specifically, the authors propose a two-stage SFT+RL training approach. First, notebook updates generated by a teacher model based on past task trajectories are used for SFT, enabling the notebook model to develop basic knowledge abstraction capabilities. In the second stage, RL with position-matched causal credit assignment is used to further improve the notebook model's update policy. This is achieved by comparing rewards across multiple rollouts at the same task position and assigning each memory update rewards only from subsequent tasks it could influence, addressing the delayed reward and credit assignment problems in experience learning. Training is conducted on shorter task sequences to reduce the computational cost of long-horizon RL. The experimental results show that NoteTaker can significantly improve performance on three commonly used benchmarks, including AppWorld, ALFWorld, and MiniGrid. Specifically, using the Qwen3.5-9B model, the final results show an improvement of 23.6 percentage points on AppWorld Test-Challenge over SFT. The results also show that the notebook model can generalize to longer 40-task sequences and unseen tasks, despite being trained on shorter sequences with only eight notebook updates.

Critical strengths

S1. The work focuses on how an agent can continuously accumulate reusable experience from past tasks—a research problem with significant practical value. By decoupling task execution from experience learning, the research objectives are rendered more clearly defined.

S2. The experimental results are highly compelling, demonstrating significant performance gains across three benchmarks: AppWorld, ALFWorld, and MiniGrid. Specifically, on the AppWorld Test-Challenge, the approach achieves an improvement of 23.6 percentage points over SFT (Supervised Fine-Tuning).

S3. The model exhibits a degree of generalization capability; a "Notebook Model" trained using only relatively short task sequences is able to generalize to long sequences of 40 tasks, as well as to previously unseen "Task Models."

Critical weaknesses

W1. The overall framework is quite similar to existing RL-based memory update methods, such as MemoPilot. While the primary innovation lies in credit assignment, the technical distinction from existing approaches is not sufficiently pronounced.

W2. The study lacks independent ablation analyses for "Position Matching" and "Causal Suffix Return." Furthermore, the performance gains attributed to credit assignment are relatively modest (1.9–4.3 percentage points), and the results lack validation across multiple random seeds or a broader range of benchmarks.

W3. The knowledge extracted consists primarily of environment-specific rules and operational procedures. Moreover, since a separate "Notebook Model" is trained for each benchmark, the model’s capacity for learning generalizable experience—applicable across different environments—has not yet been adequately demonstrated.

Questions for authors

Q1. I am not fully convinced about the novelty of the proposed method compared with existing RL-based memory update methods, particularly MemoPilot. Both approaches train a separate model to update memory based on past task trajectories and use downstream task performance as feedback, while keeping the task model frozen. Is the main difference the position-matched causal credit assignment, or are there other important differences in the training framework? I think a more detailed comparison with MemoPilot would help clarify the actual contribution.

Q2. In Figure 5(a), the authors compare the proposed causal credit assignment with a trajectory-level outcome reward. However, the proposed method combines position-matched reward comparison and causal suffix returns. It is unclear how much each component contributes to the final improvement. Could the authors provide separate ablations for these two components? Also, the improvement from the proposed credit assignment seems relatively modest (1.9 and 4.3 percentage points for 4B and 9B). Since the ablation is only conducted on AppWorld with a single training seed, it would be helpful to see whether the improvement is consistent across different seeds and the other two benchmarks.

3 comments

We thank the reviewer for their time and for recognizing the value of the problem and our compelling experimental results. To address the concerns, we conducted all suggested experiments showing that (1) our causal accumulated reward outperforms MemoPilot's next-task credit, (2) crediting all later tasks accounts for most of the gain in the ablation, (3) the improvement holds with a second training seed, and (4) notebook models trained on one benchmark also improve the others.

W1: I am not fully convinced about the novelty of the proposed method compared with existing RL-based memory update methods, particularly MemoPilot. Is the main difference the position-matched causal credit assignment, or are there other important differences?

The main difference is the task setting and the resulting credit assignment rule. We trained MemoPilot's next-task credit inside our framework. In our setting, our causal accumulated reward works best.

The tasks that the two works focus on are very different, which leads to different design choices in RL. In MemoPilot, each episode repeats one game against one fixed opponent, so the information in a memory update can be used immediately in the next game. Conversely, our study uses long streams of agentic tasks where knowledge extracted from one task may not help immediately but may be reused several tasks later. This difference brings new challenges, and motivates us to study how we should train on long streams.

As a result, MemoPilot uses short RL training rollouts (3 games and 2 memory updates), whereas we use 8 memory updates per rollout. Given this difference in setting, MemoPilot reports that crediting only the next task works better than using a cumulative return, whereas our additional experiment finds the opposite, as shown in the following table (held-out AppWorld success rate, %, after 160 training steps).

Credit assignment4B9B
Next task only (MemoPilot)11.119.7
Trajectory-level outcome reward12.320.8
Causal accumulated reward (ours)14.524.6

We will discuss this in the related work in our revision.

W2: The study lacks independent ablation analyses for "Position Matching" and "Causal Suffix Return."

We conducted the suggested ablations on AppWorld. All estimators start from the same SFT notebook model and use the same RL setup and number of steps, and differ only in how the advantage of each update is computed. We report the success rate (%) on held-out split. Following the notation in the paper, each group has GG rollouts, and rg(ℓ)r_g^{(\ell)} is the reward of rollout gg at task position ℓ\ell. Let r~g(ℓ)=rg(ℓ)−1G−1∑h≠grh(ℓ)\tilde r_g^{(\ell)}=r_g^{(\ell)}-\frac{1}{G-1}\sum_{h\ne g}r_h^{(\ell)} be this reward minus the mean of the other rollouts at the same position, rˉg\bar r_g the mean reward of rollout gg over its tasks, rˉ=1G∑hrˉh\bar r=\frac{1}{G}\sum_h\bar r_h the mean over all rollouts in the group, and bg=1G−1∑h≠grˉhb_g=\frac{1}{G-1}\sum_{h\ne g}\bar r_h the same mean computed without rollout gg. For the trajectory-level outcome reward and our method, we report the average of two training runs with different random seeds.

EstimatorAdvantage of update kk in rollout gg4B9B
Trajectory-level outcome rewardrˉg−rˉ\bar r_g-\bar r: the same value for every update12.3 (−2.2)20.8 (−3.8)
Next task onlyr~g(k)\tilde r_g^{(k)}: the next task only11.1 (−3.4)19.7 (−4.9)
Pooled suffix∑ℓ≥k(rg(ℓ)−bg)\sum_{\ell\ge k}(r_g^{(\ell)}-b_g): later tasks, one baseline pooled over positions14.4 (−0.1)23.3 (−1.3)
Causal accumulated reward (ours)∑ℓ≥kr~g(ℓ)\sum_{\ell\ge k}\tilde r_g^{(\ell)}: later tasks, position-matched baseline14.524.6

Crediting each update with the rewards of all later tasks it can influence accounts for most of the gain. At both model sizes, the two estimators that sum over later tasks outperform the trajectory-level outcome reward, which gives every update the same credit, and next-task credit, which ignores delayed payoffs.

W2: The gains attributed to credit assignment lack validation across multiple random seeds.

We trained an independent second run of both methods with different random seeds. The table reports the held-out AppWorld success rate after 160 training steps. The causal accumulated reward stays ahead of the trajectory-level outcome reward in both runs at 4B and 9B.

EstimatorModelSeed 1Seed 2Average
Trajectory-level outcome reward4B12.811.812.3
Causal accumulated reward (ours)4B14.714.314.5
Trajectory-level outcome reward9B19.821.820.8
Causal accumulated reward (ours)9B24.125.124.6

W3: The knowledge extracted consists primarily of environment-specific rules. Since a separate notebook model is trained for each benchmark, the capacity for learning generalizable experience across environments has not been adequately demonstrated.

We conducted an additional analysis of cross-benchmark generalization. We find that the ability to abstract reusable knowledge also generalizes across benchmarks. We ran each trained 9B notebook model on the other benchmarks (9B task model, success %). Compared with the untrained base model, training on each benchmark increases the score on the other benchmarks in most cases, with few exceptions. This indicates that the model learns not only environment-specific rules but also a more general update policy. We speculate that jointly training on multiple benchmarks can further improve generalization, and leave this to future work.

Notebook model / evaluated onAppWorldALFWorldMiniGrid
Untrained Qwen3.5-9B1.267.935.5
AppWorld-trained29.177.640.6
ALFWorld-trained0.286.644.4
MiniGrid-trained12.077.141.5

We would be very happy to continue the discussion and address any remaining concerns. If our responses have resolved your concerns, we would kindly ask you to consider reassessing your evaluation of our paper.

Thank you again for taking the time to review our paper. As the discussion period is closing soon, we would greatly appreciate your feedback on our responses and the additional results. If you have any remaining concerns or would like further clarification, we would be happy to address them!

Comments on the paper

1 comment

We thank the AC and the reviewers for their constructive feedback.

Strengths noted by the reviewers

  • The problem of accumulating reusable experience across tasks has significant practical value (PyZQ).
  • The idea of a learned notebook is interesting and distinct from previous works (4MkL).
  • Freezing the task model cleanly isolates the contribution of the notebook (bSus, PyZQ).
  • RL consistently improves over SFT across three benchmarks and two model sizes (PyZQ, 4MkL, bSus).
  • The notebook model generalizes to longer task streams and unseen task models (PyZQ, bSus).

How we address the concerns

  • Novelty vs. MemoPilot (PyZQ): we evaluate MemoPilot's next-task credit within our framework and find that our design works better in our setting, where a notebook update often pays off several tasks later. We also clarify the differences in task setting and credit rule in the related work (Section 2, Section 6.1, Appendix A.4).
  • Separate ablations of the credit-assignment components (PyZQ): we conduct the suggested ablation, which isolates the causal suffix return and the position-matched baseline (Appendix A.4).
  • Robustness of the credit-assignment gains (PyZQ, bSus): we conduct the suggested experiment with a second independent training seed (Section 6.1, Appendix A.4).
  • Generalization across environments (PyZQ): we conduct the suggested cross-benchmark transfer evaluation (Appendix A.7).
  • Why learn a notebook rather than the task model (4MkL): we clarify the motivation and discuss unified models as future work.
  • Role of the teacher in SFT (4MkL): we show this with an additional run of RL without teacher SFT (Appendix A.5).
  • Compute and notebook size limit (4MkL): we clarify the training compute and report how often notebook updates reach the size limit (Appendix B).
  • Whether continued updating helps (bSus): we conduct the suggested experiment with a control that freezes the notebook after eight updates (Appendix A.6).
  • Post-task feedback (bSus): we add an evaluation with success/failure-only feedback, and we clarify the feedback setting for each benchmark with examples (Appendix A.8, Appendix C.1).

We are grateful for the reviewers' suggestions, which have strengthened the paper. All new experiments and clarifications are included in the revised PDF. We believe the revision addresses concerns raised in the reviews, and we hope the AC and the reviewers will take these improvements into account in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.