acceptodds
Under review as a conference paper at ICLR 2027

Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM Agents

10 views

Abstract

Agentic memory systems reuse past experience to improve future performance, yet most existing designs curate memory at write time: once a task is completed, its trajectory is distilled into a fixed artifact, such as a reflection, workflow, skill, or reasoning strategy, that is later retrieved by similarity. This forces the system to decide what is worth remembering before the future query is known, irreversibly discarding information and producing a query-independent summary that must serve many possible downstream tasks. Learning such a write-time curator is also difficult because the value of a storage decision may only become apparent when a relevant query arrives, potentially many tasks later, creating a long-horizon credit-assignment problem. We instead retain raw trajectories and defer curation until read time, when the current task is known. Given the retrieved traces and the new task, a memory curator synthesizes a compact, task-adaptive payload tailored to the immediate need. Because this payload is consumed on the same task, the curator can be trained directly from immediate task success, avoiding delayed utility signals and the need to artificially group related tasks. Across ALFWorld, WebShop, and -bench, our Just-in-Time Memory (JITMEM) consistently outperforms no-memory agents as well as heuristic and learned write-time memory methods, improving over the strongest baseline by 16.2, 16.3, and 3.9 absolute success-rate points, respectively. Notably, even an untrained curator is already competitive with or surpasses these baselines, showing that task-adaptive read-time curation itself is a major source of the gain; training the curator further compounds the improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Agentic memory, Memory curation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fGWuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of how the current agent systems reuse past experience to improve future performance. Most existing designs curate memory at write time: once a task is completed, its trajectory is distilled into a fixed artifact, such as a reflection, workflow, skill, or reasoning strategy, that is later retrieved by similarity. This forces the system to decide what is worth remembering before the future query is known, irreversibly discarding information and producing a query-independent summary that must serve many possible downstream tasks. Learning such a write-time curator is also difficult because the value of a storage decision may only become apparent when a relevant query arrives, potentially many tasks later, creating a long-horizon credit-assignment problem.

Instead, this study develops a JITMem strategy. It retains raw trajectories and defers curation until read time, when the current task is known. Given the retrieved traces and the new task, a memory curator synthesizes a compact, task-adaptive payload tailored to the immediate need. Because this payload is consumed on the same task, the curator can be trained directly from immediate task success, avoiding delayed utility signals and the need to artificially group related tasks.

Their experiments show that across ALFWorld, WebShop, and τ 2-bench, Just-in-Time Memory (JITMEM) consistently outperforms no-memory agents as well as heuristic and learned write-time memory methods, improving over the strongest baseline by 16.2, 16.3, and 3.9 absolute success-rate points, respectively. Notably, even an untrained curator is already competitive with or surpasses these baselines, showing that task-adaptive read-time curation itself is a major source of the gain; training the curator further compounds the improvement.

The study is interesting and timely, and based on my view, it does make progress on the agent handling and management although the curator may also be costly in its own which may need to be further examined in the discussion and evaluation.

Critical strengths

S1. The study poses a critical problem of the current agent system, that is, most existing designs curate memory at write time and distill the trajectory into a fixed artifact, such as a reflection, workflow, skill, or reasoning strategy, and later retrieved them by similarity. But such a strategy has a fundamental problem: The value of a storage decision may only become apparent when a relevant query arrives, potentially many tasks later, creating a long-horizon credit-assignment problem. This observation is good.

S2. The study works out a new strategy, called JITMem, which retains raw trajectories and defers curation until read time, when the current task is known. Given the retrieved traces and the new task, a memory curator synthesizes a compact, task-adaptive payload tailored to the immediate need. Because this payload is consumed on the same task, the curator can be trained directly from immediate task success, avoiding delayed utility signals and the need to artificially group related tasks.

S3: The performance study on three benchmark datasets, ALFWorld, WebShop, and τ 2-bench, shows JITMEM consistently outperforms no-memory agents as well as heuristic and learned write-time memory methods, improving over the strongest baseline 16.2, 16.3, and 3.9 absolute success-rate points, respectively. Notably, even an untrained curator is already competitive with or surpasses these baselines, showing that task-adaptive read-time curation itself is a major source of the gain; training the curator further compounds the improvement. This is very good progress.

Critical weaknesses

W1. The design is interesting. But when it defers curation until read time, when the current task is known. It may raise two problems: (1) how could you retrieve the previous traces that need to be curated now correctly and accurately? When the traces passed for quite some time, to correctly retrieve such traces may itself becomes a problem. This seems to be a concern, and (2) the cost of curation.

W2: Given the retrieved traces and the new task, a memory curator synthesizes a compact, task-adaptive payload tailored to the immediate need, this curation itself could be costly and nontrivial, which may need to be counted and joining into the cost study more seriously.

W3: How to curate in time, by following what standard, could be a theme of discussion. I also wonder whether there could be cases that the same retrieved cases should be curated differently based on different usages in later cases. What should be the standard and most efficient strategy to accomplish such a task.

Overall, the study is interesting and makes a good contribution to the generation of agent memory. I am a little on the borderline between weak accept and weak reject, depending on how the rebuttal could successfully address my concern above.

Questions for authors

I would like to see how the authors would address my concerns raised in the weakness. I am open for in-depth discussion of the paper.

5 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer fGWu (Part 1)

We thank the reviewer for their thoughtful review and are glad that you found our work "interesting and makes a good contribution to the generation of agent memory". We address each comment below.

 

W1. The design is interesting. But when it defers curation until read time, when the current task is known. It may raise two problems: (1) how could you retrieve the previous traces that need to be curated now correctly and accurately? When the traces passed for quite some time, to correctly retrieve such traces may itself becomes a problem. This seems to be a concern, and (2) the cost of curation.

We understand the concern to be that, as time passes, the bank accumulates more traces, so the relevant ones must be found among more and more candidates. Three properties of JITMEM address this.

First, retrieval does not depend on when a trace was stored. BM25 ranks stored trajectories only by how well their task descriptions match the current task, and the bank never discards entries, so an old trace is as retrievable as a recent one.

Second, a larger bank has not hurt performance in our experiments. In Table 5 of Section 4.2, we show that pre-filling the test bank with 100 additional trajectories from the training set changes SR by at most 1.3 points.

Third, read-time curation tolerates imperfect retrieval. The curator reads all retrieved trajectories together with the current task and first identifies which of them are relevant.

Regarding the choice of retriever, we follow prior work SkillOS [1], so that the comparison is fair and the performance gain can be attributed to task-adaptive read-time curation rather than to a better retrieval method. All methods use the same retriever BM25, with the current task description as the query and past task descriptions as the keys. We acknowledge that BM25 is a simple choice, and an embedding-based or model-based retriever could perform better. However, we chose it to keep the experimental setup fair, and we note this as a limitation in the conclusion of our submission.

 


 

W2: Given the retrieved traces and the new task, a memory curator synthesizes a compact, task-adaptive payload tailored to the immediate need, this curation itself could be costly and nontrivial, which may need to be counted and joining into the cost study more seriously.

We thank the reviewer for this point. We agree that curation cost should be counted explicitly. Table 11 below gives a cost breakdown of the curator and the executor. Costs are computed from the token counts in Table 12.

  • Curation is cheap. As shown in Table 11, the curator costs only \0.00065pertask,whichis1.70.00065 per task, which is 1.7% of JITMEM's total cost of \\0.0382. This is because the JITMEM curator produces a short payload once per task, with about 0.31K output tokens as reported in Table 12, and the curator model is cost-efficient.
  • Better curation reduces execution cost. As a result, JITMEM has the lowest total cost among all memory methods, \0.0382 per task, compared to \0.0684 for ReasoningBank and \0.0770forSkillOS−base.ItsexecutorcostisevenlowerthanNoMemory,at0.0770 for SkillOS-base. Its executor cost is even lower than No Memory, at \\0.0376 vs. \$0.0435. There are two reasons. (i) JITMEM generates compact, task-specific payloads, so it injects much less context into the executor prompt than the other memory baselines, with 9.8K input tokens compared to 19.7K and 22.4K. (ii) The payloads are adaptive and useful, so, compared to No Memory, the executor completes tasks in 11.6 steps instead of 17.8 and generates 0.87K output tokens instead of 1.40K.

All experiments are on ALFWorld with a Qwen3-8B curator and a GPT-5.4 executor. Costs are based on \0.2per1MinputandoutputtokensforQwen3−8B[2],and0.2 per 1M input and output tokens for Qwen3-8B [2], and \\2.5 per 1M input tokens and \$15 per 1M output tokens for GPT-5.4 [3].

Table 11. Cost breakdown per task (USD) on ALFWorld with Qwen3-8B curator and GPT-5.4 executor. Costs are computed from the token counts in Table 12.

MethodCuratorExecutorTotal
No Memory–0.04350.0435
ReasoningBank0.000240.06820.0684
SkillOS-base0.000580.07640.0770
JITMEM-base0.000650.04230.0429
JITMEM0.000650.03760.0382

Table 12. Token breakdown per task (K) on ALFWorld with Qwen3-8B curator and GPT-5.4 executor. Columns marked with * are reported in Table 4 of the submission.

MethodCurator InCurator OutExecutor In*Executor Out*Steps*
No Memory––9.01.4017.8
ReasoningBank1.040.1519.71.2616.2
SkillOS-base2.670.2322.41.3616.9
JITMEM-base2.950.2910.91.0013.2
JITMEM2.940.319.80.8711.6

Reference

[1] Ouyang et al. SkillOS: Learning Skill Curation for Self-Evolving Agents

[2] https://pricepertoken.com/pricing-page/model/qwen-qwen3-8b

[3] https://developers.openai.com/api/docs/models/gpt-5.4

AuthorsOpenReview ↗

Authors' Response to Reviewer fGWu (Part 2)

W3: How to curate in time, by following what standard, could be a theme of discussion. I also wonder whether there could be cases that the same retrieved cases should be curated differently based on different usages in later cases. What should be the standard and most efficient strategy to accomplish such a task.

We thank the reviewer for this question.

Standard. In JITMEM, the curation standard is not hand-designed but learned from the task reward. Since the curated payload is applied to the same task it was produced for, we train the curator with GRPO using the task success of the executor as the reward.

Strategy. Curating the same retrieved trajectory adaptively based on the purpose of each later task is the design of JITMEM. Specifically, the curator reads the retrieved traces together with the current task, and then synthesizes a compact, task-conditioned payload. Because the curator knows the current task, it extracts the specific segments useful for that task, so the same stored trajectory can yield a different payload for each later tasks. This strategy parallels the cognitive-science view that episodic memory is reconstructive rather than replayed, with retrieval shaped by current goals and cues [1]. This discussion is in lines 50 to 84 of the submission.

Our case study in Figure 3 and Section 4.3 shows that the same retrieved trajectory is curated differently for two later tasks by JITMEM. Both tasks retrieve the trajectory "put a cool egg in microwave", a sequence of steps that includes a state-transition step (e.g., cooling an object) as well as object localization and placement steps. Each later task draws strategies from different steps. For "put a hot potato in fridge", the curator focuses on the state-change step and specializes it into guidance for heating the potato with a microwave. For "put a newspaper in sofa", it instead emphasizes the placement steps, curating guidance such as verifying the target location.

Reference

[1] Schacter and Addis. The cognitive neuroscience of constructive memory: remembering the past and imagining the future

AuthorsOpenReview ↗

A gentle reminder that the deadline for the author–reviewer discussion is approaching.

Dear Reviewer fGWu,

Thank you for your constructive feedback, which helped us improve our paper. As the discussion period is closing soon, we would like to follow up and see if the Reviewer has had a chance to consider our response. We have provided additional results and clarifications in our response. Please let us know if we should include anything further in the revised draft. We are more than happy to clarify if anything is unclear.

Best regards,

Authors

Reviewer_YiX6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduced JitMem (Just-in-Time Memory), moving memory curation for LLM agents from write time to read time, like a shift. Memoyr bank stores raw trajectories without any distillation. The BM25 retriever fetches the top K trajectories and a curator model will synthesize a task-adaptive payload conditioned on the current task, then inject into a frozen executor's context. A curator (Qwen3 8B) can be trained with GRPO using immediate task reward from the frozen executor, avoiding the delayed credit assignment.

Experiments on ALFWorld, Webshop, and tau-2 bench showed some amazing gains over multiple model settings.

Critical strengths

*Simple idea but works. The basic experimental frame is pretty clean and well-motivated.

  • It's indeed an algorithmic trick; deferring curation to read time removes the delayed-utility credit assignment problem of learned write-time memory, allowing a curator to be trained with only the immediate task reward and no task grouping.
  • Figure 5-6 showed stable learning curves, and promising results.
  • Evaluation is broad. 3 popular benchmarks and 3 executors.
Critical weaknesses

*Gemini 2.5 Pro is definitely too old, a model from 2025. But GPT 5.4 and Qwen3 8B kind of offset this issue.

  • The baseline comparisons are not fully controlled: RL-trained curator + no retrieved traj still can reach 62.6 SR on ALFWorld (above 60.5 untrained JitMem-base), suggesting that GRPO on each benchmark's training split teaches the curator benchmark-specific hints. Figure 4 also kind of suggested this. However, a curator trained with GRPO + no retrieval would be needed to separate "learning to distill memory" from "learning environment-specific prompts", I believe.
  • Table 1's no memory rows for Qwen3-8B and Gemini2.5Pro match the numbers in SkillOS, instead of the author's reproductions in Table 6. This is kind of vague. It's unclear which baselines are reproduced versus copied, and whether all methods share the sample prompts.
Questions for authors
  1. which numbers in Tables 1, 2, and 9 are copied from previous work, and which are your owns? Are ReasoningBank, MemP, and SkillOS all re-run with the same executor prompts (including the added WebShop search guidance), the same thinking settings, the same retriever, and the same streaming/batching protocol as JitMem?
  2. Benchmarks seem correlate and help with each other, e.g., ALFWorld => WebShop. Should we set a separate curator per environment/docker?
  3. Can you run experiment, a curator trained with GROP but without retrieved traj and test the training and test time? It's mainly to quantify how much RL gain comes from distilling memory vs. learning benchmark-specific guideance.
  4. Newer Gemini models?
7 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer YiX6

We thank the reviewer for their positive review and are glad that you found our work "Simple idea but works", "pretty clean and well-motivated". We address each comment below with additional experiments and clarification.

 

W1 The baseline comparisons are not fully controlled: RL-trained curator + no retrieved traj still can reach 62.6 SR on ALFWorld (above 60.5 untrained JitMem-base), suggesting that GRPO on each benchmark's training split teaches the curator benchmark-specific hints. Figure 4 also kind of suggested this. However, a curator trained with GRPO + no retrieval would be needed to separate "learning to distill memory" from "learning environment-specific prompts", I believe.

Q3 Can you run experiment, a curator trained with GROP but without retrieved traj and test the training and test time? It's mainly to quantify how much RL gain comes from distilling memory vs. learning benchmark-specific guideance.

We thank the reviewer for suggesting this experiment and agree that it is needed to understand this point. We train the curator with GRPO without retrieved trajectories, keeping all other settings unchanged such as the number of RL steps. We then test this curator both without and with retrieved trajectories. We ran it on WebShop with the Qwen3-8B executor and curator.

The results show that most of JITMEM's gain comes from learning to distill memory. JITMEM improves the SR of No Memory from 9.8 to 32.8, while the curator trained without retrieval only improves it to 18.9. Moreover, giving this curator retrieved trajectories at test time does not help and even lowers its SR to 15.6, which shows that it has not learned to use memory.

This is also corroborated by our results of testing JITMEM without retrieval at test time. Its SR drops from 32.8 to 17.6, a loss of 15.2 points. This shows that RL training mainly gives the JITMEM curator the ability to distill retrieved trajectories, rather than the ability to generate benchmark-specific prompts.

Table 5. Ablating trajectory retrieval during RL training and at test time on WebShop with Qwen3-8B as executor and curator.

MethodSR (mean ± std)
No Memory9.8 ± 0.5
RL-trained w/o retrieved traj, test w/o retrieved traj18.9 ± 1.0
RL-trained w/o retrieved traj, test w/ retrieved traj15.6 ± 1.0
JITMEM (RL-trained w/ retrieved traj, test w/o retrieved traj )17.6 ± 1.1
JITMEM (RL-trained w/ retrieved traj, test w/ retrieved traj)32.8 ± 1.7

 


 

W2 Table 1's no memory rows for Qwen3-8B and Gemini2.5Pro match the numbers in SkillOS, instead of the author's reproductions in Table 6. This is kind of vague. It's unclear which baselines are reproduced versus copied, and whether all methods share the sample prompts.

Q1 which numbers in Tables 1, 2, and 9 are copied from previous work, and which are your owns? Are ReasoningBank, MemP, and SkillOS all re-run with the same executor prompts (including the added WebShop search guidance), the same thinking settings, the same retriever, and the same streaming/batching protocol as JitMem?

We clarify that all methods share the same experimental setup to ensure a fair comparison. We re-run all baselines under the same setup as JITMEM, including the same executor prompts (with the WebShop search guidance for Qwen3-8B executor), the same thinking settings, the same BM25 retriever, and the same batch size. For every baseline, both no-memory and memory-based, we have both our reproduced number and the number reported by SkillOS, and we always report the higher of the two in all comparisons. This keeps our gains conservative. It is also why, as the reviewer pointed out, the no-memory numbers in Table 1 match SkillOS while our reproduced numbers in Table 6 are lower.

 


 

Q2 Benchmarks seem correlate and help with each other, e.g., ALFWorld => WebShop. Should we set a separate curator per environment/docker?

In our current setup, we train a separate curator for each benchmark. The JITMEM curators are trained on the ALFWorld and WebShop training sets separately and evaluated on their own test sets. We do not train a shared curator across benchmarks.

 


 

Gemini 2.5 Pro is definitely too old, a model from 2025. But GPT 5.4 and Qwen3 8B kind of offset this issue. Q4 Newer Gemini models?

We use Gemini-2.5-Pro to align our experimental setup with SkillOS [1] and ensure a fair comparison with the baseline methods. As the reviewer noted, we also include the more recent GPT-5.4 and Qwen3-8B as executors, and JITMEM shows consistent gains across all three executors.

Reference

[1] Ouyang et al. SkillOS: Learning Skill Curation for Self-Evolving Agents

I thank the authors for taking care of my review, and for the detailed response and additional experiment.

  • W1/Q3 - new ablation addressed my main concern. So the wording “most of the gain” should be stated with this decomposition. Since my original concern arose from the ALFWorld results (62.6 vs. 60.5), I encourage the authors to include the same ablation for ALFWorld in the final version.
  • W2/Q1 - appreciate the clarification. Reporting the higher of the reproduced and SkillOS-reported numbers for each baseline is conservative, and I probably find this acceptable. However, this protocol is not described in the paper. Please state it explicitly and mark which entries in Tables 1, 2 and 9 are taken from SkillOS (including their std) versus reproduced, ideally reporting both numbers in the appendix.
  • Q2/Q4 - clarifications noted.

Overall, my concerns are largely addressed, and I tend to maintain my positive assessment (weak accept). I would to let AC/PC and other reviewers support acceptance if the clarifications above are further incorporated.

AuthorsOpenReview ↗

Authors' Further Response to Reviewer YiX6

We thank the reviewer for the follow-up and the positive assessment. The draft has been updated to incorporate the clarifications, with changes highlighted. We stated “most of the gain comes from learning to distill memory” with the results decomposition at L431 and added the protocol description at L949. The source of each baseline result is now annotated in L1085 Table 10 (previously Table 9). This table collects all ablation and baseline results from the main text. We will include the additional ablation study in the final version.

Thank you for your constructive feedback, which has helped improve our paper.

Reviewer_tHvpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that agentic memory should be curated at read time, when the current task is known, rather than at write time. JITMEM keeps a bank of raw, unabstracted trajectories (admitted only when an executor-as-judge deems the task solved), retrieves the top three by BM25 over task descriptions, and runs a separate curator model that reads the retrieved traces together with the current task and writes a compact task-conditioned briefing, which is prepended to a frozen executor’s prompt. Because the briefing is consumed on the same task that produced it, the curator can be trained with GRPO from the executor’s immediate task reward, which removes the delayed-credit and task-grouping machinery that learned write-time curators such as SkillOS need. Experiments on ALFWorld, WebShop, and τ²-bench with Qwen3-8B, Gemini-2.5-Pro, and GPT-5.4 executors compare against no-memory, ReasoningBank, MemP, and SkillOS (untrained, frontier-prompted, and RL-trained variants).

Critical strengths
  1. The credit-assignment reformulation makes curator training simple, stable, and transferable. After the reformulation, the reward is a direct function of the payload for the same task, so training needs only the task reward, with no auxiliary judge model and no task grouping.
  2. The ablations are pretty thorough.
Critical weaknesses
  1. The No Memory rows in Table 1 are identical to the SkillOS paper’s numbers in Table 6, while the authors’ own reruns of the same setting come out up to 5.8 points lower. Also JITMEM uses a different inference mode, a modified WebShop prompt, and a different evaluation protocol. To be honest I don't know if the reported gains are trustworthy and consistent.
  2. When retrieval is turned off entirely, the RL-trained curator still reaches 17.6 SR with Qwen3-8B, beating every baseline including trained SkillOS (16.5). In other words, the curator learned useful WebShop strategies during training and can write helpful plans without looking at any stored experience, which makes me question if much of the WebShop improvement does not come from memory at all.
  3. With Qwen3-8B on WebShop, the untrained JITMEM scores below the no-memory agent and below every memory baseline.
Questions for authors
  1. Could you add two simple experiments, one is testing with untrained curator given only the task and no retrieved trajectories, and the other is feeding the retrieved raw trajectories straight to the executor with no curator at all.
4 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer tHvp (Part 1)

We thank the reviewer for their thoughtful review. We address each comment below with additional experiments and clarification.

W1 The No Memory rows in Table 1 are identical to the SkillOS paper’s numbers in Table 6, while the authors’ own reruns of the same setting come out up to 5.8 points lower. Also JITMEM uses a different inference mode, a modified WebShop prompt, and a different evaluation protocol. To be honest I don't know if the reported gains are trustworthy and consistent.

Our experimental setup follows SkillOS as closely as the information in their paper allows. All methods share the same evaluation protocol, and it is identical to that of SkillOS: the memory bank is updated with a batch size of 10, retrieval uses BM25. We run each evaluation three times with shuffled task orders, and report mean and std. For every baseline, both no-memory and memory-based, we have our reproduced number and the number reported by SkillOS, and we always report the higher of the two in the method comparison. This is why, as the reviewer pointed out, the no-memory numbers in Table 1 match SkillOS but our reproduced numbers (Table 6, copied below) are lower. Since JITMEM is evaluated in the setup that yields these lower non-memory scores but is compared against the higher reported ones, its gains are measured conservatively.

The differences of inference mode and webshop prompt the reviewer points out stem from settings that SkillOS leaves unspecified for the Qwen3-8B executor. There are no such "differences" with GPT-5.4 and Gemini-2.5-Pro as executors.

  • Inference mode. SkillOS does not specify its inference settings (e.g., thinking vs. non-thinking mode for Qwen3-8B), and we could not find them in any public resource. We therefore ran the no-memory baseline in both modes and used the mode that comes closest to their reported numbers for JITMEM.
  • As already clarified in line 882 of submission, the default thinking-format prompt caused degenerate outputs and had to be removed. With this change, our reproduced no-memory SR was still far below the reported 9.8. We therefore added a short search-guidance paragraph and calibrated it so that the no-memory SR becomes closer to the score reported by SkillOS but remains below it. Thus, JITMEM gains no advantage from this prompt. All methods with the Qwen3-8B executor share this prompt, including the no-memory baseline and our reruns of ReasoningBank, MemP, and SkillOS, so the comparison is fair.

We have added this clarification in the line 944 of the revised paper.

Table 7. No-memory baseline reproduction. “Reported” from SkillOS, “Reproduced” from our runs. In all cases, our reproduced performance is at or below the reported results, ensuring gains are measured conservatively.

SourceALFWorld SRWebShop ScoreWebShop SR
Executor: Qwen3-8B
SkillOS Reported47.933.39.8
Reproduced (non-thinking)34.536.48.6
Reproduced (thinking)42.129.04.4
Executor: Gemini-2.5-Pro
SkillOS Reported66.448.638.4
Reproduced63.647.337.5

 


 

W2 When retrieval is turned off entirely, the RL-trained curator still reaches 17.6 SR with Qwen3-8B, beating every baseline including trained SkillOS (16.5). In other words, the curator learned useful WebShop strategies during training and can write helpful plans without looking at any stored experience, which makes me question if much of the WebShop improvement does not come from memory at all.

We clarify that most of the WebShop improvement comes from distilling raw experience. The table below shows the results from Figure 2 of our submission. As the reviewer notes, without retrieved trajectories, the RL-trained curator improves over SkillOS by only 1.1 points. When the same curator has access to retrieved trajectories and synthesizes its payload from them, the improvement rises to 16.3 points. This means that 93.25% of the gain comes from distilling stored trajectories.

MethodSR
SkillOS (RL-trained)16.5
JITMEM (RL-trained w/o retrieved traj)17.6 (+1.1)
JITMEM (RL-trained + retrieved traj.)32.8 (+16.3)
AuthorsOpenReview ↗

Authors' Response to Reviewer tHvp (Part 2)

W3 With Qwen3-8B on WebShop, the untrained JITMEM scores below the no-memory agent and below every memory baseline.

We first clarify that on WebShop with the Qwen3-8B executor, the untrained JITMEM-base improves the success rate of the no-memory agent from 9.8 to 11.7, outperforms ReasoningBank (11.4), and is comparable to MemP (12.0). We acknowledge that it underperforms the other memory baselines in this setting. However, this is the only one of the 7 settings across 3 benchmarks and 3 executors where JITMEM-base trails the baselines. In all other settings, it outperforms every training-free baseline that uses the same Qwen3-8B curator.

This is likely because synthesizing task-adaptive memory zero-shot is challenging when the stored trajectories are noisy, as in this setting. On WebShop, the Qwen3-8B executor succeeds on only about 10% of tasks. Following SkillOS, the same executor model also serves as the judge, and we find that it misjudges 47.4% of the trajectories. As a result, the memory bank and the retrieved trajectories contain many mistakes, and an untrained 8B curator struggles to extract useful information from them. However, RL training improves the same Qwen3-8B curator from 11.7 to 32.8 SR, outperforming all baselines including RL-trained SkillOS (16.5). In addition, when the same untrained Qwen3-8B curator is paired with a stronger executor and judge on WebShop, such as Gemini-2.5-Pro and GPT-5.4, it achieves higher SR than all baselines.

 


 

Q1 Could you add two simple experiments, one is testing with untrained curator given only the task and no retrieved trajectories, and the other is feeding the retrieved raw trajectories straight to the executor with no curator at all.

We thank the reviewer for suggesting these experiments. We report the results in the table below and have added them to the revised paper.

For the first experiment, “JITMEM-base (w/o retrieved traj.)”, we give the untrained curator only the current task, with no retrieved trajectories. It reaches 63.3 SR, with no clear gain over the no-memory baseline 62.6 SR. This confirms that the gain of JITMEM-base comes from distilling past trajectories rather than from the parametric knowledge of the curator.

For the second experiment, “Trajectory-in-Context”, we place the retrieved raw trajectories directly in the executor’s context, with no curator. This reaches 73.8 SR, which is 5.5 points below JITMEM-base and 12.9 points below JITMEM. This baseline is also more expensive. JITMEM and JITMEM-base use a cheap 8B curator to compress the raw trajectories into a compact payload of about 0.3K tokens per task. In contrast, Trajectory-in-Context feeds the raw trajectories directly into the expensive GPT-5.4 executor, and these are roughly 9x longer than the payload.

Table 13. Results on ALFWorld with GPT-5.4 executor. The experiments use the same setup as main experiments.

MethodSR (mean ± std)
No Memory62.6 ± 0.3
JITMEM-base (w/o retrieved traj.)63.3 ± 2.2
Trajectory-in-Context73.8 ± 3.6
MemP72.6 ± 1.7
SkillOS70.0 ± 3.3
ReasoningBank77.9 ± 4.2
JITMEM-base79.3 ± 3.6
JITMEM86.7 ± 0.7

Thank you authors for the detailed response and the additional experiments. The clarification that all baselines were re-run with the shared prompt and that Table 1 reports the higher of reproduced and reported numbers resolves my main concern about baseline fairness. The two new baselines in Table 13 confirm that the untrained curator alone does not help and that curation outperforms placing raw trajectories in context, and the judge-noise explanation for the Qwen3-8B WebShop setting is quite informative.

Given that these new experiments have resolved most of my original concerns, I have raised my soundness assessment from 2 to 3.

Dear Reviewer tHvp,

Thank you for your positive feedback and for raising the rating to 3. We sincerely appreciate your time.

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.