acceptodds
Desk rejected at ICLR 2027

Joint Agent Memory and Exploration Learning via Novelty Signals

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Exploration and memory are two fundamental capabilities for autonomous agents in open-ended environments. However, both are difficult to learn. Effective exploration requires reasoning about the possible consequences of actions and identifying actions that can lead to new states, while a good memory system needs to compress large-scale interaction histories into a form that is easy for an agent to use. Instead of considering these two capabilities separately, we observe that memory and exploration form a mutually dependent loop: sustained exploration requires memory to distinguish exhausted behaviors from unseen ones, while novelty-seeking interactions provide supervision for learning useful memory representations. Based on this insight, we introduce **J**oint **A**gent **M**emory and **E**xploration **L**earning (**JAMEL**), a framework that jointly trains agentic memory and an exploration policy through novelty-driven interaction. JAMEL compresses interaction histories into latent memory tokens and jointly trains memory and policy through novelty-guided rejection sampling fine-tuning on filtered exploration prefixes. We establish evaluation settings for open-environment exploration across GUI, navigation, and household domains. Across these domains, JAMEL consistently outperforms the evaluated baselines, supporting the effectiveness of jointly learning memory and exploration through novelty signals.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Memory, Latent Memory, Exploration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HivVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework that jointly trains agentic memory and an exploration policy termed JAMEL. The core idea is to compress previous interactions into memory tokens, and then learn an exploration policy by designing a novelty reward based on that memory. The history compression is done by a VLM, where the last token’s hidden state is used to represent that memory. The method is evaluated in several domains, with comparable or better results compared to baselines.

Critical strengths
  • The overall idea is simple, though some training details are missing.
  • While on several domains JAMEL is comparable to ReAct, it significantly outperforms it on Navigation.
  • Limitations are discussed.
Critical weaknesses
  • Novelty reward definition: it is not entirely clear in the main text how “novelty” is defined per-domain. The authors mention that it “determines whether the current interaction introduces behavior not recorded in the accumulated history”, but it is not clear what that actually means. An example in one of the domains would have been helpful. This information is found in Appendix A, and it seems that designing the reward requires privileged domain/environment information and manual tuning. This is in contrast to “classic” exploration methods that are using general signals such as prediction error (e.g., with world models).
  • Abstract: it was hard to understand what domain or field this work falls under. In addition, the abstract contains repeated sentences (“jointly trains agentic memory and an exploration policy through novelty-driven interaction”). As a reader, it was hard for me to understand what to expect. For example, the utilization of VLMs should be mentioned.
  • Stages: it is unclear what components are pre-trained, how the policy is structured and trained, and what components it uses. It is not clear how training is done, for how long, what algorithms etc.
  • Complexity: It is not entirely clear how many parameters JAMEL has, how many are trained or fine-tuned (is it 2B +7B = 9B?) . Unclear if the baselines have similar numbers of parameters and if the comparison is compute-matched. It is also not clear whether the baselines are trained in capacity or taken as-is.
  • Latent vs. text memory: it seems that the gains of using learned latent memory are not significant compared to just using a text summary.
  • No examples of exploration behavior: the authors do not demonstrate what exploration looks like in each domain. I’d expect a sample output of the VLM when something novel is discovered.
  • Overall impression: given the text-summary ablation, I’m not entirely convinced the method is necessary and that careful prompt engineering cannot achieve similar results. In addition, my general impression of the writing is that several parts require more clarification as I mentioned above. I appreciate the authors’ candor regarding the utilization of AI in ideation and writing though.
Questions for authors
  • Please address the weakness above.
  • What is the intended application of the proposed method given the new iterations of large-scale models (e.g. Opus, Astra)? Do the authors think that JAMEL will be useful as future iterations of QWEN improve? My concern here is whether this is just a temporary technique to stitch a problem that would be solved in the next iterations of large-scale agents.
6 comments
AuthorsOpenReview ↗

1. Definition of novelty and the information it requires (Weakness 1)

We sincerely thank you for your effort in reviewing our paper and for your candid assessment. During the discussion period, we revised the abstract and the method description, added the training details and model sizes, added an ablation on novelty-based filtering and representative exploration examples, and clarified the comparison with textual memory. We address each point below.

What novelty means, with an example. In every domain, an action receives rt=1r_t = 1 if, and only if, it reveals at least one behavior unit that has not been recorded earlier in the same episode. In the GUI domain, the unit is executed JavaScript code: clicking a tab that loads a new page component executes code that has not run before and receives a reward, whereas clicking the same tab again executes only already-covered code and receives zero. In navigation, the unit is an object instance: rotating to bring a previously unseen mug into view receives a reward; seeing it again does not. In ALFWorld, the unit is a coverage atom such as a new location type, object type, or interaction: opening a drawer that reveals a new object type receives a reward.

This design has two benefits: because a discovered unit stays familiar, repeating anything already tried earns nothing, so the reward directly requires the policy to remember its own history; and because the signal is binary and read from environment state rather than from a learned predictor, it is consistent across actions of very different scale and cannot be inflated by unpredictable but uninformative observations, a known weakness of prediction-error curiosity.

Privileged information and manual tuning. We would like to clarify that code coverage (GUI) and newly observed objects (navigation) are environment-native state feedback, not human-annotated rewards. Code coverage is reported by the JavaScript engine of any instrumented application without modifying the application, and object visibility is simulator metadata. No per-environment tuning is involved: the same rule is applied unchanged to all 96 applications and all RoboTHOR scenes. The signal does not require a learned world model or a task reward. The rule is defined once per domain, not per environment, so it generalizes across a whole domain: the same code-coverage rule applies to any web application, since every modern browser can report JavaScript coverage, and the same object-discovery rule applies to any simulated indoor scene, since simulators expose object visibility.

AuthorsOpenReview ↗

2. Abstract, training stages, and model complexity (Weaknesses 2–4)

Domain of the work and the abstract. Thank you sincerely for pointing this out. JAMEL is a general learning method for LLM/VLM agents, and its subject is how an agent's memory and its exploration policy can be learned jointly. It is motivated by the observation that the two are mutually dependent: sustained exploration requires memory to tell exhausted behaviors from unseen ones, and novelty-seeking interaction in turn supervises what memory should retain. GUI, navigation, and household environments are therefore testbeds, not the field of the work; applying essentially the same method to all three is meant to show that it is not tied to a particular domain. We have rewritten the abstract to state this positioning up front, to mention the use of vision-language models, and to remove the repeated sentence. We have added this in the revised paper.

Stages and training. JAMEL is trained by novelty-based filtering of exploration trajectories followed by supervised fine-tuning of the model on the retained steps, so that memory and exploration are learned jointly from the same supervision without reinforcement learning. The settings are:

ScaleWoBRoboTHORALFWorld
Policy / compressorQwen2.5-VL-7B-Instruct / Qwen3-VL-2B-Instructsamesame
Training environments86 applications60 scenes1,200 games
Training samples24k28.8k13k
OptimizerAdamW, lr 2e-5, cosine, warm-up 0.05, weight decay 0.01AdamW (β = 0.9, 0.95), lr 4e-5, cosine, warm-up 0.05, weight decay 0.01same as ScaleWoB
Global batch size / optimizer steps16 / 4,39828 / 1,35032 / 812
Maximum sequence length8,1922,0483,072

We have added this in the revised paper.

Complexity. The name JAMEL-9B counts the 7B policy, the aligner and the 2B compressor. All exploration baselines are used as released, without further training: the open-weight GUI baselines (Qwen3-VL-8B-Instruct, MAI-UI-8B, GUI-Owl-1.5-8B) are of comparable size to JAMEL's 7B policy, and the RoboTHOR and ALFWorld baselines use the same Qwen2.5-VL-7B-Instruct backbone. The downstream ReAct baselines are fine-tuned on the same trajectories as JAMEL.

AuthorsOpenReview ↗

3. Latent versus text memory, examples of exploration, and prompt engineering (Weaknesses 5–6, Overall impression)

Latent versus text summary. We agree that the text-summary variant is a strong alternative. On ScaleWoB, latent memory covers 24% more code bytes (5,409,879 vs. 4,354,632) at similar novelty, and on RoboTHOR it improves both novelty and coverage (6.1 / 27.7% vs. 5.9 / 26.3%), while costing one token per past step instead of a regenerated summary. On ALFWorld, the two are tied (35.6 vs. 35.1 novelty, a gap smaller than that between the seen and unseen splits of the same model). This is expected: ALFWorld is a highly symbolic, text-only environment in which one observation–action pair takes only about 50-70 tokens and a per-step summary about 27 tokens, so a short text summary loses almost nothing; latent memory pays off when the history is long and multimodal, as in the GUI and navigation domains. Importantly, the text-summary variant is trained with the same novelty-guided procedure, so the contribution of the paper—novelty signals as supervision for memory and exploration—does not depend on which memory format is chosen.

Is careful prompting sufficient? The text-summary variant is not a prompted agent. It is trained with exactly the same novelty-guided filtering and supervised fine-tuning as JAMEL; only the memory format differs. Its strong results therefore come from our novelty supervision, not from prompting. Two results show that prompting alone does not achieve the same:

  1. Carefully engineered open-source agents fall far behind. MAI-UI and Mobile-Agent-v3.5, which use elaborate prompts with explicit notes or sliding-window memory, reach only 8.4 and 5.9 novelty on the held-out applications, versus 20.7 for JAMEL with a 7B policy under the same budget.
  2. Removing novelty supervision while keeping the prompt unchanged loses most of the gain in all three domains (our ablation on novelty-based filtering).
DomainMetricJAMEL (novelty-filtered)w/o novelty filtering
ScaleWoBNovelty / Coverage (Bytes)20.7 / 5,409,87915.4 / 5,188,235
RoboTHORNovelty / Coverage (%)6.1 / 27.73.6 / 16.7
ALFWorldNovelty / Coverage35.1 / 69.917.4 / 47.1

The improvement therefore comes from what the policy is trained to imitate, which prompt engineering cannot change.

Examples of exploration behavior. We have added annotated rollouts for all three domains. In each, JAMEL keeps moving to parts of the environment it has not yet visited (new GUI views after a reset, new rooms in RoboTHOR, new receptacles in ALFWorld), whereas the baselines tend to revisit what they have already seen. Please see Appendix F of the revised paper for the full rollouts.

Reviewer_AgqSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents JAMEL, a framework that jointly learns an exploration policy and a compact latent memory. JAMEL compresses each past observation–action pair into a single latent token and conditions the policy on these tokens alongside the current observation. To identify useful exploration trajectories for rejection fine-tuning, the framework employs domain-specific binary novelty signals: code coverage in GUI environments, newly observed objects in navigation tasks, and structured coverage atoms in household environments. The method is evaluated on ScaleWoB, RoboTHOR, and ALFWorld. Overall, JAMEL achieves greater exploration novelty and coverage than the evaluated baselines, with particularly clear improvements in navigation and household environments. The paper also examines downstream task performance, alternative memory designs, training-data scaling, novelty accumulation, token usage, and application-level exploration patterns.

Critical strengths
  • The idea that novelty signals can be used to supervise both exploration behavior and memory retention is intuitive and potentially useful. Applying essentially the same approach across GUI, navigation, and household environments suggests that the method is not tied to a particular domain.
  • The paper is generally well organized. The memory ablations, data-scaling study, novelty curves, token-usage analysis, and per-application results provide useful insight into the method’s empirical behavior. These analyses strengthen the evaluation beyond the main benchmark results.
Critical weaknesses
  • The introduction appears to contrast retaining complete interaction histories with using agent memory, but these approaches are not necessarily mutually exclusive. When a trajectory fits within the context window, the complete history can often be retained while still benefiting from KV-cache reuse. JAMEL, by contrast, compresses every interaction from the outset and requires an additional compressor. Because the paper reports input-token counts but not end-to-end latency or computational cost, it remains unclear whether the method provides meaningful efficiency gains in practice. The paper should also state more explicitly that the evaluated memory is episode-local rather than a form of long-term memory shared across tasks.
  • The current presentation makes it difficult to develop a concrete intuition for the exploration tasks. One or two representative rollout examples would help illustrate what the agent discovers and how its memory influences subsequent decisions. In addition, the ALFWorld evaluation primarily compares JAMEL against relatively old or modest baselines, such as ReAct, Reflexion, and AdaPlanner. This makes it difficult to assess JAMEL’s advantages over more recent memory-based or exploration-oriented agents.
Questions for authors
  • Is JAMEL primarily intended as memory within a single episode, or as long-term memory across tasks? If it is episode-local, why compress interactions from the first step rather than retain the exact history until the context window becomes limiting? How is KV-cache reuse handled in the reported implementation?
  • Can the authors report end-to-end inference latency or computational cost, including the compressor, and compare against a same-backbone full-history baseline? Also, are more recent memory or exploration baselines applicable to ALFWorld? If not, a clearer justification of the current baseline selection would be helpful.
6 comments
AuthorsOpenReview ↗

1. Scope of the memory, when to compress, and KV-cache reuse (Weakness 1, Question 1)

We sincerely thank you for the detailed review and for recognizing the generality of novelty supervision across the three domains and the value of the analyses. During the discussion period, we clarified the scope of the memory, added an efficiency analysis beyond input tokens, added representative rollout examples, and explained the baseline selection on ALFWorld. We address each point below.

Scope of the memory. JAMEL's memory is an environment-specific exploration memory: it accumulates what the agent has tried and discovered in the current environment, and persists throughout the exploration process. This knowledge is task-agnostic and transfers across tasks in the same environment, as shown by our downstream task experiments; what it does not do is transfer across environments, which we leave to future work. We have added this in the revised paper.

Why compress from the first step. Retaining the exact history and compressing it are indeed not mutually exclusive. We compress from the first step for two practical reasons. (i) Uniform cost: each new observation–action pair is compressed once by the 2B compressor, and the policy input grows by one token per step, so per-step cost is essentially constant over a 50-step GUI episode or a 300-step navigation episode. (ii) Multimodal histories exceed the context quickly: in ScaleWoB, a ReAct agent with the exact history already averages 37,877 input tokens per step over a 50-step episode, and a 300-step RoboTHOR episode with per-step images far exceeds the training context. A hybrid that keeps the exact recent history and compresses the rest is compatible with JAMEL. We have added this in the revised paper.

KV-cache reuse. In our reported evaluations, every step is an independent request for all methods, which is the setting of our token counts. Since prefix caching can amortize the prefill cost of a full-history agent, we additionally measured the same-backbone ReAct agent with and without prefix KV-cache reuse on three 50-step ScaleWoB episodes (one A100-80GB, batch size 1):

MethodPrompt tokens prefilled per stepLatency per step (s)Compute per step (TFLOPs)KV cache per episode at step 50
ReAct, no reuse26,0523.82771.7–
ReAct, prefix KV-cache reuse2,8390.9774.52.8 GB
JAMEL1,7210.7737.0< 0.1 GB

Prefix caching removes most of the re-prefill cost, but does not remove the dependence on history length: the latency of the full-history agent still grows with the episode, its KV cache must be kept for every active episode, and the complete prompt is still processed (and, for hosted models, billed) as input. JAMEL's prompt and cache stay nearly constant throughout the episode. Please see Appendix B of the revised paper for the per-step breakdown (Tables 7–8 and Figure 5). We have added this in the revised paper.

AuthorsOpenReview ↗

2. End-to-end latency and computational cost (Weakness 1, Question 2)

Thank you for this suggestion. We measured per-step inference latency for JAMEL and for a same-backbone ReAct agent under identical conditions (one A100-80GB, bf16, batch size 1), replaying three 50-step ScaleWoB episodes (150 decisions per configuration). JAMEL runs through its evaluation code path, in which the compressor encodes the latest observation–action pair (earlier pairs are cached) and the Qwen2.5-VL-7B-Instruct policy generates the action. The ReAct agent uses the Qwen2.5-VL-7B-Instruct backbone without memory and receives the complete history in its prompt. It is timed both as stateless requests and with prefix KV-cache reuse (only the new suffix is prefilled, as in automatic prefix caching).

MethodPrompt tokens (mean / step 50)Latency per step: all / steps 1–10 / steps 41–50 (s)KV cache per episode at step 50Peak GPU memory
JAMEL (compressor 0.10 s + policy 0.63 s)1,721 / 1,8310.77 / 0.73 / 0.78< 0.1 GB20.7 GB
ReAct, stateless26,052 / 52,3273.82 / 0.81 / 7.38–32.2 GB
ReAct, prefix KV cache26,052 / 52,3270.97 / 0.55 / 1.362.8 GB22.7 GB

JAMEL's cost is flat over the episode (its policy prompt stays at about 1.7k tokens), whereas the stateless ReAct agent grows linearly and is 9.5× slower than JAMEL in the last ten steps. Prefix caching removes most of the re-prefill cost, but ReAct's latency still increases with the history (1.36 s versus 0.78 s in steps 41–50), it must retain a 2.8 GB KV cache per episode, and the complete prompt is still processed as input; a 300-step navigation episode would multiply ReAct's history by six while leaving JAMEL's cost unchanged.

Computational cost. On the same replayed episodes, we counted the floating-point operations of every decision with the PyTorch FLOP counter, including the vision encoders, the compressor, prefill, and decoding:

MethodTFLOPs per step (mean)TFLOPs at step 50PFLOPs per 50-step episode
JAMEL (compressor + policy)37.038.61.85
ReAct, stateless771.71,912.038.59
ReAct, prefix KV cache74.5102.93.72

JAMEL's compute per step stays nearly constant, and the compressor accounts for only a small fraction of it. The stateless ReAct agent re-processes every past screenshot and accessibility tree at each step, so its compute grows quickly with the episode. Prefix caching avoids most of this recomputation, but attention over the growing cached history still makes its compute increase step by step, and over a full episode it still exceeds that of JAMEL. We have added this in the revised paper.

AuthorsOpenReview ↗

3. Representative rollout examples (Weakness 2)

We agree that concrete examples help, and we have added annotated rollouts for each domain:

  • ScaleWoB. After a reset returns both agents to the start page, JAMEL turns to controls it has not used yet, whereas the variant without memory repeats exactly the clicks it made before the reset and later revisits views it has already covered (Figure 6 and Table 10).
  • RoboTHOR. The base model stays in the few rooms around its start and stops discovering early, whereas JAMEL keeps moving through the apartment and is still finding new objects late in the episode (Figure 7).
  • ALFWorld. In the same bathroom, JAMEL visits every receptacle in turn (sink, cabinets, bathtub, countertop, shelf, garbage can, towel holders, toilet) and almost every action reveals something new, whereas the variant without memory shuttles between the sink, toilet, and one cabinet and then picks up and puts back the same cloth for the rest of the episode (Table 11).

Please see Appendix F of the revised paper for the full rollouts with screenshots and step-by-step traces.

Reviewer_8VU4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JAMEL, a framework that jointly trains agent memory and exploration policies using novelty signals. It compresses interaction histories into latent memory tokens and uses novelty-guided rejection sampling fine-tuning. Experiments on GUI, navigation, and household environments show improvements in exploration performance and token efficiency.

Critical strengths
  1. Well-motivated approach. Jointly learning memory and exploration through novelty signals is a reasonable idea that addresses an important problem in autonomous agents.

  2. Broad evaluation. The experiments cover three different environments and show consistent improvements in exploration performance. The reduction in input-token consumption is also encouraging.

Critical weaknesses
  1. Insufficient ablation studies. The paper lacks comparisons with training without novelty-based filtering or without joint optimization. It is therefore unclear which components contribute most to the improvements.

  2. Limited advantages of latent memory. Latent memory only slightly outperforms textual memory in some settings and performs worse on ALFWorld. This weakens the motivation for using latent memory over simpler alternatives. In addition, differences in model and training configurations make some baseline comparisons less convincing.

Questions for authors
  1. How does JAMEL perform with standard fine-tuning instead of novelty-guided rejection sampling?

  2. Can the authors provide an ablation comparing joint training with separately training the memory compressor and exploration policy?

  3. Why does textual memory outperform latent memory on ALFWorld?

7 comments
AuthorsOpenReview ↗

1. Ablation on novelty-based filtering (Weakness 1, Question 1)

We greatly appreciate your valuable comments and insightful feedback. Also, thank you for recognizing that jointly learning memory and exploration through novelty signals addresses an important problem in autonomous agents, and for appreciating the broad evaluation across three environments and the reduction in input-token consumption. During the discussion period, we conducted additional experiments to strengthen the validation of our method, including an ablation on novelty-based filtering and a comparison between joint and separate learning of memory and exploration in all three domains, and we clarified the advantages of latent memory over textual memory. We address each point below.

Training without novelty-based filtering. We trained a variant with the same architecture, optimizer, and training recipe, but on all collected trajectories without the trajectory-level threshold (Eq. 8) and prefix truncation (Eq. 9), i.e., standard fine-tuning on the raw exploration data. In RoboTHOR, the unfiltered set contains all 126 trajectories (37,800 steps) collected in the same training scenes, including the 49 trajectories that the novelty filter rejects (their mean final object coverage is 51.2%, versus 75.2% for the full set).

DomainMetricJAMEL (novelty-filtered)w/o novelty filtering
ScaleWoBNovelty / Coverage (Bytes)20.7 / 5,409,87915.4 / 5,188,235
RoboTHORNovelty / Coverage (%)6.1 / 27.73.6 / 16.7
ALFWorldNovelty / Coverage35.1 / 69.917.4 / 47.1

Removing novelty-based filtering consistently degrades exploration in all three domains, in both novelty and coverage. The drop is largest in the embodied and household environments, where the unfiltered model falls back to the level of the untrained baselines in Table 1, and even in the GUI domain it loses the advantage that JAMEL holds over the strongest baseline. This happens although the unfiltered model is trained on more samples.

These results indicate that the gain comes from which interactions are imitated rather than from the amount of supervision. Trajectories that stop producing new discoveries mostly consist of repeated or ineffective actions. Imitating them teaches the policy to revisit exhausted behaviors, which is exactly the failure mode that memory-conditioned exploration is meant to avoid. Novelty-based filtering is therefore an essential component of JAMEL. We have added this in the revised paper.

AuthorsOpenReview ↗

2. Joint learning versus learning memory and exploration separately (Weakness 1, Question 2)

Thank you for raising this point. We realize that what is learned jointly was not stated explicitly enough. In JAMEL, "joint" refers to learning memory and exploration together: the same novelty-selected trajectories supervise both which past interactions the memory pathway must convey and which action the policy should take given them, through a single action-likelihood objective. Learning them separately can therefore break the loop from either side: (a) exploration is learned without learning memory, or (b) a memory-conditioned policy is learned without exploration-driven supervision. We evaluated both:

DomainMetricJAMEL (joint)(a) learning policy only(b) w/o novelty-based filtering
ScaleWoBNovelty / Coverage (Bytes)20.7 / 5,409,87916.2 / 5,199,20015.4 / 5,188,235
RoboTHORNovelty / Coverage (%)6.1 / 27.73.8 / 16.93.6 / 16.7
ALFWorldNovelty / Coverage35.1 / 69.917.9 / 50.217.4 / 47.1

In variant (a), the novelty-selected exploration data and the training recipe are unchanged, but only the policy module is trained; the memory module is not optimized with the exploration supervision, so the policy must use memory representations that were never shaped by what exploration needs to remember. In variant (b), the full memory-conditioned model is trained, but on all collected trajectories without the novelty threshold and prefix truncation, so the memory is learned from ordinary interaction data rather than from exploration-driven supervision.

Both separations fall clearly short of joint learning in all three domains, and they degrade to a similar extent in each domain: whichever side of the loop is removed, most of the gain disappears. The drop is largest in the embodied and household environments, where both separated variants return close to the level of the untrained baselines. This supports the central claim of the paper: memory is most useful for exploration when it is learned from exploration, and exploration benefits most from memory that has been trained to encode what has already been tried. Neither half alone reproduces the gain of learning the two together. We have added this in the revised paper.

AuthorsOpenReview ↗

3. Advantages of latent memory and why textual memory is slightly better on ALFWorld (Weakness 2, Question 3)

Why textual memory is competitive on ALFWorld. ALFWorld is a highly symbolic, text-only environment: one observation–action pair takes only about 50-70 tokens, and a per-step summary about 27 tokens. A short text summary therefore loses almost nothing in this domain. The two variants are effectively tied: their gap (0.5 novelty-positive actions per episode) is small, and both outperform the no-memory variant.

Where latent memory helps. The advantage appears when the history is long and multimodal. Each ScaleWoB step carries a screenshot and an accessibility tree, so an agent that keeps the exact 50-step history needs about 37.9k input tokens per step, versus about 2.1k for JAMEL; a 300-step RoboTHOR episode with full image and text history reaches about 85.6k tokens at the last step, versus about 0.6k for JAMEL. Such histories cannot be summarized in short text without losing visual and structural detail, and accordingly latent memory outperforms the text summary on both ScaleWoB and RoboTHOR. It also costs one token per past step, whereas a text summary must be regenerated and re-read at every step.

Model and training configurations of the baselines. All learned baselines share JAMEL's Qwen2.5-VL-7B-Instruct backbone in RoboTHOR and ALFWorld; in ScaleWoB, the open-weight baselines are of comparable size (8B), while Gemini 3.1 Flash-Lite is a closed model. The downstream ReAct baselines are fine-tuned on the same trajectories as JAMEL.

Thank you again for your constructive feedback. Please let us know whether our responses address your concerns. We would be happy to discuss further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their constructive feedback. During the discussion period, we conducted additional experiments and revised the manuscript to address four main areas of concern.

1. Novelty-based filtering and joint learning of memory and exploration (Section 4.4, Table 4; Appendix E).

To isolate the role of novelty-based data selection (8VU4 W1&Q1), we trained a variant on all collected trajectories without the novelty threshold and prefix truncation. Removing the filter consistently degrades exploration in all three domains, e.g., 15.4 versus 20.7 novelty on ScaleWoB and 17.4 versus 35.1 on ALFWorld, although the unfiltered model is trained on more samples. To test whether memory and exploration must be learned together (8VU4 W1&Q2), we further compared JAMEL with learning the policy only and with learning memory without novelty-based filtering. Both separations fall clearly short of joint learning in every domain (e.g., 3.8 and 3.6 versus 6.1 novelty on RoboTHOR).

2. Inference cost beyond input tokens (Appendix B, Tables 7–8, Figure 5; Section 3.1).

To address end-to-end cost and KV-cache reuse (AgqS W1&Q1–Q2), we measured per-step latency, KV-cache size, and FLOPs against a same-backbone full-history ReAct agent, with and without prefix KV-cache reuse. JAMEL's cost stays flat over an episode (0.77 s and 37.0 TFLOPs per step), whereas the stateless agent grows linearly (3.82 s and 771.7 TFLOPs per step). Prefix caching removes most of the re-prefill cost, but the baseline still requires twice the compute of JAMEL and a 2.8 GB KV cache per episode. We also clarified the scope of the memory: it is an environment-specific exploration memory that is task-agnostic and transfers across tasks in the same environment.

3. Representative rollouts and memory variants (Appendix F, Figures 6–7, Tables 10–11; Section 4.4).

To illustrate exploration behavior (AgqS W2; HivV W6), we added annotated rollouts for all three domains. After a reset, JAMEL turns to unused GUI controls, whereas the variant without memory repeats its earlier clicks; in RoboTHOR, JAMEL keeps moving through the apartment while the base model stays near its start; in ALFWorld, JAMEL visits every receptacle while the variant without memory repeatedly handles the same object. Regarding latent versus textual memory (8VU4 W2&Q3; HivV W5), we explained that ALFWorld is a text-only environment in which one observation–action pair takes only about 50–70 tokens, so a short textual summary loses little information.

4. Presentation, training details, and model complexity (Abstract; Sections 3.3 and 4.1; Appendix D).

We revised the abstract to state that this work studies how a language-model-based agent learns memory and exploration jointly, to mention the use of vision-language models, and to remove the repeated sentence (HivV W2). We summarized the training procedure as novelty-based trajectory filtering followed by supervised fine-tuning, and added a table of training data and optimization settings for each domain (HivV W3). We clarified that JAMEL-9B counts the 7B policy and the 2B compressor, and that the exploration baselines are used as released with comparable model sizes (HivV W4; 8VU4 W2).

Finally, following Reviewer 8VU4's follow-up question, we also added to the Limitations that all experiments use a single backbone pairing (a Qwen2.5-VL-7B-Instruct policy with a Qwen3-VL-2B-Instruct compressor) and that other policy backbones such as Qwen3-VL-8B-Instruct have not been evaluated (Section 6).

We sincerely thank the reviewers and the Area Chair for their time and valuable feedback, which has helped us strengthen the manuscript. Please let us know whether these revisions address your concerns; we welcome further questions and discussion.

Best regards,

Authors of Submission 1888

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.