- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a framework that jointly trains agentic memory and an exploration policy termed JAMEL. The core idea is to compress previous interactions into memory tokens, and then learn an exploration policy by designing a novelty reward based on that memory. The history compression is done by a VLM, where the last token’s hidden state is used to represent that memory. The method is evaluated in several domains, with comparable or better results compared to baselines.
Critical strengths
- The overall idea is simple, though some training details are missing.
- While on several domains JAMEL is comparable to ReAct, it significantly outperforms it on Navigation.
- Limitations are discussed.
Critical weaknesses
- Novelty reward definition: it is not entirely clear in the main text how “novelty” is defined per-domain. The authors mention that it “determines whether the current interaction introduces behavior not recorded in the accumulated history”, but it is not clear what that actually means. An example in one of the domains would have been helpful. This information is found in Appendix A, and it seems that designing the reward requires privileged domain/environment information and manual tuning. This is in contrast to “classic” exploration methods that are using general signals such as prediction error (e.g., with world models).
- Abstract: it was hard to understand what domain or field this work falls under. In addition, the abstract contains repeated sentences (“jointly trains agentic memory and an exploration policy through novelty-driven interaction”). As a reader, it was hard for me to understand what to expect. For example, the utilization of VLMs should be mentioned.
- Stages: it is unclear what components are pre-trained, how the policy is structured and trained, and what components it uses. It is not clear how training is done, for how long, what algorithms etc.
- Complexity: It is not entirely clear how many parameters JAMEL has, how many are trained or fine-tuned (is it 2B +7B = 9B?) . Unclear if the baselines have similar numbers of parameters and if the comparison is compute-matched. It is also not clear whether the baselines are trained in capacity or taken as-is.
- Latent vs. text memory: it seems that the gains of using learned latent memory are not significant compared to just using a text summary.
- No examples of exploration behavior: the authors do not demonstrate what exploration looks like in each domain. I’d expect a sample output of the VLM when something novel is discovered.
- Overall impression: given the text-summary ablation, I’m not entirely convinced the method is necessary and that careful prompt engineering cannot achieve similar results. In addition, my general impression of the writing is that several parts require more clarification as I mentioned above. I appreciate the authors’ candor regarding the utilization of AI in ideation and writing though.
Questions for authors
- Please address the weakness above.
- What is the intended application of the proposed method given the new iterations of large-scale models (e.g. Opus, Astra)? Do the authors think that JAMEL will be useful as future iterations of QWEN improve? My concern here is whether this is just a temporary technique to stitch a problem that would be solved in the next iterations of large-scale agents.
6 comments
2. Abstract, training stages, and model complexity (Weaknesses 2–4)
Domain of the work and the abstract. Thank you sincerely for pointing this out. JAMEL is a general learning method for LLM/VLM agents, and its subject is how an agent's memory and its exploration policy can be learned jointly. It is motivated by the observation that the two are mutually dependent: sustained exploration requires memory to tell exhausted behaviors from unseen ones, and novelty-seeking interaction in turn supervises what memory should retain. GUI, navigation, and household environments are therefore testbeds, not the field of the work; applying essentially the same method to all three is meant to show that it is not tied to a particular domain. We have rewritten the abstract to state this positioning up front, to mention the use of vision-language models, and to remove the repeated sentence. We have added this in the revised paper.
Stages and training. JAMEL is trained by novelty-based filtering of exploration trajectories followed by supervised fine-tuning of the model on the retained steps, so that memory and exploration are learned jointly from the same supervision without reinforcement learning. The settings are:
| ScaleWoB | RoboTHOR | ALFWorld | |
|---|---|---|---|
| Policy / compressor | Qwen2.5-VL-7B-Instruct / Qwen3-VL-2B-Instruct | same | same |
| Training environments | 86 applications | 60 scenes | 1,200 games |
| Training samples | 24k | 28.8k | 13k |
| Optimizer | AdamW, lr 2e-5, cosine, warm-up 0.05, weight decay 0.01 | AdamW (β = 0.9, 0.95), lr 4e-5, cosine, warm-up 0.05, weight decay 0.01 | same as ScaleWoB |
| Global batch size / optimizer steps | 16 / 4,398 | 28 / 1,350 | 32 / 812 |
| Maximum sequence length | 8,192 | 2,048 | 3,072 |
We have added this in the revised paper.
Complexity. The name JAMEL-9B counts the 7B policy, the aligner and the 2B compressor. All exploration baselines are used as released, without further training: the open-weight GUI baselines (Qwen3-VL-8B-Instruct, MAI-UI-8B, GUI-Owl-1.5-8B) are of comparable size to JAMEL's 7B policy, and the RoboTHOR and ALFWorld baselines use the same Qwen2.5-VL-7B-Instruct backbone. The downstream ReAct baselines are fine-tuned on the same trajectories as JAMEL.
3. Latent versus text memory, examples of exploration, and prompt engineering (Weaknesses 5–6, Overall impression)
Latent versus text summary. We agree that the text-summary variant is a strong alternative. On ScaleWoB, latent memory covers 24% more code bytes (5,409,879 vs. 4,354,632) at similar novelty, and on RoboTHOR it improves both novelty and coverage (6.1 / 27.7% vs. 5.9 / 26.3%), while costing one token per past step instead of a regenerated summary. On ALFWorld, the two are tied (35.6 vs. 35.1 novelty, a gap smaller than that between the seen and unseen splits of the same model). This is expected: ALFWorld is a highly symbolic, text-only environment in which one observation–action pair takes only about 50-70 tokens and a per-step summary about 27 tokens, so a short text summary loses almost nothing; latent memory pays off when the history is long and multimodal, as in the GUI and navigation domains. Importantly, the text-summary variant is trained with the same novelty-guided procedure, so the contribution of the paper—novelty signals as supervision for memory and exploration—does not depend on which memory format is chosen.
Is careful prompting sufficient? The text-summary variant is not a prompted agent. It is trained with exactly the same novelty-guided filtering and supervised fine-tuning as JAMEL; only the memory format differs. Its strong results therefore come from our novelty supervision, not from prompting. Two results show that prompting alone does not achieve the same:
- Carefully engineered open-source agents fall far behind. MAI-UI and Mobile-Agent-v3.5, which use elaborate prompts with explicit notes or sliding-window memory, reach only 8.4 and 5.9 novelty on the held-out applications, versus 20.7 for JAMEL with a 7B policy under the same budget.
- Removing novelty supervision while keeping the prompt unchanged loses most of the gain in all three domains (our ablation on novelty-based filtering).
| Domain | Metric | JAMEL (novelty-filtered) | w/o novelty filtering |
|---|---|---|---|
| ScaleWoB | Novelty / Coverage (Bytes) | 20.7 / 5,409,879 | 15.4 / 5,188,235 |
| RoboTHOR | Novelty / Coverage (%) | 6.1 / 27.7 | 3.6 / 16.7 |
| ALFWorld | Novelty / Coverage | 35.1 / 69.9 | 17.4 / 47.1 |
The improvement therefore comes from what the policy is trained to imitate, which prompt engineering cannot change.
Examples of exploration behavior. We have added annotated rollouts for all three domains. In each, JAMEL keeps moving to parts of the environment it has not yet visited (new GUI views after a reset, new rooms in RoboTHOR, new receptacles in ALFWorld), whereas the baselines tend to revisit what they have already seen. Please see Appendix F of the revised paper for the full rollouts.
1. Definition of novelty and the information it requires (Weakness 1)
We sincerely thank you for your effort in reviewing our paper and for your candid assessment. During the discussion period, we revised the abstract and the method description, added the training details and model sizes, added an ablation on novelty-based filtering and representative exploration examples, and clarified the comparison with textual memory. We address each point below.
What novelty means, with an example. In every domain, an action receives rt=1 if, and only if, it reveals at least one behavior unit that has not been recorded earlier in the same episode. In the GUI domain, the unit is executed JavaScript code: clicking a tab that loads a new page component executes code that has not run before and receives a reward, whereas clicking the same tab again executes only already-covered code and receives zero. In navigation, the unit is an object instance: rotating to bring a previously unseen mug into view receives a reward; seeing it again does not. In ALFWorld, the unit is a coverage atom such as a new location type, object type, or interaction: opening a drawer that reveals a new object type receives a reward.
This design has two benefits: because a discovered unit stays familiar, repeating anything already tried earns nothing, so the reward directly requires the policy to remember its own history; and because the signal is binary and read from environment state rather than from a learned predictor, it is consistent across actions of very different scale and cannot be inflated by unpredictable but uninformative observations, a known weakness of prediction-error curiosity.
Privileged information and manual tuning. We would like to clarify that code coverage (GUI) and newly observed objects (navigation) are environment-native state feedback, not human-annotated rewards. Code coverage is reported by the JavaScript engine of any instrumented application without modifying the application, and object visibility is simulator metadata. No per-environment tuning is involved: the same rule is applied unchanged to all 96 applications and all RoboTHOR scenes. The signal does not require a learned world model or a task reward. The rule is defined once per domain, not per environment, so it generalizes across a whole domain: the same code-coverage rule applies to any web application, since every modern browser can report JavaScript coverage, and the same object-discovery rule applies to any simulated indoor scene, since simulators expose object visibility.