acceptodds
Under review as a conference paper at ICLR 2027

Vision-Language Agents Can Learn in Their Own Latent Worlds

Abstract

Learning from experience is essential for building the next generation of agents that continually expand their capabilities. Pretrained vision-language models (VLMs) already internalize rich world knowledge, yet this knowledge is rarely leveraged to drive policy improvement through online interaction. Realizing this potential efficiently remains challenging, as policy updates shift the agent’s interaction distribution, requiring the world model to adapt accordingly, while continually updating large pretrained models is computationally expensive. To address this challenge, we introduce **Dwell**, a model-based reinforcement learning (RL) framework that reframes a VLM as a recurrent latent transition operator, endowing it with the ability to predict, adjust, and act in latent space. Leveraging the VLM’s own representations and its dual role as both world model and policy, the agent continually learns from real interactions to anticipate and evaluate action outcomes, and from latent imagination to improve its behavior. Across five interactive visual environments, our method achieves the strongest overall performance among trainable methods while substantially reducing the compute required for imagination, showing that effective world models for VLM agents need not reconstruct the future—they need to predict futures that are useful for action.

Primary area:
reinforcement learning
Keywords:
VLM A gents, Model-based RL, Online Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TgEBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes an algorithm to train a latent world-model to synthesise imaginary trajectories that are used to accelerate the learning process.

Critical strengths

The paper demonstrates that JEPA-style latent world model can be implemented upon a VLM backbone with LoRA adapters.

Critical weaknesses
  1. Neither the proposed algorithm nor the empirical results are mind-blowing enough to make me lean towards clear acceptance. There have been multiple prior offline RL works that adopt JEPA-style world models (Zhou et al., 2024; Sobal et al., 2026). The learning algorithm is also similar to the classical model-based dataset augmentation method Dyna-Q. The usage of LoRA adapters to share a common backbone between the model and the policy is clever but not so surprising.

  2. The ablation of Ldyn\mathcal{L}_{dyn} seems meaningless. If I've understood correctly, the Ldyn\mathcal{L}_{dyn} is the only loss function related to z^\hat{\mathbf{z}}, which means dropping Ldyn\mathcal{L}_{dyn} will leave z^\hat{\mathbf{z}}, which is what is used during the "planning phase", untouched.

References

Sobal, U., Zhang, W., Cho, K., Balestriero, R., Rudner, T. G., & LeCun, Y. (2026). Learning from reward-free offline data: A case for planning with latent dynamics models. Advances in Neural Information Processing Systems, 38, 43905-43941.

Zhou, G., Pan, H., LeCun, Y., & Pinto, L. (2024). Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983.

Questions for authors
  1. It is unclear how the networks are implemented. I know that LoRA has been used, but what are the inputs and outputs? What's the dimension of the latent states? How do you feed the latent state, which is neither language nor image, to the VLM? So many key details are missing from the paper.

  2. Is there a reason why AdaWorld results are not included in Table 1?

  3. Adding these ablation studies might be interesting:

    • Using et\mathbf{e}_t, with linear projection to align the dimensions if needed, instead of zt\mathbf{z}_t. This experiment is necessary in order to justify the adoption of predicting posterior states.
    • Always initializing posteriors with 0 posterior steps.
    • Full fine-tuning instead of LoRA.
    • Full fine-tuning two different VLMs (one for policy, the other for model)
    • Sensitivity to the latent state dimension.
    • Planning with GRPO instead of PPO.
  4. Is there a reason for not using the real samples for training?

  5. Consider adding a brief explanation on how DWELL differs from LeWorldModel and AdaWorld.

Miscellaneous comments

  1. First sentence of Introduction: Team et al., 2026 → Kimi Team et al., 2026
  2. First sentence of the Interaction Setting paragraph of Method: (POMDP) Åström(1965);Kaelblingetal.(1998) → (POMDP; Åström, 1965; Kaelbling et al., 1998)
  3. The rationale for why prior and posterior state prediction are both required comes after defining them. Consider placing it before the definitions so that readers wouldn't be wondering why we need zt\mathbf{z}_t while reading Eq. (2).
  4. The expressions prior and posterior latent states are unconventional and confusing.
  5. The if used in the first sentence of the second paragraph of Recurrent Latent Model is awkward. Consider rewriting the sentence.
  6. Section 3.1 Baselines paragraph: LeWorldModel → LeWorldModel (Maes et al., 2026)
5 comments
AuthorsOpenReview ↗

Response to the Official Review by Reviewer TgEB

Thank you for suggesting a direct test of the posterior prediction target. Your comparison with a frozen visual target helps us explain what DWELL adds to earlier latent world models.


W1/W2/Q3(a) What does DWELL add to latent world-model RL, and why predict the posterior state?

1. Relation to prior work. Prior work has established latent dynamics prediction [1, 2, 5] and policy optimization from imagined experience [3, 4]. DWELL addresses a distinct question: how a pretrained VLM can continually improve its policy online using the internal states it forms during real interaction. Training VLMs online can lead to representation drift and high computational cost.

In DWELL, after a real action, the VLM updates its recurrent state with the new observation via lightweight LoRA adapters. This observation-conditioned posterior zt+1z_{t+1} serves both as the state the actor reads for its next decision and as the prediction target for the dynamics model before the observation becomes available (Eq. 3). Consequently, imagined PPO trains the actor on estimates of the exact states it will encounter when returning to the real environment, achieving stable representations and higher computational efficiency than both pixel-reconstruction world models (e.g., AdaWorld [7]) and model-free online RL (PPO/GRPO).

2. Why predict the posterior? Consider two distinct interaction histories that end in the same visual image. The frozen visual target et+1=E(ot+1)e_{t+1}=E(o_{t+1}) is identical for both, whereas their posteriors zt+1z_{t+1} can differ because each also depends on the preceding belief ztz_t and action ata_t. Furthermore, et+1e_{t+1} lives in the external encoder's feature space rather than the VLM's decision space. DWELL therefore separates the two roles: it retains the frozen visual feature et+1e_{t+1} as a visual anchor on a projection of the posterior to prevent representation drift (Eq. 4), while training the dynamics predictor to match the history-conditioned posterior zt+1z_{t+1} that the VLM actor directly reads (Eq. 3).

We agree that removing LdynL_{\mathrm{dyn}} only serves as a sanity check that dynamics supervision matters. Following your suggestion, we kept the exact same action-conditioned VLM predictor and its 2048-dimensional recurrent slots, and changed only the target used to train the predictor from zt+1z_{t+1} to the frozen V-JEPA 2 feature et+1=E(ot+1)e_{t+1}=E(o_{t+1}).

Table T1. Real-environment success rate by prediction target (mean ± std, 3 seeds).

Target for the next-state predictionSokobanFrozenLake
Observation-conditioned state zt+1z_{t+1} (DWELL)0.53 ± 0.0120.51 ± 0.021
Frozen visual feature et+1e_{t+1}0.37 ± 0.0340.23 ± 0.043

Results: Both tasks favor the posterior target. Alignment to et+1e_{t+1} rewards fidelity to the next visual observation but gives the VLM little guidance about which features matter for its next decision. The posterior target may make the VLM’s pretrained knowledge more useful during imagination because it expresses the action’s outcome in the representation the policy already uses to choose its next move.

We also thank Reviewer DBhY for suggesting a standalone predictor. With the Qwen policy initialization and training data held fixed, replacing the VLM transition with V-JEPA 2 and a four-layer Transformer lowered success on both tasks. We have added your suggested prediction-target comparison to our ablation study in Appendix A.3.

AuthorsOpenReview ↗

Response to the Official Review by Reviewer TgEB (2)


Q2 Why was AdaWorld omitted from Table 1?

We originally included AdaWorld in Section 3.3 to compare learning curves and computation (Figures 2–3). We agree that readers should also be able to compare final success rates in the main table.

Table T2. Final real-environment success rate. AdaWorld uses 3 seeds on Sokoban&FrozenLake and one seed on each of the other three environments.

MethodSokobanFrozenLakeNavigationPrimitiveSkillEB-ALFRED
AdaWorld [7]0.37 ± 0.0500.35 ± 0.1050.330.500.233
DWELL (paper Table 1, corrected)0.53 ± 0.0120.51 ± 0.0210.390.210.18 ± 0.048

Results: DWELL get better performance on each grid task in the three-seed comparison. AdaWorld’s single-seed score is lower on Navigation and higher on PrimitiveSkill and EB-ALFRED than DWELL’s reported averages. Once the remaining three-seed evaluations are complete, we will add the five-environment comparison to Table 1 and update Section 3.2 accordingly.


Q3(b–f) How sensitive is DWELL to rollout initialization, fine-tuning, latent width, and the policy objective?

We tested the five variants you suggested. In (b), setting t=0t=0 in Algorithm 1 starts every imagined rollout at episode reset. In (e), linear projections map each recurrent slot from 2048 to 1408 dimensions for storage and back to 2048 when Qwen reads it. In (f), GRPO compares the imagined returns of G=8G=8 action sequences from the same starting state, using the same imagined-transition and actor-update budgets as PPO.

Table T3. Real-environment success rate for the additional ablations. Within the rebuttal period, we completed 1 seed per task for (b)–(f).

VariantSokobanFrozenLake
DWELL, native 2048-dimensional slots0.53 ± 0.0120.51 ± 0.021
(b) Start all imagined rollouts at episode reset0.570.44
(c) Fully fine-tune one shared VLM0.430.48
(d) Fully fine-tune two VLMs, one per role0.250.14
(e) Projected recurrent slots, d=1408d=14080.380.44
(f) Train the actor with GRPO on imagined rollouts0.560.68

Results: We get some interesting preliminary findings from the ablations: We find that the best starting state for an imagined rollout may vary by task. Under full fine-tuning, both tasks favored a shared backbone. Because the actor reads the dynamics model’s predicted slots directly, sharing weights may help the actor track changes in how the model encodes a state as training proceeds. Both tasks also favored native-width slots, although the narrower variant changes both the width and the projections. Imagined GRPO scored highest on FrozenLake. When two of its eight continuations from the same tile reach different outcomes, their relative returns can tell the actor which route to favor from that tile. Appendix A.3 reports all five variants, with one training seed per variant and task.


Q4 Why are real samples not used for training?

Real samples are used throughout DWELL, with a clear division between off-policy replay and on-policy optimization:

  1. Where real samples are used: Real trajectories in the replay buffer D\mathcal{D} train and continually refresh the world and reward models, supply nn-step return targets for critic warm-up, supervise EB-ALFRED behavior cloning, and provide the real posterior starting states ztz_t for imagination.
  2. Why PPO updates on imagined rollouts: Because PPO is an on-policy algorithm, directly reusing historical trajectories in D\mathcal{D} quickly becomes off-policy as the actor updates. At the start of each PPO iteration, the current policy generates a fresh batch of imagined rollouts from posterior states inferred from real trajectories in D\mathcal{D}. PPO then reuses this batch across multiple update epochs without additional environment calls. This lets us study the benefit of latent imagination under a fixed real-interaction budget, though mixing fresh real transitions with imagined rollouts is also compatible with our framework. We have clarified the data source for each update in Section 2.3 and Appendix A.2.
AuthorsOpenReview ↗

Response to the Official Review by Reviewer TgEB (3)


Q5 How does DWELL differ from LeWorldModel and AdaWorld?

The three methods differ in what the world model predicts and how the VLM policy consumes that prediction:

  • AdaWorld [7] (pixel generation): Predicts future RGB observations that the VLM must re-encode through its vision tower at every imagined step, incurring substantially higher compute (38–58% less GPU-hours for Dwell; Figure 3).
  • LeWorldModel [6] (standalone latent dynamics): Learns a separate visual encoder and compact latent dynamics outside the VLM. Even with a learned projection to bridge its states to the Qwen actor–critic, decoupling the prediction and decision spaces degrades policy learning (consistent with our bridged-interface ablation in Appendix A.3, Table 2).
  • DWELL (native VLM latent dynamics): Predicts recurrent states directly in the VLM’s own token embedding space, allowing the actor and dynamics model to read predicted states immediately without pixel synthesis, re-encoding, or cross-space projection.

We have added this comparison to Section 3.1 and Appendix A.2.


Miscellaneous comments

Thank you for identifying the unclear wording and citations. We have addressed all six comments in the revised manuscript. In detail, we corrected the Kimi Team attribution (Sec 1) and POMDP citation (Sec 2), added Maes et al. in Section 3.1, and rewrote the conditional sentence in “Recurrent Latent Model.” Before Eq. (2) (Sec 2.1), we explain which information is available during real interaction and imagination, then clarify that posterior denotes the state updated with the current observation and prior denotes the state predicted without that observation. Both are deterministic latent representations.

References

[1] Zhou et al. DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning. ICML, 2025.

[2] Sobal et al. Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models. NeurIPS, 2025.

[3] Sutton. Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming. ICML, 1990.

[4] Hafner et al. Dream to Control: Learning Behaviors by Latent Imagination. ICLR, 2020.

[5] Zhang et al. DeepSight: Long-Horizon World Modeling via Latent States Prediction for End-to-End Autonomous Driving. ICML, 2026.

[6] Maes et al. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. 2026.

[7] Gao et al. AdaWorld: Learning Adaptable World Models with Latent Actions. ICML, 2025.

Reviewer_DBhYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

DWELL uses separate adapters on a frozen VLM for world modeling and policy learning. The policy trains on imagined latent trajectories, avoiding future-image generation, while environment interactions update the world model in the online settings.

Critical strengths

s1. The method improves performance on Sokoban and FrozenLake under the reported interaction budgets.

s2. Latent imagination reduces compute compared with the image-generating baseline, and the paper reports end-to-end GPU costs.

s3. The ablations show benefits from predictive training, visual grounding, and continued world-model updates.

Critical weaknesses

w1. The overall approach is close to established latent world-model RL. The experiments do not clearly establish what using the VLM itself as the transition model adds over a separate dynamics model.

w2. The online-learning claim does not apply equally across tasks. EB-ALFRED uses scripted demonstrations, behavior cloning, and a frozen world model without new environment rollouts.

w3. Table 1 contains reporting inconsistencies. The four PrimitiveSkill scores average to about 0.21, rather than 0.28, and the text says GRPO performs better even though the table shows otherwise.

Questions for authors

q1. Could you replace the VLM-based world model with a smaller standalone dynamics model, keeping the policy and training data unchanged, to test whether using the VLM for prediction is necessary?

q2. On EB-ALFRED, how much does imagined RL improve over the same behavior-cloning initialization and training data without imagination?

q3. Could you clarify the PrimitiveSkill average and check the overall score and corresponding discussion?

3 comments
AuthorsOpenReview ↗

Response to the Official Review by Reviewer DBhY (1)

Thank you for suggesting a standalone predictor and a comparison from the same behavior-cloning (BC) initialization. We report both comparisons below.


W1/Q1 What does using the VLM as the transition model add over a smaller standalone model?

1. Why use the VLM for prediction? Our goal is to help a pretrained VLM agent continually improve its policy from limited real interaction by imagining the consequences of its actions. We choose the VLM itself as the dynamics predictor for two reasons.

  • Using pretrained knowledge to form the policy’s belief. After executing an action ata_t in real interaction, the agent must integrate its prior belief ztz_t, the action ata_t, and the new observation ot+1o_{t+1} to update its understanding of the environment. Leveraging the VLM's pretrained grounding of visual content and action semantics, DWELL fuses these three inputs into the posterior state zt+1z_{t+1}—the exact representation the policy reads to select its next action.

  • Turning real beliefs into self-consistent imagination: Each real transition naturally supplies a supervision target: what belief zt+1z_{t+1} did the VLM actually form after observing the action's outcome? By training the dynamics LoRA adapter on the same VLM to predict this posterior from (zt,at)(z_t, a_t) alone, the predicted state z^t+1\hat{z}_{t+1} can be consumed directly by the policy for the next decision and unrolled by the dynamics model for subsequent steps—turning the representations formed during real interaction into the native states used to act and learn in imagination.

2. Standalone predictor. Following your suggestion, we replaced the VLM transition with a V-JEPA 2 visual encoder [2] and a four-layer action-conditioned Transformer. As in DINO-WM [3], it predicts the next observation’s visual features. During imagination, the Transformer rolls forward on its own predictions and supplies 36 latent slots to the Qwen actor. We kept the policy initialization, training data, imagination horizon, PPO update budget, and evaluation episodes the same.

Table D1. Real-environment success rate (mean ± std, 3 seeds, 400 steps).

MethodSokobanFrozenLake
V-JEPA 2 + four-layer Transformer0.39 ± 0.0600.39 ± 0.071
LeWorldModel [4] (latent dynamics)0.44 ± 0.0210.29 ± 0.061
AdaWorld [5] (generative)0.37 ± 0.0500.35 ± 0.105
DWELL0.53 ± 0.0120.51 ± 0.021

Results: With the Qwen policy initialization and training data held fixed, DWELL achieves better performance. In our paper, we also notice that our method outperforms LeWorldModel’s standalone latent JEPA and AdaWorld’s diffusion predictor on both tasks, supporting our choice to use the VLM itself to predict the state its policy will act on next. Following your suggestion, we explain how the policy reads the predicted state in section 2.1, and reported the 3 seed comparison with the standalone predictor in Appendix A.3.


W2/Q2 Was the EB-ALFRED world model updated online, and how much does imagined RL add after BC?

1. Online training. You correctly identified an inconsistency in Table 5. The EB-ALFRED result in paper Table 1 came from the online runs. We mistakenly copied the schedule for a separate frozen-world-model run into Table 5. During PPO, the policy collected four new real episodes every four actor updates. We added them to replay and refreshed the dynamics adapter every 20 actor updates. We have corrected Table 5 and included the EB-ALFRED online training configuration in our anonymous code repository.

Table D2. Online versus frozen dynamics on EB-ALFRED (3 seeds). We average success over six subsets within each seed

Dynamics during imagined PPOEB-ALFRED
Online refresh0.18 ± 0.048
Frozen after offline fitting0.07 ± 0.018

Results: Both policies train with imagined PPO. Mean policy success is approximately 11 percentage points higher when the dynamics adapter is refreshed online.

2. Policy improvement after BC initialization. For your no-imagination comparison, we use model-free PPO with the same 500-step BC initialization procedure. We also report DWELL with auxiliary BC removed to show how much continued demonstration supervision contributes during imagined PPO.

Table D3. EB-ALFRED policy success rate (1 seed).

MethodEB-ALFRED
Model-free PPO, with BC initialization0.080
DWELL, w/o initialization and auxiliary BC0.143
DWELL, w/o auxiliary BC0.163
Full DWELL0.187

Results: After BC initialization, model-free PPO reaches 0.08 and DWELL trained without auxiliary BC reaches 0.163 (and 0.187 with auxiliary BC). We have clarified these settings in Appendix A.2.

AuthorsOpenReview ↗

Response to the Official Review by Reviewer DBhY (2)


W3/Q3 PrimitiveSkill average, Overall score, and the comparison with GRPO

Thank you for catching both errors. We divided the sum of DWELL’s four subtask scores by three. With equal weight for all four subtasks, the correct average is

(0.33+0.21+0+0.29)/4=0.2075≈0.21.(0.33+0.21+0+0.29)/4=0.2075\approx 0.21.

We have corrected the PrimitiveSkill average from 0.28 to 0.21 and the five-environment Overall score from 0.38 to 0.36 in Table 1, and have also corrected Section 3.2.

References

[1] Hafner et al. Dream to Control: Learning Behaviors by Latent Imagination. ICLR, 2020.

[2] Assran et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. 2025.

[3] Zhou et al. DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning. ICML, 2025.

[4] Maes et al. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. 2026.

[5] Gao et al. AdaWorld: Learning Adaptable World Models with Latent Actions. ICML, 2025.

AuthorsOpenReview ↗

Follow-up Discussion (Discussion Closing Soon)

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards,

The Authors

Reviewer_AjCKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DWELL enables a vision-language agent to learn from imagined trajectories represented directly in latent space. Instead of generating future images and encoding them again, the world model predicts representations that the policy can consume directly.

The framework combines latent dynamics learning, representation alignment, reward prediction, and policy optimization through imagined rollouts. The VLM backbone remains frozen, with separate trainable adapters for the policy and world model.

Experiments cover five visual environments spanning puzzles, navigation, manipulation, and long-horizon embodied tasks. The reported benefits are task dependent: DWELL performs well on several environments, while GRPO remains stronger on navigation.

Critical strengths
  1. Clear computational motivation. Avoiding image generation and repeated visual encoding can make imagined experience less expensive.
  2. Controlled backbone comparison. Trainable methods use the same VLM backbone, helping reduce differences caused by model capacity.
  3. Useful component analysis. The paper includes three training seeds and ablations addressing the shared representation interface, training objectives, and world-model updates.
Critical weaknesses
  1. Task-specific reward transformations, update rollback mechanisms, and the auxiliary behavior-cloning loss used for EB-ALFRED may contribute substantially to performance. Their effects should be separated from the contribution of latent imagination, and relevant baseline settings should be matched.
  2. The online co-improvement claim can be better jusfitied. The world model is frozen in the EB-ALFRED setting. This experiment therefore does not directly demonstrate continuous joint improvement of the policy and world model.
Questions for authors

Are there computational typos in Table 1 text descriptions? DWELL’s four PrimitiveSkill scores are 0.33, 0.21, 0, and 0.29. Their equal-weighted average is approximately 0.21, whereas the table reports 0.28. The text also states that GRPO is slightly better on PrimitiveSkill, which does not align with the displayed scores.

3 comments
AuthorsOpenReview ↗

Response to the Official Review by Reviewer AjCK (1)

Thank you for recognizing the computational motivation for latent imagination and for asking how much of the improvement comes from the training choices below.


W1 Contribution of reward transformations, update rollback, and auxiliary behavior cloning

1. Reward transformations. We agree that the task-specific mappings could affect performance. For sparse-reward tasks, learning the reward model is challenging due to the class imbalance between positive rewards and per-step costs. A model that always predicts “no success” can therefore classify most transitions correctly while providing little guidance for policy learning. For this reason, we formulate reward learning as event classification so that the reward head explicitly estimates the probability of success from predicted latent states (e.g., in FrozenLake it distinguishes ongoing, goal, and hole events). Table 4 of the paper specifies how much reward a predicted success earns and when it is counted.

To test your concern directly, we replaced this design with a scalar head trained on the same latent inputs and training data. It predicts the recorded environment reward at each step. PPO uses this prediction directly during imagination, with the original termination rules.

Table R1. Real-environment success rate (1 seed).

MethodSokobanFrozenLakePrimitiveSkill
Model-free PPO0.290.220.07
DWELL, task-specific rewards0.530.510.21
DWELL, direct reward prediction0.510.440.25

Results: Direct prediction lowers success by 2 percentage points on Sokoban and 7 on FrozenLake, and raises it by 4 on PrimitiveSkill. DWELL still exceeds model-free PPO on all three tasks. We have added this comparison and clarified the mappings in Appendix A.5.

2. Update rollback. To check whether rollback accounts for the grid-task gains, we repeated training with the gate disabled. The gate is motivated by conservative policy updates [1–3]. It rejects a candidate PPO update if action rankings deteriorate on a fixed panel initialized from held-out real trajectories. A frozen world model and reward head supply the reference rankings.

Table R2. Rollback ablation (mean ± std, three seeds).

MethodSokobanFrozenLake
With rollback0.53 ± 0.0120.51 ± 0.021
w/o rollback0.52 ± 0.0720.49 ± 0.050

Results: Enabling rollback raises mean success by approximately 1.0 and 2.0 percentage points, respectively. We also notice that the results are more stable with rollback. The mean rollback rate across runs is about 5.87%, which is not used frequently in our training. We have describes the rollback criterion and reports the 3 seed ablation in Appendix A.6.

3. Auxiliary behavior cloning on EB-ALFRED. You are right that demonstrations can contribute at two stages. DWELL uses 500 steps of behavior cloning (BC) before RL (all baselines use it) and an auxiliary term 0.1LBC0.1L_{\mathrm{BC}} during PPO. The latter provides action-level supervision from successful scripted demonstrations when terminal success gives little feedback on intermediate actions [4]. We removed the two sources of supervision in turn.

Table R3. EB-ALFRED success rate. The DWELL ablation uses one independent training seed.

Methodw/o initialization & auxiliary BCw/o auxiliary BCFull Method
DWELL0.1430.1630.187
PPO0.0470.080---

Results: In this run, initialization alone raises success from 0.143 to 0.163. Adding auxiliary BC during DWELL PPO raises it to 0.187. For comparison, pure model-free PPO without any BC achieves 0.047. We have reported both stages separately in Appendix A.2.

AuthorsOpenReview ↗

Response to the Official Review by Reviewer AjCK (2)


W2 Online world-model updates on EB-ALFRED

Thank you for pointing out this parameter setting in Table 5. The EB-ALFRED results in Table 1 came from the online runs. We mistakenly copied the schedule for a separate frozen-world-model run into Table 5. During online PPO, we collected four new environment episodes every four actor updates, added them to replay, and refreshed the world-model dynamics adapter every 20 actor updates. We have corrected Table 5 and included the EB-ALFRED online training configuration in our anonymous code repository.

Table R4. Online versus frozen dynamics on EB-ALFRED. Each seed’s success rate is averaged over the six EB-ALFRED subsets. We report mean ± std across three seeds.

Dynamics during PPOEB-ALFRED
Online refresh (paper Table 1)0.18 ± 0.048
Frozen after offline fitting0.07 ± 0.018

Both settings use imagined PPO. Mean policy success was higher with online refresh. We have updated Table 5 and reported this comparison in Appendix A.3.


Q1 PrimitiveSkill average and the comparison with GRPO in Table 1

Thank you for catching both errors. We divided the sum of the four DWELL subtask scores by three. With equal weight for all four subtasks, the correct average is

(0.33+0.21+0+0.29)/4=0.2075≈0.21.(0.33+0.21+0+0.29)/4=0.2075\approx 0.21.

We have corrected the PrimitiveSkill average from 0.28 to 0.21 and the five-environment Overall score from 0.38 to 0.36 in Table 1, and updated Section 3.2 accordingly. While GRPO scores higher on Navigation (0.73 vs. 0.39), DWELL still achieves the highest score on PrimitiveSkill (0.21 vs. 0.17 for GRPO) and the highest Overall performance (0.36 vs. 0.28 for GRPO) among all trainable methods.

References

[1] Schulman et al. Trust Region Policy Optimization. ICML, 2015.

[2] Achiam et al. Constrained Policy Optimization. ICML, 2017.

[3] Laroche et al. Safe Policy Improvement with Baseline Bootstrapping. ICML, 2019.

[4] Rajeswaran et al. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. RSS, 2018.

AuthorsOpenReview ↗

Follow-up Discussion (Discussion Closing Soon)

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers for their constructive feedback and careful evaluation. Here, apart from our point-by-point responses, we would like to thank the reviewers for recognizing the strengths of our work and highlight the new results and clarifications added during the rebuttal.

We are encouraged that the reviewers appreciate and recognize our contributions:

  • Clear computational motivation and practical efficiency of latent imagination, avoiding future-image generation and repeated visual encoding [AjCK, DBhY].

  • Unified VLM world-model and policy architecture, demonstrating that a JEPA-style recurrent latent world model and policy can be instantiated on a shared VLM backbone via lightweight LoRA adapters [TgEB, AjCK].

  • Controlled empirical comparisons and insightful component ablations, including predictive learning, visual grounding, and world-model updates [AjCK, DBhY], alongside clear presentation [TgEB].

In the rebuttal and revised manuscript, we have directly addressed the main technical questions with new controlled experiments and clarifications:

  • We further explained the roles of posterior prediction and visual anchoring, and added 3-seed comparisons with frozen visual targets and a small standalone dynamics model, supporting prediction in the policy's native latent space [DBhY, TgEB].

  • We clarified the online training protocol used in the original EB-ALFRED experiments and released the corresponding configuration. An additional 3-seed comparison shows higher success with online dynamics updates than with frozen dynamics (0.18 vs. 0.07) [AjCK, DBhY].

  • We added ablations of reward prediction, update rollback, and behavior cloning. Gains over model-free PPO persist with direct reward prediction or without auxiliary BC (one seed each), while rollback has a limited effect on mean success (3 seeds) [AjCK, DBhY].

  • We made network inputs/outputs, latent-state interfaces, and real versus imagined data usage more explicit, and further explained DWELL's differences from LeWorldModel and AdaWorld [TgEB, DBhY].

We also thank the reviewers for identifying typo, which have been corrected in the revised manuscript. These clarifications and additional results further support DWELL's core design and its contribution to efficient online VLM agent learning. We would appreciate any remaining questions or concerns and welcome the opportunity to clarify them before the discussion closes. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.