acceptodds
Under review as a conference paper at ICLR 2027

MemJEPA: Erasure-Trained Memory Lets Latent World-Model Planners See Through Occlusion

Abstract

A latent world model can imagine the consequences of its actions in a learned state space, enabling model predictive control in robotics. However, an agent often loses sight of what it controls, such as when its arm moves out of the camera's frame. A windowed joint-embedding predictive architecture (JEPA) planner is then blind to the hidden actor, and adding memory trained only on visible clips does not help. We introduce Memory-Augmented JEPA (MemJEPA), which carries a recurrent belief through the planner's imagination and trains it with actor erasure: the actor is removed from the input clips but kept in the prediction targets. Theoretically, we show that a fixed-window latent planner cannot react to a hidden actor's motion that leaves no trace in its window, making memory of earlier frames necessary, and that erasure training against clean targets provides a learning signal for a useful carried state. Empirically, MemJEPA was tested on three image-based control tasks (reaching, pushing, navigation) with the frozen encoder of LeWorldModel (LeWM). MemJEPA raises average success under actor occlusion from 13% to  75% against a memoryless twin trained on the same data with the same actor erasure, while matching or exceeding its performance with the actor in view. The hidden-view gains transfer, at a smaller size, to a DINO-WM backbone.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, JEPA, latent planning, occlusion, recurrent memory, frozen encoders, model predictive control, partial observability, robot manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_afRCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the problem of using a JEPA-type planner in the presence of severe occlusion, where the agent may not be visible at all during inference, the idea being to add some sort of memory term to the prediction function and to train using frames were the agent has been erased.

Critical strengths

Adding some notion of memory to a planner is a good idea. The proposed method improves planning performance in the case where the agent is not visible.

Critical weaknesses

The use cases considered in the paper are not realistic. Although it is true that (part of a) robot arm may leave the camera field of view during a manipulation task, or be partially occluded, this would normally never happen in the three tasks considered in Figure 2. In addition, when the agent is completely invisible, it is hard to see how one should predict its behavior except for short time intervals since its interactions with the environment (say, obstacles or manipulated objects) during that time are not correlated with the visual inputs.

The paper is poorly written and difficult to read. The conditions under which the success rates given in the introduction and Figure 1 are obtained are not specified in the introduction. The motivation for erasing the agent during training while keeping it in the prediction targets is not well justified. Neither is the reason for using an additive memory term in Eqs. (1) and (2). The experimental protocol is not clearly described, in particular in which real conditions would the agent be hidden at inference time for the three tasks described?

At this stage unfortunately the paper is not ready for publication. It needs a better definition of the addressed problem, some justification for the proposed method, and a better definition of the experimental protocol.

Questions for authors

Many points should be clarified. For example

  • What are (C1) and (C2)? Steps? Conditions?
  • Is Eq. (2) supposed to define MemJEPA-S?
  • Clarify what you mean by reading a belief.
  • Explain your experimental setting more clearly. There are lots of numbers but it is difficult to understand what you do.
3 comments

We thank the reviewer for their feedback and questions, especially for noting that "Adding some notion of memory to a planner is a good idea". We have substantially rewritten the paper to make it clearer and to improve its flow and the quality of the writing.

The revised paper presents one model, MemJEPA, which is the submission's MemJEPA-S: LeWM's planner with PlaNet's recurrent state-space model (RSSM) as its memory. Both parts already exist, so the model is not a contribution. The results of the submission's second model are kept in App. B. The paper makes three contributions to the training recipe and mechanisms that let recurrent memory work in latent world-model planning:

  1. Erasure training, which hides the entity, the part the robot moves (its arm, its pusher or, in navigation, the agent itself), in the input frames but keeps it in the prediction targets. It makes the memory usable (§4.1).
  2. Counterfactual training, FIRM-WM's simulator replays under actions not taken. It mostly improves planning on PushT.
  3. The mechanism behind what the trained memory does. It mostly tracks where the robot's actions take the hidden entity (§4.3).

In response to your review, we added the following after the submission.

  • A rollout on the actions alone. An RSSM never trained with erasure follows only the robot's actions through the hidden frames (the RSSM open-loop rollout). It plans about as well as MemJEPA or better in five of six tests (Table 3).
  • Slips that no action causes. A slip moves the hidden pusher once; only the visible block's motion could reveal it. MemJEPA's success falls from 60.6% to 14.0% (App. F.5).
  • Occluders drawn over the scene. Under a square over the PushT pusher, MemJEPA trained with it reaches 54.8%, and the RSSM open-loop rollout 66.8%. A rectangle unseen in training breaks every model (§4.4, Table 4).
  • A test of the erasure design. Trained against erased targets, MemJEPA's RSSM succeeds in 9% of PushT episodes with the pusher hidden, against 42% with clean targets (development set, §4.1).
  • New main test sets. §1's rates now come from main sets built after the submission, with MemJEPA's bars fixed before they were run (Tables 2 and 21).

The revision also states these points plainly.

  • The problem. A planner must reach a goal image while the entity is erased from its frames, given its frames, its past actions and the goal image, with no proprioception and no mask (§3.1).
  • The model. MemJEPA is defined by Eq. 2 and has no additive term (§3.2).
  • Why erase the input but not the target. To lower its loss on a target that still shows the entity, the model must remember where the entity went (§3.3, Proposition 2).
  • The numbers in §1. MemJEPA's 78% against 15% without memory is success with the entity hidden, averaged over the three tasks, on the main sets (Table 2).
  • A test episode. The entity is visible in the first two frames (three on TwoRoom), then erased; the planner takes over once its last three frames no longer show it (§4, Figure 3).
  • Not done: any 3D task, real robot or long-horizon task. The three tasks are 2D simulation benchmarks.

The uploaded PDF holds the main text (pp. 1-10), the Limitations (p. 15) and Appendices A and C; Appendices B and D to L are in the supplementary file appendix_B_D-L.pdf, under the same letters and table numbers.

We now address the issues raised in the review.

Writing, and the numbers in §1

"The paper is poorly written and difficult to read. The conditions under which the success rates given in the introduction and Figure 1 are obtained are not specified"

We significantly rewrote the paper to address this. The revision defines each name at first use, and §4 opens with the test episode and how to read the results. Figure 1's caption now says that its bars show the entity hidden and its ticks in view, and gives the sets, seeds and intervals.

Realistic use cases

"The use cases considered in the paper are not realistic." "in which real conditions would the agent be hidden at inference time"

We agree. In our three tasks the entity would not be hidden in practice, and §1 now says so: "we erase it to control what the planner can see." Nor would a single robot with joint sensors need pixel memory for its own body (Limitations, item 3; not tested). §1's example is now PushT, not an arm leaving the camera's view.

Our results suggest where memory would have real work to do. When the hidden part is the robot's own, its actions determine it, and a rollout on them does about as well (§4.3). Memory would matter for an entity the robot neither controls nor senses, such as another robot's arm in a shared workspace, where neither proprioception nor the robot's own actions recover it. We have not tested that setting; it is the future direction our Conclusion names.

The drawn occluders above are our step toward real occlusion (§4.4).

This is Part 2 of our reply. Part 1 listed what we added and clarified, and answered the writing and the use cases; this part answers your other concerns, and Part 3 answers your four questions.

Predicting a hidden agent

"it is hard to see how one should predict its behavior except for short time intervals …"

We agree, and the revision tests this (§4.3). In our tasks the hidden entity goes where the robot's actions send it, which is why the RSSM open-loop rollout does about as well. Your case, a hidden entity moved by something other than the robot's actions, is what the slip test in Part 1 checks, and there MemJEPA's state does not correct itself from the visible block (App. F.5). Success also falls as hidden spans grow: on PushT, from 61.7% at 2 hidden frames to 10.2% at 40 (§4.4). Our own tests therefore support your point. Prediction works over short spans and where the actions determine the entity, and it degrades when something else moves the entity or the hidden span grows.

Our second control is the no-memory model, LeWM's three-frame predictor, shown where the entity was last seen. No test puts MemJEPA clearly ahead of both the RSSM open-loop rollout and the no-memory model shown the last seen position.

Why erase the input but not the target

"The motivation for erasing the agent during training while keeping it in the prediction targets is not well justified."

With the entity hidden in the input but shown in the target, the model can lower its loss only by remembering where the entity was and how it moved; §3.3 now states this. If the target hid it too, on Reacher and TwoRoom nothing in the loss would depend on where the entity is (Proposition 2, App. A.4). On PushT an erased target still shows the block that the hidden pusher moves, so we tested it there; the erased-target test in Part 1 shows that clean targets matter. Trained on ordinary clips, the same memory loses track of the entity (§4.1).

The additive memory term

"the reason for using an additive memory term in Eqs. (1) and (2)"

MemJEPA, the revised paper's model, has none; its memory is PlaNet's RSSM state, defined in Eq. 2 (§3.2). Eq. 1 is now LeWM's predictor without memory. The term belonged to the submission's second model. Addition changes LeWM's trained predictor least: it shifts the latent without transforming it, and its scale starts at zero, so training begins from LeWM's predictor (App. B.1).

Problem, method and protocol

"It needs a better definition of the addressed problem, some justification for the proposed method, and a better definition of the experimental protocol."

Done. The problem is a planner that must reach a goal image from frames in which one entity is erased, given the actions it has taken, with no proprioception and no mask (§3.1).

The method follows from it. LeWM's predictor sees only its last three frames, so two episodes that end in the same erased frames and actions look the same to it, and nothing it predicts depends on where the entity really is (App. A.4). A recurrent state can carry that information. PlaNet's RSSM fits because PlaNet plans with the same cross-entropy method as LeWM (§3.2), and erasure training is what makes its state carry the entity (§3.3).

The protocol is a two-part test episode: a scripted part that shows the entity briefly, then a planning part without it (§4, Figure 3; Part 3 gives it in plain words). One test was planned in advance with its bars fixed, the unmarked cells of Table 2, and §4 states that every other result was added later.

This is Part 3 of our reply. Parts 1 and 2 answered your concerns; this part answers your four questions.

We now answer the reviewer's questions.

"What are (C1) and (C2)? Steps? Conditions?"

Conditions, not steps. (C1) was erasure training, which hides the entity in the input frames but keeps it in the prediction targets; it is now §3.3. (C2) was advancing the memory state inside every imagined plan, so each candidate action sequence carries its own copy of the state; it is now §3.4. Without (C1), the state stays near the no-memory level (§4.1, Figure 4). With it, swapping the state between two episodes removes the gain, so the plans follow the state that (C2) carries. The labels are gone.

"Is Eq. (2) supposed to define MemJEPA-S?"

In the revision, yes. Eq. 2 defines MemJEPA, the submission's MemJEPA-S (§3.2; old names in App. C.4). In the submission it did not: MemJEPA-S had no equation, and Eq. (2) described a variant that the second model, MemJEPA-B, did not use. In words, the new Eq. 2 has a recurrent part updated from the previous state and action, a stochastic part corrected by the latent of each received frame, and a decoder that estimates the clean latent. On imagined frames nothing is received, so the decoder's output is the prediction the planner scores.

"Clarify what you mean by reading a belief."

The revised paper no longer uses the phrase; it says "state", MemJEPA's recurrent state in Eq. 2, which each received frame and action update. To plan, each candidate action sequence advances its own copy of the state on its actions and is scored by the distance of its last imagined latent to the goal image's latent. The copies are then discarded, so imagined frames never change the real state (§3.4, Eq. 4). In the submission, the phrase meant feeding the second model's memory state into its predictor (App. B.1).

"Explain your experimental setting more clearly. There are lots of numbers but it is difficult to understand what you do."

This was the rewrite's main aim. We use LeWM's three simulated tasks: a Reacher arm, a PushT pusher that moves a T-shaped block, and a TwoRoom agent that navigates two rooms. Each test episode starts with a scripted part in which the entity is visible only in the first two frames (three on TwoRoom); the simulator renders every later frame without it. The planner takes over once its last three frames no longer show the entity, so LeWM's three-frame predictor alone cannot tell where it is. From these frames, the actions taken and a goal image that shows the entity, it must reach the goal; success is the environment's own test.

We measure success with the entity hidden, and in view as a check, and compare MemJEPA with each control. The no-memory model sees only its last three frames and tests whether memory is needed. The last-seen model is also shown where the entity was last seen, and tests whether the last view suffices. The open-loop rollouts get only the actions once the entity is hidden, and test whether the actions suffice. §3.1 and §4's first paragraph describe this with Figure 3, and Table 1 lists the controls.

Conclusion

We hope these answers address your concerns, and thank you again for feedback that shaped the rewrite. If the paper now reads more clearly, we would be grateful if you would revisit your assessment; we are happy to discuss further.

Reviewer_LH3cOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies latent-space planning when the controlled entity is erased from the camera frames mid-episode. The proposed MemJEPA adds a recurrent state and trains it with "actor erasure": on half the training clips, the entity is removed from the input frames after the first two frames, while the prediction targets remain the clean next latents. At planning time, each CEM candidate advances its own copy of the state through the imagined rollout.

On the three LeWM tasks with LeWM's frozen encoder, hidden-entity success rises from 7.5/3.3/29.3% for a memoryless twin to 93.0/52.0/79.0% (MemJEPA-B) and 93.2/57.0/83.3% (MemJEPA-S), verifying the paper's claim.

Critical strengths
  1. Motivation is clear: the memory problem of JEPA is important, and the proposed resolution is feasible.

  2. Transparent reporting: experiment results are clear and reliable. The paper fixes success bars and case lists in advance, marks post-hoc comparisons and reused sets, discloses deviations, and reports failed ports. This makes the claims easy to audit.

  3. A clean mechanism ablation in the experiment. Figure 5c and Table 12 isolate the two ingredients: a carried state without erasure and erasure without a carried state both stay at the memoryless level, and swapping the state between paired episodes removes the gain. A probe decodes the hidden fingertip from the state. Within the chosen setting, this convincingly shows that the state's content is what the planner uses.

Critical weaknesses
  1. Self-constructed problem, known fix. The hidden variable is the agent's own state, determined by the last visible frame and its own actions. A real robot has proprioception, and test-time occlusion is the same simulator-mask erasure used in training. Training memory on masked inputs against clean targets is standard practice (observational dropout, Loci-Looped, MAE/V-JEPA-style masking), and MemJEPA-S is simply an RSSM trained this way.

  2. This setting cannot work in a realistic reward-free offline setting. The methodology can only work in simulations, which can simulate such block-masked data.

  3. Toy experiments. All three tasks are 2D point tracking, selected by a ≥20-point occlusion drop; the only 3D manipulation, Cube, Robomimic, failed before any memory test, and there is no real robot. Introducing long-horizon tasks where memory is important could improve this paper's quality.

  4. Missing baselines: proprioceptive state, a world-model rollout from the last visible frame with the executed actions, and a tuned long-context predictor trained with erasure. The untuned 10-frame window already ties on TwoRoom (78.0 vs. 79.0%), a 40-frame window wins at 33 hidden frames (45.5 vs. 24.8%), and a last-visible cache wins on recall by 35.5 points.

Questions for authors
  1. Why is the agent denied proprioception and its own action history? Which real robot setting needs pixel memory for its own state?
  2. How do the proprioceptive state and a predictor rolled forward from the last visible frame with the executed actions, compare with MemJEPA?
  3. Does random spatiotemporal masking (as in VideoMAE or V-JEPA) work as well as entity-specific erasure with simulator masks?
4 comments

We thank the reviewer for their thorough and expert review, especially for noting that our reporting "makes the claims easy to audit" and for the "clean mechanism ablation". We have substantially rewritten the paper to make it clearer and to improve its flow and the quality of the writing.

The revised paper presents one model, MemJEPA, which is the submission's MemJEPA-S: LeWM's planner with PlaNet's recurrent state-space model (RSSM) as its memory. Both parts already exist, so the model is not a contribution. The results of the submission's second model are kept in App. B. The paper makes three contributions to the training recipe and mechanisms that let recurrent memory work in latent world-model planning:

  1. Erasure training, which hides the entity in the input frames but keeps it in the prediction targets. With the entity hidden, MemJEPA averages 78% success against 15% for the no-memory model (Table 2).
  2. Counterfactual training, FIRM-WM's simulator replays under actions not taken. It mostly improves planning on PushT.
  3. The mechanism behind what the trained memory does. It mostly tracks where the robot's actions take the hidden entity.

In response to your review, we added the following after the submission.

  • Your world-model rollout, on the executed actions. An RSSM never trained with erasure, told which frames are erased and advanced on the actions alone (the RSSM open-loop rollout), plans about as well as MemJEPA or better in five of six tests (Table 3); LeWM's predictor rolled out the same way restores Reacher and TwoRoom but not PushT (23.0% against 54.2%), likely because of its three-frame predictor (Table 2).
  • Tests where the actions do not determine the hidden thing. When a slip that no action commands moves the visible pusher with the block hidden, MemJEPA leads the RSSM rollout by 18.0 points but the no-memory model shown the last seen position (the last-seen model, your "last-visible cache") stays within 7.0, and when the hidden pusher slips, MemJEPA falls from 60.6% to 14.0% (§4.3, App. F.5).
  • Your tuned long-context predictor. MemJEPA leads a tuned longer-window model trained with erasure by 15.3 points on PushT and 5.2 on TwoRoom, short of our 20-point bar, and the window does better in view on PushT (Table 53; not run on Reacher).
  • Proprioception, as two privileged stand-ins. A no-memory model given the true state matches MemJEPA on Reacher (94.5% against 94.8%), and one shown the pusher where the actions put it ties MemJEPA on PushT (56.4% against 54.2%; App. F.2, F.3).
  • Masks without the simulator, and random masks. Colour-segmenter masks in training and at test keep a 45.2-point lead over the no-memory model on PushT, while random blocks and a random disc gain 15.6 and 5.6, short of the 20-point bar (§4.4).
  • Occluders drawn over the scene. Trained with a square over the PushT pusher, MemJEPA reaches 54.8% against the RSSM rollout's 66.8%, and under occluders unseen in training memory meets none of our bars (§4.4).
  • New main test sets. Table 2's rates come from sets built after the submission (Table 21); your summary's rates are on the earlier sets (Table 18).

The revision also states these points plainly.

  • The claim. No test puts MemJEPA clearly ahead of both the RSSM open-loop rollout and the no-memory model shown the last seen position; hiding what the actions determine is not by itself a test of memory (§1).
  • What is known and what is new. Erasure training combines two known ideas (§2); the contribution is which training makes the state usable and why ordinary training fails (§4.1).
  • The model. MemJEPA is PlaNet's RSSM trained with erasure, chosen after both forms' results were known (App. B).
  • Scope. The three tasks are 2D simulation benchmarks, chosen because hiding the entity costs the no-memory model at least 20 points (Limitations items 1 and 2).
  • Not done: a trained proprioceptive pathway; any 3D task, real robot or long-horizon task; the tuned longer-window model on Reacher; random masks beyond PushT.

The uploaded PDF holds the main text (pp. 1-10), the Limitations (p. 15) and Appendices A and C; Appendices B and D to L are in the supplementary file appendix_B_D-L.pdf, under the same letters and table numbers.

Our answers to the issues you raised follow in two further comments. Part 2 covers the self-constructed problem, simulation and toy experiments. Part 3 covers the known fix, the missing baselines and your three questions.

This is Part 2 of our reply. Part 1 listed what we added. This part answers the self-constructed problem, simulation and toy experiments. Part 3 answers the known fix, the missing baselines and your questions.

We now address the issues raised in the review.

Self-constructed problem

"The hidden variable is the agent's own state, determined by the last visible frame and its own actions."

This is correct for our three tasks, and the paper's third contribution now measures it with your rollout (Part 1).

The rollout also explains why ordinary training fails. It runs the same checkpoints as an RSSM trained on ordinary clips, which loses the entity because it updates on frames that no longer show it (§4.1). Skipping that update requires knowing which frames are erased (Table 1, row 5); an erasure-trained state needs no mask at test, though it was trained with simulator masks (§4.4).

A test of memory needs something the actions do not determine. We ran three such tests, with MemJEPA retrained with the block erased for the block tests. Success (%), entity hidden:

TestMemJEPARSSM rolloutLast-seen model
PushT block hidden63.859.454.4
Block hidden, visible pusher slips27.19.120.1
TwoRoom agent hidden in one region45.232.644.0

Hiding the block does not separate MemJEPA from the rollout. Where MemJEPA does lead the rollout, after the slip and in the region, the last-seen model stays close (Tables 3 and 4).

Two further results go against memory. When the hidden pusher slips, the state barely corrects itself from the visible block (§4.3). A displaced pusher that then pushes the hidden block gave no answer (App. F.8).

"A real robot has proprioception"

Agreed. We did not train a proprioceptive pathway; instead we ran two privileged stand-ins. One gives the no-memory model the entity's true state at every real frame; the other draws the pusher into the erased frames where the actions put it. Success (%), entity hidden:

ModelReacherPushT
MemJEPA94.854.2
No memory + true state94.543.0
No memory + drawn pusherno pusher56.4

On PushT the true-state model under-uses a state it receives only at real frames (Limitations item 4), and on TwoRoom it did not learn to use it (App. F.2). Released LeWM shown the drawn pusher is 15.4 points ahead of MemJEPA (App. F.3).

So in our tasks a planner given the robot's own state does not need pixel memory. Our planner, like LeWM's, gets no proprioception (§3.1), and Limitations item 3 says "a single robot with joint sensors would not need pixel memory for its own body (not tested)" (p. 15).

Simulator masks, in training and at test

"test-time occlusion is the same simulator-mask erasure used in training"

"The methodology can only work in simulations, which can simulate such block-masked data."

The first is correct for our main test (Table 2), and the second in part. We varied both sides.

In training, masks computed from the recorded frames keep most of the gain; random masks do not. Lead over the no-memory model trained the same way (points), against our 20-point bar:

Training maskTaskLead
Colour segmenter, also at testPushT45.2
Background subtractionTwoRoom39.7
Random blocks, as in V-JEPAPushT15.6
Random disc of the pusher's areaPushT5.6

The segmenter is 5.8 points below simulator masks, and TwoRoom used simulator erasure at test (§4.4, App. G.1). So the mask need not come from a simulator, but it must follow the entity.

At test, under a square drawn over the PushT pusher, the state must be trained with the square, and even then the RSSM rollout does better (Part 1; App. G.5 to G.7).

Some needs remain:

  • the entity visible in every recorded frame (§3.3);
  • a resettable simulator for counterfactual training (§4.2);
  • simulator-rendered frames; whether natural occlusion trains the state is untested (Limitations item 1, p. 15).

Toy experiments

"All three tasks are 2D point tracking, selected by a ≥20-point occlusion drop; the only 3D manipulation … failed before any memory test, and there is no real robot"

On scope we agree (Part 1). Given this selection, the Limitations add that "the gains say nothing about how often that holds" (item 2, p. 15).

The Cube and robomimic failures are in Table 22 (App. C.5, I.3). Not done: any 3D task, real robot or long-horizon task; the hidden-span test (§4.4) is not one.

The 2D tasks do allow controlled tests. For example, swapping the state between two episodes that differ only in the hidden entity's path removes the gain (§4.1).

This is Part 3 of our reply. Part 1 listed what we added, and Part 2 answered the self-constructed problem, simulation and toy experiments. This part answers the known fix, the missing baselines and your three questions.

Known fix

"Training memory on masked inputs against clean targets is standard practice … MemJEPA-S is simply an RSSM trained this way."

To our knowledge, this is the first study of what makes a recurrent memory usable for planning by a frozen-encoder JEPA planner of the DINO-WM and LeWM line once the entity it moves is hidden. These planners plan from their last few frames and are evaluated fully observed, so they lose track of an entity hidden for longer (§1, §2). We show that adding a recurrent state the standard way does not fix this, identify why, and give the training recipe that makes the state usable. The nearest work differs: FIRM-WM carries a recurrent state in CEM on LeWM's tasks, fully observed; ELVIS plans under occlusion in an RSSM trained by reconstruction, not a JEPA planner; CLWM, concurrent and evaluated under occlusion, trains its encoder end to end with a contrastive objective (App. K).

The ingredients are known, and §2 cites observational dropout, Loci-Looped, MAE and V-JEPA for them; MemJEPA is that RSSM. What is new is which training makes the state usable:

  • Ordinary clips: the RSSM stays near the no-memory level, because it updates on erased frames.
  • Erased targets: an RSSM reaches 9% with the PushT pusher hidden, against 42% with clean targets (development set, tested once).
  • Random masks: short of the bar (Part 2).

So the mask must follow the entity and the target must show it.

Missing baselines

"proprioceptive state, a world-model rollout … and a tuned long-context predictor trained with erasure"

The rollout and stand-ins are in Parts 1 and 2. The tuned longer-window model, trained with erasure, was not run on Reacher:

PushTTwoRoom
MemJEPA's lead, entity hidden (points)15.35.2
Window, in view (%)79.782.5
MemJEPA, in view (%)69.686.5

MemJEPA leads the window by less than our 20-point bar, and on PushT the window plans better in view (Table 53).

"The untuned 10-frame window already ties on TwoRoom … and a last-visible cache wins on recall by 35.5 points."

These were measured on the submission's MemJEPA-B (App. B) and hold. The tuned window also matches it on TwoRoom, and on a static recall variant of PushT, run with that form only, the last-seen model beats it by 35.5 points (App. I.2).

On the Reacher 33-frame set, the 40-frame window reaches 46.3% against MemJEPA's 79.8%, but the LeWM open-loop rollout reaches 92.3% (Table 49).

We now answer the reviewer's questions.

"Why is the agent denied proprioception and its own action history? Which real robot setting needs pixel memory for its own state?"

The action history is not withheld; every model receives its executed actions (§3.1). Proprioception is withheld because we keep LeWM's pixel-only interface, to ask what a pixel planner can recover. For the robot's own state we cannot name a real setting that needs pixel memory, and our stand-ins agree (Part 2; Limitations item 3).

The setting that motivates the work, untested, is multi-robot work, where one robot loses sight of another robot's arm or of a part another robot moves out of view. We expect proprioception not to cover it, since it would report the robot's own joints but not:

  • another robot's arm, unless the two share state in a common calibrated frame;
  • a tool or object the robot is not rigidly holding;
  • anything moved by another robot or by contact.

There, neither proprioception nor rolling the robot's own actions forward would recover the hidden part, so a state would have to carry it. §1 and the Conclusion name this as untested future work; in our slip tests, the nearest evidence, the state barely corrects for motion the actions did not cause.

"How do the proprioceptive state and a predictor rolled forward … compare with MemJEPA?"

Part 1's first bullet gives the rollouts, and Part 2's table the stand-ins. For the robot's own body, the better form of each does about as well as memory.

"Does random spatiotemporal masking … work as well as entity-specific erasure with simulator masks?"

No, on PushT (Part 2's table). The random disc overlaps the pusher in only 2.0% of frames (App. G.1), so the loss rarely asks the state to carry it. Random masks were not run on Reacher or TwoRoom.

Conclusion

Thank you for a review that changed the paper's central claim. If the revised claim now matches the evidence, we would be grateful if you would revisit your assessment, and we are happy to discuss further.

I thank the authors for the extensive revision and the new experiments. The rebuttal confirms my main concerns. MemJEPA is PlaNet's RSSM trained with known masking ideas. No test puts it clearly ahead of the open-loop rollout or the last-seen baseline. Also, the evaluation is still limited to 2D simulation. I therefore keep my score.

Reviewer_QYvPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem in World Models where the entity being controlled moves out of the frame of visual observation. The authors suggest the use of memory in order to maintain a belief on the current position of the entity, thus avoiding a collapse of planning. The authors also argue that this memory is ineffective if the training of set always contain the entity and thus augment training samples by erasing the entity in recent frames, forcing the model to carry information about the entity from past frames in order to predict the next latent state. The empirical results are positive and show that MemJEPA outperforms naive baselines in environments where the entity is hidden, and remains competitive in the standard setting when it is observable.

Critical strengths

The setting of the paper is practically relevant and the methods used to address the problem are reasonable and sound. The intuition is clear and the writing is easy to follow and understand. The empirical results are promising and sufficiently thorough.

Critical weaknesses

I don't see any critical weaknesses of the problem setting or the approach, however, I would've liked some experiments that actually impose the practical problem that the paper is based on. Specifically, the authors motivate the paper using the example of a robot where the arm moves out of frame during a task. However, experiments are carried out in synthetic environments where all but the entity that is being controlled is hidden. I find that these two are mildly disconnected.

As an example for a more related experimental setup, consider the two-room setting in Figure 3 where the lower half of the starting room is always occluded. This would mean that the entity in this quadrant is always hidden, but is visible at all other places. The current setup is more analogous to a practical case where the robot temporarily loses access to its camera but is still expected to make actions towards a goal.

I am also mildly bothered by the first two "contributions" of that the authors list. These are necessary experimental validations of their motivation rather than actual contributions and should not be treated as major novelties.

Questions for authors

I thank the authors for their work on this paper. I found it a good paper to read. My only concern is detailed as a weakness, but I find that either including experiments to fit the motivating example better, or simply changing the motivating example itself would address my concern. I do note that I would lean more towards the former and only use the latter if such experimentation is infeasible.

3 comments

We thank the reviewer for their thoughtful and constructive review, especially for noting that "The intuition is clear and the writing is easy to follow and understand", for finding the submission "a good paper to read", and for proposing a concrete test: an erased region in TwoRoom's starting room. We ran that test, drew occluders over the scene and changed the motivating example.

The region test came out against the claim that trained memory is needed. Two results still stand. With the entity hidden, MemJEPA averages 78% success, against 15% for the no-memory model (LeWM's three-frame predictor, fine-tuned on the same erased clips), and it meets every bar fixed in advance (Table 2). Swapping its state between two episodes that differ only in the hidden entity's path removes the gain (Figure 4).

In response to your review we made the following changes.

  • We ran your region test on three new sets (§4.4, Tables 4 and 52). MemJEPA leads the no-memory model on every set, but on sets 2 and 3 a no-memory model shown the agent's last seen position does about as well.
  • We drew occluders over the scene (§4.4, Table 4). Under a square it was trained with, MemJEPA beats the no-memory model, but an RSSM never trained with erasure, told which frames are erased and advanced on the robot's actions alone (the RSSM open-loop rollout), does better. A standing square gave no valid test, and a rectangle unseen in training breaks every model.
  • We tested the case of a robot that loses its camera (§4.3, Table 3). The RSSM open-loop rollout, which advances on the actions alone, is within 4 points of MemJEPA on Reacher and TwoRoom and ahead on PushT.
  • We changed the motivating example to PushT, a task we test, and §1 now says that in our tasks the entity would not be hidden in practice.
  • We rewrote the contribution list (p. 2). It lists no model, and counterfactual training is credited to FIRM-WM, with no novelty claimed.
  • Not done: no experiment has an arm leaving a camera's view, and nothing is in 3D or on a real robot.

Pointers to the main text (pp. 1-10), the Limitations list (p. 15), Appendix A and Appendix C refer to the uploaded PDF; Appendices B and D to L, with the tables they hold, are in the supplementary material (appendix_B_D-L.pdf in the zip), under the same letters and table numbers.

Part 1 answers the gap between the motivating example and the experiments. Part 2 answers your region test, the case of a robot that loses its camera, the first two contributions and your question.

We now address the issues raised in the review.

Experiments that impose the practical problem

"I would've liked some experiments that actually impose the practical problem … these two are mildly disconnected."

We agree, and thank you for raising this point. Table 2's test erases the entity after a short visible prefix and draws nothing over it (Scope, p. 2). §1 now says this plainly:

"In PushT and our other two tasks the entity would not be hidden in practice; we erase it to control what the planner can see."

To come closer to the practical problem, we then drew occluders over the scene. These experiments are new and post hoc (§4.4, Table 4):

  • A square over the PushT pusher from step 10. Trained with the square, MemJEPA plans at 54.8%, against 8.0% for the no-memory model. However, the RSSM open-loop rollout reaches 66.8% (App. G.7).
  • A square standing in the scene from the first frame. Both designs we tried failed the validity bar fixed beforehand, so this gave no valid test of memory (App. G.8).
  • A rectangle drawn over the lower half of both rooms, which covers your region, unseen in training. It breaks every model; each reaches at most 3.2% (App. G.6).

What we have not done: no experiment has an arm leaving a camera's view, and nothing is in 3D or on a real robot. The three tasks are 2D simulation benchmarks (Limitations, p. 10).

This is Part 2 of our reply. Part 1 summarized what we changed and answered the gap between the motivating example and the experiments; this part answers your region test, the case of a robot that loses its camera, the first two contributions and your question.

As in Part 1, the no-memory model is LeWM's three-frame predictor fine-tuned on the same erased clips, and the RSSM open-loop rollout is an RSSM never trained with erasure, told which frames are erased and advanced on the robot's actions alone.

The erased region in the starting room

"consider the two-room setting in Figure 3 where the lower half of the starting room is always occluded"

We appreciate this concrete suggestion, and we ran it on three new sets (post hoc; §4.4, Tables 4 and 52). The agent is erased while it is in that half of the room. A scripted path shows the agent first, then walks it into the region. Success (%):

Set (cases)MemJEPANo memory
1 (200)42.77.5
2 (193)41.59.7
3 (185)45.210.1

On set 3, whose bars were fixed first, we also ran the controls. The RSSM open-loop rollout reaches 32.6%. The no-memory model shown the agent's last seen position (the last-seen model) reaches 44.0%.

MemJEPA leads the RSSM rollout by 12.6 [7.7, 17.5], which clears its 10-point bar. It leads the LeWM open-loop rollout (LeWM's predictor fed its own predictions on the executed actions) by 5.6, short of the same bar; on sets 1 and 2 that lead is 8.2 and 9.7. However, the last-seen model does about as well as MemJEPA: MemJEPA's lead over it is +1.3 [-5.8, +8.5] on set 3. On set 2 that lead is -2.8 [-9.7, +4.0] (used cases, rerun on a second machine; App. G.4).

Two variants failed. MemJEPA trained with the agent erased in the region missed all three bars fixed for it. With the region erased from the first step, 66 of 71 cases show no agent before planning, and MemJEPA fails (13.6%, against 93.0% in view); we think this is because it never saw its start.

So no test puts MemJEPA clearly ahead of both the RSSM open-loop rollout and the no-memory model shown the last seen position.

When the robot loses its camera

"The current setup is more analogous to a practical case where the robot temporarily loses access to its camera"

We agree, and §4.3 now tests what follows from it. The RSSM open-loop rollout advances on the robot's actions alone, as a robot without its camera would. With the robot's own body hidden, it is within 4 points of MemJEPA on Reacher and TwoRoom, and 5.2 points ahead on PushT (Table 3).

The first two contributions

"These are necessary experimental validations of their motivation rather than actual contributions"

We agree, and thank you for this point. The failure without memory and its control are now §1's problem statement, and §1 lists no model as a contribution. The list now reads (p. 2, abridged):

  1. Erasure training. We hide the entity in the input frames but keep it in the prediction targets. … Trained on ordinary clips, the RSSM … updates on frames without the entity and loses track of it.
  2. Counterfactual training. … following FIRM-WM … it mostly improves planning (§4.2).
  3. Mechanism on what the trained memory does. We ask whether it does more than track where the robot's actions take the entity. Mostly not: an RSSM never trained with erasure, rolled out open loop on those actions, plans about as well as MemJEPA or better in five of six tests … Hiding what actions determine is not by itself a test of memory.

Item 2 is borrowed from FIRM-WM, and we claim no novelty for it (§3.3).

We now answer the reviewer's question.

Experiments, or a new motivating example

"either including experiments to fit the motivating example better, or simply changing the motivating example itself"

We did both. You preferred experiments, so we first ran your region test (above) and drew occluders over the scene (Part 1). They come closer to the original example, but none has an arm leaving a camera's view.

So we also changed the example. §1 now uses PushT, a task we test. It also says why hiding the robot's own body is not by itself a test of memory: the pusher goes where the robot's commands send it.

Conclusion

We hope our responses have addressed your concern, and we thank you again for your thoughtful and constructive feedback; your region test is now part of the paper. If you find that the revised paper fits its evidence better, we would be grateful if you would consider strengthening your support for it. We are more than happy to discuss any further questions.

Reviewer_QYvPOpenReview ↗

Reply to author response

I thank the authors for their thorough work on the rebuttal. I will keep my positive recommendation.

Comments on the paper

1 comment

We thank the reviewers for their thorough and constructive reviews, and for noting that "the memory problem of JEPA is important" and that the "experiment results are clear and reliable" (LH3c), that the "empirical results are promising and sufficiently thorough" (QYvP), and that the method "improves planning performance in the case where the agent is not visible" (afRC).

Contributions

The revised paper presents one model, MemJEPA, the submission's MemJEPA-S: LeWM's planner with PlaNet's recurrent state-space model (RSSM), both existing parts. The submission's second model is kept in App. B. The revised paper makes three contributions to the training recipe and mechanisms that would allow recurrent memory to be successful in latent world model planning:

  1. Erasure training, which hides the entity (the part the robot moves) in the input frames but keeps it in the prediction targets. With the entity hidden, MemJEPA averages 78% success, against 15% for the no-memory model (Table 2). Trained on ordinary clips, the same RSSM stays near the no-memory level.
  2. Counterfactual training, FIRM-WM's simulator replays under actions not taken. It adds 10.9 points on PushT with the pusher hidden and 8.3 in view, so it is mostly a planning gain (§4.2).
  3. The mechanism behind what the trained memory does. The RSSM open-loop rollout, an RSSM never trained with erasure and advanced on the robot's actions alone, plans about as well as MemJEPA or better in five of six tests (Table 3).

MemJEPA clears its 10-point bar over the RSSM rollout only in test 6, where the block is hidden and the visible pusher slips (+18.0), and in the region test (+12.6); in both, the last-seen model (the no-memory model shown the last seen position) is within 7.0 and 1.3 points of MemJEPA. No test puts MemJEPA clearly ahead of both the RSSM open-loop rollout and the last-seen model (Tables 3, 4 and 52).

What is new, and why it matters

Our motivating setting is multi-robot work, where one robot cannot see another robot's arm or a part that leaves its view. Our tasks hide the robot's own body, a controlled stand-in; we agree this is far from real use (§1).

Planners of the DINO-WM and LeWM line predict from their last few frames and are tested fully observed (§2). Once the entity is hidden longer, nothing they receive says where it is (§3.1), and a recurrent state trained on ordinary clips loses track of it (§4.1).

To our knowledge, this is the first study of what makes recurrent memory usable by a frozen-encoder JEPA planner of this line with the entity it moves hidden. The recipe, with a mask that follows the entity, gives a state the planner uses with no mask at test. The mechanism, that this state mostly tracks where the robot's actions take the entity, shows that a real test of memory needs something hidden the actions do not determine, as in the multi-robot case, our Conclusion's next direction.

What the revision adds

  • Controls that decide the claim (LH3c). The RSSM and LeWM open-loop rollouts on the executed actions (§4.3), and for proprioception only two privileged stand-ins (App. F.2, F.3); LH3c's central objection held, and the paper is rebuilt around it.
  • A tuned longer-window model trained with erasure (LH3c). MemJEPA leads it by 15.3 points on PushT and 5.2 on TwoRoom, short of the 20-point bar (§4.1, Table 53).
  • Closer to real occlusion (QYvP, afRC). The region test on three new sets and drawn occluders (§4.4).
  • Masks without the simulator (LH3c). Masks from recorded frames keep most of the gain, random ones fall short (§4.4).
  • New main test sets (Tables 2 and 21), with bars fixed in advance; Table 2's rates differ from the submitted ones.
  • The writing (LH3c, afRC). Plain names, one equation per subsection (§3.1 to §3.4) and one table of baselines (Table 1).

LeWM's tasks, data, frozen encoder, planner and evaluator are unchanged; MemJEPA was chosen after both models' results were known. We have substantially rewritten the paper to make it clearer and to improve the flow of the writing.

Future directions

For future work, we plan to train a planner with proprioception as its own pathway on real robot hardware, particularly for long horizon tasks.

We thank the reviewers again for reviews that changed the paper, and welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.