- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of using a JEPA-type planner in the presence of severe occlusion, where the agent may not be visible at all during inference, the idea being to add some sort of memory term to the prediction function and to train using frames were the agent has been erased.
Critical strengths
Adding some notion of memory to a planner is a good idea. The proposed method improves planning performance in the case where the agent is not visible.
Critical weaknesses
The use cases considered in the paper are not realistic. Although it is true that (part of a) robot arm may leave the camera field of view during a manipulation task, or be partially occluded, this would normally never happen in the three tasks considered in Figure 2. In addition, when the agent is completely invisible, it is hard to see how one should predict its behavior except for short time intervals since its interactions with the environment (say, obstacles or manipulated objects) during that time are not correlated with the visual inputs.
The paper is poorly written and difficult to read. The conditions under which the success rates given in the introduction and Figure 1 are obtained are not specified in the introduction. The motivation for erasing the agent during training while keeping it in the prediction targets is not well justified. Neither is the reason for using an additive memory term in Eqs. (1) and (2). The experimental protocol is not clearly described, in particular in which real conditions would the agent be hidden at inference time for the three tasks described?
At this stage unfortunately the paper is not ready for publication. It needs a better definition of the addressed problem, some justification for the proposed method, and a better definition of the experimental protocol.
Questions for authors
Many points should be clarified. For example
- What are (C1) and (C2)? Steps? Conditions?
- Is Eq. (2) supposed to define MemJEPA-S?
- Clarify what you mean by reading a belief.
- Explain your experimental setting more clearly. There are lots of numbers but it is difficult to understand what you do.
3 comments
This is Part 2 of our reply. Part 1 listed what we added and clarified, and answered the writing and the use cases; this part answers your other concerns, and Part 3 answers your four questions.
Predicting a hidden agent
"it is hard to see how one should predict its behavior except for short time intervals …"
We agree, and the revision tests this (§4.3). In our tasks the hidden entity goes where the robot's actions send it, which is why the RSSM open-loop rollout does about as well. Your case, a hidden entity moved by something other than the robot's actions, is what the slip test in Part 1 checks, and there MemJEPA's state does not correct itself from the visible block (App. F.5). Success also falls as hidden spans grow: on PushT, from 61.7% at 2 hidden frames to 10.2% at 40 (§4.4). Our own tests therefore support your point. Prediction works over short spans and where the actions determine the entity, and it degrades when something else moves the entity or the hidden span grows.
Our second control is the no-memory model, LeWM's three-frame predictor, shown where the entity was last seen. No test puts MemJEPA clearly ahead of both the RSSM open-loop rollout and the no-memory model shown the last seen position.
Why erase the input but not the target
"The motivation for erasing the agent during training while keeping it in the prediction targets is not well justified."
With the entity hidden in the input but shown in the target, the model can lower its loss only by remembering where the entity was and how it moved; §3.3 now states this. If the target hid it too, on Reacher and TwoRoom nothing in the loss would depend on where the entity is (Proposition 2, App. A.4). On PushT an erased target still shows the block that the hidden pusher moves, so we tested it there; the erased-target test in Part 1 shows that clean targets matter. Trained on ordinary clips, the same memory loses track of the entity (§4.1).
The additive memory term
"the reason for using an additive memory term in Eqs. (1) and (2)"
MemJEPA, the revised paper's model, has none; its memory is PlaNet's RSSM state, defined in Eq. 2 (§3.2). Eq. 1 is now LeWM's predictor without memory. The term belonged to the submission's second model. Addition changes LeWM's trained predictor least: it shifts the latent without transforming it, and its scale starts at zero, so training begins from LeWM's predictor (App. B.1).
Problem, method and protocol
"It needs a better definition of the addressed problem, some justification for the proposed method, and a better definition of the experimental protocol."
Done. The problem is a planner that must reach a goal image from frames in which one entity is erased, given the actions it has taken, with no proprioception and no mask (§3.1).
The method follows from it. LeWM's predictor sees only its last three frames, so two episodes that end in the same erased frames and actions look the same to it, and nothing it predicts depends on where the entity really is (App. A.4). A recurrent state can carry that information. PlaNet's RSSM fits because PlaNet plans with the same cross-entropy method as LeWM (§3.2), and erasure training is what makes its state carry the entity (§3.3).
The protocol is a two-part test episode: a scripted part that shows the entity briefly, then a planning part without it (§4, Figure 3; Part 3 gives it in plain words). One test was planned in advance with its bars fixed, the unmarked cells of Table 2, and §4 states that every other result was added later.
This is Part 3 of our reply. Parts 1 and 2 answered your concerns; this part answers your four questions.
We now answer the reviewer's questions.
"What are (C1) and (C2)? Steps? Conditions?"
Conditions, not steps. (C1) was erasure training, which hides the entity in the input frames but keeps it in the prediction targets; it is now §3.3. (C2) was advancing the memory state inside every imagined plan, so each candidate action sequence carries its own copy of the state; it is now §3.4. Without (C1), the state stays near the no-memory level (§4.1, Figure 4). With it, swapping the state between two episodes removes the gain, so the plans follow the state that (C2) carries. The labels are gone.
"Is Eq. (2) supposed to define MemJEPA-S?"
In the revision, yes. Eq. 2 defines MemJEPA, the submission's MemJEPA-S (§3.2; old names in App. C.4). In the submission it did not: MemJEPA-S had no equation, and Eq. (2) described a variant that the second model, MemJEPA-B, did not use. In words, the new Eq. 2 has a recurrent part updated from the previous state and action, a stochastic part corrected by the latent of each received frame, and a decoder that estimates the clean latent. On imagined frames nothing is received, so the decoder's output is the prediction the planner scores.
"Clarify what you mean by reading a belief."
The revised paper no longer uses the phrase; it says "state", MemJEPA's recurrent state in Eq. 2, which each received frame and action update. To plan, each candidate action sequence advances its own copy of the state on its actions and is scored by the distance of its last imagined latent to the goal image's latent. The copies are then discarded, so imagined frames never change the real state (§3.4, Eq. 4). In the submission, the phrase meant feeding the second model's memory state into its predictor (App. B.1).
"Explain your experimental setting more clearly. There are lots of numbers but it is difficult to understand what you do."
This was the rewrite's main aim. We use LeWM's three simulated tasks: a Reacher arm, a PushT pusher that moves a T-shaped block, and a TwoRoom agent that navigates two rooms. Each test episode starts with a scripted part in which the entity is visible only in the first two frames (three on TwoRoom); the simulator renders every later frame without it. The planner takes over once its last three frames no longer show the entity, so LeWM's three-frame predictor alone cannot tell where it is. From these frames, the actions taken and a goal image that shows the entity, it must reach the goal; success is the environment's own test.
We measure success with the entity hidden, and in view as a check, and compare MemJEPA with each control. The no-memory model sees only its last three frames and tests whether memory is needed. The last-seen model is also shown where the entity was last seen, and tests whether the last view suffices. The open-loop rollouts get only the actions once the entity is hidden, and test whether the actions suffice. §3.1 and §4's first paragraph describe this with Figure 3, and Table 1 lists the controls.
Conclusion
We hope these answers address your concerns, and thank you again for feedback that shaped the rewrite. If the paper now reads more clearly, we would be grateful if you would revisit your assessment; we are happy to discuss further.
We thank the reviewer for their feedback and questions, especially for noting that "Adding some notion of memory to a planner is a good idea". We have substantially rewritten the paper to make it clearer and to improve its flow and the quality of the writing.
The revised paper presents one model, MemJEPA, which is the submission's MemJEPA-S: LeWM's planner with PlaNet's recurrent state-space model (RSSM) as its memory. Both parts already exist, so the model is not a contribution. The results of the submission's second model are kept in App. B. The paper makes three contributions to the training recipe and mechanisms that let recurrent memory work in latent world-model planning:
In response to your review, we added the following after the submission.
The revision also states these points plainly.
The uploaded PDF holds the main text (pp. 1-10), the Limitations (p. 15) and Appendices A and C; Appendices B and D to L are in the supplementary file
appendix_B_D-L.pdf, under the same letters and table numbers.We now address the issues raised in the review.
Writing, and the numbers in §1
We significantly rewrote the paper to address this. The revision defines each name at first use, and §4 opens with the test episode and how to read the results. Figure 1's caption now says that its bars show the entity hidden and its ticks in view, and gives the sets, seeds and intervals.
Realistic use cases
We agree. In our three tasks the entity would not be hidden in practice, and §1 now says so: "we erase it to control what the planner can see." Nor would a single robot with joint sensors need pixel memory for its own body (Limitations, item 3; not tested). §1's example is now PushT, not an arm leaving the camera's view.
Our results suggest where memory would have real work to do. When the hidden part is the robot's own, its actions determine it, and a rollout on them does about as well (§4.3). Memory would matter for an entity the robot neither controls nor senses, such as another robot's arm in a shared workspace, where neither proprioception nor the robot's own actions recover it. We have not tested that setting; it is the future direction our Conclusion names.
The drawn occluders above are our step toward real occlusion (§4.4).