acceptodds
Under review as a conference paper at ICLR 2027

Controllability of Pretrained Visual Representations under Start-Controlled Evaluation on Multi-Object Scenes

Abstract

Turning pretrained visual representations into behavior requires a procedure for sequential decision making, such as test-time search (CEM) via model-predictive control in the latent space or a goal-conditioned inverse-dynamics model (GC-IDM). We contribute a start-controlled evaluation protocol, an extended evaluation suite with progressively greater object counts, and a long-horizon predictive architecture for learning visual representations. Our start-controlled protocol shows that test-time search methods fail on 3D cube manipulation (0.0% success rate) and that GC-IDM still succeeds (68.9–84.9% success rate). Our extended evaluation suite scales the cube count up to four cubes, revealing that prior work's (LeWM and Sub-JEPA) representations become insufficient for downstream control under GC-IDM. The prior work's random-start evaluation protocol hides both issues, inflating LeWM's four-cube performance from 17.1% to 72.6%. In contrast, on four-cube tasks our proposed architecture learns representations that outperform those of LeWM and Sub-JEPA by up to 20.4 and 35.6 percentage points respectively. Our architecture predicts 50 environment steps of future latents in one pass and, through a structured attention mask, simultaneously learns inverse dynamics from its internal representations of consecutive states; a patch decoder maps each latent back to the encoder's patch representation, retaining object-level detail. Our representations are also more robust to colour distractors in one-cube pick-and-place. Ablations show that our method depends strongly on the training horizon, inverse-dynamics loss, and patch representation reconstruction loss, with the inverse-dynamics term alone worth a third to two fifths of control performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Pretrained vision representations, Self-supervised learning, Representation learning for control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_ujJ1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to tackle long-horizon planning in JEPA type world models. The main contributions are:

  1. A longer-horizon training objective, instead of predicting next state, predict 10 latent states (50 frames, 5 skip) and actions into the future
  2. A modification of evaluation setting where instead of random start, always start at the beginning

The authors perform simple simulation evaluations on tasks like pick and place, stacking, cyclic rearrangement for up to 4 objects, and show that their method is more robust to increased object number and in the controlled-start setting which requires long-horizon planning. While their method does not uniformly surpass baselines, it wins more often.

While it is interesting to see the effect of longer-horizon training objective, I have questions about whether the training objective makes sense, since there are many possibilities of future states and actions conditioned on current state and action. I wonder whether this training objective just overfits to the task distribution better - the small perturbations in object numbers etc. does not really test generalization. I feel the method would require more justification and thorough evaluation if the claim is that this training recipe yields better general representation.

Critical strengths
  1. The long horizon angle is interesting - it is an important aspect in world modeling and action planning
  2. The authors caught the effect of an evaluation setting difference, i.e. random frame start vs. controlled start
  3. Though small-scale, the ablations are quite comprehensive
Critical weaknesses
  1. The method design does not seem well-justified - could the authors explain why the model should predict a specific future state and action sequence in long horizon only given the current state and action?
  2. The evaluation is only on 3 tasks in sim environments. While increasing object numbers does perturb environments slightly, it does not really test generalization or how good the learned representation is in general.
Questions for authors
  1. Could the authors explain why the model should predict a specific future state and action sequence in long horizon only given the current state and action?
  2. Could the authors comment on why the proposed method could be more robust to object number change, whereas LeWM is better at 2 objects in both controlled start and random?
  3. For all methods, controlled start/multiple object performance is quite low. Could you explain this or shed light on what might improve the performance? Since the authors present the controlled-start and multi-object as important findings, it would be good to provide rationale on what these settings are testing.
1 comment

Thank you all for your thoughtful reviews and the time spent on them.

W1 / Q1 (why predict a future state and action sequence). The predictor is a training signal for the encoder, not a planning model: it is discarded after training and the controller uses only the frozen latent. Regressing 10 future latents (50 steps) and the actions between them from the current state and first action forces the latent to carry what determines how the scene will evolve under the expert, including object positions far from the gripper; with a deterministic loss it learns the expected expert continuation rather than one sampled future. The ablations (§7, Fig. 6) support this: a one-step horizon returns LH-JEPA to LeWM's level and collapses position decodability, the inverse-dynamics term improves control without changing decodability, and the same supervision applied conventionally (to observed frame pairs, on LeWM) lowers control from 14.4 to 2.0 on pnp4, so predicting actions jointly with future latents is what helps.

W2 (scope of the evaluation). We added LIBERO-Object (§6.3, Figs. 2–3): ten tasks with household objects in a cluttered scene, goal offsets up to 100 steps, a controlled clutter-free variant (LO-Simple), a pick-3 task (LO-Pick3) and search baselines, all with three training seeds. With GC-IDM, LH-JEPA leads LeWM at every offset in every scene (e.g. 95.0±1.795.0\pm1.7 vs. 72.6±1.572.6\pm1.5 at 50 steps). The distractor study (App. E) evaluates encoders trained on the clean task in scenes with added distractor objects of varying colour and size. We agree that real-robot data is the next step (§8).

Q2 (robustness to object count; LeWM at two objects). The decodability analysis (§6.4) shows why: from two to four objects LeWM's held-out position R2R^2 falls on every task family (and its MLP error grows 54–87×), while LH-JEPA's stays above 0.8, so LeWM's latent stops encoding the cubes the controller must act on. With two cubes this capacity limit does not bind and LeWM is ahead; §8 states that no representation leads on every task.

Q3 (low start-controlled performance). From the true start the controller must approach, grasp and move a cube within the budget, whereas random starts often begin mid-manipulation or already at the goal: a policy that outputs zero actions succeeds on 25.3% of random-start pnp1 episodes. Start-controlled success therefore measures the full manipulation, which is why it is lower. On LIBERO the same protocol separates the representations sharply once grasping and carrying are required (69.4 vs. 19.2 at 75 steps). The LIBERO-Object experiments also exposed a second cause of low performance, the training data distribution: with the number of training frames held fixed, moving from LO-Full to the longer LO-Pick3 episodes collapses LeWM from 72.6% to 11.9% at 50 steps and from 5.9% to 0.0% at 100 steps, while LH-JEPA falls only from 95.0% to 89.7% and from 48.3% to 35.0%.


Here we summarise the key revisions to the paper:

  1. Extending to another manipulation domain (LIBERO-Object), with controlled, progressively harder datasets and evaluation across goal horizons.
  2. Providing training-variance results.
  3. Refined claims.
  4. Motivated our design choices.
  5. Presentation-based improvements.
Reviewer_2uBPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper empirically investigates two pillars in latent world model, joint-embedding predictive architectures (JEPAs), research: (i) how the current evaluation suite hides the true performance of recent latent world models, and (ii) which action inference strategy between test-time search and inverse dynamics inference performs better in goal-conditioned behavior learning. The authors present a training recipe that incorporates three components into the latent world model learning: (i) multi-step JEPA with single forwarding, (ii) inverse dynamics action prediction via structured attention masking, and (iii) patch-level reconstruction objective for visual fine details. The authors propose a start-controlled protocol, which removes already-succeeded evaluation conditions, and evaluate their method within the protocol, analyzing not only the average success rate (controllability) but also decodability (representation quality) in the stable-world-model benchmark.

Critical strengths
  1. Characterization of an overlooked problem in latent world model research: The authors pinpoint the urgent but hidden problem in recent JEPA-based world model research: the evaluation protocol can mislead the actual performance of algorithms. Personally, I noticed the same problem when I tried to reproduce the reported scores of related approaches: I found the same gap; evaluating an algorithm on the offline dataset ensures reachable state-goal pairs but also includes false-positive pairs that are already succeeded or nearly succeeded. The authors characterize the problem and propose a solution by controlling the initial state distribution and goal budget, enabling fairer comparisons with baselines.
  2. Failure mode characterization of test-time search: The authors compare the test-time search strategies while fixing the same world model backbone in the main experiment. While it is notable that none of the test-time search methods achieve sufficient success rates, the authors' investigations into why such methods fail to plan over the horizon, given the well-predicting models, are very impressive in Appendix C.2. I agree that the current planning objective, which minimizes the L2 distance from the current to goal latent states, may become the bottleneck for test-time search world models.
Critical weaknesses
  1. Weak presentation: While I appreciate the problem formulation and experimental results from the authors, it costs substantial effort and time to fully understand the manuscript: missing descriptions of notations, exhaustively detailed hyperparameters, missing training procedures for GD-IDM, and the entire training process. I summarize my concerns in the questions below. I believe that improving readability and coherence remains a major task for the authors.
  2. Missing justification for inverse-dynamics loss: The proposed method learns the predictor of the latent world model to predict future multi-step sequential data, given a single-step state-action pair. By leveraging the structured attention mask, which makes the state tokens attend causally over the interleaved token sequence and the action tokens attend to consecutive state tokens, the proposed method learns implicit inverse dynamics. However, this implicit learning objective is different from the conventional inverse dynamics loss in deep reinforcement learning, where neither multi-step sequential data nor attention masking is necessary. Hence, I think the authors should justify how their algorithm differs from the conventional inverse dynamics loss (i.e., L(st,st+1)=∣f(st,st+1)−at∣22\mathcal{L}(s_t,s_{t+1})=\\| f(s_t,s_{t+1}) - a_t \\|^2_2) in the main text.
Questions for authors

Presentation

  1. Why do you name your algorithm "Ours"? If the method differs from prior approaches and does not fall directly under a backbone method like GC-IDM, naming the algorithm would be better for future researchers and readers.
  2. In Section 3, the authors exhaustively explain the details of their method, including hyperparameters and network architectures. However, I found that the authors describe the list of implementation details in Appendix A, where many of them duplicate Section 3. I recommend describing the extra details in Appendix A and using general variable placeholders in the main text; e.g., zt=fθ(ot)z_t=f_\theta(o_t), where oto_t is the image frame at timestep tt and fθf_\theta is the encoder. In place of such details, formulating the learning problem within a Markov decision process and explaining your algorithm's learning procedure would increase readability.
  3. I think the authors explain how they train GC-IDM given the frozen encoder explicitly. Since the original GC-IDM's encoder requires single-step latent states, the proposed method that predicts multi-step latent states cannot transfer to GC-IDM's training recipe directly.
  4. How do you train each baseline for multi-object study? Are they trained with a single-object task variant and evaluated with multi-object scenes in a zero-shot setting?

Minor requests

  1. Would you add a citation for stable-worldmodel in L197? I know that stable-worldmodel is cited at L148, but I think adding the citation to the experimental setup would be better for future readers.
  2. In L242-243, the authors state that they add task-irrelevant distractor objects that are not seen during the fixed task. However, I found that the yellow distractor cube is already presented when there is no distractor in the +1 (yellow) column of Figure 2-(b). Could you explain this?
  3. Could you describe the experimental setup more precisely? For instance, the term "pnpx", which stands for the Pick-and-place task with x cubes in OGBench, is widely used across Sections 4 and 5 without proper description. Furthermore, the font style of "pnp" is not consistent across the manuscript; some are in a plain font, and some are in a typewriter font. For the baselines, there are no explicit descriptions for "L+rec", O-rec", and "Ours-recon". What does "n" stand for in L422?
1 comment

Thank you all for your thoughtful reviews and the time spent on them.

W1 / Q2 (presentation and structure). We added a problem formulation as a goal-conditioned MDP (§3) and restructured the experimental setup (§5) into the environments and datasets (§5.1) and the encoder and GC-IDM training procedure, with a short overview of the experiments (§5.2). Each experiment's details now sit in its own results section.

Q1 (method name). The method is now named LH-JEPA (Long-Horizon JEPA) throughout the text, tables and figures.

Q3 (GC-IDM training). §5.2 now states the procedure explicitly. GC-IDM uses only the frozen encoder; the predictor and decoder are discarded, so it never consumes multi-step predictions. Each frame is encoded once to a single latent ztz_t, and the policy regresses the recorded action from (zt,zt+k,k)(z_t, z_{t+k}, k), with kk drawn uniformly up to min⁡(50,steps left)\min(50, \text{steps left}).

Q4 (baseline training). Every encoder is trained from scratch on the data of the task it is evaluated on (e.g. the pnp4 encoder on pnp4); nothing is evaluated zero-shot across object counts. This is now stated in §5.2. The only exception is the distractor study (App. E), which by design evaluates encoders trained on the clean task in altered scenes.

W2 (inverse-dynamics loss). The conventional loss regresses ata_t from a pair of encoded observations (zt,zt+1)(z_t, z_{t+1}); ours predicts the actions inside the predictor, from its own predicted latents, jointly with ten future latents from a single state-action anchor. We now test the conventional form directly (§7, Fig. 6): added to LeWM, it lowers control from 14.4 to 2.0 on pnp4 and from 15.1 to 1.0 on cyclic4, although the loss itself trains normally, while removing our joint action term costs LH-JEPA 10–12 points.

Minor. (1) The stable-worldmodel citation is added to the setup. (2) The "+1" distractor is lime, while pnp4's fourth task cube is orange, which renders yellowish; we relabelled the distractor and give both colours in the caption. (3) pnpN, stackN and cyclicN are now defined and set in one font; every model abbreviation (S-JEPA, DINO, L+rec, LH-rec) is defined; and nn in the ablation is defined as the number of predicted future latents, each 5 steps apart.


Here we summarise the key revisions to the paper:

  1. Extending to another manipulation domain (LIBERO-Object), with controlled, progressively harder datasets and evaluation across goal horizons.
  2. Providing training-variance results.
  3. Refined claims.
  4. Motivated our design choices.
  5. Presentation-based improvements.
Reviewer_cjmKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies compact visual representations for downstream control. It compares random mid-episode initialization with evaluation from the beginning of a demonstration, finding that the tested latent-search planners fail on OGB-Cube under the latter protocol. It then evaluates representations in scenes containing up to four cubes and proposes an architecture combining long-horizon latent prediction, action-prediction supervision, and patch-feature reconstruction. The resulting encoder is frozen and evaluated through a separately trained goal-conditioned inverse-dynamics policy (GC-IDM).

Critical strengths

1.The paper highlights a useful evaluation concern: random mid-episode starts can include already-solved states or bypass difficult manipulation stages. Including a zero-action baseline helps quantify trivial success.

2.The search diagnostics are informative. Sweeping CEM configurations and showing decreasing predicted latent cost without physical object displacement provides evidence that increasing search compute alone does not resolve the observed failure.

3.The representation-learning experiments include useful ablations of prediction horizon, action supervision, and patch reconstruction. The observation that action supervision improves control without substantially changing position decodability is particularly interesting.

4.The proposed representation improves GC-IDM performance on the evaluated three- and four-object settings while retaining the same compact 192-dimensional interface at inference.

Critical weaknesses
  1. The headline evaluation finding is narrower than the paper’s framing suggests. The complete collapse of search-based control occurs on OGB-Cube. Table 1 shows heterogeneous effects elsewhere: Push-T performance generally decreases, Two-Room exhibits varying changes, and CEM/iCEM performance on Reacher improves under start-controlled evaluation. Thus, the results establish sensitivity to initialization and task difficulty, rather than a general defect of random-start evaluation. The subsequent multi-object experiments remain within the cube environment and do not provide independent confirmation across manipulation domains. This limits the breadth and significance of the central empirical finding.

  2. The proposed method only partially addresses the limitations motivating the paper. Although it improves absolute GC-IDM performance on several start-controlled tasks, it remains strongly sensitive to initialization: four-object stacking falls from 66.6% under random starts to 17.4%, and cyclic rearrangement falls from 68.2% to 24.4%. Moreover, the method is not shown to remedy the complete failure of latent-search planners. The results therefore support improved representation learning for a particular controller and evaluation distribution, rather than a general resolution of the controllability limitations identified by the study. The paper should distinguish its diagnostic contribution from the narrower improvements delivered by its method.

  3. The multi-object evaluation measures short-horizon goal reaching rather than complete multi-object task execution. Both the goal offset and evaluation budget are fixed to 50 environment steps, while the underlying demonstrations can last several hundred steps. A four-object scene therefore does not necessarily require manipulating four objects within the evaluated segment. The paper should distinguish increased visual scene complexity from increased sequential manipulation complexity. Statistics on the number of objects that must move, and evaluations using complete-task goals, would clarify the scope of the results.

  4. The increasing-object-count trend has potential confounds. Separate models are trained for each task, and changing object count also changes the demonstration and goal distributions. The reported improvement from two to four objects could partly reflect differences in the initial manipulation segment rather than an intrinsic ability to scale with object count. LeWM also outperforms the proposed method on all three two-object task families. A controlled comparison preserving the required manipulation while varying surrounding objects would strengthen the interpretation.

  5. The representation objective needs clearer positioning and stronger controls. The predictor receives only the initial latent and first action block, then jointly predicts subsequent states and actions. It therefore learns expert-trajectory continuations rather than general dynamics conditioned on an arbitrary future action sequence. A comparison with conventional inverse-dynamics supervision on observed-state pairs would help establish the benefit of the proposed structured prediction mechanism. Furthermore, matching optimization steps does not match compute or frame exposure when the proposed model uses longer windows and additional losses.

  6. Reported uncertainty does not capture training variability. The standard deviations are computed across evaluation seeds. Independent encoder and policy training runs are needed to establish the robustness of the architectural gains, particularly for smaller differences.

Questions for authors

1.Scope of the proposed improvement: Your method improves absolute success on several start-controlled tasks but still exhibits large random-start-to-start-controlled drops, and is not evaluated with search-based planners. Which of the diagnosed controllability limitations is the method intended to address, and what evidence demonstrates that it addresses them?

2.Generality of the evaluation finding: The complete search collapse occurs on OGB-Cube, while Table 1 shows mixed effects elsewhere, including improvements on Reacher. Does the finding extend to another manipulation environment requiring approach, grasp, and transport?

3.Separating trivial success from task difficulty: How do the conclusions change after excluding initially successful episodes and stratifying random starts by manipulation stage? This would help distinguish inflated scores from legitimate success on easier, partially completed tasks.

4.Meaning of multi-object success: How many objects actually need to move between initialization and the 50-step goal for each task and object count? Do the method rankings and object-count trends persist with longer horizons or complete-task goals?

1 comment

Thank you all for your thoughtful reviews and the time spent on them.

W1 / Q2 (generality of the evaluation finding). We agree and have scoped the claim (§6.1): the complete collapse of search is specific to 3D manipulation, and on the 2D environments random starts reorder methods rather than uniformly inflate them. To test a second manipulation domain we added LIBERO-Object (§6.3, Figs. 2–3), whose goals require approach, grasp and transport. Run from the true start with twice GC-IDM's step budget, CEM succeeds on 4.0±4.44.0\pm4.4 to 25.5±2.625.5\pm2.6% of 25-step goals, against 76.7–100% for GC-IDM, and fails in every run on both representations once the object must be grasped and carried (75 steps), as on OGB-Cube.

W2 / Q1 (scope of the method). The Limitations (§8) now separate the two contributions: the diagnostic one (protocol and multi-object suite) and a narrower method, a frozen representation from which an amortized controller succeeds more often from the true episode start. We now also run search on LH-JEPA's latents: they make CEM more effective than LeWM's at short offsets (25.5±2.625.5\pm2.6 vs. 4.0±4.44.0\pm4.4 at 25 steps), but do not repair it.

W3 / Q4 (what multi-object success measures). At offset 50 every start-controlled cube episode requires moving exactly one cube, so the object-count axis varies scene complexity at fixed manipulation complexity (§5.1). It does, however, change the training distribution, since episodes with more cubes are longer. On LIBERO-Object we control for exactly this (§5.1): LO-Simple and LO-Full replay identical actions and differ only in scene clutter, and LO-Pick3 lengthens the episodes while matching the number of training frames (1.28M vs. 1.26M). We also report the object displacement each offset requires (0, 2.5, 21 and 33 cm at 25–100 steps). On LO-Pick3, whose demonstrations place three objects in the basket (Fig. 2, bottom), LH-JEPA leads at every offset (99.9, 89.7, 58.8, 35.0% vs. 5.8, 11.9, 1.3, 0.0% at 25–100 steps), and every LH-JEPA seed beats every LeWM seed. We did not evaluate complete-task execution: following the prior work, we evaluate goal reaching, as we now specify in the problem formulation (§3).

W4 (object-count confounds). LO-Simple and LO-Full form the controlled comparison you suggest: LO-Full contains the same episodes as LO-Simple with five non-target objects added to the scene, so the required manipulation is identical frame for frame and only the surrounding objects change. The ranking holds in both scenes (at 100 steps, 76.8±2.276.8\pm2.2 vs. 17.2±5.017.2\pm5.0 without distractors). The distractor study (App. E) likewise keeps the task cubes' trajectories identical while adding objects.

W5 (positioning against inverse-dynamics supervision; compute). We ran the comparison you suggest (§7, Fig. 6): adding conventional inverse-dynamics supervision on observed frame pairs to LeWM lowers it from 14.4 to 2.0 on pnp4 and from 15.1 to 1.0 on cyclic4, although its action loss trains normally; its latents still decode cube positions as well as LeWM's on pnp4 (probe R2R^2 0.50 vs. 0.50), whereas our joint prediction of actions with future latents raises control (removing it costs 10–12 points). We agree the predictor learns expert continuations rather than dynamics under arbitrary actions; it is a training signal for the encoder and is discarded at control time (§8). Optimisation steps are matched (200k), but frame exposure and compute are not, since our window is longer: on one RTX 5090 a run takes about 21 h for LH-JEPA against 9 h for LeWM (2.4×2.4\times). Inference cost is identical.

W6 (training variability). We retrained LH-JEPA and LeWM with three independent training seeds on pnp4 (encoder and GC-IDM retrained per seed). LH-JEPA outperforms LeWM on every seed (34.5±4.534.5\pm4.5 vs. 12.7±1.612.7\pm1.6, mean ± sd over training seeds). These retrains use training sets regenerated with the original collectors. All LIBERO results also use three training seeds (on LO-Pick3, two of LeWM's three seeds score below 1% at every offset); clutter lowers both models, but LH-JEPA retains a larger share of its performance (at 100 steps it keeps 63% of its LO-Simple success, LeWM 34%) and leads in both scenes.

Q3 (excluding trivially successful episodes). A policy that outputs zero actions measures the trivially solved stratum directly: it succeeds on 25.3% (pnp1), 12.9% (pnp4) and 13.9% (cyclic4) of random-start episodes, those that begin at the goal. Start-controlled evaluation removes these and every partially completed start by construction, and our conclusions are drawn there: from t=0t=0 search scores 0.0% on OGB-Cube while GC-IDM reaches 68.9–84.9%, and LH-JEPA leads at four cubes. We were unable to complete the stratification by manipulation stage within the rebuttal period.

Reviewer_nLhgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether compact, frozen visual representations support downstream goal-conditioned control. It shows that evaluating from random mid-episode states can substantially overestimate performance relative to starting from the beginning of a demonstration. It then evaluates scenes containing up to four cubes and proposes an encoder-training architecture combining long-horizon latent prediction, an auxiliary inverse-dynamics objective, and patch-feature reconstruction. The encoder is subsequently frozen and used by a goal-conditioned inverse-dynamics controller, GC-IDM.

Critical strengths

The paper provides a valuable evaluation audit, showing that random mid-episode starts can substantially inflate control success, supported by zero-action baselines and search-budget experiments.

Critical weaknesses

The multi-object evaluation targets expert states only 50 steps after initialization, rather than completed tasks, limiting the evidence for reliable full-task manipulation as object count increases.

Questions for authors
  1. Can you report the start-distribution × goal-offset comparison, including the original 25-step setting, and evaluate longer or terminal task goals?
  2. Are control evaluation episodes held out from both encoder and GC-IDM training? How many independent training runs underlie the results?
  3. Are transition counts and controller update budgets matched across object counts, and how much manipulation is required within each 50-step target?
  4. Does the proposed attention structure outperform a simpler predictor with matched objectives and training compute? What is the measured training and inference cost?
2 comments

Thank you all for your thoughtful reviews and the time spent on them.

W1 / Q1 (goal offsets and longer goals). We fixed the cube offset at 50 because at 25 steps start-controlled OGB-Cube is already solved (100%), as under the success criterion of the stable-worldmodel harness every episode starts in a successful state, so the shorter goal does not separate controllers (§6.1). On LIBERO-Object (§6.3, Fig. 3) we sweep the goal offset from 25 to 100 steps, where the target object must be grasped and carried up to 33 cm. LH-JEPA leads LeWM on every LIBERO-Object dataset (LO-Simple, LO-Full, LO-Pick3; Fig. 2) at every offset (e.g. 48.3±2.348.3\pm2.3 vs. 5.9±0.75.9\pm0.7 on LO-Full at 100 steps).

Q2 (held-out episodes; independent runs). Following the evaluation protocol of the prior work, start and goal states are drawn from the expert demonstrations the encoder and GC-IDM are trained on; the controller still acts in closed loop from the true start, and the success check is on the simulated state. On LIBERO-Object, start states are drawn from the task's initial-state distribution (§3). pnp4 and all LIBERO results use three independent training seeds (encoder and GC-IDM retrained), each evaluated on four evaluation seeds × 200 episodes; LH-JEPA beats LeWM on every seed (pnp4 34.5±4.534.5\pm4.5 vs. 12.7±1.612.7\pm1.6).

Q3 (budgets and required manipulation). On the cube suite, every task has 10,000 expert episodes, every encoder trains for 200k optimisation steps and every GC-IDM for 100 epochs. Transition counts are therefore not matched across object counts, because episodes with more cubes are longer (348–360 steps for pnp4). Within a 50-step start-controlled goal exactly one cube must move at every object count (§5.1), so the cube suite varies scene complexity at fixed manipulation, but it also changes the training distribution. LIBERO-Object controls for both (§5.1): LO-Simple and LO-Full replay identical actions and differ only in scene clutter, and LO-Pick3 lengthens the episodes (350–487 steps) while collecting fewer of them (3,150), so the number of training frames is matched (1.28M vs. 1.26M).

Q4 (attention structure; cost). The ablations (§7) isolate what matters: shortening the prediction horizon to a single 5-step latent returns LH-JEPA to LeWM's level (pnp4 17.4 vs. 17.1), and removing the inverse-dynamics loss or the patch decoder costs 10–12 and 1–10 points. With the same objectives, a plain causal mask lowers pnp4 (26.9 vs. 37.5) and matches cyclic4 (24.1 vs. 24.4), so the mask's contribution to control is task-dependent and the consistent gains come from the horizon and the two losses (§7, Fig. 6). Inference cost is identical to LeWM (the same ViT-Tiny encoder and 192-d latent; predictor and decoder are discarded). Training is about 2.4×2.4\times LeWM's: on one RTX 5090 a 200k-step run takes about 21 h against 9 h.


Here we summarise the key revisions to the paper:

  1. Extending to another manipulation domain (LIBERO-Object), with controlled, progressively harder datasets and evaluation across goal horizons.
  2. Providing training-variance results.
  3. Refined claims.
  4. Motivated our design choices.
  5. Presentation-based improvements.

Thanks for the authors' reply. I will keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.