acceptodds
Under review as a conference paper at ICLR 2027

DSSF: Robust Latent State Construction for JEPA-Based Visual Planning

Abstract

Latent world models based on the Joint-Embedding Predictive Architecture provide an efficient approach to dynamics modeling for visual planning by predicting future representations rather than reconstructing pixels. However, latent prediction primarily optimizes the predictability of visual information and does not guarantee that the retained factors are relevant to control. Consequently, task-irrelevant backgrounds and visual distractors may be encoded into the latent state, perturbing state prediction and goal matching when previously unseen visual distractors occur at test time. We propose DSSF, a two-stage latent-state construction method for improving planning robustness under unseen visual distractors. In the first stage, learnable prototypes model the feature-space support of non-distractor visual factors and filter out local factors that are not supported by the training distribution. In the second stage, DSSF sparsely selects information relevant to action-conditioned dynamics from the retained factors to construct a compact latent state. In this way, DSSF separately models whether a visual factor is supported by the training distribution and whether it is relevant to control, thereby suppressing both unseen distractors and in-distribution yet control-irrelevant information. We evaluate unseen distractor instances and cross-family distractors to systematically assess planning robustness under increasingly severe shifts in distractor appearance and count. Across multiple continuous-control tasks, DSSF better preserves clean-setting planning performance and consistently outperforms strong baselines under unseen distractors and cross-family distractor-appearance shifts, indicating that its robustness does not rely on memorizing the appearances of training distractors.

Primary area:
applications to robotics, autonomy, planning
Keywords:
JEPA, Latent World Model, Out-of-Distribution Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CSU9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents DSSF, Distribution-Supported Sparse Control Factors, a latent-state representation learning method for Joint-Embedding Predictive Architecture (JPEA)-based latent world models. DSSF targets zero-shot visual generalization during test time when unseen visual distractors or seen, control-irrelevant factors may contaminate goal-conditioned planning. By leveraging foreground masking that divides task-relevant spatial information from control-irrelevant distractors, DSSF composes the compact latent state from high-dimensional visual inputs, where potential in-distribution, control-irrelevant, or out-of-distribution, task-irrelevant distractors are included. DSSF demonstrates strong performance on 4 goal-conditioned visual planning tasks, spanning from maze-escaping to pick-and-place tasks, across latent world model baselines.

Critical strengths
  1. Novel problem formulation and algorithm: Many researchers have been diving into the JEPA-based latent world model recently. While improving visual planning over the long horizon by increasing effective horizons or incorporating hierarchical architectures has been suggested, learning a compact latent world model that is robust to visual distractors has not been explored enough, despite its necessity. DSSF can be incorporated into a JEPA-based latent world model as a plug-and-play module, since DSSF only changes how the encoder is trained and does not touch the underlying dynamics prediction task or test-time planning. Substantial qualitative and quantitative experimental evidence supports that DSSF can manage test-time unseen visual distractors effectively.
Critical weaknesses

Weak connection to prior literature in visual generalization: While visual generalization in JEPA-based latent world models has yet to be explored, zero-shot or few-shot visual generalization research has been investigated significantly in reinforcement learning. For instance, data augmentation, pretrained vision backbones, and representation learning affect the test-time visual generalization performance of model-free RL agents [1-4]; learning the bisimulation principle or distinguishing task-relevant from task-irrelevant features enables robust learning for model-based RL [5-8]. While these works may not connect directly to JEPA-based latent world models with visual generalization, providing a holistic view of visual generalization for decision-making could further improve legibility and understanding of this work. Unclear description of the algorithm components and experimental setup: It takes substantial effort and time to fully understand what DSSF is and how DSSF works under the experimental setup. I summarize my concerns and remaining questions below. I believe resolving the concerns could improve the quality of this paper significantly.

Reference

[1] Hansen, Nicklas, Hao Su, and Xiaolong Wang. "Stabilizing deep q-learning with convnets and vision transformers under data augmentation." NeurIPS 2021

[2] Huang, Yangru, et al. "Spectrum random masking for generalization in image-based reinforcement learning." NeurIPS 2022

[3] Yuan, Zhecheng, et al. "Pre-trained image encoder for generalizable visual reinforcement learning." NeurIPS 2022

[4] Batra, Sumeet, and Gaurav S. Sukhatme. "Zero-shot generalization of vision-based rl without data augmentation." ICML 2025

[5] Zhang, Amy, et al. "Learning invariant representations for reinforcement learning without reconstruction." ICLR 2021

[6] Fu, Xiang, et al. "Learning task informed abstractions." ICML 2021

[7] Bharadhwaj, Homanga, et al. "Information prioritization through empowerment in visual model-based RL." ICLR 2022

[8] Zhu, Chuning, et al. "Repo: Resilient model-based reinforcement learning by regularizing posterior predictability." NeurIPS 2023

Questions for authors

Regarding the algorithm

  1. The two-stage latent-state construction procedure is not described completely:
  • How are the shallow and deep features, pt,ip_{t,i} and ft,if_{t,i}, extracted from the encoder? Could you define them formally (e.g., an equation)?
  • What does AA stand for in o~t=A(ot;d)\tilde{o}_t=A(o_t;d) (L149)?
  • What are the formal definitions of location ii for the shallow and deep features (L195-196, L226)?
  • Could you add proper citations for the terminology, e.g., prototype, support gate, support descriptor? I think these expressions are not commonly used in latent world model research, so adding citations could help future readers understand better.
  • What is WoW_o in L242?
  1. Could you add a pseudocode table to describe how DSSF trains its encoder? I think Figure 2 does not provide enough context to understand Section 3.

Regarding the experimental setup

  1. How is DSSF sensitive to the threshold τ\tau? Could you perform a sweep search on a toy example?
  2. How does an ambiguous set I\mathcal{I} negatively affect the two-stage latent-state learning if the set is included in a positive or negative set?
  3. How did you place the random visual distractors in the clean image when augmenting the original view? Did you allow the distractors to occlude the task-relevant regions, e.g., the T-shaped bar in the Push-T task or a manipulator arm in the Cube task?
  4. Could you add an extra baseline, i.e., JEPA-Bisim*, for the main robustness performance comparison in Figure 4? I think JEPA-Bisim would be a good comparison baseline, since it also investigates visually robust latent world models.
  5. Could you explain how Gaussian noise is applied during test time for the internal ablation in Section 4.5?
  6. How do you determine whether a pixel contains non-background or not throughout Appendix E? What is the criterion?

*Toso, Leonardo F., et al. "Learning invariant visual representations for planning with joint-embedding predictive world models." arXiv 2026

5 comments
AuthorsOpenReview ↗

Response to Reviewer CSU9 (1/5)

General Response

We appreciate your recognition of the problem formulation and algorithmic contribution, DSSF's integration into an existing JEPA planning pipeline, and the qualitative and quantitative evidence for handling unseen distractors. We address your comments point by point below. The revisions expand the connection to visual RL generalization and add formal definitions, training pseudocode, a threshold sweep, explicit augmentation and evaluation protocols, and updated baseline comparisons.

R3-Critical Weaknesses — Connections to Prior Work on Visual Generalization

Reviewer comment

Weak connection to prior literature in visual generalization: While visual generalization in JEPA-based latent world models has yet to be explored, zero-shot or few-shot visual generalization research has been investigated significantly in reinforcement learning. For instance, data augmentation, pretrained vision backbones, and representation learning affect the test-time visual generalization performance of model-free RL agents [1–4]; learning the bisimulation principle or distinguishing task-relevant from task-irrelevant features enables robust learning for model-based RL [5–8]. While these works may not connect directly to JEPA-based latent world models with visual generalization, providing a holistic view of visual generalization for decision-making could further improve legibility and understanding of this work.

Response

We have incorporated all eight suggested references into Section 2, “Robust Visual Representations under Nuisance Shifts.” The first added sentence connects visual RL generalization to stable data augmentation (Hansen et al., 2021), spectral masking (Huang et al., 2022), pretrained visual encoders (Yuan et al., 2022), and latent disentanglement with associative memory (Batra and Sukhatme, 2025). The second added sentence discusses complementary approaches to learning task-relevant representations, including bisimulation (Zhang et al., 2021), task-informed abstractions (Fu et al., 2021), empowerment-based information prioritization (Bharadhwaj et al., 2022), and posterior-predictability regularization (Zhu et al., 2023). We also revised the closing sentence of this paragraph to clarify how DSSF builds on these foundations by explicitly separating training-distribution support from control relevance through prototype-based Support Filtering and sparse Control-Factor Selection for offline, reward-free JEPA planning.



R3-1 — Shallow and Deep Feature Extraction

Reviewer comment

How are the shallow and deep features, pt,ip_{t,i} and ft,if_{t,i}, extracted from the encoder? Could you define them formally (e.g., an equation)?

Response

The shallow and deep descriptors come from spatially aligned feature maps extracted after the 5th and 9th main-path convolutional layers of ResNet-18 (He et al., 2016), corresponding to its first and second residual stages.

Manuscript changes — Section 3.1, Eq. (2): we have added the extraction points, feature-map dimensions, and formal alignment operations, together with the definitions of the resulting channel vectors.

For a 224 × 224 input, the raw maps are 64 × 56 × 56 and 128 × 28 × 28. We align them as

\mathbf F_t=\Pi_{1\times1}(\mathbf H_t^d),$$ where $\mathcal U_2$ denotes PixelUnshuffle(2) and $\Pi_{1\times1}$ is a learnable 1 × 1 convolution with 192 output channels. The aligned maps are 256 × 28 × 28 and 192 × 28 × 28; $p_{t,i}$ and $f_{t,i}$ are their channel vectors at location $i$. This supplies local appearance information for Support Filtering and more contextual features for state construction. --- --- ### **R3-2 — Definition of the Visual Distractor Augmentation Operator** **Reviewer comment** > What does A stand for in $\tilde o_t=A(o_t;d)$ (L149)? **Response** $A$ is the **visual distractor augmentation operator**. The nuisance factor $d$ specifies attributes including appearance, position, scale, and opacity. **Manuscript change — Section 3.1:** we have explicitly defined $A$ where the notation is introduced. **Appendix A.2** documents the background-placement constraints. For a single distractor, alpha compositing can be written as $$A(o_t;d)=(1-\alpha_d)\odot o_t+\alpha_d\odot D_d,$$ where $D_d$ is the image-aligned distractor RGB image and $\alpha_d$ is its opacity mask. Multiple distractors are applied sequentially under the placement constraints. This changes the visual observation while preserving the task state and environment dynamics.
AuthorsOpenReview ↗

Response to Reviewer CSU9 (2/5)

R3-3 — Spatial Indexing and Feature Alignment

Reviewer comment

What are the formal definitions of location i for the shallow and deep features (L195–196, L226)?

Response

The index ii refers to a location on the shared 28 × 28 spatial grid after shallow/deep feature alignment.

Manuscript change — Section 3.1, following Eq. (2): we have added the flattened-index definition and the correspondence between the two descriptors.

The shallow map initially has 56 × 56 locations. PixelUnshuffle(2) packs the four 64-dimensional feature vectors in each 2 × 2 block into one 256-dimensional vector, producing a 28 × 28 map. The deeper map already has 28 × 28 locations; the 1 × 1 convolution converts each 128-dimensional vector into a 192-dimensional vector independently, keeping all grid locations unchanged.

On these two aligned maps, uu and vv denote the row and column, with u,v∈{0,…,27}u,v\in\{0,\ldots,27\}, and i=28u+vi=28u+v. The vectors pt,i∈R256p_{t,i}\in\mathbb R^{256} and ft,i∈R192f_{t,i}\in\mathbb R^{192} are read from the same row uu and column vv in their respective maps. For example, i=30i=30 selects row 1, column 2 from both maps.



R3-4 — Terminology, Definitions, and Supporting References

Reviewer comment

Could you add proper citations for the terminology, e.g., prototype, support gate, support descriptor? I think these expressions are not commonly used in latent world model research, so adding citations could help future readers understand better.

Response

Learnable prototypes have an established representation-learning background; Support Gate and Support descriptors are names for specific components in DSSF.

Changes in Section 3.2.1 and the bibliography:

  • Added SwAV (Caron et al., 2020) as conceptual background for learnable visual prototypes.
  • Added PatchCore (Roth et al., 2022) as related work on local feature matching using a feature bank.
  • Explicitly defined the Support Gate and Support descriptors in the method description.

The Support Gate is the binary spatial mask obtained by thresholding the normalized Support response. Support descriptors are the deep features retained after positional encoding and Support Filtering and passed to Control-Factor Selection. The citations provide conceptual context: PatchCore's feature memory bank differs from our learned prototype bank, and the DSSF-specific terms are defined within our method.

References

Caron, M., et al. (2020). Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. NeurIPS 33, pp. 9912–9924.

Roth, K., et al. (2022). Towards Total Recall in Industrial Anomaly Detection. CVPR, pp. 14318–14328.



R3-5 — Definition of the Output Projection

Reviewer comment

What is WoW_o in L242?

Response

Wo∈R192×192W_o\in\mathbb R^{192\times192} is a learnable output projection matrix applied to the weighted aggregation of selected Control factors before the subsequent projector.

Manuscript change — Section 3.2.2: we have added this definition immediately after the aggregation expression, distinguishing WoW_o from the projector. It is jointly optimized with the encoder and the other trainable components through the overall objective.



R3-6 — Pseudocode for Encoder and Latent Dynamics Training

Reviewer comment

Could you add a pseudocode table to describe how DSSF trains its encoder? I think Figure 2 does not provide enough context to understand Section 3.

Response

We have added a standalone training pseudocode to connect the architecture in Figure 2 with the operations and losses in Section 3.

Added in the revision:

  • Appendix J, Algorithm 2: DSSF Encoder and Latent Dynamics Training: the complete training sequence.
  • Section 3.3: a direct reference to the new pseudocode.

Algorithm 2 describes how we apply the same distractor layout to every frame in each training sequence and generate the corresponding distractor masks. It also covers aligned shallow/deep feature extraction; prototype-based Support Filtering and mask supervision; sparse Control-Factor Selection and latent construction; and joint optimization using latent prediction, SIGReg (Balestriero and LeCun, 2025), Support supervision, and auxiliary action recovery. It also specifies straight-through gradient estimation for the hard Support Gate and Top-K selection.

AuthorsOpenReview ↗

Response to Reviewer CSU9 (3/5)

Regarding the experimental setup

R3-7 — Sensitivity to the Support Threshold

Reviewer comment

How is DSSF sensitive to the threshold τ\tau? Could you perform a sweep search on a toy example?

Response

We have added a fixed-checkpoint threshold sweep on Push-T, measuring how stricter gating trades distractor rejection against rejection of valid clean features.

Added in the revision — Appendix B, Figure 9 and the Threshold sensitivity paragraph: two panels show distractor rejection and valid-clean false rejection as τ\tau varies from 0.1 to 0.99, accompanied by quantitative analysis in the text.

Each family contains 500 fixed images with five distractors. We hold the model and images fixed. The table below reports distractor rejection / valid-clean false rejection (%): the proportion of distractor locations removed by the gate and the proportion of valid clean locations incorrectly removed, respectively. The ambiguous boundary band is excluded from valid-clean statistics. The default threshold is highlighted in bold.

τ\tauHeld-out in-familyCross-family ShapesCross-family Real-object
0.199.670 / 0.00086.662 / 0.49099.617 / 0.080
0.399.680 / 0.00286.954 / 0.74499.663 / 0.132
0.599.692 / 0.00987.174 / 0.95099.688 / 0.187
0.799.703 / 0.02587.301 / 1.21299.717 / 0.262
0.999.719 / 0.09487.627 / 1.89299.761 / 0.526
0.9599.734 / 0.13787.788 / 2.51099.770 / 0.824
0.9999.766 / 0.26088.145 / 4.73299.820 / 1.855

For τ∈[0.1,0.9]\tau\in[0.1,0.9], distractor rejection remains within 99.67–99.72% for held-out in-family assets and 99.62–99.76% for Real-object distractors. At τ=0.5\tau=0.5, their valid-clean false rejection rates are 0.01% and 0.19%.

For Shapes, raising τ\tau from 0.5 to 0.99 increases distractor rejection from 87.17% to 88.15%, while increasing clean false rejection from 0.95% to 4.73%. Thus, rejection has a broad intermediate-threshold plateau, while stricter gating increasingly removes valid clean locations. These measurements quantify the threshold sensitivity of the Stage-I gate.



R3-8 — Ambiguous Regions in Support Supervision

Reviewer comment

How does an ambiguous set I\mathcal I negatively affect the two-stage latent-state learning if the set is included in a positive or negative set?

Response

The ambiguous set I\mathcal I is excluded from the Support loss, while its forward selection remains governed by the learned Support Gate and Control-Factor Selector.

Manuscript change — Section 3.2.1: we have added the reason for excluding ambiguous labels and clarified that loss exclusion does not force a location to be retained or discarded in the forward pass.

Locations near a distractor boundary can contain mixed clean/distractor information through their receptive fields even when no distractor pixel directly covers the location. Labeling them positive may encourage nuisance-contaminated descriptors to enter the second stage. Labeling them negative may remove useful scene information before Control-Factor Selection. Excluding these ambiguous labels avoids imposing either potentially misleading target.

Reviewer_GjxHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DSSF constructs a planning state through two stages: a prototype-based gate filters local visual features, and a sparse selector aggregates the retained features into a latent state. Training combines JEPA-style latent prediction and regularization with synthetic distractor-mask supervision and auxiliary action recovery. The paper evaluates visual planning in four environments under held-out distractor appearances, including cross-family overlays and mismatched current-goal layouts.

Critical strengths

1、The distinction between visually supported information and control-relevant information is useful. The method is integrated into an existing latent-prediction and CEM-planning pipeline, making the proposed change reasonably clear. 2、The reported robustness gains are substantial. On Push-T with five static shape distractors, success rises from 28.60% for LeWM to 70.13%, while clean success remains 87.93%. Gains extend to four environments and to real-object cutouts (Tables 1 and 7). 3、The matched-backbone 2-by-2 ablation and the internal stage ablation are informative. They show the importance of action recovery and reveal different behavior under localized overlays and Gaussian noise, rather than claiming that every component always helps (Tables 2-3). Appendix H documents hard selection and its straight-through gradient approximation.

Critical weaknesses
  1. The benefit of prototype support modeling is not isolated from supervised nuisance segmentation. Stage I receives spatial labels derived from the synthetic overlay process. The CNN-LeWM ablation controls the backbone and augmentation, but does not establish an advantage over a conventional shallow binary mask predictor trained with the same labels and paired with the same action-recovery objective. Since the support score is learned discriminatively from positive and negative regions, improved planning alone does not establish that the prototype formulation provides a distinct support-estimation benefit. The paper appropriately avoids claiming calibrated OOD detection; the remaining architectural attribution still needs a matched control.
  2. The separation between development choices and final evaluation needs clarification. Appendix E chooses the 30% selection budget using foreground occupancy from 1,500 clean initial evaluation observations per task. Appendix B also states that the held-out in-family assets were used in validation forward passes. These facts do not by themselves imply outcome-label leakage, but they mean that exclusion from parameter updates is not equivalent to an untouched test protocol. It is unclear which data informed backbone, checkpoint, threshold, and budget choices, and whether the reported three seeds include independently trained models.
  3. The demonstrated robustness is narrower than a general solution to unseen visual nuisances. All structured distractor families share one compositing mechanism. Moreover, Dynamic means different fixed layouts in the current and goal images, not moving distractors throughout execution (Appendix A.2). Push-T Shapes success falls from 70.13% in Static to 32.93% in Dynamic, showing a substantial remaining failure mode. Support Only matches Full DSSF at five structured distractors, while Control Only has slightly higher Gaussian-noise means (Table 3). These are useful findings, but the strongest conclusion is complementary robustness within the tested corruption mechanisms.
Questions for authors
  1. How does a conventional supervised mask gate perform when given the same synthetic masks, backbone, action-recovery loss, and planning budget? This would clarify the specific contribution of the prototype bank.
  2. Please distinguish training, model-selection, and final evaluation data, including the use of held-out assets and evaluation-image occupancy. Do the three reported seeds retrain the full model or only vary evaluation randomness?
  3. For the existing Push-T Dynamic runs, is the main failure caused by nuisance retention, rejection of task-relevant features, or latent goal matching? Can you quantify this without adding a new benchmark?
2 comments
AuthorsOpenReview ↗

Response to Reviewer GjxH (1/2)

General Response

We appreciate your recognition of the distinction between visually supported and control-relevant information, the robustness gains, and the informative component ablations. We respond to each point below, starting with the new matched-supervision gate comparison, followed by the evaluation-protocol clarification and new encoder diagnostics for the existing Push-T Dynamic cases.

R2-1 — Prototype Filtering versus a Supervised MLP Mask Gate

Reviewer comment

How does a conventional supervised mask gate perform when given the same synthetic masks, backbone, action-recovery loss, and planning budget? This would clarify the specific contribution of the prototype bank.

Response

We added a direct comparison between a two-layer MLP mask gate and our prototype-based Support gate.

Added comparison — Appendix E, Table 7: Both variants use the same synthetic masks, augmentation, backbone, latent predictor, auxiliary action-recovery loss, and per-episode CEM planning budget. Both omit Control-Factor Selection and densely aggregate features passing their gates; the architectural change is confined to the gate.

Planning success rates on Push-T are reported below as mean ± standard deviation across three evaluation seeds. Structured-distractor conditions use 500 episodes per seed for both gates; Gaussian-noise conditions use 500 for MLP and 50 for Prototype. The higher mean in each condition is bold.

Push-T conditionMLP gatePrototype gate
SR@080.93 ± 0.1286.20 ± 0.35
SR@183.80 ± 1.8081.27 ± 1.63
SR@369.80 ± 1.0475.47 ± 0.61
SR@564.93 ± 0.9570.13 ± 0.61
Gaussian Low34.00 ± 5.3336.00 ± 5.29
Gaussian Medium16.80 ± 1.5618.00 ± 2.00
Gaussian High10.53 ± 5.3510.00 ± 5.29

All values are percentages. SR@n denotes success with n structured distractors. Gaussian Low, Medium, and High use σ=0.03\sigma=0.03, 0.080.08, and 0.150.15, respectively, with RGB values scaled to [0,1][0,1].

Prototype filtering improves clean success from 80.93% to 86.20%, SR@3 from 69.80% to 75.47%, and SR@5 from 64.93% to 70.13%. The MLP gate performs better with one distractor (83.80% vs. 81.27%). The evidence therefore supports an empirical benefit under stronger structured interference, with performance depending on the perturbation condition. Matching features to multiple learned reference vectors provides an explicit inductive bias for estimating support.

Under Gaussian noise, both gate-only variants degrade substantially, with small differences relative to the reported variability. The existing Table 4 separately shows that adding Control-Factor Selection to the prototype gate increases medium-noise success from 18.00% to 64.67%. This separates the Stage-I comparison from the complementary contribution of Stage II.



R2-2 — Training, Model Selection, and Evaluation Protocol

Reviewer comment

Please distinguish training, model-selection, and final evaluation data, including the use of held-out assets and evaluation-image occupancy. Do the three reported seeds retrain the full model or only vary evaluation randomness?

Response

For DSSF, we use the epoch-10 checkpoint on all four tasks, consistent with the 10-epoch training setting reported by LeWM (Maes et al., 2026). The three reported seeds vary evaluation randomness for this fixed checkpoint; they do not retrain the full model. The standard deviations therefore characterize evaluation variability.

Changes and protocol details in the revised manuscript:

  • Section 4.1 and Appendix J, Table 10: clarified the checkpoint epoch and that the three seeds repeat evaluation with fixed model weights.
  • Appendix D, Table 6: ResNet-18 is chosen using the backbone comparison of planning robustness, parameter count, and FLOPs.
  • Appendix B, Figure 9: the threshold analysis uses 500 validation images per distractor family with a fixed checkpoint and supports the default τ=0.5\tau=0.5 by quantifying the rejection/false-rejection tradeoff.
  • Appendix B, Scope and limitations: held-out in-family assets from PrismLayersPro (Chen et al., 2025) are excluded from gradient-based optimization but are used in validation forward passes.
  • Appendix G, Figure 12 and Eq. (7): the 30% Control budget is informed by occupancy in 1,500 clean initial evaluation observations per task.
  • Appendix G, Figure 11: the prototype count is selected from qualitative comparisons of 20, 30, 40, and 50 prototypes, using the smallest empirically effective setting, 20.

Exclusion from parameter updates does not imply absence of development exposure. The held-out condition therefore measures transfer beyond optimization assets. Likewise, the occupancy images contributed to a development choice and should not be described as an untouched test set.

AuthorsOpenReview ↗

Response to Reviewer GjxH (2/2)

R2-3 — Diagnosing Failures under Dynamic Distractors

Reviewer comment

For the existing Push-T Dynamic runs, is the main failure caused by nuisance retention, rejection of task-relevant features, or latent goal matching? Can you quantify this without adding a new benchmark?

Response

Nuisance retention, rejection of task-relevant features, and latent goal matching can interact: the first two can perturb the representations used for goal matching. We therefore quantify their observable effects rather than assign each failure to a single cause.

Added diagnostic analysis, reported below: we reuse 60 preselected cases from the existing Push-T evaluation, retaining the original distractor identities, sizes, and Static/Dynamic layouts. We run only the frozen encoder, with no new benchmark, training, or policy rollouts.

Distractor familyDynamic distractor-cell retentionReadout weight on distractor cellsDynamic foreground-proxy rejectionGoal-layout sensitivity RR (95% CI)
In-family0.21%0.014%10.15%0.618 [0.506, 0.746]
Shapes14.64%0.389%5.64%0.459 [0.379, 0.555]
Real-object0.13%0.004%7.58%0.557 [0.459, 0.668]

All rates are case means. Distractor-cell retention is the fraction of nuisance-labeled grid cells passing the Support Gate; readout weight is the total weight assigned to those cells in the final feature aggregation. Foreground rejection is measured on foreground-proxy cells, with a reference rejection rate of 0.028% on clean goal images.

For goal-layout sensitivity, we hold the clean goal image fixed and apply its original Static and Dynamic distractor layouts. Using the complete encoder outputs, we compute

{\operatorname{Mean}_{c}\operatorname{MSE}(z_{0,c}^{\mathrm{clean}},z_{g,c}^{\mathrm{clean}})}.$$ Here, $c$ indexes cases, and $z_0$ and $z_g$ denote initial and goal representations. Thus, $R$ compares the mean squared change caused solely by goal-layout variation with the mean squared clean initial–goal separation. The 95% confidence intervals use **2,000 paired bootstrap resamples within each evaluation seed** and are conditional on the fixed checkpoint and selected case pool; they do not include retraining variability. The diagnostics provide three complementary observations: - **Nuisance retention:** Shapes cells pass the gate more often, but the mean readout weight on distractor cells is below **0.4%** for every family. Gate retention alone therefore does not measure their influence on the final representation. - **Foreground rejection:** foreground-proxy rejection is higher under distractors than the clean reference. This reveals spatial rejection of foreground proxies, but the rejected-cell fraction is not a percentage of task information lost. - **Goal-representation sensitivity:** changing only the goal-side distractor layout produces a mean squared latent shift of **0.46–0.62 times** the mean squared clean initial–goal distance. The final goal representation therefore remains sensitive to irrelevant layout changes; $R$ is not a failure rate. **Our working hypothesis is that residual layout sensitivity can perturb latent goal matching, potentially together with foreground rejection, even when direct distractor-cell readout is small.** These measurements do not establish which mechanism dominates Dynamic failures: low weight on labeled distractor cells does not exclude nuisance contamination at other locations, and goal-latent drift alone does not show that the planner misranks candidate actions. If the clarifications and additional evidence help resolve your concerns, we hope you will consider updating your overall assessment of the paper accordingly. We would also welcome any further feedback and would be happy to clarify any remaining points.
Reviewer_XqSxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper adds a learned filtering and selection module before a JEPA latent predictor. Prototype matching filters local features using supervision from synthetic distractor masks, and sparse selection with an action-recovery objective constructs the final state. The method achieves substantial planning improvements under the tested distractors. My main concern is whether the synthetic problem setting captures a practically important planning challenge.

Critical strengths
  • The method is sensible: irrelevant visual content can interfere with latent prediction and goal matching, and filtering it before state construction is a reasonable approach.
  • The planning gains are substantial across four simulated tasks, with clean performance generally preserved.
  • The matched-backbone and augmentation ablations provide useful evidence that the proposed components contribute to the improvements.
Critical weaknesses
  1. The practical relevance of the evaluation is unclear. The experiments paste distractors onto simulated observations while leaving the physical environment unchanged. This provides a controlled robustness test, but the paper does not sufficiently establish that these corruptions represent visual distractions that meaningfully limit planning in realistic settings. My concern is primarily with the significance of the demonstrated problem, rather than the method’s ability to solve the benchmark.

  2. Training and evaluation share the same corruption mechanism. Training provides exact masks of synthetic overlays, and evaluation changes their appearances while retaining the same compositing process. The cross-family results demonstrate useful appearance transfer, but leave open whether the learned separation extends beyond this particular construction. Real-object cutouts remain pasted images, and the “Dynamic” condition changes current-versus-goal placement rather than testing continuously moving distractors.

Questions for authors

Does DSSF retain its advantage when the distractor-generation mechanism changes? What evidence suggests that robustness to these overlays translates into a meaningful benefit for practical visual planning?

Suggestions / Comments

  • Demonstrate transfer to a different distractor-generation mechanism, such as task-irrelevant objects rendered within the scene.
  • Show examples connecting the benchmark to failures under naturally occurring visual distractions. A focused experiment would be sufficient; a comprehensive real-robot study is unnecessary.
  • Compare against a targeted visual robustness method with comparable access to augmentation and synthetic-mask supervision.

Limitations

The paper acknowledges the shared overlay mechanism and the absence of evidence for broader visual shifts. These are appropriate limitations, but they also restrict the practical conclusions supported by the experiments.

Overall assessment

I lean weak reject. The method is reasonable and the experiments support its effectiveness within the chosen setup. However, I am not sufficiently convinced by the realism or practical significance of that setup to recommend acceptance.

3 comments
AuthorsOpenReview ↗

Response to Reviewer XqSx (1/2)

General Response

We appreciate your recognition of the rationale for filtering irrelevant visual information before state construction, the planning gains across four tasks, and the evidence from the controlled ablations. We address your comments point by point below. The revision adds a scene-rendered 3D clutter evaluation, asset and planning-case visualizations, and additional baseline and matched-supervision comparisons.

R1-1 — Transfer Across Distractor Mechanisms and Planning Benefits

Reviewer comment

Does DSSF retain its advantage when the distractor-generation mechanism changes? What evidence suggests that robustness to these overlays translates into a meaningful benefit for practical visual planning?

Response

Our original 2D overlays provide a controlled way to vary observation-level visual interference while keeping the task state and environment dynamics unchanged. Our zero-shot evaluations cover held-out in-family instances and cross-family distractor appearances under both Static and Dynamic current–goal layouts. However, these experiments vary appearances and layouts within the same 2D compositing process and therefore do not establish transfer across distractor-generation mechanisms.

Added in the revision:

  • Scene-rendered 3D distractor evaluation — Section 4.3.1, Table 2: planning success and degradation from clean performance on OGBench-Cube.
  • 3D asset-bank visualization — Appendix A.1, Figure 7: the 30 objects used for sampling.
  • Paired clean/cluttered planning case — Appendix C, Figure 10: successful and failed executions under the same cluttered task conditions.

The new evaluation renders three task-irrelevant objects directly in the scene, sampled from a bank containing 26 HouseCat6D scans (Jung et al., 2024) and four procedurally generated models. DSSF uses the existing checkpoint trained with PrismLayersPro (Chen et al., 2025) 2D distractor augmentation, without fine-tuning or test-time adaptation. It achieves 69.80 ± 1.60% success, versus 53.27 ± 2.08% for LeWM (Maes et al., 2026). The respective drops from clean performance are 5.07 and 14.73 percentage points. Each condition uses 500 episodes per evaluation seed across three seeds.

This supports transfer to a different distractor-generation mechanism and shows a downstream planning benefit in more realistic simulated scenes. Figure 10 makes that benefit concrete: both methods succeed in the clean case, while only DSSF reaches the goal after 3D clutter is introduced.

References:

Jung, H., et al. (2024). HouseCat6D - A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios. CVPR, pp. 22498–22508.

Chen, J., et al. (2025). PrismLayers: Open Data for High-Quality Multi-Layer Transparent Image Generative Models. arXiv:2505.22523.



R1-2 — Zero-Shot Transfer to Scene-Rendered 3D Clutter

Reviewer comment

Demonstrate transfer to a different distractor-generation mechanism, such as task-irrelevant objects rendered within the scene.

Response

The added OGBench-Cube evaluation changes the distractor-generation mechanism from RGB overlays to task-irrelevant 3D objects rendered within the scene.

Added evidence, shared with the preceding response:

  • Section 4.3.1, Table 2: the scene-rendered 3D planning comparison.
  • Appendix A.1, Figure 7: the distractor asset bank.
  • Appendix C, Figure 10: a paired clean/cluttered planning example and an explicit statement that the 3D distractors are unseen during training.

The bank comprises 26 scans from HouseCat6D (Jung et al., 2024) and four procedurally generated objects: a bottle, a box, a pyramid, and a sphere. All 30 assets are eligible for sampling; each scene contains three objects. The same DSSF checkpoint trained with PrismLayersPro (Chen et al., 2025) 2D distractor augmentation is evaluated without adaptation. DSSF achieves 69.80 ± 1.60% success, compared with 53.27 ± 2.08% for LeWM (Maes et al., 2026). This directly tests transfer from image compositing to scene-level 3D clutter.

AuthorsOpenReview ↗

Response to Reviewer XqSx (2/2)

R1-3 — Planning Successes and Failures under 3D Clutter

Reviewer comment

Show examples connecting the benchmark to failures under naturally occurring visual distractions. A focused experiment would be sufficient; a comprehensive real-robot study is unnecessary.

Response

The added case study connects scene-rendered household clutter to a concrete planning failure under paired clean and cluttered task conditions. The underlying asset bank combines HouseCat6D scans (Jung et al., 2024) with procedurally generated models.

Added in the revision:

  • Appendix C, Figure 10: initial, intermediate, and final states for DSSF and LeWM (Maes et al., 2026) in the paired case.
  • Section 4.3.1, Table 2: aggregate success rates supporting the qualitative comparison.

Both methods reach the goal in the clean scene. After a bottle, a cup, and a teapot are added, DSSF still reaches the goal, whereas LeWM leaves the cube on the table and fails to complete the task, as shown by the final states. DSSF uses its checkpoint trained with 2D distractor augmentation, without adaptation to these 3D objects.

The aggregate results are 69.80 ± 1.60% for DSSF versus 53.27 ± 2.08% for LeWM. Together, the paired example and success rates show a planning benefit under more naturalistic simulated clutter.



R1-4 — Targeted Robustness Baselines and Matched-Supervision Comparisons

Reviewer comment

Compare against a targeted visual robustness method with comparable access to augmentation and synthetic-mask supervision.

Response

We have added SMWM as a targeted robustness baseline. For supervision fairness, we compare our LeWM-based DSSF with a LeWM variant given the same augmentation and synthetic-mask supervision, and further test a conventional MLP gate.

Added in the revision:

(1) Targeted robustness — Figure 4 and Appendix I, Table 9. We evaluated SMWM (Ivashkov et al., 2026), a recent JEPA-style method that uses inverse dynamics to learn action-relevant representations and suppress uncontrollable distractors, across all four tasks under our 0/1/3/5-distractor protocol. With five distractors, we achieve 70.13% vs. 24.13% on Push-T, 98.93% vs. 87.80% on Two-Room, 79.27% vs. 48.20% on Reacher, and 73.33% vs. 58.47% on OGBench-Cube. SMWM has higher clean success on OGBench-Cube (88.13% vs. 74.87%), while we retain higher success under distractors.

(2) Matched augmentation and synthetic-mask supervision — Section 4.5, Table 4. Since LeWM (Maes et al., 2026) does not natively support spatial mask supervision, we equip it with mask-supervised Support Filtering while omitting Sparse Control-Factor Selection. This gives us a mask-supervised LeWM variant, reported as Support Only in Table 4, with the same augmentation, synthetic masks, backbone, latent predictor, auxiliary action-recovery loss, and per-episode CEM budget as our full model. Under matched augmentation and synthetic-mask supervision, our full DSSF improves SR@3 from 75.47% to 79.40% and medium Gaussian-noise success from 18.00% to 64.67%. These gains demonstrate the additional benefit of our Control-Factor Selection stage when mask supervision is held fixed.

We additionally compare this prototype-based variant with a conventional two-layer MLP gate under the same masks, training settings, and per-episode CEM budget; both omit Control-Factor Selection (Appendix E, Table 7). Our prototype gate improves SR@3 from 69.80% to 75.47% and SR@5 from 64.93% to 70.13%, although the MLP gate has a higher mean with one distractor. Table 3 provides further LeWM-based module ablations under the same augmentation and planning protocol. The structured-distractor comparisons use 500 episodes per evaluation seed; the Table 4 noise comparison uses 50 for both Support Only and Full DSSF.

References

Ivashkov, P., Balestriero, R., and Schölkopf, B. (2026). Sensorimotor World Models: Perception for Action via Inverse Dynamics. arXiv:2606.20104.

If the clarifications and additional evidence below address your concerns, we hope you will consider updating your overall assessment of the paper accordingly.

Reviewer_XqSxOpenReview ↗

Rebuttal acknowledgment

Thanks for running the scene-rendered clutter experiment. Testing the same 2D-trained checkpoint on 3D clutter without adaptation directly addresses my main concern about transfer beyond pasted overlays. The new controls also help clarify the contribution of the method. The 3D evidence is limited to one simulated task, but it is sufficient for me to raise my rating to 3 (weak accept).

AI assistance was used to draft this acknowledgment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.