- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper presents DSSF, Distribution-Supported Sparse Control Factors, a latent-state representation learning method for Joint-Embedding Predictive Architecture (JPEA)-based latent world models. DSSF targets zero-shot visual generalization during test time when unseen visual distractors or seen, control-irrelevant factors may contaminate goal-conditioned planning. By leveraging foreground masking that divides task-relevant spatial information from control-irrelevant distractors, DSSF composes the compact latent state from high-dimensional visual inputs, where potential in-distribution, control-irrelevant, or out-of-distribution, task-irrelevant distractors are included. DSSF demonstrates strong performance on 4 goal-conditioned visual planning tasks, spanning from maze-escaping to pick-and-place tasks, across latent world model baselines.
Critical strengths
- Novel problem formulation and algorithm: Many researchers have been diving into the JEPA-based latent world model recently. While improving visual planning over the long horizon by increasing effective horizons or incorporating hierarchical architectures has been suggested, learning a compact latent world model that is robust to visual distractors has not been explored enough, despite its necessity. DSSF can be incorporated into a JEPA-based latent world model as a plug-and-play module, since DSSF only changes how the encoder is trained and does not touch the underlying dynamics prediction task or test-time planning. Substantial qualitative and quantitative experimental evidence supports that DSSF can manage test-time unseen visual distractors effectively.
Critical weaknesses
Weak connection to prior literature in visual generalization: While visual generalization in JEPA-based latent world models has yet to be explored, zero-shot or few-shot visual generalization research has been investigated significantly in reinforcement learning. For instance, data augmentation, pretrained vision backbones, and representation learning affect the test-time visual generalization performance of model-free RL agents [1-4]; learning the bisimulation principle or distinguishing task-relevant from task-irrelevant features enables robust learning for model-based RL [5-8]. While these works may not connect directly to JEPA-based latent world models with visual generalization, providing a holistic view of visual generalization for decision-making could further improve legibility and understanding of this work. Unclear description of the algorithm components and experimental setup: It takes substantial effort and time to fully understand what DSSF is and how DSSF works under the experimental setup. I summarize my concerns and remaining questions below. I believe resolving the concerns could improve the quality of this paper significantly.
Reference
[1] Hansen, Nicklas, Hao Su, and Xiaolong Wang. "Stabilizing deep q-learning with convnets and vision transformers under data augmentation." NeurIPS 2021
[2] Huang, Yangru, et al. "Spectrum random masking for generalization in image-based reinforcement learning." NeurIPS 2022
[3] Yuan, Zhecheng, et al. "Pre-trained image encoder for generalizable visual reinforcement learning." NeurIPS 2022
[4] Batra, Sumeet, and Gaurav S. Sukhatme. "Zero-shot generalization of vision-based rl without data augmentation." ICML 2025
[5] Zhang, Amy, et al. "Learning invariant representations for reinforcement learning without reconstruction." ICLR 2021
[6] Fu, Xiang, et al. "Learning task informed abstractions." ICML 2021
[7] Bharadhwaj, Homanga, et al. "Information prioritization through empowerment in visual model-based RL." ICLR 2022
[8] Zhu, Chuning, et al. "Repo: Resilient model-based reinforcement learning by regularizing posterior predictability." NeurIPS 2023
Questions for authors
Regarding the algorithm
- The two-stage latent-state construction procedure is not described completely:
- How are the shallow and deep features, and , extracted from the encoder? Could you define them formally (e.g., an equation)?
- What does stand for in (L149)?
- What are the formal definitions of location for the shallow and deep features (L195-196, L226)?
- Could you add proper citations for the terminology, e.g., prototype, support gate, support descriptor? I think these expressions are not commonly used in latent world model research, so adding citations could help future readers understand better.
- What is in L242?
- Could you add a pseudocode table to describe how DSSF trains its encoder? I think Figure 2 does not provide enough context to understand Section 3.
Regarding the experimental setup
- How is DSSF sensitive to the threshold ? Could you perform a sweep search on a toy example?
- How does an ambiguous set negatively affect the two-stage latent-state learning if the set is included in a positive or negative set?
- How did you place the random visual distractors in the clean image when augmenting the original view? Did you allow the distractors to occlude the task-relevant regions, e.g., the T-shaped bar in the Push-T task or a manipulator arm in the Cube task?
- Could you add an extra baseline, i.e., JEPA-Bisim*, for the main robustness performance comparison in Figure 4? I think JEPA-Bisim would be a good comparison baseline, since it also investigates visually robust latent world models.
- Could you explain how Gaussian noise is applied during test time for the internal ablation in Section 4.5?
- How do you determine whether a pixel contains non-background or not throughout Appendix E? What is the criterion?
*Toso, Leonardo F., et al. "Learning invariant visual representations for planning with joint-embedding predictive world models." arXiv 2026
5 comments
Response to Reviewer CSU9 (2/5)
R3-3 — Spatial Indexing and Feature Alignment
Reviewer comment
What are the formal definitions of location i for the shallow and deep features (L195–196, L226)?
Response
The index refers to a location on the shared 28 × 28 spatial grid after shallow/deep feature alignment.
Manuscript change — Section 3.1, following Eq. (2): we have added the flattened-index definition and the correspondence between the two descriptors.
The shallow map initially has 56 × 56 locations. PixelUnshuffle(2) packs the four 64-dimensional feature vectors in each 2 × 2 block into one 256-dimensional vector, producing a 28 × 28 map. The deeper map already has 28 × 28 locations; the 1 × 1 convolution converts each 128-dimensional vector into a 192-dimensional vector independently, keeping all grid locations unchanged.
On these two aligned maps, and denote the row and column, with , and . The vectors and are read from the same row and column in their respective maps. For example, selects row 1, column 2 from both maps.
R3-4 — Terminology, Definitions, and Supporting References
Reviewer comment
Could you add proper citations for the terminology, e.g., prototype, support gate, support descriptor? I think these expressions are not commonly used in latent world model research, so adding citations could help future readers understand better.
Response
Learnable prototypes have an established representation-learning background; Support Gate and Support descriptors are names for specific components in DSSF.
Changes in Section 3.2.1 and the bibliography:
- Added SwAV (Caron et al., 2020) as conceptual background for learnable visual prototypes.
- Added PatchCore (Roth et al., 2022) as related work on local feature matching using a feature bank.
- Explicitly defined the Support Gate and Support descriptors in the method description.
The Support Gate is the binary spatial mask obtained by thresholding the normalized Support response. Support descriptors are the deep features retained after positional encoding and Support Filtering and passed to Control-Factor Selection. The citations provide conceptual context: PatchCore's feature memory bank differs from our learned prototype bank, and the DSSF-specific terms are defined within our method.
References
Caron, M., et al. (2020). Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. NeurIPS 33, pp. 9912–9924.
Roth, K., et al. (2022). Towards Total Recall in Industrial Anomaly Detection. CVPR, pp. 14318–14328.
R3-5 — Definition of the Output Projection
Reviewer comment
What is in L242?
Response
is a learnable output projection matrix applied to the weighted aggregation of selected Control factors before the subsequent projector.
Manuscript change — Section 3.2.2: we have added this definition immediately after the aggregation expression, distinguishing from the projector. It is jointly optimized with the encoder and the other trainable components through the overall objective.
R3-6 — Pseudocode for Encoder and Latent Dynamics Training
Reviewer comment
Could you add a pseudocode table to describe how DSSF trains its encoder? I think Figure 2 does not provide enough context to understand Section 3.
Response
We have added a standalone training pseudocode to connect the architecture in Figure 2 with the operations and losses in Section 3.
Added in the revision:
- Appendix J, Algorithm 2: DSSF Encoder and Latent Dynamics Training: the complete training sequence.
- Section 3.3: a direct reference to the new pseudocode.
Algorithm 2 describes how we apply the same distractor layout to every frame in each training sequence and generate the corresponding distractor masks. It also covers aligned shallow/deep feature extraction; prototype-based Support Filtering and mask supervision; sparse Control-Factor Selection and latent construction; and joint optimization using latent prediction, SIGReg (Balestriero and LeCun, 2025), Support supervision, and auxiliary action recovery. It also specifies straight-through gradient estimation for the hard Support Gate and Top-K selection.
Response to Reviewer CSU9 (3/5)
Regarding the experimental setup
R3-7 — Sensitivity to the Support Threshold
Reviewer comment
How is DSSF sensitive to the threshold ? Could you perform a sweep search on a toy example?
Response
We have added a fixed-checkpoint threshold sweep on Push-T, measuring how stricter gating trades distractor rejection against rejection of valid clean features.
Added in the revision — Appendix B, Figure 9 and the Threshold sensitivity paragraph: two panels show distractor rejection and valid-clean false rejection as varies from 0.1 to 0.99, accompanied by quantitative analysis in the text.
Each family contains 500 fixed images with five distractors. We hold the model and images fixed. The table below reports distractor rejection / valid-clean false rejection (%): the proportion of distractor locations removed by the gate and the proportion of valid clean locations incorrectly removed, respectively. The ambiguous boundary band is excluded from valid-clean statistics. The default threshold is highlighted in bold.
| Held-out in-family | Cross-family Shapes | Cross-family Real-object | |
|---|---|---|---|
| 0.1 | 99.670 / 0.000 | 86.662 / 0.490 | 99.617 / 0.080 |
| 0.3 | 99.680 / 0.002 | 86.954 / 0.744 | 99.663 / 0.132 |
| 0.5 | 99.692 / 0.009 | 87.174 / 0.950 | 99.688 / 0.187 |
| 0.7 | 99.703 / 0.025 | 87.301 / 1.212 | 99.717 / 0.262 |
| 0.9 | 99.719 / 0.094 | 87.627 / 1.892 | 99.761 / 0.526 |
| 0.95 | 99.734 / 0.137 | 87.788 / 2.510 | 99.770 / 0.824 |
| 0.99 | 99.766 / 0.260 | 88.145 / 4.732 | 99.820 / 1.855 |
For , distractor rejection remains within 99.67–99.72% for held-out in-family assets and 99.62–99.76% for Real-object distractors. At , their valid-clean false rejection rates are 0.01% and 0.19%.
For Shapes, raising from 0.5 to 0.99 increases distractor rejection from 87.17% to 88.15%, while increasing clean false rejection from 0.95% to 4.73%. Thus, rejection has a broad intermediate-threshold plateau, while stricter gating increasingly removes valid clean locations. These measurements quantify the threshold sensitivity of the Stage-I gate.
R3-8 — Ambiguous Regions in Support Supervision
Reviewer comment
How does an ambiguous set negatively affect the two-stage latent-state learning if the set is included in a positive or negative set?
Response
The ambiguous set is excluded from the Support loss, while its forward selection remains governed by the learned Support Gate and Control-Factor Selector.
Manuscript change — Section 3.2.1: we have added the reason for excluding ambiguous labels and clarified that loss exclusion does not force a location to be retained or discarded in the forward pass.
Locations near a distractor boundary can contain mixed clean/distractor information through their receptive fields even when no distractor pixel directly covers the location. Labeling them positive may encourage nuisance-contaminated descriptors to enter the second stage. Labeling them negative may remove useful scene information before Control-Factor Selection. Excluding these ambiguous labels avoids imposing either potentially misleading target.
Response to Reviewer CSU9 (1/5)
General Response
We appreciate your recognition of the problem formulation and algorithmic contribution, DSSF's integration into an existing JEPA planning pipeline, and the qualitative and quantitative evidence for handling unseen distractors. We address your comments point by point below. The revisions expand the connection to visual RL generalization and add formal definitions, training pseudocode, a threshold sweep, explicit augmentation and evaluation protocols, and updated baseline comparisons.
R3-Critical Weaknesses — Connections to Prior Work on Visual Generalization
Reviewer comment
Response
We have incorporated all eight suggested references into Section 2, “Robust Visual Representations under Nuisance Shifts.” The first added sentence connects visual RL generalization to stable data augmentation (Hansen et al., 2021), spectral masking (Huang et al., 2022), pretrained visual encoders (Yuan et al., 2022), and latent disentanglement with associative memory (Batra and Sukhatme, 2025). The second added sentence discusses complementary approaches to learning task-relevant representations, including bisimulation (Zhang et al., 2021), task-informed abstractions (Fu et al., 2021), empowerment-based information prioritization (Bharadhwaj et al., 2022), and posterior-predictability regularization (Zhu et al., 2023). We also revised the closing sentence of this paragraph to clarify how DSSF builds on these foundations by explicitly separating training-distribution support from control relevance through prototype-based Support Filtering and sparse Control-Factor Selection for offline, reward-free JEPA planning.
R3-1 — Shallow and Deep Feature Extraction
Reviewer comment
Response
The shallow and deep descriptors come from spatially aligned feature maps extracted after the 5th and 9th main-path convolutional layers of ResNet-18 (He et al., 2016), corresponding to its first and second residual stages.
Manuscript changes — Section 3.1, Eq. (2): we have added the extraction points, feature-map dimensions, and formal alignment operations, together with the definitions of the resulting channel vectors.
For a 224 × 224 input, the raw maps are 64 × 56 × 56 and 128 × 28 × 28. We align them as
\mathbf F_t=\Pi_{1\times1}(\mathbf H_t^d),$$ where $\mathcal U_2$ denotes PixelUnshuffle(2) and $\Pi_{1\times1}$ is a learnable 1 × 1 convolution with 192 output channels. The aligned maps are 256 × 28 × 28 and 192 × 28 × 28; $p_{t,i}$ and $f_{t,i}$ are their channel vectors at location $i$. This supplies local appearance information for Support Filtering and more contextual features for state construction. --- --- ### **R3-2 — Definition of the Visual Distractor Augmentation Operator** **Reviewer comment** > What does A stand for in $\tilde o_t=A(o_t;d)$ (L149)? **Response** $A$ is the **visual distractor augmentation operator**. The nuisance factor $d$ specifies attributes including appearance, position, scale, and opacity. **Manuscript change — Section 3.1:** we have explicitly defined $A$ where the notation is introduced. **Appendix A.2** documents the background-placement constraints. For a single distractor, alpha compositing can be written as $$A(o_t;d)=(1-\alpha_d)\odot o_t+\alpha_d\odot D_d,$$ where $D_d$ is the image-aligned distractor RGB image and $\alpha_d$ is its opacity mask. Multiple distractors are applied sequentially under the placement constraints. This changes the visual observation while preserving the task state and environment dynamics.