acceptodds
Under review as a conference paper at ICLR 2027

JEPA-Bisim: Learning Robust Visual Representations for Planning with Joint-Embedding Predictive World Models

Abstract

World models learned from high-dimensional visual observations allow agents to make decisions and plan directly in latent space, avoiding pixel-level reconstruction. However, recent latent predictive architectures (JEPAs), including the DINO world model (DINO-WM), display a degradation in test time robustness due to their sensitivity to “slow features". These include visual variations such as background changes and distractors that are irrelevant to the task being solved. We address this limitation by augmenting the predictive objective with a bisimulation encoder that enforces control-relevant state equivalence, mapping states with similar transition dynamics to nearby latent states while limiting contributions from slow features. We evaluate our model on a navigation task (PointMaze) and on a manipulation task (PushT) under different test-time background changes and visual distractors. Across all benchmarks, our model consistently improves robustness to slow features while operating in a reduced latent space, up to smaller than that of DINO-WM. Moreover, our model is agnostic to the choice of pretrained visual encoder and maintains robustness when paired with DINOv2, SimDINOv2, and iBOT features.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint-Embedding Predictive Architectures (JEPA), world models, bisimulation, invariant representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GReROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JEPA-Bisim projects frozen visual features into a smaller patch-level latent space and jointly trains action-conditioned dynamics with a reward-free bisimulation-inspired loss. A PCA-based variance/covariance regularizer is intended to prevent collapse while permitting removal of dominant nuisance directions. Planning uses CEM with terminal latent distance. Experiments on PointMaze and PushT evaluate robustness to background changes and distractors, with additional visual-encoder comparisons. The paper also proposes an on-policy cost-to-go bound as a theoretical justification for invariance.

Critical strengths
  1. The robustness problem is relevant. Frozen visual features can retain nuisance variation, so adapting the representation used by a planner is a reasonable direction. Testing background shifts and distractors directly addresses the intended failure mode.

  2. The implementation is modular and compact. A patch-wise projection preserves spatial structure while reducing each DINOv2 patch from 384 to 32 channels, allowing the approach to be used with several pretrained encoders.

  3. PointMaze shows encouraging robustness. Table 1 reports 78% native success and small changes under the tested shifts for DINO-Bisim, whereas the DINO-WM baseline degrades substantially under several background changes. The paper includes a domain-randomization baseline and acknowledges that lower relative degradation need not mean higher absolute success.

Critical weaknesses
  1. Theorem 3.1 is false under its stated approximate-preservation assumption. The assumption permits representation-distance error up to epsilon, but the bound contains no epsilon term. Consider two states with identical transitions, hence zero reward-free bisimulation distance, and scalar representations h(z) = 0 and h(z') = delta, where 0 < delta <= epsilon. With goal representation zero and T = 1, the assumed approximation and boundedness hold, yet the cost difference is delta squared while Equation (5) bounds it by zero. A corrected theorem needs an approximation term and explicit metric assumptions. Moreover, a fixed-policy cumulative-cost bound does not directly guarantee the terminal-cost CEM planner's decisions.

  2. The practical target is not justified as a Wasserstein bisimulation estimator. Independently sampled next states form a product coupling, not generally the optimal coupling required by Wasserstein distance. Even identical transition distributions can have positive expected distance between independent draws, although their Wasserstein distance is zero. The standard recursive metric also specifies the ground pseudometric (Castro, 2020); the paper leaves this unclear. With no reward or other distinguishing term, the usual discounted recursive construction admits the all-zero bounded fixed point. Anti-collapse regularization can create useful features, but does not establish the claimed control equivalence, especially for actions outside the behavior policy.

  3. Relative robustness hides a large absolute control-performance loss on PushT. With full data, native success falls from 78% for DINO-WM to 44% for DINO-Bisim. Converting the rounded relative degradations in Table 2 gives approximately 66.3% versus 44.0% success under SC and 49.9% versus 37.8% under LC. DINO-Bisim is worse on four of the five shifted conditions in this full-data setting, although its relative degradation is smaller. Thus, the main manipulation result does not demonstrate a generally better robust planner.

  4. The contribution of bisimulation is not isolated in the available experiments. Compact projection, predictive training, bisimulation loss, and PCA-based regularization all change together. Matched 32-dimensional dynamics without bisimulation, standard variance/covariance regularization, and PCA-only controls are needed. Figure 4 does not establish that the dominant component is always nuisance: scene layout may be essential for obstacle avoidance. The exact PCA loss and the referenced ablations are deferred to an appendix absent from the supplied PDF.

  5. The empirical scope and uncertainty are limited. Evaluation covers two environments with 50 initial-goal pairs per condition, without reported repeated-training variation or paired intervals. This makes small improvements such as negative relative degradation difficult to interpret. Dimension reduction alone also does not establish an equivalent reduction in complete planning latency.

Questions for authors
  1. How is the T = 1 counterexample excluded by the stated assumptions of Theorem 3.1? Please supply the proof and a corrected bound with explicit approximation dependence.
  2. What ground metric defines the Wasserstein term, and what justifies independent action/transition pairs as its estimator? How is off-policy action distinguishability preserved for CEM?
  3. Can you report absolute success under every shift, with uncertainty, and explain the 78% to 44% native-performance drop on full-data PushT?
  4. Which gains remain against matched compact predictive models and PCA/variance regularization without the bisimulation term?
4 comments
AuthorsOpenReview ↗

Point-to-point response to Reviewer GReR (1/2)

We thank the Reviewer for recognizing the relevance of guaranteeing robustness to distractors and background changes at test-time. And also for recognizing our compact and modular patch-wise encoder, and the PointMaze results, including the domain-randomization comparison. We address all five weaknesses and the corresponding questions below.

1. Theorem 3.1 and the T=1T=1 counterexample.

The counterexample is valid under the stated assumptions. An additive distance approximation permits d(z,z′)=0d(z,z')=0 while ∣∣hη(z)−hη(z′)∣∣2>0||h_ \eta(z)-h_ \eta(z')||_ 2>0. The cost difference therefore cannot be bounded by a multiple of d(z,z′)d(z,z') alone. We correct the statement and Appendix A.11.

Our response to Reviewer 2Xdi, point 2, provides the corrected statement and proof. For a specified pseudometric dd, Wd(Pπ(⋅∣z),Pπ(⋅∣z′))≤Ld(z,z′)W_ d(P_ \pi(\cdot\mid z),P_ \pi(\cdot\mid z'))\leq Ld(z,z'), bounded embeddings, and ∣∣hη(z)−hη(z′)∣∣≤d(z,z′)+ε||h_ \eta(z)-h_ \eta(z')||\leq d(z,z')+\varepsilon,

\leq4H_ w\left[\sum_ {t=0}^{T-1}(\gamma L)^t d(z,z')+ \varepsilon\sum_ {t=0}^{T-1}\gamma^t\right].$$ At $T=1$ and $d=0$, the bound is $4H_w\varepsilon$, rather than zero, so it covers the suggested counterexample. Under $\gamma L\leq1$, the corrected bound retains the original $T d(z,z')$ dependence. For fixed $0\leq\gamma<1$, the added term is at most $4H_w\varepsilon/(1-\gamma)$, so it vanishes as $\varepsilon \to 0.$ **2. Ground metric, independent pairs, and off-policy actions.** Our objective uses transition-distance supervision to adapt pretrained features jointly with action-conditioned prediction and anti-collapse regularization. The target in Section 2.3 is $$\gamma||T_ \phi(w,a)-T_ \phi(w',a')||_ 2.$$ Independent transition pairs provide a tractable comparison signal, but do not generally realize the optimal Wasserstein coupling. For exact successor samples and ground pseudometric $\rho$, $$W_ \rho(P_ \pi(\cdot\mid z),P_ \pi(\cdot\mid z')) \leq \mathbb E[\rho(U,V)],$$ with equality only when the independent coupling is optimal. We therefore describe this term as a *bisimulation-inspired transition-distance regularizer*, rather than an exact Wasserstein estimator. This preserve the reward-free learning procedure and its empirical robustness results while stating precisely what the objective establishes. **3. Relation to terminal-cost CEM planning.** As detailed in our response to Reviewer 2Xdi (point 3), the argument extends to the terminal objective. For every common candidate sequence $\boldsymbol a$, $$|J_ z(\boldsymbol a)-J_ {z'}(\boldsymbol a)| \leq 4H_ w[L^Td(z,z')+\varepsilon].$$ Thus, when two observations have small distance $d(z,z')$, the same candidate action sequence has similar expected terminal costs from either observation, up to the representation approximation error $\varepsilon$. We will include this extension in the revision. **4. Absolute PointMaze and PushT performance.** The revised paper reports absolute success for every condition in Table 1 (PointMaze), Table 2 (matched-width PushT), and Appendix A.12 (PushT training-data ablation). The compact model has lower native success on PushT, particularly with the full dataset. With $1{,}000$ samples, increasing DINO-Bisim's width from $32$ to $384$ raises native success from $36$\% to $40$\%. At matched width, DINO-Bisim improves absolute success under every shift, with native success of $40$\% versus $48$\% for DINO-WM. See the new PushT table in our general response. The additional Wall results (see our general response) further show higher absolute success on all five selected patterned backgrounds, with comparable native success ($90$\% versus $92$\%). **5. Component attribution, PCA, and layout information.** The appendix is included in our **submitted supplementary materials**. In particular, Appendix A.3 presents the PCA objective and visualizations of the learned representations before and after the regularization. Appendix A.10 evaluates seven planning horizons. and Table 1 compares DINOv2, SimDINOv2, iBOT, and end-to-end training. This evaluates representation learning metrics, planning horizon, and encoder choice. The ablation table in our general response adds the compact projector, PCA-only, and bisimulation ablations, with all metrics defined there. Our objective is to suppress changes in appearance that leave the physical task unchanged. Obstacle geometry must remain available to the planner because it determines feasible motion. The PCA visualizations concern the tested background variations and do not establish that PC1 is universally control-irrelevant. As clarified in our response to Reviewer 2Xdi, the implementation uses a fixed PCA basis with explicit variance targets, together with covariance regularization in the original per-patch coordinates. We will make this more clearer in our revision.
AuthorsOpenReview ↗

Point-to-point response to Reviewer GReR (2/2)

6. Scope, uncertainty, and planning latency.

As discussed in the general response and our response to Reviewer 2Xdi (point 7), we evaluate 5050 initial-goal pairs per condition following the same evaluation protocol as DINO-WM.

The additional Wall evaluation is reported in our general response.

Our compactness claim relies on latent dimension: 196×384196\times384 is reduced to 196×32196\times32, exactly 12×12\times fewer coordinates. We will clarify that this does not imply a measured 12×12\times reduction in total planning latency, however, it yields to planning on a much more compact latent space.

Dear Reviewer,

We would like to follow up on our responses to your concerns and questions and kindly ask you to adjust your score if your concerns/question have been addressed. Thank you!

Dear Reviewer,

We thank you for your time reviewing our work. We are happy to discuss any additional question you may have. If our responses have addressed your questions and concerns, we would appreciate if the score can be adjusted accordingly. Thank you!

Reviewer_wWJAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to mitigate the sensitivity of JEPA WMs by augmenting the predictive objective with a bisimulation encoder that enforces control-relevant state equivalence. It demonstrates improvement on robustness under different test-time background changes and visual distractors.

Critical strengths
  • The sensitivity of JEPA WMs to slow features is an important question, and the paper provides a new approach to mitigate it
  • The implementation is simple and doesnt require new training data
  • Tested under different background changes and visual distractors (though limited to only two environments)
Critical weaknesses
  • The contribution of individual modules is unclear: need an ablation on PCA. The baseline should also include DINOWM+projector but without bisim.
  • It would strengthen the paper to include more environments
Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Point-to-point response to Reviewer wWJA

We thank the Reviewer for recognizing the relevance of the slow-feature sensitivity in JEPA world models, the simplicity of our implementation, and the fact that it requires no additional training data. We respond to both weaknesses below.

1. PCA and projector-only ablations.

Our submitted version already includes several complementary experiments and ablations: Table 1 compares three pretrained encoders and end-to-end training. Table 2 evaluates three training-data sizes. In addition, in Appendix A.10 (please refer to our supplementary materials) reports planning horizons 5,10,15,20,25,50,1005,10,15,20,25,50,100. The horizon ablation improves mean success over NC/SC/C from .78.78 at horizon 55 to .92.92 at horizon 2525, showing that the default short horizon is not the only regime in which the model plans successfully. Appendix A.3 also illustrates the latent geometry with the standard VICReg regularization at epoch 5050 and after the PCA-based VICReg regularization is enabled at epoch 9090.

The table in our general response adds the projector and PCA ablations. The transition-distance term improves checker-background matching accuracy with VICReg. The modified PCA variant provides a smaller gain under the checker background. Absolute planning success is reported separately in the Wall results.

2. Additional environments.

PointMaze and PushT evaluate navigation and contact-based manipulation under five visual perturbations. We also evaluate the Wall task. Our tables in the general comment report the results. See the Wall subsection in the revised paper for the backgrounds figure.

Dear Reviewer,

We would like to follow up on our responses to your concerns and questions and kindly ask you to adjust your score if your concerns/question have been addressed. Thank you!

Thanks for adding the new ablation and wall env! I maintain my positive rating. It would be interesting to see the method applied to more challenging robotic manipulation tasks in the future.

We thank the reviewer for their time evaluating our work and for their positive assessment! Thank you!

Reviewer_2XdiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the sensitivity of visual world models to background changes and distractors. JEPA-Bisim adds a trainable patchwise projection to frozen visual features and learns latent dynamics using prediction and reward-free bisimulation-style losses. A PCA-based variance–covariance regularizer is used to prevent collapse and reduce sensitivity to nuisance features. The learned model is used for goal-conditioned planning with CEM.

Experiments cover PointMaze and PushT under several visual shifts, with additional PointMaze results using different pretrained encoders. The paper also presents a bound relating latent-state similarity to differences in planning cost.

Critical strengths
  1. The PointMaze results are convincing under the stronger background shifts. Native-condition success is close to DINO-WM (0.78 versus 0.80), while performance under LCG improves from approximately 0.48 to 0.78. The domain-randomization baseline also performs worse under this condition. These results support the usefulness of adapting pretrained features for planning under appearance changes.

  2. The adaptation is relatively lightweight and preserves the spatial patch structure. Reducing each patch embedding from 384 to 32 dimensions gives a substantially smaller representation without retraining the visual backbone. The experiments with DINOv2, SimDINOv2, and iBOT also suggest that the approach can be applied to more than one feature extractor.

Critical weaknesses
  1. I have concerns about the definition of reward-free bisimulation and the theoretical justification built on it. Under the usual recursive definition, where the Wasserstein distance uses the bisimulation pseudometric as its ground metric, removing the reward term leaves the zero pseudometric as the unique bounded discounted fixed point. It is unclear what anchors a nontrivial distance in the proposed formulation. Variance regularization can prevent the learned representation from collapsing, but the connection between that representation and the claimed reward-free bisimulation metric still needs to be established.

There is also a gap in Theorem 3.1. The theorem allows an approximation error ε, but the final bound contains no corresponding error term. In Appendix A.11, the proof first obtains an additive 4H_w ε term and then assumes ε ≤ d^π(z,z′). This condition is absent from the theorem and cannot hold for distinct zero-distance states when ε > 0. The bound therefore does not follow from the assumptions as stated.

  1. The empirical evidence is mixed on PushT, and the contribution of the individual components remains unclear. With the full dataset, native-condition success drops from 0.78 for DINO-WM to 0.44 for DINO-Bisim. Converting the relative degradations in Table 2 back to absolute success gives an average over the five shifted conditions of approximately 0.524 and 0.371, respectively. DINO-Bisim is worse in four of these five conditions. The paper acknowledges the lower native performance, but the reduced relative degradation alone provides limited evidence of improved practical performance on this task.

The experiments also lack a matched 32-dimensional projection baseline without the bisimulation loss, as well as quantitative planning ablations separating bisimulation from PCA-based regularization. These controls are important for understanding which component produces the PointMaze gains. Results are reported over 50 evaluation episodes per condition, without uncertainty estimates or variation across independent training runs, making smaller differences difficult to assess.

Questions for authors
  1. What is the ground metric in the reward-free Wasserstein term? If it is d^π itself, how does the formulation obtain a nontrivial bounded fixed point after removing rewards? If another ground metric is intended, please clarify how it connects to the definition used in Appendix A.11.

  2. What is the corrected statement of Theorem 3.1 when ε > 0? Please account for the approximation-error term and clarify the additional assumptions needed. It would also help to explain how the fixed-policy, accumulated-cost result supports the terminal-cost CEM planner used in the experiments.

  3. Is the PCA basis recomputed from each current batch or estimated once and then fixed? Appendix A.3 appears to describe both. With a basis computed from the same covariance matrix being regularized, the covariance in PCA coordinates is diagonal, so the stated off-diagonal penalty is zero. Please specify the implementation and training schedule used for the reported planning results.

  4. Could you provide matched planning ablations with the bisimulation loss removed and with standard variance–covariance regularization replacing the PCA variant? Absolute success rates under all shifts, together with variability across training runs, would help determine whether the gains persist after accounting for native-performance differences.

4 comments
AuthorsOpenReview ↗

Point-to-point response to Reviewer 2Xdi (1/2)

We thank the Reviewer for recognizing our convincing PointMaze results under strong background shifts, the lightweight patch-wise adaptation that leads to a much more compact latent predictive dynamics, and its applicability across pretrained encoders. We address each concern below.

1. Ground metric, removing the reward term, and nontriviality. Our implementation learns a representation through the combined predictive, transition-distance, and variance regularization objectives. It does not compute the fixed point obtained by removing rewards from the standard bisimulation recursion. The Reviewer is correct that the recursion admits only the zero bounded fixed point, i.e., for γ<1\gamma<1,

∣∣d∣∣∞=∣∣γWd(Pπ,Pπ)∣∣∞≤γ∣∣d∣∣∞⟹d≡0.||d||_ \infty = ||\gamma W_ d(P_ \pi,P_ \pi)||_ \infty \leq \gamma||d||_ \infty \quad\Longrightarrow\quad d\equiv0.

We will adjust the interpretation accordingly. In our training objective, the transition-distance target is

γ∣Tϕ(w,a) ⁣∣Tϕ(w,a)−Tϕ(w′,a′)∣ ⁣Tϕ(w,a)∣2.\gamma\left|\vphantom{T_\phi(w,a)}\right.\!\left|T_ \phi(w,a)-T_ \phi(w',a')\right|\!\left.\vphantom{T_\phi(w,a)}\right|_ 2.

This term encourages latent distances to reflect differences in predicted successor embeddings, while the predictive loss constrains action-conditioned dynamics and variance regularization prevents total collapse. The learned representation is shaped by all three objectives. We will then refer to the implemented term as a bisimulation-inspired transition-distance regularizer.

2. Refined Theorem 3.1, including ε>0\varepsilon>0. Let dd be a specified measurable pseudometric on a Markov state space, and let WdW_d denote the 11-Wasserstein distance with that same ground pseudometric. Suppose that

Wd(Pπ(⋅∣z),Pπ(⋅∣z′))≤Ld(z,z′),∣∣hη(z)−hη(z′)∣∣2≤d(z,z′)+ε,∣∣hη(z)∣∣2,∣∣wg∣∣2≤Hw.W_ d(P_ \pi(\cdot\mid z),P_ \pi(\cdot\mid z'))\leq Ld(z,z'),\qquad ||h_ \eta(z)-h_ \eta(z')||_ 2\leq d(z,z')+\varepsilon,\qquad ||h_ \eta(z)||_ 2,||w_ g||_ 2\leq H_ w.

We define ST(x):=∑t=0T−1xtS_T(x):=\sum_{t=0}^{T-1}x^t. Then, we have

\tag{1}$$ In particular, if $\gamma L\leq1$, we obtain $$|J_T^\pi(z)-J_T^\pi(z')| \leq4H_w\left[Td(z,z')+ \varepsilon\frac{1-\gamma^T}{1-\gamma}\right].$$ ***Effect of the correction.*** *We emphasize that under the stated assumptions and* $\gamma L\leq 1$, *the bound keeps the original linear dependence on* $T d(z,z')$ *and adds* $4H_w\varepsilon S_T(\gamma)$. *For fixed* $0\leq\gamma<1$, *the geometric sum is bounded by* $1/(1-\gamma)$, *so the correction is* $O(\varepsilon)$ *uniformly in* $T$ *and vanishes as* $\varepsilon\to0$. *Proof.* The squared goal cost satisfies $|c(h_\eta(u))-c(h_\eta(v))|\leq4H_w[d(u,v)+\varepsilon]$. Then, let $\mu_t,\nu_t$ be the state laws at time $t$ starting from $z,z'$. The kernel condition then implies that $W_d(\mu_t,\nu_t)\leq L^t d(z,z')$. Integrating the cost inequality yields $$|\mathbb E_{\mu_t}c(h_\eta)-\mathbb E_{\nu_t}c(h_\eta)| \leq4H_w[L^t d(z,z')+\varepsilon],$$ as we do in the paper. Then, summing with weights $\gamma^t$ proves (1). This proof keeps the additive approximation error. **3. Relation to terminal-cost CEM planning.** The connection to our planner can be made explicit through the terminal objective. Suppose that $$W_d(P(\cdot\mid z,a),P(\cdot\mid z',a)) \leq Ld(z,z'),$$ holds uniformly over actions. For any common action sequence $\boldsymbol a$ of length $T$, propagating this inequality and applying the Lipschitz bound for the squared goal cost yields $$|J_z(\boldsymbol a)-J_{z'}(\boldsymbol a)| \leq 4H_w\big[L^Td(z,z')+\varepsilon\big] =:b_T,$$ where $J_z(\boldsymbol a):=\mathbb E[c(h_\eta(Z_T))\mid Z_0=z,\boldsymbol a]$. Hence, nearby initial states have similar terminal costs under the same candidate actions. Because the condition is uniform over actions, this bound holds across the planner's candidate sequences. We will include this in our revision also. **4. Absolute PointMaze and PushT performance.** The revised paper reports absolute success for every condition in Table 1 (PointMaze), Table 2 (matched-width PushT), and Appendix A.12 (PushT training-data ablation). The compact model has lower native success on PushT, particularly with the full dataset. With $1{,}000$ samples, increasing DINO-Bisim's width from $32$ to $384$ raises native success from $36$\% to $40$\%. At matched width, DINO-Bisim improves absolute success under every shift, with native success of $40\%$ versus $48$\% for DINO-WM. See the new PushT table in our general response. The additional Wall results (see our general response) further show higher absolute success on all five selected patterned backgrounds, with comparable native success ($90$\% versus $92$\%). **5. Ablations.** The ablation table in our general response compares the compact projector with VICReg, PCA-based VICReg, and the addition of bisimulation. Adding the transition-distance term to VICReg increases checker-background matching accuracy from $15.6$\% to $60.2$\%.
AuthorsOpenReview ↗

Point-to-point response to Reviewer 2Xdi (2/2)

6. PCA basis and training schedule.

We clarify that the implementation estimates the PCA basis once, when PCA regularization is activated, and keeps it fixed thereafter. Centered embeddings from subsequent batches are projected onto this fixed basis to compute the variance loss. After activation, the PCA penalty matches explicit variance targets. It does not merely omit a variance floor on PC1. The covariance penalty is computed separately in the original per-patch latent coordinates, not in a basis that diagonalizes the current covariance. Therefore, the Reviewer's zero off-diagonal argument does not apply to our implemented regularization. We will make it more clearer in Section 3.3 and Appendix A.3.

7. Uncertainty and repeated training.

Following the evaluation protocol of DINO-WM, each condition is evaluated on 5050 independently sampled initial-goal pairs. Our main robustness results characterize substantial changes in success rate: on PointMaze under LCG (large color gradient change), DINO-Bisim achieves 7878% success compared with 4848% for DINO-WM, while their native success rates are comparable (7878% versus 8080%).

Dear Reviewer,

We would like to follow up on our responses to your concerns and questions and kindly ask you to adjust your score if your concerns/question have been addressed. Thank you!

Dear Reviewer,

We thank you for your time reviewing our work. We are happy to discuss any additional question you may have. If our responses have addressed your questions and concerns, we would appreciate if the score can be adjusted accordingly. Thank you!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to the Reviewers (1/2)

We thank the Reviewers for their constructive comments and feedback. The reviews highlight the importance of adapting pretrained visual features for planning under visual appearance changes, the simplicity of our patch-wise bisimulation encoder, and the strong results for navigation. Next, let us first address the concerns shared across reviews, followed by a response to each individual reviewers.

What the experiments establish. Our contribution is a compact adaptation of frozen visual features, trained jointly with an action-conditioned world model without reward supervision or additional environment interaction. On PointMaze, our DINO-Bisim model achieves 7878% success rate, compared with 8080% for DINO-WM on the default background (native), and preserves 78%78\% under LCG (large color gradient change), where DINO-WM falls to 4848% and domain randomization achieves 6464%. The encoder preserves all 196196 spatial patches and reduces each patch from 384384 to 3232 channels. We then provide a 12×12\times reduction in latent space which leads to a much more compact latent world model.

On PushT, the 3232-channel DINO-Bisim model has lower mean relative degradation across all three dataset sizes. Our revised version (please see our revised version) reports the absolute rates in the PushT training-data ablation. With 1,0001{,}000 samples and 384384 channels per patch for both models, DINO-Bisim improves absolute success under all five shifts, from 25.225.2% to 42.042.0% on average, see the table below.

Model (channels per patch)NCSCCLCLCGDAvg.
DINO-WM (384)0.480.380.380.140.240.120.252
DINO-Bisim (384)0.400.420.440.400.420.420.420

Existing ablations.

Our submitted version already includes the following ablations and diagnostics (please refer to our supplementary materials):

  • Visual encoder: DINOv2, SimDINOv2, iBOT, and end-to-end training (Table 1).
  • Training data: Three PushT dataset sizes (PushT training-data ablation in the revised appendix).
  • Planning horizon: Seven horizons ranging from 55 to 100100 (Appendix A.10).
  • Representation learning: Latent space visualization before and after the switch to PCA-based regularization (Appendix A.3).
  • Reward-distance diagnostics: Training metrics examining the reward-distance signal alongside the transition and regularization losses (Appendix A.8).
  • Planning rollouts: Additional trajectories under visual shifts for PointMaze and PushT (Appendices A.4 and A.5).

Additional environment. We also evaluate our model in the Wall environment, following the DINO-WM evaluation protocol with 5050 initial-goal pairs per condition. The table below reports absolute success rates for five patterned backgrounds alongside No Change. The revised paper includes the full background figure and results in the Wall subsection. DINO-Bisim achieves higher success on all five shifts, with mean success increasing from 16.4%16.4\% to 47.2%47.2\%, while native success remains comparable (90%90\% versus 92%92\%). Results for all 1818 visual shifts are provided in the table below.

ModelNo ChangeNewspaperStripesHoneycombGridCrownAvg.
DINO-WM0.920.000.100.280.140.300.16
DINO-Bisim0.900.520.520.520.400.400.47
BackgroundDINO-WMDINO-Bisim
No Change.92.90
Slight change.92.94
Large colour.90.92
Large colour gradient.90.88
Distractors.42.42
Newspaper.00.52
Stripes.10.52
Honeycomb.28.52
Camo.02.22
Heavy distractors.10.30
Crown.30.40
Grid.14.40
Rings.02.18
Stripes + distractors.02.18
Crown + distractors.08.16
Noise.10.16
Polka dots.48.52
Bricks.62.60
Noise + distractors.10.08

We would also like to emphasize that these Wall results were obtained without environment-specific hyperparameter tuning, leaving room for further improvement.

AuthorsOpenReview ↗

General comment to the Reviewers (2/2)

New ablations. We compare 196×32196\times32 visual representations on Wall using the same architecture, training data, optimizer settings, and training seeds. A projector maps each patch from 384384 to 3232 features. Bisim adds the transition-distance loss.

  • EpE_p: squared embedding-prediction error divided by the squared distances of the encoded next images from their batch mean.
  • EsE_s: squared position-prediction error from a linear model fitted on training visual embeddings.
  • Background matching (%): we replace each image's background with a solid color or checker pattern, keeping the agent and walls unchanged. We then compare its embedding with those of 128128 original images. A match is correct if the smallest Euclidean distance selects the original image with the same agent position and wall layout. We report the percentage of correct matches in the table below.
ModelEp↓E_p\downarrowEs↓E_s\downarrowSolid ↑\uparrowChecker ↑\uparrow
Projector + VICReg1.8810.11922.77.8
Projector + PCA-based VICReg4.6280.10350.015.6
Projector + Bisim + PCA-based VICReg2.7500.10461.760.2

Adding Bisim to PCA-based VICReg increases checker matching from 15.615.6% to 60.260.2%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.