acceptodds
Under review as a conference paper at ICLR 2027

Why Can Latent Dynamics Models Adapt to Visual Shifts from a Few Examples?

Abstract

Latent dynamics models built on pretrained foundation representations often adapt to dynamics-preserving visual shifts from only a few target-domain episodes. We explain this sample efficiency through the structure of the required functional correction. Much of this correction is globally shared across inputs and both statistically and optimization-wise accessible through low cross-episode latent variability and low-dimensional parameter updates. The state-dependent remainder is learned more gradually but concentrates in a substantially lower-dimensional subspace than under source-specific representations. Transport analysis link these structures to the interaction between representation geometry and pretrained dynamics. These findings show that foundation representations facilitate adaptation not merely through transferable features, but by structuring the functional change required to reuse source-trained dynamics across domains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent dynamics models, world models, few-shot adaptation, foundation representations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LZFWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why latent dynamics models (world models) built on frozen, pretrained foundation representations (such as DINOv2) can adapt to dynamics-preserving visual shifts using only a few target-domain episodes. Rather than proposing a new adaptation algorithm, the authors provide a mathematical and empirical framework characterizing the required functional correction between the source-trained predictor and the true target dynamics.

Critical strengths
  1. Good mathematical framework which resolves counter-intuitive empirical observations. Specifically, the orthogonal decomposition of fine-tuning progress and the cross-episode sample complexity bound are mathematically elegant, cleanly separating initial zero-shot domain mismatch from the sample complexity required for adaptation. This framework provides a principled theoretical basis for two notable phenomena: a. Fast adaptation despite poor initialization: Why a foundation-based world model can suffer higher initial zero-shot prediction MSE than a baseline, yet adapt dramatically faster within a few episodes. b.The zero-shot vs. few-shot gap: Why strong zero-shot transfer does not inherently guarantee rapid few-shot adaptation.

  2. Affine transport fits across foundation representations The transport analysis in Table 3 comparing DINOv2, Web-SSL (MAE), and SigLIP2 provides concrete geometric intuition for why self-supervised visual features are often favoured over vision-language features (like SigLIP2) for fine-grained world modelling and control. This is a nice point.

Critical weaknesses

The two points are connected to each other and are my main concerns. If the authors will provide some information about what to do in 3D scenario it will strenghen the paper

  1. Limited experimental scope and synthetic 2D shifts: All empirical evaluations are restricted to two 2D planar environments under uniform synthetic rendering shifts (eg zoom-out, Background color). In 2D top-down views, uniform background color changes or camera zoom-outs naturally act globally across spatial tokens. It remains unclear whether η\eta remains large or the cross-domain transport remains approximately affine under realistic environment such as 3D camera viewpoint rotation.

  2. Not relevant for 3D In Section 7 and the Appendix, the authors restrict the affine transport fit to the bilinear form PZR+B, where P acts across spatial patch tokens and R acts across feature channels. While this is a computationally sensible constraint, it enforces a Kronecker-separable structure between spatial location and feature channels. Synthetic 2D zoom-out and uniform colour shifts naturally satisfy this factorization. Under 3D perspective shifts, spatial token mixing depends on the underlying scene geometry and feature content, which would likely violate the PZR+B parameterization.

Questions for authors

The two weaknesses are connected somehow and are my main concern with this paper. Can the authors explain or give some intuition on the 3D scenatio?

Also, maybe I missed it but I am not sure that I understood why in Figure 7, the zoom-out and stripe shifts achieve different task success rates at 5 and 10 episodes despite having nearly identical functional recovery curves. Can you please elaborate this point?

1 comment

We thank the reviewer for their time and constructive feedback. We provide our responses below.

[W1] All empirical evaluations are restricted to two 2D planar environments under uniform synthetic rendering shifts (eg zoom-out, Background color). It remains unclear whether η\eta remains large or the cross-domain transport remains approximately affine under realistic environment such as 3D camera viewpoint rotation.

Thank you for raising this point. We agree that zoom-out and background-color shifts are relatively simple, synthetic rendering changes. However, simple changes in observation space need not induce simple latent transformations through a deep pretrained foundation encoder.

We also consider the block-color shift, whose visual effects follow the moving block and therefore depend on physical state. In this case, η\eta is lower and global recovery requires more target episodes than under zoom-out or background-color shifts. Our analysis thus captures both reduced global dominance and slower recovery. Nevertheless, the cross-domain transport retains measurable structure: Table 3 shows that the adopted affine approximation closes 92.6% of the transport gap under Web-SSL and 71.3% under DINO. This demonstrates that state-dependent visual changes need not eliminate regularity in latent transport.

Realistic 3D viewpoint shifts may introduce more complex, geometry-dependent changes, potentially reducing global dominance and increasing the importance of state-dependent adaptation. Importantly, our framework does not assume either a large η\eta or approximately affine transport. Although the block-color experiment does not establish generalization to 3D, it demonstrates how our analysis characterizes differences in adaptation difficulty beyond globally uniform visual shifts.

[W2] The authors restrict the affine transport fit to the bilinear form PZR+B. While this is a computationally sensible constraint, it enforces a Kronecker-separable structure between spatial location and feature channels. Under 3D perspective shifts, spatial token mixing depends on the underlying scene geometry and feature content, which would likely violate the PZR+B parameterization.

We agree that the bilinear parameterization imposes a restrictive separability assumption between spatial location and feature channels. However, its use does not assume that transport must be bilinear or affine: a successful fit under such a restricted family provides evidence of structured latent correspondence, whereas a poor fit does not necessarily imply unstructured transport. As mentioned above, Table 3 shows that even under this restrictive parameterization, Web-SSL captures 92.6% of the transport gap for the more state-dependent block-color shift, where visual changes follow the moving block instead of occurring uniformly across fixed spatial locations or feature channels. This suggests that even state-dependent visual shifts can induce latent correspondences with substantial structure under foundation representations.

[Q] Why in Figure 7, the zoom-out and stripe shifts achieve different task success rates at 5 and 10 episodes despite having nearly identical functional recovery curves?

Thank you for the question. Figure 7 highlights the distinction between functional recovery and downstream control performance. Our metrics measure aggregate, normalized prediction-error reduction over held-out target data. Similar recovery therefore need not imply equal residual error magnitudes or distributions across states and actions. Closed-loop control is sensitive to these differences, particularly in control-critical regions where small prediction errors can alter planning decisions.

In Wall considered in Fig. 7, where scene geometry is largely static, both shifts exhibit rapid global recovery but the remaining state-dependent errors affect control-critical predictions differently. This motivates identifying the residual correction directions most consequential for control. Our results provide a promising starting point by showing that the state-dependent correction remains concentrated in a low-dimensional subspace, despite its potential importance for control performance. We will clarify the distinction and its implications in the revision.

Reviewer_nEgtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors observe that latent dynamics models using frozen pretrained visual representations adapt to dynamics-preserving visual shifts much more efficiently than models relying on task-specific, self-learned representations. The paper aims to investigate the underlying mechanisms that explain this difference in adaptation efficiency.

The authors define the functional correction required for a model to accurately predict dynamics in a new visual domain and decompose it into two components: a globally shared mean correction and a state-dependent residual correction. They then analyze the structure and properties of these two components through theoretical and empirical investigations.

Under DINO representations, the global correction accounts for a large fraction of the total functional correction. The authors show that greater consistency in episode-level mean corrections enables more accurate estimation of the global correction from only a few episodes. They further demonstrate that the global correction can be realized through parameter updates restricted to a random low-dimensional subspace, while the resulting functional changes are concentrated in a relatively small number of modes. In addition, the state-dependent residual correction exhibits substantially lower effective dimensionality under pretrained representations.

The authors validate these findings across different visual shifts in the PushT and Wall tasks. Furthermore, by analyzing cross-domain transport under different pretrained representations and deriving theoretical relationships under simplified linear dynamics and affine transport assumptions, they suggest that the structure of the required functional correction may arise from the interaction between representation geometry and pretrained dynamics.

Critical strengths

The paper proposes a systematic framework for analyzing the functional corrections required for target-domain adaptation. By decomposing the correction into global and state-dependent components and examining its statistical estimation and parameter-space realization, the framework provides a clear and quantifiable perspective on how pretrained representations facilitate few-shot adaptation.

Critical weaknesses

Critical Weakness 1: Lack of Cross-Representation Comparisons
Figure 6 demonstrates that, under DINO representations on the PushT zoom-out shift, the global correction can be effectively realized through restricted parameter-space optimization, while the resulting functional updates exhibit strong spectral concentration. However, the paper does not provide corresponding restricted fine-tuning and gamma analyses for self-learned representations. Consequently, it remains unclear whether this low-complexity realization is a distinctive advantage of foundation representations. I suggest that the authors include cross-representation comparisons under comparable experimental settings.

Critical Weakness 2: Insufficient Empirical Validation of the Transport–Dynamics Mechanism
Section 7 derives that, under the assumptions of linear dynamics and affine transport, the structure of the state-dependent correction is constrained by the commutator (MA-AM). However, the corresponding empirical analysis primarily evaluates how well cross-domain transport can be approximated by an affine transformation, without directly examining the relationship between the transport–dynamics mismatch and the actual functional correction. Furthermore, Tables 2 and 3 evaluate different sets of representations, and thus do not directly establish a cross-representation relationship between transport regularity and correction structure. Additional comparisons would strengthen the empirical support for the proposed mechanistic explanation.

Questions for authors

Q1. Cross-representation comparison of low-complexity realization
Figure 6 does not include restricted fine-tuning experiments for self-learned representations, making it difficult to determine whether the observed low-complexity realization is a distinctive advantage of foundation representations. Have the authors conducted corresponding experiments using self-learned representations? If so, do the results support the same conclusion?

Q2. Empirical validation of the Transport–Dynamics explanation
(1) Have the authors directly compared the functional correction predicted by the Transport–Dynamics mismatch in Equation 4 with the actual correction required in the target domain, using the nonlinear dynamics models studied in the paper? (2) Tables 2 and 3 evaluate different sets of representations. Have the authors examined the relationship between transport regularity and correction structure under matched representation and visual shift settings?

4 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for their time and constructive feedback. We provide our responses below.

[W1] The paper does not provide corresponding restricted fine-tuning and gamma analyses for self-learned representations. Consequently, it remains unclear whether this low-complexity realization is a distinctive advantage of foundation representations

A: Thank you for this suggestion. We report below the functional recovery decomposition for the frozen self-learned representations in Fig. 2, using the same held-out evaluation procedure as for DINO. Their global correction fractions are substantially smaller than DINO’s: η=0.0087\eta=0.0087 for zoom-out and 0.1940.194 for background color, so the required correction is predominantly state-dependent. For zoom-out, negative overall progress indicates that few-shot fine-tuning increases held-out target prediction error relative to the source model. Normalization by the very small global correction energy amplifies the magnitude of negative global recovery values. For background color, global recovery reaches g50=0.97g _{50}=0.97, but state-dependent recovery remains limited at α50=0.33\alpha _{50}=0.33, yielding P50=0.46P _{50}=0.46.

Functional recovery with frozen self-learned representations on PushT.

ShiftMetricN=5N=10N=20N=50
Zoom-out (η=0.0087)gNg _N-32.5-64.5-18.6-3.8
αN\alpha _N-0.37-0.70-1.46-0.03
PNP _N-0.65-1.26-1.60-0.06
Background (η=0.194)gNg _N-0.41-0.730.700.97
αN\alpha _N-0.22-0.460.050.33
PNP _N-0.25-0.510.180.46

These results explain why the restricted global fitting and functional spectrum analyses in Section 5.2 focus only on foundation representations. For DINO, the global correction accounts for most adaptation energy, making its low-complexity realization consequential for few-shot recovery. For self-learned representations, restricted global fitting would not capture the dominant state-dependent adaptation difficulty. Our central finding is the combination of a substantial global correction, statistical accessibility, and efficient realization in foundation representations, while we do not claim that low-dimensional global realization is exclusive to them.

AuthorsOpenReview ↗

Response 2

[W2] The corresponding empirical analysis primarily evaluates how well cross-domain transport can be approximated by an affine transformation, without directly examining the relationship between the transport–dynamics mismatch and the actual functional correction

A: Thank you for highlighting this gap. We agree that the empirical analysis in Section 7 evaluates transport regularity but does not quantify how well the transport–dynamics mismatch explains the required correction. We therefore compare the estimated mismatch with the ground-truth correction. For paired source and target latent inputs CIC^\mathrm{I} and COC^\mathrm{O}, the estimated affine transport κ^ϕ\hat{\kappa} _\phi gives the exact decomposition

\Delta^{\phi} _{\ast}(C^{\mathrm{O}},a) &= F^{\mathrm{O}} _{\ast}(C^{\mathrm{O}},a) - F _0(C^{\mathrm{O}},a) \\\\ &= \underbrace{\hat{\kappa} _{\phi}(F _0(C^{\mathrm{I}},a)) - F _0(C^{\mathrm{O}},a)} _{\text{Approximate transport-dynamics mismatch},\\;\hat{\Delta} _{\mathrm{trans}}} \\\\ &\quad + \underbrace{\hat{\kappa} _{\phi}(F^{\mathrm{I}} _{\ast}(C^{\mathrm{I}},a)) - \hat{\kappa} _{\phi}(F _0(C^{\mathrm{I}},a))} _{\text{Approximate transported source-model error},\\;\hat{\Delta} _{\mathrm{src}}} \\\\ &\quad + \underbrace{F^{\mathrm{O}} _{\ast}(C^{\mathrm{O}},a) - \hat{\kappa} _{\phi}(F^{\mathrm{I}} _{\ast}(C^{\mathrm{I}},a))} _{\text{Transport-fitting error}}. \end{aligned}$$ Let $\mu _X = \mathbb{E}[X]$ denote the global component of a correction term $X$, and let $\mu _{\ast} = \mathbb{E}[\Delta _{\ast}^{\phi}]$. We report three quantities: $$R _g = 1 - \frac{\\|\mu _{\hat{\Delta} _{\mathrm{trans}}}-\mu _{\ast}\\| _2^2}{\\|\mu _{\ast}\\| _2^2}, \quad R _{\mathrm{total}} = 1 - \frac{\mathbb{E}\\|\hat{\Delta} _{\mathrm{trans}}-\Delta _{\ast}^{\phi}\\| _2^2}{\mathbb{E}\\|\Delta _{\ast}^{\phi}\\| _2^2}, \quad \rho _{\mathrm{src}}^g = \frac{\\|\mu _{\hat{\Delta} _{\mathrm{src}}}\\| _2^2}{\\|\mu _{\ast}\\| _2^2}$$ Here, $R _g$ measures how well the estimated transport-dynamics mismatch recovers the global correction, $R _{\mathrm{total}}$ measures how much of the total correction energy is explained by the estimated transport-dynamics mismatch, and $\rho _\mathrm{src}^g$ quantifies the relative magnitude of the global source-model error. **Transport–dynamics mismatch analysis for DINO on PushT.** | Metric | Zoom-out | Background color | |:---|:---:|:---:| | Affine transport gap closed | 91.9% | 88.6% | | Global recovery (`R_g`) | **0.934** | **0.944** | | Overall correction recovery (`R_total`) | 0.727 | 0.780 | | Global source-error energy ratio (`rho_src^g`) | 0.060 | 0.060 | The estimated transport–dynamics mismatch closely reproduces the dominant global correction, achieving global recovery scores of **0.934 and 0.944** for zoom-out and background color, respectively. It also explains a substantial fraction of the total correction energy, with recovery scores of **0.727 and 0.780**. The global component of the transported source-model error has a squared norm equivalent to only approximately **6%** of that of the global correction. Although the state-dependent remainder is less accurately reconstructed, consistent with the limited expressivity of the affine transport approximation, the estimated mismatch still captures most of the required correction under both shifts. These results provide direct empirical support for our interpretation: **the interaction between structured cross-domain transport and pretrained dynamics largely determines the dominant global correction and accounts for much of the overall adaptation requirement.** We will include this new analysis in the revised manuscript.
AuthorsOpenReview ↗

Response 3

[Q1] Figure 6 does not include restricted fine-tuning experiments for self-learned representations, making it difficult to determine whether the observed low-complexity realization is a distinctive advantage of foundation representations. Have the authors conducted corresponding experiments using self-learned representations? If so, do the results support the same conclusion

We have not conducted restricted global-fitting experiments for frozen self-learned representations because their global component accounts for only 0.870.87\\% of the required correction energy under zoom-out and 19.419.4\\% under background color, much lower than for foundation representations. The additional recovery decomposition in W1 likewise shows that overall adaptation depends primarily on state-dependent recovery. Thus, restricted global fitting would not characterize their dominant adaptation difficulty. Our analysis focuses on foundation representations, where the global component accounts for most correction energy, making its low-dimensional realization consequential for few-shot recovery.

[Q2] Have the authors directly compared the functional correction predicted by the Transport–Dynamics mismatch in Equation 4 with the actual correction required in the target domain, using the nonlinear dynamics models studied in the paper?

We added additional analysis in response to W2 showing that the transport–dynamics mismatch, computed using the fitted affine transport, closely reproduces the actual global correction, which dominates the required adaptation energy.

[Q3] Tables 2 and 3 evaluate different sets of representations. Have the authors examined the relationship between transport regularity and correction structure under matched representation and visual shift settings?

We clarify that Tables 2 and 3 examine complementary aspects: Table 2 characterizes correction structure, while Table 3 evaluates transport regularity across foundation encoders. They are not intended to establish a direct correlation between transport accuracy and correction complexity. To bridge this gap, as mentioned earlier, we added the matched analysis in W2, jointly evaluating transport and correction under the same representation and visual shifts. We will clarify this distinction and the scope of these comparisons in the revision.

Thank you for the detailed responses and additional analyses.

The new results on functional recovery for self-learned representations help clarify the distinction between foundation and self-learned representations. I also appreciate the clarification that low-dimensional global realization is not claimed to be exclusive to foundation representations.In particular, the additional analysis directly comparing the transport–dynamics mismatch with the actual functional correction provides convincing empirical support for the proposed interpretation.

Overall, I find that the additional evidence and clarifications have substantially addressed my original concerns. I have increased my score to weak accept.

Reviewer_CLj4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper seeks to explain why latent dynamics models built on foundation representations can adapt to dynamics-preserving visual shifts from a small number of target episodes. It decomposes the required target-domain functional correction into a global mean component and a state-dependent residual, defines the global energy fraction and cross-episode variability, studies adaptation in random parameter subspaces and through a path-averaged Jacobian spectrum, and relates the resulting correction to cross-domain latent transport. Experiments on PushT and Wall compare DINOv2 features with source-specific learned representations under synthetic visual shifts.

Critical strengths
  1. The paper provides a coherent diagnostic language for analyzing adaptation. The decomposition in Equations 7–9 distinguishes globally shared recovery from state-dependent recovery, while the episode-mean dispersion and functional spectrum offer complementary views of statistical and realization difficulty. Table 1 and Figures 2–4 provide locally convincing evidence that these quantities differ between DINO and a source-specific encoder on the studied PushT shifts.

  2. The paper usefully connects representation geometry and dynamics. The affine example in Equations 17–19 illustrates how the commutator between latent transport and dynamics can determine whether a correction is global or state-dependent. This is a potentially fruitful perspective for future representation design.

Critical weaknesses
  1. The paper does not establish that its oracle post-hoc quantities explain or predict the behavior of the actual few-shot learner. Appendix A (p. 12) states that eta, C_global, and the component-recovery terms are computed from ground-truth next observations over the target evaluation distribution. Such oracle quantities can be useful explanatory diagnostics, but the paper does not connect them to what can be estimated from the learner's few-shot data or to the optimization dynamics of fine-tuning. Lemma 1 is the exact orthogonal decomposition induced by defining a mean-zero residual, and Lemma 2 is the standard variance identity for an i.i.d. sample mean; neither result shows that SGD recovers the global component first. The block-color discussion selects a 1% estimation threshold and observes recovery around 50 episodes on the same shift, but does not prospectively test whether the criterion predicts adaptation on held-out shifts. Moreover, Appendix A constructs target data by re-rendering the same expert trajectories used in the source domain, so the experiments hold state-action coverage fixed and do not test few-shot adaptation under new target-domain coverage.

  2. The transport explanation omits an unmeasured source-model-error term and uses representation-dependent quantities as if they were intrinsic. Equation 4 (p. 4) explicitly neglects transported source-model error, but the paper provides no bound or direct measurement showing that this term is small relative to the reported global and residual corrections. The argument also assumes a deterministic transport kappa exists, which can fail whenever two physical states collide in the source latent but not in the target latent. Finally, eta and 95%-energy rank change under an invertible anisotropic rescaling of latent coordinates. Table 2 reports only normalized rank across representations with different ambient dimensions, while Table 3 compares raw affine MSE across encoders of different dimension and unspecified scale. These measurements therefore do not establish an intrinsic reduction in adaptation complexity across representations.

Questions for authors
  1. Can the authors provide the exact correction identity including transported source-model error and quantify the omitted term relative to the reported global and residual components? Under what conditions does a deterministic transport kappa exist when the source encoder is non-injective?

  2. Can the proposed quantities prospectively predict adaptation difficulty on held-out tasks and visual shifts, using only the target data available to the learner? In particular, please evaluate on target trajectories with state-action coverage not used for source training and characterize sensitivity to the random projection construction used in the intrinsic-difficulty analysis.

  3. How are latent scale and ambient dimension controlled when comparing eta, energy rank, and affine MSE across self-learned, DINO, Web-SSL, and SigLIP representations? Please report whitened or otherwise coordinate-calibrated comparisons and absolute ranks.

5 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for their time and constructive feedback. We provide our responses below.

[W1.1] The proposed quantities require ground-truth target observations over the target evaluation distribution, and their connection to quantities estimable from few-shot data is unclear

A: We agree that the reported diagnostics use a target-domain dataset larger than the adaptation datasets. This is intentional: our goal is to understand few-shot adaptability from an explanatory perspective, using sufficient evaluation data to enable a reliable assessment of the adapted models.

To examine their estimability from limited data, we recompute η\eta and CglobalC _{\mathrm{global}} using only the target episodes available for adaptation. As shown below, even estimates from N=5N=5 episodes reasonably approximate the full-data values and preserve the qualitative differences across shifts, while increasing NN generally improves estimation accuracy.

ShiftMetricN=5N=10N=20N=35N=50Paper
Zoom-outη^N\hat{\eta} _N.731±.019.716±.008.701±.010.702±.011.698±.007.699
C^global,N\hat{C} _{\mathrm{global},N}.131±.024.134±.010.142±.006.143±.012.146±.008.144
Backgroundη^N\hat{\eta} _N.782±.009.769±.017.760±.012.761±.011.759±.006.756
C^global,N\hat{C} _{\mathrm{global},N}.120±.027.128±.019.134±.006.133±.012.133±.006.131
Block-redη^N\hat{\eta} _N.451±.030.442±.015.428±.008.434±.017.429±.012.428
C^global,N\hat{C} _{\mathrm{global},N}.467±.135.469±.085.468±.038.462±.039.468±.024.488

These results suggest that the proposed diagnostics can inform relative correction structure and adaptation difficulty using only a few target episodes. Comparing a new domain’s estimates with those of previously characterized shifts may thus offer preliminary guidance on data requirements. Nevertheless, our primary objective remains explanatory, and we will clarify the distinction between this objective and potential predictive applications in the revision.

[W1.2] Lemmas 1 and 2 are standard statistical identities, but neither shows that SGD recovers the global component first during adaptation

A: We clarify that Lemmas 1 and 2 characterize adaptation structure and statistical accessibility, respectively, linking representation geometry to adaptation requirements. We do not aim to establish SGD convergence. Proving that SGD recovers the global component first would require controlling relative recovery rates of the global and state-dependent components, while accounting for their coupling through shared parameters, the evolving parameter Jacobian, and stochastic gradient noise. Instead, our study provides a complementary explanation for the observed data-efficient adaptation through correction dominance identified using Lemma 1, statistical accessibility characterized by Lemma 2, and low-dimensional realization demonstrated in Section 5.2.

[W1.3] Lack of evaluation of target-domain adaptation under new state-action coverage

A: Thank you for this point. We repeated the zoom-out experiment in Fig. 2 using new expert demonstrations for target-domain adaptation, instead of rerendered source-training trajectories. These demonstrations were generated with the source-trained model F0F_0 over in-domain visual observations, retaining only successful trajectories. The progress terms below closely match the original values, preserving the pattern of rapid global recovery followed by more gradual state-dependent recovery. We will add this results in the revision.

MetricNotationPaper N=5N=10N=20N=50New N=5N=10N=20N=50
Global recoverygNg_N.981.990.994.997.981.989.994.997
State-dependent recoveryαN\alpha_N.432.532.569.600.441.559.592.624
Overall progressPNP_N.816.852.867.878.826.866.879.890
AuthorsOpenReview ↗

Response 2

[W2.1] Equation 4 explicitly neglects transported source-model error, but the paper provides no bound or direct measurement showing that this term is small relative to the reported global and residual corrections

A: We clarify that Eq. 4 omits the transported source-model error solely to isolate the transport-dynamics mismatch for mechanistic interpretation, and our correction analysis retains it. Under the exact condition, the full correction is

\Delta^{\phi} _{\ast}(\kappa _{\phi}(C),a) &= F^{\mathrm{O}} _{\ast}(\kappa _{\phi}(C),a) - F _{0}(\kappa _{\phi}(C),a) \\\\ &= \kappa _{\phi}(F^{\mathrm{I}} _{\ast}(C,a)) - F _{0}(\kappa _{\phi}(C),a) \\\\ &= \underbrace{\kappa _{\phi}(F _{0}(C,a)) - F _{0}(\kappa _{\phi}(C),a)} _{\text{Transport-dynamics mismatch (Eq. 4)}} \\\\ &\quad + \underbrace{\kappa _{\phi}(F^{\mathrm{I}} _{\ast}(C,a)) - \kappa _{\phi}(F _{0}(C,a))} _{\text{Transported source-model error}}. \end{aligned}$$ The second term represents the source-model prediction error $F^\mathrm{I} _{\ast}-F _0$ after cross-domain transport. Its omission is motivated by the predictive accuracy of $F _0$, which is directly optimized during source-domain training. Consistent with this, the source-domain prediction errors reported below are low and comparable between training and held-out data. | Task | Source-domain prediction error | | |:---|:---:|:---:| | | **Train** | **Held-out** | | PushT | 0.040 | 0.041 | | Wall | 0.015 | 0.018 | Importantly, all reported analyses directly use the target correction $F _{\ast}^{\mathrm O}-F _0$, computed from ground-truth observations, and the learned correction $F _N-F _0$. **The transported error thus remains included regardless of its magnitude.** **[W2.2] The argument assumes a deterministic transport kappa exists, which can fail whenever two physical states collide in the source latent but not in the target latent** **A:** **A deterministic transport does not require the source representation to be injective.** Such a transport exists on the considered state support as long as any two physical states sharing the same source latent also share the same target latent, i.e., $\phi(R _I(s _1))=\phi(R _I(s _2)) \implies \phi(R _O(s _1))=\phi(R _O(s _2))$. Thus, non-injectivity alone does not invalidate deterministic transport. It fails only when states indistinguishable in the source latent remain distinguishable in the target latent. Moreover, as clarified above, **our correction decompositions are independent of the existence of such transport.** We do not claim that foundation representations are immune to latent collapses. Rather, the transport formulation provides a conditional mechanistic explanation of how representation geometry can structure the required correction. This interpretation is further supported empirically: Section 7 demonstrates that transports can be well approximated by affine mappings under the considered shifts, providing evidence of structured latent transport in our study. **[W2.3] Eta and 95%-energy rank change under an invertible anisotropic rescaling of latent coordinates** **A:** We agree that $\eta$ and $r _{95}^{\delta}$ (Section 6) are not invariant under arbitrary coordinate rescaling. However, our analysis specifically concerns the foundation encoders’ native representations, so these quantities characterize correction structure in the geometry used directly for dynamics training and evaluation. For the realized functional energy rank $r^\gamma$ (Section 5.2), we additionally assess sensitivity to independent random projections. **Consistent estimates suggest that the observed spectral concentration is robust to projection choice.** We will include this sensitivity analysis in the revision. | N | $r _{0.5}$ | $r _{0.6}$ | $r _{0.7}$ | $r _{0.8}$ | $r _{0.9}$ | $r _{0.95}$ | |:---|:---:|:---:|:---:|:---:|:---:|:---:| | N=5 (5 seeds) | 14.0 ± 1.7 | 27.8 ± 0.8 | 77.2 ± 4.1 | 175.0 ± 4.4 | 312.8 ± 4.0 | 420.2 ± 1.9 | | N=10 (5 seeds) | 11.2 ± 1.6 | 24.8 ± 2.8 | 78.8 ± 3.7 | 170.4 ± 4.4 | 302.8 ± 4.9 | 407.8 ± 5.9 |
AuthorsOpenReview ↗

Response 3

[W2.4] Table 2 reports only normalized rank across representations with different ambient dimensions, while Table 3 compares raw affine MSE across encoders of different dimension and unspecified scale.

In Table 2, we report the normalized r95δ/dzr _{95}^{\delta}/d _z to quantify the fraction of each latent space needed to capture 95% of the state-dependent correction energy. DINO has dz=384×256d _z=384\times256 dimensions, whereas the self-learned encoder has dz=192d _z=192. Absolute ranks alone overlook this disparity.

Meanwhile, we clarify that Table 3 already reports normalized affine transport MSE: the per-element reconstruction MSE divided by each representation’s mean squared latent magnitude, averaged over frames, spatial patches, and channels. This expresses reconstruction error relative to latent energy, accounting for dimensionality and overall feature scale. Table 3 also reports the relative gap closed by the affine approximation against each representation’s own transport reference. Both metrics therefore assess transport quality relative to each representation. We will make the MSE normalization explicit in the revised manuscript.

[Q1] Can the authors provide the exact correction identity including transported source-model error and quantify the omitted term relative to the reported global and residual components?

A: As detailed in W2.1, the corrections analyzed in our study already account for the transported source-model error omitted from Eq. 4. This omission is intended to simplify the mechanistic interpretation and highlight the role of transport-dynamics mismatch. The injectivity of the source representation depends on the chosen foundation encoder. We do not assume that foundation representations preserve all distinctions between physical states or are universally free of latent collapse. Nevertheless, Section 7 shows highly structured cross-domain correspondences across the considered encoders and shifts.

In addition, we have added a new analysis using the estimated transports to quantify the relative contribution of transport-dynamics mismatch to the total correction. We examine PushT zoom-out and background-color shifts under DINO. The background-color transport is newly fitted for this analysis, achieving 88.6% affine gap closure, compared with 91.9% for zoom-out.

Denote the estimated transport-dynamics mismatch by Δ^trans\hat{\Delta} _{\mathrm{trans}} and the source-model error by Δ^src\hat{\Delta} _{\mathrm{src}}. Let μX=E[X]\mu _X = \mathbb{E}[X] denote the global component of a correction term XX, and let μ∗=E[Δ∗ϕ]\mu _{\ast} = \mathbb{E}[\Delta _{\ast}^{\phi}]. We report three quantities:

R _{\mathrm{total}} = 1 - \frac{\mathbb{E}\\|\hat{\Delta} _{\mathrm{trans}}-\Delta _{\ast}^{\phi}\\| _2^2}{\mathbb{E}\\|\Delta _{\ast}^{\phi}\\| _2^2}, \quad \rho _{\mathrm{src}}^g = \frac{\\|\mu _{\hat{\Delta} _{\mathrm{src}}}\\| _2^2}{\\|\mu _{\ast}\\| _2^2}.$$ Here, $R _g$ measures how well the estimated transport-dynamics mismatch recovers the global correction, $R _{\mathrm{total}}$ measures how much of the total correction energy is explained by the estimated transport-dynamics mismatch, and $\rho _\mathrm{src}^g$ quantifies the relative magnitude of the global source-model error. **Transport–dynamics mismatch analysis for DINO on PushT.** | Metric | Zoom-out | Background color | |:---|:---:|:---:| | Affine transport gap closed | 91.9% | 88.6% | | Global recovery (`R_g`) | **0.934** | **0.944** | | Overall correction recovery (`R_total`) | 0.727 | 0.780 | | Global source-error energy ratio (`rho_src^g`) | 0.060 | 0.060 | The estimated transport–dynamics mismatch closely reproduces the dominant global correction, achieving global recovery scores of **0.934 and 0.944** for zoom-out and background color, respectively. It also explains a substantial fraction of the total correction energy, with recovery scores of **0.727 and 0.780**. The global component of the transported source-model error has a squared norm equivalent to only approximately **6%** of that of the global correction. Although the state-dependent remainder is less accurately reconstructed, consistent with the limited expressivity of the affine transport approximation, the estimated mismatch still captures most of the required correction under both shifts. These results provide direct empirical support for our interpretation: **the interaction between structured cross-domain transport and pretrained dynamics largely determines the dominant global correction and accounts for much of the overall adaptation requirement.** We will include this new analysis in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.