acceptodds
Under review as a conference paper at ICLR 2027

Identifiable World Models from Pretrained Diffusion Representations

Abstract

Diffusion-based world models can generate and predict trajectories in high-dimensional dynamical systems, but predictive accuracy does not imply that their latent coordinates recover the underlying state variables or causal interactions. We ask whether a frozen pretrained diffusion model can be equipped with identifiable coordinates without retraining its generative backbone. We show that auxiliary-variable nonlinear ICA guarantees can be transferred to Contrastive Diffusion Alignment (ConDA), which learns only a lightweight alignment map on top of frozen diffusion latents. Under standard TCL/GCL assumptions, the aligned representation identifies latent dynamical states up to permutation and componentwise invertible transformations, preserves the latent dynamic structural causal model, and reduces lagged graph recovery to transition-Jacobian sparsity. We evaluate TCL-, GCL-, and CEBRA-based ConDA against TDRL, CaRiNG, IDOL, temporal SuaVE, and iVAE across physical and robotic video systems. TCL and GCL achieve near-perfect blockwise state recovery and competitive lagged graph recovery, including exact recovery in a simulated falling-body system. In a simulated bipedal robot, learned dynamics recover the sign and temporal structure of responses to held-out control perturbations. These results show that a frozen generative diffusion model can be equipped with coordinates that are identifiable, structurally interpretable, and useful for analyzing intervention-relevant dynamics.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Causal Representation Learning, Diffusion World Models, Identifiability, Latent Dynamics, Causal Graph Recovery, Interventional Validation.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jqwbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce the strategy of compressing diffusion latents into a low-dimensional space using contrastive learning methods. They demonstrate that under the equivalence class of permutations and componentwise invertible scalar transformations, structural causal models are preserved, and show the possibility that their approach can recover that transition model. They validate their approach across three datasets, comparing the quality of recovery of their latent states and of the true environmental dynamics. They compare to several recent conditional representation learning methods, showing that their approach is comparable to and outperforms many of those methods.

Critical strengths

The main strength of this paper is the set of experimental results. It seems as if their approach is effective at learning a consistent set of latents, especially when using TCL/GCL.

Critical weaknesses

I have one major concern and one main weakness. The primary concern is with respect to section 3, where a portion of the work (in particular theorem 1 and lemma 1) are presented as work from the authors without proper attribution in the main text. Some of the proper attribution is done in the appendix for theorem 1, but should be done in the main text as well. Lemma 1 is also remarkably similar to prior results (e.g. noise-contrastive estimation Gutman & Hyv"arinen 2010, contrastive predictive coding van den Oord et al. 2019), without reference to those results in the lemma.

The weakness is with respect to the results. It is not clear what these results add to our understanding of diffusion models or contrastive learning, or what the benefits of this approach are over just running GCL/TCL/CEBRA on the data, rather than diffusion latents. These experiments should be done to demonstrate some kind of benefit of this approach over existing methods.

Questions for authors

For the diffusion latents, what diffusion timestep is used as input for contrastive learning? Does the timestep used impact the results?

1 comment
AuthorsOpenReview ↗

Response to Reviewer jqwb

We thank the reviewer for raising the attribution issue and asking for a direct pixel comparison. We should have credited the nonlinear ICA and density-ratio results more explicitly where we stated them. We have now corrected this in Section 3. We also ran the comparison you requested. On Physical Fall, direct pixel TCL and diffusion-latent TCL recovered the state equally well. The new Figure 6 shows why one might still want to learn coordinates in a pretrained generator. We can change a robot’s command, predict its motion in the learned coordinates, render that prediction, and compare it with what the robot actually does in the simulator.

W1. Attribution of Theorem 1 and Lemma 1

Revised Theorem 1 now credits Hyvärinen et al. (2019, Theorem 3) for identification of sufficient statistics up to invertible affine mixing and cites the related TCL and iVAE results. Revised Theorem 2 credits Hyvärinen et al. (2019, Theorem 1) for the stronger coordinatewise result and states the assumptions needed to apply it to a frozen representation. Lemma 1 now cites Gutmann and Hyvärinen for the density-ratio argument and van den Oord et al. for its use in contrastive predictive coding.

Our theoretical contribution concerns what follows for the dynamics. A linear mixing can preserve enough information to reconstruct the physical state while changing which variables appear to influence one another. Under coordinatewise identification, the lagged parent sets are preserved. New Proposition 1 addresses the practical question of how to detect those dependencies in a fitted transition model. With state-independent additive noise in the physical coordinates and the stated regularity conditions, the forward conditional-mean Jacobian preserves the zero pattern of the physical transition Jacobian. Under derivative faithfulness, that pattern identifies the lagged graph.

W2. What does diffusion add over direct TCL?

We trained the new direct TCL baseline on the original Physical Fall videos. Both rows use the same evaluation. Values are means ± SD across five seeds.

TCL inputCoordinate MCCLinear readout R²Graph F1AUPRC
Original pixels0.695 ± 0.0011.000 ± 0.0001.000 ± 0.0001.000 ± 0.000
Frozen diffusion latents0.694 ± 0.0010.998 ± 0.0001.000 ± 0.0001.000 ± 0.000

On Physical Fall, the 4,096-dimensional diffusion representation retained enough state information to match TCL trained directly on 256×256 RGB frames. The exact graph scores in both rows use supervised state alignment and known parent counts.

The reason to learn coordinates in a pretrained generator is that it already knows how to render the system to realistic video. ConDA keeps that capability and learns a small map into coordinates where we can model the dynamics. We can then ask what a change in command would do, predict its effect in those coordinates, and render the motion that follows, all without retraining the diffusion generator.

The new Figure 6 in Appendix A.12.1 shows this in Microduck. Starting from the same robot state, we change the forwrd velocity command from zero to -0.25 m/s and render the next second of predicted motion. We show it alongside MuJoCo rollouts under both commands, so the predicted change can be compared with the change the command actually caused. We also measured the velocity response separately over 100 matched rollouts. At one second, the learned dynamics predicted the correct direction and about a third of the magnitude of the backward response (-0.0042 versus -0.0129 m/s in MuJoCo).

Q1. Diffusion timestep and sensitivity

We use the final latent returned by the eight-step Rex-RK4 inversion. On Physical Fall, we varied the number of integration steps from 2 to 20. Eight steps gave the highest mean reconstruction PSNR, though two steps gave a similar result (39.68 versus 39.55 dB). This sweep tested reconstruction sensitivity to the solver step count. We did not compare latents extracted at different diffusion times, so their effect on state and graph recovery remains open.

Reviewer_s1WkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper applies TCL/GCL alignment to frozen diffusion representations and studies conditions for identifiable coordinates and preservation of lagged causal structure. Experiments measure state readout, graph recovery, and control-intervention responses in three simulated systems. Empirical graph recovery operates on representations aligned to known physical states and uses reference parent counts for binary graphs.

Critical strengths
  1. Evaluating state recovery, graph structure, and intervention responses goes beyond reconstruction quality. Five-seed comparisons and threshold-free AUPRC are useful, and the Microduck results expose response-magnitude errors as well as successful sign recovery.

  2. Keeping the generative representation frozen while learning an alignment map is a practical setting. The paper distinguishes generic contrastive objectives from the narrower TCL/GCL identification guarantees and acknowledges the supervised alignment used in evaluation.

Critical weaknesses
  1. Theorem 1 appears stronger than its stated assumptions justify. Exponential-family modulation with sufficient rank can leave linear mixing of sufficient statistics; coordinatewise invertible recovery needs additional conditions. Appendix A.1 only cites prior work. Theorem 2 assumes that the applicable result already provides coordinatewise recovery, leaving the basis for the later causal guarantees unresolved. This is a concern about the main result, not just presentation.

  2. Near-perfect nonlinear state readout coexists with much lower coordinate MCC, while graph estimation uses physical-state alignment and known parent counts. The Robotics reference graph is hand specified rather than established by interventions. The gap between coordinatewise theory and supervised state-readout evaluation is the main reason for weak rejection.

  3. The empirical forward-Jacobian scores are linked to Theorem 4's inverse-noise Jacobian only under additive noise in the aligned coordinates (Eq. 15). That property does not follow from identification up to nonlinear componentwise transformations. Its applicability to the evaluated systems needs justification before the graph-recovery guarantee can be connected to the estimator.

Questions for authors
  1. Which precise identification theorem and extra assumptions give Eq. (6), rather than a linear mixture of sufficient statistics? Please show how the implemented TCL/GCL classifier and any unmixing satisfy those conditions. See Hyvarinen et al. (2019), Theorem 3: https://proceedings.mlr.press/v89/hyvarinen19a/hyvarinen19a.pdf.

  2. What supervision and data splits are used to obtain the physical-state readout? How does graph recovery perform using only the allowed permutation and componentwise transformations, without a general supervised state decoder?

  3. How is the Robotics reference graph validated against simulator dynamics, including possible joint-to-end-effector dependencies? Please distinguish recovery of a chosen reference graph from recovery of the system's causal graph.

2 comments
AuthorsOpenReview ↗

Response to Reviewer s1Wk

Thank you for identifying a gap in our method. Theorem 1 allowed mixing that could change the graph, and the supervised decoder could undo that mixing before graph estimation. We revised Theorems 1 and 2, added the requested graph experiment in Table 7, and derived a new forward-Jacobian result in Proposition 1.

W1 / Q1. What justifies coordinatewise recovery?

The assumptions in the submitted Theorem 1 were insufficient for the coordinatewise claim in the submitted Eq. (6). Revised Theorem 1 gives the weaker result in Eq. (7). Following Hyvärinen et al. (2019, Theorem 3), the sufficient statistics are identified up to invertible affine mixing. Such mixing can change graph sparsity.

Revised Theorem 2 derives the coordinatewise relation in Eq. (14) under the assumptions of Hyvärinen et al. (2019, Theorem 1). The composite map from physical state to learned coordinates must be a smooth diffeomorphism. The sources must be conditionally independent given the classifier’s auxiliary variable, and the coordinate-separable classifier must attain the population optimum. At each state, Eq. (11) requires 2d+12d+1 auxiliary contexts whose log-density derivatives yield 2d2d linearly independent differences.

The new proof in Appendix A.3 takes mixed partials of the score identity across contexts. Full rank forces each row of the inverse-map Jacobian to have at most one nonzero entry. Invertibility and connectedness fix a single permutation, and the product-domain assumption gives an invertible scalar transformation per coordinate. This supplies the premise for graph preservation.

Our GCL classifier uses r(x,u)=∑iψi(hi(x),u)r(x,u)=\sum_i\psi_i(h_i(x),u) to distinguish observed pairs from pairs with randomized uu. Under Theorem 2, the result applies directly to hi(x)h_i(x) without further unmixing. TCL uses segment classification and does not inherit this guarantee. Conditional independence and derivative variability remain unverified on CausalVerse.

W2 / Q2. What survives removal of the decoder and parent counts?

The original evaluation used physical-state labels to fit the readout and reference parent counts to select binary edges. On Physical Fall, a linear readout of the GCL representation achieved R2=0.999R^2=0.999, while individually matched coordinates had MCC of 0.700. The state was readily decodable, but this result leaves open whether its graph can be recovered in the learned coordinates.

We ran the requested experiment and added Table 7 to Appendix A.12.1. We fit lag-1 Lasso models directly to learned coordinates, choose the penalty by predictive validation, and select edges at a fixed bootstrap stability threshold. Neither state labels nor parent counts enter these steps. Once the graph is fixed, we use training-set state labels only to match coordinate names by permutation for scoring. We split by video, with 4,000 training and 1,000 held out for Physical Fall and Microduck, and 2,160 training and 540 held out for Robotics.

On Physical Fall, TCL and GCL each achieved all-edge AUPRC of 0.787±0.0420.787 \pm 0.042 across five seeds, down from 1.000 under the original protocol. Their F1 of 0.800 matches what one obtains by predicting only the six self-lags. We therefore restrict the exact cross-edge recovery claim to the state-aligned evaluation with known parent counts. On Robotics, GCL reached F1 of 0.826, compared with 0.850 for CaRiNG and 0.849 for IDOL. On Microduck, five methods tied at 0.705.

W3. When does the forward Jacobian identify the graph?

You are right that a nonlinear coordinate change can destroy noise additivity. New Proposition 1 in Section 3 starts from the physical transition si′=fi(s,u)+εis_i'=f_i(s,u)+\varepsilon_i, with noise independent of the current state conditional on uu. After matching coordinates, the forward conditional mean satisfies

∂mi∂zj=E[ϕi′(fi(s,u)+εi)∣u]ϕj′(sj)∂fi∂sj.\frac{\partial m_i}{\partial z_j}=\frac{\mathbb{E}[\phi_i'(f_i(s,u)+\varepsilon_i)\mid u]}{\phi_j'(s_j)}\frac{\partial f_i}{\partial s_j}.

Under the stated regularity conditions, a finite, nonzero multiplier preserves the structural Jacobian’s zero pattern almost everywhere. Under derivative faithfulness, that pattern identifies the lagged graph. The deterministic case follows by the chain rule. Determinism of the full simulator alone is insufficient if the observed state omits relevant variables. The fitted Jacobian scores therefore remain empirical estimates.

Q3. How was the Robotics reference validated?

We did not independently validate the CausalVerse Robotics graph by simulator intervention. Its seven cross-edges link each joint velocity to its next joint position. A genuine joint-to-end-effector dependency could therefore be scored as a false positive. We now interpret the Robotics scores as agreement with this specified kinematic reference. Microduck uses a separate reference estimated by fitting one-step controlled dynamics to simulator states and evaluating the Jacobians on held-out trajectories.

Thank you for the detailed rebuttal, theoretical revisions, and new Table 7 experiments. While these additions clarify the scope, removing the supervised readout and known parent counts causes cross-edge recovery to drop substantially (e.g., Physical Fall F1 drops to 0.800, matching self-lags only), and key Theorem 2 assumptions remain unverified on CausalVerse. Because restricting exact recovery to state-aligned settings significantly narrows the main claims, I will maintain my rating of 2 (Weak rejection).

Reviewer_PhgDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks whether a frozen pretrained diffusion model can be equipped with identifiable latent coordinates without retraining the backbone. The authors train a lightweight alignment map on inverted diffusion latents with TCL, GCL or CEBRA objectives, and argue from nonlinear ICA theory that the result recovers the latent state up to permutation and componentwise transformations while preserving the lagged causal graph.

Critical strengths
  • The paper addresses a relevant question, namely whether a frozen diffusion model can provide identifiable state coordinates.
  • The evaluation goes beyond prediction accuracy and covers state recovery, graph recovery and intervention response.
  • The method is lightweight, since only a small alignment map is trained on top of the frozen latents.
Critical weaknesses
  • The theory and experiments do not seem to rely much on properties specific to diffusion models, so the backbone may be largely interchangeable with other injective encoders.
  • The paper motivates the diffusion backbone by its ability to generate trajectories, and Figure 1 shows lifting back to the diffusion latent, but I could not find any experiment in the paper that uses generation.
  • Since self-lag edges are added by construction, only 3 cross edges on Physical Fall and 7 on Robotics need to be inferred, which makes the graph results less informative than they look.
  • Many gains over baselines are within one standard deviation, and iVAE is often comparable or better, so the advantage of ConDA is not clear to me.
Questions for authors
  • SHD seems to be normalized differently across datasets (e.g., SHD 0.400 and accuracy 0.900 for CaRiNG on Physical Fall do not match), so could the authors clarify?
1 comment
AuthorsOpenReview ↗

Response to Reviewer PhgD

Thank you for pointing out that Figure 1 showed how an intervention could be rendered through the frozen model, but the submitted paper never showed the resulting trajectory. In response, we added Figure 6 and Tables 6 and 7 to Appendix A.12.1. Figure 6 shows a rendered Microduck rollout under a changed command. Table 6 tests which cross-variable edges we recover after excluding self-lags. Table 7 tests what we recover directly from the learned coordinates without a supervised state decoder or reference parent counts.

W1. Role of the diffusion backbone

A pretrained diffusion model can render a walking robot. Our question is whether we can use that same model to predict how the robot would move under a different command. ConDA learns a small alignment map on the frozen diffusion latents. Under the componentwise identification assumptions, lagged dependencies among the aligned coordinates correspond to those among the physical state variables. We can then change the command, roll the learned dynamics forward, and render the predicted motion with the original generator.

We agree that other encoders can satisfy the identification assumptions. In our new Physical Fall comparison, direct pixel TCL and diffusion-latent TCL gave nearly identical coordinate recovery (MCC 0.695 and 0.694, respectively). What the diffusion backbone adds is a way to take dynamics learned from video and render the consequences of an intervention without retraining the generator.

W2. Counterfactual trajectory generation

The new Figure 6 is a proof of concept. Starting from the same Microduck state, we change the forward velocity command from zero to −0.25 m/s and render a one-second rollout of the ConDA-TCL dynamics. We show six frames alongside MuJoCo rollouts under both commands. This lets the reader see what the command changed in the simulator and compare it with the motion generated by ConDA.

We separately measured the backward and lateral velocity responses over 100 matched rollouts. The model predicted the direction of both. At one second, it predicted about a third of the backward response (−0.0042 vs. −0.0129 m/s in MuJoCo) and came closer on the lateral response (0.0107 vs. 0.0084 m/s).

W3. Self-lags and cross-variable graph recovery

In the new Table 6, we exclude self-lags for all eight methods. We keep the original state alignment and give each method the reference number of parents per variable. The question is whether it picks the right parents. On Physical Fall, TCL and GCL found all three cross-edges in each of the five seeds. On Robotics, only seven of 600 candidate cross-edges belong to the reference graph. CaRiNG, IDOL, and TDRL each achieved 0.988 accuracy despite missing all seven.

In the new Table 7, we fit the graph directly to the learned coordinates without supplying parent counts. On Physical Fall, TCL and GCL each reached an F1 of 0.800 across all edges. Predicting just the six self-lags gives the same F1. Our claim of exact cross-edge recovery therefore applies to the state-aligned evaluation with known parent counts in Table 6.

W4. Comparison with iVAE

We agree that iVAE is competitive. The new cross-edge AUPRC results from Table 6 are means ± SD across five seeds.

DatasetiVAEConDA-TCLConDA-GCL
Physical Fall0.940 ± 0.1341.000 ± 0.0001.000 ± 0.000
Robotics0.500 ± 0.0800.407 ± 0.0630.378 ± 0.077
Microduck0.515 ± 0.0420.556 ± 0.0590.538 ± 0.089

iVAE led on Robotics. TCL had the highest mean on Microduck, but the difference from iVAE was modest relative to the variation across seeds. Strong state readout also warrants a narrower interpretation. On Physical Fall, ConDA-GCL achieved linear readout R2=0.999R^2=0.999 with coordinate MCC of 0.700. The physical state was readily decodable even though individual coordinates were less closely aligned. The case for using ConDA here rests on the ability to render and check an intervention, even when another method recovers the graph equally well or better.

Q1. Normalized SHD

We corrected the definition in Appendix A.11. If HH is the number of mismatched adjacency entries, normalized SHD is H/∣Eref∣H/|E_{\mathrm{ref}}| and accuracy is 1−H/∣Ecand∣1-H/|E_{\mathrm{cand}}|. For the CaRiNG row you flagged, Physical Fall has 36 candidate edges and nine reference edges. The same 3.6 errors averaged across seeds give accuracy 1−3.6/36=0.9001-3.6/36=0.900 and normalized SHD 3.6/9=0.4003.6/9=0.400. Excluding the six self-lags leaves 30 candidates and three reference edges, giving the corresponding Table 6 values of 0.880 and 1.200.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.