acceptodds
Under review as a conference paper at ICLR 2027

DELTA: Dual-stream Encoding with Latent Temporal Anticipation for Frozen Visual World Models

Abstract

Frozen vision backbones such as V-JEPA 2.1 provide strong visual features for manipulation, but not an action-conditioned model of dynamics. Existing world models introduce that dynamics under different modeling assumptions. Predictors defined on frozen features take a short temporal window and regress the next embedding, so they maintain no state beyond that window and do not cast the prediction as an update of the observed frame. Predictors that retain a longer history instead advance an image code or a learned latent, and therefore depart from the frozen representation. We propose DELTA, a reconstruction-free world model on frozen V-JEPA 2.1 (ViT-B/384) features. A dual-stream latent carries state across time and is updated by an action-conditioned causal transition. A residual head writes the resulting short-horizon change onto the observed tokens. Actions enter only as conditioning. On manipulation demonstrations, under a shared feature-cache protocol, DELTA lowers multi-step open-loop, replan, and CEM feature-prediction error relative to these baselines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, robotic manipulation, visual dynamics, representation learning, JEPA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KVjeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DELTA, an action-conditioned world model built on frozen V-JEPA 2.1 visual features. The method combines dual-stream latent encoding, action-conditioned state transitions, and residual feature prediction. Specifically, two temporal observation windows are encoded into a persistent latent state using cross-attention, which is then updated through FiLM modulation and a Transformer conditioned on robot actions. A residual head predicts future feature changes instead of reconstructing complete visual representations. Experiments on LIBERO Object evaluate the method against several baselines, including DINO-WM and STORM, using open-loop prediction, replanning, and CEM-based action selection. The results demonstrate competitive multi-step feature prediction accuracy, while ablation studies highlight the importance of residual prediction and action conditioning.

Critical strengths

Clear organization and completeness of presentation:The paper is well-structured and presents a coherent narrative, progressing logically from the research background and motivation to the proposed methodology and conclusions. The key components of the proposed framework are described with mathematical formulations and an overall architecture diagram, making the methodology relatively easy to follow. Overall, the manuscript is reasonably complete in its organization and technical presentation.

Critical weaknesses

Insufficient justification and validation of the semantic-dynamics stream distinction: The distinction between the semantic and dynamics streams is primarily based on the lengths of their temporal input windows (two versus four frames), while both streams use the same frozen V-JEPA 2.1 encoder. The paper does not provide explicit supervision or empirical evidence demonstrating that the two streams actually capture distinct semantic and motion-related information. Since both temporal windows may contain overlapping semantic and dynamic information, the motivation for assigning these specific roles remains unclear. Furthermore, the single-stream ablation only demonstrates the benefit of adding another stream, rather than establishing the necessity of semantic-dynamics specialization. Additional controlled experiments using different temporal window combinations or feature-level analyses would be needed to substantiate this central architectural assumption. Insufficient justification for the additional cross-stream attention: Before concatenating the two streams, DELTA introduces an additional cross-attention module that allows the dynamics stream to attend to the semantic stream. However, the subsequent action-conditioned Transformer already employs self-attention over all concatenated tokens, enabling interactions between the two streams. Therefore, the necessity of this additional cross-attention and the choice of its asymmetric direction remain unclear. The paper lacks a direct-concatenation baseline or alternative fusion strategies to demonstrate that this design provides benefits beyond the subsequent self-attention module.

Questions for authors

1.What evidence supports the distinction between the semantic and dynamics streams? The two streams primarily differ in their temporal input windows (two versus four frames), while both use the same frozen V-JEPA 2.1 encoder. What evidence demonstrates that they learn distinct semantic and motion-related representations rather than largely overlapping information? Could the authors provide controlled comparisons with alternative temporal window combinations (e.g., 2+2 or 4+4 frames), or feature-level analyses to justify this distinction? This is important for assessing whether the proposed dual-stream design provides meaningful specialization beyond additional temporal context and model capacity. 2.Why is an additional asymmetric cross-attention module necessary before concatenating the two streams? The dynamics stream attends to the semantic stream before concatenation, but the subsequent action-conditioned Transformer already performs self-attention over all concatenated tokens. What additional benefit does this asymmetric interaction provide? Could the authors compare it with direct concatenation without the extra cross-attention module? This comparison would help establish whether the proposed fusion mechanism is necessary for the reported performance improvements. 3.Could the authors clarify the evaluation protocol and its consistency with the supplementary implementation? The main paper reports results on 20 test episodes, whereas the supplementary README describes a fixed eight-episode evaluation subset. Which episodes were used to generate the main results in Tables 1–3? Additionally, the provided evaluation scripts call predict_features() without supplying the base features required when residual prediction is enabled. Could the authors clarify which evaluation implementation and model configuration produced the reported results? Resolving these discrepancies is important for assessing the reliability and reproducibility of the experimental comparisons.

1 comment
AuthorsOpenReview ↗

Response to Reviewer KVje

We thank the reviewer for a constructive and specific report. All numbers below are the metric used throughout the paper (pooled semantic-token MSE, 10^2), on the same episodes, at the same checkpoints.

Q1 -- Evidence for the semantic/dynamics distinction.

We do not claim that the two streams learn different content from the same encoder. We claim that they read different temporal extents and that both are needed. Three controls, now added as Table 4 of the revision:

Varianth=5h=20h=40
2+4 (ours)0.350.921.67
2+20.340.982.13
4+40.350.981.77
Single stream, 4-frame0.340.991.91

Shortening the long window costs 27% at h=40, lengthening the short window costs 6%, and cutting to one stream costs 14%. The 4+4 gap sits close to the seed spread (4%, measured across two seeds of the full model), so we do not claim more than this: both windows are needed, and the asymmetric pairing is the best of the three. Short horizons are unaffected in every case (0.34 to 0.35 at h=5), which is consistent with the streams serving the long-horizon carry rather than the one-step term.

Q2 -- Is the cross-stream cross-attention redundant given the later self-attention?

We replaced the cross-stream step with a concat plus a linear projection, keeping both stream readers and matching capacity (the variant has slightly fewer parameters, 5.39M against 5.52M):

Varianth=1h=5h=10h=20h=40
cross-attention (ours)0.200.350.450.921.67
concat + projection0.210.340.450.961.74

The 0.07 gap at h=40 is smaller than the seed spread, so we do not claim that the coupling operator is the source of the gain, and we have removed any phrasing that implied it. What the coupling does buy is that the two streams can carry different token counts (K and 2K) while sharing one conditioning path, which is what the 2- and 4-frame windows require; a concat forces a common token count. We thank the reviewer for pressing on this, since the original wording overclaimed.

Q3 -- Which episodes produced Tables 1-3, and the evaluation script.

The tables use a fixed evaluation subset of the held-out test split, defined in the released split file and applied identically to every method and every table. The paper now names it that way rather than saying "the test split", which is where the discrepancy with the README came from. For completeness, the full 20-episode test split gives 0.21 / 0.35 / 0.46 / 0.97 / 1.76 at h=1/5/10/20/40 and 0.70 under replan at r=10; every ordering in Tables 1-3 is unchanged on it.

On the script: the reviewer is correct that the shipped predict_features() call omitted the base features. That call is valid only when use_feat_residual is disabled; with the residual head enabled it silently drops the anchor the head was trained on, so the shipped script could not reproduce the residual rows. The updated supplement passes the base pair and feeds the predicted pair back between replanning steps, matching the training-time interface, and the README records the change. The numbers in the paper were produced by the corrected path. We are grateful for the catch.

We thank the reviewer again for a report that led to concrete improvements in both the paper and the released code.

Reviewer_jPR9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper argues that feature-space world models on frozen encoders has a trade-off between anchoring predictions on observed features and carrying state across time. To this end, the authors introduce a reconstruction-free action-conditioned world model called Delta. DELTA treats the observed features as an anchor to be corrected rather than regenerated, and relies on a persistent, action-driven latent state to carry information beyond the current frame. Evaluated by pooled semantic-token MSE on LIBERO-Object demonstrations, DELTA reports the lowest or tied error against adapted baselines.

Critical strengths

None

Critical weaknesses
  • Novelty issue

    • The proposed method seems to provides technical approaches that enhance the quality of the representations. But these techniques themselves are insufficient to prove their novelty
  • The writing needs polishing. Sentences are unnatural and paragraphs are not organized well overall

    • For example, the abstract part does not sufficiently deliver concise summary of the paper
  • Narrow range of validation

    • Insufficient quantity of experiments to validate the effectiveness of the proposed method
    • All experiments use only LIBERO-Object demonstrations with narrow diversity (e.g., few number of seeds). Evaluation on various environments or benchmarks seem necesary.
  • In table 1, long-horizon advantage disappears when h reaches 40. The table shows that DELTA results matches with the scores of MLP-Δ. Thus, there should be further verifications with larger h.

  • no qualitative results throughout the paper and supplementary materials

Questions for authors
  • polishing issue
    • The title breaks its line at a strange position
1 comment

We first want to thank you for the time you have put into this review, and for how specific the points are -- a report of this detail is genuinely useful to us.

We write with some hesitation, and we apologise in advance if we have misread anything. The Summary field of your review describes a paper we do not recognise as our own: VLA state aliasing, seven frozen encoders, an auxiliary inverse-dynamics objective with pseudo time-reversal augmentation, and evaluation on VLM4VLA, CALVIN and SimplerEnv. We may of course be missing something, but our submission is a latent world model built on frozen V-JEPA 2.1 features and evaluated on LIBERO, and we do not use an inverse-dynamics objective at any point in training.

We raise this only so that it can be checked, and not to dispute any part of the review. We are glad to respond to each comment in the discussion, and we are not asking for the review to be removed or reconsidered -- that is entirely a matter for you and the Area Chair. If the Summary was carried over from another review, we would be very grateful if that could be confirmed; if we are the ones who have misunderstood, please do correct us and accept our apologies for the trouble. We had also mentioned this to the Area Chair, in case that is the more appropriate channel, and we are happy to leave the decision to you both.

Thank you again for your patience and your time.

Reviewer_kYsfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DELTA, a reconstruction-free, action-conditioned world model built on frozen V-JEPA 2.1 (ViT-B/384) features. Two frozen-feature streams are read into a dual-stream latent via cross-attention: a 2-frame “semantic” stream and a 4-frame “dynamics” stream. This latent is carried across time and updated by a FiLM-conditioned transformer transition. A zero-initialized head then predicts the next pooled feature tokens as a residual on the current (or previously predicted) features. The model is trained with three losses: a teacher-forced one-step loss, an 8-step rollout loss, and a latent-motion regularizer. Evaluation uses LIBERO Object demonstrations and measures pooled semantic-token MSE under open-loop rollout, periodic replanning, and an offline CEM probe. Under these protocols, DELTA reports the lowest or tied error against DINO-WM, Copy, MLP-Δ, and residual adaptations of STORM, TWM, and EMERALD.

Critical strengths
  • Well-motivated problem and design rationale. The paper identifies a concrete tension in feature-space world models. Residual prediction is accurate at short horizons but carries no memory. Decoding from a latent carries memory but starts with higher error. The proposed dual-timescale design, F̂ₜ₊₁ = pool(F) + Δ(Zₜ₊₁), is a reasonable way to combine the two. The method and Figure 2 are clearly presented.
  • Controlled evaluation protocol. All methods share one frozen feature cache, which isolates the predictor from encoder differences. The three probes (open-loop, replan, CEM) examine drift, recovery, and action sensitivity rather than one-step error alone.
  • Consistent mid-horizon gains at comparable model size. DELTA has 5.52M predictor parameters, versus 5.27M for DINO-WM. At h=10, its open-loop error is 0.45, versus 0.65 for the best baseline. At r=10, its replan error is 0.67, versus ≥1.02 for all other methods. It is also the only model whose error clearly rises under random actions (0.45 → 0.62), which indicates it actually uses the action input.
Critical weaknesses
  • The central claim is not directly tested, and the source of the gains is unclear. The core argument (Fig. 1b) is that residual prediction without a carried latent drifts at long horizons. However, no DELTA variant keeps the residual head while removing the carried latent, for example by resetting the queries to learned constants at every step. Since Figure 1 is schematic, this half of the dual-timescale argument is asserted rather than shown. The inputs are not matched across methods. Sec. 4.2 states that all baselines consume only the semantic (2-frame) tokens, while DELTA additionally reads the 4-frame dynamics stream. When that stream is removed (single-stream row of Table 3): DELTA is worse than MLP-Δ and STORM at every horizon. It is no better than Copy at h≥10 (0.70 vs. 0.70 at h=10; 1.30 vs. 1.28 for replan at r=10). The main-table margin may therefore come largely from the extra temporal input rather than from the architecture. In addition, this ablation removes four components at once: the 4-frame input, half the latent tokens, the cross-stream coupling, and the dynamics loss term. Their individual effects cannot be separated. L_roll and L_reg are not ablated. The paper also does not state whether baselines were trained with the same 8-step rollout loss, and that loss alone could explain much of the reduced drift.
  • The baselines appear weak, and the training protocol is described inconsistently. Several trained baselines fail to beat the Copy baseline, which has no dynamics at all: at h=1, DINO-WM (0.44 vs. 0.41); at h=10 and h=40, DINO-WM, TWM, and EMERALD. STORM, TWM, and EMERALD also give identical errors under demonstration and random actions, which suggests they ignore the action entirely. Even zero-action DELTA (0.56 at h=10) outperforms every action-conditioned baseline. Together these results suggest that the baseline adaptations to 32 pooled tokens are under-tuned. The training budgets are described inconsistently. Sec. 4.1 and 5.2 state that all models were “trained to completion” under “the same training budget.” In contrast, App. A.1 states that the reported DELTA checkpoint is step 2600 of a 5000-step run, while the ablations use a 2500-step budget. The “Full” row in Table 3 was therefore not trained under the same budget as the ablations, and the paper does not explain how step 2600 was selected. All results come from a single seed on 20 test episodes, with no variance or confidence intervals. Small differences, such as 0.20 vs. 0.21 at h=1, cannot be assessed.
Questions for authors
  • Could the authors report results for the following variants? (a) DELTA with the residual head but without a carried latent; (b) a single-stream variant that reads the 4-frame features; (c) baselines given the same dynamics tokens. Were baselines trained with the k-step rollout loss? How does DELTA perform without L_roll and L_reg?
  • Please clarify the data split and the computational cost: Does “task-wise” mean held-out tasks or stratification by task? The split totals 454 episodes, whereas LIBERO Object nominally provides 10 tasks × 50 demonstrations. Were some episodes filtered? Could the authors report wall-clock inference and planning time, including the encoder, and, if feasible, closed-loop success on at least one suite?
2 comments
AuthorsOpenReview ↗

Response to Reviewer kYsf

We thank the reviewer for naming the missing control so precisely. All numbers below use the metric of Table 3 (pooled semantic-token MSE, 10^2), on the same episodes and checkpoints.

The central claim. We accept the criticism as it applies to the submitted text. Figure 1b is a schematic that asserted a conclusion; we have removed that assertion from the text, and the caption now states only what the panel plots: a residual path is sharp at short horizons (0.20 at h=1), and decoding the full token from a compact latent starts from a much higher one-step error (0.89 at h=1). The control you specify - residual head retained, carried latent removed, queries re-initialised each step - is the right one and is scheduled; we will report it under the same recipe, split, and metric rather than quote a number not checked to the same standard.

The inputs are not matched. The token budget is identical (K=32) for every method; what differs is which windows feed those tokens. We separated the confounds you identified and added them, with two controls of our own, as Table 4 (open-loop MSE):

Varianth=1h=5h=10h=20h=40
DELTA (2+4)0.200.350.450.921.67
2+2 window0.210.340.440.982.13
4+4 window0.210.350.460.981.77
Single stream, 4-frame0.200.340.460.991.91
Concat coupling0.210.340.450.961.74

Your variant (b) closes the short-horizon gap completely (0.20, 0.46) but still lands at 1.91 at h=40 against 1.67, so the long-horizon margin is not the extra temporal input alone. Four frames on both streams gives 1.77 and two frames on both gives 2.13, so it is the asymmetry of the windows, not temporal context, that the design turns on. The four components do come apart: dropping the coupling costs 0.07, the second stream 0.24, the long window 0.46.

The baselines. We push back gently: TWM ties DELTA at h=1 (0.20) and is within 0.06 at h=5; STORM is within 5% at h=40 (1.76 vs 1.67). What they carry is an interface cost: DINO-WM predicts a patch grid and is scored on 32 pooled tokens, TWM and EMERALD are built around spatial token grids, and STORM's planner is never called. No method was retuned beyond this interface, as Section 4.2 now states. Copy is the zero-residual reference a residual head has to beat, and TWM (2.68) and EMERALD (3.60) do not, while DINO-WM (1.97) is within noise of Copy (1.92). Their flat action sensitivity is a property of how their outputs are formed under this interface, which is why we report it as a separate column rather than inferring it.

The rollout loss. Yes: every trained baseline optimises the same teacher-forced and k=8 rollout losses, now stated in Section 5.1. Being shared, it cannot account for differences between methods; it accounts for the magnitude of drift, since removing it from DELTA raises h=40 error from 1.67 to 79.68.

Training budgets. The submitted text contradicted itself and both places are rewritten. Every model ran to the end of its scheduled budget and its checkpoint was chosen by validation error on the monitoring episodes, the same rule for DELTA, every ablation, and every baseline. For DELTA that selects step 2600 of the scheduled 5000-step run, where h=40 error is lowest (1.79 over 70 episodes) and rises again at later steps.

Single seed. A second seed gives 0.20 / 0.45 / 1.74 at h=1/10/40 and 0.52 / 0.69 / 1.28 under replan, against 0.20 / 0.45 / 1.67 and 0.52 / 0.67 / 1.23. The h=40 spread is a little over 4%, and differences below it are no longer treated as separations: we withdraw the h=40 margin over STORM (1.67 vs 1.76). What survives is h=10, 0.45 against 0.65 for the best baseline, and 0.67 under replan at r=10 against at least 1.02.

Q(a) answered above; scheduled. Q(b) 0.20 / 0.46 / 1.91: matches or beats MLP-Delta through h=10, loses at h=40. Q(c) not run: changing a baseline's input changes its architecture, so it would no longer be the published method under one interface. The 4+4 row is the closest bound we can offer (1.77).

Split and cost. "Task-wise" means stratified by task: all ten Object tasks appear in the training, monitoring, and test portions, and no task is held out. The 454 episodes are the public release as shipped, with no filtering of our own; the nominal 10x50 count is not what the released archive contains. Wall-clock inference and planning time will be added in the revision. Closed-loop task success is downstream of this world model and outside what the paper claims.

Thank you for the detailed response and the additional experiments. The clarifications on the shared rollout loss, the training budgets and checkpoint selection, and the data split are helpful. The second seed, together with the correspondingly narrowed claims, makes the results more credible. In particular, the 2+2 variant shows that the mid-horizon gain holds even when DELTA receives the same input as the baselines, which addresses my main concern about input asymmetry. I have raised my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.