acceptodds
Under review as a conference paper at ICLR 2027

JEPA-DIT: FROM-SCRATCH PREDICTIVE WORLD-MODEL PRETRAINING FOR MANIPULATION

Abstract

World-action models—transformers that jointly denoise future video and actions—represent the world almost exclusively in the latent of a reconstruction VAE inherited from a video-generation prior. That is a liability for control: a re- construction objective must preserve every pixel-level nuisance a decoder has to repaint, whereas a policy needs only the scene’s controllable structure. A policy- free diagnostic makes the cost concrete—under domain randomisation the recon- struction latent moves nearly twice as far from its clean distribution as a predictive (JEPA) latent (ρ= 2.05 vs. 1.16), while a cross-task control shows both register a change of task equally: invariance, not collapse. The field keeps the VAE any- way, because a JEPA latent has no generative prior to build a world model on. We present JEPA-DiT, to our knowledge the first world-action model whose dif- fusion transformer is pretrained from random initialisation in a JEPA latent, car- ried unchanged from egocentric pretraining through teleoperation mid-training to downstream manipulation, and delta-JEPA, which makes the two representations complementary rather than competing by having the predictive stream carry the temporal difference from the present observation. Training from scratch proves to be not the price of a predictive latent but the reason it works: in a width-matched ablation the from-scratch stream develops a language pathway at 25×the floor uniform attention would give while a warm-started run reaches 2.1×, barely above chance, and passes that deficit downstream. Across all 50 RoboTwin 2.0 tasks with randomisation held out of training, JEPA-DiT reaches 84.5% clean and 59.5% out of distribution—the best OOD score and best clean/OOD average among world- action models on this protocol. Each representation alone falls short of the pair: with a mature prior but no predictive stream the model scores 1.9% under the shift, with a predictive stream but no prior 11.9%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
JEPA, DIT, WORLD MODEL, PRETRAINING, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_xqnxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies predictive JEPA representations for world-action modeling. The authors pretrain a diffusion transformer from scratch in a frozen V-JEPA feature space and combine a VAE stream for absolute scene information with a delta-JEPA stream for changes relative to the current observation. Experiments on RoboTwin 2.0 show improved success under held-out domain randomization.

Critical strengths

Using predictive features as the state space of a generative world-action model is a meaningful direction. Controlled re-rendering experiments show that the selected JEPA encoder is less sensitive to texture changes, providing concrete support for this motivation. Evaluation covers all 50 RoboTwin tasks with randomized scenes held out of downstream training. The predictive-stream ablation improves OOD success by 6 percentage points, supporting the practical benefit of the added branch.

Critical weaknesses
  1. The benefit of the JEPA representation is not isolated. The VAE-only and JEPA-only models differ in backbone and pretraining, while removing the predictive stream also reduces capacity. The matched-step comparison does not establish whether the OOD gap would persist after both models are trained to a performance plateau. These limitations leave the claimed representation advantage difficult to distinguish from capacity and optimization effects.
  2. The contribution of delta-JEPA is not established. There is no comparison against absolute JEPA features within the same dual-stream architecture. The existing branch ablation therefore cannot determine whether temporal differencing improves robustness beyond simply adding a second representation, leaving a central design claim unsupported by a direct test.
Questions for authors
  1. Can the authors provide a VAE-only baseline under the same clean-only protocol? If possible, a clean-only adaptation of LingBot-VA would also help assess the gain over a strong VAE-based model.
  2. Does the VAE baseline catch up with further training?
  3. What changes when delta-JEPA is replaced with absolute JEPA features, keeping the dual-stream architecture, mask, and training setup fixed? This would isolate the contribution of temporal differencing.
1 comment

We thank Reviewer xqnx for the positive assessment of our problem formulation, diagnostic controls, and 50-task evaluation, and for pinpointing the exact controlled ablations needed. All three requested experiments are now in §§4.4-4.5 (Tables 3 & 4, Figure 3).

1. Matched Clean-Only VAE-Only Baseline, LingBot-VA & Granularity Controls (W1 & Q1)

a) Same-backbone VAE-only baseline (JEPA-DiT-ReconOnly, lambda_p = 0): On our exact 32-layer OmniJointAttention backbone, pretraining lineage, and 2,500 demo_clean trajectories, disabling only the predictive stream (lambda_p = 0) isolates representation from backbone and pretraining (Tables 3 & 4a):

Configuration (All 50 Tasks, Clean-Only)Clean (%)OOD (%)Avg (%)
Exploratory (bs=256, 10k): Without JEPA (lambda_p=0)77.035.356.2
Exploratory (bs=256, 10k): With delta-JEPA (lambda_p=1)80.441.360.9
Peak (bs=512, 11k): JEPA-DiT-ReconOnly (lambda_p=0)85.046.065.5
Peak (bs=512, 11k, 100 trials/task): 4-stream JEPA-DiT85.259.072.1
Scaled multi-source + 34-DoF EEF: JEPA-DiT-EEF90.365.177.7

At step 11k, ReconOnly matches JEPA-DiT on Clean (85.0% vs. 85.2%), confirming full manipulation capacity on its own, while adding delta-JEPA yields a pure +13.0 point OOD gain (46.0% -> 59.0% over 10,000 rollouts; a second 10,000-rollout run gives 84.6% / 59.2%, <0.6% variance). Standalone from-scratch 3-stream JEPA-DiT-PredOnly (zero VAE tokens, matched bs=512, lr=3e-5) also reaches 76.0% Clean / 35.2% OOD.

b) Clean-only LingBot-VA adaptation: Published LingBot-VA (92.2% / 91.0%) uses mixed training (50 clean + 500 randomised demos/task in-distribution). Our Wan-5B LingBot-VA reproduction matches this under mixed training (90.0% / 90.4%), but adapting Wan-5B strictly on demo_clean drops to 68.8% Clean / 0.0%-14.0% OOD (§4.4), mirroring Fast-WAM (77.8% -> 1.9%).

c) Identical temporal granularity & capacity check: Under delta-JEPA (§3.3, Eq. 2), V-JEPA 2.1 (tau=2) encodes boundary pairs [I_0,I_0], [I_16,I_16], [I_32,I_32], giving F=3 slots (G=2 future steps x N=16 actions), matching the causal VAE's F=3 grid 1-to-1. Moreover, under recon ⊥ jepa and txt -> {txt}, adding the delta-JEPA expert adds zero parameters or attention keys to text/reconstruction, and Absolute-JEPA below confirms extra stream capacity alone hurts OOD without differencing.

2. Does Longer Training Close the OOD Gap? (Q2)

No (Table 4c, §4.5). Across all 50 tasks under clean-only training, extending past step 11k slightly raises Clean while degrading OOD by -10.2 points due to clean-appearance memorisation:

Step (All 50 Tasks)8k9k11k (Peak OOD)13k
Clean / Held-out OOD (%)84.0 / 55.682.6 / 51.484.5 / 59.585.6 / 49.3

ReconOnly similarly peaks at step 11k (85.0% / 46.0%) without catching up thereafter.

3. Direct 4-Stream Comparison: Delta-JEPA vs. Absolute-JEPA (W2 & Q3)

Holding the exact 4-stream architecture, frozen encoders, and recon ⊥ jepa mask fixed, we compare predicting phi(g) - phi(g-1) (Delta-JEPA) vs. un-differenced phi(g) (Absolute-JEPA) on a 4-task bimanual suite (place_container_plate, place_empty_cup, stack_bowls_two, stack_bowls_three; Table 4b, §4.5):

4-Stream Model (Same Backbone & recon ⊥ jepa Mask)Clean (%)OOD (%)Avg (%)
4-Stream Absolute-JEPA (phi(g), matched LoRA)72.57.540.0
4-Stream Absolute-JEPA (phi(g), slot-tuned)77.52.540.0
4-Stream Delta-JEPA (phi(g)-phi(g-1), matched LoRA)60.040.050.0
4-Stream Delta-JEPA (phi(g)-phi(g-1), full fine-tune)94.087.090.5

Without differencing, Absolute-JEPA reaches 72.5%-77.5% Clean yet collapses to 2.5%-7.5% OOD because static scene redundancy lets the policy exploit clean appearance shortcuts. Subtracting phi(g-1) isolates pure state transitions, lifting matched-budget OOD by +32.5 points (>5x) and full fine-tuning to 87.0% OOD.

Reviewer_fa7HOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

JEPA-DiT studies latent video target choice and backbone initialization for flow-matching World-Action Models. The authors argue that predictive tubelets discard appearance nuisance while reconstruction VAEs entangle appearance with dynamics. They couple experts of different modality via masked joint self-attention with a trainable text stream, and ultimately propose a four-stream model that pairs a reconstruction VAE stream with a mutually blind V-JEPA 2.1 stream predicting temporal differences against a head-padded present-observation anchor. Evaluated across all 50 RoboTwin 2.0 tasks under demo_clean to demo_randomized co-training, the four-stream model reaches 84.5% Clean and 59.5% OOD success.

Critical strengths

Complementary world representations: The paper combines an absolute reconstruction stream with a residual predictive stream, allowing action generation to draw on both scene detail and appearance-robust temporal features, offering a principled alternative to choosing a single representation, with the predictive-stream ablation supporting improved OOD generalization.

Critical weaknesses
  1. Unproven benefit of from-scratch JEPA pretraining: Although the paper centers its title and motivation on pretraining from random initialization in a JEPA latent space to avoid a cross-modal attention bottleneck, this benefit is not sufficiently proven in experiments. The initialization comparison only measures open-loop attention weights under mismatched learning rates without evaluating downstream control, and in closed-loop benchmarks the model actually pretrained from scratch in JEPA space performs poorly on OOD tasks (11.9%), whereas the flagship four-stream model warm-starts its predictive stream onto a pretrained reconstruction backbone.
  2. Organization and writing: the paper put too many visuals and results in the supplementary while leaving very few in the main text. Almost all main sections are directing to the appendix for more details. I would highly recommend the author to manually review the writing and organization the main tex into a mostly self-contained story instead of almost depleting the alphabet letters in the appendix.
Questions for authors
  • Please refer to weakness.
  • There are strong indications and patterns of AI-aided writing and is making me hard to follow, especially it involves topics I am less familiar with. For example, the excessive use of em dashes throughout the entire manuscript, and even the "Use of AI assistance" disclosure feels like AI-generated. Would the authors confirm and share more details on the workflow on how they ensure the manuscript is well organized and does not contain any hallucinated contents?
1 comment

We thank Reviewer fa7H for the candid and constructive critique regarding (1) the narrative tension between our original title and the 4-stream model's initialisation, and (2) the reliance on 24 appendices and em-dash punctuation. We have overhauled both our experimental controls and the manuscript to resolve every point.

1. Unified Co-Evolution Narrative, Matched-LR Controls & Closed-Loop Payoff (W1)

a) Retitling and unifying the co-evolution story: We agree our original title over-indexed on standalone from-scratch pretraining. We retitled the paper "JEPA-DiT: Complementary Predictive and Reconstructive World-Action Pretraining for Manipulation" and unified §§1, 3, and 4.3 around one cross-modal co-evolution principle: a trainable Text Expert (over frozen VLM tokens) forms a deep language pathway only when co-trained from random initialisation alongside a visual stream.

  • Standalone 3-stream predictive WAM (JEPA-DiT-PredOnly, zero VAE tokens): With no JEPA video-text prior available, co-training Text + JEPA from scratch builds a 25.1x language pathway (vs. 2.1x warm-started). Crucially, our initial Table 3 reported an early unaligned run (63.3% / 11.9%). Trained at the exact same batch size (512) and learning rate (3e-5) as our main model, from-scratch JEPA-DiT-PredOnly reaches 76.0% Clean and 35.2% OOD across all 50 tasks (Table 3, §4.4), keeping 9.8x language attention (0.203) and 0.796 act -> jepa reliance, beating published VAE-only WAMs (1.9%-3.2% OOD) by >10x.
  • Complementary 4-stream JEPA-DiT: Phase 1 applies the same rule by co-training Text + Reconstruction from scratch, anchoring task language in the reconstruction stream (recon -> txt = 0.248; instruction swaps shift reconstruction predictions by Delta cos = 0.17-0.26 vs. <=0.022 for delta-JEPA). This frees Phase-2 delta-JEPA to specialise purely on appearance-invariant dynamics (phi(g) - phi(g-1)), lifting 50-task performance from 85.0% / 46.0% (ReconOnly) to 85.2% / 59.0% (JEPA-DiT, 10,000 rollouts) and 90.3% / 65.1% (JEPA-DiT-EEF).

b) Controlling for learning rate and visual convergence (Table 2, §4.3):

  1. Matched visual convergence at Ego step 4k: On 60 validation clips, the warm-started run (lr=5e-5) reaches slightly better visual prediction than the cold-start run (lr=1e-4): Delta cos = 0.495 vs. 0.486 (rel-L2 = 0.970 vs. 0.979), proving 5e-5 did not visually under-train the JEPA stream; rather, newborn JEPA projections failed to align with already-converged text weights (2.1x vs. 25.1x).
  2. Strictly identical lr = 5e-5 in AgiBot mid-training: Carrying both ego checkpoints into AgiBot mid-training under identical lr = 5e-5, Muon+, widths (2048/4096/1280), mask, and step (5k), the gap widens to 23.4x (21.2x vs. 0.9x floor; 27/32 layers below floor in warm-start vs. 0/32 from scratch), raising downstream world reliance (act -> jepa = 0.847 vs. 0.641).

2. Self-Contained 10-Page Main Body (W2)

Every core figure, diagnostic, and ablation now resides inside Pages 1-10:

  • Figure 1 (p. 4): Full 4-stream architecture (frozen vs. trainable modules, feature differencing, joint mask).
  • Table 1 (p. 7): Invariance ratio rho, cross-task controls, and zero-motion (max|Delta q|=0) texture re-rendering.
  • Figure 2 & Table 2 (pp. 7-8): 3-panel attention evolution and matched-LR initialisation controls.
  • Figure 3 & Table 3 (pp. 8-9): 50-task RoboTwin 2.0 comparison (JEPA-DiT, JEPA-DiT-EEF, PredOnly, ReconOnly, and published WAMs).
  • Table 4 (p. 10): Single-variable stream removal (+13.0 OOD), Delta-JEPA vs. Absolute-JEPA (+32.5 OOD), and step trajectory (8k-13k).
  • Appendices were consolidated from 24 sections (A-X) down to 6 (A-F).

3. Complete Author Prose Rewrite (W3)

All hypotheses, code, derivations, and metrics were implemented and programmatically verified against primary logs by the authors; LLMs were used only for grammar polishing by non-native speakers. We manually rewrote the entire manuscript, removing all em-dashes (0 occurrences, down from 60+) and stating our AI disclosure plainly on Page 11.

Reviewer_9ebSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper attempts to improve performance of world action models (WAMs) under OOD settings that introduce domain randomization. The work identifies the use of image VAE encoders trained using reconstruction as a limitation in most of the current WAMs, and includes a predictive-objective (JEPA) based diffusion transformer that encodes changes in image sequences, and is trained from scratch along with a language encoding branch. The results demonstrate that the proposed method can achieve higher success rate of 59.5% on OOD tasks of bimanual Robotwin benchmark compared to the next best WAM implementation that only achieve 48.7% success rate.

Critical strengths
  1. The work attempts to include additional experiments to demonstrate the different information captured in reconstructive and predictive encoders.
  2. Combining reconstructive and predictive encoders is interesting and is also recently gaining traction in WAM literature [1,2].
  3. The results show the proposed method achieves best WAM performance on OOD settings of delta-JEPA.

[1] Flex-π: A Multi-Stream World-Action Model with Compute Flexibility. arxiv 2026 [2] What Visual Representations Do World Action Models Need? arxiv 2026.

Critical weaknesses
  1. The presentation can be significantly improved. The writing at places (eg. L115) seems casual and hard to follow. The motivation studies for different reconstructive and predictive encoders are a little disorganised across the paper (introduced in method section 3.4).

    a. The main approach figure (Figure 1) is also a misrepresentation of the complete approach and the authors provide another version of the figure in Figure 4. The authors emphasize language encoding pathway are developed together from frozen but the main approach figure suggests these are frozen, and passes raw images (and not changes) to this pathway.

  2. Optimal representation for motion. The paper encodes differences in images using the JEPA-like pathway. Image differences can still capture nuisance factors like textures of objects. Prior work has explored using optical flow for robot policies [2, 8] but the paper does not compare to alternate representations free of nuisance factors.

  3. The necessity of training encoders from scratch is not fully clear. Training textual encoder from scratch prevents generalization to unseen language instructions and unique objects the model has not encountered before. It is not clear why the proposed approach cannot be adapted from pretrained video backbones [1].

  4. The current ablations only show importance of including reconstruction branch and do not completely justify the need for including predictive branch, training from scratch and using image changes in place of raw images with careful ablations that evaluate on the same benchmarks. Further, comparison to LDA-1B that uses predictive latents is missing.

  5. The evaluation is limited to a single benchmark, and the gains are limited to OOD settings of the benchmark. The paper doesn't study more simulation benchmarks or demonstrate the effectiveness method on real-robot. The evaluation to prior works is unfair as they do not necessarily use the same datasets.

  6. The paper lacks methodological novelty. Training encoders from scratch [4, 7], using predictive encoders [6, 8], training future prediction networks from scratch [5, 7] and using flow-like representations [2, 3, 8] have been explored before in the space of action-conditioned world models and behavior cloning, and some also in world-action models [5]. The paper combines some of the ideas to world action models.

[1] Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. Kim et al. ICLR 2026.

[2] FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Model. Gao et al. ICLR 2025

[3] FlowWAM: Optical Flow as a Unified Action Representation for World Action Models. Chen et al. arxiv July 2026.

[4] LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. Maes et al. arxiv March 2026.

[5] Unified Video Action Models. Li et al. RSS 2025.

[6] LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion. Lyi et al. RSS 2026.

[7] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. Zhou et al. ICML 2025.

[8] EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data. arxiv July 2026. CoRL 2026. (More recent)

Questions for authors
  1. What pretrained weights does the work uses in the warm start comparison to motivate the need for training textual encoder from scratch?
  2. L161. What does "comparable across representations" mean?
  3. L233: Shouldn't "least stable" be "most stable"?
  4. What are benefits of delta-JEPA? How does it compare to a baseline that inputs the same raw images to JEPA encoder?
1 comment

We thank Reviewer 9ebS for the detailed comments and related-work pointers. Several central concerns stem from ambiguities in our original presentation and Figure 1. We clarify each point below and updated Figure 1 and §§2-4 accordingly.

1. Clarification of Core Technical Points (Soundness)

1) Representation-space differencing, NOT pixel/image differencing (W1, W4, Q4): JEPA-DiT never takes pixel differences. As shown in revised Figure 1 and §3.3 (Eq. 2), RGB frames at [0,0], [16,16], [32,32] are encoded first by frozen V-JEPA 2.1 phi(.), and differences are computed in V-JEPA latent space: j_tilde_0 = phi(g_0) and j_tilde_g = phi(g) - phi(g-1) (g >= 1).

  • Nuisance suppression: Appearance nuisances are filtered by phi(.) before differencing (rho = 1.16 vs. 2.05 for VAE; zero-motion texture re-rendering retains cos = 0.47-0.53 in V-JEPA vs. 0.29-0.36 in VAE, Table 1).
  • Direct comparison to un-differenced Absolute-JEPA (Q4; new Table 4b, §4.5): In the exact same 4-stream architecture and recon ⊥ jepa mask on our 4-task suite, predicting raw V-JEPA features phi(g) collapses to 2.5%-7.5% OOD due to static redundancy, whereas delta-JEPA (phi(g)-phi(g-1)) reaches 40.0% OOD under matched LoRA (+32.5 pts) and 87.0% OOD under full fine-tuning.

2) The pretrained VLM encoder IS frozen; open-vocabulary generalisation is preserved (W3): We never train the VLM tokeniser/encoder from scratch. As shown in Figure 1 and §3.1, tokens come from a frozen pretrained VLM encoder (preserving unseen object/instruction semantics), feeding a trainable 32-layer Text Expert inside OmniJointAttention. Held-out demo_randomized explicitly tests unseen instruction paraphrases and clutter (§4.1), where our models reach 59.0% OOD (JEPA-DiT, 10,000 rollouts) and 65.1% OOD (JEPA-DiT-EEF), including 80%-85% OOD on semantic disambiguation tasks (place_cans_plasticbox 85%, pick_diverse_bottles 80%, place_a2b_right 80%; Table 9).

3) Single-variable predictive ablation & dataset fairness (W4):

  • Isolating the predictive stream (Tables 3 & 4a): Disabling only the predictive stream (lambda_p=0) on the identical backbone and 2,500 clean demos gives 35.3% -> 41.3% OOD at step 10k (bs=256), and 46.0% -> 59.0% OOD (+13.0 pts at matched 85% Clean) at peak convergence (bs=512, step 11k).
  • Protocol fairness: Every Table 3 row trains strictly on the 2,500 demo_clean demos (0 randomised). Works reporting >90% "Random" (FlowWAM, LingBot-VA, R^2-WAM) train on 50 clean + 500 randomised demos/task (in-distribution); our Wan-5B LingBot-VA reproduction gets 90.0% / 90.4% under mixed training, but drops to 68.8% Clean / 0.0%-14.0% OOD under clean-only training (§4.4).

2. Comparison with All 10 Suggested Works (W1, W2, W4; §2 & Appendix A)

  • Optical flow (FLIP, FlowWAM): Pixel flow relies on brightness constancy and degrades under lighting/shadow shifts (and FlowWAM uses mixed training), whereas delta-JEPA differences after V-JEPA invariance.
  • SSL/multi-stream WAMs (LDA-1B, EgoWAM, Flex-pi, R^2-WAM, Cosmos Policy, UVA): LDA-1B/EgoWAM use per-frame DINO or 3D flow, while Flex-pi/R^2-WAM fuse undifferenced features. Table 4b proves undifferenced fusion collapses on held-out OOD (2.5%-7.5%) without delta-JEPA + recon ⊥ jepa.
  • Planning JEPAs (DINO-WM, LeWorldModel): Both run test-time MPC on low-DoF tasks rather than generative language-conditioned WAM control across 50 bimanual tasks. (Mid-training is validated on 28.3M real-robot AgiBot windows in Table 7; physical shift tests are noted in Appendix F.)

3. Specific Questions & Edits (Q1-Q4)

  • Q1 (Warm-start init): Loads Text Expert and shared attention weights from a converged 2-stream (Text + Causal VAE) video-text DiT while JEPA Expert starts randomly; Table 2 adds a matched lr=5e-5 control (21.2x vs. 0.9x floor).
  • Q2 (L161 loss units): Frozen VAE (d=16) and V-JEPA (d=1664) have different feature scales, so raw MSEs L_recon and L_pred have incomparable units; only L_action shares normalized action units (§3.1).
  • Q3 (L233 "least stable"): Originally phrased as a rhetorical reader objection before refuting it; rewritten plainly in §3.3, with §3.4 moved to §4.2 and colloquialisms removed.
  • Q4 (delta-JEPA vs. raw JEPA): 40.0% vs. 2.5%-7.5% OOD (LoRA) and 87.0% OOD (full tune) in Table 4b.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Area Chair and All Reviewers

We sincerely thank the Area Chair and all three reviewers for their constructive feedback. We have uploaded a substantially revised manuscript (strictly within the 10-page main text limit, with appendices consolidated from A-X into A-F) addressing every experimental, architectural, and presentation concern.

Summary of New Experiments and Key Revisions:

  1. Direct 4-Stream Delta-JEPA vs. Absolute-JEPA Ablation (Reviewer xqnx W2/Q3, Reviewer 9ebS W4/Q4; Table 4b, §4.5): Holding the 4-stream architecture, frozen encoders, and mutually blind mask (recon ⊥ jepa) strictly fixed, predicting absolute V-JEPA features (phi(g)) collapses to 2.5%-7.5% OOD due to cross-stream static redundancy, whereas delta-JEPA (phi(g) - phi(g-1)) reaches 40.0% OOD under matched LoRA (+32.5 OOD pts, >5x) and 87.0% OOD under full fine-tuning.
  2. Matched Same-Backbone VAE-Only Baseline, Horizon Curve & Clean-Only LingBot-VA (Reviewer xqnx W1/Q1/Q2, Reviewer 9ebS W4; Tables 3 & 4a/c, §§4.4-4.5):
    • On our exact 32-layer backbone and pretraining lineage, disabling only the predictive stream (lambda_p = 0, JEPA-DiT-ReconOnly) peaks at 85.0% Clean / 46.0% OOD, while adding delta-JEPA reaches 85.2% Clean / 59.0% OOD (+13.0 OOD pts at matched clean accuracy across 10,000 official rollouts at 100 trials/task).
    • Extending clean-only training from 8k to 13k steps raises Clean (84.0% -> 85.6%) while degrading OOD past 11k (59.5% -> 49.3%), proving longer training overfits clean appearance rather than closing the OOD gap.
    • Reproducing LingBot-VA (Wan-5B) confirms mixed training (with 500 randomised demos/task in-distribution) reaches 90.0% / 90.4%, whereas clean-only training drops to 68.8% Clean / 0.0%-14.0% OOD.
  3. Unified Co-Evolution Narrative, Matched-LR Controls & Stronger Standalone Predictive Model (Reviewer fa7H W1, Reviewer 9ebS W3/Q1; Table 2, §4.3):
    • At matched bs=512, lr=3e-5, standalone from-scratch 3-stream JEPA-DiT-PredOnly (zero VAE tokens) reaches 76.0% Clean / 35.2% OOD across all 50 tasks (updating the early unaligned 63.3% / 11.9% run).
    • We add matched-LR (lr=5e-5) AgiBot mid-training controls (21.2x vs. 0.9x language-attention floor) and ego visual-convergence controls (Delta cos = 0.486 vs. 0.495), and retitle the paper to reflect co-evolution and dual-stream complementarity.
    • Scaling multi-source mid-training with 34-DoF EEF control (JEPA-DiT-EEF) reaches 90.3% Clean / 65.1% OOD / 77.7% Avg, surpassing all published WAMs on clean, OOD, and average metrics.
  4. Self-Contained 10-Page Paper & Full Author Rewrite (Reviewer fa7H W2/W3, Reviewer 9ebS): Figure 1 now shows the full 4-stream architecture; core ablations, causal re-rendering, and attention plots reside in Pages 1-10 (Figures 1-3, Tables 1-4); 10 suggested related works are added (§2, Appendix A); and the entire manuscript was manually rewritten (0 em-dashes, down from 60+).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.