acceptodds
Under review as a conference paper at ICLR 2027

Qantara: Bridge-Flow Training for Multi-Paradigm JEPA Inference

Abstract

Joint-Embedding Predictive Architectures (JEPAs) underpin a growing family of latent world models for control from raw pixels. Published JEPA world models for pixel-based control (PLDM, DINO-WM, LeWM, V-JEPA-2-AC) each commit at training time to a single inference paradigm, trajectory optimisation in a learned dynamics model, so a checkpoint trained to plan cannot be read off as a policy without retraining. Deferring that choice to inference would let the constraints that bind at deployment, per-step latency and whether a goal observation can be supplied, pick the path. We present Qantara, an end-to-end JEPA of about 21M parameters whose joint training objective pairs a Brownian-bridge interpolant between consecutive clean latents on the state axis with noise-to-data flow matching on the action axis. One checkpoint serves three inference paradigms without retraining: latent planning, behaviour-cloning action sampling, and a video-inverse composition that predicts the next latent without action conditioning, then recovers the action bridging the two. Training samples the two noise levels along the edges of the (action-noise, state-noise) square, one edge per inference dispatch, so each path is trained where it is queried. Replacing that sampler with uniform sampling over the square at matched compute drops Push-T planning from 90.1 to 53.3 success rate (SR). On the LeWM control suite at a matched 10-epoch budget, Qantara averages 91.2 SR over the four environments and reaches 93.7 on OGBench-Cube, +7.7 over DINO-WM and +17.0 over LeWM, each measured against the stronger of that baseline's published score and our retrain of it. From the same weights the two goal-blind paths, which receive no goal image, reach 82.1-83.2 SR on Push-T and 70.8-72.8 on Cube at roughly 15-65x lower inference cost, and fall to near-noise on Reacher, whose released training corpus comes from a random policy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
JEPA, world models, visuomotor control, flow matching, Brownian bridge, goal-conditioned planning, behaviour cloning, inverse dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YJF4OpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Qantara trains a roughly 21M-parameter visual encoder and action–state transformer jointly, using the LeWM representation-learning recipe with SIGReg. Actions are learnt through Gaussian-to-data flow matching. State prediction uses a Brownian-bridge interpolant between consecutive latents and an endpoint-regression head with a zero-initialised residual. Training covers four edges and the diagonal of the action–state noise square.

One checkpoint per environment supports goal-conditioned CEM planning, goal-blind behaviour cloning, and a goal-blind video–inverse procedure that predicts a future latent before recovering an action.

The complete model performs well on the LeWM benchmark, particularly Cube. The main uncertainty is whether the experiments establish the benefit of the proposed formulation over simpler ways of combining latent prediction and action learning. There is also a specific error in the probabilistic description of the state sampler

Critical strengths
  1. The sampling ablation is useful and properly controlled. Replacing the five selected modes with five uniformly sampled copies, at the same replication budget, reduces Push-T planning from 90.1 to 53.3 and video–inverse performance from 83.2 to 49.7. This supports the proposed training distribution within the Qantara formulation.
  2. The Cube results are substantial. Qantara reaches 93.7, compared with 76.7 for the LeWM reproduction, 80.0 for the DINO-WM reproduction and 74.7 for the larger LeWM control. The Cube-Double results also favour Qantara’s planner: 49.6 against 36.9 and 36.3. These are the strongest empirical results in the paper.
  3. The shared checkpoint has useful capabilities. Planning and two action-generation procedures work without retraining or a pixel decoder at inference. Demonstrating this in a compact model trained from scratch is worthwhile.
  4. The reporting includes important negative results. The authors distinguish reproduced from published baselines, identify unstable seeds, explain the goal-blind limitation, and report that longer LeWM training removes the Reacher advantage. Most success-rate comparisons use three training seeds and 250 evaluation episodes per seed.
Critical weaknesses
  1. The paper does not establish the advantage over a simpler model with the same interfaces.

A natural comparison is a shared JEPA encoder with conventional heads for forward prediction, action generation, action-free prediction and inverse dynamics. Such a model would test whether the joint denoising formulation offers an advantage beyond learning several related tasks together. The mode-removal experiments do not answer this question: they remove supervision from Qantara while retaining its architecture and training recipe.

A frozen-LeWM controller is another relevant comparison. Nguyen et al. report strong goal-conditioned performance from a roughly 1.5M-parameter head. Their numbers cannot be compared directly with Qantara’s goal-blind paths, and their extra training cost excludes learning the LeWM backbone. Nevertheless, a reproduction under Qantara’s protocol would help establish the practical value of training everything jointly.

For the goal-blind paths, conventional BC and Diffusion Policy baselines with the same observations, history and data would be more informative than the current comparisons. The UVA reproduction is difficult to interpret: low error on its training trajectories establishes fitting, but does not explain its weak closed-loop performance. It should carry limited weight in claims about superiority over other policy-learning methods.

  1. Equal epochs do not provide an equal training-compute comparison.

Appendix B gives Qantara an effective batch of 640 through five-mode replication of 128 source rows. LeWM and PLDM use 128 source rows at the same optimiser-step count. The resulting training costs are not reported side by side.

This does not imply exactly five times the total computation: encoder work may be shared, and the architectures differ. It does mean that the comparison controls data exposure and optimiser steps without establishing equal computational cost.

The larger LeWM control addresses parameter count. The 20-epoch experiment addresses one longer-training setting and removes the Reacher margin. Neither resolves the overall compute comparison. Similarly, comparisons between one-, two- and five-mode models change the number of processed examples per update.

Please report measured training cost and compare the principal baseline and specialists at comparable budgets. The five-mode sampling comparison and comparisons among the four-mode rows remain informative because their budgets match.

  1. The contribution of the bridge remains unresolved.

Table 5 omits the most important control: a Gaussian source with the residual head retained. The explanation that this is “structurally unreachable” describes the implementation, rather than a conceptual restriction. Without this cell, the bridge has not been isolated in the configuration used by the successful model.

The residual comparison also changes the output-layer initialisation. It therefore tests the residual and initialisation together. The reported 12.7-point Reacher gain from the bridge is obtained with the residual disabled and cannot establish the same benefit in the default configuration.

The other ablations warrant a restrained interpretation. A single predictor call performs similarly to four calls, so iterative state refinement has little demonstrated benefit on these tasks. This does not show that bridge corruption during training is unnecessary. Likewise, the noise-scale sweep establishes sensitivity, particularly on Push-T, but does not identify why noise helps.

The mode-removal results support an empirical stability observation. They do not yet establish a general co-regularisation mechanism. With three seeds and occasional large failures, changes in optimisation or effective batch size remain plausible explanations. The claim should be stated at that level unless further controls separate these effects.

  1. Appendix G’s marginal-preservation claim is incorrect in general.

The state head is trained by squared-error endpoint regression. For a fixed representation and conditioning information, the optimal prediction is a conditional mean. Equation 12 substitutes that estimate for the unknown endpoint and adds fresh bridge noise. This does not generally reproduce the training bridge marginal.

For a simple counterexample, fix the current latent at zero and let the next latent be equally likely to be −1 or +1 under identical conditioning. Use a unit bridge-noise coefficient. The optimal prediction at the source is zero. At the halfway point, the training bridge has variance 1/2, whereas the first update of a two-step Equation 12 recursion has variance 1/4. The uncertainty in the endpoint is missing.

The authors should either provide assumptions and a derivation that justify the claim, or describe Equation 12 as a heuristic refinement procedure. This correction concerns the interpretation of the sampler; it does not invalidate the measured control results or the action flow-matching objective.

The video–inverse procedure has a related limitation: a conditional-mean future latent can lie between incompatible outcomes and be an unsuitable inverse-dynamics target. Equation 10 already acknowledges the point-estimate approximation. A discussion or controlled example of its failure would be sufficient.

  1. The deployment claims need a clearer success–cost comparison.

BC and video–inverse are distinct inference procedures, but both perform goal-blind imitation. Their success therefore does not establish that the checkpoint can preserve goal-reaching ability when switching from planning to a faster mode.

The paper explains why imitation succeeds in its evaluation: goals are later states of the trajectory supplying the initial state, Push-T has a shared task target, and success tolerances accommodate some deviation. This is a legitimate setting. It leaves the usefulness of the fast paths for independently specified goals largely untested.

The Reacher failure is consistent with this scope. Goal-blind imitation of random behaviour should not be expected to solve a precise goal-reaching task. A goal-conditioned inverse model succeeding on the same data would not contradict that explanation. Nor would querying Qantara’s local inverse model with a distant goal necessarily solve the problem.

The timing comparison also needs a stronger planning reference. Table 6 gives a suite average of approximately 90.7 with one state-predictor call, against 91.2 with four calls. Table 7 measures the four-call configuration. The reported 15–65× ratios are valid for those settings, but do not show the advantage over a cheaper planner with nearly the same success rate.

Please report success against measured runtime for the one-call planner and a modest CEM-budget sweep. Also separate amortised cost per environment step from latency when a new plan or action is actually computed. Both quantities matter, since CEM executes a 25-step buffer while the imitation paths compute every five steps.

Questions for authors
  1. How does Qantara compare with a simpler shared-encoder model or a frozen-LeWM controller under the same evaluation protocol and measured training budget?
  2. What happens when the Gaussian-source control retains the residual head and its initialisation?
  3. Do the main planning gains survive matching training cost, particularly on Cube and Two-Room?
  4. Can the marginal claim following Equation 12 be justified, or will the sampler be presented as heuristic?
  5. What is the measured success–runtime trade-off for one versus four state-predictor calls, including latency at an actual planning decision?
5 comments

Response to Reviewer YJF4 (1/2)

Bottom line. We trained the simpler alternatives you named under our protocol. A shared-encoder LeWM with conventional heads, at several times Qantara's FLOPs, reaches 25–30 on the within-corpus Push-T goal-blind paths against Qantara's 83; its planner trails on Reacher (15.8–18.2), is not separated on Cube seeds 11–55 and trails on seeds 101–105 (5.3–6.7), is level on Two-Room and 2.6–3.5 ahead on Push-T. Qantara costs 1.07× LeWM's FLOPs per step. With the residual kept, the bridge beats a Gaussian source on Push-T and Reacher, and the Gaussian source is 3.7 higher on Cube (W3). Equation 12 is now a heuristic (W4).

W1 / Q1: a simpler shared-encoder model, a frozen-LeWM controller, and policy baselines. We took Qantara's protocol and trained all three: the shared-encoder LeWM with conventional heads, the frozen-LeWM controller, and the policy baselines. It turned out that on Push-T, Qantara's goal-blind paths are ahead of each goal-blind alternative among them.

Shared encoder with conventional heads (general §4: LeWM with forward, BC, action-free and inverse heads, MLP or flow BC, at Qantara's optimiser steps and predictor rows, ≥4.7× its FLOPs, head losses weighted 1, 5 seeds). Its Push-T policy paths are far below Qantara's and its planner is below on Reacher; it is level on Two-Room, not separated on Cube at five seeds and above on Push-T planning.

Qantara minus it (MLP / flow BC)Push-TTwo-RoomCubeReacher
planning−2.6 / −3.5tienot separated+15.8–18.2
BC, video–inverse53–58 (within-corpus pairs)video–inverse about 6

Frozen-LeWM controller and policy baselines. Heads on the frozen latents of our 10-epoch LeWM (50 epochs at batch 640) and policies with the same observations, history and data, 3 seeds each. With a goal, our variant (general §6), from one training stage, is level with the goal-conditioned heads on Two-Room and Cube, ahead of the frozen-LeWM one on Reacher and below it on Push-T.

SR, Push-T / Two-Room / Cube (/ Reacher)
goal-blind BC, frozen LeWM19.9 / 64.4 / 64.5 / 4.1
inverse dynamics, frozen LeWM26.7 / 59.3 / 51.3 / 4.1
Qantara goal-conditioned variant93.0 / 99.8 / 100.0 / 68.2
goal-conditioned inverse dynamics as in Nguyen et al., frozen LeWM95.7 / 100.0 / 100.0 / 40.0
Nguyen et al., as reported (released LeWM, their recipe, n = 50)84.7 / 100.0 / 99.3 / 100.0
MLP BC21.6 / 63.7 / 64.4
Diffusion-Policy-style flow-matching chunk policy26.5 / 61.2 / 66.5
Qantara BC path (5 seeds)83.1 / 68.6 / 67.0

The revision scopes the UVA row (14.4) to the suite's protocol (72gF Q5).

W2 / Q3: equal epochs are not equal compute. The revision reports training cost: 1.07× LeWM's FLOPs per step (shared encoder pass) and 1.0–1.1× its wall time. More compute does not close the Cube gap, and LeWM keeps its Push-T planning lead, as in the paper (general §1–3).

CEM SRLeWMQantara
Cube: plain LeWM at 5× steps or 5× batch (better of two LRs); Qantara at LeWM's predictor-row count72.9–73.893.5, +20.9 [+18.5, +23.0] over LeWM at its own batch (72.6), same seeds
Two-Room, LeWM at 5× batch76.5100.0
Push-T, LeWM at its own batch91.388.7

At matched rows the five-mode model leads a forward-only model by +5.4 [+3.1, +8.5] on Push-T and +4.8 on Two-Room.

Response to Reviewer YJF4 (2/2)

W3 / Q2: Gaussian source with the residual head and its initialisation kept. We took the variant you proposed and tested it: the next latent is generated from pure noise (a Gaussian source) instead of the bridge, while the residual head and its zero initialisation stay as in the original model. We also ran the same model with standard instead of zero initialisation of the residual head, and two variants that start from the current latent zₜ, one without noise and one with noise. All runs used peak LR 1e-4 and the same 5 seeds (table in general §5). In the submitted configuration, the bridge beats the start from noise on Push-T by 6.2 points [+4.6, +8.0] and on Reacher by 10.7 [+5.2, +16.1]. On Reacher, the bridge also beats both starts from zₜ. On Push-T and Cube, the start from zₜ without noise does as well as the bridge. On Cube, the start from noise is 3.7 higher than the bridge. With 60 seeds for the full recipe and three drop-one arms, the revision states the mode-removal results as an empirical stability observation, as you suggest. With the residual kept at LR 1e-4, the default initialisation moves CEM by at most 0.3 on Push-T and Cube; on Reacher the initialisation comparison is inconclusive at 5 seeds. The revision states the bridge result per environment, replaces the "structurally unreachable" sentence with these cells, and reports the γ sweep as inconclusive.

W4 / Q4: the marginal claim after Equation 12. Equation 12 does not preserve the training-bridge marginal in general. We checked your counterexample numerically, and it holds: with 2,000,000 samples, the bridge-midpoint variance is 0.4995 (exact 0.5) and after the first of two recursion steps 0.2497 (exact 0.25), because the conditional mean replaces the uncertain endpoint. The revised Appendix J (was G) gives this counterexample, presents Equation 12 as a heuristic refinement and drops the marginal-preservation sentence. After Equation 10 we added that a conditional-mean next latent can fall between incompatible outcomes and be a poor inverse-dynamics target.

W5 / Q5: goals, success–cost trade-off, and latency at a decision.

Held-out start–goal pairs (goal 25 steps later in the same episode). The planner scores 87.3 and 90.7 on unseen pairs; the goal-blind paths hold on Cube and drop on Push-T, for a policy-only specialist too (−50.3 vs −46.3), so the drop is not specific to the joint model.

3 train seedsCEMBC / video–inverse
Push-T, 21 episodes neither model trained on87.3 ± 1.235.3 / 34.9; four-head LeWM 15.1–21.5 (train seeds 11, 22, 33), 13.7–20.3 below (revised App. E)
Cube, 200 freshly collected episodes90.769.5 / 73.9
Cube, goals from another episode25.6 vs LeWM's 18.3

The Limitations state that the goal-blind Push-T numbers are within-corpus.

  • A goal-conditioned fast path (on the suite's start–goal protocol). The goal-conditioned variant (general §6) runs through the BC path with one extra input and two denoising steps per action chunk (not timed separately) and reaches 93.0 / 99.8 / 100.0 / 68.2, above the same checkpoints' planner on Push-T and Cube.
  • One-call planner and CEM budget. With one predictor call, CEM scores 89.1 / 100.0 / 93.3 / 80.5 (suite 90.7 vs 91.2 with four; revised Table 9). On a fixed K × N × iteration grid, the cheapest CEM setting within 3 SR of the best one without BC seeding is K = 1 on Push-T, Two-Room and Cube, and K = 4 with 10 iterations on Reacher. Against it, the goal-blind paths are 7–57× cheaper per decision and 1.4–11.5× amortised per environment step; at the default planner, 10–19× amortised, and one CEM planning call takes 4.8–12.7 s (five episodes batched). The revision tabulates each grid point's success rate and plan-call time, with the Pareto set marked.

Update: the four-head LeWM on Cube, five further seeds. We ran both models again, new runs from one code tree, paired by seed. On five Cube seeds (101–105), Qantara's CEM gets 90.5 ± 0.5, against 83.8 ± 1.5 with MLP heads (H1) and 85.2 ± 0.3 with the flow BC head (H2). So the difference is +6.7 [+5.3, +8.3] and +5.3 [+4.6, +5.9] (95% paired bootstrap over seeds), with both intervals above zero. On the other seeds, 11–55, the difference did not separate (revised App. E).

Reviewer_JBfPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Qantara is an end-to-end pixel JEPA world model trained offline on trajectory data using three distinct loss components, following a simplified training recipe that omits common stabilisation mechanisms. It employs a block-causal transformer to jointly predict action and next-latent outputs for each block, with a specialised sampling scheme for noise levels that covers multiple operational modes and expands the effective batch size accordingly. A single trained checkpoint enables several different capabilities, including goal-conditioned latent planning, goal-blind behaviour cloning sampling, and a combined video-inverse inference mode. Evaluated across a set of standard robotic environments, the model shows strong success rates for planning, while its goal-blind variants achieve competitive performance at substantially lower computational cost; ablation studies validate the importance of the proposed noise sampling strategy and the joint multi-mode setup, which the authors refer to as trunk co-regularisation. Additional supplementary experiments reveal context-dependent performance comparisons against prior methods, with some performance gaps closing when baseline models are given longer training budgets, and other established approaches struggling on the same training data.

Critical strengths
  1. Clear and practical goal: one compact (~21M) JEPA checkpoint that supports planning, BC and inverse-dynamics queries without retraining.

  2. A simple, well-motivated sampling design (edges plus diagonal of the noise square) with a large effect against uniform sampling on Push-T.

  3. Strong planning results on Cube (93.7; +17.0 over the in-pipeline LeWM, +13.7 over the in-pipeline DINO-WM), on Two-Room, and on Cube-Double. The baseline comparisons are applied conservatively.

  4. Rigour above the norm in several places: LeWM and PLDM retrained in-pipeline, 750 episodes per cell, the Reacher-corpus audit, and honest disclosure of the 20-epoch result.

Critical weaknesses
  1. The "three paradigms" claim is overstated. Both cheap paths are goal-blind, they succeed mainly through trajectory replay or a fixed goal, and they fail on Reacher. Video–inverse is statistically indistinguishable from BC on Push-T and Two-Room.

  2. No matched specialist baselines: separate policy plus world model at equal parameters and compute, heads on frozen LeWM latents, or GCBC in-pipeline.

  3. "Matched budget" matches optimiser steps, not compute. The 5× replicated batch confounds comparisons with LeWM and the Table 3 specialist rows.

  4. Novelty is incremental relative to UWM (point-mass edges when co-training on video), UVA masking, Diffusion Forcing and Uniform-t̄. Intermediate sampler variants are missing.

  5. The "co-regularisation" contribution is not supported: three seeds, the non-monotone {forward, policy} result, and widespread seed collapse point to a fragile recipe.

  6. The bridge's benefit at inference is not established: K = 1 ties K = 4, the factorial ablation is incomplete, and the Push-T results are mixed.

  7. The latency framing hides 75–330× per-decision gaps and depends on the CEM settings. The UVA comparison is not credible as reported. Several Table 8 entries are inaccurate or labelled inconsistently. There are minor table inconsistencies, and the headline K is the best of a sweep on the test set.

Questions for authors

Q1. Please report published-only and reproduction-only margins, suite averages with Reacher excluded or at 20 epochs, and the Push-T deficit in the abstract and conclusion.

Q2. What are the measured GPU-hours and FLOPs per step for Qantara vs the LeWM reproduction? Is the encoder pass shared across the five replicas? Can you add LeWM at 5·B and Qantara at a total batch of B?

Q3. Can you add matched specialist baselines (separate ~10M policy plus ~10M world model; BC and inverse heads on frozen LeWM latents; GCBC in-pipeline) and a goal-conditioned cheap path, or BC used as CEM's proposal distribution?

Q4. Can you evaluate the goal-blind paths with goals decoupled from the start trajectory, and clarify how the Push-T goal is defined? Can you add sampler variants (edges plus p·interior, uniform plus point mass, Beta near edges, i.i.d. per-axis levels, Uniform-t̄), and explain

Q5. why BC survives uniform sampling while CEM collapses?

Q6. Can you run ≥10 seeds on the key Table 3 rows, and add loss and latent-rank curves, gradient-conflict analysis, re-weighting at matched compute, and a frozen-encoder control? How do you reconcile the stable {forward, policy} row with the claimed co-regularisation?

Q7. Can you complete the bridge × residual factorial, and add a deterministic noise-augmentation baseline? Why not deploy K = 1?

Q8. Can you report per-decision latency with cadences, and state whether the 50-step budget counts environment steps or decisions?

Q9. For UVA: what resolution, action convention, chunk/frame-skip mapping, history length and training stages did you use? Can you reproduce UVA's ~0.98 under its own protocol, and add a Diffusion-Policy baseline in your harness?

Q10. Can you correct Table 8: consistent "no" vs "not shown" labels, sourced parameter counts, Cosmos Policy's value-guided planning, and UVA's goal-conditioned policies?

3 comments

Response to Reviewer JBfP (1/3)

Bottom line. Qantara costs 1.07× LeWM's FLOPs per step, and its Cube lead holds at LeWM's predictor-pass count (93.5 vs 72.6). The baselines you asked for are added under our protocol: a separate ~10M policy plus ~10M world model, heads on frozen LeWM latents, and GCBC in our pipeline. The fast paths stay goal-blind; a separately trained variant adds the goal. The sampler result holds at 20 seeds per sampler (−33.8 for uniform), with four of the variants you listed. The three drop-one arms run at 60 seeds do not separate in collapse rate, so the co-regularisation claim is replaced by the compute-matched analysis.

Q1: margins, suite means, Push-T deficit. If we average over the four environments, Qantara is better than each baseline, both with and without Reacher. On Push-T, Qantara trails LeWM: by 2.2 against our reproduction and by 5.9 against the published number. Qantara's lead per baseline, Table 1 (3 seeds; negative = behind):

Push-TTwo-RoomCubeReachersuite meanwithout Reacher
Published LeWM−5.9+13.0+19.7−5.191.2 vs 85.894.6 vs 85.7
Published DINO-WM+16.10.0+7.7+1.991.2 vs 84.894.6 vs 86.7
LeWM reproduction−2.2+10.1+17.0+16.691.2 vs 80.894.6 vs 86.3

With Reacher at 20 epochs for both (79.7 vs 80.0), the means are 90.9 vs 84.7. The revised abstract and conclusion state the Push-T deficit (90.1 vs 92.3 retrained, 96.0 published).

Q2 / W3: training cost. Yes, the encoder pass is shared, so a Qantara step costs about as much as a LeWM step: per step Qantara encodes 128 clips and runs 640 predictor rows, LeWM 128 and 128. "Matched budget" meant optimiser steps (57k–155k per 10-epoch run at B = 128); the revised Setup adds these costs.

GFLOPs per stepH100 h per 10-epoch run, Push-T / Two-Room / Cube / Reacher
Qantara5,730 (1.07×)9.26 / 3.24 / 8.98 / 8.30
LeWM5,3718.33 / 3.17 / 8.48 / 8.48
LeWM at batch 64026,855

Giving LeWM 5× the batch does not close the gap on Two-Room or Cube and lowers its own Push-T score (general §1–3): LeWM at 5·B scores 85.0 / 76.5 / 72.9 / 77.3 on Push-T / Two-Room / Cube / Reacher (better of two LRs on Push-T and Cube) against Qantara's 88.7 / 100.0 / 83.8 (same seeds; general §2) / 78.6. Qantara at a total of B (26 clips × 5 modes = 130 predictor rows) scores 93.5 on Cube, +20.9 [+18.5, +23.0] over LeWM on the same seeds, and 77.4 on Push-T at LR 1e-4.

Q3 / W2: matched specialists and baselines. On Push-T the joint checkpoint leads the matched specialists on planning and video–inverse and is level with them on BC. Five seeds unless stated.

  • Full-size specialists (only their path's modes, ≥ 640 rows), Push-T: planning 88.7 vs 83.3, video–inverse 83.1 vs 23.1, BC 83.1 vs 84.5; on Cube none is separated.
  • ~10M policy plus ~10M world model (depth-3 predictors, 10.1M each; joint 21.6M), Push-T: the joint checkpoint is ahead of the world model (+4.5 [+2.7, +6.2]) and level with the policy (83.1 vs 81.1); Cube is inconclusive.
  • BC as CEM's proposal (3 seeds): +5.6 [+4.7, +6.5] on Push-T at K = 1, N = 100 (88.8 vs 83.2) at equal planning time; no material change elsewhere (Cube inconclusive).

Policy baselines (3 seeds; same observations, history and data) score below Qantara's BC path on Push-T; only the goal-conditioned GCBC is above it on Two-Room and Cube.

SR, Push-T / Two-Room / Cube (/ Reacher)
Qantara BC (5 seeds)83.1 / 68.6 / 67.0
in-pipeline GCBC42.3 / 100.0 / 85.6
goal-blind MLP BC21.6 / 63.7 / 64.4
flow-matching chunk policy26.5 / 61.2 / 66.5
goal-blind BC on frozen LeWM latents19.9 / 64.4 / 64.5 / 4.1
inverse dynamics on frozen LeWM latents26.7 / 59.3 / 51.3 / 4.1

The suite authors' GCBC (75 / 100 / 84 in Table 2) uses DINOv2 patch embeddings; ours learns from pixels.

Response to Reviewer JBfP (2/3)

Q4 / W1: decoupled goals; the Push-T goal. Each Push-T episode starts at a corpus state, and the goal is the 5-D state 25 steps later in that trajectory; success means agent and block position within 20 units and block angle within π/9, within 50 environment steps. The green marker in the render is fixed and is not the goal; masking it changes BC and video–inverse by at most 2. On unseen start–goal pairs (same-episode goals) the planner scores 87.3 on Push-T and 90.7 on Cube (LeWM 92.3 and 68.5); the goal-blind paths hold on Cube and drop on Push-T.

3 train seedsCEMBCvideo–inverse
Push-T, 21 episodes neither model trained on87.3 ± 1.2 (LeWM 92.3)35.334.9
Push-T, 100 training episodes, same conversion81.682.8
Cube, 200 freshly collected episodes90.7 (LeWM 68.5)69.573.9
Cube, goals from another episode25.6 ± 0.9 (LeWM 18.3 ± 0.5)3.23.9

A goal-blind policy-only specialist falls by a similar amount on Push-T (−50.3 vs −46.3), so the drop is not specific to the joint checkpoint. The goal-blind paths do not see the goal; the separately trained goal-conditioned variant (general §6) takes it as an input.

On W1: in the submitted checkpoint the planner is the goal-reaching path and the two goal-blind paths serve when no goal image is available. Video–inverse is 1.1 above BC on Push-T and 2.0 on Cube (Table 2); the claim is only that one checkpoint serves both. On Reacher, as the abstract states, the goal-blind paths fall to near-noise on a random-policy corpus; the planner reaches 80.9.

Q5 / W4: novelty and sampler variants. The cited methods, on action-labelled data, draw the noise level for the action and for the next latent independently and uniformly, and that is exactly our uniform-sampling baseline in Table 3 (UWM: action and observation timesteps independent and uniform, the action timestep fixed at its maximum only for action-free video; Diffusion Forcing: per-token levels i.i.d. uniform; UVA: random masking of future-frame tokens). This is precisely where our sampler differs from them: sampling only along the edges and the diagonal beats their scheme by 33.8 points on Push-T planning. The intermediate variants you asked for split on one feature, whether the mass stays on the edges: the two where it stays are within 2 of ours, and the two where it leaves the edges lose 11–12.

Variant minus edges and diagonalPush-T CEMother paths
uniform, 20 seeds per arm, no collapse−33.8video–inverse −26.0; Cube −3.8 (10 seeds)
edges plus 10% interior+0.8Cube +1.6
uniform square, point mass on each axis end+1.8 [+0.0, +3.9]
Beta near the edges: Beta(1, 10) at 0, Beta(10, 1) at 1−12.3 [−15.9, −8.7]video–inverse −7.4
Uniform-t̄ without reweighting−11.1 [−13.3, −8.2]

The four run on the same five Push-T seeds at LR 1e-4. Uniform-t̄ shares our principle of matching training noise levels to inference, for the mean level over many image patches; our queries sit on the edges and diagonal of a two-variable square, so that is where we put the mass. BC is unaffected by the sampler (−0.4 [−1.6, +0.7]) while planning and video–inverse fall, and those two are the paths that consume a predicted next latent: the planner chains forward-edge predictions over its rollout, video–inverse feeds the video edge's prediction into the inverse edge, and BC holds the state slot at zₜ and reads one denoised action.

Q6 / W5: seeds. We ran 60 Push-T seeds for the full model and three variants without one mode each, and their collapse rates do not separate: 11 collapses for the full model, 21 without policy (Holm p = 0.093), 17 without joint (0.28), 10 without inverse; without video, 6 of 20 seeds; all modes with 20% less supervision, 11. So the revision drops the co-regularisation claim: we report the collapse rates, and contribution (3) is now the compute-matched analysis. At these rates three seeds cannot separate the variants, even for the stable {forward, policy} pair. At peak LR 1e-4 no variant collapses: full 0 of 40, without policy or joint 0 of 12 each. Loss and rank curves, gradient conflict, re-weighting and a frozen encoder would test a mechanism we no longer claim, so we skip them.

Response to Reviewer JBfP (3/3)

Q7 / W6: bridge factorial, noise baseline, K. General §5 now has the 2×2 at LR 1e-4 on Push-T and Reacher (Cube: residual on only); with the residual off the bridge still beats a Gaussian source (+3.0, +8.2). Your deterministic and noise-augmentation baselines are the plain-zₜ and zₜ-plus-noise inputs; the bridge beats both on Reacher (+6.9, +5.8) and is within 2 of both on Push-T and Cube. On a fixed K × N grid, the cheapest CEM setting within 3 SR of the best one without BC seeding uses K = 1 on Push-T, Two-Room and Cube; on Reacher K = 1 falls 3.1 short. The revision keeps K = 4 as the default, as in Table 1, reports K = 1 beside it (suite 90.7 vs 91.2), and gives the grid in the appendix. K was picked on the test setting; the suite means over the tested K span 89.8–91.2.

Q8 / W7: latency. The 50-step budget counts environment steps; CEM plans once per 25, the goal-blind paths act once per 5-step block. Against that CEM setting, the goal-blind paths are 7–57× cheaper per decision and 1.4–11.5× amortised. At the default planner, amortised: 10–19× in our timing runs, and one CEM planning call takes 12.7 / 6.1 / 4.8 / 4.9 s on Push-T / Two-Room / Cube / Reacher on our stack (five episodes batched).

Q9: UVA. 96 px observations, relative actions in [−1, 1], 16-frame history, 8-action open-loop chunks of one environment step each (about 7 replans per 50-step episode; our BC acts once per 5-step block), two stages, 248k steps (UVA's own: 247k). Retrained on its own data and scored by its own evaluator: 0.956 (released 0.943; published 0.98, best over epochs). Under our protocol: 14.4; per-step replanning adds 5.6, and the row is scoped to this protocol.

Q10: Table 8 (revised Table 12). "Not shown" where a capability is not demonstrated; footnotes for Cosmos Policy's value-guided best-of-N and for UVA's goal-conditioned policies with reward re-ranking; each parameter count names its configuration (UVA 0.5B on Libero10, PAD XL/2 0.66B, Cosmos Policy from 2B Cosmos-Predict2); "VAE latent" for UVA, PAD, UWM and Cosmos Policy. On the table inconsistencies: the LeWM Cube entry of Table 1 now reports the population std (76.7 ± 1.0), the paper's convention.

Reviewer_72gFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Qantara, a small JEPA world model (about 21M parameters) trained from pixels that supports three inference modes using the same weights. It can plan with CEM towards a goal image, directly sample actions as a goal-blind BC policy, or predict the next latent without actions and then recover the action through an inverse dynamics query. The predictor uses interleaved action and latent tokens. Actions are trained with standard flow matching from Gaussian noise, while the next latent is trained with a Brownian bridge starting from the previous latent.

The main design choice is how the two noise levels are sampled during training. Rather than sampling uniformly over the unit square, the method samples only along the four edges (one for each inference query) and the diagonal. On the four LeWM-suite tasks, the planner achieves an average SR of 91.2, with the largest gain on OGBench-Cube (93.7 vs. 76.7 for their LeWM retrain). The goal-blind modes are 15–65× cheaper and achieve around 82–83 on Push-T and around 70 on Two-Room and Cube. The ablations cover uniform vs. edge sampling, dropping individual modes, the residual head, gamma, and the number of denoising steps.

Critical strengths
  1. The evaluation is careful by the standards of this area. Baselines are retrained in the same pipeline, every cell is 750 episodes over 3 training seeds, and the authors report things that hurt their own story, like the Reacher margin going away at 20 epochs, the released Reacher data actually coming from a random policy, and the goal-blind paths failing on Reacher. I trust these numbers more than I usually do in JEPA world model papers, and the code is included.

  2. Edge-aligned sampling is simple and the effect is big. Switching to uniform sampling at matched compute drops Push-T planning from 90.1 to 53.3 and video-inverse from 83.2 to 49.7. Anyone training a joint state/action denoiser can reuse this, whether or not they care about JEPAs.

Critical weaknesses
  1. The gains over LeWM are not compute-matched, and an obvious baseline is missing. Since each sample is replicated across five modes, Qantara gets 5× as many predictor passes per step as the LeWM retrain. Matching 10 epochs means matching the amount of data seen, not the compute. The 1-mode forward model in Table 3 (planning only, 1× compute) performs no better than LeWM (62.0 on Push-T and 76.3 on Cube), so it is unclear whether the gain on Cube comes from the objective or simply from the extra compute. I would also like to see LeWM jointly trained with BC and inverse dynamics heads on its latents, as this seems like the simplest way to support all three paradigms with a single checkpoint.

  2. The claims about the bridge and co-regularisation are not well supported. The state head regresses the clean endpoint, and the K sweep is nearly flat, so the bridge seems to act mostly as input noise around a residual predictor. The bridge is only ablated with the residual off, where it actually hurts performance on Push-T (74.1 vs. 86.5). The co-regularisation result is only shown on Push-T with 3 seeds, and Appendix C shows that the 3-mode model was stable under the earlier λ_z=1 setting (84.4 ± 0.9). This makes me wonder whether the observed effect is mainly due to tuning. Since these are two key aspects that distinguish the method from UWM-style per-modality noise, stronger evidence is important for the novelty claim.

Minor. The paper is quite dense, and the results sections contain long strings of numbers, which made them difficult to read. The Figure 3 caption refers back to Fig. 3 itself, and calling the action-marginal latent prediction the "video" edge is a bit confusing, since no video is actually decoded.

Questions for authors
  1. Could you provide a compute-matched comparison between LeWM and the 1-mode forward model trained with 5× the batch size or 5× the training steps, at least on Cube and Push-T? If the gap on Cube largely holds, I would be willing to raise my score.

  2. Please add a baseline with LeWM jointly trained with BC and inverse dynamics heads (a small flow head would be fine) on Push-T, Two-Room, and Cube.

  3. Could you test the residual on with the bridge off (using a Gaussian source or plain ztz_t input without noise) under the default training recipe? This would help clarify whether the bridge itself matters.

  4. For the drop-one rows, have you tried any retuning (e.g., λz\lambda_z or the learning rate) or running more seeds? I would like to see whether the co-regularisation effect still holds.

  5. Clarification: UVA achieves 14.4 SR compared to 1.6 for random, which is much lower than what UVA reports on Push-T in its original setting. Given that it fits the training data well, could you clarify what differs in the evaluation protocol (e.g., action chunk execution, frame skip, resolution, or start states)? As it stands, I would be hesitant to draw conclusions from this result.

6 comments

Response to Reviewer 72gF (1/2)

Bottom line. Q1 asked whether the Cube gap holds at matched compute. At LeWM's predictor-pass count (130 against 128) Qantara scores 93.5 against LeWM's 72.6 on the same seeds, and LeWM at 5× steps or 5× batch stays at 72.9–73.8. LeWM with BC and inverse heads scores 76.9 on Cube; the full four-head model is not separated from Qantara on seeds 11–55 and trails it by 5.3–6.7 on seeds 101–105 (follow-up below). On Push-T, LeWM keeps its planning lead at its own batch, and the bridge result is stated per environment (W2).

W1 / Q1: compute-matched comparison on Cube and Push-T. On Cube the gap essentially holds. If we give Qantara the same effective batch as LeWM, that is, as many predictor passes per step (26 clips × 5 modes = 130 against 128), Qantara's success is 93.5 and LeWM's is 72.6, on the same seeds. The one-mode forward model, with just the forward mode at a 5× larger effective batch, gets 78.2, while plain LeWM at 5× steps or 5× batch gets 72.9–73.8, so still lower. On Push-T, LeWM at its own batch keeps a slight planning lead, as the paper says. Training Qantara costs only a little more: 1.07× LeWM's FLOPs per step, so only 7% more, because the five modes all share the encoder pass. Wall time per step is at most 1.1×, so about 10% more.

CEM SRFLOPs/step vs LeWMCubePush-T
LeWM, batch 128, 10 epochs1.0×72.6 ± 2.191.3 ± 1.0
LeWM, 5× steps (50 epochs)1.0×73.8 ± 1.289.4 ± 1.1
LeWM, 5× batch (640), better of LR 5e-5 and 2.5e-45.0×72.9 ± 0.885.0 ± 1.4
1-mode forward model, 640 predictor rows (Table 3: 128)78.2 ± 1.283.3 ± 3.0
Qantara at LeWM's row count (26 clips × 5 modes = 130 rows)< 1× (est.)93.5 ± 1.577.4 at LR 1e-4 (13.4 at 3e-4)
Qantara, submitted recipe, the same five seeds1.07×83.8 ± 17.388.7
Qantara, full recipe, further seeds1.07×91.3 ± 1.6 (20 seeds, LR 3e-4, none below 70)88.3 at LR 1e-4 (40 seeds)
Difference, same seeds [95% interval]CubePush-TTwo-Room
Qantara at LeWM's row count − LeWM+20.9 [+18.5, +23.0]
forward model − LeWM at 5× batch+5.4 [+4.0, +6.6]−1.7 [−4.8, +1.0]
five-mode − forward model+5.5 [−12.0, +15.0]+5.4 [+3.1, +8.5]+4.8 [+3.0, +6.6]

The ladder uses release 0.1.1 of the suite's evaluation library and LeWM at predictor dropout 0, Qantara's setting; Table 1 used 0.0.6 and LeWM dropout 0.1, and under 0.1.1 its Cube checkpoints score 72.8 (LeWM) and 91.5 (Qantara).

W1 / Q2: LeWM jointly trained with BC and inverse-dynamics heads. We ran two versions. With MLP inverse and BC heads alone, LeWM's planner mean is below Qantara's in every environment (table). The full conventional model (forward, BC, action-free and inverse heads, as MLPs or with the small flow BC head you suggested; general §4) trained at Qantara's steps and predictor rows and ≥4.7× its FLOPs, falls far below Qantara's Push-T policy paths and plans below Qantara on Reacher; it is level on Two-Room, not separated on Cube at five seeds and 2.6–3.5 ahead on Push-T planning.

CEM SR, 5 seedsPush-TTwo-RoomCubeReacher
LeWM + inverse and BC heads, 5× batch, LR 5e-5 (gradients reach the encoder)87.484.676.974.4
Full conventional model, against Qantara (MLP / flow BC)2.6 / 3.5 aheadtie83.6 / 84.3, not separated15.8–18.2 below
Qantara (same seeds)88.7100.083.8 ± 17.378.6

Its Push-T BC and video–inverse reach 25–30 against Qantara's 83 (revised App. E).

Response to Reviewer 72gF (2/2)

W2 / Q3: bridge off, residual on and off. In the end we ran four variants, all with the residual head kept on (general §5). Two are the ones you proposed: a Gaussian source, and plain zₜ without noise. We also decided to run two more: the same noise without mixing with the target, to check your hypothesis that the bridge acts mostly as input noise, and standard initialisation of the residual head instead of zero. It turned out that plain zₜ works no worse than the bridge on Push-T and Cube, but not on Reacher. Table 5's (revised Table 8) residual-off bridge row, 74.1 ± 12.3 against 86.5 for a Gaussian source, has one low seed, so these cells run at peak LR 1e-4, where no arm collapses. At that LR we also reran both arms without the residual (second row; interaction in general §5).

Bridge minus variant, same 5 seeds [95% interval]Push-TCubeReacher
Gaussian source+6.2 [+4.6, +8.0]−3.7+10.7 [+5.2, +16.1]
Gaussian source, both without the residual (Push-T, Reacher)+3.0 [+1.0, +5.2]—+8.2 [+1.4, +15.7]
plain zₜ / zₜ + bridge noise+0.1 / −1.8−0.4 / −1.8+6.9 / +5.8

Under the submitted recipe (LR 3e-4), the Gaussian source with the residual collapses on 0 of 20 Push-T seeds (80.6 ± 5.3) vs 11 of 60 for the bridge.

Q4: retuning and more seeds for the drop-one rows. Both. At the submitted recipe the drop-one arms' collapse rates are not separated, and at peak LR 1e-4 no tested arm collapses.

Push-T, submitted recipefulldrop-policydrop-jointdrop-inversedrop-video20% less supervision
collapses11/6021/6017/6010/606/2011/60
Holm-adjusted p vs full0.0930.28

The full recipe's rate is 18.3% [9.5, 30.4], with unconditional CEM 71.9 ± 32.4.

Retuning, Push-TcollapsesCEMvariant − full
full recipe, LR 1e-4, fresh seeds0/40 (one-sided Fisher p = 0.0024 vs 11/60)88.3 ± 2.7
drop-policy, LR 1e-40/1284.8 ± 8.5−3.5 [−9.0, +0.7]
drop-joint, LR 1e-40/1290.5 ± 1.2+2.2 [+1.2, +3.3]
λ_z = 1, submitted LR0/12 (one-sided p = 0.11)85.4 ± 2.1

λ_z = 1 is consistent with your Appendix C observation but not significant at that n. At 1e-4, Cube and Two-Room hold (91.2, 99.7) and Reacher is 11.4 lower, so the revision reports the 1e-4 results in the appendix and keeps Table 1 at the submitted recipe. The mode-set ablation reports these rates, and contribution (3) becomes the compute-matched analysis of your Q1.

Q5: UVA protocol. The 14.4 is UVA's two-stage recipe trained on the suite's Push-T corpus and scored under the suite's protocol: start at a corpus state and reach, within 50 steps, the 5-D state 25 steps later in that trajectory (agent and block position within 20 units, block angle within π/9). Actions are relative in [−1, 1], 224 px renders are downsampled to 96 px, and UVA keeps its 16-frame history and 8-action open-loop chunks. Its native setting scores coverage of a fixed target: our retraining on its own data reaches mean coverage 0.956, 47 of 50 episodes at ≥ 0.95 (released 0.943; published 0.98). In our protocol it fits the corpus open loop (action MAE 0.0156) and replanning every step adds 5.6, and the revision scopes the row to this protocol.

Minor. The Figure 3 continuation caption is fixed; new tables go into the appendices, and the main text states only the headline numbers. The paper defines the "video" edge at first use as next-latent prediction without actions; nothing is decoded.

Update: the four-head LeWM on Cube, five further seeds. We ran both models again, new runs from one code tree, paired by seed. On five Cube seeds (101–105), Qantara's CEM gets 90.5 ± 0.5, against 83.8 ± 1.5 with MLP heads (H1) and 85.2 ± 0.3 with the flow BC head (H2). So the difference is +6.7 [+5.3, +8.3] and +5.3 [+4.6, +5.9] (95% paired bootstrap over seeds), with both intervals above zero. On the other seeds, 11–55, the difference did not separate (revised App. E).

Comments on the paper

3 comments

General response: summary

In short. On Cube the gain is not from extra compute. Qantara gets about as many predictor passes as LeWM (130 against 128) and still scores 93.5 against LeWM's 72.6 on the same seeds. LeWM itself, given 5× more steps or a 5× larger batch, stays at 72.9–73.8. A Qantara training step costs only 7% more FLOPs than LeWM's, because all five modes share one encoder pass. The submitted recipe holds at 91.3 on 20 new seeds. We also ran a shared encoder with conventional heads, and it does not give this result: even at several times our compute, it trails Qantara's fast goal-blind paths (BC and video–inverse) on Push-T by 53–58 points on start–goal pairs from the corpus and by 13.7–20.3 on unseen pairs. The sampler result also holds: edges and diagonal beat uniform sampling on Push-T planning, at 20 seeds each.

QuestionAnswer
Is the gain over LeWM a compute effect? (72gF, JBfP, YJF4)Not on Cube or Two-Room. Cost 1.07× LeWM's FLOPs per step. Cube: at LeWM's predictor-row count Qantara scores 93.5, +20.9 [+18.5, +23.0] over LeWM on the same seeds; plain LeWM stays at 72.6–73.8 at 1×, 5× steps or 5× batch (the better of two LRs). The submitted recipe scores 91.3 ± 1.6 over 20 further seeds. Two-Room: LeWM 85.7 / 76.5 at its own / 5× batch, Qantara 100.0. Push-T planning: LeWM leads at its own batch (91.3 vs 88.7), as in the paper.
Would a shared encoder with conventional heads do as well? (72gF, YJF4)On the Push-T policy paths (within-corpus start–goal pairs), no: its BC and video–inverse reach 25–30 against Qantara's 83. On planning, at ≥4.7× Qantara's FLOPs, it is 15.8–18.2 below Qantara on Reacher, level on Two-Room, not separated on Cube at five seeds (5.3–6.7 behind at five more; 72gF thread), and 2.6–3.5 ahead on Push-T (general §4).
Does the sampler result survive more seeds? (JBfP)Yes: at 20 seeds per arm and LR 1e-4 (no collapse), uniform sampling (independent uniform noise levels, as UWM and Diffusion Forcing draw them) costs −33.8 [−36.2, −31.5] on Push-T planning (Table 3, at 3e-4: −36.8) and −26.0 on video–inverse.
Is training stable? (72gF, JBfP, YJF4)At peak LR 1e-4, 0 of 40 Push-T seeds collapse, with planning at 88.3 ± 2.7.
Can the fast path use a goal? (JBfP, YJF4)With a goal input, yes: a separately trained variant whose BC mode also takes the goal latent, with the same objective, reaches 93.0 / 99.8 / 100.0 / 68.2 on Push-T / Two-Room / Cube / Reacher.
Does the bridge matter, residual head on or off? (72gF, JBfP, YJF4)At LR 1e-4 the bridge beats a Gaussian source on Push-T and Reacher with the residual on (+6.2, +10.7) and off (+3.0, +8.2); on Cube, residual on, a Gaussian source is 3.7 higher.

Every new cell is 5 training seeds × 5 evaluation seeds × 50 episodes unless stated; ± is the population std over training seeds, and brackets are 95% bootstrap intervals over them.

Revised PDF (uploaded with this response; main text within 10 pages): measured training cost; appendices with the compute ladder, the conventional multi-head model, the policy baselines and frozen-LeWM heads; the goal-conditioned variant as its own row; the 60- and 20-seed arms; the bridge × residual cells; Appendix J (the submission's G) as YJF4 asked; inference cost per step and per decision.

General response (details): cost, compute-matched baselines, bridge, goal-conditioned path

1. Training cost. The five modes replicate only the predictor rows; the encoder pass, which dominates the cost, is shared. Per training step Qantara uses 5,730 GFLOPs and LeWM 5,371 (1.07×); LeWM at batch 640, Qantara's predictor-row count, uses 26,855 (5.0×). Wall time per optimiser step on one H100 is 1.0–1.1× LeWM's.

2. Cube under every budget we gave LeWM (CEM SR):

CubeFLOPs/step vs LeWMSR
LeWM, batch 128, 10 epochs1.0×72.6 ± 2.1
LeWM, 5× steps (50 epochs)1.0×73.8 ± 1.2
LeWM, 5× batch (640), better of two LRs5.0×72.9 ± 0.8
… plus inverse-dynamics and BC heads≥5.0×76.9 ± 0.8
… plus forward, BC, action-free and inverse heads (§4, H1 / H2)≥5.0×83.6 / 84.3
Qantara at LeWM's predictor-row count (26 clips × 5 modes = 130 rows vs 128; LeWM's optimiser steps, a fifth of its clips)< 1× (est.)93.5 ± 1.5
Qantara, submitted recipe, 20 new seeds1.07×91.3 ± 1.6 (0/20 below 70)
Qantara, the seeds paired with the LeWM rows1.07×83.8 ± 17.3 (4 at 91.6–93.2, 1 at 49.2)

At LeWM's row count Qantara is +20.9 [+18.5, +23.0] above LeWM on the same seeds, with fewer estimated FLOPs per step than LeWM.

3. Other environments. On Two-Room plain LeWM stays below Qantara at either batch; on Reacher Qantara leads LeWM at its batch and is level at 5× batch.

CEM SRTwo-RoomReacher (Qantara minus LeWM)
Qantara100.078.6
LeWM, batch 12885.768.1 (+10.6 [+7.3, +13.9])
LeWM, 5× batch76.577.3 (+1.4 [−2.5, +5.0])
LeWM + two / four heads84.6 / 99.7

4. Conventional multi-head model (72gF Q2, YJF4 Q1): LeWM with forward, BC, action-free next-latent and inverse-dynamics heads on shared latents, as MLP heads (H1) or with a small flow-matching BC head (H2). Batch 640 for 50 epochs: Qantara's optimiser steps and predictor rows, 5× its distinct clips, ≥4.7× its FLOPs; head-loss weights fixed at 1. Qantara minus H1 / H2, from unrounded means (bold: |Δ| ≥ 3 with the interval excluding 0):

CEMBCvideo–inverse
Push-T−2.6 / −3.5+57.6 / +55.1+56.0 / +52.7
Two-Room+0.3 / +0.4+4.5 / +1.2+6.2 / +5.8
Cube+0.2 / −0.6+2.4 / −1.2+4.0 / +3.7
Reacher+18.2 / +15.8near floornear floor

On within-corpus Push-T start–goal pairs its BC and video–inverse reach 25–30 against Qantara's 83. On Reacher the extra heads lower LeWM's own planner (60.5 / 62.9 against 77.3 without them).

5. Bridge with and without the residual head (72gF Q3, YJF4 Q2, JBfP Q7). CEM SR at peak LR 1e-4, where no arm collapses; the same 5 seeds per arm; Δ = full recipe minus variant, from unrounded means [95% bootstrap interval, arms resampled independently]:

full recipeGaussian sourceplain zₜ, no noisezₜ + bridge noise, no mixingdefault init
Push-T86.980.6; Δ +6.2 [+4.6, +8.0]86.8; Δ +0.188.7; Δ −1.887.2; Δ −0.3
Cube91.595.2; Δ −3.7 [−5.0, −2.3]91.9; Δ −0.493.3; Δ −1.891.6; Δ −0.1
Reacher67.356.6; Δ +10.7 [+5.2, +16.1]60.4; Δ +6.9 [+2.8, +11.3]61.5; Δ +5.8 [+0.6, +12.0]62.4; Δ +4.9 [−0.5, +11.0]

Without the residual (same LR and seeds, Push-T and Reacher; paired intervals): bridge 86.5 ± 0.9 vs Gaussian source 83.4 ± 1.9 on Push-T, Δ +3.0 [+1.0, +5.2]; 64.3 ± 2.4 vs 56.1 ± 6.7 on Reacher, Δ +8.2 [+1.4, +15.7]. Interaction (Δ with minus Δ without the residual): +3.2 [+0.2, +5.6] on Push-T, +2.5 [−6.4, +11.4] on Reacher.

6. A goal-conditioned policy path (JBfP Q3, YJF4 W5). We trained a separate Qantara variant with the goal latent as one extra input to its BC mode (same objective and sampler, peak LR 1e-4, 5 seeds, no collapse); it runs through the BC path. It reaches 93.0 ± 0.6 / 99.8 ± 0.2 / 100.0 ± 0.0 / 68.2 ± 14.6 on Push-T / Two-Room / Cube / Reacher, and 30.0 / 60.2 / 49.8 / 4.3 with its goal input shuffled across episodes (each episode still scored against its own goal). Its planner is level with the unconditioned recipe's (+1.4, +0.1, −0.5 on the first three).

Revised PDF: where the changes are

Main text (10 pages, the discussion-phase limit):

  • Abstract: training cost (1.07× LeWM's FLOPs per step), the Push-T deficit, and the 10–19× amortised speed-up.
  • §1: contribution (3) is now the compute-matched analysis.
  • §2.2: a conditional-mean next latent can be a poor inverse-dynamics target. §2.3: removing the joint diagonal gave an inconclusive increase in Push-T collapses.
  • §3.1: training cost per step (1.07× LeWM's FLOPs, 1.0–1.1× its wall time); Push-T collapses over 60 seeds; the peak-LR 1e-4 recipe (0 of 40 seeds collapse).
  • §3.2, Table 1: the LeWM Cube entry reports the population std.
  • §3.3: Table 2 adds the goal-conditioned variant column; the text adds a GCBC trained in our pipeline, latency against the cheapest CEM setting, and UVA retrained on its own data.
  • §3.4: the sampler result at LR 1e-4 (−33.8, 20 seeds per sampler); the mode-set ablation at 60 seeds, stated as an observation without a mechanism; K and γ.
  • §5: Limitations (goal-blind Push-T numbers are within-corpus).

Appendix (D–F are new; the submission's D–M are now G–P):

  • B: per-step FLOPs and wall time. C: the peak-LR 1e-4 recipe.
  • D (new): compute-matched ladder from LeWM to Qantara (Table 4).
  • E (new): conventional multi-head LeWM on all three paths (Table 5); the text adds Cube seeds 101–105.
  • F (new): policy baselines in our pipeline and heads on frozen LeWM latents (Table 6).
  • H (was E): residual and bridge ablations (Table 8); the text adds the peak-LR 1e-4 cells.
  • J (was G): Equation 12 as a heuristic, with the counterexample; the CEM K sweep.
  • L (was I): the CEM planner grid and per-dispatch timing (Tables 10–11).
  • References: updated venues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.