- Rating
- 3(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Qantara trains a roughly 21M-parameter visual encoder and action–state transformer jointly, using the LeWM representation-learning recipe with SIGReg. Actions are learnt through Gaussian-to-data flow matching. State prediction uses a Brownian-bridge interpolant between consecutive latents and an endpoint-regression head with a zero-initialised residual. Training covers four edges and the diagonal of the action–state noise square.
One checkpoint per environment supports goal-conditioned CEM planning, goal-blind behaviour cloning, and a goal-blind video–inverse procedure that predicts a future latent before recovering an action.
The complete model performs well on the LeWM benchmark, particularly Cube. The main uncertainty is whether the experiments establish the benefit of the proposed formulation over simpler ways of combining latent prediction and action learning. There is also a specific error in the probabilistic description of the state sampler
Critical strengths
- The sampling ablation is useful and properly controlled. Replacing the five selected modes with five uniformly sampled copies, at the same replication budget, reduces Push-T planning from 90.1 to 53.3 and video–inverse performance from 83.2 to 49.7. This supports the proposed training distribution within the Qantara formulation.
- The Cube results are substantial. Qantara reaches 93.7, compared with 76.7 for the LeWM reproduction, 80.0 for the DINO-WM reproduction and 74.7 for the larger LeWM control. The Cube-Double results also favour Qantara’s planner: 49.6 against 36.9 and 36.3. These are the strongest empirical results in the paper.
- The shared checkpoint has useful capabilities. Planning and two action-generation procedures work without retraining or a pixel decoder at inference. Demonstrating this in a compact model trained from scratch is worthwhile.
- The reporting includes important negative results. The authors distinguish reproduced from published baselines, identify unstable seeds, explain the goal-blind limitation, and report that longer LeWM training removes the Reacher advantage. Most success-rate comparisons use three training seeds and 250 evaluation episodes per seed.
Critical weaknesses
- The paper does not establish the advantage over a simpler model with the same interfaces.
A natural comparison is a shared JEPA encoder with conventional heads for forward prediction, action generation, action-free prediction and inverse dynamics. Such a model would test whether the joint denoising formulation offers an advantage beyond learning several related tasks together. The mode-removal experiments do not answer this question: they remove supervision from Qantara while retaining its architecture and training recipe.
A frozen-LeWM controller is another relevant comparison. Nguyen et al. report strong goal-conditioned performance from a roughly 1.5M-parameter head. Their numbers cannot be compared directly with Qantara’s goal-blind paths, and their extra training cost excludes learning the LeWM backbone. Nevertheless, a reproduction under Qantara’s protocol would help establish the practical value of training everything jointly.
For the goal-blind paths, conventional BC and Diffusion Policy baselines with the same observations, history and data would be more informative than the current comparisons. The UVA reproduction is difficult to interpret: low error on its training trajectories establishes fitting, but does not explain its weak closed-loop performance. It should carry limited weight in claims about superiority over other policy-learning methods.
- Equal epochs do not provide an equal training-compute comparison.
Appendix B gives Qantara an effective batch of 640 through five-mode replication of 128 source rows. LeWM and PLDM use 128 source rows at the same optimiser-step count. The resulting training costs are not reported side by side.
This does not imply exactly five times the total computation: encoder work may be shared, and the architectures differ. It does mean that the comparison controls data exposure and optimiser steps without establishing equal computational cost.
The larger LeWM control addresses parameter count. The 20-epoch experiment addresses one longer-training setting and removes the Reacher margin. Neither resolves the overall compute comparison. Similarly, comparisons between one-, two- and five-mode models change the number of processed examples per update.
Please report measured training cost and compare the principal baseline and specialists at comparable budgets. The five-mode sampling comparison and comparisons among the four-mode rows remain informative because their budgets match.
- The contribution of the bridge remains unresolved.
Table 5 omits the most important control: a Gaussian source with the residual head retained. The explanation that this is “structurally unreachable” describes the implementation, rather than a conceptual restriction. Without this cell, the bridge has not been isolated in the configuration used by the successful model.
The residual comparison also changes the output-layer initialisation. It therefore tests the residual and initialisation together. The reported 12.7-point Reacher gain from the bridge is obtained with the residual disabled and cannot establish the same benefit in the default configuration.
The other ablations warrant a restrained interpretation. A single predictor call performs similarly to four calls, so iterative state refinement has little demonstrated benefit on these tasks. This does not show that bridge corruption during training is unnecessary. Likewise, the noise-scale sweep establishes sensitivity, particularly on Push-T, but does not identify why noise helps.
The mode-removal results support an empirical stability observation. They do not yet establish a general co-regularisation mechanism. With three seeds and occasional large failures, changes in optimisation or effective batch size remain plausible explanations. The claim should be stated at that level unless further controls separate these effects.
- Appendix G’s marginal-preservation claim is incorrect in general.
The state head is trained by squared-error endpoint regression. For a fixed representation and conditioning information, the optimal prediction is a conditional mean. Equation 12 substitutes that estimate for the unknown endpoint and adds fresh bridge noise. This does not generally reproduce the training bridge marginal.
For a simple counterexample, fix the current latent at zero and let the next latent be equally likely to be −1 or +1 under identical conditioning. Use a unit bridge-noise coefficient. The optimal prediction at the source is zero. At the halfway point, the training bridge has variance 1/2, whereas the first update of a two-step Equation 12 recursion has variance 1/4. The uncertainty in the endpoint is missing.
The authors should either provide assumptions and a derivation that justify the claim, or describe Equation 12 as a heuristic refinement procedure. This correction concerns the interpretation of the sampler; it does not invalidate the measured control results or the action flow-matching objective.
The video–inverse procedure has a related limitation: a conditional-mean future latent can lie between incompatible outcomes and be an unsuitable inverse-dynamics target. Equation 10 already acknowledges the point-estimate approximation. A discussion or controlled example of its failure would be sufficient.
- The deployment claims need a clearer success–cost comparison.
BC and video–inverse are distinct inference procedures, but both perform goal-blind imitation. Their success therefore does not establish that the checkpoint can preserve goal-reaching ability when switching from planning to a faster mode.
The paper explains why imitation succeeds in its evaluation: goals are later states of the trajectory supplying the initial state, Push-T has a shared task target, and success tolerances accommodate some deviation. This is a legitimate setting. It leaves the usefulness of the fast paths for independently specified goals largely untested.
The Reacher failure is consistent with this scope. Goal-blind imitation of random behaviour should not be expected to solve a precise goal-reaching task. A goal-conditioned inverse model succeeding on the same data would not contradict that explanation. Nor would querying Qantara’s local inverse model with a distant goal necessarily solve the problem.
The timing comparison also needs a stronger planning reference. Table 6 gives a suite average of approximately 90.7 with one state-predictor call, against 91.2 with four calls. Table 7 measures the four-call configuration. The reported 15–65× ratios are valid for those settings, but do not show the advantage over a cheaper planner with nearly the same success rate.
Please report success against measured runtime for the one-call planner and a modest CEM-budget sweep. Also separate amortised cost per environment step from latency when a new plan or action is actually computed. Both quantities matter, since CEM executes a 25-step buffer while the imitation paths compute every five steps.
Questions for authors
- How does Qantara compare with a simpler shared-encoder model or a frozen-LeWM controller under the same evaluation protocol and measured training budget?
- What happens when the Gaussian-source control retains the residual head and its initialisation?
- Do the main planning gains survive matching training cost, particularly on Cube and Two-Room?
- Can the marginal claim following Equation 12 be justified, or will the sampler be presented as heuristic?
- What is the measured success–runtime trade-off for one versus four state-predictor calls, including latency at an actual planning decision?
5 comments
Response to Reviewer YJF4 (2/2)
W3 / Q2: Gaussian source with the residual head and its initialisation kept. We took the variant you proposed and tested it: the next latent is generated from pure noise (a Gaussian source) instead of the bridge, while the residual head and its zero initialisation stay as in the original model. We also ran the same model with standard instead of zero initialisation of the residual head, and two variants that start from the current latent zₜ, one without noise and one with noise. All runs used peak LR 1e-4 and the same 5 seeds (table in general §5). In the submitted configuration, the bridge beats the start from noise on Push-T by 6.2 points [+4.6, +8.0] and on Reacher by 10.7 [+5.2, +16.1]. On Reacher, the bridge also beats both starts from zₜ. On Push-T and Cube, the start from zₜ without noise does as well as the bridge. On Cube, the start from noise is 3.7 higher than the bridge. With 60 seeds for the full recipe and three drop-one arms, the revision states the mode-removal results as an empirical stability observation, as you suggest. With the residual kept at LR 1e-4, the default initialisation moves CEM by at most 0.3 on Push-T and Cube; on Reacher the initialisation comparison is inconclusive at 5 seeds. The revision states the bridge result per environment, replaces the "structurally unreachable" sentence with these cells, and reports the γ sweep as inconclusive.
W4 / Q4: the marginal claim after Equation 12. Equation 12 does not preserve the training-bridge marginal in general. We checked your counterexample numerically, and it holds: with 2,000,000 samples, the bridge-midpoint variance is 0.4995 (exact 0.5) and after the first of two recursion steps 0.2497 (exact 0.25), because the conditional mean replaces the uncertain endpoint. The revised Appendix J (was G) gives this counterexample, presents Equation 12 as a heuristic refinement and drops the marginal-preservation sentence. After Equation 10 we added that a conditional-mean next latent can fall between incompatible outcomes and be a poor inverse-dynamics target.
W5 / Q5: goals, success–cost trade-off, and latency at a decision.
Held-out start–goal pairs (goal 25 steps later in the same episode). The planner scores 87.3 and 90.7 on unseen pairs; the goal-blind paths hold on Cube and drop on Push-T, for a policy-only specialist too (−50.3 vs −46.3), so the drop is not specific to the joint model.
| 3 train seeds | CEM | BC / video–inverse |
|---|---|---|
| Push-T, 21 episodes neither model trained on | 87.3 ± 1.2 | 35.3 / 34.9; four-head LeWM 15.1–21.5 (train seeds 11, 22, 33), 13.7–20.3 below (revised App. E) |
| Cube, 200 freshly collected episodes | 90.7 | 69.5 / 73.9 |
| Cube, goals from another episode | 25.6 vs LeWM's 18.3 |
The Limitations state that the goal-blind Push-T numbers are within-corpus.
- A goal-conditioned fast path (on the suite's start–goal protocol). The goal-conditioned variant (general §6) runs through the BC path with one extra input and two denoising steps per action chunk (not timed separately) and reaches 93.0 / 99.8 / 100.0 / 68.2, above the same checkpoints' planner on Push-T and Cube.
- One-call planner and CEM budget. With one predictor call, CEM scores 89.1 / 100.0 / 93.3 / 80.5 (suite 90.7 vs 91.2 with four; revised Table 9). On a fixed K × N × iteration grid, the cheapest CEM setting within 3 SR of the best one without BC seeding is K = 1 on Push-T, Two-Room and Cube, and K = 4 with 10 iterations on Reacher. Against it, the goal-blind paths are 7–57× cheaper per decision and 1.4–11.5× amortised per environment step; at the default planner, 10–19× amortised, and one CEM planning call takes 4.8–12.7 s (five episodes batched). The revision tabulates each grid point's success rate and plan-call time, with the Pareto set marked.
Update: the four-head LeWM on Cube, five further seeds. We ran both models again, new runs from one code tree, paired by seed. On five Cube seeds (101–105), Qantara's CEM gets 90.5 ± 0.5, against 83.8 ± 1.5 with MLP heads (H1) and 85.2 ± 0.3 with the flow BC head (H2). So the difference is +6.7 [+5.3, +8.3] and +5.3 [+4.6, +5.9] (95% paired bootstrap over seeds), with both intervals above zero. On the other seeds, 11–55, the difference did not separate (revised App. E).
Response to Reviewer YJF4 (1/2)
Bottom line. We trained the simpler alternatives you named under our protocol. A shared-encoder LeWM with conventional heads, at several times Qantara's FLOPs, reaches 25–30 on the within-corpus Push-T goal-blind paths against Qantara's 83; its planner trails on Reacher (15.8–18.2), is not separated on Cube seeds 11–55 and trails on seeds 101–105 (5.3–6.7), is level on Two-Room and 2.6–3.5 ahead on Push-T. Qantara costs 1.07× LeWM's FLOPs per step. With the residual kept, the bridge beats a Gaussian source on Push-T and Reacher, and the Gaussian source is 3.7 higher on Cube (W3). Equation 12 is now a heuristic (W4).
W1 / Q1: a simpler shared-encoder model, a frozen-LeWM controller, and policy baselines. We took Qantara's protocol and trained all three: the shared-encoder LeWM with conventional heads, the frozen-LeWM controller, and the policy baselines. It turned out that on Push-T, Qantara's goal-blind paths are ahead of each goal-blind alternative among them.
Shared encoder with conventional heads (general §4: LeWM with forward, BC, action-free and inverse heads, MLP or flow BC, at Qantara's optimiser steps and predictor rows, ≥4.7× its FLOPs, head losses weighted 1, 5 seeds). Its Push-T policy paths are far below Qantara's and its planner is below on Reacher; it is level on Two-Room, not separated on Cube at five seeds and above on Push-T planning.
Frozen-LeWM controller and policy baselines. Heads on the frozen latents of our 10-epoch LeWM (50 epochs at batch 640) and policies with the same observations, history and data, 3 seeds each. With a goal, our variant (general §6), from one training stage, is level with the goal-conditioned heads on Two-Room and Cube, ahead of the frozen-LeWM one on Reacher and below it on Push-T.
The revision scopes the UVA row (14.4) to the suite's protocol (72gF Q5).
W2 / Q3: equal epochs are not equal compute. The revision reports training cost: 1.07× LeWM's FLOPs per step (shared encoder pass) and 1.0–1.1× its wall time. More compute does not close the Cube gap, and LeWM keeps its Push-T planning lead, as in the paper (general §1–3).
At matched rows the five-mode model leads a forward-only model by +5.4 [+3.1, +8.5] on Push-T and +4.8 on Two-Room.