- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
JEVA learns a latent-action world model from image trajectories with sparse action labels. It jointly trains a visual encoder, a Gaussian inverse-dynamics model, a forward predictor in embedding space, and an action decoder. Its main technical proposal is variance-reduced routing (VRR): the forward predictor receives posterior means on labeled transitions and posterior samples on unlabeled transitions, while action decoding remains stochastic. The paper analyzes the resulting bias–variance trade-off and evaluates the learned model through CEM planning and a separately trained goal-conditioned behavior-cloning policy on five simulated tasks.
Critical strengths
- The paper studies a concrete training problem and provides direct evidence for its proposed intervention. In the PushT ablation with 5% labels, adding VRR to the otherwise regularized variational model improves reported CEM success from 61.0% to 75.0%. This is more informative than an uncontrolled architectural comparison. The broader results also demonstrate promising label efficiency, particularly on Cube-Double and Cube-Triple, and evaluate both planning and learned-policy deployment.
- The bias–variance analysis provides a useful, appropriately qualified local explanation. The paper explicitly acknowledges that posterior-mean conditioning is biased, including for an affine predictor, and distinguishes it from exact Rao–Blackwellization. The small-posterior analysis helps explain when mean substitution could improve predictor-gradient estimation. The discussion of Reacher also identifies a meaningful limitation of the learned action bottleneck rather than presenting uniformly positive results.
Critical weaknesses
-
The necessity of label-dependent routing and its proposed mechanism are not sufficiently validated.
The central comparison establishes that using posterior means on labeled transitions improves the tested configuration. It does not establish why posterior sampling should be retained specifically on unlabeled transitions. The most direct missing control is mean conditioning on both branches, with the action decoder still consuming samples. Without this comparison, the contribution could primarily be mean conditioning rather than the proposed selective routing rule.
The theoretical motivation further depends on sufficiently concentrated posteriors. The paper argues that action supervision produces narrow labeled posteriors, but does not report posterior covariance statistics demonstrating this regime. Action reconstruction constrains uncertainty in action-sensitive directions; it does not necessarily make the entire latent posterior narrow.
Moreover, VRR both reduces sampling variance and changes the prediction objective by removing covariance-dependent terms. A comparison with multi-sample Monte Carlo estimation would help distinguish these effects. The theorem analyzes predictor gradients under a fixed posterior, rather than the stability of the complete jointly trained system. Consistently, the single-stage model without VRR still retains six active dimensions and achieves 61% success, so describing VRR as what makes single-stage training possible overstates the evidence.
-
The contribution and empirical positioning relative to existing semi-supervised latent-action methods need clarification.
Joint inverse–forward dynamics learning and action grounding are established ideas. CLAM already jointly trains its latent-action model and action decoder using labeled and unlabeled data; its subsequent policy-training stage should not be confused with delayed grounding or inverse-model warm-up. CoLA-World’s warm-up addresses interaction with a pretrained video world model, and its paper also reports non-collapsing joint training when both components are randomly initialized. JEVA’s distinctive contribution should therefore center on its variational JEPA formulation and routing rule, rather than a broad claim of first enabling joint training from scratch. CLAM, CoLA-World.
The main baselines use only labeled trajectories, whereas JEVA additionally uses the complete observation pool. This is a valid label-efficiency comparison, but does not establish an advantage over methods capable of exploiting the same unlabeled observations. The cumulative ablation also lacks the complete JEVA recipe trained using only the labeled trajectory pool, which would isolate the contribution of additional observations.
Questions for authors
- Controlled evidence: How does the proposed routing compare with using posterior means on both labeled and unlabeled branches? Can reduced-variance sampling, such as multi-sample Monte Carlo, help distinguish variance reduction from the change in training objective?
- Mechanism clarification: What are the posterior covariance distributions on labeled and unlabeled data throughout training? Do they support the small-posterior assumption? Please also define the criterion and threshold used to count active latent dimensions.
- Controlled evidence: How does the complete JEVA recipe perform without the additional unlabeled trajectories, keeping the remaining components fixed? A comparison with a directly relevant method that can consume the same unlabeled pool would also materially strengthen the empirical positioning.
- Figure–table clarification: In Figure 3(a), the LeWM point at 5% labels appears near 15%, whereas Table 2 reports 51.3±6.4%. Which result is correct, and can the plot and numerical table be reconciled?
- Statistical clarification: Does Table 1 use the same three-seed, 100-episode binary-success evaluation protocol? Please provide the per-seed success counts and definition of “±,” particularly for 48.33±1.27, which I could not reconcile with conventional across-seed standard deviations under that protocol.
2 comments
Thank you for the additional controls and for correcting the figure and evaluation statistics. The mean-only and multi-sample comparisons address my main question about selective routing. I also found the posterior measurements helpful, particularly the estimate of the sampling penalty relative to the prediction error on labeled transitions. Together, these results give a more convincing account of why routing helps in the tested setting.
The pseudo-labeling baseline using the same observation pool strengthens the empirical comparison. I appreciate the narrower claim about joint training and am raising my score from 2 to 3. I would encourage including the numerical results for both routing controls in the revised paper, since they are central to the argument.
Thank you for a careful review. Each controlled comparison you asked for is below.
Q1. Routing controls (PushT, 5%, CEM):
Sampling is required on unlabeled transitions, so the routing has to depend on labels; mean conditioning alone is not the contribution. MC-16 separates the two effects you named. MC-K and single-sample training share the same expected objective, ‖r_t‖² + tr(J_t Σ J_tᵀ) to leading order (Eq. 8), and differ only in estimator variance. VRR optimizes ‖r_t‖² and drops the tr(JΣJᵀ) term on labeled transitions, where the action decoder already constrains ℓ. Since MC-16 does not recover the gain, the change in objective accounts for it. We agree that Proposition 1, which concerns predictor gradients under a fixed posterior, does not explain stability of the joint system; the penalty in Eq. 8 is the argument we now lead with. We also agree that Table 1b contradicts "VRR is what makes single-stage training work". The new runs show that reaching the best success rate needs both pieces: without VRR, success is 61.0%, and without SIGReg on ℓ it is 67.3%, against 75.0% for full JEVA. A warm-up without VRR collapses every dimension. The abstract and contributions are revised to match.
Q2. Posterior covariance and active dimensions. Action reconstruction narrows the posterior in the action-sensitive directions, as you suspected, and not elsewhere. On the two dominant latent dimensions (PushT's action is 2-D), which carry most of the predictor's dependence on ℓ, posterior std is 0.12–0.18 against a spread of the means of ≈ 1.0: posterior variance is 1.6–3.4% of signal variance throughout training. The other dimensions have wider posteriors, but the predictor is largely insensitive to them: sampling there moves its prediction by 0.2–0.3% of the predicted-latent variance. The local analysis needs JΣJᵀ to be small relative to the signal, not Σ itself, and that holds. Labeled and unlabeled posteriors match until about 60k steps; after that, labeled posteriors are slightly tighter. A dimension counts as active if Var(μ_d) > 0.01, i.e. std(μ_d) > 0.1 (Burda et al., 2016). JEVA has 7 of 8 active dims (std(μ_d) ∈ [0.14, 1.02] on active dimensions vs 0.06 on the inactive one). Small relative to the signal does not mean small relative to the loss: the relevant scale for the penalty is the term it is added to in Eq. 8, the residual ‖r_t‖² = ‖P(z_t, μ_t) − z_{t+1}‖² (the prediction error at the posterior mean). Without VRR, tr(JΣJᵀ), estimated as the variance of the prediction over posterior samples, is 20–36% of ‖r_t‖² on labeled transitions throughout training, so a substantial part of the sampled labeled loss penalizes the predictor's use of ℓ.
Q3. Full recipe without the unlabeled pool, and a method using the same pool. Without D_U, JEVA reaches 19.0 ± 2.6, against 75.0 ± 6.6 with it (Table 2b). With no unlabeled transitions, every predictor input is a posterior mean, the routing that never trains in Table 2a, so this run conflates the amount of data with the routing. A deterministic latent action has no routing to change; labeled-only training with it reaches 49.7 ± 5.5 (Table 1a, first row), still 25 points below JEVA with the full pool (Appendix C.5). The gain comes from the unlabeled data, not the architecture alone. For a method using the same pool, pseudo-labeling with an IDM trained on D_L and then training LeWM on everything gives 64.0 ± 9.5. CLAM at the same budget is in the general response (16.0%). We have also corrected the CLAM and CoLA-World descriptions as you suggested and dropped the claim to be first to train jointly from scratch.
Q4. Figure 3(a). The results table (Table 2 in the submission, Table 3 in the revision) is correct (51.3 ± 6.4). The plotted point was wrong and is fixed.
Q5. Statistics. Yes: Table 1 uses 3 evaluation seeds × 100 episodes with binary success, and ± is the sample standard deviation across seeds (ddof = 1). Two entries were wrong and are corrected in the revision. The warm-up entry under 3 × 100 should be 67.3 ± 2.5. The entry 48.33 ± 1.27 was also an error: per-seed counts are 47 / 48 / 50, giving 48.3 ± 1.5.
We hope these controls answer your questions about the mechanism and positioning, and would be grateful if you would reconsider your evaluation. We are glad to run further controls during the discussion if any remain unconvincing.