acceptodds
Under review as a conference paper at ICLR 2027

JEVA: Joint Encoders for Variational Action World Models

23 views

Abstract

Learning world models for robotic control typically requires large amounts of action-labeled data, which is expensive to collect, whereas action-free demonstrations such as video are abundant. Latent action models address this imbalance by inferring the actions that explain observed transitions, but training the inverse dynamics model (IDM) jointly with the forward model is unstable. Existing methods therefore fix the latent action space before training the world model, or add a separate inverse-dynamics warm-up stage. We introduce JEVA, a latent-action world model that co-trains an IDM and a forward world model end-to-end from scratch in a single stage, on a pool of trajectories in which only a small fraction of transitions carry action labels. JEVA treats the action between two visual states as a continuous Gaussian latent that can be planned over and decoded into continuous controls. Variance-reduced routing (VRR) is what makes single-stage training work. On labeled transitions, where the action decoder keeps posterior forward predictor on the posterior mean instead. This removes reparameterization noise from the predictor gradient at the cost of a small bias, and it also removes a penalty that otherwise drives latent actions toward collapse: with a warm-up stage and no VRR every latent-action dimension collapses. With 5% of episodes labeled, a policy trained in JEVA's latent action space matches or exceeds every baseline trained with 20% of labels on OGBench Cube-Double and Cube-Triple. Under CEM planning, JEVA leads the strongest world model baseline by 11-22 points on all three Cube tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ivSsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JEVA learns a latent-action world model from image trajectories with sparse action labels. It jointly trains a visual encoder, a Gaussian inverse-dynamics model, a forward predictor in embedding space, and an action decoder. Its main technical proposal is variance-reduced routing (VRR): the forward predictor receives posterior means on labeled transitions and posterior samples on unlabeled transitions, while action decoding remains stochastic. The paper analyzes the resulting bias–variance trade-off and evaluates the learned model through CEM planning and a separately trained goal-conditioned behavior-cloning policy on five simulated tasks.

Critical strengths
  1. The paper studies a concrete training problem and provides direct evidence for its proposed intervention. In the PushT ablation with 5% labels, adding VRR to the otherwise regularized variational model improves reported CEM success from 61.0% to 75.0%. This is more informative than an uncontrolled architectural comparison. The broader results also demonstrate promising label efficiency, particularly on Cube-Double and Cube-Triple, and evaluate both planning and learned-policy deployment.
  2. The bias–variance analysis provides a useful, appropriately qualified local explanation. The paper explicitly acknowledges that posterior-mean conditioning is biased, including for an affine predictor, and distinguishes it from exact Rao–Blackwellization. The small-posterior analysis helps explain when mean substitution could improve predictor-gradient estimation. The discussion of Reacher also identifies a meaningful limitation of the learned action bottleneck rather than presenting uniformly positive results.
Critical weaknesses
  1. The necessity of label-dependent routing and its proposed mechanism are not sufficiently validated.

    The central comparison establishes that using posterior means on labeled transitions improves the tested configuration. It does not establish why posterior sampling should be retained specifically on unlabeled transitions. The most direct missing control is mean conditioning on both branches, with the action decoder still consuming samples. Without this comparison, the contribution could primarily be mean conditioning rather than the proposed selective routing rule.

    The theoretical motivation further depends on sufficiently concentrated posteriors. The paper argues that action supervision produces narrow labeled posteriors, but does not report posterior covariance statistics demonstrating this regime. Action reconstruction constrains uncertainty in action-sensitive directions; it does not necessarily make the entire latent posterior narrow.

    Moreover, VRR both reduces sampling variance and changes the prediction objective by removing covariance-dependent terms. A comparison with multi-sample Monte Carlo estimation would help distinguish these effects. The theorem analyzes predictor gradients under a fixed posterior, rather than the stability of the complete jointly trained system. Consistently, the single-stage model without VRR still retains six active dimensions and achieves 61% success, so describing VRR as what makes single-stage training possible overstates the evidence.

  2. The contribution and empirical positioning relative to existing semi-supervised latent-action methods need clarification.

    Joint inverse–forward dynamics learning and action grounding are established ideas. CLAM already jointly trains its latent-action model and action decoder using labeled and unlabeled data; its subsequent policy-training stage should not be confused with delayed grounding or inverse-model warm-up. CoLA-World’s warm-up addresses interaction with a pretrained video world model, and its paper also reports non-collapsing joint training when both components are randomly initialized. JEVA’s distinctive contribution should therefore center on its variational JEPA formulation and routing rule, rather than a broad claim of first enabling joint training from scratch. CLAM, CoLA-World.

    The main baselines use only labeled trajectories, whereas JEVA additionally uses the complete observation pool. This is a valid label-efficiency comparison, but does not establish an advantage over methods capable of exploiting the same unlabeled observations. The cumulative ablation also lacks the complete JEVA recipe trained using only the labeled trajectory pool, which would isolate the contribution of additional observations.

Questions for authors
  1. Controlled evidence: How does the proposed routing compare with using posterior means on both labeled and unlabeled branches? Can reduced-variance sampling, such as multi-sample Monte Carlo, help distinguish variance reduction from the change in training objective?
  2. Mechanism clarification: What are the posterior covariance distributions on labeled and unlabeled data throughout training? Do they support the small-posterior assumption? Please also define the criterion and threshold used to count active latent dimensions.
  3. Controlled evidence: How does the complete JEVA recipe perform without the additional unlabeled trajectories, keeping the remaining components fixed? A comparison with a directly relevant method that can consume the same unlabeled pool would also materially strengthen the empirical positioning.
  4. Figure–table clarification: In Figure 3(a), the LeWM point at 5% labels appears near 15%, whereas Table 2 reports 51.3±6.4%. Which result is correct, and can the plot and numerical table be reconciled?
  5. Statistical clarification: Does Table 1 use the same three-seed, 100-episode binary-success evaluation protocol? Please provide the per-seed success counts and definition of “±,” particularly for 48.33±1.27, which I could not reconcile with conventional across-seed standard deviations under that protocol.
2 comments

Thank you for a careful review. Each controlled comparison you asked for is below.

Q1. Routing controls (PushT, 5%, CEM):

Predictor input, labeled / unlabeledSR (%)
sample / sample (VRR off)61.0 ± 4.4
mean / mean (decoder still uses samples)≤ 5.0 at every checkpoint, 10k–140k
MC-16 / sample60.7 ± 1.5
mean / sample (VRR)75.0 ± 6.6

Sampling is required on unlabeled transitions, so the routing has to depend on labels; mean conditioning alone is not the contribution. MC-16 separates the two effects you named. MC-K and single-sample training share the same expected objective, ‖r_t‖² + tr(J_t Σ J_tᵀ) to leading order (Eq. 8), and differ only in estimator variance. VRR optimizes ‖r_t‖² and drops the tr(JΣJᵀ) term on labeled transitions, where the action decoder already constrains ℓ. Since MC-16 does not recover the gain, the change in objective accounts for it. We agree that Proposition 1, which concerns predictor gradients under a fixed posterior, does not explain stability of the joint system; the penalty in Eq. 8 is the argument we now lead with. We also agree that Table 1b contradicts "VRR is what makes single-stage training work". The new runs show that reaching the best success rate needs both pieces: without VRR, success is 61.0%, and without SIGReg on ℓ it is 67.3%, against 75.0% for full JEVA. A warm-up without VRR collapses every dimension. The abstract and contributions are revised to match.

Q2. Posterior covariance and active dimensions. Action reconstruction narrows the posterior in the action-sensitive directions, as you suspected, and not elsewhere. On the two dominant latent dimensions (PushT's action is 2-D), which carry most of the predictor's dependence on ℓ, posterior std is 0.12–0.18 against a spread of the means of ≈ 1.0: posterior variance is 1.6–3.4% of signal variance throughout training. The other dimensions have wider posteriors, but the predictor is largely insensitive to them: sampling there moves its prediction by 0.2–0.3% of the predicted-latent variance. The local analysis needs JΣJᵀ to be small relative to the signal, not Σ itself, and that holds. Labeled and unlabeled posteriors match until about 60k steps; after that, labeled posteriors are slightly tighter. A dimension counts as active if Var(μ_d) > 0.01, i.e. std(μ_d) > 0.1 (Burda et al., 2016). JEVA has 7 of 8 active dims (std(μ_d) ∈ [0.14, 1.02] on active dimensions vs 0.06 on the inactive one). Small relative to the signal does not mean small relative to the loss: the relevant scale for the penalty is the term it is added to in Eq. 8, the residual ‖r_t‖² = ‖P(z_t, μ_t) − z_{t+1}‖² (the prediction error at the posterior mean). Without VRR, tr(JΣJᵀ), estimated as the variance of the prediction over posterior samples, is 20–36% of ‖r_t‖² on labeled transitions throughout training, so a substantial part of the sampled labeled loss penalizes the predictor's use of ℓ.

Q3. Full recipe without the unlabeled pool, and a method using the same pool. Without D_U, JEVA reaches 19.0 ± 2.6, against 75.0 ± 6.6 with it (Table 2b). With no unlabeled transitions, every predictor input is a posterior mean, the routing that never trains in Table 2a, so this run conflates the amount of data with the routing. A deterministic latent action has no routing to change; labeled-only training with it reaches 49.7 ± 5.5 (Table 1a, first row), still 25 points below JEVA with the full pool (Appendix C.5). The gain comes from the unlabeled data, not the architecture alone. For a method using the same pool, pseudo-labeling with an IDM trained on D_L and then training LeWM on everything gives 64.0 ± 9.5. CLAM at the same budget is in the general response (16.0%). We have also corrected the CLAM and CoLA-World descriptions as you suggested and dropped the claim to be first to train jointly from scratch.

Q4. Figure 3(a). The results table (Table 2 in the submission, Table 3 in the revision) is correct (51.3 ± 6.4). The plotted point was wrong and is fixed.

Q5. Statistics. Yes: Table 1 uses 3 evaluation seeds × 100 episodes with binary success, and ± is the sample standard deviation across seeds (ddof = 1). Two entries were wrong and are corrected in the revision. The warm-up entry under 3 × 100 should be 67.3 ± 2.5. The entry 48.33 ± 1.27 was also an error: per-seed counts are 47 / 48 / 50, giving 48.3 ± 1.5.

We hope these controls answer your questions about the mechanism and positioning, and would be grateful if you would reconsider your evaluation. We are glad to run further controls during the discussion if any remain unconvincing.

Thank you for the additional controls and for correcting the figure and evaluation statistics. The mean-only and multi-sample comparisons address my main question about selective routing. I also found the posterior measurements helpful, particularly the estimate of the sampling penalty relative to the prediction error on labeled transitions. Together, these results give a more convincing account of why routing helps in the tested setting.

The pseudo-labeling baseline using the same observation pool strengthens the empirical comparison. I appreciate the narrower claim about joint training and am raising my score from 2 to 3. I would encourage including the numerical results for both routing controls in the revised paper, since they are central to the argument.

Reviewer_8aV4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JEVA jointly learns an inverse dynamics model, a latent forward model, and an action decoder from videos with sparse action labels. Its main proposal, variance-reduced routing, uses posterior means for forward prediction on labeled transitions and posterior samples on unlabeled transitions. The paper provides a local gradient-error analysis and evaluates planning and learned policies on five simulated control tasks.

Critical strengths

The results show useful gains in action-label efficiency, particularly on Cube-Double and Cube-Triple. Evaluating both CEM planning and learned policies helps show that the learned representation is useful beyond a single control procedure. The weaker results on Reacher also identify a concrete limitation in fine-grained control.

Critical weaknesses
  1. The novelty claim relative to CLAM needs correction. Section 2 describes CLAM as a two-stage alternative to joint training, but CLAM already jointly trains inverse dynamics, forward dynamics, and action grounding during representation learning; its second stage trains the policy. JEVA’s variational formulation and VRR remain distinct, but joint training itself is not new. CLAM is also absent from the experiments, while the reported baselines use only labeled data. The comparisons therefore do not establish JEVA’s advantage over a closely related method that can also exploit unlabeled demonstrations.
  2. The claim that VRR enables stable single-stage training is stronger than the evidence. In Table 1b, single-stage training without VRR retains six active dimensions and achieves 61% success; complete collapse occurs in the warm-up variants. Proposition 1 establishes a local predictor-gradient advantage for sufficiently narrow posteriors, with posterior parameters and targets fixed, rather than stability of joint training. The ablations show that VRR helps, but do not establish why routing must depend on label availability or whether using posterior means on both branches would suffice.
Questions for authors
  1. How does JEVA compare with CLAM under the same labeled/unlabeled data budget, and which contribution remains beyond CLAM’s joint representation training?
  2. How does VRR compare with using posterior means on both branches, and what evidence connects the local gradient analysis to collapse prevention during joint training?
  3. Is the policy loss in Eq. (19) trained exclusively on the labeled subset DLD_L?
2 comments

Thank you. You are right on both points about the paper's wording, and we changed it. The new controls also show more precisely what VRR contributes.

Q1 / W1. CLAM. You are right: CLAM trains its inverse dynamics, forward dynamics and action decoder together, with labeled batches during that stage; only the policy is trained afterwards. We misdescribed it in Section 2. We removed our claim to be first to co-train these from scratch, and our claim that prior continuous latent-action models ground their actions only afterwards. Under the same budget (PushT, 5% labels, full unlabeled pool, CLAM's published architecture and loss weights), a reactive policy trained on the labeled episodes and supervised with ground-truth actions through CLAM's frozen action decoder reaches 16.0 ± 1.0%; JEVA reaches 75.0 ± 6.6. CLAM's policies target fixed-goal tasks while PushT changes the goal in every episode, so we also compare on action prediction: CLAM's action decoder fits the labeled data (R² = 0.85) but does not recover actions on unlabeled training transitions (R² = −0.53) or held-out ones (R² = −0.69), while JEVA's decoder reaches R² = 0.84 on held-out transitions. With sparse labels, CLAM's deterministic latent actions do not decode beyond its labeled transitions (Appendix C.4). Training the policy as in CLAM's own pipeline instead, imitating the latent actions its inverse model infers on the full dataset (labeled and action-free) and decoding them at test time, lowers success to 6.3 ± 2.1%: on action-free transitions those latent actions do not decode to the true actions, so the additional data gives the policy incorrect targets. What remains beyond CLAM:

  • (a) the latent action is stochastic, a Gaussian posterior regularized with KL and SIGReg, so the space can be sampled and searched; CLAM trains a deterministic autoencoder over latent actions;
  • (b) the forward model predicts in latent space and is kept for planning; CLAM's forward model reconstructs pixels and is used only during training;
  • (c) VRR. Stochastic latent actions bring posterior variance into the prediction loss (Q2 below), which CLAM's deterministic latents never face. Routing the posterior mean on labeled transitions removes it, and success rises from 61.0 to 75.0.

Q2 / W2. Means on both branches, and the link to collapse. Using posterior means on both branches never trains: at most 5.0% at every checkpoint from 10k to 140k steps. Sampling is needed on unlabeled transitions; the mean helps on labeled ones. To test whether the gain comes from lower gradient variance, we averaged the labeled-branch loss over 16 posterior samples. That reduces estimator variance but keeps the sampled objective, and it reaches 60.7%, no better than VRR off. So the gain is not lower estimator variance, which is what Proposition 1 is about. It comes from dropping the tr(JΣJᵀ) term in Eq. 8, which penalizes the predictor's sensitivity to ℓ in proportion to posterior width. That is the link to collapse: a predictor that ignores ℓ gives the inverse model no reason to keep the posterior informative. On labeled transitions the action decoder already constrains ℓ, so this penalty only costs; on unlabeled transitions it is the only limit on what ℓ carries, which is why means on both branches fail. We measured the predictor's reliance on ℓ directly, by setting ℓ to zero and recording the increase in forward-prediction loss: at the reported checkpoints it rises by 0.172 with VRR against 0.156 without it, and 6 latent dimensions are functionally active against 4 (a dimension is functionally active if ablating it alone raises the loss by more than 1%). Consistent with this, a warm-up stage without VRR collapses every latent dimension (Table 1b). We agree the abstract overstated VRR's role. Reaching the best success rate needs both pieces: without VRR, success is 61.0%, and without SIGReg on ℓ it is 67.3%, against 75.0% for full JEVA. The revised abstract says so.

Q3. Eq. (19), now Eq. 14. Yes, the policy is trained only on D_L. The revision says so.

We hope the corrected positioning against CLAM and the routing controls answer your concerns, and that you will consider raising your score. If the revised Section 2 wording still misstates CLAM, please let us know and we will change it.

Thanks for the additional experiments and clarifications. My W1 concern is partly addressed, but the same-pool baselines cover only PushT and Cube-Single. Cube-Double and Cube-Triple, which support the main label-efficiency claim, still lack this comparison. The CLAM experiment on PushT is useful, though the acknowledged task and policy differences limit what it shows. I remain unconvinced that the gains on the harder Cube tasks would hold against baselines using the same unlabeled data, so I maintain my score of 2.

Reviewer_WNToOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JEVA learns a world model when you have lots of video but real robot actions for only 5 to 20% of episodes. An inverse model guesses a latent action from two frames, a forward model predicts the next frame latent from it, and on labeled episodes a small decoder maps the latent action back to the real action. All trained from scratch in one go. The new trick is VRR: on labeled data the forward model gets the mean of the guessed action instead of a noisy sample. The paper says this cuts gradient noise and stops the model from learning to ignore the action. Results on PushT, Reacher and three Cube tasks: with 5% labels JEVA's BC policy matches baselines that use 20% labels on the two harder Cube tasks, and its CEM planner beats other world-model planners by 11 to 22 points on Cube. It loses on Reacher. On PushT, unlabeled video only helps once the bottleneck, regularizer and VRR are all in.

Critical strengths
  • Realistic problem, simple model. One loss, three small parts.

  • VRR is a one-line change that adds 14 points on PushT, Table 1a.

  • The ablation is honest. Unlabeled data alone does nothing; it only helps with the right bottleneck. That is the most useful finding.

  • Fair representation comparison. Same BC head on LeWM's latent and JEVA's.

Critical weaknesses
  1. Baselines never see the unlabeled video. JEVA trains on all frames; baselines train on 5 to 20% of them. The easy fix is pseudo-labeling: train an inverse model on the labeled part, label the rest, train LeWM on everything. Without that, the 11 to 22 point gap may just be more data.
  2. No latent-action baseline. CLAM, AdaWorld, CoLA-World and LPWM do this exact job from unlabeled data. All are cited, none are run.
  3. The abstract overstates VRR. It says VRR is what makes single-stage training work. Table 1b shows it works without VRR, at 61% with 6 active dims. VRR adds 14 points and prevents collapse only under warm-up. Say that instead.
  4. The theory does little. Proposition 1 only covers tight posteriors and the appendix admits the proof isn't what carries the argument. Two pages of math for using the mean on the labeled branch.
  5. Ablations cover one setting. PushT, 5% labels, CEM. Nothing on Cube or under BC.
  6. Only one unlabeled amount. Always the full dataset. No curve showing more video keeps helping.
  7. Unfinished draft. Broken sentence in the abstract, template text in the acknowledgments, a Table 1b entry with no std, typos, a duplicated citation.
Questions for authors
  1. Add a pseudo-label baseline. How much of the gap survives?
  2. Run CLAM at the same label fractions.
  3. Fix the abstract to match Table 1b.
  4. Repeat the ablation on Cube-Double and under BC.
  5. Sweep the amount of unlabeled data at 5% labels.
  6. How many seeds behind the 68.6 in Table 1b?
  7. Does the latent action leak next-frame content? A probe would show it.
3 comments

Thank you for the clear checklist. Answers follow your question numbers. Shared results are in the general response.

Q1. Pseudo-label baseline. We ran the baseline you proposed (LeWM-PL; details in Appendix C.3): an IDM trained on the 5% labeled episodes pseudo-labels the other 95%, and LeWM trains on the union. LeWM-PL reaches 64.0 ± 9.5, vs 51.3 ± 6.4 for LeWM on the real labels alone and 75.0 ± 6.6 for JEVA. Pseudo-labels close about half of the gap between LeWM and JEVA. We are running the same pseudo-label baseline on Cube-Single and will post the result here when it finishes.

Q2. CLAM at the same label fraction. We ran CLAM on PushT at 5% labels with the full unlabeled pool and the architecture and loss weights of the CLAM paper. A reactive BC policy acting through CLAM's action decoder reaches 16.0 ± 1.0%. CLAM's policies target fixed-goal tasks, while PushT changes the goal in every episode, so we also compare on action prediction, which any CLAM policy relies on: CLAM's action decoder fits the labeled data (R² = 0.85) but does not recover actions on unlabeled training transitions (R² = −0.53) or held-out ones (R² = −0.69), whereas JEVA's decoder reaches R² = 0.84 on held-out transitions. With sparse labels, CLAM's deterministic latent actions do not decode beyond its labeled transitions (Appendix C.4).

Q3. Abstract vs. Table 1b. You are right that the sentence overstates it: Table 1b shows single-stage training without VRR at 61%. The new runs show that reaching the best success rate needs both latent-action regularization and VRR: without SIGReg on latent actions, success is 67.3%, and without VRR it is 61.0%, against 75.0% for full JEVA. With a warm-up stage and no VRR, every latent dimension collapses. The new abstract sentence is in the general response.

Q4 / W5. Ablation on other tasks and BC. On Cube-Single (5%, CEM), VRR also helps: 80.7 ± 2.5 vs 73.3 ± 2.5 without it. Table 3's 78.0 ± 4.4 (Table 2 in the submission) is from the original run; both new arms are matched retrains, reported in Appendix C.7. Under BC (PushT, 5%), VRR on and off reach the same success (88.0 ± 2.0 vs 88.0 ± 5.2). This is what the mechanism predicts: VRR changes how much the forward predictor relies on ℓ, which CEM plans through and the BC head does not use. The Cube-Double ablation is still running; we will post it here when it finishes.

Q5. Amount of unlabeled data (PushT, 5% labels). Success rates (Table 2b): no unlabeled data 19.0; ⅓ of D_U 67.3; ⅔ 72.7; all 75.0. A third of the pool recovers most of the gain.

Q6. Seeds for 68.6. The entry was computed over 3 seeds × 500 episodes. Under the 3 × 100 protocol used elsewhere it is 67.3 ± 2.5, and the revision reports that (Appendix C.1). All ± entries are now sample std (ddof = 1).

Q7. Does the latent action leak next-frame content? On held-out transitions with real actions, an MLP probe predicting z_{t+1} from z_t alone reaches R² = 0.905, from (z_t, a_t) reaches R² = 0.960, adding ℓ_t gives 0.967, and adding a shuffled ℓ_t gives 0.960. ℓ carries little about the next frame beyond what the true action already explains (+0.007 R²). A copy channel would recover most of the remaining variance; ℓ recovers about a sixth, consistent with ℓ encoding the action and some of its effect rather than the next frame. An unconditional probe agrees (Appendix C.8).

W4. The theory. We agree that Proposition 1 does not explain the main effect, and the new controls show why. The proposition is about gradient-estimator variance, and MC-16 lowers exactly that variance with no gain. The argument that carries the paper is the one in Eq. 8. Sampling adds a posterior-variance penalty, tr(JΣJᵀ), to the prediction loss, which pushes the predictor to ignore ℓ; on labeled transitions the action decoder already constrains ℓ, so VRR removes the penalty there. We measured this directly by setting ℓ to zero and recording the increase in forward-prediction loss: at the reported checkpoints it rises by 0.172 with VRR against 0.156 without it, and 6 latent dimensions are functionally active against 4 (a dimension is functionally active if ablating it alone raises the loss by more than 1%). The revision puts this mechanism in the main text (Section 3.4) and shortens Proposition 1 to a remark (Appendix A.2).

W7. Draft quality. Fixed: abstract sentence, acknowledgments template, duplicate citation, typos, missing std.

With baselines that see the unlabeled pool, the corrected VRR claim and the routing controls, we hope the revised paper addresses your main concerns, and we would be grateful if you would reconsider your score. If any item is only partly addressed, please tell us which, and we will follow up during the discussion.

AuthorsOpenReview ↗

Additional Results

Two additional results we promised are now in.

Q1, pseudo-label baseline on Cube-Single (5% labels, CEM): LeWM-PL reaches 69.7±4.269.7 \pm 4.2, against 66.7±10.066.7 \pm 10.0 for LeWM on the real labels alone and 78.0±4.478.0 \pm 4.4 for JEVA.

Q4, VRR ablation on Cube-Double (5% labels, CEM): without VRR, JEVA reaches 48.0±3.548.0 \pm 3.5, against 55.0±3.655.0 \pm 3.6 with VRR (Table 3), a gap of 7 points. VRR therefore improves CEM planning on both Cube tasks we tested (Cube-Single: 80.780.7 vs 73.373.3).

Reviewer_WNToOpenReview ↗

Response to rebuttal.

Thanks for the rebuttal. Pseudo-labeling closes about half the gap, CLAM is run, abstract fixed. The mean/mean and MC-16 controls settle the mechanism for me: VRR changes the objective, it doesn’t reduce variance. Raising to 3.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of new results and revisions (1/2)

Summary of new results and revisions (1/2)

We thank the reviewers for detailed reviews. Reviewers found the problem realistic and the model simple (WNTo), the PushT ablation a controlled test of the proposed change (ivSs, WNTo), and the label-efficiency gains on Cube-Double and Cube-Triple useful (8aV4, ivSs).

All three raised the same three points: (1) baselines that can use the unlabeled pool, (2) whether VRR really is what makes single-stage training work, and (3) the relation to CLAM and CoLA-World. We ran the requested controls. JEVA's margin holds against baselines that use the unlabeled pool, and the routing controls pin down how VRR helps. One abstract sentence overstated VRR's role (Table 1b already shows single-stage training at 61% without VRR); the revised PDF corrects it and leads with the mechanism.

All new numbers are PushT, 5% labeled episodes, CEM planning, 3 seeds × 100 episodes, mean ± sample std (ddof = 1). Numbers refer to the revised PDF; part (2/2) maps them to the submitted version.

(1) Methods that use the same unlabeled pool (Table 2b)

MethodUses D_USR (%)
LeWM (labeled only, Table 3)no51.3 ± 6.4
LeWM-PL: IDM trained on D_L pseudo-labels D_U, LeWM trained on allyes64.0 ± 9.5
CLAM, reactive BC through its action decoderyes16.0 ± 1.0
JEVA without D_U (Gaussian latent action)no19.0 ± 2.6
JEVA without D_U (deterministic latent action, Table 1a)no49.7 ± 5.5
JEVAyes75.0 ± 6.6

Success with 0, ⅓, ⅔ and all of D_U: 19.0, 67.3, 72.7, 75.0. A third of the pool recovers most of the gain.

CLAM at the same budget (5% labels, full unlabeled pool, CLAM's published architecture and loss weights). CLAM's policies target fixed-goal tasks, while PushT changes the goal in every episode, so we compare the two methods on what both policies rely on: recovering actions from latent actions. We report R², the fraction of action variance explained by the decoded actions (1 is perfect, 0 matches always predicting the mean action, and negative is worse). CLAM's action decoder fits the labeled data (R² = 0.85) but does not recover actions on unlabeled training transitions (R² = −0.53) or held-out ones (R² = −0.69). JEVA's decoder reaches R² = 0.84 on held-out transitions. With sparse labels, CLAM's deterministic latent actions do not decode beyond its labeled transitions (Appendix C.4).

(2) What VRR does, and a corrected claim (Table 2a)

Predictor input on labeled / unlabeled transitionsSR (%)
sample / sample (VRR off)61.0 ± 4.4
mean / mean5.0 ± 2.6
MC-16 (loss averaged over 16 samples) / sample60.7 ± 1.5
mean / sample (VRR)75.0 ± 6.6
VRR, without SIGReg on latent actions67.3 ± 0.6

The reviewers are right that "VRR is what makes single-stage training work" overstates it: without VRR, single-stage training still reaches 61% with 6 active dimensions (Table 1b). Reaching the best success rate needs both pieces: without SIGReg on ℓ, success is 67.3%, and without VRR it is 61.0%, against 75.0% for full JEVA. With a warm-up stage and no VRR, every latent dimension collapses (Table 1b). The table also shows what VRR does, and it is not generic variance reduction. Using posterior means on both branches never trains (5.0 ± 2.6 %), so sampling on unlabeled transitions is essential. Averaging 16 posterior samples on the labeled branch lowers gradient-estimator variance but gives 60.7%, no better than VRR off. What VRR changes is the objective. Sampling the latent action adds tr(JΣJᵀ) to the expected prediction loss (Eq. 8). This term acts as an information bottleneck: it penalizes the predictor's reliance on ℓ in proportion to posterior width, which limits how much ℓ can carry. On unlabeled transitions, nothing else limits ℓ, and removing the bottleneck there is fatal (means on both branches: ≤ 5%). On labeled transitions, the action decoder already ties ℓ to the true action, so the bottleneck is redundant and only pushes the predictor to ignore ℓ. VRR therefore removes it only where labels are available. The revision leads with this mechanism (Section 3.4, Appendix A.1) and presents Proposition 1 as a secondary, local result (Appendix A.2).

Revised abstract sentence: "Single-stage training needs two pieces to reach its best success rate: regularized latent actions and variance-reduced routing (VRR). VRR removes a posterior-variance penalty from the labeled prediction loss. Without it, a warm-up stage collapses every latent dimension, and single-stage training loses 14 points.”

AuthorsOpenReview ↗

Summary of new results and revisions (2/2)

Summary of new results and revisions (2/2)

Numbering. Table, figure, equation and appendix numbers in our replies refer to the revised PDF. Changes from the submitted version: Table 2 is new (the controls in this response), so the full-results table, submitted Table 2, is now Table 3, and submitted Tables 3–5 are now Tables 4–6. Table 1, Figures 1–4 and Proposition 1 keep their numbers. The posterior-variance expansion (submitted Eqs. 12 and 21) is now Eq. 8 in Section 3.4, the policy loss (submitted Eq. 19) is Eq. 14, and the other equations shift accordingly. Section 5.3 (ablations) and Appendix C (evaluation protocol and all new controls) are new, so submitted Appendices C–H are now D–I.

(3) Relation to CLAM and CoLA-World

Our Section 2 was misleading about CLAM. Its second stage trains only the policy; its inverse model, forward model and action decoder are trained together, with labeled batches grounding the latent actions during that stage. CoLA-World also reports that joint training from random initialization does not collapse. We removed the claim to be first to co-train from scratch, and the claim that prior continuous latent-action models ground their actions only afterwards. What JEVA adds is (a) stochastic latent actions that can be sampled and searched; (b) a latent forward model that is itself the planner, whereas CLAM's forward model reconstructs pixels as a training signal and the CLAM paper never plans with it; and (c) VRR, which handles the posterior variance that stochastic latent actions bring into the prediction loss, a problem CLAM's deterministic latents never meet. Section (1) compares against CLAM at the same budget.

Other fixes in the revision (changes in blue): abstract sentence, acknowledgments, duplicate citation, typos; Table 1b: 68.6 → 67.3 ± 2.5 (the earlier value was computed over 500 episodes per seed, not 100) and 48.33 ± 1.27 → 48.3 ± 1.5 (per-seed counts 47/48/50); all ± entries now sample std; Fig. 3a LeWM point now matches Table 3; active-dimension criterion defined (Var(μ_d) > 0.01).

AuthorsOpenReview ↗

Summary for the AC

Strengths noted by reviewers: realistic problem and simple model (WNTo); a controlled test of the proposed intervention and an appropriately qualified bias–variance analysis (ivSs, WNTo); label-efficiency gains on Cube-Double/Triple under both planning and learned policies (8aV4, ivSs); an honest account of the Reacher limitation (8aV4, ivSs).

A summary of concerns and what we did:

  • Baselines did not use the unlabeled pool (all reviewers): added pseudo-label LeWM (PushT 64.064.0 vs JEVA 75.075.0; Cube-Single 69.769.7 vs 78.078.0), JEVA without the pool (19.019.0) and a pool-size sweep. CLAM at the same budget: its action decoder does not recover actions beyond the labeled data (R2=−0.53R^2 = -0.53 on unlabeled transitions, vs 0.840.84 for JEVA), and its policies reach 16.0%16.0\%.
  • VRR's role was overstated (all reviewers): wording revised. New controls show that the best success rate needs both VRR and latent-action regularization, that routing must depend on labels (mean/mean ≤5%\leq 5\%), and that VRR works by removing a posterior-variance penalty from the labeled loss, not by lowering gradient variance (MC-16 60.760.7 vs 75.075.0). VRR also helps CEM on Cube-Single (80.780.7 vs 73.373.3) and Cube-Double (55.055.0 vs 48.048.0), and ties under BC, as the mechanism predicts.
  • Positioning vs CLAM/CoLA-World (8aV4, ivSs): Section 2 corrected; "first to co-train from scratch" removed.
  • Statistics and presentation (WNTo, ivSs): fixed in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.