acceptodds
Under review as a conference paper at ICLR 2027

Latent World Models Need to Also See the Details: Keyframe-Aware Visual Alignment

Abstract

Latent world models built on the Joint Embedding Predictive Architecture (JEPA), such as LeWorldModel, learn environment dynamics entirely in latent space and avoid the cost of pixel-level reconstruction. This purely latent paradigm, however, has two fundamental limitations. In fine-grained manipulation, the target object typically occupies only a tiny fraction of the frame, and the encoder's information bottleneck can suppress visual details critical to success. Moreover, the absence of any decoder that maps latents back to an observable space makes the long-horizon accuracy of latent rollouts unverifiable, allowing an overconfident yet physically implausible plan to pass unnoticed. We introduce **KAVA** (Keyframe-Aware Visual Alignment), a hybrid framework that augments a JEPA next-embedding objective with a temporal-saliency key-frame selector—combining the JEPA-loss gradient norm with an action-perturbation uncertainty score—and applies a pixel-level reconstruction loss only to a small subset of information-dense frames per trajectory. The reconstruction gradient is stopped at the predictor, preserving the latent signal the JEPA objective is designed to learn, and the decoder simultaneously serves as an online probe for the consistency of imagined rollouts. Across four image-based control benchmarks, KAVA achieves the best planning success rate among four representative latent world-model baselines. Selective, saliency-driven observation supervision is therefore a promising recipe for retaining the efficiency of reconstruction-free world models without sacrificing visual fidelity or rollout verifiability.

Primary area:
reinforcement learning
Keywords:
keyframe-aware visual alignment, joint embedding predictive architecture, latent world models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ct8VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents KAVA to address insufficient preservation of fine-grained visual information and limited reliability monitoring in JEPA-based latent world models. KAVA combines JEPA-loss gradients and action-perturbation sensitivity to select keyframes for sparse reconstruction supervision, with reconstruction gradients updating the encoder and decoder but not directly the dynamics predictor. It also introduces cSSIM, which measures the consistency of decoded images under small latent perturbations to filter unstable candidate rollouts during planning. Experiments on four visual control benchmarks show a 4.75% improvement in average success rate over LeWM, supported by rollout-filtering ablations, hyperparameter sensitivity experiments, and representation analyses.

Critical strengths

1、KAVA presents a coherent framework that uses a shared decoder for selective visual supervision during training and rollout stability screening during planning.

2、Experiments across four visual control benchmarks demonstrate improved average performance over LeWM, while ablations consistently support the contribution of rollout screening.

Critical weaknesses

1、Insufficient validation of keyframe selection. Without comparisons against random or uniform selection under the same reconstruction budget, the benefits of the proposed selection strategy remain unclear.

2、Limited validation of cSSIM as a reliability indicator. cSSIM measures local decoder stability, but its relationship with actual prediction errors is not evaluated. Improved planning performance alone does not establish reliable detection of erroneous rollouts.

Questions for authors

1、How does KAVA’s keyframe selection compare with random or uniform selection under the same reconstruction ratio and training budget?

2、Does cSSIM correlate with actual rollout prediction errors? Could you provide quantitative evidence of its ability to detect erroneous trajectories?

Reviewer_aiPNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KAVA, which augments a JEPA world model with selective image reconstruction and decoder-based rollout pruning. During training, it selects keyframes using a combination of latent prediction-loss gradients and sensitivity to action perturbations. Reconstruction supervision updates the encoder and decoder without directly updating the dynamics predictor. During planning, the decoder measures image consistency under small latent perturbations and rejects candidates with low consistency. Experiments cover four simulated control environments, with additional analyses of reconstruction quality, object-position decodability, attention, and latent geometry.

Critical strengths
  1. Interesting adaptive supervision mechanism. Combining prediction-loss sensitivity with action sensitivity provides a plausible way to allocate a limited reconstruction budget. The selector adapts during training and requires no task labels.

  2. Meaningful results for the complete system. KAVA improves average planning success over LeWM from 90.50% to 95.25%, including a substantial increase from 76% to 88% on OGB-Cube. It matches or exceeds the compared JEPA methods on all four tasks.

  3. Useful experimental controls and reporting. LeWM and Sub-JEPA are retrained with the same backbone and training configuration, and the main results include means and standard deviations over five seeds. The rollout-pruning ablation helps distinguish training changes from planning changes.

  4. Supporting representation analyses. Improved position probes provide quantitative evidence that task-relevant geometry remains accessible. The authors also appropriately acknowledge that decoder stability cannot establish physical correctness.

Critical weaknesses
  1. The benefit of the central keyframe-selection mechanism is insufficiently isolated. There are no comparisons against random or uniformly spaced frames at the same reconstruction budget, or against dense reconstruction using the same architecture. The perturbation ablation supports one component of the selector, but does not establish the advantage of adaptive selection over simpler reconstruction schedules. The proposed explanation also remains incomplete. Selected frames still receive a whole-image reconstruction loss, which may emphasize background appearance rather than small objects or contact regions. Attention maps and position probes provide supporting evidence, but do not identify where reconstruction accuracy improves or establish that these improvements cause better planning.

  2. The interpretation of cSSIM as a rollout-reliability measure lacks direct validation. cSSIM measures the decoder’s response to latent perturbations. Low consistency need not imply an inaccurate predicted state, while a smoothly incorrect prediction can receive high consistency. The paper acknowledges the latter limitation, but does not directly demonstrate that low-cSSIM predictions have larger errors or more physical violations. This matters because pruning accounts for much of the observed improvement. Without pruning, KAVA averages 90.00%, compared with 90.50% for LeWM; enabling pruning increases success to 95.25%. These results support the complete system’s usefulness, but leave uncertainty about whether selective reconstruction improves the dynamics representation or primarily supplies a useful planning filter.

Questions for authors
  1. Selector contribution: How does the proposed selector compare with random and uniformly spaced frames at the same reconstruction ratio, with rollout pruning held fixed? Is there a dense-reconstruction comparison using the same encoder and decoder?
  2. Spatial relevance: Equation 7 reconstructs the entire selected image. What evidence shows that temporal selection directs supervision toward task-relevant details? Do reconstruction improvements occur primarily around objects/contact regions or in the background?
  3. Pruning reliability: Does cSSIM correlate with prediction error against actual future states? Please clarify the candidate rejection rate and sensitivity to the pruning threshold. Would applying the same filter using the post-hoc baseline decoders recover a substantial portion of the gain?
  4. Architecture clarification: Appendix C specifies a ViT-Tiny encoder but describes the transposed-convolutional decoder as “mirroring” it. Does this refer only to reversing the image-to-latent mapping? Please specify the decoder’s initial spatial shape and layer configurations.
Reviewer_SHZxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

KAVA adds sparse reconstruction and rollout filtering to LeWM. Frames are selected using JEPA-loss gradients and action-perturbation sensitivity. At planning time, cSSIM measures how much a decoded prediction changes under a small latent perturbation, and low-scoring candidates are rejected. Table 1 reports a four-task mean success rate of 95.25%, compared with 90.50% for LeWM (PDF p.5).

Critical strengths
  • Control results The full system matches or exceeds both JEPA baselines on all four tasks, with five-seed means and standard deviations. The OGB-Cube improvement over LeWM, from 76% to 88%, is worth explaining

  • Ablations The paper reports what happens without filtering and studies selection hyperparameters on PushT. It also explicitly acknowledges that action sensitivity is not epistemic uncertainty and that decoder smoothness does not establish physical correctness .

Critical weaknesses
  • Frame selection Without filtering, KAVA averages 90.00%, versus LeWM's 90.50%; filtering brings it to 95.25%. Reconstruction could still help by producing a representation on which filtering works better. However, that interaction has not been tested: LeWM with the same filter is missing, as is a random-frame reconstruction control. I therefore cannot tell whether the proposed selector matters, whether sparse reconstruction is enough, or whether the gain mainly comes from filtering. The PushT gradient-only result does not settle this.

  • cSSIM and prediction error Equation 8 tests whether a small perturbation of a latent changes the decoder's output. A wrong prediction can still be locally smooth, as the authors acknowledge. Figure 5 does report rollout-quality curves, but it does not show whether cSSIM identifies erroneous candidate rollouts or how often accurate ones are rejected. The planning gains support using the filter in these tasks; they do not establish its accuracy as a diagnostic. I would like to see cSSIM scores checked against simulator state errors.

  • Real-world validation All four evaluations use simulated control tasks. The paper is motivated by small, task-critical visual details, yet it does not show whether the selector or cSSIM remains useful with real camera noise, clutter, occlusion, and uncertain contact. Real robotic trajectories and a focused manipulation experiment would directly test this gap and give firmer support for the claims about visual fidelity and rollout reliability.

Questions for authors

Please addressing the above concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.