acceptodds
Under review as a conference paper at ICLR 2027

MORE: Learning Motion-Oriented Registers for Driving World Models and Planning

Abstract

By predicting future states, latent world models provide rich self-supervision for end-to-end driving, but often inherit dense features from pretrained visual encoders as their world-state interface. This creates a mismatch: visual pretraining produces information-rich, spatially dense features, but their adaptation for motion-conditioned world modeling and planning is deferred to downstream training. The dense grid also makes future prediction more computationally expensive as visual resolution increases. We therefore argue that video pretraining should directly learn state representations suited to downstream world modeling and planning, rather than relying on downstream training to adapt inherited visual features. We propose ***MORE***, a V-JEPA framework that learns **M**otion-**O**riented **RE**gisters during video pretraining as a shared state interface for driving world models and planning. MORE injects temporally aligned ego motion into a small set of registers and uses them as the sole context for masked patch prediction, channeling dense patch-level supervision through a compact state bottleneck. A complementary blockwise register-masking objective further shapes predictive structure in the register space. During downstream training, the same register space defines the future states predicted by the world model and provides shared scene memory for trajectory generation and scoring. On NAVSIM-v2, MORE outperforms the state of the art by 1.6 in EPDMS, while achieving superior efficiency with only 1/10 the inference latency of DriveLaW. These results support learning the state interface for world modeling and planning directly during video pretraining.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, World Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6ehJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MORE learns compact, motion-conditioned register tokens during video pretraining and reuses them for driving world modeling and planning. The method builds on V-JEPA and introduces two complementary objectives: predicting masked patch features using only register tokens as context, and predicting masked register features. During downstream training, a world model predicts future registers, which are combined with observed registers to generate and score candidate trajectories.

The paper reports 92.5 EPDMS on NAVSIM-v2 and 93.2 PDMS on NAVSIM-v1, together with zero-shot closed-loop results on 436 HUGSIM scenarios. Ablations examine motion conditioning, the pretraining objectives, future-state prediction, and the number of registers. The reported inference latency is 110 ms on a single RTX 4090.

Critical strengths

The paper makes a convincing case for training the registers during pretraining. Table 4 shows that simply adding registers has little effect. The larger improvement comes from making them the only source of context for masked patch prediction: EPDMS rises from 85.6 to 88.5. Register masking and motion conditioning bring further gains. This is a useful finding because it shows how to make a compact representation retain information that matters for planning.

The approach performs well across several evaluations. MORE achieves strong aggregate results on both NAVSIM versions, and the closed-loop HUGSIM evaluation provides additional evidence that the method remains useful when its actions affect later observations. The reported latency of 110 ms also makes the approach practically interesting. Taken together, these results support the value of using the same compact representation for prediction and planning.

Critical weaknesses

It is still unclear how much the future-prediction objective contributes. In Table 4, removing future prediction reduces EPDMS from 91.8 to 91.1. However, this also removes the predicted tokens from the planner’s memory, so the comparison changes more than the prediction objective. The gain could come partly from the extra computation or memory available to the planner. A comparison that keeps those tokens but removes their future-state supervision would make the contribution clearer. This matters because learning a state space suited to future prediction is a central claim of the paper.

Some conclusions are stronger than the evidence supports. The paper does not report variation across training runs, so it is difficult to judge the smaller gains in the ablations. The HUGSIM results also deserve a more balanced discussion: MORE has the best average scores, but its advantage is concentrated in Easy and Medium scenarios. It does not lead on Hard or Extreme scenarios, and its Extreme HD-Score is 11.1, compared with 18.1 for Latent-WAM. The overall results are encouraging, but they provide less support for robustness in the most difficult settings.

Questions for authors
  • Could you keep the future tokens in the planner but train them only with the planning losses, without the future-prediction loss? This would help show whether the improvement comes from predicting future states or simply from adding more tokens and computation.

  • How well does the model predict future registers on unseen data, and how does accuracy change when predicting further ahead? Does it perform better than simply keeping the current registers unchanged? Since each variant learns different representations, please use the same evaluation method to make the comparison fair.

  • Can you report variation across training seeds for the full model and the key ablations?

  • Please specify the patch and register masking ratios, block-sampling rules, and downstream world-model architecture for reproducibility.

Reviewer_vZV8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that driving world models usually inherit dense features from pretrained visual encoders and leave their adaptation to motion-conditioned prediction and planning to downstream training. MORE proposes to learn a compact state during video pretraining instead. In Stage 1, V-JEPA 2 is further pretrained on OpenScene with 8 learnable registers per 2-frame chunk. Each register receives the ego motion of its chunk. Masked patch features must be predicted using only the registers as context and masked blocks of registers are predicted from the visible patches. In Stage 2, only the registers are kept. A non-autoregressive world model predicts 32 future registers corresponding to 4s from 16 observed registers, the past ego motion and the current ego state, supervised by EMA-encoder targets. The observed and predicted registers form a shared memory from which a generator produces 64 candidate trajectories and a scorer ranks them. The paper reports SOTA results on NAVSIM-v1 and NAVSIM-v2 and show zero-show closed loop evaluation on HUGSIM scenarios. Table 4 ablates registers, register-mediated prediction, register masking and motion conditioning design components.

Critical strengths
  • S1. The method proposes to have only one compact state for world modeling and planning. Routing all masked-patch prediction through a small set of registers and then reusing the same register space as the world-model state and the planners memory is a simple and coherent design. Ablation in Table 4 compares different proposed design choices very well including -- register-mediated prediction, blockwise register masking and motion conditioning. The approach is a practical alternative to predicting dense future grids and the ablation shows which parts of the model are critical to achieve this.

  • S2. The proposed method show strong results on open-loop and closed-loop benchmarks at moderate cost. The method reaches 93.2 PDMS on NAVSIM-v1 and 92.5 EPDMS on NAVSIM-v2 (corrected evaluator), the best average route completion and HD-Score on 436 HUGSIM scenarios without HUGSIM-specific training

  • S3. The paper provides an anonymized code package, full hyperparameters and data splits.

Critical weaknesses
  • W1. The paper defines the mismatch through three properties of the state, namely information sufficiency, ego-motion sensitivity and predictive structure, but none of them is measured for dense features versus registers. The only evidence is end-task scores in Table 4, where ego-motion conditioning alone brings the dense baseline level with the register model without motion. Since the encoder is also fine-tuned in Stage 2, the gains cannot be attributed to the register state interface rather than to motion conditioning and fine-tuning.

  • W2. The contribution of future prediction is not fully established. The evaluation of the world model as a predictor is missing. There is not future register prediction error reported or no comparison with simply copying the last observed state. The only evidence is the PDMS score, which is not fair because observation only uses 16 tokens vs 48 tokens (with future). Additionally, no run-to-run variance is reported. This matters because the paper presents MORE as a world model for planning.

  • W3. Table 3 has no backbone or pretraining column, although MORE uses a ViT-L, while many baselines use smaller backbones. On NAVSIM-v2, MORE reports only the corrected evaluator, so it cannot be compared with baselines that report only the original evaluator (for example Drive-JEPA).

  • W4. MORE performs worse on Hard and the Extreme splits, so the gains come from the Easy and Medium splits.

Questions for authors
  • Q1 (W1). The paper defines the state-interface mismatch through information sufficiency, ego-motion sensitivity and predictive structure, but supports it only with end-task scores. Could you measure these properties directly for #1, #2, #3 and #8 in Table 4, for example with a linear probe for ego motion and the future-state prediction error? Could you also run Stage 2 with the encoder frozen, for both MORE and the V-JEPA baseline? If the registers resolve the mismatch, they should probe better, predict better and lose less when frozen.

  • Q2 (W1). For #1 and #2 in Table 4, was the encoder pretrained on OpenScene for the same number of steps as MORE?

  • Q3 (W1). Ego motion enters both the observed registers and the world model. Could you ablate the motion input to the world model separately and clarify whether #6 also removes motion in Stage 2?

  • Q4 (W1). The controlled evidence comes only from V-JEPA 2 variants, where ego-motion conditioning alone raises EPDMS from 84.9 to 89.4 and the register design adds 2.4 more. Other driving world models also learn their state during video pretraining with action conditioning, such as Epona (ICCV 2025) and Orbis (NeurIPS 2025). Could you use the frozen states of one of these models as the state interface in the same downstream world model and planner? This would show whether the gain comes from the register design or from motion-conditioned pretraining in general.

  • Q5 (W2). Could you report the prediction quality of the world model, namely the future-register error per future chunk, compared with simply copying the last observed registers?

  • Q6 (W2). Could you add two rows to Table 4 - One that would keep the world model and its 32 output tokens but remove the future-prediction loss. The other one - that would give the planner the same 48-token memory from observations only. These would separate the effect of predicting the future from the added capacity. Could you also report run-to-run variance over three seeds for Table 4?

  • Q7 (W2). The future target registers are encoded with the logged future ego motion, so the predicted registers may mainly carry the expert future motion. How do the results change if future motion is zeroed in the target encoder? How well can the experts future motion be decoded from the predicted registers?

  • Q8 (W2). Could you describe the architecture of the world model?

  • Q9 (W3). Could you also report the NAVSIM-v2 score of MORE with the original evaluator, so that it can be compared with methods that report only that version, such as Drive-JEPA?

Reviewer_XvohOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MORE, which learns motion-oriented registers during video pretraining and reuses them as a shared state interface for driving world modeling and planning. The approach combines ego-motion conditioning, masked patch prediction exclusively through registers, and blockwise register masking. A non-autoregressive world model then predicts future registers, which are combined with observed registers for trajectory generation and scoring.

Critical strengths
  1. The paper addresses the mismatch between pretrained visual features and the states needed for prediction and planning. Requiring patch prediction to pass exclusively through registers is a particularly compelling way to train the compact interface.

  2. Without motion conditioning, pretrained registers (#6) outperform registers introduced only downstream (#3) by 3.4 EPDMS. Register-mediated patch prediction alone improves #4 over #5’s predecessor setting by 2.9 points, showing that the training objective matters beyond simply adding tokens.

Critical weaknesses
  1. The motion-conditioned setting lacks a matched downstream-compression baseline. The #3–#6 comparison supports the register pretraining package without motion conditioning. However, there is no configuration combining patch-motion pretraining, as in #2, with registers introduced only downstream, as in #3. This would more directly test whether learning registers during pretraining retains an advantage when both approaches use motion information. It would also sharpen the distinction from existing compact planning-token approaches such as DrivoR and NTR.

  2. The contribution and scope of future prediction need clarification. Removing future prediction reduces EPDMS from 91.8 to 91.1, but #7 also changes the planner’s inputs and potentially its computation and training objectives. The comparison therefore does not isolate the value of future-state supervision. Moreover, Eq. (12) predicts one shared future representation without conditioning on candidate trajectories. This provides predictive scene context, but does not model the distinct consequences of candidate actions. The paper should state this boundary explicitly and moderate its claim that the ablation confirms decision-relevant scene evolution.

  3. Closed-loop gains are not consistent across difficulty levels. MORE trails UniAD on both Hard route completion and HD-Score, and its Extreme HD-Score is 11.1 versus Latent-WAM’s 18.1. The near-stop failures in the appendix further motivate a more targeted analysis. The results support improved average closed-loop performance, but not a consistent advantage in difficult scenarios.

Questions for authors
  1. Could you compare MORE against patch-motion pretraining followed by downstream register compression, using the same initialization, inputs, register count, and training budget?

  2. What exactly is retained in #7 during training and inference? Could you preserve the world-model computation, future queries, and planner inputs while removing only LpredL_{\mathrm{pred}}, to isolate the contribution of future-state supervision?

  3. Is future prediction entirely independent of candidate trajectories? Please clarify the intended scope of the world model and how its shared future representation supports candidate scoring.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.