acceptodds
Under review as a conference paper at ICLR 2027

Motion-JEPA: Physically Structured Residual Predictive Representation Learning for Autonomous Driving

Abstract

Driving world models must predict and represent future scenes relevant to driving decisions. Changes in high-dimensional representations arise both from referenceframe changes caused by ego motion and from temporal evolution of the surrounding environment. When a model directly predicts the complete future, a predictor with limited capacity must fit both known geometric effects and unknown scene changes. This can impair future-scene prediction, produce representations that are less useful for driving decisions, and impose substantial computational costs. We propose MotionJEPA, a physically structured residual predictive representationlearning framework. It uses low-dimensional, interpretable ego-motion variables to construct a physical reference evolution for high-dimensional representations, and then learns the conditional residual not explained by that reference. The method combines explicit SE(2) motion transport, residual prediction around a physical reference, and static-dynamic diagnostics of residual sources. In 10,000 training steps on NAVSIM, model A3, which combines motion transport with an environment-residual predictor, reduces masked Smooth-L1 error by 7.76% relative to A1, which uses motion conditioning alone. A3 also achieves a 7.35% lower best validation Macro Smooth-L1 and a 7.69% lower overall test Smooth-L1 than A1. The additional validation metric, mean Intersection over Union (mIoU), also improves.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Joint-Embedding Predictive Architecture, Motion Prior, Latent Representation Learning, Vehicle Dynamics, Predictive World Modeling, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bwySOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how ego motion can help predict future driving scenes. MotionJEPA first moves the current bird’s-eye-view (BEV) map into an estimated future ego frame. A neural network then predicts a signed correction to this transported map.

The experiments use seven-channel semantic BEV maps built from NAVSIM annotations, with prediction horizons of 0.5, 1.0, and 1.5 seconds. The paper compares direct prediction, motion-conditioned prediction, geometric transport alone, and transport with a learned residual. Compared with the motion-conditioned baseline, the proposed method reduces test Macro Smooth-L1 by 7.69%. Validation mIoU increases from 0.5906 to 0.6585. The paper also uses ground-truth future poses to study errors caused by motion extrapolation.

Critical strengths

Models A1 and A3 use the same motion information and have the same number of trainable parameters. Their difference shows the effect of using motion as an explicit geometric operation rather than only as a network condition. Model A3 improves both test Smooth-L1 and validation mIoU.

It notes that the residual may include motion-estimation errors, sampling errors, visibility changes, and object motion. The ground-truth-pose comparison on common support is a useful check. This makes the interpretation more credible than claiming that the network separates only true environmental motion.

Critical weaknesses

The input and target are fixed semantic BEV maps built from annotations. The reported setup does not demonstrate a learned joint-embedding target or evaluate representation quality. Transport followed by learned correction is also closely related to the physics-plus-learning and feature-warping approaches discussed in the paper. The related-work section does not clearly explain what new capability or insight distinguishes MotionJEPA from these ideas.

The study uses one dataset setup, short prediction horizons, and 10,000 training steps. It does not report results across different random seeds or confidence intervals. At 1.5 seconds, A3’s test Smooth-L1 improvement over A1 is only about 1.07%. A3 also has higher MAE than transport alone and does not improve pedestrian or bicycle IoU over that baseline. These results do not disprove the method, but they make it important to show that the gains are reliable and explain which errors the residual corrects. The current comparisons also combine geometric alignment with the additive residual, so their separate effects are unclear.

Questions for authors
  • What makes the evaluated Stage 1 model a joint-embedding predictive architecture? Are any context or target representations learned beyond the supervised semantic prediction network described in the paper?

  • Were the results obtained from a single training seed? Please provide the missing training settings and, if available, variation across seeds or paired confidence intervals across test logs. Does the A1-A3 gap remain after longer training?

  • How does A3 compare with a model that receives the transported BEV and the same motion condition, but directly predicts the future BEV without adding the transport baseline? This would help separate the value of alignment from the value of residual prediction.

  • Can you report the common-support CTRA-versus-ground-truth-pose errors by static and dynamic class and by horizon? Together with ego-pose extrapolation errors, this would help assess how much of the residual addresses motion-prior errors rather than changes in other road users.

Reviewer_szNfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MotionJEPA predicts future semantic BEV grids constructed from NAVSIM annotations. It extrapolates ego motion from historical states, transports the current grid into the predicted future ego frame and learns a signed residual correction. The main experiment compares this design with a motion-conditioned direct predictor under matched motion inputs and parameter counts. The appendix examines individual classes, prediction horizons and the effect of using ground-truth poses.

Critical strengths
  1. The main comparison controls important sources of confounding. A1 and A3 receive the same historically extrapolated motion and have equal parameter counts. Fixed semantic inputs separate prediction structure from representation quality. The improvement is supported by held-out continuous errors and validation semantic quality. A3 also improves over transport alone in validation mIoU, including a clear vehicle-class gain. The learned correction therefore adds value beyond preserving the geometric reference.

  2. The interpretation of the physical reference is careful. The appendix acknowledges that the residual includes motion-prior error, representation effects and environmental changes. Its comparison between extrapolated and ground-truth poses holds visible support fixed. True future poses are used as motion inputs only in the diagnostic baseline. This makes the study more informative than an aggregate comparison between two predictors.

Critical weaknesses
  1. The main contribution is a controlled demonstration of a particular prediction structure. Warp to the Future combines learned feature transport with direct prediction, while APHYNITY augments physical dynamics with learned corrections. MotionJEPA's specific increment is to use historical CTRA extrapolation for deterministic geometric transport and learn a zero-initialized, signed correction around that reference. The matched A1 and A3 comparison supports this structure, and the comparison with A2 shows that learned correction adds value beyond transport alone. The class, horizon and common-support pose diagnostics further characterize these gains. These are useful empirical findings. However, the methodological advance remains a particular application of established transport and correction principles. The claimed quantitative explanation of the transport contribution is also incomplete, as detailed below. In my assessment, the resulting methodological and explanatory contribution is insufficient for acceptance.

  2. Contribution 3 promises to quantify the proportion of scene change explained by transport across static and dynamic classes and prediction horizons. The reported Copy and A2 errors already provide an aggregate transport comparison, and the common-support comparison with true poses quantifies the effect of the motion prior. The class and horizon results also show where prediction quality improves. The missing step is an explicit definition of the claimed transport-explained proportion and its breakdown across those groups. The paper should connect its existing measurements to this claim or narrow the claim. This matters because the diagnostic analysis is the main opportunity to establish explanatory insight beyond the forecasting improvement.

Questions for authors
  1. What quantity represents the transport-explained proportion claimed in Contribution 3, and which reported results establish it across static and dynamic classes and prediction horizons? The existing class, horizon and pose diagnostics are useful, but their connection to this specific claim remains unclear.

  2. Were the motion-conditioned direct predictor and the residual predictor trained with the same optimizer, learning-rate schedule and batch size? How many independent training runs underlie their comparison, and how does the observed variation compare with the aggregate and horizon-wise performance differences?

Reviewer_efeVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

MotionJEPA predicts future semantic BEV maps by first applying an explicit SE(2) transformation based on extrapolated ego motion, then learning a signed correction to the transformed map. Experiments use seven-channel BEVs constructed from NAVSIM annotations, with prediction horizons of 0.5–1.5 seconds. The paper compares direct prediction, motion-conditioned prediction, geometric transport, and transport with residual prediction. The proposed model reduces test Macro Smooth-L1 by 7.69% relative to the motion-conditioned baseline and improves validation mIoU.

Critical strengths
  1. A clear geometric motivation supported by a controlled comparison. A1 and A3 receive the same motion information and have identical parameter counts (Table 1). A3 reduces test Macro Smooth-L1 from 0.004546 to 0.004196 and increases validation mIoU from 0.5906 to 0.6585 (Tables 3 and 5). These results support explicitly applying ego-motion geometry before learned prediction in the evaluated setting.
  2. Useful diagnostics and appropriately cautious interpretation. The paper distinguishes environmental changes from pose-extrapolation and sampling errors. Its ground-truth-pose diagnostic compares transport methods on the same valid region (Table 8), avoiding a comparison over different spatial supports. This analysis helps explain the limitations of geometric transport and the errors a residual predictor must correct.
Critical weaknesses
  1. The experiments do not establish the claimed representation-learning contribution. The evaluated inputs and targets are fixed semantic maps generated from annotations (Section 4; Appendix B). No learned context–target encoder setup or transfer experiment is presented. The results therefore demonstrate supervised semantic-BEV forecasting, while the connection to JEPA representation learning and useful driving representations remains untested. The stated Stage 1 scope should be reflected more clearly in the paper’s claims.

  2. The baselines do not isolate the benefit of residual prediction. A1 and A3 differ in both spatial preprocessing and output structure. A predictor that directly reconstructs the future from the transported BEV would help separate the benefit of alignment from the additive residual. An ordinary residual predictor operating on the original BEV is also missing. This matters because pose-driven warping followed by learned refinement already appears in related work such as GAST, Section 3.3.

  3. Evaluation excludes part of the future scene. All A0–A3 results use the CTRA validity mask and cover only locations sampled from the current field of view (Section 4). This is a fair comparison on overlapping regions, but it leaves prediction of newly exposed regions untested. Results cover only short horizons, and A3’s test Smooth-L1 advantage over A1 falls to 1.07% at 1.5 seconds (Table 4), limiting evidence for benefits beyond short-term alignment.

  4. The gains are not consistent across metrics or dynamic classes. A3 has higher test MAE than transport alone: 0.020921 versus 0.014766 (Table 3). On validation, pedestrian IoU decreases slightly relative to A2, while bicycle IoU is unchanged (Table 6). The vehicle improvement is useful, but these results leave uncertainty about how reliably the learned correction improves predictions for other moving road users. Test-set semantic metrics would strengthen this assessment.

5.Training and efficiency evidence is incomplete. Results are reported for a 10,000-step budget without variation across training runs. The manuscript also omits optimizer, learning-rate, and batch-size details. No timing or memory measurements support the motivation concerning computational cost. Consequently, the stability of the gains and any practical training-efficiency advantage remain unclear.

Questions for authors
  1. Which components learn a representation beyond the annotation-derived semantic maps? What experimental evidence supports describing the evaluated system as JEPA representation learning?

  2. How does A3 compare with direct prediction from the transported BEV and with residual prediction from the original BEV, using comparable initialization and training budgets?

  3. Can the authors report test mIoU separately on overlapping regions and the full future grid, including locations outside the current field of view?

  4. Do the pedestrian and bicycle results persist on the test set and across training runs? What changes does the residual actually make in these channels?

What are the complete training settings, variation across seeds, and measured training/inference costs? Does A3 retain its advantage with a longer training budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.