acceptodds
Withdrawn from ICLR 2027

Throw D4RT into the Future: Motion World Models from 4D Reconstruction Prior

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Feedforward 4D reconstruction models learn a spatiotemporal prior of dynamic scenes from large collections of 4D-annotated video. Motion forecasters, in contrast, still rely on 3D trajectory annotations that are expensive to collect and limited in domain. We present FutureD4RT, a world model that forecasts in the feature space of a frozen 4D reconstruction model and is trained on raw video and timestamps alone. A tokenizer compresses each transition between consecutive scene states into one continuous delta token, a predictor forecasts future tokens, and the frozen decoder of the reconstruction model reads 3D positions from the predicted states. We build the same pipeline on two backbones, D4RT and Point4D. The delta token carries transition-specific content, since the correct token lowers reconstruction loss by 95.1% against copying the preceding state while tokens of other transitions do not. Forecasts beat copying the last state in feature space, and updating the random query vectors of the predictor from the waypoints of one supplied track lowers the average 3D error of the other points on PointMotionBench by 12 to 22% with D4RT and 6 to 20% with Point4D. These gains use an encoding of the full clip, as in training. From three observed frames alone, the forecast stays at the level of holding the last observed position, which we report as the main limitation. The results indicate that a frozen 4D reconstruction model can supply the state space of a motion world model without motion labels.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Motion Forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aZikOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FutureD4RT, a motion world model that predicts future 3D geometry in the feature space of frozen 4D reconstruction models. It uses a delta tokenizer to compress state transitions, a transformer to predict future tokens, and the frozen reconstruction decoder for geometric readout.

Critical strengths
  1. Leveraging pretrained 4D reconstruction models for motion forecasting without additional 3D trajectory annotations is an interesting direction.
  2. The compact delta-token representation and trajectory-based steering mechanism are interesting. The experiments demonstrate that the learned features capture meaningful motion information.
Critical weaknesses

The main limitation is that the proposed method does not demonstrate effective forecasting from observed frames alone. Its performance is comparable to holding the last position and worse than simple damped velocity extrapolation. The positive results largely rely on full-clip context, where the bidirectional encoder has access to future frames.

Questions for authors

Same as Weakness 1. Given that the method fails to outperform simple motion extrapolation under observed-only inference, what evidence supports its effectiveness as a future motion forecasting model? Could the authors address the train-test discrepancy caused by full-clip encoding?

Reviewer_WBjXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents a method to predict future motion states from a current scene state. The paper is built upon D4RT, a unified transformer for predicting dynamic scene geometry from a global scene representation. The paper further leverages DeltaTok to represent future motion states in a compressed token space based on the scene representation from D4RT. The training has the follow steps. In Stage 1, it first transforms the input video into a global scene representation using the encoder in D4RT, producing a dense state over time. A tokenizer encoder is then trained to predict a delta token that explains the difference of two consecutive dense states. A tokenizer decoder can then reconstruct a future state given a current state and its delta token. In Stage 2, the tokenizer encoder and decoder are frozen, and a predictor network is trained to predict future a sequence of delta tokens, which could be subsequently decoded to predict future dense scene states.

Critical strengths

Motion prediction is an important topic with potential applications in world modelling for video generation, robotics or autonomous driving. The proposed method is a timely contribution to this research direction.

The paper is built upon state-of-the-art techniques such as D4RT (CVPR 2026) and Point4D (a recent arxiv paper).

Critical weaknesses

The paper suffers from the following major issues.

  1. Limited novelty. The architecture is based on D4RT and DeltaTok. The conceptual novelty in the proposed method is rather limited. Two backbones are tested for scene states encoding, via D4RT and via Point4D. However, despite some empirical performance difference, the insight into adapting D4RT or Point4D is not well explained.

  2. Weak performance. The main results in Table 1 does not outperform previous work. All baseline results in Table 1 are taken from Table 1 in Molmo-Motion (as acknowledged in the table caption). It is not clear whether the comparison is fair as the baselines are not re-executed, but from the numbers the proposed method has inferior performance compared to the baselines. Results in Table 2 and Table 3 seem inconclusive.

  3. Presentation issues. The paper is somewhat hard to follow. I am a bit confused when reading introduction for the first time. The first five paragraphs are quite technical. These paragraphs, unfortunately, do not clearly explain to the reviewer what the problem is, why it is important, and how this paper offers a better solution than the previous work. At first glance, the three contribution paragraphs, especially the last two, remain too technical to understand the significance of the contributions.

Questions for authors

The authors are encouraged to address the following questions in the rebutttal.

  1. Novelty. Please clarify what conceptual and technical novelty the paper claims. How does the novelty generalise to new backbones beyond D4RT and Point4D?

  2. Evaluation. Please explain why the baselines in Table 1 are taken from Molmo-Motion. Is it a fair comparison or the baseline should be re-run? Please clarify the position of the performance of the proposed method w.r.t. the baselines.

Reviewer_34dPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes FUTURE D4RT, a 3D motion forecasting framework built upon frozen 4D reconstruction models. It compresses changes between consecutive scene states into continuous delta tokens, predicts future tokens, and uses the frozen reconstruction decoder to recover 3D point positions. The authors evaluate the framework with D4RT and Point4D on PointMotionBench and investigate token reconstruction, feature-space forecasting, and trajectory steering. However, under the observed-only setting with three input frames, the method performs comparably to a simple persistence baseline, leaving its effectiveness for genuine future motion forecasting insufficiently demonstrated.

Critical strengths

The paper presents an interesting framework that uses frozen 4D reconstruction models as the state space for 3D motion forecasting. The delta-token design provides a compact representation of state transitions, and the token reconstruction experiments offer evidence that these tokens encode transition-specific information.

Critical weaknesses

The main forecasting results rely on full-clip context. Because the reconstruction encoders use bidirectional attention, the context states may contain information from future frames. Therefore, these results do not establish effective forecasting from past observations alone.

Questions for authors

Can the authors provide forecasting results under a strictly observed-only setting, ensuring that no future-frame information is available to the encoder or predictor, and demonstrate improvements over persistence and velocity-based baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.