acceptodds
Withdrawn from ICLR 2027

One 4RROW, Every Target: for 4D Reconstruction and Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

27 views

Abstract

Feedforward models have made rapid progress on reconstructing 4D geometry and 4D tracking from dynamic video, and separately on incorporating semantics into static 3D reconstructions. However, no single model combines all three tasks. This paper introduces arrow, the first feedforward model to jointly perform 4D reconstruction, point tracking, and recognition from a single input video of arbitrary length, enabling extraction of semantic 4D instances of dynamic objects. Following D4RT, we use a powerful encoder coupled with an efficient point-wise decoder. We introduce object queries for instance recognition in such a way that they seamlessly operate alongside the point queries, and generalize the point query mechanism to videos of arbitrary length. To enable strong semantics we adapt DINOv3 for video and make it temporally aware, turning it into TimeDINO. On combined geometry and semantic prediction, arrow outperforms prior work on the static Scannet++ dataset while establishing a baseline on the dynamic Waymo dataset. Finally, we show our unified model can even outperform specialized video instance segmentation models in an open-vocabulary setting while also producing strong 4D reconstruction results.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
4D reconstruction, semantic segmentation, instance segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zU7zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes 4RROW, which combines TimeDINO video encoding, D4RT-style point-wise queries, and instance queries in the same feed-forward model to jointly handle dynamic geometry and instance recognition. The model can extract object motion and aggregate points from different frames to a selected reference time. The experiments report strong open-vocabulary VIS and competitive depth and point-cloud results. Joint geometry and semantic prediction is also evaluated on static ScanNet++ and dynamic Waymo

Critical strengths

The model performs strongly on open-vocabulary VIS while keeping competitive depth and point-cloud accuracy. In Figures 3–4 and the supplementary demonstrations, instance semantics can separate objects moving together, keep the static parts of objects, and support object-level point-cloud aggregation. These results make the value of the unified architecture more concrete

Both ScanNet++ and Waymo experiments include geometry quality in the evaluation. The oracle semantics and oracle geometry experiments on ScanNet++ further show which aspect limits the performance. This is more helpful than only reporting 2D segmentation results

Critical weaknesses

Figures 1, 3, 4 and the supplementary material show cross-frame aggregation, but I am still not clear how accurate the aggregated shapes are. I am also not sure whether the point-cloud errors in Table 1 already cover cross-time queries. DAP combines instance masks and depth, but does not directly evaluate the aggregated shapes. In Figure 4, the two methods also use different ways to extract the foreground, so the difference in geometry quality is not easy to see clearly. This is my main concern at present, because it relates to the new capability emphasized by the paper

Questions for authors

Which source/target time combinations are used in the point-cloud evaluation of Table 1?

What training and cross-frame instance association settings are used for the Pano3D baseline in the Waymo experiments?

1 comment
AuthorsOpenReview ↗

Answers to questions

We thank the reviewer for their careful feedback. We are happy that our main contribution, which is also identified as strength #1 by the other reviewers, was highlighted in the summary: To our knowledge, our work presents the first method to combine reconstruction, tracking, and segmentation in a single framework, which enables multi-view reconstruction of individual dynamic objects.

Cross-frame aggregation

We agree this is an important point. As explained in our submission (L294) there are no established benchmarks for 4D reconstruction and recognition on dynamic scenes. Moreover, for real-world scenes obtaining ground-truth for cross-frame aggregation under partial visibility is practically impossible. Therefore in our submission we showed this capability only visually. Nevertheless, it will be possible to establish a new protocol to measure this on a synthetic dataset and we will do so in the final version of our manuscript. Since to our knowledge (as also noted by Reviewer imRq) no other model supports this capability, this evaluation will establish a reference baseline for future work.

Point-Cloud Evaluation

Point cloud evaluation always uses the same source and target times according to established existing protocol. The reviewer is correct that it does not cover cross-time queries; Similarly, the new protocol promised above is intended to fill this gap.

Cross-frame on Waymo

Both Pano3D and 4RROW use object queries which cross attend to all tokens of all frames, giving them a global view and directly producing temporally consistent instances across the clip. In contrast, for this task the point queries reference the specific frame (t_src == t_tgt == t_cam) since the ground-truth segmentation annotations label the observed pixels per frame.

Extracting foreground for D4RT and 4RROW

4RROW predicts instances directly. D4RT is incapable of this so we can only use the inferior technique of thresholded motion prediction, which also means that with D4RT one cannot isolate static objects. Illustrating this difference, rather than comparing geometry quality, is the purpose of Figure 4; we will clarify this in the caption.

Reviewer_imRqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes 4RROW, a unified feedforward model for monocular-video 4D reconstruction, point tracking, instance segmentation, and open-vocabulary recognition. Building on D4RT's point-query decoder, it introduces TimeDINO with temporal RoPE and interleaved frame and global attention, sequence-level object queries inside the encoder, and content-aware frame pooling in place of fixed temporal lookup tables. Experiments cover three video instance segmentation datasets, geometry evaluation on Sintel and ScanNet, joint geometry and semantics on ScanNet++, and dynamic Waymo evaluation with a new DAP metric.

Critical strengths
  1. The paper unifies 4D reconstruction, point tracking, and instance recognition, enabling instance-conditioned dynamic geometry that is not provided by prior specialized systems.
  2. The same framework obtains strong open-vocabulary results on three video instance segmentation datasets while retaining competitive geometry prediction and providing joint evaluations on ScanNet++ and dynamic Waymo.
Critical weaknesses
  1. Key component ablations and a modular D4RT plus independent video-segmentation baseline are missing, so the necessity and synergy of unification are not established.
  2. Joint training noticeably degrades geometry relative to D4RT, especially increasing Sintel depth AbsRel from 0.171 to 0.251, without analysis of multitask interference.
  3. Content-aware frame pooling represents time through mean frame content and may struggle with small moving objects against static backgrounds or near-duplicate frames.
  4. Point tracking and genuine unseen-category generalization are not evaluated quantitatively, leaving the tracking and open-vocabulary claims under-supported.
  5. The model is trained on fixed 24-frame clips despite arbitrary-length claims, while DAP uses ground-truth scale alignment without adequate sensitivity analysis.
Questions for authors
  1. Could the authors add key component ablations and compare with D4RT plus an independent video instance segmentation model?
  2. How do single-task and joint training compare, and what causes the geometry degradation relative to D4RT?
  3. How does content-aware frame pooling compare with continuous timestamp Fourier embeddings on near-duplicate frames and small moving objects?
  4. Could the authors report standard point-tracking results and open-vocabulary VIS results with genuinely unseen categories?
  5. How do accuracy and cost scale with sequence length, and is DAP stable to scale alignment and its evaluation thresholds?
3 comments
AuthorsOpenReview ↗

Ablations part 1

We thank the reviewer for their thorough and constructive review. We are glad that our main contribution is identified as strength #1: Our work presents the first method to combine reconstruction, tracking, and segmentation in a single framework, which enables multi-view reconstruction of individual dynamic objects.

Ablation studies

As suggested by the reviewers, we have done several ablation studies. To ensure readability and highlight the overarching trends, our new studies report performance averaged across datasets rather than exhaustive per-dataset tables; we will include the complete breakdowns in the final paper and gladly provide any specific numbers during the discussion phase.

Effect of variable-frame support and TimeDINO on geometry - 500k steps, BS=64

In the following experiment we go from the original VideoMAEv2 backbone (results from D4RT paper) to using Content-Aware Frame-Pooling (CAFP) which enables variable-frame inference. Then we switch to the TimeDINO backbone. Finally we increase resolution from 256x256 to 384x384 which is the resolution at which most of the experiments in the paper are done since a higher resolution is essential for segmentation.

MethodDepth Estimation Mean S AbsRel (↓)Point Cloud Reconstruction Mean Scaled L1 (↓)Camera Pose Estimation Mean Pose AUC (↑)3D Tracking Mean 3D AJ (↑)
SpatialTrackerv2 (External Baseline)0.0880.7020.6670.182
Pi^3 (External Baseline)0.0870.5850.690unsupported
VideoMAE G — D4RT paper0.0810.4220.7350.274
VideoMAE G — CAFP0.0850.5250.7240.268
TimeDino H+ — CAFP0.0910.5030.7010.269
TimeDINO H+ — CAFP @3840.0830.4760.7120.280

As can be seen, when using the VideoMAE backbone, moving from fixed learned embeddings to content-aware frame-pooling (CAFP) results in a drop for point cloud reconstruction, while tracking, depth, and camera pose remain mostly on par. Moving then to TimeDINO recovers some of the point cloud reconstruction but results in a slight drop in camera pose estimation. The subsequent increase of resolution is beneficial across the board but the final model remains behind the original D4RT on point cloud reconstruction. Please note that the final results are still significantly better than the SpatialTrackerV2+VGGT and Pi^3 baselines.

Conclusion: CAFT and TimeDINO are on par with the non-variable length VideoMAE backbone of D4RT in all categories except point cloud reconstruction. Results are still significantly better Pi^2 and SpatialTrackerV2.

Effect of using VideoMAE / DinoV3 on Visual Instance Segmentation - 100k steps, BS=64

In earlier stages of the project, we used the VideoMAE backbone in our experiments and found VIS results to be poor compared to SOTA. We tried to improve this by using distillation using DinoV3 and by feeding the decoder additional DinoV3 features, neither of which managed to improve results significantly. We ultimately found TimeDINO, our modification of DinoV3, to provide the best results. See the results in the following table.

MethodModificationYTVIS19 AP (↑)
VideoMAE-56.2
VideoMAEDistillation loss from frozen DinoV358.6
VideoMAEAdditional Frozen DinoV3 image features fed to the decoder.55.5
TimeDINO-63.8

Conclusion: TimeDINO improves over VideoMAE by 7.6 AP; distillation recovers only part of this gap (+2.4 AP) and feeding DINOv3 features to the decoder does not help.

AuthorsOpenReview ↗

Ablations part 2

Effect of Temporal RoPE - 100k steps, BS=64

We did a 100k-step temporal encoding ablation on TimeDino H+ (256×256, 24 frames, batch size 64) to better understand the effects of Temporal RoPE. Specifically we varied the amount of temporal and spatial RoPE and also experimented with STRING (Schenk et al. 2025) which enables learning the frequencies. As before, we average over datasets used in the original D4RT paper:

MethodSpatial RoPE %Temporal RoPE %3D Tracking Mean 3D AJ (↑)Point Cloud Mean Scaled L1 (↓)Depth Mean S AbsRel (↓)Camera Pose Mean Pose AUC (↑)
No temporal RoPE100%0%0.1970.5150.1060.604
Overlapping RoPE (Submission)100%50%0.2330.5010.1000.641
50/50 split RoPE50%50%0.2330.4790.0970.629
50/50 split learned RoPE frequencies50%50%0.2380.4720.0980.640

As can be seen, temporal RoPE improves results across the board. However, our subsequent ablation demonstrates that a strict separation of temporal and spatial rotation is beneficial. Finally, using STRING (i.e. learned RoPE frequencies) yields a minor, yet consistent improvement.

Conclusion: Temporal RoPE significantly improves results while separating spatial and temporal channels completely yields additional benefits.

Object Queries: EoMT vs Two-way transformer

Our paper uses the Encoder-only Mask Transformer (EoMT). However, another possibility would be to add object queries after the video encoder and before the decoder (i.e. in a ‘neck’). We tried this using a two-way transformer. Our best variant of this was a two-layer two-way transformer but this consistently fell behind EoMT.

250k steps, BS=64, 384 resolution:

MethodYTVIS19YTVIS21
EoMT66.860.8
Two-way transformer65.459.7

Conclusion: EoMT is better than a two-way transformer: +1.4 AP on YTVIS19 and +1.1 AP on YTVIS21.

Single Task vs Joint Training

We compare 4RROW (jointly trained) with the same setup but then trained only using the original D4RT tasks and losses. We contextualize results by showing also results of Pi^3 and SpatialTrackerV2.

MethodYTVis19 APDepth Estimation Mean S AbsRel (↓)Point Cloud Mean Scaled L1 (↓)Camera Pose Mean Pose AUC (↑)3D Tracking Mean 3D AJ (↑)
SpatialTrackerv2 (External Baseline)unsupported0.0880.7020.6670.182
Pi^3 (External Baseline)unsupported0.0870.5850.690unsupported
D4RT using CAFP + TimeDINOunsupported0.0830.4760.7120.280
4RROW (ours)69.00.0900.4530.6540.228

4RROW improves slightly on point cloud reconstruction, but loses on camera pose estimation and 3D tracking. This demonstrates that there is a trade-off between focusing on visual instance segmentation and geometry. Nevertheless, our results are still competitive or better than Pi^3 and SpatialTrackerV2 despite being able to handle segmentation.

Open Vocabulary vs Closed Vocabulary

We used “the open vocabulary setting as defined by DVIS++” (L312). However, this is more of a joint training scenario on multiple datasets as opposed to optimizing for a single dataset and we will rename it accordingly. We do not claim generalization to unseen categories. Instead, we train jointly on multiple datasets for multiple tasks.

Thank you for the detailed response and additional experiments. The new ablations provide useful evidence for TimeDINO, temporal RoPE, EoMT, and point tracking. The single-task comparison also clarifies that joint training introduces a trade-off: point-cloud reconstruction improves slightly, while depth, camera pose, and tracking degrade. I also appreciate the clarification that the current setting does not evaluate genuine unseen-category generalization.

Several concerns nevertheless remain. CAFP is not compared with explicit continuous timestamp embeddings or tested on near-duplicate frames and small moving objects. Long-sequence accuracy, runtime and memory scaling, as well as the sensitivity of DAP to scale alignment and evaluation thresholds, are still unreported. Moreover, the new results are mainly averages across datasets, making dataset-specific failures difficult to assess. The discussion also confirms that the current quantitative evaluation does not directly measure cross-time instance reconstruction.

Overall, the response strengthens the empirical support for several components, but the remaining gaps concern important claims about variable-length processing, evaluation robustness, and cross-time reconstruction. I therefore retain my original rating.

Reviewer_Ef8xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes 4RROW, a unified feed-forward framework for 4D reconstruction, point tracking, and object recognition from an input video. The method is built upon D4RT, incorporating a TimeDINO video encoder and object queries to facilitate instance segmentation and open-vocabulary classification. This paper also redesigns the point queries for variable-frame videos. Experimental results on video instance segmentation, geometric reconstruction and joint geometry-semantic prediction demonstrate the effectiveness of 4RROW.

Critical strengths
  1. Resonable unified framework design. This paper intergrates dynamic geometry, point tracking and instance recognition into a unified framework, enabling reconstruction of individual dynamic objects in video.
  2. Supporting variable-length video. This paper uses frame features to generate querate query conditions, which avoids reliance on D4RT static temporal lookup tables.
  3. Convincing video instance segmentation results. This paper outperforms existing open-vocabulary baselines on three VIS benchmarks on OVIS, while also maintaining competitive geometric reconstruction ablity as shown in the supplemental material.
Critical weaknesses
  1. Missing ablation study. This paper does not conduct ablation on the proposed components, i.e., TimeDINO with temporal RoPE, learnable object queries, and the content-aware frame pooling.
  2. Insuffucient quantitative evaluation. This paper does not report point tracking metrics on benchmarks such as TAPVid-3D, which is evaluated by D4RT.
  3. Performance on individual tasks. In table 1, the proposed method underperforms D4RT on all reported geometry metrics. In table 2, the semnatic performance is lower than Pano3D on 3D mIoU with oracle geometry, and both metrics in the 9 scenes. The authors hypothesize that the degradation results from 2 of the 9 bad results, which hurts the average. But the results are not explictely shown or experimentally validated.
Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Ablations part 1

We thank the reviewer for their thorough and constructive review. We are glad that our main contribution is identified as strength #1: Our work presents the first method to combine reconstruction, tracking, and segmentation in a single framework, which enables multi-view reconstruction of individual dynamic objects.

Ablation studies

As suggested by the reviewers, we have done several ablation studies. To ensure readability and highlight the overarching trends, our new studies report performance averaged across datasets rather than exhaustive per-dataset tables; we will include the complete breakdowns in the final paper and gladly provide any specific numbers during the discussion phase.

Effect of variable-frame support and TimeDINO on geometry - 500k steps, BS=64

In the following experiment we go from the original VideoMAEv2 backbone (results from D4RT paper) to using Content-Aware Frame-Pooling (CAFP) which enables variable-frame inference. Then we switch to the TimeDINO backbone. Finally we increase resolution from 256x256 to 384x384 which is the resolution at which most of the experiments in the paper are done since a higher resolution is essential for segmentation.

MethodDepth Estimation Mean S AbsRel (↓)Point Cloud Reconstruction Mean Scaled L1 (↓)Camera Pose Estimation Mean Pose AUC (↑)3D Tracking Mean 3D AJ (↑)
SpatialTrackerv2 (External Baseline)0.0880.7020.6670.182
Pi^3 (External Baseline)0.0870.5850.690unsupported
VideoMAE G — D4RT paper0.0810.4220.7350.274
VideoMAE G — CAFP0.0850.5250.7240.268
TimeDino H+ — CAFP0.0910.5030.7010.269
TimeDINO H+ — CAFP @3840.0830.4760.7120.280

As can be seen, when using the VideoMAE backbone, moving from fixed learned embeddings to content-aware frame-pooling (CAFP) results in a drop for point cloud reconstruction, while tracking, depth, and camera pose remain mostly on par. Moving then to TimeDINO recovers some of the point cloud reconstruction but results in a slight drop in camera pose estimation. The subsequent increase of resolution is beneficial across the board but the final model remains behind the original D4RT on point cloud reconstruction. Please note that the final results are still significantly better than the SpatialTrackerV2+VGGT and Pi^3 baselines.

Conclusion: CAFT and TimeDINO are on par with the non-variable length VideoMAE backbone of D4RT in all categories except point cloud reconstruction. Results are still significantly better Pi^2 and SpatialTrackerV2.

Effect of using VideoMAE / DinoV3 on Visual Instance Segmentation - 100k steps, BS=64

In earlier stages of the project, we used the VideoMAE backbone in our experiments and found VIS results to be poor compared to SOTA. We tried to improve this by using distillation using DinoV3 and by feeding the decoder additional DinoV3 features, neither of which managed to improve results significantly. We ultimately found TimeDINO, our modification of DinoV3, to provide the best results. See the results in the following table.

MethodModificationYTVIS19 AP (↑)
VideoMAE-56.2
VideoMAEDistillation loss from frozen DinoV358.6
VideoMAEAdditional Frozen DinoV3 image features fed to the decoder.55.5
TimeDINO-63.8

Conclusion: TimeDINO improves over VideoMAE by 7.6 AP; distillation recovers only part of this gap (+2.4 AP) and feeding DINOv3 features to the decoder does not help.

AuthorsOpenReview ↗

Ablations part 2

Effect of Temporal RoPE - 100k steps, BS=64

We did a 100k-step temporal encoding ablation on TimeDino H+ (256×256, 24 frames, batch size 64) to better understand the effects of Temporal RoPE. Specifically we varied the amount of temporal and spatial RoPE and also experimented with STRING (Schenk et al. 2025) which enables learning the frequencies. As before, we average over datasets used in the original D4RT paper:

MethodSpatial RoPE %Temporal RoPE %3D Tracking Mean 3D AJ (↑)Point Cloud Mean Scaled L1 (↓)Depth Mean S AbsRel (↓)Camera Pose Mean Pose AUC (↑)
No temporal RoPE100%0%0.1970.5150.1060.604
Overlapping RoPE (Submission)100%50%0.2330.5010.1000.641
50/50 split RoPE50%50%0.2330.4790.0970.629
50/50 split learned RoPE frequencies50%50%0.2380.4720.0980.640

As can be seen, temporal RoPE improves results across the board. However, our subsequent ablation demonstrates that a strict separation of temporal and spatial rotation is beneficial. Finally, using STRING (i.e. learned RoPE frequencies) yields a minor, yet consistent improvement.

Conclusion: Temporal RoPE significantly improves results while separating spatial and temporal channels completely yields additional benefits.

Object Queries: EoMT vs Two-way transformer

Our paper uses the Encoder-only Mask Transformer (EoMT). However, another possibility would be to add object queries after the video encoder and before the decoder (i.e. in a ‘neck’). We tried this using a two-way transformer. Our best variant of this was a two-layer two-way transformer but this consistently fell behind EoMT.

250k steps, BS=64, 384 resolution:

MethodYTVIS19YTVIS21
EoMT66.860.8
Two-way transformer65.459.7

Conclusion: EoMT is better than a two-way transformer: +1.4 AP on YTVIS19 and +1.1 AP on YTVIS21.

Additional metrics

We compare 4RROW w.r.t Pi^3 and SpatialTrackerV2 also in terms of tracking and camera pose estimation (in addition to what was presented in the paper).

MethodYTVis19 APDepth Estimation Mean S AbsRel (↓)Point Cloud Mean Scaled L1 (↓)Camera Pose Mean Pose AUC (↑)3D Tracking Mean 3D AJ (↑)
SpatialTrackerv2 (External Baseline)unsupported0.0880.7020.6670.182
Pi^3 (External Baseline)unsupported0.0870.5850.690unsupported
4RROW (ours)69.00.0900.4530.6540.228

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.