acceptodds
Under review as a conference paper at ICLR 2027

ARROW: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild

32 views

Abstract

Dynamic scenes may be captured by a moving camera, multiple video streams, or images taken at different times. These observations reveal complementary aspects of scene geometry and motion, yet bringing them together requires establishing correspondence across viewpoints, capture times, and visibility changes. We introduce ARROW, a feed-forward model that unifies 3D reconstruction and 3D point tracking from arbitrary image sets. At its core is a novel order-invariant querying approach, which allows the association of queries with observations across arbitrary inputs. We show that exposing the model to more diverse sets of inputs during training results in improved task performance. Moreover, the resulting model is capable of generalization to a wider range of tasks including multi-view tracking. Trained with this strategy, ARROW establishes a new state of the art in 3D tracking on WorldTrack and TAPVid-3D and outperforms dedicated multi-view trackers on an adapted RGB-only MVTracker benchmark, while remaining competitive across 3D reconstruction tasks. Code and weights will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D point tracking, 4D reconstruction, 3D perception, 4D perception, multi-view tracking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xjfyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ARROW, a feed-forward framework for joint 3D reconstruction and point tracking from arbitrary sets of RGB observations. Its key contribution is an order-invariant querying mechanism that repurposes image-conditioned camera tokens from a pretrained DA3 encoder as identity embeddings, enabling flexible source, target, and reference selection without explicit temporal or camera indexing. By training on diverse static image collections, dynamic videos, and multi-view sequences, ARROW generalizes beyond temporally continuous video. It achieves state-of-the-art results on several 3D tracking benchmarks and outperforms dedicated multi-view trackers under an RGB-only setting, while remaining competitive in 3D reconstruction and camera pose estimation.

Critical strengths
  1. The method is clearly presented and easy to understand.

  2. The motivation is well aligned with the method design, and the distinction between order-aware and order-invariant representations is well motivated.

  3. I appreciate the unified framework for 3D reconstruction and both monocular and multi-view tracking.

  4. The method achieves state-of-the-art performance on several benchmarks.

Critical weaknesses
  1. Lack of video demonstrations. The paper would benefit from video demos that showcase the method's performance in dynamic scenes, particularly its ability to maintain temporally consistent 2D and 3D point tracks.
Questions for authors
  1. Since the method predicts UV correspondences, a more comprehensive evaluation of 2D tracking would strengthen the paper. Both quantitative results and qualitative visualizations of 2D point trajectories would help clarify the effectiveness of the correspondence prediction mechanism.

  2. Track4World and SpatialTrackerV2 also support 2D and 3D tracking. Qualitative comparisons with these methods, particularly visualizations of 2D trajectories and 3D point tracks on the same video sequences, would help better illustrate the advantages and limitations of the proposed method. Such comparisons could provide a clearer understanding of its tracking quality, temporal consistency, and robustness in challenging scenarios.

1 comment

We thank the reviewer for the positive assessment of the motivation, unified framework, and benchmark results. We agree that video demonstrations would provide additional evidence of the reconstruction and tracking quality.

Qualitative demonstrations and comparisons. Our anonymous website provides a video overview, interactive reconstruction and tracking examples, and predicted 2D trajectories illustrations. We also provide the website source as supplementary material (ARROW-supplementary.zip) for local viewing.

2D tracking. Following D4RT, we supervise image-space coordinates as an auxiliary objective, while focusing on consistent 3D reconstruction and tracking. The UV loss coefficient accounts for only 5% of the loss weights (Appendix B.1). Similarly, we do not claim state-of-the-art 2D tracking performance. For completeness, we report below the monocular TAP-Vid-DAVIS results. Since D4RT does not report 2D tracking accuracy, we include OpenD4RT as an available implementation of D4RT-style querying, alongside Point4D, which also utilizes a similar auxiliary 2D supervision in its training.

MethodAJAPDOAEPE (px)
CoTracker363.9176.4190.943.58
CoWTracker63.2878.6289.702.41
Track4World61.2574.2786.103.45
SpatialTrackerV260.9374.2189.303.94
Point4D21.7831.7582.6415.53
OpenD4RT31.3945.0384.498.78
ARROW43.3956.7188.276.43

Under the same evaluation protocol, ARROW outperforms OpenD4RT and Point4D on all four metrics, while trailing dedicated 2D trackers and methods that rely on explicit 2D motion estimation for 3D tracking. These results illustrate the remaining gap in this domain to dedicated 2D trackers, consistent with the local-precision limitation discussed in Appendix E.

The aforementioned website also provides illustration of the uv-regressed predictions on video sequences from the TAP-Vid-DAVIS benchmark.

Comparisons with Track4World and SpatialTrackerV2. Fig. 12 adds qualitative 3D tracking comparisons with Track4World, SpatialTrackerV2, and Open-D4RT on the same three video sequences. These complement the existing comparisons in Fig. 4, which we further extend, as well as the reconstruction and tracking results provided in Appendix G.

We hope that the additionally provided results and qualitative illustration address the raised concerns.

Reviewer_mZcwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets the inductive bias toward motion and appearance continuity in video-based tracking. It proposes an order-invariant querying approach that uses content-derived identity tokens to associate queries with input observations. Combined with a diverse training mixture of videos, static image collections, and multi-view sequences, the approach improves tracking performance while remaining competitive on 3D reconstruction.

Critical strengths
  • The training strategy is well designed to support unordered observations. It combines non-sequential sampling across time and camera views with static and dynamic training data. This makes the proposed query interface useful across a broader range of input configurations.

  • Table 6 provides a detailed comparison between index-based and ID-based querying. The improvements from replacing order-dependent query embeddings with content-derived ID tokens support the value of the proposed order-independent querying approach.

Critical weaknesses
  • V-DPM already uses backbone-derived target-time tokens for decoder conditioning and states that its design does not assume a particular temporal ordering. What is the conceptual novelty of ARROW’s ID-based conditioning beyond extending this idea to independently selectable source, target, and reference observations?

  • The motivation for order-invariant tracking is not fully substantiated. In lines 053–055, the paper argues that training on temporally local video introduces an inductive bias toward motion and appearance continuity, potentially limiting generalization. However, this concern could also be addressed by increasing and varying temporal strides while retaining temporal order. D4RT already employs random temporal strides during training. The paper lacks a controlled analysis distinguishing the benefits of broader temporal sampling from those of order-invariant querying, leaving the necessity of the proposed formulation for tracking unclear.

Questions for authors
  • Broader temporal sampling may reduce reliance on local motion continuity without removing temporal order. How does the proposed approach compare with ordered-video training using larger or more varied temporal strides?

  • Table 6 demonstrates the benefit of the expanded training mixture. How does ARROW compare with existing reconstruction and tracking methods at a comparable training scale, even on a smaller shared dataset? Specifically, MVTracker and TAPIP3D use only Kubric-derived data for tracker training, whereas ARROW uses a much broader training mixture. A comparison at the same training-data scale would help clarify the source of the performance gains. A similar comparison with representative reconstruction methods, even on a smaller shared dataset, would also be informative.

  • The multi-view evaluation uses synchronized streams. Does ARROW also support tracking across asynchronous camera streams, and how does temporal misalignment affect performance?

  • The model predicts image-space coordinates as well as 3D positions. How do its monocular and multi-view 2D tracking results compare with dedicated 2D trackers, such as CoTracker3, CowTracker, and MV-TAP?

  • What are the inference time and peak memory usage for monocular and multi-view tracking, and how do they compare with the baselines under the same input settings?

  • How do inference time and memory usage change as the number of queries increases?

5 comments

We thank the reviewer for the careful review and appreciate the constructive feedback regarding the suggested approach.

Conceptual novelty and motivation. Shared response A addresses novelty concern and positioning to prior work, now also expanded in the revised Section 2 and Appendix A.

Separating querying, temporal encoding, and sampling. We refer to table 6 for a controlled analysis on the benefits of both our order-invariant mechanism and broader sampling strategy including multi-view tracks supervision:

  1. S1→S2 explicitly isolates our order-invariant querying approach, replacing discrete temporal embeddings with content-derived identities, while retaining encoder temporal encodings, sampling strategy and data. We see that the proposed change already improves across both monocular and multi-view evaluations, before broader sampling is introduced.
  2. S2→S3 removes encoder temporal encodings, still keeping sampling fixed. We observe that this does not lead to performance drops. Thus, we retain this design as it allows us to avoid imposing temporal order on arbitrary observation sets and the need to interpolate for longer contexts.
  3. S3→S4 introduces discontinuous and cross-camera observations, keeping both architecture and data fixed. This transition demonstrates the benefits of broadening the input configuration setting.

To test broader temporal sampling while retaining order, we add two ablations: S3.5 compares an intermediate sampling strategy, sampling video frames in strict temporal order with a random interval up to stride 10 (ordered-random-intervals). S1* retains S1's D4RT-style discrete querying and encoder temporal encodings, while training with an increased and varied temporal stride (ordered-random-stride).

StageQueryPESequentialDataEvalMono APDMono EPEMV APDMV EPEMono AUC@30Mono F1MV AUC@30MV F1
S1D4RT✓✓videointerpolated76.760.23059.810.778343.4845.230.9825.30
S1*D4RT✓ord-rnd-stridevideointerpolated77.610.219210.580.752249.6663.463.3137.74
S3ARROW✗✓videofull78.010.220019.600.566843.5557.6225.2031.69
S3.5ARROW✗ord-rnd-intervalvideofull78.300.214431.870.310751.1969.5645.4537.59
S4ARROW✗✗videofull78.490.212748.450.163450.1975.4362.7346.99

The results in both experiments further demonstrate the benefit of a less temporally biased training supervision. However, they still underperform compared to S4, illustrating the benefits of both the query-invariant mechanism and the arbitrary sampling strategy.

As for the necessity of the proposed formulation, order-invariant querying provides three additional practical advantages:

  1. Longer contexts: no interpolation or extrapolation of timestep embeddings beyond the training context length.
  2. Multiple and asynchronous streams: observations remain independently identifiable when timestamps coincide, without requiring a shared temporal index.
  3. Flexible observation sets: video frames, static images, and images from pre-existing scans can share one encoding without imposing artificial temporal order.

Training scale. Both MVTracker and TAPIP3D models operate in different setting than ARROW, already receiving 3D points as inputs. To evaluate in 3D from RGB inputs, they rely on large pretrained 3D foundation models, which require large training resources (Sec. C.1). Additional scaling experiments are unlikely to isolate the sources of performance gains.

Asynchronous streams. ARROW does not require synchronized inputs: training already samples across cameras and times without requiring an observation from every camera at each timestep (Appendix B.3). We additionally evaluate simulated temporal misalignment on PStudio, selecting 24 frames from each of four 150-frame camera streams. Synchronized sampling uses a common frame every six frames; asynchronous sampling independently selects one frame per camera within the corresponding six-frame intervals. The selected observations are fixed across methods.

MethodSync APDAsync APDRelative drop
ARROW77.8177.690.15%
OmniX66.8566.470.57%
4RC57.6657.001.14%

ARROW retains its performance under the tested temporal misalignment.

2D tracking and inference costs. Our response to Reviewer xjfy addresses 2D tracking. Shared response B addresses runtime and memory.

Please let us know if any concerns remain unaddressed. We are happy to discuss them.

Reviewer_mZcwOpenReview ↗

Official Comment by Reviewer mZcw

Thank you for the thorough rebuttal and the additional experiments. Most of my concerns have been resolved, particularly through the clarification of the conceptual differences from existing methods and the detailed ablations separating the effects of query design, temporal encoding, and sampling.

The remaining concerns are the multi-view aspects of my questions about 2D tracking and inference costs. While the monocular results are helpful, could you also provide multi-view 2D tracking results against relevant baselines, such as MV-TAP, along with inference cost?

If these remaining concerns are adequately addressed, I would be inclined to raise my score to Weak Acceptance.

We appreciate your response and are glad that most concerns have been resolved. Below, we provide the requested multi-view 2D tracking comparisons and inference costs.

We evaluate on our existing MVTracker benchmark, with four views and 24 timestamps per clip. For 2D tracking, we follow MV-TAP’s evaluation protocol and pool scores across all four cameras. Cells report AJ / APD / OA (%) ↑, with AJ and APD computed at thresholds {1, 2, 4, 8, 16} in 256×256-normalized coordinates.

MethodPStudioDexYCBKubric
MV-TAP59.03 / 77.90 / 82.6367.99 / 81.58 / 86.2850.82 / 68.89 / 78.81
MVTracker + DA3*— / 42.63 / —— / 31.07 / —— / 39.12 / —
OmniX*— / 47.96 / —— / 33.30 / —— / 14.25 / —
Point4D16.22 / 26.14 / 69.949.77 / 17.43 / 75.3010.26 / 21.89 / 68.02
ARROW48.32 / 65.37 / 84.1736.49 / 52.16 / 84.1628.02 / 44.55 / 81.21

* These methods do not provide camera-specific visibility predictions, so OA and AJ are unavailable; their APD is computed from projected 3D tracks.

Inference time and memory. We profile the 2D evaluation above on a single H100 GPU. Cells report mean seconds per clip / maximum peak allocated GiB ↓.

MethodPStudioDexYCBKubric
MV-TAP3.02 / 9.542.99 / 9.543.06 / 10.16
MVTracker + DA3*2.62 / 10.632.84 / 10.863.69 / 11.54
OmniX*54.44 / 58.5193.37 / 77.2455.37 / 59.29
Point4D6.91 / 16.006.03 / 15.198.50 / 16.65
ARROW2.25 / 10.083.50 / 10.365.85 / 12.35

ARROW outperforms Point4D’s regressed 2D predictions and achieves higher APD than the reprojected tracks from OmniX and MVTracker across all three datasets. MV-TAP as a dedicated 2D tracker retain an advantage in 2D accuracy. As discussed in the initial response regarding monocular 2D tracking, ARROW uses UV prediction as an auxiliary objective while prioritizing consistent 3D tracking. Our evaluation therefore focuses on trajectories in a shared 3D coordinate system, which we believe better captures the underlying cross-camera consistency.

We hope these additional results address your remaining concerns.

Reviewer_zXYsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper (ARROW) extends the D4RT method from monocular sequential video frames to unordered arbitrary input views. D4RT used VideoMAE as the backbone, while ARROW is pretrained from DepthAnything3. Order-invariant querying is described as the core contribution of this method. Trained on 19 public datasets with 32 H100 for 200K iterations, ARROW achieves state-of-the-art 3D point tracking performance on WorldTrack, TAPvid3D, and MVTracker.

Critical strengths
  1. The method is clearly presented. The evaluations are thourough. And the visual results are impressive. The most convincing parts of the paper are the clear advance of performance on public benchmarks, indicating great engineering efforts and excellent execution.
Critical weaknesses
  1. The order-invariant querying is described as the major contribution of this paper. However, such component already exists in baselines such as V-DPM and 4RC, which are not credited in the paper. Therefore, the major contribution should be instead stated as bringing order-invariant querying to D4RT.
  2. D4RT-like methods benefit from the sparse query + RGB crop design so that are very good at modeling details. However, it is computationally more expensive compared to other models like V-DPM and 4RC for dense restruction and tracking. There lacks runtime analysis on this difference, e.g., (a) with how many query pixels ARROW can infer at a comparable FPS to 4RC, (b) how long ARROW takes to densely reconstruct a 100-frame sequence in comparison to 4RC, both given input resolution 518 or 504
Questions for authors
  1. The 200K-iteration model achieves APD 88 on average on WorldTrack's four tracking subsets, while the 100K-iteration model achieves 80. How many hours or days does the 200K-iter training take? Will the performance further scale up with even more iterations? Was the 100K-iter model also trained with 32 H100?
2 comments

We thank the reviewer for recognizing the thorough evaluation, strong benchmark results, and execution quality. We appreciate the constructive feedback which we address below.

Clarification on contribution. Our contribution statement already describes the contribution as an "order-invariant querying approach, aligning the embedding space of the queries and the global context" (L124–125). This statement refers to our proposed querying mechanism (Figure 2c) building on D4RT's query formulation and unique to ARROW. Shared response A discusses the differences to V-DPM and 4RC in detail. The revised Section 2 and Appendix A now explicitly credit their formulation for unordered inputs. Moreover, the contribution goes beyond conceptual support. We deliberately utilize the unordered-input formulation to extend the training mixture to include multi-view streams and broader sampling. Table 6 separates the effects of the query mechanism, broader sampling, and broader training data.

How many queries give comparable FPS to 4RC? On representative 64-frame WorldTrack clips, 4RC operates at approximately 36 FPS. Under the same conditions, ARROW achieves comparable throughput for approximately 102,400 queries, corresponding to 1,600 source pixels tracked across all 64 frames. This comparison is now included in Appendix D.3. Shared response B provides the requested runtime and memory analysis, including dense tracking over 100 frames.

Main run versus ablation runs. The ablation models train for 100k steps on 16 H100 GPUs, using a DA3-initialized ViT-L encoder and six decoder layers. The full model trains for 200k steps on 32 H100 GPUs, using ViT-G and eight decoder layers, and takes approximately 3.5 days. Thus, the reported performance difference reflects changes in both model capacity and training budget. We have added these additional details to Section 4 in the revised pdf.

Additional training iterations. Further gains are plausible given the variety of query supervision, input types, and camera–time relationships, but we have not measured scaling beyond the reported 200k-step budget. We therefore cannot yet quantify the benefit of larger training scale.

Please let us know if further clarification would be helpful.

I thank the authors for their reply. I will maintain my original positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Shared Questions and Concerns

We thank the reviewers for their constructive feedback, and for recognizing the clarity, thorough evaluation, benchmark improvements, and execution quality. We also appreciate their positive assessments of the training strategy, ablations, and the unified framework. Below, we address the reviewers’ shared concerns and provide additional analyses, with further details in the reviewer-specific responses.

A. Comparison to existing approaches Section 2 and Appendix A now provide an extended discussion on ARROW conceptual differences from V-DPM and 4RC, as summarized below:

  1. Point querying versus dense conditioning. V-DPM and 4RC condition dense outputs, without independent point queries with selectable source, target, and reference observations. ARROW extends D4RT's unified point-query interface to unordered sets through content-derived identities for all three roles.

  2. Retrieval from one shared state. Both reuse backbone encodings but refine features during target-conditioned decoding. V-DPM refines the full context per target: tracking even one pixel across T targets requires T global refinements. 4RC restricts refinement to source–target features via target-conditioned AdaLN. ARROW retrieves point states through cross-attention to one fixed global context shared by all queries.

  3. Supervision beyond monocular videos. Unordered-input support differs from exploiting diverse camera–time relationships during training. V-DPM states, “During training, we sample video snippets” [1, Sec 3.3], while 4RC specifies, “Video frames are sampled in strict temporal order with a random interval ranging from 1 to 5 frames.” [2, App. A.3]. Their training centers on monocular video. ARROW jointly trains and evaluates reconstruction and tracking across static image collections, monocular videos, and multi-view streams. Our response to Reviewer mZcw discusses additional ablations with broader temporal strides alone.

B. Runtime and memory analysis We report runtime and peak memory for sparse and dense tracking workloads. Appendix D.3 was added to the revised pdf and discusses query chunking and workload comparison.

Query-chunk memory–time tradeoff. We track all 9,031,680 queries (141,120 source pixels) of a 280 × 504 source image across a 64-frame sequence, matching the typical WorldTrack setting, varying only chunk size. Each query requests one point at one target observation; Encoding is performed once and included in the total inference time.

Queries/chunkEncode (s)Single chunk decode (s)Total decoding (s)Total (s)Peak (GiB)
1k1.04—257.29258.337.96
25k1.030.1867.6268.659.02
150k1.051.0462.8463.8920.64
250k1.031.7462.7263.7529.95
750k1.055.2863.6464.6976.46
1M1.03OOMOOMOOMOOM

Throughput largely saturates beyond 150k queries per chunk, while memory continues to grow; 1M runs out of memory. We use 150k queries per chunk: 73% less peak memory than 750k at comparable runtime.

Comparison with baselines. All methods run on a single H100 at 280 × 504 resolution; times include encoding and decoding. Sparse WorldTrack tracking averages approximately 650 source points across 64 frames (41.6k queries). Dense tracking follows all pixels of one source image across 64 or 100 frames (9.03M or 14.11M queries), using 150k queries per chunk for the query methods.

Method64-frame sparse time (s)Peak (GiB)64-frame dense time (s)Peak (GiB)100-frame dense time (s)Peak (GiB)
4RC1.8214.051.8214.053.2718.74
V-DPM97.4919.9397.4919.93291.1527.65
OmniX18.7130.8118.7140.8159.8577.67
Point4D1.5718.7270.5820.74171.0524.75
ARROW1.3416.8665.2720.64155.5221.61

ARROW, a query based approach completes the sparse workload in the shortest time. For dense prediction it remains faster than V-DPM, uses less peak memory than OmniX, and has comparable runtime to Point4D. 4RC, a dense model is indeed substantially faster when full dense output is required. In general, query-based interfaces can provide a fast initial sparse reconstruction and progressively densify without repeating the encoding. Chunking controls peak decoding memory, while all requested point states continue to access the fixed global representation. That being said, decoding millions of point states is costly, as acknowledged in Appendix E.

C. 2D tracking Our response to Reviewer xjfy provides the full TAP-Vid-DAVIS results and discusses auxiliary UV supervision.


[1] Sucar et.al "V-DPM: 4D Video Reconstruction with Dynamic Point Maps", CVPR'26

[2] Luo et. al. "4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere", ICML'26

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.