acceptodds
Withdrawn from ICLR 2027

Flow4RT: Seeing Motion through Geometry

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing and tracking dynamic scenes requires choosing a reference against which motion is measured: the background and independently moving objects support different rigid alignments across views. Directly regressing global geometry or camera poses makes this choice implicit and difficult to change after prediction. We present Flow4RT, a feedforward model that extends Flow4R's local geometry and relative motion formulation from image pairs to joint multi-view inference. The model aggregates information across all input views and predicts motion in both directions between each view and an anchor, supporting reconstruction and tracking within one formulation. Our key insight is that motion is displaced geometry. We predict local geometry for each view, then condition its features on a target view to predict where the same points move in viewpoint and time. A single dense head decodes both local and displaced point maps with their respective confidence and rigidity weights, avoiding duplicated geometry predictions and separate output heads. A closed-form, rigidity-weighted alignment recovers the transformations needed for reconstruction and tracking without a pose regression head. Adjusting the alignment weights selects a different rigid reference object while reusing the same dense predictions. On WorldTrack, Flow4RT achieves the best average tracking performance, improving APD by 2.4 points and reducing EPE by 17% over the strongest prior result for each metric, while ranking second in reconstruction on both metrics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D, 4D, Feedforward, Reconstrution, Tracking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M8TaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Flow4RT presents a feedforward multi-view framework for unified 4D reconstruction and world-space tracking. It extends Flow4R from pairwise inference to joint multi-view reasoning, predicting one local point map per view and target-conditioned displaced point maps for selected view pairs. A rigidity-weighted closed-form alignment recovers camera motion and enables the same dense predictions to support both reconstruction and tracking under different rigid reference regions. On WorldTrack, Flow4RT achieves the best average tracking performance and ranks second in reconstruction.

Critical strengths

This paper supports multi-frame joint inference and achieves superior tracking performance compared to traditional pairwise prediction settings.

Critical weaknesses

Many technical modules rely heavily on existing technologies such as Flow4R. The main contribution of the paper appears to be the extension of pairwise prediction to multi-frame scenes; however, it neither explicitly identifies the specific challenges posed by long sequences nor addresses them. Consequently, the advantages of long-sequence modeling over simply increasing the number of views remain insufficient. The only exception is the advantage observed in low-overlap scenes, but this has not yet been fully validated.

Questions for authors
  1. I believe that joint multi-frame inference has the potential to improve low-overlap correspondences and cross-frame consistency. However, could the authors analyze performance under different source-to-target frame overlap ratios and compare setups with and without intermediate frames to quantify the benefits of joint inference? This is because the current ablation experiments do not directly validate this motivation.
  2. I am more concerned with how to verify the geometric consistency between reconstruction and tracking. Given that the research motivation is to mitigate inconsistencies caused by independent pairwise predictions, could the authors provide quantitative evidence demonstrating that joint inference improves the consistency of bidirectional transformations or cross-frame predictions? If the authors can provide convincing evidence, I am willing to raise my score.
1 comment

I thank the authors for their detailed response, which addresses most of my concerns. However, the technical contribution beyond Flow4R remains insufficiently clarified. The comparisons between two-view and multi-view inference demonstrate the benefit of additional visual context, but do not by themselves explain the contribution of the specific methodological changes beyond Flow4R. Could the authors clarify which technical changes account for the reported improvements, and how their effects relate to the benefit of additional input views? In particular, Flow4RT already outperforms Flow4R when both methods receive only two views, but the source of this improvement remains unclear. At present, I find it difficult to identify the distinct technical contribution and would appreciate a more detailed explanation from the authors. Clarifying this point would help me better understand the paper’s contribution relative to prior work and assess its suitability for publication.

Reviewer_ifjAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Flow4RT, a feed-forward framework for joint multi-view 3D reconstruction and tracking in dynamic scenes. Building on Flow4R, it extends pairwise motion estimation to multi-view inference, using shared dense prediction and rigidity-weighted alignment to recover camera transformations without an explicit pose regression head. The method achieves state-of-the-art tracking performance and ranks second in reconstruction on WorldTrack. However, its conceptual novelty beyond Flow4R and the consistency between reconstruction and tracking remain insufficiently explored.

Critical strengths
  1. The paper is clearly written and easy to follow.

  2. The analysis in Figure 1 is interesting and insightful, effectively illustrating the flexibility of the proposed reference selection mechanism.

  3. The method achieves strong performance, ranking first in average 3D tracking and second in 3D reconstruction on WorldTrack.

Critical weaknesses
  1. The novelty appears relatively limited. The main contributions are primarily technical and architectural extensions of Flow4R, rather than fundamentally new conceptual insights. The paper would benefit from a clearer articulation of the key conceptual advances beyond the existing framework.
Questions for authors
  1. Beyond architectural modifications, it would be helpful to clarify what fundamentally new capability or insight Flow4RT introduces, and why its design offers advantages over existing methods.
  2. The paper uses the same framework for reconstruction and tracking, but it is unclear how consistent the two prediction paths are. For the same scene points at the same timestamp, do the reconstructed point clouds and tracked points align well after transforming them into the same anchor coordinate system? It would be helpful to quantify this cross-consistency, ideally separately for static and dynamic regions, to better validate the geometric consistency of the proposed method.
Reviewer_gUDTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Flow4RT, a feedforward multi-view model for dynamic scene reconstruction and world-space 3D tracking that extends Flow4R from image pairs to joint multi-view inference. Instead of regressing world-frame point maps or camera poses directly, the model predicts reference-agnostic quantities: a local point map P per view and, for each requested source-target pair, a displaced point map P_vt locating the same pixels at the target's viewpoint and time. The choice of which scene region is treated as stationary is isolated in a predicted rigidity map W, and the relative transformation is recovered by a closed-form, W-weighted Procrustes alignment. W is trained without rigidity labels by requiring the resulting transformation to agree with the ground-truth camera pose. A single DPT head decodes both (P, C) and (P_vt, W), and local geometry is computed once per view and reused across pairs. On WorldTrack, Flow4RT reports the best average tracking (83.7 APD, 0.164 EPE) and second-best average reconstruction (90.4 APD, 0.131 EPE), with additional video depth, camera pose, and runtime evaluations.

Critical strengths

1. Explicit, adjustable motion reference. Methods such as VGGT and MonST3R regress world-frame geometry or poses directly, so the reference against which motion is measured is fixed implicitly during training and cannot be changed after prediction. Flow4RT separates this choice from dense prediction: the network predicts quantities that do not depend on a reference, and the reference enters only through W in a closed-form alignment. Weighting the background yields camera pose, while weighting a moving rigid object yields an object-relative alignment without rerunning the network. This is a conceptually clean formulation, and it is potentially useful for downstream tasks that require different references (e.g., fusing observations of a moving object into a single coherent reconstruction).

2. Label-free rigidity supervision. W is supervised only through whether the recovered transformation matches the ground-truth camera motion (Eq. 9), so no motion-mask annotations are required. Unlike a binary dynamic mask, this allows the model to down-weight points that are stationary but geometrically unreliable, which is a more faithful signal for pose estimation.

Critical weaknesses

1. The central claim (selectable reference) is not empirically validated. The abstract and introduction motivate the method primarily through reference selectability, yet all quantitative evaluations in Tables 1 and 2 are background-referenced, so this capability is never exercised. The only evidence is the qualitative example in Fig. 1, and the paper does not describe how W is redirected to a moving object at inference. There is also no ablation isolating the contribution of W or the closed-form alignment (e.g., learned W vs. uniform W, learned W vs. ground-truth or segmentation masks, Procrustes vs. a pose regression head); Table 3 covers only head sharing and initialization. Moreover, dynamic-scene pose is where W should matter most, yet on TUM-Dynamics (Table 8) Flow4RT does not lead on ATE (0.0136 vs. 0.0107 for 4RC) or RPE_r, while it leads on static ScanNet.

2. The source of the reported gains is unclear. The backbone is initialized from 4RC, so it is difficult to attribute the large improvement over Flow4R (73.1 to 83.7 tracking APD) to the proposed formulation rather than to the stronger backbone and multi-view context. Against 4RC itself, reconstruction is lower (90.4 vs. 91.9 APD, 0.131 vs. 0.112 EPE). The tracking lead is also uneven across datasets: Flow4RT leads clearly only on DR, is behind OmniX on ADT, ties V-DPM on PO, and is marginally behind SM4RT on PS. Finally, the claim in Fig. 1 that Flow4RT leads every prior method relies on an average of the tracking and reconstruction scores defined by the paper, under which a method that is not first on reconstruction becomes first overall.

Questions for authors

Clarifications:

  1. How is W redirected from the background to a moving object at inference (e.g., a manual mask, an external segmentation model, or thresholding the learned W)? Is the learned W ever used directly for non-background references?
  2. What accounts for the improvement over Flow4R, and why is reconstruction below 4RC despite initializing from it? An experiment training Flow4RT's motion branch on a backbone comparable to Flow4R's would help separate these factors.
  3. What is the rationale for averaging tracking and reconstruction into a single metric in Fig. 1, rather than reporting each task separately?

Requested experiments that could change my assessment:

  1. Ablations isolating the reference mechanism: learned W vs. uniform W, learned W vs. ground-truth or segmentation masks, and closed-form Procrustes vs. a pose regression head.
  2. Any quantitative evaluation of object-relative alignment or object-centric reconstruction, which would directly support the main claim.

Suggestion for future work:

  1. Demonstrating a downstream task that benefits from switching references (e.g., object reconstruction or manipulation) would substantially strengthen the motivation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.