acceptodds
Under review as a conference paper at ICLR 2027

ReLift: Dense 3D Tracking as Trajectory Repair

Abstract

Pretrained 2D trackers and geometry models already provide strong estimates for monocular 3D tracking where their observations are reliable. Lifting 2D tracks with estimated depth and cameras yields trajectories that are accurate in most places but jitter along the viewing ray, jump to other surfaces near occlusions, and go missing while points are hidden. We recast dense 3D tracking as repairing this lift. A feed-forward corrector reads only the geometry of the lifted trajectories and predicts their corrections. Since it never sees an image, we train it only on clean PointOdyssey trajectories with simulated lifting errors, and one trained model serves every tracker and geometry source we evaluate. On WorldTrack, it achieves state-of-the-art mean accuracy, and with VGGT- geometry it is competitive with the strongest 3D trackers at a fraction of their runtime.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer vision, 3D tracking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5NWXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReLift splits monocular dense 3D tracking into two parts: (i) a "lift" that combines 2D tracks from an off-the-shelf tracker (AllTracker or CoWTracker) with range and cameras from a frozen geometry model (Depth Anything 3, VGGT-Ω, or 4RC's own geometry), and (ii) a feed-forward corrector (19M parameters, alternating temporal and across-track attention over ray–point tokens) that reads only the lifted geometry and predicts 3D residuals. The corrector is trained exclusively on clean PointOdyssey trajectories with simulated lifting errors (range flicker, drift and outliers, small ray noise, synthetic occluders and dropouts), so a single model serves all six tracker × geometry pairs. On WorldTrack, CoWTracker + 4RC geometry gives the best mean APD3D (89.0 / 87.0 All / Dyn vs 86.9 / 81.6 for 4RC), and the VGGT-Ω variants are competitive with 4RC and V-DPM at 2.7–4.4× lower pipeline runtime. Ablations cover corruption families, chunk and group size, ray masking, and simulated versus provider-generated training lifts.

Critical strengths
  1. The decomposition is clean, and the paper documents a useful fact: a naive lift of a modern 2D tracker with off-the-shelf geometry already matches or exceeds learned 3D trackers given the same geometry (Tab. 2 "Lift" row; matched-geometry comparisons with TrackCraft3R and DELTAv2 in Tab. 1). This is consequential because it tells the community where the remaining difficulty in monocular 3D tracking actually lies.
  2. One image-free corrector trained purely on simulated corruptions transfers across six provider combinations without retraining, and its design choices are ablated (ray masking, Tab. 9; group size, Fig. 6; corruption families, Tab. 2). Full-pipeline runtime and memory are reported honestly (Tabs. 10–11).
Critical weaknesses
  1. Positioning and novelty. The problem the paper addresses — lift pixels to 3D with monocular depth and refine the trajectories in 3D with a transformer — is the SpatialTracker line of work. SpatialTracker (Xiao et al., CVPR 2024) and TAPIP3D are not cited; SpatialTrackerV2 is mentioned in one sentence and not benchmarked, although it is a direct world-frame competitor. The elements that would distinguish ReLift (an image-free corrector; training on simulated corruptions rather than real lifts) are never argued or tested against that line. As written, the contribution reads as a re-derivation of an existing recipe with the image features removed.
  2. The learned repair is compared only with its own crude initialization. The baseline for every corrector gain is the lift with nearest-pixel depth lookup, linear gap interpolation and a held terminal position (Sec. 3.2, App. A.1). No classical or learned repair baseline is reported: temporal smoothing or spline fitting, robust along-ray outlier rejection, rigidity/ARAP regularization across tracks (as in SpatialTracker), or TracksTo4D-style fitting on the same lifts. The corrector adds +1.6–2.6 APD (All) and +2.4–3.6 (Dyn) over the lift (Tab. 2), with the largest gains on occluded frames where the baseline is a held position; most of the state-of-the-art margin comes from the providers (the best row requires 4RC's own geometry and is slower than 4RC; on Kubric, Tab. 5, the DA3/VGGT-Ω variants trail 4RC and V-DPM substantially). The necessity of a learned corrector is therefore not established.
Questions for authors

Clarifications that could change my assessment:

  1. Please report non-learned repair baselines on the same lifts (temporal smoothing/spline, robust along-ray outlier rejection, cross-track rigidity) and, if possible, SpatialTracker/SpatialTrackerV2 under the WorldTrack protocol. How much of the +1.6–3.6 APD survives?
  2. Training treats the ray as trusted (angular noise ≤ 0.005 rad vs range corruptions up to ±50%), yet the residual is a free 3D vector (l. 256) and APD2D ends up 1.5 / 3.8 points below the native 2D tracker. Does an along-ray parameterization (Δr) or a reprojection term for available frames recover APD2D without losing APD3D? Relatedly, how are gross 2D failures (drift, identity swaps reported with high visibility) handled, given that neither persistent wrong-surface tracking nor correspondence switches are simulated (App. A.3)?
  3. Please break down the occluded-point gains into terminal gaps (track invisible until the end, initialized as a held position) versus interior gaps, and report whether the gain depends on co-moving tracks remaining visible in the group. An ablation of the dynamic-track fraction/weighting in training would also show whether the corrector learns a motion prior or mainly multi-view denoising.

Suggestion (not affecting the score): the introduction motivates the repaired trajectories by downstream tasks but demonstrates none; initializing a per-video dynamic optimizer (e.g., MoSca, Shape of Motion) with repaired versus naive-lift tracks would make the practical value concrete.

Reviewer_1iPGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReLift, a feed-forward framework for dense monocular 3D tracking that combines pretrained 2D trackers and geometry models with learned trajectory correction. Instead of learning tracking and geometry jointly, it lifts 2D tracks into 3D and uses a corrector trained on synthetically corrupted PointOdyssey trajectories to repair depth errors and complete missing observations. A single corrector generalizes across different trackers and geometry providers without retraining. Experiments on WorldTrack demonstrate state-of-the-art mean tracking accuracy, while achieving competitive performance with substantially lower runtime than leading methods.

Critical strengths
  1. The motivation is well-founded, and the proposed solution is reasonable and intuitive.

  2. The paper is clearly written and easy to follow, with a well-organized presentation of the method and experiments.

  3. The paradigm of combining pretrained 2D tracking with depth-based 3D lifting and trajectory refinement is appealing. Unlike direct-regression approaches such as 4RC, which may exhibit inconsistencies between reconstructed point clouds and tracked points even after transforming them into a shared anchor coordinate system, this approach explicitly leverages 2D correspondences and refines the lifted 3D trajectories to improve geometric consistency, particularly when tracking failures occur.

  4. The video demonstrations are convincing, and the experimental results are strong, showing consistent improvements across different combinations of 2D trackers and geometry providers.

Critical weaknesses
  1. Dependence on pretrained models. The method relies on pretrained 2D trackers and geometry estimators, which may limit its scalability and generalization to larger datasets and more diverse scenarios. This is my main concern, as the approach's potential for further scaling remains constrained by the capabilities of these external models.
Questions for authors
  1. Comparison with Track4World. Since Track4World also supports both 2D and 3D tracking, it would be valuable to include a direct comparison to better clarify the advantages of the proposed decomposition. Furthermore, could the proposed trajectory corrector be applied to refine the 3D trajectories predicted by Track4World, particularly its depth-based branch? Such an experiment would help assess whether ReLift's correction mechanism can complement existing joint 2D/3D tracking methods and improve their 3D tracking accuracy.
Reviewer_HfDeOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ReLift, a dense 3D tracking framework that repairs lifted trajectories by targeting three failure modes: jitter along viewing rays, jumps to incorrect surfaces, and missing observations during occlusion. Motivated by the strong performance of dense 2D tracks lifted using estimated depth and cameras, which can already match or outperform existing 3D tracking methods, ReLift refines these trajectories with an attention-based corrector. The corrector s trained on clean PointOdyssey trajectories corrupted with simulated lifting errors.

Critical strengths
  • The approach is well motivated by three concrete failure modes in lifting 2D tracks to 3D. The strong performance of dense 2D trackers combined with estimated geometry supports formulating 3D tracking as trajectory repair. This provides a convincing rationale for building on existing models with a dedicated corrector.

  • A single corrector trained with simulated lifting errors improves all six evaluated tracker–geometry combinations. This demonstrates transfer across different sources of geometric errors and supports the practical value of the approach.

Critical weaknesses

The evaluation does not clearly establish how well ReLift addresses each of the three motivating failure modes. Table 2 examines training corruptions, and Figure 3 shows gains on ground-truth occluded points, but neither separates test cases with ray-direction jitter, wrong-surface jumps, and missing observations. Evaluating correction gains separately for these cases would clarify which failures the method reliably repairs. This limits the evidence connecting the reported improvements to the specific problems motivating the approach.

Questions for authors
  • How much does correction improve test cases with ray-direction jitter, wrong-surface jumps, and missing observations separately? These cases could be identified using temporally varying depth estimation errors, object segmentation (e.g., SAM 3 [1]) to detect jumps to other objects, and the tracker’s observation-availability mask, respectively.

  • Table 12 shows lower 2D accuracy than the original 2D trackers. Could this partly result from correcting XYZ coordinates while keeping the estimated cameras fixed? Could an optimization approach such as bundle adjustment address this by jointly refining cameras and trajectories while preserving the 3D gains?

  • The corrector appears to serve a similar role to trajectory refinement modules in existing trackers (e.g., CoTracker3 [2] and TAPIP3D [3]). Can it also improve the outputs of dense 3D trackers such as DELTAv2 and TrackCraft3R? Would the same trained corrector transfer directly, or would retraining be necessary?

[1] Carion et al., SAM 3: Segment Anything with Concepts, ICLR2026

[2] Karaev et al., CoTracker3: Simpler and Better Point Tracking by Pseudo-Labeling Real Videos, ICCV 2025

[3] Zhang et al., TAPIP3D: Tracking Any Point in Persistent 3D Geometry, NeurIPS 2025

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer HfDe,

Thank you for your constructive feedback and for helping us improve our work. With the discussion period nearing its end, we hope that our additional experiments and clarifications have helped address your concerns.

Please let us know if you have any remaining questions. We would be happy to clarify further.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.