- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ReLift splits monocular dense 3D tracking into two parts: (i) a "lift" that combines 2D tracks from an off-the-shelf tracker (AllTracker or CoWTracker) with range and cameras from a frozen geometry model (Depth Anything 3, VGGT-Ω, or 4RC's own geometry), and (ii) a feed-forward corrector (19M parameters, alternating temporal and across-track attention over ray–point tokens) that reads only the lifted geometry and predicts 3D residuals. The corrector is trained exclusively on clean PointOdyssey trajectories with simulated lifting errors (range flicker, drift and outliers, small ray noise, synthetic occluders and dropouts), so a single model serves all six tracker × geometry pairs. On WorldTrack, CoWTracker + 4RC geometry gives the best mean APD3D (89.0 / 87.0 All / Dyn vs 86.9 / 81.6 for 4RC), and the VGGT-Ω variants are competitive with 4RC and V-DPM at 2.7–4.4× lower pipeline runtime. Ablations cover corruption families, chunk and group size, ray masking, and simulated versus provider-generated training lifts.
Critical strengths
- The decomposition is clean, and the paper documents a useful fact: a naive lift of a modern 2D tracker with off-the-shelf geometry already matches or exceeds learned 3D trackers given the same geometry (Tab. 2 "Lift" row; matched-geometry comparisons with TrackCraft3R and DELTAv2 in Tab. 1). This is consequential because it tells the community where the remaining difficulty in monocular 3D tracking actually lies.
- One image-free corrector trained purely on simulated corruptions transfers across six provider combinations without retraining, and its design choices are ablated (ray masking, Tab. 9; group size, Fig. 6; corruption families, Tab. 2). Full-pipeline runtime and memory are reported honestly (Tabs. 10–11).
Critical weaknesses
- Positioning and novelty. The problem the paper addresses — lift pixels to 3D with monocular depth and refine the trajectories in 3D with a transformer — is the SpatialTracker line of work. SpatialTracker (Xiao et al., CVPR 2024) and TAPIP3D are not cited; SpatialTrackerV2 is mentioned in one sentence and not benchmarked, although it is a direct world-frame competitor. The elements that would distinguish ReLift (an image-free corrector; training on simulated corruptions rather than real lifts) are never argued or tested against that line. As written, the contribution reads as a re-derivation of an existing recipe with the image features removed.
- The learned repair is compared only with its own crude initialization. The baseline for every corrector gain is the lift with nearest-pixel depth lookup, linear gap interpolation and a held terminal position (Sec. 3.2, App. A.1). No classical or learned repair baseline is reported: temporal smoothing or spline fitting, robust along-ray outlier rejection, rigidity/ARAP regularization across tracks (as in SpatialTracker), or TracksTo4D-style fitting on the same lifts. The corrector adds +1.6–2.6 APD (All) and +2.4–3.6 (Dyn) over the lift (Tab. 2), with the largest gains on occluded frames where the baseline is a held position; most of the state-of-the-art margin comes from the providers (the best row requires 4RC's own geometry and is slower than 4RC; on Kubric, Tab. 5, the DA3/VGGT-Ω variants trail 4RC and V-DPM substantially). The necessity of a learned corrector is therefore not established.
Questions for authors
Clarifications that could change my assessment:
- Please report non-learned repair baselines on the same lifts (temporal smoothing/spline, robust along-ray outlier rejection, cross-track rigidity) and, if possible, SpatialTracker/SpatialTrackerV2 under the WorldTrack protocol. How much of the +1.6–3.6 APD survives?
- Training treats the ray as trusted (angular noise ≤ 0.005 rad vs range corruptions up to ±50%), yet the residual is a free 3D vector (l. 256) and APD2D ends up 1.5 / 3.8 points below the native 2D tracker. Does an along-ray parameterization (Δr) or a reprojection term for available frames recover APD2D without losing APD3D? Relatedly, how are gross 2D failures (drift, identity swaps reported with high visibility) handled, given that neither persistent wrong-surface tracking nor correspondence switches are simulated (App. A.3)?
- Please break down the occluded-point gains into terminal gaps (track invisible until the end, initialized as a held position) versus interior gaps, and report whether the gain depends on co-moving tracks remaining visible in the group. An ablation of the dynamic-track fraction/weighting in training would also show whether the corrector learns a motion prior or mainly multi-view denoising.
Suggestion (not affecting the score): the introduction motivates the repaired trajectories by downstream tasks but demonstrates none; initializing a per-video dynamic optimizer (e.g., MoSca, Shape of Motion) with repaired versus naive-lift tracks would make the practical value concrete.
Follow-up on Our Response
Dear Reviewer HfDe,
Thank you for your constructive feedback and for helping us improve our work. With the discussion period nearing its end, we hope that our additional experiments and clarifications have helped address your concerns.
Please let us know if you have any remaining questions. We would be happy to clarify further.
Best,
The Authors