- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper addresses object tracking under large viewpoint changes and long-term disappearance and reappearance, and proposes an online object tracking model named 3AM. The core idea is to incorporate frozen MUSt3R features into SAM2.1-L. Features from different sources are fused into SAM2 through channel-wise modulation. Routing tokens read the current frame and historical memories through SAM2’s Memory Attention and generate fusion parameters for the next frame, allowing the model to dynamically adjust its reliance on appearance and geometric information.
3AM achieves clear improvements over previous methods on the ScanNet++ Selected Subset constructed by the authors. It also maintains good performance on conventional dynamic tracking benchmarks and transfers zero-shot to Ego-Exo4D.
Critical strengths
-
The paper studies an important failure mode of existing memory-based trackers: after an object disappears for a long period and reappears under a substantially different viewpoint, the tracker may fail to recover the identity of the same instance. This is a valuable problem and remains challenging in spatial reasoning.
-
The paper provides relatively comprehensive baseline comparisons, ablation studies, efficiency measurements, and feature analyses. Overall, the paper is clearly written and the method is well explained.
Critical weaknesses
1. This is my primary concern: the paper emphasizes online deployment, but its main evaluation protocol does not fully match this setting.
On ScanNet++ and Replica, the authors first inspect all ground-truth masks of the target across the entire sequence, select the frame in which the target has the largest visible mask as the prompt frame, and then track both forward and backward from that frame. Such a maximum-area prompt frame usually provides a larger, more complete, and less occluded target, a higher-quality initial memory, and a more stable MUSt3R geometric anchor.
The current evaluation therefore measures inference conditioned on a selected high-quality mask. In real online deployment, however, the system operates in real time and cannot inspect the entire video in advance, select the most suitable frame, and then perform both forward and backward propagation. The current experiments mainly demonstrate that the model has strong cross-view propagation ability when initialized from the best keyframe selected using full-sequence ground truth. They do not establish whether the large improvements would persist under realistic online initialization.
I suggest that the authors at least provide results using the first visible frame as the prompt. Results averaged over randomly sampled visible prompt frames, including the mean and variance, or results grouped by prompt-mask size or target visibility, would also be helpful.
2. Minor concern: the core behavior of the routing mechanism lacks direct validation.
The paper claims that this “switch” changes frame by frame within a video according to the object’s state. However, the experiments only report average routing coefficients on two different datasets: 50.7% geometric contribution on ScanNet++ and 17.8% on DAVIS17.
These results could also be explained by a simpler behavior:
- The model recognizes a ScanNet++-style static indoor video and relies more heavily on geometry throughout the sequence.
- The model recognizes a DAVIS-style dynamic video and relies less heavily on geometry throughout the sequence. This would not necessarily demonstrate frame-by-frame switching.
The claim could be directly validated with a simple experiment: 1. Select a video containing a “static → moving → static again” transition. 2. Mark the ground-truth motion speed of the object over time. 3. Examine whether the geometric gate exhibits the expected “high → low → high” pattern.
Questions for authors
-
Could the authors address my primary concern and provide the corresponding additional experimental results?
-
Could the authors provide direct validation of the routing mechanism described in Concern 2?
I would be inclined to accept the paper if my concerns are adequately addressed.
I thank the authors for the detailed rebuttal and additional experiments.
Naming and task positioning
I appreciate the clarification of the term "3egment Anything." While the naming remains somewhat ambiguous, I consider this a minor presentation issue.
Geometric consistency and backbone dependence
The additional comparisons with DINOv3, VGGT, and LingBot-Map (Table 24) provide useful evidence for the contribution of geometry-aware features and the generality of the proposed framework. The explanation of CUT3R's performance drop is also helpful. These results largely address my concerns.
Methodological clarity
The revised Section 3.2 and the added pseudocode (Algorithm 1) clarify the routing mechanism and its interaction with memory attention. This addresses my concern.
Overall
The rebuttal addresses most of my concerns, and the additional experiments strengthen the paper. I am therefore raising my score to 3.