acceptodds
Under review as a conference paper at ICLR 2027

3AM: 3egment Anything with Adaptive Geometric Consistency in Videos

Abstract

Consistent video object tracking must preserve identity under large viewpoint changes, temporary absence, and nearby distractors. Memory-based trackers such as SAM2 rely on appearance matching and fail in these cases, whereas 3D-based methods preserve identity across views but need camera poses, depth, or a reconstructed point cloud, run offline, and are not promptable. We present 3AM, a promptable online tracker that achieves geometric consistency from RGB alone by coupling memory-based tracking with the online multi-view reconstruction model MUSt3R. We identify the MUSt3R layers whose features preserve instance identity across views and fuse them into the tracker's image features by feature-wise modulation whose parameters are decoded from routing tokens. Geometry alone is insufficient, since a location cue holds only while the object stays in place; the routing tokens therefore traverse the tracker's memory attention and decide per frame how much the fusion should rely on geometry, so that one model handles static and moving objects without a hand-designed fallback. On wide-baseline benchmarks built from ScanNet++ and Replica, 3AM reaches 90.6% IoU and 65.7% tracking recall on our selected subset of ScanNet++ (+12.3 and +28.2 points over the best prior tracker), matches or exceeds SAM2 on standard VOS and VOT benchmarks, and transfers zero-shot to ego-exo correspondence on Ego-Exo4D.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D, Video Segmentation, Correspondence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sXoGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper addresses object tracking under large viewpoint changes and long-term disappearance and reappearance, and proposes an online object tracking model named 3AM. The core idea is to incorporate frozen MUSt3R features into SAM2.1-L. Features from different sources are fused into SAM2 through channel-wise modulation. Routing tokens read the current frame and historical memories through SAM2’s Memory Attention and generate fusion parameters for the next frame, allowing the model to dynamically adjust its reliance on appearance and geometric information.

3AM achieves clear improvements over previous methods on the ScanNet++ Selected Subset constructed by the authors. It also maintains good performance on conventional dynamic tracking benchmarks and transfers zero-shot to Ego-Exo4D.

Critical strengths
  1. The paper studies an important failure mode of existing memory-based trackers: after an object disappears for a long period and reappears under a substantially different viewpoint, the tracker may fail to recover the identity of the same instance. This is a valuable problem and remains challenging in spatial reasoning.

  2. The paper provides relatively comprehensive baseline comparisons, ablation studies, efficiency measurements, and feature analyses. Overall, the paper is clearly written and the method is well explained.

Critical weaknesses

1. This is my primary concern: the paper emphasizes online deployment, but its main evaluation protocol does not fully match this setting.

On ScanNet++ and Replica, the authors first inspect all ground-truth masks of the target across the entire sequence, select the frame in which the target has the largest visible mask as the prompt frame, and then track both forward and backward from that frame. Such a maximum-area prompt frame usually provides a larger, more complete, and less occluded target, a higher-quality initial memory, and a more stable MUSt3R geometric anchor.

The current evaluation therefore measures inference conditioned on a selected high-quality mask. In real online deployment, however, the system operates in real time and cannot inspect the entire video in advance, select the most suitable frame, and then perform both forward and backward propagation. The current experiments mainly demonstrate that the model has strong cross-view propagation ability when initialized from the best keyframe selected using full-sequence ground truth. They do not establish whether the large improvements would persist under realistic online initialization.

I suggest that the authors at least provide results using the first visible frame as the prompt. Results averaged over randomly sampled visible prompt frames, including the mean and variance, or results grouped by prompt-mask size or target visibility, would also be helpful.

2. Minor concern: the core behavior of the routing mechanism lacks direct validation.

The paper claims that this “switch” changes frame by frame within a video according to the object’s state. However, the experiments only report average routing coefficients on two different datasets: 50.7% geometric contribution on ScanNet++ and 17.8% on DAVIS17.

These results could also be explained by a simpler behavior:

  • The model recognizes a ScanNet++-style static indoor video and relies more heavily on geometry throughout the sequence.
  • The model recognizes a DAVIS-style dynamic video and relies less heavily on geometry throughout the sequence. This would not necessarily demonstrate frame-by-frame switching.

The claim could be directly validated with a simple experiment: 1. Select a video containing a “static → moving → static again” transition. 2. Mark the ground-truth motion speed of the object over time. 3. Examine whether the geometric gate exhibits the expected “high → low → high” pattern.

Questions for authors
  1. Could the authors address my primary concern and provide the corresponding additional experimental results?

  2. Could the authors provide direct validation of the routing mechanism described in Concern 2?

I would be inclined to accept the paper if my concerns are adequately addressed.

Reviewer_5MjuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces 3AM, a promptable online video object tracking framework designed to maintain object identity under large viewpoint changes, object reappearance, and motion. It integrates SAM2 with geometric features from MUSt3R through multi-layer feature fusion and adaptive routing tokens, enabling the model to dynamically balance appearance and geometric information. Experiments on ScanNet++, Replica, and several video tracking benchmarks demonstrate substantial improvements in tracking consistency under large viewpoint changes. The method also shows zero-shot generalization on Ego-Exo4D.

Critical strengths
  1. Practical Relevance. The paper addresses the important challenge of maintaining object identity under large viewpoint changes and long-term reappearance, where existing appearance-based video object tracking methods often struggle. The proposed RGB-only, online tracking framework is well motivated and has practical value.

  2. Extensive Experiments. The proposed method is evaluated on diverse benchmarks, including ScanNet++, Replica, DAVIS17, LaSOT, and Ego-Exo4D. The results demonstrate substantial improvements over existing methods, particularly in challenging cross-view tracking scenarios.

Critical weaknesses
  1. Unclear naming and task positioning. The term "3egment Anything" in the title is confusing and is not clearly explained in the paper. The proposed method essentially performs 2D video object segmentation, with geometric priors provided by a pretrained 3D foundation model. The current naming may mislead readers into expecting a 3D segmentation or tracking method.

  2. Insufficient evidence for geometric consistency. The paper does not convincingly demonstrate that the performance gains stem from 3D geometric consistency rather than stronger cross-view correspondence features provided by MUSt3R. Since the proposed method mainly incorporates intermediate MUSt3R features into SAM2 through feature fusion and routing, it remains unclear whether the improvements are due to geometry-specific information or simply better pretrained representations for cross-view matching. To better isolate the contribution of 3D geometry, it would be useful to compare against strong 2D visual foundation models, such as DINO.

  3. Strong dependence on the 3D foundation model. The proposed method relies heavily on the pretrained MUSt3R backbone, making it unclear how much of the improvement comes from the proposed architecture itself. Although the authors provide a replacement experiment with CUT3R, its substantial performance drop raises further concerns about backbone dependence. More comprehensive experiments with alternative 3D foundation models would help establish the generality of the proposed approach.

  4. Insufficient methodological details. The method section is relatively brief, especially considering that adaptive geometric consistency is presented as the main contribution. The core routing mechanism is described at a high level, with limited explanation of how routing tokens interact with memory attention and update the fusion parameters across frames.

Questions for authors
  1. The paper introduces the term "3egment Anything" in the title, but its intended meaning is not clearly explained. Since the proposed method primarily performs 2D video object segmentation with geometry-aware features, the naming may suggest a broader 3D segmentation or tracking capability. Could the authors clarify the motivation behind this terminology and how they position the proposed method relative to conventional 2D video segmentation and 3D tracking methods?

  2. The paper attributes the performance improvements to the introduction of geometric consistency through MUSt3R features. However, since the method mainly incorporates intermediate features from a pretrained reconstruction model, it is unclear whether the gains specifically come from 3D geometric information or simply from stronger cross-view feature representations. I would like to see a comparison with strong 2D visual foundation models, such as DINOv2, using the same feature fusion and routing framework. This would help clarify whether 3D geometric priors are essential to the observed improvements.

  3. Relatedly, the effectiveness of the proposed method appears to depend heavily on the choice of the 3D foundation model. Table 15 shows a substantial performance drop when replacing MUSt3R with CUT3R, with Tracking Recall decreasing from 65.7% to 27.5%. Could the authors provide more analysis of why this gap is so large? It would also be helpful to evaluate the framework with additional online 3D foundation models. At the moment, it remains unclear whether the proposed fusion and routing mechanisms are broadly effective or particularly tailored to MUSt3R features.

  4. Finally, the routing mechanism is a central component of the proposed method, but its description in the main paper is rather brief. I would appreciate a clearer explanation of how routing tokens interact with memory attention and how the routing states are initialized and updated across frames. In particular, how does the routing mechanism determine when geometric features should be emphasized or suppressed? A more detailed formulation or pseudocode would help clarify the underlying mechanism.

2 comments

I thank the authors for the detailed rebuttal and additional experiments.

Naming and task positioning

I appreciate the clarification of the term "3egment Anything." While the naming remains somewhat ambiguous, I consider this a minor presentation issue.

Geometric consistency and backbone dependence

The additional comparisons with DINOv3, VGGT, and LingBot-Map (Table 24) provide useful evidence for the contribution of geometry-aware features and the generality of the proposed framework. The explanation of CUT3R's performance drop is also helpful. These results largely address my concerns.

Methodological clarity

The revised Section 3.2 and the added pseudocode (Algorithm 1) clarify the routing mechanism and its interaction with memory attention. This addresses my concern.

Overall

The rebuttal addresses most of my concerns, and the additional experiments strengthen the paper. I am therefore raising my score to 3.

We sincerely thank the reviewer for the positive feedback and very helpful reviews, and will continue to improve our presentation in the final version!

Reviewer_aZqUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

3AM addresses object tracking through large viewpoint changes, disappearance and reappearance, and object motion. It fuses frozen MUSt3R geometry features with SAM2.1 appearance features. Object-specific routing tokens read the current fused features and tracking memory and set the next frame's fusion weights. The model is trained on static scenes and dynamic videos. The main results show large gains on wide-baseline tracking while retaining competitive performance on dynamic videos.

Critical strengths
  1. Relevant controls support the contribution. On the ScanNet++ Selected Subset, 3AM reaches 65.7 Recall versus 40.9 for SAM2 fine-tuned on the authors' data. Learned fixed fusion reaches 59.6 Recall and 86.9 DAVIS J&F, versus 65.7 and 90.4 with routing (Table 7).
  2. The evaluation identifies where the method helps. In Table 10, the Recall gain over SAM2 increases from 15.2 points at 0–15° camera rotation to 40.4 at 120–180°, with scene-bootstrap intervals and within-object comparisons. Whole-set ScanNet++ results reduce dependence on the selected subset. ADT Recall improves from 69.1 for the best listed SAM2-based tracker to 76.9, while dynamic-video performance remains competitive (Table 1).
Critical weaknesses
  1. The evidence for motion-aware routing and feature selection is incomplete. Table 7 supports adaptive over fixed fusion, but each frame's gates are set by the previous frame, and routing reads fused features rather than each unweighted geometry source. The paper does not show how this design responds to motion onset or re-enables suppressed geometry. Table 11 changes both motion and image domain, so it cannot isolate motion adaptation. The layer probes are useful, but the claimed benefit of the selected layers and explicit-geometry branch lacks a quantitative ablation.
  2. The strongest controls and the streaming protocol need broader validation. SAM2† and fixed fusion are compared on 286 selected ScanNet++ objects, about 7% of the evaluated instances, but not on the Whole Set, ADT or Ego-Exo4D. Whole-set gains are substantial, yet these controls are needed to attribute them to the design.
Questions for authors
  1. Can you compare adaptive and fixed fusion on the same ADT objects at motion start/stop and reappearance, with routing responses? Please explain recovery from suppressed geometry and report the gate statistics behind the statement that Table 7(f) never opens its geometry paths.
  2. Can you extend SAM2† and fixed fusion to ScanNet++ Whole Set and key transfer or mixed-motion tasks?
  3. Which frames inform each MUSt3R feature in forward/backward evaluation, and how are states and caches shared or reset?
  4. Appendix B.1 claims no validation-scene overlap with MUSt3R training, while MUSt3R includes Replica. Which checkpoint and scene exclusions reconcile these statements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.