- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The study presents a general, data-driven approach that frames the essential memory mechanism of the SAM family of models as a sequence decision-making task. Extensive experiments show that the approach improves SAM performance and generally outperforms hand-crafted rules across a broad range of benchmarks for visual object tracking and video object segmentation.
Critical strengths
-
The approach is well motivated and clearly formulated. The memory mechanism is an important component of SAM family of models. Learning to manage memory is well motivated for SAM-based video trackers as decisions about which observations to retain are critical to the predictions of later frames. The framework including controller inputs, action space, reward, and training procedure are clearly described, which makes the approach straightforward to follow. The proposed light controller that improves the use of a limited memory budget could be useful in practice and provide foundation for future work on adaptive memory management in SAM.
-
The experiments cover a broad range of benchmarks and demonstrate solid improvements. The experiments cover a substantial range of tracking and video object segmentation benchmarks and evaluate both SAM2.1 and SAM3, showing the robustness of the framework across different backbones. The experiment results show compelling improvements over SAM3 and competing methods that use hand-crafted memory rules on several challenging benchmarks.
Critical weaknesses
- The controller design needs further justification. A fixed budget and a lightweight MLP are reasonable choices. However, the paper does not sufficiently explain why the bank must remain full after initialization under the single-entry eviction formulation. For example, does this design offer advantages over allowing the controller to retain a variable number of memories within the same maximum capacity when some memories are unreliable or redundant?
- The feature ablations do not isolate specific representation choices. Appendix E.1 provides useful feature-group ablations but does not separately examine foreground mean pooling with alternative pooling methods or evaluate alternatives to the chosen pairwise similarity measures.
Questions for authors
- Can you explain the intuition behind keeping the memory bank full after initialization? Have you compared single-entry eviction with allowing the controller to retain a variable number of memories within the same maximum capacity, particularly when some memories are unreliable or redundant?
- Have you compared foreground mean pooling with alternative pooling methods, while keeping the other components fixed?
- Have you evaluated alternative measures for computing similarities between memories, beyond the feature-group ablations in Appendix E.1?
- In Table 4, do SAM3 ours and SAM3RL use the same auxiliary-pointer selection rules and temporal encoding? Can you clarify whether the learned memory-selection policy is the only difference. If not, could you provide a comparison while the other settings are fixed?
2 comments
Response to Authors' Comment
Dear Author,
Thank you for your clarifications and for revising the manuscript. I have carefully read the revised version. The revisions make the paper clearer. I continue to find the work well motivated and technically sound, and I am happy to maintain my positive assessment.
Best regards
Thank you for your careful review, and for finding our approach well motivated, our controller practical, and our benchmark coverage broad. Below we answer each of the weaknesses and questions you raised.
W1 / Q1 — Full bank vs. variable number of memories
We have not compared against a variable-sized bank. We keep the bank full mainly for consistency with prior work. The native rules of SAM 2 and SAM 3 also evict exactly one entry per update, and prior rules such as DAM4SAM, SAMURAI and SAM2Long likewise keep the bank full. A full bank also keeps the design simple, as each action maps to exactly one slot and parallel rollouts can be batched without padding or masking. Even with a full bank, unreliable or redundant memories can be handled, since the policy can evict any stored frame or reject the incoming one at every step. Whether a variable-sized bank could still help, for example by keeping only a few reliable memories, is an interesting question for future work. We added this justification to Section 3 and mention a variable-sized bank as future work in Section 6.
W2 / Q2 + Q3 — Foreground mean pooling and similarity measures
Foreground mean pooling. We did not train actors with alternative pooling methods. We preferred foreground pooling over whole-frame pooling, as it keeps the memories more distinguishable. Specifically, on our LaSOT validation split, the mean cosine similarity between memory slots is 0.99 with whole-frame pooling but 0.87 with foreground pooling, likely because the mostly unchanged background dominates the small changes of the object. Feeding the unpooled features instead would require a much heavier actor, which we avoid to keep the controller lightweight. We agree this choice deserved more explanation, which we added to Section 3.1 and Appendix E.1.
Similarity measures. We did not try any other similarity measures. We use cosine similarity, as it is the standard measure for comparing embeddings. We expect other measures to matter little, so we did not consider them worth a full ablation. If you consider it important, we are happy to add such an ablation to the camera-ready version.
Q4 — Gains attributable to the learned eviction
SAM 3 (ours) in the submitted Table 4 indeed used SAM 3’s native object-pointer selection and pointer temporal encoding, not those of SAM3RL. This is because Table 4 was designed as a stride ablation, in which SAM 3 (ours) served as a minimal fix of the official strided SAM 3. The official version collapses because it reselects a shifted bank at every frame, likely a setting SAM 3 never sees in training. The fix keeps the bank unchanged between updates, as our method does, and changes nothing else. To avoid confusion, this row is now called SAM 3 (fix).
We agree that isolating the learned eviction is important. The name SAM (ours) now denotes a new row, which runs SAM 3’s eviction rule inside our MDP (Section 3.1). It shares the stride, the recently evicted pointer rule and the pointer temporal encoding with SAM3RL, so only the eviction policy differs. SAM3RL outperforms SAM 3 (ours) on all seven benchmarks, by 0.9 on average and most on VOT-LT2022 (+2.2) and VOTS2026 (+1.3). Going from SAM 3 (fix) to SAM 3 (ours), which changes only the pointer rule and encoding, adds 0.6 on average. Notably, SAM 3 (ours) alone already outperforms all prior methods on VOTS2026, VOTS2025 and LaSOT, but still falls short of our learned policy. We updated Table 4 and Section 5 accordingly.