- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates re-acquisition failures in SAM2 when a long interruption makes useful pre-interruption memories inaccessible to its fixed recency window. It proposes Persistent Memory Reinjection (PMR), which retains previously encoded memories and re-keys the freshest entries into existing attention slots when the window is incomplete; separate always-on and confidence-triggered variants address continuously processed videos. The authors evaluate a controlled frame-skip protocol on MOSEv2 and DAVIS-2017, along with natural failure episodes, buffer-size ablations, fine-tuning, and a comparison with SAM2Long.
Critical strengths
-
The paper identifies a concrete memory-access failure in a popular streaming segmenter. The fixed-window analysis explains why previously computed memories can become inaccessible following an interruption, and the controlled experiments test that diagnosis directly.
-
PMR is an economical inference-time intervention. Its gap-length and buffer-size analyses provide evidence that access to a few recent pre-interruption memories can recover object identity without retraining the segmenter.
Critical weaknesses
-
The main evaluation does not establish an improvement for occlusions encountered during ordinary video propagation. In Sec. 4.1, the controlled benchmark skips the entire interval, including the RGB frames, so missing entries are created in the recent-memory window. This directly enables the proposed structural gate. During continuous propagation, however, the model still writes memory on occluded frames, and the gate does not fire (Sec. 4.8). The gated method therefore produces exactly the SAM2 baseline results in Table 4. For the always-on variant designed for natural videos, Supplementary Table A2 reports J=0.7744 versus 0.7750 for SAM2, with no significant improvement in full-sequence accuracy. Recovery on episodes selected because the baseline already failed is useful evidence, but it does not replace evaluation of the net effect over complete videos. I am not convinced that the large gains in Table 1 support the broader occlusion-recovery claim.
-
The comparisons are too limited to establish the incremental contribution of this memory policy. The authors do compare SAM2Long [1], but Supplementary Table A4 shows statistically comparable improvements across the 700 natural failure episodes, with +0.156 for PMR and +0.165 for SAM2Long. The significant PMR advantage is confined to 40 beyond-window occlusion episodes (Supplementary Table A5), mostly with relatively short absences. No corresponding comparison establishes an advantage on full continuous-video benchmarks, and the relevance of PMR to newer memory-based SAM 3 systems [3] is untested. Training-free occlusion-aware memory management also predates this work, including MA-SAM2 [2]. A simple last-valid-memory retention control would help isolate the benefit of re-keying from the benefit of preserving a reliable memory. Moreover, the learned and oracle retrieval ablations change buffer admission along with selection (Sec. 4.7), so they do not isolate memory-selection quality.
[1] SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree, ICCV 2025.
[2] Memory-Augmented SAM2 for Training-Free Surgical Video Segmentation, MICCAI 2025.
[3] SAM 3: Segment Anything with Concepts, arXiv 2025.
Questions for authors
-
In Supplementary Table A2, the always-on variant improves the selected failure episodes but not the full-sequence mean. Can the authors quantify how many frames or sequences improve versus deteriorate, using the existing continuous-video predictions?
-
Under the same FIFO bank and attention-slot budget, how does PMR compare with simply retaining the last reliable pre-gap memory? Does the improvement depend on the proposed temporal re-keying, rather than on keeping that memory accessible?
2 comments
Thanks for the rebuttal. I appreciate the additional comparisons, but my main concern remains, so I will keep my rating.
We thank the reviewer for their comments and provide responses below.
Always-on PMR, MOSEv2 held-out, 800 videos, all objects: 175 videos better, 164 worse, 461 within 0.5 J (131 bit-identical); 10.3% of frames better and 10.0% worse (of 57,554, 1 J tolerance); full-video J +0.30 [-0.32, +0.91]. Confidence-gated PMR: 102 / 86 / 612 (494 identical), +0.39 [-0.13, +0.90]. The gated structural variant is bit-identical to SAM2 on all 800 videos.
Corrected frame-skip protocol, post-gap J vs baseline:
The improvement comes from keeping pre-gap memory accessible: re-keying has no measurable effect (paired -0.1 / -0.1 / -1.2 / +0.1, all n.s.), and a single retained memory, not re-keyed, gives +11.2* / +11.1* / +7.2* / +9.4*. We will simplify the method accordingly.
With admission held fixed (K = 16 FIFO), selecting 4 buffered memories at random gives +11.1* / +11.0* / +8.8* / +11.8*, against +11.8* / +13.0* / +8.8* / +8.9* for recency; recency is a sufficient rule.
MA-SAM2's released code depends on a modified
sam2_baseand a loader that were not released, so it could not be run; we will cite it. On SAM 3.1, PMR gives +4.4* / +8.1* / +5.3* at g = 10 / 20 / 40. SAM 3 as released already keeps confidently present frames in memory with no time horizon, so PMR does not add to it; gated PMR is bit-identical to each base model on continuous video.