- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MEMOIR, a region-level graph memory framework for long-term video object segmentation. The method consists of three components: Structured Graph Memory (SGM), which organizes historical region features through spatial, similarity, and temporal relations; Query-Conditioned Graph Retrieval (QCGR), which selectively retrieves relevant historical evidence; and Budget-Aware Memory Management (BAM), which maintains informative memories under bounded storage. The framework improves long-term segmentation and target reappearance performance without modifying or retraining the backbone model.
Critical strengths
Well-motivated memory design.
The paper addresses a meaningful limitation of frame-level memory in long-term video segmentation. Organizing historical information at the region level is a reasonable approach to preserving informative partial appearances.
Effective graph-based retrieval.
The combination of region-level representation, relational retrieval, and budget-aware memory management is technically meaningful. The ablation studies demonstrate the effectiveness of graph-based retrieval, particularly in long-term scenarios.
Comprehensive experimental validation.
The method is evaluated on seven VOS benchmarks, with additional experiments on visual object tracking and SAM2.1. The improvements in target reappearance scenarios support the proposed motivation.
Critical weaknesses
Significant computational overhead.
MEMOIR substantially reduces inference throughput compared with SAM3. According to Table 14, the speed decreases from 35.4 FPS to 8.2 FPS on LVOSv2, while the overall J&F improvement is only 1.6 points. This raises concerns about the practical accuracy-efficiency trade-off.
Insufficient comparisons under matched computational budgets.
Although the authors compare different memory representations and retrieval strategies, comparisons with simpler memory approaches under matched inference time and memory-attention budgets remain insufficient. It is therefore difficult to determine whether the additional graph complexity is justified.
Limited analysis of unreliable region associations.
The framework relies on feature-based region partitioning and temporal correspondence to construct its memory graph. Incorrect associations may introduce misleading historical evidence, particularly under rapid camera motion, severe appearance changes, or similar distractors. The robustness of these associations is not sufficiently analyzed.
Questions for authors
- Can the authors provide an accuracy-efficiency comparison against simpler long-term memory baselines under matched inference time and memory budgets?
- How robust is the proposed graph construction to incorrect temporal associations or region fragmentation? Have the authors evaluated its behavior under rapid camera motion or severe identity confusion?
- How sensitive is MEMOIR to the graph node budget, retrieval token budget, and region partitioning thresholds? Can the authors provide quantitative sensitivity analysis?
4 comments
(continued...)
Comment 3. Robustness to unreliable region associations.
Response. We thank the reviewer for the question. To directly address this question, we present additional experimental results for each of its points. First, on incorrect temporal associations, we measure the proportion of incorrect temporal edges in the graph. Since the datasets do not annotate correspondences between regions, we estimate this proportion with the ground-truth mask, and a temporal edge is counted as incorrect when one of its two ends lies mostly on the target and the other lies mostly on the background. Over all 239 objects of LVOSv2, such edges account for only 0.84% of the 1.5M temporal edges. For comparison, if the two ends of each edge were drawn independently from all 1.75M stored regions, such edges would account for 10.1%. Second, on region fragmentation, we evaluate MEMOIR with fragmented trajectories and with different region partitions. When the trajectories are fragmented by the removal of the modified z-score criterion (Table 8), MEMOIR obtains a J&F score of 89.8 on LVOSv2 and a J&Ḟ score of 61.1 on MOSEv2, against 88.8 and 60.0 for SAM 3 (Table 9). When the region partition is changed by varying from 1.15 to 1.35 (Table R3), MEMOIR stays at or above 89.3 on LVOSv2 and 61.0 on MOSEv2. Third, on fast motion, identity confusion, and appearance change, MEMOIR improves the J&F score of SAM 3 on the videos that LVOSv2 officially labels with fast motion (88.5 to 89.6), motion blur (83.5 to 87.8), background clutter (88.0 to 89.3), cross-temporal confusion (88.0 to 89.3), and appearance change (88.5 to 90.0), as reported in Table R5 below. These results validate that the graph construction of MEMOIR remains effective in each of these cases. We thank the reviewer again for the question, and hope that these results answer it. The revised paper includes these results in Appendix H.5.
Comment 4. Sensitivity to the graph-node budget, the retrieval token budget, and the partition threshold.
Response. We thank the reviewer for the question. To directly address this question, we present an additional sensitivity study in Table R3 below, in which one hyperparameter is varied at a time. The storage budget, which comprises the graph-node budget and the token-storage budget, is halved and doubled. The retrieval budget is varied from 2 to 6 frames of tokens, and the partition threshold is varied from 1.15 to 1.35. Across all of these settings, the score of MEMOIR varies by at most 1.1 points on SA-V val, SA-V test, and LVOSv2, and by at most 1.4 points on MOSEv2. Moreover, MEMOIR remains above SAM 3 at every setting on SA-V val (83.6 to 84.7 against 83.2), LVOSv2 (89.3 to 90.3 against 88.8), and MOSEv2 (61.0 to 62.4 against 60.0). These results validate that the performance of MEMOIR is stable with respect to these hyperparameters. We thank the reviewer again for the question, and hope that these results answer it. The revised paper includes Table R3 in Appendix H.3.
(to be continued...)
(continued...)
Table R1. SAM 3-based memory variants, run from the SAM 3 implementations of Alansari et al. (2025) under our protocol, in which each object is tracked from its ground-truth mask and scored with the same toolkit as MEMOIR. Each dataset reports the standard metric (J&F, and J&Ḟ on MOSEv2), together with the disappearance score D and the reappearance score R of Table 2. SAM 3 is our reproduction. Bold marks the highest value of each column.
| SA-V val | SA-V test | LVOSv2 | MOSEv2 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | J&F | D | R | J&F | D | R | J&F | D | R | J&Ḟ | D | R |
| SAM 3 | 83.2 | 82.6 | 61.5 | 84.6 | 83.2 | 62.8 | 88.8 | 67.2 | 75.4 | 60.0 | 64.4 | 42.7 |
| SAMURAI-3 | 78.9 | 63.1 | 63.5 | 81.5 | 69.5 | 63.7 | 88.4 | 52.2 | 77.8 | 56.9 | 45.3 | 48.1 |
| SAM3Long | 82.7 | 81.0 | 61.7 | 85.0 | 82.7 | 64.5 | 88.8 | 64.6 | 76.0 | 59.3 | 61.5 | 44.5 |
| DAM4SAM-3 | 80.4 | 77.1 | 61.9 | 82.7 | 81.7 | 64.1 | 89.1 | 64.1 | 76.8 | 57.2 | 57.3 | 44.5 |
| SAMITE-3 | 81.4 | 69.0 | 64.9 | 82.4 | 70.6 | 65.5 | 88.8 | 55.2 | 77.9 | 54.1 | 41.2 | 45.7 |
| HiM2SAM-3 | 80.3 | 70.2 | 62.9 | 81.3 | 71.4 | 62.3 | 85.8 | 50.8 | 75.8 | 50.0 | 38.0 | 43.0 |
| MEMOIR (ours) | 84.3 | 84.2 | 63.9 | 85.0 | 84.0 | 63.6 | 90.1 | 67.1 | 79.1 | 62.4 | 65.2 | 46.7 |
Table R2. Equal-budget baseline, in which SAM 3 additionally reads five older full frames (25,920 extra tokens per frame). The metrics follow Table R1.
| SA-V val | SA-V test | LVOSv2 | MOSEv2 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | J&F | D | R | J&F | D | R | J&F | D | R | J&Ḟ | D | R |
| SAM 3 | 83.2 | 82.6 | 61.5 | 84.6 | 83.2 | 62.8 | 88.8 | 67.2 | 75.4 | 60.0 | 64.4 | 42.7 |
| SAM 3 + five older frames | 83.0 | 82.7 | 61.1 | 84.5 | 83.0 | 62.5 | 89.5 | 67.3 | 77.2 | 60.0 | 63.3 | 44.1 |
| MEMOIR (ours) | 84.3 | 84.2 | 63.9 | 85.0 | 84.0 | 63.6 | 90.1 | 67.1 | 79.1 | 62.4 | 65.2 | 46.7 |
Table R3. Sensitivity to the hyperparameters that the reviewer named (J&F, and J&Ḟ on MOSEv2). One hyperparameter is varied at a time, and the default setting is in bold. SAM 3 reaches 83.2, 84.6, 88.8, and 60.0 on the four datasets. The revised paper reports the complete table.
| Parameter | Setting | SA-V val | SA-V test | LVOSv2 | MOSEv2 |
|---|---|---|---|---|---|
| / (frames) | 2 | 84.7 | 84.4 | 89.6 | 61.8 |
| 4 | 84.3 | 85.0 | 90.1 | 62.4 | |
| 6 | 84.0 | 84.6 | 90.0 | 61.5 | |
| Storage budget | x0.5 | 83.6 | 83.9 | 89.7 | 62.1 |
| x1 | 84.3 | 85.0 | 90.1 | 62.4 | |
| x2 | 84.2 | 84.8 | 90.3 | 62.3 | |
| 1.15 | 84.5 | 84.5 | 89.5 | 61.4 | |
| 1.25 | 84.3 | 85.0 | 90.1 | 62.4 | |
| 1.35 | 84.0 | 84.6 | 89.3 | 61.0 |
Table R5. LVOSv2 val by official attribute (J&F). The five subsets contain 71, 40, 82, 79, and 96 of the 140 videos. The BC and CTC subsets are different sets of videos, and their scores differ in the second decimal.
| Subset | SAM 3 | MEMOIR (ours) |
|---|---|---|
| All | 88.8 | 90.1 |
| Fast motion (FM) | 88.5 | 89.6 |
| Motion blur (MB) | 83.5 | 87.8 |
| Background clutter (BC) | 88.0 | 89.3 |
| Cross-temporal confusion (CTC) | 88.0 | 89.3 |
| Appearance change (AC) | 88.5 | 90.0 |
Thank you for the detailed responses and the additional experiments in Tables R1–R5. They resolve my concerns and I will keep my score.
We appreciate the reviewer's time and effort spent on the review, and would like to respond as follows. The table and figure numbers in our responses follow the revised paper, and we kindly refer the reviewer to it. Unless otherwise noted, the scores in our responses are reported in the standard metric of each benchmark, which is J&F on SA-V and LVOSv2 and J&Ḟ on MOSEv2.
Response. We thank the reviewer for the comment, and respectfully clarify that efficiency is not the subject of this paper. This paper investigates long-term memory in video object segmentation, in which recovering a target after prolonged absence requires historical evidence that lies beyond the recency window of SAM 3. The additional cost of MEMOIR is bounded, and its gain is concentrated on the cases that require long-term memory. The improvement on LVOSv2 that the reviewer cites is an average over all frames, whereas long-term memory is evaluated in the reappearance regime, in which historical evidence is the only usable cue. In this regime, the reappearance score improves by +2.4, +0.8, +3.7, and +4.0 points on SA-V val, SA-V test, LVOSv2, and MOSEv2 (Table 2). The gain is the largest where the history is the longest: +8.9 points in the reappearance score after absences longer than 8 s (Fig. 7), and +5.7 points in J&Ḟ on videos longer than 80 s (Fig. 6). Moreover, the existing memory designs do not improve SAM 3 consistently. To directly address this question, we present an additional comparison with five SAM 3-based memory variants under our protocol in Table R1 below. All five variants fall below the score of SAM 3 on SA-V val and MOSEv2, and MEMOIR is the only method in Table R1 that improves SAM 3 on all four datasets. The cost of these gains is bounded. Throughput depends on the hardware and on the implementation, and the cost is therefore measured by the number of tokens that memory attention reads: MEMOIR reads 42.8k to 47.1k tokens per frame, which is 1.3 times the 34.1k to 36.0k tokens of SAM 3 (Table 15), and its stored memory stays within a fixed budget regardless of the video length (Fig. 3). Furthermore, an optimized implementation of MEMOIR, which produces identical outputs, runs 31% faster on DAVIS 2017 and 43% faster on the five longest videos of LVOSv2 than the original implementation on the same GPU. We thank the reviewer again for the comment, and hope that this response addresses it.
Response. We thank the reviewer for the suggestion, and respectfully clarify that the graph is the component that keeps the history beyond the recency window accessible within a fixed budget, and that a simpler long-term memory with the same memory-attention budget does not reproduce the gain of MEMOIR. To directly address this question, we present an additional equal-budget baseline in Table R2 below. As in our response to Comment 1, the budget is measured in the number of tokens that memory attention reads. In this baseline, SAM 3 additionally reads five older full frames, which amounts to 25,920 extra tokens per frame and equals the long-term read budget of MEMOIR. The five frames are selected by recency and quality score, which extends the memory-selection rule of SAM 3 itself to the frames outside its native window. This baseline changes the score of SAM 3 by -0.2, -0.1, +0.7, and 0.0 points on SA-V val, SA-V test, LVOSv2, and MOSEv2, whereas MEMOIR improves it by +1.1, +0.4, +1.3, and +2.4 points. In this setting, the two methods share the same long-term read budget. In practice, the baseline reads 56.1k and 61.3k tokens per frame on DAVIS 2017 and LVOSv2, which is more than the 42.8k and 47.1k tokens of MEMOIR (Appendix H.2 and Table 15). The difference therefore lies in the history that this budget covers, rather than in the amount of memory that is read. Specifically, the five full frames cover a short span of recent history. In contrast, the graph of MEMOIR keeps the history beyond the recency window in a compressed and organized form, and the same read budget therefore reaches the regions of this history that are most relevant to the current frame. Furthermore, the five SAM 3-based memory variants of Table R1 all fall below the score of SAM 3 on SA-V val and MOSEv2. We thank the reviewer again for the suggestion, and hope that this comparison addresses it.
(to be continued...)