acceptodds
Under review as a conference paper at ICLR 2027

MEMOIR: Region-Level Graph Memory for Long-Term Video Object Segmentation

Abstract

Long-term video object segmentation must preserve target identity across prolonged absence, severe occlusion, and similar distractors, a requirement that ties each prediction to historical appearance evidence far beyond the recent observations. Existing memory designs retain this history as whole frames within a bounded window and expose all of it to every prediction. Informative partial appearances therefore vanish with the discarded frames, and every prediction attends to the entire store regardless of what it requires. To address this limitation, we propose Memory with Evidence Modeled On Interconnected Regions (MEMOIR), a graph memory that stores, maintains, and exposes historical evidence at the level of spatially coherent regions. MEMOIR organizes the region-level memories of a frozen SAM-style base model into a graph whose edges encode spatial, similarity, and temporal relations. Retrieval exposes long-term evidence only where recent observations fail to support the prediction, and management maintains the graph within explicit node and token budgets according to temporal support and retrieval utility rather than age. The base model itself remains unchanged, and no training is required. Across seven video object segmentation benchmarks, MEMOIR improves on the SAM 3 base model with the largest gains on LVOSv2 and MOSEv2, whose long and crowded videos stress memory most, and exceeds SAM 3 on all four reappearance splits, where historical evidence determines the prediction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video object segmentation, long-term memory, graph memory, memory retrieval, training-free, Segment Anything Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6iJLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MEMOIR, a region-level graph memory framework for long-term video object segmentation. The method consists of three components: Structured Graph Memory (SGM), which organizes historical region features through spatial, similarity, and temporal relations; Query-Conditioned Graph Retrieval (QCGR), which selectively retrieves relevant historical evidence; and Budget-Aware Memory Management (BAM), which maintains informative memories under bounded storage. The framework improves long-term segmentation and target reappearance performance without modifying or retraining the backbone model.

Critical strengths

Well-motivated memory design.

The paper addresses a meaningful limitation of frame-level memory in long-term video segmentation. Organizing historical information at the region level is a reasonable approach to preserving informative partial appearances.

Effective graph-based retrieval.

The combination of region-level representation, relational retrieval, and budget-aware memory management is technically meaningful. The ablation studies demonstrate the effectiveness of graph-based retrieval, particularly in long-term scenarios.

Comprehensive experimental validation.

The method is evaluated on seven VOS benchmarks, with additional experiments on visual object tracking and SAM2.1. The improvements in target reappearance scenarios support the proposed motivation.

Critical weaknesses

Significant computational overhead.

MEMOIR substantially reduces inference throughput compared with SAM3. According to Table 14, the speed decreases from 35.4 FPS to 8.2 FPS on LVOSv2, while the overall J&F improvement is only 1.6 points. This raises concerns about the practical accuracy-efficiency trade-off.

Insufficient comparisons under matched computational budgets.

Although the authors compare different memory representations and retrieval strategies, comparisons with simpler memory approaches under matched inference time and memory-attention budgets remain insufficient. It is therefore difficult to determine whether the additional graph complexity is justified.

Limited analysis of unreliable region associations.

The framework relies on feature-based region partitioning and temporal correspondence to construct its memory graph. Incorrect associations may introduce misleading historical evidence, particularly under rapid camera motion, severe appearance changes, or similar distractors. The robustness of these associations is not sufficiently analyzed.

Questions for authors
  • Can the authors provide an accuracy-efficiency comparison against simpler long-term memory baselines under matched inference time and memory budgets?
  • How robust is the proposed graph construction to incorrect temporal associations or region fragmentation? Have the authors evaluated its behavior under rapid camera motion or severe identity confusion?
  • How sensitive is MEMOIR to the graph node budget, retrieval token budget, and region partitioning thresholds? Can the authors provide quantitative sensitivity analysis?
4 comments

We appreciate the reviewer's time and effort spent on the review, and would like to respond as follows. The table and figure numbers in our responses follow the revised paper, and we kindly refer the reviewer to it. Unless otherwise noted, the scores in our responses are reported in the standard metric of each benchmark, which is J&F on SA-V and LVOSv2 and J&Ḟ on MOSEv2.

Comment 1. The computational overhead of MEMOIR relative to its accuracy gain.

Response. We thank the reviewer for the comment, and respectfully clarify that efficiency is not the subject of this paper. This paper investigates long-term memory in video object segmentation, in which recovering a target after prolonged absence requires historical evidence that lies beyond the recency window of SAM 3. The additional cost of MEMOIR is bounded, and its gain is concentrated on the cases that require long-term memory. The improvement on LVOSv2 that the reviewer cites is an average over all frames, whereas long-term memory is evaluated in the reappearance regime, in which historical evidence is the only usable cue. In this regime, the reappearance score improves by +2.4, +0.8, +3.7, and +4.0 points on SA-V val, SA-V test, LVOSv2, and MOSEv2 (Table 2). The gain is the largest where the history is the longest: +8.9 points in the reappearance score after absences longer than 8 s (Fig. 7), and +5.7 points in J&Ḟ on videos longer than 80 s (Fig. 6). Moreover, the existing memory designs do not improve SAM 3 consistently. To directly address this question, we present an additional comparison with five SAM 3-based memory variants under our protocol in Table R1 below. All five variants fall below the score of SAM 3 on SA-V val and MOSEv2, and MEMOIR is the only method in Table R1 that improves SAM 3 on all four datasets. The cost of these gains is bounded. Throughput depends on the hardware and on the implementation, and the cost is therefore measured by the number of tokens that memory attention reads: MEMOIR reads 42.8k to 47.1k tokens per frame, which is 1.3 times the 34.1k to 36.0k tokens of SAM 3 (Table 15), and its stored memory stays within a fixed budget regardless of the video length (Fig. 3). Furthermore, an optimized implementation of MEMOIR, which produces identical outputs, runs 31% faster on DAVIS 2017 and 43% faster on the five longest videos of LVOSv2 than the original implementation on the same GPU. We thank the reviewer again for the comment, and hope that this response addresses it.

Comment 2. Comparison with simpler memory approaches under matched budgets.

Response. We thank the reviewer for the suggestion, and respectfully clarify that the graph is the component that keeps the history beyond the recency window accessible within a fixed budget, and that a simpler long-term memory with the same memory-attention budget does not reproduce the gain of MEMOIR. To directly address this question, we present an additional equal-budget baseline in Table R2 below. As in our response to Comment 1, the budget is measured in the number of tokens that memory attention reads. In this baseline, SAM 3 additionally reads five older full frames, which amounts to 25,920 extra tokens per frame and equals the long-term read budget of MEMOIR. The five frames are selected by recency and quality score, which extends the memory-selection rule of SAM 3 itself to the frames outside its native window. This baseline changes the score of SAM 3 by -0.2, -0.1, +0.7, and 0.0 points on SA-V val, SA-V test, LVOSv2, and MOSEv2, whereas MEMOIR improves it by +1.1, +0.4, +1.3, and +2.4 points. In this setting, the two methods share the same long-term read budget. In practice, the baseline reads 56.1k and 61.3k tokens per frame on DAVIS 2017 and LVOSv2, which is more than the 42.8k and 47.1k tokens of MEMOIR (Appendix H.2 and Table 15). The difference therefore lies in the history that this budget covers, rather than in the amount of memory that is read. Specifically, the five full frames cover a short span of recent history. In contrast, the graph of MEMOIR keeps the history beyond the recency window in a compressed and organized form, and the same read budget therefore reaches the regions of this history that are most relevant to the current frame. Furthermore, the five SAM 3-based memory variants of Table R1 all fall below the score of SAM 3 on SA-V val and MOSEv2. We thank the reviewer again for the suggestion, and hope that this comparison addresses it.

(to be continued...)

(continued...)

Comment 3. Robustness to unreliable region associations.

Response. We thank the reviewer for the question. To directly address this question, we present additional experimental results for each of its points. First, on incorrect temporal associations, we measure the proportion of incorrect temporal edges in the graph. Since the datasets do not annotate correspondences between regions, we estimate this proportion with the ground-truth mask, and a temporal edge is counted as incorrect when one of its two ends lies mostly on the target and the other lies mostly on the background. Over all 239 objects of LVOSv2, such edges account for only 0.84% of the 1.5M temporal edges. For comparison, if the two ends of each edge were drawn independently from all 1.75M stored regions, such edges would account for 10.1%. Second, on region fragmentation, we evaluate MEMOIR with fragmented trajectories and with different region partitions. When the trajectories are fragmented by the removal of the modified z-score criterion (Table 8), MEMOIR obtains a J&F score of 89.8 on LVOSv2 and a J&Ḟ score of 61.1 on MOSEv2, against 88.8 and 60.0 for SAM 3 (Table 9). When the region partition is changed by varying τg\tau_g from 1.15 to 1.35 (Table R3), MEMOIR stays at or above 89.3 on LVOSv2 and 61.0 on MOSEv2. Third, on fast motion, identity confusion, and appearance change, MEMOIR improves the J&F score of SAM 3 on the videos that LVOSv2 officially labels with fast motion (88.5 to 89.6), motion blur (83.5 to 87.8), background clutter (88.0 to 89.3), cross-temporal confusion (88.0 to 89.3), and appearance change (88.5 to 90.0), as reported in Table R5 below. These results validate that the graph construction of MEMOIR remains effective in each of these cases. We thank the reviewer again for the question, and hope that these results answer it. The revised paper includes these results in Appendix H.5.

Comment 4. Sensitivity to the graph-node budget, the retrieval token budget, and the partition threshold.

Response. We thank the reviewer for the question. To directly address this question, we present an additional sensitivity study in Table R3 below, in which one hyperparameter is varied at a time. The storage budget, which comprises the graph-node budget and the token-storage budget, is halved and doubled. The retrieval budget is varied from 2 to 6 frames of tokens, and the partition threshold τg\tau_g is varied from 1.15 to 1.35. Across all of these settings, the score of MEMOIR varies by at most 1.1 points on SA-V val, SA-V test, and LVOSv2, and by at most 1.4 points on MOSEv2. Moreover, MEMOIR remains above SAM 3 at every setting on SA-V val (83.6 to 84.7 against 83.2), LVOSv2 (89.3 to 90.3 against 88.8), and MOSEv2 (61.0 to 62.4 against 60.0). These results validate that the performance of MEMOIR is stable with respect to these hyperparameters. We thank the reviewer again for the question, and hope that these results answer it. The revised paper includes Table R3 in Appendix H.3.

(to be continued...)

(continued...)

Table R1. SAM 3-based memory variants, run from the SAM 3 implementations of Alansari et al. (2025) under our protocol, in which each object is tracked from its ground-truth mask and scored with the same toolkit as MEMOIR. Each dataset reports the standard metric (J&F, and J&Ḟ on MOSEv2), together with the disappearance score D and the reappearance score R of Table 2. SAM 3 is our reproduction. Bold marks the highest value of each column.

SA-V valSA-V testLVOSv2MOSEv2
MethodJ&FDRJ&FDRJ&FDRJ&ḞDR
SAM 383.282.661.584.683.262.888.867.275.460.064.442.7
SAMURAI-378.963.163.581.569.563.788.452.277.856.945.348.1
SAM3Long82.781.061.785.082.764.588.864.676.059.361.544.5
DAM4SAM-380.477.161.982.781.764.189.164.176.857.257.344.5
SAMITE-381.469.064.982.470.665.588.855.277.954.141.245.7
HiM2SAM-380.370.262.981.371.462.385.850.875.850.038.043.0
MEMOIR (ours)84.384.263.985.084.063.690.167.179.162.465.246.7

Table R2. Equal-budget baseline, in which SAM 3 additionally reads five older full frames (25,920 extra tokens per frame). The metrics follow Table R1.

SA-V valSA-V testLVOSv2MOSEv2
MethodJ&FDRJ&FDRJ&FDRJ&ḞDR
SAM 383.282.661.584.683.262.888.867.275.460.064.442.7
SAM 3 + five older frames83.082.761.184.583.062.589.567.377.260.063.344.1
MEMOIR (ours)84.384.263.985.084.063.690.167.179.162.465.246.7

Table R3. Sensitivity to the hyperparameters that the reviewer named (J&F, and J&Ḟ on MOSEv2). One hyperparameter is varied at a time, and the default setting is in bold. SAM 3 reaches 83.2, 84.6, 88.8, and 60.0 on the four datasets. The revised paper reports the complete table.

ParameterSettingSA-V valSA-V testLVOSv2MOSEv2
KretK_{ret} / BTretB_T^{ret} (frames)284.784.489.661.8
484.385.090.162.4
684.084.690.061.5
Storage budgetx0.583.683.989.762.1
x184.385.090.162.4
x284.284.890.362.3
τg\tau_g1.1584.584.589.561.4
1.2584.385.090.162.4
1.3584.084.689.361.0

Table R5. LVOSv2 val by official attribute (J&F). The five subsets contain 71, 40, 82, 79, and 96 of the 140 videos. The BC and CTC subsets are different sets of videos, and their scores differ in the second decimal.

SubsetSAM 3MEMOIR (ours)
All88.890.1
Fast motion (FM)88.589.6
Motion blur (MB)83.587.8
Background clutter (BC)88.089.3
Cross-temporal confusion (CTC)88.089.3
Appearance change (AC)88.590.0

Thank you for the detailed responses and the additional experiments in Tables R1–R5. They resolve my concerns and I will keep my score.

Reviewer_bMNAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MEMOIR adds a training-free long-term memory to a frozen SAM 3 for semi-supervised VOS. SAM 3 reads only the prompt frame and the six most recent object-present frames, so after a long absence its memory often has nothing that can re-identify the target. MEMOIR keeps older history as a graph of regions. Each frame's memory tokens are split into spatially connected regions with homogeneous features, and the regions are linked by adjacency, same-frame similarity and cross-frame correspondence edges (SGM).

At each frame, query regions that overlap little with regions from the last 10 frames trigger retrieval. Directly matched nodes seed a personalized PageRank, and the top-ranked nodes are deduplicated along each trajectory and appended to SAM 3's native memory (QCGR). The graph is kept under node and token budgets: regions that never get a temporal match are dropped, near-duplicate segments are merged, nodes are evicted by their average retrieval relevance, and evicted segments are compressed into anchor tokens (BAM).

The claimed contributions are regions as the memory unit, separating what is stored from what is read, and retention by utility rather than recency. Validation covers seven VOS settings, disappearance/reappearance scores on four splits, component ablations, a port to SAM 2.1, five tracking benchmarks, runtime and memory measurements, and two propositions on accessibility and retrieval cost.

Critical strengths
  1. The gains show up where the stated problem is. Over a same-protocol SAM 3, reappearance scores improve on all four splits (+0.8 to +4.0 J&Ḟ_r), while disappearance scores barely move (−0.1 to +1.6). The largest standard gains are on LVOSv2 (+1.6) and MOSEv2 (+2.1). The method also helps SAM 2.1 without retraining (LVOSv2 84.1→87.3, MOSEv2 49.5→52.8). This is the main reason I take the paper seriously: it shows that SAM 3's recency window is a real bottleneck on long videos, and that changing only the memory side recovers part of the loss.
  2. Within the full pipeline, region nodes beat frame and grid nodes on all four long-term settings (Tables 3 and 5; LVOSv2 89.3 / 89.7 / 90.1, MOSEv2 60.8 / 61.6 / 62.4). This is the most direct evidence for the paper's central claim that the memory unit matters.
Critical weaknesses

Main weakness: the paper does not show that the region-graph machinery is necessary.

  • The gains are small. On the standard metrics, the change relative to SAM 3 ranges from −0.4 (DAVIS) to +2.1, with 0.0 on MOSEv1.
  • The cost is large.
    • Speed drops to a third or a quarter of SAM 3 (DAVIS 38.1→12.7 FPS, LVOSv2 35.4→8.2 FPS).
    • The graph holds 155k–259k tokens, i.e. 30–50 frames' worth, versus 7 frames for SAM 3.
    • Table 12 lists 33 fixed settings. The paper gives no sensitivity analysis and does not say how they were chosen.
  • The baselines are insufficient.
    • In VOS, the only same-backbone baseline is vanilla SAM 3. SAM 3-based memory variants (SAM3Long, DAM4SAM-3, SAMURAI-3, SAMITE-3, HiM2SAM-3) already appear in the authors' own tracking table, and there MEMOIR does not win: it trails DAM4SAM-3 on all three LaSOText metrics (63.9/75.3/73.7 vs 66.0/81.3/78.6) and SAMITE-3 on LaSOT success (76.9 vs 77.4).
    • There is no simple equal-budget baseline, such as reading a few older full frames within the same token budget.
  • The gain depends on a specific combination of components. On LVOSv2, removing graph propagation (87.0), removing consolidation (88.4) or dropping the middle-term frames (87.5) each falls below SAM 3 (88.5). Meanwhile, the frame-node variant, which keeps the rest of the pipeline, already gets half of the LVOSv2 gain (89.3).

Why this matters: from the current evidence I cannot tell whether the improvement comes from region-level graph memory, or simply from reading more old memory plus many tuned rules. This determines whether the design is a contribution or an over-engineered way of keeping more history.

Secondary weaknesses:

  • The mechanism is never measured directly.
    • The paper does not report how many regions are routed per frame, what fraction of retrieved or retained nodes lie on the target, or whether utility actually preserves target appearance. Utility is target-agnostic, and while the target is stably visible its regions are rarely routed, so the target may never accumulate high utility.
    • Figs. 6–7 do not state which datasets are pooled or how many videos fall in each bin. The long-video bins are likely dominated by LVOSv2, so length and dataset are confounded. The 4–8 s bin's CI also includes zero.
  • The source of the baseline numbers is unclear. Table 9 says its SAM 3 row comes from the same reproduction as Table 2, yet 6 of its 7 numbers are identical to Table 1, and the SAM 3 row in Table 1 is not marked as reproduced.
  • The theory adds little.
    • Prop. 1(b) holds by the definition of "accessible": every node in the graph is a retrieval candidate.
    • Prop. 2 bounds only retrieval cost. It excludes graph construction and management, which Appendix D calls the most expensive part. Any fixed-budget memory, including SAM 3's, already has length-independent cost.
  • Contribution 3 is overstated. XMem already evicts long-term memory by usage normalized by time in memory. The paper does not discuss this.
  • Reproducibility. The reproducibility statement says code is in the supplementary material, but the supplementary contains only a link. Several details are missing: the merge order and how R_max and N_min are enforced, the attention mask that restricts long-term memory to routed regions, whether consolidation refills the freed budget, and the positional encoding of anchor tokens.
Questions for authors

Requests that could change my rating:

  1. Please report VOS results for SAM 3-based memory variants (at least SAM3Long and DAM4SAM-3) under your per-object protocol, on LVOSv2, MOSEv2 and SA-V, including D/R scores.
  2. Please add an equal-budget baseline: SAM 3 plus extra older full frames, within the same read budget (about 26k extra tokens per frame) and the same storage budget. The frames could be selected by recency plus quality score, by uniform sampling, or by similarity to the current frame. If this matches MEMOIR, the region graph is not what drives the gain.
  3. How sensitive are the results to τ_route, K_ret/B_T, τ_g and the storage budget? How were the Table 12 values chosen, and on which data? Were they tuned on the validation sets you report?
  4. In the w/o Graph variant, what exactly is retrieved? Are Route and consolidation still on, with the same budget? Would a confidence threshold on s_v close the gap? I want to know whether PPR beats a reasonable similarity baseline, or only one that is worse than doing nothing.
5 comments

We appreciate the reviewer's time and effort spent on the review, and would like to respond as follows. The table and figure numbers in our responses follow the revised paper, and we kindly refer the reviewer to it.

Comment 1. Whether the gain of MEMOIR comes from the region-level graph memory, or from reading more memory from older frames together with tuned settings.

Response. We thank the reviewer for the question, and respectfully clarify that the gain comes from the region-level graph itself rather than from reading more memory from older frames or from tuned settings. To directly address this question, we present the four additional ablation analyses that the reviewer requested in Tables R1 to R4 below, in the order of the requests: five SAM 3-based memory variants under our protocol, an equal-budget baseline, a sensitivity study, and a similarity-retrieval baseline. Unless otherwise noted, the numbers below are reported in the standard metric of each benchmark, which is J&F on SA-V and LVOSv2 and J&Ḟ on MOSEv2, and they are listed for SA-V val, SA-V test, LVOSv2, and MOSEv2 in this order.

Reading more memory from older frames under the same read budget does not reproduce the gain. In the baseline that the reviewer suggested, SAM 3 additionally reads five older full frames, which amounts to 25,920 extra tokens per frame and equals the long-term read budget of MEMOIR. The five frames are selected by recency and quality score, which extends the memory-selection rule of SAM 3 itself to the frames outside its native window. The baseline also operates under the storage budget of MEMOIR: it stores up to 50 older full frames and trims them to the 30 most recent ones. This baseline changes the score of SAM 3 by -0.2, -0.1, +0.7, and 0.0 points, whereas MEMOIR improves it by +1.1, +0.4, +1.3, and +2.4 points (Table R2). In this setting, the two methods share the same read budget, and the difference lies in the history that this budget covers. Specifically, the five full frames cover a short span of recent history. In contrast, the graph of MEMOIR keeps the history beyond the recency window in a compressed and organized form, and the same read budget therefore reaches the regions of this history that are most relevant to the current frame. Section 3.1 formulates the problem in these terms, as a separation of the memory that is kept from the memory that is exposed to memory attention at each prediction.

The granularity of the memory nodes and the relations of the graph both contribute to this result. First, with every other part of the pipeline fixed, region-level nodes outperform frame-level nodes on all four datasets, with the largest margin on MOSEv2 (62.4 against 60.8 in J&Ḟ, Table 5). Second, with routing, consolidation, and budgets fixed, propagation over the graph contributes +0.2, +0.9, +1.5, and +1.3 points beyond retrieval by direct similarity (Table R4). This similarity baseline replaces the relevance propagated by Personalized PageRank (PPR) with the softmax of the direct relevance svs_v of Eq. (3) in every place where MEMOIR uses it, namely ranking, consolidation, the selection of object pointers and anchors, and utility, while routing and the budgets remain unchanged. This baseline is therefore the reasonable similarity baseline that the reviewer requested. In comparison, the w/o Graph variant of the original submission ranked the stored nodes by svs_v with routing and the budgets unchanged, and additionally turned off consolidation, which relies on the temporal edges of the graph. Furthermore, a confidence threshold on svs_v, calibrated on 20 SA-V training videos, did not improve over retrieval without a threshold, and the baseline in Table R4 therefore uses no threshold. The ablations that the reviewer cites lead to the same conclusion. In particular, the w/o Graph variant and the variant without the middle-term frames fall below SAM 3 on SA-V, LVOSv2, and MOSEv2, and without consolidation, most of the gain over SAM 3 disappears (Tables 6 and 9), which confirms that each of these components is necessary.

(to be continued...)

(continued...)

The settings of MEMOIR were not tuned on the reported benchmarks. Specifically, only τg\tau_g was calibrated, on 20 SA-V training videos, to yield about 15 regions per frame. Every other value was fixed before evaluation and is shared across all benchmarks: the memory sizes follow SAM 3, and the token budgets are whole multiples of the token count of one frame. Furthermore, when each hyperparameter that the reviewer named is varied, MEMOIR remains above SAM 3 on SA-V val, LVOSv2, and MOSEv2 at every setting (Table R3). Moreover, the existing memory designs do not improve SAM 3 consistently. To examine this, we adapted the public SAM 3 implementations of five memory variants (Alansari et al., 2025) to semi-supervised VOS and evaluated them under our per-object protocol, in which each object is tracked from its ground-truth mask and scored with the same toolkit as in the paper. These variants, SAM3Long and DAM4SAM-3 among them, all fall below the score of SAM 3 on SA-V val and MOSEv2, and MEMOIR is the only method in Table R1 that improves SAM 3 on all four datasets. On the disappearance and reappearance scores that the reviewer requested, the variants that raise the reappearance score do so at the expense of the disappearance score. In particular, every variant falls below the disappearance score of SAM 3 on every dataset, by up to 26.4 points, which indicates a shifted balance between reporting and suppressing the target. In contrast, MEMOIR improves the reappearance score on all four datasets while its disappearance score remains on par with or above that of SAM 3, and it is the only method in Table R1 that achieves both.

The gains of MEMOIR are concentrated on the cases that require long-term memory: the reappearance score improves by +2.4, +0.8, +3.7, and +4.0 points (Table 2), and by +8.9 points after absences longer than 8 s (Fig. 7), while the short benchmarks, on which recent memory suffices, remain on par with SAM 3. The cost of these gains is bounded: memory attention reads 42.8k to 47.1k tokens per frame, which is 1.3 times the 34.1k to 36.0k tokens of SAM 3 (Table 15), and the stored memory stays within a fixed budget regardless of the video length (Fig. 3). On tracking, to which MEMOIR transfers without modification, it improves the success score of SAM 3 on LaSOT and TNL2K and its average overlap on GOT-10k (Table 10), and the two variants that lead on LaSOText and LaSOT, DAM4SAM-3 and SAMITE-3, fall below SAM 3 on MOSEv2 (Table R1). The revised paper includes Tables R1 to R4 in Appendix H. We hope that these analyses address the question of the reviewer.

(to be continued...)

(continued...)

Table R1. SAM 3-based memory variants, run from the SAM 3 implementations of Alansari et al. (2025) under our protocol, in which each object is tracked from its ground-truth mask and scored with the same toolkit as MEMOIR. Each dataset reports the standard metric (J&F, and J&Ḟ on MOSEv2), together with the disappearance score D and the reappearance score R of Table 2. SAM 3 is our reproduction. Bold marks the highest value of each column.

SA-V valSA-V testLVOSv2MOSEv2
MethodJ&FDRJ&FDRJ&FDRJ&ḞDR
SAM 383.282.661.584.683.262.888.867.275.460.064.442.7
SAMURAI-378.963.163.581.569.563.788.452.277.856.945.348.1
SAM3Long82.781.061.785.082.764.588.864.676.059.361.544.5
DAM4SAM-380.477.161.982.781.764.189.164.176.857.257.344.5
SAMITE-381.469.064.982.470.665.588.855.277.954.141.245.7
HiM2SAM-380.370.262.981.371.462.385.850.875.850.038.043.0
MEMOIR (ours)84.384.263.985.084.063.690.167.179.162.465.246.7

Table R2. Equal-budget baseline, in which SAM 3 additionally reads five older full frames (25,920 extra tokens per frame). The metrics follow Table R1.

SA-V valSA-V testLVOSv2MOSEv2
MethodJ&FDRJ&FDRJ&FDRJ&ḞDR
SAM 383.282.661.584.683.262.888.867.275.460.064.442.7
SAM 3 + five older frames83.082.761.184.583.062.589.567.377.260.063.344.1
MEMOIR (ours)84.384.263.985.084.063.690.167.179.162.465.246.7

Table R3. Sensitivity to the hyperparameters that the reviewer named (J&F, and J&Ḟ on MOSEv2). One hyperparameter is varied at a time, and the default setting is in bold. SAM 3 reaches 83.2, 84.6, 88.8, and 60.0 on the four datasets.

ParameterSettingSA-V valSA-V testLVOSv2MOSEv2
τroute\tau_{route}0.784.184.789.462.0
0.884.385.090.162.4
0.984.484.589.662.3
KretK_{ret} / BTretB_T^{ret} (frames)284.784.489.661.8
484.385.090.162.4
684.084.690.061.5
Storage budgetx0.583.683.989.762.1
x184.385.090.162.4
x284.284.890.362.3
τg\tau_g1.1584.584.589.561.4
1.2584.385.090.162.4
1.3584.084.689.361.0

Table R4. Similarity-retrieval baseline. The similarity baseline replaces the PPR relevance with the softmax of the direct relevance svs_v wherever MEMOIR uses it, and keeps routing, consolidation, and budgets unchanged. The metrics follow Table R1.

SA-V valSA-V testLVOSv2MOSEv2
MethodJ&FDRJ&FDRJ&FDRJ&ḞDR
SAM 383.282.661.584.683.262.888.867.275.460.064.442.7
w/o Graph (original submission)82.080.561.183.179.162.187.060.177.059.761.343.5
Similarity baseline84.184.362.784.182.362.488.665.978.061.163.245.3
MEMOIR (PPR)84.384.263.985.084.063.690.167.179.162.465.246.7

(to be continued...)

Reviewer_Mgh2OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript presents a study toward video object segmentation in long videos. The proposed algorithmic treatment is through effective information association (in high level).

The core to the proposed method is using extra storage space to the benefit of performance gain. And such a solution naturally leads to a engineering design problem - how to build up the needed extra memory storage, what operations should be executed and when. The goal is to assist information association between distant frames. The choices made (i.e., read-in/out/matching) are inspired a lot by standard graph operations coupled with cost-effective budget management.

On experimental results, the proposed memeory-assisted method boosts the performance by 1-2% on average, which seems to be decent given existing methods typically being better than their precedant by the similar amount.

Critical strengths
  • Intereting research problem, sovled by careful engineering design.

  • Very detailed documentation of implemenation configurations.

  • Solid improvements over benchmarks.

Critical weaknesses
  • The writing is quite dense, making it difficult to connect related information scattered across different sections. Too many symbols and acronyms. Could be greatly simplified to assist the reading, as well as pouching on the key take-away message rather than getting distracted a lot.

  • Minor 1: The overall solution, to this reviewer, falls into the category of engineering solution design. Maybe engineering effort is exactly the needed recipe to this problem. The meaningful knowledge is less obvious to this reviwer. The long-distant information association low efficiency might comes from the training recipe of the base foundation model (i.e., SAM3). And the engineer solution presents one reasonable possibility without clearly revealing new knowledge / information on. A more interesting research question could be \textit{how historical information processing flow, i.e., primarily storing, filtering, and retrieving, can be consolidated into a learned representation that can flexibly associate information across long time horizons.}.

  • Minor 2: Due to the engineering nature, the work contains quite amount of heuristics, which is undertandable yet it is hard to gain deeper understanding on why it works, to what extent does it works and when it does not work. Present a few corner cases could be helpful.

Questions for authors
  • NA
3 comments

We appreciate the reviewer's time and effort spent on the review, and would like to respond as follows. The table and figure numbers in our responses follow the revised paper, and we kindly refer the reviewer to it. Unless otherwise noted, the scores in our responses are reported in the standard metric of each benchmark, which is J&F on SA-V and LVOSv2 and J&Ḟ on MOSEv2.

Comment 1. The density of the writing and the number of symbols.

Response. We sincerely thank the reviewer for the suggestion on the presentation. With the additional page that ICLR allows in the rebuttal phase, we have adjusted the layout of the main text to provide more space, and the changes are highlighted in color. The revised paper also includes a notation table (Table 13 in the appendix), which summarizes the symbols next to the hyperparameters of Table 12. The camera-ready version will further simplify the presentation along the suggestion of the reviewer. We hope that these revisions improve the readability of the paper.

Comment 2. The contribution of MEMOIR and the knowledge that it provides.

Response. We thank the reviewer for the comment, and respectfully clarify that the main contribution of this paper is a substantial advance in long-term memory for video object segmentation. In a long video, the target leaves the view, is occluded, and returns after a prolonged absence, and its earlier appearances are then the only evidence that identifies it. The native memory of SAM 3 reads only the conditioning frame and the six most recent retained frames (Proposition 1(a)), and this evidence is therefore no longer available to it. MEMOIR keeps this evidence accessible within a fixed budget and recovers the target after such absences. This advance lies beyond the level of engineering, since a simple assembly of existing memory components does not reproduce it. To directly address this question, we present an additional comparison with five SAM 3-based memory variants under our protocol in Table R1 below. All five variants fall below the score of SAM 3 on SA-V val and MOSEv2, and MEMOIR is the only method in Table R1 that improves SAM 3 on all four datasets. The design of MEMOIR is supported by both theory and experiments. The graph memory stays within a fixed node budget and keeps each validated region accessible until memory management removes it (Proposition 1(b)), and the cost of graph retrieval is bounded independently of the video length (Proposition 2). In the reappearance regime, MEMOIR improves SAM 3 by +2.4, +0.8, +3.7, and +4.0 points on SA-V val, SA-V test, LVOSv2, and MOSEv2 (Table 2), and by +8.9 points after absences longer than 8 s (Fig. 7). This paper further locates the bottleneck of long-term propagation in the unit of memory: with every other part of the pipeline fixed, region-level nodes outperform frame-level and grid-level nodes on all four datasets (Table 5). Moreover, this bottleneck lies in the memory rather than in the training of SAM 3. MEMOIR keeps every weight of the base model frozen and replaces only the memory that the model reads, and this replacement alone recovers targets that SAM 3 loses (IoU from 0.00 to 0.91 in Fig. 1, and J from 0.00 to 0.88 in Fig. 5). The same design also improves a different base model, SAM 2.1, by +3.1, +3.2, and +3.3 points on SA-V val, LVOSv2, and MOSEv2 (Table 11). The learned representation that the reviewer suggests is a promising direction, and the findings above identify the evidence that such a representation needs to preserve. We thank the reviewer again for the comment, and hope that this response clarifies the contribution of this paper.

(to be continued...)

(continued...)

Table R1. SAM 3-based memory variants, run from the SAM 3 implementations of Alansari et al. (2025) under our protocol, in which each object is tracked from its ground-truth mask and scored with the same toolkit as MEMOIR. Each dataset reports the standard metric (J&F, and J&Ḟ on MOSEv2), together with the disappearance score D and the reappearance score R of Table 2. SAM 3 is our reproduction. Bold marks the highest value of each column.

SA-V valSA-V testLVOSv2MOSEv2
MethodJ&FDRJ&FDRJ&FDRJ&ḞDR
SAM 383.282.661.584.683.262.888.867.275.460.064.442.7
SAMURAI-378.963.163.581.569.563.788.452.277.856.945.348.1
SAM3Long82.781.061.785.082.764.588.864.676.059.361.544.5
DAM4SAM-380.477.161.982.781.764.189.164.176.857.257.344.5
SAMITE-381.469.064.982.470.665.588.855.277.954.141.245.7
HiM2SAM-380.370.262.981.371.462.385.850.875.850.038.043.0
MEMOIR (ours)84.384.263.985.084.063.690.167.179.162.465.246.7

Comment 3. The principles behind the design of MEMOIR, its ablations, and its corner cases.

Response. We thank the reviewer for the comment, and respectfully clarify that the design of MEMOIR rests on theory and on principled mechanisms. As in our response to Comment 2, Propositions 1 and 2 establish that the history stays accessible within a fixed budget and that the cost of graph retrieval is bounded independently of the video length. Retrieval propagates relevance with Personalized PageRank over the memory graph (Section 3.5), and the graph organizes the regions of the history through temporal, spatial, and similarity edges (Section 3.4), which provides the memory with a systematic representation of the evidence. These components follow from theory and from established algorithms rather than from tuning on the reported benchmarks. The selection of the evidence to keep and to discard is also measured directly. On LVOSv2, the regions of the target are routed in 97% of the cases, and only 2.3% of the nodes evicted by the utility rule lie on the target, compared with 27.8% if the oldest nodes were evicted at the same eviction events (Appendix H.6). These mechanisms determine the result: without the graph, in which case consolidation is turned off as well, or without the middle-term frames, MEMOIR falls below SAM 3 on SA-V, LVOSv2, and MOSEv2, and without consolidation alone, most of the gain over SAM 3 disappears (Tables 6 and 9). Each main component is examined by an ablation, and we kindly refer the reviewer to Table 3 for the unit of memory, Table 4 for the retrieval stages and the memory management, Table 8 for the graph construction, and Table 9 for the memory layers. Furthermore, the same hyperparameters are shared across all benchmarks (Table 12). Regarding the corner cases, the gains of MEMOIR concentrate on long videos with prolonged absence, which is the regime that long-term memory targets. On short videos, the recent memory already contains the evidence that a prediction requires, and MEMOIR remains on par with SAM 3 (Appendix G). We thank the reviewer again for the comment, and hope that this response clarifies the principles behind MEMOIR.

Thanks for the rebuttal.

It becomes more clear to this reviewer what is missing from the paper and what causes the gap on understanding the contribution - a categorization, revelation and quantification of the shortcomings of existing method, i.e., SAM3, on this video understanding task. All the abovementioned issues with SAM3 model are real, observable, however, the significance and impact are not well captured and measured.

The correct way should be surfacing the issues, quantify the impact and present the solution. This work seems to focuses more on the later (understanding that most work tends to focuses on addressing problem without clearly defining problem first).

The response iterates the contribution in experimental efforts and outcomes, which is substantial, understandable and yet still missing depth in knowlege gain; mostly because the stage has not been properly set up - the problem and issues of SAM3 model on long-term video are not well defined and measured.

The reviewer's concerns remain unchanged at this stage. But, a decision on weak acceptance remains the same, for the efforts, performance improvements and engineering endeavor.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.