acceptodds
Withdrawn from ICLR 2027

Toward Open-World Video Segmentation over Long Horizons

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long videos challenge open-world segmentation systems to continually discover objects and preserve their identities through disappearance and reappearance. Savvy, a zero-shot, semi-online, class-agnostic system for persistent object discovery and identity maintenance, and OGA, an evaluation suite that credits coherent part-level predictions when their granularity differs from the reference annotations. Savvy combines modular mask discovery, deferred admission based on accumulated evidence, and track consolidation to maintain an evolving object set. OGA allows multiple predictions to support one reference while preserving prediction IDs, pairing granularity-aware fidelity with diagnostics of identity bleeding and fragmented support. Across 142 ScanNet videos and 106 HM3D trajectories, evaluated using adaptively sampled prefixes capped at 1,500 source frames, Savvy outperforms DEVA+SAM and EntitySAM in VPQ∞, STQ, and AQ under both conventional and OGA evaluation. Conventional VPQ∞ reaches 18.44 versus DEVA+SAM’s 11.52 on ScanNet and 17.31 versus 6.58 on HM3D. VIPSeg comparisons reveal conventional one-to-one VPQ’s sensitivity to annotation granularity. Controlled identity-severing and flicker tests show that OGA detects temporal failures even when frame-level masks remain unchanged. Together, these results demonstrate complementary advances: Savvy improves long-horizon segmentation and association, and OGA distinguishes coherent part-level support from temporal identity failures.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-world video segmentation, long-horizon video understanding, object discovery, identity maintenance, granularity-aware evaluation, zero-shot segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9cSpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper adresses class-agnostic open world video segmentation (OVS) over long videos. No category names or initial objects are given. Objects must be discovered as they appear and keep the same ID when they leave the view and come back.

Contributions.

  1. Savvy, a training-free pipeline built on SAM (mask discovery) and SAM2.1 (propagation). It adds an object "lifecycle":
    • mask discovery prompts SAM only in regions no current track explains;
    • deferred admission follows new candidates for a short window before accepting, merging or discarding them;
    • track consolidation checks returning tracks against their stored appearance, settles overlapping tracks, and absorbs parts into newly found wholes.
  2. OGA (Open-world Granularity-Agnostic evaluation), a metric that does not penalize predictions splitting a ground-truth object into parts (e.g., a sofa predicted as seat, backrest and cushions). Standard metrics like VPQ/STQ use 1:1 matching, so only one part can match the sofa and the others count as errors. OGA allows many predictions to match one ground-truth object (n:1). It also adds identity diagnostics (IP/IC) to detect fragmentation and ID switches.

Validation.

  • Datasets: 142 ScanNet and 106 HM3D indoor videos (first 1,500 frames), plus VIPSeg.
  • Baselines: DEVA+SAM and EntitySAM.
  • Main result: Savvy outperforms both on long-horizon metrics under conventional and OGA evaluation (conventional VPQ∞: 18.44 vs 11.52 on ScanNet; 17.31 vs 6.58 on HM3D).
  • Additional experiments: a cumulative ablation of the three lifecycle stages, and synthetic stress tests (dropout, flicker, sever, clutter, spatial fragmentation) intended to show that their introduced metric OGA reacts differently to different failure modes.
Critical strengths

Consistent long-horizon gains under standard metrics. Savvy outperforms DEVA+SAM and EntitySAM on long-window association on both indoor benchmarks, and under conventional 1:1 evaluation as well as OGA. Conventional VPQ∞ is 18.44 vs 11.52 on ScanNet and 17.31 vs 6.58 on HM3D; conventional STQ is 53.84 vs 46.66 and 40.35 vs 31.34. Fig. 3 shows the same pattern qualitatively: object IDs are preserved after the camera returns. This matters most to my assessment because the main empirical claim does not depend on the proposed metric. Its strength is limited, however, by the use of only two baselines and the absence of variance estimates.

Critical weaknesses

1. Section 3.2 (OGA) is very hard to follow. The section is dense with notation, but the concepts behind it are never explained in plain terms. Key terms are undefined: "virtual tube," "support state," "active identity set," "sever-induced fragment," and even VPQ's "tubes" and "windows." The symbol DD means two different things (an intersection in Eq. 6, a "discovery factor" in Eq. 9). There is no worked example for Eqs. 5–8. Section 3.1 ("provisional," "recent-eviction" masks) and the perturbations in Fig. 5 have the same problem: they are only defined in the appendices. Since the metric is half of the contribution, the paper cannot be verified or reused from the main text. (See Question 1.)

2. OGA does not penalize unnecessary splitting of objects. OGA gives full credit when a correct object is split into several predictions. This is intended for meaningful parts (e.g., a sofa's cushions and backrest), but it applies equally to an imprecise segmenter that cuts an object arbitrarily. For example, two predictions each covering half of a chair receive full OGA credit, whereas conventional 1:1 matching counts the chair as missed. The IP/IC diagnostics do flag the split, but they flag valid part decompositions in the same way, so they cannot tell the two cases apart. A reader therefore cannot know from OGA scores whether a method found meaningful parts or simply over-fragmented. (See Question 2.)

3. Missing related work on track management, and limited experimental rigor. Two of Savvy's three lifecycle mechanisms have close precedents in multi-object tracking that the paper does not discuss. DeepSORT (Wojke et al., 2017) keeps new tracks "tentative" until they are matched over several frames, and accepts a re-association only if it is consistent with the track's stored appearance descriptors. This is close to deferred admission and reappearance gating. The paper should position itself against this line of work and state what is new, such as mask-level retroactive relabeling or part–whole absorption. In addition, the system relies on about 20 hand-set thresholds, the ablation gains are modest and reported without variance, and the baseline settings for DEVA are not given. (See Question 3.)

Questions for authors

Q1. Section 3.2 needs to be rewritten in plain terms. Could you define each quantity in words and add a small worked example (e.g., one ground-truth object, two predictions, a few frames)? As written, I could not follow what Eqs. 5–8 compute.

Q2. Consider a chair predicted as two masks: one covers the left half and the other the right half. Neither half corresponds to a meaningful part of the chair. Under OGA, both masks are assigned to the chair and together receive full credit, exactly as a meaningful decomposition (e.g., seat and backrest) would. How should a reader tell from OGA scores whether a method found meaningful parts or simply cut objects arbitrarily?

Q3. Which settings did you use for DEVA+SAM? With its default settings, DEVA deletes an object after about 30 frames without a match, which handicaps it when the camera comes back to an object. How does it perform with a longer memory?

Reviewer_zxnxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Savvy, a method for class-agnostic instance segmentation in long videos. The method combines SAM-based hierarchical mask proposals with SAM2 propagation into a training-free inference time mask discovery and tracking framework for long videos. In addition, the authors propose the metric extension OGA, namely open-world granularity-agnostic evaluation. OGA allows evaluation of predicted parts to count towards one object. The metric is addressing the mismatch between open-world predicted granularities and the fixed and human defined granularity of the ground truth annotations . The authors apply the granularity-aware extension to the established VPQ and STQ video panoptic segmentation metrics across the ScanNet, HM3D, and VIPSeg dataset comparing Savvy to the related works EntitySAM and DEVA+SAM.

Critical strengths

S1: Relevant Problem OGA addresses an important problem. In the era of foundation models and their zero-shot application across domains the alignment between model predictions and annotations can not be guaranteed. A method may predict reasonable object parts that do not match the annotation’s granularity. Accounting for this in evaluation is a relevant research direction.

Critical weaknesses

W1: Limited technical contribution The proposed method Savvy primarily combines pretrained foundation models at inference time with rules for accepting new objects, rejecting duplicates, and merging tracks. I appreciate the simplicity and acknowledge that the proposed changes lead to improved segmentation and tracking performance (cf. Table 2). However, the technical contribution appears limited, and the long-horizon comparison includes only two related methods, DEVA+SAM and EntitySAM.

W2: OGA metric extension requires a more comprehensive evaluation and differentiation from existing work The idea of allowing part-level predictions to explain one annotated object is useful, but related many-to-one matching already exists, as acknowledged in the paper (e.g., Appendix B.6). In general, the uniqueness and, more importantly, the benefit of the metric for applications or the assessment of foundation models remain unclear. Furthermore, the comparative experiments include only two related methods and the proposed method, although the metric could theoretically be applied to a much broader range of approaches. The paper should therefore clarify the benefit of extending the matching framework of Großkopf et al. to the temporal domain through its additional support-selection and identity-scoring rules, and demonstrate this benefit beyond the narrow set of evaluated methods.

Taken together, the two main contributions are not sufficiently strong to justify acceptance at a top-tier conference in their current form. However, extending the evaluation of OGA to a broader set of methods and a broader set of tasks, providing a reproducible benchmark and reference results as a foundation for future comparisons, would be a valuable direction.

Minor weakness: Presentation. The large number of abbreviations makes parts difficult to follow. Some appear before being explained (e.g., IP and IC in l. 244–245). MD, DA, and TC are introduced earlier, but brief reminders at the start of Section 3.1 would help. Citation and reference formatting also needs cleanup. The bibliography uses inconsistent names for the same venues (e.g., CVPR l. 507 vs. l. 559 vs. l. 628). Figure 1 is crowded and does not provide an intuitive introduction to the paper’s contributions. Given the highly visual nature of the task, qualitative video examples in the supplementary material would help reviewers to assess the task and method better.

Questions for authors

Q1: Reason for using a panoptic metric for class-agnostic video instance segmentation? Since the task is class-agnostic video instance segmentation, more reasoning on choosing video panoptic metrics to build up on with OGA instead of video instance segmentation metrics like AP/AR [A] would be interesting. On the other hand, what is the reason to not evaluate the metric beyond the class-agnostic setup (e.g. open-vocabulary segmentation).

Q2: What measurable benefit comes from the OGA temporal rules? Compared with the many-to-one matching discussed in Appendix B.6, what evidence isolates the benefit of temporal support selection and identity penalties? Have you compared OGA against a simpler union-based alternative on the same predictions, showing which important errors or distinctions the additional rules capture?

Q3: Does Savvy improve over a broader set of relevant methods? Are additional comparisons available against methods such as MASA, VideoSAM, or OV2Seg/InstFormer? Appendix B.8 explains adaptation requirements, but which of these actually prevent a meaningful comparison using their existing segmentation and tracking outputs?

[A] Yang el al., Video Instance Segmentation, In ICCV 2019

Reviewer_YiVZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Savvy, a framework for open-world video segmentation. It aims to discover new objects and maintain their identities over long videos. Savvy combines three components: Hierarchical Mask Discovery (HMD), Deferred Admission (DA), and Track Consolidation (TC). It uses pretrained SAM and SAM2 models. The authors also introduce OGA, an evaluation framework that allows multiple predicted segments to match a single ground-truth object. This approach is designed to handle differences in segmentation granularity.

Critical strengths
  1. Discovering new objects and maintaining their identities over long videos is challenging, especially when objects disappear and reappear from different viewpoints. The paper addresses these practical challenges.

  2. Separating object discovery, admission, and track consolidation makes the framework easy to understand. OGA is also useful because traditional one-to-one matching may penalize reasonable part-level predictions.

  3. Savvy outperforms DEVA+SAM and EntitySAM on ScanNet and HM3D, especially in long-term object association. The ablation studies and synthetic tests also provide useful insights into the different components.

Critical weaknesses
  1. Most components build on existing SAM/SAM2 models and use familiar tracking techniques, such as IoU matching, appearance similarity, and track merging. Although their combination works well, it is difficult to identify the main technical novelty beyond integrating these techniques.

  2. The main experiments only compare Savvy with DEVA+SAM and EntitySAM. Other relevant methods, such as VideoSAM and MASA, are not included. Savvy also runs at only 1.34 FPS, compared with 4.31 FPS for DEVA+SAM. These limitations make it difficult to fully assess its competitiveness and computational efficiency.

  3. Most experiments use indoor datasets with largely static objects. The evaluation also uses sampled frames from only the first 1,500 source frames. It remains unclear how well Savvy performs on longer videos or scenes with moving objects and frequent occlusions.

  4. Appendix H.1 shows that splitting predictions into more spatial fragments increases VPQ0, while STQ changes very little. This suggests that OGA may reward unnecessary fragmentation. In Table 17, some IP and IC scores also change with the severing threshold, although these metrics do not directly depend on it. Both results need further clarification.

Questions for authors
  1. Could the authors clarify the main technical novelty of Savvy compared with existing tracking pipelines? Which components go beyond standard IoU matching, appearance verification, and track merging?

  2. Why are VideoSAM and MASA not included in the main comparisons? Could the authors provide additional results or compare these methods under similar computational budgets?

  3. How does Savvy perform on longer videos with independently moving objects? Could the authors provide quantitative results on more dynamic scenes or longer sequences?

  4. How does OGA distinguish meaningful part-level segmentation from arbitrary spatial fragmentation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.