- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper adresses class-agnostic open world video segmentation (OVS) over long videos. No category names or initial objects are given. Objects must be discovered as they appear and keep the same ID when they leave the view and come back.
Contributions.
- Savvy, a training-free pipeline built on SAM (mask discovery) and SAM2.1 (propagation). It adds an object "lifecycle":
- mask discovery prompts SAM only in regions no current track explains;
- deferred admission follows new candidates for a short window before accepting, merging or discarding them;
- track consolidation checks returning tracks against their stored appearance, settles overlapping tracks, and absorbs parts into newly found wholes.
- OGA (Open-world Granularity-Agnostic evaluation), a metric that does not penalize predictions splitting a ground-truth object into parts (e.g., a sofa predicted as seat, backrest and cushions). Standard metrics like VPQ/STQ use 1:1 matching, so only one part can match the sofa and the others count as errors. OGA allows many predictions to match one ground-truth object (n:1). It also adds identity diagnostics (IP/IC) to detect fragmentation and ID switches.
Validation.
- Datasets: 142 ScanNet and 106 HM3D indoor videos (first 1,500 frames), plus VIPSeg.
- Baselines: DEVA+SAM and EntitySAM.
- Main result: Savvy outperforms both on long-horizon metrics under conventional and OGA evaluation (conventional VPQ∞: 18.44 vs 11.52 on ScanNet; 17.31 vs 6.58 on HM3D).
- Additional experiments: a cumulative ablation of the three lifecycle stages, and synthetic stress tests (dropout, flicker, sever, clutter, spatial fragmentation) intended to show that their introduced metric OGA reacts differently to different failure modes.
Critical strengths
Consistent long-horizon gains under standard metrics. Savvy outperforms DEVA+SAM and EntitySAM on long-window association on both indoor benchmarks, and under conventional 1:1 evaluation as well as OGA. Conventional VPQ∞ is 18.44 vs 11.52 on ScanNet and 17.31 vs 6.58 on HM3D; conventional STQ is 53.84 vs 46.66 and 40.35 vs 31.34. Fig. 3 shows the same pattern qualitatively: object IDs are preserved after the camera returns. This matters most to my assessment because the main empirical claim does not depend on the proposed metric. Its strength is limited, however, by the use of only two baselines and the absence of variance estimates.
Critical weaknesses
1. Section 3.2 (OGA) is very hard to follow. The section is dense with notation, but the concepts behind it are never explained in plain terms. Key terms are undefined: "virtual tube," "support state," "active identity set," "sever-induced fragment," and even VPQ's "tubes" and "windows." The symbol means two different things (an intersection in Eq. 6, a "discovery factor" in Eq. 9). There is no worked example for Eqs. 5–8. Section 3.1 ("provisional," "recent-eviction" masks) and the perturbations in Fig. 5 have the same problem: they are only defined in the appendices. Since the metric is half of the contribution, the paper cannot be verified or reused from the main text. (See Question 1.)
2. OGA does not penalize unnecessary splitting of objects. OGA gives full credit when a correct object is split into several predictions. This is intended for meaningful parts (e.g., a sofa's cushions and backrest), but it applies equally to an imprecise segmenter that cuts an object arbitrarily. For example, two predictions each covering half of a chair receive full OGA credit, whereas conventional 1:1 matching counts the chair as missed. The IP/IC diagnostics do flag the split, but they flag valid part decompositions in the same way, so they cannot tell the two cases apart. A reader therefore cannot know from OGA scores whether a method found meaningful parts or simply over-fragmented. (See Question 2.)
3. Missing related work on track management, and limited experimental rigor. Two of Savvy's three lifecycle mechanisms have close precedents in multi-object tracking that the paper does not discuss. DeepSORT (Wojke et al., 2017) keeps new tracks "tentative" until they are matched over several frames, and accepts a re-association only if it is consistent with the track's stored appearance descriptors. This is close to deferred admission and reappearance gating. The paper should position itself against this line of work and state what is new, such as mask-level retroactive relabeling or part–whole absorption. In addition, the system relies on about 20 hand-set thresholds, the ablation gains are modest and reported without variance, and the baseline settings for DEVA are not given. (See Question 3.)
Questions for authors
Q1. Section 3.2 needs to be rewritten in plain terms. Could you define each quantity in words and add a small worked example (e.g., one ground-truth object, two predictions, a few frames)? As written, I could not follow what Eqs. 5–8 compute.
Q2. Consider a chair predicted as two masks: one covers the left half and the other the right half. Neither half corresponds to a meaningful part of the chair. Under OGA, both masks are assigned to the chair and together receive full credit, exactly as a meaningful decomposition (e.g., seat and backrest) would. How should a reader tell from OGA scores whether a method found meaningful parts or simply cut objects arbitrarily?
Q3. Which settings did you use for DEVA+SAM? With its default settings, DEVA deletes an object after about 30 frames without a match, which handicaps it when the camera comes back to an object. How does it perform with a longer memory?