- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces S4VY: given a set of RGB observations, it processes them with a pretrained feed-forward visual geometry model, predicts cross-observation consistent masks through shared instance queries, and supports point, box, and natural-language specification of targets. The language module combines object-query matching, box prediction, Active Tree Search, and candidate-mask adjudication by a Critic. The authors evaluate prompt-free segmentation, prompted segmentation, and language grounding on static multi-view scenes and dynamic videos.
Critical strengths
-
The unified task design has practical value. The same instance representation serves both prompt-free discovery and interactive prompting and language grounding; the model also accepts sparse or unordered observations. This is more useful than solving only a single prompt type or a single video protocol.
-
The validation spans multiple scenes of differing difficulty. The authors report static, short-video, long-video, and driving-scene results, and additionally provide the complete ScanNet200 validation split and component ablations.
Critical weaknesses
-
The current experiments cannot yet confirm the paper's most central mechanistic contribution. The main novelty of this paper relative to existing methods (e.g., SeqFormer, PanSt3R, IGGT) lies in replacing per-frame association or feature clustering with global space-time queries, thereby achieving cross-observation consistent segmentation under a unified instance representation. However, the performance gap between S4VY and the baselines currently confounds at least three variables: a stronger visual geometry backbone (VGGT-Ω-1B), a much larger training corpus (about 6M frames, 33k scenes), and the instance-association scheme itself. The existing ablations focus on LoRA, positional encoding, and the language module, and do not separately compare "global space-time queries" against "per-frame prediction + cross-frame association/feature clustering" under a fixed backbone and fixed training data/budget. Consequently, it is difficult to disentangle whether the reported main gains come from the claimed mechanistic contribution or mainly from representation capacity and data scale.
-
The unified promptable interface does not show an advantage under clean box prompts. The paper takes as one of its core contributions a unified instance interface supporting point, box, and language prompts. Yet Appendix Table 8 shows that under clean box prompts SAM 2 clearly overtakes S4VY: 0.801 vs. 0.878 on ScanNet K=8 box, 0.779 vs. 0.827 on DAVIS K=4 box, and 0.796 vs. 0.828 on DAVIS K=16 box. S4VY's advantage is concentrated mainly in point prompts and corrupted-prompt settings. Given that box prompts are among the most common forms of user input for interactive segmentation, this result weakens the persuasiveness of the "unified promptable interface" as a primary selling point, and also suggests that the current training sampling or box encoding strategy for box prompts may be insufficient.
Questions for authors
-
To isolate the effect of the instance-association mechanism, please provide a controlled comparison under the same VGGT-Ω-1B backbone, the same training subset and optimization budget, and the same input observations: (a) your global space-time query decoder; (b) per-frame mask prediction followed by IoU-based association; (c) IGGT-style feature clustering. Please report a table for (i) the same 10 ScanNet validation clips used in Table 1 with T-mIoU/T-SR/J&F, and (ii) fixed 10-video subsets of DAVIS and LVOS with 4D-IoU/4D-SR/J&F. In addition, define and report an identity-consistency metric on re-appearance after occlusion, e.g., number of identity switches per re-appearance event and re-appearance ID accuracy, with the occlusion/re-entry event definition and association threshold stated. A small-scale but strictly controlled experiment would affect my judgment of the mechanistic novelty.
-
How does the paper explain S4VY trailing SAM 2 under clean box prompts? Does the paper consider increasing training sampling for box prompts, or improving box encoding or post-processing, so that box-prompt performance at least matches the model's advantages in point-prompt performance and robustness? Please provide the relevant analysis or additional experiments.
4 comments
Response to Reviewer 2Gr6 (2/2): Box Prompts
Weakness 2 and Question 2 (box prompts). The behavior under box prompts reflects a trade-off between two prompt designs: SAM 2 benefits from clean boxes given in multiple frames, whereas S4VY is far more robust to noisy prompts. With a single prompt, S4VY outperforms SAM 2 with box prompts on ScanNet (0.806 vs. 0.753 J&F in Tab. 8) and with point prompts on both ScanNet (0.793 vs. 0.649) and DAVIS (0.793 vs. 0.771). SAM 2 gains more from additional prompted frames (0.753 to 0.878 with box prompts on ScanNet), whereas the accuracy of S4VY is largely independent of their number.
S4VY is natively an exhaustive auto-segmentation model, whose learnable object queries establish a global instance partition that is consistent across space and time. Its prompt encoder samples image features in the prompted regions and adaptively pools them into one promptable object query (Sec. 3.2 and App. A.2), which the space-time decoder resolves against the learnable object queries to predict the prompted mask in all observations (App. A.1). SAM 2 is instead natively a promptable model. It does not support exhaustive auto-segmentation by itself and relies on a grid of point prompts (App. B.3), which leaves it well behind S4VY in the automatic setting of Tab. 1 (e.g., 0.588 vs. 0.781 J&F on ScanNet). Its prompt encoder encodes only the prompt coordinates as positional tokens, which the mask decoder decodes against the features of the prompted frame.
These designs behave differently with multi-frame prompts. In SAM 2, each prompt determines only the mask of its own frame and the other frames receive the identity through memory propagation, so additional prompted frames bring incremental gains. For the same reason, noisy prompts interfere severely with the propagation. S4VY adaptively pools the prompts from any number of frames into a single space-time consistent query (Sec. 3.2), which attends to the features of all observations simultaneously and is therefore inherently robust to noisy prompts (e.g., 0.665 vs. 0.253 J&F on ScanNet when 50% of the 8 box prompts are noisy, Tab. 8).
Regarding the training sampling and the box encoding, we provide two further analyses.
(1) Sampling: We ablate the ratio between box and point prompts. Following the setting of the space-time queries variant in Question 1, we freeze its segmenter and train only the prompt encoder for 3k steps, sampling each training prompt as a box with probability p_box and as a point otherwise (p_box = 0.4 in App. B.1). The evaluation follows the protocol of Tab. 8 on ScanNet with K = 1, 2, 4 and 8.
| p_box | Box K=1 | Box K=2 | Box K=4 | Box K=8 | Point K=1 | Point K=2 | Point K=4 | Point K=8 |
|---|---|---|---|---|---|---|---|---|
| 0.2 | 0.761 | 0.724 | 0.719 | 0.719 | 0.775 | 0.781 | 0.781 | 0.781 |
| 0.4 | 0.760 | 0.747 | 0.734 | 0.734 | 0.771 | 0.780 | 0.783 | 0.784 |
| 0.6 | 0.765 | 0.768 | 0.770 | 0.770 | 0.774 | 0.781 | 0.781 | 0.781 |
| 0.8 | 0.773 | 0.784 | 0.781 | 0.781 | 0.771 | 0.781 | 0.774 | 0.775 |
Increasing the box ratio brings box prompts on par with point prompts (0.781 vs. 0.775 with K = 8 at p_box = 0.8). Even with a higher box ratio, the accuracy remains stable as more frames are prompted, which confirms that this behavior follows from the single-query design above rather than from the training sampling.
(2) Box encoding strategy: We ablate the structure of the prompt encoder. All variants are trained on the frozen full-scale segmenter with the multi-view training scenes and evaluated with the same protocol (K = 4). We compare fixed pooling with fixed weights, the adaptive pooling of S4VY with learned weights, and a transformer encoder whose output queries attend to the sampled features through eight attention layers, using a single query, three learnable queries, or three queries initialized from the pooled features.
| Prompt encoder | ScanNet box | ScanNet point | DAVIS box | DAVIS point |
|---|---|---|---|---|
| Fixed pooling | 0.775 | 0.784 | 0.668 | 0.801 |
| Adaptive pooling (ours) | 0.777 | 0.794 | 0.762 | 0.804 |
| Transformer encoder, single query | 0.751 | 0.757 | 0.485 | 0.747 |
| Transformer encoder, three learnable queries | 0.561 | 0.609 | 0.481 | 0.642 |
| Transformer encoder, three pooled queries | 0.578 | 0.609 | 0.488 | 0.570 |
Adaptive pooling achieves the best accuracy in all four settings, so a different structure of the prompt encoder brings no further gain in our design.
At the system level, S4VY with a 4B grounder and a 2B critic further outperforms VLM+SAM 2, which prompts SAM 2 with boxes predicted by a 30.5B Qwen3-VL, on all four grounding benchmarks in Tab. 2 (3D mIoU 0.461 vs. 0.308 on ScanRefer and 0.293 vs. 0.170 on ScanNet++; J&F 0.755 vs. 0.709 on Ref-DAVIS and 0.573 vs. 0.483 on MeViS). In downstream use, where prompts are produced by a grounder rather than annotated, the robustness to imperfect prompts and the integration with the grounder therefore outweigh the gap under clean box prompts.
I have reviewed the rebuttal and additional experiments, which directly address my main concerns. The matched-budget controls and LVOS re-appearance results confirm that global space-time queries preserve long-term identity far better than per-frame association or clustering. While clean multi-frame box performance still trails SAM 2 (which should be noted in the main text), the prompt ablations and noisy-prompt robustness adequately contextualize this trade-off. As my primary concern regarding mechanistic attribution has been resolved, I am raising my rating from 3 to 4.
Reply to Reviewer 2Gr6
We thank the reviewer for the follow-up and for raising the rating. As suggested, we will note in the main text that SAM 2 remains more accurate under clean multi-frame box prompts and discuss this trade-off there. The controlled comparison, the re-appearance results and the prompt ablations will also be included in the revised paper.
Response to Reviewer 2Gr6 (1/2): Instance Decoding Mechanism
We sincerely thank the reviewer for the detailed reading and the constructive suggestions. We address the comments below.
Weakness 1 and Question 1 (instance decoding mechanism). The novelty of S4VY is a Segment Anything paradigm on feed-forward 4D visual geometry. Instead of the instance or panoptic segmentation paradigm of the methods mentioned by the reviewer, S4VY discovers every instance with a persistent identity in a single forward pass and allows any instance to be selected by point, box or language prompts. Language grounding further has its own design, comprising Active Tree Search, a dual-stream Grounder and a comparative Critic. Under the requested controlled setting, the space-time query decoder outperforms per-frame association and feature clustering on all three benchmarks and preserves identities after re-appearance far more reliably.
We compare four instance decoding mechanisms: (i) per-frame decoding, which links instances predicted independently in each frame by Hungarian matching on mask IoU; (ii) IGGT-style clustering, which replaces the query decoder with a per-pixel embedding head and clusters the embeddings lifted to 3D with HDBSCAN; (iii) IGGT4D-style clustering, which uses the same embedding head but clusters within each frame and merges the clusters across frames; and (iv) our space-time queries, each predicting one instance over all observations. Since full-scale training of every variant is prohibitively expensive, all variants share the same backbone (VGGT-Ω-1B with LoRA), training data (200 ScanNet++ scenes, 60 DAVIS and 120 LVOS videos) and 15k-step budget, and are evaluated on the same 10 ScanNet clips of Tab. 1 and the first ten validation videos of DAVIS and LVOS.
The space-time queries achieve the best results on all three benchmarks. Per-frame association remains competitive on the short and continuous DAVIS videos but fails to preserve identities across sparse ScanNet views and long LVOS sequences, and both clustering variants fall behind consistently.
For identity consistency after occlusion, we evaluate the same four variants on the 50 LVOS validation videos, which contain 96 re-appearance events; ScanNet clips are too short (six to eight observations) and DAVIS contains only 15 such events. A re-appearance event occurs when an object becomes visible again (ground-truth mask of at least 100 pixels) after at least two invisible frames. We report the following metrics instead of the number of identity switches, because a switch count does not penalize objects that are never re-detected. Re-ID Acc and Re-ID IoU are computed over all 96 events, so an object that is lost before or after the occlusion counts as an error.
The space-time queries achieve the highest identity consistency on all metrics, with a Re-ID Acc more than three times that of the other mechanisms under both thresholds.