acceptodds
Under review as a conference paper at ICLR 2027

S4VY: Segment Anything in Feed-Forward 4D Visual Geometry

Abstract

Accurate instance segmentation in dynamic scenes is important for downstream applications such as robotics and autonomous driving. Existing Segment Any- thing models operate primarily on 2D image or video masks and preserve iden- tity through sequential memory, while promptable 4D instance segmentation built upon feed-forward visual geometry remains underexplored. We introduce S4VY, a Segment Anything model built on feed-forward 4D visual geometry. From a set of RGB observations, S4VY transforms shared visual-geometric features into an exhaustive set of class-agnostic 4D instance masks through a space-time query decoder, with each persistent object query binding one entity across all observa- tions. This representation supports prompt-independent segmentation as well as point- and box-conditioned selection, without requiring a seed mask or temporal ordering. We further develop an agentic harness for natural-language grounding in the large observation space of a 4D scene. Active tree search identifies rele- vant frames without scanning every fixed window; a dual-stream grounder com- bines fine-grained VLM visual priors with geometry-consistent instance features through complementary bounding-box prediction and object-query matching; and an independent critic selects the final 4D instance mask from their predictions. Extensive experiments demonstrate state-of-the-art 4D instance segmentation and strong language-guided grounding performance under a unified evaluation span- ning static and dynamic scenes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
4D Instance Segmentation, Segment Anything, Visual Grounding, Feed-Forward Visual Geometry

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_2Gr6OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces S4VY: given a set of RGB observations, it processes them with a pretrained feed-forward visual geometry model, predicts cross-observation consistent masks through shared instance queries, and supports point, box, and natural-language specification of targets. The language module combines object-query matching, box prediction, Active Tree Search, and candidate-mask adjudication by a Critic. The authors evaluate prompt-free segmentation, prompted segmentation, and language grounding on static multi-view scenes and dynamic videos.

Critical strengths
  1. The unified task design has practical value. The same instance representation serves both prompt-free discovery and interactive prompting and language grounding; the model also accepts sparse or unordered observations. This is more useful than solving only a single prompt type or a single video protocol.

  2. The validation spans multiple scenes of differing difficulty. The authors report static, short-video, long-video, and driving-scene results, and additionally provide the complete ScanNet200 validation split and component ablations.

Critical weaknesses
  1. The current experiments cannot yet confirm the paper's most central mechanistic contribution. The main novelty of this paper relative to existing methods (e.g., SeqFormer, PanSt3R, IGGT) lies in replacing per-frame association or feature clustering with global space-time queries, thereby achieving cross-observation consistent segmentation under a unified instance representation. However, the performance gap between S4VY and the baselines currently confounds at least three variables: a stronger visual geometry backbone (VGGT-Ω-1B), a much larger training corpus (about 6M frames, 33k scenes), and the instance-association scheme itself. The existing ablations focus on LoRA, positional encoding, and the language module, and do not separately compare "global space-time queries" against "per-frame prediction + cross-frame association/feature clustering" under a fixed backbone and fixed training data/budget. Consequently, it is difficult to disentangle whether the reported main gains come from the claimed mechanistic contribution or mainly from representation capacity and data scale.

  2. The unified promptable interface does not show an advantage under clean box prompts. The paper takes as one of its core contributions a unified instance interface supporting point, box, and language prompts. Yet Appendix Table 8 shows that under clean box prompts SAM 2 clearly overtakes S4VY: 0.801 vs. 0.878 on ScanNet K=8 box, 0.779 vs. 0.827 on DAVIS K=4 box, and 0.796 vs. 0.828 on DAVIS K=16 box. S4VY's advantage is concentrated mainly in point prompts and corrupted-prompt settings. Given that box prompts are among the most common forms of user input for interactive segmentation, this result weakens the persuasiveness of the "unified promptable interface" as a primary selling point, and also suggests that the current training sampling or box encoding strategy for box prompts may be insufficient.

Questions for authors
  1. To isolate the effect of the instance-association mechanism, please provide a controlled comparison under the same VGGT-Ω-1B backbone, the same training subset and optimization budget, and the same input observations: (a) your global space-time query decoder; (b) per-frame mask prediction followed by IoU-based association; (c) IGGT-style feature clustering. Please report a table for (i) the same 10 ScanNet validation clips used in Table 1 with T-mIoU/T-SR/J&F, and (ii) fixed 10-video subsets of DAVIS and LVOS with 4D-IoU/4D-SR/J&F. In addition, define and report an identity-consistency metric on re-appearance after occlusion, e.g., number of identity switches per re-appearance event and re-appearance ID accuracy, with the occlusion/re-entry event definition and association threshold stated. A small-scale but strictly controlled experiment would affect my judgment of the mechanistic novelty.

  2. How does the paper explain S4VY trailing SAM 2 under clean box prompts? Does the paper consider increasing training sampling for box prompts, or improving box encoding or post-processing, so that box-prompt performance at least matches the model's advantages in point-prompt performance and robustness? Please provide the relevant analysis or additional experiments.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2Gr6 (1/2): Instance Decoding Mechanism

We sincerely thank the reviewer for the detailed reading and the constructive suggestions. We address the comments below.

Weakness 1 and Question 1 (instance decoding mechanism). The novelty of S4VY is a Segment Anything paradigm on feed-forward 4D visual geometry. Instead of the instance or panoptic segmentation paradigm of the methods mentioned by the reviewer, S4VY discovers every instance with a persistent identity in a single forward pass and allows any instance to be selected by point, box or language prompts. Language grounding further has its own design, comprising Active Tree Search, a dual-stream Grounder and a comparative Critic. Under the requested controlled setting, the space-time query decoder outperforms per-frame association and feature clustering on all three benchmarks and preserves identities after re-appearance far more reliably.

We compare four instance decoding mechanisms: (i) per-frame decoding, which links instances predicted independently in each frame by Hungarian matching on mask IoU; (ii) IGGT-style clustering, which replaces the query decoder with a per-pixel embedding head and clusters the embeddings lifted to 3D with HDBSCAN; (iii) IGGT4D-style clustering, which uses the same embedding head but clusters within each frame and merges the clusters across frames; and (iv) our space-time queries, each predicting one instance over all observations. Since full-scale training of every variant is prohibitively expensive, all variants share the same backbone (VGGT-Ω-1B with LoRA), training data (200 ScanNet++ scenes, 60 DAVIS and 120 LVOS videos) and 15k-step budget, and are evaluated on the same 10 ScanNet clips of Tab. 1 and the first ten validation videos of DAVIS and LVOS.

DecodingScanNet (T-mIoU / T-SR / J&F)DAVIS (4D-IoU / 4D-SR / J&F)LVOS (4D-IoU / 4D-SR / J&F)
Per-frame decoding + IoU association0.328 / 0.013 / 0.3210.725 / 0.900 / 0.8300.110 / 0.000 / 0.111
IGGT-style clustering0.554 / 0.508 / 0.4960.450 / 0.600 / 0.5680.163 / 0.000 / 0.213
IGGT4D-style clustering0.701 / 0.625 / 0.6300.687 / 0.950 / 0.7980.106 / 0.000 / 0.153
Space-time queries (ours)0.788 / 0.817 / 0.7600.745 / 1.000 / 0.8590.444 / 0.333 / 0.510

The space-time queries achieve the best results on all three benchmarks. Per-frame association remains competitive on the short and continuous DAVIS videos but fails to preserve identities across sparse ScanNet views and long LVOS sequences, and both clustering variants fall behind consistently.

For identity consistency after occlusion, we evaluate the same four variants on the 50 LVOS validation videos, which contain 96 re-appearance events; ScanNet clips are too short (six to eight observations) and DAVIS contains only 15 such events. A re-appearance event occurs when an object becomes visible again (ground-truth mask of at least 100 pixels) after at least two invisible frames. We report the following metrics instead of the number of identity switches, because a switch count does not penalize objects that are never re-detected. Re-ID Acc and Re-ID IoU are computed over all 96 events, so an object that is lost before or after the occlusion counts as an error.

  • Re-appearance ID accuracy (Re-ID Acc): in each frame, an object is associated with the predicted instance that has the highest IoU with its ground-truth mask if this IoU exceeds the association threshold, for which we report both 0.5 and 0.25. Re-ID Acc is the fraction of events in which the first identity associated within five visible frames after re-appearance equals the most frequent identity over the last five visible frames before disappearance.
  • Re-ID IoU: the IoU between the ground truth and the masks of the identity that best overlaps the object over the last five visible frames before disappearance, accumulated over the first five visible frames after re-appearance.
  • J&Ḟ_r (MOSEv2; Ding et al., 2025): the mean of region similarity and adaptive boundary accuracy over the re-appearance clips of each object, computed with the official definition and evaluation code, where each object is evaluated on the prediction matched to it over the complete video.
  • IDF1 (Ristani et al., 2016): computed over the complete videos with an association threshold of 0.5.
DecodingRe-ID Acc (> 0.5)Re-ID Acc (> 0.25)Re-ID IoUJ&Ḟ_rIDF1
Per-frame decoding + IoU association0.0520.0520.0250.0890.148
IGGT-style clustering0.0620.1460.1190.1460.222
IGGT4D-style clustering0.0830.1250.0890.1570.254
Space-time queries (ours)0.3440.4790.2670.4230.546

The space-time queries achieve the highest identity consistency on all metrics, with a Re-ID Acc more than three times that of the other mechanisms under both thresholds.

AuthorsOpenReview ↗

Response to Reviewer 2Gr6 (2/2): Box Prompts

Weakness 2 and Question 2 (box prompts). The behavior under box prompts reflects a trade-off between two prompt designs: SAM 2 benefits from clean boxes given in multiple frames, whereas S4VY is far more robust to noisy prompts. With a single prompt, S4VY outperforms SAM 2 with box prompts on ScanNet (0.806 vs. 0.753 J&F in Tab. 8) and with point prompts on both ScanNet (0.793 vs. 0.649) and DAVIS (0.793 vs. 0.771). SAM 2 gains more from additional prompted frames (0.753 to 0.878 with box prompts on ScanNet), whereas the accuracy of S4VY is largely independent of their number.

S4VY is natively an exhaustive auto-segmentation model, whose learnable object queries establish a global instance partition that is consistent across space and time. Its prompt encoder samples image features in the prompted regions and adaptively pools them into one promptable object query (Sec. 3.2 and App. A.2), which the space-time decoder resolves against the learnable object queries to predict the prompted mask in all observations (App. A.1). SAM 2 is instead natively a promptable model. It does not support exhaustive auto-segmentation by itself and relies on a grid of point prompts (App. B.3), which leaves it well behind S4VY in the automatic setting of Tab. 1 (e.g., 0.588 vs. 0.781 J&F on ScanNet). Its prompt encoder encodes only the prompt coordinates as positional tokens, which the mask decoder decodes against the features of the prompted frame.

These designs behave differently with multi-frame prompts. In SAM 2, each prompt determines only the mask of its own frame and the other frames receive the identity through memory propagation, so additional prompted frames bring incremental gains. For the same reason, noisy prompts interfere severely with the propagation. S4VY adaptively pools the prompts from any number of frames into a single space-time consistent query (Sec. 3.2), which attends to the features of all observations simultaneously and is therefore inherently robust to noisy prompts (e.g., 0.665 vs. 0.253 J&F on ScanNet when 50% of the 8 box prompts are noisy, Tab. 8).

Regarding the training sampling and the box encoding, we provide two further analyses.

(1) Sampling: We ablate the ratio between box and point prompts. Following the setting of the space-time queries variant in Question 1, we freeze its segmenter and train only the prompt encoder for 3k steps, sampling each training prompt as a box with probability p_box and as a point otherwise (p_box = 0.4 in App. B.1). The evaluation follows the protocol of Tab. 8 on ScanNet with K = 1, 2, 4 and 8.

p_boxBox K=1Box K=2Box K=4Box K=8Point K=1Point K=2Point K=4Point K=8
0.20.7610.7240.7190.7190.7750.7810.7810.781
0.40.7600.7470.7340.7340.7710.7800.7830.784
0.60.7650.7680.7700.7700.7740.7810.7810.781
0.80.7730.7840.7810.7810.7710.7810.7740.775

Increasing the box ratio brings box prompts on par with point prompts (0.781 vs. 0.775 with K = 8 at p_box = 0.8). Even with a higher box ratio, the accuracy remains stable as more frames are prompted, which confirms that this behavior follows from the single-query design above rather than from the training sampling.

(2) Box encoding strategy: We ablate the structure of the prompt encoder. All variants are trained on the frozen full-scale segmenter with the multi-view training scenes and evaluated with the same protocol (K = 4). We compare fixed pooling with fixed weights, the adaptive pooling of S4VY with learned weights, and a transformer encoder whose output queries attend to the sampled features through eight attention layers, using a single query, three learnable queries, or three queries initialized from the pooled features.

Prompt encoderScanNet boxScanNet pointDAVIS boxDAVIS point
Fixed pooling0.7750.7840.6680.801
Adaptive pooling (ours)0.7770.7940.7620.804
Transformer encoder, single query0.7510.7570.4850.747
Transformer encoder, three learnable queries0.5610.6090.4810.642
Transformer encoder, three pooled queries0.5780.6090.4880.570

Adaptive pooling achieves the best accuracy in all four settings, so a different structure of the prompt encoder brings no further gain in our design.

At the system level, S4VY with a 4B grounder and a 2B critic further outperforms VLM+SAM 2, which prompts SAM 2 with boxes predicted by a 30.5B Qwen3-VL, on all four grounding benchmarks in Tab. 2 (3D mIoU 0.461 vs. 0.308 on ScanRefer and 0.293 vs. 0.170 on ScanNet++; J&F 0.755 vs. 0.709 on Ref-DAVIS and 0.573 vs. 0.483 on MeViS). In downstream use, where prompts are produced by a grounder rather than annotated, the robustness to imperfect prompts and the integration with the grounder therefore outweigh the gap under clean box prompts.

I have reviewed the rebuttal and additional experiments, which directly address my main concerns. The matched-budget controls and LVOS re-appearance results confirm that global space-time queries preserve long-term identity far better than per-frame association or clustering. While clean multi-frame box performance still trails SAM 2 (which should be noted in the main text), the prompt ablations and noisy-prompt robustness adequately contextualize this trade-off. As my primary concern regarding mechanistic attribution has been resolved, I am raising my rating from 3 to 4.

AuthorsOpenReview ↗

Reply to Reviewer 2Gr6

We thank the reviewer for the follow-up and for raising the rating. As suggested, we will note in the main text that SAM 2 remains more accurate under clean multi-frame box prompts and discuss this trade-off there. The controlled comparison, the re-appearance results and the prompt ablations will also be included in the revised paper.

Reviewer_m5AUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

S4VY combines a VGGT-Ω geometry backbone with a Mask2Former-style decoder whose shared object queries predict persistent masks across RGB observations. A separately trained prompt encoder supports points and boxes. Language grounding uses a geometry-conditioned Qwen3-VL grounder with query and box streams, active tree search over observations, and a second VLM that selects between the resulting masks. Training consolidates approximately six million frames from 19 sources with teacher-generated pseudo-masklets. Experiments cover static and dynamic segmentation, referring expressions, prompt corruption, sparse/shuffled inputs, and additional transfer datasets in the appendix.

Critical strengths
  1. A useful unified representation with broad empirical support. Persistent queries provide a coherent interface for prompt-free, point/box and language-conditioned segmentation. Table 1 reports strong static and dynamic results, including 0.803 DAVIS J&F, 0.602 LVOS J&F and 0.795 VIPSeg STQ. Importantly, the appendix extends the small main-text ScanNet evaluation to all 312 ScanNet200 validation scenes, obtaining 0.517 T-mIoU versus 0.488 for IGGT4D, and includes zero-shot HOI4D/VISOR results (Tables 5–6). The evidence is substantially broader than the ten-clip main-text subset.
  2. The grounding components have meaningful supporting ablations. On ScanRefer, adding register-token input raises [email protected] from 20.7 to 37.7, and the critic improves 3D mask mIoU from the stronger individual stream’s 0.434 to 0.461 (Table 3). Table 13 extends the stream/critic decomposition to the other grounding benchmarks. Table 8 also tests identical point/box prompts and corrupted prompts across methods, revealing both S4VY’s robustness advantages and its limitations with clean multi-view boxes. These results substantiate the value of the integrated interfaces.
Critical weaknesses
  1. The gains are not cleanly attributed to the proposed segmentation design. S4VY adapts a strong 1B geometry backbone and trains on approximately six million frames, whereas Appendix B.3 evaluates competing systems using their released checkpoints. Table 13 varies LoRA and frame-position embeddings, but does not isolate the persistent-query decoder or pseudo-label curriculum against a simpler decoder trained with the same backbone, data and schedule. The experimental results demonstrate a stronger complete system. They provide less evidence that the proposed architectural choices explain the improvements. A limited matched-training ablation would strengthen the scientific contribution without requiring retraining every baseline.
  2. Evidence for sparse/unordered language search and complete grounding efficiency is incomplete. Algorithm 1 expands ranges using neighboring input indices and prunes non-root ranges without positive sampled boxes. Arbitrary ordering can disperse target observations and weaken this search strategy. The shuffle results in Table 7 concern segmentation, not the complete ATS grounding pipeline. Table 3(c) reports fewer windows but also query-stream drops of 1.7 and 1.9 points on the static benchmarks, so preservation is not universal. Tables 11–12 report segmentation runtime, omitting the grounder and two-order critic evaluation. These distinctions limit the claimed reach of the agentic component, although they do not negate the segmentation results.
Questions for authors
  1. Contribution attribution: Is there a comparison using the same VGGT-Ω checkpoint, training mixture, and schedule with a simpler shared-query decoder or without pseudo-mask densification/curriculum? Please identify which improvement survives this control. One focused ablation would be more useful than additional unrelated benchmarks.
  2. Search robustness: On the same observation sets, how do full-pipeline grounding accuracy, target-discovery failure rate and visited-window count change after shuffling, especially for low-presence targets? Please distinguish missing the target entirely from merely failing to retrieve every target-visible frame, and specify the maximum depth and root fallback partition.
  3. Efficiency and metric scope: What is the end-to-end latency per expression including geometry, ATS, both mask candidates and the critic? Also, please distinguish the model-lifted 4D overlap in Appendix B.4 from evaluation against independently measured 4D geometry, and specify the voxel scale. These clarifications would delimit the efficiency and geometric-accuracy claims.
3 comments
AuthorsOpenReview ↗

Response to Reviewer m5AU (1/3): Contribution Attribution

We sincerely thank the reviewer for the detailed reading and the constructive suggestions. We address the comments below.

Weakness 1 and Question 1 (contribution attribution). Both the space-time query decoder and the pseudo supervision retain their improvements under matched training, as shown by the two controls below.

(1) Architecture. We compare four instance decoding mechanisms: (i) per-frame decoding, which links instances predicted independently in each frame by Hungarian matching on mask IoU; (ii) IGGT-style clustering, which replaces the query decoder with a per-pixel embedding head and clusters the embeddings lifted to 3D with HDBSCAN; (iii) IGGT4D-style clustering, which uses the same embedding head but clusters within each frame and merges the clusters across frames; and (iv) our space-time queries, each predicting one instance over all observations. Since full-scale training of every variant is prohibitively expensive, all variants share the same backbone (VGGT-Ω-1B with LoRA), training data (200 ScanNet++ scenes, 60 DAVIS and 120 LVOS videos) and 15k-step budget, and are evaluated on the same 10 ScanNet clips of Tab. 1 and the first ten validation videos of DAVIS and LVOS.

DecodingScanNet (T-mIoU / T-SR / J&F)DAVIS (4D-IoU / 4D-SR / J&F)LVOS (4D-IoU / 4D-SR / J&F)
Per-frame decoding + IoU association0.328 / 0.013 / 0.3210.725 / 0.900 / 0.8300.110 / 0.000 / 0.111
IGGT-style clustering0.554 / 0.508 / 0.4960.450 / 0.600 / 0.5680.163 / 0.000 / 0.213
IGGT4D-style clustering0.701 / 0.625 / 0.6300.687 / 0.950 / 0.7980.106 / 0.000 / 0.153
Space-time queries (ours)0.788 / 0.817 / 0.7600.745 / 1.000 / 0.8590.444 / 0.333 / 0.510

The space-time queries achieve the best results on all three benchmarks. Per-frame association remains competitive on the short and continuous DAVIS videos but fails to preserve identities across sparse ScanNet views and long LVOS sequences, and both clustering variants fall behind consistently.

(2) Pseudo-label self-distillation. To isolate the effect of self-distillation, we keep the model configuration, training budget and evaluation of (1) unchanged and only add 800 clips with pseudo annotations: 200 from YouTube-VOS, 200 from MeViS videos and 400 from DL3DV scans. These clips are supervised by pseudo-masklets and sampled with the curriculum of App. B.1.

Training dataScanNet (J&F)DAVIS (4D-IoU)LVOS (4D-IoU)
Annotated data only (380 scenes and videos)0.7600.7450.444
+ 800 pseudo-labelled clips0.7750.7630.500

Pseudo supervision improves all three benchmarks, with the largest gain on the long LVOS videos (+0.056 4D-IoU). Such pseudo supervision is also necessary for using partially annotated videos at all: exhaustive set prediction supervises every unmatched query as no-object, whereas these videos label only a few instances per frame, so using their annotations directly would supervise the remaining instances as background and suppress their discovery.

AuthorsOpenReview ↗

Response to Reviewer m5AU (2/3): Search Robustness

Weakness 2 and Question 2 (search robustness). We would like to first clarify that, for language grounding, an arbitrary observation order is not always meaningful: many referring expressions describe temporal changes or motion, such as "The rabbit that changed its location" (Fig. 6), and their meaning depends on the order of the observations. Therefore, although our segmenter imposes no constraint on the observation order, ATS is not specifically designed for unordered inputs. As stated in Sec. 4.2, it is introduced for sparse or non-consecutive observations, for which temporal prior information is weakened, and explores the observation set adaptively based on predicted target presence while following the supplied order (App. A.4). We will state this scope explicitly in Sec. 4.2.

We therefore evaluate shuffling on expressions whose meaning does not depend on the observation order (e.g. appearance). From ScanRefer and ScanNet++, we remove expressions containing viewer-centric or ordinal terms that depend on the camera trajectory (e.g., left, right, front, behind, closest, second, "as you enter"), keep those describing appearance, relations between objects, room context or function, and sample 200 expressions per benchmark with a fixed seed. Each expression is processed by the complete pipeline on the same observation set, either in the original order or under a fixed random permutation. Results are reported as ordered → shuffled.

ExpressionsCount3D mIoUTarget-discovery failureVisited windows
ScanRefer, all2000.486 → 0.4676.0% → 4.0%6.3 → 5.2
ScanRefer, low presence (< 15% of observations)440.364 → 0.3589.1% → 9.1%6.0 → 5.3
ScanNet++, all2000.295 → 0.28113.0% → 13.0%6.1 → 5.8
ScanNet++, low presence (< 15% of observations)880.180 → 0.18422.7% → 27.3%5.9 → 5.9

Shuffling leaves the full-pipeline accuracy and the number of visited windows nearly unchanged, including the accuracy on low-presence targets. The loss caused by shuffling mainly lies in a lower recall of frame localization, which drops from 0.569 to 0.370 on ScanRefer and from 0.353 to 0.279 on ScanNet++. Nevertheless, accurate spatial grounding in the selected observations is sufficient for the segmenter to produce correct masklets over all observations, so the overall accuracy is only marginally affected. In contrast, a target that is missed entirely cannot be recovered by the segmenter. This case is measured by the target-discovery failure rate, i.e., the fraction of expressions for which none of the selected observations contains the target, which shuffling does not increase overall. A modest increase appears only on the low-presence ScanNet++ subset (22.7% → 27.3%, four additional targets out of 88). Eight ScanNet++ targets that are not visible in any observation are excluded from these frame-level statistics.

Under the same observation sets and permutations, we further compare with the agentic frame localization baselines and the trivial baselines of Tab. 10 (Uniform top-8 and Uniform top-16) using frame precision, F1 and frame-set IoU.

MethodScanRefer precisionScanRefer F1ScanRefer IoUScanNet++ precisionScanNet++ F1ScanNet++ IoU
Uniform top-80.284 → 0.2940.164 → 0.1700.093 → 0.0980.195 → 0.1870.104 → 0.0950.056 → 0.052
Uniform top-160.288 → 0.2950.246 → 0.2530.149 → 0.1550.192 → 0.1970.157 → 0.1600.087 → 0.091
VideoAgent0.311 → 0.3470.157 → 0.1750.090 → 0.1010.231 → 0.2160.101 → 0.0990.056 → 0.055
VideoTree0.289 → 0.2810.343 → 0.3360.225 → 0.2210.200 → 0.1930.237 → 0.2310.141 → 0.137
LensWalk0.363 → 0.2600.202 → 0.1340.141 → 0.0880.263 → 0.1980.152 → 0.0960.098 → 0.059
Vidi 1.50.638 → 0.4460.205 → 0.1310.133 → 0.0800.449 → 0.3280.169 → 0.1320.104 → 0.078
Ours0.711 → 0.6660.605 → 0.4340.489 → 0.3080.564 → 0.5480.389 → 0.3310.269 → 0.219

Shuffling degrades frame localization for the methods that outperform the trivial baselines, whereas VideoAgent, LensWalk and Vidi 1.5 fall below Uniform top-16 in F1 and frame-set IoU in most settings. Our method significantly outperforms all other methods both before and after shuffling, and, as analyzed above, the decrease in frame localization has little effect on its final accuracy.

Finally, ATS uses a maximum depth of D = 4 and N_w = 8 observations per window. When no observation in the root window is predicted positive, the full range is uniformly partitioned into eight sub-ranges, all of which are searched.

AuthorsOpenReview ↗

Response to Reviewer m5AU (3/3): Efficiency and Metric Scope

Question 3 (efficiency and metric scope). We measure latency on 58 ScanRefer expressions from 11 scenes, running our method on one A100-80GB GPU with the same standard implementation of the grounder and critic. The end-to-end latency of the complete pipeline, covering geometry, ATS, both mask candidates and the critic, is 44.6 s per expression at the median and 48.5 s on average, with the components listed below.

ComponentRunsMean time per run (s)Mean time per expression (s)
Geometry and segmentation15.965.96
ATS: grounder CoT13.883.88
ATS: window search5.26 visited windows6.5734.5
Mask candidates13.513.51
Critic2 candidate orders0.320.64
Total48.5

We further compare the grounding time, i.e., ATS and the two mask candidates, with the agentic frame localization baselines under the setting of Tab. 10.

MethodOutputMedian (s)Mean (s)
VideoAgentFrame IDs107.0107.4
VideoTreeFrame IDs75.082.0
LensWalkFrame IDs120.0129.2
Ours with sliding windowMasks116.8108.0
OursMasks35.341.9

Our method thus achieves a favorable trade-off between speed and accuracy, as it is faster than all agentic baselines and the sliding window while achieving higher accuracy (3D mIoU of 0.495 with ATS vs. 0.471 with the sliding window).

Regarding the metric scope, the 4D-IoU in App. B.4 is a segmentation metric computed on lifted 4D geometry. All methods share the same lifting procedure, so the comparison is fair, and it does not involve any comparison of geometric reconstruction accuracy. We lift with VGGT-Ω because DAVIS and LVOS provide no ground-truth geometry, and VGGT-Ω is the overall best-performing reconstruction method in our evaluation on dynamic videos below (9 dynamic sequences from Sintel, Bonn and TUM-dynamics).

MethodSintel ATEBonn ATETUM-dynamics ATETUM-dynamics AbsRel
MegaSaM0.1330.0170.0100.080
PAGE-4D0.1960.0220.0120.052
OpenD4RT0.2510.0390.0800.191
MapAnything0.3390.0310.0230.163
DA30.1650.0100.0100.105
π³0.1480.0090.0070.065
MoRe0.1580.0230.0090.108
Point4D0.2900.0190.0170.142
VGGT-Ω0.0460.0100.0050.040

For benchmarks with ground-truth geometry, such as ScanRefer and ScanNet++, we evaluate directly on the ground-truth geometry provided by the datasets. For 4D-IoU, the annotated and predicted 2D masks are lifted through the same depth and cameras predicted by VGGT-Ω. The points of each clip are centered at their median and normalized by their 90th-percentile radius, the voxel size is 0.02 in this normalized space, i.e., 2% of that radius, and every observation forms its own temporal slice. We will state this scope and these settings explicitly in App. B.4.

Reviewer_Be2yOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents S4VY, a unified framework for feed-forward 4D instance segmentation and language grounding. It leverages geometry-consistent features and persistent object queries to achieve promptable 4D segmentation, complemented by an agentic grounding pipeline.

Critical strengths

The paper addresses an important problem and presents a comprehensive framework that unifies 4D segmentation and language grounding. The experiments are particularly solid, demonstrating consistent improvements across diverse benchmarks, along with extensive evaluations of robustness and efficiency.

Critical weaknesses
  1. Moderate novelty. The segmentation component shares similarities with prior feed-forward geometry-based instance segmentation methods, particularly SegVGGT, making its methodological novelty somewhat limited.
  2. Insufficient analysis of training contributions. The experiments do not fully disentangle the contributions of the proposed architecture, training-data composition, and pseudo-label self-distillation to the overall performance gains.
Questions for authors

Could the authors provide additional ablations on the training-data composition and pseudo-label self-distillation strategy? This would help clarify how much of the segmentation improvement comes from the proposed architecture versus the training recipe.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Be2y

We sincerely thank the reviewer for the positive assessment and the constructive suggestions. We address the comments below.

Weakness 1 (novelty relative to SegVGGT). S4VY shares the query-based decoding formulation with SegVGGT, while its novelty lies at the paradigm level. It explores a Segment Anything paradigm on feed-forward 4D visual geometry, which leverages the jointly built geometry-consistent representation to discover every instance with a persistent identity in a single forward pass, and allows any instance to be selected by point, box or language prompts. In contrast, SegVGGT targets static scenes without prompt interfaces. Within this paradigm, the space-time query decoder maintains one instance identity per query across all observations, and (1) below verifies its advantage over alternative decoding mechanisms under matched training. Language grounding further has its own design, comprising Active Tree Search, a dual-stream Grounder and a comparative Critic. We will clarify this positioning in the revised related work.

Weakness 2 and Question (contributions of architecture, self-distillation and data composition). Following the reviewer's suggestion, we conducted three sets of experiments.

(1) Architecture. We compare four instance decoding mechanisms: (i) per-frame decoding, which links instances predicted independently in each frame by Hungarian matching on mask IoU; (ii) IGGT-style clustering, which replaces the query decoder with a per-pixel embedding head and clusters the embeddings lifted to 3D with HDBSCAN; (iii) IGGT4D-style clustering, which uses the same embedding head but clusters within each frame and merges the clusters across frames; and (iv) our space-time queries, each predicting one instance over all observations. Since full-scale training of every variant is prohibitively expensive, all variants share the same backbone (VGGT-Ω-1B with LoRA), training data (200 ScanNet++ scenes, 60 DAVIS and 120 LVOS videos) and 15k-step budget, and are evaluated on the same 10 ScanNet clips of Tab. 1 and the first ten validation videos of DAVIS and LVOS.

DecodingScanNet (T-mIoU / T-SR / J&F)DAVIS (4D-IoU / 4D-SR / J&F)LVOS (4D-IoU / 4D-SR / J&F)
Per-frame decoding + IoU association0.328 / 0.013 / 0.3210.725 / 0.900 / 0.8300.110 / 0.000 / 0.111
IGGT-style clustering0.554 / 0.508 / 0.4960.450 / 0.600 / 0.5680.163 / 0.000 / 0.213
IGGT4D-style clustering0.701 / 0.625 / 0.6300.687 / 0.950 / 0.7980.106 / 0.000 / 0.153
Space-time queries (ours)0.788 / 0.817 / 0.7600.745 / 1.000 / 0.8590.444 / 0.333 / 0.510

The space-time queries achieve the best results on all three benchmarks. Per-frame association remains competitive on the short and continuous DAVIS videos but fails to preserve identities across sparse ScanNet views and long LVOS sequences, and both clustering variants fall behind consistently.

(2) Pseudo-label self-distillation. To isolate the effect of self-distillation, we keep the model configuration, training budget and evaluation of (1) unchanged and only add 800 clips with pseudo annotations: 200 from YouTube-VOS, 200 from MeViS videos and 400 from DL3DV scans. These clips are supervised by pseudo-masklets and sampled with the curriculum of App. B.1.

Training dataScanNet (J&F)DAVIS (4D-IoU)LVOS (4D-IoU)
Annotated data only (380 scenes and videos)0.7600.7450.444
+ 800 pseudo-labelled clips0.7750.7630.500

Pseudo supervision improves all three benchmarks, with the largest gain on the long LVOS videos (+0.056 4D-IoU).

(3) Training-data composition at full scale. We compare the teacher and the student described in Sec. 3.3 and App. B.1. The teacher is trained only on the seven exhaustively annotated sources (Infinigen, ScanNet++, RealEstate10K, VIPSeg, Cityscapes-VPS, KITTI-STEP and JRDB-PanoTrack), whereas the student, which is our final model, additionally learns through teacher pseudo-masklets from ten partially annotated video datasets (SA-V, YouTube-VOS, MeViS, MOSE, UVO-Dense, DynamicReplica, SAIL-VOS, LVOS, DAVIS and VOST) and two unlabelled multi-view datasets (DL3DV and Matterport3D). We compare them on the following benchmarks from Tabs. 1 and 5.

ModelDAVIS (4D-IoU)LVOS (4D-IoU)HOI4D (J&F)VISOR (J&F)KITTI-STEP (J&F)
Teacher0.6860.5170.6070.2870.489
Student (S4VY)0.7150.5330.6240.3130.501

The partially annotated and unlabelled data improve all five benchmarks: 4D-IoU increases by +0.029 on DAVIS and +0.016 on LVOS, and J&F by +0.017 and +0.026 on the held-out egocentric benchmarks HOI4D and VISOR. The driving benchmark KITTI-STEP also improves by +0.012 J&F although the added data contain no driving-specific dataset. We will include these ablations in the revised appendix.

Thank you for the detailed response and additional experiments. The new ablations convincingly address my concerns about the contributions of the architecture and training strategy. Although the methodological novelty remains somewhat limited, I appreciate the comprehensive framework and strong experimental results. I have raised my score from 3 to 4.

AuthorsOpenReview ↗

Reply to Reviewer Be2y

We sincerely thank the reviewer for the follow-up and for raising the rating. We will update the ablations of the architecture and training strategy in the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their constructive suggestions and for recognizing the strengths of our work. We summarize the common points here.

Strengths recognized by the reviewers.

(1) Unified framework. One instance representation serves prompt-free 4D segmentation, point and box prompting, and language grounding (Be2y, m5AU, 2Gr6).

(2) Solid and broad validation. The improvements are consistent across static scenes, short and long videos and driving scenes, including the complete ScanNet200 validation split and zero-shot benchmarks, together with robustness and efficiency evaluations (Be2y, m5AU, 2Gr6).

(3) Supporting ablations of the grounding components (m5AU, 2Gr6).

Concern 1: attribution of the gains. We refine the attribution of the gains with a controlled comparison of the instance decoding mechanism and the pseudo-label self-distillation under the same backbone and training budget.

SettingScanNet (J&F)DAVIS (4D-IoU)LVOS (4D-IoU)
Per-frame decoding + IoU association0.3210.7250.110
IGGT-style clustering0.4960.4500.163
IGGT4D-style clustering0.6300.6870.106
Space-time queries (ours)0.7600.7450.444
+ 800 pseudo-labelled clips (ours)0.7750.7630.500

Concern 2: novelty. The novelty of S4VY lies at the paradigm level. It explores a Segment Anything paradigm on feed-forward 4D visual geometry, which discovers every instance with a persistent identity in a single forward pass and allows any instance to be selected by point, box or language prompts. Language grounding further has its own design, comprising Active Tree Search, a dual-stream Grounder and a comparative Critic.

The reviewer-specific questions are addressed point by point in the individual responses below. All new results will be included in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.