acceptodds
Under review as a conference paper at ICLR 2027

4DSuperG: Identity-Aware Super-Gaussians for Open-Vocabulary 4D Scene Understanding

Abstract

Open-vocabulary understanding of dynamic scenes requires text-selected objects to retain their identities as they move and become temporarily invisible. Existing 4D language fields distill per-Gaussian features through alpha-blended rendering, leaving primitive-level semantics ambiguous and often requiring feature compression. Identity-based language association addresses these issues in static scenes, but dynamic querying also requires the selected primitives to move coherently. We introduce 4DSuperG, which groups Gaussians into motion-coherent, identity-aware Super-Gaussians (SuperGs), making identity learning and motion prediction share the same unit. Each SuperG retrieves a full-dimensional CLIP feature through its object identity, without language-feature distillation. A query selects SuperGs once in canonical space, and their motion carries the selection through time and under occlusion. On HyperNeRF and Neu3D, 4DSuperG achieves state-of-the-art primitive-level open-vocabulary segmentation and competitive pixel-level performance, and supports zero-shot language-driven spatio-temporal editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Gaussian Splattings, Open-Vocabulary Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ctLKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

4DSuperG targets open-vocabulary object query in dynamic scenes. It starts with motion-based grouping, uses temporary per-Gaussian encodings to refine associations. Decoded object IDs retrieve full-dimensional CLIP feature. Text queries select canonical primitives whose learned deformation carries the identity across time. Experiments on HyperNeRF and Neu3D report improved primitive-level segmentation, with additional zero-shot language-driven spatio-temporal editing demonstrations.

Critical strengths

None

Critical weaknesses
  1. This work is an incremental combination of existing approaches: motion grouping comes from SP-GS; identity-based CLIP feature assignment comes from Segment-then-Splat and OpenGaussian; and Super-Gaussians have been explored in SuperGSeg. I do not believe this work offers meaningful insights or contributions to the task of open-vocabulary querying with 3DGS. In my view, it does not meet the bar for acceptance at ICLR.
  2. The paper is poorly written, making it difficult to understand, assess, and follow the work. I turned to the Method section for answers to my questions raised while reading the abstract and introduction, but found that it reads like a list of implementation steps without adequately justifying the design choices. Important relationships between modules need to be made explicit in the text.
Questions for authors

What is the principal advance beyond the cited components, and which experimental comparison directly isolates and supports that advance?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ctLK

We thank the reviewer for the review and answer the question directly below. Line numbers refer to the revised PDF, and all cited passages are unchanged from the submission.

Answer: The principal advance is 4DSuperG, a representation of identity-aware Super-Gaussians whose grouping is learned jointly from motion and identity, so that the unit of identity learning is also the unit of motion in dynamic scenes. This representation yields state-of-the-art primitive-level open-vocabulary 4D segmentation (Table 1), robustness to occlusion (Fig. 3, Table 11, Fig. 12), a reusable carrier for object-level features (Fig. 4), zero-shot language-driven spatio-temporal editing (Fig. 5), and efficient training and querying (Appendix D). The controlled comparison with SA4D (Table 1) isolates the contribution of the 4DSuperG carrier, and the identity-stage ablations (Table 2) that of identity-reshaped grouping.

Principal advance (W1, Question). W1 frames the task as open-vocabulary querying with 3DGS, but our paper targets dynamic 4D scenes, with complex motion and occlusion. Our insight is that, in dynamic scenes, open-vocabulary querying requires the selected object to retain its identity as it moves and through occlusion (L031–032), so the representation cannot be designed apart from the dynamics: the unit of identity learning must also be the unit of motion (L090–091). Sharing identity and motion within a group pools evidence across members and time, so visible members constrain occluded ones (L079–082). Each group carries one time-invariant identity, which a query selects once in canonical space. The paper already cites every listed work and distinguishes 4DSuperG from each:

  • SP-GS (L154–158, L173–174) groups Gaussians by motion only, for reconstruction. In 4DSuperG, identity reshapes the grouping, because motion-only groups merge co-moving objects (L082–083, Fig. 6).
  • OpenGaussian (L066–076, L126–131) targets static scenes, discretizes contrastive instance features with a learned codebook, and matches 3D instances to 2D masks. In 4DSuperG, IDs supervised by tracked masks directly index a table of CLIP features (Sec. 3.3), and the selection stays intact as the Gaussians move, whereas SA4D with the same lookup fragments (Fig. 1).
  • Segment-then-Splat (L148–150) fixes Gaussian–object membership before reconstruction. 4DSuperG learns membership during reconstruction from both motion and identity.
  • SuperGSeg (L152–154) forms Super-Gaussians in static scenes from geometric and segmentation cues and distills language features onto them. 4DSuperG's groups are also motion units, and their language features are attached by lookup instead of distillation.

None of these works couples semantics and motion in the same unit, and putting their components together does not produce one: motion-only groups merge co-moving objects, and identities attached afterwards cannot split them. Making identity reshape the grouping requires the two-stage design of Sec. 3.2: per-Gaussian identity bootstrapping, then shared identity refinement.

Experiments that support this advance (Question).

  • Against the state of the art (Table 1). Our full method improves primitive-level mIoU by 26.61 / 25.48 on HyperNeRF / Neu3D over 4DLangSplat, the strongest open-vocabulary 4D method at the primitive level in its original form.
  • Controlled comparison against per-Gaussian identities (Table 1). SA4D cannot answer open-vocabulary queries by itself. We deliberately equip it with our lookup and post-pruning, which makes it the strongest baseline, so that language association and pruning are identical and the identity carrier differs. Even without non-rigid refinement, our method remains 7.16 / 2.43 mIoU higher.
  • Identity-reshaped grouping (Table 2). Ablation (f) skips the per-Gaussian bootstrap, so identity is learned only on the motion-based groups. It is 5.96 / 24.52 mIoU below the full identity stage (g).

Writing (W2). The other reviewers found the paper clear and easy to follow. The overview of Sec. 3 states how the modules connect (L163–168), and each design choice is motivated where it is introduced: shared motion because per-Gaussian deformation fragments a selection (L172–173), identity because motion alone merges co-moving objects (L222), and per-Gaussian bootstrapping because shared identity learning and association refinement are interdependent (L227–229).

We hope this clarifies our contributions, and that the reviewer will reconsider the assessment in light of the comparisons above.

I thank the authors for your response, but my rating is still unchanged. Segment-and-Splat already assigns CLIP embedding IDs to Gaussians in open-vocabulay query in dynamic scenes. However, there is no experimental comparsions with Segment-then-Splat. The remaining novelty narrows to learned soft Gaussian membership from motion and identity, and the paper should be presented that way. The paper's ablation shows this design adds a little: (g)→(e), reduces mIoU by 2.69 and 0.56, respectively, while removing per-Gaussian bootstrap drops to 50.79 in Neu3D which is below SA4D. I therefore keep my original rating.

AuthorsOpenReview ↗

Response to Reviewer ctLK: Segment then Splat comparison, and the two identity stages

Thank you for these specific points. We are happy to address them.

Regarding comparison with Segment-Then-Splat

We did run it before the submission, and excluded it for a reason we should have stated in the paper. We add a video showing why its design does not work for 4D primitive-level open-vocabulary segmentation to the revised supplementray materials.

In short: primitive-level querying is only useful if the selected primitives keep the object through occlusion, and in 4D Segment then Splat does the opposite: its object loss deforms that geometry away to match the occlusion-truncated mask.

Its object loss, Eq. (6), supervises the isolated rendering of each object’s Gaussians against the ground-truth image masked by that object’s 2D mask: Lobj=L1(Mip⊗Ii,I^ip)L_{obj} = L1(M_i^p ⊗ I_i, Î_i^p). Where the object is occluded, the mask has a hole, so the object’s own Gaussians are penalised for occupying that region. In static 3D this is largely corrected by other views. In 4D it is not: the deformation network predicts a per-timestep deformation from canonical, so the geometry is free to be carved to the occlusion-truncated silhouette at each t. The supplementary video shows this: an object reconstructed by Segment then Splat with a concave notch where the occluder sits, which persists across viewing angles and is therefore geometry rather than a rendering artifact. We find its design against the motivation of primitive-level segmentation protocol therefore excluding it.

In addition, in its paper it reports result 69.48 and 44.00 mIoU on HyperNeRF and Neu3D in a 3D querying protocol described as “directly retrieving the Gaussians associated with an object” (Sec. 4.2), while our primitive-level results are 79.10 and 76.07. We report its numbers as published, the scene and query sets are not specified there and may differ, but it is still a large margin here and ours is obtained while actually retaining the object under occlusion, as shown in Appendix F and the reply to reviewer e3nb.

Regarding the ablation and the two identity stages

In our design, identity acts on the grouping through two stages: the per-Gaussian identity bootstrap (IB), stage one, and the shared SuperG identity refinement (SIR), stage two.

If we understand the reviews correctly, the reviewer read the table as: (g)→(e): removing SIR causes only a small drop (g)→(f): removing IB causes a large one and the gain is therefore attributed to IB, which the reviewer think that taken to be SA4D’s contribution rather than ours.

We believe this rests on a misreading of stage one. On the mechanism: SA4D learns a per-Gaussian, time-varying identity field, and has no grouping for identity to act on. In our bootstrap the identity losses act on the association logits themselves, and the per-Gaussian encodings are temporary and discarded once the shared encodings are initialised from them (Sec. 3.2, Appendix B.4). Stage one is where membership first becomes identity-aware: it is the first half of the proposed design, not a borrowed component.

The table also disapprove this. If stage one were equivalent to SA4D’s carrier, case (e), which has stage one but not stage two, should match SA4D under the same lookup. Instead (e) reaches 70.54 and 74.75 against 63.06 and 59.41, i.e. +7.48 and +15.34. With both stages, (g) is +10.17 and +15.90.

What the table shows, then, is that identity learning is more effective on the SuperG than on the individual Gaussian. Fig. 3 and the amodal evaluation in Appendix F show the same for occlusion robustness against SA4D, and Appendix D reports substantially lower training and query cost. The learned identities also carry other 2D object-level features without retraining. These are further advantages over SA4D.

A clarification on Super-Gaussians, here and in prior work : “Super-Gaussian” is a name for a group of Gaussians, taken from superpoints, and sharing the name does not mean sharing the mechanism. We hope this does not hinder our novelty.

  • SuperGSeg forms static groups from geometric and segmentation cues and distils language features onto them.
  • SP-GS groups by motion alone, so a group may cross object boundaries, which makes it unsuitable as a unit for segmentation (Fig. 6c).

Ours is formed from motion and identity jointly, which is what makes it usable as a query unit in 4D, and what yields the state-of-the-art primitive-level results together with the efficiency, occlusion robustness and reusability above.

We hope this addresses the concern, and would be grateful if you would reconsider the assessment.

Reviewer_e3nbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents 4DSuperG, a representation for open-vocabulary understanding of dynamic scenes. It groups Gaussians into local Super-Gaussians that share identity encodings and motion models. The grouping is first established through motion supervision, then refined using instance identities through per-Gaussian bootstrapping and shared Super-Gaussian identity learning. Decoded identities index object-level CLIP descriptors, enabling language queries without optimizing per-Gaussian language features. Selected primitives can subsequently be propagated through time for object isolation and editing.

Experiments on six HyperNeRF and six Neu3D scenes demonstrate improved primitive-level segmentation over the evaluated baselines, competitive pixel-level segmentation, and qualitative editing capabilities.

Critical strengths

S1. The motivation of this paper is clear. Coupling motion and identity within Super-Gaussians is a reasonable and practically meaningful design, particularly for persistent object selection under deformation and occlusion.

S2. The component ablations support the usefulness of the staged motion–identity grouping design, particularly the importance of per-Gaussian identity bootstrapping and the additional benefit of shared SuperG identity refinement.

Critical weaknesses

W1. Novelty is somewhat limited, as the proposed framework largely integrates existing components. Motion-based superpoint grouping, rendered identity supervision, structured semantic Gaussian groups, and instance-level language association all have established precedents [1-5]. R5DGS [6] also combines identity encoding, group-level motion, and offline language-feature lookup, albeit for object-level rigid extrapolation. This does not establish equivalence, but it narrows the conceptual novelty attributable to the broad combination.

W2. The quantitative evaluation primarily relies on segmentation mIoU and mAcc@50, which do not directly measure motion accuracy, temporal identity preservation, or occlusion robustness. The qualitative results are encouraging, but additional evaluations.

[1] Diwen Wan, Ruijie Lu, and Gang Zeng. Superpoint gaussian splatting for real-time high-fidelity dynamic scene reconstruction. In ICML, pp. 49957–49972. PMLR, 2024.

[2] Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. In ECCV, pp. 162–179. Springer, 2024.

[3] Shengxiang Ji, Guanjun Wu, Jiemin Fang, Jiazhong Cen, Taoran Yi, Wenyu Liu, Qi Tian, and Xinggang Wang. Segment any 4d gaussians, 2024.

[4] Yanmin Wu, Jiarui Meng, Haijie Li, Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, and Jian Zhang. Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding. In NeurIPS, 2024b.

[5] Siyun Liang et al. SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussians. In 3DV, 2026.

[6] Denis Gridusov, Maxim Popov, and Sergey Kolyubin. Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors. arXiv preprint arXiv:2605.25909, 2026.

Questions for authors

Q1. Given the multi-granularity mask preprocessing, how does the framework handle fine-grained text queries referring to different parts of the same object? Specifically, when multiple Super-Gaussians share the same object identity and CLIP descriptor, can language queries distinguish between them, or is the effective query granularity limited by the available mask-derived identities? Please clarify this capability boundary.

Q2. What is the precise methodological distinction between the proposed joint grouping and a combination of motion-based superpoints, identity supervision, and instance-level language association? Please also discuss the relationship to R5DGS, particularly the distinction between learning multiple local motion units per identity and propagating an entire semantic object through a shared rigid transformation. This clarification would materially affect my assessment of novelty.

Q3. Under the primitive-level evaluation protocol, are unselected Gaussians retained as occluders, or is the selected subset rendered independently? How are partially and fully occluded targets scored, and which reported quantitative results specifically support robustness through occlusion?

6 comments
AuthorsOpenReview ↗

Response to Reviewer e3nb (1/2): Novelty and R5DGS

Response (1/2)

We thank the reviewer for the constructive review, and for recognizing both the motivation of coupling motion and identity within SuperGs and the support from our component ablations. We answer Q2 first, then Q3, W2, and Q1. In the revised PDF, revisions for this review are marked in green\color{green}{\text{green}}, and those that also address another reviewer in purple\color{purple}{\text{purple}}.

Q2 & W1: What is the precise methodological distinction between the proposed joint grouping and a combination of motion-based superpoints, identity supervision, and instance-level language association? Please also discuss R5DGS.

Answer: In a combination, the Gaussian-to-SuperG associations are fixed by motion and identity is attached afterwards. In ours, the associations are optimized under both motion and identity losses, so identity reshapes the groups (Sec. 3.2). Removing the per-Gaussian bootstrap, which starts this reshaping, lowers mIoU by 5.96 / 24.52 (ablation (f), Table 2). R5DGS is now discussed in Related Work (L140–141).

  • Insight. In dynamic scenes, querying requires the selected object to retain its identity as it moves and through occlusion (L031–032), so the unit of identity learning must also be the unit of motion (L090–091). Sharing identity and motion within a group pools evidence across members and time, so visible members constrain occluded ones (L079–082).
  • Mechanism. Motion-only groups can merge co-moving objects, such as the hands and the container in Fig. 6, and identity attached afterwards cannot split them. In 4DSuperG, the identity variance and reconstruction losses act directly on the association logits, while the motion loss stays active, so the groups become coherent in both motion and identity.
  • Benefits of this coupling. Beyond state-of-the-art primitive-level open-vocabulary 4D segmentation (Table 1), it yields robustness to occlusion (Fig. 3, Table 11, Fig. 12), a reusable carrier for object-level features (Fig. 4), zero-shot language-driven spatio-temporal editing (Fig. 5), and efficient training and querying (Appendix D).
  • Compared with the listed components. SP-GS groups by motion only, Gaussian Grouping and SA4D learn per-Gaussian identities, OpenGaussian associates static instances with CLIP features, and SuperGSeg forms static Super-Gaussians without motion. None makes the identity unit also the motion unit, as 4DSuperG does.
  • Evidence. Ablation (f) skips the bootstrap, so identity is learned only at the SuperG level on the motion-based groups. Per-Gaussian identities with the same lookup and post-pruning (SA4D + our lookup + our post-pruning, Table 1) remain 7.16 / 2.43 mIoU below ours.
  • R5DGS addresses a complementary goal: physically grounded extrapolation beyond the observed time range, which our method does not attempt. It forms one group per object from per-Gaussian identities and, during extrapolation, propagates each object rigidly from a representative Gaussian to accelerate rendering. 4DSuperG instead learns local motion units jointly with identity, so one object can comprise several units (Fig. 8) that may move differently, with soft associations blending their transformations across part boundaries. Such units could serve as local rigid bodies for R5DGS-style extrapolation of articulated objects.
AuthorsOpenReview ↗

Response to Reviewer e3nb (2/2): Occlusion, Temporal Evaluation, and Part Queries

Response (2/2)

Q3: Under the primitive-level evaluation protocol, are unselected Gaussians retained as occluders, or is the selected subset rendered independently? How are partially and fully occluded targets scored, and which reported quantitative results specifically support robustness through occlusion?

Answer: The selected subset is rendered independently, without unselected Gaussians as occluders. The benchmarks provide ground-truth masks only for visible regions, not for occluded parts, so targets are scored against visible-region masks, and the standard metrics cannot reward occlusion robustness and even penalize it (Sec. 4.1, L352–354). To study occlusion directly, we add an experiment with amodal ground truth, in which our method keeps 96.24 amodal mIoU under full occlusion, versus 59.64 for SA4D (Appendix F).

  • Protocol. A partially occluded target is scored against its visible part, and a fully occluded target has an empty visible mask, so any part retained through occlusion counts as false positives.
  • Occlusion study (Table 11, Figs. 11–12). On a monocular 4D Infinigen sequence of 288 frames, in which two moving occluders hide and reveal a vase, our method reaches 97.42 / 96.24 amodal mIoU on partially / fully occluded frames, versus 61.75 / 59.64 for SA4D with our lookup and post-pruning, only slightly below 98.90 on fully visible frames. The updated supplementary material includes a video of this sequence comparing our selection with SA4D. As a single synthetic scene, this is a first step, and a dedicated occlusion benchmark is a direction for future work.

W2: Temporal identity and motion accuracy.

Answer: The primitive-level protocol already tests temporal identity preservation, and Q3 addresses occlusion. We do not claim more accurate motion than 4DGS, and our reconstruction quality is on par with it (Table 7).

  • Temporal identity. Primitives are selected once in canonical space and scored at every frame, so identity or motion errors in later frames directly lower the score. In time-sensitive querying (Appendix C.3, Table 6), our method reaches 59.82 vIoU versus 30.80 for 4DLangSplat.
  • Motion accuracy. As an indirect measure, our reconstruction quality with the non-rigid residual reaches 28.20 vs. 28.61 PSNR for 4DGS on HyperNeRF and 31.16 vs. 31.15 on Neu3D (Table 7).

Q1: Can language queries distinguish parts of the same object?

Answer: Not currently, but the representation leaves room for it. As in other methods that derive identities or language features from SAM masks, language queries are bounded by the mask-derived identities (Sec. 4.1; Appendices B.1, E). SuperGs that share an identity also share its CLIP descriptor, so language queries cannot distinguish them, and a part is queryable only if the SAM masks at some granularity level separate it, typically the small level. However, because SuperGs are grouped jointly by motion and identity, one object can still comprise several SuperGs (Fig. 8), which may follow different motions. This opens the possibility of per-SuperG descriptors for part-level queries, which we leave for future work.

We hope these clarifications, especially on novelty and occlusion, address the reviewer's concerns. If so, we would be grateful if you would consider raising your score. We are happy to clarify any remaining points.

Reviewer_e3nbOpenReview ↗

Response to Authors

Thank you for the detailed responses. I understand that the rebuttal period can be demanding, and I appreciate the time and effort you have put into addressing my concerns.

Your clarifications have addressed most of my questions, and I appreciate the additional experiment in Appendix F.

My remaining concern is the degree of methodological novelty. I recognize the practical value of the method and the distinction between jointly learning motion–identity groups and attaching identities after motion grouping. I do not agree with the absolute assertion that this paper makes no meaningful contribution. However, I still view the contribution as a relatively straightforward integration and incremental extension of established mechanisms, and the rebuttal has not yet sufficiently changed this assessment.

One more question is: in ablation (f), does the association matrix A continue to receive gradients from the identity losses during shared identity refinement (SIR)?

Reviewer_bVLnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents 4DSuperG for open-vocabulary object segmentation and editing in dynamic scenes. The method organizes Gaussians into Super-Gaussians, each with a shared identity encoding and a local motion model. The representation is learned through motion-based grouping, per-Gaussian identity bootstrapping, and shared identity refinement. At query time, object IDs retrieve full-dimensional CLIP features. A text query selects the target Gaussians once in canonical space, and the learned motion propagates this selection through time. Experiments on HyperNeRF and Neu3D evaluate both pixel-level and primitive-level querying, with additional demonstrations of time-sensitive queries and object editing. Compared with SA4D equipped with the same CLIP lookup and post-pruning, the method improves primitive-level mIoU by 7.16 and 2.43 percentage points on the two datasets, respectively.

Critical strengths
  1. The representation is simple and well suited to querying and editing dynamic objects. I find the use of the same SuperG for identity and local motion a useful design. An object can consist of several local motion units, while a shared object ID allows these units to be selected together. This supports selecting an object once and carrying the selection across frames for subsequent object-level operations. The visualizations also show fewer floaters and boundary streaks.

  2. The method improves over an augmented SA4D baseline. The authors equip SA4D with the same full-dimensional CLIP lookup and post-pruning, making this comparison more informative about the proposed representation. The method achieves primitive-level mIoU of 79.10 on HyperNeRF and 76.07 on Neu3D, compared with 71.94 and 73.64 for this SA4D variant. Table 2 also examines the main design choices, including identity bootstrapping, shared refinement, feature dimensionality, and post-pruning, providing further support for the method.

Critical weaknesses
  1. The visualizations leave some uncertainty about whether occluded object regions are retained. In the two cooking scenes on the right of Figure 3, the selected chef appears to end around the waist, whereas the SA4D and 4DLangSplat results include leg-like regions. The authors emphasize on page 8 that the method preserves object structure through occlusion. Looking at these examples, I wonder whether the cleaner results are due to fewer streaking artifacts or whether real parts of the chef's body are also missing. Please clarify whether the missing regions were absent from the reconstruction, assigned to the wrong identity, or removed by post-pruning.

  2. The use of masks at multiple granularities for identity supervision is unclear. Appendix B.1 retains four sets of label maps with different IDs, but the main method describes a single identity classifier and decodes one final ID per SuperG at query time. I could not determine which supervision target is used when a pixel has different labels at different granularities. If the granularities are trained separately, how are the results selected or combined for the main table? If they are trained jointly, how are overlapping labels handled? This choice affects the object boundaries learned by the model and is needed to reproduce the SA4D comparison under the same settings. Please clarify the training and inference rules.

Questions for authors

1. What causes the missing lower body of the chef in Figure 3?

Are these regions visible in other training views, and does the base reconstruction contain the corresponding Gaussians? If so, how do identity selection and post-pruning affect them? Please show relevant training views and intermediate results, particularly corresponding results with and without post-pruning.

2. How are the four SAM mask granularities used for identity learning?

Appendix B.1 states that preprocessing generates masks at four granularities (default, small, middle, and large), with different IDs and a separate label map for each level. How is the granularity selected during training, or how are multiple labels for the same pixel handled? Are separate models or classifier heads used? If the granularities are trained or evaluated separately, how are the results combined or selected for Table 1, and does SA4D follow the same rules?

3. What storage or query-time benefits accompany the longer training time?

The appendix reports average training times of approximately 1.8 times those of SA4D and 2.3–2.5 times those of 4DLangSplat. If object-level feature sharing reduces storage, GPU memory use, or query time, please quantify these benefits. Please also specify whether the reported training times include non-rigid refinement and preprocessing, so that the practical trade-offs can be assessed.

1 comment
AuthorsOpenReview ↗

Response to Reviewer bVLn

We thank the reviewer for the positive assessment, and especially for appreciating the augmented SA4D baseline, which we constructed deliberately for a more informative comparison. SA4D is an identity segmentation method and cannot perform open-vocabulary 4D segmentation by itself. We equip it with our identity-to-CLIP lookup and post-pruning, which makes it the strongest baseline in Table 1, so that the comparison focuses on the difference between per-Gaussian identities and our motion- and identity-aware SuperGs. Against the strongest open-vocabulary 4D baseline at the primitive level, 4DLangSplat, our full method improves primitive-level mIoU by 26.61 / 25.48 on HyperNeRF / Neu3D. The revised Tables 1, 4, and 5 make this explicit by labeling the SA4D rows "+ our lookup" and "+ our post-pruning".

We are also grateful to the reviewer for pointing out that we did not clearly explain how the SAM granularity levels are used, which also obscured the efficiency advantage of our method. The three questions are connected, so we answer Q2, Q3, then Q1. In the revised PDF, revisions for this review are marked in blue\color{blue}{\text{blue}}, and those that also address the other reviewers in purple\color{purple}{\text{purple}}.

Q2: How are the four SAM mask granularities used for identity learning?

Answer: Like the baselines, we produce one model per level, but we train only once, on small-level masks, and transfer the identities to the middle and large levels (Sec. 4.1, L332–336, L346–349, L361–364; Appendix E).

  • Training: Following 4DLangSplat, the default level is unused. As in all baselines, each pixel has exactly one label per level, with no joint multi-level targets or separate heads. All baselines, including SA4D, train the three level-specific models separately.
  • Cross-level transfer: We project the trained Gaussians into training views and assign each SuperG its middle- and large-level ID by majority vote over its member Gaussians, without further training. This takes 24.2 s / 34.2 s per scene on average (HyperNeRF / Neu3D) and saves two of the three training runs (Q3). The baselines do not directly support it:
    • Distillation methods such as 4DLangSplat learn continuous CLIP features with no discrete identity to relabel.
    • SA4D's identity field f(x,y,z,t)f(x,y,z,t) is time-varying, so a Gaussian can take different IDs across frames and offers no stable unit for voting.
  • Inference (Table 1): Each query is evaluated at all three levels, and the level with the highest CLIP relevance gives the output, as for SA4D and 4DLangSplat.

Q3: What storage or query-time benefits accompany the longer training time?

Answer: Counting all three levels, our training time is the lowest on both datasets. Queries are 158× faster than SA4D and use only 0.75 GB of GPU memory with 512-D CLIP features (Appendix D, Tables 8–10).

  • Training time: The paper reported baseline times per level. Three-level totals:
Average (min)OursOurs + non-rigidSA4D (3×)4DLangSplat (3×)
HyperNeRF50.157.683.265.1
Neu3D101.5114.4168.9118.8

Our times include cross-level transfer. Preprocessing, shared by all methods, is excluded.

  • Query time and memory: All prompts of a HyperNeRF scene over the full video take 0.35 s for ours vs. 55.70 s for SA4D. SA4D must recompute IDs at every frame, while our time-invariant SuperG identities are selected once. Language features are stored per object, so for a scene with NN objects, a query compares the text feature with NN 512-D CLIP features and uses 0.75 GB of GPU memory on average.

Q1: What causes the missing lower body of the chef in Figure 3?

Answer: The legs are genuinely missing, not just fewer streaks. They are reconstructed and not removed by post-pruning, but are assigned the wrong (background) identity, because SAM masks are inconsistent across granularity levels under cross-level transfer (Appendix E, Fig. 10).

  1. chef is answered at the large level, whose identities we transfer from the small-level model.
  2. The small-level SAM masks merge the legs into the background, so leg Gaussians share SuperGs with background Gaussians.
  3. Majority-vote transfer labels these mixed SuperGs as background. SA4D trains a separate large-level model and keeps the legs.

Trained directly on large-level masks (Fig. 10, no post-pruning), our method keeps the legs. Since the cause is not occlusion, our occlusion robustness claim is unaffected. This is the trade-off for training only once. All reported results already include such cases, and we still outperform SA4D at about 0.6× its training time.

We hope these clarifications resolve the concerns behind the soundness rating. If so, we would be grateful if you would consider raising your score. We are happy to clarify any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.