- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a video spatial-temporal segmentation method. By synthesizing a large amount of pseudo-labeled data, it assists the model in achieving part-level video grounding. Experiments verify the effectiveness of the proposed method.
Critical strengths
- The proposed method achieves state-of-the-art (SOTA) performance, significantly outperforming other baselines.
- The paper is clearly written, with the method, motivation, and experimental analysis well articulated.
Critical weaknesses
- This paper mainly extends the existing BIO Span-based Grounding mechanism from PLUM to the video scenario and combines it with SAM2 for temporal segmentation. Although this integration scheme achieves good performance, its main components are derived from existing works.
- There is a lack of experiments demonstrating why it is necessary to extract Part Spans from generated text via the BIO Head, rather than directly using the Part Expressions explicitly provided in the input query.
- Consequently, it is unclear whether the performance improvement stems from the proposed model design or the newly added Part-level Pseudo-label Supervision.
- The authors do not report the BIO Span prediction accuracy, nor do they compare the segmentation performance using Ground-truth Spans versus Predicted Spans. Therefore, it is difficult to assess the reliability of this key module and its impact on the final performance.
Questions for authors
The authors state, "For each training example, physical instances sharing the same semantic part phrase are merged to form the target semantic mask." Could this merging strategy negatively affect the precision of part-level grounding?
2 comments
Revised Paper Uploaded
Thank you again for your constructive review. We have now uploaded the revised paper, with changes highlighted in blue. Below, we point to the parts that address your comments:
- Why extract spans from the generated text (W2): Sec. 4.4, "BIO-Head Role," with the BIO head vs. average pooling ablation in Table 4.
- Model design vs. pseudo-label supervision (W3): Sec. 4.4, "Effect of training data" (Table 7).
- Merging instances (Q1): clarified in Sec. 3.2 (Training).
Thank you again for your suggestions.
Response to Reviewer E6L6
We thank the reviewer for the careful reading. Below we respond to each point. The new results we present will be included in the revised paper to close the gaps identified by the reviewer.
W1. Novelty relative to PLUM and SAM2
We build on PLUM's span grounding and SAM2 deliberately, since both are proven in their domains. Our contributions are what is needed to make them work for parts in video. First, a part-level training corpus for language-guided video part segmentation, derived from object-level annotations. Second, the span prompt replaces points and boxes in SAM2's prompt encoder and is supplied at every frame, so SAM2's memory tracks the part through the video. Third, an evaluation setup that adapts three part benchmarks to language queries. The resulting model outperforms all baselines on SMITE and PUMaVOS by 27.2 and 7.2 J&F (Table 1 of the paper).
W2. Why extract part spans from the generated text instead of the input query
SAM2 does not accept text, so the target has to be passed to it as an embedding. We project the pooled span representation into a sparse prompt token that replaces points or boxes in SAM2's prompt encoder. We take this embedding from the generated response because the target is often not stated in the query. Queries may require reasoning, e.g., "segment the part the pilot uses to steer," where the answer ("the yoke") appears only in the model's response.
To ablate the contribution of the BIO head explicitly, the table below compares a 4-epoch training of a BIO-head variant against a simple average pool variant. We observe that the BIO head yields moderate improvements on three out of five benchmarks with averaged scores slightly favoring the BIO head over average pooling.
J&F (4 epochs)
We will continue training the two variants to 8 epochs and report the final results in the revised draft.
W3. Model design vs. pseudo-label supervision
We separate the two by holding the data fixed and varying the model. Same data, different architecture: we trained Sa2VA-4B on our pseudo-labels.
J&F (4 epochs)
We observe gains in the performance of the Sa2VA model on three out of five benchmarks with the additional training on our proposed dataset. We also note that VideoPLUM maintains a consistent headroom over the trained Sa2VA variant on four out of five benchmarks, which attests to the effectiveness of the proposed architecture.
W4. BIO span accuracy and ground-truth vs. predicted spans
On our validation split (4,618 requests), the BIO head recovers the annotated span with 99.94% exact-match accuracy (99.99% token F1). Using ground-truth spans instead of predicted spans changes J&F by only 0.12 (68.05 vs. 67.93).
Q1. Effect of merging instances on part-level precision
Merging does not reduce mask precision, since each instance’s mask is kept unchanged. Our queries name a part, such as “the wheel of the car,” and therefore refer to all of its instances. The benchmarks score part masks in the same way, treating all instances of a part as one target. Training, validation and evaluation therefore share the same target definition, so part-level precision is measured against exactly the masks the model is trained to predict.
We will upload the revised paper, including all new results above, on Monday. We hope these responses address the reviewer's concerns and would be happy to clarify any remaining questions.