acceptodds
Under review as a conference paper at ICLR 2027

VideoPLUM: A Large Multimodal Model for Part-Level Visual Grounding in Videos

Abstract

Objects are composed of fine-grained parts, and identifying these parts can enable more precise visual reasoning and grounded language generation. Extending such fine-grained grounding to videos, however, requires identifying and consistently tracking object parts over time. Existing image-based segmentation and grounding methods do not readily address this temporal dimension at the part-level. We introduce VideoPLUM, a large multimodal model for fine-grained, part-level visual grounding in videos. Our approach combines the semantic reasoning capabilities of a vision-language model with SAM-based segmentation to generate accurate spatiotemporal masks for object parts, proposing an end-to-end approach for fine-grained, part-level grounding across video frames. To address the scarcity of fine-grained part-level grounding annotations in videos, we develop an automatic data-generation pipeline that identifies relevant object parts and temporally propagates their masks. We evaluate our model on both part-level and object-level grounding across videos and images. VideoPLUM substantially outperforms existing baselines in fine-grained part grounding, achieving 51.3 J&F across SMITE, PUMaVOS, and EgoFun3D and surpassing GLaMM + SAM2 by 12.5 points. Our method also generalizes comparably to image segmentation, extending its fine-grained visual reasoning capabilities beyond video grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual grounding, part-level segmentation, large multimodal models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_E6L6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a video spatial-temporal segmentation method. By synthesizing a large amount of pseudo-labeled data, it assists the model in achieving part-level video grounding. Experiments verify the effectiveness of the proposed method.

Critical strengths
  1. The proposed method achieves state-of-the-art (SOTA) performance, significantly outperforming other baselines.
  2. The paper is clearly written, with the method, motivation, and experimental analysis well articulated.
Critical weaknesses
  1. This paper mainly extends the existing BIO Span-based Grounding mechanism from PLUM to the video scenario and combines it with SAM2 for temporal segmentation. Although this integration scheme achieves good performance, its main components are derived from existing works.
  2. There is a lack of experiments demonstrating why it is necessary to extract Part Spans from generated text via the BIO Head, rather than directly using the Part Expressions explicitly provided in the input query.
  3. Consequently, it is unclear whether the performance improvement stems from the proposed model design or the newly added Part-level Pseudo-label Supervision.
  4. The authors do not report the BIO Span prediction accuracy, nor do they compare the segmentation performance using Ground-truth Spans versus Predicted Spans. Therefore, it is difficult to assess the reliability of this key module and its impact on the final performance.
Questions for authors

The authors state, "For each training example, physical instances sharing the same semantic part phrase are merged to form the target semantic mask." Could this merging strategy negatively affect the precision of part-level grounding?

2 comments
AuthorsOpenReview ↗

Response to Reviewer E6L6

We thank the reviewer for the careful reading. Below we respond to each point. The new results we present will be included in the revised paper to close the gaps identified by the reviewer.

W1. Novelty relative to PLUM and SAM2

We build on PLUM's span grounding and SAM2 deliberately, since both are proven in their domains. Our contributions are what is needed to make them work for parts in video. First, a part-level training corpus for language-guided video part segmentation, derived from object-level annotations. Second, the span prompt replaces points and boxes in SAM2's prompt encoder and is supplied at every frame, so SAM2's memory tracks the part through the video. Third, an evaluation setup that adapts three part benchmarks to language queries. The resulting model outperforms all baselines on SMITE and PUMaVOS by 27.2 and 7.2 J&F (Table 1 of the paper).

W2. Why extract part spans from the generated text instead of the input query

SAM2 does not accept text, so the target has to be passed to it as an embedding. We project the pooled span representation into a sparse prompt token that replaces points or boxes in SAM2's prompt encoder. We take this embedding from the generated response because the target is often not stated in the query. Queries may require reasoning, e.g., "segment the part the pilot uses to steer," where the answer ("the yoke") appears only in the model's response.

To ablate the contribution of the BIO head explicitly, the table below compares a 4-epoch training of a BIO-head variant against a simple average pool variant. We observe that the BIO head yields moderate improvements on three out of five benchmarks with averaged scores slightly favoring the BIO head over average pooling.

J&F (4 epochs)

MethodSMITEPUMaVOSEgoFun3DRef-DAVIS17MeViSAvg.
BIO span (ours)60.5556.2533.2173.5358.3756.38
Average pooling58.0955.7735.2073.5457.4656.01

We will continue training the two variants to 8 epochs and report the final results in the revised draft.

W3. Model design vs. pseudo-label supervision

We separate the two by holding the data fixed and varying the model. Same data, different architecture: we trained Sa2VA-4B on our pseudo-labels.

J&F (4 epochs)

MethodSMITEPUMaVOSEgoFun3DRef-DAVIS17MeViSAvg.
Sa2VA-4B (released checkpoint)22.4943.0530.2873.4855.3144.92
Sa2VA-4B, trained on our data51.9047.7522.7973.9450.3349.34
VideoPLUM60.5556.2533.2173.5358.3756.38

We observe gains in the performance of the Sa2VA model on three out of five benchmarks with the additional training on our proposed dataset. We also note that VideoPLUM maintains a consistent headroom over the trained Sa2VA variant on four out of five benchmarks, which attests to the effectiveness of the proposed architecture.

W4. BIO span accuracy and ground-truth vs. predicted spans

On our validation split (4,618 requests), the BIO head recovers the annotated span with 99.94% exact-match accuracy (99.99% token F1). Using ground-truth spans instead of predicted spans changes J&F by only 0.12 (68.05 vs. 67.93).

Q1. Effect of merging instances on part-level precision

Merging does not reduce mask precision, since each instance’s mask is kept unchanged. Our queries name a part, such as “the wheel of the car,” and therefore refer to all of its instances. The benchmarks score part masks in the same way, treating all instances of a part as one target. Training, validation and evaluation therefore share the same target definition, so part-level precision is measured against exactly the masks the model is trained to predict.

We will upload the revised paper, including all new results above, on Monday. We hope these responses address the reviewer's concerns and would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Revised Paper Uploaded

Thank you again for your constructive review. We have now uploaded the revised paper, with changes highlighted in blue. Below, we point to the parts that address your comments:

  • Why extract spans from the generated text (W2): Sec. 4.4, "BIO-Head Role," with the BIO head vs. average pooling ablation in Table 4.
  • Model design vs. pseudo-label supervision (W3): Sec. 4.4, "Effect of training data" (Table 7).
  • Merging instances (Q1): clarified in Sec. 3.2 (Training).

Thank you again for your suggestions.

Reviewer_2yvfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VideoPLUM, a method for language-guided, part-level visual grounding in videos. It proposes an automatic data generation pipeline to construct part-level video annotations and integrates Qwen3.5 with SAM2 for end-to-end video part segmentation. Extensive experiments demonstrate strong performance on multiple part-level and object-level video segmentation benchmarks.

Critical strengths
  1. The paper addresses an important and challenging task of part-level grounding in videos.
  2. The proposed automatic data generation pipeline alleviates the scarcity of part-level video annotations to some extent.
  3. The proposed method achieves strong performance across multiple video segmentation benchmarks.
Critical weaknesses
  1. The pipeline in Figure 3 primarily combines existing pretrained models (Qwen3.5, LocateAnything, and SAM2). The technical novelty of this pipeline is unclear. In addition, have you consider introducing several detectors to improve the detection performance?
  2. The proposed architecture largely builds upon existing techniques, including BIO-based span grounding from PLUM and temporal mask propagation from SAM2. However, the robustness of the proposed framework to different backbone choices remains unclear. For example, the authors should investigate whether the framework remains effective when replacing (1) SAM2 with more advanced segmentation models, i.e., SAM3, and (2) Qwen3.5 with other VLMs, such as Gemma4.
  3. The current experiments do not adequately disentangle the contributions of the newly generated training data, model architecture, and end-to-end joint training. For example, it remains unclear whether the reported improvements primarily come from the additional part-level training data or the proposed designs. The authors should consider more detailed ablation studies in some aspects, e.g., data contribution, BIO head.
Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2yvf

We thank the reviewer for the careful reading. Below we respond to each point. The new results we present will be included in the revised paper to close the gaps identified by the reviewer.

W1. Novelty of the data pipeline and use of multiple detectors

As in Sa2VA and VideoGLaMM, our pipeline combines existing pretrained models. A key contribution of our work is the training data. Existing video part datasets are small evaluation sets (SMITE: 48 released videos, PUMaVOS: 24, EgoFun3D: 271), none designed for language-guided part segmentation, so we adapt them for evaluation only (Sec. 4.1). For training, we derive 29,034 part requests over 8,563 parent-object tracks from object-level annotations, with no overlap with any evaluation set. Stage 1 already uses a dedicated detector: Qwen3.5 lists the visible parts of each parent object, LocateAnything detects them on parent-isolated crops, and SAM2 converts the detections into masks and links them into part tracks.

Under W3 below, we illustrate that both the training data and the proposed architectures stand their ground under extensive comparative experiments.

W2. Robustness to backbone choices

Segmenter. Below we report results of two VideoPLUM variants, one building on SAM2, the other, on SAM3. We find that the five benchmarks fare differently under the two segmenters, with PUMaVOS and EgoFun3D showing the highest sensitivity and favoring different segmenters. On average, however, the framework remains equally effective with both segmentation models.

J&F (epoch 8)

ModelSMITEPUMaVOSEgoFun3DRef-DAVIS17MeViSAvg.
VideoPLUM (SAM2)61.9156.5535.4075.1257.4257.28
VideoPLUM (SAM3)63.0362.2831.5274.5857.0157.68

VLM. The role of the VLM in this setting is trivial - to isolate the relevant part and provide a text representation for it. We posit that the choice of vision-language modeling backbone is of minimal importance in this kind of architecture, but we have queued experiments to assess this empirically and will report the result in the revised draft.

W3. Disentangling the contributions of data, architecture, and the BIO head

We take the criticism and provide results addressing each of the reviewer's concerns.

Training data. To illustrate the gains from the proposed training data, we train a Sa2VA-4B model on our training data.

J&F (4 epochs)

MethodSMITEPUMaVOSEgoFun3DRef-DAVIS17MeViSAvg.
Sa2VA-4B, original22.4943.0530.2873.4855.3144.92
Sa2VA-4B, trained on our data51.9047.7522.7973.9450.3349.34
VideoPLUM60.5556.2533.2173.5358.3756.38

We observe gains in the performance of the Sa2VA model on three out of five benchmarks with the additional training on our proposed dataset. We also note that VideoPLUM maintains a consistent headroom over the trained Sa2VA variant on four out of five benchmarks, which attests to the effectiveness of the proposed architecture.

BIO head. To ablate the contribution of the BIO head, the table below compares a 4-epoch training of a BIO-head variant against a simple average pool variant. We observe that the BIO head yields moderate improvements on three out of five benchmarks with averaged scores slightly favoring the BIO head over average pooling.

J&F (4 epochs)

MethodSMITEPUMaVOSEgoFun3DRef-DAVIS17MeViSAvg.
BIO span (ours)60.5556.2533.2173.5358.3756.38
Average pooling58.0955.7735.2073.5457.4656.01

We will continue training the two variants to 8 epochs and report the final results in the revised draft.

We will upload the revised paper, including all new results above, on Monday. We hope these responses address the reviewer's concerns and would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Revised Paper Uploaded

Thank you again for your constructive review. We have now uploaded the revised paper, with changes highlighted in blue. Below, we point to the parts that address your comments:

  • Robustness to the segmenter (W2): VideoPLUM-SAM-3 is added to Tables 1 and 2, with its design described in Sec. 3.2 and results discussed in Sec. 4.1 and 4.2.
  • Data vs. architecture (W3): Sec. 4.4, "Effect of training data" (Table 7), compares Sa2VA-4B trained on our data with VideoPLUM.
  • BIO head (W3): Sec. 4.4 and Table 4.

Thank you again for your suggestions.

I appreciate the authors for their detailed responses and additional experiments, e.g., the SAM3-based experiments. While they address some of my concerns, several concerns remain unresolved.

  1. The authors do not analyze the effects of different detectors in their data engine. For example, LocateAnything is a relatively advanced detector, how will the performance change if it is replaced with GroundingDINO?

  2. I believe the evaluation of different VLM backbones is important. I don't think that the role of VLM is trivial, since the proposed method directly relies on its text representations for conditioning. Different VLM families, such as Qwen and Gemma, may have substantially different representation spaces.

Some of my other concerns also remain unresolved, as detailed in my original review.

Overall, I appreciate the authors' efforts, but I think the paper still requires substantial improvements before it is ready for acceptance. Therefore, I maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer 2yvf

Thank you for the follow-up and for acknowledging the additional experiments. We address the two remaining concerns below.

Detector choice. We replaced LocateAnything with GroundingDINO in our Stage 1 pipeline, keeping everything else identical, and evaluated the resulting labels on EgoFun3D. GroundingDINO is primarily an object-level detector and usually localizes the whole object instead of the requested part. As a result, LocateAnything successfully labels four times as many parts as GroundingDINO (331 vs. 82 of 532). As the reviewer notes, LocateAnything is the more advanced detector, and we chose it precisely so that our pipeline would produce higher-quality training data.

VLM backbone. Following the reviewer's suggestion, we trained VideoPLUM with Gemma 4 E4B in place of Qwen3.5-4B, using the same recipe (4 epochs):

J&F (4 epochs)SMITEPUMaVOSEgoFun3DRef-DAVIS17MeViS (Valid_u)
VideoPLUM, Gemma 4 E4B54.8250.9128.8572.2753.44
VideoPLUM, Qwen3.5-4B60.5556.2533.2173.5358.37

We found out that the backbone matters: Qwen3.5 is better on all five benchmarks, by 1.3–5.7 J&F. At the same time, the framework transfers to a different VLM family without modification, and the Gemma variant still outperforms all other baselines on SMITE, PUMaVOS, Ref-DAVIS17, and MeViS (Tables 1 and 2).

Together, these results show that VideoPLUM, like any modular system, benefits from strong components, while the framework itself remains general.

We hope we have addressed your comments and would be happy to resolve any remaining concerns.

Reviewer_wQ4KOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Brief Summary: The paper tackles the task of part-level visual grounding in videos. The input is a text prompt such "beak of the bird" and a given video, and the output is a segmentation mask across the timestamps. The main idea is to use qwen3.5 with LoRA to get a BIO tagging for the relevant tokens, and combine the corresponding embedding token with SAM2 to get the corresponding segmentation masks. Experiments on multiple datasets for video-part-segmentation show significant improvements over baselines over 13 pts on J&F metric (mean of mask-iou and boundary F-measure). This is also seen on tasks such as video object segmentation with 5-15 pts improvement.

Critical strengths

Pros:

  1. Video tracking, part-based visual grounding, especially with natural language can have variety of downstream applications in general video understanding. As such, the task is well motivated.

  2. Authors compare across different tasks such as video-object-segmentation, video-part-segmentation, image-part-grounding. Each task has more than one dataset. Further, authors show qualitative results (Fig4), ablation of various components (Table 4), and improvements on video-QA tasks (Table 5)

Critical weaknesses

Cons:

  1. My main concern is obvious missing baselines. Given SAM3 is available, it is unclear why authors choose to use SAM2 here? Further, SAM3 already takes text-prompts as inputs, so why not do simple Qwen3.5 to get text prompts to provide as SAM3. This is an obvious no-training baseline.

  2. It is unclear why BIO is even needed and authors don't provide an ablation for it. Given that LoRA is already being used, why not just do simple average pool. Even then the obvious case seems to be Qwen3.5 to give the required text prompt that can be used with any separate text-embedding scheme.

  3. The proposed method is worse than SegLLM on image-part-grounding? But authors dont discuss this? If I understand correctly, SegLLM also uses different backbone and not Qwen3.5?

  4. It seems VideoPLUM leads to worse VideoQA accuracy? Even though only LoRA parts are trained? This is quite unclear why this is happening?

  5. The comparisons in most tables are not using same backbones so very difficult to compare. For instance, VideoGLAMM uses Phi-3 Mini, which can't really be compared with Qwen3.5. So most of the comparisons with previous methods don't make sense. Either authors should same backbone as compared baselines, or use show previous methods with qwen3.5.

Questions for authors

see weakness section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wQ4K (1/2)

We thank the reviewer for the careful reading. Below we respond to each point. The new results we present will be included in the revised paper to close the gaps identified by the reviewer.

W1. Missing SAM3 baselines

We used SAM2 so that VideoPLUM is compared under the same conditions as the strongest prior methods: Sa2VA, VideoGLaMM, and the MLLM + SAM2 pipelines all decode with SAM2. We agree SAM3 is a natural baseline and have evaluated it in two zero-shot settings:

  • SAM3 (user query): SAM3 receives the user query directly.
  • Qwen3.5-4B → SAM3 (zero-shot): Qwen3.5-4B sees the video frames and the query and outputs the part name which is passed to SAM3.

In addition, we have trained VideoPLUM with a SAM3 decoder for a more fair comparison against the zero-shot baselines.

J&F (epoch 8)

MethodSMITEPUMaVOSEgoFun3DRef-DAVIS17MeViS
SAM3 (user query)5.139.8420.5011.539.16
Qwen → SAM3, zero-shot, part34.1251.2148.9743.4243.74
VideoPLUM (SAM2)61.9156.5535.4075.1257.42
VideoPLUM (SAM3)63.0362.2831.5274.5857.01

The results show a clear advantage for both VideoPLUM variants over the two zero-shot baselines on four out of five benchmarks. EgoFun3D is an interesting outlier, which we continue to analyze. The sizable gains on the other four benchmarks, however, weigh in favor of the trained VideoPLUM approach.

We also note that between the two VideoPLUM variants, there is no clear winner between SAM2 and SAM3 - VideoPLUM (SAM2) wins on three benchmarks, VideoPLUM (SAM3) on the other two, and both variants come short of the zero-shot Qwen → SAM3 baseline on EgoFun3D.

W2. Necessity of the BIO head vs. average pooling

VideoPLUM adopts the BIO-head design choice following PLUM. However, we agree with the reviewer that this choice merits an ablation. The table below compares a 4-epoch training of a BIO-head variant against a simple average pool variant. We observe that the BIO head yields moderate improvements on three out of five benchmarks with averaged scores slightly favoring the BIO head over average pooling.

J&F (epoch 4)

MethodSMITEPUMaVOSEgoFun3DRef-DAVIS17MeViSAvg.
BIO span (ours)60.5556.2533.2173.5358.3756.38
Average pooling58.0955.7735.2073.5457.4656.01

We will continue training the two variants to 8 epochs and report the final results in the revised draft.

On the suggestion of using a separate text-embedding scheme after Qwen's generation: Qwen's text representations should be sufficiently informative to condition the SAM2 segmentation step. Introducing an additional text embedder would complicate the system unnecessarily. Yet, we recognize that a VideoPLUM variant built on SAM3 could directly take Qwen's text output as a prompt. Below we report results from two trained model variants built from Qwen3.5-4B and SAM3:

  • Qwen → SAM3, trained: takes the text output of Qwen and passes it as a prompt to SAM3; this is basically a trained variant of the zero-shot baseline reported above.
  • VideoPLUM (SAM3): takes Qwen's output embeddings and uses those to condition SAM3's segmentation.

J&F (epoch 8)

MethodSMITEPUMaVOSEgoFun3DRef-DAVIS17MeViS
Qwen → SAM3, trained41.0854.5544.0869.3655.28
VideoPLUM (SAM3)63.0362.2831.5274.5857.01

This shows that VideoPLUM's architecture outperforms the model trained with text prompting alone on both part- and object-level datasets, with the exception of EgoFun3D. We hypothesize that Qwen may provide higher-quality text representations than SAM3's own text embedder, which, with sufficient training to learn an alignment from Qwen's latent geometry to SAM3's one can yield better conditioning for SAM3's segmentation. At this time, this is merely a conjecture, which we continue to explore. The results, however, clearly favor the VideoPLUM approach.

AuthorsOpenReview ↗

Response to Reviewer wQ4K (2/2)

W3. Comparison with SegLLM on image part grounding

On PARTONOMY-Core (Table 3), VideoPLUM is best on Intersection and second to SegLLM on the other four tasks, with gaps of 1.3–3.4 micro gIoU. We note three differences between the models. First, VideoPLUM is trained only on video data and evaluated here zero-shot on images, whereas SegLLM is trained on image referring segmentation (MRSeg), which includes hierarchical part–whole queries similar to the Part2Whole and Whole2Part tasks. Second, SegLLM is built on LLaVA-v1.5-7B, while VideoPLUM uses a 4B backbone. Third, VideoPLUM outperforms the image-based pretrained PLUM-13B on all ten metrics, despite being about three times smaller and trained only on video.

W4. VideoQA accuracy drop despite LoRA

The LoRA adapters introduction in our architecture changes the visual tokens and attention for every query, not only for segmentation. Training also covers only short part-describing responses, which pulls output behavior away from multiple-choice QA. The effect is small: VideoPLUM keeps 97.5% of the base accuracy on MVBench (−1.53) and 93.1% on Video-MME short (−4.67, 900 questions), and outperforms VideoGLaMM, another segmentation video LMM, by more than 20 points on both. Because the adapters can be removed, the base model's QA performance can be recovered exactly by turning LoRA off for pure-QA queries.

W5. Comparisons across different backbones

We agree that backbone differences make cross-method comparisons less clean. This is common in the field. VideoLISA (Phi-3, 3.8B) compares against baselines with different backbones, including LISA and TrackGPT with LLaVA-7B/13B (Table 1 of VideoLISA), and VideoSeg-R1 (Qwen2.5-VL-3B/7B) compares against LISA (LLaVA-7B), VISA (Chat-UniVi-7B) and VideoLISA (LLaVA-Phi-3-V-3.8B) (Table 2 of VideoSeg-R1). Additionally, the job of the LLM is quite simple, it is used to map from a user query to a part.

We will upload the revised paper, including all new results above, on Monday. We hope these responses address the reviewer's concerns and would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Revised Paper Uploaded

Thank you again for your helpful feedback. We have uploaded the revised paper, with all changes highlighted in blue. Below, we point to the parts that address your comments:

  • SAM-3 (W1): VideoPLUM-SAM-3 is added to Tables 1 and 2 and described in Sec. 3.2 and Sec. 4.1. The zero-shot SAM-3 baselines (user query, and Qwen3.5 → SAM-3) are reported in Sec. 4.4 and Table 6.
  • BIO head vs. average pooling (W2): Sec. 4.4 and Table 4. The suggested alternative of passing Qwen's text output to SAM-3 is evaluated in Table 6, both zero-shot and trained on our data.
  • SegLLM (W3): discussed in Sec. 4.3.

Thank you again for your suggestions.

AuthorsOpenReview ↗

Response to Reviewer wQ4K

Thank you again for your review. As the discussion period is coming to an end, we wanted to follow up on our response and the revised paper. In summary, we added SAM-3 baselines and a VideoPLUM-SAM-3 variant (Tables 1, 2 and 6), an ablation of the BIO head against average pooling (Table 4), and a discussion of SegLLM (Sec. 4.3).

We hope these address your concerns, and we would be glad to answer any remaining questions before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.