acceptodds
Under review as a conference paper at ICLR 2027

Cross Granularity Decoupled Decoder and Temporal Query Evolution for Referring Video Object Segmentation

Abstract

Referring Video Object Segmentation (RVOS) aims to segment the target specified by a language expression throughout a video. Recent diffusion based RVOS methods benefit from text aligned spatiotemporal representations, but converting diffusion features into accurate and temporally stable referring masks remains challenging. Frame wise query decoding may suffer from target drift under occlusion, disappearance, and similar distractors. Meanwhile, a unified mask prediction head has to handle both target localization and complete mask generation in the same prediction space. To address these limitations, we propose a prediction stage adaptation framework for diffusion based RVOS. First, we introduce Temporal Query Evolution (TQE), which updates object queries with short term temporal context and text aligned long range references to maintain target identity over time. Second, we design a Cross Granularity Decoupled Decoder (CGD), which decomposes mask prediction into discriminative localization and semantic coverage. The Discriminative Instance Branch performs query conditioned instance filtering to identify the referred target, while the Holistic Semantic Branch provides broader semantic activation for mask completion. We employ Temporal Context Mask Refinement (TCMR) to refine low resolution masks with temporal boundary cues and improve high resolution mask quality. Experiments on five RVOS benchmarks validate the framework's effectiveness and generalization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Referring Video Object Segmentation, Vision-Language Segmentation, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_msU9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a prediction-stage adaptation framework based on VD-IT for diffusion-based referring video object segmentation (RVOS). It proposes three modules: Temporal Query Evolution for temporally consistent object queries, Cross Granularity Decoupled Decoder for target discrimination and mask completion, and Temporal Context Mask Refinement for improving mask quality and temporal coherence. Experiments on five RVOS benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper addresses important challenges of RVOS, particularly in temporal consistency and accurate mask prediction.
  2. Extensive experiments on five benchmarks and ablation studies demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. The performance improvement is not significant, and some comparisons are not fair. On Ref-DAVIS17, CGD improves FlowRVS by only +1.3 J&F, and FlowRVS’s 73.3 is reported as zero-shot, while CGD is finetuned on it. Consequently, this margin does not demonstrate the performance improvement.
  2. In my understanding, the designs of HSB and DIB do not support the claim of decomposing the mask prediction into semantic coverage and target localization. The structures of HSB and DIB are not explicitly designed for semantic coverage and target localization, since no extra supervision is used to force the modules to learn corresponding capability. The notations such as P_m,P_s,P_b,Y^t in Section 3.3 is not defined, which makes me confused.
  3. The temporal operation in TCMR does not support the claim of bringing short-term motion cues. The Eq. (7) formulates to perform temporal convolution on features of single frame t, which is confusing.
  4. Some experiment settings are not clear. For example, the details of unified head in Table 3 and VAE, CNN in Table 6 are not specified in paper, which makes the ablation experiments not clear.
Questions for authors
  1. No explicit mechanism or supervision enforces HSB to specialize in semantic coverage and DIB in target localization, please provide additional experiments to support your claim.
  2. Please provide more experiments to verify that the top-k policy of selecting text-aligned long-range frames is practical. For instance, conduct an experiment comparing with selecting random k frames.
  3. The equations in 3.3 contains undefined notations (P_m, P_s, P_b), and some equations such as eq. (5) are not consistent with release code (concatenation in equation while summarization in release code).
  4. The performance drops when only using language initialization in Table 5. Please provide more experiments to demonstrate the claim that language initialization alone will therefore hurt performance without temporal adaptation. For example, dense temporal attention over all frames / sparse temporal attention over random selected k frames + language initialization.
  5. The details of unified head in Table 3 and VAE, CNN in Table 6 are not specified, please provide the module details.
6 comments
AuthorsOpenReview ↗

Reply to Reviewer msU9

Thank you for pointing out the difference in training protocols and the ambiguities in our method description.

W1. Ref-DAVIS17 comparison. The original 74.6 J&F used Ref-DAVIS17 training data. Without using Ref-DAVIS17 for training or fine-tuning, CGD scored 74.7 J&F on its validation set, 1.4 points above the zero-shot score of 73.3 reported by FlowRVS [1]. The two CGD settings differ by just 0.1 point, which we regard as within experimental variation. Of the 336,881 training clips across both stages, 268,388 came from MeViS, 61,453 from Ref-YouTube-VOS, and 7,040 (2.09%) from Ref-DAVIS17.

Q1 / W2. Branch roles. We disagree that there is no architectural basis for the different roles of HSB and DIB. HSB has 64 trainable parameters of its own. It scores each pixel by a dot product with the query, favoring broad coverage of regions that match the expression. DIB has 1,266,496 trainable parameters of its own (about 1.27M). It generates an instance-specific kernel from the query and uses pixel coordinates relative to the query reference point to distinguish the target from similar objects. These designs give the branches different biases under a shared mask loss.

Across our mask visualizations, HSB tends to cover more of the object, whereas DIB responds more strongly to the referred instance and suppresses similar distractors. Fig. 8 shows representative examples. To measure how much the trained model uses each branch, we fixed one CGD checkpoint and disabled one branch at a time at inference on Ref-YouTube-VOS:

PredictionJ&FJF
Full CGD72.0369.9674.10
Without HSB52.4150.4054.43
Without DIB69.4466.9671.91

Both branches contribute to the final prediction, and their masks show complementary behavior despite sharing supervision.

Q2 / Q4. Reference selection and language initialization. To follow up on the drop with language initialization alone in Table 5, we considered dense attention and random references. With language initialization enabled, dense attention scored 71.84 J&F and text-guided Top-3 scored 72.03 on Ref-YouTube-VOS (Table 7). We also compared selection rules on 20 videos with occlusion, reappearance or distractors, 10 each from Ref-YouTube-VOS and MeViS. We used the same checkpoint trained with text-guided Top-3, kept the local neighbors fixed, and changed only how we selected the three long-range references at inference. Against the same ground-truth masks, text-guided selection scored 51.42 J&F, random selection 47.98 (one draw, seed 42), and uniform selection 48.11.

Q3. Notation and mask fusion. The supplementary code inadvertently retained the element-wise sum used in an ablation. The main model concatenates the branch outputs and projects them as in Eq. (5). We tested both:

FusionJ&FJF
Element-wise sum71.9669.9773.95
Concatenation + projection72.0369.9674.10

We will correct the supplementary code and define the notation in Sec. 3.3. PmP_m projects the channel-wise concatenation of the two mask outputs to a coarse mask. PsP_s and PbP_b map the frame query QtQ^t to its confidence score StS^t and bounding box BtB^t. YtY^t is the ground-truth annotation used for Hungarian matching at frame tt.

W3. Temporal convolution. Let GtG_t be the spatial feature from frame tt and G=(G1,…,GT)G=(G_1,\ldots,G_T) the clip's feature sequence in temporal order. Eq. (7) uses the stride-four feature F4t=Gt+[TConv(G)]tF_4^t=G_t+[\mathrm{TConv}(G)]_t. We convolve along the clip's time axis before taking the output at tt. The feature thus incorporates neighboring frames without explicitly estimating motion.

Q5 / W4. Ablation baselines. Unified in Table 3 is VD-IT's original dynamic-convolution head: the query generates a kernel that maps visual features to one coarse mask, without HSB–DIB fusion. All four configurations were fine-tuned on Ref-YouTube-VOS only. In Table 6, VAE and CNN use the same refinement path, combining the coarse mask with features at 1/8 and 1/4 resolution. VAE takes the 1/4-resolution feature from the VAE encoder, as in VD-IT. CNN extracts it from each RGB frame with two stride-2 convolutions (3×3 kernels), normalization, and a residual block. With noise prediction off, VAE, CNN, and TCMR scored 71.01, 71.22, and 72.03 J&F, respectively.

Reference

[1] Zanyi Wang, Dengyang Jiang, Liuzhuozheng Li, Sizhe Dang, Chengzu Li, Harry Yang, Guang Dai, Mengmeng Wang, and Jingdong Wang. Deforming Videos to Masks: Flow Matching for Referring Video Segmentation. International Conference on Learning Representations (ICLR), 2026.

Thank you for your response. I have one more question.

I am very surprised that using Ref-DAVIS17 to finetuning reduces the performance from 74.7 to 74.6. This seems quite unusual in machine/deep learning, and I would like to understand why. If the model is not previously trained on Ref-DAVIS17, why does adding training data from the target dataset lead to worse performance? On the other hand, if the model uses Ref-DAVIS17 data for training, is the comparison with the zero-shot FlowRVS results fair?

I would appreciate further reply.

AuthorsOpenReview ↗

Reply to Reviewer msU9

Thank you for asking about the Ref-DAVIS17 results. The 74.7 and 74.6 J&F reported in Table 1 are scores for the same CGD model before and after DAVIS fine-tuning. The checkpoint trained on Ref-YouTube-VOS scores 74.7 in the zero-shot setting. Continuing from that checkpoint using only the Ref-DAVIS17 training split gives 74.6. Both scores use the official Ref-DAVIS17 evaluation protocol and are reported to one decimal place.

Zero-shot comparison with FlowRVS. For the published DAVIS result, FlowRVS [1] pretrained its model on RefCOCO, RefCOCO+, and RefCOCOg, then continued training on Ref-YouTube-VOS. We pretrained CGD on MeViS and also continued training on Ref-YouTube-VOS. Both models exclude Ref-DAVIS17 from RVOS training and are evaluated directly on its validation set. Under this matched zero-shot protocol, CGD scores 74.7 J&F and FlowRVS reports 73.3 J&F. ReferFormer [2] also evaluates its Ref-YouTube-VOS model directly on Ref-DAVIS17 (supplement, Sec. B).

CGD across three training seeds. We ran a separate experiment starting from CGD checkpoints fine-tuned on Ref-YouTube-VOS alone. We then continued fine-tuning for two epochs using only the Ref-DAVIS17 training split. We evaluated on the Ref-DAVIS17 validation split before and after this step for seeds 42, 52, and 1024.

Training seedBefore (J&F)After 2 DAVIS epochs (J&F)Change (points)
4274.7274.66−0.06
5274.5774.64+0.07
102474.6874.71+0.03
Mean ± sample SD74.657 ± 0.07874.670 ± 0.036+0.013 ± 0.067

Two seeds improved and one declined. The average changed by +0.013 J&F points after two epochs.

FlowRVS during DAVIS fine-tuning. The Ref-YouTube-VOS checkpoint behind the published 73.3 J&F in [1] was unavailable in the public release. For the supplementary run, we initialized from the public MeViS-trained FlowRVS checkpoint and followed the same DAVIS-only fine-tuning setup. At each checkpoint, we averaged J&F over the expression sets from both Ref-DAVIS17 annotators.

DAVIS fine-tuning epochsMean J&F (both annotators)Change from initialization (points)
063.56—
264.67+1.11
467.36+3.80
667.16+3.60

FlowRVS's mean validation score rose through epoch 4, then fell by 0.20 points at epoch 6 while staying above the starting score. This illustrates how validation performance can vary during continued training on the target dataset.

For CGD, the three paired runs show essentially unchanged mean performance and no consistent decline after two DAVIS fine-tuning epochs. This conclusion across seeds is limited to the tested two-epoch schedule.

References

[1] Zanyi Wang, Dengyang Jiang, Liuzhuozheng Li, Sizhe Dang, Chengzu Li, Harry Yang, Guang Dai, Mengmeng Wang, and Jingdong Wang. Deforming Videos to Masks: Flow Matching for Referring Video Segmentation. International Conference on Learning Representations (ICLR), 2026.

[2] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as Queries for Referring Video Object Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 4974–4984.

Reviewer_r8neOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a prediction-stage adaptation framework for diffusion-based referring video object segmentation (RVOS), consisting of Temporal Query Evolution (TQE), Cross Granularity Decoupled Decoder (CGD), and Temporal Context Mask Refinement (TCMR). TQE uses local temporal interactions and text-guided reference-frame selection, CGD combines holistic mask prediction with instance discrimination, and TCMR refines masks using multi-scale features and temporal convolution. Experiments on five RVOS benchmarks show competitive results. However, the proposed components mainly build upon existing techniques, and the paper does not clearly establish their methodological novelty beyond these architectural modifications.

Critical strengths
  1. The paper presents a coherent prediction-stage adaptation framework for diffusion-based RVOS, with components addressing temporal query modeling, mask prediction, and mask refinement.
  2. The method is evaluated on five RVOS benchmarks, and the reported results are competitive. The ablation studies provide initial evidence that the proposed components contribute to the overall performance.
Critical weaknesses
  1. Limited novelty. The proposed method mainly combines existing techniques. TQE uses temporal query interaction and reference-frame selection, CGD combines dot-product-based mask prediction with dynamic convolution, and TCMR uses multi-scale feature fusion and temporal convolution. Although the overall framework is reasonable, the paper does not clearly explain what is fundamentally new beyond these existing techniques.
  2. Insufficient validation of the proposed designs. The ablation studies show that each component improves performance, but do not fully justify the specific design choices. For TQE, the improvement of Top-3 sparse attention over dense attention is relatively small. For CGD, the experiments do not show why the proposed two-branch design is better than alternative prediction heads. For TCMR, the contribution of temporal modeling is not clearly separated from multi-scale fusion. These issues make it difficult to assess the actual contribution of each proposed design.
Questions for authors
  1. What is the main novelty of the proposed framework beyond combining existing techniques? How are the three components connected beyond addressing different stages of the prediction pipeline?
  2. CGD combines dot-product-based mask prediction and dynamic convolution. How does this design differ from existing segmentation methods, and why is the two-branch design necessary?
  3. The improvement of Top-3 sparse attention over dense attention is relatively small. What is the main benefit of the proposed reference-frame selection strategy?
  4. TCMR combines multi-scale feature fusion and temporal convolution. Can the authors clarify the contribution of temporal modeling independently of multi-scale fusion?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer r8ne

Thank you for your review.

Q1. What is the main contribution, and how do the three modules work together?

We found that MeViS pretraining hurt single-target segmentation with the Unified head. A MeViS expression can refer to multiple instances, whereas a Ref-YouTube-VOS expression refers to one. Existing methods also handle the two settings separately. In their main experiments, ReferDINO and FlowRVS each train a separate checkpoint for evaluation on MeViS; their Ref-YouTube-VOS and Ref-DAVIS17 results use weights trained without MeViS. This suggests that adapting a model to both settings may be a broader challenge.

Table 3 tests whether the mask head affects this transfer. All four configurations were fine-tuned on Ref-YouTube-VOS only. With MeViS pretraining, the Unified head lost 1.23 J&F points on average, while CGD gained 0.95. The direction of both effects held across all three training seeds. Changing the head thus turned the extra supervision from a dataset with multi-instance expressions into a benefit for single-target segmentation.

TQE and TCMR use temporal information at different levels. The coarse mask predicted by CGD connects the two. TQE updates the current frame's query using neighboring frames and distant frames relevant to the expression. CGD turns this query into a coarse mask of the referred target. TCMR then uses fine-scale features from neighboring frames to refine that target's boundary as it moves. We apply Dice and focal losses to both the coarse and final masks, supervising both predictions against the same referred target.

Q2. Why use the dot-product and dynamic-convolution branches in parallel?

HSB predicts a mask by taking the dot product of the target query with each pixel's features. DIB uses the query to generate a convolution kernel and applies it to the visual features. HCD also uses these predictors in parallel. In our model, both branches receive the same TQE-updated query and predict separate masks, which we then fuse.

We kept both branches, the head parameter count, and the training and evaluation settings fixed. On Ref-YouTube-VOS, the parallel connection scored 72.0 J&F versus 69.8 for the serial connection. We also disabled one branch at a time at inference in the trained parallel model. Its score fell from 72.03 to 52.41 without HSB and to 69.44 without DIB. The first comparison favors the parallel connection; the second shows that the model uses both outputs. We did not test a head with two branches of the same type.

Q3. What is the benefit of text-guided reference selection?

TQE retains neighboring frames and selects three long-range references by their relevance to the expression. On Ref-YouTube-VOS, Top-3 scored 72.03 J&F, compared with 71.84 for dense attention (Table 7). Restricting long-range references to three gave similar accuracy to attending to all frames.

To compare selection rules with the same three-frame budget, we selected 10 videos each from Ref-YouTube-VOS and MeViS featuring occlusion, target reappearance, or distractors. We computed J&F against the corresponding ground-truth masks. We used one checkpoint trained with text-guided Top-3 and changed only the selection rule at inference. On these 20 videos, text-guided selection scored 51.42 J&F, compared with 47.98 for random selection (sampling seed 42) and 48.11 for uniform selection: gains of 3.44 and 3.31 points, respectively.

Q4. What does temporal modeling contribute with multi-scale fusion held fixed?

In Table 6, both the framewise CNN and TCMR fuse the coarse mask with features at 1/8 resolution, then refine it with features at 1/4 resolution. The CNN extracts refinement features from the current frame alone; TCMR uses temporal convolution to incorporate neighboring frames. Replacing the CNN with TCMR raised J&F from 71.22 to 72.03 (+0.81) with noise prediction off and from 71.06 to 71.69 (+0.63) with it on. With the fusion steps held fixed, the temporal feature branch improved segmentation in both settings.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer r8ne,

Thank you for taking the time to review our paper and share your feeedback. With the discussion deadline approaching, we'd appreciatehearing whether our response addresses your concerns. If any points remain unclear, we'd be happy to discuss them further. If the additional results address your concerns,we'd be grateful if you would consider raising your score.

Best regards, The authors of Submission #14927

Reviewer_QgEROpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies prediction-stage adaptation for referring video object segmentation with a pretrained text-to-video diffusion backbone. Temporal Query Evolution combines neighboring frames with text-selected long-range references; a Cross Granularity Decoupled Decoder combines dot-product and dynamic-convolution mask branches; and Temporal Context Mask Refinement introduces temporal cues into high-resolution refinement. The method is evaluated on five benchmarks, with J&F scores of 54.9 on MeViS, 72.0 on Ref-YouTube-VOS, and 74.6 on Ref-DAVIS17.

Critical strengths
  1. The empirical contribution is substantial enough to be useful. Results cover five datasets, including A2D-Sentences and JHMDB-Sentences without training or fine-tuning on those datasets (Tables 1–2, pp. 6–7).

  2. The controlled evidence goes beyond comparisons with published systems. Table 3 tests the interaction between pretraining and head design; Tables 4–6 examine the two mask branches, query evolution, language initialization, refinement, and noise prediction. For example, the decoupled head improves J&F from 69.23 to 72.03 under the MeViS-pretraining setting, and removing either branch reduces performance. The design addresses complementary prediction-stage requirements while retaining a pretrained backbone. Query adaptation, target discrimination, mask coverage, and boundary refinement form a coherent system. The paper distinguishes system-level comparisons from controlled comparisons and clearly states its offline inference setting.

I recommend acceptance. The strongest contribution is a well-evaluated prediction-stage design for diffusion-based RVOS, supported by competitive results across five benchmarks and several useful controlled studies. The individual operations build on established components, and the mechanism claims should be stated more carefully. Nevertheless, I consider the empirical contribution and coherent design sufficient for acceptance; the remaining questions primarily concern attribution and reproducibility rather than evidence that the complete method is ineffective.

Critical weaknesses
  1. The empirical benefit of combining the branches is clearer than the proposed explanation of functional decoupling. Equations (3)–(5) supervise the fused prediction, and the 1×1 dynamic-convolution branch is closely related to a feature–weight inner product. Parameter-matched wider single heads and same-type dual heads would help distinguish functional specialization from additional capacity or complementary parameterization (pp. 4–5, Table 4).

  2. The temporal analysis can be strengthened. Appendix C measures changes in frame-level J&F, which does not directly measure mask consistency or identity switches. Moreover, text-selected sparse attention improves over dense attention by only 0.19 J&F without reported seed variation (Table 7). This limits the strength of the specific sparse-selection claim, although the broader temporal adaptation results remain useful. Reproducibility would benefit from a complete optimization configuration, input resolution and clip length, loss weights, and module-level runtime. The reported 19 FPS should be accompanied by a matched baseline and a clear treatment of the interleaved long-video inference protocol (Section 4.2 and Appendix A).

Questions for authors
  1. Are the pretraining/head variants in Table 3 identical in TQE, TCMR, LoRA, training budget, and fine-tuning data? Does the pretraining interaction persist across seeds?
  2. What distinguishes the dynamic-convolution branch from the dot-product branch beyond its projections? Could the authors provide a parameter-matched single-head or same-type dual-head control?
  3. Does text-based reference selection improve over uniform or random references on occlusion/reappearance and distractor subsets?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer QgER

Thank you for distinguishing two questions: whether both branches contribute to the prediction, and what each branch learns. Your reading of Table 2 is correct: we trained the RVOS model only on MeViS and Ref-YouTube-VOS, and evaluated it directly on A2D-Sentences and JHMDB-Sentences.

Q1. Does the effect of pretraining on each head persist across seeds?

Yes. All four configurations in Table 3 used the same TQE, TCMR, and LoRA settings and the same fine-tuning budget and data (Ref-YouTube-VOS only). The two heads also had the same training budget within each pretraining condition. We repeated all four configurations with seeds 52 and 1024, changing only the seed from the original run (42). For both heads, each new run with MeViS started with a fresh round of pretraining followed by fine-tuning.

Mask headWithout MeViSWith MeViSChange with pretraining
Unified70.44 ± 0.0569.22 ± 0.02−1.23 ± 0.04
CGD71.08 ± 0.0472.04 ± 0.01+0.95 ± 0.04

Values are Ref-YouTube-VOS J&F (mean ± sample SD over the three training seeds). MeViS pretraining lowered the Unified score and raised the CGD score at every seed. The difference between the two changes (CGD minus Unified) was 2.16, 2.19, and 2.19 points for seeds 42, 52, and 1024, respectively: 2.18 ± 0.02 points on average.

Q2 / W1. How do the branches differ, and what does the equal-parameter comparison show?

HSB scores each pixel by taking a dot product between its projected visual features and the projected target query. In DIB, the query generates the parameters of two dynamic convolution layers with a ReLU between them. The input also includes each pixel's coordinates relative to the query's reference point. DIB is thus both nonlinear and position-dependent. These structural differences allow the branches to learn different roles under a shared mask loss. We see this pattern across the branch masks we visualized, without imposing separate supervision on either branch.

We compared serial and parallel connections with the same HSB and DIB, the same head parameter count, and the same training and evaluation settings. Parallel fusion scored 72.0 J&F on Ref-YouTube-VOS, versus 69.8 for serial fusion. In a separate test, we fixed the trained parallel model and disabled one branch at a time at inference. J&F fell from 72.03 to 52.41 without HSB and to 69.44 without DIB. The first comparison favors the parallel connection; the second shows that the trained model uses both outputs.

Q3. Does text-guided reference selection help on difficult videos?

We selected 10 videos from Ref-YouTube-VOS and 10 from MeViS and evaluated all three selection rules against the corresponding ground-truth masks. We used one checkpoint trained with text-guided Top-3. The local neighbors and all other inference settings were fixed; only the choice of the three long-range references changed.

Reference selectionOcclusion/reappearance (12)Distractors (8)All 20
Text similarity52.350.151.42
Random (sampling seed 42)48.746.947.98
Uniform48.447.748.11

Text-guided selection gave the highest J&F in both groups. Across all 20 videos it scored 3.44 points above random and 3.31 above uniform selection. On the full Ref-YouTube-VOS benchmark, Top-3 scored 72.03 versus 71.84 for dense attention (Table 7). We did not measure variation across training seeds for this 0.19-point difference. Appendix C measures fluctuations in frame-level segmentation accuracy; it does not directly measure mask alignment across frames or identity switches.

W2. Training details and inference speed. We trained the new modules and the LoRA adapters on the query and value projections (rank 16, alpha 32), keeping the original backbone, VAE, and text encoders frozen. MeViS pretraining took 10 epochs and fine-tuning took 6, both on eight H20 GPUs. We applied Dice and focal losses to both the coarse and final masks. The supplementary code gives the timing scope for each module and the steps for sampling and merging long-video predictions.

We measured 19 FPS on MeViS using one H20 GPU for offline inference. Videos exceeding the model's input length are split into interleaved subsequences that share the first frame. We process each subsequence and restore the predictions to their original frame order. The reported throughput is for this full MeViS inference procedure.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reply to All Reviewers

We thank all three reviewers for their careful reading. QgER highlighted the ablations and cross-dataset transfer results; r8ne noted how query modeling, mask prediction, and refinement fit together; and msU9 recognized the breadth of the evaluation and pointed out an important difference in the Ref-DAVIS17 training protocols.

Our main finding is that whether MeViS pretraining helps on Ref-YouTube-VOS depends on the mask head. We repeated all four configurations in Table 3 with two additional seeds. All were fine-tuned on Ref-YouTube-VOS only. For the two configurations with MeViS pretraining, we also reran pretraining at each new seed. The result held across all three seeds: pretraining lowered J&F by 1.23 points on average with the original VD-IT head and raised it by 0.95 points with CGD.

Our individual replies cover the following:

  • QgER: We report the results across training seeds, compare serial and parallel designs at equal head parameter count, and give reference-selection results separately for 12 videos with occlusion or reappearance and 8 with distractors.
  • r8ne: We explain how the coarse mask predicted by CGD connects the query updated by TQE to mask refinement in TCMR. We also compare three ways to choose reference frames on the same 20 difficult videos. With one checkpoint and a budget of three long-range references, text-guided selection scored 51.42 J&F, random selection 47.98, and uniform selection 48.11.
  • msU9: Without training or fine-tuning on Ref-DAVIS17, CGD scored 74.7 J&F there, compared with 73.3 reported by FlowRVS under the same target-data restriction. We also identify the fusion rule used for the main result, distinguish it from the sum variant in the supplied code, and describe the ablation baselines in Tables 3 and 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.