- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a prediction-stage adaptation framework based on VD-IT for diffusion-based referring video object segmentation (RVOS). It proposes three modules: Temporal Query Evolution for temporally consistent object queries, Cross Granularity Decoupled Decoder for target discrimination and mask completion, and Temporal Context Mask Refinement for improving mask quality and temporal coherence. Experiments on five RVOS benchmarks demonstrate the effectiveness of the proposed method.
Critical strengths
- The paper addresses important challenges of RVOS, particularly in temporal consistency and accurate mask prediction.
- Extensive experiments on five benchmarks and ablation studies demonstrate the effectiveness of the proposed method.
Critical weaknesses
- The performance improvement is not significant, and some comparisons are not fair. On Ref-DAVIS17, CGD improves FlowRVS by only +1.3 J&F, and FlowRVS’s 73.3 is reported as zero-shot, while CGD is finetuned on it. Consequently, this margin does not demonstrate the performance improvement.
- In my understanding, the designs of HSB and DIB do not support the claim of decomposing the mask prediction into semantic coverage and target localization. The structures of HSB and DIB are not explicitly designed for semantic coverage and target localization, since no extra supervision is used to force the modules to learn corresponding capability. The notations such as P_m,P_s,P_b,Y^t in Section 3.3 is not defined, which makes me confused.
- The temporal operation in TCMR does not support the claim of bringing short-term motion cues. The Eq. (7) formulates to perform temporal convolution on features of single frame t, which is confusing.
- Some experiment settings are not clear. For example, the details of unified head in Table 3 and VAE, CNN in Table 6 are not specified in paper, which makes the ablation experiments not clear.
Questions for authors
- No explicit mechanism or supervision enforces HSB to specialize in semantic coverage and DIB in target localization, please provide additional experiments to support your claim.
- Please provide more experiments to verify that the top-k policy of selecting text-aligned long-range frames is practical. For instance, conduct an experiment comparing with selecting random k frames.
- The equations in 3.3 contains undefined notations (P_m, P_s, P_b), and some equations such as eq. (5) are not consistent with release code (concatenation in equation while summarization in release code).
- The performance drops when only using language initialization in Table 5. Please provide more experiments to demonstrate the claim that language initialization alone will therefore hurt performance without temporal adaptation. For example, dense temporal attention over all frames / sparse temporal attention over random selected k frames + language initialization.
- The details of unified head in Table 3 and VAE, CNN in Table 6 are not specified, please provide the module details.
6 comments
Thank you for your response. I have one more question.
I am very surprised that using Ref-DAVIS17 to finetuning reduces the performance from 74.7 to 74.6. This seems quite unusual in machine/deep learning, and I would like to understand why. If the model is not previously trained on Ref-DAVIS17, why does adding training data from the target dataset lead to worse performance? On the other hand, if the model uses Ref-DAVIS17 data for training, is the comparison with the zero-shot FlowRVS results fair?
I would appreciate further reply.
Reply to Reviewer msU9
Thank you for asking about the Ref-DAVIS17 results. The 74.7 and 74.6 J&F reported in Table 1 are scores for the same CGD model before and after DAVIS fine-tuning. The checkpoint trained on Ref-YouTube-VOS scores 74.7 in the zero-shot setting. Continuing from that checkpoint using only the Ref-DAVIS17 training split gives 74.6. Both scores use the official Ref-DAVIS17 evaluation protocol and are reported to one decimal place.
Zero-shot comparison with FlowRVS. For the published DAVIS result, FlowRVS [1] pretrained its model on RefCOCO, RefCOCO+, and RefCOCOg, then continued training on Ref-YouTube-VOS. We pretrained CGD on MeViS and also continued training on Ref-YouTube-VOS. Both models exclude Ref-DAVIS17 from RVOS training and are evaluated directly on its validation set. Under this matched zero-shot protocol, CGD scores 74.7 J&F and FlowRVS reports 73.3 J&F. ReferFormer [2] also evaluates its Ref-YouTube-VOS model directly on Ref-DAVIS17 (supplement, Sec. B).
CGD across three training seeds. We ran a separate experiment starting from CGD checkpoints fine-tuned on Ref-YouTube-VOS alone. We then continued fine-tuning for two epochs using only the Ref-DAVIS17 training split. We evaluated on the Ref-DAVIS17 validation split before and after this step for seeds 42, 52, and 1024.
| Training seed | Before (J&F) | After 2 DAVIS epochs (J&F) | Change (points) |
|---|---|---|---|
| 42 | 74.72 | 74.66 | −0.06 |
| 52 | 74.57 | 74.64 | +0.07 |
| 1024 | 74.68 | 74.71 | +0.03 |
| Mean ± sample SD | 74.657 ± 0.078 | 74.670 ± 0.036 | +0.013 ± 0.067 |
Two seeds improved and one declined. The average changed by +0.013 J&F points after two epochs.
FlowRVS during DAVIS fine-tuning. The Ref-YouTube-VOS checkpoint behind the published 73.3 J&F in [1] was unavailable in the public release. For the supplementary run, we initialized from the public MeViS-trained FlowRVS checkpoint and followed the same DAVIS-only fine-tuning setup. At each checkpoint, we averaged J&F over the expression sets from both Ref-DAVIS17 annotators.
| DAVIS fine-tuning epochs | Mean J&F (both annotators) | Change from initialization (points) |
|---|---|---|
| 0 | 63.56 | — |
| 2 | 64.67 | +1.11 |
| 4 | 67.36 | +3.80 |
| 6 | 67.16 | +3.60 |
FlowRVS's mean validation score rose through epoch 4, then fell by 0.20 points at epoch 6 while staying above the starting score. This illustrates how validation performance can vary during continued training on the target dataset.
For CGD, the three paired runs show essentially unchanged mean performance and no consistent decline after two DAVIS fine-tuning epochs. This conclusion across seeds is limited to the tested two-epoch schedule.
References
[1] Zanyi Wang, Dengyang Jiang, Liuzhuozheng Li, Sizhe Dang, Chengzu Li, Harry Yang, Guang Dai, Mengmeng Wang, and Jingdong Wang. Deforming Videos to Masks: Flow Matching for Referring Video Segmentation. International Conference on Learning Representations (ICLR), 2026.
[2] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as Queries for Referring Video Object Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 4974–4984.
Reply to Reviewer msU9
Thank you for pointing out the difference in training protocols and the ambiguities in our method description.
W1. Ref-DAVIS17 comparison. The original 74.6 J&F used Ref-DAVIS17 training data. Without using Ref-DAVIS17 for training or fine-tuning, CGD scored 74.7 J&F on its validation set, 1.4 points above the zero-shot score of 73.3 reported by FlowRVS [1]. The two CGD settings differ by just 0.1 point, which we regard as within experimental variation. Of the 336,881 training clips across both stages, 268,388 came from MeViS, 61,453 from Ref-YouTube-VOS, and 7,040 (2.09%) from Ref-DAVIS17.
Q1 / W2. Branch roles. We disagree that there is no architectural basis for the different roles of HSB and DIB. HSB has 64 trainable parameters of its own. It scores each pixel by a dot product with the query, favoring broad coverage of regions that match the expression. DIB has 1,266,496 trainable parameters of its own (about 1.27M). It generates an instance-specific kernel from the query and uses pixel coordinates relative to the query reference point to distinguish the target from similar objects. These designs give the branches different biases under a shared mask loss.
Across our mask visualizations, HSB tends to cover more of the object, whereas DIB responds more strongly to the referred instance and suppresses similar distractors. Fig. 8 shows representative examples. To measure how much the trained model uses each branch, we fixed one CGD checkpoint and disabled one branch at a time at inference on Ref-YouTube-VOS:
Both branches contribute to the final prediction, and their masks show complementary behavior despite sharing supervision.
Q2 / Q4. Reference selection and language initialization. To follow up on the drop with language initialization alone in Table 5, we considered dense attention and random references. With language initialization enabled, dense attention scored 71.84 J&F and text-guided Top-3 scored 72.03 on Ref-YouTube-VOS (Table 7). We also compared selection rules on 20 videos with occlusion, reappearance or distractors, 10 each from Ref-YouTube-VOS and MeViS. We used the same checkpoint trained with text-guided Top-3, kept the local neighbors fixed, and changed only how we selected the three long-range references at inference. Against the same ground-truth masks, text-guided selection scored 51.42 J&F, random selection 47.98 (one draw, seed 42), and uniform selection 48.11.
Q3. Notation and mask fusion. The supplementary code inadvertently retained the element-wise sum used in an ablation. The main model concatenates the branch outputs and projects them as in Eq. (5). We tested both:
We will correct the supplementary code and define the notation in Sec. 3.3. Pm projects the channel-wise concatenation of the two mask outputs to a coarse mask. Ps and Pb map the frame query Qt to its confidence score St and bounding box Bt. Yt is the ground-truth annotation used for Hungarian matching at frame t.
W3. Temporal convolution. Let Gt be the spatial feature from frame t and G=(G1,…,GT) the clip's feature sequence in temporal order. Eq. (7) uses the stride-four feature F4t=Gt+[TConv(G)]t. We convolve along the clip's time axis before taking the output at t. The feature thus incorporates neighboring frames without explicitly estimating motion.
Q5 / W4. Ablation baselines. Unified in Table 3 is VD-IT's original dynamic-convolution head: the query generates a kernel that maps visual features to one coarse mask, without HSB–DIB fusion. All four configurations were fine-tuned on Ref-YouTube-VOS only. In Table 6, VAE and CNN use the same refinement path, combining the coarse mask with features at 1/8 and 1/4 resolution. VAE takes the 1/4-resolution feature from the VAE encoder, as in VD-IT. CNN extracts it from each RGB frame with two stride-2 convolutions (3×3 kernels), normalization, and a residual block. With noise prediction off, VAE, CNN, and TCMR scored 71.01, 71.22, and 72.03 J&F, respectively.
Reference
[1] Zanyi Wang, Dengyang Jiang, Liuzhuozheng Li, Sizhe Dang, Chengzu Li, Harry Yang, Guang Dai, Mengmeng Wang, and Jingdong Wang. Deforming Videos to Masks: Flow Matching for Referring Video Segmentation. International Conference on Learning Representations (ICLR), 2026.