- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies target preservation in a modular MLLM-based referring video object segmentation pipeline. It traces proposals, single-frame reference masks, and propagated predictions, and finds a large loss at reference formation in a Qwen3-VL/SAMTok/SAM3 configuration. A support-aware controller can abstain, while conditional recovery revisits withheld queries using additional observations.
Critical strengths
The study measures a concrete failure funnel rather than relying only on qualitative examples. Of 793 expressions, 763 receive proposals, 738 select a target-visible frame, and 501 yield an accurate selected reference; 481 of those maintain final J&F of at least 0.5.
Critical weaknesses
-
The principal comparative results are measured on a split used for framework development. Section 5.1 explicitly states that MeViS valid_u informed prompts, routing, configurations, and reporting thresholds. The independent evaluations in Table 3 are valuable, but do not include the same comparative baseline and operating-point analysis. On the larger MeViS valid split, output coverage is 73.17% and J&F is 0.5485, underscoring that the primary operating point does not automatically transfer. This does not imply optimization on evaluation masks, but it limits the independence of the headline reliability comparison; the small High-J@90 difference is also unaccompanied by video-clustered uncertainty estimates.
-
The bottleneck diagnosis cannot yet separate semantic target resolution from mask generation, or quantify propagation's contribution across the full query population. Section 3.1 acknowledges that reference IoU is only a proxy for semantic preservation. The 237 target-visible failures can arise from descriptions, selection, or segmentation, while the GT-reference intervention is limited to the 501 cases already possessing accurate predicted references. Threshold sensitivity in Table 9 supports the funnel ordering, but does not remove this eligibility restriction. Consequently, the evidence supports reference formation as an important failure location in the chosen pipeline more strongly than a general causal diagnosis of the dominant RVOS limitation.
Questions for authors
-
Which comparative reliability conclusions remain supported on evaluation data that did not inform framework development?
-
How should the bottleneck conclusion be interpreted given the unresolved proposal-versus-mask errors and the 501-case restriction of the GT-reference intervention?
2 comments
Response to Reviewer DibX (2/2)
Q1: Comparative conclusions supported by independent evaluation data.
Thank you for this important question. The independent comparison supports competitive full-set segmentation at the reported output coverage, with lower GPU time than Refer-Agent-9B. The following results use MeViS valid, comprising 140 videos and 2,236 expressions with no video or expression overlap with valid_u:
| Method | Full-set J&F | GPU-hours |
|---|---|---|
| Sa2VA-4B | 0.4746 | 2.78 |
| Refer-Agent-9B | 0.5374 | 116.09 |
| Ours-4B | 0.5485 | 10.73 |
1. Interpreting the reported coverage. Our configuration uses 32 uniformly sampled observations and one reference, with 73.17% output coverage. All 2,236 expressions enter full-set J&F, including missing outputs evaluated as empty predictions. The 0.5485 score therefore already incorporates their effects and remains competitive with the baselines.
2. Uncertainty in the valid_u comparison. For the separate valid_u quality-tail comparison, paired video-clustered 95% confidence intervals for Ours minus Refer-Agent are:
| Metric difference | Estimate (percentage points) | 95% confidence interval (percentage points) |
|---|---|---|
| High-J@90 | +1.26 | [−4.28, +7.12] |
| Low-J@30 | −1.26 | [−5.70, +3.69] |
Both intervals include zero, so these small valid_u quality-tail differences are presented as descriptive outcome profiles. The independent valid comparison supports the reported full-set J&F and GPU-hour results. The paired quality-tail analysis remains on MeViS valid_u. On MeViSv2 valid_u, our 4B framework also achieves higher full-set J&F (0.6179 vs. 0.5851) and native absent recall (0.6159 vs. 0.3969) at 13.37 vs. 97.12 GPU-hours compared to Refer-Agent (Table 2). The results appear in Section 5.1, Table 2, Appendix B.1, and Appendix D.2 (Table 18(b)).
Q2: Interpretation of the revised diagnosis and the expanded GT-reference intervention.
Thank you for raising this important concern. We distinguish the sources of observed reference errors, expand the GT-reference intervention beyond the original 501 cases, and examine what the resulting video predictions reveal about each stage.
1. Error sources. We assign one primary observational category to each of the 237 target-visible reference failures: 108 target-interpretation errors, 47 local-description errors, 30 unsuitable reference frames, 39 mask-generation failures, and 13 annotation or expression ambiguities. Target interpretation and local descriptions account for 65.4%, compared with 16.5% primarily attributed to mask generation. A low reference IoU can therefore reflect different problems in the semantic inputs and local mask generation, which require separate inspection.
2. Expanded GT-reference intervention. We replace predicted references with GT masks on the same selected frames for all 738 expressions with target-visible selected frames, keeping video inputs and propagation fixed. Final failures denote J&F below 0.5.
| Predicted-reference group | N | Mean final J&F: predicted → GT oracle | Failures: predicted → GT oracle |
|---|---|---|---|
| Accurate, local IoU ≥ 0.5 | 501 | 0.8414 → 0.8789 | 20 → 9 |
| Inaccurate, local IoU < 0.5 | 237 | 0.1562 → 0.8792 | 216 → 10 |
| All target-visible selected frames | 738 | 0.6214 → 0.8790 | 236 → 19 |
3. What the expanded intervention establishes. The previously excluded inaccurate-reference group reaches essentially the same oracle quality as the accurate-reference group: 0.8792 versus 0.8789. The fixed propagator can therefore produce high-quality video predictions for many expressions whose predicted references failed, once supplied with correct initialization. The large improvement from 0.1562 to 0.8792 quantifies oracle headroom associated with reference inputs, including errors that originate before mask generation. The 19 residual failures under GT initialization also show that correcting the reference cannot ensure success throughout the video.
The remaining 55 expressions comprise 30 without proposals and 25 with GT-empty selected frames. All are accounted for in the complete 793-expression diagnostic flow. This preserves the distinction between failing to obtain a target-visible proposal and failing after reference formation.
The resulting diagnostic insight is that selected-frame visibility, local reference accuracy, and final video success must be checked separately. The framework's intermediate descriptions and masks allow errors at these interfaces to be examined alongside their downstream effects. This stage-wise diagnosis makes the sources and consequences of target-preservation failures measurable, providing a clearer basis for evaluating reference formation and video propagation separately. The findings are reported in Sections 3.1–3.2, Figure 2, and Appendix C.1, Table 10.
Response to Reviewer DibX (1/2)
Thank you for your careful reading and constructive comments. We address evaluation independence and strengthen the interpretation of the stage-wise diagnosis below.
Thank you for your careful assessment. We address this concern by clarifying the evaluation protocol, adding independent baseline comparisons, and quantifying uncertainty in the valid_u quality-tail differences. The Experimental Setup and Limitations explicitly identify MeViS valid_u as a split used for framework development and diagnosis. The disjoint MeViS valid split provides the independent comparison.
The independent results support competitive full-set segmentation at the reported coverage. Table 2 now includes Refer-Agent-9B and Sa2VA-4B. Our framework achieves 0.5485 full-set J&F at 73.17% output coverage. Coverage denotes the proportion producing outputs. All 2,236 expressions enter the evaluation, with missing outputs represented by empty predictions. Thus, 0.5485 already accounts for the consequences of missing outputs and remains competitive with the baselines. At the reported configurations, our framework uses 10.73 GPU-hours, compared with 116.09 GPU-hours for Refer-Agent-9B (Appendix D.2, Table 18(b)).
Cross-split operating points. Refer-Agent-9B changes from 0.6289 on valid_u to 0.5374 on valid (−9.15 percentage points), and Sa2VA-4B from 0.5628 to 0.4746 (−8.82 points). Our valid_u SAC-only setting reaches 0.5331 at 71.37% coverage, while the valid setting reaches 0.5485 (+1.54 points) at 73.17% coverage. The valid_u SAC-only setting allows up to three references, whereas valid uses 32 uniform observations and one reference. The main valid_u configuration includes conditional recovery and reaches 0.6272 at 89.79% coverage. These comparisons describe performance at their respective operating points.
We also add paired video-clustered 95% confidence intervals, resampling videos to preserve dependence among their expressions. Both intervals for the small valid_u High-J@90 and Low-J@30 differences include zero, so we present these as descriptive quality profiles. Q1 gives the independent comparisons, computational costs, and uncertainty estimates. The revisions appear in Section 5.1, Table 2, and Appendices B.1 and D.2.
Thank you for this insightful concern. We have removed the bottleneck framing from the revised manuscript. The Abstract, Introduction, and Sections 3.1–3.2 now present a stage-wise diagnosis of target-preservation failures in the evaluated pipeline. Manual attribution and the expanded ground-truth (GT) reference intervention clarify three findings.
First, errors observed in reference masks often originate before mask generation. Among the 237 target-visible reference failures, 65.4% are primarily attributed to target interpretation or local descriptions, compared with 16.5% attributed to mask generation. This supports inspecting the inputs supplied to the segmenter alongside its generated masks.
Second, the GT-reference intervention now includes inaccurate as well as accurate predicted references. With GT masks supplied on the same selected frames, the 237 previously excluded cases reach mean final J&F 0.8792, close to 0.8789 for the original 501 accurate-reference cases. The strong oracle outcome therefore also holds for expressions whose predicted reference formation failed, resolving the original eligibility restriction.
Third, correct initialization reveals both propagation potential and remaining failures. Across all 738 expressions with target-visible selected frames, mean final J&F rises from 0.6214 to 0.8790, while failures fall from 236 to 19. This quantifies oracle performance headroom and shows that accurate initialization alone does not ensure video-level success. The framework's intermediate descriptions and masks make these stage conditions separately inspectable and connect them to final predictions. Q2 provides the complete attribution, paired results, and accounting for all 793 expressions. The revisions appear in Sections 3.1–3.2, Figure 2, and Appendix C.1, Table 10.