acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Target Preservation for Reliable MLLM-Based Referring Video Object Segmentation

Abstract

Referring video object segmentation requires identifying a language-specified target and preserving its identity throughout a video. However, mean segmentation accuracy alone provides limited insight into system reliability and the sources of failure. In this work, we investigate target preservation in a controlled multimodal large language model-based pipeline by tracing frame-level proposals, reference masks, and propagated predictions. We find that reference formation is a key bottleneck in this pipeline, even when the target is visible in the selected frame, and that propagation failures can persist despite ground-truth initialization. Guided by these findings, we propose a modular framework that separates video-level target resolution, single-frame reference formation, and video propagation. Support-aware proposal control withholds proposals when visual evidence is insufficient, while conditional recovery revisits these queries with additional observations. Experiments on MeViS demonstrate competitive segmentation accuracy, while controlled comparisons reveal trade-offs among output coverage, mask quality, and target-absent recall, highlighting the need to evaluate reliability beyond mean accuracy. The source code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Referring video object segmentation, Multimodal large language models, Target preservation, Reliability, Reference mask generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DibXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies target preservation in a modular MLLM-based referring video object segmentation pipeline. It traces proposals, single-frame reference masks, and propagated predictions, and finds a large loss at reference formation in a Qwen3-VL/SAMTok/SAM3 configuration. A support-aware controller can abstain, while conditional recovery revisits withheld queries using additional observations.

Critical strengths

The study measures a concrete failure funnel rather than relying only on qualitative examples. Of 793 expressions, 763 receive proposals, 738 select a target-visible frame, and 501 yield an accurate selected reference; 481 of those maintain final J&F of at least 0.5.

Critical weaknesses
  1. The principal comparative results are measured on a split used for framework development. Section 5.1 explicitly states that MeViS valid_u informed prompts, routing, configurations, and reporting thresholds. The independent evaluations in Table 3 are valuable, but do not include the same comparative baseline and operating-point analysis. On the larger MeViS valid split, output coverage is 73.17% and J&F is 0.5485, underscoring that the primary operating point does not automatically transfer. This does not imply optimization on evaluation masks, but it limits the independence of the headline reliability comparison; the small High-J@90 difference is also unaccompanied by video-clustered uncertainty estimates.

  2. The bottleneck diagnosis cannot yet separate semantic target resolution from mask generation, or quantify propagation's contribution across the full query population. Section 3.1 acknowledges that reference IoU is only a proxy for semantic preservation. The 237 target-visible failures can arise from descriptions, selection, or segmentation, while the GT-reference intervention is limited to the 501 cases already possessing accurate predicted references. Threshold sensitivity in Table 9 supports the funnel ordering, but does not remove this eligibility restriction. Consequently, the evidence supports reference formation as an important failure location in the chosen pipeline more strongly than a general causal diagnosis of the dominant RVOS limitation.

Questions for authors
  1. Which comparative reliability conclusions remain supported on evaluation data that did not inform framework development?

  2. How should the bottleneck conclusion be interpreted given the unresolved proposal-versus-mask errors and the 501-case restriction of the GT-reference intervention?

2 comments
AuthorsOpenReview ↗

Response to Reviewer DibX (1/2)

Thank you for your careful reading and constructive comments. We address evaluation independence and strengthen the interpretation of the stage-wise diagnosis below.

W1: Comparative claims rely on development data and lack uncertainty estimates.

Thank you for your careful assessment. We address this concern by clarifying the evaluation protocol, adding independent baseline comparisons, and quantifying uncertainty in the valid_u quality-tail differences. The Experimental Setup and Limitations explicitly identify MeViS valid_u as a split used for framework development and diagnosis. The disjoint MeViS valid split provides the independent comparison.

The independent results support competitive full-set segmentation at the reported coverage. Table 2 now includes Refer-Agent-9B and Sa2VA-4B. Our framework achieves 0.5485 full-set J&F at 73.17% output coverage. Coverage denotes the proportion producing outputs. All 2,236 expressions enter the evaluation, with missing outputs represented by empty predictions. Thus, 0.5485 already accounts for the consequences of missing outputs and remains competitive with the baselines. At the reported configurations, our framework uses 10.73 GPU-hours, compared with 116.09 GPU-hours for Refer-Agent-9B (Appendix D.2, Table 18(b)).

Cross-split operating points. Refer-Agent-9B changes from 0.6289 on valid_u to 0.5374 on valid (−9.15 percentage points), and Sa2VA-4B from 0.5628 to 0.4746 (−8.82 points). Our valid_u SAC-only setting reaches 0.5331 at 71.37% coverage, while the valid setting reaches 0.5485 (+1.54 points) at 73.17% coverage. The valid_u SAC-only setting allows up to three references, whereas valid uses 32 uniform observations and one reference. The main valid_u configuration includes conditional recovery and reaches 0.6272 at 89.79% coverage. These comparisons describe performance at their respective operating points.

We also add paired video-clustered 95% confidence intervals, resampling videos to preserve dependence among their expressions. Both intervals for the small valid_u High-J@90 and Low-J@30 differences include zero, so we present these as descriptive quality profiles. Q1 gives the independent comparisons, computational costs, and uncertainty estimates. The revisions appear in Section 5.1, Table 2, and Appendices B.1 and D.2.

W2: Unresolved error attribution and the restricted GT-reference intervention.

Thank you for this insightful concern. We have removed the bottleneck framing from the revised manuscript. The Abstract, Introduction, and Sections 3.1–3.2 now present a stage-wise diagnosis of target-preservation failures in the evaluated pipeline. Manual attribution and the expanded ground-truth (GT) reference intervention clarify three findings.

First, errors observed in reference masks often originate before mask generation. Among the 237 target-visible reference failures, 65.4% are primarily attributed to target interpretation or local descriptions, compared with 16.5% attributed to mask generation. This supports inspecting the inputs supplied to the segmenter alongside its generated masks.

Second, the GT-reference intervention now includes inaccurate as well as accurate predicted references. With GT masks supplied on the same selected frames, the 237 previously excluded cases reach mean final J&F 0.8792, close to 0.8789 for the original 501 accurate-reference cases. The strong oracle outcome therefore also holds for expressions whose predicted reference formation failed, resolving the original eligibility restriction.

Third, correct initialization reveals both propagation potential and remaining failures. Across all 738 expressions with target-visible selected frames, mean final J&F rises from 0.6214 to 0.8790, while failures fall from 236 to 19. This quantifies oracle performance headroom and shows that accurate initialization alone does not ensure video-level success. The framework's intermediate descriptions and masks make these stage conditions separately inspectable and connect them to final predictions. Q2 provides the complete attribution, paired results, and accounting for all 793 expressions. The revisions appear in Sections 3.1–3.2, Figure 2, and Appendix C.1, Table 10.

AuthorsOpenReview ↗

Response to Reviewer DibX (2/2)

Q1: Comparative conclusions supported by independent evaluation data.

Thank you for this important question. The independent comparison supports competitive full-set segmentation at the reported output coverage, with lower GPU time than Refer-Agent-9B. The following results use MeViS valid, comprising 140 videos and 2,236 expressions with no video or expression overlap with valid_u:

MethodFull-set J&FGPU-hours
Sa2VA-4B0.47462.78
Refer-Agent-9B0.5374116.09
Ours-4B0.548510.73

1. Interpreting the reported coverage. Our configuration uses 32 uniformly sampled observations and one reference, with 73.17% output coverage. All 2,236 expressions enter full-set J&F, including missing outputs evaluated as empty predictions. The 0.5485 score therefore already incorporates their effects and remains competitive with the baselines.

2. Uncertainty in the valid_u comparison. For the separate valid_u quality-tail comparison, paired video-clustered 95% confidence intervals for Ours minus Refer-Agent are:

Metric differenceEstimate (percentage points)95% confidence interval (percentage points)
High-J@90+1.26[−4.28, +7.12]
Low-J@30−1.26[−5.70, +3.69]

Both intervals include zero, so these small valid_u quality-tail differences are presented as descriptive outcome profiles. The independent valid comparison supports the reported full-set J&F and GPU-hour results. The paired quality-tail analysis remains on MeViS valid_u. On MeViSv2 valid_u, our 4B framework also achieves higher full-set J&F (0.6179 vs. 0.5851) and native absent recall (0.6159 vs. 0.3969) at 13.37 vs. 97.12 GPU-hours compared to Refer-Agent (Table 2). The results appear in Section 5.1, Table 2, Appendix B.1, and Appendix D.2 (Table 18(b)).

Q2: Interpretation of the revised diagnosis and the expanded GT-reference intervention.

Thank you for raising this important concern. We distinguish the sources of observed reference errors, expand the GT-reference intervention beyond the original 501 cases, and examine what the resulting video predictions reveal about each stage.

1. Error sources. We assign one primary observational category to each of the 237 target-visible reference failures: 108 target-interpretation errors, 47 local-description errors, 30 unsuitable reference frames, 39 mask-generation failures, and 13 annotation or expression ambiguities. Target interpretation and local descriptions account for 65.4%, compared with 16.5% primarily attributed to mask generation. A low reference IoU can therefore reflect different problems in the semantic inputs and local mask generation, which require separate inspection.

2. Expanded GT-reference intervention. We replace predicted references with GT masks on the same selected frames for all 738 expressions with target-visible selected frames, keeping video inputs and propagation fixed. Final failures denote J&F below 0.5.

Predicted-reference groupNMean final J&F: predicted → GT oracleFailures: predicted → GT oracle
Accurate, local IoU ≥ 0.55010.8414 → 0.878920 → 9
Inaccurate, local IoU < 0.52370.1562 → 0.8792216 → 10
All target-visible selected frames7380.6214 → 0.8790236 → 19

3. What the expanded intervention establishes. The previously excluded inaccurate-reference group reaches essentially the same oracle quality as the accurate-reference group: 0.8792 versus 0.8789. The fixed propagator can therefore produce high-quality video predictions for many expressions whose predicted references failed, once supplied with correct initialization. The large improvement from 0.1562 to 0.8792 quantifies oracle headroom associated with reference inputs, including errors that originate before mask generation. The 19 residual failures under GT initialization also show that correcting the reference cannot ensure success throughout the video.

The remaining 55 expressions comprise 30 without proposals and 25 with GT-empty selected frames. All are accounted for in the complete 793-expression diagnostic flow. This preserves the distinction between failing to obtain a target-visible proposal and failing after reference formation.

The resulting diagnostic insight is that selected-frame visibility, local reference accuracy, and final video success must be checked separately. The framework's intermediate descriptions and masks allow errors at these interfaces to be examined alongside their downstream effects. This stage-wise diagnosis makes the sources and consequences of target-preservation failures measurable, providing a clearer basis for evaluating reference formation and video propagation separately. The findings are reported in Sections 3.1–3.2, Figure 2, and Appendix C.1, Table 10.

Reviewer_PuKsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper diagnoses where target identity is lost in referring video object segmentation, separating proposal, reference formation, and mask propagation. It uses ground-truth reference interventions to test the role of initialization. The proposed system combines single-frame reference formation with a learned proposal/abstention policy and an extra observation pass for withheld proposals. The experiments report both segmentation quality and reliability trade-offs across video grounding benchmarks.

Critical strengths
  1. The stage-wise diagnosis includes a matched ground-truth reference intervention, which helps distinguish reference-mask errors from residual propagation failures.

  2. The paper reports the costs of recovery and additional references as well as their gains. This gives a more useful picture of reliability than average segmentation accuracy alone.

Critical weaknesses
  1. The frozen-versus-adapted comparison uses verbose proposals, not the compact proposal policy used in the final pipeline. Adaptation improves local mask quality among proposed queries but reduces coverage and the all-query accurate-reference rate, so this comparison does not establish a net benefit from the learned controller in the deployed system.
Questions for authors
  1. With the compact prompt, candidate budget, observations, and downstream modules fixed, how do frozen and adapted controllers compare? Please evaluate both before recovery and after applying the same recovery rule, including withheld queries and target-absent frames.

  2. Among queries newly given an output by recovery, how often is the intended target recovered and how often does the system instead predict foreground on target-absent frames? This can be checked from the existing saved masks.

  3. For cases worsened by extra references, are the added reference masks already wrong, or does propagation deteriorate even when all references are accurate? Please split the saved cases along this distinction.

6 comments
AuthorsOpenReview ↗

Response to Reviewer PuKs (1/4)

Thank you for your careful reading and constructive comments. We have strengthened the matched evaluation of support-aware proposal control (SAC) and added analyses of recovery outcomes and reference redundancy. We address your concerns and questions below.

W1: The controller's benefit is not established under the compact proposal policy.

Thank you for highlighting this important concern. We have clarified the motivation for support-aware proposal control (SAC) and conditional recovery and added a matched compact-prompt Base/SAC × Recovery comparison. These revisions evaluate how proposal decisions affect reference quality, output availability, and final segmentation under controlled settings.

Motivation and evidence. Incorrect target interpretations can enter reference formation and affect subsequent propagation. SAC therefore assesses whether sampled observations support generating a candidate before forming a reference mask, while recovery revisits initially unanswered expressions. Under the matched compact setting, SAC increases covered J&F from 0.6281 to 0.6456, while output coverage decreases from 96.85% to 85.12%. Full-set J&F decreases by 5.13 percentage points before recovery (95% CI [−7.75, −2.65]) and 4.11 points after recovery (95% CI [−6.71, −1.71]). Applying the same recovery rule adds accurate references for 12 SAC expressions versus 2 Base expressions, with a larger recovery gain for SAC by 1.03 points (95% CI [0.42, 1.67]), as detailed in Q1.

These results clarify the roles and trade-offs of proposal control and recovery. SAC changes which reference inputs proceed to propagation, and recovery can provide useful predictions for some initially unanswered expressions. Evaluating both stages through reference accuracy, coverage, segmentation quality, and target-absent errors makes their effects explicit. The revisions appear in Sections 4.3 and 5.2, Table 5(b), and Appendix C.3.

AuthorsOpenReview ↗

Response to Reviewer PuKs (2/4)

Q1: Base and SAC under matched compact settings, before and after recovery.

Thank you for this helpful comment. The “frozen” and “adapted” settings in your question correspond to Base and SAC, respectively. Base uses the same framework without SAC training. When an incorrect target interpretation is encoded in a reference mask, propagation can carry that error through the video. We therefore study whether the current observations support a candidate before reference formation, together with recovery for initially unanswered expressions.

Both configurations use the same compact presence-checked prompt, 32 initial uniformly sampled frames, a maximum of three candidates, and frozen SAMTok/SAM3. Both retain this prompt during recovery, using 32 relevance-guided and 32 uniformly sampled frames, with duplicate frames removed. All 793 expressions enter full-set evaluation, with missing outputs evaluated as empty predictions.

ConfigurationOutput coverageAll-query accurate-reference rateFull-set J&FCovered J&FNative absent recall
Base96.85%68.10%0.61920.62810.4506
SAC85.12%61.03%0.56790.64560.5577
Base + recovery97.35%68.35%0.62070.62760.4450
SAC + recovery88.15%62.55%0.57960.63970.5344

The accurate-reference rate counts expressions with at least one reference reaching local IoU ≥ 0.5, over all 793 expressions. Covered J&F uses each configuration's own output set.

1. Quality and availability of reference inputs. SAC yields higher covered J&F both before recovery (0.6456 vs. 0.6281) and afterward (0.6397 vs. 0.6276). Before recovery, the accurate-reference proportion among outputs also rises from 70.31% (540/768) to 71.70% (484/675), using each configuration's output count as the denominator. These gains describe each configuration's own output set and are accompanied by lower output coverage, fewer expressions with accurate references overall, and lower full-set J&F. The matched comparison therefore characterizes how SAC training changes the quality and availability of reference inputs entering propagation.

2. Recovery of accurate references. The same recovery rule adds 12 accurate-reference expressions for SAC and 2 for Base, raising full-set J&F by 1.18 and 0.15 percentage points, respectively. The difference in recovery gains is 1.03 points, with a paired video-clustered 95% CI of [0.42, 1.67]. An initial failure to produce a candidate therefore need not permanently exclude the expression: reassessment can still provide an accurate reference and improve its final prediction.

3. Target-absent behavior. Among the 179 expressions containing GT-empty frames, expressions without outputs contribute 0.0894/0.2067 to Base/SAC native absent recall before recovery and 0.0782/0.1676 afterward. These contributions use all 179 eligible expressions as the denominator. The initial increase in this contribution (0.1173) exceeds the total recall increase (0.1071). This decomposition separates the contribution of missing outputs from correct empty predictions within generated videos, so higher native absent recall alone is not interpreted as better frame-level absence recognition.

The resulting design implication is to evaluate support-aware proposal decisions together with opportunities for reassessment. The first pass determines which reference inputs are immediately available, while recovery can add accurate references for some initially unanswered expressions. Full-set quality, coverage, and target-absent errors measure their combined outcome. The comparison and recall decomposition are reported in Section 5.2, Table 5(b), and Appendix C.3.

AuthorsOpenReview ↗

Response to Reviewer PuKs (3/4)

Q2: Recovered segmentation quality and foreground errors on target-absent frames.

Thank you for this helpful comment. Recovery gives initially unanswered expressions another opportunity to obtain a usable reference while preserving predictions already produced. In Table 5(a), 227 expressions initially lack outputs: 219 valid empty candidate responses and eight parsing failures. With revised proposal instructions and mixed observations, recovery produces outputs for 146/227 (64.32%), while the original 566 outputs remain unchanged.

Segmentation recovered beyond the initial empty predictions. On these 146 expressions, mean J&F rises from 0.0626 to 0.5737, a paired gain of 0.5111. Initial empty predictions can receive positive scores on GT-empty frames, so this measures improvement over their actual initial predictions. We further compare recovery with forced proposals from the initial observations on the same 146 expressions:

PolicyMean J&FFinal J&F ≥ 0.5Empty-frame FPR
Conditional recovery0.573788/1460.5738
Forced proposals from initial observations0.481671/1460.5423

We use final J&F ≥ 0.5 as the operational criterion for successful recovery. Empty-frame FPR averages each expression's fraction of GT-empty frames receiving non-empty predictions over the 39 recovered expressions containing such frames.

Useful recovered outputs and their errors. Recovery yields 88 qualifying outputs (60.27%), compared with 71 under initial forcing, and higher mean J&F on this paired subset. All 146 expressions already contain visible targets in their initial observations. This shows that reassessment can recover useful segmentations even from observations that initially fail to produce a usable proposal. The higher empty-frame FPR (0.5738 vs. 0.5423) quantifies the accompanying increase in incorrect foreground during target absence. On all 227 initially unanswered expressions, forced proposals achieve mean J&F 0.5364, with 123 expressions reaching J&F ≥ 0.5 (Table 13(b)). This broader comparison shows that direct forcing can also recover useful segmentations beyond those obtained through conditional recovery.

Recovery therefore converts some initially unanswered expressions into useful video predictions. Its benefit is assessed through segmentation quality and target-absent errors as well as output coverage. The analysis is reported in Section 5.2 and Appendix C.3, Table 13(b).

Reviewer_zVxZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies where MLLM-based referring video object segmentation (RVOS) fails, using a three-stage pipeline: Qwen3-VL proposes candidate frames with short local descriptions, frozen SAMTok forms a reference mask on the selected frame, and frozen SAM3 propagates it through the video. Tracing 793 MeViS valid_u queries through these stages, the authors find that most losses occur at reference formation (237 failures despite a visible target, versus 20 propagation failures after an accurate reference), and that some propagation failures remain even with ground-truth references. Building on this, they propose a modular framework with a LoRA-trained controller that withholds proposals when the sampled frames do not support the target, and a recovery pass that retries those queries with more frames. They report mean J&F comparable to Refer-Agent at about a third of the GPU time, the highest target-absent recall among compared systems, results on four further benchmarks, and controlled comparisons of proposal format, reference inputs, candidate budget, observation count, and model size.

Critical strengths
  1. The paper evaluates RVOS reliability beyond mean J&F. It traces each query through proposal, reference mask, and propagated video, uses ground-truth references as a control for propagation, and reports output coverage, High-J/Low-J tails, and target-absent behavior alongside the mean. This is consequential because it shows that systems with similar averages can fail in different ways, which is a useful evaluation practice for the field.
  2. The framework is efficient and openly reports its trade-offs. With a 4B MLLM and frozen segmenters, it reaches mean J&F close to Refer-Agent-9B (0.6272 vs. 0.6289) at 12.38 vs. 40.66 GPU-hours, and the paper states where its choices cost coverage, quality, or absent recall. This is consequential because it makes the design practical and the reported gains easy to interpret.
Critical weaknesses
  1. The central diagnostic finding is not well identified. The 237 "reference formation" failures with a visible target mix wrong-instance descriptions, poor frame selection, and SAMTok mask errors, and the paper states that these are not separated. Fig. 8's wrong-instance case, for example, is a target-resolution error counted at this stage. Propagation is also checked only after an accurate reference, so the 237 vs. 20 comparison is asymmetric by construction, and the high reference-to-final correlation (Pearson 0.94) is expected when the propagator follows its seed. The finding also comes from one fixed set of components. This is consequential because the framework is motivated by this diagnosis.
  2. The evaluation does not yet support the reliability claims. The main comparison (Table 1) uses MeViS valid_u, which the authors also used to develop prompts, routing rules, configurations, and thresholds. On the held-out MeViS valid split, only the proposed method is reported (0.5485), and Tables 3–7 have no baselines. Target-absent recall counts GT-empty frames with empty predictions, so withholding a whole query earns full recall on that query; the highest recall in Table 1 and the rise to 0.6946 in quality-first mode therefore partly reflect abstention. Finally, the support-aware controller lowers the all-query accurate-reference rate (59.39% to 56.75%), and the 30B base model without it gives the best mean J&F, so the benefit of the proposed control is unclear. This is consequential because the comparisons in Tables 1 and 2 cannot be trusted as evidence of improved reliability.
Questions for authors
  1. How many MeViS valid_u queries contain GT-empty frames, and how much of the target-absent recall of Ours and Ours quality-first comes from queries that were withheld entirely? What is the recall on GT-visible frames (i.e., how often is a visible target left empty)?
  2. Of the 237 GT-visible reference failures, how many involve the wrong instance or a description that does not match the expression, and how many are mask errors on the correct instance? Even a manual audit of a random subset would help.
  3. Can you report baselines (at least Sa2VA-4B and Refer-Agent) on the held-out MeViS valid split, or on another benchmark in Table 3, so that the main comparison does not rely on the development split?
  4. What does the 30B-A3B model give with the support-aware controller and recovery? This would show whether the proposed control helps beyond a stronger proposal model.
10 comments
AuthorsOpenReview ↗

Response to Reviewer zVxZ (1/4)

Thank you for your careful reading and constructive comments. We address your concerns and questions below.

W1: The central diagnostic finding is not well identified.

Thank you for raising this insightful concern. We address this concern by separating reference-error sources, examining propagation across reference-quality levels, and extending the matched GT-reference intervention. These analyses clarify the role of reference initialization within the evaluated pipeline.

1. Error sources. We manually audit all 237 inaccurate references with target-visible selected frames, assigning one primary observed error category to each. 185 (78.1%) primarily involve incorrect target interpretation, insufficient local descriptions, or unsuitable reference frames. Another 39 (16.5%) involve mask-generation failures, and the remaining 13 involve possible annotation errors or expression ambiguity. Thus, errors observed at the reference interface frequently originate before mask generation. For more details on the complete attribution, please refer to Q2.

2. Propagation across reference-quality levels. The 237/738 (32.1%) inaccurate references among target-visible selected frames and the 20/501 (4.0%) final failures among locally accurate references belong to different conditional subsets. We therefore report propagation outcomes for all 763 expressions with proposals, including 25 with GT-empty selected frames:

Selected reference IoUExpressionsMean final J&FFinal J&F < 0.5
[0, 0.3)2270.0865100.00%
[0.3, 0.5)350.518840.00%
[0.5, 0.7)770.679214.29%
[0.7, 0.9)2360.83861.69%
[0.9, 1]1880.91152.66%

Reference quality is strongly associated with final segmentation success. 481/501 (96.0%) locally accurate references achieve final J&F ≥ 0.5, while 20/501 (4.0%) fail, including 5 with IoU ≥ 0.9. Conversely, 21/35 (60.0%) references in [0.3, 0.5) still achieve final J&F ≥ 0.5. Thus, accurate references usually support successful propagation in this pipeline, although a small fraction fail and some weak references recover. The matched intervention below separately measures the effect of correcting initialization.

3. Matched GT-reference intervention. We replace predicted references with GT masks on the same selected frames for all 738 expressions with target-visible selected frames, keeping video inputs and propagation fixed. Mean final J&F rises from 0.6214 to 0.8790, and failures below J&F 0.5 fall from 236 to 19. The 237 inaccurate-reference cases improve from 0.1562 to 0.8792, while the 501 accurate-reference cases improve from 0.8414 to 0.8789. This oracle diagnostic shows substantial headroom from correcting reference inputs, alongside residual propagation failures.

4. Relationship to framework design. Target interpretation and local descriptions account for 155/237 inaccurate references. The framework makes these inputs explicit through video-level target resolution and frame-local descriptions, enabling the stage-wise tracing and controlled intervention. To evaluate its initialization interface, we use the same 566 expressions, candidate frames, and frozen SAM3, initializing propagation either directly with candidate-local boxes and positive/negative points or with SAMTok masks formed from the selected frame and its local description. Mask initialization improves mean final J&F from 0.5130 to 0.6667 and native target-absent recall from 0.1944 to 0.5221. This supports the mask interface within the studied pipeline. As acknowledged in the revised Limitations (Section 7), the generalizability of these diagnostic findings beyond the Qwen3-VL/SAMTok/SAM3 pipeline remains untested. In future work, we plan to examine whether these findings hold for other MLLM backbones and mask generators.

The analyses are now included in Sections 3.1–3.2 and 5.2, Figure 2, and Appendices C.1, C.4, and C.5. Tables 10 and 14 report the intervention and reference-quality bins, respectively. We remove the bottleneck framing from the revised Abstract, Introduction, and Sections 3.1–3.2, presenting the findings as a stage-wise diagnosis within the evaluated pipeline.

AuthorsOpenReview ↗

Response to Reviewer zVxZ (2/4)

W2: The evaluation does not yet support the reliability claims.

Thank you for your thoughtful comment. We address this concern through independent baseline comparisons, matched evaluation of support-aware proposal control (SAC), and direct analysis of recovery outcomes. These distinguish segmentation quality from missed targets and errors on target-absent frames.

1. Independent comparison. On the disjoint MeViS valid split, our framework achieves full-set J&F 0.5485, compared with 0.5374 for Refer-Agent-9B and 0.4746 for Sa2VA-4B. These results support competitive segmentation beyond the development split. Q3 provides the detailed evaluation settings and GPU-hour comparisons.

2. Role of SAC. In RVOS, the referred target may be absent from sampled observations or from some video frames. Generating unsupported proposals can introduce incorrect reference masks and propagate errors through the video. SAC therefore allows the model to withhold proposals when visual evidence is insufficient, prioritizing reliable reference formation over indiscriminate output coverage. To evaluate this mechanism, we compare Base and SAC on the same 793 expressions, using an identical compact presence-checked prompt, observation and candidate budgets, and frozen SAMTok/SAM3 (Table 5(b)). SAC increases covered J&F from 0.6281 to 0.6456, the accurate-reference proportion among outputs from 70.31% to 71.70%, and native target-absent recall from 0.4506 to 0.5577. However, output coverage decreases from 96.85% to 85.12%, the all-query accurate-reference rate from 68.10% to 61.03%, and full-set J&F from 0.6192 to 0.5679. These results reveal a quality–coverage trade-off. SAC achieves higher covered quality on a smaller, different output set, while also excluding useful references. Moreover, the higher absent recall is largely associated with missing outputs evaluated as empty predictions. Thus, selective proposal control is useful for avoiding unsupported downstream computation, while its effectiveness depends on balancing abstention with opportunities for recovery.

3. Value of recovery. As shown in Table 5(a), recovery revisits 227 expressions without initial outputs: 219 valid empty candidate responses and eight parsing failures. With revised proposal instructions and mixed observations, coverage rises from 71.37% to 89.79%, and the GT-visible non-empty rate from 70.55% to 88.50%. On the same 146 expressions receiving recovery outputs, mean J&F is 0.5737, versus 0.4816 for forced proposals from the initial observations. All 146 already contain visible targets in those initial observations. Recovery thus produces useful segmentations for initially unresolved, already-visible targets, and its foreground errors on GT-empty frames explain the accompanying decrease in native absent recall from 0.6946 to 0.5696, as detailed in Q1.

The revisions are in Sections 5.1–5.2, Tables 2 and 5, and Appendix C.3, Table 13. Besides, we further examine control and recovery with the stronger 30B-A3B proposal model in Q4.

AuthorsOpenReview ↗

Response to Reviewer zVxZ (3/4)

Q1: Expressions containing GT-empty frames and empty predictions on GT-visible frames.

Thank you for your helpful comment. Recovery reduces the GT-visible empty-prediction rate from 29.45% to 11.50%, a reduction of 17.95 percentage points. Correspondingly, the paper's GT-visible non-empty rate increases from 70.55% to 88.50%. These rates measure the proportion of GT-visible frames receiving empty or non-empty predictions within each expression, averaged over all 793 expressions. They directly quantify how often a visible target is left without a predicted mask.

Among these 793 expressions, 179 (22.57%) also contain GT-empty frames. The table reports visible-target omissions alongside the decomposition of native target-absent recall, and both recall contributions use the same denominator of 179.

MetricOurs quality-first (SAC-only)Ours (SAC + recovery)
GT-visible empty-prediction rate (macro)29.45%11.50%
GT-visible non-empty rate (macro)70.55%88.50%
No output (all)22781
Expressions with GT-empty frames179179
No output in this set7536
Contribution from missing outputs0.41900.2011
Contribution from outputs0.27560.3685
Total absent recall0.69460.5696

The improvement combines fewer visible-target omissions with better full-set segmentation. Recovery adds outputs for 146 initially unanswered expressions, preserves the original 566 outputs, and raises full-set J&F from 0.5331 to 0.6272. J&F evaluates segmentation correctness alongside the improved availability measured by the non-empty rate. The decrease in native absent recall from 0.6946 to 0.5696 records foreground errors on GT-empty frames within the newly generated outputs. These results quantify recovery's benefit on visible targets and its accompanying errors during target absence. We also expand the dataset characterization in Appendix B.4, Table 9, including GT-empty and small-target frequencies across benchmarks. The prediction-level results above are reported in Sections 5.1–5.2, Table 5(a), and Appendix C.3, Table 13(a).

Q2: Attribution of the 237 inaccurate references.

Thank you for your constructive comment. We manually audit all 237 inaccurate references with a target-visible selected frame, assigning one primary observational category to each:

Primary observed error categoryCountProportion
Target-interpretation error10845.6%
Incorrect or insufficient local description4719.8%
Unsuitable reference frame3012.7%
Mask-generation failure3916.5%
GT error or expression ambiguity135.5%
Total237100%

Target interpretation or local descriptions account for 155/237 cases (65.4%), compared with 39/237 (16.5%) primarily attributed to mask generation. Including unsuitable frames, 185/237 (78.1%) involve upstream inputs to reference formation. Thus, errors observed in reference masks frequently originate before mask generation, supporting separate evaluation of target resolution and mask formation. The audit is reported in Section 3.2, Figure 2(d), and Appendix C.1.

Q3: Baseline comparisons on the disjoint MeViS valid split.

Thank you for your helpful comment. We add both requested baselines on the same 140 videos and 2,236 expressions of MeViS valid, with no video or expression overlap with valid_u:

MethodFull-set J&FGPU-hours
Sa2VA-4B0.47462.78
Refer-Agent-9B0.5374116.09
Our framework, 4B0.548510.73

Our valid configuration uses 32 uniformly sampled observations and one reference, with 73.17% output coverage. Full-set J&F includes all 2,236 expressions, assigning empty predictions where outputs are missing.

We additionally compare our framework with Refer-Agent-9B on MeViSv2 valid_u (907 expressions). Our 4B framework achieves higher full-set J&F (0.6179 vs. 0.5851) and native target-absent recall (0.6159 vs. 0.3969) at 13.37 vs. 97.12 GPU-hours (7.26×). It achieves higher vIoU (0.5602 vs. 0.5350), while Refer-Agent achieves higher tIoU (0.9029 vs. 0.8491).

These comparisons strengthen the evidence for competitive full-set segmentation and computational efficiency beyond the development split and across benchmarks. The results are reported in Section 5.1, Table 2, and Appendix D.2, Table 18(b).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

Dear Reviewers,

We sincerely thank all reviewers for their careful reading and constructive feedback. We appreciate the recognition of our quantitative stage-wise analysis (zVxZ, PuKs, DibX), reliability evaluation beyond mean accuracy (zVxZ, PuKs), and practical efficiency (zVxZ). Your comments help us sharpen the diagnosis, strengthen framework validation, and broaden reliability evaluation. Substantive revisions are highlighted in red, and all references below refer to the revised manuscript.

1. A more precise stage-wise diagnosis. We remove the bottleneck framing in the Abstract, Introduction, and Sections 3.1–3.2, clarifying the scope of the stage-wise diagnosis within the evaluated pipeline. We manually attribute all 237 inaccurate references with target-visible selected frames and expand the matched ground-truth reference intervention to all 738 expressions with target-visible selected frames. These analyses distinguish error sources and quantify initialization effects, while identifying propagation failures that persist with correct reference masks. (Sections 3.1–3.2, Figure 2, Appendices C.1 and C.4.)

2. Stronger validation of framework design. We add matched evaluations of support-aware proposal control (SAC) and conditional recovery, spatial-initialization comparisons, and multi-reference failure analyses. These experiments clarify how proposal decisions affect prediction quality and availability, support the reference-mask interface, and show that locally accurate additional references can still worsen video segmentation. (Sections 4–5.3, Tables 3-7, Appendices C–D.)

3. More comprehensive reliability evidence. We add baseline comparisons on the disjoint MeViS valid split, decompose target-absent recall, and introduce video-clustered uncertainty estimates. Further 30B-A3B experiments assess control under different observation policies and conditional recovery. These additions support competitive performance beyond the development split and clarify how reliability varies with proposal decisions and observations. (Sections 5.1–5.3, Tables 1–2 and 5, Appendices B–D.)

Thank you again for helping us strengthen the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.