acceptodds
Under review as a conference paper at ICLR 2027

CER-RVS: Contrastive Evidence Reflection for Agentic Referring Video Object Segmentation

Abstract

Referring Video Object Segmentation (RVOS) requires identifying and segmenting a language-referred object throughout a video, often based on complex cues involving actions, temporal relations, and object interactions. Recent training-free agentic approaches leverage the zero-shot reasoning capabilities of Multimodal Large Language Models (MLLMs) to avoid costly task-specific adaptation, yet remain limited by unreliable reasoning and imperfect object tracks. We propose CER-RVS, an evidence-verified, training-free agentic framework that jointly improves reasoning reliability and object-track quality. First, Contrastive Evidence Reflection replaces one-sided candidate pruning with explicit evidence competition: two reasoning agents independently collect supporting and counter-evidence for each candidate track, while an independent arbitrator contrasts their judgments, resolves conflicts, and provides targeted feedback for iterative revision. Second, Anchor-Guided Track Refinement selects reliable anchor masks based on mask reliability and temporal consistency, uses them to guide bidirectional refinement of candidate tracks, and incorporates identity-consistent refinements to improve mask quality and temporal completeness. Together, these components form a complementary perception–reasoning loop that strengthens visual evidence while enabling more verifiable and revisable reasoning. Extensive experiments on three challenging language-guided video object segmentation benchmarks demonstrate that CER-RVS establishes new state-of-the-art performance without additional training, substantially outperforming existing training-free approaches and surpassing representative supervised fine-tuning methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Model, Reasoning Video Object Segmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BgzqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CER-RVS, a training-free agentic framework for referring video object segmentation (RVOS) that addresses two key limitations of existing methods: incomplete candidate tracks and unreliable candidate selection. Anchor-Guided Track Refinement (AGTR) selects a reliable anchor frame for each candidate based on SAM3's predicted mask confidence and relative mask area, then performs bidirectional re-tracking to recover missing observations and improve track completeness. Contrastive Evidence Reflection (CER) employs a Proponent and a Critic to independently examine supporting and contradicting evidence for each candidate, while an Arbitrator resolves conflicting judgments and guides targeted re-examination of uncertain candidates. Experiments show that, using the same MLLM backbone, CER-RVS outperforms AgentRVOS by 3.5, 1.9, and 5.0 J&F points on MeViS, ReVOS, and ReasonVOS, respectively.

Critical strengths
  1. Motivation is clear. The paper identifies two key problems in track-level agentic RVOS and addresses them with two complementary modules. Their nearly additive gains in Table 2(c) support this design.
  2. The method is effective. CER-RVS shows consistent improvements across three benchmarks and two MLLM backbones, including a 5.0-point J&F gain on ReasonVOS over AgentRVOS.
  3. The paper provides an interesting ablation study. Beyond final J&F scores, the paper evaluates AGTR using track coverage and missing rate, and CER using the same candidate pools. These results help explain why each module works.
Critical weaknesses
  1. The paper does not compare with Refer-Agent[1], a closely related training-free agentic RVOS method.
  2. The paper lacks a hyperparameter sensitivity analysis for N, alpha, and R.

[1] https://arxiv.org/abs/2602.03595

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Brief Summary of Responses to Reviewer Bgzq

Dear Reviewer Bgzq,

We sincerely thank you for your time and effort in reviewing our paper and for your constructive suggestions. To facilitate a quick review, we summarize the key points of our rebuttal below.

  • Comparison with Refer-Agent: We have conducted additional experiments comparing CER-RVS with Refer-Agent under the same Ovis2.5-9B MLLM backbone. Our method outperforms Refer-Agent by 9.8, 0.2, and 4.5 J&F points on MeViS, ReVOS (Overall), and ReasonVOS, respectively, demonstrating its effectiveness across different RVOS benchmarks. (Please refer to Weakness 1 in our response for details.)

  • Hyperparameter Sensitivity Analysis: We have conducted additional sensitivity experiments on ReasonVOS to investigate the effects of N, α, and R. The results demonstrate that our default settings (N = 5, α = 0.6, R = 3) achieve the best performance among the tested configurations, balancing anchor reliability, candidate diversity, and reflection effectiveness while considering inference efficiency. These findings further validate our design choices. (Please refer to Weakness 2 in our response for details.)

We sincerely hope that these additional experiments and analyses have adequately addressed your concerns and further strengthened our submission. We would greatly appreciate it if you could raise your score in light of these new results. Should you have any further questions or concerns, we would be happy to provide additional clarification.

Thank you again for your valuable time and thoughtful feedback.

Best regards,

Authors

Sorry for my late reply. After carefully reading all the reviews and the authors’ rebuttal, I have decided to maintain my original score. Best of luck with your paper!

AuthorsOpenReview ↗

Sincere Thanks to Reviewer Bgzq

Dear Reviewer Bgzq,

We sincerely thank you for taking the time to carefully review our rebuttal. We fully understand that this is an especially busy period during the rebuttal process, as everyone also has their own research and manuscripts to attend to. Therefore, we are particularly grateful for the additional time and effort you have devoted to revisiting our paper and thoughtfully evaluating our responses.

We also deeply appreciate your insightful and constructive feedback, which has helped us identify important areas for improvement and further strengthen the clarity and rigor of our work.

Once again, thank you very much for your valuable time, thoughtful consideration, and encouragement. We truly appreciate your recognition and support. We also sincerely wish you the very best and hope that all your own submissions receive favorable outcomes.

Best regards,

Authors

Reviewer_j3KrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CER-RVS, a training-free referring video object segmentation framework designed to address incomplete candidate trajectories and unreliable target judgment. The method first generates candidates via SAM3, then refines them through Anchor-Guided Trajectory Refinement (AGTR), which selects anchors based on mask reliability and relative target area for bidirectional re-tracking. Subsequently, Contrastive Evidence Reflection (CER) employs proponent and critic agents to evaluate candidate-query alignment, while an arbiter synthesizes evidence and provides iterative feedback for unresolved cases. Evaluated on MeViS, ReVOS, and ReasonVOS, CER-RVS outperforms AgentRVOS by 3.5, 1.9, and 5.0 J&F points. Extensive ablations and trajectory quality analyses further validate the contributions of trajectory refinement and multi-agent reflection.

Critical strengths

Strength 1: Rational Problem Decomposition with Methods Targeting Practical Failure Modes. The paper simultaneously addresses two critical issues: incomplete candidate trajectories and unreliable candidate judgment. AGTR employs instance mask initialization for bidirectional tracking, while CER manages uncertain candidates through a structured support-opposition-arbitration mechanism. The two components exhibit a clear functional division of labor. Furthermore, the framework’s independence from task-specific training enhances its practical utility.

Strength 2: Consistency of Primary Performance Gains and Ablations. As shown in Table 1, under the identical Qwen3-VL-8B-Thinking backbone, the proposed method achieves J&F improvements of 3.5, 1.9, and 5.0 points over AgentRVOS across three datasets, respectively. The Ovis2.5-9B variant also yields competitive results. Additional experiments provide substantive evidence for the system's effectiveness, demonstrating concurrent improvements in both target retention and distractor suppression.

Critical weaknesses

Weakness 1: "Re-examining Visual Evidence" Is Effectively Text-Only Reflection. Section 3.3 and Figure 3 characterize the error correction process as re-examining visual facts such as motion direction and temporal intervals. However, I witness Algorithm 1 explicitly stipulate that video is accessed only in the initial round, with subsequent iterations restricted to text-only reflection. I do not mean that textual reflection is ineffective; rather, it supports "re-interpreting existing descriptions" but cannot substantiate "re-verifying visual facts." If a critical action is missed in the first round, or if the default 16 frames fail to capture the event, subsequent text-only iterations cannot recover the missing information. The visual arbiter ablation in Table A1 fails to resolve this issue, as it modifies the arbiter's input rather than the subsequent visual agent's access to evidence.

Weakness 2: The Unique Contribution of Adversarial Roles Remains Undistinguished from General Multi-Pass Reasoning. In the appendix, the Proponent and Critic employ the same model, with prompts primarily corresponding to "satisfies description" versus "violates description." Although Table A1 matches invocation counts, it fails to compare the adversarial setup against a baseline of two neutral independent reviewers under identical arbitration, feedback, and stopping rules. Thus, what I can tell from the result is current gains may stem from multiple judgment passes, arbitration, and deferred decision-making. If both roles commit correlated errors, the current pipeline offers no mechanism for subsequent correction. The paper lacks an analysis of such correlated failure modes.

Questions for authors

Question 1: To What Extent Do AGTR’s Gains Stem from Target Judgment Versus Mask Refinement? Could you provide a cross-factor experiment that independently controls whether the MLLM views coarse or refined trajectories, and whether the final output uses coarse or refined masks? Please report candidate discrimination ability when the output mask is fixed, and segmentation gains when candidate selection is fixed. It will be better if the oracle selection upper bound is provided.

Question 2: Is the Adversarial Role Design Strictly Necessary for the Observed Gains? Could you introduce a control condition featuring "two neutral reviewers + arbitration + feedback" under identical invocation budgets, arbiters, feedback mechanisms, and stopping rules? This would distinguish the contribution of adversarial opposition from that of general ensemble reasoning and reveal the performance ceiling imposed by irreversible decisions.

1 comment
AuthorsOpenReview ↗

Brief Summary of Responses to Reviewer j3Kr

Dear Reviewer j3Kr,

We sincerely thank you for your thoughtful and constructive feedback. To facilitate a quick review, we summarize the key clarifications and additional experimental findings from our rebuttal below.

  • Genuine Visual Re-examination in CER: We clarify that Algorithm 1 contains a pseudocode error. In our actual implementation, the Proponent and Critic receive visual inputs in every reflection round, while only the Arbitrator operates on textual evidence. Moreover, uncertainty-guided temporal resampling allows agents to access previously unseen frames and re-verify unresolved visual cues. We will correct Algorithm 1 and clarify this mechanism in Section 3.3. (Please refer to Weakness 1 for details.)

  • Disentangling Target Judgment and Mask Refinement: We conduct a new 2×2 cross-factor experiment on ReasonVOS, independently controlling the trajectories used for reasoning and the masks used for output. Refined trajectories improve target discrimination by 1.7–1.8 J&F points, while refined output masks independently contribute 1.1–1.2 points. The full configuration achieves 74.3 J&F, with an oracle selection upper bound of 82.3, confirming the complementary contributions of both improvements. (Please refer to Question 1 for details.)

  • Necessity of Adversarial Role Specialization: We introduce a Two Neutral Reviewers + Arbitration + Feedback baseline under identical MLLM backbones, invocation budgets, arbitrators, feedback mechanisms, and stopping rules. Our Proponent–Critic design achieves 74.3 J&F, outperforming the neutral baseline (71.2) by 3.1 points. This controlled comparison demonstrates that adversarial role specialization provides benefits beyond general multi-pass reasoning. (Please refer to Weakness 2 and Question 2 for details.)

  • Evidence Diversity and Correlated Error Analysis: We further analyze the mechanism underlying these gains. Compared with neutral reviewers, our adversarial design reduces the cosine similarity between reasoning representations from 0.76 to 0.43 and the joint error rate from 23.2% to 7.6%. These results provide additional evidence that complementary reasoning objectives reduce correlated errors and improve decision reliability. (Please refer to Weakness 2 and Question 2 for details.)

We sincerely hope that our responses have adequately addressed the points you raised. If you have further concerns, please let us know and we will continue actively responding to your comments and improving our submission. We would deeply appreciate it if you could raise your score.

Thank you again for your valuable time and thoughtful consideration.

Best regards,

Authors

Reviewer_j1PpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CER-RVS, a training-free framework for referring video object segmentation. It introduces two components: Anchor-Guided Track Refinement (AGTR), which selects reliable instance masks and performs bidirectional re-tracking, and Contrastive Evidence Reflection (CER), which uses a Proponent, Critic, and Arbitrator to evaluate candidate tracks. Experiments on MeViS, ReVOS, and ReasonVOS show improvements over existing methods. Additional analyses examine track completeness, candidate discrimination, reasoning strategies, and inference time.

Critical strengths
  1. The method addresses two relevant failure modes. Incomplete object tracks and incorrect candidate selection can independently limit referring video segmentation. Improving both the visual evidence and its subsequent evaluation is well motivated. The framework also works without task-specific training.

  2. The paper provides useful controlled analyses. Table A1 compares reasoning strategies using the same MLLM and number of model calls. The candidate-controlled evaluation also shows improved target retention alongside reduced distractor survival. These results support a benefit beyond simply retaining more candidates or increasing the number of calls.

Critical weaknesses
  1. The methodological novelty over existing agentic video-understanding frameworks is limited. Critic-guided reasoning, iterative feedback, and separate verification roles are already explored in methods such as CAViAR, ReAgent-V, and VideoHV-Agent. Within referring video segmentation, Refer-Agent also uses reflection, while AgentRVOS already reasons over candidate tracks and revisits uncertain candidates. CER-RVS therefore primarily contributes a specific arrangement of supporting and opposing agents with an arbitrator, combined with a heuristic for selecting tracking anchors. The reported ablations support the usefulness of this arrangement, but do not establish a substantial advance over existing critic–refinement or hypothesis–verification protocols.

  2. The efficiency comparison with competing methods is insufficient. Table A3 reports that CER-RVS increases average inference time from 106 seconds for the baseline to 174 seconds per sample. This quantifies the overhead of the proposed components, but the paper does not provide a systematic comparison of end-to-end latency and computational cost against competing methods such as Refer-Agent and other verification-based approaches.

Questions for authors
  1. Novelty: What distinguishes CER’s verification protocol from existing critic-guided video-understanding methods such as CAViAR and ReAgent-V? Can you compare against an adapted critic–refinement baseline using the same candidate tracks, backbone, and inference budget?

  2. Comparative efficiency: Can you report end-to-end latency, throughput, and computational cost alongside segmentation quality for CER-RVS, AgentRVOS, and Refer-Agent under matched evaluation conditions? Please include all segmentation and language-model calls and clarify any caching or parallel execution.

  3. Reflection mechanism: Algorithm 1 specifies text-only reflection after the first round. Can subsequent rounds access visual context or acquire new observations? If not, how should “targeted re-examination” be interpreted, particularly when the initial evidence is incomplete?

2 comments
AuthorsOpenReview ↗

Brief Summary of Responses to Reviewer j1Pp

Dear Reviewer j1Pp,

We are sincerely grateful for the time and effort you have invested in reviewing our paper. To conserve your valuable time, we summarize the key points of our rebuttal below to facilitate a quick read.

  • Clarification of Methodological Novelty: Unlike existing critic-guided frameworks, CER introduces a candidate-level, conflict-driven verification protocol that combines opposing evidence, independent arbitration, and targeted reflection to prevent premature target rejection. We further conducted controlled comparisons with adapted CAViAR, ReAgent-V, and VideoHV-Agent under identical candidate tracks, backbone, and inference budgets. CER consistently outperforms the strongest alternative by 3.8, 3.3, and 3.3 J&F points on MeViS, ReVOS, and ReasonVOS, respectively. (Please refer to Weakness 1 and Question 1 in our response for details.)

  • Comparative Efficiency and Computational Cost: We conducted additional end-to-end efficiency evaluations against AgentRVOS and Refer-Agent under matched conditions, accounting for all segmentation and MLLM calls without caching. CER-RVS achieves 74.3 J&F on ReasonVOS, outperforming AgentRVOS by 7.2 points with only 10.1% additional latency, while surpassing Refer-Agent by 5.1 points with 9.8% lower latency. These results demonstrate a favorable accuracy–efficiency trade-off. (Please refer to Weakness 2 and Question 2 in our response for details.)

  • Clarification of Visual Reflection Mechanism: We identified a pseudocode error in Algorithm 1. In the actual implementation, both the Proponent and Critic receive visual inputs in every reflection round. Subsequent rounds employ uncertainty-guided temporal resampling to access previously unseen frames and re-examine unresolved evidence, while only the Arbitrator operates on textual evidence. We will correct the pseudocode and clarify this procedure in the revised manuscript. (Please refer to Question 3 in our response for details.)

We humbly hope that our responses have adequately addressed your concerns. We would deeply appreciate it if you could raise your score. If you have further concerns, please let us know and we will continue actively responding to your comments and improving our submission.

Thank you again for your valuable time and constructive feedback.

Best regards,
Authors

AuthorsOpenReview ↗

A Gentle Reminder of the Post-Rebuttal Feedback

Dear Reviewer j1Pp,

Thank you again for your time and valuable feedback on our paper. While we understand that you may have a busy schedule, we would like to confirm whether our responses have addressed your concerns. If you have any further questions or feel that we have misunderstood your feedback, please let us know. We are happy to discuss with you. Furthermore, if our responses and the revised paper have addressed your concerns, we would be grateful if you would consider updating your evaluation.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.