acceptodds
Withdrawn from ICLR 2027

SEVA: Self-Evolving VLM Agent for Video Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing video object segmentation (VOS) methods process each video independently and discarding accumulated knowledge and re-initializing object representations from scratch, unlike human visual tracking. Separately, self-evolving agent frameworks have shown strong iterative improvement in coding and text domains, but remain unexplored for VOS tasks. We propose SEVA (Self-Evolving VLM Agent), which reformulates tracking as a continuous, memory-augmented reasoning process driven by a vision-language model. SEVA maintains evolving object representations across frames and videos, produces interpretable frame-by-frame reasoning, and self-corrects over batches of videos without additional fine-tuning, reporting a transparent summary of what it has learned after each batch. Evaluated on video benchmarks emphasizing 3D spatial reasoning under occlusion and viewpoint change, SEVA achieves state-of-the-art performance while showing measurable improvement as it observes more videos, demonstrating that self-evolving agentic reasoning extends applicably across the visual task domain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent, VLM, video segmentation, video tracking, self-evolving

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_aRjWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SEVA, a self-evolving vision-language agent for video object segmentation (VOS). The framework introduces three nested feedback loops: an inner loop for mask verification and local repair, a middle loop for adapting inference strategies based on recent failures and retrieved experiences, and an outer loop for updating prompts and policies through accumulated feedback. All backbone models remain frozen, allowing the system to evolve without additional model training.

Critical strengths

Interesting framework for adaptive video segmentation.

The paper introduces a three-loop framework that integrates self-verification, local repair, and cross-video experience accumulation. The overall design is reasonable and addresses practical challenges in video segmentation.

Promising experimental performance.

SEVA achieves competitive results on several VOS benchmarks, particularly MeViS and Refer-DAVIS-17, without additional model training.

Detailed implementation and failure analysis.

The paper provides implementation details and analyzes several practical failure cases, including unsuccessful design alternatives. These discussions offer useful insights into the proposed framework.

Critical weaknesses

Lack of reliable feedback for self-evaluation.

It remains unclear how SEVA can reliably determine whether previous segmentation results are correct without ground-truth annotations. Relying on the VLM's own judgments and indirect signals such as repair counts may introduce unreliable feedback and reinforce existing errors.

Limited novelty and insufficient evidence of self-evolution.

The proposed self-evolution mainly consists of updating prompts, sampling strategies, and thresholds, which closely resembles conventional prompt or policy optimization. Moreover, the experiments do not sufficiently demonstrate consistent improvements as the system accumulates experience.

Inconclusive ablation results.

In Tables 4 and 5, disabling self-verification does not reduce segmentation performance, while disabling episodic memory slightly improves it. These results raise questions about the actual contributions of the proposed components.

Limited evaluation of computational efficiency.

SEVA introduces additional verification and repair operations, resulting in considerable inference overhead. However, the paper lacks a sufficiently controlled comparison of computational costs against the baseline, making its practical efficiency difficult to assess.

Questions for authors
  • How does SEVA reliably evaluate segmentation correctness without ground-truth annotations during real-world deployment? Have the authors evaluated the reliability of its self-verification signals?

  • What distinguishes the proposed self-evolution mechanism from conventional prompt or policy optimization? Can the authors provide quantitative evidence of progressive improvement on unseen videos across evolution rounds?

  • How do the authors explain the ablation results where disabling self-verification does not reduce performance and removing episodic memory slightly improves it?

  • Can the authors provide a controlled comparison of inference latency, computational costs, and segmentation accuracy against the baseline?

Reviewer_B5FDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SEVA implements a video object segmentation agent by combining a frozen vision-language model, an object detector, and SAM2. The agent employs mechanisms for in-video verification and correction, adaptation based on recent experiences, and an external loop that evaluates candidate objects, derived from separate datasets, to refine its prompts and policies. The experiments cover both reference-based and inference-based segmentation, with results analyzed through component ablation studies and evaluations of inference costs. The core argument of this research is that an agent with frozen weights can improve its performance across episodes, rather than treating each video in isolation, by leveraging a persistent procedural memory.

Critical strengths
  1. Separating recovery within a given video from policy modification across videos is a useful approach to system design. The use of candidate policy verification and an explicit recovery budget provides a more concrete adaptation mechanism than an unconstrained agent loop, while the reliance on fixed components reduces the need for additional model training.

  2. This paper reports the results of ablating key components, including some negative outcomes, and provides empirical measurements of latency and call counts in Appendix A.10. Removing the outer loop improved the J&F score from 0.546 to 0.602 or 0.620 (depending on the promotion criteria), offering encouraging evidence that policy adaptation can be significant in the environments studied.

Critical weaknesses
  • The evaluation protocol lacks sufficient clarity, making it impossible to ensure independent generalization across episodes or to derive comparable benchmark results. A footnote in Appendix A.1 describes a full MeViS split consisting of 473 videos, whereas Table 7 lists 50 videos and 186 evaluation pairs used for the official MeViS test split, along with a subset comprising 8 videos. While these may represent distinct data streams, their relationship to the primary results table and the "promotion" (model advancement/update) experiments is not specified. Section 3.4 and Appendix A.5 indicate that ground-truth results are used for failure logging, while held-out data is used for promotion; consequently, merely keeping weights fixed is insufficient to prove that label-based adaptation has been excluded. A clear explanation is required regarding the separation of source videos across experience collection, promotion validation, and final evaluation, as well as the policy reset or reuse procedures employed throughout the benchmark. Although we do not assert that data leakage has occurred, the current description is insufficient to rule it out or to allow for the reproduction of the reported comparative results.

  • The evidence supports a more limited scope of adaptive benefits rather than the broader claim of "extensive memory/self-evolution." Tables 4 and 5 show that disabling episodic memory actually increases the J&F score from 0.546 to 0.554, while disabling only the validation step leaves the baseline accuracy unchanged, though the text notes that this reduces the computational cost by approximately half. While the positive outcomes from the outer loop are useful, a single aggregate result is insufficient to demonstrate reliable improvement derived from cumulative experience across a sequence of streams, nor can it be distinguished from the effects of selecting a superior static prompt. Appendix A.12 acknowledges that a cost-equivalent comparison is not feasible.

Questions for authors
  1. Appendix A.1 mentions a MeViS stream consisting of 473 videos, while Table 7 lists 50 official test videos. Please map each key result and ablation to the corresponding sets—namely, the experience, promotion, and final evaluation sets—and identify any overlaps with the source videos. (Weakness 1)

  2. Section 3.4 and Appendix A.5 allow for the use of ground-truth scores in failure records. At what stage are these scores or masks accessible, and when are the memory and policy states reset? Please distinguish between "stream-adapted" evaluations (which utilize these scores) and evaluations performed on unmodified videos. (Weakness 1)

  3. The "outer-loop gain" shown in Table 4 might simply reflect the selection of a better, fixed policy. Is it possible to implement a control experiment using a static policy, selected with the same candidate search budget and access to validation data, and evaluate it on the same independent set of videos? (Weakness 2)

  4. Table 4 reports aggregated results rather than learning curves. Please show performance on the hold-out set as a function of the number of episodes processed—including both the initial and promoted policies—and, if possible, illustrate any variations based on the stream sequence. Please also clarify whether only a single test sequence was used. (Weakness 2)

  5. Removing the validation step leaves the baseline J&F score unchanged, yet the "strict" outer-loop performance drops from 0.620 to 0.521 (Tables 4–5). Was the validation step removed from the policy configuration phase, the final inference phase, or both? Furthermore, how do these variants compare in terms of accuracy and the number of function calls? (Weakness 2)

  6. Table 8 reports the cost of pairwise inference without accounting for the overhead associated with the evolutionary process. Does this include candidate generation, the execution of promotions, and rejected candidates? If these are excluded, please report the adaptation costs separately and explain how they are distributed across the evaluated pairs. (Weakness 2)

Reviewer_PgX7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a self-evolving agent for referring and reasoning video object segmentation that adapts at test time, learning from mistakes on past videos without any weight updates. The method wraps existing foundation models (Qwen3-VL-8B, Grounding DINO, SAM 2) in three nested loops that implement self-verification and repair within a video, session-level adaptation, and memory-based cross-video prompt evolution. The paper claims state-of-the-art performance on multiple video segmentation benchmarks.

Critical strengths

The idea of a self-evolving video segmentation agent is reasonable.

Critical weaknesses

1. Presentation of extremely poor quality. The overall description of the methodology and experiments is largely vague and under-specified. The paper mentions a VLM, SAM 2, and Grounding DINO, but their positioning within the proposed architecture is completely unclear. The difference between the three introduced loops is also cryptic; for example, the caption of Fig. 2 introduces "three nested loops", but only the inner and the outer are named.

The few technical sections of the paper contain undefined symbols (F\mathcal{F}, wfw_f, and dd in the pair (d,τc)(d, \tau_c)), symbols which are never used (ϕn\phi_n), inconsistent naming (τc\tau_c, τρ\tau_\rho are used in the main text, while c∗c^*, ω\omega are used in the appendix), and undefined terms ("seed-and-propagate"). The paper contains several non-existent references (Tab. 6 refers to hyperparameters from absent Eqs. 9, 10, and 13) and a malformed bibliography (MeViS, TrackGPT, VideoSeg-R1, and more, which either omit real authors or include fictional ones).

The appendix seems mostly LLM-generated, as it includes code-specific references, such as seva/tools/continuity_detector.py and version = current.version + 1, making it completely unreadable, especially as the code itself is not provided. All this despite the authors' Reproducibility Statement including "We aim to make SEVA fully reproducible" and AI Statement stating "We reviewed all AI-assisted content". Moreover, no prompt templates are specified even though the introduced method is mostly a designed harness around an existing MLLM.

Critically, the ablations are insufficiently described; for example, in the "strict promotion (τ=0.03\tau = 0.03)" row of Tab. 4, τ\tau is never defined except in Tab. 6, which back-references Tab. 4. All the aforementioned issues, whose list is far from complete, thus raise significant concerns about the soundness of the reported results, which can hardly be reproduced.

2. Unsupported claims.

  • First, the central claim about the method "showing measurable improvement as it observes more videos" is not supported by the experiments: the J&F vs. number of processed videos announced in §4.1 is never reported.
  • Second, the ablations show that disabling self-verification performs equally well, while it is supposed to be one of the main contributions of the paper.
  • Also, Tabs. 1 and 2 state that the introduced method is online, unlike some other methods. However, the method relies on bidirectional propagation, which the ablations show is a crucial component, and this contradicts the method being online.
  • Moreover, the paper claims to reach the state of the art, but it omits a comparison with CoT-RVS-Gemma3-12B and CoT-RVS-GPT-4o in Tab. 2, which, according to the CoT-RVS paper, achieve better results there. Meanwhile, they compare with CoT-RVS-Gemma3-12B in Tab. 3, where it performs worse than their method.
Questions for authors

The main concerns are not resolvable within the scope of a rebuttal to change my rating. They require a substantial revision of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.