acceptodds
Under review as a conference paper at ICLR 2027

Learning to Reason with Persistent Object States for Video Instance Segmentation

Abstract

Video segmentation models preserve object identities by carrying information about each instance across frames. They typically update this history as new observations arrive. Under prolonged occlusion, reappearance, or interactions between similar instances, however, an incorrect update can overwrite a valid history and turn a local mismatch into persistent identity drift. This raises a question that current pipelines largely leave implicit: when should an observation be allowed to change an object's state? We introduce **POSReasoner**, a trainable, plug-and-play framework that makes this decision explicit by reasoning over persistent object states. For each object, POSReasoner carries a compact state that records identity, geometry, visibility history, and confidence. A sparse interaction graph links these states to candidate observations. Over several reasoning steps, POSReasoner compares each candidate with the object's history, resolves conflicting associations, and determines whether the state should be retained, updated, or reactivated. Intermediate supervision guides association and presence prediction, while a learned gate blocks unreliable updates. The resulting state is carried forward, so each decision affects subsequent association and segmentation rather than merely reranking a fixed set of predictions. POSReasoner is trained with standard video annotations while the base model remains frozen. This lightweight design allows it to augment diverse VOS and VIS architectures. Across long-term VOS and VIS benchmarks, POSReasoner consistently improves strong baselines, with the largest gains under occlusion and object reappearance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
computer vision, video understanding, representation learning, video segmentation, object tracking, temporal reasoning, video object segmentation, video instance segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WxxSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes POSReasoner, a trainable adapter for maintaining object identities in video segmentation with a frozen host. Persistent identity states are associated with candidate observations through a sparse graph. A shared Propose–Verify procedure estimates associations, presence, and transition actions before a gated write updates the state. Experiments cover several VIS hosts on YouTube-VIS and OVIS, and a SAM3-based VOS configuration on LVOS.

Critical strengths

The separation of association from state admission is useful, and the shuffled-state and host-cue controls provide evidence about the value of identity-specific history

Critical weaknesses
  1. The implementation description does not establish which experiments exercise the full recurrent mechanism. Section 3 states that verified transitions influence subsequent association and segmentation, whereas Appendix B.1 describes frozen trajectory inputs, Appendix B.2 describes VOS as replaying frozen identity-indexed masks with fitted or calibrated policies, and some VIS hosts use a compact readout without frame-wise identity sequences. Fixed candidate masks do not inherently invalidate recurrent reasoning, but the paper needs to specify what actually changes at each frame and how the stated losses, transition advantage in Eq. (3), and write rule are realized in each implementation. Otherwise, the results do not clearly distinguish the proposed executor from learned selection or rescoring of exported trajectories.

  2. Evidence for the specific long-gap recovery claim is less decisive than the aggregate results suggest. Every paired confidence interval in Table 6 includes zero, and LVOS v2 shows negative mean changes for 5–10-frame and over-100-frame gaps; the longest-gap groups contain only 6 and 9 events. The aggregate LVOS improvement remains useful, but these diagnostics do not establish reliable improvement after prolonged absence.

Questions for authors
  1. Which host configurations execute the full recurrent transition model, and how do the replay-based VOS and compact VIS readouts implement its state updates and supervision?

  2. Given the confidence intervals and negative LVOS v2 subsets in Table 6, what conclusion do you draw specifically about recovery after long absences?

Reviewer_RTrNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper works towards enhancing identity conservation in VOS and VIS by addressing the identity persistence drift due to incorrect updates to persistent memory, which is caused by scenes with occlusions, reappearance or similar-object interactions. This issue is addressed with POSReasoner, a trainable two-stage plug-and-play framework with a Purpose-Verify procedure to propose associations by building a sparse observation map to refine and verify before performing a gated memory update, while keeping the host model frozen. Experiments and ablations across datasets show a consistent improvement in performance while supporting the use of the proposed persistent states and reactivation.

Critical strengths
  1. The problem of incorrect identity updates propagating identity errors across subsequent frames is a clear and relevant problem in video segmentation. The proposed method to improve temporal identity consistency is intuitive, with its separation of object association and memory association.

  2. PosReasoner has shown to be an adaptable framework with its ability to work well across VOS and VIS architectures without host model retraining, while achieving consistent improvements across benchmarks with clear gains in the occlusion-heavy OVIS benchmark.

  3. Proposed components are cleanly investigated and analyzed with clear ablations that display the importance of identity-specific historical information, persistence states and reactivation.

Critical weaknesses
  1. The idea on when an observation should update an object's memory has been explored in prior works like LTMU (Dai et al., CVPR 2020), which explicitly learns update readiness; LOMM (Lee et al., ICCV 2025), which both maintains a presence-aware object memory and handles object association; and recent works like SAM3-DMS (Shen et al., 2026) that explore object-specific memory selection. While the proposed two-stage association and reasoning remain distinct, its advantage over the existing approaches are not sufficiently demonstrated.

  2. With the main motivation of the paper being object persistence, the current gating mechanism is designed to suppress uncertain updates. But looking into the gating mechanism, it seems to suggest that it does not necessarily prevent incorrect updates when the model is confidently wrong. A clear ablation and failure case study is needed to best analyze the characteristics of the proposed gating mechanism.

  3. While the paper makes it a focus to work on prolonged occlusion and object reappearance, Table 6 seems to point that the largest gains come from intermediate gaps while getting smaller or even losing in longer gaps. This makes it difficult to check if the proposed framework works primarily on short-term associations or extends well to long-term absences.

Questions for authors
  1. How is the proposed method different from prior works like LTMU or LIMM?

  2. How does the proposed method handle confidently incorrect associations that could still pass through the learned write gate?

  3. Given that the motivation of the paper is towards long-term identity maintenance, why are the strongest reappearance concentrated in the intermediate interval gaps, with limited or negative gains for longer absences? Can the authors provide further analysis of the failure cases and clarify how the wide confidence intervals in Table 6 support claims made on long-term recovery?

Reviewer_a6qzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to determine the association between current predictions and historical objects in video segmentation, and whether these predictions should update historical identity states to reduce persistent identity drift.

The authors formulate long-term identity maintenance as reasoning over persistent object states and propose POSReasoner, a trainable, plug-and-play framework for VOS and VIS. The framework consists of four main components: persistent object states, a sparse state–observation graph, Propose–Verify reasoning, and selective state updates. These components jointly address object association and historical state updates.

Experiments on YouTube-VIS, OVIS, and LVOS demonstrate performance improvements across multiple host models. The authors also provide analyses of occlusion, object reappearance, and component ablations, although the improvements vary under different reappearance conditions.

Critical strengths

1. The paper extends the object association problem in video segmentation to determining whether current observations should be accepted to update historical states. By integrating identity preservation, association competition, lifecycle reasoning, and selective memory updates into a unified framework, the proposed method provides a more systematic approach to long-term identity maintenance than methods focusing only on frame-level association.

2. POSReasoner can be integrated with different VIS/VOS host models without updating their original weights. Experiments on YouTube-VIS, OVIS, and LVOS demonstrate consistent performance improvements, supporting the effectiveness and applicability of the proposed framework.

Critical weaknesses

1. Insufficient direct validation of the core mechanism. Although the paper demonstrates improvements through component ablations, controlled experiments, and reappearance analyses, these experiments mainly reflect the contributions of different components to overall segmentation performance. They do not sufficiently verify whether the improvements primarily come from reducing incorrect associations and unreliable state updates. Quantitative analyses of association errors, incorrect memory updates, and subsequent identity drift are missing. Therefore, it remains difficult to distinguish the benefits of persistent state maintenance from the independent contribution of selective memory updates.

2. Insufficient evaluation of plug-and-play adaptability and cross-host generalization. Although POSReasoner keeps the host models frozen, its VIS and VOS implementations still require additional training or calibration. The experiments mainly demonstrate performance improvements after adaptation to different hosts, but do not show whether the same trained Reasoner can be directly transferred to other hosts without retraining. Moreover, although the paper reports the parameter count and inference latency, the adaptation and training costs for different hosts are not sufficiently quantified. These limitations leave the practical applicability of the plug-and-play design unclear.

Questions for authors

Q1. Could the authors report association error rates, identity switches, and performance improvements under different occlusion levels, numbers of similar objects, and absence durations? Such analyses would help verify whether the method mainly improves identity maintenance in challenging scenarios.

Q2. Although the paper provides ablations on state maintenance, reactivation, and one-step/two-step reasoning, the independent contributions of the Learned Write Gate, Transition Advantage, and Null Observation remain unclear. Could the authors provide separate ablations of these components to clarify where the performance improvements come from?

Q3. The write gate of POSReasoner depends on state-transition probabilities and association confidence. When an incorrect association has high confidence, can the gate still prevent unreliable information from being written into the persistent state with a large weight? A controlled experiment on high-confidence incorrect associations would help evaluate their impact on subsequent identity drift.

1 comment

Thank you to all the reviewers for your time and thoughtful feedback. With the discussion deadline approaching, please let us know if you have any remaining questions or concerns. We’d be happy to clarify.

Comments on the paper

1 comment

Thank you again to all reviewers for your time and thoughtful feedback. As the discussion period is approaching its deadline, we would greatly appreciate it if you could let us know whether there are any remaining questions or concerns regarding our response or the paper. We are happy to provide further clarification, additional details, or supporting evidence as needed. We hope our rebuttal has adequately addressed the points raised, and we sincerely appreciate your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.