acceptodds
Under review as a conference paper at ICLR 2027

Sekai: Online Temporal Reasoning for Generalized Video Segmentation

Abstract

Online reasoning video segmentation (Online RVS) requires a model to segment query-referred objects as a video streams in, using only the frames observed so far. Existing formulations largely assume a fixed target set throughout the video, overlooking a fundamental property of streaming reasoning: as new events occur, the objects that satisfy a query may change over time. To address this, we propose a new task, Generalized Reasoning Video Segmentation (GRVS), where the target set depends on query-relevant events accumulated over the observed video history. As new evidence accumulates, the target set can evolve from empty to one or multiple objects, requiring a model to continuously maintain target validity while localizing the valid objects currently in view. To evaluate GRVS, we build Isekai, a diagnostic benchmark of 60 videos and 197 queries across 11 temporal query families, with 12,569 scoring points whose target sets are causally annotated over time. We further propose Sekai, an online framework with two complementary components: an Event Memory Reasoner (EMR) that maintains query-relevant events and evolving target states in structured memory, and a Memory-Grounded Decoder (MGD) that identifies valid targets from this memory and localizes them in the current frame. Experiments show that Sekai significantly outperforms existing online RVS methods, achieving 50.33 J&F and improving over the strongest baseline by 3.2 points. Code, example annotations, and evaluation scripts are available at https://anonymous.4open.science/r/Sekai.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
reasoning video object segmentation, video object segmentation, multimodal video understanding, temporal grounding, online video understanding, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u7xwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Generalized Reasoning Video Segmentation, where observed events determine a changing target set and target eligibility is separated from visibility. It builds Isekai, with 60 videos and 197 queries across 11 temporal query families, and proposes Sekai, which combines structured event memory with memory-guided localization. Sekai achieves 50.33 J&F, compared with 47.16 for the strongest reported baseline.

Critical strengths
  1. Separating event-based eligibility from visibility gives a clear and useful formulation of changing segmentation targets.
  2. Isekai's event-based annotations and Sekai's memory-free and module controls provide a useful basis for studying when historical context helps segmentation.
Critical weaknesses
  1. Isekai reuses masks from existing datasets (Sec. 3.1), but NEVER requires coverage of relevant nonparticipants, while EVER requires masks after an event ends. Appendix A excludes missing annotations, but the paper does not fully show how coverage was checked for these expanded requirements or which cases were excluded. This leaves uncertainty about the benchmark's coverage, rather than establishing that its labels are wrong.
  2. Secs. 5.3–5.4 acknowledge that a current-frame appearance cue can sometimes recover the target without event memory. The memory-free control is useful, but query-family and evidence-age scores do not show how often history must distinguish competing objects. Results around target additions, removals, and reappearances would test the claimed ability more directly than average union-mask J&F alone.
  3. In Fig. 6, the memory contains one event, yet the decoder outputs a target for a query requiring two occurrences. Thus, errors arise in both memory writing and memory use. Synthetic-memory training and the proposed ID readout are useful, but the current box-only output makes these errors hard to separate. Explicit target IDs and supporting events could make the selection step easier to check and improve.
Questions for authors
  1. How were relevant objects and post-event masks checked for completeness, especially for NEVER and EVER? Which cases were excluded because of missing annotations?
  2. On target-change and reappearance frames, does Sekai retain its advantage over the memory-free control? How many such transitions does Isekai contain?
  3. With manually checked event memories, how often does the decoder still select the wrong target set?
4 comments
AuthorsOpenReview ↗

Response to u7xw

W1/Q1: Relevant nonparticipants and post-event masks

We appreciate the distinction between insufficiently explained coverage and established label errors. Indeed, reusing source masks alone does not ensure coverage for a new query. The revision has separately explained checks of candidate scope and mask availability.

For EVER, our audit identifies 1,693 valid query-frame records across 59 videos where a confirmed qualifying event has ended but a qualifying object remains visible. These involve 1,828 object-frame records with retained source masks. For NEVER, the candidate review included relevant nonparticipants. Also, during construction we rejected 39 candidate NEVER queries out of Isekai, for their absence of same-category masks, unresolved scope and insufficient valid evaluation conditions.

We checked the integrity of 9,387 mask files and alignment of the scored records with the released labels. All 9,387 mask files passed the integrity check and the scored records align with the released labels.

W2/Q2: Additions, removals, reappearances and competing objects

Using saved predictions, we identified 205 addition, 109 removal and 40 reappearance query-frame records. Additions and removals require an object to remain visible while its eligibility changes. Reappearance requires continuously retained eligibility during temporary invisibility.

Because Sekai refreshes target prompts every eight frames, we distinguish the change frame from the first scheduled refresh at which the reference set has remained unchanged since the transition. The latter yields 179, 97 and 25 records, respectively. Removal is the hardest transition at the first scheduled refresh. Its main failure mode is stale state: previously eligible objects remain selected after their eligibility ends, as discussed in Sec. 5.3.

Per-transition J&F for Sekai and the memory-free control at both the change frame and the first scheduled refresh are reported in the revised Table 5 (Sec. 5.3).

W3/Q3: Decoder errors under manually checked memories

The counting failure in Figure 6 indeed exposes both an event-record error and a selection inconsistent with the recorded event count.

We audited 1,847 archived decoder calls from the submitted Sekai configuration on Isekai. Query-relevant memory facts were sufficient and correct in 414 calls; 365 of these also had clear instance bindings and fully assessable target sets. The decoder nevertheless selected an incorrect target set in 85 of these 365 calls (23.29%), spanning 33 videos.

Reviewer_u7xwOpenReview ↗

Response to the authors

Thank the authors for the detailed response and the additional experiments. They have resolved most of my concerns about annotation coverage, especially the checks on NEVER-related objects and the masking applied after an EVER event ends. I suggest briefly documenting these checking rules and the main audit results in the appendix.

Table 5 shows that Sekai outperforms the no-memory control after the refresh, but is not consistently better at the exact moment a change occurs. Also, even when the memory is correct, there are still 85 out of 365 target-selection errors, which indicates that the read-out issue is not just a few isolated cases. I hope the revision retains these limitations accurately.

My current concern is the data quality of COUNT and LATEST. I would ask for an independent annotation audit targeted at these two query types: other annotators, without seeing the original labels, should independently judge the number of event occurrences and their temporal order. For example, merging two identical actions into one may not affect EVER, but it changes the answer to COUNT; if it is unclear who completed the action first, LATEST is affected. The current re-check only covers NOW, EVER, and NEVER, so the reported high agreement cannot yet be taken as evidence that these two categories are equally reliable. I hope the authors can add such targeted verification, or explicitly state this as a limitation.

AuthorsOpenReview ↗

Response to Follow-up Comments

Thank you for the close reading and follow-up. We have updated the paper accordingly, with today's changes marked in dark orange.

1. NEVER and EVER checks (Appendix A.2)

We added the checking rules and audit numbers to Appendix A.2. For NEVER, we check mask availability for relevant nonparticipating objects, and we now list the breakdown of the 39 out of 60 candidate queries excluded when building Isekai-60 (8 had fewer than eight frames with both prior participants and nonparticipants, 11 had unresolved scope, 18 had fewer than two candidate instances, and 2 lacked masks for same-category objects; 21 were kept). For EVER, source masks of qualifying objects are retained after the event ends, covering 1,693 query-frame records across 59 videos on Isekai-60 (1,828 object-frame records), and all 9,387 mask files passed integrity and alignment checks.

2. Transition-frame and decoder read-out limitations

Both points are stated directly in the main text. Alongside Table 5 in Section 5.3, we note that Sekai scores lower than the memory-free control at the change frame across all three transition types (62.89 vs. 65.17, 9.24 vs. 9.57, and 7.22 vs. 11.11) and higher at the first scheduled refresh (68.82 vs. 65.32, 16.18 vs. 9.79, and 31.51 vs. 30.11). In Section 6, we retain the archived-call audit on Isekai-60 showing 85 target-selection errors out of 365 calls with verified memories (23.29% across 33 videos), noting that the read-out issue is systematic rather than isolated.

3. COUNT and LATEST agreement check (Appendix A.3, Table 9b)

We compared two returned annotation files across all 9 COUNT queries (676 points) and 18 LATEST queries (1,151 points) over 26 videos in Isekai-60 (19 queries use directly annotated qualification states, and 8 are recomputed from event intervals). Comparing target sets on all frames where both files give determinate answers (excluding 2 indeterminate pairs out of 1,827) gives:

MetricChunk endpoints (243 pts)All frames (1,825 pts)
Exact output-set agreement97.12% (236/243)96.88% (1,768/1,825)
Mean output-set Jaccard0.97120.9688
Empty/nonempty agreement98.77%98.68%
Empty/nonempty Cohen's κ\kappa0.97180.9717

All 57 disagreeing points have zero overlap (24 empty vs. nonempty, 33 selecting a different instance). This check is in a similar range to the earlier NOW/EVER/NEVER audit (93.0%, κ=0.962\kappa = 0.962, Table 9a), though it relies on a single comparison annotator and covers Isekai-60 only.

Finally, we have moved the main tables to the 200-video version while keeping all original Isekai-60 tables in Appendix D.7 (see the general response for a full comparison). Note that the manual audits above cover Isekai-60 only, and on the 200-video set Sekai's margin over CoT-RVS is +2.07 (+3.17 on Isekai-60) with a +0.45 history-subset gain over no memory (+2.04). Thank you again for your constructive comments.

Reviewer_u7xwOpenReview ↗

Response to the authors

Thank the authors for the further additional experiments and reasoning, which have resolved my new concern. I suggest that the revised manuscript take into account the issues raised during this round of discussion. Overall, all of my concerns have now been addressed. Together with the authors’ earlier responses to my other questions, I am keeping Rating 3.

Reviewer_GpyROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new task, Generalized Reasoning Video Segmentation (GRVS), which is a kind of online video object segmentation in which the queries are more fine-grained in their temporal aspects (e.g., Now, Ever, Latest, and Never). To instantiate this task, the authors propose a new benchmark, Isekai, which consists of 60 videos and 197 queries. Furthermore, the paper introduces a new method to address the GRVS problem, Sekai, which consists of two modules: the Event Memory Reasoner, a text-based memory, and the Memory-Grounded Decoder, which uses a LoRA-adapted VLM to localize the targets that satisfy the query based on the memory. Sekai achieves state-of-the-art performance on the Isekai benchmark and competitive performance on existing reasoning video object segmentation benchmarks.

Critical strengths
  • Simple and practical method
    • Sekai only adapts a lightweight LoRA adapter on the locator while keeping the writer VLM and SAM 3.1 frozen, and its text memory has a fixed 128-token budget. This makes the approach easy to reproduce and to apply on top of existing VLMs and segmentors.
  • Strong performance
    • Sekai achieves the best overall J&F on Isekai (50.33, +3.2 over the strongest baseline) and the best results among online methods on ReasonVOS and ReVOS, showing that the approach is effective beyond the proposed benchmark.
Critical weaknesses
  1. Limited novelty and scale of the task and benchmark.
    • GRVS does not appear to be a substantially new task. Similar settings already exist in ORVOSB [1], which provides causal per-frame labels and referent shifts, and already formulates causality with past-event examples (e.g., “which watermelon has been cut”). The main additions of Isekai are multi-target annotation and explicit temporal operator families such as NOW, EVER, and NEVER.
    • Although the authors introduce a new task, the dataset (60 videos, 197 queries) is substantially smaller than related benchmarks such as MeViS v2 (2,006 videos, 33,072 expressions), Long-RVOS [2] (2,193 videos, 24,689 descriptions), GroundMoRe (1,715 clips, 7,577 questions), ReVOS (1,042 videos, 35,074 instruction–mask pairs), and even ORVOSB (210 videos, 512 queries). Furthermore, all videos are taken from existing datasets and re-annotated.
    • Isekai does not provide a training split, which limits its contribution to the field.
    • Despite the strong similarity to ORVOSB, the paper does not report results on ORVOSB and compares with ORVOS only on ReVOS.
  2. Limited methodological novelty.
    • The method mainly combines a text memory written by a frozen VLM with a LoRA-adapted VLM that predicts box prompts for SAM 3.1 at each chunk endpoint, which closely resembles existing detect-then-segment pipelines (e.g., Grounded-SAM). I could not identify a substantially novel technical component.

[1] Liu et al., Online Reasoning Video Object Segmentation, arXiv:2604.11411, 2026.

[2] Liang et al., Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation, CVPR 2026.

Questions for authors

Q1. MeViS v2 split. Could the authors clarify which MeViS v2 split the 7 Isekai videos come from? If they come from the training split, baselines trained on MeViS (e.g., StreamingRVOS, SAMWISE) may have seen these videos, which would constitute data leakage.

Q2. Related work. The related work omits standard RVOS benchmarks such as Ref-YouTube-VOS and Ref-DAVIS, as well as many recent RVOS methods. Could the authors discuss them?

Q3. Non-salient targets. All examples in Figure 6 involve a single salient actor, which a model could select even without memory. How does Sekai perform when the qualifying object is not the most salient one, e.g., among multiple similar instances?

Q4. Novelty of the task. NOW queries essentially reduce to per-frame image segmentation, and EVER queries resemble existing video segmentation with motion expressions. Where exactly does the novelty of GRVS lie beyond combining the two?

3 comments
AuthorsOpenReview ↗

Response to GpyR

W1/Q4: The contribution is not established or refuted by decomposing query names

We believe Q4 rests on a misreading of the task. Under strict causality, NOW is not per-frame image segmentation: eligibility requires that an event has been confirmed and has not been observed to end, both of which are properties of the prefix. Appendix D.4 reports 866 NOW scoring points whose target set changes when only in-window events are replayed. Likewise, EVER differs from motion expressions in MeViS in that the target set is causally time-varying and monotonically growing, and a target remains eligible after its evidence has left the observation window; this is precisely the regime in which the window-based baseline in Figure 1 fails and in which Sekai leads by 9.28 J&F (Table 3). ORVOSB, by its own definition, predicts "a binary mask corresponding to the object referred to by q" at each step (Sec. 3 of ORVOS); GRVS instead admits zero-to-many targets whose membership grows, shrinks, or switches, and decouples eligibility from visibility (Eq. 1). None of these properties follows from combining NOW and EVER.

ORVOSB availability, scale and the evaluation-only design

We agree that ORVOSB is a relevant comparison, and the submission discusses the corresponding task and method in Section 2 and Tables 1 and 4. However, the benchmark has not been released: the official repository contained no data, annotations or evaluation code when we checked at submission (Sept. 26), upon receiving the reviews (Nov. 6), and again today (Nov. 11). We will report results if the benchmark becomes available during the discussion period.

Section 3.1 explicitly discloses reuse of source videos and masks. Isekai adds event, participant and causal eligibility annotations, and is evaluation-only. Reusing source pixels does not make the event-level labels equivalent to the original masks. About the expansion of Isekai, see general response.

W2: Method contribution and controlled evidence

Grounded-SAM is a stateless detect-then-segment pipeline. Our ablations include two controls that approximate this pipeline: the memory-free locator (45.61 J&F, −4.72; −11.11 on empty-target points, Table 11) and the unadapted locator (27.66, −22.67). The gap between these controls and Sekai is the contribution: a bounded, editable event memory whose fields map one-to-one onto the task's event formalism (Sec. 4.1), a state machine whose removal makes the memory worse than none on the history subset (Table 6), and a decoder trained with synthetic memories to apply the query to the memory rather than to the chunk. We agree with the reviewer that the design is simple; we do not believe simplicity is a weakness.

Q1–Q3: Provenance, related work and non-salient targets

For Q1, all seven MeViS v2 videos come from the official valid_u validation split.

For Q2, see rebuttal revision table 15. And Table R1 for the results on Ref-YouTube-VOS and Ref-DAVIS17.

For Q3, on 2,977 nonempty scoring points from 45 videos whose ground-truth target-mask union occupies less than 5% of the frame area, Sekai obtains 58.72 J&F and CoT-RVS obtains 21.96, averaging within each video and then equally across videos. The basketball EVER example was added in the rebuttal revision Fig.6. In Frame 80, Sekai obtains 97.13 J&F and CoT-RVS obtains 0.

These clarifications separate verifiable questions about provenance and evaluation from judgments about significance. We respectfully ask that the contribution be assessed through the specific task extension, event supervision and testable selection mechanism, rather than inferred from component names or the solvability of some individual queries.

Thank you for the expanded results. The 200-video benchmark addresses some of my scale concerns, but the audit covers only the original 60 videos, and Sekai’s lead over the shared-backbone baseline narrows to 2.07 points. My concerns about the task’s novelty and the method’s contribution also remain. I therefore maintain my rating.

Our final update explicitly reported both the +2.07 J&F margin and the Isekai-60 scope of the manual audits. Your reply restates these two disclosures, but does not specify where the task distinctions, controlled ablations, or soundness checks in our November 11 response fall short. Specifically:

  1. The +2.07 overall margin on the 200-video benchmark (Table 3). Across all 35,517 scoring points, aggregate J&F (50.15 vs. 48.08) averages over opposing baseline skews: shared-backbone CoT-RVS scores high on Now (58.95) and Other (57.15) but collapses on historical retention, where Sekai leads it by +48.76 on Hist. (71.29 vs. 22.53), +23.94 on EVER (62.83 vs. 38.89), and +16.75 on LATEST (40.26 vs. 23.51). Conversely, StreamingRVOS retains history (Hist. 71.20) but drops on Now (31.08) and Other (33.25), trailing Sekai by 6.79 overall (50.15 vs. 43.36). Sekai is the only method on the 200-video set that avoids collapse across both regimes, and scaling from 60 to 200 videos without re-tuning preserved overall J&F rankings within every comparison group.
  2. Verification scope (Section 3.1; Appendices A.2–A.3, B.3). All 200 videos underwent human event review under the same annotation rules, release-gate checks, and locator training-data deduplication. Fine-grained manual audits and second-annotator agreement checks cover Isekai-60 (93.0% exact output-set agreement across 20 videos for NOW/EVER/NEVER, and 96.88% across 26 videos for COUNT/LATEST; Table 9).
  3. Task and method contributions on 200 videos (Sections 4, 5.3; Tables 3, 6). GRVS decouples prefix-confirmed eligibility from visibility over zero-to-many dynamic targets (Eq. 1); even on NOW, carrying event memory across 8-frame chunks improves over chunk-only localization (Sekai w/o memory) by +6.04 J&F (33.92 to 39.96). Against 200-video controls that reduce Sekai to stateless chunk-level detection and segmentation, event memory adds +4.33 J&F overall (+10.69 on compound Other queries), adapting the locator on writer and synthetic memories adds +18.35, and removing explicit state labels drops Hist. below having no memory (68.69 vs. 70.84).
  4. Technical soundness (Soundness: 1). Our revisions provided targeted verification of MeViS valid_u provenance across all 20 source videos and locator training-data deduplication across all 200 videos (Appendices A.2, B.3), audited causal schedules and 200-video GT-box oracle bounds (85.81 J&F; Tables 6, 11; Appendix C), causal evaluation on five existing benchmarks (Tables 4, 14, 15), competing-actor analysis (Figure 6), and explicit transition/read-out limitations (Table 5, Section 6). We respectfully ask that you specify which methodological or evidential concern remains, why the corresponding evidence is insufficient, and how it supports a Soundness rating of 1.
Reviewer_9D1xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new task Generalized Reasoning Video Segmentation (GRVS), where the target set depends on query-relevant events accumulated over the observed video history. Besides, it constructs a small benchmark Isekai with only 60 videos and a baseline method Sekai. The experiments show Sekai achieve the best J&F on its Isekai.

Critical strengths

1.This paper clearly illustrates its task, benchmark and method with detailed figures. 2.The organization of this paper is clear.

Critical weaknesses

1.Compared to benchmarks (ReVOS, MeViS) consisting of hundreds of videos in the field, only 60 videos in benchmark is too few to sufficiently illustrate and validate the challenges of the new task. Besides, the paper lacks a comparison of statistics with existing datasets. 2.The evaluation results are confused. Why does Sekai show a significant lead in only one specific dimension of the Isekai benchmark? Does this imply that a baseline equipped with only a single memory is insufficient to address the newly proposed tasks? 3.More importantly, the proposed method Sekai does not perform particularly well on MeViS. The overall benchmark in Table 3, Table 4 mixes methods with varying backbones which cannot clearly illustrate how Sekai performs. This paper also lacks evaluation results on regular benchmarks such as Ref-YTB, Ref-DAVIS.

Questions for authors

see the weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9D1x

W1: Scale and dataset statistics

We agree that 60 videos limit the breadth of conclusions. Isekai is a diagnostic evaluation benchmark with 197 fixed queries and 12,569 scored query-frame records, including 5,379 empty-target and 1,028 multiple-target records.

A statistical comparison has been added in the rebuttal revision, which distinguishes total dataset size from evaluation-split size and states the units of expressions, frames and annotated records.

Following submission, we have expanded Isekai from 60 to 200 videos (600 queries). See global response.

W2: Concentrated gains and the role of memory

The query families empirically exercise different state operations. For example, EVER retains previously qualified participants, LATEST must replace older participants with the latest ones, while NEVER is particularly vulnerable to false event entries.

Family averages also reflect different empty-target frequencies and output tendencies. Table 13 reports 75.76 J&F on empty targets and 21.45 on single targets for CoT-RVS, versus 5.63 and 70.74 for StreamingRVOS. Family averages therefore partly reflect output bias toward or against empty predictions, not only reasoning quality.

We will distinguish the expressive role of a bounded event record from the current model's ability to write, update and use it correctly. Missing events, false entries, stale states and incorrect condition execution identify limitations of the implementation. They do not by themselves establish that a single bounded event memory is inherently insufficient for the task.

W3: MeViS, backbone differences and standard benchmarks

We respectfully point to Table 14: Sekai obtains 50.45 on MeViS v2, 6.55 points above the highest reference entry in that table.

The shared-backbone and shared-segmentor comparison in Appendix C.4/Table 11 addresses part of the configuration concern. In addition, we have added the backbone and segmentor of every compared method. The new conventional-benchmark results appear in Table R1.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank the reviewers for their specific questions about benchmark coverage, target changes and memory use.

We have completed evaluations on Ref-YouTube-VOS and Ref-DAVIS17, also transition analyses using saved predictions. We first clarify three facts already present in the submission or verified from the source data.

  1. MeViS v2 results. Table 14 and Appendix D.5 report 50.45 J&F for Sekai, compared with 43.90 for the strongest reference entry in that table, LMPM++, a difference of 6.55 points. Sekai uses an online protocol, whereas others are offline.
  2. A shared-backbone comparison already exists. Appendix C.4 and Table 11 compare Sekai and our causal CoT-RVS adaptation using Qwen3-VL-8B and SAM 3.1, the same arrival schedule and the same scoring points: 50.33 versus 47.16 J&F. This fixes the backbone family and segmentor, while locator adaptation and prompting still differ.
  3. MeViS source split. All seven Isekai videos drawn from MeViS v2 belong to the official valid_u validation split. Their identifiers and provenance have been added in A.2.

Table R1. Additional conventional-benchmark evaluations. Ref-YouTube-VOS uses the official 202-video validation split, obtained by excluding the 305 test videos from the combined 507-video validation/test metadata. We evaluate all 202 videos and 834 expressions in the authors' released valid.zip, with expression-equal J&F. Sekai achieves 63.84 J&F on Ref-YouTube-VOS under strictly causal inference, without fine-tuning on this dataset; its locator is adapted on MeViS. This result is comparable to OnlineRefer (63.5), while MUTR (68.4) and SAMWISE (69.2) obtain higher scores. All three comparison methods are trained on Ref-YouTube-VOS.

BenchmarkExpressionsJ&F
Ref-YouTube-VOS83463.84
Ref-DAVIS1724470.76

Benchmark scale. We acknowledge that the released Isekai (60 videos, 197 queries, 12,569 scoring points) is a diagnostic-scale benchmark. Since submission we have continued annotation under the same protocol (Appendix A.1–A.2). Isekai-V2 currently comprises 200 videos and 600 queries, with event annotation complete and the release gate of A.2 in progress. All claims in this paper are made on the released 60-video set; we report V2 only to clarify the scope of the ongoing effort, and it will be released with the code.

Summary of changes in the revised PDF

Text revisions are highlighted in blue.

  • Additional benchmark evaluations: Added Ref-YouTube-VOS and Ref-DAVIS17 results, evaluation protocols, and comparisons with existing methods (Section 5.2; Appendix D.6, Table 15).
  • Baseline configurations: Specified backbone models and added reported results for additional model configurations (Table 4).
  • Target transitions and memory limitations: Added results for addition, removal, and reappearance at both the change frame and the first scheduled refresh, and clarified the effects of stale memory (Sections 5.2–5.3, Table 5).
  • Dataset documentation: Clarified the MeViS source split and annotation coverage for NEVER and EVER, and added dataset-size comparisons (Appendix A.2, Table 8).
  • Task positioning: Clarified the relationship to generalized referring segmentation and updated the benchmark comparison (Section 2, Table 1).
  • Qualitative example: Added a basketball example showing selection of an earlier shooter while another player currently holds the ball (Section 5.4, Figure 6).
AuthorsOpenReview ↗

Brief Follow-up on Rebuttal and Revised Manuscript

Dear Area Chair and Reviewers,

As the author–reviewer discussion period approaches its November 18 conclusion, we would like to kindly follow up on our detailed responses and revised manuscript posted on November 11.
Following your valuable suggestions, we have incorporated new evaluations on Ref-YouTube-VOS and Ref-DAVIS17, controlled baseline comparisons, clarifications on data provenance (MeViS valid_u) and NEVER/EVER annotation coverage, dataset statistics alongside our ongoing 200-video benchmark expansion, and fine-grained analyses covering target transitions, non-salient targets, and decoder errors.
We would greatly appreciate your feedback on whether these responses and revisions adequately address your concerns and inform your overall assessment of our work. If any further questions remain, we would be happy to provide additional clarification before the discussion closes. Thank you very much for your time and consideration! Best regards, The Authors

AuthorsOpenReview ↗

Final Update: Main Results Moved to the 200-Video Isekai-V2 — What Holds and What Changed

Dear Area Chair and Reviewers,

We have uploaded our final revision. New revisions are in dark orange; earlier ones are in blue.

1. Main results updated to the 200-video benchmark (Isekai-V2) Following up on the scale concerns raised by Reviewers 9D1x and GpyR, we have finished verifying the 200-video expansion (600 queries, 35,517 points; referred to simply as Isekai in the revised PDF) and updated Tables 2, 3, 6, 8, 10, and 11 accordingly. No models were retrained or re-tuned, and the benchmark remains evaluation-only. All original 60-video results (Isekai-60) remain in Appendix D.7 (Tables 16–18); both sets of numbers are shown below for direct comparison.

GroupConfiguration60-vid J&F200-vid J200-vid F200-vid J&F
MainSekai50.3348.8251.4750.15
MainCoT-RVS (shared backbone)47.1647.6048.5648.08
MainStreamingRVOS41.2841.5045.2343.36
MainSAMWISE35.3036.9340.3038.62
MainOnlineRefer30.7332.0637.1134.59
AblationNo state labels48.5748.0350.6049.31
AblationSekai w/o memory45.6144.4147.2245.82
AblationUnadapted locator27.6629.4134.1931.80
SegmentorSAM2.1 (Hiera-Large)47.9247.8850.4949.19
SegmentorSEEM (FocalT-v0)29.2630.2829.2529.76
SegmentorGrabCut28.6730.8027.4929.14
SegmentorFilled rectangle24.4827.4118.2322.82
OracleGT boxes + SAM 3.187.8085.0886.5585.81

What holds. Within-group J&F rankings are unchanged. Removing memory drops J&F by 4.33 (4.72 on Isekai-60), removing locator adaptation drops 18.35 (22.67), and using SAM2.1 costs 0.96 (2.41). Memory without state labels still underperforms no-memory on Hist. (68.69 vs. 70.84; 62.87 vs. 63.93). GT boxes reach 85.81 (87.80), confirming that target selection remains the main bottleneck.

What shifts.

  • The lead over shared-backbone CoT-RVS narrows from +3.17 to +2.07.
  • On Hist., the gain over no-memory shrinks (+2.04 to +0.45), while the gap to StreamingRVOS turns positive (−2.74 to +0.09).
  • The EVER lead over StreamingRVOS moves from +9.28 to +6.44; LATEST/NEVER deficits vs. no-memory remain (−4.16/−6.47 vs. −5.97/−10.51).
  • CoT-RVS's lead on Other grows (5.21 to 10.83), and some per-family rankings shift (e.g., no-memory leads NEVER).

2. Audits and protocol notes

  • Audit scope (A.2–A.3): Manual audits cover Isekai-60 only (the 140 added videos were checked under the same annotation rules without second-annotator re-annotation; source/family breakdowns are in Tables 2, 7, and A.2). Table 9b adds a 27-query COUNT/LATEST check prompted by Reviewer u7xw (96.88% exact agreement over 1,825 points; see our reply to u7xw).
  • Other details: Appendix D.6 documents local causal scoring on Ref-YouTube-VOS (202 videos, 834 expressions: 63.84 untrained vs. 63.5–69.2 trained). Transition-frame (Table 5) and decoder read-out (Sec. 6) limitations remain in the main text, and earlier Isekai-60 numbers map to Tables 16–18.

Code, evaluation scripts, and the full 200-video annotations will be released. Thank you to the AC and reviewers.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.