- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Generalized Reasoning Video Segmentation, where observed events determine a changing target set and target eligibility is separated from visibility. It builds Isekai, with 60 videos and 197 queries across 11 temporal query families, and proposes Sekai, which combines structured event memory with memory-guided localization. Sekai achieves 50.33 J&F, compared with 47.16 for the strongest reported baseline.
Critical strengths
- Separating event-based eligibility from visibility gives a clear and useful formulation of changing segmentation targets.
- Isekai's event-based annotations and Sekai's memory-free and module controls provide a useful basis for studying when historical context helps segmentation.
Critical weaknesses
- Isekai reuses masks from existing datasets (Sec. 3.1), but NEVER requires coverage of relevant nonparticipants, while EVER requires masks after an event ends. Appendix A excludes missing annotations, but the paper does not fully show how coverage was checked for these expanded requirements or which cases were excluded. This leaves uncertainty about the benchmark's coverage, rather than establishing that its labels are wrong.
- Secs. 5.3–5.4 acknowledge that a current-frame appearance cue can sometimes recover the target without event memory. The memory-free control is useful, but query-family and evidence-age scores do not show how often history must distinguish competing objects. Results around target additions, removals, and reappearances would test the claimed ability more directly than average union-mask J&F alone.
- In Fig. 6, the memory contains one event, yet the decoder outputs a target for a query requiring two occurrences. Thus, errors arise in both memory writing and memory use. Synthetic-memory training and the proposed ID readout are useful, but the current box-only output makes these errors hard to separate. Explicit target IDs and supporting events could make the selection step easier to check and improve.
Questions for authors
- How were relevant objects and post-event masks checked for completeness, especially for NEVER and EVER? Which cases were excluded because of missing annotations?
- On target-change and reappearance frames, does Sekai retain its advantage over the memory-free control? How many such transitions does Isekai contain?
- With manually checked event memories, how often does the decoder still select the wrong target set?
4 comments
Response to the authors
Thank the authors for the detailed response and the additional experiments. They have resolved most of my concerns about annotation coverage, especially the checks on NEVER-related objects and the masking applied after an EVER event ends. I suggest briefly documenting these checking rules and the main audit results in the appendix.
Table 5 shows that Sekai outperforms the no-memory control after the refresh, but is not consistently better at the exact moment a change occurs. Also, even when the memory is correct, there are still 85 out of 365 target-selection errors, which indicates that the read-out issue is not just a few isolated cases. I hope the revision retains these limitations accurately.
My current concern is the data quality of COUNT and LATEST. I would ask for an independent annotation audit targeted at these two query types: other annotators, without seeing the original labels, should independently judge the number of event occurrences and their temporal order. For example, merging two identical actions into one may not affect EVER, but it changes the answer to COUNT; if it is unclear who completed the action first, LATEST is affected. The current re-check only covers NOW, EVER, and NEVER, so the reported high agreement cannot yet be taken as evidence that these two categories are equally reliable. I hope the authors can add such targeted verification, or explicitly state this as a limitation.
Response to Follow-up Comments
Thank you for the close reading and follow-up. We have updated the paper accordingly, with today's changes marked in dark orange.
1. NEVER and EVER checks (Appendix A.2)
We added the checking rules and audit numbers to Appendix A.2. For NEVER, we check mask availability for relevant nonparticipating objects, and we now list the breakdown of the 39 out of 60 candidate queries excluded when building Isekai-60 (8 had fewer than eight frames with both prior participants and nonparticipants, 11 had unresolved scope, 18 had fewer than two candidate instances, and 2 lacked masks for same-category objects; 21 were kept). For EVER, source masks of qualifying objects are retained after the event ends, covering 1,693 query-frame records across 59 videos on Isekai-60 (1,828 object-frame records), and all 9,387 mask files passed integrity and alignment checks.
2. Transition-frame and decoder read-out limitations
Both points are stated directly in the main text. Alongside Table 5 in Section 5.3, we note that Sekai scores lower than the memory-free control at the change frame across all three transition types (62.89 vs. 65.17, 9.24 vs. 9.57, and 7.22 vs. 11.11) and higher at the first scheduled refresh (68.82 vs. 65.32, 16.18 vs. 9.79, and 31.51 vs. 30.11). In Section 6, we retain the archived-call audit on Isekai-60 showing 85 target-selection errors out of 365 calls with verified memories (23.29% across 33 videos), noting that the read-out issue is systematic rather than isolated.
3. COUNT and LATEST agreement check (Appendix A.3, Table 9b)
We compared two returned annotation files across all 9 COUNT queries (676 points) and 18 LATEST queries (1,151 points) over 26 videos in Isekai-60 (19 queries use directly annotated qualification states, and 8 are recomputed from event intervals). Comparing target sets on all frames where both files give determinate answers (excluding 2 indeterminate pairs out of 1,827) gives:
| Metric | Chunk endpoints (243 pts) | All frames (1,825 pts) |
|---|---|---|
| Exact output-set agreement | 97.12% (236/243) | 96.88% (1,768/1,825) |
| Mean output-set Jaccard | 0.9712 | 0.9688 |
| Empty/nonempty agreement | 98.77% | 98.68% |
| Empty/nonempty Cohen's | 0.9718 | 0.9717 |
All 57 disagreeing points have zero overlap (24 empty vs. nonempty, 33 selecting a different instance). This check is in a similar range to the earlier NOW/EVER/NEVER audit (93.0%, , Table 9a), though it relies on a single comparison annotator and covers Isekai-60 only.
Finally, we have moved the main tables to the 200-video version while keeping all original Isekai-60 tables in Appendix D.7 (see the general response for a full comparison). Note that the manual audits above cover Isekai-60 only, and on the 200-video set Sekai's margin over CoT-RVS is +2.07 (+3.17 on Isekai-60) with a +0.45 history-subset gain over no memory (+2.04). Thank you again for your constructive comments.
Response to the authors
Thank the authors for the further additional experiments and reasoning, which have resolved my new concern. I suggest that the revised manuscript take into account the issues raised during this round of discussion. Overall, all of my concerns have now been addressed. Together with the authors’ earlier responses to my other questions, I am keeping Rating 3.
Response to u7xw
W1/Q1: Relevant nonparticipants and post-event masks
We appreciate the distinction between insufficiently explained coverage and established label errors. Indeed, reusing source masks alone does not ensure coverage for a new query. The revision has separately explained checks of candidate scope and mask availability.
For EVER, our audit identifies 1,693 valid query-frame records across 59 videos where a confirmed qualifying event has ended but a qualifying object remains visible. These involve 1,828 object-frame records with retained source masks. For NEVER, the candidate review included relevant nonparticipants. Also, during construction we rejected 39 candidate NEVER queries out of Isekai, for their absence of same-category masks, unresolved scope and insufficient valid evaluation conditions.
We checked the integrity of 9,387 mask files and alignment of the scored records with the released labels. All 9,387 mask files passed the integrity check and the scored records align with the released labels.
W2/Q2: Additions, removals, reappearances and competing objects
Using saved predictions, we identified 205 addition, 109 removal and 40 reappearance query-frame records. Additions and removals require an object to remain visible while its eligibility changes. Reappearance requires continuously retained eligibility during temporary invisibility.
Because Sekai refreshes target prompts every eight frames, we distinguish the change frame from the first scheduled refresh at which the reference set has remained unchanged since the transition. The latter yields 179, 97 and 25 records, respectively. Removal is the hardest transition at the first scheduled refresh. Its main failure mode is stale state: previously eligible objects remain selected after their eligibility ends, as discussed in Sec. 5.3.
Per-transition J&F for Sekai and the memory-free control at both the change frame and the first scheduled refresh are reported in the revised Table 5 (Sec. 5.3).
W3/Q3: Decoder errors under manually checked memories
The counting failure in Figure 6 indeed exposes both an event-record error and a selection inconsistent with the recorded event count.
We audited 1,847 archived decoder calls from the submitted Sekai configuration on Isekai. Query-relevant memory facts were sufficient and correct in 414 calls; 365 of these also had clear instance bindings and fully assessable target sets. The decoder nevertheless selected an incorrect target set in 85 of these 365 calls (23.29%), spanning 33 videos.