- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents VideoSEAM, a training-free framework for long-form video question answering through sequential, query-independent memory construction. VideoSEAM propagates a bounded Transition Memory containing entity references, recent events, and hierarchical summaries to condition the interpretation of subsequent video windows. The extracted entity-event records are persistently stored in an Episode Memory, which a downstream reasoning agent accesses through memory retrieval and targeted source-video inspection. Experiments on MM-Lifelong, EgoLifeQA, and LVBench demonstrate improvements over several existing video understanding systems, with additional ablations investigating memory components and the QA agent.
Critical strengths
- The paper is well-organized and easy-to-follow.
- The proposed framework achieves improvements over several baselines across multiple video QA benchmarks.
Critical weaknesses
- The paper claims that history-conditioned memory construction improves video QA through cross-window entity-event coherence. However, the ablations in Table 3 conflate this mechanism with additional contextual information. Removing Transition Memory eliminates both entity references and historical context, while removing entities also discards useful visual and textual information. Based on this concern, the contribution of coherent memory construction remains unclear.
- Insufficient comparison with closely related methods. MemDreamer, which reports stronger LVBench results (90.7%), is omitted from the comparison. While its unreleased code makes direct reproduction difficult, the authors should discuss this performance gap (90.7% vs. 79.5%) and clarify potential differences in evaluation protocols. Moreover, the included ReMA and WorldMM baselines are evaluated using different VLM backbones.
- Another concern is that VideoSEAM primarily propagates entity references between adjacent windows, but what if an entity disappears and reappears after several windows? Could this lead to inconsistent identity assignments and significantly degrade downstream QA performance? The reported reuse precision (78.8%) does not capture such identity fragmentation.
- The fixed 300-second window segmentation appears heuristic. How sensitive is the proposed memory construction to different window sizes and boundaries? Could splitting the same event across different windows affect entity-event coherence and downstream QA accuracy?
Questions for authors
- Could the authors provide a controlled comparison using the same Gemini 3.1 Pro backbone, video sampling, memory token budget, and downstream QA pipeline (including retrieval, video inspection, and reasoning)? Specifically, compare full VideoSEAM against a history-conditioned baseline that retains the same entity descriptions, visual references, event histories, and summaries, but uses window-local entity IDs without explicit cross-window alignment. This would more directly verify the benefit of the proposed coherence mechanism.
- Could the authors clarify the performance gap between VideoSEAM and MemDreamer on LVBench?
- Could the authors provide same-backbone comparisons with compatible, publicly reproducible training-free methods, such as ReMA, WorldMM, or VideoARM, under comparable inference budgets?
- Could the authors evaluate identity consistency and QA performance across different entity reappearance intervals?
- Could the authors provide a sensitivity analysis regarding different window sizes and boundaries during memory construction?
3 comments
"Could the authors provide same-backbone comparisons with compatible, publicly reproducible training-free methods, such as ReMA, WorldMM, or VideoARM, under comparable inference budgets?" (Q3)
We evaluated VideoSEAM, VideoARM, and WorldMM on MM-Lifelong with Gemini 3.1 Pro as their common multimodal backbone. For VideoARM, we used Gemini for the controller and multimodal tools and retained its original audio transcription setup. For WorldMM, we used Gemini for memory construction and QA and retained its original embeddings. Each system used its own retrieval and QA procedure.
Table R2. MM-Lifelong results with Gemini 3.1 Pro as the common multimodal backbone. ACC and Ref@300 are reported in %.
| Method | Test@Day ACC | Test@Day Ref@300 | Test@Week ACC | Test@Week Ref@300 | Val@Month ACC | Val@Month Ref@300 |
|---|---|---|---|---|---|---|
| VideoSEAM | 44.0 | 38.6 | 26.0 | 37.3 | 22.5 | 36.0 |
| DVD | 39.3 | — | 22.0 | — | 17.2 | — |
| VideoARM | 35.5 | 9.4 | 21.0 | 10.5 | 19.7 | 8.6 |
| WorldMM | 11.8 | 3.8 | 25.5 | 15.7 | — | — |
VideoSEAM outperforms VideoARM on both metrics across all three splits (Table R2). Compared with WorldMM, VideoSEAM achieves 32.2 and 34.8 pp higher ACC and Ref@300, respectively, on MM-Lifelong Test@Day. On Test@Week, the two systems are close in ACC (26.0% vs. 25.5%), while VideoSEAM leads by 21.6 pp in Ref@300 (37.3% vs. 15.7%). The WorldMM Val@Month run did not finish within the available evaluation time.
"The reported reuse precision (78.8%) does not capture such identity fragmentation." (W3)
"Could the authors evaluate identity consistency and QA performance across different entity reappearance intervals?" (Q4)
The reported 78.8% reuse precision measures how often an existing ID was assigned correctly. To quantify identity fragmentation, we audited 399 generated nodes and found that 16 of 349 ground-truth identities (4.6%) were split across multiple IDs, producing 49 redundant nodes.
We then examined how often an entity's ID changed between successive appearances, grouping transitions by the number of intervening windows in which the entity was absent. The ID changed in 46 of 440 unambiguous transitions. The change rate rose from 0.5% (2/371) for appearances in consecutive windows to 80.0% (32/40) after at least 11 intervening windows. This pattern suggests that maintaining identity across long gaps is difficult, although the longest-gap group contains only 40 transitions. We will discuss representative cases of long-gap fragmentation in the supplementary material.
We manually labeled 200 questions according to whether answering them required identifying a specific entity. VideoSEAM achieved 45.1% accuracy on the 152 entity-dependent questions and 40.6% on the 48 questions without a specific entity dependency. This 4.5-point difference describes performance by entity dependency. How QA accuracy varies with the entity's reappearance interval remains open.
"Could splitting the same event across different windows affect entity-event coherence and downstream QA accuracy?" (W4)
"Could the authors provide a sensitivity analysis regarding different window sizes and boundaries during memory construction?" (Q5)
W4. Events that cross a window boundary. The default construction window is 300 seconds. If an event spans a boundary, its observations can enter Episode Memory as separate, time-stamped event records. The next window receives recent events and active entity references, allowing it to interpret the continuation in context (Sec. 3.2.3). This allows VideoSEAM to maintain a coherent understanding of events that span multiple windows.
Q5. Sensitivity to window length and boundary placement. Table R3 compares three construction-window lengths on MM-Lifelong Test@Day.
Table R3. Construction-window ablation on MM-Lifelong Test@Day.
| Window length | ACC (%) | Ref@300 (%) |
|---|---|---|
| 150 s | 42.5 | 30.4 |
| 300 s (default) | 44.0 | 38.6 |
| 600 s | 44.0 | 39.2 |
Shortening the default window from 300 s to 150 s reduces ACC by 1.5 pp and Ref@300 by 8.2 pp. The 300 s and 600 s windows both achieve 44.0% ACC, with a small increase in Ref@300 from 38.6% to 39.2% at 600 s. Given that a 600-second window substantially increases inference latency, we adopt a 300-second window to balance accuracy and computational efficiency.
W1 / Q1. Comparison with a window-local memory baseline
We evaluated a window-local memory baseline on MM-Lifelong Test@Day. It constructs memory independently for each window, assigning local IDs to entities and events and linking each event to its participating entities. This baseline achieves 40.3% ACC and 34.2% Ref@300. Full VideoSEAM achieves 44.0% ACC and 38.6% Ref@300, gains of 3.7 and 4.4 pp, respectively. These results show that carrying information across windows improves both answer accuracy and reference grounding.
W2 / Q2. Comparison with MemDreamer on LVBench
We evaluated MemDreamer using its previously released code and the provided LVBench memory database. After fixing a ground-truth leakage issue in the released inference code, we ran MemDreamer with Gemini 3.1 Pro on the same 1,534 questions used to evaluate VideoSEAM. In our reproduction, MemDreamer achieves 79.9%, compared with 79.5% for VideoSEAM (Table R1).
Table R1. LVBench accuracy (%), grouped by evidence duration.
VideoSEAM leads our MemDreamer reproduction by 1.5 pp on the Long subset and by 1.5 pp on questions whose evidence spans more than 30 minutes. It trails MemDreamer by 0.2 pp on Short and 1.4 pp on Medium. This pattern is consistent with VideoSEAM's focus on tracking entities and events across windows. Questions with shorter evidence spans (< 5 min) may depend less on recognizing the same entity across appearances or following an event across window boundaries.
In addition, since MemDreamer does not release its memory construction code, we could not extend the comparison to MM-Lifelong to evaluate both methods on substantially longer videos (> 24 hours).