acceptodds
Under review as a conference paper at ICLR 2027

VideoSEAM: Stitching Coherent Video-Level Memory for Long-Form Video Understanding

13 views

Abstract

Long-form video understanding requires reasoning over temporally distributed entities, events, and interactions. Existing training-free methods broadly follow either query-conditioned approaches, often relying on sparse frame search that may miss critical moments, or query-independent approaches, where independently processed local windows can lead to inconsistent identities and fragmented events. We introduce VideoSEAM, a training-free framework for query-independent sequential video memorization. Rather than constructing local records independently and recovering their relations downstream, VideoSEAM maintains a coherent entity-event state during sequential processing and directly produces structured records with consistent cross-window references. A dual-horizon memory separates a bounded multimodal transition memory for continued understanding from a persistent episode memory that preserves structured history for future queries. The resulting memory supports a memory-grounded reasoning agent for retrieval, bounded video inspection, and answer generation. Experiments on MM-Lifelong, EgoLifeQA, and LVBench demonstrate strong performance against existing training-free long-video methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Form Video Understanding, Multimodal Agents, Long-Term Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BLavOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VideoSEAM, a training-free framework for long-form video question answering through sequential, query-independent memory construction. VideoSEAM propagates a bounded Transition Memory containing entity references, recent events, and hierarchical summaries to condition the interpretation of subsequent video windows. The extracted entity-event records are persistently stored in an Episode Memory, which a downstream reasoning agent accesses through memory retrieval and targeted source-video inspection. Experiments on MM-Lifelong, EgoLifeQA, and LVBench demonstrate improvements over several existing video understanding systems, with additional ablations investigating memory components and the QA agent.

Critical strengths
  1. The paper is well-organized and easy-to-follow.
  2. The proposed framework achieves improvements over several baselines across multiple video QA benchmarks.
Critical weaknesses
  1. The paper claims that history-conditioned memory construction improves video QA through cross-window entity-event coherence. However, the ablations in Table 3 conflate this mechanism with additional contextual information. Removing Transition Memory eliminates both entity references and historical context, while removing entities also discards useful visual and textual information. Based on this concern, the contribution of coherent memory construction remains unclear.
  2. Insufficient comparison with closely related methods. MemDreamer, which reports stronger LVBench results (90.7%), is omitted from the comparison. While its unreleased code makes direct reproduction difficult, the authors should discuss this performance gap (90.7% vs. 79.5%) and clarify potential differences in evaluation protocols. Moreover, the included ReMA and WorldMM baselines are evaluated using different VLM backbones.
  3. Another concern is that VideoSEAM primarily propagates entity references between adjacent windows, but what if an entity disappears and reappears after several windows? Could this lead to inconsistent identity assignments and significantly degrade downstream QA performance? The reported reuse precision (78.8%) does not capture such identity fragmentation.
  4. The fixed 300-second window segmentation appears heuristic. How sensitive is the proposed memory construction to different window sizes and boundaries? Could splitting the same event across different windows affect entity-event coherence and downstream QA accuracy?
Questions for authors
  1. Could the authors provide a controlled comparison using the same Gemini 3.1 Pro backbone, video sampling, memory token budget, and downstream QA pipeline (including retrieval, video inspection, and reasoning)? Specifically, compare full VideoSEAM against a history-conditioned baseline that retains the same entity descriptions, visual references, event histories, and summaries, but uses window-local entity IDs without explicit cross-window alignment. This would more directly verify the benefit of the proposed coherence mechanism.
  2. Could the authors clarify the performance gap between VideoSEAM and MemDreamer on LVBench?
  3. Could the authors provide same-backbone comparisons with compatible, publicly reproducible training-free methods, such as ReMA, WorldMM, or VideoARM, under comparable inference budgets?
  4. Could the authors evaluate identity consistency and QA performance across different entity reappearance intervals?
  5. Could the authors provide a sensitivity analysis regarding different window sizes and boundaries during memory construction?
3 comments

W1 / Q1. Comparison with a window-local memory baseline{\color{#2563eb}\text{W1 / Q1. Comparison with a window-local memory baseline}}

"However, the ablations in Table 3 conflate this mechanism with additional contextual information." (W1)

"...compare full VideoSEAM against a history-conditioned baseline that retains the same entity descriptions, visual references, event histories, and summaries, but uses window-local entity IDs without explicit cross-window alignment." (Q1)

We evaluated a window-local memory baseline on MM-Lifelong Test@Day. It constructs memory independently for each window, assigning local IDs to entities and events and linking each event to its participating entities. This baseline achieves 40.3% ACC and 34.2% Ref@300. Full VideoSEAM achieves 44.0% ACC and 38.6% Ref@300, gains of 3.7 and 4.4 pp, respectively. These results show that carrying information across windows improves both answer accuracy and reference grounding.

W2 / Q2. Comparison with MemDreamer on LVBench{\color{#2563eb}\text{W2 / Q2. Comparison with MemDreamer on LVBench}}

"MemDreamer, which reports stronger LVBench results (90.7%), is omitted from the comparison." (W2)

"Could the authors clarify the performance gap between VideoSEAM and MemDreamer on LVBench?" (Q2)

We evaluated MemDreamer using its previously released code and the provided LVBench memory database. After fixing a ground-truth leakage issue in the released inference code, we ran MemDreamer with Gemini 3.1 Pro on the same 1,534 questions used to evaluate VideoSEAM. In our reproduction, MemDreamer achieves 79.9%, compared with 79.5% for VideoSEAM (Table R1).

Table R1. LVBench accuracy (%), grouped by evidence duration.

MethodShort (< 30 s)Medium [30 s, 5 min]Long (> 5 min)Overall
MemDreamer79.181.080.979.9
VideoSEAM78.979.682.479.5

VideoSEAM leads our MemDreamer reproduction by 1.5 pp on the Long subset and by 1.5 pp on questions whose evidence spans more than 30 minutes. It trails MemDreamer by 0.2 pp on Short and 1.4 pp on Medium. This pattern is consistent with VideoSEAM's focus on tracking entities and events across windows. Questions with shorter evidence spans (< 5 min) may depend less on recognizing the same entity across appearances or following an event across window boundaries.

In addition, since MemDreamer does not release its memory construction code, we could not extend the comparison to MM-Lifelong to evaluate both methods on substantially longer videos (> 24 hours).

Q3. Same-backbone comparisons on MM-Lifelong{\color{#2563eb}\text{Q3. Same-backbone comparisons on MM-Lifelong}}

"Could the authors provide same-backbone comparisons with compatible, publicly reproducible training-free methods, such as ReMA, WorldMM, or VideoARM, under comparable inference budgets?" (Q3)

We evaluated VideoSEAM, VideoARM, and WorldMM on MM-Lifelong with Gemini 3.1 Pro as their common multimodal backbone. For VideoARM, we used Gemini for the controller and multimodal tools and retained its original audio transcription setup. For WorldMM, we used Gemini for memory construction and QA and retained its original embeddings. Each system used its own retrieval and QA procedure.

Table R2. MM-Lifelong results with Gemini 3.1 Pro as the common multimodal backbone. ACC and Ref@300 are reported in %.

MethodTest@Day ACCTest@Day Ref@300Test@Week ACCTest@Week Ref@300Val@Month ACCVal@Month Ref@300
VideoSEAM44.038.626.037.322.536.0
DVD39.3—22.0—17.2—
VideoARM35.59.421.010.519.78.6
WorldMM11.83.825.515.7——

VideoSEAM outperforms VideoARM on both metrics across all three splits (Table R2). Compared with WorldMM, VideoSEAM achieves 32.2 and 34.8 pp higher ACC and Ref@300, respectively, on MM-Lifelong Test@Day. On Test@Week, the two systems are close in ACC (26.0% vs. 25.5%), while VideoSEAM leads by 21.6 pp in Ref@300 (37.3% vs. 15.7%). The WorldMM Val@Month run did not finish within the available evaluation time.

W3 / Q4. Identity fragmentation and entity reappearance intervals{\color{#2563eb}\text{W3 / Q4. Identity fragmentation and entity reappearance intervals}}

"The reported reuse precision (78.8%) does not capture such identity fragmentation." (W3)

"Could the authors evaluate identity consistency and QA performance across different entity reappearance intervals?" (Q4)

The reported 78.8% reuse precision measures how often an existing ID was assigned correctly. To quantify identity fragmentation, we audited 399 generated nodes and found that 16 of 349 ground-truth identities (4.6%) were split across multiple IDs, producing 49 redundant nodes.

We then examined how often an entity's ID changed between successive appearances, grouping transitions by the number of intervening windows in which the entity was absent. The ID changed in 46 of 440 unambiguous transitions. The change rate rose from 0.5% (2/371) for appearances in consecutive windows to 80.0% (32/40) after at least 11 intervening windows. This pattern suggests that maintaining identity across long gaps is difficult, although the longest-gap group contains only 40 transitions. We will discuss representative cases of long-gap fragmentation in the supplementary material.

We manually labeled 200 questions according to whether answering them required identifying a specific entity. VideoSEAM achieved 45.1% accuracy on the 152 entity-dependent questions and 40.6% on the 48 questions without a specific entity dependency. This 4.5-point difference describes performance by entity dependency. How QA accuracy varies with the entity's reappearance interval remains open.

W4 / Q5. Sensitivity to construction window size and boundaries{\color{#2563eb}\text{W4 / Q5. Sensitivity to construction window size and boundaries}}

"Could splitting the same event across different windows affect entity-event coherence and downstream QA accuracy?" (W4)

"Could the authors provide a sensitivity analysis regarding different window sizes and boundaries during memory construction?" (Q5)

W4. Events that cross a window boundary. The default construction window is 300 seconds. If an event spans a boundary, its observations can enter Episode Memory as separate, time-stamped event records. The next window receives recent events and active entity references, allowing it to interpret the continuation in context (Sec. 3.2.3). This allows VideoSEAM to maintain a coherent understanding of events that span multiple windows.

Q5. Sensitivity to window length and boundary placement. Table R3 compares three construction-window lengths on MM-Lifelong Test@Day.

Table R3. Construction-window ablation on MM-Lifelong Test@Day.

Window lengthACC (%)Ref@300 (%)
150 s42.530.4
300 s (default)44.038.6
600 s44.039.2

Shortening the default window from 300 s to 150 s reduces ACC by 1.5 pp and Ref@300 by 8.2 pp. The 300 s and 600 s windows both achieve 44.0% ACC, with a small increase in Ref@300 from 38.6% to 39.2% at 600 s. Given that a 600-second window substantially increases inference latency, we adopt a 300-second window to balance accuracy and computational efficiency.

Reviewer_UsREOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes VideoSEAM, a training-free framework for long-form video understanding with a sequential entity-event alignment memory. It processes video windows chronologically while carrying forward a bounded Transition Memory to maintain cross-window entity and event continuity. The constructed Episode Memory then serves as a persistent, query-independent structured database. Furthermore, a memory-grounded QA agent is introduced, utilizing a ReAct loop with Memory Retrieval and Targeted Video Inspection tools to efficiently answer questions. Extensive experiments on MM-Lifelong, EgoLifeQA, and LVBench demonstrate that VideoSEAM achieves superior performance compared to existing training-free methods.

Critical strengths
  1. The core idea of combining a bounded Transition Memory for sequential processing with a persistent Episode Memory is well-motivated and elegant. It effectively solves the notorious cross-window entity inconsistency and event fragmentation issues in query-independent video understanding.

  2. Separating the active context needed for continued tracking from the persistent historical record balances information retention with token/computation efficiency.

  3. VideoSEAM achieves effective results (or ranks top among machine methods) across multiple challenging long-form video benchmarks, including MM-Lifelong, EgoLifeQA, and LVBench.

Critical weaknesses
  1. As discussed in the paper's analysis, entity reuse precision drops for infrequently observed entities (65.5% compared to 97.4% for frequent entities). While reasonable given the challenging nature of long videos, global re-identification remains an area for future improvement.

  2. The current implementation heavily relies on proprietary models (e.g., Gemini 3.1 Pro) and fixed temporal sampling/inspection budgets, which may limit generalizability and incur additional API costs.

Questions for authors
  1. Although the paper provides a token consumption analysis in Appendix showing a slight overall token reduction, it would be helpful to briefly discuss the wall-clock latency trade-offs between sequential memory construction and query-time ReAct loops in the main text.

  2. It would strengthen the discussion to include more qualitative examples where entity association fails due to drastic visual appearance changes or complex occlusion, providing clearer guidance for future work.

2 comments

W1 / Q2. Entity ID reuse across appearance changes and gaps in visibility{\color{#2563eb}\text{W1 / Q2. Entity ID reuse across appearance changes and gaps in visibility}}

"entity reuse precision drops for infrequently observed entities" (W1)

We agree that ID reuse is less reliable for infrequently observed entities, including some minor characters. Across the 669 ID reuse decisions evaluated in Figure 3, reuse precision was 97.4% for frequently observed entities and 65.5% for the remaining entities. This is because minor characters often appear only briefly, giving the constructor fewer views on which to base a match when they reappear. We will investigate ways to improve ID reuse when visual evidence is sparse in future work.

"include more qualitative examples where entity association fails due to drastic visual appearance changes or complex occlusion" (Q2)

Appearance changes. A drastic change in appearance can make it difficult to recognize the same entity across windows. We will add failure cases to the supplementary material, showing the frames before and after the change alongside the assigned IDs and the correct correspondence. Figure 3 in the main paper also shows a successful case: the protagonist retains its original ID after transforming from a monkey into a white fox.

Occlusion. Brief occlusions within a single observation window rarely disrupted entity association in the examples we examined. Because the constructor sees the entity before and after the occlusion in the same window, it can use the surrounding frames to recognize it when it reappears. We will add examples of both successful and failed associations under occlusion to the supplementary material.

Duplicate IDs after gaps in visibility. When an entity disappears for a longer interval and returns in a later window, the constructor may assign it a new ID, splitting its event history across multiple records. In a separate manual audit of 399 generated nodes, we identified 349 distinct entities, 49 redundant nodes corresponding to entities already represented, and one spurious node.

W2. Dependence on proprietary models and fixed inspection budgets{\color{#2563eb}\text{W2. Dependence on proprietary models and fixed inspection budgets}}

"The current implementation heavily relies on proprietary models (e.g., Gemini 3.1 Pro) and fixed temporal sampling/inspection budgets, which may limit generalizability and incur additional API costs." (W2)

Model dependence. We evaluated GPT-5.6-sol as the backbone for both memory construction and question answering. On MM-Lifelong Test@Day, GPT-5.6-sol achieves 28.5% ACC and 35.3% Ref@300 with 1,024 uniformly sampled frames. VideoSEAM with GPT-5.6-sol improves performance to 48.3% ACC and 38.4% Ref@300.

In Table 4(b), Gemini 3.1 Pro constructs the memory, and Qwen 3.8 Max handles query-time reasoning and video inspection. With this controller, VideoSEAM improves ACC from 14.5% to 33.3%. These results show that VideoSEAM performs well with both GPT-5.6-sol and Qwen 3.8 Max without relying exclusively on Gemini 3.1 Pro.

Sampling, inspection, and cost. During construction, we sample 300-second windows at 1 FPS, so a brief event can fall between sampled frames. At query time, the agent chooses which intervals to inspect and when to stop. It can inspect up to six intervals per question at up to 1 FPS. Requests longer than 150 seconds are cropped around their midpoint, which can omit evidence near either end. These limits bound video access per question.

On the 200-question workload in Table R1, VideoSEAM used more QA tokens than DVD (144.9M vs. 46.9M), but 7.3% fewer tokens overall (223.1M vs. 240.6M) because its lower one-time construction cost offset the additional QA tokens.

Q1. Wall-clock latency of memory construction and query-time reasoning{\color{#2563eb}\text{Q1. Wall-clock latency of memory construction and query-time reasoning}}

"Although the paper provides a token consumption analysis in Appendix showing a slight overall token reduction, it would be helpful to briefly discuss the wall-clock latency trade-offs between sequential memory construction and query-time ReAct loops in the main text." (Q1)

On MM-Lifelong Test@Day, one-time memory construction took 13.4 hours for VideoSEAM and 24.3 hours for DVD with the same Gemini 3.1 Pro backbone. VideoSEAM processes 300-second windows sequentially because each window uses context from its predecessor. DVD processes independent 10-second clips, and our run used 10 concurrent API calls.

Once the memory was built, VideoSEAM took 7.2 minutes per question on average across 200 questions, compared with 9.4 minutes for DVD. Thus, although VideoSEAM used more QA tokens, its measured response time was shorter in this setting. This suggests that VideoSEAM also offers a modest runtime advantage over DVD.

Table R1. Wall-clock latency, token consumption, and answer accuracy on MM-Lifelong Test@Day with Gemini 3.1 Pro.

MetricDVDVideoSEAM
Memory construction time ↓24.3 h (concurrency = 10)13.4 h (sequential)
Mean QA latency per question ↓9.4 min7.2 min
Memory construction tokens ↓193.7M78.2M
QA tokens ↓46.9M144.9M
Total tokens ↓240.6M223.1M
ACC (%) ↑39.344.0
Reviewer_xtcBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VideoSEAM, a training-free framework for long-form video understanding that integrates coherent sequential entity–event memory construction with an agentic Observation–Reason–Action loop for memory retrieval and source-video inspection. The proposed framework introduces a dual-horizon memory architecture consisting of Transition Memory and Episode Memory. Transition Memory maintains a bounded context of entity references, recent events, and hierarchical summaries to support coherent interpretation across consecutive video windows, while Episode Memory persistently stores structured entity–event records for subsequent question answering. The framework is evaluated on three long-form video understanding benchmarks, MM-Lifelong, EgoLifeQA, and LVBench, demonstrating strong performance compared to existing long-video understanding methods.

Critical strengths
  • Well-motivated sequential memory construction. The paper presents a compelling motivation for constructing an evolving, coherent video-level memory rather than independently processing local video windows. In particular, maintaining entity references and event context across consecutive windows is a reasonable approach to mitigating identity inconsistencies and fragmented event representations in long-form video understanding.

  • Promising empirical results. The proposed framework achieves strong performance on MM-Lifelong and LVBench, while demonstrating competitive results on EgoLifeQA. In particular, the ablation results suggest that carrying forward contextual information through Transition Memory can substantially improve question-answering performance.

Critical weaknesses

Major Weakness

  • Insufficient analysis of the proposed memory architecture. Although the paper emphasizes coherent sequential memory construction and a dual-horizon memory architecture as its main contributions, the experimental analysis does not fully isolate the effectiveness of the proposed memory components. In particular, Transition Memory consists of entity references, recent events, segment summaries, and chapter summaries, but Table 3 does not separately evaluate the contributions of segment-level and chapter-level summarization. Therefore, it remains unclear how much the hierarchical summarization mechanism contributes beyond maintaining recent entity and event context. Furthermore, while the paper reports QA performance comparisons with existing memory-based methods such as M3-Agent and WorldMM, a more direct analysis of memory construction mechanisms under comparable backbone and QA settings would help substantiate the claimed advantages of coherent sequential memory construction.

  • Limited evaluation of backbone generalizability and dependence on the retrieval agent. The framework is primarily evaluated using Gemini 3.1 Pro, raising questions about its generalizability to smaller open-source models. Although Table 4(b) includes results with Qwen 3.8 Max as the controller, it remains unclear whether the proposed memory construction and retrieval pipeline can maintain its effectiveness with smaller open-source models, such as Qwen3.5-9B. In addition, the memory-grounded QA pipeline places substantial demands on the reasoning capabilities of the retrieval agent, including query formulation, retrieval strategy selection, evidence assessment, and iterative source-video inspection. Consequently, it is difficult to determine how much of the overall performance improvement comes from the proposed memory representation versus the capabilities of a strong reasoning controller. The limited implementation details of the retrieval mechanism further make it difficult to assess its reproducibility and scalability.

Minor Weakness

  • Concrete memory examples: Although Figure 2 provides an overview of the proposed memory architecture, including concrete examples of the constructed Transition Memory and Episode Memory would improve clarity. In particular, illustrating how entity references, event records, and hierarchical summaries evolve across consecutive video windows would help readers better understand the sequential memory construction process.
  • Benchmark and metric descriptions: More detailed descriptions of the evaluation benchmarks and their task categories would improve readability. In particular, the definition and computation of the Ref@300 metric should be clarified.
  • Terminology: The relationship between the Agentic Observation–Reason–Action Loop introduced in Section 3.3.2 and the ReAct terminology used in Table 4 could be clarified for consistency.
Questions for authors
  • Hierarchical memory components. Table 3 evaluates several components of Transition Memory, but does not independently assess the contribution of historical summarization. Could the authors provide an ablation that removes segment-level and chapter-level summaries while retaining entity references and recent events? This would help clarify whether hierarchical summarization provides additional benefits beyond propagating recent context.

  • Backbone generalizability. Table 4(b) evaluates different controller models, but it is unclear whether the memory construction model is also changed. Could the authors clarify this setting? Additionally, have the authors investigated whether the framework remains effective with smaller open-source models, such as Qwen3.5-9B? Such an analysis would help assess the dependence of the proposed memory construction and retrieval pipeline on model capability.

  • Agentic retrieval behavior and video inspection. Although Section 3.3 and Appendix A.5.2 describe the available retrieval and inspection tools, the practical decision process of the QA agent remains insufficiently analyzed. How does the agent select retrieval strategies and determine which source-video intervals to inspect? In particular, Section 4.1 limits inspection to six intervals of at most 150 seconds each. How are intervals selected or divided when the retrieved events span longer durations? Providing representative retrieval trajectories, including tool selections and inspected intervals, would help clarify the behavior and efficiency of the agent.

  • Comparison with existing memory architectures. While the paper reports QA performance comparisons with M3-Agent and WorldMM, could the authors further clarify the advantages of the proposed dual-horizon memory over existing working-memory and long-term-memory architectures? In particular, a more direct comparison of memory construction mechanisms, ideally under comparable backbone and QA settings, would help establish the distinctive contribution of coherent sequential memory construction.

4 comments

Major 1 / Question 1. Contribution of hierarchical summaries{\color{#2563eb}\text{Major 1 / Question 1. Contribution of hierarchical summaries}}

"Could the authors provide an ablation that removes segment-level and chapter-level summaries while retaining entity references and recent events?"

We removed both levels of historical summaries from Transition Memory while retaining entity references, recent events, and the same downstream QA configuration. The results are shown in Table R1. On MM-Lifelong Test@Day, removing historical summaries reduces ACC by 12.2 pp and Ref@300 by 24.8 pp. Thus, even when the model has access to recent context, carrying historical summaries improves both answer accuracy and temporal grounding.

Table R1. Hierarchical summaries in Transition Memory on MM-Lifelong Test@Day.

ConfigurationACC (%)Ref@300 (%)
Full VideoSEAM44.038.6
Without segment-level and chapter-level summaries31.813.8

Major 1 / Question 4. Comparison with existing memory architectures{\color{#2563eb}\text{Major 1 / Question 4. Comparison with existing memory architectures}}

Q4(a). "While the paper reports QA performance comparisons with M3-Agent and WorldMM, could the authors further clarify the advantages of the proposed dual-horizon memory over existing working-memory and long-term-memory architectures?"

M3-Agent organizes episodic and semantic memories around entities, while WorldMM retrieves evidence from episodic, semantic, and visual memories. In VideoSEAM, the two memories serve distinct roles during memory construction. When a character reappears in the next window, Transition Memory provides its earlier description and visual reference, allowing the agent to write the new event using the same entity ID. Episode Memory retains both time-stamped events and their shared ID for later retrieval, even after the character leaves Transition Memory. This lets the agent preserve cross-window links while keeping the context for each construction step bounded.

Q4(b). "In particular, a more direct comparison of memory construction mechanisms, ideally under comparable backbone and QA settings, would help establish the distinctive contribution of coherent sequential memory construction."

Same-backbone comparison. Given the time and compute available during rebuttal, we prioritized WorldMM, the stronger of the two baselines highlighted by the reviewer on both LVBench and EgoLifeQA (main paper, Table 2). We evaluated WorldMM on MM-Lifelong using Gemini 3.1 Pro for memory construction and QA, retaining its original embeddings. Table R2 also includes the same-backbone results from our main Table 1 for DeepVideoDiscovery (DVD), which builds a searchable memory by independently captioning 10-second clips. All three systems use Gemini 3.1 Pro, and each retains its own QA procedure.

Table R2. MM-Lifelong results with Gemini 3.1 Pro as the common multimodal backbone. Acc and Ref@300 are reported as percentages. DVD results are from Table 1 of the main paper. Dashes indicate unavailable results.

MethodTest@Day AccTest@Day Ref@300Test@Week AccTest@Week Ref@300Val@Month AccVal@Month Ref@300
VideoSEAM44.038.626.037.322.536.0
DVD39.3--22.0--17.2--
WorldMM11.83.825.515.7----

VideoSEAM exceeds DVD in accuracy by 4.7, 4.0, and 5.3 pp on Day, Week, and Month, respectively. Compared with WorldMM, VideoSEAM achieves better performance on both Test@Day (44.0% vs. 11.8% Acc; 38.6% vs. 3.8% Ref@300) and Test@Week (26.0% vs. 25.5% Acc; 37.3% vs. 15.7% Ref@300).

The WorldMM Month run did not finish within the rebuttal period. Completing it would require approximately another five days on our available GPUs.

Major 2 / Question 2. Backbone generalizability{\color{#2563eb}\text{Major 2 / Question 2. Backbone generalizability}}

"Table 4(b) evaluates different controller models, but it is unclear whether the memory construction model is also changed. Could the authors clarify this setting? Additionally, have the authors investigated whether the framework remains effective with smaller open-source models, such as Qwen3.5-9B?"

a) Memory construction model in Table 4(b). In the Qwen setting, Qwen 3.8 Max handles controller reasoning and source-video inspection at query time. We changed only the reasoning model and kept the memory construction model fixed. We will specify both model assignments in the table caption.

b) Changing the backbone at both stages. As shown in Table R3, with GPT-5.6-sol for memory construction and QA, VideoSEAM achieves 48.25% ACC and 38.43% Ref@300 on MM-Lifelong Test@Day. The same model given 1,024 uniformly sampled frames directly achieves 28.50% and 35.33%, respectively. The gains are 19.75 points in ACC and 3.10 points in Ref@300. These gains show that VideoSEAM continues to improve performance with a stronger backbone.

Table R3. Answer accuracy on MM-Lifelong Test@Day. VideoSEAM uses the same backbone for memory construction and QA.

Memory Model & ControllerVideoSEAMACC (%) ↑
GPT-5.6-sol--28.5
GPT-5.6-sol✓48.3+19.75\mathbf{48.3}_{\color{#c80000}\mathbf{+19.75}}
Gemini 3.1 Pro--23.0
Gemini 3.1 Pro✓44.0+21.0\mathbf{44.0}_{\color{#c80000}\mathbf{+21.0}}

c) Smaller open-source models. To construct memory for each video window, VideoSEAM interprets the current frames together with the Transition Memory carried over from earlier windows. This multimodal input can reach approximately 100K tokens for a single window. At this length, Qwen3.5-9B struggled to reason reliably across the current observations and prior memory. As a result, we were unable to obtain an end-to-end result for this model under our current setup. In future work, we will explore how to compress the visual input and Transition Memory while preserving the information smaller models need to link entities and events across windows.

Major 2 / Question 3. Retrieval decisions and video inspection{\color{#2563eb}\text{Major 2 / Question 3. Retrieval decisions and video inspection}}

"How does the agent select retrieval strategies and determine which source-video intervals to inspect? In particular, Section 4.1 limits inspection to six intervals of at most 150 seconds each. How are intervals selected or divided when the retrieved events span longer durations?"

At each step, the controller uses the question and previous tool outputs to choose its next action. For retrieval, it selects a search mode (keyword, exact phrase, entity, temporal, or hybrid), a result order (by relevance, chronological, or reverse-chronological), and a time constraint. A retrieved event can anchor the next search, for example by restricting it to events after that timestamp.

Interval selection. The controller uses retrieved timestamps to request a source-video interval and poses a focused visual question. If the requested interval exceeds 150 seconds, the tool inspects a single 150-second crop centered on that interval. The full event boundaries are retained in memory. Each question allows at most six inspected intervals. We will make the center-crop rule explicit in Section 4.1.

Example. We provide a concrete example in Figure 4 of the supplementary material (Section A.1, "Qualitative Results"). As shown in the figure, searching by weapon name returned a different staff, so the agent switched to the boss's name. Using retrieve_video_evidence, it located the event in which the player defeated the boss, then searched subsequent events to find where the player forged the target staff. Since the forging record did not specify the weapon's attributes, the agent used inspect_video_segments to examine the crafting menu at 20:40:12–20:42:21 and verify the changes in attack power and critical-hit rate.

Contribution of adaptive retrieval. When each tool runs once in a fixed round and the model then answers directly, ACC falls from 44.0% to 24.5% (main paper, Table 4(a)). This 19.5-point drop measures the effect of adaptive tool use with the memory and tools held fixed.

Minor 1. Concrete memory examples{\color{#2563eb}\text{Minor 1. Concrete memory examples}}

"...including concrete examples of the constructed Transition Memory and Episode Memory would improve clarity. In particular, illustrating how entity references, event records, and hierarchical summaries evolve across consecutive video windows would help readers better understand the sequential memory construction process." (minor weakness 1)

We will provide detailed visualizations in the supplementary material to illustrate how entity references, event records, and hierarchical summaries evolve across consecutive video windows.

Minor 2. Benchmark descriptions and evaluation metrics{\color{#2563eb}\text{Minor 2. Benchmark descriptions and evaluation metrics}}

"More detailed descriptions of the evaluation benchmarks and their task categories would improve readability. In particular, the definition and computation of the Ref@300 metric should be clarified." (minor weakness 2)

We will expand the benchmark descriptions and explicitly identify the task categories and metric definitions. For MM-Lifelong, both ACC and Ref@300 follow the official evaluation protocol:

  • ACC. Answers are evaluated using GPT-5. Evaluation scores are mapped to 0, 0.5, or 1, averaged across questions, and reported as a percentage.
  • Ref@300. We use the official MM-Lifelong evaluation function, which maps predicted and annotated evidence intervals to 300-second bins and computes the intersection-over-union of the two bin sets.
  • EgoLifeQA categories. The category columns in Table 2 of the main paper correspond to EntityLog, EventRecall, HabitInsight, RelationMap, and TaskMaster, followed by Average.

Minor 3. Consistency between Observation–Reason–Action and ReAct{\color{#2563eb}\text{Minor 3. Consistency between Observation–Reason–Action and ReAct}}

"The relationship between the Agentic Observation–Reason–Action Loop introduced in Section 3.3.2 and the ReAct terminology used in Table 4 could be clarified for consistency." (minor weakness 3)

The Observation–Reason–Action loop in Section 3.3.2 and "ReAct" in Table 4 refer to the same mechanism. We will consistently use Observation–Reason–Action loop throughout the paper, including the ablation label in Table 4.

I appreciate the authors' detailed response and additional experiments, which have addressed some of my concerns. However, while the additional results support the effectiveness of the overall framework, I remain uncertain about the extent to which the proposed memory and retrieval mechanisms provide distinct benefits beyond the strong context understanding and adaptive reasoning capabilities of frontier models. I will therefore maintain my original score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their careful reading of our paper and for highlighting important issues and areas for improvement. We greatly appreciate the time and effort you have devoted to reviewing our work and helping us strengthen the paper. We will upload an updated version of the paper to OpenReview shortly, incorporating the clarifications and revisions discussed in our responses.

Dear Reviewers,

Thank you again for your time and effort in reviewing our work and providing valuable feedback. We would like to inform you that we have submitted the updated version of our paper, incorporating all the additional experiments, analyses, and revisions addressing the reviewers' comments.

Due to time constraints, some of the additional visualizations discussed during the rebuttal have not yet been included in the submitted version. Nevertheless, all other experimental results, analyses, and manuscript revisions have been incorporated.

We sincerely appreciate your consideration and look forward to your further feedback.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.