acceptodds
Withdrawn from ICLR 2027

LongVidGraph: Progressive Graph Memory for Retrieval-Augmented Omni-Modal Long-Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-video understanding with omni-modal models remains challenging due to limited context budgets and long-range temporal dependencies. Existing methods rely on dense frame sampling or simple context accumulation, which fail to maintain coherent information over long durations. We propose LongVidGraph, a retrieval-augmented framework with a progressive graph memory. The model incrementally constructs a structured graph of entities, actions, and dialogues from audio-visual signals in long videos. This graph serves as a compact and temporally consistent memory. To enable the model to learn this incremental and structured representation, we build a large-scale clip-graph supervision pipeline that produces aligned video-graph training data at scale. We fine-tune Qwen2.5-Omni on this data for progressive graph construction. At inference time, we perform query-conditioned retrieval over the graph memory to obtain relevant structured evidence. The retrieved evidence is used to support downstream understanding tasks. Experiments on multiple long-video benchmarks show consistent improvements over strong open-source baselines, with more significant gains on long-duration videos. The results demonstrate the effectiveness of progressive graph memory and retrieval for long-video understanding with omni models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omni Model, Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZuSWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LongVidGraph is a progressive graph memory framework for omni-modal long-video understanding. The method divides videos into minute-long clips and fine-tunes Qwen2.5-Omni to incrementally extract and merge entities, actions, and dialogues. For question answering, it retrieves relevant graph records and uses their timestamps to extract keyframes, combining textual and visual evidence. The authors construct approximately 15K clip–graph training examples and report improvements over the Qwen2.5-Omni baseline on several benchmarks under different settings.

Critical strengths
  1. The method improves overall performance over the Qwen2.5-Omni baseline with more pronounced gains on some longer-video subsets, which provides empirical support for the framework’s effectiveness on existing long-video QA benchmarks.

  2. The ablations show that retrieved text and keyframes provide complementary information, and the constructed memory improves multiple QA models on LVBench. Therefore, the retrieved evidence has value across models, with benefits extending beyond the fine-tuned model itself.

Critical weaknesses

Evaluation by video duration does not establish long-range dependency modeling, and the paper lacks targeted evaluations of multi-hop temporal reasoning, state tracking, or tasks requiring high-FPS input to demonstrate that the graph preserves critical temporal details. The contribution of audio beyond transcripts and the necessity of an omni-modal model are also not established through audio ablations or questions that require audio.

Questions for authors
  1. How much does graph memory improve over clip-embedding retrieval under comparable video coverage, frame counts, audio inputs, and QA models?

  2. What are the time and computational costs of graph construction, incremental updates, and per-question retrieval for different video durations? Including graph construction, how does the end-to-end cost compare with clip-embedding retrieval?

  3. What audio information does the graph preserve beyond transcripts, speaker information, and dialogue summaries? Are there audio ablations that establish the need for an omni-modal model over a vision-based model supplied with transcripts?

Reviewer_ANM3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces LongVidGraph, a retrieval-augmented framework with progressive graph memory for long-video understanding. The authors construct a video–graph training dataset and fine-tune Qwen2.5-Omni to generate incremental graph updates from sequential video clips. The resulting memory provides textual evidence and timestamp-grounded keyframes for downstream question answering. Experiments on four long-video benchmarks show improvements over the base model, suggesting the effectiveness of the overall system.

Critical strengths
  • The framework addresses an important problem in long-video understanding by organizing information across video segments into persistent structured memory.
  • The video–graph dataset provides supervision for learning incremental memory construction, covering entities, actions, and dialogues. Experimental results demonstrate consistent improvements over the base model across multiple benchmarks, with substantial gains on several long-duration subsets.
  • The framework incorporates both audio and visual inputs and explicitly represents dialogues with speaker, listener, and timestamp information, providing a richer memory representation for long-video understanding.
Critical weaknesses
  • The experimental setup does not fully disentangle the effect of fine-tuning from the benefit of graph-based retrieval. A direct video-QA baseline using the fine-tuned backbone without graph retrieval is missing. Table 3 also does not explicitly clarify whether the retrieved text and keyframes are held fixed across QA models.
  • The graph representation is insufficiently specified. Although source–target and speaker–listener relationships are described, the distinction between nodes, edges, and attributes remains unclear. Since retrieval operates over linearized graph elements, it is also unclear how graph connectivity contributes beyond storing structured textual records.
  • The graph-construction training pipeline is evaluated only with Qwen2.5-Omni. Although Table 4 shows that the retrieved evidence benefits Qwen3-Omni and MiniCPM-o 2.6, this does not establish whether the proposed training and progressive graph construction generalize to other backbones. Applying the same pipeline to Qwen3-Omni would strengthen the evaluation.
  • The discussion of closely related work is incomplete. VGent (NeurIPS 2025) already proposes graph-based retrieval-augmented generation for long-video understanding, including cross-clip entity merging and dynamic graph updates. The paper neither discusses nor compares against this method, making it difficult to assess the novelty and additional benefit of the proposed progressive graph memory.
Questions for authors

The paper presents an intuitive approach to long-video memory construction and reports promising results. However, clearer specification of the graph structure and more controlled experiments are needed to establish the contributions of graph-based memory and its associated training procedure.

Please also see details in critical weaknesses。

Reviewer_gudjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LongVidGraph, a retrieval-augmented framework for long-video question answering using progressive graph memory. The authors construct clip–graph supervision and fine-tune Qwen2.5-Omni to generate graph updates containing entities, actions, and dialogues, conditioned on historical context. The resulting memory is used to retrieve relevant textual records and video frames for answering questions. Experiments on various benchmarks show improvements over the original Qwen2.5-Omni.

Critical strengths
  1. The method performs well across four long-video benchmarks and consistently improves over the Qwen2.5-Omni baseline.

  2. The authors build a substantial clip–graph dataset and use it to train the model to draw on earlier observations when processing new clips. The data construction and training work give the paper a concrete contribution to learning memory updates across video segments.

  3. The retrieved evidence also improves different answer models, and combining graph text with keyframes produces good overall results.

Critical weaknesses
  1. The distinction from GraphVideoAgent and EGAgent is not sufficiently clear. These methods already explore dynamic entity relations and temporal graph memory for video QA. I suggest revising how the paper’s contributions are summarized.

  2. The contribution of graph-construction training is also not directly established. Table 3 compares QA models but does not isolate the effect of training the memory constructor.

I would be willing to raise my score if the authors satisfactorily address these questions.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.