acceptodds
Withdrawn from ICLR 2027

VideoLedger: Index-and-Revisit Memory for Long-Horizon Video Stream Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large multimodal models (LMMs) remain limited when reasoning over continuously growing visual histories spanning hours, weeks, or months. Reprocessing the accumulated video for every new question becomes increasingly impractical, while relying on compact memory alone risks losing details that may become decisive for future questions. We introduce **VideoLedger**, a training-free framework that follows an *index-and-revisit* paradigm for long-horizon streaming video understanding. As each clip arrives, VideoLedger appends timestamped visual narratives and audio transcripts to chronological Markdown timelines, forming a question-agnostic index linked to the source video. At question time, an off-the-shelf LMM navigates the index using only standard file operations—lookup and scoped reading—and selectively revisits timestamp-linked source clips when the textual records are insufficient. This design separates efficient long-range evidence navigation from fine-grained visual verification. Stored as persistent text outside the active reasoning context, the index is naturally harness-friendly and remains accessible after context compaction, without a learned retriever or graph traversal. Notably, this simple navigation interface proves highly effective: across ten hour-, week-, and month-scale benchmarks, VideoLedger achieves the best results among compared methods in ultra-long, robot-centric, and general long-video settings, with consistent gains across Qwen, GPT, and Gemini backbones. Further analyses show complementary benefits from textual and revisited visual evidence, accurate temporal grounding, and a favorable accuracy–latency trade-off. Our code and evaluation logs will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Fd6oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VideoLedger, a training-free framework for long-horizon streaming video understanding. VideoLedger incrementally stores timestamped visual narrations and speech transcripts as chronological text, creating a question-agnostic index linked to the original video. At inference time, an off-the-shelf LMM searches and reads this textual memory to locate relevant evidence and selectively revisits the corresponding raw video clips when finer visual details are needed. This index-and-revisit design avoids repeatedly processing the entire video history while preserving access to the original visual evidence. Experiments across ten long-video QA benchmarks and multiple LMM backbones show strong performance across hour- to month-scale videos, while ablation and efficiency analyses demonstrate the complementary benefits of textual indexing and selective video revisiting.

Critical strengths

Strong and consistent empirical results - VideoLedger achieves large gains across ten benchmarks covering hour- to month-long videos, with consistent improvements across Qwen, GPT, and Gemini backbones. Also, the authors showed efficiency evaluation on latency, token usage, indexing cost and inference cost.

Well-validated index-and-revisit design - The ablations clearly show that speech information and selective source-video revisiting provide complementary improvements over the textual visual timeline alone. Such in-depth analysis supports the effectiveness of the proposed method.

Critical weaknesses

Limited robustness of lexical search - VideoLedger relies on keyword-based lookup, which can miss relevant evidence when the question and memory use different words. I acknowledge that the authors stated this point in the limitation. However, I think this is too big limitation which would weaken the robustness of the proposed method. Therefore, I think related experiment need to be provided rather than stating this point in the limitation.

Different visual budgets used for direct-inference comparisons - The direct baselines uniformly sample up to 1,024 frames, while VideoLedger selectively revisits at most 10 clips. A matched-budget comparison would better isolate the benefit of targeted retrieval from differences in visual access.

Limited methodological novelty - The main contribution is primarily a system design that combines existing components, including textual video descriptions, lexical search, and selective source-video access. While the resulting system is simple and effective, the methodological novelty beyond prior memory-based and agentic video retrieval systems appears limited.

Questions for authors

Please refer to the weaknesses

Reviewer_QQJZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VideoLedger, a training-free framework for long-horizon streaming video understanding based on an index-and-revisit paradigm. The proposed approach incrementally converts incoming video clips into timestamped visual narratives and speech transcripts, which are stored as persistent Markdown timelines.

Critical strengths
  • The paper addresses an important problem in long-horizon streaming video understanding. The index-and-revisit paradigm is intuitive, and the overall framework is clearly explained and easy to follow.
  • VideoLedger achieves substantial improvements across hour-, week-, and month-level benchmarks, including egocentric and robot-centric scenarios. Evaluations using Qwen, GPT, and Gemini backbones further demonstrate the practical applicability of the approach.
  • The paper investigates the contributions of visual narratives, speech transcripts, and source revisiting. The additional analyses of temporal grounding, inference latency, and token consumption provide useful evidence of the framework's effectiveness.
Critical weaknesses
  • The core pipeline of VideoLedger closely resembles several established approaches. DrVideo (CVPR 2025) already converts long videos into textual documents and iteratively retrieves and augments visual evidence for question answering. VideoAgent (ECCV 2024) employs structured temporal memory, video segment localization, and tool-based visual inspection. More recently, WorldMM (CVPR 2026) integrates textual episodic memory with query-adaptive retrieval of visual evidence, while VideoSeek (CVPR 2026) introduces iterative tool-guided navigation to locate and inspect relevant video segments. Compared with these methods, VideoLedger mainly replaces more structured memory and retrieval mechanisms with timestamped Markdown files and standard grep/read operations. Although this simplification may improve practicality and deployment, it remains unclear whether it constitutes a sufficiently novel algorithmic contribution. The paper would benefit from a more explicit technical comparison with these prior works and controlled experiments demonstrating the distinct advantages of its design.
  • The paper compares VideoLedger with several existing systems, but these methods may differ in their underlying models, memory representations, and inference procedures. A more informative comparison would use identical visual narratives, reasoning backbones, and inference budgets while varying only the retrieval strategy. Without such controls, it is difficult to establish whether the proposed navigation mechanism offers a meaningful advantage over standard text retrieval approaches.
  • Although VideoLedger is motivated by continuously growing video streams and persistent memory, most reported experiments focus on question answering over long recorded histories. The paper does not sufficiently demonstrate online performance under continuously arriving data, repeated queries, memory growth, or actual context-compaction events.
Questions for authors
  • Could the authors compare VideoLedger with BM25, embedding-based retrieval, and a simple retrieve-and-revisit baseline using the same timestamped narratives, reasoning backbone, and computational budget?
  • How does VideoLedger perform when the video history grows continuously and questions arrive at different times?
  • How frequently does the agent revisit the source video, and how much does revisiting improve accuracy relative to text-only reasoning?
Reviewer_gwJvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VideoLedger, a training-free framework for ultra-long streaming video understanding, aiming to address the high computational costs and information loss associated with continuously growing video histories. The proposed method adopts an Index-and-Revisit mechanism, which converts video content into timestamped textual indices and retrieves relevant information through standard file operations, selectively revisiting the original video for fine-grained visual verification when necessary. The framework can be integrated with existing large multimodal models and agent systems, supporting persistent memory and long-horizon video reasoning. Experiments conducted on 10 long-video understanding benchmarks demonstrate the effectiveness of the proposed method across Qwen, GPT, and Gemini models, showing promising results in terms of accuracy, generalizability, and inference efficiency.

Critical strengths
  1. The proposed Index-and-Revisit mechanism is simple and effective. The paper adopts an Index-and-Revisit mechanism that divides video understanding into two stages: first, efficiently locating relevant information through a textual index, and then selectively revisiting the original video for fine-grained verification when necessary. This design avoids repeatedly processing the entire video history while preserving access to detailed visual information, achieving a good balance between computational efficiency and information completeness.
  2. The framework requires no additional training and offers good generalizability and scalability. VideoLedger does not require training a dedicated retrieval model or relying on complex graph structures or graph traversal algorithms. Instead, it leverages existing large multimodal models and standard file operations for information retrieval and evidence verification. This reduces implementation and deployment costs and facilitates integration with different models and application scenarios.
Critical weaknesses
  1. Limited Methodological Novelty The proposed Index-and-Revisit mechanism essentially converts video clips into textual indices, retrieves relevant information based on user queries, and selectively revisits the original video when necessary. However, similar ideas, including textual indexing, retrieval-augmented generation (RAG), and on-demand access to original data, have already been widely explored in previous studies. Therefore, the paper's main contribution appears to lie more in integrating existing techniques than in introducing fundamentally new algorithms or methodologies. The authors should further clarify the key differences between VideoLedger and existing approaches to long-video memory and retrieval-augmented video understanding.
  2. Potential Information Loss in Textual Indexing VideoLedger converts video clips into textual descriptions and relies on these descriptions as the primary basis for subsequent retrieval. However, fine-grained visual information, such as spatial relationships between objects, subtle motion changes, and transient events, may not be adequately preserved in textual representations.
  3. Insufficient Analysis of Indexing Errors and Long-Term Error Accumulation VideoLedger relies on multimodal models to generate visual narratives and audio transcripts, which may contain inaccuracies, omissions, or hallucinations. Since the index is continuously constructed as new video clips arrive, errors introduced during the indexing process may persist over time and negatively affect subsequent evidence retrieval and reasoning. This concern becomes particularly important in scenarios involving video histories spanning weeks or months.
Questions for authors
  1. A critical question is: If important visual details are omitted during index construction, how can the system locate the corresponding original video clips during subsequent retrieval? Although the Revisit mechanism allows access to the original video, it still depends on the textual index to provide accurate retrieval cues. Consequently, information loss in the textual index may prevent relevant clips from being retrieved in the first place, rather than merely affecting the accuracy of the final answer. The authors are encouraged to investigate how different levels of information loss affect retrieval performance and question-answering accuracy.

2.How does the framework ensure the accuracy and reliability of a continuously growing textual index? The authors should further evaluate the impact of indexing errors on downstream performance and discuss whether mechanisms for error detection, index correction, or updating are available.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.