acceptodds
Withdrawn from ICLR 2027

Compressing a Lifetime: Exploiting Long-Horizon Temporal Redundancy for Persistent Visual Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Persistent visual memory is essential for always-on smart-glass assistants, but its storage cost is prohibitive: recording 2K video at 10 frames per second produces roughly 17 PB of raw RGB data over a 25-year lifetime, far beyond practical storage limits. Existing systems often reduce storage by selecting salient observations, irreversibly discarding visual evidence that may matter for unforeseen future queries. We instead formulate persistent visual memory as a memory-compression problem: retaining a compact representation of the complete observation stream rather than replacing it with selected frames or textual summaries. Our key insight is that egocentric video contains substantial long-horizon redundancy, with similar observations recurring across days and months, yet existing video codecs mainly exploit spatial and short-range temporal redundancy. We introduce a system that leverages this recurring context during both storage and reconstruction. A multi-rate neural video codec supports a wide range of compression levels, a novelty-aware rate selector assigns different rates to observations, and a memory-guided generative decoder reconstructs highly compressed content. Across three egocentric benchmarks, our system achieves compression ratios closed to , beyond the practical range of existing baselines, while substantially improving reconstruction quality and downstream video understanding under extreme compression.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Memory, Neural Video Compression, AR/VR

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4QaQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to build persistent visual memory for smart-glass assistants under limited storage. Its central observation is that egocentric video contains substantial redundancy across hours and days because users repeatedly encounter the same environments and objects. The proposed system combines a multi-rate neural codec, a novelty-aware rate selector that preserves unfamiliar observations at higher quality, and a memory-guided diffusion decoder that reconstructs heavily compressed frames using previously stored reference images. It retains a compressed representation of every sampled frame to support unforeseen future questions. The authors also introduce EgoCompress, a benchmark targeting fine-grained visual attributes in seven days of egocentric recordings. Experiments across four egocentric question-answering benchmarks report substantial storage savings while preserving downstream accuracy.

Critical strengths
  • The paper addresses a practical challenge in storing visual experiences for questions that are unknown at recording time.

  • The evaluation measures whether compressed memories remain useful for answering questions, including fine details through the new EgoCompress benchmark.

Critical weaknesses
  • There is a placeholder/hallucinated citation in the paper. Line 152-153 mentions cites Memorize-and-recall but its citation is simply a placeholder.

  • A few baselines are missed by the paper. The authors should at least discuss why they are not compared.

    • StableCodec: Taming One-Step Diffusion for Extreme Image Compression
    • DLF: Extreme Image Compression with Dual-generative Latent Fusion
  • The generative decoder restores details using older reference images. If an object’s label, location, or state has changed, it could introduce outdated or invented details. The metrics used in the paper (accuracy and perceptual similarity) do not directly measure these false memories, especially problematic when preserving evidence for unforeseen questions is the central motivation. To strength the paper, the author could add controlled tests where the current observation differs subtly from references, measuring errors in text, object presence, counts, and state changes.

  • Section 3.1 explicitly assumes the target frame is known. Thus, the experiments test reconstruction and answering after evidence selection; they do not establish whether an assistant can locate the right observation within a growing archive. The main multi-day benchmark also covers only one person over seven days, and the paper reports no archive-scale retrieval or query-latency results.

Questions for authors

See weakness.

Reviewer_myeQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper compresses persistent egocentric visual memory for questions unknown at recording time. It combines a variable-rate image codec (ENVC), novelty-aware keyframe storage and rate selection (NRS), and a diffusion decoder conditioned on historical keyframes (MGD). EgoCompress adds 500 fine-grained visual questions from seven days of one user's recordings. Evaluation covers four datasets at 512 × 512 and 1 FPS, with two VQA models on three datasets. On EgoCompress, the adaptive system achieves 61.4% Qwen accuracy at 3,414× overall compression, versus 63.1% uncompressed accuracy (Table 1). The results are promising, but central comparative and attribution claims need stronger validation.

Critical strengths
  1. A meaningful objective and coherent design. Preserving visual observations for unforeseen questions is valuable, and the three modules offer complementary mechanisms for storage allocation and reconstruction. Figure 1 and Table 4 establish recurrence over hours and days, motivating historical context as a source of compression gains.

  2. Promising results across datasets and evaluators. Table 1 evaluates four datasets, with Qwen and Gemini on three. On EgoSchema, the adaptive system achieves 75.3% Qwen accuracy at 2,766×, versus 78.8% uncompressed accuracy and 61.5% for DrVideo at 3,073×. Tables 2–3 provide preliminary support for multiple references and a specialized decoder, suggesting value beyond the new benchmark.

Critical weaknesses
  1. The comparative advantage and long-horizon contribution are not adequately isolated. Section 5.1 runs H.265 and DCVC-UF in intra-only mode because inputs are sampled at 1 FPS. A one-second interval does not establish that temporal prediction is ineffective. The results therefore establish superiority over the selected image-coding configurations, rather than practical temporal video coding more broadly.

    Table 2 varies references from one to three, and Table 4 establishes their age, but neither isolates the gain from distant observations over short-history references or generative restoration alone. Existing VQA and reconstruction metrics provide useful evidence, yet do not specifically test whether historical references overwrite changed text, object states, or novel objects. This distinction matters for preserving recorded evidence. No-reference controls should be appropriately trained or adapted, rather than created solely by removing references at inference.

  2. Rate accounting and evaluation documentation leave the headline tradeoff insufficiently verified. Section 5.1 defines complete-memory compression, whereas Appendix B uses non-keyframe ratios for reconstruction curves. Section 5.2 discusses approximately 9,000× reconstruction, but complete NRS+MGD systems in Table 1 reach at most 6,556×. These quantities need consistent separation.

    Section 3.3 also omits the NRS-A question source, held-out protocol, labeling model, and operating-point selection procedure. Appendix A excludes evaluation-video overlap for MGD, but gives no corresponding account for NRS-A or the internal ENVC dataset. Moreover, a single unweighted pass over 500 binary-scored questions produces 0.2-percentage-point increments, unlike values such as 63.1% and 74.9%. Repeated runs or other aggregation could explain this, but must be documented. These are verification gaps, not evidence of leakage or incorrect reporting.

Questions for authors
  1. Temporal baselines: How do inter-enabled H.265 and DCVC-UF perform on the same 1 FPS inputs? Please report documented settings and storage–accuracy comparisons at overlapping rates, at least on a representative subset.

  2. Historical context and fidelity: At comparable storage budgets, what gains remain over short-history references and an appropriately trained or adapted no-reference decoder? Do changing-state, text, and novel-object examples show outdated details being imported?

  3. Evaluation protocol: What data and VQA model train NRS-A, how are splits and operating points selected, and what ENVC overlap controls apply? How are the reported accuracies aggregated? Please provide uncertainty for the principal adaptive-versus-fixed comparison.

  4. Storage accounting: What are the reconstruction curves under complete-memory accounting? Are reference identifiers, rate metadata, and indexing overhead included, and how are matched references recovered during decoding? Does Figure 6 use Appendix B's non-keyframe convention?

Reviewer_2qU6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a lifelong visual-memory compression system to tackle the problem of storing the continuous video stream recorded by always-on smart glasses. Their pipeline is built around long-horizon temporal redundancy: they argue that frames recorded hours or even days apart are often highly redundant, so later observations can be stored at a much lower rate.

The system consists of three components. An Egocentric Neural Video Codec (ENVC) compresses each frame on the device, at one of many compression levels, before it is sent to the cloud. In the cloud, a Novelty-aware Rate Selector (NRS) compares the decoded frame with the keyframes already stored in memory and decides whether to store it as a new keyframe or as a highly compressed non-keyframe, and at which rate. At query time, a Memory-guided Generative Decoder (MGD) reconstructs each non-keyframe using its most similar keyframes as visual reference.

Along with the system, the authors introduce EgoCompress, a new benchmark built on EgoLife that measures how compression affects downstream VQA on fine-grained visual details. The system is evaluated on EgoCompress, EgoSchema, HD-EPIC, and EgoEverything, reaching compression ratios several times higher than the provided baselines while staying close to the accuracy obtained on uncompressed frames.

Critical strengths

Framing persistent visual memory as an extreme-compression problem, rather than as salient-frame selection or text summarization, is well motivated and it avoids irreversibly discarding evidence that unforeseen future queries may need.

Well-engineered system: The three components (ENVC, NRS, MGD) fit together coherently. In particular, a single ENVC model covers compression ratios from about 200× to nearly 10,000×, extending the operating range of existing codecs by almost an order of magnitude.

EgoCompress targets fine-grained visual details (color, texture, visible text, object state), which are the first to be lost under compression. It discriminates between compression methods far better than existing benchmarks such as EgoSchema, where even heavily compressed or text-only inputs remain close to uncompressed accuracy.

Critical weaknesses

Benchmark construction is under-specified: The paper does not state when each question is posed relative to the moment the evidence was observed. It is therefore impossible to know how much time has passed between seeing the evidence and answering the question, which is exactly the setting the benchmark is meant to measure.

Retrieval is assumed, not evaluated: The paper states that “our evaluation assumes that the target frame is known” (Section 3.1). The system therefore never has to retrieve the relevant visual evidence for a query, the target frame is selected with oracle information and then regenerated. The motivating scenario (“What brand of coffee did I drink last week?”) is never evaluated end to end, and this limitation should be stated clearly in Section 4.

Possible data leakage. Appendix A states that MGD is trained on HD-EPIC and EgoLife, which are also used for evaluation. The authors state that the training videos do not overlap with the evaluation videos, but this does not rule out leakage. EgoLife participants share the same house, and HD-EPIC is recorded in a small number of kitchens, so the generative model has likely seen the same rooms, furniture, and objects it is later asked to reconstruct. This could substantially inflate the results. I would ask the authors to report results with MGD trained without EgoLife and HD-EPIC.

DINOv2 similarity cannot detect detail changes. The pipeline appears to work overall, but my main concern is the use of DINOv2 global similarity to decide which frames are redundant. DINOv2 embeddings are largely invariant to fine details, and Figure 1(c) itself shows that two frames with different foreground content still reach a similarity of 0.9. If a detail has changed since the matched keyframe was recorded, the non-keyframe (stored at a very low rate, down to a 4×4 latent) cannot preserve it. MGD may then regenerate the old, unchanged detail from the keyframe, and the answer will be wrong but look plausible. Conversely, if MGD does reconstruct the changed detail correctly, this may be explained by the data leakage described above rather than by the method. I would ask the authors to split EgoCompress into questions where the queried attribute changed between the keyframe and the target frame and questions where it did not, and to report accuracy separately.

Questions for authors

For each question, how much time passes between the frame containing the evidence and the moment the question is asked? Could you report the distribution of these time gaps and break down accuracy by gap (e.g., minutes, hours, days)?

Retrieval. Since the evaluation assumes the target frame is known, how would the system find the relevant frame for a real query? Could you report results with an actual retrieval step instead of the oracle target frame, even with a simple retriever (e.g., text-to-image similarity over the stored keyframes)?

Data leakage:

  • How exactly were the training and evaluation splits of EgoLife and HD-EPIC defined (by video, by participant, or by environment)?
  • Could you report results with MGD trained without EgoLife and HD-EPIC, so that the evaluation environments are never seen during training?

Changed details and hallucination:

  • Could you split EgoCompress into questions where the queried attribute changed between the matched keyframes and the target frame and questions where it did not, and report accuracy separately for each?
  • On EgoCompress, ENVC + NRS-F + MGD at OP1 (64.2) outperforms uncompressed frames (63.1) with Qwen. How do you explain a reconstruction scoring higher than the original frame? Does this indicate that MGD transfers content from the keyframes rather than preserving the target frame?
  • How does the system behave when the keyframes are misleading, e.g., when an object has been moved, replaced, or changed state since the keyframe was recorded?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.