acceptodds
Withdrawn from ICLR 2027

ReSight: Reconstructing Fine-Grained Visual Evidence for Streaming Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding a changing world requires reasoning about evidence that is no longer visible. Streaming video understanding demands this ability under bounded memory, yet observations arrive before their relevance to future questions is known. We introduce ReSight, which reconstructs fine-grained visual evidence from a compact persistent state. Our key insight is to give memory a stable visual reconstruction target while learning separately how recovered evidence should influence an answer. ReSight maintains a fixed-size visual field through recursive least-squares updates in fixed temporal and appearance coordinates. At query time, a lightweight reader reconstructs spatially resolved features and pools them into a bounded visual context, deferring question-dependent interpretation to the decoder. To account for imperfect reconstruction, asymmetric attention preserves a perception reference within a shared decoder, and a learned signed gate regulates the deviation proposed by history-informed reasoning. This separates preserving observations from deciding their influence, without replaying past frames or optimizing parameters at query time. Trained on only 10% of StreamTTT's training data, ReSight surpasses StreamTTT on seven of eleven task metrics across OVO-Bench and StreamingBench and matches it on one. It improves action sequence identification by 10.1 percentage points and sequential question answering by 7.20 points, while raising real-time visual understanding accuracy by 2.76 points. Moreover, ReSight reduces per-frame ingestion time by 19% and persistent state storage by 57% relative to StreamTTT. These results highlight the effectiveness of reconstructive memory with higher data efficiency and inference efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Memory for streaming LMMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UgrBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on streaming video understanding and proposes ReSight. It stores observations in a fixed-size visual field via ridge regression, reconstructs past visual evidence by querying that memory at a few time points, and processes it through a two-branch Transformer where a special attention mask protects current perception from reconstructed history. It evaluates on OVO-Bench (Real-Time Perception and Backward Tracing) and StreamingBench (RTVU and SQA) on selected subsets.

Critical strengths
  1. The fixed-size visual field seems new, which keeps storage bounded and makes ingestion cheaper than StreamTTT.
  2. The erasure ablation shows the memory carries usable history beyond the native cache
Critical weaknesses
  1. Benchmark coverage is partial and the paper does not clearly say so, omitting OVO-Bench's Forward Active Responding and most of StreamingBench's other task types.
  2. It does not report the results of the base model Qwen3-VL-4B, so readers cannot tell how much gain comes from the method versus the stronger base model.
  3. The writing makes the method harder to follow: dense notation, lacking necessary citations, and motivations often stated as claims rather than argued.
  4. The repeated "10% data" claim comes with no data-scaling curve, so it is unclear how the 10% was chosen or whether the advantage holds at other fractions.
  5. Despite the paper's "fine-grained" framing, there is no evaluation of how faithfully the reconstructed evidence quality match the original observations.
Questions for authors

Please refer to the weaknesses.

Reviewer_1CRbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses a fundamental challenge in streaming video understanding: preserving historical visual evidence under bounded memory when future queries are unknown. The authors propose ReSight, which maintains a Fixed-Coordinate Visual Memory through recursive least-squares updates before queries arrive, reconstructs historical features into a bounded evidence prefix at query time, and integrates them with current perception through a Protected Perception Reference and Signed Fusion.

Critical strengths
  • The paper systematically discusses existing streaming memory mechanisms and identifies an important challenge in query-independent preservation of historical visual evidence.
  • Formulating persistent visual memory as a regularized reconstruction problem is interesting. The reported improvements over StreamTTT, particularly on ASI and SQA, are encouraging.
Critical weaknesses
    1. Unclear clarity of many key concepts. The method section repeatedly introduces memory-related concepts without adequately clarifying their individual roles, relationships, or connections to the proposed memory mechanism. For instance, Section 2.1 introduces fixed temporal coordinates, appearance coordinates, and the visual field, followed by visual evidence reconstruction, the evidence prefix, and the protected perception reference in subsequent sections. Although the mathematical operations are specified, the paper does not clearly establish how these concepts collectively define what the memory represents, how historical evidence is preserved and reconstructed, and how it is utilized for reasoning.
    1. The proposed method does not convincingly overcome the limitations identified in the introduction. ReSight still compresses historical observations into a fixed-capacity visual field and subsequently reconstructs and pools them into bounded evidence tokens. As video duration increases under a fixed memory budget, interference and reconstruction errors may cause fine-grained visual details to be lost. Moreover, the proposed gated fusion regulates historical influence rather than recovering information already lost during memory construction. The paper provides insufficient discussion or experimental evidence demonstrating how such details remain recoverable over long video streams.
    1. Insufficient explanation of performance improvements. ReSight improves ASI by 10.1 points, compared with only 1.1 points on EPM. However, it remains unclear why the proposed visual reconstruction mechanism particularly benefits action-sequence reasoning. Meanwhile, HLD decreases by 10.8 points, raising concerns about the reliability of reconstructed historical evidence.
Questions for authors

Q1. Could the authors clarify the definition and interpretation of the visual field, particularly how the temporal, appearance, and spatial coordinates interact?

Q2. Why is the ridge-regression objective in Eq. (1) particularly suitable for historical visual evidence preservation? How do the regularization strength and fixed temporal basis affect reconstruction quality and long-term retention?

Q3. How does reconstruction fidelity change with increasing video duration and evidence-query delay? Can the fixed-capacity visual field reliably preserve short-lived, fine-grained visual details in long videos? Direct reconstruction-quality measurements and fine-grained historical recognition experiments would strengthen this claim.

Q4. What explains the substantially larger improvement on ASI than EPM? Which component contributes most to temporal action-sequence reasoning, and why does HLD deteriorate considerably?

Reviewer_tdCkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReSight for streaming video understanding under bounded memory. In this setting, frames arrive continuously, and evidence often appears before anyone knows whether a future question will need it. Storing all visual tokens makes memory grow with the stream. Its core idea is to decouple preserving observations from deciding how history influences the answer.

  1. Fixed-coordinate visual memory. Each temporal patch’s visual features are RMS-normalized, projected onto a fixed PCA basis, and resampled to a 12×12 grid with a log-scale channel. These targets are keyed by fixed Fourier time encodings. A fixed-size linear memory is fitted to them online via recursive least-squares, so no past frames are retained.
  2. Visual evidence reconstruction. At query time, 16 evenly spaced timestamps are probed to reconstruct historical feature grids. A lightweight residual decoder refines them, and they are average-pooled into 1,024 evidence tokens. Both probing and pooling are question-independent.
  3. Reasoning with a protected perception reference. A single shared decoder computes a perception path that cannot see the reconstructed evidence and a history path that can, separated by an asymmetric attention mask. A learned signed gate fuses the two at each decoding step. Auxiliary losses supervise the history path and penalize fusion when it underperforms perception on real-time questions.
Critical strengths
  1. The method is interesting. By using the closed-form RLS solution, the memory is updated analytically without any test-time gradient optimization, and the paper shows clear advantages over StreamTTT.

  2. The problem is well defined. Not knowing future questions in advance is a key characteristic of streaming memory, and separately handling writing and reading memory under this constraint is insightful.

  3. The donor-memory substitution and zero-evidence ablation are interesting diagnostics for testing whether the stored memory actually carries video-specific information.

  4. The perception protection design is reasonable. The asymmetric attention mask structurally guarantees that the perception path is not affected by the reconstructed evidence.

Critical weaknesses
  1. Most memory-related experiments are evaluated on the ER subset of OVO-Bench, which contains only a few dozen questions. Results at this scale are prone to fluctuation and noise. Could the authors evaluate on additional benchmarks, particularly ones with longer videos?

  2. Latency is only compared against StreamTTT. Could the authors also compare with a standard VLM baseline? I am curious whether the reconstruction step introduces additional latency, especially at query time. It would also be helpful to see curves showing how latency and memory grow as the input length increases.

  3. The temporal encoding is built from periodic functions. With the paper’s parameters, timestamps that are far apart may end up more correlated than nearby ones, which could cause interference in long streams.

  4. It is unclear how the readout mechanism recovers fine-grained information. A clearer and more concise explanation would be helpful.

Questions for authors

The qualitative examples suggest that fine-grained objects can be captured from the reconstructed features, which is very interesting. Could the authors explain this capability in more detail?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.