acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Temporal Resource-Aware Chains of Evidence for Long-Range Video Understanding

Abstract

Streaming long-video understanding requires a model to answer user queries at any moment while maintaining visual and conversational context across multiple rounds. However, existing methods either reduce memory through flat representations that lose the temporal relations among events, or maintain such relations in structures that grow continuously with the stream. To address these issues, we propose TRACE, a training-free online framework that organizes streaming memory into queryable chains of evidence with resource-aware retention and retrieval. During ingestion, we compare each observation with its predicted semantic state and retain surprising event changes in full detail while merging repeated content. The retained observations are then parsed into subject-action-object event nodes and organized into a spatio-temporal chain that preserves entity trajectories and event relations. When a query arrives, an intention-driven resource-aware router formulates evidence access as a budgeted selection problem and dynamically allocates the limited resource budget to maximize the utility of complementary evidence for response generation. Extensive experiments show that our framework achieves superior performance with accuracy of 76.2% and answer-quality score of 3.88 on StreamBench. It also demonstrates high throughput of 94.05 FPS with low latency of 3.36 s and maintains generalization capabilities on the other complementary benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Range Video Understanding, Streaming Video Dialogue, Entity-Centric Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FaycOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TRACE, a training-free framework for streaming long-video understanding built around an SAO-centric memory mechanism. Unlike visual-feature-centric memory approaches, TRACE primarily organizes historical observations through text-only Subject-Action-Object (SAO) triples and entity-centric temporal trajectories. Based on these textual semantic representations, Semantic Surprise-Guided Event Filtering (SSEF) selectively retains key-frame or full-segment visual features, while an Intention-Driven Resource-Aware Router (IRR) retrieves historical evidence under resource constraints.

Critical strengths

The paper aims to address an important challenge in streaming long-video understanding: preserving historical evidence and temporal relationships under resource constraints. The proposed SAO-centric memory provides an intuitive and interpretable way to organize historical events. Its training-free design may be appealing in settings with limited training resources.

Critical weaknesses

W1. Fundamental limitations of SAO-centric memory for general video understanding. My primary concern is whether representing historical observations primarily through text-only, coarse-grained SAO triples (e.g., "man2, look at, man1") is sufficiently expressive for general video understanding. Such representations may omit substantial fine-grained perceptual information, including object attributes, scene text, spatial relationships, and subtle state changes. Although TRACE selectively retains visual features, these decisions rely on SAO-based semantic similarity and surprise. Semantic redundancy does not necessarily imply visual redundancy, and discarded visual evidence cannot be recovered for future queries. Even retained visual features may be difficult to retrieve when relevant attributes are absent from the semantic index.

Moreover, it remains unclear how such a restricted representation supports complex cross-temporal state reasoning, entity identity consistency, and evolving interactions among multiple objects. These are not exceptional cases but common requirements across practical streaming scenarios and established video understanding benchmarks. Consequently, the proposed memory appears better suited to a relatively narrow subset of coarse-grained event retrieval tasks than to the broad video understanding setting claimed in the paper. The current experiments do not adequately establish its general applicability to either offline long-video understanding or streaming video understanding.

W2. Unquantified memory costs and missing modality-specific ablations. Although TRACE claims to achieve resource-aware memory retention by selectively preserving visual features alongside SAO-based textual memory, the actual memory footprint and relative contributions of textual and visual evidence remain unclear. SSEF determines merge/key/full retention using fixed semantic thresholds, yet the proportions of retained key-frame and full-segment features, their storage costs, and memory growth with video duration are not reported. More importantly, the cumulative ablations cannot establish whether the improvements primarily originate from the SAO-based textual memory, the retained visual features, or their combination.

Questions for authors

My questions arise mainly from the methodology. It would be helpful for my understanding of this work if the authors could clarify or address them.

Q1. How would TRACE answer common visual understanding questions such as "What color was the cup previously on the table?" or "What was written on the black car that passed earlier?" Such questions require fine-grained visual evidence that may not be represented in SAO triples. How can SSEF ensure that this evidence is retained, and how can IRR retrieve it if the relevant attributes are absent from the semantic index? Would the proposed memory mechanism fail in such cases?

Q2. How does EETC preserve evolving object states, spatial relationships, and complex interactions across distant timestamps? How does it distinguish visually similar entities or repeated but temporally separate events, particularly as the number of entities increases?

Q3. How necessary and effective is SAO-centric memory? What advantages does the proposed SAO-centric memory offer over the richer text-structured event memory (e.g., OASIS, cvpr2026)? Could the authors compare SAO-only memory, SAO memory with retained visual features, and representative alternative memory structures under matched storage budgets? Such experiments are essential to isolate the contributions of textual and visual evidence and justify the restricted SAO representation.

Q4. What are the end-to-end computational and latency costs of SAO extraction, semantic prediction, and entity-trajectory updates? For example, how much wall-clock time is required to construct the proposed memory for a video longer than three minutes? Does the reported processing throughput include these operations, and how do the costs scale with video duration?

Reviewer_P8WzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TRACE, a training-free pipeline for long-range streaming video understanding. TRACE filters events using semantic surprise, organizes retained observations into entity-centric temporal chains, and routes query-time evidence at multiple fidelities under context and compute budgets. Using LongVA-7B-DPO together with frozen language and embedding models, the paper reports gains on StreamBench and several additional video benchmarks, as well as 94.05 FPS, 3.36-second total response latency, and 27.0 GB peak GPU memory.

Critical strengths
  1. Organizing long-range evidence around entity trajectories is intuitive and potentially more interpretable than flat token or frame memories. The separation between semantic event filtering, structured storage, and budgeted query-time retrieval is a useful systems decomposition.

  2. The paper attempts broad evaluation across streaming and conventional video benchmarks and reports accuracy, answer quality, latency, throughput, and GPU memory rather than accuracy alone.

Critical weaknesses
  1. TRACE does not provide the bounded memory claimed throughout the paper. Its own complexity is O(NKd_g + N_key D_key + N_full D_full), where the number of entities, transitions, key nodes, and full-fidelity nodes can all increase with stream duration. The text argues that K is typically much smaller than the number of frames, but gives no hard capacity, eviction rule, or approximation guarantee for an arbitrarily long stream. The router bounds the evidence exposed at query time; it does not bound the persistent memory accumulated during ingestion.

  2. Neither the 94.05-FPS real-time claim nor the cross-benchmark ranking is currently reliable. Event segmentation, subject-action-object extraction, identity linking, the router utility U_j,h(q), and the greedy multi-budget allocation are incompletely specified. More importantly, the reported FPS is defined using decoding/preprocessing time and the query latency starts only when the question arrives; the paper does not clearly include the online Qwen2.5-7B semantic-prediction calls, event parsing, entity-chain construction, and memory updates that are essential to TRACE. The common "accuracy" and answer-quality scores are also produced by a 4-bit Meta-Llama-3-8B-Instruct judge rather than official metrics or a human-validated evaluator.

  3. The semantic-surprise mechanism is not adequately positioned against LiveStar (NeurIPS 2025). LiveStar's SVeD uses changes in language-model perplexity to detect semantic divergence and decide when an online response has become invalid; its Peak-End memory compression further reuses model-derived semantic importance to retain keyframes and recent event summaries. TRACE applies prediction error at event ingestion rather than response verification, but both rely on model-derived semantic discrepancy to decide which moments deserve downstream computation or retention. The paper should clarify whether TRACE introduces a substantively different surprise signal, calibration objective, or retention behavior, rather than presenting semantic-surprise filtering as an isolated novelty.

Questions for authors
  1. What exact mechanism places a hard upper bound on persistent TRACE memory for an arbitrarily long stream?

  2. Does the reported 94.05 FPS include every ingestion-stage operation, particularly semantic prediction, event extraction, entity resolution, chain insertion, and embedding computation? Please provide mean/P95 end-to-end update latency and real-time factor over long streams.

  3. How does TRACE compare under matched settings with Dynamic Hub-and-Spoke Memory (arXiv:2608.30294), which also stores typed textual observations in an entity-centered structure and retrieves query-relevant evidence?

Reviewer_HdCXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRACE keeps track of who did what in a video and looks back at the relevant moments when a question comes in; the authors test it on StreamBench and six other datasets.

Critical strengths
  1. It does well on the questions where memory really matters. On StreamBench, TRACE scores 69.3% on long-term memory and 70.9% on object search, compared with ReKV’s 62.7% and 56.5%. Those results make the event-chain idea convincing to me.

  2. The speed and memory numbers are good too. It reports a 3.36-second response time and 27 GB peak GPU use without fine-tuning. That matters for a system meant to handle video as it arrives.

Critical weaknesses
  1. How does TRACE segment the video, turn clips into subject–action–object events, and keep entity IDs consistent? And what happens to answer quality when those steps go wrong?

  2. The retention rule makes a permanent call before it knows the question. An event can look repetitive now and matter later. If its visual detail is dropped, the router can’t bring it back.

  3. “Unbounded stream” needs a qualifier. Merging repeats helps, but every new event or entity can still add a node. What actually bounds memory growth?

  4. The routing objective sounds sensible, but the utility calculation is hard to pin down. How are evidence values computed, and how well does greedy downgrading optimize the two-budget objective? Without that, it’s hard to tell whether the resource-aware routing drives the gains.

Questions for authors

see critical weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.