acceptodds
Withdrawn from ICLR 2027

Understanding Streaming Memory for Progressive Context Expansion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming video understanding requires models to reason over continuously growing history under strict online computation budgets. Existing memory systems rely on increasingly complex compression, organization, and retrieval mechanisms, yet their tightly coupled designs obscure the factors that truly govern performance and efficiency. We systematically revisit streaming memory, disentangle its key design choices, and derive principles for effective memory utilization. Based on these insights, we propose TRACE (Triggered Retrieval and Adaptive Context Expansion), a training-free progressive memory framework that optimizes Recent, Short, and Long memory levels and expands historical access according to predictive uncertainty. TRACE starts from low-cost recent context and introduces additional memory only when needed. On OVO-Bench, TRACE improves Qwen3.5-9B by 9.13 percentage points overall, with gains of 10.88 points on real-time perception and 6.81 points on backward tracing, while reducing per-query latency from 19.10 s to 1.86 s. Consistent gains on StreamingBench and across multiple VLM backbones demonstrate the effectiveness of selective and progressive memory access for balancing long-horizon reasoning and efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Streaming Memory Progressive, Context Expansion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PAbZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to use memory effectively in streaming video understanding. Rather than introducing another complex memory architecture, the authors first examine three design choices: how to access past information, what evidence to present to the model, and when to retrieve additional history.

Based on these observations, they propose TRACE, a training-free framework that progressively expands the context from recent video frames to sparse short-term memory and, when needed, retrieved historical events. The decision to expand is based on the probability gap between the two most likely predicted tokens. Historical events are indexed using captions but presented to the model as visual frames with temporal information. TRACE also reuses KV caches across stages to avoid redundant computation.

On OVO-Bench, TRACE improves Qwen3.5-9B from 63.49% to 72.62% overall accuracy, while reducing per-query inference latency from 19.10 s to 1.86 s. The authors also report improvements on StreamingBench and with other VLM backbones.

Critical strengths
  1. The comparisons of memory access strategies, visual evidence, and temporal information help explain why adding more history can reduce accuracy.
  2. Without additional training, TRACE improves Qwen3.5-9B by 9.13 percentage points on OVO-Bench while substantially reducing query-time latency.
Critical weaknesses
  1. The benefit of the intermediate SM stage needs clearer evidence. Most components of TRACE have already been explored in prior work, particularly OASIS. Table 8 compares access strategies within TRACE, but does not fully separate the benefit of three-stage access from differences between TRACE and earlier approaches.

  2. TRACE gains 1.91 points over SimpleStream at roughly three times the query latency and FLOPs. Table 8 also shows gains of 0.62 and 1.16 points from successive memory expansions. Results grouped by historical evidence distance, together with statistical tests, would help assess whether these gains justify the extra cost.

  3. High-confidence errors may prevent retrieval. TRACE uses the top-two token probability margin to decide whether to stop, but the paper does not report how often it stops with an incorrect answer when relevant historical evidence is missing from the current context.

  4. The reported latency differs between Section 4.2 and Table 5: 2.31 s versus 1.86 s. Separately, some analyses use different context configurations and evaluation protocols, making direct comparisons difficult.

Questions for authors
  1. OASIS already uses uncertainty-triggered historical retrieval. Could the authors compare it with TRACE's three-stage strategy under the same memory representation, retriever, and computational budget? This would help clarify what the additional SM stage contributes.

  2. How often does TRACE stop at RC or SM with a high-confidence incorrect answer, even when the correct evidence is available in long-term memory? How much do these cases affect overall accuracy?

  3. The reported latency excludes memory construction and updates. How does TRACE compare with SimpleStream and OASIS when these costs are included, especially at different query frequencies?

1 comment
AuthorsOpenReview ↗

Response To Reviewer PAbZ

Thank the reviewer for the constructive feedback.

1. (W1, Q1) Contribution of the Intermediate SM Stage

We agree that uncertainty-triggered retrieval has been explored in prior work, including OASIS. TRACE instead analyzes complementary historical evidence and integrates it through progressive context expansion.

Table 8 compares access strategies with identical backbone, memory, and retrieval:

AccessBackwardReal-TimeOverall
RC only54.8382.9270.84
RC → SM56.8982.4471.46
RC → RM54.2082.4470.30
RC → SM → RM58.9582.9272.62

Direct RC → RM reduces Overall by 0.54 points versus RC only. Adding SM instead improves Overall by 0.62 points and Backward by 2.06 points, allowing some queries to be answered using sparse visual context and summaries. The full pipeline improves Overall by another 1.16 points over RC → SM, and by 2.32 points over RC → RM (Backward: +4.75 points). SM thus provides a complementary evidence layer, not merely extra computation.

2. (W2) Accuracy–Efficiency Trade-off and Evidence Distance

We acknowledge that TRACE requires more query-time computation than SimpleStream-4. TRACE prioritizes historical reasoning while preserving real-time perception: compared with SimpleStream-4, it trails Real-Time by 0.35 pp (82.92 vs. 83.27), but gains 4.91 pp on Backward (58.95 vs. 54.04) and 1.91 pp Overall (72.62 vs. 70.71).

Since OVO-Bench lacks complete evidence timestamps, we manually annotated 100 questions and grouped them by query-to-evidence distance dd:

Method≤10s (27)10–30s (31)>30s (42)
Vanilla70.3767.7064.28
SimpleStream77.7761.2961.90
OASIS70.3770.9669.04
TRACE74.0774.1971.40

SimpleStream performs best for near-term evidence (≤10s), where retrieval may be unnecessary. For more distant evidence, TRACE exceeds SimpleStream by 12.90 points (10–30s) and 9.50 points (>30s), and OASIS by 3.23 points and 2.36 points, respectively. This exploratory result supports TRACE's advantage on distant evidence.

3. (W3, Q2) Overconfidence and Premature Stopping

We agree that the top-1/top-2 probability margin cannot reliably detect every case of insufficient evidence. Overconfident errors can prematurely stop routing at RC or SM and miss relevant evidence.

In an additional question-level comparison across 1,468 questions, 191 answers changed from incorrect to correct, while 130 changed from correct to incorrect. Historical access can both correct errors and introduce distracting evidence. TRACE therefore balances retrieval benefits against distraction, rather than maximizing invocation.

4. (W4) Latency and Evaluation Protocol

We apologize for the inconsistency. The correct TRACE (Qwen3.5-9B) query latency is 1.86s, as reported in the Abstract and Table 5; the 2.31s in Section 4.2 is outdated and will be corrected.

Section 2 conducts controlled analyses of Access, Information, and Invocation, sometimes using fixed contexts, golden evidence, or different evidence-presentation protocols to isolate individual factors. Section 4 instead evaluates the full progressive-routing pipeline under a strict online streaming protocol.

5. (Q3) Total Cost Across Query Frequencies

We agree that query latency alone does not capture total streaming-system cost. Appendix B.2 (Table 10) reports online memory construction: 27.8s/min of video for TRACE, versus 116.8s/min for OASIS.

Let qq denote queries per minute of video. Combining average memory-maintenance and QA costs gives the following estimated cumulative compute time (seconds per minute of video):TRACE: 27.8+1.86q27.8 + 1.86q, OASIS: 116.8+13.84q116.8 + 13.84q, Vanilla: 19.10q19.10q, SimpleStream-4 / -16: 0.59q0.59q / 1.79q1.79q

TRACE is cheaper than OASIS across query frequencies. SimpleStream is cheaper still without persistent memory, but weaker on historical reasoning. Compared with Vanilla, TRACE reaches compute break-even at approximately 1.61 queries/min.

Memory construction is asynchronous and query-independent, running as video arrives. Hence the measured query-to-answer latency is 1.86s per query. Our efficiency claim concerns online responsiveness and accuracy–cost trade-offs.

Reviewer_JZy9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates streaming video memory design for online video understanding along three dimensions: memory access, information representation, and invocation. Based on empirical analyses, the authors propose TRACE, a training-free framework that progressively expands the model's context from recent frames to sparse memory and retrieved historical evidence. TRACE uses the margin between the top two predicted answer probabilities to trigger memory expansion and reuses KV caches to reduce inference overhead. Experiments on OVO-Bench (real-time and backward) and StreamingBench (real-time) demonstrate competitive performance against existing streaming video understanding methods, with improvements particularly on backward-tracing tasks.

Critical strengths

The paper is well motivated and provides a systematic analysis of the perception-memory trade-off in streaming video understanding, exploring how memory about the past can be better utilized without compromising recent context perception.

Critical weaknesses
  • W1. [Novelty] The paper largely revisits the perception-memory trade-off in OVO-Bench already highlighted by SimpleStream, and several of its central observations and design principles overlap with it. While the systematic analysis is useful, it remains unclear what fundamentally new insights emerge beyond those established by SimpleStream.

    • W1.1. In terms of results, SimpleStream with only 4 recent frames performs within approximately 2% of TRACE overall. On backward tracing, TRACE improves over the 4-frame and 16-frame SimpleStream baselines by approximately 5% and 2%, respectively. These relatively modest gains come at the cost of a considerably more complex memory pipeline involving event captioning, summarization, embedding-based retrieval, and uncertainty-guided routing, alongside higher inference latency. I am not convinced that the demonstrated improvements sufficiently justify this additional complexity.
  • W2. [Method] I find the motivation behind uncertainty-guided context expansion insufficiently justified. TRACE uses the difference between the top two predicted answer probabilities as a confidence signal to determine whether additional historical memory should be accessed. However, prediction confidence does not necessarily indicate whether relevant historical evidence is missing, particularly when the model initially has access only to recent frames. For instance, when asked about an object observed several minutes earlier, the model may confidently guess an answer based on the recent context or its language priors, without recognizing that the available context lacks the necessary evidence. This raises a fundamental question about whether the proposed confidence signal can reliably identify queries that would benefit from additional memory. The paper does not directly evaluate this failure mode. Furthermore, potential miscalibration of VLM token probabilities may further undermine the reliability of this confidence-based decision rule - this is also not discussed.

  • W3. [Experiments]

    • W3.1. [Comparison with vanilla models] Much of the discussion in Sec. 4.2 emphasizes improvements over vanilla VLMs. I find this comparison less informative, as the paper itself builds on the premise that processing extensive video history is inefficient and can degrade performance. Improvements over recency-based baseline such as SimpleStream and on-demand memory baseline like OASIS are marginal. Furthermore, it is unclear why the vanilla Qwen models exhibit such high inference latency. Since their video context could also be KV-cached, I would expect lower query-time latency comparable to TRACE, which additionally incurs retrieval and context-expansion overhead. The authors should clarify whether KV caching is applied consistently across baselines.

    • W3.2. [Scope of Benchmarks] Although the paper positions TRACE as a general framework for streaming video understanding, its evaluation focuses primarily on real-time perception and backward tracing in OVO-Bench, with additional real-time perception results on StreamingBench. These benchmarks largely assess question answering over evidence available before a query timestamp. Another important aspect of streaming video understanding, particularly proactive VideoQA [a, b], are not evaluated or discussed. This limits the extent to which the reported results support the broader claims about streaming video understanding.

  • W4. [Presentation and Clarity] The paper would benefit from more careful organization and technical exposition, particularly in Section 2. Several experimental configurations and their motivations are not clearly introduced, making the analyses difficult to follow. More generally, the manuscript repeatedly reiterates the same design principles without substantially developing the underlying arguments. A more concise presentation, with clearer definitions and a tighter connection between the empirical observations and the resulting methodological choices, would improve readability.

I would appreciate it if the authors could address the above concerns in their rebuttal, with W3.2 being a lower priority, as it primarily concerns the scope of the evaluation.

[a] ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models, arXiv 2025

[b] Don’t Pause! Every prediction matters in a streaming video, NeurIPS 2026 E&D

Questions for authors

Q1. How was the context-expansion threshold of 0.3 selected, and is the same threshold used for StreamingBench?

Some typos I noticed:

  • L149, L158: W → B?
  • L318: vistual → visual
2 comments
AuthorsOpenReview ↗

Response To Reviewer JZy9(1/2)

(W1) Novelty and Sufficiency of Improvements

We agree that the trade-off between recent perception and historical memory has been explored in prior work, including SimpleStream. Rather than introducing a fundamentally new memory mechanism, TRACE systematically investigates three key dimensions of streaming memory: Access (how to access history), Information (what evidence to provide), and Invocation (when to expand memory). Through controlled analyses, we derive empirical design principles and integrate them into a unified progressive context expansion framework.

Our goal is not merely to maximize real-time accuracy, but to improve historical reasoning while preserving strong real-time perception. Compared with SimpleStream-4, TRACE improves Backward Tracing by 4.91 points (58.95% vs. 54.04%) and Overall accuracy by 1.91 points (72.62% vs. 70.71%), with only a 0.35-point decrease in Real-Time accuracy, albeit at approximately three times the query-time cost.

More importantly, compared with SimpleStream-16 at similar latency (1.86s vs. 1.79s), TRACE reduces FLOPs by approximately 25% while improving Backward Tracing, Real-Time, and Overall accuracy by 1.74, 3.11, and 2.52 points, respectively.

Table 8 further supports the progressive design: directly expanding from RC to RM achieves only 70.30% Overall accuracy, whereas RC → SM → RM reaches 72.62%, highlighting the complementary value of the intermediate SM stage.

Thus, TRACE's contribution lies in deriving systematic memory design principles and demonstrating a favorable trade-off among real-time perception, historical reasoning, and inference efficiency. We will clarify this positioning and discuss the associated computational costs in the revision.

(W2) Reliability of Confidence-based Context Expansion

We acknowledge that the top-1/top-2 probability margin is only a proxy for evidence sufficiency, and overconfident incorrect predictions may cause premature stopping. We adopt this signal because it is training-free, lightweight, and requires no separate assessment model.

To quantify this limitation, we analyzed all 1,468 questions. Among them, 303 were answered incorrectly after high-confidence stopping at RC (margin ≥ 0.3). Counterfactual evaluation shows that 52 of these errors (17.16%) could be corrected under the RM context, including 40 Backward Tracing errors (19.80%) and 12 Real-Time errors (11.88%). Of these 52 recoverable cases, 27 could already be corrected at SM, while the remaining 25 required RM.

These results highlight both the limitation and rationale of our design: high-confidence errors can prevent beneficial memory access, yet most such errors (82.84%) remain incorrect even under the full RM context. Moreover, approximately half of the recoverable cases can be resolved at SM, supporting the value of progressive rather than unconditional memory expansion.

We will include this failure analysis and further clarify the limitations of margin-based routing under model overconfidence and miscalibration.

AuthorsOpenReview ↗

Response To Reviewer JZy9(2/2)

(W3.1, Q1) Experimental Comparisons and Threshold Selection

Baseline Comparison.

We would like to emphasize that the primary advantage of TRACE is its ability to improve historical reasoning while preserving strong real-time perception performance. Compared with SimpleStream-4, TRACE improves Backward Tracing accuracy from 54.04% to 58.95% (+4.91 points) and Overall accuracy from 70.71% to 72.62% (+1.91 points), with only a 0.35-point difference in Real-Time accuracy (82.92% vs. 83.27%). Even compared with SimpleStream-16, TRACE achieves an approximately 2-point improvement in Backward Tracing accuracy.

More importantly, compared with OASIS, which also supports historical memory access, TRACE achieves improvements in both accuracy and efficiency. While obtaining higher Overall and Backward Tracing accuracy, TRACE reduces query latency from 13.84s to 1.86s (approximately 7.4× speedup) and online memory construction cost from 116.8s/min to 27.8s/min of video (approximately 76.2% reduction).

These results suggest that TRACE not only addresses the limitations of recent-only approaches in historical reasoning, but also achieves a more favorable accuracy–efficiency trade-off compared with existing memory-based methods.

Vanilla Model Latency.

We clarify that the vanilla baseline uses KV caching. Its relatively high query latency is not simply due to repeatedly encoding the historical context, but rather to processing the entire visible video history sampled at 0.5 FPS. As the video duration increases, the accumulated visual tokens introduce substantial attention computation and memory-access overhead, which cannot be completely eliminated by KV caching.

Routing Threshold (Q1).

The routing threshold of 0.3 was selected on the OVO-Bench validation set and kept fixed across all subsequent experiments, including evaluations with different backbones and cross-dataset evaluation on StreamingBench. No additional benchmark-specific or backbone-specific threshold tuning was performed.

Furthermore, Figure 4(b) in Section 4.3 already presents a threshold ablation study, demonstrating that TRACE maintains relatively stable performance across a reasonable range of threshold values and does not rely on finely tuned thresholds.

We will further clarify the threshold selection procedure and its consistent application across different models and datasets in the revised manuscript, providing a more transparent account of the generalization capability and tuning requirements of our approach.

(W3.2) Scope of Streaming Video Understanding

We agree with the reviewer that streaming video understanding encompasses a broad range of tasks, whereas our current evaluation primarily focuses on Streaming Video Question Answering, particularly Real-Time Perception and Backward Tracing.

The central research question of this work is how a model can effectively access historical evidence under strict causal visibility constraints and a bounded active context. We therefore select OVO-Bench and StreamingBench to evaluate these capabilities.

Proactive VideoQA introduces additional requirements, such as autonomously determining when to generate predictions, initiate interactions, or take actions. These requirements differ from our current query-triggered memory access setting.

We acknowledge the importance of this direction. However, the current experiments do not provide sufficient evidence to claim that TRACE generalizes to all streaming video understanding scenarios. We will clarify the scope of our study in the revised manuscript and discuss Proactive VideoQA as a promising direction for future work, including the potential applications and challenges of progressive memory access in proactive interactive settings.

(W4) Presentation and Clarity

We thank the reviewer for the suggestions regarding the organization and clarity of the manuscript.

In the revised version, we will reorganize Section 2 to clarify the three design dimensions—Access, Information, and Invocation—and explicitly present the research questions and controlled variables for each analysis, with detailed experimental settings already provided in Appendix B. We will also strengthen the connections between empirical findings and TRACE's design choices while removing repetitive descriptions, making the progression from systematic analysis to method development clearer and more coherent.

Finally, we will carefully review and correct the notation and typographical issues identified by the reviewer, including the inconsistent W/B notation in Lines 149 and 158, and the typo “vistual” in Line 318.

Reviewer_j6LvOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper deconstructs streaming video memory along three dimensions: Access (sparse buffer / summary / retrieval), Information (frames / captions / temporal structure), and Invocation (when extra memory is needed). Oracle-combination experiments, controlled factor ablations, and MLP probing yield three principles: keep a fixed form of history by default and retrieve on demand; expose visual and temporal information directly rather than semantic compression; invoke memory progressively, since verbalized confidence is unreliable while internal signals work. Guided by these, TRACE maintains a recent window, sparse buffer, persistent event memory, and an incremental global summary; at query time it expands context RC → SM → RM using the top-1 vs top-2 probability margin as a sufficiency signal, reusing KV cache across nested contexts. On OVO-Bench with Qwen3.5-9B, TRACE improves overall accuracy by +9.13 points with per-query latency around 1.86s.

Critical strengths

1 The analysis section has standalone value. The oracle experiments quantify the headroom of on-demand access ({B,S,R} oracle 77.53 vs 63.15 for the best fixed combination), the factor analysis grounds the "keep visual detail, avoid semantic compression" principle (frames +1.12, captions -0.64), and the contrast between verbalized confidence (~50% balanced accuracy, 98% self-rated sufficient) and MLP probing (74%) is a well-executed finding for the whole streaming-memory area.

2 Principles are realized with a strong accuracy-latency trade-off. TRACE reaches 72.62 overall at 1.86s/query versus SimpleStream-16 at 70.10/1.79s, both expansion stages contribute in ablation, the margin threshold shows a wide performance plateau, and results hold across backbones and benchmarks.

Critical weaknesses

1 Inconsistent headline numbers and incomplete efficiency accounting. The abstract reports 19.10s → 1.86s while Section 4.2 reports 19.10s → 2.31s; at least one is wrong. More fundamentally, FLOPs/latency exclude memory construction, yet online maintenance is heavy (a VLM caption every 16 frames, summary updates every four chunks). The vanilla baseline pays for full-history context while TRACE shifts cost into streaming pre-processing; against SimpleStream-16 latency is essentially tied, so the speed claim is overstated as presented.

2 Limited method-level novelty and single-run evidence. Uncertainty-triggered retrieval exists in Self-RAG (cited) and WeaveTime (a baseline here); TRACE's increment is the three-level progressive design with margin routing and KV reuse. All results are single runs without significance reporting; TRACE is also not best under every comparison (SimpleStream-4 achieves higher Real-Time average, 83.27 vs 82.92).

Questions for authors

1 Which is the correct TRACE (Qwen3.5-9B) latency, 1.86s or 2.31s, and why do the abstract and Section 4.2 differ?

2 What is the amortized memory-maintenance cost per video second, and does TRACE still win end-to-end total compute over vanilla on hour-scale streams?

3 How much does margin-based routing lose relative to the MLP probe's 74% balanced accuracy as an upper reference, and on what split/layers was the probe trained?

4 Event memory retains 8 frames per event without semantic compression, so storage grows linearly with the stream; how is this reconciled with the "fixed-form history by default" principle, and is there an eviction strategy?

1 comment
AuthorsOpenReview ↗

Response To Reviewer j6Lv

Thank the reviewer for the careful reading feedback. We will clarify each concern below and will incorporate the clarifications into the revised manuscript.

1. (W1, Q1, Q2) Latency and Memory Maintenance Cost

We apologize for the inconsistent latency numbers. The correct query latency of TRACE (Qwen3.5-9B) is 1.86s, consistent with the Abstract and Table 5. The 2.31s reported in Section 4.2 is an outdated value and will be corrected.

We acknowledge that online memory construction incurs computational costs. These costs have been separately reported in Appendix B.2 (Table 10), including construction latency, GPU memory, and storage overhead. TRACE requires 27.8s/min of video, substantially lower than OASIS (116.8s/min).

To further account for memory construction, consider a video of length TT minutes with NN queries. The total computational time can be approximated as: TRACE: 27.8T+1.86N27.8T + 1.86N and Vanilla: 19.10N19.10N

The break-even point is approximately 1.61 queries per minute of video. Above this frequency, TRACE achieves lower cumulative computational time under this simplified model. Importantly, TRACE separates query-independent memory construction from query-dependent answering. Memory is maintained continuously as the video arrives, rather than reconstructed for each query. Thus, the user-perceived query-to-answer latency remains 1.86s.

We will clarify this distinction and refine our efficiency claims.

2. (W2) Method-level Novelty and Experimental Results

We agree that uncertainty-triggered retrieval is not entirely new, as explored in Self-RAG and WeaveTime. Our contribution is not a novel retrieval or confidence estimation algorithm, but rather a systematic investigation of three fundamental design dimensions of streaming memory and their integration into a unified progressive framework.

Guided by our empirical findings, TRACE organizes RC, SM, and RM into progressively expandable, nested contexts, with KV-cache reuse to avoid redundant computation. We consider this analysis-driven unified design and its empirical validation the primary methodological contribution.

Regarding reproducibility, Appendix C.1 details our standardized inference protocol, designed to minimize inference variability. We also observed consistent results across repeated runs under identical configurations.

We acknowledge that SimpleStream-4 slightly outperforms TRACE on Real-Time accuracy (83.27% vs. 82.92%). However, TRACE aims to balance real-time perception with historical reasoning rather than maximize accuracy on every individual category.

Compared with SimpleStream-4:

MetricTRACESimpleStream-4Difference
Real-Time82.92%83.27%−0.35%
Backward58.95%54.04%+4.91%
Overall72.62%70.71%+1.91%

These results demonstrate that TRACE preserves competitive real-time perception while substantially improving historical reasoning, addressing a key limitation of recent-context-only approaches.

3. (Q3) Margin-based Routing vs. MLP Probe

We clarify that the MLP Probe serves primarily as a diagnostic tool to investigate whether intermediate representations encode evidence-sufficiency signals, rather than as the router deployed during inference.

As detailed in Appendix C.4, we extract the 4096-dimensional hidden state of the final prompt token from the 24th decoder block (out of 32) of Qwen3.5-9B during the original QA prefill.

In an additional comparison, the fixed-threshold Margin Router (θ=0.3\theta=0.3) achieves 71.25% classification accuracy, compared with 75.78% for the supervised MLP Probe.

We do not claim that margin-based routing outperforms supervised probing. Instead, we adopt margin-based routing for its training-free nature, simplicity, and low overhead, while demonstrating its effectiveness in progressive context expansion.

4. (Q4) Linear Growth of Persistent Event Memory

We agree that TRACE's persistent event memory grows linearly with video duration, as visual frames and retrieval indices are retained for each event. The current implementation does not employ explicit event eviction.

However, this does not contradict our principle of “Fixed-form History by Default”, which concerns the amount of historical information exposed to the model, rather than total external storage. While TRACE's persistent event storage grows with video duration, its active historical context remains bounded through selective memory access, preventing proportional growth in per-query VLM context.

This is a deliberate trade-off: retaining event-level visual evidence avoids irreversible information loss from aggressive compression, while progressive access controls inference-time context.

We agree that long-term storage management remains important, but may compromise evidence completeness. We therefore leave these mechanisms for future investigation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.