acceptodds
Under review as a conference paper at ICLR 2027

ConcurTree: A Novel Memory Structure for Long-Video Streaming

Abstract

Vision-Language Models (VLMs) have advanced rapidly in visual perception and reasoning, yet long-video understanding in streaming applications remains challenging due to strict memory constraints and incomplete reasoning over extended temporal horizons. Existing streaming VLMs face a key trade-off: structured query-agnostic systems track objects or locations with unbounded token growth, whereas fixed-budget architectures sequentially merge adjacent events, losing interleaved activity threads when tasks are interrupted and resumed. To address these limitations, we introduce ConcurTree, a novel query-agnostic memory framework that maintains distinct trees under a fixed token budget. ConcurTree's memory structure organizes events into activity threads causally with an exception-based routing mechanism, while handling memory constraints via similarity-weighted token merging that compresses historical evidence in-place rather than evicting or flattening it. ConcurTree achieves state-of-the-art long-horizon performance among peer streaming systems across three held-out benchmarks under controlled evaluation. On our primary pooled long-horizon endpoint (n=1,054), pooled across three benchmarks, ConcurTree reaches 50.4% accuracy, compared with 43.1% for the strongest matched fixed-budget streaming baseline. On EgoLifeQA and Ego-R1, ConcurTree reaches 48.6% and 39.5% when evidence is more than 24 hours old, respectively, and achieves 52.3% on TeleEgo's long-horizon categories. These results highlight the importance of addressing activity structure for long-horizon reasoning in bounded streaming video.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-horizon memory, VLM, computer vision, egocentric video, memory representations for vision models, streaming

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WhvqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ConcurTree, a query-agnostic memory framework for long-video streaming under a fixed token budget. It organizes interleaved activities into persistent trees using exception-based routing, return linking, and similarity-based token merging. A fine-tuned reader is trained to use the structured memory.

Experiments on three egocentric video benchmarks show improved long-horizon performance. ConcurTree achieves 50.4% accuracy on the pooled long-horizon evaluation, compared with 43.1% for the strongest matched fixed-budget streaming baseline.

Critical strengths
  1. The paper addresses an important limitation of chronological video memory, where interrupted and resumed activities may become separated. Organizing related events into persistent activity threads is a reasonable direction for long-horizon video understanding.
  2. ConcurTree combines activity-based memory organization, exception-based routing, and budget-constrained compression. The use of a dedicated reader also addresses the difficulty of interpreting structured memory representations.
  3. The paper evaluates three benchmarks and compares against several streaming memory systems under different evaluation constraints. The matched-budget comparisons and long-horizon analyses provide useful evidence of the framework's effectiveness.
  4. The authors examine memory structures, routing policies, compression strategies, and reader adaptation. The results also identify where the method is most effective, particularly on episodic questions requiring information from temporally separated events.
Critical weaknesses
  1. The independent benefit of the multi-tree structure is unclear. Appendix Table 3 shows that replacing ConcurTree with a single-tree memory achieves the same 50.4% accuracy on the primary pooled long-horizon evaluation. Although multiple trees improve EgoLifeQA episodic performance, their necessity for the main overall result is not sufficiently established.
  2. The main mechanism lacks direct validation. The paper aims to preserve interrupted and resumed activities, but the evaluated benchmarks do not provide activity-thread annotations. Without measuring routing correctness or return-link accuracy, it is difficult to determine whether the proposed trees capture the intended activity structure.
  3. Performance gains depend heavily on reader adaptation. Under the frozen reader, ConcurTree does not significantly outperform alternative memory structures, while its advantage appears after structure-specific fine-tuning. This makes it difficult to separate improvements in memory organization from better adaptation to the resulting memory representation.
Questions for authors
  1. Appendix Table 3 shows that the single-tree variant matches ConcurTree on the primary pooled long-horizon evaluation. What evidence supports the independent contribution of maintaining multiple activity trees?
  2. Could the authors provide a small annotated evaluation of activity routing, particularly the precision and recall of Return decisions? This would help verify whether the proposed structure actually preserves interrupted and resumed activities.
  3. Why does ConcurTree outperform alternative memory structures only after reader adaptation? Could the authors clarify which properties of the activity-tree representation account for the improvement beyond general reader fine-tuning?
  4. Could the authors report separate uncertainty estimates for the long-horizon improvements on EgoLifeQA, Ego-R1, and TeleEgo? This would help assess whether the benefits generalize beyond the TeleEgo-dominated pooled evaluation.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Whvq (W1Q1)

We thank the reviewer for the thoughtful feedback and for recognizing the motivation behind ConcurTree, its activity-based memory design, and the evaluation across three benchmarks. We also appreciate the positive assessment of our results on episodic questions involving temporally separated events. We conducted additional experiments to address the four concerns: the contribution of multi-tree memory (W1/Q1), direct validation of activity routing (W2/Q2), the role of reader adaptation (W3/Q3), and benchmark-specific uncertainty (Q4).

W1Q1 (contribution of the multi-tree structure): To clarify this result and better establish the contribution of multi-tree memory, we conducted additional analyses and controlled experiments addressing 2 questions: (1) under what conditions does multi-tree memory provide an advantage, and (2) what aspects of the multi-tree structure contribute to this advantage?

1. Multi-tree benefits are concentrated in questions requiring older evidence.

Of the 1,054 questions in the pooled long-horizon endpoint, 816 (77.4%) come from TeleEgo. Although TeleEgo streams are long (≥\geq 12h) and require continuous memory maintenance, the annotated evidence spans for its long-horizon questions have a median duration of 0.41 h and a maximum of 2.08 h. In contrast, EgoLifeQA and Ego-R1 include questions whose supporting evidence lies more than 12 to 24 hours in the past. Therefore, the pooled result therefore places relatively little weight on the long, even multi-day evidence regime that motivates activity resumption. We analyzed performance by benchmark and evidence age to better understand where multi-tree memory helps.

ConcurTree − single tree (same reader, 4,096 tokens)nΔ (points)95% CIp
Pooled long-horizon (paper endpoint; 77% TeleEgo)1,054+0.0[−1.4, +1.4]1.00
TeleEgo, evidence >1 h102+2.9[−1.0, +7.8].38
EgoLifeQA, evidence >24 h109+6.4[0.0, +12.8].09
Ego-R1, evidence >24 h129+1.6[−1.6, +4.7].63
Week-scale >24 h (EgoLifeQA + Ego-R1 together)238+3.8[+0.4, +7.1].049
EgoLifeQA episodic questions (EventRecall / TaskMaster / HabitInsight)250+6.0[+2.4, +10.0].004
Flat store read by an adapter trained on its own states, pooled long-horizon1,054+3.0[+0.4, +5.7].033

The benefit is not uniform across long-horizon questions. It is clearest on the combined week-scale ≥24\geq24 h subset and EgoLifeQA episodic questions, where ConcurTree significantly outperforms the single-tree variant. We will make this distinction explicit in the revised paper.

2. Controlled experiments further establish the contribution of multi-tree memory organization.

Beyond identifying where multi-tree memory is most effective, we also ran controlled experiments to examine whether the gains come from memory organization rather than storage capacity. For these, we used 500 EgoLifeQA questions (of which 109 questions whose evidence is older than 24h) using the ConcurTree model with a 4,096 token budget. On these 500 questions, our full system scores 38.8 compared to 35.4 from the single-tree system, and on the 109 >> 24h evidenced questions, our model scores 48.6 compared to the single-tree system's 42.2.

a) The gain is not explained by additional memory capacity, but is associated with differences in how the retained evidence is organized. Both systems use the same 4,096-token budget and have matched token occupancy at all 500 question-time snapshots, with nearly identical numbers of node merges. Both continuously compress events as the video progresses, but group them differently. Single-tree nodes represent contiguous temporal intervals, with a maximum span of 3.2 hours at the end of the traced week. In contrast, 11 retained ConcurTree nodes span 6 hours to 1.7 days because they group observations from activities that were interrupted and later resumed. These spans measure the time between contributing events, not continuous footage coverage.

b) Resuming previously active threads contributes to the gain. We conducted an additional ablation on the ConcurTree structure, disabling the Return step, so once a thread becomes dormant, it is never resumed. New events can only continue the currently active thread or start a new thread. When we disable the Return step, accuracy falls to 36.2 for all 500 questions and 44.0 for the >> 24h evidenced questions (both p << 0.05), and is no longer distinguishable from the single-tree structure. This provides further evidence that the Return mechanism and its resulting memory organization contribute to the gains, beyond simply maintaining multiple trees. In addition, our Return decisions are predominantly correct from a human audit, as detailed in our response to Q2.

We address the other questions in future responses.

AuthorsOpenReview ↗

Response to Reviewer Whvq (W2Q2)

W2Q2 (direct validation of activity routing):

We agree that QA accuracy alone cannot fully establish the correctness of the inferred activity threads. We tried our best but to our best knowledge, there are no public available benchmarks to directly evaluate on. Therefore, we manually reviewed 119 gated routing decisions from the EgoLifeQA week (21 Returns and 98 new-thread decisions), plus 40 randomly sampled Continue decisions, for 159 cases. Each decision was evaluated using the original video frames and the corresponding activity-thread history. A Return is correct if the event resumes some activity, including same activities on another day (e.g. dishwashing, baking, a work session, a grocery trip).

Measure (n=159)Result
Return precision — a Return re-attaches the event to the correct activity20/21 = 95%
Return recall — dormant threads that should have been resumed20/37 = 54%
Judge decisions over gated events: correct link / wrong link / missed link / correctly no link20 / 1 / 17 / 78
Unnecessary switch gate: new thread opened inside an ongoing activity at decision time22/119 = 18%
vs. at the end of video processing, after cross-tree merging10/119 = 8%
Continue: correct continues35/39 = 90%

ConcurTree makes very few questionable Return decisions. Of the 21 Returns it made, 20 were clearly correct and one was ambiguous: it linked an evening party to table-setting performed that morning. Under a narrower event-level definition these may be treated as separate activities, whereas under our purpose-directed thread definition the table-setting can reasonably be considered preparation for the party.

The more common error is failing to make a Return when it should. Of the 17 missed Returns, 11 involved resuming the same episode after an interruption, for example, returning to a dance session after a five-minute detour or to a presentation after 2.7 hours. The other 6 involved recurring activities on the same or following day (which may or may not be considered errors, depending on the human annotator's disrection).

The switch gate also sometimes divides one activity into several threads. This happened in 22 of 119 decisions, such as splitting a supermarket visit again while the participant walked toward the checkout. However, the later merging stage corrected 12 of these 22 unnecessary splits.

In summary, the return precision is relatively high when the input captions and frames are reliable, but recall and activity-boundary detection remain imperfect. Here we want to mention that recall is more subjective, because it depends on what the human annotator decides is one activity and how an activity is defined. The splitting depends on the model's internal trade-off: combining related events saves storage but produces less detailed threads, while finer splitting preserves detail but creates more threads that later need to be merged for budget reasons. This actually supports our intended purpose for the model, where two activities that belong to the same overarching activity should be split up in the beginning when memory is unrestricted, but should be merged smartly as the memory hits its budget.

We added the annotation protocol, complete and filtered results, representative correct and incorrect routing examples, and limitations to the revised appendix. Together with the response for W1/Q1, this analysis provides complementary evidence for the contribution of activity-based routing while identifying opportunities to improve its recall and boundary detection.

We will address the last two questions in the next response.

AuthorsOpenReview ↗

Response to Reviewer Whvq (W3Q3)

W3Q3 (role of reader adaptation):

We agree that ConcurTree does not consistently outperform alternative memories with a frozen reader. This reflects an important distinction between constructing informative compressed memory and enabling the reader to interpret that representation: the frozen base Qwen3-VL-8B reader was pretrained on raw video frames, and has an internal representation according to that input type. However, all the memory methods we compare, including our own, provide compressed 128-token units that the frozen reader has never encountered during its pretraining. Therefore, reader adaptation is an intended component of our framework, rather than an assumption that the activity-tree representation should be directly interpretable by a frozen VLM. To clarify whether the improvement comes solely from reader fine-tuning or also from memory organization, we report two controlled experiments:

1. Fine-tuning does not eliminate differences between memory structures. Under the policy-specific training protocol in Figure 3, each memory structure is evaluated with its own trained adapter using the same training recipe. ConcurTree achieves 50.4%50.4\% versus 47.3%47.3\% for flat PEMF on pooled long-horizon QA (+3.0 points, p=.033), and 38.8%38.8\% versus 32.8%32.8\% on EgoLifeQA (+6.0 points, p=.005). This indicates that applying reader adaptation to both structures does not eliminate the performance difference in these evaluations.

2. The memory structure contributes even when the reader is held fixed. We tested five valid ConcurTree-trained adapter variants on both memory structures. ConcurTree performs better with four and approximately ties with one, averaging +1.7 points on EgoLifeQA and +2.9 points on questions with evidence older than 24 hours. The magnitude varies across adapters, but the result is not limited to one reader configuration.

The likely contribution of the memory structure is how evidence is grouped during compression. ConcurTree can combine observations from an activity resumed much later, whereas the single-tree variant merges temporally adjacent events. We included the additional adapter comparisons and clarify this distinction in the revised manuscript.

Reviewer_bzqMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a ConcurTree as a new memory structure of long videos. It consists of a short-term memory to tackle “now and short” windows, a definition of event node, the construction and evolution of node tree based on exception-based routing, and memory compression that dictate the tree memory under fixed token budget. Combined with a LoRA fine-tuned reader LLM, the memory is shown to boost performance of long-video QA in 3 benchmarks.

Critical strengths

The core memory structure in the format of ConcurTree is logical and plausible. The mechanisms are well-designed and explained clearly. The experiments are comprehensive and the results show substantial improvements in the targeted tasks, namely long-horizon video QA.

Critical weaknesses

On the motivation, while interleaved events are prevalent and present a critical challenge, it is not the only barrier to effective video understanding. For example, visual diversity: some events are inherently more dynamic with large scene changes while others may be more stable; Events duration: it can be short or long based on the event nature. By focusing on event interleaving, it is not clear whether the other event characteristics are dually taken care of. This issue may have an impact on the validity of exception-based routing.

Questions for authors

How does the proposed method tackle alternative challenges in event-based memory on top of concurrence? The mechanism of freezing nodes in the now-short window is based on similarity between event units. what is the basis/rationale of this freezing mechanism? What happen if u3 and u1 are least similar? How does a change of token budget (4096) affect the model performance? In other words, any guidelines on how to set the budget threshold based on horizon length? Are there any examples and qualitative results to illustrate the use cases, especially success and failure cases, pertaining to the methodology?

4 comments
AuthorsOpenReview ↗

Response to Reviewer bzqM (W1/Q1)

We thank the reviewer for the constructive feedback and for recognizing the strengths of ConcurTree's memory design and experimental evaluation. We agree that activity interleaving is not the only challenge in long-video understanding. Our primary focus is organizing and preserving interleaved activities under a fixed memory budget, while event segmentation and visual dynamics remain important related challenges. In response to the review, we clarify how our framework handles variations in visual dynamics and event duration (W1/Q1), explain the short-term freezing mechanism and provide additional memory-budget experiments (Q2), and present qualitative success and failure cases with visual evidence and memory traces (Q3). We will incorporate these clarifications and results into the revised manuscript.

W1/Q1 (alternative challenges in event-based memory): We agree that visual diversity and event duration are important challenges beyond activity interleaving. We would like to clarify that ConcurTree primarily focuses on organizing and preserving interleaved activities under a fixed memory budget. Accordingly, we place less emphasis on developing new event segmentation techniques or comprehensively modeling variations in visual dynamics. Nevertheless, our framework already incorporates SOME adaptive mechanisms to account for these factors, as detailed below.

Visual diversity. Event boundaries are detected using SigLIP feature changes normalized against a running window of 21 recent observations. This allows segmentation to adapt to local visual dynamics rather than relying on a fixed threshold. The activity-switch gate similarly uses an adaptive threshold based on running statistics of event differences.

Event duration. Event spans vary between 4 and 16 frames, with eight frames uniformly sampled for encoding. Longer activities can extend across multiple memory nodes, which are subsequently organized and compressed within their activity threads. Thus, the framework does not assume fixed-duration activities.

We also conducted some sensitivity experiments on EgoLifeQA to assess the impact of these components. Replacing adaptive segmentation with fixed-length chunks reduces accuracy from 38.8% to 33.8%, and from 48.6% to 41.3% on questions requiring evidence older than 24 hours. Coarser segmentation further reduces >24 h accuracy to 40.4%, while finer boundaries preserve the original 48.6%. Changing the switch-gate sensitivity also lowers >24 h accuracy to 44.0% or 42.2%. A separate human audit found that 20 of 21 judgeable Return decisions (95%), correctly linked related activities.

We have also included the sensitivity analysis in the appendix and clarify that more robust handling of highly dynamic scenes, subtle activity transitions, and variable event duration is an important direction for future work.

We continue our response in the next comment.

AuthorsOpenReview ↗

Response to Reviewer bzqM (Q2, Q3)

Q2 (freezing mechanism and token budget): The rationale of the freezing mechanism is based on the idea that we shouldn't immediately decide which tree to compress the most recent memory to, but at the same time, we cannot indefinitely store too many nodes in an intermediate space and not compress them. We want to let boundaries between nodes fall at a locally lowest similarity boundary instead of as a fixed rule of "always compress nodes by twos" or "always insert each node into the tree structure immediately". We choose a size of 3 most recent events to consider for compression and not yet compress into the memory structure; out of the three, we either put the single most recent event into the structure, or we merge the two most recent events into the structure. Regarding the scenario if u1 and u3 are least similar, we don't compare non-adjacent events, because in any case of pushing a node into the memory structure, we will definitely push u3 (the most recent node) inside, we just need to decide whether the next event, u2, is more similar to u3 than u1, in which case we also compress it and add it to the memory structure in the same pass.

For memory budget, we conducted a sweep from 512 to 4,096 tokens on EgoLifeQA, keeping all other settings fixed.

Budget (tokens)5121,0241,5362,0484,096
ConcurTree (all)34.436.434.635.838.8
ConcurTree (>24 h)39.444.039.442.248.6
Single tree (all)36.235.435.435.835.4
Single tree (>24 h)43.145.045.945.942.2

The results show that ConcurTree benefits from sufficient memory capacity, achieving its best performance at 4,096 tokens among the tested budgets. Importantly, this remains a strictly bounded memory setting: on the 52-hour EgoLifeQA stream, the 4,096-token store reaches capacity after only 3.36 hours of recorded footage and must continuously compress subsequent events into approximately 32 memory units. Under this constraint, ConcurTree outperforms the single-tree baseline by 3.4 percentage points overall and 6.4 points on questions requiring evidence older than 24 hours. This demonstrates the benefit of activity-aware organization when substantial video history must be preserved within a fixed memory budget.

For budget selection, rather than using video duration alone, we recommend considering the number of distinct activities that must remain accessible over the target horizon. A budget of BB tokens provides approximately B/128B/128 memory units, making the required capacity dependent on both activity density and horizon length. Our experiments also reveal a limitation: below 4,096 tokens, ConcurTree does not consistently outperform the single-tree baseline, suggesting that multi-tree organization requires enough capacity to retain distinct activity threads. If the number of distinct activity threads is so small, then the performance will naturally similar as a single tree. We will include this guideline and the budget sweep in the revised manuscript.

Q3 (examples and qualitative results): We thank the reviewer for this suggestion. We have added qualitative success and failure examples to the revised manuscript, showing the original video frames, activity-thread timelines, memory states at query time, and model predictions under evidence removal. Two representative cases are summarized below (as the figures cannot be embedded directly in this response, we only provided text discussion here) and more are provided in the appendix.

Success (q199). A question on Day 3 asks who accompanied the participant during an earlier supermarket visit on Day 1. ConcurTree answers correctly (C), while the single-tree baseline predicts B. The relevant evidence is retained in thread T6, which contains 13 events and one Return. Removing the evidence-containing memory unit changes ConcurTree's prediction from C to B, whereas removing a control unit leaves the correct answer unchanged. This illustrates how activity-based memory organization can support the use of earlier evidence.

Failure (q260). A question on Day 4 asks what the participant last took from the refrigerator on Day 1. Although ConcurTree retains the relevant evidence in thread T7, it incorrectly predicts ice cream instead of vegetables, while the single-tree baseline answers correctly. Removing the evidence unit does not change ConcurTree's prediction, suggesting that the reader fails to use the retained information effectively.

These examples illustrate both the benefit and limitation of our approach: activity-based memory can help preserve and use temporally distant evidence, but successful retention does not guarantee correct reasoning. Since we can not include the figure here, the revised manuscript will include the corresponding visual examples and memory traces for further inspection.

AuthorsOpenReview ↗

Follow-up on Rebuttal Responses

Hello Reviewer bzqM,

We sincerely thank you for your thoughtful review and valuable feedback. We have carefully addressed your concerns in our rebuttal, including additional experiments and clarifications where appropriate and revised our manuscript to incorporate your feedbacks.

We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns or if any questions remain. If there is anything further we can clarify or provide that might help you revisit your rating, we would be happy to do so.

Thank you again for your time and consideration.

I'd like to thanks the authors' for the clarification. I will retain my positive rating of 3.

Reviewer_yH9ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ConcurTree, a query-agnostic, fixed-budget memory framework for streaming long-video understanding. Unlike conventional chronological memory, ConcurTree organizes events into persistent activity trees to capture interrupted and resumed activities. It employs exception-based routing for activity assignment and similarity-weighted token merging for memory compression. A fine-tuned Qwen3-VL reader is used for question answering. Experiments on three egocentric benchmarks demonstrate improvements.

Critical strengths
  1. The paper addresses an important challenge in streaming video understanding: preserving historical evidence from interleaved and resumed activities under bounded, query-agnostic memory constraints.
  2. The proposed activity-tree structure and exception-based routing provide a reasonable approach to associating temporally distant events belonging to the same activity, while within-tree merging maintains a fixed memory budget.
Critical weaknesses

W1. Insufficient characterization of the dynamic memory structure. Although maintaining persistent activity trees is the central contribution, the paper provides limited analysis of how these trees evolve as video duration increases. The number of trees, their token allocations, and their compression dynamics are important for understanding the behavior and scalability of ConcurTree.

W2. Limited analysis of the computational overhead introduced by the return-linking judge. The exception-based routing mechanism invokes Qwen-VL to compare the current event with candidate activity threads. However, the paper does not sufficiently quantify the frequency and latency of these judgments or their contribution to overall streaming inference cost, particularly as the number of dormant trees increases.

W3. Insufficient evidence for the proposed memory structure under tighter token budgets. The default 4,096-token long-term memory budget is relatively generous for resource-constrained streaming applications. Although the authors provide a 1,024-token evaluation, a more systematic budget–accuracy analysis is needed. Additionally, the single-tree ablation achieves the same 50.4% pooled long-horizon accuracy as the full model, leaving the practical advantage of maintaining multiple activity trees insufficiently clear.

Questions for authors

Q1. How does the number and size of activity trees evolve with video duration? Appendix Table 3 mentions approximately 32 trees for an EgoLife week. Could the authors provide, on at least one benchmark, statistics stratified by video duration (e.g., number of trees, tokens per tree, and their distributions)? Visualizing tree growth and token allocation over time would help clarify the actual behavior of the proposed memory structure under a fixed budget.

Q2. What are the exact implementation details and computational costs of the return-linking judge? Could the authors provide the complete input prompt and output format used by the Qwen-VL judge, including how candidate activity captions are compared and how the final decision is obtained? More importantly, what are the average and maximum inference latencies per judgment, and how frequently is the judge invoked during streaming?

Q3. How does ConcurTree perform under more restrictive memory budgets? Could the authors extend the current 1,024/4,096-token comparison to a finer-grained sweep, such as approximately 500, 1,000, 1,500, and 2,000 tokens (or corresponding 64-token-aligned budgets)?

4 comments
AuthorsOpenReview ↗

Response to reviewer yH9Z (W1/Q1)

We thank the reviewer for recognizing the importance of preserving evidence from interleaved activities under fixed memory constraints, and the value of our activity-tree structure and exception-based routing. To address the reviewer's concerns, we provide additional analyses of memory evolution (W1/Q1), Return-judge implementation and computational cost (W2/Q2), and performance under tighter memory budgets (W3/Q3). We will incorporate these results and clarifications into the revised manuscript.

W1/Q1 (characterization of the dynamic memory structure): We agree that understanding how the memory evolves is important for evaluating ConcurTree. We provide detailed memory-evolution statistics from an EgoLifeQA week (two tables below), along with additional observations from a TeleEgo day. The two reports are provided here for your convenience and have also be included in the updated Appendix for the manuscript.

Stream hour (A1)1246813243652
Trees117131829323232
Nodes61439373536323233
Tokens7041,6643,9684,0964,0324,0964,0964,0964,096
Tokens/tree: median / max704 / 7041,664 / 1,664192 / 1,920128 / 1,024128 / 768128 / 384128 / 128128 / 128128 / 192
Within-tree merges / root merges0 / 00 / 00 / 021 / 040 / 090 / 0198 / 21308 / 49455 / 71
Gated switches / Returns0 / 00 / 07 / 114 / 221 / 435 / 774 / 22107 / 27133 / 31

Overall, ConcurTree scales in storage by converting continued stream growth into additional compression operations under a fixed 4,096-token budget through three stages: (1) Initial growth: Its number of trees initially grows with the number of detected activities. (2) Within-tree compression and tree expansion: Once the memory budget is reached, new entries trigger compression of adjacent nodes within existing trees, freeing capacity for new activities. Consequently, the number of trees increases from 7 to approximately 32 while the total memory remains bounded. (3) Steady state: The principal consequence of longer or denser video is therefore greater compression pressure, rather than growth in memory size.

We also report phases of memory construction on EgoLife. The 4,096-token budget corresponds to 32 units of 128 tokens each.

PhaseStream hoursTreesTokens per treeWhat happens
Initial growth0 → 4.5 h1 → 7One tree holds everything: 1,664 tokens at 2 h; median 192, maximum 1,920 at 4 hEvents are appended as 128-token units; the gate starts after 16 events.
Within-tree compression and tree expansion4.5 → 14.4 h7 → 32Maximum 1,920 → 128Every new entry forces one merge of two adjacent nodes within a tree; the budget shifts from a few large trees to many single-unit threads.
Steady state14.4 → 52 h32Typically 128; transiently larger during updatesA Return attaches to a thread and is merged into its unit. A new thread forces a cross-tree merge of two time-adjacent roots; the first occurs at 14.8 h, with 71 by the end.

We observe similar behavior on the denser TeleEgo stream: the memory reaches approximately 32 trees after 2.5 hours and performs 303 cross-tree merges. The number of trees stabilizes near 32 because of the fixed token budget, rather than growing indefinitely with video duration. Importantly, the activity structure still preserves temporally distant relationships: at the end of the week, 11 ConcurTree memory units span more than six hours between their contributing events, with a maximum span of 39.6 hours, compared with a maximum of 3.2 hours for the single-tree baseline under the same budget.

We continue our response in the next comment.

AuthorsOpenReview ↗

Response to reviewer yH9Z (W2/Q2, W3/Q3)

Here we continue our response for Q2 and Q3.

W2/Q2 (Return-judge implementation and computational cost): The return-linking judge, for computational efficiency, is invoked only when the visual switch gate detects a potential activity change, using an adaptive threshold based on the running statistics of event differences. When triggered, the judge was asked once per dormant thread with the following prompt:

Two moments from the same kitchen video:
A: <caption of the new event>
B: <latest caption of the dormant thread>
Are A and B part of the same ongoing activity (for example, the same recipe or the same chore)? Answer Yes or No.

The judge uses the base Qwen-VL weights with the answerer adapter disabled. We compute the normalized first-token probability of Yes versus No for each candidate. P(Yes) = p(Yes variants) / (p(Yes) + p(No)). The best P(Yes) is at least 0.01, the event to that thread's latest node (Return). Otherwise, a new thread is opened.

We measured the computational cost using real event captions on an NVIDIA H100 GPU (300 caption pairs). Frequencies are measured from traced runs on EgoLifeQA and TeleEgo. Latency is measured on an NVIDIA H100 using 300 real caption pairs.

QuantityEgoLifeQA A1 (52 h)TeleEgo P1 (13.4 h)
Events triggering judge133/843 (15.8%)578/6,021 (9.6%)
Calls per triggered eventMedian: 6 (first 6 h) → 18 → 31–32 (steady state); max 36Median: 8 (first hour) → 31 (steady state); max 34
Total judge calls3,58616,315
Per-call latency: mean / p50 / p95 / max35.9 / 35.0 / 41.8 / 58.9 ms35.9 / 35.0 / 41.8 / 58.9 ms
Latency per triggered event (10 / 25 / 32 candidates)0.36 / 0.90 / 1.15 s0.36 / 0.90 / 1.15 s
Total estimated judge time≈ 2.1 min≈ 9.8 min
Estimated share of ingest compute≈ 3%≈ 2%

The cost per triggered routing decision grows linearly with the number of dormant trees, but is bounded by the budget. At 4,096 tokens, there are typically approximately 31 dormant threads (= the number of nodes 32 - 1) at steady state, although transient updates can produce more (maximum 36 observed)). Per-judgment latency averages 35.9 ms (median 35.0 ms, p95 41.8 ms, maximum 58.9 ms). With 32 candidate threads, a triggered routing decision takes approximately 1.15 seconds using sequential comparisons. The measurements indicate that the judge adds a relatively small fraction of the overall ingestion cost, which is dominated by visual event encoding. We have also included the complete prompt, decision rule, and latency measurements in the updated appendix.

The calls across the dormant trees are independent and could be batched, although we used sequential comparisons in the reported experiments.. For completeness, a human audit of 21 Return decisions on the EgoLife week finds the judge's Returns correct in 20 of the 21 cases (a Return counts as correct if it resumes the episode or the same activity on another day).

W3/Q3 (performance under tighter memory budgets): To examine memory-budget sensitivity, we conducted a sweep from 512 to 4,096 tokens on EgoLifeQA, keeping all other settings fixed for both ConcurTree and the single-tree baseline.

Budget (tokens)5121,0241,5362,0484,096
ConcurTree (all)34.436.434.635.838.8
ConcurTree (>24 h)39.444.039.442.248.6
Single tree (all)36.235.435.435.835.4
Single tree (>24 h)43.145.045.945.942.2

The results show that ConcurTree benefits from sufficient memory capacity, achieving its best performance at 4,096 tokens among the tested budgets. Importantly, this remains a strictly bounded memory setting: on the 52-hour EgoLifeQA stream, the 4,096-token store reaches capacity after only 3.36 hours of recorded footage and must continuously compress subsequent events into approximately 32 memory units. Under this constraint, ConcurTree outperforms the single-tree baseline by 3.4 percentage points overall and 6.4 points on questions requiring evidence older than 24 hours. This demonstrates the benefit of activity-aware organization when substantial video history must be preserved within a fixed memory budget.

Regarding the pooled single-tree tie (50.4%), we continue our response in the next comment.

AuthorsOpenReview ↗

Response to reviewer yH9Z (W3) pooled single-tree tie (50.4%)

W3 (Additionally, the single-tree ablation achieves the same 50.4% pooled long-horizon accuracy as the full model, leaving the practical advantage of maintaining multiple activity trees insufficiently clear.)

To clarify this result and better establish the contribution of multi-tree memory, we conducted additional analyses and controlled experiments addressing 2 questions: (1) under what conditions does multi-tree memory provide an advantage, and (2) what aspects of the multi-tree structure contribute to this advantage?

1. Multi-tree benefits are concentrated in questions requiring older evidence.

Of the 1,054 questions in the pooled long-horizon endpoint, 816 (77.4%) come from TeleEgo. Although TeleEgo streams are long (≥\geq 12h) and require continuous memory maintenance, the annotated evidence spans for its long-horizon questions have a median duration of 0.41 h and a maximum of 2.08 h. In contrast, EgoLifeQA and Ego-R1 include questions whose supporting evidence lies more than 12 to 24 hours in the past. Therefore, the pooled result therefore places relatively little weight on the long, even multi-day evidence regime that motivates activity resumption. We analyzed performance by benchmark and evidence age to better understand where multi-tree memory helps.

ConcurTree − single tree (same reader, 4,096 tokens)nΔ (points)95% CIp
Pooled long-horizon (paper endpoint; 77% TeleEgo)1,054+0.0[−1.4, +1.4]1.00
TeleEgo, evidence >1 h102+2.9[−1.0, +7.8].38
EgoLifeQA, evidence >24 h109+6.4[0.0, +12.8].09
Ego-R1, evidence >24 h129+1.6[−1.6, +4.7].63
Week-scale >24 h (EgoLifeQA + Ego-R1 together)238+3.8[+0.4, +7.1].049
EgoLifeQA episodic questions (EventRecall / TaskMaster / HabitInsight)250+6.0[+2.4, +10.0].004
Flat store read by an adapter trained on its own states, pooled long-horizon1,054+3.0[+0.4, +5.7].033

The benefit is not uniform across long-horizon questions. It is clearest on the combined week-scale ≥24\geq24 h subset and EgoLifeQA episodic questions, where ConcurTree significantly outperforms the single-tree variant. We will make this distinction explicit in the revised paper.

2. Controlled experiments further establish the contribution of multi-tree memory organization.

Beyond identifying where multi-tree memory is most effective, we also ran controlled experiments to examine whether the gains come from memory organization rather than storage capacity. For these, we used 500 EgoLifeQA questions (of which 109 questions whose evidence is older than 24h) using the ConcurTree model with a 4,096 token budget. On these 500 questions, our full system scores 38.8 compared to 35.4 from the single-tree system, and on the 109 >> 24h evidenced questions, our model scores 48.6 compared to the single-tree system's 42.2.

a) The gain is not explained by additional memory capacity, but is associated with differences in how the retained evidence is organized. Both systems use the same 4,096-token budget and have matched token occupancy at all 500 question-time snapshots, with nearly identical numbers of node merges. Both continuously compress events as the video progresses, but group them differently. Single-tree nodes represent contiguous temporal intervals, with a maximum span of 3.2 hours at the end of the traced week. In contrast, 11 retained ConcurTree nodes span 6 hours to 1.7 days because they group observations from activities that were interrupted and later resumed. These spans measure the time between contributing events, not continuous footage coverage.

b) Resuming previously active threads contributes to the gain. We conducted an additional ablation on the ConcurTree structure, disabling the Return step, so once a thread becomes dormant, it is never resumed. New events can only continue the currently active thread or start a new thread. When we disable the Return step, accuracy falls to 36.2 for all 500 questions and 44.0 for the >> 24h evidenced questions (both p << 0.05), and is no longer distinguishable from the single-tree structure. This provides further evidence that the Return mechanism and its resulting memory organization contribute to the gains, beyond simply maintaining multiple trees. In addition, our Return decisions are predominantly correct from a human audit, as detailed in our response to Q2 from reviewer Whvq.

AuthorsOpenReview ↗

Follow-up on Rebuttal Responses

Hello Reviewer yH9Z,

We sincerely thank you for your thoughtful review and valuable feedback. We have carefully addressed your concerns in our rebuttal, including additional experiments and clarifications where appropriate and revised our manuscript to incorporate your feedbacks.

We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns or if any questions remain. If there is anything further we can clarify or provide that might help you revisit your rating, we would be happy to do so.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.