- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ConcurTree, a query-agnostic memory framework for long-video streaming under a fixed token budget. It organizes interleaved activities into persistent trees using exception-based routing, return linking, and similarity-based token merging. A fine-tuned reader is trained to use the structured memory.
Experiments on three egocentric video benchmarks show improved long-horizon performance. ConcurTree achieves 50.4% accuracy on the pooled long-horizon evaluation, compared with 43.1% for the strongest matched fixed-budget streaming baseline.
Critical strengths
- The paper addresses an important limitation of chronological video memory, where interrupted and resumed activities may become separated. Organizing related events into persistent activity threads is a reasonable direction for long-horizon video understanding.
- ConcurTree combines activity-based memory organization, exception-based routing, and budget-constrained compression. The use of a dedicated reader also addresses the difficulty of interpreting structured memory representations.
- The paper evaluates three benchmarks and compares against several streaming memory systems under different evaluation constraints. The matched-budget comparisons and long-horizon analyses provide useful evidence of the framework's effectiveness.
- The authors examine memory structures, routing policies, compression strategies, and reader adaptation. The results also identify where the method is most effective, particularly on episodic questions requiring information from temporally separated events.
Critical weaknesses
- The independent benefit of the multi-tree structure is unclear. Appendix Table 3 shows that replacing ConcurTree with a single-tree memory achieves the same 50.4% accuracy on the primary pooled long-horizon evaluation. Although multiple trees improve EgoLifeQA episodic performance, their necessity for the main overall result is not sufficiently established.
- The main mechanism lacks direct validation. The paper aims to preserve interrupted and resumed activities, but the evaluated benchmarks do not provide activity-thread annotations. Without measuring routing correctness or return-link accuracy, it is difficult to determine whether the proposed trees capture the intended activity structure.
- Performance gains depend heavily on reader adaptation. Under the frozen reader, ConcurTree does not significantly outperform alternative memory structures, while its advantage appears after structure-specific fine-tuning. This makes it difficult to separate improvements in memory organization from better adaptation to the resulting memory representation.
Questions for authors
- Appendix Table 3 shows that the single-tree variant matches ConcurTree on the primary pooled long-horizon evaluation. What evidence supports the independent contribution of maintaining multiple activity trees?
- Could the authors provide a small annotated evaluation of activity routing, particularly the precision and recall of Return decisions? This would help verify whether the proposed structure actually preserves interrupted and resumed activities.
- Why does ConcurTree outperform alternative memory structures only after reader adaptation? Could the authors clarify which properties of the activity-tree representation account for the improvement beyond general reader fine-tuning?
- Could the authors report separate uncertainty estimates for the long-horizon improvements on EgoLifeQA, Ego-R1, and TeleEgo? This would help assess whether the benefits generalize beyond the TeleEgo-dominated pooled evaluation.
5 comments
Response to Reviewer Whvq (W2Q2)
W2Q2 (direct validation of activity routing):
We agree that QA accuracy alone cannot fully establish the correctness of the inferred activity threads. We tried our best but to our best knowledge, there are no public available benchmarks to directly evaluate on. Therefore, we manually reviewed 119 gated routing decisions from the EgoLifeQA week (21 Returns and 98 new-thread decisions), plus 40 randomly sampled Continue decisions, for 159 cases. Each decision was evaluated using the original video frames and the corresponding activity-thread history. A Return is correct if the event resumes some activity, including same activities on another day (e.g. dishwashing, baking, a work session, a grocery trip).
| Measure (n=159) | Result |
|---|---|
| Return precision — a Return re-attaches the event to the correct activity | 20/21 = 95% |
| Return recall — dormant threads that should have been resumed | 20/37 = 54% |
| Judge decisions over gated events: correct link / wrong link / missed link / correctly no link | 20 / 1 / 17 / 78 |
| Unnecessary switch gate: new thread opened inside an ongoing activity at decision time | 22/119 = 18% |
| vs. at the end of video processing, after cross-tree merging | 10/119 = 8% |
| Continue: correct continues | 35/39 = 90% |
ConcurTree makes very few questionable Return decisions. Of the 21 Returns it made, 20 were clearly correct and one was ambiguous: it linked an evening party to table-setting performed that morning. Under a narrower event-level definition these may be treated as separate activities, whereas under our purpose-directed thread definition the table-setting can reasonably be considered preparation for the party.
The more common error is failing to make a Return when it should. Of the 17 missed Returns, 11 involved resuming the same episode after an interruption, for example, returning to a dance session after a five-minute detour or to a presentation after 2.7 hours. The other 6 involved recurring activities on the same or following day (which may or may not be considered errors, depending on the human annotator's disrection).
The switch gate also sometimes divides one activity into several threads. This happened in 22 of 119 decisions, such as splitting a supermarket visit again while the participant walked toward the checkout. However, the later merging stage corrected 12 of these 22 unnecessary splits.
In summary, the return precision is relatively high when the input captions and frames are reliable, but recall and activity-boundary detection remain imperfect. Here we want to mention that recall is more subjective, because it depends on what the human annotator decides is one activity and how an activity is defined. The splitting depends on the model's internal trade-off: combining related events saves storage but produces less detailed threads, while finer splitting preserves detail but creates more threads that later need to be merged for budget reasons. This actually supports our intended purpose for the model, where two activities that belong to the same overarching activity should be split up in the beginning when memory is unrestricted, but should be merged smartly as the memory hits its budget.
We added the annotation protocol, complete and filtered results, representative correct and incorrect routing examples, and limitations to the revised appendix. Together with the response for W1/Q1, this analysis provides complementary evidence for the contribution of activity-based routing while identifying opportunities to improve its recall and boundary detection.
We will address the last two questions in the next response.
Response to Reviewer Whvq (W3Q3)
W3Q3 (role of reader adaptation):
We agree that ConcurTree does not consistently outperform alternative memories with a frozen reader. This reflects an important distinction between constructing informative compressed memory and enabling the reader to interpret that representation: the frozen base Qwen3-VL-8B reader was pretrained on raw video frames, and has an internal representation according to that input type. However, all the memory methods we compare, including our own, provide compressed 128-token units that the frozen reader has never encountered during its pretraining. Therefore, reader adaptation is an intended component of our framework, rather than an assumption that the activity-tree representation should be directly interpretable by a frozen VLM. To clarify whether the improvement comes solely from reader fine-tuning or also from memory organization, we report two controlled experiments:
1. Fine-tuning does not eliminate differences between memory structures. Under the policy-specific training protocol in Figure 3, each memory structure is evaluated with its own trained adapter using the same training recipe. ConcurTree achieves versus for flat PEMF on pooled long-horizon QA (+3.0 points, p=.033), and versus on EgoLifeQA (+6.0 points, p=.005). This indicates that applying reader adaptation to both structures does not eliminate the performance difference in these evaluations.
2. The memory structure contributes even when the reader is held fixed. We tested five valid ConcurTree-trained adapter variants on both memory structures. ConcurTree performs better with four and approximately ties with one, averaging +1.7 points on EgoLifeQA and +2.9 points on questions with evidence older than 24 hours. The magnitude varies across adapters, but the result is not limited to one reader configuration.
The likely contribution of the memory structure is how evidence is grouped during compression. ConcurTree can combine observations from an activity resumed much later, whereas the single-tree variant merges temporally adjacent events. We included the additional adapter comparisons and clarify this distinction in the revised manuscript.
Response to Reviewer Whvq (W1Q1)
We thank the reviewer for the thoughtful feedback and for recognizing the motivation behind ConcurTree, its activity-based memory design, and the evaluation across three benchmarks. We also appreciate the positive assessment of our results on episodic questions involving temporally separated events. We conducted additional experiments to address the four concerns: the contribution of multi-tree memory (W1/Q1), direct validation of activity routing (W2/Q2), the role of reader adaptation (W3/Q3), and benchmark-specific uncertainty (Q4).
W1Q1 (contribution of the multi-tree structure): To clarify this result and better establish the contribution of multi-tree memory, we conducted additional analyses and controlled experiments addressing 2 questions: (1) under what conditions does multi-tree memory provide an advantage, and (2) what aspects of the multi-tree structure contribute to this advantage?
1. Multi-tree benefits are concentrated in questions requiring older evidence.
Of the 1,054 questions in the pooled long-horizon endpoint, 816 (77.4%) come from TeleEgo. Although TeleEgo streams are long (≥ 12h) and require continuous memory maintenance, the annotated evidence spans for its long-horizon questions have a median duration of 0.41 h and a maximum of 2.08 h. In contrast, EgoLifeQA and Ego-R1 include questions whose supporting evidence lies more than 12 to 24 hours in the past. Therefore, the pooled result therefore places relatively little weight on the long, even multi-day evidence regime that motivates activity resumption. We analyzed performance by benchmark and evidence age to better understand where multi-tree memory helps.
The benefit is not uniform across long-horizon questions. It is clearest on the combined week-scale ≥24 h subset and EgoLifeQA episodic questions, where ConcurTree significantly outperforms the single-tree variant. We will make this distinction explicit in the revised paper.
2. Controlled experiments further establish the contribution of multi-tree memory organization.
Beyond identifying where multi-tree memory is most effective, we also ran controlled experiments to examine whether the gains come from memory organization rather than storage capacity. For these, we used 500 EgoLifeQA questions (of which 109 questions whose evidence is older than 24h) using the ConcurTree model with a 4,096 token budget. On these 500 questions, our full system scores 38.8 compared to 35.4 from the single-tree system, and on the 109 > 24h evidenced questions, our model scores 48.6 compared to the single-tree system's 42.2.
a) The gain is not explained by additional memory capacity, but is associated with differences in how the retained evidence is organized. Both systems use the same 4,096-token budget and have matched token occupancy at all 500 question-time snapshots, with nearly identical numbers of node merges. Both continuously compress events as the video progresses, but group them differently. Single-tree nodes represent contiguous temporal intervals, with a maximum span of 3.2 hours at the end of the traced week. In contrast, 11 retained ConcurTree nodes span 6 hours to 1.7 days because they group observations from activities that were interrupted and later resumed. These spans measure the time between contributing events, not continuous footage coverage.
b) Resuming previously active threads contributes to the gain. We conducted an additional ablation on the ConcurTree structure, disabling the Return step, so once a thread becomes dormant, it is never resumed. New events can only continue the currently active thread or start a new thread. When we disable the Return step, accuracy falls to 36.2 for all 500 questions and 44.0 for the > 24h evidenced questions (both p < 0.05), and is no longer distinguishable from the single-tree structure. This provides further evidence that the Return mechanism and its resulting memory organization contribute to the gains, beyond simply maintaining multiple trees. In addition, our Return decisions are predominantly correct from a human audit, as detailed in our response to Q2.
We address the other questions in future responses.