- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents StreamArena, an evaluation benchmark for continuous, interactive, long-duration video understanding. It addresses the limitation of existing video understanding evaluations, which rely on short videos and multiple-choice questions and thus fail to measure real streaming comprehension capabilities.
In addition, the authors propose StreamMind, a two-layer multi-agent architecture. It achieves coordination between low-latency responses and long-term multimodal memory via front-end real-time interaction and active monitoring, as well as back-end asynchronous memory construction and retrieval. Experiments demonstrate that StreamMind outperforms the evaluated streaming baselines across all four capabilities.
Critical strengths
- The dataset has advantages in scale and annotation quality. The benchmark consists of 243 videos with an average duration of 88.8 minutes and 3,646 open-ended question-answering tasks spanning seven distinct scenarios.
The authors adopt multi-stage manual cross-validation and independent review, and attach precise timestamp annotations for questions and supporting evidence, which helps enhance evaluation reliability and causal consistency.
- The experimental results show a favorable performance-latency trade-off. StreamMind outperforms the evaluated streaming baselines on all four tasks.
In comparisons using the same base model, the system reduces average response latency by 66.2% while retaining approximately 89.7% of the overall accuracy. This illustrates the potential of continuous memory construction and state reuse for improving efficiency.
Critical weaknesses
- StreamMind adopts Qwen3.5-397B-A17B as its base model, while its primary streaming baselines AURA, MiniCPM-o-4.5, StreamForest and ThinkStream employ Qwen3-VL-8B-Instruct (8B), Qwen3-8B (8B), Qwen2-7B (7B), and Qwen2.5-VL-3B (3B), respectively.
These base models exhibit substantial discrepancies in scale and capability. Therefore, the performance advantage of StreamMind across the four tasks cannot be fully attributed to the proposed multi-agent architecture.
- The paper lacks independent ablation studies on key components including the Memory Writer, hierarchical event memory, entity relation graph, and keyframe retrieval. As such, the actual contribution of each module to the final performance gains cannot be clearly identified.
I recommend the authors supplement streaming architecture comparisons under the same backbone and module ablation experiments to strengthen the persuasiveness of the experimental conclusions.
Questions for authors
Refer to Critical Weaknesses.
-
For weakness 1, how abour performance of our work with baseline's backbone LLM? Or performance of baselines with our backbone LLM?
-
For weakness 2, I recommend the authors supplement module ablation experiments to strengthen the persuasiveness of the experimental conclusion.
qualitative explanations are provided in Section 3.2 on Page 8.
- History recall: The authors argue that continuously maintained multimodal memory, especially the joint retrieval of structured events and keyframes, is the major contributor to improved long-duration understanding.
- Tool use: The authors attribute the 228.1% relative performance gain to the collaboration among Router, Recall and Search Workers.
- Proactive interaction: The authors identify independently running Monitor Workers as the key to performance gains in proactive interaction.
- Response latency: Section 3.3 attributes latency reduction to persistent state reuse, which continuously builds memory before questions arrive and avoids reprocessing historical videos for every query.
However, these are all the authors’ interpretations of the overall system experimental results, rather than conclusions validated via module-level experiments. I recommend the authors supplement quantitative and rigorous module ablation experiments.
5 comments
Response to Weakness 2 (2/2)
Response to Q2. Worker roles and memory ablations
Thank you for highlighting the value of StreamArena and for asking how its components contribute. The Recall Worker retrieves past evidence for historical questions, the Search Worker looks up external information for tool questions, and the Monitor Worker issues an alert when a requested event occurs. Removing any of these workers would disable the corresponding operation. For example, without the Monitor Worker, StreamMind could never issue an alert and would score zero on proactive interaction.
We therefore keep all workers and the Qwen3.5-397B-A17B backbone fixed and vary one memory component at a time across all 877 historical retrospection (HR) questions. The variant without the event hierarchy keeps micro events and removes the macro and super levels. We report accuracy (%) overall and by the time between the supporting evidence and the question.
| Configuration | HR avg. | L1 (≤5 min) | L2 (5–15 min) | L3 (15–30 min) | L4 (>30 min) |
|---|---|---|---|---|---|
| Full StreamMind | 34.9 | 31.5 | 46.7 | 34.6 | 17.1 |
| No event hierarchy | 30.1 | 30.4 | 41.8 | 27.0 | 11.6 |
| No entity graph | 34.1 | 30.9 | 45.8 | 33.5 | 16.6 |
| No historical key frames | 27.4 | 26.5 | 36.4 | 26.5 | 12.7 |
Historical key frames contribute most to overall HR accuracy. Removing them lowers accuracy from 34.9% to 27.4%, a drop of 7.5 percentage points. Without these frames, Recall cannot revisit visual details omitted from the event descriptions. Removing the event hierarchy reduces overall accuracy by 4.8 points. Among the three ablations, this change causes the largest accuracy drop on L4, from 17.1% to 11.6%. This suggests that grouping related events helps Recall locate evidence more than 30 minutes after it appears. Removing the entity graph has the smallest overall effect (0.8 points). Its links support more specific queries, such as tracing a character’s relationships across scenes.
We have added the memory ablation results for all 877 HR questions, including the breakdown by evidence-to-question gap, to Appendix C.8 (Table 12).
Reponse to Rebuttal
While I appreciate the authors' additional experiments, I believe Weakness 2 is only partially addressed. The new ablations provide useful evidence regarding the contributions of hierarchical event memory, the entity relation graph, and historical keyframes to HR performance. However, the Memory Writer itself remains unevaluated, and the contributions of the overall multi-agent coordination mechanism are still unclear. In particular, the current experiments do not sufficiently validate the authors' explanations for the improvements in tool utilization, proactive interaction, and response latency.
The authors also provide insufficient explanations and justifications for the individual components and design choices. Your rebuttal also seems to merely list numerical results without providing sufficient analysis of the underlying mechanisms or explaining why these improvements occur.
I also note that another reviewer, TpSn, raised essentially the same concern: "StreamMind employs multiple specialized workers and integrates hierarchical event memory, an entity relation graph, and keyframe storage. However, the paper lacks sufficient module-level ablation studies to clarify the individual contributions of these components." This suggests that the concern is not merely a minor experimental detail, but a fundamental limitation in the current evaluation.
Overall, StreamMind presents an interesting engineering effort, but its architecture appears highly handcrafted, with multiple specialized workers and heuristically designed memory components. The interactions among these components are insufficiently analyzed, making the overall design difficult to interpret and its individual contributions difficult to establish.
There have already been numerous similar works on Video Agents. Without a clear explanation of the novelty and rationale behind their design choices, I am left with the impression that this work is merely an amalgamation of existing ideas.
Although the system demonstrates promising empirical results, I do not think the current evidence sufficiently supports the proposed architectural contributions at the level expected for publication at ICLR. Consequently, I am lowering my rating from 3 (Weak Accept) to 2 (Weak Reject).
Response to Weakness 1 (1/2)
Response to W1. Streaming methods with the same 397B backbone
Thank you for recognizing the value of StreamArena and for raising the question of how much backbone scale contributes to the results. In Table 4, we use each baseline’s released checkpoint and inference procedure. For AURA and VST, those checkpoints incorporate method-specific fine-tuning. Simply replacing them with a larger model would discard what the original models learned during that training. We therefore evaluated the released systems first, while recognizing that their scores reflect both the backbone and the method.
To compare the methods at the same model scale, we have adapted the inference procedures of AURA, VST, StreamForest, and ThinkStream to the same Qwen3.5-397B-A17B checkpoint used by StreamMind, without additional fine-tuning. In these 397B runs, AURA uses at most 30 recent frames, while VST uses at most 8,192 visual tokens per segment, following their respective original settings. The comparison covers all 243 StreamArena videos (3,646 questions), with the same scoring rules for every method.
Accuracy is reported in percent. HR L1–L4 group questions by the time from supporting evidence to the question (≤5 min, 5–15 min, 15–30 min, and >30 min). Pro. L1–L3 group questions by the time from the monitoring instruction to the target event (≤30 s, 30 s–4 min, and >4 min).
A dash (-) denotes a capability unsupported by the method’s evaluation interface.
With the backbone held fixed, StreamMind reaches 34.9% HR accuracy, compared with 30.2% for VST, the strongest baseline on this task.
† With the 397B backbone, HR accuracy improves by 4.0 percentage points for StreamForest and 2.8 for ThinkStream. These gains may be modest because we used the checkpoint without training it to use StreamForest’s merged visual tokens or the reasoning tokens that ThinkStream keeps in its KV cache after evicting older visual tokens.
We have added the comparison using the shared 397B backbone to Appendix C.9 (Table 13).