- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies long term memory of continuous dialogue streams, which needs an agent to maintain a bounded memory state and answer questions during ongoing interactions. The paper introduces STEM-bench, a benchmark constructed from LongDialQA with 14k QA pairs covering High-Fidelity Perception, Structural Logical Reasoning, and Dynamic Global Awareness. Then, the author proposes ProStream, combining semantic buffering, hierarchical memory organisation, frequency and recency retention, etc. Experiments compare ProStream with retrieval, summarisation, and memory systems, reporting improvements on several quality metrics and lower query latency. The paper also presents component ablations, parameter sensitivity studies, and qualitative analyses.
Critical strengths
- I think the motivation/research problem of the paper is practically important.
The paper tries to address a practically important problem, which is to support on-demand recall when conversation arrives continuously and memory capacity is limited. Also, evaluating response quality together with latency is valuable for long term conversational agents.
- The experiment of memory framework comparison and relevant analysis is comprehensive to some degree.
ProStream integrates buffering, hierarchical memory organisation, retention and structured retrieval into a unified pipeline. The experiments evaluate multiple memory retrieval baselines and provide ablations, comparisons, etc. The main results show ProStream is effective to some degree.
Critical weaknesses
- I am still confused about the novelty and evaluation validity of STEM-Bench as a streaming memory benchmark.
The paper repeatedly positions STEM-Bench as the first benchmark for streaming memory. However, the DialSim paper which introduced the LongDialQA dataset used by STEM-Bench, seems to have already simulated unfolding conversation incrementally, randomly triggered questions, online memory updates, and time constrained responses. I am confused what actually additional streaming capabilities STEM-Bench evaluates beyond this existing framework.
In addition, although STEM-Bench introduces bounded state constraints and additional QA categories, it seems the experiments do not sufficiently show their distinct contributions. In particular, the paper does not systematically examine how memory occupancy, information loss caused by retention decisions, and query performance evolve as the dialogue stream grows under a fixed memory budget.
I am also confused about some places in the benchmark construction. The original dialogues are converted into synthesised speech and then transcribed back into text, while all methods ultimately receive ASR-derived text. I am confused about the effectiveness of the process. It remains unclear what additional streaming memory capability this transformation evaluates compared with chronological text replay. Furthermore, the distinction between the new QA categories and those already available in LongDialQA is insufficiently analysed. (In other words, I am still a little confused what is the difference between the dataset and LongDialQA after I read the paper)
- The methodology contribution/novelty and experimental evidence do not fully establish the necessity of the proposed proactive hierarchical memory design.
ProStream includes semantic boundary detection、hierarchical summarization、entity extraction, and frequency/recency-based pruning etc., which seems closely related to existing online and hierarchical memory management approaches, including GAM, EverMemOS, xMemory, etc. The paper does not clearly explain what distinctive mechanism makes this particular combination better suited to streaming recall than existing incremental memory architectures.
In addition, the experiments seem not fully isolate the contributions of the central components. Table 1 shows that removing the short term sensing buffer largely reduces answer quality compared to other components. Is it means for long term dialogue streams, short term buffer is the most important even compared with long term maintaince or hierarchical memory tree? But I feel pity that I have not seen any deeper analysis, and any analysis about the independent contribution of hierarchical long-term memory.
Questions for authors
- Could you explain the distinction between STEM-Bench and existing streaming benchmarks?
It seems DialSim already supports online dialogue updates, response time constraints etc using LongDialQA. What specific evauation capability does STEM-Bench introduce beyond DialSim? In addition, could the authors clarify whether the main novelty lies in bounded state evaluation, new query dependencies, or the audio interface, and provide an analysis to show the additional difficulty introduced by these changes?
- Streaming Constraints and Benchmark Validity。
Could the authors report how actual memory occupancy, retention behaviour, memory-update throughput, and query latency change with increasing stream length under a fixed storage budget? In addition, why does STEM-Bench convert original text into synthetic speech and then back to text? Has the benchmark been compared with direct text streaming?
- Methodological Contribution and Component Effectiveness.
What are the key methodological distinctions between ProStream and existing online or hierarchical memory frameworks, such as GAM, EverMemOS, and xMemory, etc? In particular, which components are specifically designed to address challenges unique to streaming recall, rather than general long-term hierarchical memory organisation?
Moreover, Table 1 shows that removing the Short-Term Sensing Buffer causes substantially larger decreases compared to other components. Does this suggest that recent-context buffering plays a more important role than hierarchical long-term memory in the evaluated setting? Could the authors provide a more detailed analysis to clarify the independent contribution of hierarchical memory to long-range recall and reasoning?
- Could the authors clarify the total computational cost of online memory maintenance and query answering, including the effect of different input and query rates, rather than reporting query-response latency alone?
Overall, I think the problem is meaningful and appreciate the experimental effort. However, my concerns concern the core framing, method justification, and empirical positioning, rather than presentation alone, and may be difficult to fully address through clarification alone.