acceptodds
Withdrawn from ICLR 2027

Proactive Memory for Ad-Hoc Recall over Streaming Dialogues

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world dialogue usually unfolds as an infinite stream. It thus requires bounded-state memory mechanisms to operate within an infinite horizon. However, existing read-then-think memory is fundamentally misaligned with this setting, as it cannot support ad-hoc memory recall while streams unfold. To explore this challenge, we introduce **STEM-Bench**, the first benchmark for **ST**reaming **E**valuation of **M**emory. It comprises over 14K QA pairs in dialogue streams that assess perception fidelity, temporal reasoning, and global awareness under infinite-horizon constraints. The preliminary analysis on STEM-Bench indicates a critical *fidelity-efficiency dilemma*: retrieval-based methods use fragment context, while full-context models incur unbounded latency. To resolve this, we propose **ProStream**, a proactive memory framework for streaming dialogues built on a hierarchical structure. It enables ad-hoc memory recall on demand by reasoning over continuous streams with multi-granular distillation. Moreover, it employs Adaptive Spatiotemporal Optimization to dynamically optimize retention based on expected utility. It enables a bounded knowledge state for lower inference latency without sacrificing reasoning fidelity. Experiments show ProStream delivers higher reasoning fidelity than prior baselines while maintaining substantially lower latency than full-context alternatives.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Streaming dialogues, Proactive memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jrzjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies long term memory of continuous dialogue streams, which needs an agent to maintain a bounded memory state and answer questions during ongoing interactions. The paper introduces STEM-bench, a benchmark constructed from LongDialQA with 14k QA pairs covering High-Fidelity Perception, Structural Logical Reasoning, and Dynamic Global Awareness. Then, the author proposes ProStream, combining semantic buffering, hierarchical memory organisation, frequency and recency retention, etc. Experiments compare ProStream with retrieval, summarisation, and memory systems, reporting improvements on several quality metrics and lower query latency. The paper also presents component ablations, parameter sensitivity studies, and qualitative analyses.

Critical strengths
  1. I think the motivation/research problem of the paper is practically important.

The paper tries to address a practically important problem, which is to support on-demand recall when conversation arrives continuously and memory capacity is limited. Also, evaluating response quality together with latency is valuable for long term conversational agents.

  1. The experiment of memory framework comparison and relevant analysis is comprehensive to some degree.

ProStream integrates buffering, hierarchical memory organisation, retention and structured retrieval into a unified pipeline. The experiments evaluate multiple memory retrieval baselines and provide ablations, comparisons, etc. The main results show ProStream is effective to some degree.

Critical weaknesses
  1. I am still confused about the novelty and evaluation validity of STEM-Bench as a streaming memory benchmark.

The paper repeatedly positions STEM-Bench as the first benchmark for streaming memory. However, the DialSim paper which introduced the LongDialQA dataset used by STEM-Bench, seems to have already simulated unfolding conversation incrementally, randomly triggered questions, online memory updates, and time constrained responses. I am confused what actually additional streaming capabilities STEM-Bench evaluates beyond this existing framework.

In addition, although STEM-Bench introduces bounded state constraints and additional QA categories, it seems the experiments do not sufficiently show their distinct contributions. In particular, the paper does not systematically examine how memory occupancy, information loss caused by retention decisions, and query performance evolve as the dialogue stream grows under a fixed memory budget.

I am also confused about some places in the benchmark construction. The original dialogues are converted into synthesised speech and then transcribed back into text, while all methods ultimately receive ASR-derived text. I am confused about the effectiveness of the process. It remains unclear what additional streaming memory capability this transformation evaluates compared with chronological text replay. Furthermore, the distinction between the new QA categories and those already available in LongDialQA is insufficiently analysed. (In other words, I am still a little confused what is the difference between the dataset and LongDialQA after I read the paper)

  1. The methodology contribution/novelty and experimental evidence do not fully establish the necessity of the proposed proactive hierarchical memory design.

ProStream includes semantic boundary detection、hierarchical summarization、entity extraction, and frequency/recency-based pruning etc., which seems closely related to existing online and hierarchical memory management approaches, including GAM, EverMemOS, xMemory, etc. The paper does not clearly explain what distinctive mechanism makes this particular combination better suited to streaming recall than existing incremental memory architectures.

In addition, the experiments seem not fully isolate the contributions of the central components. Table 1 shows that removing the short term sensing buffer largely reduces answer quality compared to other components. Is it means for long term dialogue streams, short term buffer is the most important even compared with long term maintaince or hierarchical memory tree? But I feel pity that I have not seen any deeper analysis, and any analysis about the independent contribution of hierarchical long-term memory.

Questions for authors
  1. Could you explain the distinction between STEM-Bench and existing streaming benchmarks?

It seems DialSim already supports online dialogue updates, response time constraints etc using LongDialQA. What specific evauation capability does STEM-Bench introduce beyond DialSim? In addition, could the authors clarify whether the main novelty lies in bounded state evaluation, new query dependencies, or the audio interface, and provide an analysis to show the additional difficulty introduced by these changes?

  1. Streaming Constraints and Benchmark Validity。

Could the authors report how actual memory occupancy, retention behaviour, memory-update throughput, and query latency change with increasing stream length under a fixed storage budget? In addition, why does STEM-Bench convert original text into synthetic speech and then back to text? Has the benchmark been compared with direct text streaming?

  1. Methodological Contribution and Component Effectiveness.

What are the key methodological distinctions between ProStream and existing online or hierarchical memory frameworks, such as GAM, EverMemOS, and xMemory, etc? In particular, which components are specifically designed to address challenges unique to streaming recall, rather than general long-term hierarchical memory organisation?

Moreover, Table 1 shows that removing the Short-Term Sensing Buffer causes substantially larger decreases compared to other components. Does this suggest that recent-context buffering plays a more important role than hierarchical long-term memory in the evaluated setting? Could the authors provide a more detailed analysis to clarify the independent contribution of hierarchical memory to long-range recall and reasoning?

  1. Could the authors clarify the total computational cost of online memory maintenance and query answering, including the effect of different input and query rates, rather than reporting query-response latency alone?

Overall, I think the problem is meaningful and appreciate the experimental effort. However, my concerns concern the core framing, method justification, and empirical positioning, rather than presentation alone, and may be difficult to fully address through clarification alone.

Reviewer_ntkmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces STEM-Bench, a dialogue-memory benchmark with 14,938 question-answer pairs constructed from three television-dialogue corpora and issued after their supporting evidence. It proposes ProStream, which combines a recent-context buffer, scene/event/entity-level distillation, utility-density retention, and generation from recent, pending, and retrieved memory. Table 1 reports higher composite quality scores and lower mean query latency than Full-Context, with comparisons to several retrieval and memory systems. Appendices add limited theoretical properties, sensitivity analyses, per-flush maintenance costs, and indicative LoCoMo/LongMemEval transfer results. The central claim is that proactive organization supports efficient ad-hoc recall while maintaining bounded state over a continuing stream.

Critical strengths
  1. Secs. 3 and 5 address probes issued during an evolving conversation rather than only after a completed history. The combination of recent context, newly distilled material, and retrieved hierarchical memory is a plausible way to trade query-time cost against retention. The proposed benchmark includes perception, cross-event reasoning, and temporal/aggregate tasks rather than only factual lookup.

  2. Table 1 reports query latencies of 0.408/0.373/0.316 seconds for ProStream versus 7.333/4.870/4.776 for Full-Context across the three corpora. The composite score also improves. The paper includes component ablations, encoder and capacity sensitivity, a separate maintenance-cost breakdown, and transfer results. Appendix A appropriately limits its theory to the local pruning rule and retained-state maintenance, without claiming a general knapsack approximation or a bound on every end-to-end cost.

Critical weaknesses
  1. In Sec. 5.2 (p. 6, lines 282-285), pending representations are explicitly outside the long-term retention budget, while Sec. 5.4 includes them in generation context. The paper does not give an explicit pending-state capacity, bounded output size, or sufficiently precise mounting/arrival schedule from which a total-state bound follows. Theorem A.3 bounds retained nodes under a fixed budget and explicitly excludes distillation and other processing; it cannot fill this gap. Appendix D varies retained budgets from 50k to 150k tokens, but does not report actual total occupancy, pending backlog, evictions, or old-evidence accuracy as the stream grows beyond capacity. Appendix E does report maintenance cost, and Appendix B.3 clearly separates it from query latency. However, an average maintenance time per flush and an average query response time do not establish sustainable processing at a particular arrival rate on shared resources. Required modification: specify the bounds and scheduling of every persistent/pending component, the actual main-run budget and token accounting, and the hardware/resource protocol. Add a fixed-budget chronological stress test long enough to force sustained eviction, showing total memory, backlog, and query latency—including probes overlapping updates—against stream position, with an explicit input/probe arrival schedule. A finite stress test and a sound invariant are sufficient; a literally infinite experiment is unnecessary.

  2. In Figure 7(a) and Appendix G (p. 19, lines 1013-1017), the question asks what message Michael wanted Jan to pass to Alan, while falsely describing Alan as the sole creator. The model answers the requested message, "Tell Alan that the Mets suck," which is supported by the displayed dialogue. The reference instead says Jan and Alan created the program together. That may be a desired premise correction, but it answers a different aspect of the question; the model's printed answer does not itself assert sole authorship. An explicit false-premise evaluation policy is needed before this can establish factual inconsistency. More broadly, Table 1 shows lower Gemini correctness-judge scores for ProStream than Full-Context on all three corpora (0.509/0.476/0.534 versus 0.521/0.505/0.574), as well as lower KEM. Higher Q, which mixes answer metrics with retrieved-context similarity, therefore does not establish fidelity preservation. Removing the tree reduces the judge score by only 0.009/0.010/0.008; the remaining system still has recent and pending context. Aggregate results do not establish performance on genuinely distal evidence or on the advertised temporal/aggregate tasks. Required modification: clarify and audit the QA/scoring contract, reporting human-audit size and agreement and adjudicating the illustrated case. Then report answer correctness by task and evidence age, including probes whose evidence is outside both recent and pending buffers, using the existing no-tree ablation and a no-memory control with paired uncertainty. Present correctness and latency as a trade-off unless fidelity preservation is independently supported.

Questions for authors
  1. What are the exact capacities, units, output limits, and mounting rules for B and Mpend, and what Tmax was used in Table 1? How do flush capacity 5 and carry-over window 10 interact? Which stored text, embeddings, relation attributes, and other persistent structures are charged to the reported budget? This determines whether the system's total state is bounded.

  2. What are the utterance/probe arrival schedule, hardware and resource sharing, and treatment of a probe arriving during an unfinished update? Can the authors report occupancy/backlog and tail query latency over a sustained capacity-limited replay? The separately reported maintenance cost is useful, but does not answer this scheduling question.

  3. Under the intended scoring policy, should an answer to Figure 7(a) give the supported requested message, correct the false premise, or do both? What was the human audit's sample size, sampling scheme, disagreement rate/agreement statistic, and correction procedure? Please also provide the correctness-judge prompt and its human calibration.

  4. How does full ProStream compare with the existing no-tree ablation and a no-memory model on old-evidence probes and each HFP/SLR/DGA task? Does utility-based retention improve those results over recency-only/FIFO retention at the same total budget? These controls would directly test the claimed long-term and adaptive-memory contribution. The transfer tables should also identify ProStream's actual backbone, evaluated sample counts, budget, and scoring protocol, as Appendix F itself notes.

Reviewer_ACBsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces STEM-Bench, a dialogue-memory evaluation with 14,938 question-answer pairs and probes placed after their supporting evidence. It proposes ProStream, which combines a short-term buffer, pending representations, a hierarchical memory tree, and a frequency/recency-based retention policy. The reported experiments compare answer quality and query latency with retrieval, summarization, and full-context baselines, and include component ablations and transfer results.

Critical strengths
  1. The paper studies a practically important problem: retaining useful information from ongoing conversations while keeping response latency low. Combining recent context with persistent memory is a reasonable approach to this problem.
Critical weaknesses
  1. The released implementation does not support the claimed memory bound. With the stated defaults, updated concept weights are at least 0.6, while deletion requires a weight below 0.1. This differs from the pruning rule in Section 5.3 and cannot enforce the cap. Please confirm which code version produced the results.

  2. The benefit of long-term memory is not convincingly demonstrated. Removing the memory tree reduces the judge score by only 0.008–0.010, whereas removing the short-term buffer causes much larger losses (Table 1). A recent-buffer-plus-summary baseline and results on distant-evidence queries would help establish what the tree contributes.

  3. The novelty of the streaming setting needs clarification. The cited DialSim framework already supports spontaneous questions, online memory updates, and response deadlines. The paper should explain what STEM-Bench adds beyond these capabilities, particularly which additional constraints or reasoning requirements it evaluates.

  4. The overall quality claim depends on the choice of aggregate metric. Full context performs better on judge scores and entity matching in all three corpora. Averaging these with lexical and evidence-similarity measures can obscure this difference; the discussion should distinguish lower latency from improved answer correctness.

  5. The main experimental configuration is underspecified. The exact answer-model checkpoint, GPU configuration, and default memory budget are unclear. Memory and throughput measurements should also include pending representations and update queues, since a bound on the retained tree alone does not establish bounded online operation.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.