- Rating
- 2
- Confidence
- 1(−2)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies long-term memory for LLM conversational agents. The authors show that LLM-based memory construction loses evidence and costs a lot when conversations get long and mixed. They propose Threader, which keeps raw dialogue, splits it into topic segments with a small BERT model, and retrieves segments with multi-view embeddings, BM25 and reranking. Experiments on LongMemEval and PersonaMem show better QA accuracy.
Critical strengths
-
The motivation study is clear. The session concatenation experiment and the case studies show well how Mem0 and MemU lose evidence.
-
Threader get the best overall accuracy on both LongMemEval and PersonaMem, and the gain holds with GPT and Qwen backbones.
-
The paper gives many useful details in the appendix, such as the judge prompt, the response prompts and the segmenter training settings.
Critical weaknesses
-
The novelty is limited. The core idea, splitting memory into topic segments, was already studied in SeCom. Authors basically turn SeCom’s prompt-based segmentation into a trained model, and the later recall, rerank and answer steps are incremental.
-
As a method that improve QA by better chunking, the paper do not report detailed retrieval results. LongMemEval provide standard retrieval metrics like Recall and NDCG, but authors only report their self-defined Full-R and Mean-R, which is not reasonable.
-
The released code can not reproduce the reported results. The key input, the segmented data, is not provided. Also the segmenter and its training code, and the baseline scripts are missing.
-
The paper says Threader is slower than Self-RAG because of the extra segmentation step, but the timing code don’t include segmentation at all. Authors should explain this gap between the code and the paper.
Questions for authors
-
Could the authors analyze the main differences between Threader and SeCom, besides replacing prompt-based segmentation with a trained model?
-
Could the authors report standard retrieval metrics, such as Recall@k and NDCG@k, following its official protocol?
-
Could the authors release the segmented data, the segmenter weights and its training code, and the scripts for running baselines?
-
The authors should explain whether the 18.33s construction time includes the segmentation step. Could the authors report the segmentation time separately?
6 comments
Response to W2 & Q2
W2/Q2: Official LongMemEval metrics and retrieval results
W2: As a method that improve QA by better chunking, the paper do not report detailed retrieval results. LongMemEval provide standard retrieval metrics like Recall and NDCG, but authors only report their self-defined Full-R and Mean-R, which is not reasonable.
Q2: Could the authors report standard retrieval metrics, such as Recall@k and NDCG@k, following its official protocol?
Our Full-Recall@k uses the same definition as LongMemEval’s official recall_all@k. We use the name Full-Recall to distinguish it from Mean-Recall, an additional metric we report. Specifically, Full-Recall@k measures the fraction of queries for which all gold evidence is retrieved within the top-k results, whereas Mean-Recall@k averages the fraction of gold evidence retrieved per query.
We additionally report retrieval results following LongMemEval’s official retrieval evaluation protocol, we report Recall@k and NDCG@k and compare Threader with SeCom:
| Method | Recall@5 (%) | Recall@10 (%) | Recall@12 (%) |
|---|---|---|---|
| SeCom | 60.00 | 75.11 | 77.87 |
| Threader | 82.55 | 91.28 | 93.40 |
| Method | NDCG@5 | NDCG@10 | NDCG@12 |
|---|---|---|---|
| SeCom | 0.6168 | 0.6623 | 0.6704 |
| Threader | 0.8158 | 0.8369 | 0.8412 |
Threader consistently outperforms SeCom across all tested k values, providing more complete evidence coverage and better ranking quality.
Response to W3&Q3 AND W4&Q4
W3/Q3: Released data and code for reproducibility
W3: The released code can not reproduce the reported results. The key input, the segmented data, is not provided. Also the segmenter and its training code, and the baseline scripts are missing.
Q3: Could the authors release the segmented data, the segmenter weights and its training code, and the scripts for running baselines?
We apologize for the incomplete initial release. The segmented data were initially omitted because their size exceeded GitHub’s file-size limits. We have now released the segmented data, segmenter weights, and training code on Anonymous Hugging Face. The provided code and input data enable reproduction of Threader's reported results. For baselines, we use scripts from the respective methods' official GitHub repositories. We are currently constructing an integrated evaluation environment incorporating multiple baselines, and will release this integrated evaluation framework upon publication.
W4/Q4: Construction time breakdown and segmentation overhead
W4: The paper says Threader is slower than Self-RAG because of the extra segmentation step, but the timing code don’t include segmentation at all. Authors should explain this gap between the code and the paper.
Q4: The authors should explain whether the 18.33s construction time includes the segmentation step. Could the authors report the segmentation time separately?
The 18.33 s per 100K input tokens reported in Table 4 is measured end-to-end and includes 6.07 s for segmentation. The released indexing script starts from pre-segmented data, so its timer does not cover segmentation.
Excluding segmentation, Threader's construction time is 12.26 s, slightly below Self-RAG's 13.18 s. Threader also achieves lower query-time latency (8.66 s vs. 93.52 s) and higher LongMemEval accuracy with GPT-5-mini (90.60% vs. 76.60%).
Response to W1 & Q1
Thank you for your thoughtful comments! We address your comments point by point below.
W1/Q1: Differences from SeCom and additional comparisons
Threader and SeCom Both retrieve topic-organized dialogue. The key differences are:
We additionally tested SeCom on LongMemEval and PersonaMem. On LongMemEval, accuracy under comparable retrieved-context token budgets is:
Threader outperforms SeCom across all three backbones, with gains of 5.60 and 9.80 percentage points on GPT-5-nano and GPT-5-mini, respectively. The advantage is especially pronounced on Qwen3.5-plus, reaching 31.40 percentage points (87.00% vs. 55.60%).
On PersonaMem, SeCom's prompt-based segmenter could not process the long sessions in our tested configuration, while Threader's incremental segmenter succeeded, highlighting its practical value for long conversational histories.
Regarding cost, SeCom construction takes 127.28 s per 100K input tokens and incurs $0.07 in segmentation-related LLM charges; Threader incurs no such LLM charges. Retrieval latency is 8.13 s (SeCom) versus 8.66 s (Threader). Threader thus delivers higher accuracy with comparable retrieval latency and no segmentation-related LLM costs.