- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes LGMem, a long-term memory system for conversational question answering. It links original sentences and dialogue segments through extracted entities and keywords, without generative model calls during memory construction. At query time, graph-based retrieval gathers evidence, while a reflection step identifies missing information and guides further retrieval. On LongMemEval-S and BEAM, LGMem improves the primary evaluation scores over the compared methods while using fewer generative tokens for memory construction and query-time reasoning (Tables 1–2).
Critical strengths
-
The design addresses a real difficulty in long-term memory. Keeping the original sentences, rather than only summaries, helps preserve details that may matter to future questions. Checking for evidence gaps before answering is particularly useful when facts are spread across conversations.
-
The evaluation supports the main result. The gains appear across different history lengths, and the ablations show that both graph retrieval and reflection contribute. The paper also reports confidence intervals for its main score comparisons (Section 3.3 and Table 9).
Critical weaknesses
-
The advantage over other iterative memory-retrieval approaches is not yet clear. The paper compares against several memory systems, but not directly against the iterative approaches discussed in its related work. The ablations establish that LGMem’s components help, but do not fully clarify what this particular combination adds relative to those approaches.
-
The efficiency claim has a narrower scope than “lightweight” may suggest. The main token comparison excludes final answer generation and does not capture all non-generative computation or storage. The runtime analysis is helpful, but is limited to one benchmark setting, making the end-to-end trade-off at larger history scales harder to judge.
Questions for authors
-
Clarification: What is the key advantage of LGMem over prior iterative memory-retrieval approaches under a comparable retrieval and reflection budget? Which of the current results best isolate that advantage?
-
Clarification: How should readers interpret the reported token savings in terms of end-to-end cost? In particular, could encoding, retrieval, or graph storage meaningfully change the efficiency comparison at the 500K and 1M history scales?
5 comments
Table 3. BEAM Judge macro scores and token costs with GPT-5.6-Luna.
BEAM-100K (400 questions)
| Method | Judge (%) ↑ | Build Tokens (M) ↓ | Reasoning + Answer Tokens (M) ↓ | Total Tokens (M) ↓ | Calls ↓ |
|---|---|---|---|---|---|
| MRAgent | 56.51 | 1.083 | 6.050 | 7.133 | 2,137 |
| REMem-I | 58.21 | 5.389 | 4.780 | 10.168 | 1,718 |
| LGMem | 67.12 | 0.000 | 8.812 | 8.812 | 1,292 |
BEAM-500K (700 questions)
| Method | Judge (%) ↑ | Build Tokens (M) ↓ | Reasoning + Answer Tokens (M) ↓ | Total Tokens (M) ↓ | Calls ↓ |
|---|---|---|---|---|---|
| MRAgent | 57.74 | 6.125 | 12.366 | 18.491 | 4,496 |
| REMem-I | 59.81 | 38.919 | 8.570 | 47.489 | 3,433 |
| LGMem | 64.82 | 0.000 | 20.188 | 20.188 | 2,988 |
BEAM-1M (700 questions)
| Method | Judge (%) ↑ | Build Tokens (M) ↓ | Reasoning + Answer Tokens (M) ↓ | Total Tokens (M) ↓ | Calls ↓ |
|---|---|---|---|---|---|
| MRAgent | 57.65 | 8.074 | 13.613 | 21.686 | 5,003 |
| REMem-I | 57.06 | 70.690 | 8.658 | 79.347 | 3,421 |
| LGMem | 65.34 | 0.000 | 18.083 | 18.083 | 2,614 |
At BEAM scales of 100K, 500K, and 1M, LGMem exceeds the second-best method in this comparison by 8.9, 5.0, and 7.7 points, respectively, with the fewest LLM calls. Relative to REMem-I, total tokens decrease by 13.3%, 57.5%, and 77.2%. These results support the combination of lightweight graph construction and stateful reflection for QA quality and token efficiency.
MRAgent constructs persistent memory from user messages only, which account for approximately 8.5%, 7.0%, and 5.4% of full-history tokens at the three scales. This reduces construction input but leaves information appearing only in assistant messages outside persistent memory.
Together, these comparisons show that lightweight graph construction and stateful reflection give LGMem an advantage in the balance of QA quality and total token consumption over the compared iterative memory systems.
(3) Fine-grained ablations of the reflection module. Following the method-level and system-level comparisons above, we ablate explicit state, temporal processing, and reflection with GPT-5.6-Luna, covering 35 histories and 700 questions.
Table 4. Fine-grained ablations of the reflection module on BEAM-1M.
| Configuration | Judge (%) ↑ | Δ (pp) |
|---|---|---|
| LGMem | 66.33 | — |
| w/o State | 63.40 | −2.93 |
| w/o Temporal | 64.84 | −1.49 |
| w/o Reflection | 58.66 | −7.67 |
Removing reflection, explicit state, and temporal processing lowers scores by 7.67, 2.93, and 1.49 points, respectively, indicating the largest contribution from iterative retrieval, with state maintenance and temporal processing also aiding evidence integration.
W2 / Q2. Scope of “lightweight” and end-to-end cost at 500K/1M
“The main token comparison excludes final answer generation and does not capture all non-generative computation or storage. […] Could encoding, retrieval, or graph storage meaningfully change the efficiency comparison at the 500K and 1M history scales?”
Response. Thank you for suggesting a more complete efficiency analysis. We added total token consumption including final answers, complete-pipeline timing including all LLM calls, and index-storage and runtime-memory measurements at 500K/1M.
(1) Total tokens including final answers. As shown in Tables 1 and 2 of the submitted manuscript, LGMem offers advantages in both QA quality and token consumption. At BEAM scales of 500K and 1M in Table 2, LGMem reduces memory-processing tokens by 66.57% and 82.16%, respectively, relative to the second-best method by score, SimpleMem. Table 3 in this response compares the iterative memory systems on BEAM, with total tokens including final answering. LGMem scores higher than both MRAgent and REMem-I at 500K and 1M. Relative to REMem-I, it improves Judge scores by 5.0 and 8.3 points while reducing total tokens by 57.5% and 77.2%, respectively. Thus, LGMem retains advantages in both QA quality and token efficiency when final answering is included. Below, we further examine complete-pipeline time and storage costs.
(2) Complete-pipeline timing. Using one L20 and the same preloaded Qwen3.6 service, each method processes the same history and all 20 questions at each scale. We report amortized time per question, including initialization, construction, and final answering.
Table 5. Complete-pipeline amortized time per question (s).
| Method | 100K | 500K | 1M |
|---|---|---|---|
| Dense RAG | 10.82 | 8.18 | 9.33 |
| LinearRAG | 9.24 | 11.56 | 16.35 |
| RAPTOR | 33.92 | 75.62 | 99.48 |
| LightMem | 42.38 | 122.37 | 165.36 |
| SimpleMem | 64.45 | 137.88 | 457.26 |
| REMem-I | 47.30 | 69.03 | 147.83 |
| MRAgent | 169.20 | 340.70 | 388.70 |
| LGMem | 35.84 | 45.82 | 54.56 |
LGMem takes 35.84, 45.82, and 54.56 s per question across the three scales, below LightMem, SimpleMem, REMem-I, and MRAgent at every scale. Dense and LinearRAG remain faster, reflecting the tradeoff between multi-round evidence integration and runtime.
(3) LGMem stage times. We break down the same complete pipelines, counting construction once per history.
Table 6. LGMem complete-pipeline components (s).
| Scale | Initialization | Build | 20-question QA | Total |
|---|---|---|---|---|
| 100K | 7.61 | 17.49 | 691.69 | 716.78 |
| 500K | 7.50 | 61.14 | 847.75 | 916.39 |
| 1M | 7.66 | 129.85 | 953.68 | 1091.19 |
At 500K and 1M, graph construction accounts for 6.7% and 11.9% of total runtime, respectively, and is not the main time cost. Including construction preserves the relative efficiency conclusions above for these long-history settings.
(4) Index storage and runtime memory. We further report index size and peak local-process RAM to assess storage and runtime memory costs for long histories.
Table 7. LGMem index storage and peak RAM.
| Scale | Index (MiB) | Peak RAM (GiB) |
|---|---|---|
| 100K | 20.63 | 1.52 |
| 500K | 83.51 | 1.54 |
| 1M | 165.23 | 1.76 |
At 1M, the index size is 165.23 MiB and peak local-process RAM is 1.76 GiB. Together with the complete-pipeline timing results, these measurements show that non-generative processing and storage costs remain manageable and do not offset LGMem's relative efficiency advantages at the tested long-history scales, further supporting its lightweight design.
Dear Reviewer 67v6,
Thank you for your careful review of our work and your constructive feedback. Our detailed responses are provided below.
W1 / Q1. Advantage over prior iterative access under comparable settings
Response. We appreciate your suggestion to clarify the comparison with prior iterative methods. LGMem combines lightweight graph retrieval with explicit task state: the graph connects related original evidence, while reflection uses available information and unresolved gaps to guide subsequent retrieval. We examine this design through comparisons of reflective access methods on the same graph, evaluations against existing iterative memory systems, and fine-grained ablations of the reflection module.
(1) Reflective access on shared graph memory. We compare IRCoT-style and Self-RAG-style access on the same LGMem graph over all 500 LongMemEval-S questions, using GPT-5.6-Luna and the same final-answer settings, and report scores and token consumption.
Table 1. Access controllers on shared graph memory: GPT-5.6-Luna.
LGMem outperforms the best Graph + IRCoT setting (k=8) and Graph + Self-RAG by 5.00 and 9.60 points, respectively. Compared with IRCoT at k=12, total token usage is similar (8.599M versus 8.227M), while LGMem scores 5.60 points higher. Relative to Graph + Self-RAG, LGMem reduces total tokens by 7.9%. These results demonstrate LGMem's QA advantage among the compared access strategies and support organizing follow-up retrieval and task state around unresolved information needs.
(2) Comparisons with other iterative memory systems. We added comparisons with other iterative memory systems, MRAgent (ICML 2026) and REMem-I (ICLR 2026), on LongMemEval-S and BEAM, reporting QA scores and total token consumption.
Table 2. LongMemEval-S performance and token costs (500 questions per setting).
GPT-5.6-Luna
Qwen3.6-27B-FP8
On LongMemEval-S with Luna, LGMem improves over REMem-I and MRAgent by 2.00 and 6.60 points while reducing total tokens by 97.2% and 93.5%, respectively. With Qwen3.6, the gains are 5.40 and 12.40 points, with total-token reductions of 96.6% and 93.0%. REMem-I approaches LGMem in Luna accuracy but consumes 36.3 times as many total tokens.