- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes REALM, a retrieval-driven memory system with a cognitive graph, adaptive search, and post-answer reconsolidation. The core idea is to update the activated subgraph after each QA step so memory evolves with use. Experiments on LoCoMo and LongMemEval report gains from reconsolidation, but also clear failures on preference-style questions.
Critical strengths
- The overall design is fairly complete, and the closed-loop setup matches a reasonable cognitive story. Claims of automatic organization, adaptive retrieval, and memory enhancement also line up with practical needs.
- The method works with multiple memory organizations and retrieval styles, which makes the system more usable in real engineering settings.
Critical weaknesses
- Reconsolidation feedback includes the ground-truth answer text, not only a correct/incorrect bit. The topology update is therefore driven by oracle labels. That is much stronger than the “task feedback in practice”, and it makes the closed-loop claim hard to trust in realc system.
- The gains are not especially large, and they are uneven across benchmarks. LoCoMo improves by about 7 points over the strongest reported average, but LongMemEval only by about 2, and reconsolidation itself adds roughly 2 points. Preference is far behind strong baselines. So the method does not look generally better, it looks stronger on some settings and weak on others.
Questions for authors
- With feedback reduced to correct/incorrect only, and without the gold answer string, how do Tables 1–3 change?
- Appendix C says preference fails mainly because seed retrieval misses the topic, so reconsolidation cannot help if the subgraph is never activated. If the gold preference evidence is forced into the seed set, how high does preference accuracy go?
Thank you for looking closely at our experimental setup and comparisons. One of your central questions is whether REALM can still organize its memory effectively when the updater has no reference answer. On LoCoMo, overall accuracy held up both with correctness labels alone and with neither form of feedback. We also re-scored the LoCoMo answers with MAGMA’s published rubric and the LongMemEval answers with its released evaluation code. We discuss those results and address your questions about the evaluation set, answer prompts, and cost.
[W1/Q1 — What happens when we remove answer feedback?]
When REALM first organizes its memory, it has yet to see what users will ask. A query can bring together pieces of information that were stored separately but are useful for the same task. REALM looks at the retrieved content and its topic to decide which local connections to record for later queries. As you pointed out, we supplied reference answers to the updater in the original experiments, and these may be unavailable in practice. We therefore tested how much the method depends on this feedback.
We ran two ablations on the same 1,540 answerable LoCoMo questions. One keeps the correctness label and removes the reference answer. The other removes both. The table reports the mean and sample standard deviation over four question orders, including the historical original order and three shuffles with seeds 0, 42, and 2026. It includes binary accuracy and MAGMA’s graded score on a 0–100 scale.
The mean with full feedback is 74.84%. The historical runs in W1/Q2 averaged 74.98%. The two means differ by about 0.15 percentage points, less than the variation across question orders in either set of runs. Both ablations yield slightly higher mean LoCoMo accuracy than the full-feedback setting. In the no-feedback setting, the judge still uses the reference answer for scoring. The updater receives no reference answers, correctness labels, or annotated evidence IDs.
You also ask how much reconsolidation helps without feedback. Relative to the paper’s single archived run without reconsolidation (73.96%), the four-order means are higher by 0.88 percentage points with full feedback, 2.14 with correctness labels alone, and 1.59 with no feedback. The sample standard deviations describe how the scores vary across the four question orders. The largest accuracy SD is 0.88 percentage points, in the full-feedback setting.
[W1 / Q2 — Question order and gain size]
Tables 1 and 3 report seed 2026, the best-performing order, with a gain of 2.01 percentage points. In the archived runs with full feedback, accuracy was 74.48% for seed 0, 74.48% for seed 42, 75.00% for the original order, and 75.97% for seed 2026. The mean is 74.98%, about 1.02 percentage points above the paper’s 73.96% reference without reconsolidation, close to your estimate of about 1.1 points. We will report all four results in the revision so readers can see how the gain varies with question order.
[W1 — Why we added warm-up questions to LongMemEval?]
Each LongMemEval sample pairs a separate history with one target question. To see whether earlier queries could improve the memory used to answer that question, we generated warm-up questions from selected sessions and let the system retrieve information and update its memory before answering the target. In this setup, the number of correct answers rose from 296 to 306 out of 470. You identified two details that matter for interpreting this result. We used annotations to select sessions containing evidence, and the prompt for generating warm-up questions included examples from the test set. The generator did not receive the current target question. We will make both details explicit in the revision and report the gain of 10 correct answers under this setup.