- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GRAVITY, a host-independent auxiliary memory layer that consolidates dialogue into entity profiles, temporal event traces, and cross-session topic summaries. It retrieves these records, expands host queries, and injects selected anchors alongside host memories. The evaluation reports improvements across five memory systems on LongMemEval and LoCoMo under two model configurations, with additional experiments examining evidence organization, source interaction, and alternative auxiliary representations.
Critical strengths
(S1) GRAVITY improves all host–benchmark pairs under both model configurations. Under the primary configuration, average accuracy increases by 9.2% on LongMemEval and 7.5% on LoCoMo. Alternative evaluators also preserve the positive direction of these improvements.
(S2) Anchor construction uses raw dialogue, and the same store is reused across hosts through query and prompt interfaces. This way is practical to augment heterogeneous memory systems without modifying their internal representations.
(S3) The source diagnostics distinguish improvement over host baselines from complementarity with anchors alone. Table 6 shows consistent host contributions beyond anchors on LongMemEval and mixed contributions on LoCoMo. The empirical conclusion is strong.
Critical weaknesses
(W1) That 'same-evidence' baseline doesn't actually prove the evidence is equivalent. 4.3 claims the 80.9% to 82.4% bump is all down to organization, but Appendix A.3.2 points out those records came straight from the retrieved sessions, which may contain answer-bearing facts absent from the compressed host entries. So limiting the sources doesn't cleanly isolate the organization from just finding the evidence.
(W2) The component ablations do not separate complementary views from a larger auxiliary context. The default five-record budget per bank allows up to 5/10/15 anchors in single-/pair-/three-bank configurations. Without matching the total auxiliary-context budget, the full configuration’s advantage does not establish three-view complementarity.
(W3) The uncertainty of the smaller reported advantages is unclear. LoCoMo’s 1540 questions share only ten conversation histories, but no paired uncertainty or conversation-level variation is reported for the 1.5% organization gain or 3.6% representation margin.
(W4) The claimed representation edge seems unproven. While Table 5(a) contrasts different auxiliary setups, it seems to gloss over their extraction schemas, prompt designs, combination rules, retrieval workflows, and context window lengths. Simply keeping the host, model, prompt position, and retrieval budget the same isn't enough to prove the test actually isolates representation.
Questions for authors
(Q1) For Table 5(b), did the extractor receive only the exact memory snippets already shown to the host-only generator, or the complete source sessions containing those snippets? Please clarify the construction inputs and explain why this diagnostic’s host-only accuracy is 80.9%, compared with 77.8% in the main cross-model results.
(Q2) Were total anchor counts and injected tokens matched across the component ablations? Please report these quantities and compare the full configuration against the strongest pair under the same auxiliary-context budget.
(Q3) How consistently do the 1.5% organization gain (Table 5(b)) and 3.6% representation (Table 5(a)) advantage hold across the ten conversations? Please provide conversation-level differences and paired confidence intervals that account for questions sharing the same history.
(Q4) How were the 6 auxiliary representations constructed, consolidated, retrieved, and serialized? Please provide their prompts, schemas, and actual context budgets so that the claimed representation advantage can be assessed independently of differences in extraction and retrieval.
2 comments
Response to Reviewer FYRT (cont.)
W3 / Q3. Uncertainty across conversations
We now report conversation-level results for both margins. We resampled the ten conversations 10,000 times, keeping each conversation's questions together and pairing the two conditions:
| Comparison | Accuracy | Difference | 95% CI | Conversations with a gain | Leave-one-out range |
|---|---|---|---|---|---|
| S + R(S) vs. S | 80.91% → 82.40% | +1.49 | [0.52, 2.40] | 8 of 10 | 1.27 to 1.85 |
| GRAVITY vs. GSW-style | 80.32% → 83.90% | +3.57 | [2.90, 4.32] | 10 of 10 | 3.31 to 3.78 |
Per-conversation differences (percentage points):
| Conversation | 26 | 30 | 41 | 42 | 43 | 44 | 47 | 48 | 49 | 50 |
|---|---|---|---|---|---|---|---|---|---|---|
| S + R(S) − S | +1.32 | −1.23 | +1.32 | +3.02 | +0.56 | +4.07 | +2.67 | −1.05 | +1.92 | +1.90 |
| GRAVITY − GSW-style | +4.61 | +3.70 | +1.97 | +3.52 | +3.37 | +3.25 | +6.00 | +2.09 | +4.49 | +3.16 |
Neither interval includes zero, and dropping any single conversation does not change the sign. The smaller 1.5-point gain is less consistent, with two conversations going the other way, while the 3.6-point margin holds in every conversation. Section 4.3 now cites both intervals (L346–361), and full counts are in Appendix A.8.5 (L1800–1835).
W4 / Q4. Construction and comparability of the auxiliary representations
We agree the original description was too thin. Table 31 (Appendix A.8.6, L1836–1941) now describes, for every pipeline, how it is built and merged, what text it can see, how it retrieves, what the answer model actually receives (including caps and truncation), and how many auxiliary tokens it adds. The prompts are in the anonymous code repository linked in Appendix A.1 (L762), under auxiliary_pipelines/prompts/, together with the shared answer and judge templates. GraphRAG- and LightRAG-style use the default prompts of langchain-graphrag 0.0.9 and lightrag-hku 1.5.4, and GSW-style uses the prompts from its public code.
All pipelines share the LightMem store, top-60 retrieval, answer template, and generator (Claude Sonnet 4.6), with GRAVITY in its default configuration. A short version:
| Pipeline | Built from | What the answer model sees | Aux. tokens |
|---|---|---|---|
| Free-form summary | Retrieved sessions | A free-form summary of those sessions | 1,205 |
| GRAVITY | Full history | Up to 5 entity, 5 event, and 5 topic records | 2,673 |
| GraphRAG-style | Full history | 15 entities, 20 relations, 5 community reports | 3,043 |
| RAPTOR-style | Full history | Up to 5 tree nodes | 2,314 |
| HippoRAG2-style | Full history | 5 facts and 8 passages | 1,807 |
| LightRAG-style | Full history | Up to 30 entities, 30 relations, 15 chunks | 5,526 |
| GSW-style | Full history | Up to 15 actors, states, and operators; 10 QA pairs; background | 2,156 |
(Aux. tokens: mean per question on conv-26, 152 questions, under the cross-model configuration of Table 5(a). The 1,719.5 tokens for EVT under W2/Q2 are measured under the GPT-4o-mini configuration on all LoCoMo questions, so the two numbers are not directly comparable.)
As the table shows, the pipelines differ in more than representation: in what they are built from, how they retrieve, and how much they put in the prompt. You are right that our setup does not isolate representation. We also think it cannot be fully isolated, since each representation comes with its own way of being built and searched. We have therefore narrowed the claim. Under a shared host, template, and generator, GRAVITY outperforms each of these pipelines, including the GSW-style one that reads the complete history (+3.57 points, positive in all ten conversations), but we no longer say this isolates representation alone. The wording is updated in the abstract (L23–25), introduction (L63–65, L77–78), Related Work (L99–101), Section 4.3 (L346–352), Discussion (L517–519), Conclusion (L534–537), and Appendix A.3.2 (L949–952).
Response to Reviewer FYRT
Thank you for the very careful review, and for highlighting that GRAVITY improves every host–benchmark pair under both configurations, that one anchor store built from raw dialogue can be reused across hosts in practice, and that the source diagnostics separate gains over the host from complementarity with it. Your questions pointed to several places where the paper was unclear or claimed too much, and we have revised it accordingly.
W1 / Q1. Evidence equivalence and the 80.9% versus 77.8% baselines
Thank you for catching this. The extractor gets the complete retrieved sessions, but so does the baseline: in Table 5(b), the baseline generator also sees the full text of those sessions, not LightMem's compressed entries. We did not say this clearly.
Concretely, let S be the full text of the sessions behind LightMem's retrieved entries. The baseline sees S, the records R(S) are built from S only, and the augmented model sees S plus R(S), with query expansion turned off. Both sides have access to exactly the same dialogue, so the 80.9% to 82.4% gain shows that adding records helps without widening the source evidence. This setup rules out evidence from outside S, but it does not separate the structure of the records from the extra processing, repeated presentation, and longer input they also bring.
This also answers why the baseline is 80.9% here but 77.8% in Tables 2 and 5(a): those tables give the generator LightMem's compressed entries, while this one gives it the full session text. For the same reason, S + R(F) (85.5%) is higher than LightMem + GRAVITY in the main results (83.9%): the former pairs the records with full session text, the latter with compressed memories, so the two are different input settings (Appendix A.3.2). In the revision, the rows of Table 5(b) are labeled by what the generator sees, and the protocol is spelled out in the caption (L379–383), Section 4.3 (L353–361), and Appendix A.3.2 (L917–940).
W2 / Q2. Total context budget in the component ablations
No, they were not matched, and you are right that this matters. With up to five records per bank, configurations with more banks simply received more context:
We therefore reran the comparison between all three banks (EVT) and the two best pairs (ET and VT, both 74.0% in Table 3) under the same per-question input cap, on all 1,540 LoCoMo questions with LightMem. The host memories were identical across conditions (query expansion off, so EVT here matches the −exp row of Table 3). For each question, EVT's prompt length set the cap, and ET and VT filled the freed space with more records from their own banks.
EVT has a slightly higher observed accuracy, by 0.65 and 0.78 points, but we report no paired intervals for these differences, and ET and VT used somewhat less of the auxiliary budget. We therefore read the three configurations as close: part of the original gap came from the extra context, and the remaining difference is not evidence of three-view complementarity. We have added these results to Appendix A.8.4 (L1751–1798) and softened the claim in Section 4.3 (L316–323) and Appendix A.3.3 (L963–965).