acceptodds
Under review as a conference paper at ICLR 2027

Same Memory, Different Interpretations : Multi-Form Memory Retrieval with Coarse-to-Fine Reranking

Abstract

Personalized memory agents aim to identify relevant information from long-term conversational histories that can support responses to the query. However, long conversation histories often contain large amounts of redundant information, making it difficult for large language models (LLMs) to capture the relevant information within a limited context window. Existing methods organize long histories into compact memories, but their fixed memory forms are limited in capturing different views of the same memory. To address these limitations, we propose **Sa**me **M**emory, Different **I**nterpretations, a multi-form memory retrieval framework with coarse-to-fine reranking. Specifically, we first construct multiple memory forms to capture factual, thematic, and contextual views of historical memories. We then retrieve each form independently and project the retrieved units back to their source memories to form a unified candidate set. Finally, we perform coarse-to-fine reranking by first establishing an overall ranking of the candidates and then adaptively refining the top-ranked memories using complementary ranking information. Extensive experiments conducted on three long-term memory datasets demonstrate that our **SaMI** framework outperforms the state-of-the-art competitors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Personalized Memory, Memory Retrieval, Coarse-to-Fine Reranking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6PAzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SMDI, which derives three memory units from the same source turn: Facet-Unit, Summary-Unit , and Expansion-Unit. Each form is retrieved independently with hybrid sparse/dense retrieval plus a cross-encoder, projected back to source turns through provenance links, and fused at the source level by summing reciprocal ranks. A coarse stage fuses projection and cross-encoder ranks, and a fine stage adaptively weights three branches using score separation and Spearman rank consistency. The top-3 source memories are passed to the LLM.

Critical strengths
  1. Memory construction, multi-path retrieval, source projection, two-stage reranking, and generation are all described, with pseudocode and hyperparameters in the appendix.
  2. The ablation is more granular than typical. Single-form, two-form combinations, reranking branches, and cumulative stages are all examined, with both retrieval and generation metrics.
Critical weaknesses
  1. The strongest baseline, MemGAS (Xu et al., ICLR 2026), already constructs multi-granularity memory units (session, turn, summary, keyword), associates them with a Gaussian mixture model, and adaptively weights granularities with an entropy-based router, followed by LLM-based filtering.
  2. The paper devotes one sentence to MemGAS and then presents single-form interpretation as an open gap, without explaining why the proposed multi-form scheme is not an equivalent restatement of multi-granularity memory.
  3. Figure 4 reports 76.9M input tokens for SMDI versus 52.9M for MemGAS (+45%), dismissed with one sentence, and the construction cost of deriving three units per turn through DeepSeek-V4-Flash is not quantified.
Questions for authors

The contribution is therefore better characterised as a well-tuned multi-view retrieval pipeline than as a new methodology.

4 comments
AuthorsOpenReview ↗

Relationship to MemGAS and concrete methodological differences

Thank you for recognizing the method description and the detailed ablations. We clarify SMDI's contribution through multi-form memory derivations, source memory projection, and coarse-to-fine memory candidate reranking.

MemGAS (Xu et al., ICLR 2026) is a direct reference for our task and evaluation. It combines session/turn/summary/keyword representations, GMM associations, entropy-based granularity weighting, and Personalized PageRank retrieval. SMDI studies long-term conversational-memory question answering on the same benchmarks.

SMDI combines multi-form retrieval, source memory projection, and coarse-to-fine memory candidate reranking. Its design has three concrete parts:

  1. Multi-Form Derivations of a Native Memory. SMDI constructs Facet-Units through Intra-Memory Facet Decomposition, Summary-Units through Cross-Memory Topic Summary, and Expansion-Units through Neighboring Memory Contextual Expansion. These units provide factual, thematic, and contextual interpretations of source memories.
  2. Multi-Form Retrieval and Candidate Formation. Each memory form undergoes independent hybrid retrieval and unit-level cross-encoder reranking. In Source-Memory Projection and Fusion, provenance links project retrieved units back to their supporting source memories. Reciprocal ranks of the associated units determine each source memory's projection score and form the source candidate set.
  3. Coarse-to-Fine Memory Candidate Reranking. Coarse-grained candidate reranking combines projection fusion ranking and cross-encoder ranking. Fine-grained head reranking adaptively fuses projection fusion ranking, cross-encoder ranking, and native contextual anchor ranking. These weights act on ranking branches; MemGAS's entropy router weights memory granularities.

Multi-form retrieval finds relevant units, source memory projection combines their retrieval evidence at the source-memory level, and coarse-to-fine reranking determines the final source memory ranking. We will expand the MemGAS comparison in the introduction and related work, explicitly describe existing multi-granularity methods, and explain these methodological differences using the terminology above.

AuthorsOpenReview ↗

Differences between the multi-form and the multi-granularity memory

SMDI's multi-form scheme differs from MemGAS's multi-granularity memory in memory derivations and retrieval-evidence fusion. MemGAS associates and adaptively weights session, turn, summary, and keyword representations. SMDI derives factual, thematic, and contextual interpretations through Facet-Units, Summary-Units, and Expansion-Units. Independent multi-form retrieval supplies unit-level rank evidence that source memory projection integrates into source candidates for coarse-to-fine memory candidate reranking (Sections 2.2-2.4).

The ablation in Section 3.3, Table 3 quantifies the effect of combining these interpretations. Using all three forms increases Recall-all@3 from 78.80 for the strongest single form, Expansion-Unit, to 82.20, and Recall-all@10 from 93.20 to 97.00. The full model also outperforms each two-form combination. Scores are percentages.

Memory formsRecall-all@3Recall-all@5Recall-all@10
Facet-Unit only50.4058.4063.60
Summary-Unit only59.4065.8067.40
Expansion-Unit only78.8086.4093.20
w/o Facet-Unit80.0088.2096.20
w/o Summary-Unit81.4089.8096.00
w/o Expansion-Unit60.0067.2069.00
Full model82.2090.2097.00

Appendix E.1, Table 9 further characterizes the retrieval results underlying this comparison. Pairwise Jaccard@3 ranges from 0.387 to 0.505, showing partially overlapping source-memory sets across forms. Source memory projection uses provenance links to aggregate their unit-level rank evidence into common source candidates (Section 2.3, Equation (1)).

Compared memory formsJaccard@3Jaccard@5Jaccard@10
Expansion-Unit vs. Facet-Unit0.3870.3230.280
Expansion-Unit vs. Summary-Unit0.4250.3510.316
Facet-Unit vs. Summary-Unit0.5050.4380.397

The gains also extend to answer generation. Appendix E.2, Table 10 reports the corresponding single-form and full-model generation ablation on LongMemEval-M and LoCoMo. The full model improves GPT-4o-J over the strongest single form by 4.00 points on LongMemEval-M (53.80 to 57.80) and 4.72 points on LoCoMo (41.78 to 46.50).

Memory formsLongMemEval-M GPT-4o-JLoCoMo GPT-4o-J
Expansion-Unit only53.8036.02
Facet-Unit only42.6041.78
Summary-Unit only44.2037.94
Full model57.8046.50

These results support the use of complementary multi-form derivations for source-memory retrieval and answer generation. Source memory projection integrates their retrieval evidence into the source candidate set for coarse-to-fine memory candidate reranking.

AuthorsOpenReview ↗

Trade-off between performance gains and computational cost

Existing cost discussion: Section 3.4, Figure 4. It compares token consumption and generation F1 on LongMemEval-S and includes memory organization and response generation in the token accounting. Below we reproduce its input-token counts alongside the LongMemEval-S F1 values from Section 3.2, Table 2.

MethodLongMemEval-S input tokens (M), Figure 4LongMemEval-S generation F1, Table 2
MemGAS52.920.38
SMDI76.922.13

SMDI incurs additional token consumption compared with MemGAS. Figure 4 reports 76.9M input tokens for SMDI and 52.9M for MemGAS, an increase of 24.0M tokens or 45.4%. SMDI also achieves higher question-answering accuracy across all three benchmarks. As Section 3.2, Table 2 shows, GPT-4o-J accuracy increases by 4.60, 12.40, and 5.43 percentage points on LongMemEval-S, LongMemEval-M, and LoCoMo, respectively. We will expand the cost discussion to present the additional token consumption alongside these performance gains. Scores below are percentages; differences are percentage points.

DatasetMemGAS GPT-4o-JSMDI GPT-4o-JDifference
LongMemEval-S60.2064.80+4.60
LongMemEval-M45.4057.80+12.40
LoCoMo41.0746.50+5.43

The construction overhead arises from generating fine-grained Facet-Units and processing them again to construct Summary-Units. MemGAS generates session summaries and keywords. SMDI first performs Intra-Memory Facet Decomposition: the LLM reads source dialogue and generates multiple Facet-Units containing factual claims, temporal information, and provenance metadata. Cross-Memory Topic Summary then reads the Facet-Units grouped by subject within each session and generates a Summary-Unit for each group. These stages consume input tokens for both source dialogue and extracted facts, and output tokens for both Facet-Units and Summary-Units. Neighboring Memory Contextual Expansion constructs Expansion-Units through text concatenation without a separate LLM call.

In the revised version, we will report the per-history LLM call counts and input/output tokens separately for Intra-Memory Facet Decomposition and Cross-Memory Topic Summary. We will also report embedding and indexing time separately to quantify the construction overhead by stage.

Thank you for the authors' detailed response. Although the additional experiments further demonstrate the effectiveness of individual components, I remain concerned about the limited methodological novelty of this work.

First, recent long-term memory studies, including MemGAS [1], HiGMem [2], and MemReranker [3], have already explored multi-granularity memory organization, adaptive retrieval, hierarchical memory selection, and memory reranking. In comparison, SMDI's three memory representations are largely based on established techniques such as fact extraction, summarization, and contextual expansion. Its multi-path retrieval, source projection, and coarse-to-fine reranking also primarily combine existing retrieval and ranking strategies. The distinction between Multi-Form and Multi-Granularity emphasized in the rebuttal does not sufficiently establish a substantial methodological contribution.

Second, while the ablation results demonstrate performance improvements from combining different components, they mainly validate the effectiveness of the engineering design rather than establish fundamental methodological novelty. Direct comparisons with recent relevant approaches such as HiGMem and MemReranker are also missing.

Furthermore, SMDI incurs approximately 45% more input tokens than MemGAS. Although the authors provide additional explanations, the construction overhead of individual stages and its trade-off against performance gains remain insufficiently quantified.

Overall, I consider this work closer to a well-engineered multi-view memory retrieval and reranking pipeline than a fundamentally novel long-term memory approach. The rebuttal does not sufficiently resolve my concerns regarding methodological novelty. Therefore, I maintain my original rating.

[1] From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents.

[2] HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents.

[3] MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval.

Reviewer_tXqEOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SMDI for long-term conversational memory retrieval. It constructs Facet-Units, Summary-Units, and Expansion-Units, performs retrieval independently for each form, and projects the results back to source memories through provenance links. Coarse-grained candidate reranking and adaptive fine-grained head reranking then select evidence for answer generation. Experiments on LoCoMo and the two LongMemEval settings report improvements in both retrieval and generation, supported by memory-form ablations and comparisons across different base models.

Critical strengths
  1. Source-memory projection provides a useful way to combine factual, thematic, and contextual retrieval signals while preserving their connection to the original dialogue. Tables 3 and 9 show that Facet-Units and Summary-Units recover relevant evidence that Expansion-Units alone miss. This gives concrete support for the complementarity of the three forms.
  2. The reported improvements extend to answer quality and are particularly substantial on longer histories. On LongMemEval-M, Tables 1–2 report Recall@3 of 64.20% for SMDI versus 51.06% for MemGAS, and GPT4o-J scores of 57.80% versus 45.40%. Here, Recall@3 follows Eq. (7): the proportion of questions for which the top three results contain all gold memories.
  3. Appendix G tests three retrievers, three rerankers, and three generators. SMDI outperforms the reported non-oracle baselines across the retriever and generator comparisons. Table 10 also shows that the full model outperforms each individual memory form on LongMemEval-M and LoCoMo, extending the evidence beyond the main LongMemEval-S ablation.
Critical weaknesses
  1. The additional value of adaptive head fusion is less established than the overall retrieval gains. Table 11 includes fixed uniform fusion, but the improvements on LongMemEval-S are only 1.00 point in Recall@3 and 0.12 points in NDCG@3. Neither confidence intervals nor generation results are provided for this comparison, leaving its contribution to answer quality unclear.
  2. The efficiency analysis does not yet give a complete cost breakdown. Figure 4 includes memory organization and generation tokens and shows higher total consumption than MemGAS. Table 2 reports generation latency. However, construction, evidence filtering, retrieval and reranking, and generation costs are not clearly separated. The amortized cost is also not reported for different numbers of queries per history.
  3. The native contextual anchor ranking branch needs a more precise definition. Appendix F links it to Expansion-Units, but the scoring function and the handling of missing or constant scores remain unspecified. These choices affect the score separation degree in Eq. (3) and the resulting fusion weights.
Questions for authors
  1. Do all methods use the same questions and denominators for Recall@k, including the same treatment of unanswerable questions? On a common evaluation set, please report generation results and paired confidence intervals for adaptive versus fixed uniform fusion.
  2. Which stages are included in Table 2's average latency? How does the full cost per query, including construction and evidence filtering, change with the number of queries served from one history?
  3. How are anchor scores and ranks computed from Expansion-Units? How are missing scores and a zero denominator in Eq. (3) handled?
  4. Section 3.2 reports a Recall@3 gain of 3.66 points, whereas Table 1 gives 82.20 − 78.51 = 3.69. Assuming that Table 15 uses the same 500 questions and query-type counts as Table 7, weighting its per-type results gives approximately 82.40%, rather than the reported Overall value of 82.20%. Could the authors reconcile these figures?
1 comment

Evaluation Protocol and Paired Retrieval Results*

We follow the benchmark settings of MemGAS (Xu et al., ICLR 2026). Its LongMemEval retrieval evaluation excludes unanswerable questions. For LongMemEval-S, this leaves 470 answerable questions after excluding the same 30 unanswerable questions. We have recomputed adaptive and fixed uniform fusion using exactly these 470 question IDs and the same denominator at every k. We will correct the previous 500-question aggregation in Appendix F, Table 11 and apply this evaluation rule consistently across methods.

The comparison uses the same projected source candidates and ranking branches and changes only the fusion weights in fine-grained head reranking. We calculate 95% paired bootstrap confidence intervals for the score difference—adaptive minus fixed uniform fusion—using 10,000 resamples of the 470 question pairs. Scores are percentages; differences and confidence intervals are in percentage points. Adaptive fusion increases Recall-all@3 by 1.06 percentage points, while both methods have identical Recall-all@10.

MetricFixed uniform fusionAdaptive fusionDifference95% paired CI
Recall-all@382.1383.19+1.06[0.00, 2.13]
Recall-all@591.2891.06-0.21[-0.64, 0.00]
Recall-all@1097.2397.230.00[0.00, 0.00]

Computational Cost and Latency

SMDI’s additional token use supports its multi-form memory derivations and accompanies higher answer accuracy. Section 3.4, Figure 4 reports 76.9M input tokens for SMDI and 52.9M for MemGAS on LongMemEval-S, an increase of 45.4%. This accounting includes memory organization and response generation. Section 3.2, Table 2 reports the following answer accuracy and SMDI token and latency statistics.

DatasetMemGAS GPT-4o-JSMDI GPT-4o-JSMDI avg. tokensSMDI avg. latency (s)
LongMemEval-S60.2064.808,8021.73
LongMemEval-M45.4057.808,8101.96
LoCoMo41.0746.501,4371.76

The memory construction cost arises from two LLM processing stages: Intra-Memory Facet Decomposition reads the source dialogue to construct Facet-Units, and Cross-Memory Topic Summary then reads grouped Facet-Units to construct Summary-Units. Neighboring Memory Contextual Expansion constructs Expansion-Units through text concatenation without a separate LLM call. For repeated queries over the same history, the construction cost is shared across queries. In the revised version, we will report memory construction costs separately from query processing costs, including LLM call counts, input/output tokens, and processing time. We will also clarify which stages the token and latency averages in Table 2 cover and provide separate measurements for embedding, indexing, retrieval, reranking, evidence filtering, and answer generation.

Native Contextual Anchor Ranking and Adaptive Fusion

For a source candidate mm with Expansion-Unit hits, we define:

s_a(m)=\frac{1}{r_a(m)}.$$ Here, retrieval ranks start at 1. A candidate without an Expansion-Unit hit receives zero anchor contribution. The weighted anchor contribution to the final fusion score is $w_a·s_a(m)$. **Consistency of Reported Results** Thank you for identifying these numerical inconsistencies. In the revised version, we will check and correct the values in Section 3.2 and Tables 1, 7, and 15, and update the corresponding text. We will apply consistent question selection and denominator rules to the overall and query-type retrieval results, ensuring that the overall score agrees with the question-count-weighted average of the query-type scores and that the reported improvements match the corrected tables. The reported Recall-all@3 of 82.20% comes from 411 successful retrievals among the full 500 LongMemEval-S questions, including 30 unanswerable questions. A retrieval is counted as successful when the top-3 source memories contain all annotated gold memories for the question. The following breakdown uses the question counts in Table 7 and the results from the same adaptive fusion run. | Question type | Number of questions | Successful retrievals at top-3 | Recall-all@3 (%) | | --- | ---: | ---: | ---: | | single-session-user | 70 | 68 | 97.14 | | single-session-assistant | 56 | 56 | 100.00 | | single-session-preference | 30 | 24 | 80.00 | | multi-session | 133 | 92 | 69.17 | | knowledge-update | 78 | 72 | 92.31 | | temporal-reasoning | 133 | 99 | 74.44 | | **Overall** | **500** | **411** | **82.20** | The overall score is the question-count-weighted average of the query-type scores, calculated before rounding: $$\mathrm{Recall\text{-}all@3} =\frac{68+56+24+92+72+99}{70+56+30+133+78+133}\times100\% =\frac{411}{500}\times100\% =82.20\%.$$
Reviewer_ZdxdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SMDI, a retrieval framework that derives facet, summary, and expansion interpretations of the same source turn, projects retrieved units back to source turns, and reranks candidates in a coarse stage followed by an adaptive fine stage. It is evaluated on LongMemEval-S, LongMemEval-M, and LoCoMo for retrieval and generation.

Critical strengths
  1. Beating MemGAS by a large margin on the harder LongMemEval-M set is a real result, since MemGAS is the closest multi-granularity competitor and uses the same generation protocol.
  2. Projecting units back to source turns is a sensible design choice. It lets different forms vote for the same evidence while keeping the final context coherent for generation.
  3. Robustness across retrievers, rerankers, and generators is shown directly. The advantage over MemGAS holds with MiniLM, MPNet, and Contriever, and with GPT-4o-mini, Qwen3-8B, and Qwen3-1.7B as generators.
  4. The ablations and appendix analyses are very thorough.
Critical weaknesses

The main problem with this work is that it doesn't have potential to be applied in the real world.

  1. The topics in Facet-Unit and merges in Summary-Unit can enter an endless loop without defined taxonomy or ontology for the multi-form derivations. The main text only hints at four facet types and three broad summary topics, while the prompts leave Memory Type and topic grouping open to LLM judgment, with no closed list, clustering rule, or merge threshold.
  2. No conflict handling is provided in construction or utilization. Construction has no detection, supersession, or versioning rule for contradictory facets or summaries, and utilization relies only on chronological ordering plus LLM filtering and generation, with the evolving-state case study explicitly admitting that no supersession operation is encoded.
  3. Aggregation has no completeness or deduplication protection. The Figure 2 trip example is a closed case with exactly three gold trips totaling 15 hours, but there is no check that a sum covers all and only the intended trips, no event identity resolution, and final generation is capped at top-3 source memories.
  4. Construction cost and updating are conceded as future work, and that concession matters here. Deriving three LLM-generated forms per turn raises total token consumption above previous methods, and no incremental update mechanism is shown for continuously evolving histories.

And here are some weaknesses on the design: 5. The coarse plus fine reranking story is questionable. Adding coarse reranking alone drops Recall at 3 from 80.00 to 76.80 on LongMemEval-S, and only fine reranking lifts it to 82.20, yet no fine-only ablation is reported to show whether coarse is needed at all beyond selecting the head set. 6. The multi-form claim is weaker than the headline suggests. Expansion alone dominates, and removing facet or summary on LongMemEval-S costs only about 2 and 1 points, so most of the gain may come from neighbor context plus reranking rather than from three equal interpretations. 7. Adaptive fine fusion adds little over a fixed baseline. It beats uniform fusion by only about 1 point on Recall at 3 across sets, despite the added separation and consistency machinery.

Questions for authors
  1. Use the correct ICLR templete.
  2. How are facet types defined, what is the closed list if any, and how are summary topics grouped, with what clustering rule, similarity threshold, and entity resolution?
  3. What is fine-only performance, without the coarse stage, on the initial ranking, and what happens if a gold memory falls outside the top-10 head? It is unclear whether coarse is needed at all.
  4. What is expansion-only performance with the full coarse and fine reranking pipeline, rather than as a single-form retrieval pathway alone? It looks like facet and summary add little on LongMemEval-S.
  5. How does the system verify that an aggregation answer covers all and only the intended evidence, with deduplication by event, when final generation sees only top-3 source memories?
3 comments
AuthorsOpenReview ↗

Task and experimental setting of long-term conversational retrieval

Thank you for recognizing the improvements on LongMemEval-M, source memory projection, and the experiments across models. SMDI studies the same long-term conversational memory retrieval and question-answering task as MemGAS (Xu et al., ICLR 2026). We use the same three benchmarks, LongMemEval-S, LongMemEval-M, and LoCoMo, and follow its generation/evaluation protocol: top-3 retrieved memories, GPT-4o-mini for answer generation, and GPT-4o for evaluation (Section 3.1; Appendix D.3).

Under this benchmark setting, the task is to retrieve evidence from the given dialogue history and generate an answer. Event-level deduplication and checks for omitted events are not separate evaluation tasks; their effects enter the evaluation through evidence coverage and answer correctness. The reviewer’s concerns relate to challenging knowledge-update and multi-session questions, which require reasoning over changing states and combining evidence across sessions. On both query types, SMDI achieves higher F1 and ROUGE-L than MemGAS on LongMemEval-S (Section 3.4, Figure 5; Appendix G.4, Figure 9(c,d)).

AuthorsOpenReview ↗

Memory construction and Analysis of retrieval and generation errors

Section 2.2, Appendix A, Table 5, and Appendix J, Figures 14 and 15 describe the construction process. Intra-Memory Facet Decomposition extracts individual facts with subject, temporal, and provenance information. The LLM assigns a descriptive Memory Type to each fact, using semantic categories such as event, state, preference, goal, relationship, skill, object, reflection, and plan. The four aspects in Section 2.2 describe fact content; Memory Type records its semantic category. Cross-Memory Topic Summary groups Facet-Units by identical subject strings within each session and generates one Summary-Unit per group. Grouping and summarization run once; generated summaries do not re-enter the construction process. We will state the exact grouping rule and these steps explicitly in the revised version.

For knowledge-update questions, SMDI retains timestamps and orders selected evidence chronologically; the generator uses the question's time scope to interpret earlier states and later updates (Appendix K, Figure 24). For aggregation, top-3 counts source memories: LongMemEval uses complete sessions, each of which can contain multiple facts or events, while LoCoMo uses dialogue turns. Source memory projection combines unit hits linked to the same source ID. In Appendix K, Figure 23, the generator combines three Big Sur camping days and five Yellowstone camping days, excludes a seven-day Utah trip that involved no camping, and answers eight days.

We additionally analyzed the existing LongMemEval-M results. For the 470 answerable questions, complete coverage means retrieving all annotated supporting sessions among the top-3 source memories; answer correctness follows the GPT-4o judgments. The following table separates supporting-session coverage from answer correctness.

Annotated supporting-session coverageCorrect answerIncorrect answer
Complete25062
Incomplete19139

We also examined 81 answerable multi-session aggregation questions: 35 require counting, 44 summation, and two averaging. Of these, 30 answers are correct and 51 incorrect. We classified the 51 incorrect answers as follows.

Aggregation error patternCount
Incomplete session coverage; reported aggregate value below the correct value29
Incomplete session coverage; no explicit aggregate value provided19
Estimate based on assumed values1
Repeated counting in the final answer1
Inclusion of out-of-scope items1
Total51

We will add these statistics and correct/incorrect cases to the appendix. In the movie-counting case, repeated mentions of Avengers: Endgame still lead to the correct answer of two distinct movies. In the appointment case, filtering identifies two completed visits, but generation counts one twice and answers three. In the spending case, retrieval misses a record of USD 500, yielding an answer of USD 2,000 instead of USD 2,500. We will show the original records, retrieved evidence, filtered content, and final answer for each case.

AuthorsOpenReview ↗

Memory Forms and Reranking

Section 3.3, Table 3 evaluates Expansion-Unit retrieval with the full coarse-to-fine memory candidate reranking pipeline. Including all three memory forms increases Recall-all@3 from 78.80 to 82.20. Removing Facet-Units or Summary-Units reduces the score to 80.00 or 81.40, respectively. The following results come from Table 3.

Memory formsRecall-all@3Recall-all@10
Facet-Unit only50.4063.60
Summary-Unit only59.4067.40
Expansion-Unit only78.8093.20
w/o Facet-Unit80.0096.20
w/o Summary-Unit81.4096.00
w/o Expansion-Unit60.0069.00
Full model82.2097.00

Section 3.3, Table 4 reports the results of introducing coarse-grained candidate reranking and fine-grained head reranking. To examine the role of coarse-grained candidate reranking, we also apply fine-grained head reranking directly to the initial top-10 or all 20 source candidates. These comparisons use the same 20 projected source candidates and source-level cross-encoder scores. All comparisons that include fine-grained head reranking retain projection fusion ranking, cross-encoder ranking, and native contextual anchor ranking.

Additional reranking comparisonRecall-all@3Recall-all@10NDCG@3
Projection fusion ranking79.6096.0088.52
Coarse-Grained Candidate Reranking only76.8097.0085.81
Fine-Grained Head Reranking on initial top-1081.4096.0090.11
Fine-Grained Head Reranking on all 2081.6096.6089.92
Coarse-to-Fine Memory Candidate Reranking81.8097.0089.77

Section 2.4 reports adaptive fusion weights from score separation degree and pairwise ranking consistency across the three ranking branches. Appendix F, Table 11 reports the following comparison with fixed uniform fusion:

DatasetFixed uniform Recall-all@3Adaptive Recall-all@3
LongMemEval-S81.2082.20
LongMemEval-M63.6064.20
LoCoMo65.7966.40

Computational Cost and Paper Format

The additional token cost supports SMDI's multi-form memory derivations, which provide complementary evidence for answer generation and improve answer accuracy. Section 3.4, Figure 4 reports 76.9M input tokens for SMDI and 52.9M for MemGAS on LongMemEval-S, an increase of 45.4%. This count includes memory organization and response generation. We will add a detailed cost breakdown for memory construction, embedding, indexing, and query processing in the revised version. Section 3.2, Table 2 shows the corresponding GPT-4o-J gains over MemGAS across all three benchmarks; scores are percentages and differences are percentage points. We will use the correct ICLR template in revised version.

DatasetMemGAS GPT-4o-JSMDI GPT-4o-JDifference
LongMemEval-S60.2064.80+4.60
LongMemEval-M45.4057.80+12.40
LoCoMo41.0746.50+5.43

Comments on the paper

1 comment

As the rebuttal period is drawing to a close, we have responded to all three reviewers’ comments but have not yet received any follow-up feedback. We would appreciate your help in reminding the reviewers to consider our responses and share any remaining concerns. We welcome further discussion and are ready to provide additional clarification. Thank you for your time and support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.