acceptodds
Under review as a conference paper at ICLR 2027

Memory as Code: Declarative Retrieval over Codified Long-Term Memory

Abstract

In long-term memory systems, memories are constructed from interaction histories and retrieved in response to queries, enhancing the reasoning ability of LLM agents. In complex scenarios, facts are scattered across sessions and their relationships evolve over time, making it difficult for existing memory systems to compose them to adapt to diverse queries. Motivated by the modularity and composability of code, we propose Memory as Code (MaC), a unified memory framework that jointly designs memory construction and retrieval through a shared code abstraction. MaC couples codified construction, which establishes executable memory abstractions, with declarative retrieval, which translates queries into structured retrieval programs. Specifically, codified construction builds a three-layer executable memory structure that progresses from an atomic function layer, through a relational layer, to a modular composition layer. Declarative retrieval recasts retrieval from relevance-based fact selection into declarative query processing over memory. Experiments on LoCoMo and LongMemEval show relative gains of up to 15.91% in overall LLM-judge accuracy over the strongest baseline in each setting, while substantially reducing token consumption, highlighting the effectiveness of memory as code for long-horizon memory tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agent memory, long-term memory, memory question answering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ENe2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Memory as Code (MaC), which organizes conversational facts, temporal replacement relations, and reusable memory computations into a callable library. A declarative retrieval component generates a query-specific program that composes these capabilities to retrieve supporting evidence. Experiments on LoCoMo and LongMemEval evaluate answer accuracy, computational cost, and component contributions.

Critical strengths
  1. The representation addresses concrete challenges in long-term memory. Canonical fact identifiers, explicit supersession relations, separate current-state and historical computations, and source provenance form a coherent interface for reasoning over evolving information. The contribution goes beyond merely wrapping text in functions.

  2. The empirical improvements are reasonably broad and consistent with the motivation. MaC achieves the highest overall LLM-judge accuracy among the evaluated methods across three LoCoMo backbones and on LongMemEval. On LongMemEval, overall accuracy improves from the strongest baseline’s 68.40% to 74.60%, with notable gains in temporal reasoning and knowledge updates. The sequential ablations also show improvements from adding both codified construction and declarative retrieval, while the cost analysis distinguishes construction from query-time overhead.

Critical weaknesses
  1. The ablations and execution analysis do not fully identify which mechanisms drive the improvement. Table 5 omits the configuration that retains declarative retrieval while removing codified construction. Moreover, codified construction jointly introduces relations, state semantics, indexing, provenance, and executable interfaces, making their individual contributions unclear. Algorithm 1 also permits program repair and fallback retrieval, but the paper does not report first-attempt execution success, repair frequency, or fallback rates. These omissions limit the explanation of how the proposed program-based retrieval operates in practice, although they do not invalidate the complete system’s gains.

  2. Important evaluation details are missing. The manuscript does not clearly identify the LongMemEval version, evaluated sample set, or truncation policy. Table 2 presents four question categories without explaining whether the overall score includes the other categories. Baseline versions and principal retrieval configurations are also insufficiently documented. Finally, the standard deviations in Table 1 come from repeated judge evaluations, rather than independent repetitions of memory construction and retrieval, and therefore do not quantify end-to-end variability.

  3. The related-work discussion should include AMA: Adaptive Memory via Multi-Agent Collaboration, which combines adaptive retrieval with conflict-aware memory maintenance and evaluates on LoCoMo and LongMemEval. Please clarify how MaC’s executable retrieval programs and explicit historical-state semantics differ from AMA’s iterative retrieval and maintenance mechanisms.

Questions for authors

See Weakness

4 comments

W1 (Ablation of codified construction): Thank you for this helpful suggestion. We have evaluated the configuration that retains declarative retrieval while removing codified construction (MaC w/o CC), using a graph for fact storage and indexing on LoCoMo. Compared with MaC, overall F1 and J decrease by 10.76 and 13.23 percentage points, respectively. These results show that codified construction contributes to the complete system’s gains when declarative retrieval is retained.

Relations, state semantics, indexing, provenance, and executable interfaces are integral parts of codified construction, jointly designed around facts stored as functions. These components are tightly coupled, making it difficult to isolate their individual contributions. We have added this ablation in Section 3.4 (Table 5) and clarified its scope in Appendix D.2.

MethodOverall F1 ↑Overall J (LLM-judge mean ± std) ↑Improv.
MaC w/o CC56.5874.16 ± 0.27+17.84%
MaC67.3487.39 ± 0.03—

W1 (First-attempt success, repair frequency, and fallback rates): We appreciate your constructive comments. We report first-attempt success, repair frequency, and fallback rates below. These results show that program-based retrieval usually succeeds on the first attempt, with repair resolving most initial failures and fallback rarely needed. We have added these execution statistics to Section 3.5, Table 7.

DatasetQuestionsFirst-attempt successRepair frequencyFallback rate
LoCoMo1,5401,538 (99.8701%)1 (0.0649%)1 (0.0649%)
LongMemEval500457 (91.4%)43 (8.6%)0 (0%)

W2 (LongMemEval version, evaluated sample set, and overall score): Thanks for your careful review. Following MRAgent’s evaluation setting, we have evaluated all 500 instances of the cleaned LongMemEval-S version (longmemeval_s_cleaned.json). We construct memory from each instance's full conversation history without additional truncation. We report the two question categories omitted from Table 2: multi-session (Multi.) and single-session-assistant (Asst.). The overall score averages over all questions across all six categories. MaC improves overall J by 17.59% relative to the strongest baseline. Thus, MaC’s overall gains hold when all six question categories are included. We have clarified the protocol in Section 3.1 and Appendix D.1 and expanded Table 2 to include all six question categories.

MethodMulti. F1 ↑Multi. J ↑Asst. F1 ↑Asst. J ↑Overall F1 ↑Overall J ↑
A-Mem35.6947.97 ± 0.3479.7796.43 ± 0.0047.1658.92 ± 0.33
MemoryOS32.1638.65 ± 0.4169.3883.93 ± 0.0043.3353.80 ± 0.24
Mem046.3162.41 ± 0.0061.7871.43 ± 0.0050.9568.00 ± 0.24
MRAgent56.7572.93 ± 0.0018.0825.00 ± 0.0043.6761.08 ± 0.27
MaC59.4974.44 ± 0.0079.1596.43 ± 0.0062.4079.96 ± 0.22

W2 (Baseline versions and principal retrieval configurations): Thanks for your valuable feedback. We follow the default settings in each baseline’s respective paper. We have provided the baseline versions and retrieval configurations in Appendix D.1 and summarize them below.

  • A-Mem: the robust implementation, retrieving 10 initial memories and expanding linked neighbors.
  • MemoryOS: memoryos-pypi, retrieving up to 5 mid-term sessions, 10 pages, and 20 entries each for user and assistant knowledge.
  • Mem0: v2.0.18, retrieving up to 20 memories with a semantic threshold of 0.1 and additional reranking disabled.
  • MRAgent: K1=80, K2=20, and at most 50 tool calls per question.

W2 (End-to-end variability): We appreciate your valuable suggestion. We have conducted five independent repetitions of memory construction and retrieval on LoCoMo and report the mean ± standard deviation to quantify end-to-end variability. The small standard deviations indicate low end-to-end variability across these independent repetitions. We have added the five-run results and stability analysis to Section 3.5, Table 6.

CategoryF1 (%)J (%)
Multi-hop52.74 ± 0.7987.45 ± 1.19
Temporal71.53 ± 0.9387.66 ± 0.78
Open-domain41.41 ± 0.9767.50 ± 1.55
Single-hop73.33 ± 0.2993.06 ± 0.40
Overall67.19 ± 0.2689.31 ± 0.25

W3 (Comparison with AMA): Thank you for this helpful suggestion. AMA [1] uses iterative retrieval guided by its Judge, while MaC generates executable retrieval programs that compose memory functions and operators. AMA updates or conditionally deletes conflicting entries. MaC retains superseded facts and uses supersession relations to compute the active state at a specified time. We have clarified these differences in Section 4 (Related Work).

[1] Weiquan Huang, Zixuan Wang, Hehai Lin, Sudong Wang, Bo Xu, Qian Li, Beier Zhu, Linyi Yang, and Chengwei Qin. 2026. AMA: Adaptive Memory via Multi-Agent Collaboration. In Findings of the Association for Computational Linguistics: ACL 2026, pages 3099–3120. Association for Computational Linguistics.

Thanks for the reply. It addresses my main concern, and I will keep my positive score.

Thank you for your positive feedback on our rebuttal and continued support for our work. We are pleased that our clarifications and additional results have helped address your concerns about codified construction’s contribution, retrieval execution reliability, evaluation settings, end-to-end stability, and MaC’s differences from AMA. We have incorporated these clarifications and results into the revised manuscript.

Reviewer_BnRROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MaC is a unified framework that jointly designs memory construction and retrieval, where both share code abstraction. Codified construction converts conversational history into callable fact functions from three-layer structure including atomic function layer, relational layer, and modular compositional layer. Declarative retrieval translates queries into structured retrieval problem ver this library to assemble evidence for answering. Experiments demonstrate improvement on LoCoMo and LongMemEval with lower token consumption.

Critical strengths
  • Explicit provenance and state handling. Preserving source references and superseded facts supports inspection and historical queries; returning an unresolved state avoids arbitrarily selecting among conflicting values.
  • Consistent empirical gains. MaC improves overall judge accuracy across three LoCoMo backbones and achieves particularly substantial temporal-reasoning gains on LongMemEval. Reporting construction costs separately from retrieval is useful.
Critical weaknesses
  • The benefit of code is not isolated. Base functions primarily return structured records, while derived functions instantiate predefined templates. Improvements could therefore come from canonicalization, temporal relations, and better retrieval rather than executable representation. Compare against other structured abstraction (e.g., SQL, json) or graph storage with identical facts, operators, and planning; complete the factorial ablation by adding declarative retrieval without codified construction.
  • Execution validity does not establish semantic correctness. Structural checks cannot verify fact extraction, supersession judgments, or evidence completeness. Equation (1) also depends on correct temporal annotations and replacement links. Evaluate these components separately, including corrections, concurrent valid states, and distinctions between assertion time and event time. Zep [1] is a relevant missing temporal-memory baseline.
  • Abstention remains unevaluated. Given MaC’s explicit provenance, unresolved-state semantics, and verification operators, evaluating LoCoMo’s adversarial category would clarify whether executable memory improves recognition of insufficient evidence. Please report correct abstention alongside unnecessary abstention on answerable questions, and clarify how the answer-repair mechanism handles justified refusals. This would distinguish reliable evidence handling from improved answer retrieval alone.
  • Impact of executable memory in factual reliability throughout the pipeline beyond the final QA accuracy. It would be great to run evaluation on HaluMem [2], which provides operation-level evaluation of memory execution, updating, and downstream QA. This would help determine whether codified construction and explicit supersession relations improve factual reliability, beyond the end-to-end accuracy gains reported on LoCoMo and LongMemEval. Reporting correctness, hallucination, and omission metrics would also clarify whether executable memory prevents upstream errors or propagates them into final answers.

[1] Zep: A Temporal Knowledge Graph Architecture for Agent Memory [2] HaluMem: Evaluating Hallucinations in Memory Systems of Agents

Questions for authors

See weaknesses

3 comments

W1 (The ablation of codified construction): Thank you for this helpful suggestion. Following your suggestion, we have replaced functions with graph storage and used the graph index while retaining declarative retrieval (DR), resulting in MaC w/o CC. Compared with MaC, overall F1 and J decrease by 10.76 and 13.23 percentage points, respectively. These results demonstrate the benefit of CC when DR is retained. In our design, functions, the index, and operators are essential to the executability of codified construction (CC) and are tightly coupled, making it difficult to isolate their individual contributions. We have added this ablation in Section 3.4 (Table 5) and clarified its scope in Appendix D.2.

MethodOverall F1 ↑Overall J (LLM-judge mean ± std) ↑Improv.
MaC w/o CC56.5874.16 ± 0.27+17.84%
MaC67.3487.39 ± 0.03—

W2 (Semantic correctness): We appreciate your constructive comments. We have evaluated memory extraction on HaluMem-Medium using six metrics that reflect fact extraction quality and evidence completeness. The results show that MaC outperforms Zep on all six metrics, supporting improved fact extraction quality and evidence completeness. We have added this evaluation to Section 3.6 (Table 9), with metric definitions and interpretation in Appendices D.3–D.4.

MethodR ↑Weighted R ↑Target P ↑Acc ↑FMR ↑F1 ↑
Zep61.35%76.65%98.76%97.06%48.00%75.69%
MaC74.56%85.61%99.16%98.16%50.94%85.12%

W2 (Temporal annotations and replacement links): We appreciate your valuable suggestion. We have separately ablated time normalization and supersedes replacement links on LoCoMo. Removing time normalization reduces overall F1 and J by 2.55 and 1.88 percentage points, respectively. These ablations support the contributions of temporal annotations and replacement links to overall performance, with a larger effect from time normalization. We have added these temporal-component ablations to Appendix D.3, Table 11.

MethodOverall F1 ↑Overall J ↑
MaC w/o supersedes67.1388.64
MaC w/o time normalization64.8587.47
MaC67.4089.35

W2 (Temporal-memory baseline): Following your suggestion, we have added Zep as a temporal-memory baseline on LoCoMo. MaC achieves an 11.34% relative improvement in overall J over Zep. This demonstrates MaC’s gains over a temporal-memory baseline. We have added the comparison with Zep to Section 3.6, Table 8.

MethodOverall F1 ↑Overall J ↑Improv. (J)
Zep60.6678.49 ± 0.10+11.34%
MaC67.3487.39 ± 0.03—

W3 (LoCoMo’s adversarial category): Thanks for your valuable suggestion. Following MRAgent’s evaluation protocol, we originally excluded LoCoMo’s adversarial category (See MRAgent’s Appendix D.1). Following your suggestion, we have evaluated this category and report the results below. MaC achieves 93.05% on both F1 and J, demonstrating competitive performance. (Since adversarial category uses binary scoring to evaluate whether the model correctly identifies insufficient evidence, F1 and J scores are identical under this evaluation setting.)

MethodF1/J
A-Mem83.63%
MemoryOS74.44%
Mem093.27%
MRAgent78.25%
Zep87.89%
MaC93.05%

W4 (Factual reliability throughout the pipeline): We appreciate your valuable suggestion. We have evaluated MaC and Zep on HaluMem-Medium. We use DeepSeek-V4-Flash as the shared LLM backbone to support future reproducibility, given GPT-4o’s scheduled API shutdown on January 14, 2027. Zep is implemented locally using graphiti-core v0.31.2. MaC achieves a 9.04% average relative improvement across all 12 metrics, with 8.89%, 14.64%, and 3.76% for memory extraction, memory updating, and question answering, respectively. These operation-level results support improved factual reliability beyond final QA accuracy. We have added these results to Section 3.6 (Table 9) and detailed the experimental setup and aggregation method in Appendices D.1 and D.4.

MethodMemory ExtractionMemory UpdatingQuestion AnsweringOverall
R ↑Weighted R ↑Target P ↑Acc ↑FMR ↑F1 ↑improv. ↑C ↑H ↓O ↓improv. ↑C ↑H ↓O ↓improv. ↑improv. ↑
Zep61.35%76.65%98.76%97.06%48.00%75.69%+8.89%48.75%3.33%47.60%+14.64%65.33%22.47%12.20%+3.76%+9.04%
MaC74.56%85.61%99.16%98.16%50.94%85.12%—59.61%3.36%36.87%—67.29%20.39%12.32%——
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for your thoughtful feedback, which has helped us improve our paper. We would appreciate hearing whether our rebuttal has addressed your concerns and whether any points need further clarification.

Reviewer_FRVoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Long conversations spread facts across sessions, and some facts change over time. Relevance-based retrieval may miss the connections between them, while step-by-step retrieval requires repeated planning. The paper proposes Memory as Code (MaC), which builds and retrieves memory through one shared design so that an LLM agent can reuse and combine facts. Codified construction (CC) turns deduplicated facts into functions. Supersession relations distinguish current facts from earlier ones, and current and history templates create reusable functions. Fact IDs and source references keep the evidence traceable. Declarative retrieval (DR) uses an index to plan a restricted Python program in a single planning call. The program runs in a sandbox, retrieves and combines evidence, and passes it to an answer model. Failed programs may be repaired or replaced by a fallback. The experiments focus on question answering over conversations. The paper reports the highest overall LLM-Judge accuracy on three LoCoMo backbones and on LongMemEval (Tables 1 and 2), with a largest relative gain over the strongest baseline of 15.91%. It also reports lower token usage, though not always lower latency. On LoCoMo, MaC uses the fewest construction tokens on all three backbones, while query-stage token consumption varies across backbones and methods. On LongMemEval, MaC uses 30.5% fewer tokens than MRAgent but takes more time. A sequential ablation (Table 5) is presented as evidence that both CC and DR contribute to the gains.

Critical strengths

The results are consistent across the tested settings: overall LLM-Judge accuracy is highest on all three LoCoMo backbones and on LongMemEval. Against the strongest baseline in each setting, the margin in overall J is 3.75 to 11.92 percentage points on LoCoMo and 6.20 points on LongMemEval.

The framework and its rules for current and historical states are clear. The current state is defined as the unique fact in a state group that no other fact within the time boundary supersedes, with an explicit unresolved marker otherwise, while the history template retains superseded facts.

Critical weaknesses

The independent value of the code abstraction remains uncertain. Table 5 does not test DR with CC removed, and the paper does not compare against a memory that uses the same index and state rules but does not store facts as functions. In Table 5, the CC step moves from extracted atomic facts only to the full codified library and raises overall J from 72.73 to 80.78, and the paper attributes this gain jointly to reusable memory computations and indexed access. This uncertainty is one reason my overall assessment is reserved. The stability of the full system remains uncertain. The error bars in Table 1 come from five LLM-judge evaluations, not from repeated construction and planning runs, and the rates of first-program success, repair, and fallback are not reported. Tables 2 and 5 report single values with no variability estimate. The retrieval stage allows up to three checking and execution attempts before a default fallback program is used. These details matter for explaining the gains and judging reliability.

Questions for authors
  1. How much of the gain comes specifically from the code abstraction? For example, how does DR perform with CC removed, and how does MaC compare with a memory that uses the same index and state rules but does not store facts as functions?
  2. How often does the first program succeed, and how often are repair or fallback needed?
6 comments

W1,Q1 (The gain from the code abstraction): Thanks for your valuable feedback. We have evaluated the gain from codified construction (CC) on LoCoMo. MaC w/o CC replaces the code abstraction with graph storage and indexing while retaining declarative retrieval (DR). Compared with MaC, overall F1 and J decrease by 10.76 and 13.23 percentage points, respectively. These results demonstrate the effectiveness of CC.

MethodOverall F1 ↑Overall J (LLM-judge mean ± std) ↑
MaC w/o CC56.5874.16 ± 0.27
MaC67.3487.39 ± 0.03

Regarding your question about retaining the same index and state rules without representing facts as functions, the current index uses functions as its retrieval units, while the state rules operate on function execution and relations between functions (Section 2.2.2). Removing the function-based representation therefore changes both what the index retrieves and what the state rules operate on. The ablation evaluates the combined contribution of the function-based representation, index, and state rules. It is difficult to isolate the contribution of the representation alone. We have added the results in Section 3.4 (Table 5) and clarified this scope in Appendix D.2.

W1,Q2 (First-attempt success, repair, and fallback): Thank you for this helpful suggestion. We report the rates of first-attempt success, repair, and fallback below. These results show that most retrieval programs succeed on the first attempt and rarely require fallback. We have added these execution statistics to Section 3.5, Table 7.

DatasetQuestionsFirst-attempt successRepairFallback
LoCoMo1,5401,538 (99.8701%)1 (0.0649%)1 (0.0649%)
LongMemEval500457 (91.4%)43 (8.6%)0 (0%)

W1 (The stability of the full system): We appreciate your constructive comments. We have conducted five repeated construction and retrieval runs on LoCoMo and report the mean ± standard deviation. The results show small standard deviations in both F1 and J across these runs, supporting the stability of the full system. We have added the five-run results and stability analysis to Section 3.5, Table 6.

CategoryF1 (%)J (%)
Multi-hop52.74 ± 0.7987.45 ± 1.19
Temporal71.53 ± 0.9387.66 ± 0.78
Open-domain41.41 ± 0.9767.50 ± 1.55
Single-hop73.33 ± 0.2993.06 ± 0.40
Overall67.19 ± 0.2689.31 ± 0.25

W1 (Variability estimates for Tables 2 and 5): Thanks for your careful review. We have conducted repeated LLM-judge evaluations for Tables 2 and 5 and report J as mean ± standard deviation. The small standard deviations in overall J support the consistency of the reported gains across repeated LLM-judge evaluations. MaC achieves the highest overall F1 and J in both tables. We have added J variability estimates to Tables 2 and 5 and clarified the evaluation protocol in Section 3.1 and Appendix D.1.

Table 2. LongMemEval results.

MethodMulti. F1Multi. JSgl. F1Sgl. JTmp. F1Tmp. JPref. F1Pref. JUpd. F1Upd. JAsst. F1Asst. JOverall F1Overall JImprov.
A-Mem35.6947.97 ± 0.3471.7484.29 ± 0.0032.4136.24 ± 0.347.0547.33 ± 3.6561.8171.03 ± 1.1579.7796.43 ± 0.0047.1658.92 ± 0.33+35.71%
MemoryOS32.1638.65 ± 0.4170.6980.00 ± 0.0030.6332.33 ± 0.006.4361.33 ± 2.9854.9968.21 ± 0.5769.3883.93 ± 0.0043.3353.80 ± 0.24+48.62%
Mem046.3162.41 ± 0.0082.6392.86 ± 0.0034.6254.29 ± 0.3419.1568.67 ± 1.8362.7175.90 ± 1.0761.7871.43 ± 0.0050.9568.00 ± 0.24+17.59%
MRAgent56.7572.93 ± 0.0073.8788.57 ± 0.0034.0356.39 ± 0.006.9450.00 ± 3.3343.2154.36 ± 0.7018.0825.00 ± 0.0043.6761.08 ± 0.27+30.91%
MaC59.4974.44 ± 0.0084.2894.86 ± 0.7853.2868.42 ± 0.0014.4971.33 ± 2.9869.6987.18 ± 0.0079.1596.43 ± 0.0062.4079.96 ± 0.22—

Table 5. LoCoMo ablation results.

MethodMulti-hop F1 ↑Multi-hop J ↑Temporal F1 ↑Temporal J ↑Open-domain F1 ↑Open-domain J ↑Single-hop F1 ↑Single-hop J ↑Overall F1 ↑Overall J ↑Improv.
MaC w/o DR+CC44.0167.59 ± 1.9250.3065.55 ± 0.5736.9759.17 ± 0.4761.0978.26 ± 0.2354.2172.47 ± 0.22+20.59%
MaC w/o DR50.5482.34 ± 1.7459.6369.16 ± 0.6239.0064.19 ± 1.3667.1585.42 ± 0.1860.7980.14 ± 0.34+9.04%
MaC w/o CC45.4572.77 ± 0.3056.2364.42 ± 0.1436.9764.38 ± 0.4762.6979.45 ± 0.3156.5874.16 ± 0.27+17.84%
MaC52.9986.81 ± 0.3070.4884.74 ± 0.0040.0164.58 ± 0.0074.0791.20 ± 0.1267.3487.39 ± 0.03—
Reviewer_FRVoOpenReview ↗

Thanks for the additional experiments; remaining questions

Thank you for the detailed response and the additional experiments. They resolve part of my concerns, and I summarize where I stand below.

  1. First-attempt success, repair, and fallback. The new statistics answer my question directly. With 1,538 of 1,540 LoCoMo programs succeeding on the first attempt and no fallback on LongMemEval, the gains do not appear to come from the fallback program. I consider this concern resolved.

  2. Contribution of the code abstraction. The new MaC w/o CC variant is useful. Read together with the other rows of Table 5, it suggests that DR adds 1.69 points of overall J without CC but 7.25 points with CC, which supports the paper's case for designing construction and retrieval jointly. The comparison I asked about, however, remains open. I am not fully convinced that it cannot be isolated: the current-state rule in Eq. 1 is defined over fact identifiers and the supersession relation set R, so it appears implementable on a store of identified facts and relations without function wrappers. I would encourage this control in a future version. It would also help to describe the graph storage used in MaC w/o CC and how operators such as search_functions and execute run on it, since the two variants without CC currently use different storage.

  3. Stability. The five repeated construction and retrieval runs are a welcome addition. Please state the backbone and judge configuration used. The five-run mean J (89.31 ± 0.25) is higher than all three MaC values in Table 1 (86.84, 87.66, and 87.39) by more than six standard deviations, while the mean F1 (67.19) is close to the DeepSeek-V4-Flash value (67.34). This suggests the setting differs from Table 1. The stability result would be stronger if measured under the reported configuration, and on LongMemEval as well.

  4. Revised LongMemEval results. In the revised Table 2, the overall F1 of MaC changes from 58.58 to 62.40, while its four previously reported per-category F1 values and the overall F1 of every baseline are unchanged. Since F1 does not depend on the judge, this change cannot come from repeated judge evaluations alone and suggests that MaC's outputs on the newly displayed categories changed. The relative gain over the strongest baseline correspondingly rises from 9.06% to 17.59%. A brief explanation of this change in the revision would make the main results easier to trust. Reporting all six categories, together with question counts and the exact data subsets for both benchmarks, would also address my earlier concern about data split details.

  5. Minor: MaC w/o DR+CC and MaC w/o CC report the same open-domain F1 (36.97), while their open-domain J differs by 5.21 points. Please check this entry.

Overall, I see this as meaningful work, and the framework is clearly presented. Because the independent value of the code abstraction remains open and the revised numbers need explanation, I will keep my score for now. I hope these suggestions help strengthen the work, and I look forward to seeing it developed further.

Comments on the paper

1 comment

Overall Review Summary

We sincerely thank the Program Chair, Senior Area Chair, Area Chair, and all three reviewers for their time, effort, and thoughtful feedback throughout the review process. The reviews highlight MaC's coherent framework design, strong empirical performance, and clear presentation. FRVo and ENe2 gave ratings of 3, with confidence 2 and 3, respectively. BnRR gave a rating of 2 with confidence 2. All three reviewers rated presentation 3.

During the rebuttal, Reviewer ENe2 confirmed that the main concern was addressed and decided to retain the positive rating. Reviewer FRVo thanked us for the detailed response and additional experiments, called the study "meaningful work", and noted that the framework is clearly presented. We promptly followed up on code abstraction, stability, and revised LongMemEval results with experiments, clarifications, and experimental details. We will include these in the final paper.

Major strengths acknowledged by reviewers

  1. Coherent framework design. All three reviewers praised current and historical state handling. Two highlighted source provenance. One noted a contribution beyond wrapping text in functions.
  2. Strong empirical performance. All three reviewers praised the results. Two noted the highest overall LLM-judge accuracy (J) on three LoCoMo backbones and LongMemEval. Two praised temporal-reasoning gains and separate cost reporting.
  3. Clear presentation. All three reviewers rated the paper clear and easy to follow.

Responses to major reviewer concerns

1. Motivation and method design

  • CC scope and implementation (FRVo, BnRR, ENe2). We clarified that MaC w/o CC measures the combined contribution of codified construction (CC). We described its fact, keyword, and tag nodes, identifiers, provenance, and search and execution operators.
  • Relationship to AMA (ENe2). We contrasted MaC's executable retrieval and historical-state computation with AMA's iterative retrieval and updates or conditional deletion.

2. Additional experiments and analyses

  • CC ablations (FRVo, BnRR, ENe2). We added two LoCoMo ablations: MaC w/o CC replaces CC with graph storage and indexing while retaining declarative retrieval (DR). MaC w/o (function in CC) stores the same identified facts in JSON instead of functions, retaining the current-state rule, supersession relations, and view index with comparable DR. Both variants underperform full MaC, supporting CC's contribution and the effectiveness of its function-based representation, respectively.
  • Execution reliability (FRVo, ENe2). We reported first-attempt success, repair, and fallback rates. First-attempt success is 99.87% on LoCoMo and 91.4% on LongMemEval. Fallback occurs for only 1/1,540 and 0/500 questions.
  • Stability (FRVo, ENe2). We added five LoCoMo construction/retrieval runs using Table 1's DeepSeek-V4-Flash backbone and judge, showing low F1/J variability. We provided per-run scores and correct-answer counts, distinguishing independent runs judged once from repeated judging of fixed answers. Judge noise and run variability may contribute to the 1.92-point mean J gap. Full LongMemEval stability runs remain planned.
  • Additional baseline and benchmark (BnRR). We added Zep as a temporal-memory baseline and HaluMem-Medium as a benchmark for factual reliability. MaC improves LoCoMo J by 11.34% relative to Zep. On HaluMem-Medium, MaC improves all six extraction metrics plus update and QA correctness, while update hallucination and QA omission remain slightly higher.
  • Temporal components (BnRR). We added separate ablations of time normalization and supersession links. Both contribute to overall QA performance, with a larger effect from time normalization.
  • Adversarial category (BnRR). We clarified that LoCoMo's adversarial category was initially excluded following MRAgent's protocol. We added this evaluation, achieving 93.05% on both F1 and J under binary abstention scoring, comparable to the strongest evaluated baseline.

3. Presentation

  • LongMemEval result changes (FRVo, ENe2). We clarified that the original overall score covered all six categories. Rerunning MaC on the two newly displayed categories raised overall F1 from 58.58 to 62.40. Other category and baseline F1 scores were unchanged.
  • Evaluation protocol (FRVo, BnRR, ENe2). We specified subsets and category counts for all ten LoCoMo conversations and all 500 cleaned LongMemEval-S instances, with full histories and no extra truncation for the latter. We documented baseline settings, HaluMem metrics and aggregation, and judge variability in Tables 2 and 5.
  • Metric interpretation (FRVo). We verified that two open-domain F1 values round to 36.97 despite a 5.21-point J gap, clarifying token overlap versus semantic judging.

We hope this summary helps the Program Chair, Senior Area Chair, and Area Chair quickly understand the overall review feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.