acceptodds
Under review as a conference paper at ICLR 2027

BINDMEM: Executable Memory for Long-Horizon Reasoning

Abstract

Long-horizon agents must interpret and combine information across past interactions, yet access to supporting evidence does not ensure correct reasoning. Even a successfully computed value may fail to satisfy the question's required scope or precision. We introduce BINDMEM, a neuro-symbolic memory interface that preserves source-local interpretations (WRITE), executes query-dependent programs (READ), and checks answer requirements before disclosing derived values (DISCLOSE). WRITE materializes source-local interpretations as provenance-linked typed records whenever their inputs are available from the source context. READ compiles each question into a typed program and an answer contract, then executes it over the retrieved records while tracking supporting evidence and unresolved conditions. DISCLOSE supplies a computed value only when it meets the requested precision and scope; otherwise it supplies the selected evidence alone. The final reader retains the original retrieved text throughout. Across six benchmark variants with two instruction-tuned readers, BINDMEM attains the highest overall score in all 12 comparisons against seven memory baselines, and it also improves a frontier reader. Fixed-retrieval controls isolate the contributions of stored interpretations, execution-selected evidence, and selective disclosure, and with identical stored records BINDMEM outperforms the evaluated Python pipelines for both readers.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM agents, long-term memory, compositional reasoning, partial evaluation, temporal reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AvmsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BINDMEM, a memory framework for long-horizon reasoning that combines source-level memory interpretation, program execution, and evidence-based answer disclosure. The system converts supported questions into executable programs, tracks evidence and unresolved conditions during execution, and only provides derived values when they satisfy the requested precision and evidence scope.

Critical strengths

Meaningful problem. Retrieving relevant memories does not guarantee correct reasoning. This is particularly evident in temporal reasoning and questions requiring information to be connected across multiple records. The paper identifies a practical limitation of existing memory systems.

Practical approach and promising results. Executing part of the reasoning through programs and checking whether the available evidence supports the requested answer is a reasonable design choice. The experiments show improvements over several memory baselines across multiple benchmarks. The controlled studies also attempt to isolate the contributions of different components.

Critical weaknesses

Limited novelty. The framework appears to combine structured memory representations, program-aided reasoning, provenance tracking, and rule-based verification. Although the integration is useful, the paper does not sufficiently establish what fundamental capability is introduced beyond these existing ideas. The novelty of the overall framework therefore remains unclear.

Dependence on correct natural language interpretation. Program execution and rule-based checks cannot guarantee that the original question or retrieved records have been interpreted correctly. Errors in event identification, semantic matching, or program construction may propagate through the execution pipeline. The paper would benefit from a clearer analysis of these errors and their impact on final answer accuracy.

Non-negligible overhead. The reported query latency increases from 0.24 seconds to 0.44 seconds per question. The framework also introduces additional LLM calls for program proposal. Although the paper discusses computational costs, the latency comparison uses different default concurrency configurations. It remains unclear whether the accuracy gains justify the additional overhead under comparable deployment conditions.

Limited applicability. The program compiler supports a constrained set of operations and compositions. Complex questions that fall outside this supported scope may still require the ordinary LLM answering path. The paper does not fully establish how well the proposed approach generalizes to more diverse forms of long-horizon reasoning.

Questions for authors

What capabilities does BINDMEM provide that existing program-aided reasoning and structured-memory systems cannot readily achieve?

How robust is the framework to errors in question interpretation, record grounding, and program construction?

Can the authors provide a controlled comparison of end-to-end latency and accuracy under matched computational budgets and concurrency settings?

8 comments
AuthorsOpenReview ↗

Response to Reviewer Avms (1/4)

W1 / Q1. What is novel beyond structured memory and program-aided reasoning?

To our knowledge, BINDMEM is the first agentic-memory to ground executable memory reasoning on both source-time and query-time dependencies. Its novelty lies in the binding-time-aware partial evaluation for persistent memory. BINDMEM analyzes template dependencies before ingestion. WRITE evaluates source-complete recipes while their local inputs are available and materializes typed results with provenance and evidence scope. Query-dependent operations remain as residual READ computation; unresolved identity and coverage obligations propagate through that computation and gate DISCLOSE. BINDMEM extends the memory architecture beyond simple structured storage and/or program execution; it is a new execution framework for agentic memory.

1. Comparison with prior approaches.

HippoRAG/A-MEM emphasize retrieval structures; TReMu pairs time-aware memory with query-time Python; PAL/PoT and LOTUS execute programs without this persistent source/query binding-time boundary. BINDMEM not only extends these approaches but also adds novel components. A summary is provided in the table below.

Method comparison with prior work (revision Table R-M2)

MethodPersistent memoryRuns programsDeclarative operatorsChecks outputsExplicit binding-time splitEvidence tracking & admission
HippoRAG [1] / A-MEM [2]✓×××××
TReMu [3]✓✓××××
PAL [4] / PoT [5]×✓××××
LOTUS [6]×✓✓✓××
CoVe [7]×××✓××
BINDMEM✓✓✓✓✓✓

✓: explicitly supported; ×: not explicitly provided by the compared system. Explicit binding-time split means dependency-guided execution of source-local computations at WRITE, leaving query-dependent operations to READ. Evidence tracking & admission means propagating provenance and unresolved identity/coverage obligations through READ and checking query-specific answer support at DISCLOSE.

2. Can post-hoc verification replace DISCLOSE?

We compare four value-admission policies on all 500 LME-S questions. All conditions use identical retrieved evidence, WRITE records, compiled programs, grounding, execution results, and evidence-only fallback. Only the policy for admitting a computed value changes.

  • Without scope checks: Admit a valid, unique computed answer without checking source coverage or unresolved evidence obligations.
  • Final-only verifier: Check the candidate answer against the question and original source evidence, then decide whether to admit it.
  • Adapted CoVe-style verifier: Generate verification questions, answer each independently from the original sources, and use those checks to decide whether to admit the candidate.
  • BINDMEM Full: Use the evidence and unresolved obligations propagated during execution to determine admission under the question's type, precision, and scope contract.

All conditions retain common execution and type checks, including answer uniqueness; both post-hoc verifiers consider the same eligible candidates and return only an admit/withhold decision. The CoVe-style arm is an admission-only adaptation.

Verification-policy comparison (LME-S)

Admission policyQwen (%)Gemma (%)
Without scope checks50.663.1
Final-only verifier52.464.2
Adapted CoVe-style verifier52.664.7
BINDMEM Full54.866.1

BINDMEM outperforms the evaluated post-hoc verifiers: +2.4/+1.9 pp over final-only verification and +2.2/+1.4 pp over CoVe-style verification for Qwen/Gemma. Thus, propagating evidence obligations during dependent execution provides benefits beyond these final-answer admission checks (Appendix C.6, Table R-C3).

(continue)

AuthorsOpenReview ↗

Response to Reviewer Avms (2/4)

Will KGs built from the same WRITE records suffice?

We construct a WRITE-derived KG retaining the same event/entity relations, values, provenance, and source-stated membership and exhaustiveness claims. An LLM proposes graph queries, which NetworkX executes; the resulting values and supporting evidence are given to the same final reader. This baseline shares the source records but does not propagate unresolved obligations through graph operations or enforce query-specific value admission. The following is a list of baselines for comparison.

  • WRITE-derived KG: Whether graph storage and query execution suffice when source records and their evidence are shared.
  • Python (same WRITE records): Whether query-time program execution can replace BINDMEM's integrated reasoning interface.
  • Dependent vs. fixed-surface execution: Whether composing operations according to intermediate dependencies improves accuracy.
  • Scope/obligation checks ON vs. OFF: Whether evidence admission contributes when programs, grounding, traces, and evidence are held fixed.

Controlled comparisons on LME-S

ComparisonQwen gainGemma gainReference
BINDMEM vs. WRITE-derived KG+6.2 pp—Table R-C4
BINDMEM vs. Python (same WRITE records)+11.0 pp+6.1 ppTable 8
Dependent vs. fixed-surface execution+3.6 pp+4.3 ppTable 5
Scope/obligation checks ON vs. OFF+4.2 pp+3.0 ppTable 6

What do these controlled comparisons establish?

With identical WRITE records, BINDMEM improves over the evaluated KG pipeline by 6.2 pp (54.8% vs. 48.6%; Table R-C4) and over the Python pipeline by 11.0/6.1 pp for Qwen/Gemma (Table 8). The dependent-execution and obligation-check ablations independently support the value of those components (Tables 5–6). Together with the matched-access and deferred-execution controls, these results support the WRITE–READ–DISCLOSE architecture: source-bound materialization, dependency-guided residual execution, and query-specific evidence admission.

Changes in revision: Main Sec. 5 (Table R-M2) for the mechanism comparison and novelty discussion; Main Secs. 3.2–3.4; Appendices A.3–A.6 for dependency-guided WRITE and matched/deferred controls; Appendix B.4 (Table R-B1) for obligation propagation; Appendix C.6 (Tables R-C3–R-C4) for verification and KG controls; Tables 5, 6, and 8 for component and Python comparisons.


W2 / Q2. How do interpretation and grounding errors affect final answers?

Checking upstream errors. We audited upstream errors across all 500 LME-S/Qwen evaluation questions. GPT-5 identifies the first observed error in WRITE interpretation, READ grounding, READ compilation, or other computation. We assign one category per question to avoid double-counting. The categories are GPT-5 judgments.

GPT-5-judged upstream errors (LME-S/Qwen, n=500)

First observed failureRate
Interpretation @WRITE6.0%
Grounding @READ3.0%
Compilation @READ2.0%
Other computation1.0%
Upstream error rate12.0%

Robustness to upstream errors: 86.7% fallback. Among questions with GPT-5-identified upstream errors, 86.7% produce evidence-only or no additional payload. This conditional fallback rate, P(fallback∣upstream error)P(\mathrm{fallback}\mid\mathrm{upstream\ error}), measures pipeline-level robustness, including failures intercepted before DISCLOSE.

Controlled checks improve final-answer accuracy. With programs, grounding, traces, and selected evidence fixed, enabling scope/obligation checks raises LME-S accuracy from 50.6% to 54.8% (+4.2 pp) for Qwen and from 63.1% to 66.1% (+3.0 pp) for Gemma (Table 6). The accuracy gains support explicit evidence checks, while the logged fallback includes failures that occur before DISCLOSE. A value admitted after an incorrect interpretation or grounding can still be wrong.

Changes in revision: Main Secs. 3.4 and 4.2; Appendix G.4 (Table R-G1) for the 500-question GPT-5-judged upstream-error audit and logged fallback; Appendix C.5 and Table 6 for disclosure reliability and scope/obligation-check ablations.


AuthorsOpenReview ↗

Response to Reviewer Avms (3/4)

W3. Does BINDMEM's overhead justify its accuracy gains?

Our original query-time measurements were amortized.

The reported 0.24–0.44 s/question figures reflect amortized wall-clock times under different default concurrency configurations and should not be interpreted as a controlled 0.20 s latency increase. We therefore provide matched serving measurements and additional cross-system end-to-end latency evaluations.

Matched serving shows substantial accuracy gains with modest overhead.

We evaluate Base and BINDMEM using the same A100 GPU, Qwen2.5-7B reader, and vLLM configuration on LME-S. Measurements include retrieval, program proposal, execution, and final answering.

Matched serving efficiency (LME-S/Qwen)

MethodAccuracyp50 / p95 (s)Throughput (q/s)
Base40.8%1.24 / 3.750.677
BINDMEM54.8%1.38 / 3.930.654

BINDMEM gains 14.0 pp accuracy with an 11.3% increase in median latency.

The p95 latency increases by 4.8%, while throughput decreases by 3.4%. Accuracy is measured on all 500 LME-S questions; latency uses 120 requests at concurrency one, and throughput uses 499 valid completed requests at concurrency eight.

Additional experiment: cross-system end-to-end latency.

We compare BINDMEM, A-MEM, HippoRAG, and MemGAS using an A100 GPU, the Qwen2.5-7B reader, and the existing serving configuration. We measure per-request latency at concurrency levels c∈{1,4,8}c\in\{1,4,8\}, where cc denotes concurrent requests. This is a separate evaluation from the Base/BINDMEM matched-serving measurement above.

Cross-system E2E latency (LME-S/Qwen; seconds)

Systemc=1 p50 / p95 / p99c=4 p50 / p95c=8 p50 / p95
A-MEM3.26 / 6.28 / 7.584.26 / 7.875.96 / 9.83
HippoRAG8.26 / 12.21 / 13.488.79 / 12.489.37 / 13.46
MemGAS1.89 / 7.38 / 7.473.86 / 12.975.08 / 22.25
BINDMEM1.38 / 3.93 / 7.023.82 / 10.075.45 / 10.77

BINDMEM has the lowest p50/p95/p99 at c=1 (1.38/3.93/7.02 s). At c=8, MemGAS has lower p50 and A-MEM lower p95.

Symbolic execution incurs negligible latency relative to LLM inference.

LLM inference accounts for 83.1% of end-to-end time, whereas READ and DISCLOSE together account for only 1.1%, approximately 76 times less. Although LLM inference includes program-proposal calls, these results show that the symbolic execution itself is not a major latency bottleneck. Combined with the matched-serving results (+14.0 pp accuracy for only 0.14 s additional median latency), this supports a favorable accuracy–latency trade-off.

BINDMEM end-to-end latency breakdown

StageTime share
Retrieval (embedding and ranking)14.1%
WRITE-record loading and rendering1.1%
READ (compilation, grounding, execution)0.6%
DISCLOSE0.5%
Prompt construction0.6%
LLM inference83.1%

READ and DISCLOSE account for 1.1% of measured E2E time; LLM inference accounts for 83.1%. Proposal-model inference is included in the latter.

Changes in revision: Main Sec. 4.2 (Table R-M1) for cross-system cost–quality context; Appendix E.4 (Tables R-E1–R-E2) for matched serving and compute; additional Appendix E tables for cross-system E2E latency and stage-level profiling.


Reviewer_3gEBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary:

The authors propose BINDMEM, a neuro-symbolic memory interface that preserves source-local interpretations (WRITE), executes query-dependent programs (READ), and checks answer requirements before disclosing derived values (DISCLOSE). The proposed method attains the highest score over several benchmarks against other state of the art memory baselines.

Critical strengths

Strengths:

  1. The authors provide a memory interface which has best accuracy numbers against various SOTA memory baselines and benchmarks.

  2. The architecture of memory interface and the write operation presented in the paper are quite interesting. In particular, write operation is deterministic and has no overhead LLM cost.

Critical weaknesses

Weakness and questions:

  1. Cost of different methods should be analysed and reported. For instance in Table 3.

  2. Write operation is very interesting. Has something similar been done in prior works? In particular, I am interested to know if there is work on resolving the retrieved documents using their source context.

  3. I would like the authors to comment on what was known in prior works more explicitly. I feel read and disclose operations are kind off already known in the community. Also, regarding the write operation, I am aware that Knowledge graphs are quite popular which capture entities and semantic linking. Would you please comment differences between KG and your write method. More broadly, I ask the authors to comment explicitly on what ideas are novel and what are already studied.

  4. All the experiments are mainly done through SLMs. Although it is interesting, I wish to know how much value the proposed method has when the LLMs are used. I wish to see experiments with flagship models, especially Figure 1 and a subset of other tables whichever the authors think are feasible. I believe that the retrieval execution gap would be lot lesser with larger LLMs. Therefore I wonder if this gap only exists in SLMs because of their limited reasoning capability.

  5. I would like to see actual typed records for the three examples given on Page 4 in the main paper.

  6. Clarification question and minor comment: I believe write operation happens even before the query is known as written in the paper. In Figure 2, it somehow feels like write operation is happening during the query execution which was a little confusing.

Questions for authors

See weaknesses.

I have put the score of weak reject for now and willing to change it after discussion with other reviewers, authors rebuttal and understanding the contributions of this paper in the context of prior work.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3gEB (1/3)

W1. What are the costs relative to other memory systems?

A cost comparison has been provided in Appendix of the original manuscript. We reprise the result table below; it reports store construction, reader-context size, storage overhead, and answer quality on the same 10 LoCoMo dialogues (272 sessions). Considering the importance of cost comparison, we have moved this table to the main text: see Table R-M1 in the revised paper.

Cross-system cost–quality comparison (LoCoMo; Table R-M1)

MethodIngest callsBuild (s)Read tok/qStorage (×raw)Score Q/G
A-MEM29917011.5K1.1935.1/42.2
HippoRAG29910210.1K4.2037.4/44.5
MemGAS544633.8K12.5638.5/44.3
Mem01,1902,7792.8K6.1034.1/40.2
BINDMEM08.36.3K1.4346.5/50.4

BINDMEM achieves a good construction-cost and accuracy trade-off. BINDMEM requires no ingestion-time LLM calls, constructs its store in 8.3 s, and achieves the highest LoCoMo scores for both readers. Its reader context is smaller than A-MEM's and HippoRAG's, although MemGAS and Mem0 use fewer tokens. A-MEM also requires less additional storage. Construction times are not concurrency-matched.

Query cost and matched serving. BINDMEM uses 1.46 read-time LLM calls/question on LoCoMo; the reported 0.44 s/question is amortized time. On matched A100 serving for LME-S, Full gains 14.0 pp over Base with 11.3% higher p50 latency and 3.4% lower throughput (Appendix E.4).

Additional cost comparison: end-to-end query latency. We evaluate BINDMEM, A-MEM, HippoRAG, and MemGAS at concurrency levels 1, 4, and 8, reporting latency distributions.

End-to-end query latency (seconds; lower is better)

Systemc=1 p50 / p95 / p99c=4 p50 / p95c=8 p50 / p95
A-MEM3.26 / 6.28 / 7.584.26 / 7.875.96 / 9.83
HippoRAG8.26 / 12.21 / 13.488.79 / 12.489.37 / 13.46
MemGAS1.89 / 7.38 / 7.473.86 / 12.975.08 / 22.25
BINDMEM1.38 / 3.93 / 7.023.82 / 10.075.45 / 10.77

BINDMEM achieves the lowest p50/p95/p99 at c=1 (1.38/3.93/7.02 s). At c=8, MemGAS has lower p50 and A-MEM lower p95.

Changes in revision: Main Sec. 4.2 (Table R-M1) for the cross-system cost–quality comparison; Appendix E.2–E.3 (Tables 16–18) for detailed construction, storage, and query costs; Appendix E.4 (Tables R-E1–R-E2) for matched serving and compute controls; we also added the new comparison on end-to-end latency and stage-level profiling reported in Appendix E.


W2. How does WRITE differ from prior source-aware memory?

WRITE incorporates binding-time-guided partial evaluation into agentic memory. Existing source-aware memories organize and link observations (A-MEM, HippoRAG) or combine time-aware summaries with query-time Python reasoning (TReMu). BINDMEM makes a different architectural decision: it explicitly analyzes what computations can be fully bound to the source at ingestion and which depend on an unknown future query. WRITE executes the source-complete portion immediately, including source-anchored temporal and local-reference resolution, and stores typed results. READ executes the residual query-dependent selection, linking, and aggregation (Appendix A.3–A.4). Our method differs from simply storing richer metadata or invoking a program after retrieval: binding time determines where memory reasoning is executed.

WRITE retains evidence obligations with typed interpretations. Event, time, scalar, and scope records preserve provenance, precision, and source-supported coverage: "only transport expenses" supports completeness, whereas "included the flight and train" does not. READ composes the records; DISCLOSE admits values only when their obligations are met. This source/query execution boundary is the key architectural novelty.

Matched controls confirm the WRITE-stage distinction. Typed WRITE improves Base from 41.6% to 45.2% with the same restored inputs. Full retains 52.8% under Base-matched source access (54.8% standard). Complete-input deferral matches WRITE (2,486/2,486), but retrieval-only deferral drops to 52.8%, demonstrating the value of materializing source-bound results before their inputs disappear.

Changes in revision: Main Secs. 3.2 and 5; Appendix A.3–A.4 for dependency-guided materialization, Appendix A.5–A.6 for matched-input and deferred-execution controls, and Appendix B.4 for evidence-obligation semantics.


AuthorsOpenReview ↗

Response to Reviewer 3gEB (2/3)

W3. What is novel beyond knowledge graphs and program-aided reasoning?

To our knowledge, BINDMEM is the first agentic-memory to ground executable memory reasoning on both source-time and query-time dependencies. Its novelty lies in the binding-time-aware partial evaluation for persistent memory. BINDMEM analyzes template dependencies before ingestion. WRITE evaluates source-complete recipes while their local inputs are available and materializes typed results with provenance and evidence scope. Query-dependent operations remain as residual READ computation; unresolved identity and coverage obligations propagate through that computation and gate DISCLOSE. BINDMEM extends the memory architecture beyond simple structured storage and/or program execution; it is a new execution framework for agentic memory.

Comparison with prior work.

HippoRAG/A-MEM emphasize retrieval structures; TReMu pairs time-aware memory with query-time Python; PAL/PoT and LOTUS execute programs without this persistent source/query binding-time boundary. BINDMEM not only extends these approaches but also adds novel components. A summary is provided in the table below.

Method comparison with prior work (revision Table R-M2)

MethodPersistent memoryRuns programsDeclarative operatorsChecks outputsExplicit binding-time splitEvidence tracking & admission
HippoRAG [1] / A-MEM [2]✓×××××
TReMu [3]✓✓××××
PAL [4] / PoT [5]×✓××××
LOTUS [6]×✓✓✓××
CoVe [7]×××✓××
BINDMEM✓✓✓✓✓✓

Here, evidence tracking and admission means propagating unresolved identity/coverage requirements through dependent operations and checking question-specific answer support at disclosure. The table does not treat generic provenance tracking or output verification as equivalent to this mechanism.

Why are evidence tracking and admission necessary beyond structured storage?

Consider two candidate flights ($200 and $300), one of which was booked, plus a $50 train fare. The source states that the booked flight and train were the only transport expenses, but does not identify which flight was booked. The possible totals are $250 and $350. WRITE preserves the source-supported coverage without resolving the missing identity. JOIN carries the ambiguity, REDUCE retains both totals, and DISCLOSE withholds an unsupported exact answer. A stored set of facts does not by itself impose these checks on subsequent operations.

Will KGs built from the same WRITE records suffice?

We construct a WRITE-derived KG retaining the same event/entity relations, values, provenance, and source-stated membership and exhaustiveness claims. An LLM proposes graph queries, which NetworkX executes; the resulting values and supporting evidence are given to the same final reader. This baseline shares the source records but does not propagate unresolved obligations through graph operations or enforce query-specific value admission. The following is a list of baselines for comparison.

  • WRITE-derived KG: Whether graph storage and query execution suffice when source records and their evidence are shared.
  • Python (same WRITE records): Whether query-time program execution can replace BINDMEM's integrated reasoning interface.
  • Dependent vs. fixed-surface execution: Whether composing operations according to intermediate dependencies improves accuracy.
  • Scope/obligation checks ON vs. OFF: Whether evidence admission contributes when programs, grounding, traces, and evidence are held fixed.

Controlled comparisons on LME-S

ComparisonQwen gainGemma gainReference
BINDMEM vs. WRITE-derived KG+6.2 pp—Table R-C4
BINDMEM vs. Python (same WRITE records)+11.0 pp+6.1 ppTable 8
Dependent vs. fixed-surface execution+3.6 pp+4.3 ppTable 5
Scope/obligation checks ON vs. OFF+4.2 pp+3.0 ppTable 6

What do these controlled comparisons establish?

With identical WRITE records, BINDMEM improves over the evaluated KG pipeline by 6.2 pp (54.8% vs. 48.6%; Table R-C4) and over the Python pipeline by 11.0/6.1 pp for Qwen/Gemma (Table 8). The dependent-execution and obligation-check ablations independently support the value of those components (Tables 5–6). Together with the matched-access and deferred-execution controls, these results support the WRITE–READ–DISCLOSE architecture: source-bound materialization, dependency-guided residual execution, and query-specific evidence admission.

Changes in revision: Main Sec. 5 (Table R-M2) for the mechanism comparison and novelty discussion; Main Secs. 3.2–3.4; Appendices A.3–A.4 for dependency-guided WRITE, Appendix B.4 (Table R-B1) for obligation propagation, and Appendix C.6 (Table R-C4) for the KG control.

AuthorsOpenReview ↗

Response to Reviewer 3gEB (3/3)

W4. Does the retrieval–execution gap persist with a flagship reader?

The gap exists for GPT-5 reader, and BINDMEM can narrow the gap. We evaluate GPT-5 on 500 LoCoMo questions using gold supporting turns and session timestamps. Base and BINDMEM receive the same oracle evidence, in the original order. This removes supporting-turn retrieval differences and tests whether BINDMEM's execution and evidence-admission interface still helps when the relevant history is available.

GPT-5 with oracle supporting evidence on LoCoMo

Question typeBaseBINDMEMGain
Single-hop (n=200)94.0%96.0%+2.0 pp
Temporal (n=200)84.0%90.0%+6.0 pp
Multi-hop (n=100)90.0%94.0%+4.0 pp
Overall (n=500)89.2%93.2%+4.0 pp

The paired oracle improvement is 4.0 pp.

BINDMEM answers 466/500 questions correctly, compared with 446/500 for Base. It corrects 28 Base errors while Base corrects 8 BINDMEM errors (exact two-sided McNemar p=0.00119). The largest improvement is on temporal questions (+6.0 pp). The total number of errors falls from 54 to 34, a 37.0% relative reduction. The audit uses automatic binary scoring (exact match or token F1 ≥ 0.5); its scores are separate from the official LoCoMo aggregate metric. Because the gold supporting evidence is shared, this result establishes a residual benefit under matched evidence access, although it does not isolate DISCLOSE alone.

The original evaluation independently supports this result under standard retrieval.

Table 7 reports GPT-5 improvements from 88.5% to 92.7% on LoCoMo (+4.2 pp) and from 77.2% to 80.8% on LME-S (+3.6 pp). We report the oracle audit separately and do not pool its 500 questions with the standard-retrieval evaluations.

Changes in revision: Main Sec. 4.2 (Table 7) for the existing GPT-5 evaluation and oracle-evidence result; Appendix F.3 (Table R-F1) for the 500-question oracle experiment and paired significance test.


W5–W6. Actual records and the timing of WRITE

Actual records. We added the actual records to Appendix A.7 for the executed Section 3.2 examples: date records giving a 6-day difference, pottery-class event records with evidence-only disclosure, and Paris-trip scalar/scope records admitting $350 only with exhaustive coverage.

Clarification of WRITE timing. We modified Fig. 2 to clarify that WRITE runs at ingestion; READ and DISCLOSE run at query time. We added numbers (1) WRITE, (2) READ, and (3) DISCLOSE.

Changes in revision: Main Sec. 3.1 and Fig. 2 for ingestion/query timing; Appendix A.7 for the three stored-record and READ/DISCLOSE examples, including the Paris scope contrast.

Reviewer_LVHJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BINDMEM, a neurosymbolic memory interface for reasoning over conversational histories. WRITE constructs typed records with provenance from the local source context. READ compiles questions into programs and executes them over retrieved records. DISCLOSE checks answer type, precision, and evidence scope before supplying computed values to the reader. When these requirements are unresolved, the system provides available evidence without a derived value while retaining the original retrieved text.

Critical strengths
  1. Consistent improvements supported by controlled experiments. BINDMEM achieves the highest reported score in all twelve comparisons in Table 3. Table 4 fixes retrieved identities and ordering, with LongMemEval S scores increasing from 40.8 to 54.8 for Qwen and from 50.1 to 66.1 for Gemma. The GPT 5 results in Table 7 further support its usefulness with a stronger reader.
  2. Selective disclosure is well motivated and empirically supported. The method distinguishes successful computation from sufficient support for an answer. Table 4 isolates the benefit of admitted values beyond execution selected evidence. Table 6 evaluates scope and obligation checks while holding programs, grounding, execution traces, and selected evidence fixed. These controls support checking answer requirements rather than relying on execution success alone.
Critical weaknesses
  1. The benefits of early materialization are not isolated from representation and information differences. The comparison between Base and Write establishes the benefit of stored interpretations, but not of constructing them at WRITE rather than READ. Appendix A.4 permits deferred execution with the same constructors, yet no matched comparison is reported. Appendix A.1 also allows stored interpretations to preserve information from neighboring turns absent from later retrieval. Identical retrieved identities therefore do not necessarily imply equivalent information. These limitations affect attribution to early materialization, but do not invalidate the overall improvements or the comparison between Full and Evidence only, which shares the stored records.

  2. The reliability of admitted values is insufficiently characterized. Table 5 reports 38 correct traces among 50 cases accepted by the compositional recognizer after regex selection. This audit does not establish the correctness of admitted values across the full system. The operator intent audits in Appendix G also do not verify grounding, scope, or numerical correctness. The audit identifies incorrect traces, but does not show whether DISCLOSE prevents their erroneous values from reaching the reader. Reporting these outcomes would clarify the effectiveness of the disclosure checks.

Questions for authors
  1. Information comparability: How often do WRITE records depend on sources or metadata unavailable to Base? Do the improvements persist when the available source information is matched?
  2. Materialization timing: How does WRITE compare with deferred READ execution using the same constructors, inputs, rendering, and downstream processing?
  3. Disclosure reliability: Among the incorrect traces in Table 5, how many produced an incorrect admitted value, evidence without a value, or no payload? How did these outcomes affect final answers? A representative audit of admitted values would help assess reliability.
3 comments
AuthorsOpenReview ↗

Response to Reviewer LVHJ (1/2)

W1 / Q1. Does WRITE benefit from information unavailable to Base?

WRITE benefits from source-local materialization. Base and BINDMEM retrieve identical turns. WRITE materializes source-local computations (e.g., resolving "yesterday" using a source timestamp) into typed records, without external or oracle information. With identical complete inputs, eager WRITE and deferred READ produce identical records, traces, prompts, and answers (2,486/2,486; Appendix A.6). Thus, partial evaluation preserves source-supported interpretations without changing their computational semantics.

We isolate the materialization benefit under partial retrieval. Without materialization, reconstructing an interpretation may require retrieving the source turn together with its local dependencies (e.g., temporal anchors or neighboring turns). Partial retrieval can omit these dependencies, as observed in 26/500 questions (5.2%). To isolate this effect, we regenerate WRITE records using only Base-accessible inputs and rerun READ, DISCLOSE, and final answering.

Source-access controls (LME-S, Qwen, n=500)

ConditionCorrectAccuracy
Base (original)20440.8%
Base + restored raw inputs20841.6%
Typed WRITE, same restored inputs22645.2%
Full, Base-matched source access26452.8%
Full (standard)27454.8%

Most gains persist under matched source access. Restricting WRITE to inputs available to Base still achieves 52.8%, compared with 40.8% for Base and 54.8% for standard Full. Thus, 12.0 of the original 14.0 pp improvement (85.7%) remains without this input-availability advantage.

Changes in revision: Main Sec. 4.2; Appendix A.5 (Table R-A1, 26/500 input-availability audit and regenerated Base-matched WRITE records); Appendix A.6 (complete-input deferred-execution equivalence).


Q2. How does WRITE compare with deferred READ execution?

WRITE resolves source-local dependencies through binding-time-aware partial evaluation. Before ingestion, template-level dependency analysis classifies source-local inputs as static and future-query inputs as dynamic. At ingestion, WRITE executes the source-complete recipes, for example, anchoring relative dates to source timestamps and resolving local references, and stores their typed results with provenance and evidence scope. Query-dependent selection, linking, and aggregation remain at READ. This placement matters when the original source-local inputs are unavailable in later retrieval.

Deferred READ is equivalent when all original source-local inputs remain available. We replayed 500 LME-S and 1,986 LoCoMo questions using identical constructors, complete versioned source-local inputs, grounding, rendering, and downstream reader settings. Records, execution traces, reader prompts, and final answers were identical in 2,486/2,486 cases. Thus, executing the same constructors earlier does not intrinsically improve accuracy.

Partial retrieval reveals the benefit of resolving dependencies at ingestion. We additionally evaluate deferred execution using only the inputs available in the retrieved context, without restoring missing source-local dependencies.

Eager vs. deferred execution (LME-S, Qwen, n=500)

ConditionCorrectAccuracy
Eager WRITE (standard Full)27454.8%
Deferred READ (complete inputs)27454.8%
Deferred READ (retrieval-only)26452.8%

Retrieval-only deferral loses 2.0 pp relative to WRITE, consistent with the 52.8% matched-access control and 26/500 source-access mismatches (W1/Q1).

The distinction is when source-local dependencies are resolved. With complete source snapshots, deferred READ matches WRITE. Under partial retrieval, those inputs may be missing; WRITE preserves their computed interpretations for subsequent query-dependent READ operations.

WRITE also avoids recomputing shared interpretations: LME-S has 1,100 constructor requests over 460 distinct inputs (2.39 requests/input) without caching. Caching removes repetition but cannot restore missing inputs.

Changes in revision: Main Secs. 3.2 and 4.2; Appendix A.6 (Tables R-A2–R-A3), with complete-input equivalence (2,486/2,486), retrieval-only deferred execution (52.8%), and constructor reuse analysis.

AuthorsOpenReview ↗

Response to Reviewer LVHJ (2/2)

W2 / Q3. What happens to incorrect traces, and are admitted values supported?

Incorrect execution traces do not necessarily propagate to final answers. In Table 5 of the original paper, 12 of the 50 audited execution traces were incorrect. We further traced these 12 cases through DISCLOSE and final answering. None disclosed an incorrect computed value: five produced evidence-only payloads, and seven produced no additional payload. Relative to paired final-answer controls, one answer improved, eleven were unchanged, and none worsened.

Why do scope checks matter? In Fig. 2, the same flight ($300) and train ($50) produce a $350 subtotal. But that subtotal is the total transport expense only when the source establishes exhaustiveness.

Same arithmetic, different evidence sufficiency (example)

Source evidenceWithout scope checksWith scope checks
"The flight and train were my only transport expenses"Admit $350Admit $350
"My transport expenses included the flight and train"May admit $350Withhold value

The no-check condition still verifies type validity and answer uniqueness; it removes only scope/obligation checks (Table 6). The table illustrates the rule.

Two controlled tests confirm the practical benefit. In 50 matched pairs with identical operands and arithmetic but different evidence completeness, BINDMEM correctly admits the complete case in 48/50, withholds the incomplete case in 46/50, and makes both decisions correctly in 45/50 pairs. With programs, grounding, traces, and evidence fixed, admission checks also improve LME-S final-answer accuracy (Table 6):

Effect of scope/obligation checks (LME-S; Table 6)

ReaderChecks OFFChecks ONGain
Qwen50.6%54.8%+4.2 pp
Gemma63.1%66.1%+3.0 pp

How reliable are admitted values? Among 500 LME-S/Qwen questions, DISCLOSE admitted a computed value in 185 cases. We randomly selected 100 of these cases and used GPT-5 to check each value against the original source evidence.

Most admitted values were correct and supported. In 98/100 cases, the value, referent, unit, and precision were correct. In 93/100 cases, the source evidence also supported the full scope of the requested answer. The remaining 7 cases lacked sufficient evidence for the value or its scope.

Do unsupported values affect final answers? We removed the disclosed values in these 7 cases and reran final answering with all other inputs unchanged. Four answers improved, one worsened, and two remained unchanged. This is a post-hoc test of the impact of unsupported values.

The 100 audited admissions were sampled randomly; GPT-5 judged source support, while GPT-4.1 remained the end-to-end evaluator.

Changes in revision: Main Secs. 3.4 and 4.2 (admission-reliability summary); Appendix C.5 (Tables R-C1–R-C2), with incorrect-trace outcomes, completeness contrasts, and the 185-admission/100-case source-support and withholding audits.

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback. We have made every effort to address your concerns through additional experiments, analyses, and revisions.

As the discussion period is nearing its end, we would sincerely appreciate hearing whether our responses have adequately addressed your concerns, or if any issues remain.

Thank you very much for your time and consideration.

Authors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Summary of Rebuttal and Revisions

We sincerely thank the reviewers for their thoughtful feedback and the AC for facilitating the discussion and encouraging further evaluation of our responses. We have made substantial revisions and conducted targeted experiments to address the main concerns.

Key additional evidence:

  • WRITE and source access: Matched-input and deferred-execution controls confirm the benefit of preserving source-local interpretations under partial retrieval.
  • Novelty and mechanism: Same-WRITE KG, Python, and post-hoc verification comparisons isolate the contributions of dependency-guided execution and evidence-aware admission.
  • Reliability: Execution-trace and admitted-value audits characterize error propagation and disclosure reliability.
  • Flagship models: GPT-5 shows a significant improvement under identical oracle evidence (+4.0 pp, p=0.00119).
  • Efficiency and applicability: Matched serving demonstrates +14.0 pp accuracy with 11.3% higher median latency, while WildChat evaluates operational coverage beyond curated QA benchmarks.

These results strengthen the central contribution of BINDMEM: source-time materialization, query-time compositional execution, and evidence-aware value admission.

We hope the additional evidence and revisions help inform the final assessment.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.