acceptodds
Under review as a conference paper at ICLR 2027

Answering Is Not Repairing: Queryable World Memory for Changing Sources

Abstract

Agents repeatedly interpret heterogeneous observations, but later queries may arrive after either the source or the task has changed. We ask what should survive compression so that interpreted state remains reusable and its applicability can be reassessed. Queryable World Memory (QWM) has two conceptual components: budgeted selective compilation of observations into persistent records, and query-time reassessment of whether a record should be reused, checked, or replaced. Records separate answer state from repair certificates that identify re-observable support. Controlled SQL and synthetic longitudinal replays show that certificates can localize targeted rereading. In the longitudinal replay, QWM reaches 81/86 exact answers versus 69/86 for full rereading while using 79.2% fewer supplied observation units. A one-transition replay over natural PyPI and GitHub histories finds similar observed accuracy to full rereading (139/168 versus 140/168) with 10.1% fewer input tokens, but lower accuracy than answer-only memory (149/168). On the 22 certified cells, every evaluated arm is correct, while QWM uses 9,345 input tokens versus 64,554 for full rereading. The measured coverage–repair frontier quantifies one cost of reusable interpretation: support evidence saves rereading where retained but displaces answer content. The three main comparisons do not verify a learned writer, new-query generalization, or persisted multistep updates; an exploratory appendix reports a bounded synthetic execution of a fixed model writer without upgrading those claims.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Autonomous Agents, Agent Memory, Semantic Drift, Knowledge Repair, Tool Use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_qaC7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper measures the trade-off, under a fixed memory budget, between caching more past answers and storing source-dependency metadata (e.g. spending more memory budget for storing not only each answer but also where to find it) that makes answers cheaper to recheck after sources change.

Critical strengths

It could have been an interesting research problem if the experiments and presentations were done thoroughly.

Critical weaknesses
  1. Unclear novelty. The authors state that query-time probing, structured caching, provenance links, temporal validity and selective verification are prior work. They also state QWM has no advantage over an equally informative cache. TierMem (Zhu et al., 2026) already links lossy summaries to immutable evidence and escalates when evidence is insufficient.

  2. Half the framework never runs. QWM is defined as two components (Budgeted selective compilation and Applicability reassessment), but only compilation is executed. No model or rule performs applicability reassessment (Section 3.3) in any main experiment.

  3. No existing-system baseline. Records are built deterministically from exact old answers. No Mem0, A-MEM, Zep, AgeMem or TierMem baseline is run. The natural-track "text summary" is a JSON rendering of the old answer.

Questions for authors
  1. What should a practitioner take from this paper?
  2. Can the author run an experiment where the agent itself decides between reuse and re-exploration?
2 comments
AuthorsOpenReview ↗

Response to Reviewer qaC7

The revision executes reassessment and persistent updates and includes an actual TierMem adaptation. Appendix I distinguishes these additions from the original replay and from later exploratory variants.

The original main replay did not execute a general query-time controller. Added experiments now execute model-selected PROBE/READ/ANSWER/ABSTAIN actions and reload committed memory. The matched policy intervention separately measures mandatory checking. We explicitly distinguish autonomous choices from host-enforced actions and retain all failures (Appendix E).

We recognize TierMem's existing provenance-guided escalation and write-back. Our actual adaptation adds a real-framework baseline, scoring 17/17 alongside equally accurate lightweight controls. This provides an implemented system comparison on the compatibility task, where the outcomes do not distinguish accuracy. It is an adaptation; original-benchmark reproduction and standalone comparisons with the other listed systems remain outside this evaluation (Appendix F).

In the evaluated fixed-schema setting, task-aware lightweight state can recover source selection without heavyweight support. Checking failures can dominate content errors, and fewer reads can coexist with higher model cost. Appendix G tests the latter tradeoff with the same rule solver and checking policy across all storage arms. These findings motivate conditional design choices rather than a blanket deployment recommendation.

The stable-only protocol executes 432 of 648 scheduled queries, all correct; 216 BentoML queries are unexecuted because six sources lack stable versions. A separately frozen prerelease fallback completes those 216 queries correctly and is reported separately. The preflight omission and original gap remain visible. These are simulated direct-dependency environments, not broad real-user validation or independent LLM reasoning.

Appendix H now holds the controller fixed while varying representation, without mandatory actions or a solving tool: minimal state 6/16, fixed certificates 4/16. All 32 queries read current sources. Offline diagnosis separates nine wrong answers with sufficient visible evidence from thirteen lacking usable historical constraints; this does not alter model scores or prove causal attribution.

Do the executed trajectories address the implementation concern? Is the remaining concern primarily the generality of the setting or the distinction from existing provenance-aware memory systems?

AuthorsOpenReview ↗

Follow-up on reassessment execution and the system baseline

We would like to clarify which additions address your two implementation concerns.

Appendix E includes executed PROBE/READ/ANSWER/ABSTAIN trajectories and persistent memory updates. Appendix H further evaluates model-selected actions without mandatory checking or a solving tool. Thus reassessment is now executed, although the autonomous representation comparison does not show a certificate advantage.

Appendix F includes an actual TierMem adaptation using its trained router, Mem0 and persistent Qdrant. It and the lightweight controls score 17/17. Deployment changes and disabled BM25 are disclosed; this is an implemented comparison on the compatibility task, not reproduction of TierMem’s original benchmark.

The practitioner implication is conditional: use lightweight state when dependencies are recoverable from the task, and assess retained support against both historical-access costs and repeated context costs. Do these additions resolve the execution and missing-system-baseline concerns, while leaving generality and novelty as the remaining issues?

Reviewer_FcboOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Queryable World Memory (QWM), which combines selective memory compilation with query-time reassessment using repair certificates. The authors evaluate the approach on SQL tasks, synthetic longitudinal records, and software-release histories. Results show that repair certificates can reduce rereading costs, but improvements in overall accuracy are limited and depend on memory coverage.

Critical strengths

The paper addresses a relevant problem in persistent agent memory: how to reuse stored information when external sources change. Separating answer content from evidence needed for later verification is a reasonable design choice. The evaluation also considers both accuracy and resource costs.

Critical weaknesses
  1. Limited novelty and unclear technical contribution. The proposed approach largely combines existing ideas from structured caching, provenance tracking, and selective retrieval. The paper acknowledges that an equally informative cache with the same policy behaves identically to QWM. It remains unclear what fundamentally new algorithmic or methodological contribution is established, especially for the ICLR paper level of quality.

  2. Incomplete validation of the proposed framework. The main experiments rely on deterministically constructed memory records and controlled replay settings. Key components, including learned memory compilation, autonomous query-time reassessment, and persistent multistep repair, are not evaluated. As a result, the experiments needs further validations on these parts.

  3. Unconvincing practical benefits. On the natural software-release benchmark, QWM achieves 139/168 correct answers, compared with 149/168 for answer-only memory. While repair certificates reduce rereading costs on selected cases, the overall accuracy and efficiency trade-off does not clearly justify the additional complexity of the framework. Further evidence is needed here.

Questions for authors
  1. What is the fundamental novelty of QWM compared with existing structured caches and provenance-aware memory systems?
  2. Can the authors evaluate the complete framework with model-generated memory records and autonomous query-time repair across multiple source changes?
  3. How do the authors justify the practical value of QWM when the simpler answer-only baseline achieves better overall accuracy on the natural benchmark?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Fcbo

Repair savings on retained records must be evaluated together with displaced answer coverage and subsequent evidence-use costs. QWM makes this allocation measurable; the controlled studies provide the empirical contribution.

Building on structured caching, provenance links, and selective retrieval, we evaluate three distinct dimensions: capacity allocated to answers versus support, information recoverable from the task, and the policy used during reassessment. The contribution is this empirical characterization, rather than a new caching primitive.

We added real model writing, tool selection, persistence, and a matched checking-policy intervention. DeepSeek has 59 FREE errors without observations and 24 PROBE errors after checking without rereading. GUARD has no answered-but-incorrect cases, but retains 16 invalid outputs/abstentions. This localizes failures at two decision points—obtaining observations and reading after a check—under the matched intervention (Appendix E).

The strongest task-aware lightweight control outperforms the original QWM policy, so we do not recommend extra support complexity for that fixed-schema workload. The common-solver study preserves this distinction: selective retention can reduce historical rereads while increasing model-context cost. Its high accuracy depends on host checking and a rule-solving tool; the model accepts, reuses, or abstains. The actionable lesson is to evaluate coverage and full lifecycle resources jointly, and to test whether dependencies are already recoverable from the task schema. Selective stripping is a separately labeled post-review variant, not a replacement for the original method (Appendices D/G).

A further fixed-budget comparison without host enforcement or an online solver gives minimal state 6/16 and certificates 4/16. This separates executed controller behavior from representation utility: the fixed certificates have no aggregate advantage in this small, previously inspected DEV setting (Appendix H).

The revised positioning distinguishes our shared-capacity external-change experiments from existing provenance and modular-evaluation methods. Is the remaining concern primarily the informativeness of these findings, or a specific overlap that needs closer comparison?

AuthorsOpenReview ↗

Follow-up on contribution and executed evaluation

We would appreciate your assessment of whether the added experiments address the implementation concern separately from the novelty concern.

Appendix E now executes model-written memory, tool selection and persistent updates, while distinguishing autonomous actions from enforced checking. Appendix H additionally compares representations under the same autonomous controller and byte budget, without host-enforced actions or an online solver. Minimal state scores 6/16 and certificates 4/16; this result is retained rather than presented as a certificate advantage.

The revised contribution is an empirical characterization of when support displaces useful answer coverage, when task information makes support redundant, and when reduced retrieval increases model-context cost. It does not introduce a new caching primitive or establish general autonomous superiority.

Section 5.5, Table 4 summarizes these distinctions. Could you clarify whether incomplete execution remains a concern, or whether your remaining reservation is primarily the significance of these empirical findings?

Reviewer_rhCUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The author wants to propose an agent's memory that not only answer questions, but also be efficiently verified and repaired when the environment changes. And they argue that traditional memory typically only preserves the answer state. What does matter is that the answer came from or what evidence it relied on. It cost more so they build Queryable World Memory(QWM)

  1. Selective Compilation: With a limited storage budget, selectively save the Answer State and Repair Certificate.
  2. Applicability Reassessment: During the query, the applicability of the memory is reassessed, and a decision is made to reuse it directly, conduct a targeted check, repair it, re-observe it, or discard the answer.

They use SQL repair synthetic longitudinal repair and natural software-release utility to evaluate.

Critical strengths

S1. The paper addresses an important problem: maintaining reliable agent memory under changing external sources.

S2. The proposed repair certificates demonstrate promising efficiency improvements by reducing unnecessary re-observation in selected scenarios.

Critical weaknesses

W1. Baseline missing and insufficient data persuasiveness The paper lacks direct comparisons with established agent memory systems and simpler version-aware caching strategies. It remains unclear whether QWM provides meaningful benefits beyond existing provenance-aware memory mechanisms.

W2. Lack of Lifecycle Amortization Only single-call replay and single-state transitions were evaluated, and the repaired state was not persisted for reuse. In real-world multi-turn query scenarios, the cost of loading and updating memory on demand can be amortized by subsequent queries. The current experimental setup in this paper artificially amplifies the capacity disadvantage of the repaired credentials.

W3. The cost-effectiveness of the credential design is questionable (Heavyweight Certificate vs. Lightweight Pointer): Why is it necessary to store such heavy repair credentials, which take up nearly half of the answer capacity (33 vs. 62)?

If a minimalist"source pointer + timestamp/version number (ETag)" is used in conjunction with on-demand loading, can efficient updates be achieved with almost no sacrifice in answer coverage?

Questions for authors

Q1. Can you compare to other Memory structures? Such as a native rag? tiermem? Zep and so on.

Q2. What additional capabilities do repair certificates provide beyond a lightweight source pointer combined with timestamp/version tracking and on-demand retrieval?

4 comments
AuthorsOpenReview ↗

Response to Reviewer rhCU

We directly test the lightweight alternative and measure persistent reuse costs. This identifies both a setting favoring lightweight state and the external-access cost needed to offset retained support.

In the fixed-schema workload, task-routed answer-only reaches 162/168 versus QWM's 142/168 with 59.1% fewer input tokens. Auditing finds that task schema reconstructs the certificate-selected evidence sets on all 168 project-task pairs across the original splits. This identifies a concrete setting in which lightweight state is the better allocation. The original 33/62 figures count retained answer records, not certificate bytes (Appendix D).

Added runs retain updates across queries and charge writing, checking, reading, and model decisions. In Qwen's matched lifecycle study, PROBE/GUARD both reach 144/144; GUARD uses 6.2% fewer input-plus-output tokens despite one more source read. The subsequent common-solver capacity study likewise separates historical-access savings from context cost. At full reference capacity, selective support reduces historical reads from eight to four per 24-query validation workload, but API cost increases: CNY0.039336→0.050720 for Great Expectations and 0.039876→0.050996 for MLflow. All budgets are reported. Retained support exchanges historical access for repeated context transmission, making both resources necessary for the comparison (Appendices E/G).

We executed TierMem's actual pipeline with its trained router, Mem0, and persistent Qdrant. Both this adaptation and lightweight controls score 17/17. Disabled BM25, changed endpoints/deployment, page boundaries, interface repairs, and decoding differences are disclosed. It is an adaptation, not original-benchmark reproduction or superiority evidence (Appendix F).

For the separate common-solver study at full reference capacity, the extra API cost is offset only if each saved historical access has an additional cost above approximately CNY0.002846 (Great Expectations) or CNY0.002780 (MLflow), holding other costs equal. These are hypothetical break-even thresholds, not measured deployment savings (Appendix H).

Does the task-routed control address the lightweight-baseline concern? We would appreciate clarification on any remaining mismatch in access assumptions or cost accounting.

Thank you for the detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns.

The new task-routed answer-only baseline is particularly informative. However, if this lightweight approach can reconstruct all certificate-selected evidence sets while achieving higher accuracy (162/168 vs. 142/168) with 59.1% fewer input tokens, what is the fundamental advantage of retaining the additional repair certificates? In which practical scenarios would QWM provide benefits that cannot be achieved by such a lightweight approach?

I am also concerned about whether the comparison accounts fairly for the cost of task schemas. Are task schemas treated as free external knowledge? If dependency information is implicitly encoded in task-routing rules for the lightweight baseline but explicitly charged against QWM's memory budget, the comparison may not reflect equivalent cost assumptions.

While the additional experiments address some of my concerns about baseline coverage and lifecycle costs, I remain unconvinced that the necessity and overall cost-effectiveness of QWM have been sufficiently demonstrated.

Therefore, I may be inclined to maintain my original rating.

AuthorsOpenReview ↗

Clarification on recoverable dependencies and cost accounting

Thank you for identifying the distinction between recoverable task information and retained historical information. In the fixed-schema workload, the task-routed baseline reconstructs the selected evidence sets, so retaining those same dependencies provides no demonstrated advantage. We interpret this result as identifying when additional support is redundant.

The potential benefit arises when the historical dependencies cannot be reconstructed from the current query and shared routing rules. Retaining them can then reduce historical retrieval. Our common-solver study demonstrates this resource tradeoff: historical reads decrease from eight to four at full reference capacity, while API cost increases. This supports reduced historical access, not an unconditional cost advantage. An equally informative lightweight cache could also provide this capability; we do not claim that the certificate format is uniquely necessary.

Your accounting concern is important. The reported 59.1% reduction concerns model input tokens; it does not by itself establish a reduction in total system cost, including task-schema construction and storage. A complete comparison must distinguish shared routing logic from record-specific dependencies and account for each consistently. Shared rules should also be amortized rather than charged repeatedly as independent records. We therefore do not use this token comparison as evidence of universal cost-effectiveness.

Would you consider the baseline-coverage and lifecycle-evaluation concerns addressed, with the remaining disagreement concerning the scope and significance of this conditional contribution?

Thank you for the clarification and additional experiments. I acknowledge that the new baselines and lifecycle studies have substantially addressed my concerns regarding experimental coverage.

However, I remain concerned about the necessity and significance of QWM. The additional experiments do not sufficiently demonstrate QWM's distinct advantages or overall cost-effectiveness.

Therefore, I am inclined to maintain my original rating.

Reviewer_TcLHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what an agent should keep in memory when its information sources can change. They propose Queryable World Memory (QWM) that separates the stored answer from a “repair certificate,” which records the source locations and dependencies needed to check or repair that answer later. Both consume the same memory budget, so keeping more repair evidence means keeping fewer answers.

The experiments cover SQL schema changes, synthetic longitudinal records, and natural software-release histories. Certificates reduce rereading on the records that retain them. However, on the natural source evaluation, QWM is less accurate than answer-only memory. The main finding is this coverage-repair tradeoff, rather than an overall improvement in agent performance.

Critical strengths
  • The distinction between remembering an answer and remembering enough to repair it is useful. The natural-source result makes this concrete: QWM saves input tokens, but scores 139/168 compared with 149/168 for answer-only memory. I find this negative result more informative than another small improvement on a memory benchmark. It shows why storing more provenance is not automatically better under a fixed budget.

  • The paper does a good job separating conditional results from population results. In particular, it does not present the large savings on certified records as savings over the whole workload. It also reports the model-dependent reversal in the longitudinal experiment. These details make the empirical conclusions easier to trust.

Critical weaknesses

The main experiments give the system much of the difficult work in advance. They built historical records deterministically from exact old answers, and the harness supplies the observation views before a single model call. Thus the experiments mainly test whether the model can answer from a prepared targeted view. They do not test whether an agent can reliably write the right certificate, notice when it needs checking, and choose the right observations. The appendix writer experiments are useful, but they do not close this gap. This substantially limits the contribution as an agent-memory method.

Questions for authors

In the SQL experiment, why does full rereading achieve only 12/24 even without a schema change, and 4/24 under a relevant rename? What information does the certificate run retain that the full-reread run must reconstruct?

How sensitive is the natural-source result to the certificate reward coefficient and memory capacity? Did any tested setting retain the token savings without the observed accuracy loss against answer-only memory?

2 comments
AuthorsOpenReview ↗

Response to Reviewer TcLH

We explain the SQL information asymmetry, quantify capacity sensitivity, and add executed writing and persistent reassessment. These directly address your questions about the replay setup.

We added model-written memory, model-selected tools, and durable state updates. A matched DEV intervention shares the initial writer output and interface across FREE, mandatory PROBE, and PROBE-plus-stale-READ GUARD. DeepSeek scores 62/99/128 out of 144; Qwen scores 83/144/144. The matched intervention measures the effect of checking policy under reliable simulated version signals; autonomous and enforced actions are reported separately (Appendix E).

The original contrast also preserves historical SQL in the memory arms and supplies an ordinal-based rename instruction to the certificate arm. All 12 no-drift rereading errors executed but returned nonmatching results; relevant drift produced eight wrong results, seven abstentions, and five execution errors. Generated SQL text was not retained, so finer semantic attribution is unavailable. We explicitly remove a representation-only interpretation (Appendix D).

The four-capacity/seven-reward grid shows that beta=1 at 50% capacity retains 60 answers/one certificate, versus 33 answers/11 certificates at beta=1.5. Under the same first-response map, beta=1 matches the answer-only reference at 80/84 and 75/84 with 1.85% fewer input tokens. This uses exact-prompt response reuse, not independent trials or a non-inferiority test; the inspected tasks are not an independent tuning evaluation.

With the same autonomous controller and byte budget, minimal state scores 6/16 versus fixed certificates 4/16 on two previously inspected DEV projects. Every query reads current sources; this supplies executed trajectories but no autonomous certificate advantage (Appendix H).

Does the SQL audit and capacity grid address the interpretation and sensitivity concerns, or is a specific comparison still unclear?

AuthorsOpenReview ↗

Follow-up on SQL interpretation and capacity sensitivity

Thank you again for highlighting the coverage–repair tradeoff. We would like to draw attention to two results addressing your specific questions.

Appendix D explains that the original SQL comparison also differs in available historical information: the memory arms retain historical SQL, and the certificate arm receives an ordinal-based rename instruction. We therefore do not interpret that contrast as isolating representation alone.

The capacity/reward grid also identifies a setting matching the answer-only reference at 80/84 and 75/84, with 1.85% fewer input tokens. These are exact-prompt response-reuse results on inspected tasks, rather than independent confirmation of non-inferiority.

The revised manuscript summarizes the added evidence in Section 5.5, Table 4, and the updated supplementary package provides offline scoring checks. Do these clarifications address your SQL and sensitivity questions, or does a specific issue remain unresolved?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.