- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper measures the trade-off, under a fixed memory budget, between caching more past answers and storing source-dependency metadata (e.g. spending more memory budget for storing not only each answer but also where to find it) that makes answers cheaper to recheck after sources change.
Critical strengths
It could have been an interesting research problem if the experiments and presentations were done thoroughly.
Critical weaknesses
-
Unclear novelty. The authors state that query-time probing, structured caching, provenance links, temporal validity and selective verification are prior work. They also state QWM has no advantage over an equally informative cache. TierMem (Zhu et al., 2026) already links lossy summaries to immutable evidence and escalates when evidence is insufficient.
-
Half the framework never runs. QWM is defined as two components (Budgeted selective compilation and Applicability reassessment), but only compilation is executed. No model or rule performs applicability reassessment (Section 3.3) in any main experiment.
-
No existing-system baseline. Records are built deterministically from exact old answers. No Mem0, A-MEM, Zep, AgeMem or TierMem baseline is run. The natural-track "text summary" is a JSON rendering of the old answer.
Questions for authors
- What should a practitioner take from this paper?
- Can the author run an experiment where the agent itself decides between reuse and re-exploration?
2 comments
Follow-up on reassessment execution and the system baseline
We would like to clarify which additions address your two implementation concerns.
Appendix E includes executed PROBE/READ/ANSWER/ABSTAIN trajectories and persistent memory updates. Appendix H further evaluates model-selected actions without mandatory checking or a solving tool. Thus reassessment is now executed, although the autonomous representation comparison does not show a certificate advantage.
Appendix F includes an actual TierMem adaptation using its trained router, Mem0 and persistent Qdrant. It and the lightweight controls score 17/17. Deployment changes and disabled BM25 are disclosed; this is an implemented comparison on the compatibility task, not reproduction of TierMem’s original benchmark.
The practitioner implication is conditional: use lightweight state when dependencies are recoverable from the task, and assess retained support against both historical-access costs and repeated context costs. Do these additions resolve the execution and missing-system-baseline concerns, while leaving generality and novelty as the remaining issues?
Response to Reviewer qaC7
The revision executes reassessment and persistent updates and includes an actual TierMem adaptation. Appendix I distinguishes these additions from the original replay and from later exploratory variants.
The original main replay did not execute a general query-time controller. Added experiments now execute model-selected PROBE/READ/ANSWER/ABSTAIN actions and reload committed memory. The matched policy intervention separately measures mandatory checking. We explicitly distinguish autonomous choices from host-enforced actions and retain all failures (Appendix E).
We recognize TierMem's existing provenance-guided escalation and write-back. Our actual adaptation adds a real-framework baseline, scoring 17/17 alongside equally accurate lightweight controls. This provides an implemented system comparison on the compatibility task, where the outcomes do not distinguish accuracy. It is an adaptation; original-benchmark reproduction and standalone comparisons with the other listed systems remain outside this evaluation (Appendix F).
In the evaluated fixed-schema setting, task-aware lightweight state can recover source selection without heavyweight support. Checking failures can dominate content errors, and fewer reads can coexist with higher model cost. Appendix G tests the latter tradeoff with the same rule solver and checking policy across all storage arms. These findings motivate conditional design choices rather than a blanket deployment recommendation.
The stable-only protocol executes 432 of 648 scheduled queries, all correct; 216 BentoML queries are unexecuted because six sources lack stable versions. A separately frozen prerelease fallback completes those 216 queries correctly and is reported separately. The preflight omission and original gap remain visible. These are simulated direct-dependency environments, not broad real-user validation or independent LLM reasoning.
Appendix H now holds the controller fixed while varying representation, without mandatory actions or a solving tool: minimal state 6/16, fixed certificates 4/16. All 32 queries read current sources. Offline diagnosis separates nine wrong answers with sufficient visible evidence from thirteen lacking usable historical constraints; this does not alter model scores or prove causal attribution.
Do the executed trajectories address the implementation concern? Is the remaining concern primarily the generality of the setting or the distinction from existing provenance-aware memory systems?