- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies individual delivery-rider simulation using LLMs conditioned on personal activity histories, focusing on inconsistencies between LLM-generated memories and source measurements. The authors propose Measurement-Grounded Memory, which represents memories as natural-language statements with structured numeric claims and explicit dependencies. A deterministic checker identifies conflicts against matching source measurements and retracts dependent statements, with formal guarantees for the resulting memory structure. Experiments on real rider data reveal frequent numeric conflicts in generated memories and evaluate the proposed framework through preregistered forecasting studies, additional rider cohorts, and component comparisons, showing improvements over unchecked memory in several forecasting settings.
Critical strengths
The paper presents an executable framework combining measurement-based consistency checking with dependency retraction and provides formal guarantees for the resulting memory structure.
Critical weaknesses
-
The reported high memory conflict rate is based on Qwen3.5-9B with thinking disabled, without evaluating whether similarly severe inconsistencies persist with substantially stronger open- or closed-weight models. Since the proposed method requires neither model training nor access to internal model states, evaluating stronger memory writers is practical and would be important to validate whether this main motivation problem really exists. More fundamentally, the structured numeric claims and dependencies are generated by the same LLM as the natural-language memory and may themselves be unreliable. Table 18 reports model-judge claim-binding precision of approximately 47%, suggesting substantial mismatches between structured claims and their corresponding text. Since the entire consistency-checking framework relies on these structured fields, their unreliability directly undermines the practical validity of the framework, which cannot be established by formal guarantees over the declared structure alone.
-
The benefit or necessity of dependency retraction over simpler conflict-handling methods is not sufficiently established. Although checked memory improves forecasting over unchecked memory, the comparisons between direct-only retraction (TL) and full dependency retraction (TC7) in Appendices F and J do not establish a clear advantage for the latter. In the later study involving 1,934 riders, their performance differences are also close to zero. Furthermore, TC7 does not demonstrate a clear improvement in offer log loss over simple numeric repair (SR). These results make the additional practical value of constructing dependency graphs and propagating retractions insufficiently demonstrated.
Questions for authors
-
Could the authors evaluate numeric conflict rates using substantially stronger open- or closed-weight memory writers, potentially with reasoning enabled? If these models produce more accurate memories, does the proposed consistency check still improve downstream forecasting? Or if these models produce nearly zero conflict rates, would the core motivation for the proposed framework still hold?
-
Given the low claim-binding precision reported in Table 18, how can the authors establish that LLM-generated numeric claims and dependencies faithfully represent the natural-language memory? How do incorrect bindings affect the reliability of conflict detection and retraction?
-
Given the limited predictive differences between TC7 and TL in Appendices F and J, what empirical evidence supports the additional value of dependency propagation beyond direct conflict retraction?
5 comments
Response to Reviewer u97R (2/2)
This second comment answers your questions on write-time errors and on binding precision.
How often is the written memory wrong? In the seven-day study we compared every numeric claim in all 2,609 memories with the table the writer had just read. Of 19,926 claims, 35.7% [35.0, 36.5] were outside tolerance. This is write-time error only, with no drift. At the forecast cutoff the share is 36.8% [35.9, 37.8].
The check helps where there are conflicts. Among the 1,923 riders with a direct conflict, checked minus plain memory is −0.0316 [−0.0466, −0.0169] in offer log loss. Among the 285 riders without one it is −0.0032 [−0.0069, −0.0004] (Section 5.4). The check costs 0.013 ms per memory (Table 25). On your conditional question: even a writer with zero write-time error produces conflicts at the forecast cutoff whenever the rider's drift between the two cutoffs exceeds the tolerance (Proposition 3(iv), Appendix S.2). In the 295-rider study, drift alone exceeds the tolerance in 329 attributed conflicts (Appendix C). Results with stronger writers are in our reply below.
Binding precision of about 47% (Table 18). Table 18 reports binding precision of 47.6% and 47.2% by two model judges. For retraction, what matters is whether the retracted statements are in fact contradicted. Of the statements the check retracts directly, the two judges find 91.1% and 85.8% contradicted, and both call the retraction unnecessary in 4.6% [3.0, 6.4] of 862 cases. Among the statements that stay on display, the share that both judges find contradicted falls from 26.2% to 16.1% with retraction.
Thank you for the additional experiments. However, my main concern in Weakness 1 regarding stronger memory writers has not been directly addressed. I consider this important because the high conflict rate of LLM-written memory is a central motivation and starting point of the work. While the response explains that temporal drift can still introduce conflicts even with a perfectly faithful writer, this is a different source of inconsistency from the substantial write-time errors emphasized in the original submission.
I therefore still believe that evaluating a stronger open-weight writer, or at least enabling reasoning for the current model, is necessary. The restriction on external APIs does not preclude these alternatives. The key missing evidence is whether the conflict rate remains substantial with a stronger writer and, if it becomes much lower, whether the consistency check still provides meaningful benefits.
I also encourage the authors to manually verify and edit any AI-generated text in the response, or provide a more directly human-written response. Both the submission and the rebuttal are difficult to follow, and the reviewer concerns should be addressed more directly and clearly.
Response to Reviewer u97R (1/2)
Thank you for the detailed review. This comment answers your question on what dependency propagation adds. The next one covers write-time errors and binding precision.
To answer it we repeated the forecasting study at a larger scale: 2,208 riders, a seven-day window, and the same writer and forecaster as in the paper (Qwen3.5-9B). The results are in Section 5.4 and Appendix V of the revised PDF (new text in blue).
The full check beats numeric repair. Mean offer log loss by what the forecaster is shown (Table 45):
The full check minus numeric repair is −0.0286 [−0.0412, −0.0170] in offer log loss and −0.0050 [−0.0075, −0.0027] in refusal-fraction error.
Propagation versus direct retraction only. The difference in offer log loss is small: −0.0023 [−0.0111, 0.0068]. What propagation adds is consistency. With direct retraction only, conclusions stay on display after the premise they rest on has been withdrawn: 527 such statements in 247 of 295 riders (Section 5.2). Propagation removes them, and the result does not depend on the order of processing (Proposition 1). In the seven-day study it removes 1.55 further statements per rider.