acceptodds
Under review as a conference paper at ICLR 2027

Simulating Individual Delivery Riders from Their Own Histories with Measurement-Grounded Memory

Abstract

Delivery platforms want to simulate how an individual rider will behave by conditioning a language model on that rider's own history. When such a simulator must show LLM-written memory to its forecaster, the memory's numbers have to stay tied to the rider's measurements. We build the simulator around a measurement contract: source statistics keep their numerator, denominator, unit and window; LLM-written memory is stored as typed, unverified claims with declared dependencies; a deterministic check flags a claim only against a measurement of the same definition; and dependency closure retracts every statement built on a flagged claim. We prove that the retained memory is the unique largest conflict-free, dependency-closed sub-memory, that budgeted selection is feasible and maximal, and that a premise-preserving selector keeps both guarantees after retraction. Across 316 riders, conditioning on the rider's own history lowers offer-response log loss by 0.0892 relative to population context and raises accuracy by 15.30 and 8.98 points over wrong-identity and matched-neighbor histories, so the simulator represents the intended rider. An outcome-free audit of 590 memories for 295 riders finds 41.6% of the explicit numeric claims outside tolerance even when the memory is written from the very table the measurements come from; an independent reader of the same memories reproduces every parsed claim, source measurement and retraction set. In a preregistered two-week forecasting study with a Qwen3.5-9B writer and forecaster, typed memory with the consistency check is non-inferior and superior to the same memory shown as plain text on the 22 evaluable riders of a holdout group outside every earlier cohort, with differences of in offer log loss and in refusal-fraction error, and across all 295 riders it lowers the share of conflicting metrics on which the forecast lies closer to the stale claim than to the fresh measurement from 22.3% to 4.7%. A registered validation at five new forecast cutoffs finds checked memory non-inferior to plain memory on offer log loss, and on 789 never-exposed riders of a third sample checked memory lowers offer log loss relative to plain memory by 0.1239 with interval in a registered descriptive contrast. In a registered large-sample confirmation on a 9,350-rider cohort, combining an LLM forecast that reads the rider's memory documents and measurements with a smoothed personal statistical predictor lowers refusal-fraction error by 9% relative to that predictor alone on 1,998 confirmatory riders, , under both registered tests. The memory layer thus provides measurement-grounded maintenance with structural guarantees for simulators that must use external memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model agents, rider simulation, personal behavioral histories, identity controls

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u97ROpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies individual delivery-rider simulation using LLMs conditioned on personal activity histories, focusing on inconsistencies between LLM-generated memories and source measurements. The authors propose Measurement-Grounded Memory, which represents memories as natural-language statements with structured numeric claims and explicit dependencies. A deterministic checker identifies conflicts against matching source measurements and retracts dependent statements, with formal guarantees for the resulting memory structure. Experiments on real rider data reveal frequent numeric conflicts in generated memories and evaluate the proposed framework through preregistered forecasting studies, additional rider cohorts, and component comparisons, showing improvements over unchecked memory in several forecasting settings.

Critical strengths

The paper presents an executable framework combining measurement-based consistency checking with dependency retraction and provides formal guarantees for the resulting memory structure.

Critical weaknesses
  1. The reported high memory conflict rate is based on Qwen3.5-9B with thinking disabled, without evaluating whether similarly severe inconsistencies persist with substantially stronger open- or closed-weight models. Since the proposed method requires neither model training nor access to internal model states, evaluating stronger memory writers is practical and would be important to validate whether this main motivation problem really exists. More fundamentally, the structured numeric claims and dependencies are generated by the same LLM as the natural-language memory and may themselves be unreliable. Table 18 reports model-judge claim-binding precision of approximately 47%, suggesting substantial mismatches between structured claims and their corresponding text. Since the entire consistency-checking framework relies on these structured fields, their unreliability directly undermines the practical validity of the framework, which cannot be established by formal guarantees over the declared structure alone.

  2. The benefit or necessity of dependency retraction over simpler conflict-handling methods is not sufficiently established. Although checked memory improves forecasting over unchecked memory, the comparisons between direct-only retraction (TL) and full dependency retraction (TC7) in Appendices F and J do not establish a clear advantage for the latter. In the later study involving 1,934 riders, their performance differences are also close to zero. Furthermore, TC7 does not demonstrate a clear improvement in offer log loss over simple numeric repair (SR). These results make the additional practical value of constructing dependency graphs and propagating retractions insufficiently demonstrated.

Questions for authors
  1. Could the authors evaluate numeric conflict rates using substantially stronger open- or closed-weight memory writers, potentially with reasoning enabled? If these models produce more accurate memories, does the proposed consistency check still improve downstream forecasting? Or if these models produce nearly zero conflict rates, would the core motivation for the proposed framework still hold?

  2. Given the low claim-binding precision reported in Table 18, how can the authors establish that LLM-generated numeric claims and dependencies faithfully represent the natural-language memory? How do incorrect bindings affect the reliability of conflict detection and retraction?

  3. Given the limited predictive differences between TC7 and TL in Appendices F and J, what empirical evidence supports the additional value of dependency propagation beyond direct conflict retraction?

5 comments
AuthorsOpenReview ↗

Response to Reviewer u97R (1/2)

Thank you for the detailed review. This comment answers your question on what dependency propagation adds. The next one covers write-time errors and binding precision.

To answer it we repeated the forecasting study at a larger scale: 2,208 riders, a seven-day window, and the same writer and forecaster as in the paper (Qwen3.5-9B). The results are in Section 5.4 and Appendix V of the revised PDF (new text in blue).

The full check beats numeric repair. Mean offer log loss by what the forecaster is shown (Table 45):

Memory shownOffer log loss
Plain text0.3157
Wrong numbers replaced by measured ones0.3164
Conflicting statements retracted0.2900
Retraction plus dependency propagation (full check)0.2877

The full check minus numeric repair is −0.0286 [−0.0412, −0.0170] in offer log loss and −0.0050 [−0.0075, −0.0027] in refusal-fraction error.

Propagation versus direct retraction only. The difference in offer log loss is small: −0.0023 [−0.0111, 0.0068]. What propagation adds is consistency. With direct retraction only, conclusions stay on display after the premise they rest on has been withdrawn: 527 such statements in 247 of 295 riders (Section 5.2). Propagation removes them, and the result does not depend on the order of processing (Proposition 1). In the seven-day study it removes 1.55 further statements per rider.

AuthorsOpenReview ↗

Response to Reviewer u97R (2/2)

This second comment answers your questions on write-time errors and on binding precision.

How often is the written memory wrong? In the seven-day study we compared every numeric claim in all 2,609 memories with the table the writer had just read. Of 19,926 claims, 35.7% [35.0, 36.5] were outside tolerance. This is write-time error only, with no drift. At the forecast cutoff the share is 36.8% [35.9, 37.8].

The check helps where there are conflicts. Among the 1,923 riders with a direct conflict, checked minus plain memory is −0.0316 [−0.0466, −0.0169] in offer log loss. Among the 285 riders without one it is −0.0032 [−0.0069, −0.0004] (Section 5.4). The check costs 0.013 ms per memory (Table 25). On your conditional question: even a writer with zero write-time error produces conflicts at the forecast cutoff whenever the rider's drift between the two cutoffs exceeds the tolerance (Proposition 3(iv), Appendix S.2). In the 295-rider study, drift alone exceeds the tolerance in 329 attributed conflicts (Appendix C). Results with stronger writers are in our reply below.

Binding precision of about 47% (Table 18). Table 18 reports binding precision of 47.6% and 47.2% by two model judges. For retraction, what matters is whether the retracted statements are in fact contradicted. Of the statements the check retracts directly, the two judges find 91.1% and 85.8% contradicted, and both call the retraction unnecessary in 4.6% [3.0, 6.4] of 862 cases. Among the statements that stay on display, the share that both judges find contradicted falls from 26.2% to 16.1% with retraction.

Thank you for the additional experiments. However, my main concern in Weakness 1 regarding stronger memory writers has not been directly addressed. I consider this important because the high conflict rate of LLM-written memory is a central motivation and starting point of the work. While the response explains that temporal drift can still introduce conflicts even with a perfectly faithful writer, this is a different source of inconsistency from the substantial write-time errors emphasized in the original submission.

I therefore still believe that evaluating a stronger open-weight writer, or at least enabling reasoning for the current model, is necessary. The restriction on external APIs does not preclude these alternatives. The key missing evidence is whether the conflict rate remains substantial with a stronger writer and, if it becomes much lower, whether the consistency check still provides meaningful benefits.

I also encourage the authors to manually verify and edit any AI-generated text in the response, or provide a more directly human-written response. Both the submission and the rebuttal are difficult to follow, and the reviewer concerns should be addressed more directly and clearly.

Reviewer_V5oHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies indivisual rider forecasting using LLM-written memories, which are stored/summaried from rider's historical records. The authors proposes a measurement-grounded memory interface, which associates numerical claims wiht external definitions and time window and checks these claims against measurements, and retracts contradicted statements and their dependents. Experiments on real rider records evaluate personalization, memory accuracy, and future behavior prediction. And the results show that the checked memory improves forecasts over plain memory on Qwen.

Critical strengths
  1. The paper addresses a reliability problem (LLM-written memory can be wrong and is important to future predictions) with a good formulation. Numerical statements in personal memories could be misleading because the partial observation, wrong numbers, and temporal drift. By explicitly tracking numerator, denominator, unit, definition and window, the checking procedure becomes more interpretable and reproducible .

  2. This paper has practical contributions to the agent memory system, especially handling numerical records.

  3. Evaluation uses real individual outcomes which is convincing.

Critical weaknesses
  1. The checking mechanism depends on numerical claims and the dependency relations generated by the LLM, and these can be incorrect in practice. For example, if undeclared dependencies cannot be detected, then there will be unsupported conclusions in memory. Also, if there are incorrect dependencies, then some helpful statements will be removed.

  2. Checked memory impoves forcasting with Qwen, but in Llama-3.1-8B it worsens the forecasting. This means that the improvements are not consistent across different models. I would be appriciated if the authors can provide an explanation.

  3. It's unclear why we need textual memory in this task. In Section 3.1, forecaster already receives statistics from the rider’s records. Then why we need to generate additional textual memories and check their errors?

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer V5oH (1/2)

Thank you for these questions. This comment answers why the result differs with Llama-3.1-8B. The next one covers wrong dependencies and the value of textual memory.

The gap with Llama appears before any checking. With Llama, checked minus plain memory is 0.1542 [0.1205, 0.1891] in offer log loss. The same gap is already there when the memory is only shown in typed form, with no check applied: typed minus plain is 0.1556 [0.1201, 0.1921]. The check itself changes the loss by −0.0014 [−0.0381, 0.0365].

It comes from few riders. 1,103 of the 1,934 riders have exactly the same loss with checked and with plain memory. The 97 riders (5%) with the largest typed-minus-plain differences carry 99.96% of the mean difference; among the other 1,837 riders the mean is 0.0001 (Appendix V).

Llama writes more conflicting statements and forecasts less accurately. 18,387 of the 42,202 statements in its 2,441 memories conflict directly with the measurement, 7.5 per memory against 2.7 for Qwen3.5-9B. Its mean offer log loss is 0.3629 with plain memory and 0.5170 with checked memory. On the same riders Qwen3.5-9B is between 0.2446 and 0.2708 in every condition (Table 38).

The paper's model on a larger sample. We repeated the study with Qwen3.5-9B on 2,208 riders over seven days. Checked minus plain memory is −0.0280 [−0.0409, −0.0157] in offer log loss, and checked minus typed memory is −0.0291 [−0.0428, −0.0165]. Typed minus plain is 0.0012 [−0.0109, 0.0135], against 0.1556 with Llama.

AuthorsOpenReview ↗

Response to Reviewer V5oH (2/2)

This second comment answers your points on wrong dependencies and on why textual memory is used at all.

Dependencies written by the LLM can be wrong in both directions.

Removing too much. Two model judges both call a retraction unnecessary in 4.6% [3.0, 6.4] of 862 cases (Table 18). In the seven-day study on 2,208 riders, propagating retractions to dependent statements changes offer log loss by −0.0023 [−0.0111, 0.0068] relative to direct retraction only, which is within the 0.020 margin.

Removing too little. The guarantee covers declared dependencies (Section 3.3). 1,150 of 4,363 statements declare a premise, and numeric claims are checked directly whether or not a premise is declared.

Why textual memory when the forecaster already has the statistics. Our claim is about simulators that show LLM-written memory to the forecaster, as the last sentence of the abstract says. In that setting, memory shown as written costs accuracy and the check recovers it. Typed memory as written raises offer log loss over no memory by 0.0303 [0.0178, 0.0448] (Table 46). Checked memory differs from no memory by 0.0012 [−0.0042, 0.0063] (Table N1). Our 9,350-rider experiment is such a setting, with four memory documents per rider.

Reviewer_URRKOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper's fix is a "measurement contract." Every source statistic keeps its numerator, denominator, unit, and window. The LLM's memory is stored as typed, unverified claims with declared dependencies. A deterministic checker only flags a claim when it disagrees with a measurement of the exact same definition (same metric, same window, same denominator population) — so a refusal rate over responses can't fight with a refusal rate over deliveries. Then dependency closure retracts every statement that was built on a flagged claim, truth-maintenance style. They prove the retained memory is the unique largest conflict-free sub-memory, which is cleaner than classic belief-base contraction where multiple maximal subsets survive.

Results: conditioning on the right rider's history beats population context by 0.089 log loss, and swapping in another rider's history costs 15 points of accuracy — so the simulator actually uses this rider's data. The preregistered check on a 22-rider holdout cuts stale-claim adoption from 22.3% to 4.7% and improves both primary endpoints. It replicates on new cutoffs and 789 never-exposed riders. At scale (9,350 riders), combining the LLM forecast with a smoothed personal statistical predictor cuts refusal error 9% vs. the predictor alone.

Critical strengths
  1. The core finding is genuinely important: LLM-written memory gets numbers wrong at write time, not just through drift. 41.6% error rate from a table it's literally looking at. That point is worth publishing.

  2. The "same-definition only" comparison is the right abstraction — it avoids spurious cross-window conflicts and matches how real metrics actually work.

  3. The dependency retraction gives a clean uniqueness theorem, and they actually propagate 527 extra statements beyond the directly conflicting ones.

  4. Methodologically this is unusually careful, preregistered margins, holdout riders outside every earlier cohort, multiple transfer studies, paired bootstraps with Holm correction, outcome-free audit, an independent reader that re-parses everything, and they even caught their own factor-of-7 bug and re-ran. The retraction costs 0.013ms against 81.5s of model time — basically free.

Critical weaknesses
  1. The confirmatory holdout for the check itself is n=22 riders. That's the primary confirmatory result, and the CIs are wide. The bigger studies (789 riders, 9,350 riders) are descriptive, and the 9,350-rider study doesn't even test the memory check — it shows memory verbatim. So the headline "checked memory helps" rides on a tiny sample.
  2. The numeric-repair baseline — just overwrite the conflicting value with the measurement and keep the text — is non-inferior to the full typed+retract pipeline. That raises a real question: why run the whole typed-memory ceremony when slapping the correct number back in does about as well? The retraction of dependent conclusions is the advertised extra, but the gap between TCO (silently remove) and TC (show retraction marker) is small.
  3. The guarantee only covers explicit numeric claims on registered metrics. Qualitative claims ("she prefers evenings") aren't checked, and the model-judge audit finds ~23% of kept statements still look contradicted. Declared dependencies are writer-supplied, so undeclared dependencies are invisible to the closure.
  4. The two big contributions feel somewhat glued together: the memory-check result is on n=22-282 riders, and the impressive 9,350-rider result is about forecast combination (LLM + statistical predictor), not about the check.
Questions for authors
  1. Why is the confirmatory holdout for the check only n=22? You had 789 never-exposed riders and 9,350 in the larger cohort — why not pre-register one of those as the confirmatory test of TC vs P?
  2. Since numeric repair is non-inferior, what does the typed+retract machinery actually buy you over just fixing the numbers? Is it the dependent-conclusion retraction, the markers shown to the forecaster, or something else?
  3. How often does the writer fail to declare a real dependency? The 527 dependency-only retractions are when it did declare them; what's the miss rate?
  4. Does the check generalize across model sizes? If a frontier model writes memory from a table with <5% numeric error, does the check still help?
2 comments
AuthorsOpenReview ↗

Response to Reviewer URRK (1/2)

Thank you for the careful reading. We ran the larger study you asked for and report it here. A second comment covers undeclared dependencies and model size.

Why n = 22, and the same test at scale. When the protocol was fixed, the holdout had all 27 riders outside every earlier cohort, and 22 of them had usable outcomes. To test the same question at scale we repeated the study on all 2,609 eligible riders of the four-city cohort, with a new cutoff, a seven-day window, and the paper's writer and forecaster (Qwen3.5-9B). The comparisons and their margins were chosen before the data were scored. 2,208 riders responded to at least one offer in the window (353,053 responses) and are analysed. Each entry is checked memory minus the other condition:

Checked memory versusOffer log lossRefusal-fraction error
plain memory−0.0280 [−0.0409, −0.0157]−0.0064 [−0.0095, −0.0035]
typed memory as written−0.0291 [−0.0428, −0.0165]−0.0058 [−0.0085, −0.0034]
numeric repair−0.0286 [−0.0412, −0.0170]−0.0050 [−0.0075, −0.0027]
direct retraction only−0.0023 [−0.0111, 0.0068]
no memory0.0012 [−0.0042, 0.0063]

Checked memory lowers offer log loss against plain memory, typed memory and numeric repair; the rider bootstrap and the paired t test agree. It is within the 0.020 margin of direct retraction only and of no memory. The results are unchanged without the 1, 3 and 5 most favorable riders (Appendix V).

What typing and retraction buy over fixing the numbers. Forecasts: mean offer log loss is 0.3164 with numeric repair, 0.3157 with plain memory and 0.2877 with checked memory (Table 45). Text: repair keeps every statement on display. With retraction, the share of displayed statements that two model judges both find contradicted falls from 26.2% to 16.1% (Table 18). Structure: direct retraction only leaves 527 conclusions whose premise was withdrawn, in 247 of 295 riders (Section 5.2). Propagation removes them.

AuthorsOpenReview ↗

Response to Reviewer URRK (2/2)

This second comment answers your questions on undeclared dependencies, on model size, and on the two contributions.

Undeclared dependencies and unchecked text. The closure guarantee covers declared dependencies (Section 3.3); 1,150 of 4,363 statements declare a premise (Section 5.2). Numeric claims are checked whether or not a premise is declared. Two model judges both call a retraction unnecessary in 4.6% [3.0, 6.4] of 862 cases, and among the statements left on display the share that both judges find contradicted falls from 26.2% to 16.1% (Table 18). In the seven-day study on 2,208 riders, propagating over declared dependencies keeps offer log loss within the 0.020 margin of direct retraction only.

Model size and stronger writers. With the paper's writer, 35.7% [35.0, 36.5] of the 19,926 numeric claims in 2,609 memories are outside tolerance against the very table the writer read, and 36.8% [35.9, 37.8] at the forecast cutoff. On your conditional question: even a writer with zero write-time error produces conflicts at the forecast cutoff whenever the rider's drift between the two cutoffs exceeds the tolerance (Proposition 3(iv), Appendix S.2). In the 295-rider study, drift alone exceeds the tolerance in 329 attributed conflicts (Appendix C). All experiments use open-weight models run locally, because the records stay inside a controlled environment.

Two contributions. The seven-day study tests the check in the same cohort as the simulator. On 2,208 riders, checked memory lowers offer log loss against plain memory, typed memory as written and numeric repair.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.