acceptodds
Under review as a conference paper at ICLR 2027

Diversity Is Measured, Not Assumed: Generating Memory Evaluations from Real Order Histories

Abstract

Personalization is crucial to the agentic commerce experience, and is in turn dependent on consumer memory. Most benchmarks for memory in this space evaluate an agent downstream of memory, confounding memory quality with agent behavior and harness. We present a pipeline to generate consumer memory benchmarks at scale and PantryBench, the resulting benchmark from running the pipeline on the public Delivery Hero Recommendation Dataset (DHRD). Each question is drafted by an LLM constrained by up to five semantic axes (memory type, lookup type, verb, noun, situation). LLM judges then evaluate the question-answer pair against evidence in the consumer's order history. Questions deemed unsupported then become abstention questions to test for hallucination. A deterministic parser check with Stanza, an open-source NLP tool, confirms that each question matches the intent of its fixed semantic axes. Our pipeline makes question diversity measurable as variety (Vendi count over embeddings), coverage (over category pairs within semantic axes), and complexity (lookup type, and number and selection of constrained semantic axes). We find that without constraints from semantic axes, we observe diversity collapse in question-drafting which replicates in both a private cohort and DHRD. On the Stockholm cohort, a Vendi count over sentence embeddings finds 7.6 distinct questions per 40 exported free draws, against 12.2 to 14.8 for constrained questions. Our variety and coverage findings replicate across three LLM families (Claude Opus 5, GPT-5.6, and Gemini 3.5 Flash) and all three cities in DHRD. We release PantryBench, its generation pipeline, and artifacts from Mem0 and Letta evaluation.

Primary area:
datasets and benchmarks
Keywords:
Agentic memory, Memory evaluation, Large language model agents, Grounded question generation, Benchmark generation, Long-term personalization, Abstention evaluation, E-commerce applications

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mC4ROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a method for building memory tests from customer order histories. It uses predefined categories to guide LLMs in generating questions and answers. Automatic checks and LLM judges assess the answers and identify questions that cannot be answered from the history. The authors introduce PantryBench, built from public food delivery data, and also test the method on private customer histories. The experiments measure question diversity across models and cities and compare several memory systems with a baseline that reads the full history.

Critical strengths

None

Critical weaknesses
  1. At least four references name people who did not write the cited papers or replace authors’ names with incorrect ones. This repeated pattern raises concerns about correct attribution, how carefully the references were checked, and whether the manuscript was prepared to professional academic standards.

  2. The paper uses detailed order histories from 100 real customer accounts but does not identify the approval, consent, or other approved basis for this research use. It also does not clarify whether this covered processing customer information through the reported LLM services. Removing identifiers does not resolve these questions.

  3. The reliability of the benchmark is not sufficiently established. LLMs generate and judge the questions and answers, but the authors acknowledge that a systematic human audit is missing. The 253 manually reviewed questions were selected partly to distinguish memory systems, so they do not establish the error rate of the full benchmark. Errors in the expected answers, abstention labels, or grading could affect the reported system comparisons.

Questions for authors

N/A.

2 comments

Thanks you for pointing out the bibliography errors. The majority of those errors were fixed in the final version. We were aware of it and did multiple rounds of in-depth reviews of references. Unfortunately, when I uploaded the final version together with code, detailed results, and prompts, I failed to check that it was properly uploaded. Instead you got the preliminary version. This is inexcusable and I respect the resulting decision.

"These cases are documented in my earlier comment" - Is this referring to the comment from reviewer 4VCV? In the post, no reference is mentioned.

We agree that the comment on anonymization in the paper is not sufficient. The data was approved for the specific use case, including LLM usage. We will address that in the revision.

Thank you for your response.

To clarify, my earlier comment went to the Program Chairs, Area Chairs and Reviewers and is not visible to the authors. Here are the cases so you can fix them in future versions:

Do LLMs Recognize Your Preferences? The manuscript cites: "Siyan Zhao, Mingyi Nguyen, and Aditya Grover."

The correct list of authors is: Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, Kaixiang Lin.

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments The manuscript cites: "Shuhang Liu, Jiayuan Zhu, Lei Shu, Jialong Lin, Yiheng Chen, Haozhe Zhang, Cheng Zhang, Derong Xu, Jielin Li, Bo Tang, Zhiyu Li, Feiyu Xiong, Enhong Chen, and Tong Xu."

The correct list of authors is: Shuochen Liu, Junyi Zhu, Long Shu, Junda Lin, Yuhao Chen, Haotian Zhang, Chao Zhang, Derong Xu, Jia Li, Bo Tang, Zhiyu Li, Feiyu Xiong, Enhong Chen, Tong Xu

OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents The manuscript cites: "Yuhan Hu, Zhouyang Long, Jiayi Guo, Xin Sui, Xiao Fu, Wenbo Zhao, Yanyan Zhao, and Bing Qin."

The correct list of authors is: Yulin Hu, Zimo Long, Jiahe Guo, Xingyu Sui, Xing Fu, Weixiang Zhao, Yanyan Zhao, Bing Qin

PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory The manuscript cites: Bowen Jiang, Zhuoqun Hao, Young-Min Cho, et al.

The correct list of authors is: Bowen Jiang, Yuan Yuan, Maohao Shen, Zhuoqun Hao, Zhangchen Xu, Zichen Chen, Ziyi Liu, Anvesh Rao Vijjini, Jiashu He, Hanchao Yu, Radha Poovendran, Gregory Wornell, Lyle Ungar, Dan Roth, Sihao Chen, Camillo Jose Taylor

Reviewer 4VCV’s review lists two more cases (PerMemBench and Memora). Please check every reference against its published version, not only these.

On the data: thank you for confirming the use was approved. In the revision, please name the body that approved it (for example, an institutional review, a data governance process, or the platform’s terms) and state that the approval covered processing the data through the external LLM services you report.

Reviewer_aTy4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets evaluation of consumer memory for commerce agents. It aims to test memory directly, not through downstream agent behavior. It proposes a pipeline that samples up to five semantic axes per question (memory type, lookup type, verb, noun, situation), has an LLM draft a question grounded in one account's order history, and filters the result with deterministic checks and two blind LLM judges. Questions the judges find unsupported become abstention questions. The authors measure diversity as variety (Vendi count), coverage (category pairs), and complexity. They report that unconstrained drafting collapses: 7.6 distinct questions per 40 free draws versus 12.2-14.8 for constrained ones. This holds on a private 100-account cohort and on three DHRD cities, with three authoring LLMs. They release PantryBench (built from DHRD) and evaluate raw history, a SQL agent, Letta, and three Mem0 variants (Table 4).

Critical strengths
  • Real, evidence-bound order histories with scored abstention. Most memory benchmarks use synthetic chat. Here questions are grounded in real logged behavior, values are checked against the history, and abstention is a scored class (Table 4). This fills a real gap for commerce memory evaluation.

  • Careful, transparent generation pipeline. The funnel is fully reported (Figure 1, Tables 2, 7, 18). The deterministic numeric audit and dual-judge gating are sensible. Replication across cities and authoring families, plus the public release, makes the work reusable.

Critical weaknesses
  • The headline diversity result uses a weak baseline. The "collapse" compares constrained slots against "free" slots, which are about 6% of specifications (217 sampled, 70 exported on the private run, Table 12). These are drafted in the same per-account batch prompt as the constrained slots (§3). No stronger unconstrained baseline is tested, such as a prompt that asks for diverse questions, persona or category steering as in DataMorgana, or a list of question types. So the result mainly shows that adding constraints adds variety. That is expected and does not show the five-axis grammar beats simpler steering. The Vendi numbers also come from small free-draw pools after filtering, and free draws survive filtering worst (32%).

  • Gold answers lack human validation. All labels come from LLM drafters and LLM judges. The paper says "a systematic blinded audit of gold answers is future work" (Appendix F.6) and that a human audit of axis and answerability labels "remains open" (Appendix D.3). The 253 hand-reviewed questions were filtered for "clarity, consistent grading, and discriminative power" (§4.1), so they do not estimate the error rate. Without a gold-answer accuracy estimate, the absolute scores in Table 4, and especially the abstention labels, are hard to trust. This matters because the paper claims a benchmark.

  • The memory-system evaluation is thin and possibly biased. On PantryBench, Claude Opus 5 drafts the questions, Claude Sonnet 4.5 answers them, and the same Sonnet 4.5 model grades them (§4.1). The claim that within-family bias is small is stated, not tested. Only Letta and Mem0 are tested, each with one ingestion template, and the private systems are anonymized. So the claims that the benchmark "separates memory representations" and can "guide the choice of a memory system" rest on limited evidence.

Questions for authors
  1. Can you add a stronger unconstrained baseline: a prompt that asks explicitly for diverse questions, or one with persona or type steering like DataMorgana? Please give it the same number of draws as the constrained set. Does the Vendi and coverage gap survive?
  2. What is the gold-answer error rate on a random, unfiltered sample checked by blinded human raters? Please report answerable and unanswerable questions separately.
  3. Can you re-score Table 4 with a reader and judge from a family other than the authoring model (e.g., GPT-5.6 for the systems that allow it), so we can see whether the system ranking holds?
  4. Are free and constrained slots for one account drafted in the same LLM call? If so, how might seeing the constrained slots shape what the model writes for the free slots?
1 comment
AuthorsOpenReview ↗

Thank you for the great in-depth review!

Unfortunately, our final upload failed. Otherwise, the exact prompts, code, and results would have been provided.

Regarding diverse questions, we asked to "Draft the requested number of distinct, atomic questions for each specification." and the idea was to ask for all questions at once for each account. DataMorgana is more for RAG and thus requires substantial adjustment for the memory application. Nevertheless, we tried it while integrating our axes. It turned out that the approach resulted in even better Vendi scores. However, when digging deeper, it was more about question length and the variety in how the same question got asked. A major hurdle that stopped us from moving forward was that some of the queries were so cryptic that we could not even infer the intent and correct reply.

  1. We like the idea of prompting for diverse questions separately from the constrained queries. From initial experimentation, we realized that unconstrained prompting did not give us the diversity that we wanted especially in the context of coverage. That's why we created the presented framework around it.

We ran the first of the two suggested baselines and report it below. The gap survives, though it is smaller than our original free-draw control suggested.

We drafted two new arms for the 100-customer Stockholm cohort with the same model (Claude Opus 5), seed, and 4000 draws as the constrained run. In both, every draw leaves all semantic axes open. The first arm uses our plain drafting prompt. The second adds one instruction asking the model to make the questions as diverse as possible, varying what they ask about, how they reason over the history, the time span they cover, and their phrasing. We measured both arms the way we measure the main run: Vendi over embeddings in subsamples of 40, and the number of distinct category pairs realized across the five semantic axes.

Vendi per 40 drafted questions was 10.8 for the plain unconstrained arm and 11.3 for the explicit-diversity arm. The constrained questions of our main run score 12.6 for one constrained axis and 14.6 to 14.8 for three to five, and our original free draws score 7.2. The gap to the highly constrained questions therefore survives. The gap to one-axis questions is within the bootstrap noise of about 1.

The explicit instruction added about 0.5 Vendi, which is inside that noise. Asking for diversity in the prompt does not substitute for constraining the semantic axes.

Coverage differs more. At 3000 questions per arm, the constrained run realizes 2148 distinct category pairs across the axis pairs, against 566 for the plain unconstrained arm and 578 for the explicit-diversity arm. The unconstrained arms rarely produce questions that use predicate, target, or ordering-condition categories, which the axes ask for directly.

The new arms are less collapsed than our original free draws (10.8 and 11.3 against 7.2). We think the reason is drafting context. The original free draws are about two requested questions placed among steered specifications. In the new arms the model drafts all 40 questions of a customer in one call and can see them together. We will state this in the revised text.

  1. We understand the concern as mentioned in the limitations. We are planning to add it in a final version that is accepted by the research community. Depending on the feedback, a modified data generation run might be first required, invalidating an early grading.

  2. Thank you for this request. We have not re-scored Table 4 with a judge or reader from another family, and we do not claim that the ranking is robust to that change. We can offer two measurements related to it.

First, on the private cohort the reader (GPT-5.6) and the grader (Claude Sonnet 4.5) already come from different families. On the public PantryBench study, Claude Sonnet 4.5 both reads and grades, which we state as a limitation.

Second, we tested own-family bias directly in the generation pipeline’s two support judges (Claude Sonnet 5 and GPT-5.6 economy). Across 3,315 and 3,312 judged questions per authoring family (Claude Opus 5 and GPT-5.6), both judges correct Opus-authored answers about 4 points more often than GPT-authored ones. That gap is the same for both judges, so it reflects the authors, not the judges’ families. This covers gold-answer support judging at generation time, not Table 4’s answer grading.

A full regrade would come at a high cost.

  1. This is a correct observation. The free sampling could see the instructions for the other calls and due to self-attention could be impacted by it. However, we would expect this to increase and not decrease the diversity. Previously, we saw that quite a few new proposals came for the free sampling from our flywheel approach. See the reply at 1 for it. Would you recommend always separating free and constrained pulls? How would that best be handled with cases where only a subset of the axes is free?
Reviewer_4VCVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a pipeline to generate tasks that can evaluate memory in commerce agents needing to personalize to user preferences and show that unconstrained question construction does not work and diversity needs to be engineered. Tested on DHRD, collection of food-delivery orders.

Critical strengths

None

Critical weaknesses
  1. The submission contains significant errors in bibliography. This conflicts with research integrity and the submission should not be accepted even with these errors fixed.
  • Yeonjun In, Wonjoong Kim, Sangwoo Park, Kanghoon Yoon, and Chanyoung Park. Personalize-thenstore: Benchmarking and learning personalized memory for long-horizon agents. arXiv preprint arXiv:2605.25535, 2026. URL https://arxiv.org/abs/2605.25535. Introduces the PerMemBench benchmark. -> mistake in one of the names: Sangwu Park not Sangwoo Park according to arxiv
  • Md Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral, and Guangyu Wang. From recall to forgetting: Benchmarking long-term memory for personalized agents. arXiv preprint arXiv:2604.20006, 2026. URL https://arxiv.org/abs/2604.20006. Introduces the Memora benchmark. -> Gengyu Wang instead of Guangyu Wang according to arxiv
  • Siyan Zhao, Mingyi Nguyen, and Aditya Grover. Do LLMs recognize your preferences? Evaluating personalized preference following in LLMs. arXiv preprint arXiv:2502.09597, 2025a. URL https://arxiv.org/abs/2502.09597. -> real list of authors from arxiv: Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, Kaixiang Lin, also could cite ICLR 2025 version
  • Shuhang Liu, Jiayuan Zhu, Lei Shu, Jialong Lin, Yiheng Chen, Haozhe Zhang, Cheng Zhang, Derong Xu, Jielin Li, Bo Tang, Zhiyu Li, Feiyu Xiong, Enhong Chen, and Tong Xu. PERMA: Benchmarking personalized memory agents via event-driven preference and realistic task environments. arXiv preprint arXiv:2603.23231, 2026. URL https://arxiv.org/abs/2603.23231. -> real list of authors according to arxiv: Shuochen Liu, Junyi Zhu, Long Shu, Junda Lin, Yuhao Chen, Haotian Zhang, Chao Zhang, Derong Xu, Jia Li, Bo Tang, Zhiyu Li, Feiyu Xiong, Enhong Chen, Tong Xu
  • Yuhan Hu, Zhouyang Long, Jiayi Guo, Xin Sui, Xiao Fu, Wenbo Zhao, Yanyan Zhao, and Bing Qin. OP-Bench: Benchmarking over-personalization for memory-augmented personalized conversational agents. arXiv preprint arXiv:2601.13722, 2026. URL https://arxiv.org/abs/2601.13722. -> Yulin Hu, Zimo Long, Jiahe Guo, Xingyu Sui, Xing Fu, Weixiang Zhao, Yanyan Zhao, Bing Qin
  • Bowen Jiang, Zhuoqun Hao, Young-Min Cho, et al. PersonaMem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory. arXiv preprint arXiv:2512.06688, 2025b. URL https://arxiv.org/abs/2512.06688. -> Bowen Jiang, Yuan Yuan, Maohao Shen, Zhuoqun Hao, Zhangchen Xu, Zichen Chen, Ziyi Liu, Anvesh Rao Vijjini, Jiashu He, Hanchao Yu, Radha Poovendran, Gregory Wornell, Lyle Ungar, Dan Roth, Sihao Chen, Camillo Jose Taylor
  1. The paper significantly lacks connective details between detailed paragraphs and is difficult to read.
Questions for authors

None

2 comments

Thanks you for pointing out the bibliography errors. The majority of those errors were fixed in the final version. We were aware of it and did multiple rounds of in-depth reviews of references. Unfortunately, when I uploaded the final version together with code, detailed results, and prompts, I failed to check that it was properly uploaded. Instead you got the preliminary version. This is inexcusable and I respect the resulting decision.

Would you mind elaborating on the second point? Especially for the main paper part we avoided the use of AI to make sure, everything is properly connected and readable.

By my second point, I meant that the text is not reader-friendly and makes it difficult to follow the content because high-level information and transitions are missing. Some examples:

  • The framework section comes right after the background and starts with: Evaluation boundary. The memory system under test is a black box. Each memory approach receives the same input, ... without giving the reader a smooth transition, structure for the information and any high-level details, like what are we describing now and why and how does that fit into the narrative of the paper.
  • The entire framework section describes details, but does very little to explain the decisions and connect these with other literature, and that is frequent throughout the paper.
  • The introduction is short and gives little to no background; it starts with the problem this paper is addressing without giving a brief overview of the background and other approaches.
  • Figure 1 is way too small and detailed to be readable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.