Diversity Is Measured, Not Assumed: Generating Memory Evaluations from Real Order Histories

Personalization is crucial to the agentic commerce experience, and is in turn dependent on consumer memory. Most benchmarks for memory in this space evaluate an agent downstream of memory, confounding memory quality with agent behavior and harness. We present a pipeline to generate consumer memory benchmarks at scale and PantryBench, the resulting benchmark from running the pipeline on the public Delivery Hero Recommendation Dataset (DHRD). Each question is drafted by an LLM constrained by up to five semantic axes (memory type, lookup type, verb, noun, situation). LLM judges then evaluate the question-answer pair against evidence in the consumer's order history. Questions deemed unsupported then become abstention questions to test for hallucination. A deterministic parser check with Stanza, an open-source NLP tool, confirms that each question matches the intent of its fixed semantic axes. Our pipeline makes question diversity measurable as variety (Vendi count over embeddings), coverage (over category pairs within semantic axes), and complexity (lookup type, and number and selection of constrained semantic axes). We find that without constraints from semantic axes, we observe diversity collapse in question-drafting which replicates in both a private cohort and DHRD. On the Stockholm cohort, a Vendi count over sentence embeddings finds 7.6 distinct questions per 40 exported free draws, against 12.2 to 14.8 for constrained questions. Our variety and coverage findings replicate across three LLM families (Claude Opus 5, GPT-5.6, and Gemini 3.5 Flash) and all three cities in DHRD. We release PantryBench, its generation pipeline, and artifacts from Mem0 and Letta evaluation.