- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents a method for building memory tests from customer order histories. It uses predefined categories to guide LLMs in generating questions and answers. Automatic checks and LLM judges assess the answers and identify questions that cannot be answered from the history. The authors introduce PantryBench, built from public food delivery data, and also test the method on private customer histories. The experiments measure question diversity across models and cities and compare several memory systems with a baseline that reads the full history.
Critical strengths
None
Critical weaknesses
-
At least four references name people who did not write the cited papers or replace authors’ names with incorrect ones. This repeated pattern raises concerns about correct attribution, how carefully the references were checked, and whether the manuscript was prepared to professional academic standards.
-
The paper uses detailed order histories from 100 real customer accounts but does not identify the approval, consent, or other approved basis for this research use. It also does not clarify whether this covered processing customer information through the reported LLM services. Removing identifiers does not resolve these questions.
-
The reliability of the benchmark is not sufficiently established. LLMs generate and judge the questions and answers, but the authors acknowledge that a systematic human audit is missing. The 253 manually reviewed questions were selected partly to distinguish memory systems, so they do not establish the error rate of the full benchmark. Errors in the expected answers, abstention labels, or grading could affect the reported system comparisons.
Questions for authors
N/A.
2 comments
Thank you for your response.
To clarify, my earlier comment went to the Program Chairs, Area Chairs and Reviewers and is not visible to the authors. Here are the cases so you can fix them in future versions:
Do LLMs Recognize Your Preferences? The manuscript cites: "Siyan Zhao, Mingyi Nguyen, and Aditya Grover."
The correct list of authors is: Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, Kaixiang Lin.
PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments The manuscript cites: "Shuhang Liu, Jiayuan Zhu, Lei Shu, Jialong Lin, Yiheng Chen, Haozhe Zhang, Cheng Zhang, Derong Xu, Jielin Li, Bo Tang, Zhiyu Li, Feiyu Xiong, Enhong Chen, and Tong Xu."
The correct list of authors is: Shuochen Liu, Junyi Zhu, Long Shu, Junda Lin, Yuhao Chen, Haotian Zhang, Chao Zhang, Derong Xu, Jia Li, Bo Tang, Zhiyu Li, Feiyu Xiong, Enhong Chen, Tong Xu
OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents The manuscript cites: "Yuhan Hu, Zhouyang Long, Jiayi Guo, Xin Sui, Xiao Fu, Wenbo Zhao, Yanyan Zhao, and Bing Qin."
The correct list of authors is: Yulin Hu, Zimo Long, Jiahe Guo, Xingyu Sui, Xing Fu, Weixiang Zhao, Yanyan Zhao, Bing Qin
PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory The manuscript cites: Bowen Jiang, Zhuoqun Hao, Young-Min Cho, et al.
The correct list of authors is: Bowen Jiang, Yuan Yuan, Maohao Shen, Zhuoqun Hao, Zhangchen Xu, Zichen Chen, Ziyi Liu, Anvesh Rao Vijjini, Jiashu He, Hanchao Yu, Radha Poovendran, Gregory Wornell, Lyle Ungar, Dan Roth, Sihao Chen, Camillo Jose Taylor
Reviewer 4VCV’s review lists two more cases (PerMemBench and Memora). Please check every reference against its published version, not only these.
On the data: thank you for confirming the use was approved. In the revision, please name the body that approved it (for example, an institutional review, a data governance process, or the platform’s terms) and state that the approval covered processing the data through the external LLM services you report.
Thanks you for pointing out the bibliography errors. The majority of those errors were fixed in the final version. We were aware of it and did multiple rounds of in-depth reviews of references. Unfortunately, when I uploaded the final version together with code, detailed results, and prompts, I failed to check that it was properly uploaded. Instead you got the preliminary version. This is inexcusable and I respect the resulting decision.
"These cases are documented in my earlier comment" - Is this referring to the comment from reviewer 4VCV? In the post, no reference is mentioned.
We agree that the comment on anonymization in the paper is not sufficient. The data was approved for the specific use case, including LLM usage. We will address that in the revision.