- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to adapt a large language model to one user at inference time.
Its key idea is that a user's preferences may shift from prompt to prompt. So to handle this, EXACT, first picks a few attributes, such as Concise or Formal, for each past prompt. And these attributes are chosen greedily. When a new prompt comes in, EXACT finds the most similar past prompt and adds its attributes to the new prompt.
According to the authors, this simple design is competitive with RAG few-shot on PRISM and Summarize from Human Feedback, in both pairwise accuracy and LLM-judged win rates.
Critical strengths
The paper points out that one user may prefer different styles for different prompts - this is a fair motivation for retrieval at the prompt level.
Critical weaknesses
Before assessing scientific novelty, I must raise serious research integrity concerns about this submission. I believe the authors are committed academic misconduct in this work.
It is very likely that large parts of the paper were produced by AI and then submitted without fixing the problems that the AI itself flagged, several key results cannot come from the method and data that the paper describes, and even the paper itself admits that key results are not verified. I listed evidences below.
important!! - I'm here to ask the other reviewers and PC/ACs to help check these points carefully. - I take research integrity seriously and will provide more evidence I have to the AC and other reviewers for assessment, and am willing to discuss these concerns openly.
Evidence 1. The paper itself directly says that many key parts are drafts and are not verified
These notes appear from the Introduction page1 to the Appendix - No human author would leave such content in a submission.
Main text
- Introduction (line 052): "the draft illustrations in Table 1 await source verification."
- Table 1 caption (lines 074–077): "Draft illustrations of contextual preference shifts (provenance pending)."
- Method (lines 130–131): "draft prompt-format illustrations whose source records and vocabulary mapping remain to be verified."
- Theory (lines 279–281): "Draft theory verification … require correction before this theorem can be used as a guarantee for EXACT."
- Section 4.3, human study (line 435): "study metadata and item-level records remain to be reconciled."
- Section 4.4, ablation (line 449): "The retrieval vs. non-retrieval comparison requires metric verification."
- Section 4.5, case study (lines 458–462): "The draft case record for user thott7Xe … Dataset provenance remains unresolved."
- Ethics statement (lines 500–503): the study procedures, informed consent, compensation, and IRB review "remain to be documented."
- Reproducibility statement (line 519): "Draft tables and cases with unresolved metric or provenance conflicts are marked for reconciliation."
Appendix
- Table 8 (lines 1507–1509): "These draft values require search-log verification."
- Table 9 caption (lines 1525–1527): "Verification is pending: at k = 3, 0.13 conflicts with 0.64 in Table 2."
- Table 15 (lines 1669–1681): "Draft result verification … they have not yet been verified" and "they must be confirmed or recomputed."
- Figures 2–7 (lines 1238–1241): "Figures 2–7 retain draft prompt-format illustrations … remain unverified."
In short, the authors doesn't knew that these results were not checked, and still submitted them. This also triggered my concern that the paper was produced by AI and then submitted without any check.
Evidence 2. Incorrect dataset attribution
I checked the examples against the public source records and found incorrect dataset labels:
- user5 and user9 (Tables 1 and 6) are labeled as Summarize users, but the quoted examples come from PRISM (conversation IDs:
c40,c13,c23, andc43). - RgH765... (Tables 1 and 5) is labeled as a PRISM user, but the examples match Summarize from Human Feedback, where this ID identifies an annotator.
- thott7Xe... is assigned to PRISM in Table 8, although it is also an annotator ID in Summarize from Human Feedback.
These errors raise serious concerns about example provenance. The authors should provide exact source-record mappings and clarify whether the errors also affect the experimental pipeline.
Sources: PRISM, Summarize from Human Feedback.
Evidence 3. The exhaustive search in Table 8 cannot have been run
Table 8 and lines 1504–1506 say that the authors compare greedy selection with exhaustive search at k = 3, 5, and 10. At k = 10, exhaustive search must check C(42,10) which about 1.47 billion subsets, and the paper gives this number itself (line 1508). Each subset needs two forward passes of a 7B model: one for the preferred response and one for the dispreferred response.
The paper says that all experiments run on a single H100 GPU (line 1123). Even at an optimistic speed of 500 subsets per second, one preference pair would take more than a month.
Questions for authors
Clarification requests (each answer could change my assessment):
- Please provide the per-example scores, per-user test counts, and run logs behind Table 2 and what do the ± values in Table 2 measure?
- Help me resolve these conflicts - in particular, please explain how the k=10 search in Table 8 was run on one GPU, which of 0.13 and 0.64 is the correct deviation for Qwen on PRISM, and why Table 15 reports 60.75 for EXACT instead.
- Was the train/test split done by pair or by prompt? How did you get 0% overlap?
- Who were the human raters, and is there an IRB record for the study?
Suggestions for future work: The related work could also discuss these personalization papers:
- Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When Large Language Models Meet Personalization. ACL 2024. arXiv:2304.11406
- Zhaoxuan Tan et al. Can Large Language Models Understand Preferences in Personalized Recommendation? 2025. arXiv:2501.13391
- Zhaoxuan Tan et al. Instant Personalized Large Language Model Adaptation via Hypernetwork. 2025. arXiv:2510.16282
- Zhaoxuan Tan et al. Aligning Large Language Models with Implicit Preferences from User-Generated Content. ACL 2025. arXiv:2506.04463
- Michael J. Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Held, and Diyi Yang. SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs. ACL 2025. arXiv:2506.05598 - It also builds a text-based user profile from a few preference pairs on PRISM and adds it to the prompt, which is very close to EXACT.
- Daiwei Chen, Yi Chen, Aniket Rege, Zhi Wang, and Ramya Korlakai Vinayak. PAL: Sample-Efficient Personalized Reward Modeling for Pluralistic Alignment. ICLR 2025. arXiv:2406.08469 - It studies few-shot personalization on the same Reddit (Summarize from Human Feedback) data.