- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of completing missing preferences in the context of large model personalization and proposes CP-gated LoRA. Combining collaborative filtering with LoRA, the method first extracts feature representations for each user from the records of the entire user base; it then dynamically adjusts shared LoRA weights based on the current topic and model layer, thereby leveraging information from similar users to make predictions. In terms of experiments, the authors constructed a synthetic dataset named PersonaGrid and conducted tests using multiple open-source models and the MovieLens dataset; the overall performance surpassed baselines that incorporate personal history or summaries into the prompt. Additionally, the effectiveness of the method was validated through experiments involving variations in observation density and the substitution of user representations.
Critical strengths
-
The issue raised by the authors is indeed significant and has often been overlooked; the introduction is well-written and clearly articulates the importance of studying this problem.
-
The problem statement is very clear, explicitly outlining the task setup and evaluation objectives.
-
In addition to the main experiments, the authors designed intervention experiments--such as substituting user representations--that directly validate the actual role user representations play in the prediction process.
Critical weaknesses
-
The baseline comparisons are not entirely fair. While ICL and Memory-ICL rely solely on the current user's own history, the authors' method utilizes data from other users. Consequently, it is unclear whether the performance gains stem from the modification of LoRA weights or simply from the inclusion of external information. The experiments should incorporate baselines that also leverage data from other users—such as collaborative filtering, prompt-based inclusion of similar users' information, or comparisons with methods like CoRA.
-
The experimental setup is overly idealized. The synthetic datasets were generated based on a few fixed prototypes, naturally aligning with the paper's approach of clustering users beforehand. Furthermore, the experiments were limited to closed-ended multiple-choice questions, omitting open-ended text generation tasks. This makes it difficult to determine whether the method remains effective in real-world scenarios involving personalized generation with large models.
Questions for authors
-
How does the proposed method--which modifies LoRA weights--compare to an approach where information from other users (such as records of similar users or collaborative filtering results) is directly incorporated into the prompt? What are the specific advantages of embedding this information into the weights versus simply including it in the prompt?
-
For the tasks evaluated in the paper, what kind of performance could be achieved by using traditional collaborative filtering algorithms directly, without a large language model? Under what circumstances is it necessary to encode collaborative filtering directly into the LoRA weights, rather than using collaborative filtering to calculate missing preferences first and then feeding that information to the large language model?
-
The current experiments are structured as multiple-choice tasks. Would this method remain effective if the model were required to generate free-form, full-paragraph responses or engage in everyday conversation? Could modifying the LoRA weights compromise the model's inherent quality of expression?
-
If a user has previously expressed relevant preferences--or if their tastes differ significantly from the majority of their group--how does the model decide whether to rely on that user's history or the group's aggregate results? How can the model avoid being skewed by the group's mainstream opinion?
4 comments
Follow-up on our response and the revised paper
Dear Reviewer t34Y,
Thank you again for the detailed review. Our response above covers the source of the performance gains (W1), real-world effectiveness (W2), and Q1 to Q4. The revised PDF now includes these results.
We would appreciate hearing whether the response resolves these concerns, and we are happy to provide further clarification during the discussion period.
Thanks to the authors for the response and the updated paper. I have read the rebuttal carefully and see the points the authors are making. Even so, my overall view of the paper stays the same, so I will keep my score.
Could the reviewer point to the remaining concerns?
Thank you for reading the response and the revised paper.
To make the remaining discussion period useful, could the reviewer point to the specific concerns that are still unresolved? The response addressed the source of the performance gains (W1), real-world effectiveness (W2), and Q1 to Q4, each with new experiments now included in the revision. If any of these answers is unconvincing, or if a concern lies outside these items, we would be glad to address it directly.
Thank you for your review, we addressed your questions below
We sincerely thank Reviewer t34Y for recognizing the importance of the problem, the clarity of the Introduction and the representation-intervention evidence. We address the questions below:
W1: Source of the performance gains
Sorry for the confusion about the source of the gains. We clarify that collaborative filtering is the key to the gains, instead of the modification of LoRA weights. Following the reviewer's suggestion, we added baselines that also leverage data from other users.
W2: Effectiveness in real-world scenarios
Q1: Advantages of weights > prompt
As clarified in W1, collaborative filtering is the key, and LoRA weights are one way to use it. Including the collaborative filtering results in the prompt reaches accuracy similar to embedding them into the weights, while including records of similar users in the prompt gives no gain (table in W1).
Embedding the information into the weights takes fewer tokens and needs less downstream data.
Q2: Necessity of the LLM and weights beyond traditional collaborative filtering
Q3: Effectiveness in free-form generation and quality of expression
Q4: Conflict between the user's history and the group's mainstream opinion
The model does not have to choose, because a user's group is made of users with similar histories. By construction, collaborative filtering guarantees that the user's history does not differ too much from the majority of its group. The group is used only for preferences that the history does not cover.
When the user's history disagrees with the group, the model follows the user. With two relevant ratings in the history, our CP-gated LoRA answers according to the user in 93.9% of such cases for seen users and 91.1% for unseen users, so it is not skewed by the group's mainstream opinion.
Please let us know if you have additional questions.