- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes LIFT. It is a likelihood-ratio metric for measuring whether personalization makes a language model better predict held-out text from a specific user. It compares the adapted model against the generic baseline on the same user text, and also introduces ESS-LIFT as a robustness check for rare, high-leverage tokens. The paper validates the metric across several models, datasets, and personalization mechanisms.
Critical strengths
-
I like the problem the paper is is trying to solve. Personalization is often evaluated via task performance or LLM judge, but none of these directly tells us whether the model has actually become better at anticipating a particular user. LIFT gives a simple and fairly clean way to measure this predictive aspect of personalization.
-
The empirical validation is also quite thorough. The paper tests identity sensitivity, surface-form confounds, and consistency across different personalization mechanisms, and the results are reasonably consistent across several model families and datasets. I also appreciate that the paper is quite explicit about what LIFT does and does not measure.
Critical weaknesses
-
My main concern is that LIFT measures a fairly specific notion of personalization, i.e., how predictable the user's held-out text becomes under the adapted model. This is useful, but it is clearly different from whether the personalized system is actually more helpful or preferred by the user. The paper acknowledges this distinction, so I see it more as a limitation of scope than a fundamental problem.
-
I also think the validation is necessarily somewhat indirect because there is no ground-truth scalar measure of "how personalized" a model is. The paper relies on several construct-validity tests instead. These are sensible and fairly convincing, but they still leave some room for interpretation, especially in domains where user identity and topic preference are strongly correlated.
Questions for authors
See above weaknesses.
Manuscript Revisions
We thank the reviewer again for the helpful feedback and have uploaded the rebuttal revision to reflect the reviewer's suggestions. Specifically, we expanded the discussion of predictive familiarity and its potential relationship to user-facing utility, and explained how controlled comparisons can help interpret the factors underlying LIFT.
Please let us know if any concerns remain or if further clarification would be helpful.