acceptodds
Under review as a conference paper at ICLR 2027

A Personalized Language Model Should Be Less Surprised by You

Abstract

Personalization in language models is commonly evaluated through task performance, stylistic resemblance, or automatic judgments. These proxies capture useful behaviors, but they do not test whether user-specific conditioning helps a model better anticipate its target user, as reflected in lower surprisal on held-out user text. For a fixed task, tokenizer, and base model, we define predictive personalization as the extent to which a user-specific adaptation reduces the surprisal of text produced by the target user relative to a generic baseline. We operationalize this notion as user-specific predictive alignment and introduce Likelihood Improvement from Targeted Familiarity (LIFT), a per-token log-likelihood-ratio diagnostic that measures the length-normalized surprisal reduction on held-out user text relative to a generic baseline. We also introduce ESS-LIFT as a rare-token robustness analysis that downweights rare, high-leverage tokens to test whether conclusions drawn from LIFT depend on their influence. Since there is no scalar oracle for the latent amount of personalization achieved by an adaptation, we evaluate LIFT through construct-validity stress tests, demonstrating its sensitivity to user identity beyond contextual relevance, robustness to generic fluency and raw-likelihood confounds, and ranking coherence across qualitatively different personalization mechanisms. Experiments across diverse language models, datasets, and tasks provide convergent evidence that LIFT captures user-specific predictive alignment, supporting its use as a principled diagnostic for predictive personalization in language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
predictive personalization, user-specific predictive alignment, personalization evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_D5ogOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LIFT. It is a likelihood-ratio metric for measuring whether personalization makes a language model better predict held-out text from a specific user. It compares the adapted model against the generic baseline on the same user text, and also introduces ESS-LIFT as a robustness check for rare, high-leverage tokens. The paper validates the metric across several models, datasets, and personalization mechanisms.

Critical strengths
  1. I like the problem the paper is is trying to solve. Personalization is often evaluated via task performance or LLM judge, but none of these directly tells us whether the model has actually become better at anticipating a particular user. LIFT gives a simple and fairly clean way to measure this predictive aspect of personalization.

  2. The empirical validation is also quite thorough. The paper tests identity sensitivity, surface-form confounds, and consistency across different personalization mechanisms, and the results are reasonably consistent across several model families and datasets. I also appreciate that the paper is quite explicit about what LIFT does and does not measure.

Critical weaknesses
  1. My main concern is that LIFT measures a fairly specific notion of personalization, i.e., how predictable the user's held-out text becomes under the adapted model. This is useful, but it is clearly different from whether the personalized system is actually more helpful or preferred by the user. The paper acknowledges this distinction, so I see it more as a limitation of scope than a fundamental problem.

  2. I also think the validation is necessarily somewhat indirect because there is no ground-truth scalar measure of "how personalized" a model is. The paper relies on several construct-validity tests instead. These are sensible and fairly convincing, but they still leave some room for interpretation, especially in domains where user identity and topic preference are strongly correlated.

Questions for authors

See above weaknesses.

1 comment
AuthorsOpenReview ↗

Manuscript Revisions

We thank the reviewer again for the helpful feedback and have uploaded the rebuttal revision to reflect the reviewer's suggestions. Specifically, we expanded the discussion of predictive familiarity and its potential relationship to user-facing utility, and explained how controlled comparisons can help interpret the factors underlying LIFT.

Please let us know if any concerns remain or if further clarification would be helpful.

Reviewer_9FPMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes evaluating predictive personalization by measuring how much an adaptation reduces surprisal on held-out text written by the target user. It introduces LIFT, the average per-token log-likelihood ratio between an adapted model and a generic baseline, and ESS-LIFT, a complementary analysis that downweights tokens with low baseline probability.

Critical strengths
  1. The paper investigates a useful evaluation to show whether personalization actually help a model anticipate the target user’s behavior. LIFT provides a simple, interpretable diagnostic that can be computed directly from model probabilities.
  2. The empirical coverage is substantial, spanning several model families and diverse tasks. The same-user versus different-user comparisons and the same-item, same-rating Amazon authorship tests provide useful evidence that the measured signal goes beyond straightforward topical matching.
Critical weaknesses
  1. The added value for the paper’s stated adaptation-comparison objective remains unclear. When the baseline and held-out references are fixed, the baseline term cancels when comparing adaptations. LIFT therefore gives exactly the same ranking as average per-token adapted-model log-likelihood. Its advantage in authorship verification concerns comparisons between different candidate texts, which is a different evaluation problem. Moreover, positive LIFT can arise from general task or domain adaptation that benefits many users. Although the paper acknowledges that LIFT does not identify the source of improvement, stronger evidence is needed to interpret these gains as specifically personal rather than general predictive improvement.
  2. Some central validation experiments leave important confounds unresolved. Both rewrite prompts in Appendix E explicitly remove author-specific expression while changing fluency. Preferring the original text therefore does not isolate robustness to fluency. Similarly, the JIC comparison lacks a control separating general dialogue adaptation from individual-specific learning. This matters because JIC dialogues are generated by Llama 3 70B from journal entries, rather than being directly observed user conversations. Higher likelihood after JIC fine-tuning could partly reflect adaptation to that synthetic distribution. These experiments provide suggestive evidence, but do not fully establish the claimed construct validity.
Questions for authors
  1. For the fixed-reference adaptation comparison defined in Section 3.1, what additional information does LIFT provide beyond held-out per-token log-likelihood, given that the baseline term cancels? Is its main benefit instead baseline-relative interpretation and comparisons across candidate texts?
  2. Is LLaMAdelic a single model fine-tuned across JIC users, or adapted separately for each target user? How were the evaluation dialogues and authors separated from its training data? Can you provide a control that separates general JIC adaptation from gains attributable to the correct user’s information?
  3. Can you separate fluency changes from identity removal in the rewrite experiment—for example, by comparing identity-preserving and identity-removing rewrites at comparable fluency levels? This would help determine whether the current results support fluency robustness or primarily sensitivity to author-specific expression.
1 comment
AuthorsOpenReview ↗

Manuscript Revisions and Rebuttal Summary

We thank the reviewer again for the constructive feedback, and have uploaded a rebuttal revision to address the concerns raised. In the revision, we clarified the evaluation objective and the baseline’s role, added controls to separate fluency from author expression, and clarified the JIC evaluation protocol and how we interpret shared fine-tuning gains.

In the rebuttal, we clarify that baseline subtraction preserves fixed-reference adaptation rankings, while providing user-level interpretation and affecting comparisons between candidate texts. We additionally conduct a controlled rewrite experiment that separates fluency from author-expression preservation, finding an expression-related advantage at both fluency levels among effectively personalized samples. For JIC, we clarify the shared checkpoint and dialogue-level holdout with possible author overlap, and add generation controls supporting conditional predictive benefits from shared adaptation without attributing them to author-specific parameter knowledge.

Please let us know if any concerns remain or if further clarification would be helpful.

Reviewer_NXekOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LIFT, a per-token log-likelihood-ratio diagnostic for predictive personalization. It measures the reduction in length-normalized surprisal on held-out user text from a base model to its user-specific adaptation. The paper also introduces ESS-LIFT to reduce the influence of tokens with low base-model probability. Experiments evaluate these metrics across multiple models, tasks, and personalization methods.

Critical strengths
  1. LIFT is easy to interpret and applicable across different personalization methods.
  2. The experiments are comprehensive, covering six models and seven tasks and including stress tests of user identity, retrieval relevance, and surface form.
Critical weaknesses
  1. Limited novelty of LIFT. The construction of LIFT is straightforward, and similar ideas appeared in prior work that the paper does not discuss:
    • Leveraging Similar Users for Personalized Language Modeling with Limited Data evaluates personalized LMs using Δ\Delta Perplexity relative to baselines on held-out user text, which is closely related to LIFT.
    • Contrastive Decoding: Open-ended Text Generation as Optimization uses the token-level log-probability difference between an expert and an amateur model as its contrastive score.
  2. Weak evidence on the Abstract task. In Figure 2(b), the model-averaged positive rate for LIFT is only 0.52 in both the Same-User Low-Relevance versus Wrong-User High-Relevance and Original versus Polished comparisons. Figure 6 further shows that Qwen3-8B and both Gemma models are at or below 0.5 in these comparisons.
Questions for authors
  1. Given the precedents discussed above, where does the main novelty of LIFT lie?
  2. Which token types are most affected by ESS-LIFT? Do the main conclusions hold when rare tokens like names and technical terms are masked?
  3. Is the LLaMAdelic model used in Table 3 fine-tuned separately for each user? If not, how does the experiment distinguish user-specific personalization from general adaptation to the JIC dialogue domain?
1 comment
AuthorsOpenReview ↗

Manuscript Revisions and Rebuttal Summary

We thank the reviewer again for the thoughtful feedback and have uploaded a rebuttal revision to address the concerns raised. Specifically, we clarified LIFT’s contribution relative to prior work, added the controlled Abstract and token-sensitivity analyses, and clarified the JIC fine-tuning setup and interpretation.

In the rebuttal, we clarify LIFT’s contribution as a measurement framework with controlled validation, explicitly connecting its likelihood contrast to prior work. We conduct additional analyses to strengthen the Abstract evidence conditional on effective few-shot personalization, distinguish baseline surprisal from lexical rarity in ESS-LIFT, and show that the correct-profile advantage persists under token interventions. Finally, we clarify that JIC uses one shared fine-tuned checkpoint, whose predictive gains do not establish author-specific parameter knowledge yet can still be helpful for user-specific prediction.

Please let us know if any concerns remain or if further clarification would be helpful.

Comments on the paper

1 comment

Thank you for the response. My overall assessment of the paper remains positive, and I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.