acceptodds
Under review as a conference paper at ICLR 2027

From Personalization to Paternalism: Measuring, Locating, and Mitigating Paternalism in LLMs

Abstract

Large language models (LLMs) deployed as personalized assistants are conceptual analogues to recommender systems, raising the question of whether they inherit two well-known profile-driven failures, namely collapse of output diversity and substitution of profile-based content for query-relevant content. We formalize this class of behaviors as LLM Paternalism: the unconsented imposition of the model's inferred judgment about the user on the user's available choices. Paternalism takes two forms. Restrictive paternalism silently removes options the user did not ask to have removed; active paternalism injects profile-driven content the user did not request. We introduce two corresponding deterministic metrics. DivScore measures restrictive paternalism as the relative shrinkage of effective option diversity (via Vendi Score) when a user profile is added. DriftScore measures active paternalism as the fraction of response items that are entailed by the profile but not by the query (via NLI). Both metrics operate on free-form text without a fixed catalog or LLM-as-judge, and reach substantially higher agreement with human annotators and roughly 2× greater reproducibility than LLM-as-judge baselines. We evaluate on our benchmark, a paired with-/without-profile dataset of 2,000 real-user scenarios from Reddit across ten high-stakes domains (e.g., healthcare, finance). Across eight frontier LLMs, paternalism is universal: seven of eight show positive DivScore, all eight show positive DriftScore. To understand where in the model these failures arise, we apply layer-wise linear probing and causal tracing. The two modes localize to distinct internal mechanisms. DivScore corresponds to hidden-state geometry and DriftScore to attention routing, mapping one-to-one onto the two failure modes. Building on these probes, a detection-and-steering intervention reduces DivScore by 30–100% and DriftScore by 35–68% within a 5% perplexity budget.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Personalization, LLMs, Evaluation, Mech Interp

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cyqROpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper defines LLM paternalism as a phenomenon in which personalized LLMs restrict users' options based on their profiles or insert unsolicited content into their responses. To measure this phenomenon, this paper proposes DivScore and DriftScore. Through PaterBench, which consists of 2,000 real Reddit user scenarios, it demonstrates that such behaviors are widespread across existing LLMs and that these phenomena can be detected through the hidden states and attention of LLMs, respectively. Furthermore, this paper also proposes a method that mitigates LLM paternalism without additional training while maintaining response quality.

Critical strengths
  • Concrete metrics and benchmark for addressing the identified problem: To address the paternalism problem raised in this paper, this work (1) proposes metrics and a benchmark to measure paternalism, (2) conducts analyses of existing LLMs, and (3) proposes a method based on these analyses. All three components are clear and well connected, improving the overall quality of this paper. Specifically:
    • Regarding the proposed metrics, the experiments demonstrate that the proposed approach is not only more efficient than simple LLM-based evaluation but also provides a more effective evaluation framework.
    • The analysis experiments and proposed method are clear and effectively provide useful insights not only into paternalism but also into how LLM personalization works in general.
      • Personally, I believe these insights could serve as an important milestone for future advances in LLM personalization.
Critical weaknesses

Personally, I could not identify any major weaknesses in the analyses or proposed method for addressing paternalism. However, I have the following concern regarding how the problem of paternalism is framed in this paper. I would be willing to increase my rating if this concern could be resolved or at least alleviated through discussion.

  • Concern about whether paternalism should be considered together with users' fine-grained preferences: I believe that "restrictive paternalism" and "active paternalism," which are introduced as problems in the Introduction, are essentially about "what information an LLM provides to users and how it delivers that information." In other words, for these behaviors to be considered problematic as excessive intervention beyond "well-calibrated personalization," I believe they should be considered problematic when the LLM performs them despite the user not wanting such behaviors.
    • For example, considering Figure 1, some users may actually prefer to receive only the information they truly need, selectively and quickly, with the main point presented first. In such cases, the response at the bottom of Figure 1 could actually be a more "well-calibrated personalized" response for those users.

      • Conversely, if a user with such preferences were given the response at the top of Figure 1, it might actually be less "well-calibrated" to their preferences, which could be considered a failure of LLM personalization itself.
    • In this context, I believe that to measure paternalism as a problem beyond "well-calibrated personalization," the evaluation should be controlled to measure only cases where the LLM performs such behaviors despite the user having preferences against them. I also wonder whether the analyses should have been conducted specifically on such cases.

      • Otherwise, the analysis results may mix cases where the LLM is actually performing personalization well (i.e., users who prefer more intervention) with cases of unwanted paternalism. This raises the concern that the analysis may not fully reflect paternalism itself.
Questions for authors

Could you share your thoughts on the weakness discussed above, along with supporting evidence or any additional analysis results?

Reviewer_xppVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates paternalism in personalized LLMs, where user profiles may cause models to unnecessarily restrict available options or introduce profile-driven content that is not directly supported by the user's query. The authors propose two metrics: DivScore, which measures the reduction in recommendation diversity using Vendi Score, and DriftScore, which uses embedding similarity and NLI to detect profile-driven content. They construct PaterBench with 2,000 Reddit-derived scenarios across ten domains and evaluate eight LLMs, finding that both behaviors are widespread. The paper further uses linear probing and causal tracing to investigate the internal mechanisms, and proposes an activation-steering intervention to mitigate these behaviors. Experiments report improvements in both metrics, supported by human evaluations and robustness analyses.

Critical strengths
  1. The paper studies an important but relatively underexplored failure mode of personalized LLMs. The distinction between restrictive and active paternalism provides an interesting perspective on how personalization can interfere with user autonomy, particularly in high-stakes decision-making. I appreciate that the authors go beyond conventional personalization performance and examine potential negative consequences.
  2. The paper combines metric development, benchmarking, mechanistic analysis, and mitigation into a unified framework. The evaluation across eight LLMs and ten domains provides a useful empirical picture. I also appreciate the effort to validate the proposed metrics against human annotations rather than relying entirely on LLM-as-judge evaluation.
Critical weaknesses
  1. The proposed metrics do not necessarily distinguish paternalism from legitimate personalization. My main concern is whether DivScore and DriftScore actually capture paternalism rather than general profile-induced changes. A reduction in recommendation diversity is not necessarily undesirable, especially when the user prefers concise or focused advice. Similarly, introducing profile-relevant information does not automatically imply that the model is overriding the user's choices. This distinction is particularly important because some constructed profiles explicitly contain preferences such as wanting a single recommended option (Appendix B.1). Moreover, NLI entailment is not equivalent to query relevance: a useful recommendation generally does not have to be logically entailed by the question. Although the paper discusses these limitations, I am not convinced that the experimental validation sufficiently establishes the claimed connection to paternalism.

  2. Some experimental inconsistencies and mechanistic claims require stronger justification. There are several issues that weaken my confidence in the results. Most notably, Table 6 in Appendix A.4.2 explicitly states that the DivScore LLM-judge comparison uses "synthetic shrinkage values (smoke-test override)," despite being used to support the reproducibility claim. The steering experiments also contain inconsistencies between the reported intervention layers and baseline values across sections. Beyond these reporting concerns, the probing results demonstrate that paternalism-related signals are decodable from hidden states and attention patterns, but this does not necessarily establish two distinct causal mechanisms. In particular, the theoretical argument in Appendix C.3 appears to overlook indirect information propagation through intermediate token positions. I therefore find the mechanistic interpretation stronger than the evidence currently supports.

Questions for authors
  1. How can the proposed metrics distinguish harmful paternalism from beneficial personalization? Could the authors provide human evaluations explicitly assessing whether profile-induced changes unnecessarily restrict user choices, with annotators given access to the relevant profiles?
  2. Table 6 states that the DivScore LLM-judge comparison uses synthetic shrinkage values. Could the authors clarify why synthetic values were used, whether other reported results are affected, and provide the corresponding results based on actual model outputs?
  3. Section 5 describes steering at the best DivScore probing layer, while Appendix B.3 reports steering at different layers (e.g., L55 rather than L49 for Qwen3-32B). Additionally, Appendix A.10 reports a 0% baseline DriftScore for LLaMA-70B, compared with 26.2% in Figure 5. Could the authors clarify the intervention layers, evaluation subsets, and these discrepancies?
  4. Can the authors provide stronger evidence that DivScore and DriftScore correspond to distinct mechanisms, rather than correlated signals across different representations? For example, cross-component probing or targeted causal interventions would help substantiate this claim.
Reviewer_874rOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study the LLM paternalism problem, where personalized assistants may unnecessarily restrict users' choices or introduce profile-driven content into their responses. The authors propose to distinguish restrictive paternalism from active paternalism, and introduce two new metrics: 1) DIVSCORE, based on effective response diversity, and 2) DRIFTSCORE, based on profile-conditioned content drift. Moreover, the authors also construct the PaterBench dataset, which includes 2,000 paired real-user scenarios from Reddit across 10 high-stakes domains. They have performed extensive experiments to evaluate the performance of 8 LLMs, and the experimental results support the proposed metrics. They further investigate profile-conditioning mechanisms using linear probing, causal tracing, and LogitLens, and develop a probe-guided activation steering intervention method to reduce the measured behaviors.

Critical strengths
  1. The paper provides a meaningful perspective on potential risks of personalization by distinguishing option-space contraction from profile-driven content drift.

  2. The proposed DIVSCORE and DRIFTSCORE offer reproducible measurements without relying on generative LLM judges. The empirical validation includes comparisons with human annotations, robustness analyses across metric backbones, and evaluations across eight LLMs.

  3. The authors have performed extensive analysis connecting behavioral measurements with internal representations and mitigation. The paper goes beyond output-level evaluation by investigating hidden-state representations, attention patterns, and causal effects of profile conditioning. The combination of probing, causal tracing, and activation steering provides interesting evidence that profile-related response behaviors can be detected and modified at inference time without model fine-tuning.

  4. This paper is well written and easy to follow.

Critical weaknesses
  1. The proposed metrics may not accurately measure paternalism. DIVSCORE measures reduced response diversity, while DRIFTSCORE measures profile-driven content. However, these behaviors do not always indicate harmful paternalism. They may also reflect useful personalization. The metrics do not directly evaluate whether the model unfairly limits the user choices or acts without the user consent.

  2. The mechanism analysis and mitigation experiments need stronger evidence. The probing results show only moderate prediction accuracy. It is unclear whether the two types of paternalism come from different internal mechanisms. Although activation steering reduces the proposed metrics, the human evaluation includes only 40 examples.

  3. The paper also lacks comparisons with simpler methods, such as prompting, and does not clearly show whether useful personalization is preserved.

Questions for authors
  1. Is there a human evaluation that directly distinguishes harmful paternalism from legitimate personalization, considering user consent, contextual appropriateness, and decision quality?

  2. Whether annotators in the Section 3.2 preference study are shown the user profiles?

  3. How does activation steering compare with strong system-level prompting or selective memory-use controls? Does it preserve beneficial personalization on tasks where user preferences and personal circumstances are relevant?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.