acceptodds
Under review as a conference paper at ICLR 2027

Unlocking Latent Personalization in LLMs

Abstract

Large language models (LLMs) are increasingly expected to adapt to individual users, yet effective personalization remains challenging when only limited user-specific samples are available. In this work, we take an alternative perspective: pretrained LLMs may already possess latent capacity for personalization, and a few user samples may therefore suffice to guide the model toward user-aligned behavior with minimal user-specific adaptation. From this perspective, we propose *LatentPersonal*, a framework that formulates personalization as navigation in a shared latent adaptation space. *LatentPersonal* infers a compact latent representation from a few user samples to guide user-specific model adaptation, regularized with a variational information bottleneck to encourage compact preference representations. We instantiate *LatentPersonal* with LoRA, leveraging its low-rank parameterization as a natural low-dimensional adaptation space for personalization. By simply inserting a user-specific guidance vector between the shared low-rank factors, the model can navigate toward personalized adaptations through lightweight inference of this compact representation, without updating the shared LoRA parameters. Experiments across multiple personalization datasets demonstrate that *LatentPersonal* substantially reduces user-specific adaptation overhead while achieving effective personalization from only a few user-specific interactions, with particularly strong performance in the one-shot regime.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Personalization, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FCrQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatentPersonal, a framework for few-shot LLM personalization that learns a shared low-dimensional adaptation space across users. Given a small set of interactions from an unseen user, the method infers a latent representation using a variational preference encoder and uses it to modulate shared PEFT parameters. Its main implementation inserts a user-dependent diagonal matrix between shared LoRA factors, enabling personalization without user-specific gradient updates at inference time.

Critical strengths

The paper addresses a meaningful problem and the proposed LoRA parameterization provides a simple and computationally efficient solution. This design is technically reasonable and provides a useful alternative to generating or optimizing separate adaptation parameters for each user. The empirical evaluation demonstrates that LatentPersonal performs competitively across four personalization benchmarks, with particularly strong results in the one-shot setting.

Critical weaknesses
  1. The proposed approach is closely related to existing methods that generate personalized parameters from user representations. Although using latent coefficients to modulate shared LoRA components is a simple and efficient design, it is not clear what specific benefits this parameterization offers over other shared adaptation mechanisms. The current ablations do not fully address this question, particularly since the w/o Navigation variant also changes the optimization procedure.

  2. It is unclear to what extent the learned user representations capture preferences rather than differences in interaction content. The preference encoder relies on mean-pooled interaction representations, which may also encode user-specific topics or content patterns. Since support and query samples are drawn from the same users, the model could potentially benefit from these correlations without learning stable preferences.

  3. The analysis of the information bottleneck and learned latent space could be more convincing. While removing the IB term leads to a substantial performance drop, it is unclear whether this reflects the intended information compression or simply the benefits of regularization. Similarly, the decreasing latent displacement in Figures 3 and 6 could partly be explained by averaging more interaction representations, rather than by a particular property of the learned latent space.

  4. Some of the reported improvements over Profile-to-PEFT are relatively small, and the gains are less consistent with the second backbone. Paired statistical comparisons would make it easier to assess how reliable these differences are. I also found the evaluation setup somewhat unclear, particularly how personalized reference responses are constructed from PRISM. Since the LLM-based evaluator has access to both user histories and reference responses, it would be useful to clarify how well the resulting scores distinguish preference alignment from reference similarity.

Questions for authors

Please refer to weaknesses

8 comments
AuthorsOpenReview ↗

W1: Clarifying the Benefits of Our Parameterization

We sincerely thank you for your thoughtful and constructive feedback on LatentPersonal. We appreciate your recognition of the value and efficiency of our approach. Your comments have helped us improve the clarity and rigor of our work. We address your concerns in detail below.

W1(a): Distinction from Existing Methods

Among existing methods, Profile-to-PEFT is one of the most closely related to our work, as both follow a similar pipeline: User History→User Representation→Personalized Adaptation.\text{User History} \rightarrow \text{User Representation} \rightarrow \text{Personalized Adaptation}.

However, their adaptation mechanisms are fundamentally different. Profile-to-PEFT adopts parameter generation, using a hypernetwork to generate personalized LoRA parameters. In other words, it uses one neural network to generate the parameters of another neural network. In contrast, LatentPersonal adopts parameter composition within a shared low-dimensional adaptation space:

ΔWu=Bdiag⁡(zu)A,\Delta W_u = B\operatorname{diag}(z_u)A,

where zuz_u determines their combination weights. This distinction leads to different parameter spaces and inductive biases: rather than generating user-specific LoRA factors, LatentPersonal combines shared adaptation directions through a compact latent representation.

We have also discussed this distinction in the "Personalized Adaptation of LLMs" subsection of our Related Work.

W1(b): Advantages of Parameter Composition

Regarding the specific benefits of our parameterization, as clarified in W1(a), LatentPersonal performs personalization through composition rather than parameter generation. This design offers two main advantages: (1) Parameter efficiency: only a compact latent vector is required to personalize shared LoRA factors, avoiding the generation of user-specific parameter matrices. (2) Few-shot generalization: constraining user-specific adaptations to a shared low-dimensional space, reducing the degrees of freedom for user-specific adaptation and enabling knowledge sharing across users, thereby reducing overfitting risk in low-interaction settings.

W1(c): Clarification and Extension of Ablation Studies

Our original intention of w/o Navigation was to examine the role of user-specific latent representations in personalization. Specifically, w/o Latent removes the latent coefficients, while w/o Navigation retains the inferred coefficients but fixes them and optimizes the LoRA factors instead. However, since w/o Navigation also changes the optimization procedure, this design may have caused confusion about its purpose and does not fully address the reviewer's concern regarding the benefits of our parameterization.

To address this concern while preserving our original experimental objective, we introduce a Joint Optimization variant that updates both the latent coefficients and LoRA factors during personalization. This allows us to further examine whether additional parameter optimization improves latent-guided personalization, providing a more comprehensive comparison of the different adaptation strategies.

Variant1-shot R-1MET.CP2-shot R-1MET.CP4-shot R-1MET.CP
LatentPersonal0.41620.41335.950.41600.41425.920.41560.41375.94
w/o Latent0.35500.36674.720.34990.38414.860.33080.34235.29
w/o Navigation0.38120.37775.660.39100.39295.710.39090.39925.92
w/o IB0.26840.34234.570.27030.34224.650.26950.34444.74
Joint Optimization0.32500.35084.920.36990.37255.260.33070.37575.19

The results show that Joint Optimization consistently underperforms LatentPersonal across all evaluated shot settings and metrics. This suggests that jointly optimizing the latent coefficients and LoRA factors does not necessarily improve few-shot personalization. Together with the w/o Navigation results, these comparisons provide additional evidence that personalization through inferred latent coefficients over fixed shared adaptation directions can be more effective than direct user-specific parameter optimization under limited interactions.

Overall, the key advantage of LatentPersonal lies in its shared low-dimensional adaptation space, which enables efficient few-shot personalization through compact latent coefficients rather than generating or optimizing user-specific LoRA parameters. The methodological distinction, empirical performance, and additional ablation results collectively support the effectiveness of this parameterization, particularly when user interactions are limited.

AuthorsOpenReview ↗

W2: Preference Learning and Content Correlations

W2(a): Distinction Between Preferences and Interaction Content

User preferences and interaction content are not mutually exclusive in LLM personalization.

For example, a user's long-standing interest in artificial intelligence topics can itself constitute a topic preference. User preferences encompass not only tone, formatting, and response length, but also content selection, topic interests, and information emphasis. Previous studies [1, 2] have also recognized topical interests and content preferences as important aspects of LLM personalization.

Consistent with this perspective, our Content Personalization (CP) metric explicitly considers topics, examples, information selection, and emphasis. Therefore, our objective is not to eliminate content information from zuz_u, but to preserve information relevant to personalized adaptation while discarding irrelevant interaction-specific details. This is consistent with the information bottleneck formulation in Section 3.2, which encourages the learned representation to retain preference-relevant information that generalizes across interactions.

[1] Personalized Benchmarking: Evaluating LLMs by Individual Preferences. Findings of ACL, 2026.

[2] Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas. ACL, 2025.

W2(b): Role of Mean Pooling in Preference Learning

We acknowledge that mean-pooled interaction representations may contain topical and content information. However, mean pooling serves only as a feature aggregation step, rather than directly determining the learned user preferences.

Specifically, the aggregated features are processed by a learnable variational preference encoder, whose optimization is governed by two complementary objectives (Eq. 12):

Ltotal=Eu[Ezu∼qψ(zu∣Dusup)L(Duqry;θ0+gϕ(zu))+βDKL(qψ(zu∣Dusup)∥r(z))]\mathcal{L}_{\mathrm{total}}=\mathbb{E}_u\left[\mathbb{E}_{z_u\sim q_\psi(z_u\mid D_u^{\mathrm{sup}})}\mathcal{L}(D_u^{\mathrm{qry}};\theta_0+g_\phi(z_u))+\beta D_{\mathrm{KL}}(q_\psi(z_u\mid D_u^{\mathrm{sup}})\parallel r(z))\right]

The adaptation loss encourages zuz_u to preserve information useful for personalization on held-out query interactions, while the compression term discourages encoding unnecessary interaction-specific information. Together, these objectives encourage zuz_u to capture information relevant to personalization across interactions, rather than indiscriminately preserving all information contained in the mean-pooled representations.

W2(c): Support–Query Correlations

PIR and PersonaMem-v2 contain diverse interaction content associated with user-specific preferences.

For example, PIR includes questions on distinct topics, such as cooking, weight management, and general knowledge, under the same persona. Similarly, PersonaMem-v2 incorporates different interaction scenarios, including email editing, translation, and question answering, while maintaining user-level preference information.

These dataset designs allow us to evaluate personalization across diverse interaction contexts rather than only repeated or similar topics. Furthermore, training and test users are strictly disjoint, and support and query interactions do not overlap.

Additional evidence comes from our User-Specificity Analysis (Figure 4), where matched user representations consistently outperform swapped and population-mean representations under identical evaluation queries. This demonstrates that the inferred latent representations contain user-specific information that contributes to personalization, rather than merely providing generic adaptation.

Overall, our objective is to learn compact representations that preserve information useful for user-specific adaptation, including content-related preferences where relevant. The support–query objective, diverse evaluation scenarios, and user-specificity analysis collectively provide evidence that LatentPersonal learns representations useful for personalization beyond simple interaction matching.

AuthorsOpenReview ↗

W3: Information Bottleneck and Latent Space Analysis

W3(a): Information Bottleneck vs. Regularization

We would like to clarify that information bottleneck (IB) and regularization are not mutually exclusive. In fact, IB is implemented through a specific form of regularization that encourages information compression.

Specifically, our KL regularization term provides a variational upper bound on the mutual information between user interactions and the learned latent representation, as derived in Appendix A.1:

I(zu;Dusup)≤Eu[DKL(qψ(zu∣Dusup)∥r(z))].I(z_u;D_u^{\mathrm{sup}})\leq\mathbb{E}_u\left[D_{\mathrm{KL}}\left(q_\psi(z_u\mid D_u^{\mathrm{sup}})\parallel r(z)\right)\right].

Therefore, unlike generic regularization, our IB objective explicitly constrains the information retained from user interactions, encouraging the encoder to preserve information relevant to personalization while suppressing unnecessary interaction-specific details.

To further illustrate the specific role of the IB constraint, we additionally introduce an L2 regularization baseline by replacing the KL term with LL2=Eu[∥μψ(Dusup)∥22]\mathcal{L}_{\mathrm{L2}} = \mathbb{E}_u[ \left\|\mu_\psi(D_u^{\mathrm{sup}})\right\|_2^2], while keeping the remaining model architecture and training procedure unchanged. The comparison on HiCUPID is presented below.

Variant1-shot R-1MET.CP2-shot R-1MET.CP4-shot R-1MET.CP
LatentPersonal0.41620.41335.950.41600.41425.920.41560.41375.94
No Regularization0.26840.34234.570.27030.34224.650.26950.34444.74
L2 Regularization0.35600.38265.340.35300.38995.380.35340.38115.46

The results show that L2 regularization improves performance over the no-regularization baseline, but consistently underperforms the full IB objective across all evaluated shot settings and metrics. This suggests that the benefits of IB extend beyond those of conventional weight regularization. Unlike L2, which primarily constrains the magnitude of latent representations, the variational IB objective additionally regularizes the posterior distribution and provides an explicit upper bound on the information retained from user interactions. These findings support the effectiveness of information-bottleneck regularization in learning compact representations for few-shot personalization.

W3(b): Latent Displacement vs. Mean Pooling

We would like to clarify that the decreasing latent displacement in Figures 3 and 6 is intended to illustrate the progressive stabilization of user representations as additional interactions become available, rather than to serve as the sole evidence of the learned latent space's properties.

To further distinguish the effect of mean pooling from that of the learned preference encoder, we introduce an additional comparison between the mean-pooled interaction representations and the learned latent representations. Specifically, for each user, we measure the displacement between consecutive shot settings (1→2, 2→4, and 4→8):

d_K^{z}=|z_u^{(2K)}-z_u^{(K)}|_2.$$ where $K \in {1,2,4}$ denotes the number of observed user interactions. Since the two representations have different dimensions and scales, we normalize their displacements by the corresponding 1→2-shot displacement to compare their relative stabilization trends. |Dataset| Representation|1→2|2→4|4→8| |---|---|---|---|---| |HiCUPID|Mean Pooling ($h_u$)|1.00|0.81|0.69| ||Learned Latent ($z_u$)|1.00|0.67|0.46| |PRISM|Mean Pooling ($h_u$)|1.00|0.89|0.71| ||Learned Latent ($z_u$)|1.00|0.74|0.50| |PersonaMem | Mean Pooling ($h_u$)|1.00|0.79|0.68| ||Learned Latent ($z_u$)|1.00|0.70|0.50| |PIR|Mean Pooling ($h_u$)|1.00|0.79|0.68| ||Learned Latent ($z_u$)|1.000|0.65|0.53| The results show that both mean-pooled and learned latent representations exhibit decreasing displacement as additional interactions become available. However, the learned latent representations consistently demonstrate a more pronounced relative reduction across all four datasets. This suggests that the observed stabilization is not solely attributable to the statistical effect of mean pooling, but also reflects the influence of the learned preference encoder. Together with the user-specificity and interpolation analyses in Figure 4, these findings provide complementary evidence that the learned latent space supports both stable representations and meaningful user-specific adaptation.
Reviewer_qSHAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LatentPersonal, a framework for few-shot personalization of large language models. The central idea is to formulate personalization as navigation in a shared low-dimensional latent adaptation space, rather than learning an independent personalized model from scratch for each user. The method infers a compact user-specific representation from a small number of observed interactions and uses this representation to modulate a shared parameter-efficient adaptation structure. The paper primarily instantiates this idea using LoRA. Instead of learning a separate LoRA module for every user, LatentPersonal learns a shared set of low-rank adaptation directions across users and uses the inferred user representation to determine how strongly each direction contributes to the final personalized adaptation. At inference time, the shared parameters remain fixed, and personalization only requires inferring the user-specific latent representation from a few interactions, without additional user-specific optimization. The framework is trained using an episodic support-query procedure. A subset of a user’s interactions is used to infer the user representation, while held-out interactions from the same user provide the supervision for personalized adaptation. The paper further introduces an information-bottleneck regularization term intended to encourage the user representation to retain preference-relevant information while discarding interaction-specific details. Experiments are conducted on HiCUPID, PRISM, PersonaMem, and PIR, with ICL, user-specific LoRA, PPlug, and Profile-to-PEFT as the main baselines. The results suggest that LatentPersonal is particularly competitive in low-shot settings, although the advantage is not uniform across all datasets, metrics, and numbers of observed interactions.

Critical strengths
  1. The method provides a simple and practically appealing approach to amortized personalization. LatentPersonal avoids learning or storing a separate personalized adaptation module for each user. Instead, it learns a shared adaptation structure and only infers a compact user-specific representation at test time. This makes the framework attractive for large-scale personalization, especially when only a small amount of evidence is available for a new user. The reported efficiency results also support this motivation. Test-time personalization is substantially faster than gradient-based user-specific LoRA adaptation in the low-shot regime, while avoiding per-user optimization.
  2. The experimental study is reasonably broad and includes meaningful ablations. The paper evaluates on four personalization benchmarks and across several few-shot regimes. It also compares against both prompting-based and adaptation-based baselines. The ablation study shows substantial degradation when either the user-specific latent representation or the information-bottleneck regularization is removed, suggesting that the proposed components are contributing meaningfully to performance. The paper further evaluates an additional backbone and alternative PEFT architectures in the appendix, which helps demonstrate that the method is not entirely tied to one particular model or LoRA implementation.
Critical weaknesses
  1. The central claim that the method is “unlocking” personalization already latent in the pretrained LLM is not convincingly established. The paper motivates LatentPersonal by arguing that pretrained LLMs may already contain personalization-relevant behaviors and that a few user interactions are sufficient to navigate toward these behaviors. The introduction explicitly contrasts this with learning personalized behavior from scratch. However, the actual LatentPersonal system learns both the user preference encoder and the shared adaptation structure from a population of training users. Therefore, strong few-shot generalization to unseen users does not by itself show that the personalization structure was already present in the pretrained model. An equally plausible explanation is that the population-level personalization training learns a low-dimensional user adaptation space, after which the encoder learns to infer a position in that space for a new user. This distinction matters because it is central to the paper’s conceptual framing and novelty claim. In its current form, the empirical evidence more directly supports the conclusion that few-shot personalization can be amortized through a learned shared adaptation space. I do not think the experiments clearly support the stronger conclusion that LatentPersonal is uncovering personalization capabilities that were already latent in the original pretrained model. I would consider this concern substantially alleviated if the authors either narrow the framing throughout the paper or provide a controlled experiment that explicitly separates structure inherited from pretraining from structure acquired through population-level personalization training.

  2. The empirical advantage is not as consistent as the presentation suggests, and the personalization evaluation would benefit from stronger validation. The main results are promising, especially in the low-shot regime, but LatentPersonal is not uniformly superior across all settings. For example, on some dataset, metric, and shot combinations, conventional LoRA or Profile-to-PEFT performs similarly or better. This makes the language of “consistent advantages” somewhat stronger than what I would infer from the tables. The strongest empirical story appears to be that LatentPersonal is particularly competitive in very low-shot settings, rather than uniformly outperforming competing methods. I am also concerned about the reliance on a single LLM-based evaluator for the main personalization-oriented metrics. The evaluator receives the user history, current input, reference response, and generated response, and assigns personalization scores using the same judge model across all methods. Using the same evaluator consistently across methods is reasonable, but because personalization quality is a central claim of the paper, I would like stronger evidence that these scores reliably measure user-specific alignment. Human evaluation on a representative subset, evaluation with multiple independent judges, or a calibration study against known matched and mismatched user responses would substantially strengthen the evidence. Additionally, several of the differences between strong methods are relatively small. Reporting confidence intervals is helpful, but paired significance testing across the same test users would make it much easier to assess whether these differences represent robust improvements.

Questions for authors
  1. Clarification / potentially score-changing: Can the authors provide an experiment that separates personalization structure inherited from the pretrained LLM from personalization structure learned during population-level LatentPersonal training? In the current setup, both the preference encoder and the shared adaptation space are trained using multiple users. Therefore, it is unclear whether the successful few-shot personalization reflects latent capabilities already present in the pretrained model or structure learned during personalization training.

  2. Clarification / potentially score-changing: Are the improvements over the strongest baseline statistically significant under a paired user-level comparison? In particular, can the authors report paired confidence intervals or statistical tests comparing LatentPersonal against the strongest competing method under the same test users and support-set selections?

3 comments
AuthorsOpenReview ↗

W1 & Q1: Clarifying the Unlocking Perspective

We thank you for your thoughtful and constructive review, and for recognizing the simplicity, efficiency, and strong few-shot performance of LatentPersonal. We have carefully considered your concerns regarding our core motivation and the reliability of our empirical evaluation. Below, we address these concerns through further clarification, additional experiments, and corresponding manuscript revisions.

W1(a): Clarification of Our Core Motivation

Before addressing your concern, we would like to clarify the core motivation behind LatentPersonal. Our central intuition is that pretrained LLMs may already possess personalization-relevant behavioral capabilities that can be elicited with limited user-specific evidence, rather than learning these behaviors from scratch for each user.

From a parameter perspective, LatentPersonal formulates personalized adaptation as Wu=W0+Bdiag⁡(zu)AW_u = W_0 + B\operatorname{diag}(z_u)A. We understand your concern that the learned adaptation term Bdiag⁡(zu)AB\operatorname{diag}(z_u)A may also account for part of the observed personalization gains. We agree that population-level training contributes to personalization. However, this does not contradict our core motivation. We view the learned adaptation mechanism as a means of leveraging and eliciting personalization-relevant capabilities of pretrained LLMs. In this sense, "unlocking" refers to enabling these capabilities through learned, user-conditioned adaptation, rather than assuming that the specific adaptation structure already exists in the pretrained model.

W1(b) & Q1: Disentangling Pretraining and Population-Level Training

To further address your question regarding the respective contributions of pretrained capabilities and population-level personalization training, we conduct an additional controlled experiment.

Experimental setup: We compare four settings: (1) Base LLM, the pretrained model without adaptation; (2) General LoRA, trained on general-purpose data using standard LoRA optimization; (3) LatentPersonal (General LoRA), which first trains LoRA factors A,BA,B on general-purpose data, then freezes them and trains the preference encoder on the personalization benchmark; and (4) LatentPersonal, our original jointly trained framework. All settings use the same pretrained backbone and are evaluated on HiCUPID with identical held-out test users and query samples. Further implementation details are provided in Appendix D.8.

Results:

Variant1-shot R-1MET.CP2-shot R-1MET.CP4-shot R-1MET.CP
Base LLM0.16140.21663.570.16140.21663.570.16140.21663.57
General LoRA0.16810.21533.810.16810.21533.810.16810.21533.81
LatentPersonal (General LoRA)0.38420.35615.410.39390.35835.580.39870.35905.75
LatentPersonal0.41620.41335.950.41600.41425.920.41560.41375.94

Analysis: LatentPersonal (General LoRA) substantially outperforms the Base LLM and achieves performance close to the original LatentPersonal, even without personalization-specific training of A,BA,B. This demonstrates that learning a shared adaptation space from the target user population is not necessary for effective personalization.

W1(c): Clarification of Our Claim and Manuscript Revisions

We acknowledge that our original framing did not sufficiently distinguish between personalization-relevant capabilities of pretrained LLMs and the adaptation structure learned through population-level training. We will clarify this distinction while retaining unlocking as our core motivation.

Revised claim: Pretrained LLMs may already possess personalization-relevant behaviors that can be elicited with limited user-specific evidence. LatentPersonal learns a shared adaptation mechanism to leverage these capabilities, enabling effective few-shot personalization through user-specific latent navigation.

We have revised the relevant statements throughout the manuscript accordingly.

AuthorsOpenReview ↗

W2 & Q2: Empirical Advantages and Evaluation Reliability

W2(a): Clarification of Performance Advantages

We appreciate your recognition of LatentPersonal's strong performance with limited user interactions. In particular, LatentPersonal achieves the best results across all four datasets and three reported metrics in the one-shot setting. We agree that a few statements in the original manuscript may have overstated the consistency of these advantages. Accordingly, we have refined these statements to emphasize our strong performance in low-shot settings rather than uniform superiority across all settings.

W2(b): Reliability of Personalization Evaluation

To further validate the reliability of our personalization evaluation, we conduct two complementary assessments.

Independent LLM Judge: We employ Qwen3-30B-A3B-Instruct-2507 as an independent evaluator, using the same test samples and CP/PP evaluation criteria as GPT-4.1-mini.

Results (HiCUPID, 1-shot):

EvaluatorMethodCPPP
GPT-4.1-miniProfile-to-PEFT5.646.02
GPT-4.1-miniLatentPersonal5.956.33
Qwen3-30B-A3B-Instruct-2507Profile-to-PEFT5.786.29
Qwen3-30B-A3B-Instruct-2507LatentPersonal6.016.52

The independent Qwen evaluator produces higher absolute scores but preserves the same relative ranking, with LatentPersonal outperforming Profile-to-PEFT on both CP (6.01 vs. 5.78) and PP (6.52 vs. 6.29). This suggests that the observed personalization advantages are not specific to the original GPT-4.1-mini evaluator.

Human Evaluation: We conduct a blind pairwise evaluation comparing LatentPersonal with Profile-to-PEFT in the 1-shot setting. Five independent annotators evaluate 200 test samples, selecting the response that better aligns with user preferences based on the user history and query. Method identities are hidden, response order is randomized, and ties are allowed.

Results:

DatasetLatentPersonal Win (%)Tie (%)Loss (%)
HiCUPID56.414.229.4
PRISM69.05.925.1
PersonaMem54.112.433.5
PIR49.714.935.4

LatentPersonal achieves higher win than loss rates across all four datasets, with the largest advantage on PRISM (69.0% vs. 25.1%). Since annotators evaluate responses without access to reference responses, these findings provide independent evidence of preference alignment beyond reference-based similarity.

Together, the independent LLM evaluation and blind human evaluation provide complementary evidence supporting the reliability of our personalization assessment. Further details of the human evaluation are provided in Appendix D.9.

W2(c) & Q2: Paired Statistical Significance

We conduct paired user-level significance tests comparing LatentPersonal with the strongest competing baseline under identical test users and support-set selections. We report mean performance differences (Δ\Delta), 95% paired bootstrap confidence intervals, and p-values from paired permutation tests.

Results (1-shot):

DatasetMetricΔ\Delta95% CIp-value
HiCUPIDR-1+0.0147[0.0019, 0.0275]0.0453
HiCUPIDMET.+0.0111[-0.0028, 0.0286]0.0748
HiCUPIDCP+0.31[0.109, 0.481]0.0008
PRISMR-1+0.0152[0.0081, 0.0253]0.0031
PRISMMET.+0.0049[-0.0019, 0.0167]0.2727
PRISMCP+0.13[-0.069, 0.310]0.0506
PersonaMemR-1+0.0086[0.0002, 0.0188]0.0459
PersonaMemMET.+0.0036[-0.0011, 0.0143]0.2188
PersonaMemCP+0.17[0.004, 0.348]0.0473
PIRR-1+0.0041[-0.0027, 0.0139]0.1364
PIRMET.+0.0021[0.0002, 0.0064]0.0402
PIRCP+0.10[-0.023, 0.233]0.0991

LatentPersonal achieves positive performance differences against the strongest competing baselines across all 12 comparisons, with six reaching statistical significance at the unadjusted p<0.05p<0.05 level. Particularly strong evidence is observed for HiCUPID CP and PRISM R-1, which remain significant after Holm correction for multiple comparisons. For the remaining settings, the improvements are generally smaller and do not consistently reach statistical significance. These findings provide statistical support for the effectiveness of LatentPersonal in selected settings, while also highlighting that its empirical advantages are not uniformly significant across all datasets and metrics. Additional results and implementation details are provided in Appendix D.10.

We hope that these clarifications, additional experiments, and manuscript revisions adequately address your concerns, and that you will reconsider your assessment of our work.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer qSHA,

Thank you again for your valuable feedback. We have provided detailed responses to your concerns in our rebuttal.

As the discussion period nears its end, we would greatly appreciate your feedback on whether our responses have adequately addressed your concerns or if any questions remain.

Thank you very much for your time and consideration.

Best regards, The Authors

Reviewer_UZCLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a framework called LatentPersonal for personalizing LLMs with few samples. The method represents user preferences in a shared low-dimensional latent space. The preference encoder infers a compact user representation from a small amount of historical interaction data and utilizes an information bottleneck mechanism to filter out irrelevant information. The shared LoRA components are fine-tuned using the inferred representations, enabling personalised adaptation during testing without the need to update user-specific parameters. Experiments across four personalisation benchmarks demonstrate that this method performs well.

Critical strengths
  1. This paper explores the issue of personalisation in LLMs from an novel perspective. It models personalisation as the search for an appropriate combination of shared adaptation directions. This approach is quite reasonable and provides a clear theoretical basis for the proposed framework.
  2. This method is simple and efficient. By inserting user-specific latent vectors between shared LoRA factors, the model can generate personalised parameter updates without the need to train the LoRA module separately for each user. The information bottleneck design is also a sound approach for learning compact preference representations from a limited number of interactions. This method can also be extended to other PEFT architectures.
  3. The experiments covered four personalised datasets and varying numbers of user interactions. The method achieved excellent results, particularly in single-learning scenarios. Ablation experiments validated the effectiveness of the key components.
Critical weaknesses
  1. The motivation experiment shown in Figure 1(a) is not entirely convincing. The authors used an artificially constructed preference prompt designed to demonstrate that a frozen LLM can generate personalised responses without the need to update its parameters. However, the prompt itself already provided explicit information about the user’s preferences. This experiment primarily demonstrates that pre-trained models are capable of following a given description of preferences. This does not directly demonstrate that the model is capable of identifying such preferences from a limited number of initial interactions, nor does it prove that these preferences can be represented in a shared low-dimensional adaptation space.
  2. The preference encoder first performs mean pooling on the token representations, followed by another round of mean pooling on the historical interactions. This approach may be too simplistic for extracting user preferences from a diverse history of interactions. Different tokens and interactions may contain varying amounts of preference-relevant information, whilst others may primarily reflect task content or other irrelevant details. Treating them all equally may weaken the resulting user representation.
  3. The LoRA-based implementation uses the same user-specific latent vectors to modulate the shared adaptation components across different layers. Although each layer has its own LoRA factors, using the same modulation coefficients across layers may limit the flexibility of personalised adaptation. Different user preferences may require different adjustments at different layers.
Questions for authors
  1. How does the experiment in Figure 1(a) support the hypothesis that personalised behaviours can be identified and activated from limited user interactions, rather than simply demonstrating the model's ability to follow explicit preference instructions?
  2. Have the authors considered alternative aggregation methods, such as attention-based or weighted pooling, for extracting user preferences from historical interactions? How does the current mean pooling approach handle interactions that contain little preference-relevant information?
  3. What is the motivation for using the same user-specific latent vector across all LoRA layers? Have the authors considered layer-specific modulation coefficients, and how might this affect the flexibility and performance of personalised adaptation?
5 comments
AuthorsOpenReview ↗

W1 & Q1; W2 & Q2: Motivation and Preference Learning

Thank you for your positive assessment of LatentPersonal, particularly your recognition of its latent navigation perspective, methodological simplicity, and strong few-shot personalization performance. We also appreciate your constructive suggestions, which have helped us clarify our motivation and strengthen the experimental validation of our design choices. We address your concerns in detail below.

W1(a): Clarification of the Purpose of Figure 1(a)

Figure 1(a) serves as a motivation experiment, illustrating that pretrained LLMs can exhibit personalization-relevant behaviors under appropriate preference guidance without parameter updates. Although the human-constructed prompts explicitly provide user preferences, the resulting preference-aligned responses suggest that such behaviors can be elicited from the pretrained model rather than learned through user-specific parameter updates.

W1(b) & Q1: Few-Shot Preference Identification and Activation

To directly examine whether pretrained LLMs can identify and utilize preferences from limited interactions, we conduct an additional experiment using the same frozen backbone under four settings:

  1. Base LLM: No preference guidance.
  2. Human Preference Prompt: Human-constructed preference descriptions, as in Figure 1(a).
  3. Inferred Preference Prompt: Preference descriptions automatically inferred from four historical interactions of the target user.
  4. Mismatched Preference Prompt: Preference descriptions inferred from four historical interactions of another user.

All settings are evaluated on the same held-out test users and queries from HiCUPID, without test-time parameter updates.using Presentation Personalization (PP), consistent with Figure 1(a).

SettingPP
Base LLM4.3
Human Preference Prompt5.5
Inferred Preference Prompt6.1
Mismatched Preference Prompt4.0

The inferred preference prompt substantially outperforms both the base LLM and mismatched preference guidance, and achieves a higher PP score than the human-constructed preference prompt (6.1 vs. 5.5). These results indicate that pretrained LLMs can leverage useful preference information inferred from only four historical interactions to elicit user-aligned behaviors without parameter updates.

We also acknowledge that Figure 1(a) does not establish that user preferences can be represented in a shared low-dimensional adaptation space. This is a modeling assumption of LatentPersonal, whose effectiveness is evaluated through our main experiments and ablation studies. We provide the additional experimental details and results in Appendix D.7.


W2(a): Role of Mean Pooling in Preference Learning

As described in Section 3.2, preference learning is not determined solely by mean pooling, but is further optimized through the adaptation loss (Eq. 10):

Ladapt=Ezu∼qψ[L(Duqry;θ0+gϕ(zu))].\mathcal{L}_{\mathrm{adapt}}=\mathbb{E}_{z_u\sim q_\psi}\left[\mathcal{L}(D_u^{\mathrm{qry}};\theta_0+g_\phi(z_u))\right].

Here, zuz_u is inferred from the aggregated support interactions, while DuqryD_u^{\mathrm{qry}}provides supervision for optimizing the preference encoder qψq_\psi and shared adaptation mapping gϕg_\phi. This allows the model to learn and emphasize preference-relevant information retained after aggregation, even though mean pooling assigns equal weights to the input features. Meanwhile, the information bottleneck further constrains redundant information in zuz_u. Together, these objectives help mitigate the influence of irrelevant information during preference learning.

Our original intention was to realize effective preference learning with a simple aggregation mechanism, rather than introducing additional architectural complexity. We acknowledge that mean pooling may dilute important preference signals, and that such information loss cannot always be fully compensated by subsequent learning.

W2(b) & Q2: Comparison with Attention-Based Aggregation

To examine whether more expressive aggregation improves preference learning, we additionally compare mean pooling with attention-based pooling on HiCUPID using Llama-3.1-8B-Instruct in the 4-shot setting. Specifically, we replace the interaction-level mean pooling with learnable attention pooling, while keeping the token-level pooling, preference encoder, training objective, and shared LoRA structure unchanged.

Aggregation MethodR-1MET.CP
Mean Pooling0.41560.41375.94
Attention Pooling0.41890.41875.98

The results show that attention-based pooling achieves slightly better performance across all three metrics, with improvements of 0.0033 in R-1, 0.0050 in METEOR, and 0.04 in CP. However, these gains are relatively small, suggesting that the simple mean pooling strategy already provides effective representations for preference learning. This supports our choice of mean pooling as a lightweight aggregation mechanism without introducing additional architectural complexity.

AuthorsOpenReview ↗

W3 & Q3: Shared Latent Modulation and Layer-Specific Adaptation

W3(a): Motivation for Shared Latent Modulation

Our motivation is to use a single compact user preference representation to guide personalized adaptation across different layers. As formulated in Eq. (14), the layer-specific update can be expressed as

ΔWu(l)=B(l)diag⁡(zu)A(l)=∑i=1rzu,ibi(l)ai(l)⊤.\Delta W_u^{(l)} = B^{(l)}\operatorname{diag}(z_u)A^{(l)} = \sum_{i=1}^{r} z_{u,i} b_i^{(l)} a_i^{(l)\top}.

Although the modulation coefficients zuz_u are shared, each layer learns its own adaptation directions through A(l)A^{(l)}and B(l)B^{(l)}. Thus, the same user representation can induce distinct parameter updates across layers, rather than imposing identical adaptations. This provides layer-dependent adaptation while requiring only a single compact latent representation inferred from limited user interactions.

W3(b) & Q3: Comparison with Layer-Specific Modulation

To examine whether independent modulation coefficients provide additional benefits, we compare shared latent modulation with a layer-specific variant. Specifically, we extend the preference encoder to generate a larger latent representation, which is split into separate vectors zu(l)z_u^{(l)} and assigned to the corresponding LoRA layers, rather than using a single shared vector zuz_u. To accommodate the increased output dimensionality from rr to LrLr, we also increase the preference encoder capacity, adding parameters and training overhead. The pretrained backbone, LoRA rank, and overall training procedure remain unchanged. Given the increased computational cost and the limited scope of the rebuttal, we report results on the representative 4-shot HiCUPID setting.

Modulation StrategyR-1MET.CP
Shared Latent0.41560.41375.94
Layer-Specific Latent0.41850.41916.23

The results show that layer-specific modulation provides modest improvements, reflecting its greater flexibility. However, the gains are limited, suggesting that the shared latent representation already provides sufficient flexibility for effective layer-dependent personalization through independently learned LoRA factors. Moreover, our approach requires only a single (r)-dimensional user representation, rather than (Lr) layer-specific coefficients, preserving the compactness of latent navigation in few-shot settings.

We hope that these clarifications, additional experiments, and manuscript revisions adequately address your concerns and further strengthen your confidence in LatentPersonal.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer UZCL,

Thank you again for your valuable feedback. We have provided detailed responses to your concerns in our rebuttal.

As the discussion period nears its end, we would greatly appreciate your feedback on whether our responses have adequately addressed your concerns or if any questions remain.

Thank you very much for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.