acceptodds
Under review as a conference paper at ICLR 2027

PersJEPA: Interpretable Language Model Personalization through Sparse Residual Prediction

17 views

Abstract

Language models can personalize their answers when given a user's profile or past interactions, but repeatedly supplying this information consumes context and computation. Learning personalized parameters can reduce this dependence on prompting, yet offers limited visibility into how user preferences affect the model's responses. Activation steering has attracted growing interest as a way to personalize models through interventions in hidden states without updating their parameters, yet it faces three key challenges: (1) fixed adjustments overlook how user preferences interact with each request, (2) steering confined to the initial decoding step leaves later steps without further intervention, and (3) accurate prediction of hidden representation changes does not reliably translate into personalized responses. To address these challenges, we introduce PersJEPA, a framework for interpretable personalization of frozen language models through sparse prediction conditioned on user information. Inspired by JEPA, we formulate predictive residual learning, using paired generic and personalized representations to learn how user context changes the model's hidden states. We design sparse residual decoding so that each request activates a small set of learned features whose directions are combined into a personalization residual. User conditioning determines which features contribute and how they adjust the model's representations. We further develop generation-aligned steering, applying the decoded residual throughout generation and optimizing the predictor for the likelihood of personalized responses through the frozen model while retaining residual supervision. The language model remains frozen throughout training, and inference requires no profile text in the prompt. Experiments demonstrate substantial improvements in controlled persona generation across multiple language models and gains on real user personalization benchmarks. Analyses of the learned features reveal how their contributions vary across user groups, providing a concrete basis for interpreting the learned personalization interventions. Our implementation is fully open-sourced.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Large language models, personalization, joint-embedding predictive architectures, sparse autoencoders, activation steering, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nFpnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PersJEPA takes ideas from JEPA to personalize a frozen LM while keeping the user’s profile out of the prompt at inference time. But training still uses user information. The authors pass each answer through the LM twice, once with that information and once without, and use the difference between the hidden states as a training target.

They also group users into different clusters and train a small sparse model for each cluster to predict this difference from the request alone. These smaller models get another round of tuning to make the user’s answers more likely under the frozen LM. The authors also introduce SynPer, their own LLM-generated benchmark with ten personas, to evaluate the method.

Critical strengths

The paper offers a practical way to steer a frozen language model without including the user’s profile in each inference prompt. It learns small predictors and applies their outputs throughout generation. This supports the feasibility of the approach for controlled persona generation.

Critical weaknesses
  • PerFit [1] already studies how adding user information changes hidden states. It separates this change into shared and user-specific parts, learns adjustments for both, and evaluates them on the same LaMP tasks.

  • L2S [2] trains a small network to predict the hidden-state change caused by contrastive prompts using only the input. It applies the predicted change at each generated token. This is very close to the first stage of PersJEPA.

The ablations do not show what the proposed components add - A dense predictor without residual loss performs as well as the full model (94.0% vs. 93.2%).

Fixed group vectors obtained by averaging the trained predictor’s outputs still reach 87.4%. This raises questions about whether the separately trained group-vector baseline (62.6%) was optimized well enough.


Relatively limited evidence for personalization .

On LaMP, a global predictor without user information recovers 88–98% of the gains. Only LaMP-2 shows a notable user-specific gain (+3.2 points). With just 20 groups for thousands of users, personalization largely happens at the group level. It is unclear how the method captures differences between users within a group.

The large gains come from SynPer, where the true labels assign ten synthetic personas to experts. Its classifier is trained on the same synthetic answers the method learns to reproduce, yet the paper reports neither its accuracy on reference answers nor a human evaluation. Increasing the steering scale also lowers F1.


[1] J. Liu et al. PerFit: Exploring Personalization Shifts in Representation Space of LLMs. ICLR 2026.

[2] J. Parekh et al. Learning to Steer: Input-dependent Steering for Multimodal LLMs. NeurIPS 2025.

Questions for authors

Without clarification, the following points limit my confidence in the reported results.

  1. Table 2 numbers evaluation. - If each percentage is calculated from 500 examples in a single run, the values should be multiples of 0.2%. The other values in the 9B and Llama blocks follow this pattern, but the SteerX and BiPO rows include 2.7%, 2.3%, 15.9%, 10.9%, and 21.3% - Please clarify how these scores were calculated or averaged.

  2. What does “cold start” mean in Figure 3d if these users’ histories were used to train the predictor? And how many history entries were actually used at k=20 when each user has at most eight? Why does the unsteered baseline differ from Table 1 (36.1% vs. 36.3%)?


The following points need to be addressed to make the paper more solid and convincing.

  1. Explain what PersJEPA adds beyond PerFit, L2S, and other steering methods. What do residual prediction and sparsity contribute when a dense predictor performs similarly?

2.Resolve the inconsistencies listed in Part I above, and document the settings used to produce the reported results.

Reviewer_feysOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The task is standard LLM personalization. The method is mostly standard activation steering: it adds a predicted steering vector to the hidden states of every output token to change the LLM's output. The method clusters the users into a few groups (softly), and then learns a predictor for each group. Each predictor maps the original, non-personalized state to the steering vector. Each user is routed to a predictor (or multiple predictors when doing soft grouping) based on their average activation difference (to which the steering vector will later be added). Each predictor is learned by first MSE-based pretraining and then fine-tuning on answer likelihood.

Critical strengths
  1. The ablations on SynPer are well controlled. The authors report paired bootstrap intervals and include permuted-routing and expert-swap controls.
  2. The method is straightforward.
Critical weaknesses
  1. The JEPA advertising is misleading and does not match the actual design. This method should be positioned clearly as an activation-steering-based method. Neither its problem formulation nor its design choices are directly motivated by or borrowed from JEPA. And by the paper's own ablation, the most JEPA-like part contributes little: MSE-based pretraining alone gives 36% persona accuracy against 33% unsteered.

  2. Activation steering introduces user-specific parameters, which is unavoidably unscalable, even with soft grouping or activation steering or some other tricks. The literature has moved largely to a single shared model that conditions on user context, where a shared model learns the user-to-behavior mapping across all users, handles new users with no calibration, and is not restricted to a single fixed additive direction. There is a large body of work on shared personalized models (e.g., LaMP-style fine-tuning with retrieved history, P-RLHF, PerCE, NextQuill), and none of it appears as a baseline.

  3. No experiment uses a held-out user: every evaluated user's profile items are used to pretrain the predictor, fit the routing clusters, and serve as likelihood-stage training targets, so the claims of scalability and new-user handling are never tested, and the LaMP results cannot be compared with methods evaluated on LaMP's user-based split, which is designed for users unseen in training.

  4. The strongest results depend on a synthetic, label-aligned setting. SynPer routes each example to its true persona's expert. Its personas are stylistically extreme.

Questions for authors

Please see above weaknesses

Reviewer_CKufOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PersJEPA, a new personalization approach for frozen LLMs via learned activation interventions. The proposed method constructs residual targets by comparing generic and personalized representations, and then uses sparse residual decoding to predict adjustments using a small number of active features. The predicted adjustments are then applied to the frozen LLMs to steer generation. Experiments on their new synthetic persona benchmark and existing personalization benchmarks demonstrate strong personalization performance and improvements on real user tasks.

Critical strengths
  • This paper introduces a new method for LLM personalization and shows strong results on both synthetic evaluations by the authors and existing benchmarks. The proposed methods don't require prompting additional context every time, which is practically useful.
  • The authors conduct extensive ablations to understand the core components behind the strong empirical results.
Critical weaknesses
  • Based on the ablation results, the main performance gains appear to come from directly optimizing the interventions and using user-conditioned experts, with additional gains from request-dependent adjustments. However, dense predictors achieve comparable performance without residual pretraining or the auxiliary residual loss, even though residual learning appears to be a core part of the method.
  • The results on the authors’ synthetic benchmark and LaMP-2 are strong, while gains over strong baselines on the other datasets are generally limited or inconsistent.
  • The presentation is occasionally hard to follow due to the density of technical details.
Questions for authors
  • Given that dense predictors achieve comparable performance without residual pretraining or the auxiliary residual loss, what are the motivations or advantages of using a JEPA-inspired approach?
  • Do the authors have an explanation for why the performance gain of the proposed method is somewhat limited on most of the existing benchmarks, while their method shows substantial gains on the new synthetic benchmark and LaMP-2?
1 comment
AuthorsOpenReview ↗

Response to Reviewer CKuf Part2/2

Weakness 2 & Question 2: W2: The results on the synthetic benchmark and LaMP-2 are strong, while the gains over strong baselines on the other datasets are limited or inconsistent. Q2: Do the authors have an explanation?

Answer to W2 & Q2:

  • The gains are consistent among methods that see no user text. Across Table 1, PersJEPA outperforms every published steering baseline (CAA, FinTS, SteerX) in all 21 dataset-backbone cells and is the best method without user text in 20 of them. The table gives the four datasets averaged over the three backbones (ROUGE-L, Table 1). PersJEPA also exceeds placing the user's full profile in the prompt in 11 of these 12 cells, while using no profile text.
Method (no user text at inference)LaMP-4LaMP-5LaMP-7Amazon
Unsteered10.619.136.011.2
Best published steering baseline11.219.838.711.3
Trained vectors12.839.042.211.8
PersJEPA14.940.843.512.4
  • The closest baseline is built from our own components, and we still improve on it. "Trained vectors", the BiPO [5] row of Table 1, keeps the same groups and routing, the same likelihood objective and the same persistent injection as PersJEPA, and replaces the request-conditional predictor with one constant per group. It is the "fixed vector per group" arm of Table 3. Its strength therefore supports our design. We also added PerFit [2] as a baseline, re-implemented in our setting with one vector trained per user, whereas PersJEPA trains nothing per user.
Dataset (Qwen3.5-4B)PersJEPA vs. Trained vectorsPersJEPA vs. PerFit
LaMP-2 (accuracy)+19.2+26.5
LaMP-4 (ROUGE-L)+0.9+0.9
LaMP-5 (ROUGE-L)+0.5+3.0
LaMP-7 (ROUGE-L)+0.3+0.3
Amazon (ROUGE-L)-0.4-1.6

The differences from Trained vectors are statistically significant under a user-level paired bootstrap. PersJEPA is ahead on four of the five datasets against both baselines. Amazon is the exception on this backbone: it is the one set where a user's signal is individual rather than shared by a group.

  • Why the margins are largest on SynPer and LaMP-2. These are the benchmarks where the right intervention depends most on the user and on the request. On SynPer the same request has ten different persona references, so user routing carries the gain: one global expert reaches 42.6% against 93.2%. LaMP-2 is 15-way tagging, where a constant vector cannot express which tag fits a given movie. This matches the finding that a fixed steering vector's effect varies strongly across inputs [7], and it is what our request-conditional predictor adds.
  • Why the margins are smaller under ROUGE-L. ROUGE measures term overlap with a single reference, and term-overlap metrics often fail to capture the stylistic similarity that matters in personalized generation [8]. Indeed, even placing the user's actual profile in the prompt raises ROUGE-L by at most 2.4 points on these four tasks. Modest margins are also the norm in prior work: the published steering method StyleVector reports an 8% relative improvement in personalized generation [9]. Our routing controls nevertheless isolate a statistically significant user-specific component on LaMP-4 and in an added run on Amazon.

Weakness 3: The presentation is occasionally hard to follow due to the density of technical details.

Answer to W3: We will add a one-table summary of the four stages (calibration, residual pre-training, likelihood fine-tuning, inference) with the inputs and trained parameters of each, add a notation table, and organise Section 5.1 around the roles listed in our answer to W1 & Q1.

Thank you again for the constructive feedback. We hope that these clarifications and the added experiments address your concerns.

References

[1] Huben et al. "Sparse Autoencoders Find Highly Interpretable Features in Language Models." ICLR, 2024.

[2] Liu et al. "PerFit: Exploring Personalization Shifts in Representation Space of LLMs." ICLR, 2026.

[3] Assran et al. "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture." CVPR, 2023.

[4] Huang et al. "LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures." ICLR, 2026.

[5] Cao et al. "Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization." NeurIPS, 2024.

[6] Wu et al. "ReFT: Representation Finetuning for Language Models." NeurIPS, 2024.

[7] Tan et al. "Analyzing the Generalization and Reliability of Steering Vectors." NeurIPS, 2024.

[8] Salemi et al. "ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation." Findings of ACL, 2025.

[9] Zhang et al. "Personalized Text Generation with Contrastive Activation Steering." ACL, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.