acceptodds
Under review as a conference paper at ICLR 2027

UPSHIFT: User Preference Steering from Human–AI Interaction Feedback Traces

Abstract

Personalizing large language model assistants requires adapting their responses to individual users, including the selection of content, reasoning approach, level of detail, and form of presentation. Such preferences are often latent, becoming visible only when users respond to an initial answer and request revisions. We investigate whether these feedback trajectories can be converted into a reusable personalization signal without explicit preference profiles or labels, prompt-time access to interaction histories, or model parameter updates. We introduce UPSHIFT (User Preference Steering from Human–AI Interaction Feedback Traces), a training-free method that compares the hidden representations of initial and feedback-conditioned responses within each session, averages the resulting trajectory displacements across a user's sessions, and injects the aggregated preference direction when answering a new query. The method is motivated by the implicit-preference hypothesis: although an individual conversational transition may reflect topic progression or task-specific correction, changes that recur across sessions retain a component that captures how the user wants responses to be adjusted. We validate the preference-representation capability of UPSHIFT on PrefEval. In evaluations with 40 simulated users, every tested configuration improves preference alignment over the unsteered baseline on held-out topics for both Qwen3-8B and Qwen3-32B; the best configuration increases macro-averaged Elo from 1364.0 to 1605.2 and from 1341.9 to 1591.8, respectively. Matched-user and positive-versus-negative controls further demonstrate that these gains arise from the user-specific preference direction and its learned orientation. These results show that interaction histories can provide compact, reusable supervision for lightweight personalization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Activation Steering, Preference Learning, Implicit Feedback, Human–AI Interaction, Representation Engineering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7A8fOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes UPSHIFT, a training-free method for LLM personalization that extracts user-specific steering vectors from historical human–AI interaction trajectories. The method computes differences between the hidden representations of initial assistant responses and subsequent feedback-conditioned responses, then averages these differences across sessions to obtain a user-specific preference direction. The resulting vector is injected into the model's residual stream during inference, enabling personalization without updating model parameters or including historical conversations in the input prompt.

The authors evaluate UPSHIFT on PrefEval and through experiments involving 40 simulated users with Qwen3-8B and Qwen3-32B. The results demonstrate improvements over unsteered baselines on held-out topics. Additional experiments involving mismatched user vectors and positive-versus-negative steering directions suggest that the extracted vectors contain user-specific preference signals.

Critical strengths

1. An interesting source of implicit personalization signals. The paper explores the possibility of extracting user preference information from changes in assistant responses following user feedback, without requiring explicit preference labels or user-authored target responses. Although the underlying activation steering framework is closely related to prior work, using feedback-conditioned response trajectories as a source of personalization signals is an interesting idea.

2. A simple formulation with useful control experiments. The proposed method is straightforward to implement and does not require model parameter updates. The authors also conduct user-mismatch and positive-versus-negative steering experiments, which provide useful evidence that the extracted vectors contain user-specific information rather than merely inducing arbitrary changes in model behavior.

Critical weaknesses

1. Limited methodological novelty and lack of comparisons with closely related prior work.

The methodological novelty of UPSHIFT appears limited in the context of existing activation-based personalization methods, such as StyleVector (Zhang et al., 2025), SteerX (Zhao et al., 2025), and GLASS (Chen et al., 2026). In particular, StyleVector already constructs user-specific steering vectors by averaging contrastive differences between hidden representations and applies them at inference time without updating model parameters. UPSHIFT follows essentially the same overall framework, with its primary distinction being the use of initial and feedback-conditioned assistant responses as contrastive pairs. While this is a different source of supervision, it is unclear whether it represents a sufficiently substantial methodological contribution beyond existing contrastive activation steering approaches.

More importantly, despite explicitly discussing these closely related methods in the Related Work section, the paper does not include them as experimental baselines. The main results are based on comparisons against an unsteered LLM, supplemented by user-mismatch and steering-direction controls. Although these experiments demonstrate that the extracted vectors can influence responses in a user-specific manner, they do not establish any advantage over existing personalization techniques.

This is a fundamental limitation rather than merely a missing experimental detail. Given the substantial methodological overlap with prior work, direct comparisons are essential to establish the significance of the proposed contribution. Without such comparisons, it is difficult to determine whether the proposed feedback-trajectory-based extraction offers any meaningful improvement over existing approaches, or whether similar results could already be achieved using established contrastive activation steering methods. Consequently, the paper fails to convincingly establish either its methodological novelty or its relative effectiveness.

2. Insufficient empirical validation of the central implicit-preference hypothesis.

The paper's central hypothesis is that aggregating hidden-state changes across feedback trajectories can recover stable, implicit user preferences that generalize to new tasks. However, the experimental validation does not sufficiently establish this claim.

The main personalization experiments rely entirely on 40 simulated users generated from publicly available LinkedIn profiles, with evaluations performed by a profile-conditioned LLM judge. Since the underlying profiles primarily contain biographical and professional information rather than explicit subjective response preferences, it is unclear whether the observed improvements reflect genuine preference inference or consistency between the simulated users and the evaluation judge, both of which are conditioned on the same profile information.

Furthermore, the PrefEval experiments primarily demonstrate that activation steering can represent and control preference-related information in a benchmark setting. They do not directly validate whether stable, latent preferences can be reliably recovered from natural human feedback trajectories.

Therefore, the experiments do not convincingly establish that the proposed method can discover and generalize actual human preferences. Given that this is the central motivation and claimed contribution of UPSHIFT, the reliance on simulated interactions and LLM-based evaluation substantially limits the strength of the empirical evidence.

Taken together, these concerns make it difficult to establish the paper's significance beyond existing activation-based personalization research and substantially weaken the support for its central claims.

References

[1] Zhang, J., Liu, Y., Wang, W., Liu, Q., Wu, S., Wang, L., and Chua, T.-S. (2025). Personalized Text Generation with Contrastive Activation Steering. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), pp. 7128–7141.
https://aclanthology.org/2025.acl-long.353/

[2] Zhao, X., Yan, M., Qiu, Y., Ni, H., Zhang, Y., Feng, F., Cheng, H., and Chua, T.-S. (2025). SteerX: Disentangled Steering for LLM Personalization. arXiv preprint arXiv:2510.22256.
https://arxiv.org/abs/2510.22256

[3] Chen, L., Zhang, Z., Zhang, X., Nie, S., Liu, Q., Wu, S., and Wang, L. (2026). From Profiles to Steering Vectors: Global Sparse Priors and Local Semantic Calibration for Personalized Text Generation. arXiv preprint arXiv:2607.21620.
https://arxiv.org/abs/2607.21620

Questions for authors

Q1. What is the substantial methodological contribution of UPSHIFT beyond existing contrastive activation steering approaches?

In particular, StyleVector already constructs user-specific steering vectors from averaged hidden-state differences and uses them for inference-time personalization. Apart from replacing the contrastive pairs with initial and feedback-conditioned assistant responses, what fundamentally distinguishes UPSHIFT from existing approaches? What evidence demonstrates that this particular choice of contrastive pairs offers additional benefits?

Q2. Why are closely related personalization methods not included as experimental baselines?

The paper discusses StyleVector, SteerX, and GLASS but does not compare against them. Can the authors provide comparisons against these methods, or appropriately adapted versions of their vector extraction procedures, under matched experimental conditions? In particular, can the authors isolate whether the proposed temporal feedback contrast provides improvements beyond those achievable through existing activation-based personalization techniques?

Q3. How can the authors establish that the extracted vectors represent genuine implicit user preferences rather than artifacts of the simulation and evaluation setup?

Since both the simulated users and the LLM judge are conditioned on LinkedIn-derived profiles that do not explicitly specify subjective response preferences, how is the validity of the inferred preferences established? Is there any independent human evaluation or evidence from real user feedback trajectories supporting the central implicit-preference hypothesis?

3 comments

We sincerely thank the reviewer for the thoughtful and detailed assessment of our work. We appreciate the recognition of UPSHIFT's implicit feedback-based personalization approach and its controlled experiments. We address the concerns regarding methodological novelty, baseline comparisons, and the empirical validity of the implicit-preference hypothesis below.

Q1. What is the substantial methodological contribution of UPSHIFT beyond existing contrastive activation steering approaches?

Thank you for raising this important question. We agree that the activation addition operation and cross-sample averaging themselves are established techniques. Our contribution is not to introduce a new activation-steering operator, but to investigate a different source of personalization supervision and formulate a training-free approach to extracting reusable preference directions from natural human–AI interaction trajectories.

Specifically, existing approaches such as StyleVector derive user-specific directions primarily from user-authored texts and style-neutral counterparts, targeting the reproduction of individual writing styles. SteerX further investigates the disentanglement of preference-driven information from user histories, while GLASS extracts global and local style representations from historical responses. In contrast, UPSHIFT uses the temporal contrast between an assistant's initial response and its feedback-conditioned response within the same interaction session. This provides assistant-side supervision about how users want responses to be adjusted, rather than relying on how users themselves write or on explicitly constructed preference descriptions.

We believe this distinction is important because users' preferred assistant responses do not necessarily resemble their own writing styles. For example, a user may communicate concisely while preferring detailed and rigorous explanations from an assistant. UPSHIFT aims to capture this distinction by learning from users' interventions on assistant-generated responses.

Thus, the methodological contribution lies in the feedback-trajectory-based supervision formulation and its cross-session aggregation, rather than in the underlying vector addition operation itself. We will clarify this distinction and better articulate the scope of our methodological contribution in the revised manuscript.

Q2. Why are closely related personalization methods not included as experimental baselines?

We acknowledge that direct comparisons with existing methods would provide additional evidence regarding their relative effectiveness. However, the original evaluation objectives and supervision settings of these methods differ from those of UPSHIFT.

For instance, StyleVector and GLASS primarily evaluate personalized text generation by measuring the correspondence between generated outputs and user-specific historical text. SteerX also investigates preference-driven information, but its empirical evaluation focuses on personalized review generation and topic writing, using reference-based text-generation metrics such as ROUGE, METEOR, BLEU, and BERTScore. These evaluations do not directly measure whether users prefer an assistant's responses on previously unseen topics based on implicit preferences inferred from feedback trajectories.

Consequently, directly comparing the reported results across these methods would not establish a meaningful performance ranking. A rigorous comparison would require adapting their extraction procedures to the same interaction-feedback setting and evaluating all methods under a common user-preference protocol.

Our current experiments establish its effectiveness relative to unsteered generation and provide user-specificity and directionality controls. We will make this distinction explicit and better position UPSHIFT as a complementary approach that explores a different source of personalization supervision.

AuthorsOpenReview ↗

Official Comment by Authors

Q3. How can the authors establish that the extracted vectors represent genuine implicit user preferences rather than artifacts of the simulation and evaluation setup?

We appreciate this fundamental concern and agree that relying exclusively on simulated users and profile-conditioned LLM judgments leaves room for potential bias. In particular, conditioning both simulated users and evaluators on the same LinkedIn-derived profiles may introduce correlations that do not necessarily reflect genuine subjective preferences.

To address this concern, we conducted an additional small-scale human study involving 10 volunteers. Each participant selected six topics of personal interest and completed at least three rounds of interaction with the assistant on each topic. We extracted individual preference vectors using UPSHIFT with the ALL-RM configuration. For evaluation, each participant completed ten A/B comparisons on newly sampled topics, comparing unsteered responses against responses steered by their own extracted preference vectors. We used Qwen3-32B with a steering coefficient of α = 0.05.

The results show that participants preferred UPSHIFT-steered responses in approximately 63% of the pairwise comparisons. Importantly, both the interaction histories used for vector extraction and the subsequent preference judgments came from real human participants, without relying on LinkedIn-derived profiles or profile-conditioned LLM judges. The evaluation also uses previously unseen topics, providing initial evidence that feedback-derived preference directions can generalize beyond the interactions from which they were extracted.

We acknowledge that this study is relatively small-scale and does not establish the robustness of the implicit-preference hypothesis across a broad population. Nevertheless, it provides an independent human-based validation that complements our larger simulated-user experiments and directly addresses the potential evaluation bias raised by the reviewer.

We will incorporate this additional human evaluation into the revised manuscript and clarify the limitations of both simulated and human evaluations.

We sincerely thank the reviewer again for these constructive comments, which have helped us clarify the methodological contribution and strengthen the empirical validation of UPSHIFT.

I thank the authors for their detailed response and, in particular, for conducting an additional human evaluation. The new study with real participants is a meaningful improvement over the original evaluation, as it partially addresses my concern about the reliance on simulated users and profile-conditioned LLM judgments.

However, my primary concerns remain unresolved.

First, while I appreciate the clarification that the contribution lies in feedback-trajectory-based supervision rather than a new activation steering mechanism, the rebuttal does not provide empirical evidence that this supervision formulation offers meaningful advantages over existing contrastive activation steering approaches.

Second, the lack of comparisons with closely related personalization methods remains a fundamental limitation. I understand that these methods were originally evaluated under different settings, but my concern was precisely the need for comparisons under a common evaluation protocol, rather than comparisons of previously reported results.

Third, although the additional human study is encouraging, its small sample size and the absence of user-mismatched steering controls in the human evaluation make it difficult to determine whether the observed preference improvements reflect genuinely user-specific preference inference rather than more general improvements in response quality.

Overall, I acknowledge that the rebuttal strengthens the empirical evidence, particularly through the new human study. Nevertheless, the central concerns regarding the paper's distinct contribution and its effectiveness relative to existing methods remain. Therefore, I maintain my original rating.

Reviewer_LVKXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents UPSHIFT, a training-free personalization framework for large language models based on activation steering. Motivated by the hypothesis that recurring feedback displacements across sessions capture user preferences while averaging out session-specific content, UPSHIFT extracts hidden state differences between initial and revised responses across past dialogues. The resulting mean displacement vector is injected into the model's residual stream during inference for new queries, without requiring explicit user profiles, dialogue histories in context, or weight updates. Evaluations on PrefEval and 40 simulated users using Qwen3-8B and Qwen3-32B show improvements in preference win rates against the unsteered baseline.

Critical strengths

1 Training-free perspective on feedback: Deriving preference steering directions implicitly from the natural temporal progression of multi-turn interactions, rather than relying on explicit persona profiles or synthetic contrastive pairs, is an appealing and conceptually lightweight idea. 2 Controlled causality analysis: The inclusion of matched versus mismatched user controls and positive versus negative directionality tests provides clean empirical evidence that the observed changes are tied to the specific user orientation rather than generic activation perturbation.

Critical weaknesses

1 Absence of existing personalization baselines: The experimental validation only compares UPSHIFT against the raw unsteered baseline and its own internal ablations. Standard in-context personalization approaches and recent activation-based methods are discussed in related work but omitted from the empirical tables. 2 Conflation of task progression with preference feedback: In natural multi-turn interactions, follow-ups frequently advance a task or request new information rather than correcting style or format preferences. Averaging uncurated session differences risks capturing arbitrary conversational progression rather than stable preferences, which could dilute the steering vector. 3 Lack of evaluation on general capabilities: Persistently adding an activation vector across all generation tokens can easily degrade core model capabilities, such as mathematical reasoning, instruction following. The paper does not provide evaluations to verify whether general model utility is preserved.

Questions for authors

1)Why are established personalization baselines and existing activation steering methods missing from the main comparative experiments? How does UPSHIFT compare against them under identical test conditions. 2)How does the framework distinguish between feedback that signals a persistent personal preference and ordinary multi-turn task continuation? Does averaging over uncurated multi-turn dialogues dilute the preference direction when most follow-ups are task-driven? 3)What is the impact of the added steering vector on the model's core capabilities?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for the thoughtful assessment of our work and for recognizing the value of deriving user-specific steering directions from natural interaction feedback. We appreciate the three important concerns regarding baseline comparisons, the distinction between preference feedback and task progression, and the preservation of general model capabilities. We address each question below.

Q1. Why are established personalization baselines and existing activation steering methods missing from the main comparative experiments?

We agree that positioning UPSHIFT relative to existing personalization approaches is important. We would like to clarify that the methods discussed in our paper share the broad goal of LLM personalization but differ in their optimization targets and supervision sources.

For example, StyleVector primarily focuses on reproducing user-specific writing styles from historical user-authored texts. SteerX further investigates disentangling preference-related information from user histories, but its empirical evaluation focuses on personalized text generation using historical text references. In contrast, UPSHIFT targets implicit preference alignment in human–AI interactions. It extracts response-adjustment signals from temporal changes between initial and feedback-conditioned assistant responses and evaluates whether users prefer the resulting responses on previously unseen topics.

Thus, although these approaches share activation-space steering as a technical foundation, their supervision sources and evaluation objectives are substantially different. In particular, similarity to historical user-authored texts does not directly measure whether an assistant's responses satisfy a user's latent preferences. Similarly, conventional in-context personalization approaches typically rely on explicit user profiles or historical interactions in the input context, whereas UPSHIFT aims to compress interaction feedback into a reusable steering direction without requiring historical context during inference.

A direct comparison would require adapting their methods to the same interaction-feedback setting and establishing a common preference-based evaluation protocol. We will clarify these distinctions and better position UPSHIFT relative to prior work in the revised manuscript.

Additionally, to strengthen the real-world validation of UPSHIFT, we conducted a small-scale human study with 10 volunteers. Participants preferred UPSHIFT-steered responses in approximately 63% of pairwise comparisons against unsteered responses on newly sampled topics. This provides initial evidence of its effectiveness with real human feedback.

Q2. How does the framework distinguish persistent personal preferences from ordinary task continuation? Does averaging over uncurated dialogues dilute the preference direction?

We agree that this is an important limitation of learning from natural conversational feedback. UPSHIFT does not explicitly classify each follow-up into preference feedback, task continuation, or factual correction. Instead, it relies on the implicit-preference hypothesis that recurring user-specific response adjustments can be retained through cross-session aggregation, while session-specific variations may be attenuated.

A useful statistical interpretation is:

du,s=pu+cu,s+ϵu,s,d_{u,s}=p_u+c_{u,s}+\epsilon_{u,s},

where pup_u denotes a recurring user-specific response-adjustment component, cu,sc_{u,s} captures session- and task-dependent changes, and ϵu,s\epsilon_{u,s} represents stochastic noise. If session-dependent variations are sufficiently diverse and weakly correlated across sessions, averaging can attenuate these variations while retaining recurring user-specific signals.

However, we acknowledge that this assumption does not always hold. In particular, systematic task progression or recurring task-specific requirements may introduce non-zero-mean components that remain in the aggregated vector. Therefore, cross-session averaging does not guarantee complete separation of preference signals from task-related changes.

Our existing experiments investigate two response-selection strategies: FIRST, which uses the response after the first follow-up, and ALL, which uses the final response after all follow-ups. Both strategies improve preference alignment over the unsteered baseline across the tested configurations in our simulated-user experiments. These results suggest that useful personalization signals can be extracted under different interaction depths, although they do not establish explicit disentanglement of preference feedback from task progression.

We will clarify this distinction and explicitly acknowledge the possibility of task-related signal contamination in the revised manuscript. More selective feedback modeling and context-dependent preference extraction are promising directions for future work.

AuthorsOpenReview ↗

Official Comment by Authors

Q3. What is the impact of the added steering vector on the model's core capabilities?

We appreciate this important concern. We agree that personalization through activation steering involves a trade-off between inducing the desired behavior and preserving the model's original predictive capabilities. We do not assume that such interventions are free of side effects.

This trade-off has also been theoretically investigated in recent work. Taimeskhanov et al. (2026), in Towards Understanding Steering Strength (ICML 2026), provide a theoretical analysis of the relationship between steering strength, target behavior expression, and output distribution distortion. In particular, they establish that the increase in cross-entropy exhibits a locally quadratic dependence on the steering strength α\alpha around zero. Their analysis further characterizes how the expression of a target concept increases with steering strength, suggesting that an appropriate intervention strength can enhance the desired behavior while avoiding excessive distortion of the model's output distribution. These findings highlight the importance of balancing steering effectiveness against potential performance degradation.

Following this perspective, UPSHIFT evaluates different steering coefficients rather than assuming that stronger interventions are always better. In our experiments, α=0.05\alpha=0.05 provides the best overall preference alignment among the tested configurations. However, this selection is based on preference alignment and does not establish that general capabilities are optimally preserved at this strength.

We acknowledge that our current evaluation does not directly quantify the impact of UPSHIFT on general capabilities such as mathematical reasoning and instruction following. Therefore, we do not claim that these capabilities are entirely preserved. We will clarify this limitation in the revised manuscript and emphasize that selecting an appropriate steering strength is essential for balancing personalization effectiveness against potential degradation in general model performance.

Reference

Taimeskhanov, M., Vaiter, S., & Garreau, D. (2026). Towards Understanding Steering Strength. Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 306, 118084–118133. https://proceedings.mlr.press/v306/taimeskhanov26a.html

Thank you for the detailed response. I would like to maintain my original score.

Reviewer_UvS4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a research problem: user preferences can be better inferred from multi-turn conversations with human feedback than from simple conversation history. It also proposes an implicit-preference hypothesis that a stable user preference can be extracted by aggregating the representation displacements across multiple conversations. The proposed method seems reasonable, but its evaluation relies only on simulated users, and it lacks theoretical analysis of which preferences the learned steering vector actually captures.

Critical strengths
  1. The paper identifies an interesting research problem: user preferences extracted from the representation displacements across multiple feedback-bearing sessions are more stable and task-independent, and this stable preference can be reused in new conversations. It also proposes a method, UPSHIFT, to capture this preference across sessions.
Critical weaknesses
  1. The personalization evaluations are based purely on simulated user interactions, which likely introduces bias. Real-world user feedback can be noisy and unstable.
  2. The proposed method lacks a theoretical basis beyond the implicit-preference hypothesis, which is assumed rather than analyzed, and it provides limited insight. Moreover, which personalized preferences the learned steering vector actually reflects has not been specifically evaluated.
Questions for authors
  1. Could the authors provide an evaluation on a subset of real human users to demonstrate that the problem exists in real-world settings?
  2. Could the authors provide a more detailed exploration of which preferences the learned steering vector actually captures? This would make the proposed method more theoretically solid.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for the encouraging assessment and for raising important questions about the real-world validity and interpretability of UPSHIFT. We address both questions below.

Q1. Could the authors evaluate a subset of real human users?

Thank you for this valuable suggestion. We agree that simulated interactions may not fully reflect the noise and diversity of real-world feedback. We therefore conducted an additional small-scale human study with 10 volunteers.

Each participant selected six topics of personal interest and completed at least three interaction rounds on each topic. We extracted a user-specific vector using UPSHIFT (ALL-RM). For evaluation, each participant completed ten A/B comparisons on newly sampled topics. Each comparison paired an unsteered response with an UPSHIFT-steered response. We used Qwen3-32B with a steering coefficient of α = 0.05.

Participants preferred the UPSHIFT-steered responses in approximately 63% of pairwise comparisons. Although this study is small-scale, it provides initial evidence that feedback from real human–AI interactions can yield useful personalization signals, complementing our simulated-user evaluation. We will include this study in the revised manuscript and acknowledge that larger-scale studies are needed to assess robustness and generalizability.

Q2. What preferences do the learned steering vectors actually capture?

We agree that characterizing the learned vectors is important for interpreting the implicit-preference hypothesis. We first clarify that UPSHIFT is not designed to decode a vector into a human-readable profile or a predefined preference label. Instead, it directly estimates a continuous direction in the model's representation space from feedback-induced changes in hidden states. A vector therefore need not correspond to one identifiable attribute; it may reflect a combination of recurring response tendencies, such as preferred detail, reasoning depth, presentation, or explanatory style.

A useful statistical interpretation is to decompose each session-level displacement as du,s=pu+ϵu,sd_{u,s}=p_u+\epsilon_{u,s}, where pup_u denotes the recurring user-specific response-adjustment component and ϵu,s\epsilon_{u,s} captures session-specific factors, including topic progression, task-related changes, and interaction noise. If these variations are sufficiently diverse and weakly correlated across sessions, averaging can attenuate them while retaining recurring user-specific signals. This is the statistical motivation of UPSHIFT, not a guarantee that every conversational displacement reflects a stable preference.

Our existing experiments provide indirect evidence about the functional properties of the learned directions. The PrefEval results show that steering improves preference-related classification and generation performance over the unsteered baseline. The matched-user versus mismatched-user comparisons further indicate user specificity. In addition, our positive-versus-negative directionality control shows that the learned orientation matters: with ALL-RM and α = 0.05, positive steering outperforms negative steering by approximately 240 Elo points on Qwen3-8B and 282 Elo points on Qwen3-32B. These controls were conducted with simulated users.

We acknowledge that these results demonstrate preference-related functionality, user specificity, and directional effectiveness, but do not identify the precise semantic attributes encoded by each vector. We have not conducted an explicit attribute-level decomposition. We will clarify this distinction in the revised manuscript and identify finer-grained behavioral and semantic analysis as an important direction for future work.

We thank the reviewer again for this suggestion, which helps us clarify the interpretation and limitations of UPSHIFT.

Thank you for the rebuttal, which has partially addressed my concerns. However, I believe the paper still needs further improvement, so I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.