acceptodds
Under review as a conference paper at ICLR 2027

Dopamine: Personalization as Brain Modes, Not Brains¹

Abstract

Language-model personalization is usually represented in weight space: each user receives a separate parameter update, commonly a low-rank adapter. We ask whether this places personalization at the wrong representational level. Rather than treating each user as a different model, we propose that a single frozen model can express many personalized behavioral modes, selected by small user-specific states in activation space. We instantiate this view with Dopamine, which learns one additive residual vector per transformer layer while leaving all backbone parameters unchanged. Across three model families spanning 0.36B to 14B parameters, synthetic preference tasks, and real-user data, Dopamine matches the personalization quality of LoRA while requiring 100–215× less per-user state. Unlike separately instantiated adapters, activation modes can also be selected row by row, allowing users with different preferences to share a single forward pass without cross-user interference. Mechanistic ablations show that successful modes do more than perturb confidence: additive steering changes the model’s greedy token at roughly 55% of generated positions, whereas multiplicative modulation usually does not. Finally, although learned modes occupy a statistically low-dimensional space, aggressive shared-basis compression fails to preserve personalization quality, suggesting that behaviorally important user variation is not captured by variance alone. Together, these results support a different systems and modeling abstraction for personalization: not many personalized models, but many compact modes of one shared model.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
language-model personalization, activation steering, activation-space adaptation, user modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_j2spOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether personalization can be represented by compact activation states rather than user-specific weight updates. It proposes DOPAMINE, which learns one additive residual vector per transformer layer for each user while keeping the backbone frozen. On synthetic style preferences across three model families and seven model/scale settings, DOPAMINE achieves accuracy close to LoRA-r8 with 140–215× fewer per-user parameters. On an Amazon-review rating prediction task, it also obtains comparable accuracy and lower held-out cross-entropy. The paper further reports unsuccessful multiplicative gating and PCA compression experiments, and describes how the per-user vectors can be selected row-wise in a shared batch.

Critical strengths
  1. The paper proposes a simple and parameter-efficient personalization interface. The main result is interesting: a small additive mode can match the observed performance of much larger LoRA adapters on the tested tasks. The 140–215× reduction in per-user parameters is meaningful for storing a large number of user profiles, even though an end-to-end serving speedup has not been demonstrated.
  2. The paper evaluates multiple model families and sizes, includes a three-seed comparison on SmolLM2, and tests both controlled styles and Amazon review histories. I also appreciate the negative results for gating and PCA compression, as well as the authors' care in distinguishing observed accuracy, storage savings, and actual serving performance.
Critical weaknesses
  1. Limited novelty and incomplete comparisons. The proposed intervention is essentially a form of per-user bias-only adaptation, and the paper acknowledges that closely related layer-wise additive interventions already exist (e.g., Sinii et al., 2025). The contribution is therefore primarily empirical, but the comparison focuses heavily on LoRA. Baselines such as BitFit, prompt/prefix tuning, and personalized representation steering would help establish whether this is a particularly effective interface rather than simply a smaller adaptation method. In addition, the fixed 60-step protocol and learning rates do not establish a well-tuned LoRA baseline. This matters especially because LoRA performs worse than the frozen model on the supplementary review-language-modeling task (Appendix B.4).
  2. The evaluation does not yet convincingly establish broader personalization ability. The synthetic task covers only a few surface-level style rules, and most cross-model results use just 30 users and one seed. For the Amazon rating task, the review text itself may reveal the rating, while per-user majority-rating and shuffled-history controls are missing. It is therefore difficult to tell how much of the improvement comes from learning individual user preferences rather than sentiment cues or label priors. More realistic personalization tasks or stronger user-history controls would substantially strengthen the main claim.
Questions for authors
  1. Could the authors provide a more carefully tuned LoRA comparison (e.g., learning-rate and training-step sensitivity), and ideally compare with another compact personalization baseline such as bias-only tuning or personalized steering? This would help clarify the strength of the parameter-efficiency result.
  2. For the Amazon rating task, how does DOPAMINE perform against per-user majority-rating, no-history, and shuffled-user-history controls? These would help establish whether the learned vectors capture user-specific preferences.
  3. For the PCA compression experiment, what happens if the fully trained modes are directly projected onto the PCA basis, rather than learning the low-dimensional codes from scratch? This would help distinguish a representational limitation from an optimization issue.
6 comments

(1/4)

Thank you for a precise and fair review. You were right that the Amazon review-LM LoRA was untuned and that the Amazon controls were missing; both are answered below with measured results, and section, table and appendix numbers refer to the uploaded PDF. Synthetic accuracies use a corrected scorer (v2; Sec 5.1, App A.2), which is why they are higher than in the submission; the last paragraph of (4/4) notes why other numbers differ.

Protocol: quality comparisons are paired on the same users; each trained compact baseline in Table 1 takes its learning rate from a sweep on validation users; the tuned synthetic and Amazon comparisons select learning rate and steps of both Dopamine and LoRA-r8 on validation users, and other per-user Dopamine and LoRA runs keep the submitted protocol (60 steps; lr 1e-2 and 5e-4) unless stated; equivalence means that the 90% CI of the paired difference lies inside +-0.02, a margin fixed before any new run; brackets are 95% paired bootstrap CIs over users, and diff is Dopamine minus the other method.

J1 / JQ1. Novelty, baselines and a tuned LoRA. We agree the operator itself is not new and that the contribution is empirical, but the closest works do not test what we test. Sinii et al. (2025) train the same per-layer vectors with reinforcement learning, one set per task, to elicit reasoning; BitFit tunes the bias terms of an encoder per task. Neither uses the vectors as a per-user state, compares them with per-user LoRA, or batches users. Used per user, the state shows four things: learning it is what matters (on the same 100 users, a gradient-trained vector per layer 0.936, a contrastive vector of the same shape 0.456, StyleVector 0.368, frozen 0.336; Sec 6.1); it matches per-user LoRA-r8 on stable preferences at 141x fewer parameters; users become points in a shared space, so 64 trained numbers per user over a mean and basis fit to other users match the full 30,720-number state (0.926 vs 0.927, 60 users, synthetic styles; Sec 7.3); and 64 distinct users decode in one batch at 0.96-0.97 of frozen throughput, against 0.63-0.69 for per-row LoRA-r8 (Sec 8). The baselines you asked for are in Table 1. On fixed styles, at about 30K parameters, training the state on the user's text matters more than its form; Amazon ratings separate the forms.

  • Synthetic (SmolLM2, 100 users x 3 seeds): Dopamine 0.939. Bias tuning of all projections (BitFit-style, 276,480 parameters) 0.944; 32-token prompt tuning with exactly Dopamine's 30,720 parameters 0.935; one-token prefix tuning 0.942; LoReFT rank 1 0.938; rank-1 LoRA with 30,720 parameters 0.943. Every trained baseline with 20K-276K parameters (three of them on seed 0 only) is within 0.006 of Dopamine, and all are within 0.012 of the score of the reference targets cut to the 64 generated tokens (0.947), so this task leaves little room to separate them.
  • Personalized steering: StyleVector (Zhang et al., ACL 2025), with layer and scale chosen on validation users, reaches 0.368, and a contrastive vector with Dopamine's shape 0.456, against 0.936 for Dopamine and 0.336 for the frozen model on the same users (seed 0; Sec 6.1); with oracle generic-style negatives StyleVector reaches 0.427 (App C). One vector trained at StyleVector's layer reaches 0.906 on validation users with 30x the default learning rate (StyleVector 0.379 on those users), so fitting the state to the user's text, not its size or depth, accounts for most of the gap (Sec 7.1).
  • Amazon (SmolLM2, 300 users, method minus Dopamine): bias tuning +0.002 [-0.015, +0.018], one-token prefix tuning -0.004 and rank-1 LoRA -0.009 are level with Dopamine; 32-token prompt tuning -0.028 [-0.048, -0.007], rank-1 LoReFT -0.065 [-0.086, -0.045] and StyleVector -0.028 [-0.049, -0.008] are significantly below it; LoRA-r8 is +0.013 [-0.002, +0.029], not significant.

(2/4)

J1 / JQ1 (continued).

  • Tuned LoRA (synthetic; App C). We selected learning rate (1e-4 to 2e-3) and steps (30 to 240) for LoRA-r8, and learning rate (3e-3 to 3e-2) on the same step grid for Dopamine, on 30 validation users. LoRA is insensitive to this grid: 17 of its 20 configurations score 0.941-0.943 on validation, and the submitted default (lr 5e-4, 60 steps) is one of four tied for the best (0.943; the tie-break on validation CE picked lr 1e-4 with 120 steps). On 100 test users x 3 seeds, tuned Dopamine reaches 0.943 and tuned LoRA-r8 0.944, diff -0.001 [-0.002, +0.000]; tuning helped Dopamine (+0.004 [+0.002, +0.006]), not LoRA (-0.001 [-0.002, +0.000]). At the default protocol, 0.939 vs 0.945 (diff -0.006). Both are inside the +-0.02 equivalence margin, at 141x fewer parameters per user.
  • Amazon review LM (App B.4). You were right: the submitted LoRA (CE 5.41; 5.394 in our rerun of that protocol) was overfitting with the default 60 steps. With learning rate and steps selected on validation users, LoRA-r8 reaches 2.865 (same learning rate, 15 steps, the edge of the grid) and Dopamine 2.904, both far below the frozen model's 3.506. App B.4 reports the tuned comparison.

J2. Broader personalization ability.

  • Real long-form personalization (Sec 6.4, Table 5, App B.3): LongLaMP product reviews with Qwen2.5-1.5B-Instruct, 93 users (we removed 7 whose target appears near-verbatim in the profile; one seed). Dopamine reaches 0.145 ROUGE-L, against 0.108 for the frozen model, 0.130 for BM25 retrieval of four profile items, 0.118 for StyleVector, and 0.129 for both a pooled vector and another user's vector; each of these paired CIs excludes zero. Prompt tuning with Dopamine's parameter count (at Dopamine's learning rate, no sweep) is level with it (0.142, diff +0.003 [-0.004, +0.010]), and per-user LoRA-r8 reaches 0.153 (diff -0.008 [-0.014, -0.001]) with 215x Dopamine's parameters. Asking the frozen model for the user's median review length reaches 0.135 (diff +0.010 [0.001, 0.019]) and a higher METEOR, so part of the gain is output length. Dopamine gains less with Qwen2.5-7B-Instruct, where it beats the frozen model only on ROUGE-L, and BM25 retrieval and StyleVector are level with it on ROUGE-L and lead it on ROUGE-1 (BM25 also on METEOR), and on topic writing, where with the 7B model it falls below the frozen model on ROUGE-1 and METEOR.
  • Beyond fixed rules, across scales (Sec 6.5, Table 6, App B.5): in a variant where each user's format and verbosity change with the subject (held-out subjects; 5 backbones, 0.36B-7B, up to 60 users x 2 seeds), methods do not saturate (the reference targets, cut to the 64 generated tokens, score 0.94-0.96), and LoRA-r8 leads at every size, by 0.028 (7B) to 0.088 (0.36B), with the gap narrowing from 0.049 to 0.028 within Qwen2.5 and changing little within Llama-3.2 (0.070 to 0.063), and still by 0.025-0.043 after tuning Dopamine's learning rate on four backbones (LoRA not re-tuned). On SmolLM2, bias tuning, LN-tuning and (IA)3, at their fixed-style learning rates, also lead tuned Dopamine, by 0.036-0.039 (40 user-seeds).
  • Seeds and users: the main synthetic comparison (Dopamine, LoRA-r8 and six baselines) uses 100 users x 3 seeds, matched across these methods. We also reran the original task at Qwen2.5-7B and Llama-3.2-3B with 100 users x 2 seeds (App B.1): Dopamine minus LoRA-r8 is +0.0001 and -0.0013, with both methods at the 64-token score of the reference targets (0.955). The other four cross-model cells stay at 30 users x 1 seed (v1, Table 3).
  • User-specific preferences rather than generic rule compliance (Sec 6.3, App B.2): another user's vector drops synthetic accuracy from 0.939 to 0.305, the frozen level (own minus other +0.634 [0.607, 0.659]), one vector trained on all 100 users reaches 0.327, and accuracy under a borrowed vector rises with the preference axes the two users share (0.037 with none, 0.714 with four).

(3/4)

JQ2. Amazon controls: majority rating, no history, shuffled history (Sec 6.2, 6.3). Paired over users (SmolLM2 300 users, Qwen2.5-1.5B 150 users), rating accuracy (Table 4 has every row except the in-context one, which is in the supplement's result tables):

ControlSmolLM2Qwen-1.5B
Dopamine, own history0.7230.727
Per-user majority rating0.6870.698
Global majority rating (5 stars)0.6740.700
No user history: one vector trained on 1,000 other users0.7380.583*
No user history: one LoRA-r8 trained on 1,000 other users0.7540.617*
That pooled vector + 60 steps on the user's own history0.7460.723*
Frozen model, not adapted0.5390.223
Frozen model + 8 history reviews in context0.7040.704
Shuffled history: Dopamine trained on another user's history0.5790.654

*Our Qwen pooled models are under-trained (best validation accuracy below the per-user majority rule), so that comparison is inconclusive.

A state carries its owner's ratings: trained on another user's history, it loses 0.143 [0.110, 0.178] accuracy on SmolLM2 and 0.073 [0.027, 0.119] on Qwen-1.5B. Five-star ratings make up 67-70% of the test labels; Dopamine beats each user's own majority rating by +0.035 [0.015, 0.055] accuracy on SmolLM2 and +0.028 [0.001, 0.055] on Qwen, and lowers MAE by 0.133 and 0.138; on Qwen its gain over the global five-star rule, +0.027 [-0.005, +0.059], is not significant. The no-history control you asked for shows that on SmolLM2 general task adaptation suffices for this accuracy: a state trained without the user's history is at least as accurate (pooled vector minus Dopamine +0.016 [-0.006, +0.038]). The best Dopamine state starts from that pooled vector and takes the 60 per-user steps (0.746, +0.024 [0.008, 0.040] over per-user Dopamine). The per-user step adds +0.008 [-0.012, +0.028] on average and +0.041 [0.005, 0.077] for the third of users whose mean rating is farthest from the pool's, the only third where the per-user step helps significantly (App B.2; the trend across thirds is not significant). By our rule (App A.3: own state beats both another user's and a pooled state), the states are user-specific on synthetic styles and LongLaMP product reviews; on Amazon with SmolLM2 they carry the owner's ratings, but a shared state is as accurate on average (Qwen: inconclusive).

JQ3. Direct projection of trained modes onto the PCA basis (Sec 7.3, Table 9). Thank you; this question led us to the cause of the submitted failure. The submitted code model loaded only the basis of the centered modes and never added the across-user mean, which carries 36.6% of the modes' squared norm (the centered span captures 19.7% of it), and it trained the codes with a learning rate of 1e-2.

  • Direct projection, no training (100 users, k=64): with the mean added back and a basis fit to the other 99 users, 0.802; with an in-sample basis, 0.927 (full mode 0.936). Without the mean, even the in-sample basis gives 0.447.
  • Re-optimized codes, learning rate chosen on validation users (1, i.e. 100x the submitted one), basis fit to other users: in the form mean + basis, 64-dim codes reach 0.926 vs 0.927 for the full mode (codes minus full mode -0.001 [-0.005, +0.003], 60 users); in the submitted form, 0.806.

So the submitted result reflected the missing mean and the step size, not a lack of low-dimensional structure: a user can be stored as 64 trained numbers over a mean and basis fit to other users. The submission's 96%-of-variance figure for the 64-dim basis was computed after removing that mean, which the submitted codes could not represent.

Reviewer_yNXPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is an AI-generated word salad. It claims that vector steering is a novel way to do personalization of LLMs, despite the paper itself citing such work.

Critical strengths

none; this is an AI-generated word salad

Critical weaknesses

The paper is an AI-generated word salad. It claims that vector steering is a novel way to do personalization of LLMs, despite the paper itself citing such work.

Detailed comments:

  • The fundamental hypothesis of this work is not novel at all: "Personalization may be represented at the wrong level. A common design stores a user as a change to model weights; we ask whether one frozen language model can instead express many user-conditioned modes, selected by small activation states". That exact question is the main focus of these papers:

  • 129-137: The related work already cites several other works that directly challenge the novelty claim. BTW, the related work is outdated: most work is form 2024 and earlier. Very little form 2025, and nothing from 2026.

  • The paper appears to have been automatically generated: teh authors admit the use of AI, and I believe this is a fully automatic word salad.

  • The abstract is full of tiny details about what exactly was achieved, but lacks the big picture: what is truly novel and why the reader should care about all these results.

  • 110-116: The contributions are completely off: extremely detailed word salad, e.g., here is contribution #3: "(iii) an account of its boundaries, including unsuccessful gating and compression, and a row-wise composition property with a clear distinction between algebra and measured serving performance."

  • 115-116: Then, follows this: "The broader thesis is deliberately conditional: before optimizing the cost of separate effective models, we should test whether users need them." -- I cannot make sense of this. I tought the central thesis was that the central question was this: "Are we representing personalization at the wrong level?" (as in Fig.1)

477-478: Here is how the conlusion ends: "The lesson is not that users never need adapted models. It is that a different effective model for every user should be a hypothesis to justify, rather than the starting assumption." -- What does this mean? How can a normal paper end like that?? How can this make sense to be said as teh very final sentences in the conclusion?

Questions for authors
  1. Did you check your novelty claim?

  2. Have you read what the LLM wrote for you at least once?

  3. Why do you waste everybody's time with AI-generated non-sense?

Reviewer_VRZsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores compact activation states as an alternative to weight-based personalization, demonstrating promising parameter efficiency. The methodological novelty is limited.

Critical strengths
  1. The paper raises a meaningful research question by examining whether user-specific behavior can be achieved through compact activation states rather than weight updates, offering an alternative perspective on the representation of personalization.

  2. The proposed method is simple and parameter-efficient, achieving performance comparable to LoRA-r8 with 140–215× fewer per-user parameters across multiple model families and scales.

  3. The paper includes useful negative results on multiplicative gating and PCA-based compression, highlighting the limitations of certain compact activation representations.

Critical weaknesses
  1. The experiments do not adequately distinguish user-specific preference learning from general task adaptation. Synthetic style preferences primarily test rule compliance, while Amazon rating prediction may rely heavily on textual sentiment or rating priors. Critical controls, such as shuffled user modes and shuffled user histories are missing.

  2. The proposed additive activation intervention closely resembles existing bias-only adaptation and activation steering methods. Although the paper acknowledges these connections, it lacks direct comparisons with closely related approaches such as ReFT and personalized steering, making its incremental contribution difficult to assess.

  3. The evaluation focuses primarily on synthetic style preferences and review-based rating prediction. These tasks do not adequately capture more challenging personalization scenarios, such as implicit preference inference, or contextual preference conflicts.

  4. The reported 140–215× reduction in per-user parameters is promising, but the comparison relies on fixed optimization budgets without extensive hyperparameter tuning. Moreover, parameter savings do not necessarily translate into lower training costs or faster inference.

  5. The paper provides limited insight into why additive interventions succeed while multiplicative gating and PCA-based compression fail. In particular, the substantial performance degradation despite retaining 96% of vector variance remains unexplained. More systematic analyses, such as layer-wise ablations, or cross-user mode analysis, would strengthen the findings.

Questions for authors

Pls see the weakness.

5 comments

(1/4)

Thank you for a careful review. We answer each of your five points with a measured result; section, table and appendix numbers refer to the uploaded PDF, and the last paragraph of (4/4) notes why some numbers differ from the submitted tables.

Protocol: quality comparisons are paired on the same users; each trained compact baseline in Table 1 takes its learning rate from a sweep on validation users; the tuned synthetic and Amazon comparisons select learning rate and steps of both Dopamine and LoRA-r8 on validation users, and other per-user Dopamine and LoRA runs keep the submitted protocol (60 steps; lr 1e-2 and 5e-4) unless stated; equivalence means that the 90% CI of the paired difference lies inside +-0.02, a margin fixed before any new run; brackets are 95% paired bootstrap CIs over users, and diff is Dopamine minus the other method.

What the per-user state shows. The operator is not new: Sinii et al. (2025) train the same per-layer vectors with reinforcement learning, one set per task, to elicit reasoning, and BitFit tunes the bias terms of an encoder per task. Neither uses the vectors as a per-user state. Used that way, they show four things:

  • Learning the state is what matters: on the same 100 users, a gradient-trained vector per layer reaches 0.936, a contrastive vector of the same shape 0.456 and StyleVector 0.368 (frozen 0.336; Sec 6.1).
  • On stable preferences it matches per-user LoRA-r8 at 141x fewer parameters: 0.939 vs 0.945, and 0.943 vs 0.944 with both tuned, equivalent within +-0.02 (Sec 6.1).
  • Users become points in a shared space: a common mean carries 36.6% of the states' squared norm, a linear probe reads each preference axis from a user's state (0.88-1.00 leave-one-out), and 64 trained numbers per user over a mean and basis fit to other users match the full 30,720-number state (0.926 vs 0.927, 60 users, synthetic styles; Sec 7.3).
  • Many users share one batch: with 64 distinct users per batch, decoding keeps 0.96-0.97 of the frozen model's throughput, against 0.63-0.69 for per-row LoRA-r8, and training takes about half of LoRA-r8's wall-clock (Sec 5.3, Sec 8).

W1. User-specific preferences vs general task adaptation; shuffled controls. We added these controls, paired over users (Sec 6.2, 6.3, Table 4, App B.2). We agree that the synthetic task mainly tests rule compliance; the Amazon and LongLaMP histories below are real users' reviews.

  • Shuffled user modes (synthetic, SmolLM2, 100 users x 3 seeds): evaluating each user with another user's trained vector drops accuracy from 0.939 to 0.305, the frozen level (own minus other +0.634 [0.607, 0.659]); one vector pooled over all users reaches 0.327. Scored against the axes of the user it was trained on, the borrowed vector reaches 0.940, so it carries its owner's style, not a generic "style on" direction. Accuracy under a borrowed vector rises with the number of axes the two users share, from 0.037 with none to 0.714 with four. LoRA-r8 behaves the same way.
  • Shuffled user histories (Amazon): a state trained on another user's history loses 0.143 [0.110, 0.178] accuracy on SmolLM2 (300 users) and 0.073 [0.027, 0.119] on Qwen2.5-1.5B (150 users), and on SmolLM2 Dopamine beats each user's own majority rating by +0.035 [0.015, 0.055] accuracy and lowers MAE by 0.133 [0.093, 0.174]. As you suspected, on SmolLM2 general task adaptation suffices for this accuracy: one vector trained on 1,000 other users' histories is at least as accurate as per-user training (0.738 vs 0.723, pooled minus per-user +0.016 [-0.006, 0.038]; pooled LoRA-r8 0.754). Starting from the pooled vector and training per user gives 0.746 (+0.024 [0.008, 0.040] over per-user Dopamine); this per-user step adds +0.008 [-0.012, 0.028] over the pooled vector on average and helps significantly only in the third of users whose mean rating is farthest from the pool's (+0.041 [0.005, 0.077]; the trend across thirds is not significant). Our Qwen2.5-1.5B pooled models are under-trained, so that comparison is inconclusive.
  • LongLaMP product reviews (Qwen2.5-1.5B-Instruct, 93 users): another user's vector loses 0.016 [0.008, 0.025] ROUGE-L, and a pooled vector 0.016 [0.008, 0.024].

By the rule we fixed before any new run (own state beats both another user's and a pooled state, App A.3), the states are user-specific on synthetic styles and on LongLaMP product reviews; on Amazon with SmolLM2 they carry the owner's rating behavior but do not beat a pooled state.

(2/4)

W2. Comparisons with ReFT, bias-only and steering methods. We agree the contribution is empirical and that the comparison focused too much on LoRA. Table 1 compares 13 baselines, grouped by where the per-user state lives (Sec 6.1, 6.2).

  • Synthetic (100 users x 3 seeds; LN-tuning, (IA)3 and 10-token prefix tuning on seed 0): every trained baseline with 20K-276K parameters per user lands within 0.006 of Dopamine's 0.939: LoReFT rank 1 0.938, 32-token prompt tuning with exactly Dopamine's 30,720 parameters 0.935, one-token prefix tuning 0.942, bias tuning of every projection (BitFit-style, 9x the parameters) 0.944. All are within 0.012 of the score of the reference targets cut to the 64 generated tokens (0.947; App A.2). Only 8-token prompt tuning (7,680 parameters, seed 0) falls clearly behind, by 0.056. Without training, 8 in-context examples reach 0.943 (diff -0.004 [-0.007, -0.001]), but they must be resent with every request.
  • Personalized steering: StyleVector (Zhang et al., ACL 2025), with layer and scale chosen on validation users, reaches 0.368, and a contrastive vector with exactly Dopamine's shape (one per layer) 0.456, against 0.936 for Dopamine and 0.336 for the frozen model on the same users (seed 0; Sec 6.1). BiPO (Cao et al., 2024) trains one vector at one layer with a preference loss; we did not run that objective. Our closest variant, one vector trained at StyleVector's layer with our next-token loss, reaches 0.906 on validation users with a larger learning rate, where StyleVector scores 0.379, so fitting the state to the user's text, not its size or depth, accounts for most of the gap (Sec 7.1).
  • Amazon (300 users, method minus Dopamine): bias tuning, one-token prefix tuning and rank-1 LoRA are level with Dopamine, while StyleVector (-0.028 [-0.049, -0.008]), rank-1 LoReFT (-0.065 [-0.086, -0.045]) and 32-token prompt tuning (-0.028 [-0.048, -0.007]) are significantly below it.

W3. More challenging scenarios.

  • Contextual preference conflicts (Sec 6.5, Table 6, App B.5): in a variant where each user's format and verbosity change with the subject's category (a different format in each of four categories, so no fixed style fits more than a quarter of a user's format targets; held-out subjects; five backbones, 0.36B-7B, up to 60 users x 2 seeds), LoRA-r8 leads at every size: by 0.088 [0.077, 0.099] on SmolLM2-360M, 0.070 on Llama-3.2-1B, 0.049 on Qwen2.5-1.5B, 0.063 on Llama-3.2-3B and 0.028 [0.019, 0.037] on Qwen2.5-7B, almost entirely on the two subject-dependent axes. With its learning rate selected on validation users (four backbones), Dopamine still trails by 0.025-0.043 (LoRA not re-tuned). It beats 8-shot in-context history and 32-token prompt tuning on four of the five backbones each; on SmolLM2, bias, LN and (IA)3 tuning lead tuned Dopamine by 0.036-0.039 (40 user-seeds). Section 6.5 reports this as the setting where a fixed offset falls short of LoRA.
  • Real long-form generation (Sec 6.4, Table 5, App B.3): LongLaMP product reviews with Qwen2.5-1.5B-Instruct (93 users after removing 7 whose target appears near-verbatim in the profile; one seed). Dopamine reaches 0.145 ROUGE-L against 0.108 for the frozen model, 0.130 for BM25 retrieval of four profile items, 0.118 for StyleVector, and 0.129 for a pooled vector and for another user's vector (each of these paired CIs excludes zero); prompt tuning of the same size is level (0.142), and per-user LoRA-r8, with 215x the parameters, reaches 0.153 (diff -0.008 [-0.014, -0.001]). Asking the frozen model for the user's median review length reaches 0.135 (diff +0.010 [0.001, 0.019]) and a higher METEOR, so part of the gain is output length. Dopamine gains less with Qwen2.5-7B-Instruct, where it beats the frozen model only on ROUGE-L, and BM25 retrieval and StyleVector are level with it on ROUGE-L and lead it on ROUGE-1 (BM25 also on METEOR), and on topic writing, where with the 7B model it falls below the frozen model on ROUGE-1 and METEOR.
  • Implicit preference inference, and conflicts between a stored preference and an explicit instruction, are not covered by our tasks; Sec 8 lists the first as a limitation.

** (3/4)**

W4a. Fixed optimization budgets. We selected learning rate and steps for both methods on validation users (App C).

  • Synthetic (LoRA 5 learning rates x 4 step counts, Dopamine 3 x 4, 30 validation users; test 100 users x 3 seeds): tuned Dopamine 0.943, tuned LoRA-r8 0.944, diff -0.001 [-0.002, +0.000], well inside the equivalence margin. LoRA is insensitive to the grid: 17 of its 20 configurations score 0.941-0.943 on validation, and the submitted default (lr 5e-4, 60 steps) is one of four tied for the best (0.943; the tie-break on validation CE picked lr 1e-4 with 120 steps). Tuning helped Dopamine (+0.004 [+0.002, +0.006]), not LoRA (-0.001 [-0.002, +0.000]). At the default protocol the numbers are 0.939 vs 0.945 (diff -0.006, also inside the margin), at 141x fewer parameters per user.
  • Amazon (LoRA lr 1e-4 to 2e-3, Dopamine 3e-3 to 3e-2, 30-240 steps, 50 validation users): the validation surface is flat and both selected configurations score below their defaults on test; tuned diff -0.015 [-0.031, +0.001]. That is not significant, but its 90% CI reaches past -0.02, so we do not claim equivalence on Amazon.
  • Review language modeling (App B.4): here the fixed budget did hurt LoRA. At the default 60 steps, LoRA-r8's held-out CE (5.394) is above the frozen model's (3.506), the pattern of the submitted appendix (5.41 vs 3.52 there); with learning rate and steps selected on validation users it reaches 2.865 (same learning rate, 15 steps), and Dopamine 2.904.
  • Every other trained baseline in Table 1 received its own validation learning-rate sweep.

W4b. Parameter savings vs training and inference cost. We measured both on one B200 (Sec 5.3, Sec 8, App F).

  • Storage, where the saving carries over in full: a million users need 61.4 GB of Dopamine states on SmolLM2-360M against 8.68 TB for LoRA-r8, or 128 MB as 64-dim codes (Sec 8).
  • Training (60 steps): Dopamine takes about half of LoRA-r8's wall-clock at each of the four sizes timed (SmolLM2 4.07 vs 7.53 s; Qwen-7B 3.26 vs 6.65 s; Qwen-14B 5.26 vs 11.29 s), with optimizer state of 0.25-2.0 MB vs 35-275 MB. Peak memory differs little (16.0 vs 17.5 GiB at 7B), because both backpropagate through the frozen network.
  • Inference with 64 distinct users per batch (HF, eager; Fig 2c, Table 13): Dopamine decodes at 0.97 (SmolLM2) and 0.96 (Qwen-7B) of the frozen model's throughput; per-row LoRA with batched matrix products reaches 0.69 and 0.63, and merged LoRA served one user at a time 0.017 and 0.015. In vLLM with its multi-LoRA kernels, 64 LoRA users decode at 0.21 and 0.12 of vLLM's own frozen throughput. We did not port Dopamine to vLLM, so a same-engine comparison is missing; in absolute tokens/s, vLLM multi-LoRA is faster than our HF Dopamine on SmolLM2 (4,678 vs 3,073; bf16 vs fp32) and slower on Qwen-7B (1,948 vs 4,069).
  • A mixed-user batch is numerically equivalent to single-user execution in fp32 (max logit difference 6.6e-4, 32/32 identical generations).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.