- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents PAT, a framework for personalizing text-to-image diffusion models using offline pairwise user preferences. The method first learns a shared low-dimensional visual attribute representation by compressing VLM-annotated attributes with an autoencoder. A differentiable evaluator is then trained to predict scores along these dimensions. Each user's preferences are modeled as a linear combination of the shared scores, with user-specific weights estimated using a Bradley–Terry objective. The authors further introduce alternating evaluator-weight optimization and mixed-effects modeling. Finally, the personalized reward guides a frozen diffusion model through Direct Noise Optimization (DNO).
Critical strengths
- Clear motivation and formulation. Personalizing image generation from limited user feedback is an important problem. The different components of the pipeline are clearly organized.
- Useful ablation studies. The paper examines alternating optimization, mixed-effects modeling, latent dimensionality, and regularization strength. These experiments help clarify the contribution of individual components.
- The factorization of personalized rewards into shared evaluation features and user-specific weights is intuitive and computationally convenient for preference estimation.
Critical weaknesses
- PAT seems like a combination of existing methods. The personalized reward formulation is closely related to existing methods, such as RFM [1] and PIPE [2], which also model individual preferences through linear combinations of reward features. The remaining components are all established techniques.
- According to Section 4.4, the reproduced PPD method uses only four historical preference pairs per user, whereas PAT uses more preference data. Although PPD emphasizes its ability to achieve competitive results with limited data, this does not mean that it can only use a small amount of data. For a fair comparison, the authors should evaluate both methods using the same number of historical preference pairs.
- Although PAT does not require fine-tuning the pretrained diffusion model, it relies on DNO. According to Appendix D, PAT performs 30 optimization steps per inference, each requiring 30 denoising steps. This makes inference at least 30 times more time-consuming than using the base model, even before accounting for the additional computational cost of backpropagation. Given such a high inference cost, why not simply generate dozens of images and select the one with the highest score using a Personalized Reward Model? Alternatively, why not consider methods that require only lightweight training?
[1] Barreto A, Dumoulin V, Mao Y, et al. Capturing individual human preferences with reward features[J]. Advances in Neural Information Processing Systems, 2026, 38: 82119-82154. [2] Ryu M, Hsu C W, Saha A, et al. PIPE: Personalized Image-generation via Preference Encoding[C]//Pluralistic Alignment Workshop at ICML 2026.
Questions for authors
- What is the generation win rate of DNO using the universal reward vector, with the same diffusion model and optimization budget as PAT?
3 comments
W2: Using the preference history available to each user
Thank you for raising this point. We would like to clarify that PAT's advantage here is its ability to use preference information that remains unused by PPD's four-pair profiling setup. PPD [3] specifies N = 4 preference examples for constructing the VLM-based user representation, which our reproduction follows. Even when a user has a longer history, this setup uses only four pairs to form the profile. This matters in our evaluation: more than 60% of the 631 users in Table 2 in the paper have at least 61 training pairs. These users have already provided the feedback, and PAT is designed to put it to use.
The difference lies in how additional history can be incorporated. PPD's profiling prompt jointly processes the preferred and dispreferred images from each pair [3]. Extending this procedure adds two image inputs per pair, so the multimodal context grows with the history. PAT instead evaluates images independently and can cache their 50-dimensional scores. Each additional pair contributes its relative score difference to the Bradley–Terry objective in Equation (6), while the user representation remains 50-dimensional. Both positive and negative preference information therefore contribute to the reward without requiring the full image history in one VLM prompt. Being able to use more of the user's existing feedback is itself a practical benefit enabled by PAT's design.
Following your suggestion, we tested whether incorporating more pairs actually improves preference prediction. We added a controlled study on 391 real users with at least 61 training pairs. We fit each user's λ using 4, 8, or 16 randomly sampled training pairs. For each budget, we average over five random draws per user and then across users. Every budget is evaluated on the same 50 fixed held-out pairs per user as the Table 2 protocol, so only the feedback used for fitting λ varies. At zero pairs, predictions use G directly.
Table R1. Held-out preference-prediction accuracy as the user-weight fitting budget increases, with the paper's full-history result for reference.
| Feedback used to fit user weights | Held-out accuracy |
|---|---|
| 4 pairs | 70.5% |
| 8 pairs | 71.5% |
| 16 pairs | 72.7% |
| Full history (61+ group; paper Table 2 reference) | 78.2% |
The controlled results show a consistent increase in held-out accuracy, achieving 70.5%, 71.5%, and 72.7% with four, eight, and sixteen pairs. This is evidence of empirical scaling with preference data: more feedback improves user-weight estimation for the same users on the same test pairs, reaching a gain of 2.9% over the population prior at sixteen pairs. The controlled trend directly supports the advantage motivating PAT: additional choices already recorded by the user can translate into a more accurate personalized reward.
There is a further reason why using the available history matters: a PPD profile inferred from four pairs can depend strongly on which four pairs are selected. To examine this sensitivity, we studied three real users (1967, 6022, and 6043), generating three profiles per user from disjoint sets of four preference pairs. Within each draw, the pairs have distinct captions, and caption sets do not overlap across draws for the same user. We used the original PPD profiling prompt [3] and supplied identical sampled pairs to LLaVA-OneVision and GPT-5.5. For example, two GPT-5.5 profiles for user 6022 emphasize different aesthetics:
Draw 0: “charming stylization over excessive realism, clutter, or hyper-detailed technical rendering.”
Draw 1: “hyperrealism, intricate detail, clean composition, dramatic lighting, strong textures, and visually striking central subjects.”
Table R2. Textual profile similarity across preference-history subsets and users.
| Profiling model | Text similarity metric | Same user, different subsets | Different users |
|---|---|---|---|
| LLaVA-OneVision | TF-IDF cosine | 0.107 ± 0.049 | 0.124 ± 0.075 |
| LLaVA-OneVision | Jaccard | 0.112 ± 0.052 | 0.123 ± 0.072 |
| GPT-5.5 | TF-IDF cosine | 0.146 ± 0.048 | 0.114 ± 0.045 |
| GPT-5.5 | Jaccard | 0.154 ± 0.032 | 0.131 ± 0.039 |
The contrasting descriptions show how selecting different four-pair subsets can change the inferred profile of the same user. The lexical measurements in Table R2 further illustrate this sensitivity: for LLaVA-OneVision, mean similarity between profiles of the same user from different subsets is lower than between-user similarity under both metrics. This makes the choice of a small history subset consequential for PPD's user representation. It also reinforces the motivation for PAT: by incorporating the available pairwise feedback directly, PAT gives each recorded preference a role in estimating the user's reward, making use of information beyond any single four-pair subset.
W3 & Q1: Best-of-N selection and flexible use of the personalized reward
Thank you for this helpful suggestion. We agree with the reviewer that using a personalized reward to select among generated candidates is a natural and practical approach. Your suggestion also helps us highlight an aspect of PAT: the learned personalized reward can support different generation procedures. PAT constructs the reward in two stages: it first learns a shared scoring space and evaluator, then fits each user's weights from historical pairwise feedback. The resulting reward can be reused across generation procedures while keeping the diffusion model frozen. We will also highlight this in our revised manuscript.
Following the reviewer's suggestion, we now demonstrate two uses of the same learned reward: DNO, which optimizes initial noise using reward gradients, and best-of-8, which scores independently sampled candidates. To evaluate these two applications and assess the statistical significance of personalization gains, we conducted an additional controlled study with simulated users. Recruiting users and collecting fresh judgments was not feasible within the short rebuttal period. In this experiment, the evaluator is the simulated user itself, using its originally assigned profile both to provide preference feedback and to judge new images. This directly evaluates alignment with the target simulated user's preferences without requiring a separate evaluator to infer that user's profile. Specifically, GPT-6 generated ten preference profiles, each instantiated using GPT-5.5 and GPT-4o, giving twenty simulator–profile instances. Each simulator supplied feedback on twenty SD3.5-generated image pairs across twenty training captions. We fitted the user weights from simulated feedback and evaluated new generations on ten held-out captions using the same simulator and its original profile. The universal and shuffled controls select from the same eight candidates as the personalized best-of-8 method. For DNO, we also compare the target user's own weights with the universal vector G under the same diffusion model and optimization budget.
Table R3. Simulated-user evaluation of the personalized reward.
| Comparison | Mean win rate [95% CI] | t-test p |
|---|---|---|
| PAT own λ best-of-8 vs. random | 68.3% [61%, 75%] | 0.0001 |
| PAT own λ DNO vs. random | 63.5% [49%, 77%] | 0.041 |
| Universal λ DNO vs. random | 47.7% [37%, 59%] | 0.65 |
| Universal λ best-of-8 vs. random | 56.8% [50%, 64%] | 0.037 |
| Shuffled λ best-of-8 vs. random | 55.8% [49%, 63%] | 0.060 |
| PAT own λ vs. universal λ (best-of-8) | 63.2% [59%, 67%] | <0.0001 |
| PAT own λ vs. universal λ (DNO) | 67.5% [60%, 75%] | 0.0001 |
| PAT own λ vs. shuffled λ (best-of-8) | 61.3% [57%, 66%] | 0.0001 |
| PPD-inferred profile vs. random | 56.0% [45%, 68%] | 0.163 |
| PAT best-of-8 vs. PPD-inferred profile | 56.5% [44%, 69%] | 0.168 |
The personalized reward supports preference gains with both inference procedures. PAT achieves a 63.5% win rate against random generation through DNO (p = 0.041) and 68.3% through best-of-8 (p = 0.0001). These results demonstrate two effective applications of the learned reward; they are separate comparisons against random generation and do not establish a direct ranking between the inference procedures. For DNO, the target user's own weights directly outperform universal weights with a 67.5% win rate (p = 0.0001) under the same diffusion model and optimization budget. For best-of-8, with all reward configurations selecting from the same eight images, the target user's own weights win 63.2% against universal weights (p < 0.0001) and 61.3% against shuffled weights (p = 0.0001). These controls show that personalization improves both DNO and best-of-8, with the model and optimization budget matched for DNO and the candidate images shared for best-of-8.
Applicability when generator training is restricted. We agree that lightweight fine-tuning is a useful option when adapting the generator is feasible. In practical deployments, however, the generator may be exposed only through an inference API, or the production service may not allow custom weights or adapters, so even lightweight generator adaptation cannot be deployed. PAT learns personalization outside the generator, allowing the learned reward to be used under these constraints. Best-of-N selection requires only generated images and reward scoring, while DNO provides another option when access to generator gradients is available. This extends the applicability of the personalized reward to deployment settings where updating the generator is not an option.
[1]: Capturing Individual Human Preferences with Reward Features.
[2]: PIPE: Personalized Image-generation via Preference Encoding.
[3]: Personalized Preference Fine-tuning of Diffusion Models.
Response
We thank the reviewer for the constructive feedback and for highlighting relevant prior work and alternative inference strategies. We discuss these points below.
W1: Relationship to RFM and PIPE
Thank you for highlighting these works. PAT's contribution is a visual preference-modeling framework that learns a reusable personalized reward. RFM is already cited in Section 2. Compared with PAT, RFM[1] learns shared reward features and user-specific coefficients, with experiments on language-model preferences. PAT mainly works on image domain, constructs an attribute-grounded visual scoring evaluator, trains a shared differentiable image evaluator, and estimates each user's weights from image-pair feedback. While both approaches shared linear parameterization, PAT develops the visual representation and reward-learning pipeline needed to apply it to personalized image generation.
PIPE [2] also models preferences through coefficients over shared base user types, and uses the resulting user embedding to condition generation through an IP-Adapter trained with Diffusion-DPO. PAT produces an explicit, differentiable image-scoring reward that can be used independently of a generator-side conditioning module. It fits user-specific reward weights and applies that reward to a frozen generator, without training or injecting a user-conditioning adapter. The same reward can score sampled candidates or supply gradients for noise optimization, separating preference estimation from the choice of generation procedure.
We will update the related-work discussion and clarify the scope of the formulation claim in the Introduction accordingly.