acceptodds
Under review as a conference paper at ICLR 2027

Personalize at Test Time: Learning User Preferences for Image Generation

Abstract

Diffusion models can generate high-quality images, yet aligning their outputs with individual user preferences remains challenging. A key bottleneck is accurately modeling diverse user preferences from limited feedback. Existing approaches often rely on labor-intensive manual preference annotations or vision-language models(VLM) to extract preference information from user interaction histories, introducing substantial annotation or computational costs that limit scalability. We propose an approach that learns personalized reward models directly from users’ historical image preference pairs. First, we use an autoencoder to compress hundreds of visual attributes into 50 attribute-anchored preference dimensions and train an evaluator to score images along these dimensions. We then represent each user’s preferences as a linear combination of the shared dimension scores, estimating the user-specific weights by maximizing the likelihood of their observed pairwise preferences under the Bradley–Terry model. This formulation reduces per-user adaptation to optimizing a low-dimensional weight vector, simplifying optimization and enabling data-efficient personalization from sparse feedback. The learned personalized rewards guide image generation at inference time while keeping the diffusion model frozen. Experiments on real-user preference data show that our approach achieves approximately 77% held-out pairwise preference prediction accuracy and improves the alignment of generated images with individual user preferences.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Model, Personalization, Text-to-Image

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jedGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents PAT, a framework for personalizing text-to-image diffusion models using offline pairwise user preferences. The method first learns a shared low-dimensional visual attribute representation by compressing VLM-annotated attributes with an autoencoder. A differentiable evaluator is then trained to predict scores along these dimensions. Each user's preferences are modeled as a linear combination of the shared scores, with user-specific weights estimated using a Bradley–Terry objective. The authors further introduce alternating evaluator-weight optimization and mixed-effects modeling. Finally, the personalized reward guides a frozen diffusion model through Direct Noise Optimization (DNO).

Critical strengths
  1. Clear motivation and formulation. Personalizing image generation from limited user feedback is an important problem. The different components of the pipeline are clearly organized.
  2. Useful ablation studies. The paper examines alternating optimization, mixed-effects modeling, latent dimensionality, and regularization strength. These experiments help clarify the contribution of individual components.
  3. The factorization of personalized rewards into shared evaluation features and user-specific weights is intuitive and computationally convenient for preference estimation.
Critical weaknesses
  1. PAT seems like a combination of existing methods. The personalized reward formulation is closely related to existing methods, such as RFM [1] and PIPE [2], which also model individual preferences through linear combinations of reward features. The remaining components are all established techniques.
  2. According to Section 4.4, the reproduced PPD method uses only four historical preference pairs per user, whereas PAT uses more preference data. Although PPD emphasizes its ability to achieve competitive results with limited data, this does not mean that it can only use a small amount of data. For a fair comparison, the authors should evaluate both methods using the same number of historical preference pairs.
  3. Although PAT does not require fine-tuning the pretrained diffusion model, it relies on DNO. According to Appendix D, PAT performs 30 optimization steps per inference, each requiring 30 denoising steps. This makes inference at least 30 times more time-consuming than using the base model, even before accounting for the additional computational cost of backpropagation. Given such a high inference cost, why not simply generate dozens of images and select the one with the highest score using a Personalized Reward Model? Alternatively, why not consider methods that require only lightweight training?

[1] Barreto A, Dumoulin V, Mao Y, et al. Capturing individual human preferences with reward features[J]. Advances in Neural Information Processing Systems, 2026, 38: 82119-82154. [2] Ryu M, Hsu C W, Saha A, et al. PIPE: Personalized Image-generation via Preference Encoding[C]//Pluralistic Alignment Workshop at ICML 2026.

Questions for authors
  1. What is the generation win rate of DNO using the universal reward vector, with the same diffusion model and optimization budget as PAT?
3 comments

Response

We thank the reviewer for the constructive feedback and for highlighting relevant prior work and alternative inference strategies. We discuss these points below.

W1: Relationship to RFM and PIPE

“The personalized reward formulation is closely related to existing methods, such as RFM [1] and PIPE [2], which also model individual preferences through linear combinations of reward features.”

Thank you for highlighting these works. PAT's contribution is a visual preference-modeling framework that learns a reusable personalized reward. RFM is already cited in Section 2. Compared with PAT, RFM[1] learns shared reward features and user-specific coefficients, with experiments on language-model preferences. PAT mainly works on image domain, constructs an attribute-grounded visual scoring evaluator, trains a shared differentiable image evaluator, and estimates each user's weights from image-pair feedback. While both approaches shared linear parameterization, PAT develops the visual representation and reward-learning pipeline needed to apply it to personalized image generation.

PIPE [2] also models preferences through coefficients over shared base user types, and uses the resulting user embedding to condition generation through an IP-Adapter trained with Diffusion-DPO. PAT produces an explicit, differentiable image-scoring reward that can be used independently of a generator-side conditioning module. It fits user-specific reward weights and applies that reward to a frozen generator, without training or injecting a user-conditioning adapter. The same reward can score sampled candidates or supply gradients for noise optimization, separating preference estimation from the choice of generation procedure.

We will update the related-work discussion and clarify the scope of the formulation claim in the Introduction accordingly.

W2: Using the preference history available to each user

Thank you for raising this point. We would like to clarify that PAT's advantage here is its ability to use preference information that remains unused by PPD's four-pair profiling setup. PPD [3] specifies N = 4 preference examples for constructing the VLM-based user representation, which our reproduction follows. Even when a user has a longer history, this setup uses only four pairs to form the profile. This matters in our evaluation: more than 60% of the 631 users in Table 2 in the paper have at least 61 training pairs. These users have already provided the feedback, and PAT is designed to put it to use.

The difference lies in how additional history can be incorporated. PPD's profiling prompt jointly processes the preferred and dispreferred images from each pair [3]. Extending this procedure adds two image inputs per pair, so the multimodal context grows with the history. PAT instead evaluates images independently and can cache their 50-dimensional scores. Each additional pair contributes its relative score difference to the Bradley–Terry objective in Equation (6), while the user representation remains 50-dimensional. Both positive and negative preference information therefore contribute to the reward without requiring the full image history in one VLM prompt. Being able to use more of the user's existing feedback is itself a practical benefit enabled by PAT's design.

Following your suggestion, we tested whether incorporating more pairs actually improves preference prediction. We added a controlled study on 391 real users with at least 61 training pairs. We fit each user's λ using 4, 8, or 16 randomly sampled training pairs. For each budget, we average over five random draws per user and then across users. Every budget is evaluated on the same 50 fixed held-out pairs per user as the Table 2 protocol, so only the feedback used for fitting λ varies. At zero pairs, predictions use G directly.

Table R1. Held-out preference-prediction accuracy as the user-weight fitting budget increases, with the paper's full-history result for reference.

Feedback used to fit user weightsHeld-out accuracy
4 pairs70.5%
8 pairs71.5%
16 pairs72.7%
Full history (61+ group; paper Table 2 reference)78.2%

The controlled results show a consistent increase in held-out accuracy, achieving 70.5%, 71.5%, and 72.7% with four, eight, and sixteen pairs. This is evidence of empirical scaling with preference data: more feedback improves user-weight estimation for the same users on the same test pairs, reaching a gain of 2.9% over the population prior at sixteen pairs. The controlled trend directly supports the advantage motivating PAT: additional choices already recorded by the user can translate into a more accurate personalized reward.

There is a further reason why using the available history matters: a PPD profile inferred from four pairs can depend strongly on which four pairs are selected. To examine this sensitivity, we studied three real users (1967, 6022, and 6043), generating three profiles per user from disjoint sets of four preference pairs. Within each draw, the pairs have distinct captions, and caption sets do not overlap across draws for the same user. We used the original PPD profiling prompt [3] and supplied identical sampled pairs to LLaVA-OneVision and GPT-5.5. For example, two GPT-5.5 profiles for user 6022 emphasize different aesthetics:

Draw 0: “charming stylization over excessive realism, clutter, or hyper-detailed technical rendering.”

Draw 1: “hyperrealism, intricate detail, clean composition, dramatic lighting, strong textures, and visually striking central subjects.”

Table R2. Textual profile similarity across preference-history subsets and users.

Profiling modelText similarity metricSame user, different subsetsDifferent users
LLaVA-OneVisionTF-IDF cosine0.107 ± 0.0490.124 ± 0.075
LLaVA-OneVisionJaccard0.112 ± 0.0520.123 ± 0.072
GPT-5.5TF-IDF cosine0.146 ± 0.0480.114 ± 0.045
GPT-5.5Jaccard0.154 ± 0.0320.131 ± 0.039

The contrasting descriptions show how selecting different four-pair subsets can change the inferred profile of the same user. The lexical measurements in Table R2 further illustrate this sensitivity: for LLaVA-OneVision, mean similarity between profiles of the same user from different subsets is lower than between-user similarity under both metrics. This makes the choice of a small history subset consequential for PPD's user representation. It also reinforces the motivation for PAT: by incorporating the available pairwise feedback directly, PAT gives each recorded preference a role in estimating the user's reward, making use of information beyond any single four-pair subset.

W3 & Q1: Best-of-N selection and flexible use of the personalized reward

Thank you for this helpful suggestion. We agree with the reviewer that using a personalized reward to select among generated candidates is a natural and practical approach. Your suggestion also helps us highlight an aspect of PAT: the learned personalized reward can support different generation procedures. PAT constructs the reward in two stages: it first learns a shared scoring space and evaluator, then fits each user's weights from historical pairwise feedback. The resulting reward can be reused across generation procedures while keeping the diffusion model frozen. We will also highlight this in our revised manuscript.

Following the reviewer's suggestion, we now demonstrate two uses of the same learned reward: DNO, which optimizes initial noise using reward gradients, and best-of-8, which scores independently sampled candidates. To evaluate these two applications and assess the statistical significance of personalization gains, we conducted an additional controlled study with simulated users. Recruiting users and collecting fresh judgments was not feasible within the short rebuttal period. In this experiment, the evaluator is the simulated user itself, using its originally assigned profile both to provide preference feedback and to judge new images. This directly evaluates alignment with the target simulated user's preferences without requiring a separate evaluator to infer that user's profile. Specifically, GPT-6 generated ten preference profiles, each instantiated using GPT-5.5 and GPT-4o, giving twenty simulator–profile instances. Each simulator supplied feedback on twenty SD3.5-generated image pairs across twenty training captions. We fitted the user weights from simulated feedback and evaluated new generations on ten held-out captions using the same simulator and its original profile. The universal and shuffled controls select from the same eight candidates as the personalized best-of-8 method. For DNO, we also compare the target user's own weights with the universal vector G under the same diffusion model and optimization budget.

Table R3. Simulated-user evaluation of the personalized reward.

ComparisonMean win rate [95% CI]t-test p
PAT own λ best-of-8 vs. random68.3% [61%, 75%]0.0001
PAT own λ DNO vs. random63.5% [49%, 77%]0.041
Universal λ DNO vs. random47.7% [37%, 59%]0.65
Universal λ best-of-8 vs. random56.8% [50%, 64%]0.037
Shuffled λ best-of-8 vs. random55.8% [49%, 63%]0.060
PAT own λ vs. universal λ (best-of-8)63.2% [59%, 67%]<0.0001
PAT own λ vs. universal λ (DNO)67.5% [60%, 75%]0.0001
PAT own λ vs. shuffled λ (best-of-8)61.3% [57%, 66%]0.0001
PPD-inferred profile vs. random56.0% [45%, 68%]0.163
PAT best-of-8 vs. PPD-inferred profile56.5% [44%, 69%]0.168

The personalized reward supports preference gains with both inference procedures. PAT achieves a 63.5% win rate against random generation through DNO (p = 0.041) and 68.3% through best-of-8 (p = 0.0001). These results demonstrate two effective applications of the learned reward; they are separate comparisons against random generation and do not establish a direct ranking between the inference procedures. For DNO, the target user's own weights directly outperform universal weights with a 67.5% win rate (p = 0.0001) under the same diffusion model and optimization budget. For best-of-8, with all reward configurations selecting from the same eight images, the target user's own weights win 63.2% against universal weights (p < 0.0001) and 61.3% against shuffled weights (p = 0.0001). These controls show that personalization improves both DNO and best-of-8, with the model and optimization budget matched for DNO and the candidate images shared for best-of-8.

Applicability when generator training is restricted. We agree that lightweight fine-tuning is a useful option when adapting the generator is feasible. In practical deployments, however, the generator may be exposed only through an inference API, or the production service may not allow custom weights or adapters, so even lightweight generator adaptation cannot be deployed. PAT learns personalization outside the generator, allowing the learned reward to be used under these constraints. Best-of-N selection requires only generated images and reward scoring, while DNO provides another option when access to generator gradients is available. This extends the applicability of the personalized reward to deployment settings where updating the generator is not an option.

[1]: Capturing Individual Human Preferences with Reward Features.

[2]: PIPE: Personalized Image-generation via Preference Encoding.

[3]: Personalized Preference Fine-tuning of Diffusion Models.

Reviewer_UisXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PAT, a framework for user-specific personalization from limited feedback. The method learns personalized reward functions in a compressed, low-dimensional preference space and uses these rewards to guide image generation at test time, without fine-tuning the diffusion model for each user. The authors evaluate the method through real-user preference prediction and personalized image generation experiments.

Critical strengths
  1. Clear motivation and a practically meaningful formulation. Decomposing personalized generation into learning shared evaluation dimensions and fitting user-specific weights provides a useful approach to modeling preferences from limited feedback.
  2. Experiments using real users' recorded preferences in Pick-a-Pic v1, together with multiple component-level ablations, provide empirical evidence supporting the method's effectiveness.
Critical weaknesses
  1. Insufficient evidence of improved user satisfaction with generated images. The 63.0% win rate reported in Section 4.4 is based on GPT-5.5 judgments conditioned on automatically constructed user profiles, rather than users' own choices among newly generated images. Although the historical preference-prediction experiments use real user feedback, they do not substitute for direct validation of the generated outputs. The paper does not report agreement between its profile-based evaluation and actual user judgments.
  2. Insufficiently matched feedback and inference-time computational budgets limit the attribution of performance gains. In Section 4.2, PAT learns from each user’s remaining preference pairs after holding out the test set. However, in Section 4.4, PPD randomly selects only four historical prompt–image records to extract user preferences, and Profile-as-Prompt reuses the resulting profile. Without comparisons under matched historical-feedback budgets, it is difficult to determine whether the reported advantages arise from the modeling approach or access to more user feedback.
Questions for authors
  1. Validation through a user study. I recommend conducting a user study to assess whether the model-based evaluation agrees with real users' preferences over the generated images.
  2. Comparison under matched feedback budgets. In Section 4.4, PPD uses four historical prompt–image records, whereas PAT fits user preferences using more historical preference pairs. Could the authors repeat the comparison using the same historical data source and matched amounts of feedback, and clarify whether all methods have access to both positive and negative preference information? Additionally, could the authors report PAT's direct pairwise win rate against PPD, together with user-level confidence intervals, rather than only reporting each method's win rate against vanilla SD3.5?
  3. Separating personalization gains from additional optimization. Table 3 includes a shared-weight ablation for preference prediction, but the generation experiments lack a corresponding comparison. Under the same Direct Noise Optimization (DNO) budget, could the authors compare generation using user-specific rewards, a shared population-level reward, and user-weight vectors randomly reassigned across users? Please also report per-image generation time, peak GPU memory usage, and the number of optimization steps required to achieve different preference win rates, to quantify the performance–cost trade-off.
5 comments

Response

We thank the reviewer for the careful assessment and concrete suggestions. We have added validation against real user choices and controlled comparisons of user-specific, universal, and reassigned rewards, and clarify the role of feedback and inference budgets below.

W1, Q1 & Q3: Evaluation reliability and the contribution of personalization

Thank you for these constructive suggestions on validating the evaluation, isolating personalization gains, and quantifying computational cost. They motivated the additional experiments and analysis below.

Validation against real user preferences. Following the reviewer’s suggestion, we evaluated GPT-5.5 on held-out Pick-a-Pic comparisons against the users’ recorded choices, using profiles inferred from separate historical preference pairs. The judge achieves 67.8% agreement with actual user preferences. For context, the Pick-a-Pic paper [1] reports 68.0% accuracy for human expert annotators predicting the original users’ choices (Table 1). Our result is numerically close to that reference, supporting the judge’s usefulness for this challenging task; the two figures come from different evaluation samples and protocols.

Direct evaluation by the simulated user. Recruiting users and collecting fresh judgments on generated images was not feasible within the short rebuttal period. We therefore complemented this held-out human-preference validation with a controlled simulated-user study. The simulated setting allows us to evaluate personalization directly from the target user’s perspective: the evaluator is the simulated user itself, using its original assigned profile both to provide preference feedback and to judge new outputs. In our setup, GPT-6 generated ten preference profiles, each instantiated using GPT-5.5 and GPT-4o, giving twenty simulator–profile instances. Each simulator supplied feedback on twenty SD3.5-generated image pairs across twenty training captions. We fitted the user weights from simulated feedback and evaluated new generations on ten held-out captions. We implemented all three requested reward configurations: the target user’s own weights, a universal population-level vector, and another user’s reassigned weights. We tested two generation procedures using the learned reward: DNO and best-of-8 selection. For DNO, we compare own and universal weights using the same diffusion model and optimization budget. For best-of-8, all three reward configurations select from the same eight candidates.

Table R1. Simulated-user evaluation with personalized, universal, and reassigned reward weights.

ComparisonMean win rate [95% CI]t-test p
PAT own λ best-of-8 vs. random68.3% [61%, 75%]0.0001
PAT own λ DNO vs. random63.5% [49%, 77%]0.041
Universal λ DNO vs. random47.7% [37%, 59%]0.65
Universal λ best-of-8 vs. random56.8% [50%, 64%]0.037
Shuffled λ best-of-8 vs. random55.8% [49%, 63%]0.060
PAT own λ vs. universal λ (best-of-8)63.2% [59%, 67%]<0.0001
PAT own λ vs. universal λ (DNO)67.5% [60%, 75%]0.0001
PAT own λ vs. shuffled λ (best-of-8)61.3% [57%, 66%]0.0001
PPD-inferred profile vs. random56.0% [45%, 68%]0.163
PAT best-of-8 vs. PPD-inferred profile56.5% [44%, 69%]0.168

Correctly matched user weights make a significant difference. With best-of-8, own weights win 63.2% against universal weights (p < 0.0001) and 61.3% against shuffled weights (p = 0.0001). Both comparisons exceed 50% in seventeen of the twenty simulator–profile instances. PAT also achieves 68.3% against random generation (p = 0.0001), while the universal and shuffled controls achieve lower mean win rates of 56.8% and 55.8%, respectively. These results directly support the contribution of personalization beyond the opportunity to select among multiple candidates. The DNO control leads to the same conclusion: under the same diffusion model and optimization budget, the target user's own weights achieve a 67.5% direct win rate against universal weights (p = 0.0001). Thus, personalization improves both DNO and best-of-8. We also added a direct comparison between PAT best-of-8 and the PPD-inferred-profile baseline, which uses the inferred profile in the generation prompt. PAT achieves a 56.5% mean win rate in this comparison (95% CI: [44%, 69%]; Table R1).

W1, Q1 & Q3: Evaluation reliability and the contribution of personalization(Cont'd)

Computational cost. On an NVIDIA RTX A5000, we estimate that PAT + DNO at 512 × 512 takes approximately 252 seconds (4.2 minutes) per output image, compared with 3.7 seconds for vanilla SD3.5 at the same resolution, or approximately 68× the vanilla latency. The DNO configuration uses 30 optimization iterations. Peak allocated GPU memory during optimization and rendering is approximately 9.4 GiB, measured after prompt encoding and removal of the text encoders; the latency estimate likewise excludes model loading and prompt encoding.

Preference win rate versus optimization budget. To examine the performance–cost trade-off requested by the reviewer, we selected two simulated users and varied the number of DNO optimization iterations, with GPT-5.5 evaluating ten held-out captions per user. We report the mean win rate against random generation, counting ties as half a win, and estimated time per image at each optimization budget.

Table R2. DNO optimization budget, mean preference win rate, and computational cost for two simulated users at 512 × 512 on an NVIDIA RTX A5000.

DNO optimization iterationsMean win rateEstimated time per image
567.5%~45 s
1072.5%~87 s
1570.0%~129 s
2077.5%~171 s
2582.5%~212 s
30 (paper setting)82.5%~254 s

The mean win rate shows an overall upward trend as the optimization budget increases. Across these two simulated users, it rises from 67.5% at five iterations to 77.5% at twenty iterations and 82.5% at twenty-five iterations, an improvement of 15 percentage points over the five-iteration setting. The small dip at fifteen iterations is followed by gains at twenty and twenty-five iterations. This pattern supports the benefit of additional DNO optimization for aligning generated images with the simulated users' preferences.

W2 & Q2: Feedback budgets and using longer histories

Thank you for raising this point. We would like to clarify that PAT's advantage here is its ability to use preference information that remains unused by PPD's four-pair profiling setup. PPD specifies N = 4 preference examples for constructing the VLM-based user representation, which our reproduction follows. Even when a user has a longer history, this setup uses only four pairs to form the profile. This matters in our evaluation: more than 60% of the 631 users in Table 2 in the paper have at least 61 training pairs. These users have already provided the feedback, and PAT is designed to put it to use.

The difference lies in how additional history can be incorporated. PPD's profiling prompt jointly processes the preferred and dispreferred images from each pair. Extending this procedure adds two image inputs per pair, so the multimodal context grows with the history. PAT instead evaluates images independently and can cache their 50-dimensional scores. Each additional pair contributes its relative score difference to the Bradley–Terry objective in Equation (6), while the user representation remains 50-dimensional. Both positive and negative preference information therefore contribute to the reward without requiring the full image history in one VLM prompt. Being able to use more of the user's existing feedback is itself a practical benefit enabled by PAT's design.

Following your suggestion, we tested whether incorporating more pairs actually improves preference prediction. We added a controlled study on 391 real users with at least 61 training pairs. We fit each user's λ using 4, 8, or 16 randomly sampled training pairs. For each budget, we average over five random draws per user and then across users. Every budget is evaluated on the same 50 fixed held-out pairs per user as the Table 2 protocol, so only the feedback used for fitting λ varies. At zero pairs, predictions use G directly.

Table R3. Held-out preference-prediction accuracy as the user-weight fitting budget increases, with the paper's full-history result for reference.

Feedback used to fit user weightsHeld-out accuracy
4 pairs70.5%
8 pairs71.5%
16 pairs72.7%
Full history78.2%

The controlled results show a consistent increase in held-out accuracy, achieving 70.5%, 71.5%, and 72.7% with four, eight, and sixteen pairs. This is evidence of empirical scaling with preference data: more feedback improves user-weight estimation for the same users on the same test pairs, reaching a gain of 2.9% over the population prior at sixteen pairs. The controlled trend directly supports the advantage motivating PAT: additional choices already recorded by the user can translate into a more accurate personalized reward.

There is a further reason why using the available history matters: a PPD profile inferred from four pairs can depend strongly on which four pairs are selected. To examine this sensitivity, we studied three real users (1967, 6022, and 6043), generating three profiles per user from disjoint sets of four preference pairs. Within each draw, the pairs have distinct captions, and caption sets do not overlap across draws for the same user. We used the original PPD profiling prompt and supplied identical sampled pairs to LLaVA-OneVision and GPT-5.5. For example, two GPT-5.5 profiles for user 6022 emphasize different aesthetics:

Draw 0: “charming stylization over excessive realism, clutter, or hyper-detailed technical rendering.”

Draw 1: “hyperrealism, intricate detail, clean composition, dramatic lighting, strong textures, and visually striking central subjects.”

Table R4. Textual profile similarity across preference-history subsets and users.

Profiling modelText similarity metricSame user, different subsetsDifferent users
LLaVA-OneVisionTF-IDF cosine0.107 ± 0.0490.124 ± 0.075
LLaVA-OneVisionJaccard0.112 ± 0.0520.123 ± 0.072
GPT-5.5TF-IDF cosine0.146 ± 0.0480.114 ± 0.045
GPT-5.5Jaccard0.154 ± 0.0320.131 ± 0.039

The contrasting descriptions show how selecting different four-pair subsets can change the inferred profile of the same user. The lexical measurements in Table R4 further illustrate this sensitivity: for LLaVA-OneVision, mean similarity between profiles of the same user from different subsets is lower than between-user similarity under both metrics. This makes the choice of a small history subset consequential for PPD's user representation. It also reinforces the motivation for PAT: by incorporating the available pairwise feedback directly, PAT gives each recorded preference a role in estimating the user's reward, making use of information beyond any single four-pair subset.

[1]: Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation.

Reviewer_UAH4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors address a key issue on modeling individual user preferences with limited feedback. To model the preferences with limited samples, the authors propose using an autoencoder that's anchored on 512 attributes using Qwen2.5 as the annotator. The attributes include visual categories including art styles, colors, compositions and mood. Then authors propose mathematical formulations to calculate a user specific weights by maximizing the alignment between user preferences and the 50-dim encoded vector. After that, the reward model is used for test-time generation using DNO method. The authors use Pick-a-pic dataset and evaluate on 631 users' data on SD3.5. The experiments mainly evaluate two things - whether the personal reward modelling is accurate and whether the DNO output is actually preferred by the user.

Critical strengths

I believe the strength is the proposed autoencoder method of reward modelling for personal preference alignment. The attribute supervision is human-annotation free and provides a solid basis for preference learning. the 50-dim method reduces the need of a big labelled dataset. The empirical results ensure the method is working.

Critical weaknesses

Main weakness I noticed is that the validity of the autoencoder needs more analysis. In Table 6, PCA achieves more or less the same performance compared to the autoencoder in all the k values. Given that, we need a better reason the validity of the autoencoder method and without that the paper is incomplete.

The results section shows minor improvements and needs more rigorous evaluation. First the GPT5.5 and the method of evaluation should be validated first as a useful judge using Pick-a-pic preferences (You can use this by doing the same evaluation for one or few users using held out set for each user). Given that break-even result is 50%, the improvement over PPD seems minor. Also, a human evaluation is very much needed in a preference alignment study.

Questions for authors

Provide satisfactory answers for the mentioned weaknesses

2 comments

Response

We thank the reviewer for the thoughtful feedback and constructive suggestions. We address these points below with additional experiments and clarifications.

W1: Autoencoder needs more analysis

“In Table 6, PCA achieves more or less the same performance compared to the autoencoder in all the k values.”

Thank you for highlighting the need to better justify the choice of the autoencoder. We provide additional experimental results and clarify its role in our framework. We expanded the comparison between the autoencoder, PCA, and correlation-based dimensionality reduction. The held-out pairwise preference prediction accuracies (%) are reported below(results from 3 seeds):

Table R1. Held-out preference prediction accuracy (%) for alternative dimension-reduction methods.

kCorrelationPCAAutoencoder
2075.24 ± 0.1574.66 ± 0.1575.58 ± 0.18
3075.34 ± 0.2975.02 ± 0.5276.19 ± 0.43
4075.33 ± 0.3076.09 ± 0.2876.77 ± 0.16
5075.47 ± 0.3176.35 ± 0.2677.26 ± 0.18

The autoencoder achieves the highest mean accuracy at every tested dimensionality, outperforming PCA by 0.68–1.17%. At our default setting of k=50, the improvement over PCA is 0.91 %. These results provide empirical support for our choice of the autoencoder.

We also clarify that the autoencoder is one choice for constructing the shared representation, rather than a requirement of PAT. Our main contribution is the personalized reward-modeling framework and its end-to-end pipeline: learning a compact, attribute-grounded scoring space shared across users, estimating each user’s preference weights directly from historical pairwise feedback, and using the resulting personalized reward to guide a frozen diffusion model. This formulation reduces per-user adaptation to fitting a low-dimensional vector and accommodates different methods for constructing the shared representation.

The competitive performance of PCA therefore demonstrates that PAT remains effective with an alternative representation, while the autoencoder provides a consistent empirical improvement in our experiments. We will revise the manuscript to make this distinction explicit and incorporate the additional results.

W2: Judge validation and evidence of personalized generation gains

Thank you for these suggestions. We added two complementary evaluations: validation against held-out human choices and a controlled study in which the simulated user directly evaluates the generated images.

Validation against real user preferences. Following the reviewer’s suggestion, we evaluated GPT-5.5 on held-out Pick-a-Pic comparisons against the users’ recorded choices, using profiles inferred from separate historical preference pairs. The judge achieves 67.8% agreement with actual user preferences. For context, the Pick-a-Pic paper [1] reports 68.0% accuracy for human expert annotators predicting the original users’ choices (Table 1). Our result is numerically close to that reference, supporting the judge’s usefulness for this challenging task; the two figures come from different evaluation samples and protocols.

Direct evaluation by the simulated user. To further assess the statistical significance of PAT’s personalization gains, we conducted an additional controlled experiment with simulated users. The simulated setting nevertheless allows us to evaluate personalization directly from the target user’s perspective: the evaluator is the simulated user itself, using its original assigned profile both to provide preference feedback and to judge new outputs. This removes the need for a separate evaluator to infer the target user’s preferences within the controlled setting. Specifically, GPT-6 generated ten preference profiles, each instantiated using GPT-5.5 and GPT-4o, giving twenty simulator–profile instances. Each simulator supplied feedback on twenty SD3.5-generated image pairs across twenty training captions. We fitted the user weights from simulated feedback and evaluated new generations on ten held-out captions using the same simulator and its original profile. We tested two generation procedures using the learned reward: DNO and best-of-8 selection. We compare personalized and universal rewards under both procedures, using the same diffusion model and optimization budget for DNO. Best-of-8 additionally includes shuffled user weights, with all reward configurations selecting from the same eight candidates. All results below use the same twenty simulator–profile instances.

Table R2. Simulated-user evaluation of personalized image generation.

ComparisonMean win rate [95% CI]t-test p
PAT own λ best-of-8 vs. random68.3% [61%, 75%]0.0001
PAT own λ DNO vs. random63.5% [49%, 77%]0.041
Universal λ DNO vs. random47.7% [37%, 59%]0.65
Universal λ best-of-8 vs. random56.8% [50%, 64%]0.037
Shuffled λ best-of-8 vs. random55.8% [49%, 63%]0.060
PAT own λ vs. universal λ (best-of-8)63.2% [59%, 67%]<0.0001
PAT own λ vs. universal λ (DNO)67.5% [60%, 75%]0.0001
PAT own λ vs. shuffled λ (best-of-8)61.3% [57%, 66%]0.0001
PPD-inferred profile vs. random56.0% [45%, 68%]0.163
PAT best-of-8 vs. PPD-inferred profile56.5% [44%, 69%]0.168

Statistically supported gains. PAT with best-of-8 achieves a 68.3% win rate against random generation (p = 0.0001). For best-of-8, the direct comparisons against universal and shuffled weights are also significant (p < 0.0001 and p = 0.0001), strengthening the evidence that the gains reflect personalized preference learning. DNO achieves a mean win rate of 63.5% (p = 0.041), providing results for a second generation procedure. The personalization benefit holds for both DNO and best-of-8: directly comparing the target user's own weights against universal weights yields win rates of 67.5% (p = 0.0001) for DNO and 63.2% (p < 0.0001) for best-of-8.

[1]: Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.