acceptodds
Under review as a conference paper at ICLR 2027

PTERA: Probabilistic Text Encoder for Reward Alignment in Diffusion Models

Abstract

Fine-tuning text-to-image models toward human preference rewards has become a common approach for improving text alignment. Most existing methods, however, focus on updating the diffusion model parameters, while reward-based adaptation of the text encoder remains relatively underexplored despite promising reward gain and training efficiency. Moreover, existing text encoder fine-tuning methods typically optimize a single deterministic embedding for each prompt, which can be susceptible to reward over-optimization and provides no training mechanism for exploring alternative conditioning vectors. In this work, we propose **P**robabilistic **T**ext **E**ncoder for **R**eward **A**lignment (**PTERA**), a simple yet effective text encoder fine-tuning framework based on direct reward backpropagation. PTERA replaces deterministic conditioning with a trainable probability distribution over text embeddings and derives a tractable KL-regularized objective for learning this distribution while regularizing the divergence from a stochastic reference constructed around the pre-trained text embeddings. Compared with previous text encoder adaptation, PTERA exhibits more robust reward alignment. We further introduce **PTERA-TTA**, which exploits the text embedding distribution to perform trajectory-dependent test-time alignment.

Primary area:
generative models
Keywords:
Diffusion Models, Text to Image, Human preference reward

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oVTTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PTERA, which replaces deterministic text embeddings with a learnable Gaussian distribution for reward alignment while keeping the diffusion backbone frozen. The authors also introduce PTERA-TTA, which samples from the learned embedding distribution and selects better candidates based on reward feedback at inference time. Experiments on SD1.5, SDXL, and SD3 demonstrate improvements in preference alignment and training efficiency.

Critical strengths
  1. The idea is interesting. Learning a distribution over text embeddings provides more flexibility than optimizing a single deterministic embedding, and naturally supports test-time exploration.

  2. The method is lightweight and does not require updating the diffusion backbone. The experiments and ablation studies are fairly comprehensive, showing promising improvements on SD1.5 and SDXL with relatively low training cost.

Critical weaknesses
  1. The benefit of probabilistic modeling is not entirely convincing. In Table 6, the improvement over Learned-Iso is relatively small, making it unclear how much the learned coordinate-wise variance actually contributes.

  2. The gains are less consistent on stronger models. In Table 2, the improvement on SD3 is almost negligible, and PTERA-TTA does not consistently outperform PTERA-mean on SDXL. This raises some questions about the practical benefits of the method on more advanced backbones.

Questions for authors
  1. Could the authors further clarify the advantage of learning coordinate-wise variance over using simple isotropic Gaussian perturbations? The differences in Table 6 are quite small, and results across multiple runs would be helpful.

  2. Why are the improvements on SD3 much smaller than those on SD1.5 and SDXL? Could this be related to differences in text conditioning, training hyperparameters, or reward optimization?

3 comments

We appreciate the reviewer’s valuable questions, which helped us clarify and improve our work, and provide our responses below.

Q1 (W1).

Both Learned-Iso and PTERA learn probabilistic conditioning distributions using the preference reward. Their difference lies in the covariance parameterization. Learned-Iso uses a single scalar scale shared across embedding coordinates, whereas PTERA learns a separate variance for each coordinate. This additional flexibility allows PTERA to allocate different exploration magnitudes to different coordinates, rather than scaling perturbations uniformly. Although this provides a more expressive distribution family, expressivity alone does not guarantee better performance under practical optimization.

To assess its empirical contribution, we expanded Table 6 with two additional training seeds, 43 and 44, alongside the original seed 42, and evaluated all variants on both DrawBench and the held-out HPSv2 prompts. Each evaluation uses four generated images per prompt. The expanded table reports PickScore and HPSv2 scores, together with paired 95% bootstrap confidence intervals for ΔPS=PS(PTERA)−PS(variant)\Delta\mathrm{PS}=\mathrm{PS}(\mathrm{PTERA})-\mathrm{PS}(\mathrm{variant}), separately for each training seed and benchmark.

PTERA achieves the highest mean PickScore among all variants in all six seed–benchmark combinations. Specifically, its PickScore improvements over Learned-Iso are 0.0450, 0.1111, and 0.1230 on DrawBench, and 0.0040, 0.0094, and 0.2149 on the held-out HPSv2 prompts, for seeds 42, 43, and 44, respectively. The corresponding confidence intervals exclude zero for seed 44 on both benchmarks, while the other comparisons retain uncertainty. We therefore interpret these results as evidence of a consistent positive direction in the observed PickScore differences, with modest and setting-dependent gains.

Training seedVariantDB (in-domain) PS ↑DB (in-domain) HPSv2 ↑Paired 95% CI (ΔPS)HPSv2 (held-out) PS ↑HPSv2 (held-out) HPSv2 ↑Paired 95% CI (ΔPS)
42Mean-only22.2780.266[0.0291, 0.2125]21.8050.266[-0.0069, 0.1968]
42Fixed-Iso22.3170.267[-0.0242, 0.1621]21.8130.266[-0.0193, 0.1915]
42Learned-Iso22.3390.265[-0.0529, 0.1344]21.8960.263[-0.0987, 0.1082]
42PTERA22.3840.267—21.9000.269—
43Mean-only22.2580.260[-0.0150, 0.1581]21.7850.253[0.0011, 0.2208]
43Fixed-Iso22.2770.268[-0.0676, 0.1029]21.8650.266[0.0008, 0.0895]
43Learned-Iso22.2160.266[-0.0177, 0.1653]21.8850.271[-0.0305, 0.0578]
43PTERA22.3270.271—21.8940.270—
44Mean-only22.1690.260[0.0348, 0.3495]21.8780.260[-0.0340, 0.1671]
44Fixed-Iso22.2340.263[-0.0012, 0.2794]21.7590.267[0.0732, 0.2706]
44Learned-Iso22.2720.267[0.0103, 0.1951]21.7320.262[0.0925, 0.3401]
44PTERA22.3950.267—21.9470.271—

Q2 (W2).

SD3 training configuration. Each SD3 training run takes more than 24 hours in our setup, which limited our initial exploration of architectural and optimization configurations. In the additional experiment, we retained the same PTERA residual-module architecture, whose parameter count is approximately 0.4% of that of each corresponding frozen text encoder. We increased the learning rate from 3×10−73\times10^{-7} to 10−510^{-5}, changed (β,γ)(\beta,\gamma) from (10−3,10−3)(10^{-3},10^{-3}) to (0.1,0.1)(0.1,0.1), and reduced the initial reference scale ρ\rho from 0.10.1 to 0.050.05. The training reward-query budget remained unchanged at 20K.

SD3 BenchmarkMethodPickScore ↑ImageReward ↑HPSv2 ↑CLIP ↑Aesthetic ↑
DrawBenchBase22.4940.9260.28328.6765.955
DrawBenchPTERA-mean22.5470.9460.28428.6265.948
DrawBenchPTERA-sampled22.5050.9300.28328.6245.949
DrawBenchPTERA-TTA22.5730.9810.28428.5985.968
PartiPromptsBase22.5761.1030.29727.9676.234
PartiPromptsPTERA-mean22.5991.1430.29827.9556.250
PartiPromptsPTERA-sampled22.5971.1130.29827.9656.248
PartiPromptsPTERA-TTA22.6201.1670.30227.8686.274

Under the revised configuration, PTERA-mean’s PickScore improvement over the base model increases from 0.002 to 0.053 on DrawBench and from 0.004 to 0.023 on PartiPrompts. PTERA-TTA further improves PickScore over PTERA-mean on both benchmarks. These results suggest that the original near-zero gains were partly limited by the training configuration. Since several hyperparameters were adjusted jointly, we attribute the improvement to the revised recipe as a whole. As the reviewer and Reviewer 8Zv8 both suggest, we attribute the smaller gains on SD3 to two likely factors. First, our architecture choices and training configurations for SD3 have not yet been sufficiently explored. Second, adapting only the text-conditioning network may be inherently limited. With unrestricted capacity, diffusion weight tuning, which takes xtx_t and tt as inputs, could be more advantageous. Nevertheless, we consider it an important contribution that, with a very small trainable module, PTERA achieves better efficiency than diffusion weight tuning with competitive performance (Figure 6a and Table 4).

SDXL TTA configuration. We additionally evaluated a larger candidate budget, increasing NN from 5 to 8 with the selection schedule S={50,40,30,20,10}\mathcal{S}=\lbrace 50,40,30,20,10\rbrace.

SDXLSelection schedule S\mathcal{S}Candidates NNPickScore ↑ImageReward ↑HPSv2 ↑CLIP ↑Aesthetic ↑
DrawBench--22.9990.7730.28426.7996.215
DrawBench{50,40,30,20,10}523.0000.7070.28326.6876.104
DrawBench{50,40,30,20,10}823.0170.7510.28427.0006.116
PartiPrompts--23.2070.9960.29727.7826.456
PartiPrompts{50,40,30,20,10}523.1411.0000.29827.7996.424
PartiPrompts{50,40,30,20,10}823.3000.9910.30027.6196.520

Compared with the original TTA results, PickScore increases from 23.000 to 23.017 on DrawBench and from 23.141 to 23.300 on PartiPrompts. This demonstrates that the original configuration does not exhaust the available test-time gains. Algorithm 1 exposes the selection schedule and candidate count as configurable parameters; the experiment here specifically illustrates the benefit of increasing the candidate budget, at additional inference cost. Thus, TTA provides a configurable performance and computation trade-off, with its practical benefit depending on the chosen search configuration and evaluation metric.

Dear Reviewer oVTT,

Before the discussion period ends, we would appreciate your final look at our revised manuscript. All changes are highlighted in blue. We hope our responses have addressed the reviewer's questions. We would be happy to discuss any remaining concerns or follow-up questions. Thank you.

Reviewer_HyRbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how perturbations to text embeddings can reach high-reward samples. The authors propose PTERA, a lightweight residual module on a frozen text encoder, to learn a Gaussian distribution over text embeddings. Specifically, the authors regularize the learned distribution against a local Gaussian reference around the pre-trained embeddings. PTERA-TTA further performs best-of-N selection over sampled embeddings at selected denoising steps; the paper reports gains over the SD1.5, SDXL, and SD3 base models on DrawBench and held-out HPSv2/PartiPrompts, with GenEval evaluated only in combination with DRaFT+.

Critical strengths
  1. The observation of the exploration of text embeddings for reward alignment in Figure 2 and Prop. 1 motivates the proposed PTERA and PTERA-TTA, which look inspired.
  2. The design of the loss function of PTERA is sound, as deriving Theorem 1 gives two valid upper bounds (D_diff, D_text) on the intractable mixture KL.
  3. The results show PTERA is lightweight (Table 4) and transferable zero-shot across backbones (Table 8).
Critical weaknesses
  1. The gain from stochastic embeddings is small, raising questions about whether they’re necessary. PTERA (0.1, 0.1) beats its deterministic counterpart, DTERA (0.2), by only 0.05 PickScore (Table 1), and on SD3 (Table 2), PTERA-mean is within 0.002 of the base model despite the claimed cross-backbone effectiveness. Could you add paired significance tests or per-prompt win rates?
  2. The learned distribution is reward-specific, so every new reward (or a backbone without a compatible CLIP-L interface, e.g., SD3) requires training a separate PTERA module. Consider reward-conditioned training, or test whether a PickScore-trained PTERA can be TTA-steered by other rewards.
Questions for authors
  1. Figure 4a shows AES steering on two prompts only. Could you report benchmark-level AES-steered results? Does PTERA preserve CLIP score at matched AES? Since AES is only weakly tied to prompt semantics, a text-encoder-only method can raise it by drifting toward generic aesthetic directions at the cost of text alignment.
4 comments

We thank the reviewer for the thoughtful questions, which were very helpful in strengthening our work, and respond to each of them below.

W1.

We provide additional evidence addressing both points.

Probabilistic versus deterministic parameterization. The Mean-only variant in Table 6 is DTERA, the deterministic counterpart compared with PTERA in Table 1. Table 6 evaluates this comparison using four generated images per prompt. (The result of Table 1 evaluates one image per one prompt.) We extended this evaluation to three training seeds, 42, 43, and 44, and to the held-out HPSv2 prompts.

The table below reports mean PickScore and paired 95% bootstrap confidence intervals for the difference ΔPS=PS(PTERA)−PS(DTERA)\Delta\mathrm{PS}=\mathrm{PS}(\mathrm{PTERA})-\mathrm{PS}(\mathrm{DTERA}), computed from this four images per prompt evaluation. Positive differences favor PTERA.

BenchmarkTraining seedDTERA (Mean-only)PTERAΔPS\Delta\mathrm{PS}Paired 95% bootstrap CIwin rate
DrawBench4222.27822.384+0.106[0.0291, 0.2125]0.61
DrawBench4322.25822.327+0.069[-0.0150, 0.1581]0.53
DrawBench4422.16922.395+0.226[0.0348, 0.3495]0.68
HPSv2 (held-out)4221.80521.900+0.095[-0.0069, 0.1968]0.60
HPSv2 (held-out)4321.78521.894+0.109[0.0011, 0.2208]0.65
HPSv2 (held-out)4421.87821.947+0.069[-0.0340, 0.1671]0.58

PTERA achieves a higher mean PickScore in all six seed–benchmark combinations. Three of the six confidence intervals exclude zero. These results support a consistent direction of improvement across training seeds and both prompt sets, while also showing that the gains are modest and not statistically significant in every setting.

SD3 under a revised training configuration. We agree that the originally reported SD3 improvement was very small. We revisited the conservative training configuration by increasing the learning rate from 3×10−73\times10^{-7} to 10−510^{-5}, changing (β,γ)(\beta,\gamma) from (10−3,10−3)(10^{-3},10^{-3}) to (0.1,0.1)(0.1,0.1), and reducing the noise-scale hyperparameter from 0.10.1 to 0.050.05. The training reward-query budget remained unchanged at 20K.

DatasetMethodPickScore ↑ImageReward ↑HPSv2 ↑CLIP ↑Aesthetic ↑
DrawBenchBase22.4940.9260.28328.6765.955
DrawBenchPTERA-mean22.5470.9460.28428.6265.948
DrawBenchPTERA-sampled22.5050.9300.28328.6245.949
DrawBenchPTERA-TTA22.5730.9810.28428.5985.968
PartiPromptsBase22.5761.1030.29727.9676.234
PartiPromptsPTERA-mean22.5991.1430.29827.9556.250
PartiPromptsPTERA-sampled22.5971.1130.29827.9656.248
PartiPromptsPTERA-TTA22.6201.1670.30227.8686.274

The revised configuration increases the PTERA-mean improvement over the base model from 0.002 to 0.053 on DrawBench and from 0.004 to 0.023 on PartiPrompts. This suggests that the original SD3 result was partly limited by the training configuration. Since several hyperparameters were adjusted jointly, we attribute the improvement to the revised recipe as a whole.

W2.

We agree that PTERA learns a conditioning distribution shaped by its training reward, primarily PickScore in our main experiments. However, the reward used for candidate selection during TTA can be specified independently of the training objective, allowing the same learned distribution to be explored using different user-selected rewards.

To examine this flexibility, we evaluated the PTERA checkpoint which trained on PickScore from Table 1, with (β,γ)=(0.1,0.1)(\beta,\gamma)=(0.1,0.1), using PickScore (PS), ImageReward (IR), HPSv2, CLIP, and Aesthetic Score (AES) as alternative TTA steering rewards. Across these experiments, we kept the checkpoint, selection schedule S={50,40,30,20,10}\mathcal{S}=\lbrace 50,40,30,20,10\rbrace, and candidate count N=5N=5 fixed, without additional training. The accompanying tables report all five evaluation metrics on DrawBench and the held-out HPSv2 prompts.

The results show that changing the steering reward produces corresponding changes in the evaluation metrics. In particular, PickScore, ImageReward, CLIP, and Aesthetic Score each attain their highest values among the tested steering configurations when used as the selection reward on both benchmarks. These findings demonstrate that a single PTERA checkpoint supports practical cross-reward steering at test time. Thus, although the learned proposal distribution remains influenced by its training objective, adapting the TTA selection criterion to different preferences does not necessarily require training a separate checkpoint for each reward.

TTA steer (DrawBench)PS ↑\uparrowIR ↑\uparrowHPSv2 ↑\uparrowCLIP ↑\uparrowAES ↑\uparrow
PTERA-mean22.4660.2500.26825.9875.933
PTERA-TTA (PS)22.5620.3270.26926.0745.918
PTERA-TTA (IR)22.3770.3740.26525.9055.914
PTERA-TTA (HPSv2)22.3520.2400.27026.0055.896
PTERA-TTA (CLIP)22.3410.2400.26526.0985.910
PTERA-TTA (AES)22.3440.1980.26525.8385.936
TTA steer (HPSv2)PS ↑\uparrowIR ↑\uparrowHPSv2 ↑\uparrowCLIP ↑\uparrowAES ↑\uparrow
PTERA-mean21.8130.3510.26427.2236.546
PTERA-TTA (PS)21.9440.3750.26727.2726.495
PTERA-TTA (IR)21.9130.3800.26527.6996.352
PTERA-TTA (HPSv2)21.9410.3380.26727.6526.364
PTERA-TTA (CLIP)21.9270.3410.26527.7816.373
PTERA-TTA (AES)21.9300.3300.26627.7486.598

Q1.

In our experimental setup, AES optimization required larger regularization coefficients than those used for PickScore, ImageReward, or HPSv2 to mitigate degradation in text alignment and other preference metrics. We therefore evaluated several regularization configurations for AES-trained PTERA on DrawBench and the held-out HPSv2 prompts. To examine whether PTERA’s higher CLIP scores simply reflect less aesthetic optimization, we additionally evaluated an intermediate TextCraftor checkpoint at 80% of the total training epochs. The parenthesized values for PTERA denote (β,γ)(\beta,\gamma).

Methods (AES training, DrawBench)PSIRHPSv2CLIPAES
TextCraftor (Final)18.071-2.0240.1749.2667.386
TextCraftor (80% ckpt)18.650-1.8400.18114.8466.825
PTERA (1, 1)20.763-0.6190.21320.3437.020
PTERA (1, 2)20.821-0.6390.21720.5547.047
PTERA (2, 1)21.126-0.3540.23023.4576.696
Methods (AES training, HPSv2)PSIRHPSv2CLIPAES
TextCraftor (Final)18.237-1.8580.19413.0487.398
TextCraftor (80% ckpt)18.547-1.3260.20917.6776.902
PTERA (1, 1)20.728-0.2940.22923.3867.272
PTERA (1, 2)20.676-0.3780.23023.3507.268
PTERA (2, 1)21.0680.1950.24826.4387.016

Compared with the intermediate TextCraftor checkpoint, PTERA (1, 1) achieves both higher AES and higher CLIP on both benchmarks. On DrawBench, it obtains AES 7.020 versus 6.825 and CLIP 20.343 versus 14.846. On HPSv2, it obtains AES 7.272 versus 6.902 and CLIP 23.386 versus 17.677. PickScore, ImageReward, and HPSv2 scores are also higher in both comparisons. Thus, at these operating points, PTERA’s higher text-alignment scores cannot be explained simply by a lower achieved AES.

The final TextCraftor checkpoint attains the highest AES, accompanied by substantially lower CLIP and preference scores. Within PTERA, increasing β\beta from 1 to 2 while fixing γ=1\gamma=1 further improves CLIP and the preference metrics on both benchmarks, with a corresponding reduction in AES. Together, these results provide benchmark-level evidence of the trade-off between aesthetic optimization and text alignment, and show that PTERA offers operating points with improvements in both metrics over the intermediate TextCraftor baseline.

While compiling these results, we also identified a reporting error in Figure 4(a). The PTERA AES value reported for DrawBench, 7.2720, was inadvertently taken from the HPSv2 benchmark. The correct DrawBench value is 7.0196, reported as 7.020 in the table above. We correct this value in the revised manuscript.

Dear Reviewer HyRb,

Before the discussion period ends, we would appreciate your final look at our revised manuscript. All changes are highlighted in blue. We trust that the discussions and revised manuscript addresses the reviewer's concerns. Please let us know if anything remains unclear. We would be happy to clarify any remaining concerns or questions.

Reviewer_8Zv8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PTERA learns a prompt-conditioned diagonal Gaussian distribution over text embeddings through small residual modules, freezing the pretrained encoders and diffusion backbone. Training combines reward optimization with diffusion- and embedding-level regularization; PTERA-TTA uses the learned distribution for candidate selection during sampling. Experiments cover SD1.5, SDXL, SD3, ablations, transfer, and rollout comparisons.

Critical strengths
  1. A coherent link between adaptation and exploration. The learned conditioning distribution supports both reward alignment and test-time search. The frozen-backbone design, training-resource results, and combination with DRaFT+ suggest practical value.

  2. Useful controls for probabilistic conditioning. The appendix includes parameter-matched deterministic/isotropic variants, separates embedding and initial-noise diversity, analyzes both regularizers, and reports TTA costs. These provide evidence beyond comparisons with unadapted models.

Critical weaknesses
  1. The matched probabilistic gain needs stronger generalization evidence. Table 6 reports PickScore of 22.278 (mean-only), 22.339 (learned-isotropic), and 22.384 (PTERA), with little HPSv2 change, on the training prompt set. Figure 7 supports structured exploration, but its isotropic control replaces the learned scale at inference rather than retraining a matched variant. Held-out paired gains and training-seed variability would establish how reliably the more expressive distribution helps.

  2. The implemented diffusion regularizer is underspecified. Eq. (40) uses positive-variance DDPM transitions and a trajectory expectation, while Appendix C describes DDIM training with one-step reward backpropagation. Table 7 says Eq. (40) is instantiated in training but does not specify its timestep estimator, weights, or detached paths, including the adaptation to SD3. This leaves the numerical objective and scope of its KL interpretation unclear; deterministic sampling does not invalidate the separate D_text bound.

  3. Benefits vary across backbones and inference modes. SD3's DrawBench PickScore changes only from 22.494 to 22.496 with PTERA-mean. On SDXL, TTA adds little or no PickScore improvement over the learned mean across the two benchmarks, despite additional computation. This limits the demonstrated generality of the full framework.

Questions for authors
  1. Can you extend the matched Table 6 variants to held-out prompts, reporting paired uncertainty and available training-seed variability?

  2. What D_diff estimator, timestep weights, and gradient paths are used per backbone, and how do they relate to Eq. (40) and truncated training?

  3. What explains the small SD3 PickScore gains and variable TTA gains on SDXL: optimization settings, limited conditioning-space headroom, or another factor?

4 comments

Before our responses, we sincerely thank the reviewer for the detailed and constructive questions, which have helped us improve our work.

Q1 (W1).

To address this concern, we extended Table 6 with two additional training seeds, 43 and 44, alongside the original seed 42, yielding three training runs for each ablation variant. We also evaluated all variants on the held-out HPSv2 prompts.

The table below reports PickScore and HPSv2 scores on both DrawBench (DB) and the held-out HPSv2 benchmark (generate four images per prompt), with results presented separately for each training seed. For each ablated variant, we provide a paired 95% bootstrap confidence interval for the PickScore difference, defined as (ΔPS=PS(PTERA)−PS(variant)\Delta\mathrm{PS}=\mathrm{PS}(\mathrm{PTERA})-\mathrm{PS}(\mathrm{variant})). Bold values indicate the highest mean PickScore within each seed–benchmark combination, and dashes denote self-comparisons for PTERA.

Fixed-Iso and Learned-Iso are themselves design choices within probabilistic modeling, namely restricted forms of PTERA with a fixed or a learned isotropic scale, whereas PTERA learns a diagonal scale vector and is therefore the most expressive parameterization. It consistently outperforms its deterministic counterpart DTERA (Mean-only) across all three training seeds and both prompt sets (ΔPS\Delta\mathrm{PS} between 0.069 and 0.226). PTERA also achieves the highest PickScore among the probabilistic variants in all six seed–benchmark combinations. These variants are trained separately with an isotropic scale, so, unlike the isotropic control in Figure 7, which replaces the learned scale of a trained PTERA model only at inference, this comparison is between retrained matched models. Although not every paired confidence interval excludes zero, the direction of the average gains is consistent across training seeds and both the in-domain and held-out prompt sets.

Training seedVariantDB (in-domain) PS ↑DB (in-domain) HPSv2 ↑Paired 95% CI (ΔPS)HPSv2 (held-out) PS ↑HPSv2 (held-out) HPSv2 ↑Paired 95% CI (ΔPS)
42Mean-only22.2780.266[0.0291, 0.2125]21.8050.266[-0.0069, 0.1968]
42Fixed-Iso22.3170.267[-0.0242, 0.1621]21.8130.266[-0.0193, 0.1915]
42Learned-Iso22.3390.265[-0.0529, 0.1344]21.8960.263[-0.0987, 0.1082]
42PTERA22.3840.267—21.9000.269—
43Mean-only22.2580.260[-0.0150, 0.1581]21.7850.253[0.0011, 0.2208]
43Fixed-Iso22.2770.268[-0.0676, 0.1029]21.8650.266[0.0008, 0.0895]
43Learned-Iso22.2160.266[-0.0177, 0.1653]21.8850.271[-0.0305, 0.0578]
43PTERA22.3270.271—21.8940.270—
44Mean-only22.1690.260[0.0348, 0.3495]21.8780.260[-0.0340, 0.1671]
44Fixed-Iso22.2340.263[-0.0012, 0.2794]21.7590.267[0.0732, 0.2706]
44Learned-Iso22.2720.267[0.0103, 0.1951]21.7320.262[0.0925, 0.3401]
44PTERA22.3950.267—21.9470.271—

Q2 (W2).

To address this question, we separately clarify our practical implementation and the theoretical interpretation of the diffusion regularizer.

Our reward training pipeline follows the truncated reward backpropagation of DRaFT [1] with a single step (DRaFT-KK with K=1K=1), so the reward gradient is backpropagated only through the final denoising transition. SD1.5 and SDXL use DDIM rollouts, while SD3 uses its flow-matching ODE rollout. We evaluate the diffusion regularizer only at the final denoising timestep, comparing the target and reference predictions at the same incoming latent. Specifically, we use noise predictions for SD1.5/SDXL and velocity predictions for SD3, with the squared discrepancy averaged over latent coordinates and the batch. The rollout prefix and reference prediction are detached. Gradients propagate through the target conditioning and final denoising transition, while the backbone parameters remain frozen. The regularizer itself follows the KL regularization discussed in Appendix B.8 of DRaFT [1], which adds a squared-error term between the fine-tuned and pre-trained noise predictions at the last denoising step.

For the theoretical interpretation, we consider an auxiliary Gaussian reverse model with a shared positive covariance, under which the conditional KL can be expressed as a weighted squared prediction discrepancy. We write every backbone's forward corruption as q(xt∣x0)=N(αtx0,σt2I)q(x_t\mid x_0)=\mathcal N(\alpha_t x_0,\sigma_t^2 I) (t=0t=0 data, t=1t=1 noise), with α0=1, σ0=0, α1=0, σ1=1\alpha_0=1,\ \sigma_0=0,\ \alpha_1=0,\ \sigma_1=1. SD3 uses flow matching [2] with ODE-based generation, specifically follows the linear path αt=1−t\alpha_t=1-t, σt=t\sigma_t=t. It gives x^0=xt−σtv^t\hat x_0=x_t-\sigma_t\hat v_t. We define the auxiliary reverse kernels using the Gaussian posterior mean with x0x_0 replaced by x^0\hat x_0 and a shared covariance as Qt∣sIQ_{t\mid s}I. For grid times s<ts<t, let rt∣s=αt/αsr_{t\mid s}=\alpha_t/\alpha_s and Qt∣s=σt2−rt∣s2σs2Q_{t\mid s}=\sigma_t^2-r_{t\mid s}^2\sigma_s^2, so that Ks,tφ:=pθ(xs∣xt,zφ)=N(rt∣sσs2σt2xt+αsQt∣sσt2x^0θ(xt,t,zφ),Qt∣sI)K_{s,t}^{\varphi}:=p_\theta(x_s\mid x_t,z_\varphi)=\mathcal{N}\left(\frac{r_{t\mid s}\sigma_s^2}{\sigma_t^2}x_t+\frac{\alpha_sQ_{t\mid s}}{\sigma_t^2}\hat x_0^\theta(x_t,t,z_\varphi),Q_{t\mid s}I\right). Their conditional KL at the same xtx_t is then DKL(Ks,tφ∥Ks,tφref)=αs2Qt∣s2σt4∥Δx^0,t∥22.D_{\mathrm{KL}}\big(K_{s,t}^{\varphi}\parallel K_{s,t}^{\varphi_\mathrm{ref}}\big)=\frac{\alpha_s^2Q_{t\mid s}}{2\sigma_t^4}\Vert\Delta\hat x_{0,t}\Vert_2^2 .

where Δx^0,t=x^0θ(xt,t,zφ)−x^0θ(xt,t,zφref)\Delta\hat x_{0,t}=\hat x_0^\theta(x_t,t,z_\varphi)-\hat x_0^\theta(x_t,t,z_{\varphi_\mathrm{ref}}) with Δx^0,t=−(σt/αt)Δϵ^t\Delta\hat x_{0,t}=-(\sigma_t/\alpha_t)\Delta\hat\epsilon_t (noise prediction) and Δx^0=−(σt/Dt)Δv^t\Delta\hat x_0=-(\sigma_t/D_t)\Delta\hat v_t, Dt=αtσ˙t−σtα˙tD_t=\alpha_t\dot\sigma_t-\sigma_t\dot\alpha_t (velocity prediction), we obtain DKL=C(s,t)∥Δ(prediction)∥22D_{\mathrm{KL}}=C(s,t)\Vert\Delta(\text{prediction})\Vert_2^2 with:

Backbone/pathPredictionC(s,t)C(s,t)Implemented last-step surrogate
SD1.5/SDXL, VP: αt→αˉt\alpha_t\rightarrow\sqrt{\bar \alpha_t}, σt2→1−αˉt\sigma_t^2\rightarrow1-\bar \alpha_tϵ^t\hat\epsilon_tαs2Qt∣s2αt2σt2→1−αˉt/αˉs2(αˉt/αˉs)(1−αˉt)\dfrac{\alpha_s^2Q_{t\mid s}}{2\alpha_t^2\sigma_t^2}\rightarrow\dfrac{1-\bar \alpha_t/\bar \alpha_s}{2(\bar \alpha_t/\bar \alpha_s)(1-\bar \alpha_t)}∥ϵ^θ(xt,t,zφ)−ϵ^θ(xt,t,zφref)∥22\left\Vert\hat \epsilon_{\theta}(x_t,t,z_\varphi)-\hat \epsilon_{\theta}(x_t,t,z_{\varphi_\mathrm{ref}})\right\Vert_2^2
SD3, linear FM: αt→1−t\alpha_t\rightarrow1-t, σt→t\sigma_t\rightarrow t (Dt=1D_t=1)v^t\hat v_tαs2Qt∣s2σt2→(t−s)(t+s−2st)2t2\dfrac{\alpha_s^2Q_{t\mid s}}{2\sigma_t^2}\rightarrow\dfrac{(t-s)(t+s-2st)}{2t^2}∥v^θ(xt,t,zφ)−v^θ(xt,t,zφref)∥22\left\Vert\hat v_{\theta}(x_t,t,z_\varphi)-\hat v_{\theta}(x_t,t,z_{\varphi_\mathrm{ref}})\right\Vert_2^2

Eq. (40) corresponds to the x0x_0 parameterized local KL summed over adjacent DDPM steps, with the expectation taken over the stochastic trajectory. The implemented penalty differs from it in three ways. First, it retains only the final denoising step, the same step through which the reward gradient is backpropagated. Second, it is evaluated at the state produced by the deterministic rollout. Third, it uses a unit weight and a coordinate average. Under the auxiliary Gaussian reverse model defined above, the final transition from t>0t>0 to the data endpoint has covariance σt2I\sigma_t^2 I. Its prediction-space KL coefficient is therefore 1/(2αt2)1/(2\alpha_t^2) for SD1.5/SDXL and 1/21/2 for SD3. Because training retains only this fixed final timestep, these weights, together with the coordinate normalization, can be absorbed into the overall regularization coefficient. The implemented penalty is thus proportional to the final-step term of the auxiliary KL and should be read as a KL-motivated surrogate. It does not establish equivalence to the full trajectory KL.

We update the original manuscript by adding the implementation details, together with a clearer explanation of their connection to the theoretical KL formulation, in the revised version (Appendix C blue color).

[1] "Directly fine-tuning diffusion models on differentiable rewards." ICLR 2024.

[2] "Flow matching for generative modeling." ICLR 2023.

Q3 (W3).

To address the concern about the SD3 experiment, we provide additional results under a revised training configuration.

The configuration reported in the original submission was conservative, which likely limited the reported gains. Applying PTERA to SD3 requires two design choices, the module architecture and the training configuration, and we had no prior reference for either. Because each SD3 training run takes a long time, our initial exploration of these choices was limited. To verify whether the gain of PTERA carries over to a larger backbone, we kept the architecture fixed, matching the parameter size ratio used in our main SD1.5 and SDXL experiments, and revisited only the training configuration: we increased the learning rate from 3×10−73\times10^{-7} to 10−510^{-5}, changed (β,γ)(\beta,\gamma) from (10−3,10−3)(10^{-3},10^{-3}) to (0.1,0.1)(0.1,0.1), and reduced the reference embedding noise scale ρ\rho from 0.10.1 to 0.050.05, while retaining the same budget of 20K reward queries.

The below table reports the resulting SD3 performance on DrawBench and held-out PartiPrompts. Under the revised configuration, PTERA-mean achieves PickScores of 22.547 and 22.599, respectively, and PTERA-TTA further increases these to 22.573 and 22.620. These results suggest that the previously reported gains were partly limited by the training configuration. Also, we believe that two factors contribute to the smaller gains on SD3. First, reward fine-tuning on SD3 is computationally expensive, so we were not able to explore training configurations as thoroughly as for SD1.5. In addition, SD3 is a substantially larger model, for which 100 epochs on DrawBench constitute a small scale adaptation experiment. Second, we agree that the limited conditioning-space headroom raised by the reviewer may also be an inherent limitation of adapting only the text conditioning network. In this work, we show that PTERA can outperform diffusion weight updates when the trainable module is small. With unrestricted capacity, however, diffusion weight fine-tuning, which takes xtx_t and tt as inputs and can therefore represent a richer class of functions, may be more effective at increasing the reward.

DatasetMethodPickScore ↑ImageReward ↑HPSv2 ↑CLIP ↑Aesthetic ↑
DrawBenchBase22.4940.9260.28328.6765.955
DrawBenchPTERA-mean22.5470.9460.28428.6265.948
DrawBenchPTERA-sampled22.5050.9300.28328.6245.949
DrawBenchPTERA-TTA22.5730.9810.28428.5985.968
PartiPromptsBase22.5761.1030.29727.9676.234
PartiPromptsPTERA-mean22.5991.1430.29827.9556.250
PartiPromptsPTERA-sampled22.5971.1130.29827.9656.248
PartiPromptsPTERA-TTA22.6201.1670.30227.8686.274

For SDXL, we kept the selection schedule of Algorithm 1 fixed at S={50,40,30,20,10}\mathcal{S}=\lbrace 50,40,30,20,10\rbrace and increased the number of candidates from N=5N=5 to N=8N=8. The table below includes the PTERA-mean row of Table 2 for direct comparison. With N=8N=8, PTERA-TTA exceeds PTERA-mean on both benchmarks. The gain comes at additional inference cost because the number of candidate evaluations per image increases. We therefore regard TTA on SDXL as a trade-off between performance and computation cost whose PickScore benefit over the learned mean appears with a larger candidate budget.

DatasetSelection schedule S\mathcal{S}Candidates NNPickScore ↑ImageReward ↑HPSv2 ↑CLIP ↑Aesthetic ↑
DrawBench--22.9990.7730.28426.7996.215
DrawBench{50,40,30,20,10}523.0000.7070.28326.6876.104
DrawBench{50,40,30,20,10}823.0170.7510.28427.0006.116
PartiPrompts--23.2070.9960.29727.7826.456
PartiPrompts{50,40,30,20,10}523.1411.0000.29827.7996.424
PartiPrompts{50,40,30,20,10}823.3000.9910.30027.6196.520

Dear Reviewer 8Zv8,

Before the discussion period ends, we would appreciate your final look at our revised manuscript. All changes are highlighted in blue. We hope that the rebuttal can resolve your concerns and would be grateful if our work could be reassessed. We remain available to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our response and revision.

We thank all reviewers and the area chair for their time. For clarity, we summarize the main additions made during the discussion period.

  • (Reviewers 8Zv8, HyRb, and oVTT) Multi-seed and held-out extension of the ablation in Table 6, with paired confidence intervals are given in the revised manuscript. (Table 6b)

  • (Reviewer 8Zv8) We address the issue regarding Implementation details of the diffusion regularizer and its relation to Eq. (40). We describe the implemented regularizer, a final-step squared discrepancy between the target and reference noise or velocity predictions that follows DRaFT, and clarify that it is a KL-motivated surrogate of Eq. (40). We have added this contents to Appendix C in the revised manuscript.

  • (Reviewers 8Zv8, HyRb, and oVTT) SD3 results under a revised training configuration and SDXL TTA with a larger candidate budget. SD3 training configurations and the corresponding changed performance are edited in the revised manuscript.

  • (Reviewer HyRb) Test-time alignment of a PickScore-trained PTERA checkpoint with other rewards (cross-reward experiment) are added in Appendix D.2 with Table 8b of the revised manuscript, and we report benchmark-level results for aesthetic score in the rebuttal.

  • We found a numerical error in Figure 4(a) and have corrected the reported value.

All additional contents are marked in blue in the revised manuscript. We hope these additions address the concerns raised in the reviews and will be taken into account in the subsequent discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.