acceptodds
Under review as a conference paper at ICLR 2027

ToPO: Token-Oriented Preference Optimization for Text-to-Image Diffusion Models

Abstract

Diffusion models have achieved remarkable success in text-to-image generation, yet aligning their outputs with human preferences remains challenging. Diffusion-DPO learns from image preference pairs, but their global labels do not specify how supervision should be allocated across spatial locations and denoising timesteps. The semantic roles of prompt tokens are also not explicitly considered when assigning these local weights. To address these limitations, we propose Token-Oriented Preference Optimization (ToPO), which builds a token-conditioned spatial–temporal weighting map for each pair. A frozen reference denoiser obtains spatial and temporal weights from local noise-prediction error differences under shared noise, while preferred-branch cross-attention links spatial evidence to content tokens. The map reweights Diffusion-DPO and supports an auxiliary pixel-midpoint ordering loss. It requires neither local preference annotations nor a separately trained reward model. Extensive experiments across multiple preference and compositional evaluations show that ToPO achieves state-of-the-art performance among the compared methods with only 500 training updates, demonstrating the potential of token-conditioned preference allocation for high-quality diffusion alignment within a limited update budget.

Primary area:
generative models
Keywords:
diffusion models, text-to-image generation, preference optimization, direct preference optimization, human feedback, latent diffusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JAbWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ToPO, a reweighting strategy of the Diffusion-DPO loss for text-to-image diffusion models. Specifically, it first uses the difference between the frozen reference model's local noise-prediction errors on the two images, together with the cross-attention map of the preferred image, to construct a content-aware spatial weight map and a temporal weight. The spatial and temporal weights are then integrated into the Diffusion-DPO loss, together with an auxiliary ordering loss against the encoded pixel average of the two images. Empirical results on SD-1.5 and SDXL, trained on Pick-a-Pic for 500 updates, show improvements over Diffusion-DPO and other released checkpoints on PickScore, HPSv2, ImageReward, GenEval and T2I-CompBench, together with a 16-participant user study on SDXL. Additionally, ablation studies are conducted on the weight components and the preference scale β\beta.

Critical strengths
  1. The method does not require extra annotations or a reward model. It utilizes the frozen reference model.
  2. The paper includes sensible controls for the weighting (uniform and shuffled weights) and a matched training run against Diffusion-DPO (Figure 5).
Critical weaknesses
  1. The novelty is limited, and the method and its claims are not well supported:

    i. (Novelty) Spatial credit assignment for diffusion preference learning (PatchDPO, IAPO, BiDPO), step-aware weighting over denoising time (e.g., SPO, LPO), and cross-attention as a token-to-region link (Prompt-to-Prompt, Attend-and-Excite) are all established, and STAR already allocates credit over both space and time. Spatially reweighted DPO losses also exist in related settings and are not discussed: FocusDPO weights the DPO objective toward focus regions that change across noise levels, and LocalDPO restricts the DPO loss to masked regions. ToPO combines these ideas into a heuristic reweighting.

    ii. (Soundness-1) The method is a stack of heuristics (four fixed anchors, a 50/50 prior mix, a Gaussian time kernel, attention averaging, special-token masking and a pixel-midpoint triplet term), with no derivation of why the error gap should be the right allocation, and the pair-dependent weights remove the single implicit reward that justifies DPO. The allocation signal itself, the absolute difference of reference errors, is label-invariant and shows where the reference model treats the two images differently, not why one image is preferred. The appendix propositions (SGD convergence, law of large numbers, a triangle-inequality bound) are standard and do not support the method.

    iii. (Soundness-2) The timestep factor also appears to follow the scale of the noise-prediction error rather than the motivation: in the paper's own example (Figure 11), wτw_\tau is about 0.70.7 at τ=50\tau = 50 and about 0.02 at τ=949\tau = 949, so almost no weight goes to the high-noise steps where the introduction says layout is decided.

    iv. (Significance) The multi-method tables compare released checkpoints with different initializations, data and compute, which the paper itself calls descriptive, yet the abstract claims state-of-the-art performance. On SDXL the gains are small or mixed: in the matched study Diffusion-DPO is higher on PickScore and Aesthetic (Table 5), on PartiPrompts ToPO is below Diffusion-DPO on ImageReward and CLIP (Table 7), and on HPDv2 InPO has a higher HPSv2 (Table 9).

  2. The matched comparison with Diffusion-DPO is inconsistent and appears to overstate the gain. In Figure 5, the matched Diffusion-DPO run reaches 21.344 on PickScore, 0.2862 on HPSv2 and 0.718 on ImageReward at 500 updates, against 21.542, 0.2903 and 0.812 for ToPO. However, the matched three-seed Diffusion-DPO in Table 5 reports 20.930, 0.2587 and 0.250, which are almost identical to the released Diffusion-DPO checkpoint in Table 1 (20.947, 0.2586, 0.250) and well below the SFT-winners initialization both methods supposedly start from (21.181, 0.2824, 0.642). The ImageReward spread across three training seeds is also 0.001 on SD-1.5 and 0.000 on SDXL, which shows unusually little variation for independent training runs. The paired gains in Table 5 (e.g., +0.563 ImageReward) are therefore several times larger than the matched gain in Figure 5 (+0.094). Likewise, the main-text claim that ImageReward increases from 0.250 to 0.812 compares ToPO, initialized from SFT-winners, with a released Diffusion-DPO checkpoint trained from the base model, so most of that gap comes from the initialization.

[FocusDPO] Jin et al. "FocusDPO: Dynamic Preference Optimization for Multi-Subject Personalized Image Generation via Adaptive Focus." AAAI 2026.

[LocalDPO] Huang et al. "Mind the Generative Details: Direct Localized Detail Preference Optimization for Video Diffusion Models." CVPR 2026.

Questions for authors
  1. How were the matched Diffusion-DPO numbers in Table 5 obtained, and why do they differ from the matched run in Figure 5 and coincide with the released checkpoint?
  2. What is the gain of ToPO over Diffusion-DPO trained from the same initialization with the same budget, over several seeds, on all reported benchmarks?
  3. What is the training cost per update of ToPO compared with Diffusion-DPO?
  4. Since count and spatial tokens are not up-weighted (Table 10), what explains the large counting gain on GenEval?
  5. What is the full protocol of the user study, including the number of judgments per comparison and any significance test?
  6. Does the timestep factor remain informative if the error contrast is normalized by the expected noise-prediction error at each timestep?
  7. (Suggestion) How does ToPO compare with a simpler spatial weight, such as the pixel or latent difference between the two images?
3 comments
AuthorsOpenReview ↗

Response part 1

Thank you for the careful and detailed review. We have taken each concern seriously and run additional experiments. We answer each weakness and question below, dividing the response into three parts. Revisions are marked in blue, and tables are cited by their revised numbers.


W-i. Novelty

We agree that spatial credit, step-aware weighting, and cross-attention each have precedents, and we thank you for pointing us to FocusDPO and LocalDPO, which we now discuss in Related Work. The essential difference is not which axes are weighted, but where the local signal comes from. Prior methods obtain it from outside the preference pair: from a reference image, from the way the negative is constructed, or from a trained reward model. ToPO obtains it from the pair itself.

MethodWhat is weightedWhere the local signal comes fromWhat else is needed
PatchDPOPatchesSimilarity to a given reference imageReference images, constructed pairs
FocusDPORegionsCorrespondence to a given reference imageReference images, synthetic negatives
BiDPOObject regionsBoxes from the pipeline that edits the negativesConstructed edited pairs
LocalDPOVideo regionsRandom masks used to corrupt the negativesConstructed negatives
SPO, LPODenoising stepsA separately trained step-level reward modelReward model, online sampling
STARRegions per stepText–image attention within one sampleOnline RL, 24 rollouts per prompt
ToPORegions, steps, tokensThe reference model's error gap between the two imagesNothing beyond Diffusion-DPO

This difference has three consequences.

First, ToPO addresses a harder and more general setting. In BiDPO, LocalDPO, and FocusDPO, the negative is created by editing or corrupting the positive, so the location of the difference is known by construction. PatchDPO and FocusDPO also measure fidelity to a reference image, a goal specific to personalization. In human preference data, neither shortcut exists: the two images are independent samples, there is no reference, and the decisive difference can lie anywhere. ToPO recovers it with the frozen reference model that Diffusion-DPO already uses, so it needs no new data, annotation, or model.

Second, ToPO weights differences rather than coverage. STAR's attention map shows where the prompt is rendered in one sample, regardless of whether two images differ there. ToPO combines two signals: the error gap tells where the images differ, and attention tells which differences concern the prompt. Together they concentrate the update on prompt-relevant differences, which neither signal achieves alone. FocusDPO and LocalDPO do not use prompt tokens at all.

Third, ToPO allocates over time at no extra cost. SPO and LPO must train a step-level reward model and sample candidates online, whereas ToPO's temporal weight comes from the same reference passes that give its spatial weight. STAR, the only prior method weighting both space and time, does so in online reinforcement learning with a reward signal; ToPO does so in offline DPO.

Finally, Prompt-to-Prompt and Attend-and-Excite use attention at inference to edit or guide an image, while ToPO uses it during training to turn pair contrast into token weights. Our contribution is therefore a principle, separating the direction of a preference update, given by the label, from its allocation, given by the pair itself, together with a realization that needs no annotation, reward model, or online sampling.

W-ii. Design choices and the DPO objective

We address the concern that ToPO is a stack of heuristics in three steps.

Every component is tested. Removing the spatial, temporal, or token factor, the special-token mask, or the midpoint term each lowers PickScore, by 0.17 to 0.43 points (Table 3). The revision further tests separability, the number of anchors, the attention source, error normalization, and a latent-difference weight (Tables 16 and 17).

ToPO stays within the DPO framework. Diffusion-DPO already optimizes a surrogate: an ELBO bound with a timestep weighting, evaluated at one sampled step. ToPO keeps this form and only changes the weights inside it. Because these weights are nonnegative and carry no gradient, each pair still pushes the model in the direction of its label. The same idea underlies token-weighted DPO for language models, such as TIS-DPO at ICLR 2025, which we now discuss in Related Work.

Label invariance is intended. The error gap is symmetric so that the allocation can never reverse the direction set by the label. The full weight still uses the label, since tokens are read from the preferred image's attention, and this matters: symmetric and loser-only attention both perform worse (Table 16).

We agree that the appendix propositions are standard; they state basic sanity properties and are presented as supporting analysis, not as a contribution.

AuthorsOpenReview ↗

Response part 2

W-iv and Q1. The matched comparison

Thank you for examining these numbers so carefully. Your reading exposed an error in our original description, and we apologize for it. The three-seed table, Table 5 in the original submission and Table 7 in the revision, is not a matched training comparison. It evaluates fixed checkpoints under three generation seeds, and its Diffusion-DPO rows use the officially released weights. This single fact explains each of your observations:

  • The Diffusion-DPO scores coincide with Table 1 because both use the same released checkpoint.
  • They fall below SFT-winners because the released model was trained from the base model, not from SFT-winners.
  • The standard deviations are small because only the generation seed varies, not the training run.
  • The gains are larger than in Figure 5 because ToPO starts from SFT-winners while the released model does not.

The matched comparison is Figure 5, where both methods start from the same SFT-winners checkpoint with identical settings. At 500 updates, ToPO reaches a PickScore of 21.542, HPSv2 of 0.2903, and ImageReward of 0.812, against 21.344, 0.2862, and 0.718 for Diffusion-DPO. We have corrected the description of the three-seed table, and the main text now reports this matched gain alongside the released-checkpoint comparison.

Q2. Same initialization and same budget

To strengthen the matched evidence, we trained a second, independent pair of SD-1.5 runs from the same SFT-winners checkpoint with the same data and evaluation, this time recording actual GPU-hours (Section 5.5, Figure 6):

MethodUpdatesGPU-hoursPickScore ↑HPSv2 ↑ImageReward ↑
Diffusion-DPO842.8721.3100.28600.721
ToPO473.0521.3570.28710.743
Diffusion-DPO2518.4821.3970.28870.719
ToPO1408.9921.5390.29060.790
Diffusion-DPO50016.9121.4260.28950.736
ToPO26016.6421.6310.29120.815

At every comparable budget, ToPO is ahead on all three metrics, and after only 140 updates it already exceeds Diffusion-DPO's final result while using 46.8% fewer GPU-hours. Since this run is independent of Figure 5, the two figures give two matched run pairs that agree. Multiple training seeds on every benchmark did not fit within the rebuttal period; we will add them in the revision.

Q3. Training cost

BackboneMethodSeconds per update ↓Peak memory per GPU, GiB ↓
SD-1.5Diffusion-DPO16.0723.50
SD-1.5ToPO30.2324.36
SDXLDiffusion-DPO35.4970.94
SDXLToPO70.4571.01

Measured on eight H200 GPUs, ToPO takes 1.88 times longer per update on SD-1.5 and 1.98 times on SDXL, because the frozen reference evaluates both images at four anchors. These passes need no gradients, so peak memory rises by only 0.86 and 0.07 GiB. Per-update cost alone, however, does not determine efficiency: as shown above, ToPO reaches better scores with fewer total GPU-hours. Each update costs more, but it also achieves more.

W-iv. Scope of claims

We agree that our claims should match the evidence, and we have revised the abstract from state-of-the-art to competitive performance. We also acknowledge the mixed SDXL results you note: against the released Diffusion-DPO, ToPO is slightly lower on PickScore and the aesthetic score across generation seeds and on ImageReward and CLIP on PartiPrompts, and InPO has a higher HPSv2 on HPDv2. Appendix S7.4 now discusses this variation. These cases are outweighed by ToPO's broader advantages on SDXL: it leads on most metrics, raises the GenEval average from 0.5917 to 0.6168, and is preferred over Diffusion-DPO by human raters in 63.3% to 70.7% of pairs. Our central claim rests on the matched SD-1.5 comparisons above, where ToPO leads consistently.

AuthorsOpenReview ↗

Response part 3

W-iii and Q6. The timestep factor

You are right that noise-prediction residuals are naturally larger at low noise, so we tested whether the temporal factor merely follows this scale. We divided the error contrast at each anchor by the expected noise-prediction error at that anchor, keeping everything else fixed (Table 17):

VariantPickScore ↑HPSv2 ↑ImageReward ↑Aesthetic ↑CLIP ↑
ToPO21.5420.29030.8125.36128.393
Error-normalized21.4990.28940.8015.30228.249

Normalization scores lower on all five metrics, so removing the error scale does not help. At the same time, the temporal factor is clearly informative, since removing it lowers PickScore by 0.307. The emphasis on low-noise steps also fits our motivation: Pick-a-Pic pairs often differ in fine details, and in our introductory example the decisive difference, a legible "42", forms at late, low-noise steps.

Q4. Count tokens and the counting gain

Thank you for pointing out this apparent discrepancy. The original diagnostic used SD-1.5 pairs generated with the same template and seed, differing only in the attribute word. Even so, SD-1.5 does not always confine the change to that attribute: the layout or background may shift, and the requested count or position may not appear. These side effects mainly masked count and spatial changes, so we rebuilt the diagnostic on before-and-after pairs from AnyEdit and MagicBrush, whose two images differ mainly in the edited attribute:

AttributeTarget-token weight ratio ↑95% CI
Count involving one or two1.3041.185 – 1.396
Count involving three or four1.0781.021 – 1.152
Color1.5211.375 – 1.624
Shape1.4021.293 – 1.537
Spatial1.1051.051 – 1.172

All ratios now exceed one. As the two count groups differ, we also compared individual count words:

Count wordonetwothreefourfive
Weight ratio ↑1.4291.2061.1151.0591.033

Enrichment is strongest for "one", which helps explain the counting gain: a common counting error is duplicating a subject that should appear once (Binyamin et al., CVPR 2025; Kang et al., WACV 2025), and ToPO's strong weight on the singular count word helps avoid it, as Figure 7 illustrates. Consistently, removing token conditioning lowers GenEval Count from 0.4969 to 0.4406.

Q5. User study

Sixteen participants rated all 300 prompts in both comparisons on overall preference, visual appeal, and prompt faithfulness, making a forced binary choice with identities hidden and order randomized. This gives 1,800 judgments per participant and 28,800 in total. Each pair is decided by the 16-vote majority, and an 8–8 split counts as a tie against ToPO.

ComparisonDimensionWin / Tie / LossWin rate ↑95% CIAgreement ↑
vs. BaseOverall198 / 41 / 6166.0%60.5 – 71.10.68
vs. BaseVisual220 / 42 / 3873.3%68.1 – 78.00.67
vs. BaseFaithfulness202 / 55 / 4367.3%61.8 – 72.40.63
vs. DPOOverall190 / 51 / 5963.3%57.7 – 68.60.64
vs. DPOVisual212 / 34 / 5470.7%65.3 – 75.50.70
vs. DPOFaithfulness196 / 55 / 4965.3%59.8 – 70.50.61

Excluding ties, ToPO wins 76.3% to 85.3% of decided pairs, and a two-sided sign test gives p < 1e-16 in every case. Agreement is the raw fraction of votes matching the majority.

Q7. Simpler spatial weights

VariantPickScore ↑HPSv2 ↑ImageReward ↑Aesthetic ↑CLIP ↑
ToPO21.5420.29030.8125.36128.393
Latent difference21.5530.28790.7915.35528.114

Replacing the error gap with the latent difference is slightly better on PickScore but worse on the other four metrics. Moreover, a latent difference is a single map shared by all timesteps, so it cannot provide the temporal factor that ToPO derives from the same error gap. We will add a pixel-difference variant in the revision.


We hope this clarifies our contribution and evidence, and we welcome any further questions.

Reviewer_yFmjOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Token-Oriented Preference Optimization (ToPO), a method for improving preference alignment in text-to-image diffusion models. Unlike Diffusion-DPO, which applies image-level preference supervision uniformly, ToPO constructs spatial and temporal weights using noise-prediction error differences from a frozen reference model. Cross-attention maps are used to associate these weights with prompt tokens. The resulting weights modify the Diffusion-DPO objective, together with an auxiliary pixel-midpoint ordering loss. Experiments on SD-1.5 and SDXL demonstrate improvements across preference and compositional benchmarks.

Critical strengths

The paper is well written, and the problem statement is clearly motivated. The proposed methodology is presented systematically, making the overall approach relatively easy to follow.

The idea of allocating global preference feedback across spatial locations, denoising timesteps, and prompt tokens is interesting. The method does not require additional local preference annotations or a separately trained reward model.

The authors have evaluated the method using multiple baselines, benchmarks, and evaluation metrics. I also appreciate the inclusion of a human study, matched three-seed experiments, and ablation studies examining the contributions of different components.

Critical weaknesses

W1. Insufficient validation of preference-relevant localization.

The authors use noise-prediction error differences to identify regions and timesteps for preference allocation. However, higher residual disagreement does not necessarily indicate that a particular region or timestep is responsible for the human preference. Although the authors provide routing visualizations and ablations, a more direct evaluation of the semantic correctness of the proposed localization would strengthen the central claim.

W2. Limited effectiveness of token conditioning for certain compositional attributes.

In Appendix Table 10, the token-modulation diagnostic shows improvements for color, shape, and texture, but not for counting and spatial relations. The reported ratios for counting (0.941) and spatial relations (0.918) are below one and are not statistically significant. This raises questions about how reliably the proposed token-conditioning mechanism captures different semantic constraints.

W3. Insufficient justification of some methodological choices.

The authors construct a separable spatial–temporal weighting function using four fixed denoising anchors and cross-attention from the preferred image. Although component ablations are provided, comparisons with non-separable weighting, alternative anchor configurations, and loser-branch or symmetric attention would help justify these design choices.

W4. Inconsistent improvements in matched comparisons and across benchmarks.

Although ToPO achieves encouraging results, its improvements are not consistent across all metrics. In the matched three-seed SDXL comparison (Appendix Table 5), Diffusion-DPO performs better on PickScore and aesthetic score. Additionally, the SDXL T2I-CompBench average is nearly identical to Diffusion-DPO (0.4975 vs. 0.4974). These results deserve more discussion, particularly when making broad performance claims.

W5. Missing computational efficiency comparison.

ToPO requires additional frozen-reference evaluations at four denoising anchors for both preference branches, along with cross-attention extraction and an auxiliary loss. Although the authors emphasize achieving strong results within 500 updates, the paper does not provide a matched comparison of training time, GPU-hours, or memory consumption. Equal update counts do not necessarily imply equal computational cost.

W6. Limited statistical details of the human evaluation.

The authors conduct a blind human study with 16 participants over 300 prompts, which is appreciated. However, the reporting would benefit from clearer information about the number of judgments per participant and prompt, inter-rater agreement, and confidence intervals for the reported preference rates.

Questions for authors

I think the paper is well written and presents an interesting approach to preference alignment. I also appreciate the authors' effort to provide extensive evaluations and controlled ablation studies. However, I believe that the concerns highlighted in the weaknesses section, particularly those concerning the validity of preference allocation and experimental comparisons, deserve further clarification. I would be happy to reconsider my score if the authors adequately address these concerns during the rebuttal.

3 comments
AuthorsOpenReview ↗

Response part 1

Thank you for the detailed review. We have carefully considered each concern and conducted additional experiments. Table numbers follow the revised PDF (original in parentheses); revisions are marked in blue. We divided our response into three parts.


W1. Preference-relevant localization

We agree that a large residual difference alone does not show causality, and ToPO does not rely on it: the label and the signed term δ+−δ−\delta^+-\delta^- (Eq. 12) fix the direction, while residual contrast only decides where and when the update lands. The allocation thus needs to (i) concentrate on content where the images differ and (ii) favor differences named in the prompt. We test both.

Known targets. Pick-a-Pic has no region annotations, so we use before/after pairs from AnyEdit and MagicBrush, where the edit instruction specifies the intended change. ToPO's token weights favor the target token in all five groups:

AttributeTarget/other-token weight ratio ↑ [95% CI]
Count (involving 1/2)1.304 [1.185, 1.396]
Count (involving 3/4)1.078 [1.021, 1.152]
Color1.521 [1.375, 1.624]
Shape1.402 [1.293, 1.537]
Spatial1.105 [1.051, 1.172]

Interventions (SD-1.5, matched settings, Table 3):

ConfigurationPickScore ↑ImageReward ↑GenEval Count ↑
ToPO21.5420.8120.4969
Uniform-WW21.2370.7690.4418
Shuffled-WW (same values, permuted)21.3820.7410.4375
wsw_s / wτw_\tau / wk≡1w_k\equiv121.247 / 21.235 / 21.3740.808 / 0.805 / 0.8030.4219 / 0.4500 / 0.4406

Shuffling keeps the weight values yet lowers performance, and removing any factor lowers PickScore, so gains depend on where weights go. Human-marked regions would be a more direct test, as noted in Appendix S7.4, but both required properties hold, and blind raters prefer ToPO over Diffusion-DPO in 63.3%–70.7% of SDXL pairs.

W2. Count and spatial tokens

Thank you for pointing this out. The original diagnostic, now Table 12, used SD-1.5 pairs generated with the same prompt template and seed, differing only in the target attribute word. Even so, SD-1.5 does not always confine the change to that attribute: the layout or background may shift, and the requested count or position is not always produced. The residual contrast thus mixed the intended change with unrelated ones, which mainly masked count and spatial changes, while local attributes such as color and shape still stood out.

We therefore replaced regenerated pairs with before/after pairs from AnyEdit and MagicBrush, public instruction-editing datasets in which the two images differ mainly in the edited attribute. With the final configuration of four anchors and λ=0.5\lambda=0.5, every group now has a 95% CI above one, including count and spatial (Table A in W1). Since the attribution settings also changed, we do not credit input quality alone; the original results remain in Table 12. Count and spatial stay weaker than color and shape, as their evidence spreads over several objects or locations.

The token pathway also matters for generation: removing it lowers GenEval Count from 0.4969 to 0.4406, and against Diffusion-DPO on SD-1.5, ToPO raises GenEval Count from 0.3938 to 0.4969, Position from 0.0625 to 0.0775, and T2I-CompBench Spatial from 0.1309 to 0.1816.

To understand why the two count groups differ, we compared count expressions:

Table B. Target-token enrichment ↑ by count expression, with pair-bootstrap 95% CIs (Table 4).

onetwothreefourfive
1.429 [1.281, 1.531]1.206 [1.117, 1.262]1.115 [1.034, 1.126]1.059 [0.988, 1.109]1.033 [0.956, 1.075]

Enrichment is strongest for "one" and weakens as the count grows; the intervals for four and five include one. The strong weight on the singular count word helps ToPO keep a requested single subject instead of duplicating it, a known failure of text-to-image models reported by Binyamin et al. (CVPR 2025) and Kang et al. (WACV 2025). In Figure 7, row 5, for "a capybara made of voxels", SDXL Base and MaPO generate a herd, whereas ToPO generates one. You can see them in Section 5.6 and Appendix S7.2.

AuthorsOpenReview ↗

Response part 2

W3. Design choices

Thank you for pointing this out. We trained all variants under matched SD-1.5 settings (App. S7.3, Table 16):

VariantPickScore ↑HPSv2 ↑ImageReward ↑CLIP ↑
Default ToPO21.5420.29030.81228.393
Non-separable21.5400.28920.82028.406
Two anchors {50, 949}21.3070.27910.79228.117
Symmetric attention21.4090.28510.80528.120
Loser-only attention21.2890.27210.79128.090
  • Separability: the non-separable variant is mixed and brings no consistent gain, so we keep the simpler shared spatial map.
  • Anchors: dropping the intermediate anchors lowers all four metrics; we do not claim four anchors are optimal.
  • Attention: preferred-branch attention is best on all metrics, and loser-only attention even falls below wk≡1w_k\equiv1 (21.374 / 0.2842 / 0.803 / 28.271), i.e., worse than no token conditioning.

W4. Consistency across metrics and backbones

Thank you for this careful reading. We first need to correct an error in the original submission. Table 5, now Table 7, was described as a matched comparison, but it actually evaluates our checkpoint against the released Diffusion-DPO weights over three generation seeds, so their training is not matched. We apologize for this inaccurate wording and have corrected it. The matched comparisons in the paper are Fig. 5, with the same initialization, and Fig. 6, with the same GPU-hours.

We agree that ToPO does not improve every metric on SDXL: against the released Diffusion-DPO checkpoint, it is slightly lower on PickScore and the aesthetic score, and the two are tied on the T2I-CompBench average. These gaps, however, are small. The PickScore difference of 0.032, or 0.14%, is smaller than Diffusion-DPO's own seed-to-seed standard deviation of 0.053, and the aesthetic difference of 0.015 is within seed variation. Within the CompBench tie, ToPO leads on color, shape, texture, and non-spatial, while Diffusion-DPO is slightly ahead on count and spatial.

By contrast, ToPO's gains on the same SDXL checkpoints are clearer. ImageReward improves by 10.5%, HPSv2 and CLIP also rise, and the GenEval average increases from 0.5917 to 0.6168. More importantly, blind raters prefer ToPO over Diffusion-DPO in 63.3% of pairs overall and in 70.7% for visual appeal, which indicates that the small aesthetic-predictor gap does not reflect a perceptible loss in quality. On SD-1.5, ToPO leads on every reported metric, both in matched training and across generation seeds.

Following your suggestion, we have revised the abstract to claim "competitive performance" rather than state-of-the-art, and we discuss these backbone- and metric-specific differences in Appendix S7.4.

AuthorsOpenReview ↗

Response part 3

W5. Computational cost

We agree that equal updates do not imply equal compute. ToPO is indeed slower per update, by 1.88× on SD-1.5 and 1.98× on SDXL, with little extra memory (8×H200, Table 13):

SD-1.5 DPOSD-1.5 ToPOSDXL DPOSDXL ToPO
s/update ↓16.0730.2335.4970.45
Peak GiB/GPU ↓23.5024.3670.9471.01

However, training efficiency also depends on how much each update improves the model. We therefore compared both methods on SD-1.5 from the same initialization at matched GPU-hours (Fig. 6, Table 14):

DPO updates (GPU-h)PickScore ↑ / HPSv2 ↑ / IR ↑ToPO updates (GPU-h)PickScore ↑ / HPSv2 ↑ / IR ↑
84 (2.87)21.310 / .2860 / .72147 (3.05)21.357 / .2871 / .743
167 (5.66)21.358 / .2878 / .73494 (6.05)21.530 / .2892 / .783
251 (8.48)21.397 / .2887 / .719140 (8.99)21.539 / .2906 / .790
335 (11.28)21.413 / .2896 / .739187 (11.98)21.541 / .2908 / .818
418 (14.09)21.433 / .2894 / .741234 (14.98)21.553 / .2911 / .802
500 (16.91)21.426 / .2895 / .736260 (16.64)21.631 / .2912 / .815

At every comparable budget, with rows differing by under 0.9 GPU-h, ToPO is higher on all three metrics, and after only 140 updates it already exceeds Diffusion-DPO's 500-update result with 46.8% fewer GPU-hours. Thus, although each update costs more, ToPO converts compute into preference gains more efficiently overall (Section 5.5, Appendix S7.1).

W6. Human-study details (App. S3.1, Table 6)

16 participants rated all 300 prompts in both comparisons on three dimensions (forced binary choice; identities hidden, position and order randomized): 1,800 judgments per participant, 96 per prompt, 28,800 in total. Each pair is decided by the majority of 16 votes; 8–8 counts as a tie and stays in the denominator, so ties count against ToPO (we corrected the main text, which said ties were allowed). "Agree." is the fraction of individual votes matching the pair majority.

vs.DimensionW / T / LWin rate ↑ [95% Wilson CI]Agree. ↑
BaseOverall198 / 41 / 6166.0% [60.5, 71.1]0.68
Visual220 / 42 / 3873.3% [68.1, 78.0]0.67
Faithfulness202 / 55 / 4367.3% [61.8, 72.4]0.63
DPOOverall190 / 51 / 5963.3% [57.7, 68.6]0.64
Visual212 / 34 / 5470.7% [65.3, 75.5]0.70
Faithfulness196 / 55 / 4965.3% [59.8, 70.5]0.61

Excluding ties, ToPO wins 76.3%–85.3% of decided pairs (sign test, p<10−16p<10^{-16} in all cases). Agreement rates are raw, not chance-corrected; individual judgments vary, as expected for subjective preferences, hence the 16-vote aggregation.


We hope these responses address your concerns and would be grateful if you would consider updating your score. We are happy to answer any further questions.

Reviewer_kCPCOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Token-Oriented Preference Optimization (ToPO) for aligning text-to-image diffusion models with human preferences. ToPO reweights the Diffusion-DPO objective using: Spatial residual differences between preferred and dispreferred images. Temporal weighting over denoising steps. Preferred-branch cross-attention to assign importance to prompt tokens. An auxiliary pixel-midpoint ordering loss. The method requires no local preference annotations and no separately trained reward model. Experiments on SD-1.5 and SDXL report improvements over Diffusion-DPO and several other baselines on preference and compositional benchmarks.

Critical strengths
  1. Clear and practically motivated formulation of spatial-temporal preference allocation.
  2. Drop-in extension of Diffusion-DPO without requiring a learned reward model or local annotations.
  3. Good methodological decomposition into spatial, temporal, and token-conditioned factors.
  4. Useful ablations, including uniform, shuffled, component-removal, and midpoint-loss controls.
  5. Evaluation on both SD-1.5 and SDXL.
  6. Includes compositional benchmarks rather than relying only on generic preference scores.
Critical weaknesses
  1. The core residual-gap attribution is heuristic and not sufficiently connected to human preference causality.
  2. Token conditioning is not consistently validated: count and spatial tokens receive less weight in the controlled diagnostic.
  3. Limited matched-seed evidence, especially for the stronger SDXL claims.
  4. Insufficient statistical analysis for automatic metrics and the human study.
  5. Baseline comparisons are not uniformly compute-matched or initialization-matched.
Questions for authors
  1. How does the residual contrast correlate with human-identified preference regions? Can you provide localization or intervention experiments?
  2. Why does token modulation favor color, shape, and texture but not count or spatial-relation tokens?
2 comments
AuthorsOpenReview ↗

Response part 1

Thank you for the careful review and the positive assessment. We have taken each of your comments seriously and conducted additional experiments to address your concerns. Below, we address the weaknesses and questions in three groups; all revisions in the updated PDF are marked in blue. We devided the response into two parts.


[W1, Q1] How does residual contrast relate to human preference?

Residual contrast decides where updates go, not which image is better. The preference label and the signed term δ+−δ−\delta^+-\delta^- (Eq. 12) fix the update direction; residual contrast only decides where and when that update lands. It therefore needs a weaker property than preference causality: it should concentrate on content where the two images actually differ, and token conditioning should favor differences named in the prompt.

Localization. Pick-a-Pic has no region annotations, and holistic aesthetic preferences between two independent samples rarely reduce to one well-defined region. We therefore test localization on before/after pairs from AnyEdit and MagicBrush, where the human-specified difference is known by construction: the edit instruction names the changed attribute. On these pairs, target tokens receive 1.08×1.08\times–1.52×1.52\times the weight of other content tokens, and every 95% CI lies above one (table under Q2).

Interventions (SD-1.5, matched settings, Table 3):

InterventionPickScore ↑ImageReward ↑GenEval Count ↑
ToPO21.5420.8120.4969
Uniform-WW (no allocation)21.2370.7690.4418
Shuffled-WW (same values, permuted coordinates)21.3820.7410.4375
wk≡1w_k\equiv1 (no token conditioning)21.3740.8030.4406

Shuffled-WW keeps the exact weights but moves them to other coordinates, and performance drops: placement, not only magnitude, drives the gains. Using non-preferred-branch attention also underperforms (PickScore 21.289; Table 16).

We agree that these results do not directly measure agreement with human-marked regions, and we state this in Appendix S7.4. Still, they test the two properties that allocation actually requires: concentration on real differences, shown by the edited pairs, and dependence on placement, shown by the interventions. Both hold. The blind study then confirms that the resulting updates produce images people prefer (W4 below).


[W2, Q2] Why were count and spatial tokens weaker?

Thank you for this question; it led us to find a weakness in the original diagnostic. That diagnostic regenerated SD-1.5 images with the same seed after changing one prompt attribute. Same-seed regeneration does not guarantee correct counts or positions and often changes other content, so residual contrast partly reflected unintended differences. Local edits (color, shape, texture) survived this noise; count and spatial edits, which add or move objects, did not.

We rebuilt the diagnostic on AnyEdit/MagicBrush edited pairs with the final configuration (four anchors, λ=0.5\lambda=0.5). All five groups, including count and spatial, now have 95% CIs above one:

AttributeTarget/other-token weight ratio ↑ [95% CI]
Count (involving 1/2)1.304 [1.185, 1.396]
Count (involving 3/4)1.078 [1.021, 1.152]
Color1.521 [1.375, 1.624]
Shape1.402 [1.293, 1.537]
Spatial1.105 [1.051, 1.172]

Since the attribution settings also changed, we do not credit input quality alone. Count and spatial remain weaker than color and shape because their evidence spreads over several objects or locations.

Generation results. Removing token conditioning (wk≡1w_k\equiv1) lowers GenEval Count from 0.4969 to 0.4406. Relative to Diffusion-DPO on SD-1.5, ToPO improves GenEval Count (0.3938 → 0.4969), GenEval Position (0.0625 → 0.0775), and T2I-CompBench Spatial (0.1309 → 0.1816).

Since the two count groups differ (1.304 vs. 1.078), we further compared individual count expressions:

ExpressionRatio ↑ [95% CI]
one1.429 [1.281, 1.531]
two1.206 [1.117, 1.262]
three1.115 [1.034, 1.126]
four1.059 [0.988, 1.109]
five1.033 [0.956, 1.075]

Enrichment is highest for "one" and decreases as the count grows; the intervals for four and five include one, so we treat this trend as descriptive. This emphasis is practically relevant: text-to-image models often duplicate a subject that is requested only once (Binyamin et al., CVPR 2025; Kang et al., WACV 2025), and up-weighting the singular token helps ToPO keep a single subject. Figure 7 (row 5) shows a clear example for "a capybara made of voxels sitting in a field": SDXL Base and MaPO generate a herd of capybaras, while ToPO generates a single one. Consistently, ToPO reaches a perfect GenEval single-object score on SDXL (1.000 vs. 0.9875 for Diffusion-DPO). These results are in Section 5.6 and Appendix S7.2.

AuthorsOpenReview ↗

Response part 2

[W3–W5] Matched comparisons, seeds, and statistics

Initialization and compute (W5). We agree that Table 1 compares released checkpoints and does not control initialization or compute; the paper labels it as descriptive. The controlled evidence is separate:

  • Same initialization (Fig. 5). Diffusion-DPO is retrained from the same SFT-winners checkpoint with identical data and settings. At 500 updates, ToPO reaches 21.542 / 0.2903 / 0.812 vs. 21.344 / 0.2862 / 0.718 (PickScore / HPSv2 / ImageReward), with 95% prompt-bootstrap bands.
  • Same GPU-hours (Fig. 6). ToPO costs 1.88×1.88\times per update (Table 13). Even so, after 140 updates (8.99 GPU-h) it exceeds Diffusion-DPO's 500-update result (16.91 GPU-h) on all three metrics, using 46.8% fewer GPU-hours.

Seeds and SDXL (W3). Over three generation seeds (Table 7), ToPO is higher on all five metrics on SD-1.5. On SDXL it is higher on HPSv2 (+0.0132 ± 0.0016), ImageReward (+0.098 ± 0.020), and CLIP (+0.507 ± 0.159), but not on PickScore (−0.032 ± 0.026) or Aesthetic (−0.015 ± 0.012). We have narrowed our SDXL claims accordingly: the abstract now says "competitive", and Appendix S7.4 states that SDXL gains are not uniform. Training-seed replication is not yet available; Appendix S6 states this limitation.

Two points mitigate this concern:

  • The SDXL deficits are small: PickScore differs by 0.14%, while ImageReward improves by 10.5%. On the same SDXL checkpoints, human raters prefer ToPO over Diffusion-DPO.
  • On SD-1.5, Figs. 5 and 6 come from two independently trained ToPO/Diffusion-DPO run pairs, and both show ToPO ahead on all three preference metrics.

Human-study statistics (W4). Each rater made a forced binary choice for every pair and dimension, with no tie option. A pair counts as a tie only when its 16 votes split 8–8, and ties remain in the denominator, so they count against ToPO. Under this conservative counting, Table 6 reports 95% Wilson intervals whose lowest lower bound is 57.7% (overall preference vs. Diffusion-DPO). Excluding tied pairs, ToPO wins 76.3%–85.3% of decided pairs, and a two-sided sign test gives p<10−16p<10^{-16} in all six comparisons (Appendix S3.1). We have also corrected the main-text description, which previously said that ties were allowed, to match this protocol.

These intervals are conditional on the fixed 16-rater panel. Individual judgments vary, as expected for subjective preferences: the fraction of votes agreeing with the pair-level majority is 0.61–0.70 across comparisons and dimensions (Appendix S3.1). Even so, ToPO wins a clear majority in all three dimensions against both comparators, so the conclusion does not depend on borderline cases.


We hope these clarifications address your concerns and are happy to provide further details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.