acceptodds
Under review as a conference paper at ICLR 2027

PSTune: Text-to-Image Reward Updates That Transfer Only When Prompt and State Match

Abstract

Reward fine-tuning raises a text-to-image model’s score under a learned preference reward but often lowers what the reward does not measure: tuning a LoRA on FLUX.1-dev with HPSv2.1 reduces GenEval accuracy from 67.5% to 63.1% and within-prompt LPIPS from 54.1 to 46.6. We trace this loss to how a reward update spreads. The gradient from one image changes weights that every image uses, and its first-order effect on another image is set by the cross-Jacobian of the two generations. For an adapter that reads concatenated prompt and state features, this cross-Jacobian is the sum of a prompt similarity and a state similarity, so the update reaches any image that matches in either; it should reach an image only as far as both match. PSTune builds this rule into one three-way tensor that maps the normalized prompt embedding and the centered latent–velocity feature at each location to a velocity correction for a frozen generator. With detached features and scalar SGD, the change one image’s update causes in another factors exactly into prompt similarity, state overlap, and the source gradient, and centering removes the image-wide gradient term that would bypass this product. On FLUX.1-dev, SD3.5-Medium, and SDXL, PSTune raises GenEval ImageReward over matched LoRA by 0.29–0.31 with direct training and 0.20–0.25 with policy gradients, at the frozen model’s accuracy and LPIPS.

Primary area:
generative models
Keywords:
text-to-image models, reward fine-tuning, update propagation, generative models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_cz47OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper presents a method for mitigating reward overoptimization when fine-tuning text-to-image models. Unlike traditional LoRA-based fine-tuning, which modifies the denoising model's parameters, the proposed method freezes the denoiser and trains a lightweight velocity corrector. More specifically, the corrector is parameterized by a three-way tensor conditioned on prompt and local denoising-state features, such that gradient updates transfer between samples according to the product of their prompt and state similarities. Experiments across multiple training objectives (direct reward backpropagation and Flow-GRPO) and models (SD3.5, SDXL, FLUX) demonstrate promising improvements in reward quality while largely preserving compositional accuracy and diversity compared with LoRA fine-tuning.

Critical strengths

The mathematical formulation appears sound and elegant, with a clear motivation for the proposed parameterization.

The experimental results, including the qualitative comparisons and mechanism-focused ablations, are compelling.

Critical weaknesses

My biggest concern is that the experiments may not fully disentangle the benefits of restricting the trainable model from those of the proposed gradient-localization mechanism. For example, the FLUX LoRA baseline has approximately 131M trainable parameters, whereas the proposed tensor adapter has only 6.3M. The reduction in reward overoptimization could potentially be explained by this difference in capacity, rather than by gradient localization itself.

While Section 5.2 attempts to address this through the wrong-prompt and wrong-state ablations, I would be interested in comparisons against simpler, similarly sized correctors, such as prompt-only, state-only, or additive prompt–state adapters.

Minor concern: Appendix C.2 mentions that ImageReward is used for learning-rate selection. Consequently, I would be cautious about treating ImageReward as a fully independent evaluation metric, even though HPSv2.1 is the training reward.

Questions for authors

At a higher level, while the proposed parameterization and gradient-localization principle are intuitively appealing, could this strong inductive bias also impose limitations? For example, could suppressing image-wide updates or requiring both prompt and state similarity prevent beneficial transfer of general improvements especially when looking beyond text-to-image models?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cz47 (1/2): capacity versus gradient localization

(1/2)

We are grateful for a clear and generous review. You found the formulation sound and elegant with a clear motivation, and the qualitative comparisons and mechanism-focused ablations compelling; that is the reading we hoped for. Your main concern goes to the heart of the paper, and you named the exact controls that answer it. We ran all of them.

Is the benefit capacity (6.3M vs. 131M) or gradient localization? You are right that Sec. 5.2 did not settle this. Wrong-index controls keep PSTune's capacity but corrupt its update, which shows that the index matters, not that the product beats other small adapters. The revised Sec. 5.2 adds Table R1 (main text, full metrics in Table R8, App. G.7). All adapters sit at PSTune's placement, read the same detached RMS-normalized indices, and use scalar SGD, clipping, and the same three-candidate search. All four run on SD3.5-Medium (GenEval IR / accuracy / LPIPS):

AdapterParamsKernelIRAcc.LPIPS
LoRA r32 (Table 1)47Mhidden activations0.78459.849.23
Prompt-only Θp\Theta p33kprompt0.71663.256.61
State-only Θs\Theta s (centered)0.5kstate0.90262.655.29
Additive Θ[p;s]\Theta[p;s]33ksum0.87162.054.08
Additive, parameter-matched1.0Msum0.87961.352.57
PSTune1.0Mproduct1.07764.157.95

On FLUX we repeated the additive and state-only adapters: IR 1.006 and 1.032 against 1.201 for PSTune, with accuracy and LPIPS below the base.

How we read it:

  • Capacity does not explain the gap. The parameter-matched additive adapter has exactly the tensor's parameter count (random Fourier features of pp and ss through one trainable matrix). It gains more HPS than the small additive adapter and loses more accuracy (2.4 against 1.7 points), so adding parameters to a sum makes the spread worse, not better. In the other direction, LoRA at rank 1 (1.5M parameters, close to the tensor's 1.0M) still loses 3.6 of rank 32's 3.9 accuracy points (Table R2).
  • One factor is not enough. Prompt-only can only add a per-prompt constant to the velocity and gains little. Centered state-only is the strongest alternative, yet it shares one map across all prompts, so a change written for one request reaches every request with similar local states; it stays at least 0.17 below PSTune in ImageReward and loses accuracy and LPIPS.
  • The sum is the direct test. The additive adapter reaches PSTune's training HPS (within 0.14) yet trails by 0.20 to 0.21 in ImageReward on both backbones. Adding the prompt to the state-only adapter lowers ImageReward; multiplying by it gives PSTune.
  • Adherence agrees. Both FLUX alternatives fall below the frozen base in GenEval accuracy (66.1 and 65.4 against 67.5), while PSTune stays at 67.9; GenEval checks each requested object, count, color, and position, so this is a direct measure of text adherence.
  • Update size is not the cause. Rescaling the additive update to PSTune's response norm leaves a gap of 0.22 (Table R8).

With placement, optimizer, selection, and parameter count held fixed, the remaining difference is the kernel, which is the claim of Sec. 3.2. We are grateful for this suggestion; it turned the central argument of the paper from a motivated design into a measured one.

AuthorsOpenReview ↗

Response to Reviewer cz47 (2/2): ImageReward in selection, and the limits of the inductive bias

(2/2)

Minor: ImageReward is used for learning-rate selection. Agreed, and the paper now says so where ImageReward is introduced (Sec. 5) and in App. C.2. To take selection out of the comparison, Table R3 (App. G.2) trains every learning-rate candidate of both methods to 1,024 updates with three seeds on SD3.5:

  • on SD3.5, every PSTune candidate exceeds every LoRA candidate in ImageReward, by at least 0.058;
  • if each method keeps its highest-HPS candidate, so that ImageReward plays no role, PSTune leads by 0.29, about the margin in Table 1;
  • PickScore enters neither training nor selection, and PSTune leads matched LoRA on it.

Q. Could the inductive bias block useful transfer, in T2I and beyond? Yes, and we think the honest answer is that it costs something in three measurable ways. The revised Limitations paragraph lists all three.

  1. Image-wide improvements. Forward centering removes image-wide velocity corrections, such as a global brightness or contrast shift. The forward-centering ablation measures the price: with raw forward features (F/CFF/CF), text match is better by 4.2 to 6.9 net-win points, while PSTune is preferred on realism and overall (Table 11). When a reward is mostly global, the F/CFF/CF variant is the better choice, and the update can still be centered.
  2. Generally useful transfer. The product attenuates updates that would help unrelated prompts. One visible consequence is speed: after 1,024 updates PSTune's training HPS gain stays below LoRA's on every backbone (Tables 1 and 2). At PSTune's HPS, LoRA has already paid 51 to 67% of its final accuracy and LPIPS loss (Table R2), so the attenuation slows training without costing adherence or variety.
  3. Rewards that need new content. With the GenEval reward on SD3.5 (Table R5), PSTune gains 38% of LoRA's improvement while keeping ImageReward and LPIPS. Here the restriction binds.

Beyond text-to-image, Eq. (1) holds for any adapter, so the reach of an update can be measured in any conditional generator. The particular choice in PSTune (a pooled condition times a centered local state) suits rewards that change how requested content is rendered. Settings where improvements should spread broadly, such as a global style reward or a policy whose useful skills are shared across tasks, call for a different kernel, for example a sum with a small product term or uncentered forward features. We now present the product as the right kernel for preference rewards in text-to-image fine-tuning, not as a universal rule, and the conclusion stays within that scope.

Summary for the discussion

  • The capacity explanation is tested directly: a sum-kernel adapter with exactly the tensor's parameter count, and LoRA rank 1 with about as many parameters as the tensor, both lose the accuracy and variety that PSTune keeps (Tables R1, R2).
  • The product beats prompt-only, state-only, and additive kernels at identical placement, optimizer, and search, also at matched update norm (Tables R1, R8).
  • ImageReward selection is removed without changing the conclusion (Table R3).
  • The costs of the inductive bias are now quantified and stated.

Thank you again for a review that was both encouraging and precise about what was missing. If the new comparisons address your concern about capacity, we would be very grateful if you would consider raising your score and supporting the paper in the AC discussion.

Thanks! I think this definitely addresses most of my concerns regarding the soundness of the results, and I'm happy supporting the paper wrt acceptance.

Reviewer_QJCSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how reward updates transfer between generations during text-to-image fine-tuning. PSTune introduces a bilinear adapter that maps normalized prompt embeddings and centered local latent–velocity features to velocity corrections for a frozen generator. With detached features and scalar SGD, the fixed-state response to a source update factors exactly into prompt similarity, state overlap, and the source gradient. Spatial centering removes an image-wide component of the update. Experiments on FLUX.1-dev, SD3.5-Medium, and SDXL, using direct and policy-gradient training, report improved held-out preference scores over matched LoRA training while approximately preserving the base models’ aggregate adherence and diversity.

Critical strengths
  1. The paper presents a useful and transparent perspective on reward adaptation. Explicitly designing how an update affects other generations is an appealing contribution. The fixed-state factorization and centering results appear correct under the stated assumptions. The manuscript also carefully distinguishes these exact identities from regenerated trajectories and explains the roles of detached features and scalar optimization.

  2. The empirical evaluation is substantial, with informative interventions. The experiments cover three backbones, two training objectives, several compositional benchmarks, diversity, and blinded human judgments. For example, FLUX direct training improves GenEval ImageReward from 0.904 with LoRA to 1.201, accuracy from 63.1% to 67.9%, and LPIPS from 46.63 to 54.27. Separating forward centering from update centering, together with the paired removal of the image-wide term, provides particularly useful evidence about the design.

Critical weaknesses
  1. The prompt-mismatch prediction diagnostic needs reconciliation with Eq. (13). Appendix F.4, lines 1649–1652, describes a wrong-prompt predictor that substitutes the source prompt index while predicting the same correct-update endpoint changes. If the other factors in Eq. (13) remain fixed, this implies, for nonzero denominators, $
dijwrong=dijcorrectpj⊤pwrongpj⊤pi.d^{\mathrm{wrong}}_{ij} = d^{\mathrm{correct}}_{ij} \frac{p_j^\top p_{\mathrm{wrong}}}{p_j^\top p_i}.

$ Therefore, the predictions differ in sign only when the two prompt factors have opposite signs, apart from zero cases. The reported sign-agreement decrease from 95.4% to 48.8% consequently requires such sign reversals on at least approximately 46.6% of cases under the reported averaging. The manuscript does not provide the corresponding cosine-sign statistics. Furthermore, the explanation at lines 1698–1703 suggests that the state-gradient direction also becomes incorrect, although a prompt-only substitution leaves that term unchanged. The actual implementation and empirical prompt-factor distribution are needed to reconcile this important mechanism diagnostic with the equation.

  1. The experiments do not sufficiently isolate the benefit of the product kernel from other constraints imposed by the adapter. Section 3.2 motivates PSTune through a comparison between additive and multiplicative feature combinations, but a correctly trained additive adapter is not evaluated. Relative to LoRA, PSTune changes placement, capacity, optimizer, and functional constraints simultaneously. The norm-matched controls establish that update direction matters within PSTune, but swapped indices deliberately disrupt gradient assignment and do not replace a comparison with a legitimate alternative parameterization. A focused output-level additive or centered state-only baseline would directly strengthen the central mechanism claim. Since PSTune also consistently achieves lower training HPS than LoRA, a comparison at matched HPS gain would help distinguish improved routing from more conservative adaptation.
Questions for authors
  1. Clarification: For Figure 25(d), what fraction of evaluated cases have $
(pj⊤pi)(pj⊤pwrong)<0?(p_j^\top p_i)(p_j^\top p_{\mathrm{wrong}})<0?

$. Does the wrong-prompt predictor change any state or gradient tensors in addition to the prompt index? Please specify the exact computation and reconcile the reported sign-agreement rates with Eq. (13).

  1. Targeted validation: Does PSTune retain its preference–adherence–diversity advantage over a correctly trained additive or centered state-only output adapter under comparable optimization and selection protocols? How does the comparison with LoRA change at matched training-HPS gain? These comparisons would materially affect my assessment of the proposed mechanism.
2 comments
AuthorsOpenReview ↗

Response to Reviewer QJCS (1/2): Figure 25(d) and Eq. (13), the exact computation

(1/2)

Thank you for an exceptionally careful review. You recognized what we most wanted to get across, that designing how one update affects other generations is a useful perspective, and you checked that the factorization and centering results hold under the stated assumptions and that the manuscript keeps exact identities apart from regenerated trajectories. Your algebra on Fig. 25(d) is correct, and it found a real error in our description. We are grateful for it.

W1 / Q1. Reconciling the wrong-prompt sign agreement with Eq. (13).

Your derivation holds exactly. When only the prompt index changes, dijwrong=dij (pj⊤pwrong)/(pj⊤pi)d^{\rm wrong}_{ij}=d_{ij}\,(p_j^\top p_{\rm wrong})/(p_j^\top p_i), so the sign can change only where the two cosines have opposite signs.

Fraction of cases with (pj⊤pi)(pj⊤pwrong)<0(p_j^\top p_i)(p_j^\top p_{\rm wrong})<0. Over all 6,636 FLUX/GenEval (source, target) pairs at update 1,024: 0 pairs (0.0%). Pooled text embeddings are not centered, and every cosine is positive: the minimum of pj⊤pip_j^\top p_i is 0.071 (median 0.346), and the minimum of pj⊤pwrongp_j^\top p_{\rm wrong} is 0.064 (median 0.339) (Table R4, App. G.3).

Consequence. A strictly prompt-only predictor has sign agreement 95.4%, identical to the correct predictor, as your algebra requires.

Does the predictor change any state or gradient tensor? Yes, and this is the error. The 48.8% series did not substitute the prompt index alone. It took the cyclically shifted batch sample's whole contribution, (pi′,Si′,Gi′)(p_{i'}, S_{i'}, G_{i'}), and computed −η(pj⊤pi′)⟨Hj,SjSi′⊤Gi′⟩F-\eta(p_j^\top p_{i'})\langle H_j, S_jS_{i'}^\top G_{i'}\rangle_F against the same correct-update outcome yijy_{ij}. That is a wrong-source control. Its direction SjSi′⊤Gi′S_jS_{i'}^\top G_{i'} belongs to an unrelated update, so chance-level agreement is what one should expect. App. F.4 described it as a prompt-only substitution, which was wrong, and the old explanation at lines 1698 to 1703 (a changed state-gradient direction) described the wrong-source computation, not a prompt-only one. You were right that the two did not fit together.

Changes. App. F.4 now defines the wrong-source predictor exactly; the Fig. 25 caption labels series (d) as wrong source and points to the prompt-only value; Table R4 reports both agreements; Sec. 5.3 states the prompt-only result.

The regenerated wrong-prompt arm of Table 6 follows the same algebra. Your observation also raises a question about Table 6, so we checked it. That arm does substitute only the prompt index, and only in the update (App. C.4 now says so explicitly). By Eq. (13) it keeps each target's direction and changes its weight from pj⊤pip_j^\top p_i to pj⊤pwrongp_j^\top p_{\rm wrong}. The weight is what matters. Read without its prompt factor, the source update fits targets whose requests resemble the source and misfits the average target: its alignment qij=⟨Hj,SjSi⊤Gi⟩Fq_{ij}=\langle H_j,S_jS_i^\top G_i\rangle_F rises with pj⊤pip_j^\top p_i (Spearman 0.37). The correct index weights targets by their similarity to the source; the shifted prompt weights them by their similarity to an unrelated request. The fixed-state prediction of Eq. (13) therefore gives:

FLUX/GenEval target ΔIR (10−310^{-3})fixed-state predictionregenerated (Table 6)
correct index3.874.19
prompt-only wrong index-2.36−2.80

So the negative mean of that arm comes from reweighting, not from any sign flip, and Eq. (13) predicts it. The source keeps part of its own gain through pi⊤pwrongp_i^\top p_{\rm wrong} (median 0.318), which fits the 37% of the FLUX source HPS gain that the arm retains. The paper now states the resulting division of labor (Sec. 4.3, 5.3): the prompt factor sets how strongly each target moves; the centered state factor, which takes both signs, sets where and in which direction.

AuthorsOpenReview ↗

Response to Reviewer QJCS (2/2): legitimate alternative kernels and matched training HPS

(2/2)

W2 / Q2. Legitimate alternative kernels, and LoRA at matched HPS gain. You are right that PSTune differs from LoRA in placement, capacity, optimizer, and functional constraints at once, and that swapped indices disrupt PSTune's own assignment instead of replacing it with a legitimate alternative. We ran exactly the two targeted validations you asked for.

1. Output-level alternatives under the same protocol (Table R1 in Sec. 5.2; Table R8 in App. G.7). All adapters are added after native guidance, read the same detached RMS-normalized indices, start at zero, and use scalar SGD, clipping at norm one, and PSTune's three-candidate selection. Both adapters you named run on FLUX and SD3.5. GenEval ImageReward / accuracy / LPIPS:

AdapterFLUXSD3.5
Centered state-only Θs\Theta s1.032 / 66.1 / 52.430.902 / 62.6 / 55.29
Additive Θ[p;s]\Theta[p;s]1.006 / 65.4 / 50.810.871 / 62.0 / 54.08
PSTune1.201 / 67.9 / 54.271.077 / 64.1 / 57.95
Frozen base0.814 / 67.5 / 54.140.677 / 63.7 / 57.85

The additive adapter reaches PSTune's training HPS (within 0.14) and still trails by 0.20 to 0.21 ImageReward, with accuracy and LPIPS below the base on both backbones. The centered state-only adapter, the strongest alternative, trails by at least 0.17 and also loses accuracy and LPIPS. On SD3.5, a prompt-only adapter and a parameter-matched additive adapter do no better, and rescaling the additive update to PSTune's response norm leaves a gap of 0.22. So the preference, adherence, and diversity advantage remains against legitimate alternatives, and since everything except the kernel is shared, it isolates the product.

2. Matched training-HPS gain (Table R2, App. G.1). We moved LoRA to PSTune's HPS in three ways: the 512-update checkpoints on all three backbones, and a lower learning rate and a KL penalty on SD3.5, with β\beta chosen on validation HPS only. At matched HPS (within 0.14), LoRA still trails PSTune by 0.30 to 0.33 ImageReward and already shows 51 to 67% of its final accuracy and LPIPS loss. Under your framing, this favors improved routing over more conservative adaptation: at the same training reward, LoRA pays in accuracy and variety and PSTune does not. PSTune does need more updates per HPS point, which the Limitations now state.

Summary of what changed in response to your review

  • Fig. 25(d) is reconciled with Eq. (13): zero opposite-sign pairs, prompt-only agreement 95.4%, and the 48.8% identified as a wrong-source control with a corrected description (App. F.4, G.3, Table R4).
  • The wrong-prompt arm of Table 6 is now explained by Eq. (13) itself, with the fixed-state prediction matching the sign and size of the regenerated mean.
  • The product is compared with correctly trained additive and centered state-only adapters on FLUX and SD3.5 (plus prompt-only and parameter-matched on SD3.5), and LoRA is compared at matched HPS.

Your two questions were, in your words, the comparisons that would materially affect your assessment, and they were the right ones to ask. We hope the results, and the correction of the diagnostic, address your concerns; if so, we would be grateful if you would reconsider your score. We are glad to provide any further detail, including per-pair statistics behind Table R4.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Summary of the revision (1/2): shared concerns and what changed

(1/2) Summary of the revision

We thank the three reviewers and the AC. The reviews were careful and constructive, down to the line numbers of an appendix diagnostic. Every point they raise has led to a change in the revised PDF, and several of the new comparisons were ones we should have run in the first place.

What the reviewers found valuable. Reviewer 3iLZ describes treating "where a reward update lands" as a design variable as "clean and novel in framing" and calls the mechanism evidence "careful", noting that each control "isolate[s] one component of the rule". Reviewer cz47 finds the formulation "sound and elegant" and the qualitative comparisons and mechanism-focused ablations "compelling". Reviewer QJCS calls the perspective "useful and transparent", finds the factorization and centering results "correct under the stated assumptions", and describes the evaluation as "substantial, with informative interventions".

Concerns shared across reviews.

  1. Does the gain come from the product kernel, or from PSTune's small size, placement, and optimizer? (3iLZ, cz47, QJCS)
  2. Does PSTune simply optimize less, given its lower training HPS? (3iLZ, QJCS)
  3. ImageReward also scores the learning-rate search. (3iLZ, cz47)
  4. Reviewer QJCS found that the wrong-prompt value in Fig. 25(d) cannot follow from Eq. (13) under a prompt-only substitution. We traced it: the predictor used another sample's whole contribution, and our description was wrong. Details are in our reply to QJCS.

Changes in the revised PDF. All new or changed text is blue. New tables are numbered R1 to R8 so that Tables 1 to 11 keep the numbers used in the reviews.

  • Sec. 5.2, Table R1 (main text), App. G.7, Table R8: additive, prompt-only, centered state-only, and parameter-matched additive adapters trained with PSTune's placement, detached indices, scalar SGD, and search (all four on SD3.5-Medium; additive and state-only also on FLUX).
  • Sec. 5.2, App. G.1, Table R2: LoRA at PSTune's training HPS (512-update checkpoints on all backbones; lower learning rate and KL penalty on SD3.5) and LoRA rank 1.
  • Sec. 5.1, App. C.2, G.2, Table R3: all learning-rate candidates without selection (SD3.5), and selection by HPS instead of ImageReward.
  • Sec. 5.3, App. C.4, F.4, G.3, Table R4, Fig. 25 caption: prompt-cosine statistics, the exact computation behind Fig. 25(d), and the fixed-state account of the wrong-prompt arm of Table 6.
  • App. G.4, Table R5: the GenEval reward on SD3.5, reported as a boundary of the method.
  • Sec. 5.1, App. G.5, Table R6: step masks for the final-step training question.
  • App. G.6, Table R7: Proposition 4 evaluated on real targets.
  • Abstract (one clause on where external methods lead), Sec. 4.1 and 4.3 wording, Limitations, Fig. 2 caption, AI-use statement.

The main text still ends on page 10. Every added experiment tests a claim the submission already made; none adds a component to the method. New training runs use SD3.5-Medium as the common backbone, FLUX repeats the two adapters named in the reviews, and analyses that need no training cover all three backbones (App. G, Scope). Part 2 summarizes the results.

AuthorsOpenReview ↗

Summary of the revision (2/2): results of the added experiments

(2/2) Results of the added experiments

All runs use the protocol of the paper (1024px, native samplers, HPSv2.1 on the same 8,192 prompts, 1,024 updates, seeds 17/29/43, three-seed means). New training runs use SD3.5-Medium as the common backbone; FLUX repeats the two adapters named in the reviews. IR is GenEval ImageReward.

Question (reviewers)Added comparisonResult
Product or sum? (3iLZ, cz47, QJCS)Additive adapter Θ[p;s]\Theta[p;s] with PSTune's placement, optimizer, and search, FLUX and SD3.5 (Table R1)HPS within 0.14 of PSTune, yet 0.20 to 0.21 lower IR, with accuracy and LPIPS losses PSTune does not have
Is one factor enough? (cz47, QJCS)Prompt-only (SD3.5) and centered state-only (FLUX, SD3.5) adapters (Table R1)every alternative stays at least 0.17 below PSTune in IR and loses accuracy and LPIPS that PSTune keeps
Capacity? (cz47)Additive adapter with the tensor's parameter count; LoRA rank 1 (SD3.5)the larger sum loses more accuracy (2.4 vs 1.7 points); rank 1 loses 3.6 of rank 32's 3.9 points
Matched update norm? (3iLZ)Additive update rescaled to PSTune's response norm, SD3.5 (Table R8)gap stays 0.22 IR
Optimizing less? (3iLZ, QJCS)LoRA at PSTune's HPS: 512-update checkpoints (3 backbones), lower LR and KL penalty (SD3.5) (Table R2)trails PSTune by 0.30 to 0.33 IR and already shows 51 to 67% of its final accuracy and LPIPS loss
IR used in selection (3iLZ, cz47)All learning rates, no selection, SD3.5 (Table R3)every PSTune candidate beats every LoRA candidate in IR (by at least 0.058); selecting by HPS gives a gap of 0.29
Fig. 25(d) vs Eq. (13) (QJCS)Prompt-cosine signs over 6,636 pairs (Table R4)0 pairs with opposite signs; prompt-only sign agreement 95.4%, identical to the correct predictor
Final-step training (3iLZ)Correction only on last 20% or first 80% of steps, 3 backbones (Table R6)last 20% alone keeps 67 to 74% of the IR gain; early steps add to it, with accuracy and LPIPS at the base level
Rewards needing new content (3iLZ, cz47)GenEval reward, matched PG condition, SD3.5 (Table R5)PSTune gains 38% of LoRA's improvement while keeping IR and LPIPS; stated as a limitation
Is Prop. 4 informative? (3iLZ)Bound-to-actual ratio, 2,212 targets per backbone (Table R7)median 2.9 to 5.1 with pair-wise constants; loose with global ones, so we use it qualitatively

Two results go against PSTune and are now stated in the paper. PSTune reaches a given training HPS more slowly than LoRA, and it learns content rewards only in part. Both follow from the product and from forward centering, the same design choices that protect accuracy and variety. External methods with 3 to 140 times our training images still lead on PickScore and on human text match and realism, and the abstract now says so.

We would be glad to answer further questions during the discussion.

AuthorsOpenReview ↗

Response to Reviewer 3iLZ (1/3): the sum-versus-product control and matched training reward

(1/3)

Thank you for a review that engages with the paper at the level we hoped a reader would. You describe treating "where a reward update lands" as a design variable, rather than another change to the objective, as clean and novel, and you single out exactly the controls we built to isolate one part of the rule at a time (Tables 4 to 6, Fig. 25). It is encouraging that the budget accounting, the paired CIs, and the SDXL coordinate mapping were noticed too. Your two main requests were the right ones: they are the comparisons that turn the mechanism story into a tested claim, and both are now in the paper.

W1. The product was never tested against a sum. You are right, and this was the most important gap. Table 4 corrupts PSTune's own update; it does not train a legitimate alternative. New Table R1 (Sec. 5.2, main text) and Table R8 (App. G.7) train adapters with PSTune's placement, detached RMS-normalized indices, scalar SGD, clipping, and three-candidate search:

  • additive u=Θ[p;s]u=\Theta[p;s], the sum kernel of Eq. (2);
  • prompt-only Θp\Theta p and centered state-only Θs\Theta s, one factor each;
  • a parameter-matched additive adapter that reads random Fourier features of pp and ss through exactly the tensor's parameter count (its kernel is still a sum).

All four run on SD3.5-Medium; the additive and state-only adapters also run on FLUX. GenEval results on FLUX:

AdapterHPSIRAcc.LPIPS
LoRA r32 (Table 1)34.620.90463.146.63
Additive33.471.00665.450.81
State-only (centered)33.181.03266.152.43
PSTune33.551.20167.954.27

The additive adapter reaches PSTune's training HPS (within 0.14 on FLUX and SD3.5) but trails it by 0.20 to 0.21 in ImageReward and loses accuracy and LPIPS that PSTune keeps. Every alternative stays at least 0.17 below PSTune. Rescaling the additive update to PSTune's response norm, the matched-update-norm protocol you asked for, leaves a gap of 0.22 on SD3.5. We used random-feature adapters rather than a FiLM or hypernetwork variant because their kernel is exactly a sum with nonlinear features; a FiLM or hypernetwork adds the Jacobian of its generated weights, so it does not isolate sum versus product. Since placement, optimizer, selection, and parameter count are shared, what remains is the kernel. One detail supports Eq. (2) directly: adding the prompt to the state-only adapter lowers ImageReward on both backbones, while multiplying by it gives PSTune.

W2. Lower training HPS may mean "optimize less". This alternative deserved a direct test, and Table R2 (App. G.1) gives it with the axes you listed: early stopping (the 512-update checkpoints, all three backbones), a lower learning rate, a KL penalty to the frozen velocity, and rank 1 (SD3.5).

  • At matched HPS (within 0.14), LoRA trails PSTune by 0.30 to 0.33 ImageReward. On FLUX the early-stopped LoRA has IR 0.870, accuracy 64.8, LPIPS 49.63, against 1.201 / 67.9 / 54.27 for PSTune.
  • Matched LoRA already shows 51 to 67% of its final accuracy and LPIPS loss; on SD3.5 the lower learning rate and the KL penalty land next to the early-stopped point.
  • Rank 1 (1.5M parameters on SD3.5, against the tensor's 1.0M) loses 3.6 of rank 32's 3.9 accuracy points.

So the retained accuracy and variety are not bought by optimizing less. PSTune does reach a given HPS more slowly, and the Limitations say so.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.