- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents a method for mitigating reward overoptimization when fine-tuning text-to-image models. Unlike traditional LoRA-based fine-tuning, which modifies the denoising model's parameters, the proposed method freezes the denoiser and trains a lightweight velocity corrector. More specifically, the corrector is parameterized by a three-way tensor conditioned on prompt and local denoising-state features, such that gradient updates transfer between samples according to the product of their prompt and state similarities. Experiments across multiple training objectives (direct reward backpropagation and Flow-GRPO) and models (SD3.5, SDXL, FLUX) demonstrate promising improvements in reward quality while largely preserving compositional accuracy and diversity compared with LoRA fine-tuning.
Critical strengths
The mathematical formulation appears sound and elegant, with a clear motivation for the proposed parameterization.
The experimental results, including the qualitative comparisons and mechanism-focused ablations, are compelling.
Critical weaknesses
My biggest concern is that the experiments may not fully disentangle the benefits of restricting the trainable model from those of the proposed gradient-localization mechanism. For example, the FLUX LoRA baseline has approximately 131M trainable parameters, whereas the proposed tensor adapter has only 6.3M. The reduction in reward overoptimization could potentially be explained by this difference in capacity, rather than by gradient localization itself.
While Section 5.2 attempts to address this through the wrong-prompt and wrong-state ablations, I would be interested in comparisons against simpler, similarly sized correctors, such as prompt-only, state-only, or additive prompt–state adapters.
Minor concern: Appendix C.2 mentions that ImageReward is used for learning-rate selection. Consequently, I would be cautious about treating ImageReward as a fully independent evaluation metric, even though HPSv2.1 is the training reward.
Questions for authors
At a higher level, while the proposed parameterization and gradient-localization principle are intuitively appealing, could this strong inductive bias also impose limitations? For example, could suppressing image-wide updates or requiring both prompt and state similarity prevent beneficial transfer of general improvements especially when looking beyond text-to-image models?
3 comments
Response to Reviewer cz47 (2/2): ImageReward in selection, and the limits of the inductive bias
(2/2)
Minor: ImageReward is used for learning-rate selection. Agreed, and the paper now says so where ImageReward is introduced (Sec. 5) and in App. C.2. To take selection out of the comparison, Table R3 (App. G.2) trains every learning-rate candidate of both methods to 1,024 updates with three seeds on SD3.5:
- on SD3.5, every PSTune candidate exceeds every LoRA candidate in ImageReward, by at least 0.058;
- if each method keeps its highest-HPS candidate, so that ImageReward plays no role, PSTune leads by 0.29, about the margin in Table 1;
- PickScore enters neither training nor selection, and PSTune leads matched LoRA on it.
Q. Could the inductive bias block useful transfer, in T2I and beyond? Yes, and we think the honest answer is that it costs something in three measurable ways. The revised Limitations paragraph lists all three.
- Image-wide improvements. Forward centering removes image-wide velocity corrections, such as a global brightness or contrast shift. The forward-centering ablation measures the price: with raw forward features (), text match is better by 4.2 to 6.9 net-win points, while PSTune is preferred on realism and overall (Table 11). When a reward is mostly global, the variant is the better choice, and the update can still be centered.
- Generally useful transfer. The product attenuates updates that would help unrelated prompts. One visible consequence is speed: after 1,024 updates PSTune's training HPS gain stays below LoRA's on every backbone (Tables 1 and 2). At PSTune's HPS, LoRA has already paid 51 to 67% of its final accuracy and LPIPS loss (Table R2), so the attenuation slows training without costing adherence or variety.
- Rewards that need new content. With the GenEval reward on SD3.5 (Table R5), PSTune gains 38% of LoRA's improvement while keeping ImageReward and LPIPS. Here the restriction binds.
Beyond text-to-image, Eq. (1) holds for any adapter, so the reach of an update can be measured in any conditional generator. The particular choice in PSTune (a pooled condition times a centered local state) suits rewards that change how requested content is rendered. Settings where improvements should spread broadly, such as a global style reward or a policy whose useful skills are shared across tasks, call for a different kernel, for example a sum with a small product term or uncentered forward features. We now present the product as the right kernel for preference rewards in text-to-image fine-tuning, not as a universal rule, and the conclusion stays within that scope.
Summary for the discussion
- The capacity explanation is tested directly: a sum-kernel adapter with exactly the tensor's parameter count, and LoRA rank 1 with about as many parameters as the tensor, both lose the accuracy and variety that PSTune keeps (Tables R1, R2).
- The product beats prompt-only, state-only, and additive kernels at identical placement, optimizer, and search, also at matched update norm (Tables R1, R8).
- ImageReward selection is removed without changing the conclusion (Table R3).
- The costs of the inductive bias are now quantified and stated.
Thank you again for a review that was both encouraging and precise about what was missing. If the new comparisons address your concern about capacity, we would be very grateful if you would consider raising your score and supporting the paper in the AC discussion.
Thanks! I think this definitely addresses most of my concerns regarding the soundness of the results, and I'm happy supporting the paper wrt acceptance.
Response to Reviewer cz47 (1/2): capacity versus gradient localization
(1/2)
We are grateful for a clear and generous review. You found the formulation sound and elegant with a clear motivation, and the qualitative comparisons and mechanism-focused ablations compelling; that is the reading we hoped for. Your main concern goes to the heart of the paper, and you named the exact controls that answer it. We ran all of them.
Is the benefit capacity (6.3M vs. 131M) or gradient localization? You are right that Sec. 5.2 did not settle this. Wrong-index controls keep PSTune's capacity but corrupt its update, which shows that the index matters, not that the product beats other small adapters. The revised Sec. 5.2 adds Table R1 (main text, full metrics in Table R8, App. G.7). All adapters sit at PSTune's placement, read the same detached RMS-normalized indices, and use scalar SGD, clipping, and the same three-candidate search. All four run on SD3.5-Medium (GenEval IR / accuracy / LPIPS):
On FLUX we repeated the additive and state-only adapters: IR 1.006 and 1.032 against 1.201 for PSTune, with accuracy and LPIPS below the base.
How we read it:
With placement, optimizer, selection, and parameter count held fixed, the remaining difference is the kernel, which is the claim of Sec. 3.2. We are grateful for this suggestion; it turned the central argument of the paper from a motivated design into a measured one.