- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper aligns few-step, deterministic generators under black-box rewards by freezing the generator and moving alignment into its input space. Sampling from the reward-tilted output reduces to sampling tilted noise, which NPO learns with a flow policy trained to minimize under reference regularization toward ; the policy gradient uses an ELBO-based endpoint likelihood from forward-noising pairs rather than simulated trajectories. Validation covers text-to-image (SD-Turbo, SANA-Sprint), topic-steered language (4-step ELF) and protein backbones (SiD-Protein) against direct fine-tuning, test-time search and trajectory-based noise policies, claiming a better reward-quality trade-off than fine-tuning and comparable reward to test-time search at 16–549× lower cost.
Critical strengths
- The reformulation is exact for deterministic generators and converts an intractable likelihood problem into a standard flow-fitting one, which is what makes black-box rewards and deterministic samplers share one machinery.
- The empirical design is broad and includes the right control: three modalities, four generators across 1/4/5 steps, and a baseline differing only in the likelihood estimator. The cost tables are the correct way to argue against test-time search.
Critical weaknesses
- The importance ratio in Eq. (7) is not a valid density ratio. is a single-pair residual, Eq. (6) holds only in expectation, and on one shared pair is neither unbiased nor consistent; with the averaging is thin. It is unclear whether this is a heuristic weight (then say so) or a correction (then it needs justification and clipping).
- The headline negative claim rests on baselines not shown to be tuned. Flow-GRPO drives protein designability to exactly zero, which reads as a mis-specified baseline (learning rate, KL coefficient, injected stochasticity) rather than a property of the method class. Without a best-effort baseline, "direct fine-tuning transfers poorly" is not established.
Questions for authors
- Is clipped, and what happens at ? Please justify it as a ratio or state that it is a heuristic weight.
- What sweep was run for Flow-GRPO and FAV-B, and is the designability collapse reproduced across seeds and learning rates?
- Appendix D.3 says five seeds with mean and standard deviation; why do Tables 1–2 report point estimates? Please add them, and give the image results as a table rather than only curves.
2 comments
Thank you again for your detailed review and valuable suggestions. We hope our responses have addressed your concerns. Please let us know if you have any further questions.
Rebuttal response to Reviewer akQF
We thank the reviewer for the detailed examination of our method and experiments and the helpful suggestions. We would like to address their concerns and questions below.
(W1 & Q1) Clarifying the importance ratio approximation and implementation.
Thank you for raising this point! For the importance ratio, our formulation follows [1]. Specifically, we use the ELBO-based ratio approximation and construct ρ^ψ from the difference between regression losses. This is a surrogate weight for the policy update rather than a guaranteed unbiased or consistent estimator of the exact density ratio. We have updated our paper to clarify this heuristics.
We omit ratio clipping and advantage standard-deviation normalization for methodological simplicity and fewer hyperparameters. Figure 15 in Appendix H.2 compares these choices and shows comparable performance with and without clipping and normalization.
Figure 11 in Appendix H.2 compares M=1,3,6 forward-noising pairs. Increasing M beyond three provides no observed gain in this experiment, so we use M=3 to balance performance and computational cost.
[1] Xue, et al. “Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models.” ICML, 2026.
(W2 & Q2) Addressing fair comparison concerns related to hyperparameter tuning.
We carefully tuned the hyperparameters for all methods, including baselines. Specifically, we excluded configurations where the target reward dropped abruptly or consistently decreased during training. Among the remaining configurations, we select the best setting whose target reward rises stably across 3 different seeds. We use the same strategy to select this hyperparameter for all methods, including our NPO. We use 3 seeds to select the hyperparameters and run two additional seeds for the selected ones. These selection criteria are detailed in Appendices D, E, and F.
Both Flow-GRPO and FAV-B were tuned over the learning-rate and regularization-strength ranges reported in Table 5-11 in Appendix D,E,F. To make the comparison more explicit, we have added reward–quality trade-off plot across regularization strengths averaged by 3 seeds for both NPO, Flow-GRPO and FAV-B (Figures 17, Appendix H.3). The results consistently support the same conclusion that NPO achieves a better reward–quality trade-off than direct fine-tuning.
(Q3) Standard deviations and image results as tables.
Thank you for pointing this out! In our initial draft, experiments for test-time optimization comparison (Table 1, 2, Fig 7.a) were originally reported using a single seed. We sincerely apologize for this inconsistency. We have now updated Tables 1–2, Fig 7.a to report mean ± standard deviation over five seeds and corrected the experimental details accordingly. Our conclusion remains the same with these new results.
In Appendix G, we also provide numerical tables for the image results in the main paper, with mean ± standard deviation for each metric.