acceptodds
Under review as a conference paper at ICLR 2027

APD-Flow: Reward Post-Training for Flow-Matching Image Restoration

Abstract

Flow-matching image restorers produce high-quality outputs with deterministic inference, but they are trained to match paired ground truth rather than human preferences or task-specific rewards. Reward post-training offers a way to close this gap. However, policy-gradient methods rely on stochastic exploration and action likelihoods, neither of which a deterministic sampler provides. We propose *APD-Flow*, a reward post-training framework that introduces exploration at the source state instead of the sampling trajectory. Given a degraded input, *APD-Flow* perturbs the source state in two opposite directions and restores both with the same deterministic sampler. The difference between the two restorations, weighted by their reward difference, defines an Antithetic Preference Direction (APD) that points toward the preferred output. The restorer is then optimized along this direction without action likelihoods or trajectory noise, and fidelity-aware reward adaptation keeps fidelity at the level of the pretrained restorer. The framework applies to restorers with arbitrary numbers of sampling steps and to non-differentiable rewards. Experiments on blind face restoration, blind image super-resolution, and scene-text restoration show that *APD-Flow* consistently improves the targeted perceptual or task-specific metrics while keeping the inference architecture and budget unchanged. Code and a video explanation are provided in the supplementary material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
flow matching, image restoration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ma9AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes APD-Flow, a reward post-training framework for deterministic flow-matching image restoration. It generates two restorations using antithetic perturbations of the source state, converts their reward-weighted output difference into velocity-field updates through a distance-based proxy objective, and adaptively balances preference and fidelity rewards. Experiments cover four restoration backbones, sampling schedules ranging from one to fifteen steps, and face restoration, super-resolution, and scene-text restoration. The paper reports perceptual and task-specific improvements, supported by component ablations and a human preference study, while preserving the original inference sampler and step count.

Critical strengths

Source-state exploration offers a practical way to optimize deterministic restorers with rewards, including non-differentiable feedback. Its applicability to one-step models is particularly useful, and retaining the original inference sampler makes the resulting improvements attractive for deployment.

The evaluation covers several backbones and restoration tasks, with comparisons to alternative post-training methods and targeted ablations. The 20-participant study provides additional evidence of perceptual benefit: APD-Flow is preferred over PMRF for visual quality in 64.2% of comparisons, versus 25.4% for PMRF (Table 3).

Critical weaknesses

The claim of preserving fidelity needs qualification. On PMRF, LPIPS increases from 0.331 to 0.349 despite improvements in other metrics (Table 5). Appendix C.2 also sets the normalized fidelity threshold to 0.95 for FlowIE-BFR, explicitly allowing some composite-fidelity reduction. The evidence supports a favorable perception–fidelity trade-off, but does not establish preservation of every fidelity measure.

The methodological advance is relatively focused. Section 4.2 explicitly builds on the directions and targets of FDFO and the proxy ratio of FPO. Source-state pairing and adaptive fidelity control are useful additions, but the empirical contribution of the proposed mean-response regularizer is not isolated from the rest of the objective. This leaves part of the claimed optimization contribution insufficiently established.

Questions for authors

1.How should the fidelity-preservation claim be interpreted given the LPIPS increase and the 0.95 threshold for FlowIE-BFR? Please clarify the permitted fidelity loss and whether satisfying the composite constraint can conceal deterioration in individual fidelity components. 2.With source pairs, rollout reuse, rewards, and the fidelity controller fixed, how does the proposed mean-response regularizer compare with a per-branch penalty or no regularization? This would help establish its contribution beyond the inherited proxy objective.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ma9A

Many thanks for the careful reading and the supportive assessment. We appreciate your note that the applicability to one-step models is particularly useful, that keeping the original inference sampler makes the improvements attractive for deployment, and that the human study adds evidence of perceptual benefit. Changes in the revised PDF are highlighted in blue.

W1 / Q1. Interpreting "fidelity preservation". Thank you for raising this. In our paper, fidelity preservation does not mean that every fidelity metric stays the same or improves. It means that the composite fidelity score is kept near the pretrained level, so the perceptual gains come without an overall loss of fidelity. We have revised the abstract, Sec. 1, and Sec. 4.3 to state that the method keeps a composite fidelity score at or above a target set relative to the pretrained restorer. Sec. 5.2 now describes the results as a balance between perception and fidelity.

Q1. Permitted fidelity loss. The threshold τ\tau works as a dial. Setting τ=1\tau = 1 keeps composite fidelity at the pretrained level, as used for PMRF and FlowIE-BSR. This default setting is what our claim of keeping pretrained-level fidelity refers to. A higher τ\tau asks for fidelity above the pretrained level, as used for DiT4SR (τ=1.05\tau = 1.05). A slightly lower τ\tau trades a little fidelity for better perceptual quality. FlowIE-BFR uses τ=0.95\tau = 0.95 for this reason, which allows at most a 5% drop in composite fidelity. Even so, its identity distance, landmark distance, CLIP-Sim, and FID all improve over the pretrained model (Table 1).

Q1. Can the composite constraint hide deterioration in individual metrics? In practice, it does not hide any severe loss of fidelity. All fidelity components measure agreement with the GT and are strongly positively correlated, so one component is unlikely to collapse while the others compensate for it. On PMRF, LPIPS rises slightly, while several other fidelity metrics, DISTS, identity distance, landmark distance, and CLIP-Sim, all improve (Tables 1 and 5).

W2 / Q2. Effect of the mean-response regularizer. Following your suggestion, we added a controlled ablation in Appendix F.4. With all other components fixed, we retrained PMRF with each of the three regularizers and evaluated them on a 300-image subset of CelebA-Test. Our common-mode (mean-response) regularizer gives the best perceptual quality and fidelity among the three choices.

Table R1. Regularizer ablation, post-training on PMRF (CelebA-Test, 300 images).

RegularizerMUSIQ ↑LPIPS ↓DISTS ↓
Per-branch70.790.3530.156
No regularization70.210.3580.157
Common-mode (ours)71.490.3470.154

As Eq. (A.12) shows, the per-branch penalty also suppresses the contrast between the two branches, and removing the regularizer leaves their joint drift unconstrained.

Thanks to the authors for addressing my concerns. I think their explanation has, to some extent, resolved the issues I raised earlier.

First, the additional experiments on the mean response regularization provided the experimental evidence I was looking for. Second, they added a relevant explanation regarding the cause of the discrepancy in the LPIPS metric.

Therefore, I've decided to keep my original score.

Reviewer_pDTdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes APD-Flow, a reward post-training framework for deterministic flow-matching image restoration models. The method aims to improve pretrained restorers using perceptual or task-specific rewards without modifying their original deterministic ODE samplers.The method is evaluated on four pretrained restoration models with 1, 4, 10, and 15 ODE sampling steps, covering blind face restoration, blind image super-resolution, and scene-text restoration.

Critical strengths
  1. The primary contribution of this work is shifting reward-guided exploration from intermediate ODE trajectories to the initial source states. By sharing a common base noise and introducing perturbations in opposite directions, APD-Flow constructs a pair of highly correlated restoration candidates while preserving the marginal source distribution of each branch. This design helps reduce irrelevant variations between candidates, allowing reward differences to more directly reflect local changes in restoration quality. Such a strategy is particularly well suited to image restoration tasks, where improving perceptual quality must be balanced against preserving the fidelity of the original content.
  2. The proposed method is evaluated on four pretrained restoration models with different architectures and sampling step counts, covering blind face restoration, real-world image super-resolution, and scene-text restoration. The experiments demonstrate consistent improvements across multiple settings, and the accompanying ablation studies provide reasonably comprehensive evidence for the effectiveness of the proposed framework.
Critical weaknesses
  1. FDFO has already explored reward-aligned velocity updates using pairwise output differences, while the shifted velocity targets and distance-based proxy ratio in APD-Flow are closely related to FDFO and FPO, respectively. DiffusionNFT has also demonstrated likelihood-free reward post-training with deterministic samplers. Thus, the main novelty appears to lie in source-state antithetic coupling and its integration with existing optimization techniques. However, the current experiments do not sufficiently establish whether this specific coupling mechanism provides advantages beyond more general correlated sampling strategies.
  2. Appendix C.1 shows that the adopted Flow-GRPO-style trajectory SDE substantially degrades restoration quality even before reward post-training, raising concerns about the sampling configuration rather than demonstrating an inherent limitation of trajectory-level exploration. Moreover, Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching has proposed improvements to address noise-related issues in SDE-based sampling. Without discussing or evaluating such alternatives, the claimed superiority of source-state exploration over trajectory-based approaches remains insufficiently established.
Questions for authors
  1. Can the authors further disentangle the benefits of source-state antithetic coupling from those of simply generating highly correlated candidates? A controlled comparison with alternative exploration strategies under comparable output displacements and candidate consistency would help clarify the independent contribution of the proposed sampling mechanism.

  2. Given the substantial quality degradation of the Flow-GRPO-style SDE before post-training (Appendix C.1), could the authors clarify whether the stochastic sampler was adequately adapted and tuned for PMRF? In particular, could coefficient-preserving sampling or other improved SDE-based approaches be applied, and how would APD-Flow compare against such alternatives?

1 comment
AuthorsOpenReview ↗

Response to Reviewer pDTd

We appreciate your positive assessment, in particular your view that source-state antithetic coupling is well suited to image restoration and that the ablations are reasonably comprehensive. All revisions are marked in blue in the PDF.

W1 / Q1. Antithetic coupling versus correlated candidates. A correlated pair can differ from ours only by losing the correlation or the symmetry. Any jointly Gaussian pair that keeps the source distribution of each branch and has an isotropic correlation already takes the form of Eq. (4) (Appendix A.1). We therefore test the two cases separately.

To test symmetry, Appendix F.2 compares our pair with a correlated but unequal pair z2=z1+σsδz_2 = z_1 + \sigma s\delta on PMRF. We set ss so that its output displacement is comparable to ours. Both pairs use the same 300 inputs and random draws. Branch bias is the share of inputs where branch 2 gets the higher MANIQA score, 50% without bias.

Table R1. Pair constructions on PMRF. The first three metrics are measured on pretrained PMRF, and the last three after post-training, all on a 300-image subset of CelebA-Test.

Pair constructionRMS(Δx\Delta x)ArcFace angleBranch biasMUSIQ ↑NIQE ↓ID distance (Deg.) ↓
Correlated, unequal0.01065.7°99%70.194.7030.97
Antithetic (ours)0.01157.8°52%71.494.4430.32

As Table R1 shows, at a comparable displacement, the higher-variance branch of the unequal pair wins 99% of the comparisons. PMRF turns the extra source variance into sharper texture, so the reward difference tracks the noise magnitude instead of the content. Our two branches have the same distribution, and the mirrored step cancels the even-order responses of the sampler (Appendix A.1). The bias therefore disappears, and post-training gives better MUSIQ, NIQE, and identity distance. To test correlation, Figure 7 (right) uses independent source pairs. Their reward attribution is ambiguous, and training does not converge reliably. Both correlation and symmetry are therefore needed, and Eq. (4) provides both.

On the integration with existing techniques, the key difference lies in where the exploration signal comes from. Existing methods either inject noise along the trajectory, as in Flow-GRPO, FDFO, and CPS, or draw a group of independent candidates per input, as in DiffusionNFT. Trajectory noise changes the sampler, and FDFO and CPS give identical candidates on one-step restorers. APD-Flow perturbs only the source with one antithetic pair, so the sampler, step count, and source distribution all stay unchanged, and one pair already gives a clean preference signal. The other components are built around this pair. The common-mode regularizer constrains only its shared drift and keeps its contrast (Appendix F.4), and closed-loop fidelity control adds the reference constraint that restoration needs (Sec. 4.3). Under the same budget on PMRF, the MUSIQ gain of FDFO is about one fifth of ours. DiffusionNFT uses six rollouts per input, three times our cost, yet it is worse in both perceptual quality and fidelity (Table 5).

W2 / Q2. Tuning of the trajectory SDE and comparison with CPS. We did tune the SDE for PMRF. With the official default noise level a=0.7a = 0.7, the candidates collapse before the first update, so Table C.1 reports the best tuned levels, a=0.05a = 0.05 and 0.10.1. The remaining degradation is not a tuning issue. Flow restorers differ from text-to-image flow models in both training and architecture, so trajectory-noise exploration designed for text-to-image generation does not transfer well to them. Source-state pairing leaves the sampler untouched and does not face this issue.

Following your suggestion, we evaluated Coefficients-Preserving Sampling (CPS) on PMRF. Only the exploration is replaced, and the objective, reward, fidelity controller, and update budget stay the same.

Table R2. Post-training on PMRF with CPS or source-state pairing (CelebA-Test, 3,000 images).

ExplorationMUSIQ ↑NIQE ↓DISTS ↓FID ↓
CPS (η=0.7\eta = 0.7, default)64.935.100.15513.46
CPS (η=0.1\eta = 0.1, tuned)64.925.040.15513.06
Source pair (ours)71.354.390.15410.59

CPS meets the same difficulty. With either setting, it barely changes the pretrained PMRF: MUSIQ moves only from 64.63 to 64.9, while source-state pairing reaches 71.35 (Table 7). As noted above, CPS also gives identical candidates on one-step restorers, where source-state pairing still works.

Reviewer_NRfkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose APD-Flow for the image restoration task, where they attempt performing post-training on deterministic samplers through injecting exploratory perturbations in the source state. Using the outputs obtained from the perturbed input and the oppositely perturbed input, the authors take advantage of the difference between two outputs to construct a reward signal (APD) for RL-based training. The authors perform evaluations on tasks such as blind face restoration, blind image SR, and text restoration, showing improvements in no-reference and reference-based metrics.

Critical strengths
  • The proposed framework does not modify the structure of the deterministic sampler, while only exploring the source state. This makes the proposed framework applicable to wide range of restoration models.
  • Empirical evaluations on various tasks show some improved performance over multiple baseline restorers.
  • The paper is written in a clear and straigforward manner, describing the key method with sufficient supplementary materials.
Critical weaknesses
  • How is this method fundamentally different from FDFO? Proposed method seems to only differ in how the pair is drawn.

  • There are closely related publication that was not cited by the authors. Neighbor GRPO [a] also perturbs initial noise and uses a distance-based surrogate with deterministic ODE sampling. Only difference between this work comes from the oppositely perturbed input for APD.

    • [a] He et al., Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Models, CVPR 2026
  • Evaluations under basic reference-based metrics such as PSNR and SSIM are missing in the tables (Table 1 and 5).

  • In Table 5, non-reference metric of LPIPS worsens from 0.331 to 0.349, implying that the proposed fidelity-aware reward adaptation is not working well.

Questions for authors

Please refer to the weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer NRfk

Thank you for the thoughtful review and the positive evaluation. We are glad you found that exploring only at the source state makes the framework applicable to a wide range of restoration models, and that the paper is clear and well supported by the supplementary material. New material in the revised PDF is shown in blue.

W1. Difference from FDFO. Beyond how the pair is drawn, APD-Flow differs from FDFO in three ways.

(1) Where exploration happens. FDFO separates the two rollouts by injecting noise along the trajectory. This noise vanishes at the final step, so on a one-step restorer the two FDFO candidates are identical and give no training signal. This matters in practice, because state-of-the-art restorers increasingly use a single step. APD-Flow perturbs only the source state. It therefore works on one-step and multi-step restorers alike, from 1 to 15 steps in our experiments, and trains on the same deterministic sampler used at test time.

(2) Regularization. FDFO penalizes each branch separately. Eq. (A.12) shows that this penalty equals our common-mode penalty plus an extra term that suppresses the contrast between the two branches. APD-Flow penalizes only the mean response of the pair, and Appendix F.4 compares the two choices.

(3) Fidelity control. FDFO has no reference image. APD-Flow reweights a composite fidelity reward on each batch to keep fidelity at or above a target set relative to the pretrained model (Sec. 4.3).

Under the same data, reward, and update budget on PMRF, the MUSIQ gain of APD-Flow is about five times that of FDFO, with better NIQE, DISTS, and FID (Table 5). On one-step FlowIE-BSR, where FDFO gives no signal, APD-Flow improves every metric in Table 1. Appendix F.1 discusses these differences in detail.

W2. Relation to Neighbor GRPO. Thank you for pointing out Neighbor GRPO. We now cite and discuss it in Sec. 2 and Appendix F.1. We agree that both methods perturb the initial noise and keep the ODE sampler deterministic. However, the two methods differ in the surrogate objective and in the number of candidates.

(1) Distance surrogate. At sampled ODE steps, Neighbor GRPO compares the next state predicted by the current model with the intermediate states of all 12 neighbor trajectories. A softmax over these distances gives a pseudo-likelihood, which then enters a clipped GRPO ratio. APD-Flow does not compare intermediate states. It shifts the velocity targets along the difference between the two final restorations, as defined in Eqs. (5)–(7). The velocity at every step is thus pushed along the output-space preference direction. At the rollout reference, the gradient of our preference term is proportional to the APD (Sec. 4.2). Table 8 shows that this direction agrees in sign with the true reward gradient on 96% of inputs.

(2) Number of candidates. Neighbor GRPO normalizes the rewards within each group of 12. With a single pair, this normalization gives every pair with unequal rewards the same magnitude, so a near tie and a clear preference receive the same weight. APD-Flow instead normalizes the pairwise reward differences across the batch, which keeps their relative sizes. One symmetric pair per input is therefore enough, and it needs about one sixth of the rollouts and reward evaluations.

In addition, Neighbor GRPO targets text-to-image generation without a reference image. APD-Flow also controls fidelity to the reference. Table F.2 lists these differences. The code of Neighbor GRPO is not publicly available, so a direct comparison was not possible.

W3. PSNR and SSIM. We have added PSNR and SSIM to Tables 1 and 5.

W4. LPIPS on PMRF. LPIPS is only one of the fidelity metrics we report. On PMRF, several other fidelity metrics, DISTS, identity distance, landmark distance, and CLIP-Sim, all improve, while LPIPS becomes slightly worse (Tables 1 and 5). This is expected from the design. For PMRF, the adaptation keeps a composite fidelity score at or above the pretrained level. It does not constrain each metric separately (Sec. 4.3 and Table B.2). Individual metrics therefore move slightly up or down. When shown the GT, human raters also preferred APD-Flow over PMRF for fidelity (Table 3). The LPIPS increase is much smaller than that of ReFL and DiffusionNFT (Table 5).

I thank the authors for addressing my concerns. I am inclined to keep my score of weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.