acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Mitigating State Exposure Mismatch in Flow Model Reward Alignment

Abstract

Reinforcement learning (RL) offers a flexible way to align pretrained flow models with task-specific objectives. Recent endpoint-based RL methods improve efficiency by reconstructing training states from rewarded endpoints, but these states may differ from policy-induced rollout states. We study this mismatch in state exposure during RL training of flow models. Our empirical analyses show that the discrepancy varies across timesteps, affects reward differently across denoising stages, and benefits from corrections aligned with individual trajectories. Guided by these observations, we propose Drift-Aligned Exposure Correction (DAEC), a simple method for mitigating this mismatch. DAEC perturbs only the trainable model input around analytically reconstructed states, using a readily available bridge residual to identify directions relevant to rollouts and adapting the perturbation magnitude to each timestep and sample, while preserving the original regression target and reference anchoring. DAEC introduces only lightweight tensor operations and requires no additional model evaluations per training update. Evaluations across diverse reward objectives demonstrate that DAEC improves both RL training efficiency and generation quality.

Primary area:
generative models
Keywords:
Flow Matching, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vCKbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the mismatch between analytically reconstructed training states and policy-induced rollout states in endpoint-based RL for flow models. Authors argue that this mismatch is timestep-dependent, reward-relevant, and trajectory-specific. To close this gap, they propose Drift-Aligned Exposure Correction (DAEC), which perturbs the trainable model input along a normalized bridge-residual direction. Extensive experiments on various benchmarks confirm DAEC's consistent improvement over RAM.

Critical strengths
  1. The paper identifies a concrete and underexplored limitation of endpoint-based flow-model RL.

  2. DAEC is lightweight and easy to integrate into RAM. It seems that DAEC could improve the performance across various tasks.

Critical weaknesses
  1. The authors' diagnostic findings do not fully justify the perturbation they implement. In the diagnostic experiment, what helps is pulling a rollout state against its inferred noise discrepancy. However, DAEC moves an analytic state along the positive residual direction.This can be viewed and interpreted as robustness augmentation. However, the paper does not compare the positive and negative directions. Therefore, the proposed direction does not directly follow the stated observation.

  2. DAEC evaluates the trainable model at a perturbed input while keeping the target and reference prediction unchanged at the original analytic state. The resulting target is generally not the conditional velocity at the perturbed state. This implicitly assumes local invariance of the velocity field and may act as an uncontrolled smoothing regularizer.

Questions for authors
  1. What happens when training with the negative residual direction, or with symmetric positive/negative perturbations?
  2. Renoising is a common operation in this area. I have a little doubt on the claim of exposure bias. From the perspective of ODE sampling, each state at a same timestep has a same marginal distribution. Therefore, I think directly claiming sample-level difference as exposure bias is not completely adequate.
  3. What does the image in the right down corner in Figure 1(c) mean?
  4. Please see weaknesses.

At this time, I'm still confused on some claims in the paper, and I think some design does not directly address the challenge proposed in the motivation. However, I would like to enhance my rating if authors could fully address my concerns.

4 comments
AuthorsOpenReview ↗

Response to Reviewer vCKb [1/2]

Thank you for your careful review and for recognizing the importance of the problem and the lightweight design of DAEC. We address your questions about the motivation, training objective, and interpretation below.

W1 / Q1 / Q4 Why perturb analytic states along the positive residual direction when the diagnostic uses a pullback? Is this robustness augmentation? What happens with negative or symmetric positive/negative perturbations?

Response: We apologize for not making the distinction between the diagnostic intervention and the training perturbation clearer. In Figure 1(b), and in the original-noise correction arm of Figure 1(c), we apply a one-time correction at a selected rollout timestep: we keep the instantaneous clean estimate fixed, replace the inferred noise with the original noise seed, and continue sampling with the unchanged model. The intermediate-stage reward gains show that addressing this discrepancy can benefit generation.

DAEC aims to improve the model's ability to handle related input deviations through training. During generation, no prescribed clean endpoint is available for checking intermediate states against an analytic path. During training, we have the sampled endpoint and its analytic training pair. We perturb the input and train the model against the RAM target constructed at the original state. We therefore agree that DAEC can be understood as directional robustness augmentation.

The positive sign has a concrete interpretation in this construction. Eq. (13) shows that the residual points from the assigned noise toward the noise inferred by the lagged policy. Perturbing the input along the positive residual direction is equivalent to keeping the clean endpoint fixed and changing the assigned noise in that direction, with the magnitude controlled separately. We choose this direction to construct training inputs that contain the noise deviations reflected in the model's own predictions. By training on these inputs with the original RAM supervision, we aim to improve the model's ability to handle such deviations. Figure 1 tests a correction during sampling, while DAEC trains on perturbed analytic inputs; they operate at different states with their respective local discrepancies. The positive residual need not point toward the paired rollout state.

The reported training results and random-direction comparison support this design. During method development, we also tested the opposite training direction, Xtana−ηtu^tX_t^{\mathrm{ana}}-\eta_t\widehat{u}_t, and symmetric positive/negative perturbations. Both exploratory tests encountered training divergence. Their best observed GenEval scores were approximately 0.83 for the opposite direction and 0.80 for symmetric perturbations. Based on these observations and the random-direction comparison, we adopted the positive residual direction in DAEC.

We have added this explanation in Section 4.1 and clarified the one-time diagnostic intervention in Section 3.3.

AuthorsOpenReview ↗

Response to Reviewer vCKb [2/2]

W2 / Q4 Why keep the target and reference at the original state? The unchanged target may not be the conditional velocity at the new state and may impose local invariance or uncontrolled smoothing.

Response: Keeping the original RAM target is a deliberate part of the training augmentation. We want the model to satisfy the original supervision when its input undergoes a residual-related deviation. The reference and lagged-policy predictions are components of that target, so we evaluate both at the original analytic state. Recomputing them at the perturbed state would change the supervision together with the input. We use the original target to supervise the augmented input, rather than to estimate the exact conditional velocity at the perturbed state.

We agree that this can encourage local directional smoothing, which is consistent with our aim of improving robustness to these input deviations. Section 4.3 describes this interpretation through a local expansion around a fitted training point. Prior work also supports the value of smoothness for efficient sampling. Yang et al., Lipschitz Singularities in Diffusion Models (ICLR 2024), show that mitigating temporal Lipschitz singularities improves fast sampling with DDIM and DPM-Solver. Their Section 5.4 explains that larger discretization steps place greater demands on smoothness. Their analysis concerns variation with time; DAEC encourages reduced sensitivity along a selected direction in the input state.

Our results in the inference-step block of Table 5 show that DAEC retains an advantage with few sampling steps. Using FlowMatch-Euler with CFG 4.5, RAM + DAEC achieves GenEval scores of 0.9367 and 0.9494 at 10 and 20 steps, compared with RAM's 0.9164 and 0.9218, respectively. DAEC at 10 steps also exceeds RAM at 50 steps (0.9254). We believe that improved tolerance to state deviations may contribute to this performance under fewer sampling steps.

Q2 How do paired-state differences establish exposure bias when different sample paths can share the same marginal distribution?

Response: We agree that different sample paths can share the same time-marginal distributions, so a paired-state difference alone is insufficient to establish a distribution mismatch. Our diagnostic D(t)D(t) measures the magnitude of a trajectory-conditioned local correction, rather than a distance between marginal distributions.

The motivation comes from how RAM constructs its training states. RAM independently noises generated endpoints, whereas inference follows the current model’s ODE trajectories. RAM’s analysis establishes that analytic noising recovers the controlled-SDE joint distribution at initialization and at the optimum; this does not establish the same correspondence throughout intermediate optimization. This motivates examining how supervision on analytically reconstructed states transfers to actual generation.

Our evidence therefore includes interventions beyond measuring paired differences. In Figure 1(b), a one-time correction with the model held fixed improves terminal reward at intermediate timesteps. Figure 1(c) compares this intervention with fresh-noise and norm-matched random controls, supporting the relevance of the trajectory-specific direction. These observations motivate DAEC’s directional robustness augmentation: training the model to handle related input deviations while preserving the original RAM supervision. The task and image-quality results provide further evidence of the practical value of this design.

Q3 What does the lower-right inset in Figure 1(c) show?

Response: The inset zooms in on the intermediate timestep range t∈[0.3,0.7]t\in[0.3,0.7], with ticks at 0.7, 0.5, and 0.3, to compare the three mean reward-change curves more clearly. We apologize for the confusion caused by the missing explanation. We have added this clarification to the caption of Figure 1(c).

Reviewer_vCKbOpenReview ↗

Thanks for your responses

Thank you for your responses, which help clarify several previously unclear points. I appreciate these explanations, although some concerns remain relevant to my assessment.

  1. Authors' explanation of perturbations with an unchanged estimated target appears consistent with my original understanding: this is a smoothing technique intended to improve robustness. However, I remain concerned that mapping multiple noisy states to the same target could reduce generation diversity—an important issue in online RL. Based on my experience, encouraging consistency across different denoising timesteps has a clearer rationale. The motivation for perturbing the noisy states themselves remains less convincing to me. Intuitively, current techniques in the manuscript are sufficient for enhancing reward values, but I'm not sure whether this really helps generalizable generation.

  2. I found the qualitative examples in the appendix, where the DAEC samples seem to consistently have darker backgrounds than the baseline. I wonder whether this could reflect reward hacking or lend support to the diversity concern above. A brief explanation would be sufficient here. I will not lower my rating on this aspect. :)

Overall, I appreciate the authors’ responses and find the paper internally coherent, although I am not yet fully convinced by all aspects of the method. I have not yet finalized my rating and tended to defer the final decision after checking other reviewer's concerns.

Thank you for the follow-up. We appreciate your concerns about diversity and generalization.

  1. Local robustness and diversity. RAM uses reward feedback to learn from sampled endpoints through analytically constructed states. DAEC aims to make this supervision effective around each training state, so that small state deviations during generation do not undermine the learned behavior. Each example retains its own velocity target; different generated samples are not assigned a shared output. Figure 1 shows that deviations in intermediate states can affect final generation quality, motivating us to train the model to better tolerate such deviations.

  2. Darker backgrounds and reward hacking. Although several DAEC examples have darker backgrounds, they retain the requested objects, colors, and spatial relationships. Table 2 in the revised manuscript provides further evidence: under OCR training, DAEC improves both the task reward and all four DrawBench quality metrics—ImageReward, AestheticScore, HPSv2, and PickScore—at both matched training budgets. These quality metrics are not used as the training reward. We believe these improvements indicate that DAEC enhances generation quality beyond the optimized reward, providing evidence against the concern that its gains arise solely from reward hacking.

We sincerely thank the reviewer for these constructive suggestions, which have helped us clarify the motivation and interpretation of our method and improve the paper.

Reviewer_K2VPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Endpoint-based RL methods for flow models (Diffusion-NFT, AWM, RAM) train on states rebuilt by analytic forward noising of rewarded endpoints, and these states can differ from the states the policy actually visits during sampling. The paper studies this mismatch in RAM and proposes DAEC, which moves only the trainable model input along the normalized bridge residual utu_t, with a magnitude set by a fixed time profile g(t)g(t), a warmup, and a sample gate qiq_i, while the RAM target stays anchored at XtanaX_t^{ana}.

Critical strengths
  1. The problem is real and not well studied. Endpoint-based RL is becoming common, and the paper points out a clear gap between the analytic training states and the rollout states. The three observations (where the gap is large, when correcting it helps reward, and whether the direction matters) are a reasonable way to look at it, and the fresh-noise and norm-matched random controls make the analysis more careful than a single plot would.
  2. The method is cheap and easy to add. The direction utu_t reuses the lagged-policy prediction that RAM already computes, so DAEC adds no model evaluations and about 2% wall-clock overhead. The math linking utu_t to the noise discrepancy and to the bridge score checks out, and the pseudocode makes the method easy to reproduce.
  3. The evaluation goes past a single number. GenEval gains are reported together with image quality metrics, across samplers, CFG scales, and step counts, over three training seeds, and on a second backbone (Qwen-Image). DAEC is above RAM in every inference setting tested.
Critical weaknesses
  1. At the matched budget, GenEval goes from 0.9241 to 0.9376. The 0.9462 reported as "RAM + DAEC (best)" is the best checkpoint of training seed 0; the other two seeds give 0.9349 and 0.9317. RAM is run with only one training seed, so its spread is unknown. The ablation in Table 2 is also single-seed and uses best checkpoints: the gaps between the full method (0.9462), global gating (0.9435), and uniform profile (0.9354) are all smaller than the spread across DAEC seeds (0.9317 to 0.9462). So the table does not show that each part matters. OCR and PickScore are only shown as single training curves in Fig. 3, with no table, no seeds, and no image quality checks, even though both rewards are easy to hack. The Qwen-Image result compares DAEC at 80 steps with RAM at 110 steps, which is not a matched budget.
  2. Observation II moves a rollout state toward its own seed ϵ0\epsilon_0, while DAEC moves an analytic state along +ut+u_t, which the paper calls the "outward side" of the correction axis. Fig. 6(a) shows that the alignment between utu_t and the actual rollout gap changes sign near tt about 0.54, right in the middle of the regime that g(t)g(t) targets. So for part of that regime the perturbation points away from the rollout state, and it is not clear why training there should reduce exposure bias. There is no ablation that flips the sign of utu_t. The robustness test in Fig. 4(b) perturbs along utu_t, the same direction DAEC trains on, so a smaller reward drop there is expected and is not independent evidence. The observations also use AestheticScore on a RAM model, while training uses GenEval, and the Observation II effect is small (about +0.028 AestheticScore at tt about 0.52).
  3. The paper argues that existing exposure-bias methods "may be insufficient in the RL setting", but none of them (input perturbation of Ning et al., MixFlow, scheduled sampling) is applied to RAM as a baseline. The random-direction ablation keeps g(t)g(t) and the gate, so it is not the same as plain input perturbation. DAEC is called a "plug-in method", yet it is only tested on RAM, not on Diffusion-NFT or AWM. The comparison with published baselines in Table 1 uses training steps, but the methods use different numbers of samples per step, and it is not said whether those rows are rerun or copied. The fixed profile g(t)g(t) is calibrated on a GenEval RAM model at step 100 and reused for other rewards and for Qwen-Image without checking. Related work also misses earlier reward-alignment methods that train on forward-noised endpoints and so share the same mismatch, and work on train and test mismatch in noise schedules. Suggested references: Lee et al., "Aligning Text-to-Image Models using Human Feedback" (arXiv 2023); Wallace et al., "Diffusion Model Alignment Using Direct Preference Optimization" (CVPR 2024); Fan et al., "Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization" (ICLR 2025); Lin et al., "Common Diffusion Noise Schedules and Sample Steps are Flawed" (WACV 2024).
Questions for authors
  • Does DAEC help Diffusion-NFT or AWM, given that the paper calls it a plug-in method?
  • Given that utu_t changes sign against the rollout gap near tt about 0.54, why should training along +ut+u_t reduce exposure bias in that regime?
  • Please give final tables for OCR and PickScore with image quality metrics, and say whether the Table 1 baseline rows are rerun or copied from the papers.
  • In Table 3, RAM gives the same 0.9281 for DPM-Solver++, CFG = 2.0, and CFG = 3.0. Please confirm these are separate runs.
7 comments
AuthorsOpenReview ↗

Response to Reviewer K2VP [1/5]

Thank you for your detailed review and for recognizing the value of the diagnostics, the low computational overhead, and the breadth of the evaluation. We address your concerns below.

W1(a) / W1(b) Are the matched-budget gains and the component improvements reliable given training variability and best-checkpoint selection?

Response: We agree that RL fine-tuning can exhibit substantial variation across training runs. As an additional individual-run result, RAM achieved a best evaluated GenEval accuracy of 0.9236, slightly below the reported RAM result of 0.9241. Our ablations use the same training seed and shared settings to examine how each component affects generation quality. The full method achieves the highest GenEval score in this comparison, with the largest performance drop occurring when the residual direction is replaced by a random direction. We believe these results support the combined design, particularly the choice of direction.

We also evaluate both methods across three inference seeds. Table 6 reports 0.9438±0.00220.9438 \pm 0.0022 for DAEC versus 0.9227±0.00110.9227 \pm 0.0011 for RAM, supporting the consistency of the advantage across inference randomness. Table 7 provides additional DAEC training results, with a mean accuracy of 0.9376±0.00620.9376 \pm 0.0062. At the same 190-step training budget, Table 1 also shows an improvement from 0.9241 to 0.9376, together with gains in all four image-quality metrics.

W1(c) / Q3 Can you provide OCR and PickScore result tables, multiple training seeds, and image-quality metrics to assess possible reward hacking?

Response: We have added task scores and image-quality metrics for OCR and PickScore to Table 2 in the revised manuscript and discussed them in Section 5.2, “Multi-Metric Results.” The comparisons use the same training-step budget for RAM and RAM + DAEC within each task.

TaskMethodTraining stepsTask scoreImgRwdAestheticHPSv2PickScore
OCRRAM600.97 (0.9657)-1.583.430.160.75
OCRRAM + DAEC600.97 (0.9709)0.395.280.260.85
OCRRAM1200.96-1.773.620.130.74
OCRRAM + DAEC1200.970.735.350.280.85
PickScoreRAM2000.88--6.110.290.90
PickScoreRAM + DAEC2000.90--6.040.310.92

The last four columns report Image Quality Metrics on DrawBench. Higher is better; bold indicates the better result within each matched-budget comparison.

For OCR, DAEC improves the task score from 0.97 (0.9657) to 0.97 (0.9709) at 60 training steps, and from 0.96 to 0.97 at 120 steps. All four DrawBench image-quality metrics improve at both checkpoints. Thus, the OCR reward gains are accompanied by improvements in metrics not used as the training reward.

For PickScore at 200 training steps, the held-out task score improves from 0.88 to 0.90, DrawBench PickScore from 0.90 to 0.92, and HPSv2 from 0.29 to 0.31. AestheticScore is slightly lower, changing from 6.11 to 6.04. Figure 3 also shows a consistently higher PickScore at every step. Together, these results provide task and image-quality evidence beyond a single reward score, while showing the trade-off in AestheticScore.

The OCR and PickScore experiments each use one training seed per method. We are sorry that limited computational resources prevent us from providing additional training-seed runs for these tasks.

AuthorsOpenReview ↗

Response to Reviewer K2VP [2/5]

W1(d) Can you compare Qwen-Image at matched training budgets rather than 80 versus 110 steps?

Response: We have moved the Qwen-Image results to Section 5.2 and Table 3 in the main text, showing the available checkpoints from 50 to 90 training steps. Both methods are evaluated at 512×512512\times512 resolution under the same protocol. The scores are image-level accuracy; Qwen-Image before fine-tuning scores 0.8535 under the same protocol.

Training stepsRAMRAM + DAEC
500.92090.9344
600.92450.9390
700.93440.9439
800.94030.9498
900.9349--

DAEC achieves higher accuracy at all four matched checkpoints from 50 to 80 steps. At 80 steps, the gain is 0.95 percentage points. We also include RAM at 90 steps (0.9349), which remains below DAEC at 80 steps (0.9498).

W2(a) / Q2 Why use the positive residual direction when Observation II corrects the inferred noise discrepancy and alignment with the rollout gap changes sign near t=0.54t=0.54, within the range targeted by g(t)g(t)? What happens when the training direction is reversed?

Response: We apologize for not making the distinction between the diagnostic intervention and the training perturbation clearer. In Figure 1(b), and in the original-noise correction arm of Figure 1(c), we apply a one-time correction at a selected rollout timestep: we keep the instantaneous clean estimate fixed, replace the inferred noise with the original noise seed, and continue sampling with the unchanged model. The intermediate-stage reward gains show that addressing this discrepancy can benefit generation.

DAEC aims to improve the model's ability to handle related input deviations through training. During generation, no prescribed clean endpoint is available for checking intermediate states against an analytic path. During training, we have the sampled endpoint and its analytic training pair. We perturb the input and train the model against the RAM target constructed at the original state. We therefore agree that DAEC can be understood as directional robustness augmentation.

The positive sign has a concrete interpretation in this construction. Eq. (13) shows that the residual points from the assigned noise toward the noise inferred by the lagged policy. Perturbing the input along the positive residual direction is equivalent to keeping the clean endpoint fixed and changing the assigned noise in that direction, with the magnitude controlled separately. We choose this direction to construct training inputs that contain the noise deviations reflected in the model's own predictions. By training on these inputs with the original RAM supervision, we aim to improve the model's ability to handle such deviations. Figure 1 tests a correction during sampling, while DAEC trains on perturbed analytic inputs; they operate at different states with their respective local discrepancies. The positive residual need not point toward the paired rollout state.

The reported training results and random-direction comparison support this design. During method development, we also tested the opposite training direction, Xtana−ηtu^tX_t^{\mathrm{ana}}-\eta_t\widehat{u}_t, and symmetric positive/negative perturbations. Both exploratory tests encountered training divergence. Their best observed GenEval scores were approximately 0.83 for the opposite direction and 0.80 for symmetric perturbations. Based on these observations and the random-direction comparison, we adopted the positive residual direction in DAEC.

We have added this explanation in Section 4.1 and clarified the one-time diagnostic intervention in Section 3.3.

AuthorsOpenReview ↗

Response to Reviewer K2VP [3/5]

W2(b) Does Figure 4(b) provide independent evidence for the explanation, given that it tests robustness along the same direction used during training?

Response: Figure 4(b) shows that DAEC is more robust to perturbations along the direction used during training. Beyond this test, Figure 4(c) shows a 14.9% lower trajectory-conditioned velocity gap on average during normal generation in the intermediate-noise regime, and Tables 1 and 2 show gains in task performance and image quality. We believe these results support DAEC's design of training the model to better handle residual-related state deviations.

W2(c) How do the small diagnostic gains on AestheticScore support DAEC training on GenEval?

Response: The diagnostic asks whether correcting an intermediate-state discrepancy can improve the final generation while keeping the model fixed. We apply the correction only once at a selected timestep and then continue sampling with the unchanged model. The approximately +0.028 AestheticScore gain shows that this intervention can benefit generation. Appendix G.1 and Figure 5 report similar intermediate-stage gains with DeQAScore and HPSv2.

These observations motivate training the model to better handle such state deviations. We evaluate the resulting training method directly on GenEval: at the same budget of 190 training steps, DAEC improves the score from 0.9241 to 0.9376, together with improvements in all four reported image-quality metrics in Table 1. Thus, the diagnostic provides the motivation, while the GenEval training results provide direct evidence of the method's effectiveness on compositional generation.

Reviewer_X5YdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper study the state exposure mismatch between analytically reconstructed training states and actual inference trajectories in RL of flow models. Authors study the discrepancy and get observations: timestep dependence, effect on terminal reward, and trajectory-dependent direction. Based on those observations, they propose DAEC, perturbing the trainable model's input with a normalized bridge residual, with a timestep-dependent magnitude and a sample-adaptive gate. Experimemts show its improvements

Critical strengths
  1. The observation about the state exposure mismatch between analytically reconstructed training states and model-induced rollout states is insightful.
  2. DAEC is a simple method, which only adjust the input of trainable models. the traing overhead is limited.
  3. experiments on SD 3.5M show the effectiveness of the proposed method.
Critical weaknesses
  1. Most of the expeirments are conducted on SD 3.5M, which is somewhat old model. Suggest that more recent open-source models and other models should also included to valid the effectiveness of the methods.
  2. From Table 1, the improment is marginal.
  3. The reviewer also wants to see comparisons with simple perturbation baselines, such as perturbing along -u instead of +u; standard gaussian input perturbations with matched perturbation norms, etc. The improvements may come from input regularization rather than the proposed exposure correction.
  4. Most of the quantitative results and ablations are on GenEval, which is not sufficient. Besides, user study is suggested.
Questions for authors
  1. From Table 2, it is unclear whether the two settings use the same average or RMS perturbation magnitude.
  2. The comparison with different methods uses training steps. However, they have different rollout and costs. better to compare training time/ total computation cost.
  3. Can you compare DAEC over a tuned standard input-perturbation baseline with the same perturbation norm and timestep schedule?
4 comments
AuthorsOpenReview ↗

Response to Reviewer X5Yd [1/3]

Thank you for your thoughtful review and for recognizing the relevance of the problem and the simplicity of DAEC. We address your concerns below.

W1 Can you evaluate DAEC on more recent models and additional backbones beyond SD3.5 Medium?

Response: We have also evaluated DAEC on Qwen-Image and moved the results to Section 5.2 and Table 3 in the main text, comparing the available checkpoints from 50 to 90 training steps. Before fine-tuning, Qwen-Image scores 0.8535 under the same evaluation protocol. The fine-tuned results are:

Training stepsRAMRAM + DAEC
500.92090.9344
600.92450.9390
700.93440.9439
800.94030.9498
900.9349--

DAEC achieves consistently higher scores at different steps. Both models are evaluated at 512×512512\times512 resolution under the same protocol. Together with the SD3.5M results, we believe these improvements support DAEC’s applicability across model architectures.

W2 How meaningful is the modest improvement in Table 1?

Response: We believe the practical value of DAEC lies in improving both generation quality and performance with fewer sampling steps, while requiring no additional model evaluations per training update.

At the same budget of 190 training steps in Table 1, DAEC improves the reported GenEval score from 0.9241 to 0.9376, a gain of 1.35 percentage points. All four reported image-quality metrics also improve: Aesthetic from 5.19 to 5.33, DeQA from 3.78 to 4.03, ImageReward from 0.98 to 1.01, and HPSv2 from 0.2679 to 0.2734. Thus, the task-score gain is accompanied by improvements in the reported quality measures.

The inference-step results in Table 5 show an additional practical benefit. With FlowMatch-Euler and CFG 4.5, DAEC achieves 0.9367 at 10 inference steps, compared with 0.9164 for RAM at the same step count. DAEC at 10 steps also exceeds RAM at 50 steps (0.9254). At 20 steps, the comparison is 0.9494 versus 0.9218. These results show that the DAEC retains strong performance under a reduced sampling budget.

These benefits come from changing the training input while reusing the reference and lagged-policy predictions computed by RAM. DAEC retains the same three model evaluations per training update, with a one-time calibration of the timestep profile. We believe that the combination of improved task and quality scores, useful performance at fewer inference steps, and this low additional training cost makes the method practically useful.

AuthorsOpenReview ↗

Response to Reviewer X5Yd [2/3]

W3 / Q3 How does DAEC compare with the opposite residual direction and tuned Gaussian input perturbations with matched norms and timestep schedules? Could the gains come from input regularization?

Response: We apologize for not making the distinction between the diagnostic intervention and the training perturbation clearer. In Figure 1(b), and in the original-noise correction arm of Figure 1(c), we apply a one-time correction at a selected rollout timestep: we keep the instantaneous clean estimate fixed, replace the inferred noise with the original noise seed, and continue sampling with the unchanged model. The intermediate-stage reward gains show that addressing this discrepancy can benefit generation.

DAEC aims to improve the model's ability to handle related input deviations through training. During generation, no prescribed clean endpoint is available for checking intermediate states against an analytic path. During training, we have the sampled endpoint and its analytic training pair. We perturb the input and train the model against the RAM target constructed at the original state. We therefore agree that DAEC can be understood as directional robustness augmentation.

The positive sign has a concrete interpretation in this construction. Eq. (13) shows that the residual points from the assigned noise toward the noise inferred by the lagged policy. Perturbing the input along the positive residual direction is equivalent to keeping the clean endpoint fixed and changing the assigned noise in that direction, with the magnitude controlled separately. We choose this direction to construct training inputs that contain the noise deviations reflected in the model's own predictions. By training on these inputs with the original RAM supervision, we aim to improve the model's ability to handle such deviations. Figure 1 tests a correction during sampling, while DAEC trains on perturbed analytic inputs; they operate at different states with their respective local discrepancies. The positive residual need not point toward the paired rollout state.

The reported training results and random-direction comparison support this design. During method development, we also tested the opposite training direction, Xtana−ηtu^tX_t^{\mathrm{ana}}-\eta_t\widehat{u}_t, and symmetric positive/negative perturbations. Both exploratory tests encountered training divergence. Their best observed GenEval scores were approximately 0.83 for the opposite direction and 0.80 for symmetric perturbations. Based on these observations and the random-direction comparison, we adopted the positive residual direction in DAEC.

For the random-input comparison, Table 4 includes a control that replaces the residual direction with a normalized Gaussian random vector while retaining DAEC's timestep profile, warmup, and sample-dependent magnitude rule. For a given analytic state and model, this changes the direction while preserving the perturbation norm. Table 4 reports best GenEval scores of 0.90 for the random-direction variant and 0.95 (0.9462) for full DAEC. This supports the usefulness of the residual direction in the evaluated setting. We keep DAEC's perturbation magnitude settings fixed and replace only the residual direction with a Gaussian random direction to isolate the effect of direction.

We have added this explanation in Section 4.1 and clarified the one-time diagnostic intervention in Section 3.3.

AuthorsOpenReview ↗

Response to Reviewer X5Yd [3/3]

W4 Can you provide broader task evaluation beyond GenEval and a user study?

Response: We have also evaluated DAEC on visual text rendering with an OCR reward and preference alignment with a PickScore reward. Table 2 in the revised manuscript reports task scores and image-quality metrics at matched training budgets. The task results are summarized below:

TaskTraining stepsRAMRAM + DAEC
OCR600.97 (0.9657)0.97 (0.9709)
OCR1200.960.97
PickScore2000.880.90

Table 2 also shows that OCR task gains are accompanied by improvements in all four image-quality metrics. For PickScore, both DrawBench PickScore and HPSv2 improve, while AestheticScore is slightly lower. We believe these results support DAEC's usefulness across tasks beyond GenEval.

We agree that a user study would provide a valuable complement to these automatic evaluations. Following the reviewer's suggestion, we will conduct a user study in future work.

Q1 Do the Table 4 variants use the same average or RMS perturbation magnitude?

Response: We match the average perturbation magnitude in the uniform-profile comparison. In the random-direction comparison, we retain the same magnitude rule and replace only the unit direction, preserving the perturbation norm for the same input state and model. The random-direction variant performs worse than full DAEC in Table 4, supporting the value of the residual direction beyond simply adding a perturbation.

Figure 1(c) provides a related diagnostic comparison: the random-direction control matches the original-noise correction's displacement norm for each sample. The original-noise correction yields greater reward improvements at several intermediate timesteps, supporting the importance of the trajectory-specific direction at a matched perturbation magnitude.

Q2 Can you compare training time or total computation, given that rollout procedures and per-step costs differ across methods?

Response: As the reviewer notes, rollout procedures and per-step costs differ across methods. Based on the reported GenEval training costs and our DAEC overhead estimate, we find that Flow-GRPO takes approximately 5% longer per training step than RAM, while RAM + DAEC takes approximately 2% longer under our training configuration. Thus, DAEC improves generation performance with only a small increase in per-step training time over RAM.

Dear Reviewer X5Yd,

Thank you again for your constructive feedback. As the rebuttal discussion deadline approaches, we would be grateful if you could take a moment to review our responses and the revised manuscript. We hope these clarifications address your concerns and would be happy to discuss any remaining questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We thank the reviewers for their helpful comments. We have revised the manuscript to clarify the method and expand the presentation of our results. The main changes are:

  • Sections 3.3 and 4.1, and Figure 1: Clarified that the diagnostic correction is applied only once, explained its distinction from DAEC's training perturbation and the choice of the positive residual direction, and described the inset in Figure 1(c).
  • Table 1: Clarified that the AWM and Diffusion-NFT results are taken from the paper, while the RAM baseline was reproduced by us.
  • Section 5.2 and Table 2: Added OCR and PickScore task scores and image-quality metrics at matched training budgets.
  • Section 5.2 and Table 3: Moved the Qwen-Image results into the main text and added comparisons at matched training checkpoints.
  • Appendix B: Added the suggested references and discussed earlier reward-alignment methods and training–sampling mismatch in noise schedules.
  • Appendix, “Robustness to Inference Configurations”: Clarified that the identical RAM scores under three inference configurations come from separate evaluations.

The main additions are highlighted in blue in the revised manuscript. We hope these revisions and the responses below clarify our method and address the reviewers' concerns. We look forward to the reviewers' feedback and are happy to clarify any remaining questions.

AuthorsOpenReview ↗

Summary of Contributions, Revisions, and Reviewer Discussion

Dear Area Chair,

We would like to summarize our contribution and the progress made during the rebuttal:

  • Contribution: We investigate state exposure mismatch in reward-based flow-model fine-tuning and propose DAEC, a lightweight training augmentation that improves robustness while preserving the original supervision.

  • Revisions: We expanded the explanation of the perturbation direction and its relationship to the diagnostic observations, clarified the robustness interpretation, and strengthened the experimental evidence and related-work discussion.

  • Reviewer feedback: Reviewers recognized the relevance of the problem, the informative diagnostic analysis, and DAEC’s simplicity and low computational overhead. During the discussion, vCKb acknowledged that our responses clarified several previously unclear points and found the paper internally coherent. K2VP indicated that they had no further comments after our responses. We also addressed X5Yd’s concerns through additional evidence and clarifications, although we did not receive a follow-up before the discussion period ended.

We hope this summary is helpful for your final assessment. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.