- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the mismatch between analytically reconstructed training states and policy-induced rollout states in endpoint-based RL for flow models. Authors argue that this mismatch is timestep-dependent, reward-relevant, and trajectory-specific. To close this gap, they propose Drift-Aligned Exposure Correction (DAEC), which perturbs the trainable model input along a normalized bridge-residual direction. Extensive experiments on various benchmarks confirm DAEC's consistent improvement over RAM.
Critical strengths
-
The paper identifies a concrete and underexplored limitation of endpoint-based flow-model RL.
-
DAEC is lightweight and easy to integrate into RAM. It seems that DAEC could improve the performance across various tasks.
Critical weaknesses
-
The authors' diagnostic findings do not fully justify the perturbation they implement. In the diagnostic experiment, what helps is pulling a rollout state against its inferred noise discrepancy. However, DAEC moves an analytic state along the positive residual direction.This can be viewed and interpreted as robustness augmentation. However, the paper does not compare the positive and negative directions. Therefore, the proposed direction does not directly follow the stated observation.
-
DAEC evaluates the trainable model at a perturbed input while keeping the target and reference prediction unchanged at the original analytic state. The resulting target is generally not the conditional velocity at the perturbed state. This implicitly assumes local invariance of the velocity field and may act as an uncontrolled smoothing regularizer.
Questions for authors
- What happens when training with the negative residual direction, or with symmetric positive/negative perturbations?
- Renoising is a common operation in this area. I have a little doubt on the claim of exposure bias. From the perspective of ODE sampling, each state at a same timestep has a same marginal distribution. Therefore, I think directly claiming sample-level difference as exposure bias is not completely adequate.
- What does the image in the right down corner in Figure 1(c) mean?
- Please see weaknesses.
At this time, I'm still confused on some claims in the paper, and I think some design does not directly address the challenge proposed in the motivation. However, I would like to enhance my rating if authors could fully address my concerns.
4 comments
Response to Reviewer vCKb [2/2]
W2 / Q4 Why keep the target and reference at the original state? The unchanged target may not be the conditional velocity at the new state and may impose local invariance or uncontrolled smoothing.
Response: Keeping the original RAM target is a deliberate part of the training augmentation. We want the model to satisfy the original supervision when its input undergoes a residual-related deviation. The reference and lagged-policy predictions are components of that target, so we evaluate both at the original analytic state. Recomputing them at the perturbed state would change the supervision together with the input. We use the original target to supervise the augmented input, rather than to estimate the exact conditional velocity at the perturbed state.
We agree that this can encourage local directional smoothing, which is consistent with our aim of improving robustness to these input deviations. Section 4.3 describes this interpretation through a local expansion around a fitted training point. Prior work also supports the value of smoothness for efficient sampling. Yang et al., Lipschitz Singularities in Diffusion Models (ICLR 2024), show that mitigating temporal Lipschitz singularities improves fast sampling with DDIM and DPM-Solver. Their Section 5.4 explains that larger discretization steps place greater demands on smoothness. Their analysis concerns variation with time; DAEC encourages reduced sensitivity along a selected direction in the input state.
Our results in the inference-step block of Table 5 show that DAEC retains an advantage with few sampling steps. Using FlowMatch-Euler with CFG 4.5, RAM + DAEC achieves GenEval scores of 0.9367 and 0.9494 at 10 and 20 steps, compared with RAM's 0.9164 and 0.9218, respectively. DAEC at 10 steps also exceeds RAM at 50 steps (0.9254). We believe that improved tolerance to state deviations may contribute to this performance under fewer sampling steps.
Q2 How do paired-state differences establish exposure bias when different sample paths can share the same marginal distribution?
Response: We agree that different sample paths can share the same time-marginal distributions, so a paired-state difference alone is insufficient to establish a distribution mismatch. Our diagnostic measures the magnitude of a trajectory-conditioned local correction, rather than a distance between marginal distributions.
The motivation comes from how RAM constructs its training states. RAM independently noises generated endpoints, whereas inference follows the current model’s ODE trajectories. RAM’s analysis establishes that analytic noising recovers the controlled-SDE joint distribution at initialization and at the optimum; this does not establish the same correspondence throughout intermediate optimization. This motivates examining how supervision on analytically reconstructed states transfers to actual generation.
Our evidence therefore includes interventions beyond measuring paired differences. In Figure 1(b), a one-time correction with the model held fixed improves terminal reward at intermediate timesteps. Figure 1(c) compares this intervention with fresh-noise and norm-matched random controls, supporting the relevance of the trajectory-specific direction. These observations motivate DAEC’s directional robustness augmentation: training the model to handle related input deviations while preserving the original RAM supervision. The task and image-quality results provide further evidence of the practical value of this design.
Q3 What does the lower-right inset in Figure 1(c) show?
Response: The inset zooms in on the intermediate timestep range , with ticks at 0.7, 0.5, and 0.3, to compare the three mean reward-change curves more clearly. We apologize for the confusion caused by the missing explanation. We have added this clarification to the caption of Figure 1(c).
Thanks for your responses
Thank you for your responses, which help clarify several previously unclear points. I appreciate these explanations, although some concerns remain relevant to my assessment.
-
Authors' explanation of perturbations with an unchanged estimated target appears consistent with my original understanding: this is a smoothing technique intended to improve robustness. However, I remain concerned that mapping multiple noisy states to the same target could reduce generation diversity—an important issue in online RL. Based on my experience, encouraging consistency across different denoising timesteps has a clearer rationale. The motivation for perturbing the noisy states themselves remains less convincing to me. Intuitively, current techniques in the manuscript are sufficient for enhancing reward values, but I'm not sure whether this really helps generalizable generation.
-
I found the qualitative examples in the appendix, where the DAEC samples seem to consistently have darker backgrounds than the baseline. I wonder whether this could reflect reward hacking or lend support to the diversity concern above. A brief explanation would be sufficient here. I will not lower my rating on this aspect. :)
Overall, I appreciate the authors’ responses and find the paper internally coherent, although I am not yet fully convinced by all aspects of the method. I have not yet finalized my rating and tended to defer the final decision after checking other reviewer's concerns.
Thank you for the follow-up. We appreciate your concerns about diversity and generalization.
-
Local robustness and diversity. RAM uses reward feedback to learn from sampled endpoints through analytically constructed states. DAEC aims to make this supervision effective around each training state, so that small state deviations during generation do not undermine the learned behavior. Each example retains its own velocity target; different generated samples are not assigned a shared output. Figure 1 shows that deviations in intermediate states can affect final generation quality, motivating us to train the model to better tolerate such deviations.
-
Darker backgrounds and reward hacking. Although several DAEC examples have darker backgrounds, they retain the requested objects, colors, and spatial relationships. Table 2 in the revised manuscript provides further evidence: under OCR training, DAEC improves both the task reward and all four DrawBench quality metrics—ImageReward, AestheticScore, HPSv2, and PickScore—at both matched training budgets. These quality metrics are not used as the training reward. We believe these improvements indicate that DAEC enhances generation quality beyond the optimized reward, providing evidence against the concern that its gains arise solely from reward hacking.
We sincerely thank the reviewer for these constructive suggestions, which have helped us clarify the motivation and interpretation of our method and improve the paper.
Response to Reviewer vCKb [1/2]
Thank you for your careful review and for recognizing the importance of the problem and the lightweight design of DAEC. We address your questions about the motivation, training objective, and interpretation below.
W1 / Q1 / Q4 Why perturb analytic states along the positive residual direction when the diagnostic uses a pullback? Is this robustness augmentation? What happens with negative or symmetric positive/negative perturbations?
Response: We apologize for not making the distinction between the diagnostic intervention and the training perturbation clearer. In Figure 1(b), and in the original-noise correction arm of Figure 1(c), we apply a one-time correction at a selected rollout timestep: we keep the instantaneous clean estimate fixed, replace the inferred noise with the original noise seed, and continue sampling with the unchanged model. The intermediate-stage reward gains show that addressing this discrepancy can benefit generation.
DAEC aims to improve the model's ability to handle related input deviations through training. During generation, no prescribed clean endpoint is available for checking intermediate states against an analytic path. During training, we have the sampled endpoint and its analytic training pair. We perturb the input and train the model against the RAM target constructed at the original state. We therefore agree that DAEC can be understood as directional robustness augmentation.
The positive sign has a concrete interpretation in this construction. Eq. (13) shows that the residual points from the assigned noise toward the noise inferred by the lagged policy. Perturbing the input along the positive residual direction is equivalent to keeping the clean endpoint fixed and changing the assigned noise in that direction, with the magnitude controlled separately. We choose this direction to construct training inputs that contain the noise deviations reflected in the model's own predictions. By training on these inputs with the original RAM supervision, we aim to improve the model's ability to handle such deviations. Figure 1 tests a correction during sampling, while DAEC trains on perturbed analytic inputs; they operate at different states with their respective local discrepancies. The positive residual need not point toward the paired rollout state.
The reported training results and random-direction comparison support this design. During method development, we also tested the opposite training direction, Xtana−ηtut, and symmetric positive/negative perturbations. Both exploratory tests encountered training divergence. Their best observed GenEval scores were approximately 0.83 for the opposite direction and 0.80 for symmetric perturbations. Based on these observations and the random-direction comparison, we adopted the positive residual direction in DAEC.
We have added this explanation in Section 4.1 and clarified the one-time diagnostic intervention in Section 3.3.