acceptodds
Withdrawn from ICLR 2027

Exploration–Exploitation Dilemma Introduced by Diffusion RL Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite advances in image generation, diffusion models still struggle to directly align with specific preferences, while reinforcement learning (RL) offers an effective solution but introduces challenges in coordinating exploration and exploitation. Over-exploration reduces learning efficiency, while over-exploitation traps the policy in local optima and may even cause reward hacking. This dilemma is further aggravated in diffusion model fine-tuning by sparse rewards and long-horizon denoising. Against this backdrop, we present a systematic theoretical analysis of the coordination between exploration and exploitation in diffusion fine-tuning. First, we derive an analytical expression for the optimal policy under an uncertainty-driven exploration term. We then derive three coordination cases describing how exploration and exploitation evolve. Based on these theories, we formalize an adaptive framework named \ourmethod. Extensive 24 experiments show that our method outperforms state-of-the-art approaches in diversity (8.32%), alignment (10.30%), and speed (88.64%).

Primary area:
interpretability and explainable AI
Keywords:
Diversity-Driven Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AzoiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes E2Ada, an adaptive exploration-exploitation framework for RL fine-tuning of diffusion models, aiming at reducing diversity issues and resolving reward hacking. Specifically, E2Ada introduces an intermediate exploration reward based on structural differences between consecutive predictions of clean images. The authors also provide a theoretical analysis to relate exploration rewards to policy entropy. The proposed method is evaluated across diffusion and flow matching models with multiple reward tasks and metrics, and improvements in diversity, alignments, and efficiency are reported.

Critical strengths

Instead of judging performance merely by the training reward, the authors compare image quality and diversity at similar levels of training reward, which reflects improved generation rather than merely obtaining higher scores from the reward model. Extensive experiments conducted on multiple baselines, tasks, and metrics demonstrate the effectiveness of the proposed method.

Critical weaknesses

The proof of Theorem 1 seems unclear to me. In Appendix B.1 (lines 1059-1061), the optimal value function for the combined reward is expressed as the sum of two separately optimized value functions. This does not hold in general, since the two sub-rewards may favor different subsequent actions. This might undermine the main theoretical result, and thus either a clarification on the assumptions or a corrected proof is expected.

Also, Lemma 2.1 requires specific ordering and magnitude conditions between exploration and exploitation values, but the paper does not show that the SSIM-based reward satisfies them.

Questions for authors

Please refer to the 'Critical Weaknesses' section and address the concerns raised.

Reviewer_5gzXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when a thinking model should learn from a teacher that can see the worked solution. FS-OPSD uses the same frozen starting model with and without solution access. It uses the problem-only teacher when several next tokens remain plausible, and the solution-conditioned teacher when one candidate dominates. It also softens competing candidates and keeps only the top 20 tokens in each training target. Experiments cover five main thinking models and three mathematics benchmarks, with additional small-model results in the appendix.

Critical strengths
  1. The results show useful gains across several thinking models. In Table 1, FS-OPSD beats vanilla OPSD on all 15 main model–benchmark pairs and improves the average of all five main models over Base. The largest average gains are 8.26 points over OPSD and 2.36 over Base. The training curves also suggest that the gap from OPSD persists during training. Appendix E reports the small-model exception, which helps show the limits of the result.
  2. The idea is easy to follow, and the source ablations provide useful support. On Qwen3-4B, using the problem-only teacher everywhere is 1.11 points worse, while adding solution access at forks is 1.04 points worse (Table 2). These comparisons give some evidence for choosing the supervision source by state. Appendix D clearly describes the training and evaluation procedure, and the threshold derivation acknowledges its limits.
Critical weaknesses
  1. It is still hard to separate the benefit of choosing when to use the solution from the other target changes. Vanilla OPSD does not use the same softening and top-20 restriction. Table 2 helps, but a control that uses the solution at every position with those changes is missing. The claim about preserving useful exploration is also less direct: changing the preferred token does not show that the displaced path would help solve the problem. Appendix A acknowledges that its example establishes no correctness advantage. A matched control and clearer evidence of successful correction would strengthen the main explanation.
  2. The reliability of the smaller gains is not fully clear. Appendix D reports one training seed, and each benchmark has only 30 problems with 16 responses per problem. The paper gives no uncertainty estimates or variation across training runs. Average gains over Base are only 0.41 points for Qwen3-8B and 0.62 for GLM. The responses to one problem do not provide 16 independent test problems. Tables 3–4 also use benchmark performance to support the default choices, without clearly stating whether those choices were fixed before testing. These details matter for judging the gains and ablation rankings. The gains are useful, but these concerns keep me at weak reject. A matched comparison, clearer uncertainty reporting and stronger evidence of useful exploration would make me more confident. If such evidence is unavailable, the contribution should be stated more narrowly.
Questions for authors
  1. With the same temperature adjustment and top-20 restriction, how does using the solution-conditioned teacher at every position compare with FS-OPSD?
  2. Can you report uncertainty for the main accuracy gains and key source ablations, including variation across training seeds where results are available?
  3. Were the thresholds, temperature and reported checkpoints fixed before evaluating the three benchmarks, or were these test results used to choose them?
  4. After controlling for response length, does preserving alternative continuations improve correction success or final correctness, beyond changing token probabilities and reasoning style?
1 comment
AuthorsOpenReview ↗

Review Submitted for the Wrong Paper

Dear Reviewer,

Thank you for your review. However, your comments appear to be intended for a different paper rather than our submission. Could you please provide the correct review for our paper?

Thank you for your time and consideration.

Reviewer_tkknOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the exploration–exploitation trade-off in reinforcement learning (RL) fine-tuning of diffusion models and proposes the E2Ada framework. Specifically, E2Ada introduces an intermediate exploration reward based on 1−SSIM1-\mathrm{SSIM} between clean image predictions at consecutive denoising states, adaptively adjusts the exploration weight using a denoising progress predictor, and employs an Adaptive Termination Strategy (ATS) to reduce low-benefit RL updates in the later stages of denoising.

Starting from an entropy-regularized Markov Decision Process (MDP) formulation, the authors attempt to theoretically analyze the effects of the exploration reward on the optimal policy and policy entropy. Through experiments on conventional diffusion models and Flow Matching models, they report improvements in preference rewards, diversity, and training efficiency.

Critical strengths

S1. The problem addressed in this paper is of practical importance, particularly regarding the degradation of generation diversity and excessive reward hacking caused by RL fine-tuning. The proposed framework integrates intermediate exploration rewards, state-dependent exploration weights, and adaptive optimization into a unified approach. Its compatibility with DDPO, DPOK, TDPO, and Flow-GRPO demonstrates good modularity and adaptability, offering a promising direction for improving existing generative RL methods.

S2. The paper provides extensive experiments covering multiple generative models, datasets, and reward configurations. The experimental methodology is also relatively rigorous. In particular, Table 1 compares different methods after training them to achieve approximately the same AES score, and then evaluates generation quality and diversity using metrics that are not directly optimized. This setup is more effective in revealing reward overoptimization than simply comparing the target rewards. Overall, the experiments provide substantial empirical evidence for improving the trade-off among reward maximization, generation quality, and computational efficiency.

Critical weaknesses

W1. The paper contains significant theoretical issues that undermine its central claim of addressing the exploration–exploitation trade-off. In Theorem 1 (page 3, lines 131–136), the authors attempt to derive the optimal loss function for the combined objective by linearly adding the value functions obtained from the soft Bellman recursion. However, the Log-Sum-Exp operator in the soft Bellman equation does not generally admit such a linear decomposition. This issue is not adequately addressed in Appendix B.1. Furthermore, Theorem 2 claims to rely on the conditions established in Lemma 2.1 (page 3), yet Case II (page 4) presents an opposite conclusion regarding the entropy comparison. The authors should carefully revisit the consistency and mathematical rigor of their theoretical analysis.

W2. The novelty of the proposed method remains unclear, particularly regarding its incremental contributions over existing approaches at the mechanism level. Notably, the experimental comparisons exclude AdaEnt and AdaScope, both of which are already cited in the paper and are closely related to the proposed framework in terms of adaptive entropy regulation and the timing of RL termination, respectively. The authors do not sufficiently discuss how their approach differs from these existing methods. Moreover, the analysis does not convincingly establish that the proposed SSIM-based exploration reward satisfies the value-ordering conditions required by Lemma 2.1, nor do the experiments demonstrate the conditional policy changes predicted by Theorem 2. Consequently, while the empirical results support the effectiveness of the proposed method, there remains a substantial disconnect between the theoretical analysis and the experimental evidence.

Questions for authors

Q1. (For W1) Under what conditions does the decomposition used in Theorem 1 hold? If the transformation in Theorem 1 is not universally valid, could it introduce inconsistencies between the original and decomposed objectives? If so, how might these inconsistencies affect the final training outcomes?

Q2. (For W2) Could the authors provide a more detailed analysis of the effectiveness of the 1−SSIM1-\mathrm{SSIM} exploration reward and demonstrate whether it satisfies the theoretical conditions underlying Case II? In particular, it would be valuable to provide direct empirical evidence supporting the predicted policy behavior, rather than relying solely on the diversity of the final generated images.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.