acceptodds
Withdrawn from ICLR 2027

Reinforcement Continual Learning for One-step Image Generation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce Reinforcement Continual Learning (RCL), a post-training framework that converts evaluation feedback into image-level supervision through a generate–construct–select–update loop. For each current-model output, RCL constructs candidate alternatives, determines preference eligibility, selects a suitable target, and learns from it at the original noise input and conditioning through an independently specified matching loss. Distinguishing preference eligibility, source–target compatibility, and matching allows black-box evaluation to guide learning while preserving the inference path. We instantiate RCL on a pMF-H one-step ImageNet generator with retrieved targets and ImageReward or Qwen preferences. Under their respective preference criteria, ImageReward and Qwen scores continue to improve throughout post-training. In comparison with random target-selection policies, the ImageReward-guided variant forms the empirical Pareto frontier for ImageReward and the multi-representation Fr\'echet-distance ratio fdr6. In a single-run comparison with Direct ImageReward, RCL also yields a more favorable ImageReward–fdr6 trajectory. These results demonstrate the framework's feasibility and characterize the preference–distribution trade-offs of its retrieval-based instantiation.

Primary area:
generative models
Keywords:
One-Step Image Generation, Continual Learning, Preference Alignment, Black-Box Evaluation Feedback, Retrieval-Based Target Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Dzv2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Targeting the one-step pMF generator, the paper proposes a post-training procedure that retrieves real-image candidates for each generated image, filters them by relative reward improvement, and selects the nearest eligible target in feature space.

Critical strengths

None

Critical weaknesses

[1] Performance is presented primarily through trajectory plots, without a consolidated numerical comparison.

[2] The algorithm is understandable, but Sections 4.1-4.2, Section 6.1, and Appendix D repeatedly discuss abstract interfaces and untested alternatives, including teacher-generated targets, human feedback, and alternative matching losses. These discussions could be condensed to prioritize implementation details and stronger experimental validation.

Questions for authors

[1] The “continual learning” terminology is not substantiated by the experimental setting. All experiments concern a single fixed ImageNet task, without changing tasks or domains, sequential knowledge acquisition, or retention and forgetting assessments.

[2] Black-box reward guidance, online supervision renewal, and generation-conditioned target search all have established precedents, making RCL appear primarily an engineering combination of existing techniques. What methodological novelty or nontrivial benefit does their combination introduce beyond prior approaches?

[3] Please report FID and Inception Score (IS) for both RCL and the pretrained pMF-H under the same evaluation protocol to assess how post-training affects generation quality and diversity.

[4] The composite matching objective lacks ablations justifying its MAE, DINOv2, Inception, LPIPS, and pixel components and their weights. The later deterioration in FD$r^6$ and increasingly similar backgrounds further motivate component-wise ablations and evaluation in feature spaces excluded from training to assess distribution drift and potential diversity loss.

[5] The paper lacks comparisons with DrPO, MeanFlowNFT, and DiffusionOPSD. Comparisons under matched backbones, rewards, and training budgets, are needed to establish RCL’s benefit beyond existing reward-guided post-training methods.

[6] Please ablate the preference thresholds and analyze how it affects target acceptance rates, the quality, diversity, and source compatibility of selected images, and the resulting preference-distribution trade-off.

[7] What value of λRCL\lambda_{\mathrm{RCL}} is used? Please report it explicitly and provide ablations on both λRCL\lambda_{\mathrm{RCL}} and the number of retrieved candidates KK.

Reviewer_qG4oOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a method called Reinforcement Continual Learning (RCL), which is a framework to improve image generation by incorporating expert feedback into image-level supervision. The supervision first generates/candidates alternative images, collects expert performance, selects preferred images, and learns from it. The paper tests RCL on a single pMF-H one-step ImageNet generator and uses ImageReward or Qwen preferences to post-train it and show that the methods show better performance than randomly selected preference. The work sees improvements on ImageReward-FDr^6 trajectory, showing potential for RCL.

Critical strengths
  • The paper proposes a potential way to incorporate preference feedback into one-step generator post-training while preserving one-step inference. The results show that RCL supervision can improve the generator with respect to the preference evaluator while also improving distribution alignment relative to the initial model. The results also show that IRNearest performs better on ImageReward scores, while QwenNearest performs better on Qwen scores, and both outperform random retrieval.
  • The qualitative examples show convincing improvements relative to the base model and Direct IR. Also, IRNearest achieves a better ImageReward–FDr6 trade-off trajectory than directly optimizing ImageReward. This provides encouraging visual and quantitative evidence for the approach.
Critical weaknesses
  • The proposed RCL differs from the existing Reverse XM by the introduction of preference-improvement filter, but the experiments do not cleanly ablate this component. There is no matched Reverse XM comparison or nearest-target baseline using the same retrieval pool (only Random is compared) and matching loss without preference filtering. RetrievalRandom also changes both filtering (uniformly random) and target candidate selection (not DINOv2), so its weaker performance does not establish the contribution of the preference constraint. This suggests that novelty of RCL is insufficiently supported
  • The experiments use only one generator, one dataset, and one run per configuration, which significantly limits reliability and breadth of the conclusions. It is unclear whether the improvements are consistent across runs or transfer to other generators and generation settings, which is an issue as the paper presents itself as a general post-training framework.
Questions for authors

Clarifications:

  • How were the ImageReward and Qwen margins of 0.5 and 1.8 selected? Were they calibrated to comparable acceptance rates, and what fraction of generated sources receives an eligible target throughout training?

Additional experiments:

  • Can you ablate the preference filter while keeping the same remaining setup (eight retrieved candidates, DINOv2 nearest-target selection for all instead of inception features, matching loss, and optimization settings)?
  • Can you provide comparisons to similar baselines, particularly Reverse XM and DrPO at matched hyperparameters?
  • How sensitive are the results to the preference margin and the number of retrieved candidates, K=8? Please report both performance and the associated acceptance rates and computational costs.
  • Can you repeat the main comparisons across multiple seeds and test a second generator or generation setting (e.g., multi-step or a different dataset)? This would help establish whether the observed gains are reproducible and generalize beyond the current pMF-H ImageNet experiment.
  • Can you extend the qualitative comparison to RetrievalRandom and SameClassRandom, and direct IR using the same noise inputs and class conditions, and add independent preference evaluation or user study if possible?
Reviewer_3JPLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Reinforcement Continual Learning (RCL), a post-training framework that converts evaluation feedback into image-target supervision for a one-step generator. For each generated image, the evaluated implementation retrieves eight same-class ImageNet training images using Inception features, filters them by a source-relative ImageReward or Qwen score margin, and selects the nearest eligible target in DINOv2 space. The generator then matches this target at the original noise and class input using a composite perceptual and pixel loss. Targets are refreshed as the generator changes. Experiments adapt a pMF-H generator for 600 steps and compare reward-guided selection with random target policies and direct ImageReward maximization. The reported trajectories show improved evaluator scores and an initial reduction, followed by a partial rebound, in multi-representation distribution distance.

Critical strengths
  1. RCL provides a simple route for using black-box feedback while preserving one-step inference. Evaluation selects a target, and a separate matching objective supplies the gradient. The implementation accommodates ImageReward and Qwen without backpropagating through either evaluator. This is a practical design for deterministic generators whose inference interface should remain unchanged.
  2. The controlled source inputs and joint reward/distribution trajectories provide useful preliminary evidence. The explicit reporting of later distribution degradation and single-run limitations makes the scope of these findings reasonably clear.
Critical weaknesses
  1. RetrievalRandom simultaneously removes preference filtering and replaces nearest-target selection with random selection. The crucial nearest-target baseline without a reward filter, and random selection among reward-eligible targets, are absent. There is also no comparison against targets constructed once from the initial generator, so the benefit of online renewal is not measured.
  2. The evaluation is too limited to establish reliable improvement in generative quality and coverage. All results use one generator, one dataset, and one training run per configuration. The empirical Pareto frontiers therefore have no evidence of robustness to training randomness.
  3. The paper also provides no table of exact per-method reward and FDr6 results at matched, best, or final checkpoints, readers must estimate key comparisons from the plots.
Questions for authors
  1. With the candidate pool and matching loss fixed, how do nearest-target selection without reward filtering, random selection among eligible targets, and targets frozen from initialization compare with RCL? Please report acceptance rates over training and test whether the conclusion persists when the fraction of contributing samples is controlled.
  2. Does RCL retain an advantage over a tuned direct-reward baseline with comparable real-data or distribution-preserving supervision, or an adapted close alternative such as DrPO? A comparison that separates the source of supervision from the update mechanism would materially strengthen the contribution.
  3. Can you report per-representation distances, precision/recall, within-class diversity, and nearest-training-image examples at the initial, minimum-FDr6, and final checkpoints? How concentrated is selected-target usage, and do blinded human preferences support the automatic-score improvements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.