acceptodds
Under review as a conference paper at ICLR 2027

Imago: Studying the Online RL Recipe for Visual Generation

Abstract

We introduce Imago, an online reinforcement learning (RL) post-training recipe that matures pretrained text-to-image (T2I) generators into reliable compositional models. While modern T2I generators possess rich visual representations, under compositional constraints they frequently omit entities, bind attributes incorrectly, and violate spatial relations. RL offers a principled framework to resolve this gap, yet prior research focuses primarily on optimization algorithms, leaving the reward signal and the prompt corpus comparatively underexplored. Evaluating candidate rewards on optimization stability and visual fidelity, we find that a zero-shot VQAScore evaluated by a multimodal large language model (MLLM) provides reliable guidance, whereas preference-trained rewards are less reliable, with some triggering reward hacking. Treating the prompt corpus as the exploration environment, we further show that short and long prompts develop complementary spatial and attribute capabilities. While naive corpus mixing degrades performance, a sequential short-to-long curriculum successfully captures the strengths of both distributions. We apply the resulting recipe to FLUX.2-Klein-Base and Z-Image without architectural modifications, producing Imago-4B and Imago-6B. Both models achieve leading performance in their respective size classes across compositional capability and visual appeal, with Imago-6B surpassing substantially larger open-source and proprietary models on medium- and long-prompt composition benchmarks.

Primary area:
generative models
Keywords:
reinforcement learning, text-to-image generation, training recipe

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_91QCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Imago studies which reward signals and prompt distributions make online RL effective for compositional text-to-image generation. Using DiffusionNFT with LoRA, the study compares rewards, scoring formats, reward-model sizes, and controlled prompt corpora on two generator backbones. The resulting recipe combines graded MLLM feedback with a short-to-long curriculum. Both final models improve across five compositional benchmarks, with Imago-6B achieving the highest reported medium and long-prompt scores among the evaluated systems.

Critical strengths
  1. The controlled reward comparisons offer useful guidance for visual RL. With the algorithm and corpus fixed, graded MLLM feedback improves composition across both backbones (Table 1), and compact reward models remain effective.

  2. The final recipe improves both generators across all five compositional benchmarks. Imago-6B raises T2I-CoReBench from roughly 85 to 90 and achieves the highest reported medium- and long-prompt scores among the evaluated systems, supporting the recipe's practical value.

Critical weaknesses
  1. Reward reliability needs stronger independent validation. The selected training reward and all compositional judges use Qwen; the Gemma control checks correlations on the original models, rather than evaluating RL-optimized outputs. Appendix D.3 also acknowledges that PickScore remains a proxy. These checks support the reported metric gains, but do not establish resistance to reward hacking or preservation of perceptual quality.

  2. The curriculum ablation matches update budgets but confounds training order with resetting optimizer states, EMA weights, and reward statistics (Appendix C.2). Joint and single-corpus baselines receive no equivalent reset. The gains therefore cannot be attributed specifically to staged capability acquisition, a central claim. Comparisons with matched resets and a sequential run without resets are needed to isolate the curriculum's effect.

Questions for authors
  1. Do the relative gains of VQAScore-trained models persist when their optimized outputs are assessed by a judge from another model family or by blinded human evaluation? Evidence on these outputs would directly address my main concern.

  2. Does the sequential advantage persist without resetting training states, and how do joint or single-corpus runs perform with an equivalent midpoint reset? If available, a long-to-short comparison would further clarify whether the proposed order is essential.

Reviewer_mixrOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an online RL post-training recipe, which transforms pretrained T2I generation models into reliable compositional generation models. Without any architectural modifications, it addresses problems in existing T2I models, such as missing objects, incorrect attribute binding, and inaccurate spatial relationships, solely by adjusting reward signals and training data.

Critical strengths
  1. Easy to understand and reproduce. It requires no architectural modifications, relying only on the selection of appropriate reward signals and training data.
  2. The paper conducts extensive correlation analyses using Pearson correlation coefficients and summarizes several representative findings.
  3. This paper provides a valuable insight by exploring the role of training data, ultimately proposing a sequential training strategy.
Critical weaknesses
  1. Limited novelty and unclear contributions. The paper contributes a balanced dataset, Imago-Long, and a sequential training strategy, but it does not seem to propose any new algorithms. The reward model used, VQAScore, is also an existing method.
  2. The Abstract and Introduction claim that the paper mainly addresses three problems: object omission, incorrect attribute binding, and violations of spatial relationships. However, the proposed method only uses VQAScore to generally improve image-text alignment, without introducing targeted strategies to address these three specific problems.
  3. Concerns about the reliability of the evaluation. What are the evaluation criteria used by Qwen3.5-27B as a judge in Section 3.1? VLM-based evaluators are generally not very accurate, especially when assigning specific numerical scores on a scale of 1 to 10. It is difficult for them to achieve the same level of subjective judgment as humans [1][2].

[1] Chen D, Chen R, Zhang S, et al. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark[J]. arXiv preprint arXiv:2402.04788, 2024.

[2] Hwang Y, Lee D, Min K, et al. Fooling the LVLM judges: Visual biases in LVLM-based evaluation[C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025: 23197-23216.

  1. In addition, why do the scores appear to be higher on the Long benchmark than on the Medium benchmark? In my view, this further raises concerns about the accuracy of the evaluation. In particular, based on the system prompt shown in Figure 12 of Appendix D, it seems difficult for the model to assign accurate scores. The prompt does not provide the MLLM with specific examples illustrating what level of image-text alignment corresponds to a particular score. With such a general system prompt, it is difficult for the model to assign precise scores. Alternatively, the authors should provide some comparative examples by presenting several image-text pairs and their corresponding MLLM scores to examine whether these scores are consistent with human subjective judgments.
  2. The paper mainly reports empirical findings through experiments, without sufficient theoretical justification. It computes various Pearson correlation coefficients but does not even provide the corresponding formula.
Questions for authors
  1. Regarding the "complete prompt-image alignment" mentioned in Lines 207–209, is there any ground truth available for validation? Does this require manually annotated ground truth? Otherwise, how can complete alignment be measured?
  2. Figure 11, mentioned in Lines 255–256, does not clearly demonstrate the advantages of VQAScore.
Reviewer_K5xxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Imago, an online RL post-training recipe designed to improve the compositional capabilities of text-to-image models. Using DiffusionNFT as a fixed optimization framework, the authors systematically explore different reward signals and prompt corpus designs. Their final recipe uses pretrained Qwen3.5-9B to score image–text alignment on a 1–10 scale, builds a balanced prompt corpus covering entities, attributes, and relations, and adopts a sequential short-to-long prompt curriculum. The method is applied to FLUX.2-Klein-Base-4B and Z-Image-6B through LoRA. It shows improvements across several compositional benchmarks while also boosting PickScore as a proxy for visual preference.

Critical strengths
  1. The research questions are practically relevant, and the resulting recipe is simple and effective. The paper investigates reward signals and training corpora in visual RL, providing empirical guidance on important design choices beyond optimization algorithms. These findings are useful for practitioners building RL systems for image generation.

  2. The reward analysis is systematic. The authors examine both the correlation between reward scores and downstream performance of models trained with these rewards. They further investigate scoring formats, MLLM families, and model scales. Experiments on two generation backbones strengthen the findings and demonstrate the practical potential of smaller pretrained MLLMs as reward models.

  3. The corpus experiments reveal complementary effects of prompt distributions and identify an effective strategy for combining them. Balanced expansion across entities, attributes, and relations achieves the most consistent overall performance among the four prompt designs. Under the same total number of updates, sequential short-to-long training performs well on both backbones.

Critical weaknesses
  1. The independence of the visual quality evaluation is insufficiently established. The authors select PickScore based on its lower average correlation with other preference evaluators, but lower correlation alone does not establish greater evaluation reliability.

  2. The analysis does not directly include VQAScore, the reward ultimately used for training. Please report the correlation between PickScore and VQAScore on generated samples, together with a small-scale human evaluation.

  3. A comparison with generative rewards based on verification-token likelihoods is missing. Including such baselines would help assess the advantages of the graded numerical scoring used in the paper’s VQAScore formulation.

Questions for authors
  1. Could you report the correlation between PickScore and VQAScore, on generated samples?

  2. How does the proposed 1–10 graded scoring compare with continuous rewards based on verification-token probabilities, such as P(Yes)P(\mathrm{Yes}) or P(Yes)/[P(Yes)+P(No)]P(\mathrm{Yes})/[P(\mathrm{Yes})+P(\mathrm{No})]?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.