acceptodds
Under review as a conference paper at ICLR 2027

PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning

Abstract

Reinforcement learning can improve prompt following in text-to-image (T2I) models,but obtaining high-quality reward signals remains a key challenge. Existing rewards either provide only coarse-grained global alignment signals or rely on costly human preference annotations and additional reward-model training. We introduce PromptEcho, an annotation-free reward that uses a frozen vision-language model (VLM) to compute the image-conditioned, teacher-forced next-token likelihood of the original prompt. This directly converts the VLM’s pretraining objective into a continuous token-level reward without training a separate reward model. For evaluation, we develop DenseAlignBench, a benchmark of concept-rich dense captions designed to rigorously test fine-grained prompt following. Relative to the corresponding original baselines, PromptEcho achieves net pairwise advantages of +26.8pp and +16.3pp on DenseAlignBench for Z-Image and QwenImage-2512, respectively, and yields consistent improvements on public benchmarks including GenEval, DPG-Bench, and TIIFBench without any task-specific training. Under the same Z-Image RL configuration, PromptEcho also substantially outperforms CLIP score, ImageReward-v1, and HPSv3, with a particularly pronounced advantage on long, dense prompts. Ablations with VLMs of different sizes further show that PromptEcho can directly benefit from advances in open-source VLMs. Code, checkpoints, and DenseAlignBench will be released through an anonymous project page.

Primary area:
generative models
Keywords:
text-to-image generation, reinforcement learning, reward modeling, vision-language models, annotation-free reward, prompt following, token-level reward, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_jZTjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PromptEcho, it uses a frozen VLM (Qwen3-VL-32B) as a reward signal for text-to-image RL. The reward is the mean teacher-forced log-likelihood of the original prompt conditioned on the generated image and a fixed query. The results show some improvement over a few baselines under the same RL setup.

Critical strengths
  1. The idea of this paper is interesting and the proposed method is straight forward. The reward requires no annotation, no reward-model training, and less than 5% overhead during training.
  2. Results are good. e.g., The 8B vs. 32B ablation shows that scaling the VLM helps; pairwise comparisons backed by human validation.
Critical weaknesses
  1. My main concern is that the baselines are weak in a way that makes it hard to tell where the gains actually come from. CLIP (77 tokens) and ImageReward (35 tokens) physically cannot encode the ~200-word DenseAlignBench prompts, so they are guaranteed to underperform on this benchmark regardless of reward design. InferScore uses discrete 1-5 ratings and often yields zero advantage within a group. I believe this paper could be benefited by adding the following: 1.1. discuss and compare with VQAScore (Lin et al., ECCV 2024), which is also a zero-shot, continuous VLM log-likelihood score and is the most direct point of comparison; 1.2. new baseline: a caption-then-compare reward, where the same VLM generates a caption and it is compared against the prompt as in CycleReward (ICCV 2025); learned VLM rewards like UnifiedReward or VisionReward.

  2. The score is essentially similar to VisualGPTScore's log⁡P(text∣image)\log P(\text{text} \mid \text{image}) (Lin et al., ICML 2024). Using text-image-text consistency as an annotation-free reward was studied in CycleReward, and the actual contribution is applying this likelihood as an online RL reward for T2I fine-tuning, which is a valid contribution but should be stated as such.

Questions for authors

Please refer to my weakness.

3 comments

Thank you for your detailed and constructive review. Following your suggestions, we have added further experiments and analyses, which have been very helpful in improving the quality of this paper.

1. Additional reward baselines

Following your suggestion, we have added comparisons with VQAScore, CycleReward-style Caption-cycle, and UnifiedReward. The prompt-following component of VisionReward uses VQAScore, so we do not repeat this comparison here; these methods, as well as HPSv3, which we have already compared in this paper, are not subject to the aforementioned caption truncation limitation.

For InferScore, we have added two further variants: changing the score range to 0–100, and computing the expected score from the output probabilities of scores 1–5, thereby addressing the overly coarse granularity of the original 1–5 scoring scheme. The relevant results are shown in the table below, which also includes the diverse guiding-query training metrics from the paper.

MethodGenEval ↑DPG ↑TIIF Short / Long ↑DenseAlignBench Win Rate vs. Baseline ↑DenseAlignBench Net Advantage ↑
Z-Image baseline0.752186.90440.8491 / 0.8316——
PromptEcho0.817587.92200.8850 / 0.889461.50%+26.80 pp
PromptEcho (diverse guiding queries)0.83389.230.8983 / 0.892864.70%+34.90 pp
VQAScore0.749887.07190.8329 / 0.843739.43%−18.00 pp
InferScore 0–1000.759486.33380.8384 / 0.853040.60%−15.43 pp
UnifiedReward ACS0.748685.94820.8480 / 0.851136.95%−21.12 pp
Caption-cycle (CycleReward-style)0.777087.13230.8424 / 0.866345.37%−1.26 pp
InferScore 1–5 expected score0.764786.94890.8752 / 0.872942.88%−6.88 pp

TIIF scores in the table are consistently reported on a 0–1 scale; the diverse guiding-query results are from Table 6 of the paper.

The results show that PromptEcho achieves better performance on both the public benchmarks and DenseAlignBench, especially in the long dense-caption setting of DenseAlignBench. Diverse guiding-query training further improves overall performance. We hope these experiments address your concern that the baselines are too weak.

2. The relationship between VisualGPTScore, CycleReward, and our work

We agree with your description of the relationship between our work and VisualGPTScore and CycleReward, and will add citations to these two works in the revised version and revise the related work section to accurately explain their relationship to our work.

At the same time, we would like to explain, from a theoretical perspective, the potential advantages of PromptEcho over the CycleReward-style procedure of “regenerating a caption and then comparing”:

  1. Semantic consistency of the reconstructed caption. The caption regeneration process is stochastic, and especially for long dense captions, there is no guarantee that the regenerated caption is fully consistent in meaning with the original caption used for image generation.
  2. Computational efficiency. Caption regeneration requires autoregressive decoding and is computationally time-consuming; PromptEcho uses teacher forcing and does not require caption regeneration, giving it a significant advantage in computational speed.
  3. Preservation of information in long captions. Text comparison requires first compressing the caption into an embedding and then computing cosine similarity, which may lose information in the caption, especially the details in long dense captions. PromptEcho directly reads the probabilities of the individual tokens in the original caption as the reward.

The actual RL results also show that PromptEcho provides a higher-quality reward signal.

Dear Reviewer jZTj,

Thank you again for your time and effort in reviewing our paper. With only two days remaining in the discussion period, we would like to kindly ask whether you have any further suggestions or concerns. We believe that our previous responses and additional experiments have adequately addressed your concerns. Should you have any further questions, we would greatly appreciate the opportunity to address them promptly.

Best regards, The Authors

Dear Reviewer jZTj,

Thank you again for your time and effort in reviewing our paper. As the discussion period is coming to a close, we would be grateful if you could take a look at our rebuttal and additional experimental results and share any remaining questions or concerns.

Your feedback would be very valuable to us, and we would sincerely appreciate the opportunity to address any outstanding issues promptly before the discussion ends.

Thank you for your consideration.

Best regards, The Authors

Reviewer_7eLEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an annotation-free reward for rl post-training of t2i generation models. Instead of relying on human preference annotations or training a reward model, the paper uses a frozen VLM to evaluate the teacher-forced next-token log-likelihood of the original prompt conditioned on the generated image. The average token log-likelihood is used as a continuous reward for optimization. Experiments show improvements in prompt following on the proposed DenseAlignBench and three public benchmarks (GenEval, DPG-Bench, and TIIFBench). The authors also compare with several existing reward models and conduct ablations.

Critical strengths
  1. Simple and practical reward formulation. The paper provides an elegant way to construct a continuous reward directly from a VLM's conditional token probabilities, without human preference annotations or other reward model. The approach is easy to implement and can potentially benefit from future improvements in pretrained VLMs. The simplicity and applicability are meaningful strengths.

  2. Encouraging experimental results across multiple models and benchmarks. The paper evaluates on two t2i models and report improvements on both benchmarks, additional human-eval and repeated VLM-based evaluations also strengthen the empirical evidence.

Critical weaknesses
  1. The reward comparisons do not sufficiently isolate the benefit of the proposed method. Although Tab.7 compares PromptEcho with others, the authors acknowledge that clip score and ImageReward-v1 have restrictive text-length limits (clip truncates to 77 tokens and ImageReward 35 bert tokens, at about line 373-375). Given that DenseAlignBench contains long captions averaging 200 words, these comparisons may inherently favor PromptEcho. Tab.5 compares with InferScore using the same VLM, InferScore produces discrete scores, whereas PromptEcho provides continuous likelihood-based rewards. I think the performance gap may partly reflect differences in reward granularity, rather than the specific benefit of teacher-forced likelihood scoring.

  2. Insufficient analysis of the proposed reward's fine-grained semantic sensitivity and dependence on the caption construction process. The paper emphasizes that PromptEcho provides a fine-grained, token-level reward. However, the token log-likelihoods are averaged into a sequence-level reward for rl. It remains unclear how effectively this aggregated signal distinguishes subtle semantic errors in complex prompts. App.E2 shows a controlled count-and-color experiment, which is good, but i think captions are too short and simple. More analysis involving longer and complex prompts would better support the fine-grained alignment claim. Additionally, both the training captions and the reward are constructed using Qwen3-VL-32B variants. This raises the question of whether the observed effectiveness partly depends on the compatibility between the captioning model's language distribution and the reward VLM. The current experiments do not isolate this factor.

Questions for authors

See Weaknesses

5 comments

Thank you for your detailed and constructive review. Following your suggestions, we have added further experiments and analyses, which have been very helpful in improving the quality of this paper.

1. Reward baselines and score granularity

First, we would like to emphasize that, for state-of-the-art text-to-image models such as Z-Image and QwenImage, there is still a lack of available open-source reward models that can provide high-quality feedback on prompt following, especially when the inputs are long dense captions. High-quality in-house reward models typically require expensive data annotation and dedicated reward model training, and our work fills this gap.

To address the caption truncation issue you raised, and in conjunction with the suggestions from other reviewers, we have added comparisons with VQAScore and UnifiedReward. These two methods, as well as HPSv3, which we have already compared in this paper, do not have the aforementioned caption truncation issue. To address the output granularity of InferScore, following Reviewer g1jq's suggestion, we have added two variants: changing the score range to 0–100, and computing the expected score from the output probabilities of scores 1–5. The complete results are shown in the table below.

MethodGenEval ↑DPG ↑TIIF Short / Long ↑DenseAlignBench Win Rate vs. Baseline ↑DenseAlignBench Net Advantage ↑
Z-Image baseline0.752186.90440.8491 / 0.8316——
PromptEcho0.817587.92200.8850 / 0.889461.50%+26.80 pp
PromptEcho (diverse guiding queries)0.83389.230.8983 / 0.892864.70%+34.90 pp
VQAScore0.749887.07190.8329 / 0.843739.43%−18.00 pp
InferScore 0–1000.759486.33380.8384 / 0.853040.60%−15.43 pp
UnifiedReward ACS0.748685.94820.8480 / 0.851136.95%−21.12 pp
Caption-cycle (CycleReward-style)0.777087.13230.8424 / 0.866345.37%−1.26 pp
InferScore 1–5 expected score0.764786.94890.8752 / 0.872942.88%−6.88 pp

TIIF scores in the table are consistently reported on a 0–1 scale; the diverse guiding-query results are from Table 6 of the paper. The results show that PromptEcho achieves better performance on both the public benchmarks and DenseAlignBench, especially in the long dense-caption setting of DenseAlignBench. The two enhanced InferScore variants also perform significantly worse than PromptEcho on these benchmarks. We would like to point out that HPSv3 and UnifiedReward require additional human annotation and reward model training, whereas PromptEcho does not.

2. Fine-grained semantic sensitivity in complex long captions

Following your suggestion, we constructed more complex caption–image pairs. Specifically, we constructed 32 images containing 12 apples, arranged in a 3×4 grid, with four red, four green, and four yellow apples, and randomized color positions. To increase the complexity of the prompt and introduce additional distractions, we added descriptions of the environment, background, and other details, extending the prompt to 191 tokens, including 12 color tokens corresponding to the colors of the 12 apples. This complex layout is challenging for text-to-image models, so we constructed correctly matched captions and images through multiple generation retries and further image editing.

We then sampled four caption perturbation chains for each image. Starting from the correctly paired caption, we progressively changed the apple color words to incorrect ones. At each step, we retained the existing changes and changed one additional color word corresponding to an apple, up to eight incorrect color words, while keeping all other text unchanged. After deduplication, we obtained 1054 caption–image pairs, including 32 fully correct pairs and 1022 color-mismatched pairs. We computed the PromptEcho loss for these caption–image pairs, with the results shown below.

First, for all 32 images, the correct caption had the lowest PromptEcho loss, and the rewards of all 1022 mismatched captions were lower than those of the correct captions for the same images. As the number of incorrect color words increased, the average PromptEcho loss increased overall. The reward decrease in the table below equals the CE increase of the mismatched caption relative to the fully correct caption.

Number of Incorrect Colors kCaption–Image PairsAverage Reward ↑Reward Decrease Relative to the Fully Correct Caption
0 (fully correct)32−2.185820.00000
1128−2.267300.08149
2128−2.311070.12525
3128−2.326450.14063
4128−2.337150.15133
5128−2.343880.15806
6128−2.338000.15218
7128−2.345030.15922
8128−2.345000.15918

We also measured the proportion of the net increase in the full-caption CE contributed by the incorrect color tokens:

Number of Incorrect Colors kProportion of the Net Full-Caption CE Increase Contributed by Incorrect Color Tokens
1103.1%
2104.3%
3102.1%
4100.1%
597.1%
696.1%
794.9%
894.4%

As can be seen, in captions with a total length of 191 tokens, almost the entire net increase in NTP loss comes from these 1–8 incorrect color tokens. Some proportions exceed 100% because the loss at other tokens decreases slightly, offsetting part of the increase caused by the incorrect words. We believe these results further support our conclusion: even in complex long captions, PromptEcho can still provide fine-grained, token-level reward signals.

3. Distributional compatibility between the caption construction model and the reward VLM

We would like to point out that the caption sources of the test benchmarks GenEval, DPG-Bench, and TIIFBench are all independent of the Qwen3-VL-32B used in this paper, and PromptEcho also achieves significant improvements on these benchmarks. For example, without training specifically for GenEval, diverse guiding-query training improves Z-Image's GenEval score from 0.7521 to 0.833, which is a substantial improvement.

Using Qwen3-VL-32B mainly affects the input caption distribution during RL training. We were aware of this and, in Section 4.2.3 of the paper, replaced the original single-query setting, “Describe this image in detail.”, with diverse guiding queries to enrich the training caption distribution. This further improves performance on DenseAlignBench and the public benchmarks. We hope these results address your concern that the PromptEcho method depends on the output distribution of Qwen3-VL-32B.

Here, we provide the prompt template used in the preceding analysis.

A realistic overhead photograph shows exactly twelve whole apples on a light wooden table. The apples are arranged in three horizontal rows of four, with the first row at the top of the image and the third row at the bottom. From left to right, the first row contains a {row1_col1_color} apple, a {row1_col2_color} apple, a {row1_col3_color} apple, and a {row1_col4_color} apple. The second row contains a {row2_col1_color} apple, a {row2_col2_color} apple, a {row2_col3_color} apple, and a {row2_col4_color} apple. The third row contains a {row3_col1_color} apple, a {row3_col2_color} apple, a {row3_col3_color} apple, and a {row3_col4_color} apple. Each apple occupies a separate position with clear gaps between neighboring apples. Every apple is fully visible, without cropping, overlap, containers, labels, or additional fruit. The visible skin of each apple has one clearly dominant color, without mixed red and green patches or strong reflections obscuring its appearance. Soft daylight produces natural shadows, while the viewing angle, wooden surface, composition, and surrounding space remain consistent.

Comments on the paper

5 comments

Thank you for your detailed and constructive review. Following your suggestions, we have added further experiments and analyses, which have been very helpful in improving the quality of this paper.

1. Novelty and the relationship to SpectraReward and VQAScore

We would first like to point out that the core methods of SpectraReward and PromptEcho are almost identical, but the first version of PromptEcho was submitted to arXiv on April 14, 2026, whereas the first version of SpectraReward was submitted on July 13, 2026, approximately three months later than our paper. The first version of SpectraReward also explicitly cited our work. In the interest of rigor, we cited SpectraReward in the version submitted to ICLR 2027.

VQAScore primarily asks a VLM a question and uses the probability of the affirmative answer “Yes” as the reward, which is clearly different from our method of directly computing the token likelihood of the original caption. We have added a comparison with VQAScore and will include the corresponding citation and comparison in the revised version.

2. Reward baselines and InferScore's score granularity

For state-of-the-art text-to-image models such as Z-Image and QwenImage, there is still a lack of available open-source reward models that can provide high-quality feedback on prompt following, especially when the inputs are long dense captions. High-quality in-house reward models require expensive data annotation and dedicated model training, and our work fills this gap.

To address the caption truncation issue, and in conjunction with the suggestions from other reviewers, we have added comparisons with VQAScore and UnifiedReward. These two methods, as well as HPSv3, which we have already compared in this paper, do not have the aforementioned caption truncation issue. The prompt-following component of VisionReward uses VQAScore, so we do not repeat this comparison here; PickScore also has a caption truncation issue, so we do not include it in this round of comparisons.

Following your suggestion, we have also added two InferScore variants: changing the score range to 0–100, and computing the expected score from the output probabilities of scores 1–5. The complete comparison results are shown below.

MethodGenEval ↑DPG ↑TIIF Short / Long ↑DenseAlignBench Win Rate vs. Baseline ↑DenseAlignBench Net Advantage ↑
Z-Image baseline0.752186.90440.8491 / 0.8316——
PromptEcho0.817587.92200.8850 / 0.889461.50%+26.80 pp
PromptEcho (diverse guiding queries)0.83389.230.8983 / 0.892864.70%+34.90 pp
VQAScore0.749887.07190.8329 / 0.843739.43%−18.00 pp
InferScore 0–1000.759486.33380.8384 / 0.853040.60%−15.43 pp
UnifiedReward ACS0.748685.94820.8480 / 0.851136.95%−21.12 pp
Caption-cycle (CycleReward-style)0.777087.13230.8424 / 0.866345.37%−1.26 pp
InferScore 1–5 expected score0.764786.94890.8752 / 0.872942.88%−6.88 pp

TIIF scores in the table are consistently reported on a 0–1 scale; the diverse guiding-query results are from Table 6 of the paper. The results show that PromptEcho achieves better performance on both the public benchmarks and DenseAlignBench, especially in the long dense-caption setting of DenseAlignBench. The two InferScore variants also perform significantly worse than PromptEcho. We would like to point out that HPSv3, UnifiedReward, ImageReward, and PickScore all require additional human annotation and reward model training, whereas PromptEcho does not.

3. The distribution of DenseAlignBench and generalization on public benchmarks

The metric definitions of DenseAlignBench differ from those of public benchmarks such as GenEval: the former uses pairwise comparisons and reports win rates and net advantages, whereas the latter uses absolute scores. The percentages in these two types of metrics have different meanings. For example, a 5% improvement in an absolute score may correspond to a larger win-rate advantage in pairwise comparisons. Therefore, we cannot simply conclude, based solely on the magnitude of the percentage values, that PromptEcho's improvement on DenseAlignBench is significantly larger than its improvement on out-of-distribution public benchmarks such as GenEval.

Using the single guiding query “Describe this image in detail.” and Qwen3-VL-32B in the main experiments does affect the input caption distribution during RL training. Therefore, in Section 4.2.3, we introduced diverse guiding queries to enrich the training caption distribution. This further improves performance on both DenseAlignBench and GenEval, DPG-Bench, and TIIFBench.

Specifically, diverse guiding-query training improves Z-Image's GenEval score from 0.7521 to 0.833, DPG from 86.9044 to 89.23, and TIIF Short/Long scores from 0.8491/0.8316 to 0.8983/0.8928, respectively. Given that we did not train specifically for these public benchmarks, these improvements are substantial.

Trained models typically perform better within the training distribution. This is an issue of training data construction and cannot be used to negate the effectiveness of the PromptEcho method itself. This issue can be further mitigated through the diverse guiding queries already used in this paper, or by further mixing captions generated by different VLMs.

4. Degradation in some metrics and the wording “consistent improvements”

We agree with the issue you raised and will use more cautious wording in the revised version. At the same time, overall, PromptEcho achieves broad improvements on DenseAlignBench and multiple public benchmarks, especially with diverse guiding-query training, which further improves overall performance. The relevant metrics are listed in the table above.

5. Language priors, image quality, and diversity

We have already analyzed the concern that average log likelihood may be dominated by language priors in the paper. In algorithms such as AWM and GRPO, advantages are computed within a group; in our setting, the generated images in the same group share the same caption. Therefore, the NTP loss contribution arising solely from the shared language prior is canceled out during the within-group computation. What remains is the difference in NTP loss caused by different images, providing a training signal for prompt following.

Regarding image quality, we sampled images using the Z-Image base model and the model optimized with PromptEcho, respectively, and computed FID relative to the real images in the training set. The results were 26.65 and 26.24, respectively, with no noticeable difference observed.

Regarding diversity, we observed that the variance of the reward itself gradually decreases during training. A reduction in diversity is a common phenomenon in RL learning: we increase the probability of outputs that better match our preferences within the model's output distribution, which naturally reduces diversity, but this is not an issue unique to PromptEcho.

6. Matching diverse queries and VLM scale

In diverse-query training, we deliberately maintain consistency between the caption generation query and the reward computation query at the sample level. In our setting, a mismatch between the two raises the overall NTP loss level, which in turn reduces the discriminative power of the rewards within a group and harms the quality of the reward signal. We therefore use matched queries for training.

Regarding the conclusion about 8B versus 32B, we will use more cautious wording, but the 32B model still outperforms the 8B model on the vast majority of metrics.

Dear Reviewer g1jq,

Thank you again for your time and effort in reviewing our paper. With only two days remaining in the discussion period, we would like to kindly ask whether you have any further suggestions or concerns. We believe that our previous responses and additional experiments have adequately addressed your concerns. Should you have any further questions, we would greatly appreciate the opportunity to address them promptly.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.