acceptodds
Under review as a conference paper at ICLR 2027

RubricEvo: Escaping Uniform Rewards in Rubric-based Text-to-Image Reinforcement Learning via Online Rubric Evolution

Abstract

Rubric-based rewards have recently gained attention in text-to-image RL, as decomposing prompt-following into atomic, verifiable criteria enables comprehensive verification of complex compositional semantics. However, existing rubric-based methods for text-to-image RL construct their rubric sets before training and keep them fixed throughout training. The fixed rubrics are never checked against the images that the current policy generates, so their difficulty can mismatch the ability of the policy. Specifically, many rubrics are uniform: they are satisfied or failed by nearly all rollouts. A uniform rubric assigns nearly the same score to every image in a rollout group. Rewards then carry little difference across samples, leaving the policy with almost no guidance on which samples to reinforce. To address this, we propose , an online rubric evolution framework that revises rubrics at each training step based on feedback from the policy's current rollouts, keeping the rubrics discriminative throughout training. The feedback takes two forms. The pass rate of each rubric identifies the uniform ones and tells the direction to adjust their difficulty. The visual differences among rollout images expose distinctions that the existing rubrics do not cover, from which new rubrics are constructed. Since the two forms of feedback are heterogeneous, handling them in a single MLLM call risks letting the rubric text dominate the visual evidence. RubricEvo therefore processes them with a two-stream strategy, yielding rubrics both faithful to the prompt and grounded in the actual differences among rollouts. The rollouts are then re-scored under the updated rubrics to restore discriminative rewards. Extensive experiments on widely adopted text-to-image benchmarks show that RubricEvo consistently outperforms state-of-the-art scalar-based and rubric-based methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
text-to-image generation, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sbPZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RubricEvo is an online framework for text-to-image RL that, at every training step, rewrites rubrics that all rollouts pass or all fail (using pass rates plus visual differences among the current images, via a two-stream design) so rewards stay discriminative, and it outperforms scalar-reward and static-rubric baselines.

Critical strengths

This paper proposes RubricEvo, an online rubric evolution framework that addresses the problem of static rubrics becoming uniform during training, which removes the discriminative power of group-relative advantages. Its strengths are that the idea is simple and clear and can be added on top of existing rubric methods without changing the RL algorithm or training an extra reward model. Through diagnosis, two-stream evolution and re-scoring, it turns otherwise uninformative rollouts into useful supervision rather than discarding them as rejection sampling and CGPO do. Ablations support the design choices, showing that online evolution beats offline evolution and that the two-stream design beats a single coupled call, and it improves results on both SD-3.5-Medium and FLUX.

Critical weaknesses

The paper’s central premise is that static rubrics become uniform and starve GRPO of learning signal, but it never reports how often this happens, giving neither the share of rubrics rewritten per prompt nor the share of zero-variance groups. The comparison to baselines is also not compute-matched: RubricEvo adds extra rubrics and several extra MLLM calls at every step, so the gains may come from additional judging rather than from evolution itself, the evidence is thin as well. Finally, the same 8B model writes and grades the new rubrics, and reward validity rests on one limited audit (100 groups, one external judge, agreement below 50%). It is therefore unclear whether the improvement reflects better supervision or simply a noisier, more heavily judged reward.

Questions for authors
  1. How often do static rubrics actually become uniform, and what share of rubrics is rewritten per prompt and over training?
  2. Do the single-run gains on TIIF-Bench and DPG-Bench, hold across multiple random seeds?
  3. Does the improvement come from evolution itself or simply from extra judging, when compared with compute-matched static baselines?
  4. How many additional MLLM calls and how much extra training time does RubricEvo add per step compared with each baseline?
  5. How do the results change with a different judge or evolver model, and how well do the evolved rubrics agree with human judgments?
Reviewer_DJLhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RubricEvo, a method that dynamically adapts reward rubrics during text-to-image RL, rather than using fixed rewards. So the reward function continues to distinguish better images from worse ones as the generator improves. It adopts a two-stream strategy for rubric evolution: a text stream refines existing criteria, and an image stream discovers new criteria based on visual differences among generated images. The evolved rubrics are used to re-score the current rollouts and provide more informative learning signals for GRPO. Experiments demonstrate consistent improvements over existing scalar-based and static rubric-based reward methods.

Critical strengths
  1. The paper identifies a specific limitation of static rubric-based rewards, which is a well-motivated problem formulation.
  2. The design of rubric evolution is novel.
Critical weaknesses
  1. Although in general we want reward variance in RL, higher reward variance does not guarantee better or more reliable rewards. The method could create more differences among images by asking irrelevant or noisy questions. The paper deals with this using Gemini-based pairwise audit, but it covers only 100 sampled prompt groups. It is unclear how reliably the method prevents irrelevant visual differences from becoming reward criteria.
  2. The temporary rubric set may introduce reward inconsistency across training steps. The effective reward criteria can change from one training step to another. It's unclear whether this variation introduces instability or makes the optimization objective less consistent.
Questions for authors
  1. The image stream does not see the prompt. How many evolved rubrics are still relevant to the prompt?
  2. How diverse is the evoluted rubrics with different groups?
Reviewer_3778OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RubricEvo to address uniform rewards in rubric-based text-to-image reinforcement learning. At each training step, the method identifies criteria that fail to distinguish among generated images, adjusts their difficulty using a text stream, and introduces new criteria based on visual differences using an image stream. It then re-scores the current images with the updated criteria. The paper reports improvements over a static-rubric baseline on UniGenBench++ using SD3.5-Medium and FLUX.1-dev, and evaluates the method on TIIF-Bench and DPG-Bench.

Critical strengths
  1. The paper addresses a concrete limitation of fixed binary rubrics with a practical online method. The diagnosis, evolution, and re-scoring procedure is clearly specified, and the appendix provides the prompts used for rubric evolution. This makes the approach straightforward to examine and potentially apply to existing rubric-based training pipelines.

  2. The paper demonstrates improvements over a static-rubric baseline across two base models. Impact and evidence: On UniGenBench++ short, RubricEvo improves the overall score over RubricRL from 67.2 to 69.6 with SD3.5-Medium and from 69.21 to 73.77 with FLUX.1-dev. The ablations in Figure 3 also suggest that online evolution and both streams contribute to performance, making the approach worth further investigation.

Critical weaknesses
  1. The paper does not sufficiently establish that more discriminative rubrics provide more valid rewards. The image stream generates criteria without seeing the text prompt, while the merging step does not clearly exclude distinct but prompt-irrelevant criteria. Figures 4 and 6 measure variation in individual rubric outcomes, which does not necessarily imply variation in the final group reward or better alignment with the prompt.

  2. The experiments do not sufficiently isolate the benefit of online evolution or establish the stability of smaller gains. Figure 3 provides a useful online-versus-offline comparison. However, the image stream also adds 3–5 criteria, and the method recalculates rubric weights when the number of criteria changes. The stream ablations therefore do not fully separate the benefit of the two-stream design from the effects of rubric count and additional model calls. The paper also lacks results under matched computational budgets and does not report variation across independent training runs. This matters particularly for the small overall margins on TIIF-Bench long (74.6 versus 74.0) and DPG-Bench (87.0 versus 86.6).

Questions for authors
  1. Please report whether evolved rubrics remain relevant to the text prompt and whether the final group rewards better match blinded human judgments, including on rollouts not used to generate the rubrics. These results would directly affect my assessment of the proposed reward mechanism.
  2. Please provide comparisons matched for rubric count and computational budget, along with results across independent training seeds and uncertainty estimates. These are particularly important for the small overall gains on TIIF-Bench long and DPG-Bench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.