acceptodds
Under review as a conference paper at ICLR 2027

Explore Widely, Exploit Wisely: Harnessing Text–Image Cost Asymmetry for Unified Multimodal Reinforcement Learning

Abstract

Prompt Expansion (PE) enhances text-to-image generation by transforming user prompts into explicit, visually grounded descriptions, but jointly optimizing prompt rewriting and image generation remains computationally expensive when large candidate pools are explored. We observe a pronounced cost asymmetry between text and image generation: expanding the number of PE candidates incurs relatively modest decoding overhead, whereas image generation scales approximately linearly with the number of evaluated candidates. Motivated by this asymmetry, we propose **PEAR (Prompt Exploration for Adaptive Joint Reinforcement Learning)**, a framework that enlarges the textual exploration space while selectively allocating image-side evaluation budget. PEAR first oversamples PE candidates for each prompt and estimates their relative rewards using low-cost proxy image generation with reduced precision and fewer denoising steps. It then measures the utility of each prompt by the reward separation between selected top-\(k\) and bottom-\(k\) candidates, and uses bootstrap-based expected utility gain together with a variance-based noise discount to dynamically allocate the remaining evaluation budget across prompts. After the proxy budget is exhausted, the selected high- and low-reward PE candidates are regenerated under the full generation configuration and used to jointly optimize the text and image policies. Experiments show that PEAR evaluates only 62.5% of the full candidate pool, reducing proxy screening time by 35.1% and end-to-end training-step time by 21.1% compared with exhaustive PE screening. Meanwhile, it improves the average score across WISE, HPSv2, GenEvalv2 VQAScore, and PickScore from 0.60145 to 0.60985, and outperforms a static allocation baseline under the same proxy evaluation budget. These results demonstrate that adaptive candidate evaluation can substantially reduce PE screening cost while preserving, and potentially improving, generation quality.

Primary area:
generative models
Keywords:
rl, diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_SPELOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose PEAR, oversampling PEs and evaluates an initial subset using an image proxy, averaging rewards over multiple images for each candidate. It then allocates a fixed evaluation budget across prompts according to bootstrap-estimated improvements in the mean reward gap between high- and low-reward candidates. These estimates are updated as new feedback arrives. The proxy feedback guides selection but does not directly enter the training loss. The experimental results are competitive.

Critical strengths
  1. The paper addresses a practical and interesting problem concerning the high computational cost of candidate exploration in unified multimodal reinforcement learning.
  2. The proposed low-cost proxy evaluation strategy combines FP8 quantization with reduced denoising steps, providing a practical solution for reducing the cost of large-scale candidate screening.
Critical weaknesses

The evaluation is limited to a single combination of Qwen3-8B and Stable Diffusion 3.5 Medium, making it difficult to determine whether the proposed cost-asymmetry assumptions and adaptive selection mechanism generalize to different model architectures, model scales, and generation systems.

Questions for authors
  1. The ablation studies on initial candidate pool size, the top-bottom selection ration, and the total evaluation budget are not clear. For instance, why keeping 8 runs for adaptive allocation?
  2. As the evaluation averages results across multiple image-generation seeds, why main performance tables do not report means, standard deviations?
  3. As shown in Table 3, PEAR requires 796.3 seconds per training iteration compared with only 313.6 seconds for UniGRPO, indicating that the proposed method increases the overall training time by approximately 154%.
  4. The bootstrap-based marginal-gain estimator relies entirely on observed rewards and cannot extrapolate beyond their empirical range, potentially underestimating the value of unexplored candidates. Is this a trade-off?
  5. PEAR-NoProxy comparison simultaneously changes proxy fidelity, the number of images per candidate, and the evaluation configuration. It changes too much from PEAR. Could the authors report the independent contributions of reward averaging, low-fidelity screening, and adaptive allocation in detail?
  6. Could the authors perform a human evaluation?
  7. The paper claim "Because reward-dependent selection changes the training-sample distribution, the resulting selected-sample update is not claimed to be an unbiased Monte Carlo policy-gradient estimator of Equation equation 1". Could the authors investigate Could the authors provide a deeper analysis of how PEAR's modification of the training distribution ultimately affects policy optimization?
4 comments
AuthorsOpenReview ↗

Response To the Reviewer SPEL [1]

Thank you for your careful review and for recognizing the practical importance of candidate exploration and low-cost proxy evaluation. Following your suggestions, we have supplemented the experiments and analyses and respond point by point below.

W1: Does the method generalize to other model combinations?

PEAR continues to outperform the corresponding UniGRPO baseline under the newly added Qwen3-4B + FLUX.1-dev configuration, indicating that its performance gains are not restricted to the original model pair. To examine generalization across text–image model combinations, we conducted this additional experiment by changing both the text-model scale and the image backbone. The complete results are provided in Appendix H, Table 6.

Model configurationUniGRPO AvgPEAR AvgImprovement
Qwen3-8B + SD3.5-Medium58.8560.28+1.43
Qwen3-4B + FLUX.1-dev60.3461.42+1.08

Q1: What motivates the candidate-pool size, selection ratio, total budget, and eight allocation rounds?

We agree that the roles of these parameters and the supporting evidence should be distinguished. The revision adds candidate-pool-size experiments and combines them with the existing allocation and selection ablations to explain the default configuration.

  • Candidate-pool size:​ Appendix F, Fig. 7 compares N=48, 72, and 96. After the same 125 updates, their Avg scores are 59.50, 60.06, and 60.90, respectively. Under the comparable training times reported in Fig. 7, N=96 reaches 60.90 in 10.1 hours, exceeding the 60.10 obtained with N=72 in 10.2 hours. Within the tested configurations and budgets, these results support broader candidate exploration both for improving performance at a fixed number of updates and for achieving a better performance–time trade-off.
  • Initial evaluation count:​ We set the initial evaluation count to 24, matching UniGRPO’s training-group size and satisfying the minimum candidate count required for top-12 + bottom-12 selection. Thus, even if a prompt receives no additional evaluation budget because of its low estimated exploration benefit, its initial candidates can still form a complete training group, without requiring extra candidate evaluations merely to fill the group. Selected candidates still undergo high-fidelity rollouts through the same formal training pipeline.
  • Selection ratio:​ Table 2 compares top-12 + bottom-12, top-24 only, bottom-24 only, and random-24 selection under the FullProxy setting. Two-sided selection achieves the highest Avg, supporting the value of retaining both high- and low-reward candidates. The design aims to retain candidates with more pronounced reward differences while keeping the training sample count fixed, providing more discriminative comparisons for group-relative optimization. The detailed results are discussed in Q7 below.
  • Total evaluation budget:​ In Table 2, PEAR evaluates 50% of the candidates and achieves Avg 60.28, whereas FullProxy evaluates all candidates and achieves 60.84. The former trades a 0.56-point decrease for a 39.1% reduction in proxy-preview time, illustrating the budget–performance trade-off.
  • Eight allocation rounds:​ Eight rounds are the default configuration for executing a fixed evaluation budget in batches, rather than an optimal round count required by the algorithm. After the initial evaluation, 768 candidate evaluations remain. Dividing them into eight rounds gives 96 candidates per round, or 12 blocks of eight candidates. This allows multiple allocation updates based on newly observed rewards before the full budget is exhausted, while retaining a reasonable evaluation batch size in each round. Table 2 shows that the eight-round dynamic scheme improves Avg from 59.13 to 60.28 relative to single-round static allocation, supporting the value of multi-round feedback.

Q2: Why were means and standard deviations not reported?

Thank you for pointing this out. We have added means and standard deviations across eight image-generation seeds in Appendix G, Table 5. In the main configuration, UniGRPO and PEAR achieve Avg scores of 58.85 ± 0.20 and 60.28 ± 0.21, respectively; in the FLUX configuration, they achieve 60.34 ± 0.12 and 61.42 ± 0.10. The mean improvements of 1.43 and 1.08 points are larger than the observed generation-evaluation standard deviations.

AuthorsOpenReview ↗

Response To the Reviewer SPEL [2]

Q3: PEAR increases training time per iteration by approximately 154%

PEAR does not aim to reduce per-iteration cost relative to UniGRPO; instead, it examines whether additional exploration yields a better performance–time trade-off. Appendix F, Fig. 7 provides the full wall-clock results: under a 10.4-hour budget cap, UniGRPO completes 200 updates and achieves Avg 59.01, whereas PEAR with N=96 reaches 60.90 in 10.1 hours after 125 updates, an improvement of 1.89 points.

Q4: Is the inability of bootstrap to extrapolate to unseen rewards a trade-off?

We agree that bootstrap cannot extrapolate beyond the observed reward range. To reduce reliance on the initial observations, PEAR uses dynamic estimation and budget allocation based on newly acquired feedback: instead of fixing the remaining budget using only the initial rewards, it re-estimates marginal gains and adjusts subsequent allocations after receiving rewards from newly evaluated candidates in each round. Table 2 shows that dynamic allocation improves Avg from 59.13 to 60.28, a gain of 1.15 points over static allocation, supporting the value of updating budget decisions with subsequent feedback.

Q5: How can the contributions of reward averaging, low-fidelity screening, and adaptive allocation be distinguished?

Thank you for highlighting this issue. We agree that PEAR-NoProxy and PEAR differ simultaneously in screening fidelity, images per candidate, and evaluation configuration. Their comparison alone therefore cannot attribute the overall improvement to an individual component. We use Fig. 6, Fig. 3(a), and Table 2 to provide evidence for multi-image scoring, proxy fidelity, and budget allocation, respectively.

1. Reward averaging: Multi-image scoring improves offline allocation quality. Appendix D.1, Fig. 6 compares one, two, three, and four proxy images per PE using the same reward cache. Each configuration undergoes 100 offline allocation replays:

Proxy images per PEAllocation efficiency ↑Replay standard deviation (percentage points)
175.7%4.76
276.9%3.08
381.2%3.02
4 (default)​83.5%0.16

Replacing single-image scoring with four-image averaging improves allocation efficiency by 7.8 percentage points, supporting the value of multi-image scoring for candidate evaluation and budget decisions. 2. Low-fidelity screening: Comparable training performance at a lower proxy-evaluation cost. Figure 3(a) compares FP8/BF16 precision with 4, 6, 8, and 10 denoising steps. Six-step FP8 achieves WISE 69.35 with approximately 384 seconds of proxy-preview time per iteration; six-step BF16 achieves similar WISE performance but requires approximately 570 seconds. Thus, in this comparison, six-step FP8 reduces proxy-preview time by approximately 32.6% while retaining similar final performance, supporting its use as a cost–quality trade-off. 3. Adaptive allocation: New feedback yields additional gains under the same candidate-evaluation budget. Table 2 compares fixed half-pool, single-round static, and multi-round dynamic allocation while keeping the multi-image proxy configuration, 1,536 candidate evaluations per batch, and 24 formal training samples per prompt unchanged:

Allocation strategyWISE ↑HPSv2 ↑PickScore ↑GenEvalv2 ↑Avg ↑
Static: single-round allocation67.6426.3084.6457.9359.13
PEAR: multi-round dynamic allocation69.3528.2386.3157.2360.28

Dynamic allocation improves Avg over Static by 1.15 points, supporting the value of adjusting the budget using newly observed reward feedback. In summary, the available evidence supports three distinct findings: multi-image averaging improves offline allocation quality, low-fidelity proxies reduce evaluation cost, and dynamic allocation improves training performance under a fixed candidate-evaluation budget.

AuthorsOpenReview ↗

Response To the Reviewer SPEL [3]

Q6: Can the authors conduct a human evaluation?

Thank you for this suggestion. We currently use the open-source scoring models HPSv2 and PickScore, which are trained on human preferences, to automatically evaluate images generated from PartiPrompts. We also provide qualitative comparisons in Figures 4 and 5 to illustrate differences in semantic interpretation and image generation. We acknowledge that automatic preference metrics and qualitative analyses cannot replace direct human evaluation. We will therefore supplement future work with systematic human-preference studies to more comprehensively assess generation quality, semantic consistency, and alignment with human preferences.

Q7: How does reward-dependent selection affect policy optimization?

Retaining high- and low-reward candidates for training is not unique to PEAR; it is a selective-training strategy adopted in multiple visual RL studies. For example, the optional Best-of-N extension in DanceGRPO [1] selects the top eight and bottom eight candidates from pools of 16, 64, or 256 for training. Sol-RL [2] uses a low-precision proxy to evaluate 96 image-noise seeds, retains the top 12 and bottom 12, and regenerates them in BF16 for policy updates. These studies provide methodological precedents and empirical motivation for concentrating the update budget on candidates with more pronounced reward differences.

PEAR applies this principle to textual rewrites, changing the weighting of candidates participating in updates while keeping the formal training sample count per prompt fixed. The FullProxy ablation in Table 2 further shows that two-sided selection achieves Avg 60.84, exceeding high-reward-only selection (59.05), low-reward-only selection (57.08), and random selection (58.54), supporting its optimization value in the current setting.

References

[1] Xue, Z., et al. DanceGRPO: Unleashing GRPO on Visual Generation. 2025. Sections 2.2 and 3.2; Figure 4. https://arxiv.org/abs/2505.07818

[2] Li, Y., et al. FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling. 2026. Section 3.4; Appendix B.2. https://arxiv.org/abs/2604.06916

Thank you again for helping us improve the experimental coverage, statistical reporting, and discussion of the method’s scope.

Thanks for the detailed rebuttal, I choose to main my original score.

Reviewer_KmeCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose PEAR, a unified multimodal RL framework that improves prompt expansion (PE) for text-to-image generation. The proposed framework accounts for the asymmetric computational costs of text (LLM, Qwen3-8B) and image generation (diffusion, SD3.5-M), using top-12 and bottom-12 PEs. The proposed framework achieves competitive results on the WISE benchmark which requires reasoning.

Critical strengths
  • The paper addresses a practical problem, accounting for the emprical inference costs of different generative models (LLMs and diffusion backbone) rollouts for RL training.

  • The well-desigend ablation study(Table 2, Figure 3) justifies the design choices of budet allcation and candidate selection stretegy.

Critical weaknesses
  • The idea of using lower-precision rollouts for proxy ranking to improve online RL training efficiency is not new, as Sol-RL [1] already explores this approach to select both high- and low-reward candidates. To better establish its novelty, the paper should more clearly frame its synergies that arise from applying proxy ranking to multimodal joint RL.

  • Including related works or preliminaries sections in the main manuscript would have made the paper much easier to follow. The paper is difficult to follow on an initial reading, partly because the main text lacks dedicated explainations. It presents the proposed method and experiments without sufficiently explaining the background, key challenges, design objectives, or baseline methods.

[1] FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling (Li et al., 2026)

Questions for authors
  1. Can the authors discuss whether the benefits of the proposed method are specific to reasoning-driven tasks (such as WISE) or extend to general text-to-image generation?
  2. Since PromptRL [2] already jointly optimizes prompt refinement and image generation, the proposed method could be viewed as an instantiation of PromptRL with quantized, lighter rollouts. Could the authors clarify the methodological distinctions and compare against PromptRL under a matched computational budget?
  3. How suitable are HPSv2 and PickScore as held-out evaluation metrics for reasoning-driven RL, particularly when the intended visual content is implicit in the prompt and must be inferred through reasoning? Do these metrics adequately capture both semantic alignment and human preference in such cases?
  4. How does the diversity of the generated outputs from the same prompt compare across the baselines?

[2] PromptRL: Prompt Matters in RL for Flow-Based Image Generation (Want et al., 2026)

4 comments
AuthorsOpenReview ↗

Response To the Reviewer KmeC [1]

Thank you for your careful review and for recognizing the practical importance of generation costs and our ablations on budget allocation and candidate selection. We address your comments on novelty, methodological positioning, evaluation scope, and diversity below.

W1: What distinguishes this work from Sol-RL?

We acknowledge that Sol-RL’s low-precision screening is an important inspiration for our work. PEAR differs from Sol-RL primarily in its optimization setting, selection target, and evaluation-budget allocation, rather than in low-precision screening itself.

  1. Text-side exploration for joint reinforcement learning. Sol-RL primarily screens image-noise seeds and optimizes the image-generation policy. In contrast, we exploit the computational cost difference between text and image generation to oversample prompt enhancements (PEs) during the rollout stage of joint RL. We adopt the idea of low-precision screening to reduce the image-based evaluation cost of PEs, thereby supporting broader text-side exploration.
  2. Budget allocation based on the potential benefit of additional evaluations. Different prompts do not benefit equally from further candidate evaluation. We therefore go beyond reducing the evaluation cost per candidate: we dynamically allocate a fixed total evaluation budget according to the expected improvement in the retained high- and low-reward candidate groups, updating decisions as new feedback becomes available. This gives more promising prompts additional candidate-evaluation opportunities without changing the final number of training samples contributed by each prompt.

Accordingly, low-precision screening is one of the technical means for realizing “Explore Widely, Exploit Wisely.” Our contribution is to combine text–image cost asymmetry, text-candidate exploration, and feedback-driven budget allocation to organize exploration and selection in joint RL.

W2: The background and preliminaries in the main text are insufficiently clear

Thank you for this suggestion. We have restructured Section 2.1 to describe the existing joint text–image training paradigm and its challenges in greater detail, establish our research motivation, and then introduce an overview of PEAR. Due to space constraints, the Introduction summarizes the most relevant work and background; the complete related-work discussion is provided in Appendix A.

Q2: How does PEAR differ from PromptRL, and how do their results compare?

Model configurationMethodWISEHPSv2PickScoreGenEvalv2Avg
Qwen3-8B + SD3.5-MediumPromptRL66.60 ± 0.3026.95 ± 0.0384.82 ± 0.0356.73 ± 0.3858.77 ± 0.12
Qwen3-8B + SD3.5-MediumPEAR69.35 ± 0.7628.23 ± 0.0586.31 ± 0.0757.23 ± 0.3760.28 ± 0.21
Qwen3-4B + FLUX.1-devPromptRL63.60 ± 0.3429.83 ± 0.0388.22 ± 0.0756.99 ± 0.2559.66 ± 0.11
Qwen3-4B + FLUX.1-devPEAR68.27 ± 0.3030.03 ± 0.0388.74 ± 0.0358.65 ± 0.2761.42 ± 0.10

Thank you for highlighting this connection. PromptRL is an important related work. We have added comparisons and evaluation statistics under both model configurations in the revision (Tables 1, 5, and 6), along with a discussion in Related Work. Both PromptRL and UniGRPO jointly optimize text and image policies using image feedback. In the training setting adopted in our work, one distinction is that the UniGRPO-style baseline forms its training groups from PEs and their corresponding image samples, whereas PromptRL also includes image samples conditioned on the original prompts.

PEAR operates at a different level: it is a rollout exploration and screening mechanism that is relatively decoupled from the underlying joint-optimization rules, rather than an alternative policy-update objective. PEAR exploits text–image generation cost asymmetry to oversample PEs, dynamically allocate image-evaluation budgets across candidates, and select a fixed number of samples for formal training. It is therefore designed to integrate with joint-optimization frameworks such as PromptRL, UniGRPO, and DualGRPO, rather than replace them. The PEAR and PEAR-Dual experiments have demonstrated benefits when combined with UniGRPO-style and DualGRPO-style updates, respectively.

AuthorsOpenReview ↗

Response To the Reviewer KmeC [2]

Q1 / Q3: Do the benefits extend beyond reasoning tasks? Are HPSv2 and PickScore appropriate for evaluating implicit reasoning?

PEAR’s benefits are not limited to reasoning tasks. In the current experiments, reasoning tasks show larger gains in terms of the relative improvements in the reported metrics, while general image generation and compositional prompt following also improve. Relative to the corresponding UniGRPO baselines, the two configurations achieve the following score increases (Table 1 and the updated Table 6):

Model configurationWISE gainHPSv2 gainPickScore gainGenEvalv2 gain
Qwen3-8B + SD3.5-Medium+3.09 points+0.54 points+1.12 points+0.97 points
Qwen3-4B + FLUX.1-dev+2.52 points+0.12 points+0.43 points+1.26 points

We clarify that HPSv2 and PickScore are not used to score images generated from the reasoning-oriented WISE prompts in our evaluation. Instead, they evaluate images generated from PartiPrompts [1].

References

[1] Yu, J., et al. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. 2022. https://arxiv.org/abs/2206.10789

Q4: How diverse are the outputs of different methods for the same prompt?

We have added Appendix I, Fig. 8. The results show that PEAR improves performance while reducing average sampling diversity toward the level of the base image model, rather than preserving diversity relative to every baseline.

For each of 1,000 WISE prompts, we generate eight images per method and compute the CLIP-ViT feature cosine distances for all 28 unordered image pairs. We first average within each prompt and then across prompts. Larger distances indicate greater diversity in this feature space.

MethodMean pairwise CLIP distance
Base SD3.5-Medium0.2122
Pre-RL Qwen3-8B + SD3.5-Medium0.2324
UniGRPO0.2333
DualGRPO0.2285
PEAR0.2099
PEAR-Dual0.2180

Thank you again for helping us clarify the relationship to prior work and improve our presentation of performance, computational cost, and diversity.

Thank you for the detailed responses and revisions. The clarification that HPSv2 and PickScore are evaluated on PartiPrompts addresses my concern about their use for reasoning-oriented prompts.

However, my main concern regarding novelty remains. The rebuttal clarifies PEAR’s positioning as an exploration and screening mechanism that can complement existing joint-optimization frameworks. While this is useful, compatibility with those frameworks does not itself establish a sufficiently distinct methodological contribution.

I appreciate the added PromptRL comparisons, although the response does not clearly establish whether they address the requested matched computational budget comparison.

Overall, the rebuttal resolves some evaluation and presentation concerns, but it does not sufficiently address my central reservations about novelty and the evidence supporting the claimed advantages. I therefore maintain my rating of Weak Reject.

Thank you for your follow-up. We would like to further clarify PEAR’s methodological contribution and the conclusions supported by the compute-matched experiments.

1. PEAR’s contribution lies in exploiting the computational cost asymmetry between text and image generation through low-cost candidate expansion and adaptive budget allocation, rather than introducing a new policy-update objective.

Specifically, PEAR consists of two complementary components. First, low-cost candidate expansion leverages the typically memory-bandwidth-bound nature of LLM decoding in our setting to sample additional text candidates at relatively low incremental cost. This expands the RL exploration space without increasing the final number of training samples used for policy updates. Second, adaptive budget allocation uses estimated upper limits on the candidate reward range for each prompt to assess the potential benefit of further exploration. It allocates expensive image-evaluation resources accordingly and adjusts the allocation as new feedback becomes available.

The contribution we wish to emphasize is therefore how to organize text-candidate exploration and training-sample selection more effectively under a limited image-evaluation budget, rather than low-precision screening or framework compatibility itself. This mechanism can be integrated with joint RL frameworks such as UniGRPO, Dual-GRPO, and PromptRL.

2. Figure 7 in Appendix F provides a comparison with UniGRPO under matched GPU-hour budgets.

Figure 7 shows that PEAR outperforms longer-running UniGRPO under matched computational budgets. This supports allocating additional compute to candidate exploration and selection rather than merely increasing the number of baseline training steps.

We hope these clarifications more precisely establish our methodological contribution and the scope of the existing empirical evidence. We appreciate your further assessment of both aspects.

Reviewer_hYfwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PEAR, an exploration framework for joint text-image reinforcement learning.

  • The main idea is to explore a larger pool of text rewrites using low-cost image generation, while keeping the number of full-fidelity training rollouts fixed.

  • PEAR evaluates candidate rewrites with multiple proxy images and adaptively allocates the exploration budget across prompts based on the predicted improvement in reward gaps. The selected high- and low-reward candidates are then used for training with fresh rollouts.

Experiments show improvements over existing baselines on several text-to-image benchmarks. The ablation studies also suggest that adaptive allocation and two-sided selection contribute to the performance gains.

  • However, the improvements come with considerable additional computation, and it remains unclear whether the proposed approach offers a sufficiently strong advantage under a matched total compute budget.
Critical strengths
  • Exploring more text rewrites while using cheaper image evaluations is a reasonable way to address the exploration cost in joint text-image RL. Evaluating each rewrite with multiple images also helps reduce the influence of image-generation randomness during candidate selection.

  • The ablation studies provide useful evidence for the proposed design.

    • Adaptive allocation improves the average score from 59.39 to 60.28 compared with fixed half-pool screening under the same candidate-evaluation budget, with improvements across all four metrics.
    • The results for two-sided selection are also consistent, suggesting that the gains are not simply due to evaluating more candidates.
  • The comparison with FullProxy shows a useful efficiency-performance trade-off.

    • PEAR reduces the per-iteration runtime by around 26.2% compared with evaluating the full candidate pool, while retaining most of its performance. This supports the usefulness of selective exploration within the proposed framework.
Critical weaknesses
  1. The lack of a matched total-compute comparison.
    • Although the number of training rollouts is fixed, PEAR introduces substantial additional computation for proxy evaluation.
      • According to Table 3, its per-iteration runtime is about 2.54 times that of UniGRPO, while the average improvement is 1.43 points.
      • It is unclear whether the same additional compute, if allocated to longer training or simpler exploration strategies, would lead to comparable improvements.
  2. The additional benefit of the adaptive allocation mechanism is not yet fully established.
    • Several components build on existing joint RL and contrastive screening methods, and the main distinction lies in multi-image evaluation and adaptive allocation across prompts.
      • While the adaptive strategy outperforms fixed half-pool screening, the simpler strategy already captures a substantial portion of the gains.
      • It would be helpful to compare against other lightweight allocation strategies to better understand whether the added complexity is justified.
  3. The robustness of the reported improvements is unclear.
    • The paper reports evaluation over multiple seeds, but does not provide variance across independent training runs for the main comparisons.
      • The average improvement over DualGRPO is relatively small (0.69 points), and PEAR does not outperform DualGRPO on GenEvalv2.
      • Given these margins, it is difficult to assess how consistently the improvements would hold across training runs.
Questions for authors

please refer to the Weaknesses.

3 comments
AuthorsOpenReview ↗

Response To the Reviewer hYfw

Thank you for your careful review and for recognizing the value of multi-image evaluation and our ablations on adaptive allocation and two-sided selection. We address your concerns regarding compute-budget fairness, the incremental value of adaptive allocation, and the robustness of the results below.

W1: Could the additional computation yield comparable gains if used to train the baseline longer?

Fixing the number of formal training rollouts does not ensure an equal total compute budget. Additional exploration should therefore be assessed in terms of both overall training time and final performance. Under the comparable wall-clock budgets reported in Fig. 7, PEAR still outperforms longer-running UniGRPO.

We have added performance-versus-cumulative-training-time curves in Appendix F, Fig. 7. Within a 10.4-hour budget cap, UniGRPO completes 200 updates and reaches Avg 59.01. PEAR with N=72 reaches Avg 60.10 in 10.2 hours, while PEAR with N=96 reaches Avg 60.90 in 10.1 hours after only 125 updates, improving Avg by 1.09 and 1.89 points, respectively. Within the tested configurations, these results support allocating computation to broader candidate exploration and selection rather than simply increasing the number of UniGRPO updates.

W2: Do the additional benefits of adaptive allocation justify its complexity?

We agree that outperforming fixed half-pool screening alone does not fully establish the value of the added complexity. The existing ablations provide evidence for two design choices: allocating evaluations according to their estimated value and updating allocations with new feedback. With the same multi-image scoring configuration, total number of candidate evaluations, and number of formal training samples, PEAR improves Avg over HalfProxy and Static by 0.89 and 1.15 points, respectively. Fig. 3(c) further provides offline comparisons against simple static variance- and reward-range-based strategies, supporting the motivation for estimating the benefit of additional evaluations. The estimator only resamples observed rewards, requiring neither additional models nor extra generation calls to compute allocation priorities; however, multi-round execution still incurs practical overhead.

W3: The improvement over DualGRPO is small, and PEAR does not outperform DualGRPO on GenEvalv2

Model configurationDualGRPO AvgPEAR-Dual AvgImprovement
Qwen3-8B + SD3.5-Medium59.59 ± 0.0961.21 ± 0.12+1.62
Qwen3-4B + FLUX.1-dev60.64 ± 0.1461.88 ± 0.12+1.24

PEAR and DualGRPO are not mutually exclusive: the newly added PEAR-Dual experiments show that combining them further improves performance in both model configurations. PEAR operates at the rollout stage, exploiting differences in text and image generation costs for candidate oversampling and adaptive budget allocation. In our joint-training setting, UniGRPO and DualGRPO differ primarily in how training groups are organized and how advantages are computed. PEAR’s exploration and allocation mechanism can therefore be combined with different underlying optimization schemes. The original PEAR implementation uses UniGRPO-style joint updates; we additionally integrate it with DualGRPO to construct PEAR-Dual. Compared with the corresponding DualGRPO baselines, PEAR-Dual improves all four metrics in both configurations, with GenEvalv2 increasing from 57.69 to 58.95 and from 57.26 to 58.82, respectively.

Model configurationMethodWISEHPSv2PickScoreGenEvalv2Avg
Qwen3-8B + SD3.5-MediumUniGRPO66.26 ± 0.7627.69 ± 0.0385.19 ± 0.0356.26 ± 0.1958.85 ± 0.20
Qwen3-8B + SD3.5-MediumPEAR69.35 ± 0.7628.23 ± 0.0586.31 ± 0.0757.23 ± 0.3760.28 ± 0.21
Qwen3-4B + FLUX.1-devUniGRPO65.75 ± 0.3029.91 ± 0.0588.31 ± 0.0357.39 ± 0.3560.34 ± 0.12
Qwen3-4B + FLUX.1-devPEAR68.27 ± 0.3030.03 ± 0.0388.74 ± 0.0358.65 ± 0.2761.42 ± 0.10

Regarding statistical reporting, we have added means and standard deviations across eight evaluation runs to characterize evaluation variability. For example, in the main configuration, the Avg scores of DualGRPO and PEAR are 59.59 ± 0.09 and 60.28 ± 0.21, respectively. The complete results are provided in the newly added Appendix G, Table 5, and Appendix H, Table 6.

Thank you again for helping us distinguish more clearly between the benefits of exploration, the incremental value of allocation, and the scope of our computational and statistical evidence.

Thank you for the additional experiments and clarifications.

While some of my concerns have been addressed, the robustness of the improvements across independent training runs remains unclear.

Therefore, I keep my original rating.

Model configurationMethodWISEHPSv2PickScoreGenEvalv2Avg
Qwen3-8B + SD3.5-MediumUniGRPO66.11 ± 0.8927.78 ± 0.0685.14 ± 0.0756.45 ± 0.3958.87 ± 0.24
Qwen3-8B + SD3.5-MediumPEAR69.94 ± 0.9728.38 ± 0.0786.38 ± 0.1257.23 ± 0.4860.48 ± 0.27
Qwen3-4B + FLUX.1-devUniGRPO65.47 ± 0.5629.86 ± 0.0888.35 ± 0.0657.03 ± 0.5260.18 ± 0.19
Qwen3-4B + FLUX.1-devPEAR69.07 ± 0.6130.16 ± 0.0688.96 ± 0.0758.85 ± 0.4761.76 ± 0.19

Thank you for your follow-up feedback. We agree that repeated image-generation evaluations at a fixed checkpoint cannot substitute for assessing robustness across independent training runs. Since repeating the complete joint text–image RL training pipeline takes substantial time, we first submitted the results of the other supplementary experiments that had already been completed. We now provide the newly completed multi-seed training results, with four independent RL training runs for each method–configuration pair. In light of these results, could you please let us know whether any specific concerns remain or whether any comparisons require further clarification? Thank you for your careful review.

Reviewer_fyooOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PEAR, a cost-aware framework for joint reinforcement learning of text and image policies. The method oversamples text prompt expansions, evaluates a subset using a low-cost image-generation proxy, and adaptively allocates a fixed evaluation budget according to estimated marginal improvements in a two-sided selection utility. It retains both high- and low-reward prompt expansions for fresh, high-fidelity rollouts and uses their rewards to update the text and image policies. Experiments with Qwen3-8B and Stable Diffusion 3.5 Medium show improvements over existing joint-training baselines. Compared with full-pool proxy evaluation, PEAR reduces proxy-screening time by 39.1% while achieving a similar aggregate score. The paper presents a promising approach to balancing exploration, training quality, and image-generation cost.

Critical strengths
  1. The paper identifies a practical computational asymmetry between inexpensive text generation and expensive image generation, and develops a coherent framework around this observation. The separation between proxy-based screening and fresh high-fidelity training rollouts is clearly specified, while the adaptive allocation mechanism provides a principled alternative to evaluating every candidate uniformly.

  2. The ablation studies examine several important design choices, including adaptive versus static allocation, partial versus full proxy evaluation, and two-sided versus one-sided candidate selection. PEAR achieves an average score of 60.28, compared with 58.85 for UniGRPO, and reduces proxy-screening time from 630.2 seconds to 383.8 seconds relative to full-pool proxy evaluation. The evaluation also includes HPSv2, PickScore, and GenEvalv2, providing evidence beyond the in-domain WISE metric.

Critical weaknesses
  1. The empirical validation is limited to a single text-policy and image-policy configuration, namely Qwen3-8B and Stable Diffusion 3.5 Medium. Although the evaluation includes multiple metrics, the paper does not establish whether the proposed allocation strategy generalizes to other model combinations, reward functions, or text-to-image training settings. In addition, evaluation over multiple image-generation seeds does not substitute for independent reinforcement-learning training runs. The lack of multi-seed training results and statistical uncertainty estimates makes it difficult to determine the robustness of the reported improvements, particularly given the relatively modest 1.43-point average gain over UniGRPO.

  2. The computational-efficiency claim is primarily supported by proxy-screening time rather than end-to-end training cost. PEAR generates a substantially expanded text-candidate pool, evaluates selected candidates using multiple proxy images, and incurs additional allocation and synchronization overhead. The paper explicitly acknowledges that the reported savings concern proxy evaluation rather than end-to-end training time. Consequently, the experiments do not establish whether PEAR reduces total wall-clock time or overall compute consumption under a realistic training budget. Furthermore, the adaptive allocation estimator is based on rewards already observed in each candidate pool and cannot extrapolate beyond the observed reward range. A more comprehensive end-to-end cost analysis and evaluation of allocation robustness would strengthen the practical claims.

Questions for authors
  1. Can the authors report independent training runs with confidence intervals or statistical tests, and evaluate whether the performance gains persist across different text-policy and image-policy combinations?

  2. What are the end-to-end training time, total GPU-hours, and overall compute costs of PEAR compared with UniGRPO and PEAR-FullProxy? How do the additional costs of text oversampling, proxy generation, and synchronization affect the overall efficiency gains?

  3. How robust is the adaptive allocation strategy when the initially observed reward distribution poorly represents the unseen candidates? Can the authors quantify the sensitivity of final policy performance to allocation noise and the reward-dependent selection of training samples?

3 comments
AuthorsOpenReview ↗

Response To the Reviewer fyoo

Thank you for your careful review and for recognizing the motivation arising from text–image computational cost asymmetry and the separation between proxy screening and high-fidelity training. Following your suggestions, we have added a second model configuration, training-time analyses, and evaluation-variability statistics, while further clarifying the scope of our allocation and selection analyses.

Q1: Generalization across model combinations and statistical reliability

In addition to the original Qwen3-8B + SD3.5-Medium configuration, we have added experiments with Qwen3-4B + FLUX.1-dev, changing both the text-model scale and the image backbone. The key results are summarized below (Appendix G, Table 5; Appendix H, Table 6):

Model configurationMethodWISEHPSv2PickScoreGenEvalv2Avg
Qwen3-8B + SD3.5-MediumUniGRPO66.26 ± 0.7627.69 ± 0.0385.19 ± 0.0356.26 ± 0.1958.85 ± 0.20
Qwen3-8B + SD3.5-MediumPEAR69.35 ± 0.7628.23 ± 0.0586.31 ± 0.0757.23 ± 0.3760.28 ± 0.21
Qwen3-4B + FLUX.1-devUniGRPO65.75 ± 0.3029.91 ± 0.0588.31 ± 0.0357.39 ± 0.3560.34 ± 0.12
Qwen3-4B + FLUX.1-devPEAR68.27 ± 0.3030.03 ± 0.0388.74 ± 0.0358.65 ± 0.2761.42 ± 0.10

PEAR outperforms UniGRPO in both text–image model configurations, improving all four metrics and increasing Avg by 1.43 and 1.08 points, respectively. These results indicate that the gains are not restricted to the original model pair.

The performance gains are substantially larger than the evaluation variability induced by image-generation randomness. We report means and standard deviations across eight evaluation runs for each fixed checkpoint: PEAR improves Avg over UniGRPO by 1.43 and 1.08 points in the two configurations, whereas the corresponding evaluation standard deviations range from only 0.10 to 0.21 points.

Q2: End-to-end training cost and whether additional exploration is worthwhile

We agree that fixing the number of formal training rollouts does not ensure an equal total compute budget, and that additional exploration should be assessed in terms of both overall training time and final performance. We therefore added performance-versus-cumulative-training-time curves in Appendix F, Fig. 7. Under the comparable wall-clock budgets reported in Fig. 7, PEAR still outperforms longer-running UniGRPO:​ within a 10.4-hour budget cap, UniGRPO completes 200 updates and reaches Avg 59.01; PEAR with N=72 reaches Avg 60.10 in 10.2 hours, while PEAR with N=96 reaches Avg 60.90 in 10.1 hours after only 125 updates. These correspond to improvements of 1.09 and 1.89 points, respectively. Within the tested configurations, these results support allocating computation to broader candidate exploration and selection rather than simply increasing the number of UniGRPO updates.

Q3: Initial reward-distribution mismatch, allocation noise, and reward-dependent sample selection

Allocation strategyWISE ↑HPSv2 ↑PickScore ↑GenEvalv2 ↑Avg ↑Preview time (s) ↓
PEAR-Static67.6426.3084.6457.9359.13367.2
PEAR (dynamic)69.3528.2386.3157.2360.28383.8

We recognize this concern: an unrepresentative initial reward distribution can lead to biased budget allocation. To reduce reliance on the initial observations, PEAR combines bootstrap-based marginal-gain estimation with multi-round feedback. Static allocation determines the entire remaining budget using only the initially observed rewards. Dynamic allocation instead evaluates additional candidates, incorporates newly observed rewards, and re-estimates marginal gains, adjusting subsequent allocations round by round and providing opportunities to revise the initial assessment. Table 2 shows that dynamic allocation increases Avg from 59.13 to 60.28, an improvement of 1.15 points over static allocation, supporting the value of updating allocation decisions with subsequent feedback.

Thank you for helping us present the additional evidence more precisely and clarify the scope of our conclusions regarding model generalization, computational efficiency, and training robustness.

AuthorsOpenReview ↗

Multi Seed Training Result

Model configurationMethodWISEHPSv2PickScoreGenEvalv2Avg
Qwen3-8B + SD3.5-MediumUniGRPO66.11 ± 0.8927.78 ± 0.0685.14 ± 0.0756.45 ± 0.3958.87 ± 0.24
Qwen3-8B + SD3.5-MediumPEAR69.94 ± 0.9728.38 ± 0.0786.38 ± 0.1257.23 ± 0.4860.48 ± 0.27
Qwen3-4B + FLUX.1-devUniGRPO65.47 ± 0.5629.86 ± 0.0888.35 ± 0.0657.03 ± 0.5260.18 ± 0.19
Qwen3-4B + FLUX.1-devPEAR69.07 ± 0.6130.16 ± 0.0688.96 ± 0.0758.85 ± 0.4761.76 ± 0.19

Thank you for your suggestions regarding robustness across independent training runs and generalization across model combinations. Since repeating complete RL training runs takes substantial time, we first reported the other completed supplementary results. We now provide the means and standard deviations from four independent training runs with different random seeds for each method–configuration pair. Unlike our earlier evaluations using eight image-generation seeds at fixed checkpoints, these statistics capture variability arising from the training process itself. We hope this additional evidence more fully addresses your concerns about robustness and generalization across configurations. Please let us know whether you have any further questions or suggestions regarding the new results, experimental protocol, or statistical reporting.

Thank you for the additional experiments and detailed responses. However, I remain concerned that the robustness of PEAR to misleading initial reward distributions and noisy proxy estimates has not been sufficiently established. The static-versus-dynamic comparison does not fully address this concern. Therefore, I maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We sincerely thank all four reviewers for their careful evaluation and constructive feedback. We appreciate fyoo, KmeC, and SPEL for recognizing the practical importance of accounting for candidate-exploration costs, and fyoo, hYfw, and KmeC for acknowledging the evidence provided by our ablations on budget allocation and candidate selection. We also appreciate the positive assessment of our low-cost proxy design by fyoo and SPEL, particularly the separation between screening and high-fidelity training highlighted by fyoo, as well as the value of multi-image evaluation and the efficiency–performance trade-off relative to full-pool evaluation recognized by hYfw.

To address the reviewers’ concerns, we have expanded the revised manuscript with the following experiments and analyses:

  1. Additional comparisons with PromptRL and the PEAR-Dual combination. We include PromptRL as a joint-training baseline and evaluate PEAR combined with DualGRPO to examine their compatibility and complementary gains (Tables 1 and 5).

  2. An additional text–image model configuration. We evaluate Qwen3-4B + FLUX.1-dev, extending our empirical coverage beyond the original Qwen3-8B + SD3.5-Medium configuration (Appendix H, Table 6).

  3. Training-wall-clock and candidate-pool-size analyses. We report performance-versus-training-time curves for PEAR with candidate-pool sizes of N = 48, 72, and 96, alongside longer-running UniGRPO, to examine the benefits and computational costs of broader exploration (Appendix F, Figure 7).

  4. Evaluation variability across generation seeds. We report means and standard deviations over eight image-generation seeds, providing greater transparency about generation randomness at fixed trained checkpoints and explicitly distinguishing it from variability across independent RL training runs (Appendix G, Table 5).

  5. Quantitative within-prompt diversity analysis. We evaluate 1,000 WISE prompts with eight generated images per prompt, using mean pairwise CLIP-feature cosine distance to characterize the trade-off between performance and sampling diversity (Appendix I, Figure 8).

  6. Expanded related-work discussion. We add a discussion of PromptRL and further distinguish existing joint-optimization frameworks from PEAR’s contributions to candidate exploration, screening, and evaluation-budget allocation (Appendix A).

  7. Restructured task setup and method overview. We reorganize Section 2.1 to provide a fuller description of the task setup and joint text–image training paradigm. We clarify the challenges arising from text–image generation cost asymmetry, stochastic image rewards, and the allocation of a limited evaluation budget, before introducing the PEAR overview after establishing the motivation. This creates a clearer progression from the problem formulation and design motivation to the method.

The additions are highlighted in dark red. Our point-by-point responses explain how these experiments and analyses address the individual concerns. We thank the reviewers again for helping us strengthen the empirical coverage, presentation, and positioning of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.