- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper propose PEAR, oversampling PEs and evaluates an initial subset using an image proxy, averaging rewards over multiple images for each candidate. It then allocates a fixed evaluation budget across prompts according to bootstrap-estimated improvements in the mean reward gap between high- and low-reward candidates. These estimates are updated as new feedback arrives. The proxy feedback guides selection but does not directly enter the training loss. The experimental results are competitive.
Critical strengths
- The paper addresses a practical and interesting problem concerning the high computational cost of candidate exploration in unified multimodal reinforcement learning.
- The proposed low-cost proxy evaluation strategy combines FP8 quantization with reduced denoising steps, providing a practical solution for reducing the cost of large-scale candidate screening.
Critical weaknesses
The evaluation is limited to a single combination of Qwen3-8B and Stable Diffusion 3.5 Medium, making it difficult to determine whether the proposed cost-asymmetry assumptions and adaptive selection mechanism generalize to different model architectures, model scales, and generation systems.
Questions for authors
- The ablation studies on initial candidate pool size, the top-bottom selection ration, and the total evaluation budget are not clear. For instance, why keeping 8 runs for adaptive allocation?
- As the evaluation averages results across multiple image-generation seeds, why main performance tables do not report means, standard deviations?
- As shown in Table 3, PEAR requires 796.3 seconds per training iteration compared with only 313.6 seconds for UniGRPO, indicating that the proposed method increases the overall training time by approximately 154%.
- The bootstrap-based marginal-gain estimator relies entirely on observed rewards and cannot extrapolate beyond their empirical range, potentially underestimating the value of unexplored candidates. Is this a trade-off?
- PEAR-NoProxy comparison simultaneously changes proxy fidelity, the number of images per candidate, and the evaluation configuration. It changes too much from PEAR. Could the authors report the independent contributions of reward averaging, low-fidelity screening, and adaptive allocation in detail?
- Could the authors perform a human evaluation?
- The paper claim "Because reward-dependent selection changes the training-sample distribution, the resulting selected-sample update is not claimed to be an unbiased Monte Carlo policy-gradient estimator of Equation equation 1". Could the authors investigate Could the authors provide a deeper analysis of how PEAR's modification of the training distribution ultimately affects policy optimization?
4 comments
Response To the Reviewer SPEL [2]
Q3: PEAR increases training time per iteration by approximately 154%
PEAR does not aim to reduce per-iteration cost relative to UniGRPO; instead, it examines whether additional exploration yields a better performance–time trade-off. Appendix F, Fig. 7 provides the full wall-clock results: under a 10.4-hour budget cap, UniGRPO completes 200 updates and achieves Avg 59.01, whereas PEAR with N=96 reaches 60.90 in 10.1 hours after 125 updates, an improvement of 1.89 points.
Q4: Is the inability of bootstrap to extrapolate to unseen rewards a trade-off?
We agree that bootstrap cannot extrapolate beyond the observed reward range. To reduce reliance on the initial observations, PEAR uses dynamic estimation and budget allocation based on newly acquired feedback: instead of fixing the remaining budget using only the initial rewards, it re-estimates marginal gains and adjusts subsequent allocations after receiving rewards from newly evaluated candidates in each round. Table 2 shows that dynamic allocation improves Avg from 59.13 to 60.28, a gain of 1.15 points over static allocation, supporting the value of updating budget decisions with subsequent feedback.
Q5: How can the contributions of reward averaging, low-fidelity screening, and adaptive allocation be distinguished?
Thank you for highlighting this issue. We agree that PEAR-NoProxy and PEAR differ simultaneously in screening fidelity, images per candidate, and evaluation configuration. Their comparison alone therefore cannot attribute the overall improvement to an individual component. We use Fig. 6, Fig. 3(a), and Table 2 to provide evidence for multi-image scoring, proxy fidelity, and budget allocation, respectively.
1. Reward averaging: Multi-image scoring improves offline allocation quality. Appendix D.1, Fig. 6 compares one, two, three, and four proxy images per PE using the same reward cache. Each configuration undergoes 100 offline allocation replays:
| Proxy images per PE | Allocation efficiency ↑ | Replay standard deviation (percentage points) |
|---|---|---|
| 1 | 75.7% | 4.76 |
| 2 | 76.9% | 3.08 |
| 3 | 81.2% | 3.02 |
| 4 (default) | 83.5% | 0.16 |
Replacing single-image scoring with four-image averaging improves allocation efficiency by 7.8 percentage points, supporting the value of multi-image scoring for candidate evaluation and budget decisions. 2. Low-fidelity screening: Comparable training performance at a lower proxy-evaluation cost. Figure 3(a) compares FP8/BF16 precision with 4, 6, 8, and 10 denoising steps. Six-step FP8 achieves WISE 69.35 with approximately 384 seconds of proxy-preview time per iteration; six-step BF16 achieves similar WISE performance but requires approximately 570 seconds. Thus, in this comparison, six-step FP8 reduces proxy-preview time by approximately 32.6% while retaining similar final performance, supporting its use as a cost–quality trade-off. 3. Adaptive allocation: New feedback yields additional gains under the same candidate-evaluation budget. Table 2 compares fixed half-pool, single-round static, and multi-round dynamic allocation while keeping the multi-image proxy configuration, 1,536 candidate evaluations per batch, and 24 formal training samples per prompt unchanged:
| Allocation strategy | WISE ↑ | HPSv2 ↑ | PickScore ↑ | GenEvalv2 ↑ | Avg ↑ |
|---|---|---|---|---|---|
| Static: single-round allocation | 67.64 | 26.30 | 84.64 | 57.93 | 59.13 |
| PEAR: multi-round dynamic allocation | 69.35 | 28.23 | 86.31 | 57.23 | 60.28 |
Dynamic allocation improves Avg over Static by 1.15 points, supporting the value of adjusting the budget using newly observed reward feedback. In summary, the available evidence supports three distinct findings: multi-image averaging improves offline allocation quality, low-fidelity proxies reduce evaluation cost, and dynamic allocation improves training performance under a fixed candidate-evaluation budget.
Response To the Reviewer SPEL [3]
Q6: Can the authors conduct a human evaluation?
Thank you for this suggestion. We currently use the open-source scoring models HPSv2 and PickScore, which are trained on human preferences, to automatically evaluate images generated from PartiPrompts. We also provide qualitative comparisons in Figures 4 and 5 to illustrate differences in semantic interpretation and image generation. We acknowledge that automatic preference metrics and qualitative analyses cannot replace direct human evaluation. We will therefore supplement future work with systematic human-preference studies to more comprehensively assess generation quality, semantic consistency, and alignment with human preferences.
Q7: How does reward-dependent selection affect policy optimization?
Retaining high- and low-reward candidates for training is not unique to PEAR; it is a selective-training strategy adopted in multiple visual RL studies. For example, the optional Best-of-N extension in DanceGRPO [1] selects the top eight and bottom eight candidates from pools of 16, 64, or 256 for training. Sol-RL [2] uses a low-precision proxy to evaluate 96 image-noise seeds, retains the top 12 and bottom 12, and regenerates them in BF16 for policy updates. These studies provide methodological precedents and empirical motivation for concentrating the update budget on candidates with more pronounced reward differences.
PEAR applies this principle to textual rewrites, changing the weighting of candidates participating in updates while keeping the formal training sample count per prompt fixed. The FullProxy ablation in Table 2 further shows that two-sided selection achieves Avg 60.84, exceeding high-reward-only selection (59.05), low-reward-only selection (57.08), and random selection (58.54), supporting its optimization value in the current setting.
References
[1] Xue, Z., et al. DanceGRPO: Unleashing GRPO on Visual Generation. 2025. Sections 2.2 and 3.2; Figure 4. https://arxiv.org/abs/2505.07818
[2] Li, Y., et al. FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling. 2026. Section 3.4; Appendix B.2. https://arxiv.org/abs/2604.06916
Thank you again for helping us improve the experimental coverage, statistical reporting, and discussion of the method’s scope.
Thanks for the detailed rebuttal, I choose to main my original score.
Response To the Reviewer SPEL [1]
Thank you for your careful review and for recognizing the practical importance of candidate exploration and low-cost proxy evaluation. Following your suggestions, we have supplemented the experiments and analyses and respond point by point below.
W1: Does the method generalize to other model combinations?
PEAR continues to outperform the corresponding UniGRPO baseline under the newly added Qwen3-4B + FLUX.1-dev configuration, indicating that its performance gains are not restricted to the original model pair. To examine generalization across text–image model combinations, we conducted this additional experiment by changing both the text-model scale and the image backbone. The complete results are provided in Appendix H, Table 6.
Q1: What motivates the candidate-pool size, selection ratio, total budget, and eight allocation rounds?
We agree that the roles of these parameters and the supporting evidence should be distinguished. The revision adds candidate-pool-size experiments and combines them with the existing allocation and selection ablations to explain the default configuration.
Q2: Why were means and standard deviations not reported?
Thank you for pointing this out. We have added means and standard deviations across eight image-generation seeds in Appendix G, Table 5. In the main configuration, UniGRPO and PEAR achieve Avg scores of 58.85 ± 0.20 and 60.28 ± 0.21, respectively; in the FLUX configuration, they achieve 60.34 ± 0.12 and 61.42 ± 0.10. The mean improvements of 1.43 and 1.08 points are larger than the observed generation-evaluation standard deviations.