- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the gap between set diversity and output diversity in reward-guided progressive selection for inference-time alignment. The authors identify the champion constraint, showing that retaining the highest-scoring preview candidate can prevent improvements in set diversity from translating into more diverse final outputs. To address this, they propose champion-free selection strategies, including exact max-sum diversification and density-aware selection. Experiments across four image generators demonstrate improved output diversity and rare-mode coverage while largely preserving generation quality.
Critical strengths
-
The paper highlights an important limitation of conventional diversity-aware selection: increasing survivor-set diversity does not necessarily improve the diversity of delivered outputs. The champion constraint provides a clear explanation, supported by theoretical analysis and empirical measurements.
-
The proposed champion-free strategies improve output diversity across four generative models with minimal computational overhead. The experiments include matched-quality comparisons, targeted ablations, rare-mode analysis, and human evaluations, providing substantial evidence for the practical effectiveness of the approach.
Critical weaknesses
-
The champion constraint follows directly from the deterministic final selection rule: if the preview champion is also the final reward winner, retaining it necessarily fixes the output. While useful, this observation is relatively straightforward. The additional theoretical analysis relies on simplifying assumptions, leaving the broader significance and generality of the theoretical contribution somewhat unclear.
-
The paper demonstrates the benefits of champion-free selection, but the additional value of the specific density-aware criterion is less convincing. In particular, density and exact max-sum achieve comparable mean output diversity at matched reward, while density’s main advantage is a relatively modest improvement in rare-mode coverage. It remains unclear whether simpler champion-free strategies could achieve similar quality–diversity trade-offs.
Questions for authors
Q1. Beyond the direct observation that retaining the final winner fixes the output, what are the main nontrivial insights provided by the champion constraint analysis? How robust are the locked-share predictions beyond the i.i.d. Gaussian assumptions?
Q2. Could the authors further clarify the advantages of density-aware selection over simpler champion-free alternatives, such as stochastic reward-based selection or controlled champion exclusion, under matched reward budgets?
Q3. How well does the human-preference tolerance calibrated on ImageRewardDB generalise to the evaluated generators and prompts? Can the authors provide stronger evidence that the increased diversity does not introduce meaningful perceptual quality degradation?
Q4. Since rare-mode coverage is the main advantage of density over exact max-sum, how sensitive are the conclusions to the feature representation and definition of rare modes? What evidence supports the practical significance of the reported 7%–14% relative improvement?
I find the research question meaningful and the empirical results encouraging. I am open to raising my score if the authors can adequately address these concerns during the rebuttal.
Post-Rebuttal:
Overall, I think the rebuttal has adequately addressed my major empirical concerns, and I am inclined to raise my rating from weak reject to weak accept.
6 comments
We sincerely appreciate your time and thoughtful feedback (2/4)
On the simplifying assumptions (R1/A1, continued). To clarify, they are used only by the model that predicts how the locked share changes with preview quality and pool size. The share itself is computed directly from any pipeline's preview and final scores, with no model. We test the model rather than just assuming it:
- No fitting. With no fitted parameter, it matches 11 of 12 measured shares on images within 0.02, and all 8 on text within 0.015. [new] With different preview and final reward models, it matches all 10 shares within 0.017.
- Independence holds by design, because each candidate starts from its own random seed.
- The Gaussian form matters less than it seems. The locked share depends only on how candidates are ranked, not on the scale or shape of their scores. The Gaussian form only fixes how preview and final rankings move together, and we measured this directly: the observed share is within 0.009 of the Gaussian prediction, while other dependence structures (copulas) would shift it by up to 0.15 (Appendix F.3).
- The rare-mode model is simple on purpose (one common and one rare mode), and both of its testable predictions hold (R2/A2, rare-mode coverage).
In the revision, a short paragraph after the lemma (end of Section 4) states its generality and the first result, Appendix F.3 adds the robustness tests above, and the [new] cross-reward points appear in blue in Figure 4 (Section 6.3).
W2. The paper demonstrates the benefits of champion-free selection, but the additional value of the specific density-aware criterion is less convincing. In particular, density and exact max-sum achieve comparable mean output diversity at matched reward, while density’s main advantage is a relatively modest improvement in rare-mode coverage. It remains unclear whether simpler champion-free strategies could achieve similar quality–diversity trade-offs.
Q2. Could the authors further clarify the advantages of density-aware selection over simpler champion-free alternatives, such as stochastic reward-based selection or controlled champion exclusion, under matched reward budgets?
R2/A2. Thank you for the insightful question. We agree that density and exact max-sum tie on mean output diversity, but they both follow the paper's main principle that a selector must be free to drop the champion. Beyond this principle, density adds three things, which we detail below.
It beats simpler rules that also drop the champion. [new] Following your suggestion, we have added comparisons with both alternatives you named, stochastic reward-based selection and controlled champion exclusion, in all six backbone–prompt-set settings. At matched ImageReward, density delivers more output diversity and more rare modes than each of them:
| Density minus … | DreamSim (×10⁻³) | Rare-mode share (points) | Intervals above zero |
|---|---|---|---|
| Controlled champion exclusion: random drop, then top-k [new] | +3.1 to +15.3 | +3.7 to +8.2 | 12/12 |
| Stochastic reward-based selection: reward-weighted sampling | +2.9 to +13.6 | +3.6 to +7.9 | 12/12 |
| Controlled champion exclusion: same drops as density, reward-weighted refill | +1.6 to +10.1 | +1.9 to +5.3 | 12/12 |
Dropping the champion at random recovers only 11–25% of density's gain over pruning at the same reward. So dropping the champion is necessary but not sufficient: most of the gain comes from which candidate takes its place. k-DPP sampling, which can also drop the champion, likewise falls below density at matched reward on all four backbones (Section 6.2). The full table is the new Table 30 (Appendix F.5).
Better rare-mode coverage. At matched reward and equal mean diversity, density delivers rare modes 7% and 14% more often than max-sum on the two prompt sets (+1.6 and +3.1 percentage points; intervals exclude zero). The advantage appears where Proposition 2 predicts: about three quarters of it comes from requests with at least two rare candidates. It is positive under 119 of 120 cluster definitions (Appendix D.6) and grows with pool size, from 0.4 to 2.3 points between 6 and 16 candidates. Our main setting uses six candidates, so it shows the effect near its smallest.
It still acts in settings where max-sum cannot. Max-sum rewards spread among the images it keeps, so with one survivor it keeps the best preview. Density instead marks down any candidate that looks like many others in the pool. So even with one survivor, it can skip a champion from a crowded mode and keep a good image from a rarer one. Different requests then deliver different modes more often. [new] In all six settings, at the same budget, this raises output diversity and the rare-mode share (by 4.9–9.9 points) over keeping the best preview, at a small reward cost within tolerance (Section 6.4, Appendix E.2). We hope this clarifies what density adds beyond dropping the champion.
We sincerely appreciate your time and thoughtful feedback (3/4)
Q3. How well does the human-preference tolerance calibrated on ImageRewardDB generalise to the evaluated generators and prompts? Can the authors provide stronger evidence that the increased diversity does not introduce meaningful perceptual quality degradation?
A3. Thank you for this important question. In short, on SD3.5 our own control pairs agree with the ImageRewardDB calibration, and a new human study on our generators finds no clear quality difference.
The reward cost is small. On the four backbones, density keeps 86–96% of the reward that pruning gains over native sampling: it gives up at most 0.029 ImageReward, while pruning gains 0.21–0.28. PickScore and CLIP, which the selector never sees, fall by at most 0.002 and 0.001. When PickScore selects instead, ImageReward becomes the independent judge and falls by at most 0.032.
What this means for people. The paper's tolerance is the score gap at which people prefer the higher-scoring image 51% of the time on held-out ImageRewardDB pairs, a preference so slight that detecting it would take about 20,000 paired judgements. Every loss above is within it (0.029 against 0.050 for ImageReward; 64% of tolerance when PickScore selects). Even the most conservative bound in the paper corresponds to a preference of at most 51.3% for pruning's image, under the ImageRewardDB calibration.
Does the calibration transfer to our generators? [new] On SD3.5, yes: the control pairs of our strip study (Section 6.5, native sampling against pruning; Appendix I.1) agree with it. On ImageRewardDB, preference for the higher-scoring image stays close to 50% up to a gap of 0.2 ImageReward and rises beyond it. Our raters behave the same way: they favour the higher-reward strip once the gap exceeds 0.2 (6 of 6 quality calls) and show no preference below it. All 100 density-versus-pruning strips lie below 0.2 (mean gap 0.025, largest 0.161).
[new] A direct quality test on our generators. To provide this stronger evidence, four new non-author raters, all professional annotators, compared density's image with pruning's image for the same request on SD3.5 and Klein (120 pairs per rater, stratified by reward gap).
- They can see quality differences: on curated high-gap control pairs they chose the higher-reward image 98.8% of the time, and all four passed the attention and repeat checks.
- No clear difference between density and pruning: over all requests, they preferred pruning's image in 53.1% of cases [49.6, 56.5], an interval that includes both 50% and the 51.3% reading above.
- Off-prompt rate: no clear difference (density minus pruning, +2.7 points [−5.4, +10.8]).
- They clearly see more variety: viewing one image at a time, they found density's outputs more varied in 74.5% of trials [65.1, 83.3] (reply to Reviewer 2HRr, A2).
Following your suggestion, we have added both results to Section 6.5 and Appendix I.2 (Table 35) of the revision.
We sincerely appreciate your time and thoughtful feedback (1/4)
Dear Reviewer vN8W,
We sincerely appreciate your time and thoughtful feedback. We are glad that you found the research question meaningful and the empirical results encouraging. In light of your suggestions, we have added new experiments and expanded discussions; results marked [new] are in the revised PDF, and all changes are marked in blue. Please find our detailed point-by-point responses below.
R1/A1. Thank you for raising this point. The proof is indeed short; the value of the lemma lies in the question it answers, which had been overlooked for a systematic reason. The question falls between two literatures. Set-diversity methods return the whole set, so they are rightly judged on the kept set; search methods return one output, but it is the best candidate they find. Neither asks whether a more diverse kept set changes the one delivered output, and set metrics cannot tell: selectors that keep the champion keep sets about as diverse as free ones, yet convert less than half as much of that gain into output diversity (Appendix F.2). To our knowledge, we are the first to measure this. The lemma is also general: it needs only a midway ranking and a final argmax, so it applies to any pipeline that prunes on partial scores and keeps one best survivor. It holds exactly on a language model (Appendix H), and evolutionary program search has the same structure (Section 7).
Three non-obvious results.
A quantity that had not been measured. The locked share is the fraction of requests on which the preview champion also has the best final reward. It is large, 49–72% on our four backbones, and it shows in what users receive: selectors that must keep the champion turn only 6–22% of their set-diversity gain into output diversity, against 26–66% for free ones.