acceptodds
Under review as a conference paper at ICLR 2027

From Set Diversity to Output Diversity: Progressive Selection for Inference-Time Alignment

Abstract

Inference-time alignment often samples several candidates and prunes them progressively: a reward model scores a preview of each candidate partway through generation, and only the best few, the survivors, are finished. Such selection is known to reduce diversity. Existing diversity methods for selection make the kept set diverse, but when a user receives one sample per request, what matters is output diversity, not set diversity. We characterise how the two relate and find that one candidate, the *champion* whose preview scores best, largely decides whether a more diverse set changes what the user receives. Selectors that keep the champion turn only 6% to 22% of their gain in set diversity into output diversity, against 26% to 66% for selectors free to drop it, and this *champion constraint* binds on 49% to 72% of requests, a share we prove rises with preview quality and falls with pool size. We propose champion-free selection and instantiate it with exact max-sum diversification and with density-aware selection, a single sort that penalises crowding in the candidate pool. On four text-to-image models, both selectors raise output diversity over reward-only pruning at the same budget while keeping 86% to 96% of its reward gain, and requiring them to keep the champion removes more than half of that increase. Density also delivers rare modes 7% to 14% more often than max-sum, as our analysis predicts, and blinded raters prefer its variety in 72.5% of comparisons on SD3.5 and 70.3% on FLUX.2 Klein.

Primary area:
generative models
Keywords:
density based collapse, kernel, set diversity, inference-time alignment, diffusion models, test-time scaling, diverse generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vN8WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the gap between set diversity and output diversity in reward-guided progressive selection for inference-time alignment. The authors identify the champion constraint, showing that retaining the highest-scoring preview candidate can prevent improvements in set diversity from translating into more diverse final outputs. To address this, they propose champion-free selection strategies, including exact max-sum diversification and density-aware selection. Experiments across four image generators demonstrate improved output diversity and rare-mode coverage while largely preserving generation quality.

Critical strengths
  1. The paper highlights an important limitation of conventional diversity-aware selection: increasing survivor-set diversity does not necessarily improve the diversity of delivered outputs. The champion constraint provides a clear explanation, supported by theoretical analysis and empirical measurements.

  2. The proposed champion-free strategies improve output diversity across four generative models with minimal computational overhead. The experiments include matched-quality comparisons, targeted ablations, rare-mode analysis, and human evaluations, providing substantial evidence for the practical effectiveness of the approach.

Critical weaknesses
  1. The champion constraint follows directly from the deterministic final selection rule: if the preview champion is also the final reward winner, retaining it necessarily fixes the output. While useful, this observation is relatively straightforward. The additional theoretical analysis relies on simplifying assumptions, leaving the broader significance and generality of the theoretical contribution somewhat unclear.

  2. The paper demonstrates the benefits of champion-free selection, but the additional value of the specific density-aware criterion is less convincing. In particular, density and exact max-sum achieve comparable mean output diversity at matched reward, while density’s main advantage is a relatively modest improvement in rare-mode coverage. It remains unclear whether simpler champion-free strategies could achieve similar quality–diversity trade-offs.

Questions for authors

Q1. Beyond the direct observation that retaining the final winner fixes the output, what are the main nontrivial insights provided by the champion constraint analysis? How robust are the locked-share predictions beyond the i.i.d. Gaussian assumptions?

Q2. Could the authors further clarify the advantages of density-aware selection over simpler champion-free alternatives, such as stochastic reward-based selection or controlled champion exclusion, under matched reward budgets?

Q3. How well does the human-preference tolerance calibrated on ImageRewardDB generalise to the evaluated generators and prompts? Can the authors provide stronger evidence that the increased diversity does not introduce meaningful perceptual quality degradation?

Q4. Since rare-mode coverage is the main advantage of density over exact max-sum, how sensitive are the conclusions to the feature representation and definition of rare modes? What evidence supports the practical significance of the reported 7%–14% relative improvement?

I find the research question meaningful and the empirical results encouraging. I am open to raising my score if the authors can adequately address these concerns during the rebuttal.

Post-Rebuttal:

Overall, I think the rebuttal has adequately addressed my major empirical concerns, and I am inclined to raise my rating from weak reject to weak accept.

6 comments
AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (1/4)

Dear Reviewer vN8W,

We sincerely appreciate your time and thoughtful feedback. We are glad that you found the research question meaningful and the empirical results encouraging. In light of your suggestions, we have added new experiments and expanded discussions; results marked [new] are in the revised PDF, and all changes are marked in blue. Please find our detailed point-by-point responses below.

W1. The champion constraint follows directly from the deterministic final selection rule: if the preview champion is also the final reward winner, retaining it necessarily fixes the output. While useful, this observation is relatively straightforward. The additional theoretical analysis relies on simplifying assumptions, leaving the broader significance and generality of the theoretical contribution somewhat unclear.

Q1. Beyond the direct observation that retaining the final winner fixes the output, what are the main nontrivial insights provided by the champion constraint analysis? How robust are the locked-share predictions beyond the i.i.d. Gaussian assumptions?

R1/A1. Thank you for raising this point. The proof is indeed short; the value of the lemma lies in the question it answers, which had been overlooked for a systematic reason. The question falls between two literatures. Set-diversity methods return the whole set, so they are rightly judged on the kept set; search methods return one output, but it is the best candidate they find. Neither asks whether a more diverse kept set changes the one delivered output, and set metrics cannot tell: selectors that keep the champion keep sets about as diverse as free ones, yet convert less than half as much of that gain into output diversity (Appendix F.2). To our knowledge, we are the first to measure this. The lemma is also general: it needs only a midway ranking and a final argmax, so it applies to any pipeline that prunes on partial scores and keeps one best survivor. It holds exactly on a language model (Appendix H), and evolutionary program search has the same structure (Section 7).

Three non-obvious results.

  1. Better previews make champion-keeping selectors weaker. One would expect a more accurate preview to help selection. Instead, it raises the locked share (Proposition 1), so selectors that keep the champion change less. With a perfect preview, they return exactly pruning's output, while champion-free selectors still gain 0.007–0.017 DreamSim at a cost of up to 0.038 ImageReward, within tolerance (Appendix F.6). So as generative models improve and their previews get better, letting the selector drop the champion matters more.
  2. The gap between greedy and exact max-sum is not a search problem. One may expect the gap between greedy and exact algorithms to come from searching. In our setting, it does not. At k=2, greedy max-sum returns the best pair that contains the champion (Appendix A.1), so it differs from exact max-sum only in keeping the champion. Density does not search at all, yet keeping the champion hurts it too. When the preview and final rewards are the same, requiring the champion removes over half of density's gain over pruning. When they differ, it still removes 46% on average (Appendix F.3). So the fix is not a better solver but simply letting the selector drop the champion, at the same budget.
  3. Champion-keeping selectors change the output just as often, but on requests where a change adds less. Since they cannot act on locked requests, one would expect them to change the delivered image less often. At matched reward, however, they replace pruning's image about as often as free selectors, by changing more unlocked requests, where a new image adds less diversity. So the extra diversity of free selectors comes from the locked requests: in 23 of the 25 resolved comparisons, their share of the gap is consistent with 100% (Appendix F.4). What matters, then, is not how often a selector changes the output but which requests it may change, which is why we measure the locked share.

A quantity that had not been measured. The locked share is the fraction of requests on which the preview champion also has the best final reward. It is large, 49–72% on our four backbones, and it shows in what users receive: selectors that must keep the champion turn only 6–22% of their set-diversity gain into output diversity, against 26–66% for free ones.

AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (2/4)

On the simplifying assumptions (R1/A1, continued). To clarify, they are used only by the model that predicts how the locked share changes with preview quality and pool size. The share itself is computed directly from any pipeline's preview and final scores, with no model. We test the model rather than just assuming it:

  • No fitting. With no fitted parameter, it matches 11 of 12 measured shares on images within 0.02, and all 8 on text within 0.015. [new] With different preview and final reward models, it matches all 10 shares within 0.017.
  • Independence holds by design, because each candidate starts from its own random seed.
  • The Gaussian form matters less than it seems. The locked share depends only on how candidates are ranked, not on the scale or shape of their scores. The Gaussian form only fixes how preview and final rankings move together, and we measured this directly: the observed share is within 0.009 of the Gaussian prediction, while other dependence structures (copulas) would shift it by up to 0.15 (Appendix F.3).
  • The rare-mode model is simple on purpose (one common and one rare mode), and both of its testable predictions hold (R2/A2, rare-mode coverage).

In the revision, a short paragraph after the lemma (end of Section 4) states its generality and the first result, Appendix F.3 adds the robustness tests above, and the [new] cross-reward points appear in blue in Figure 4 (Section 6.3).


W2. The paper demonstrates the benefits of champion-free selection, but the additional value of the specific density-aware criterion is less convincing. In particular, density and exact max-sum achieve comparable mean output diversity at matched reward, while density’s main advantage is a relatively modest improvement in rare-mode coverage. It remains unclear whether simpler champion-free strategies could achieve similar quality–diversity trade-offs.

Q2. Could the authors further clarify the advantages of density-aware selection over simpler champion-free alternatives, such as stochastic reward-based selection or controlled champion exclusion, under matched reward budgets?

R2/A2. Thank you for the insightful question. We agree that density and exact max-sum tie on mean output diversity, but they both follow the paper's main principle that a selector must be free to drop the champion. Beyond this principle, density adds three things, which we detail below.

It beats simpler rules that also drop the champion. [new] Following your suggestion, we have added comparisons with both alternatives you named, stochastic reward-based selection and controlled champion exclusion, in all six backbone–prompt-set settings. At matched ImageReward, density delivers more output diversity and more rare modes than each of them:

Density minus …DreamSim (×10⁻³)Rare-mode share (points)Intervals above zero
Controlled champion exclusion: random drop, then top-k [new]+3.1 to +15.3+3.7 to +8.212/12
Stochastic reward-based selection: reward-weighted sampling+2.9 to +13.6+3.6 to +7.912/12
Controlled champion exclusion: same drops as density, reward-weighted refill+1.6 to +10.1+1.9 to +5.312/12

Dropping the champion at random recovers only 11–25% of density's gain over pruning at the same reward. So dropping the champion is necessary but not sufficient: most of the gain comes from which candidate takes its place. k-DPP sampling, which can also drop the champion, likewise falls below density at matched reward on all four backbones (Section 6.2). The full table is the new Table 30 (Appendix F.5).

Better rare-mode coverage. At matched reward and equal mean diversity, density delivers rare modes 7% and 14% more often than max-sum on the two prompt sets (+1.6 and +3.1 percentage points; intervals exclude zero). The advantage appears where Proposition 2 predicts: about three quarters of it comes from requests with at least two rare candidates. It is positive under 119 of 120 cluster definitions (Appendix D.6) and grows with pool size, from 0.4 to 2.3 points between 6 and 16 candidates. Our main setting uses six candidates, so it shows the effect near its smallest.

It still acts in settings where max-sum cannot. Max-sum rewards spread among the images it keeps, so with one survivor it keeps the best preview. Density instead marks down any candidate that looks like many others in the pool. So even with one survivor, it can skip a champion from a crowded mode and keep a good image from a rarer one. Different requests then deliver different modes more often. [new] In all six settings, at the same budget, this raises output diversity and the rare-mode share (by 4.9–9.9 points) over keeping the best preview, at a small reward cost within tolerance (Section 6.4, Appendix E.2). We hope this clarifies what density adds beyond dropping the champion.

AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (3/4)

Q3. How well does the human-preference tolerance calibrated on ImageRewardDB generalise to the evaluated generators and prompts? Can the authors provide stronger evidence that the increased diversity does not introduce meaningful perceptual quality degradation?

A3. Thank you for this important question. In short, on SD3.5 our own control pairs agree with the ImageRewardDB calibration, and a new human study on our generators finds no clear quality difference.

The reward cost is small. On the four backbones, density keeps 86–96% of the reward that pruning gains over native sampling: it gives up at most 0.029 ImageReward, while pruning gains 0.21–0.28. PickScore and CLIP, which the selector never sees, fall by at most 0.002 and 0.001. When PickScore selects instead, ImageReward becomes the independent judge and falls by at most 0.032.

What this means for people. The paper's tolerance is the score gap at which people prefer the higher-scoring image 51% of the time on held-out ImageRewardDB pairs, a preference so slight that detecting it would take about 20,000 paired judgements. Every loss above is within it (0.029 against 0.050 for ImageReward; 64% of tolerance when PickScore selects). Even the most conservative bound in the paper corresponds to a preference of at most 51.3% for pruning's image, under the ImageRewardDB calibration.

Does the calibration transfer to our generators? [new] On SD3.5, yes: the control pairs of our strip study (Section 6.5, native sampling against pruning; Appendix I.1) agree with it. On ImageRewardDB, preference for the higher-scoring image stays close to 50% up to a gap of 0.2 ImageReward and rises beyond it. Our raters behave the same way: they favour the higher-reward strip once the gap exceeds 0.2 (6 of 6 quality calls) and show no preference below it. All 100 density-versus-pruning strips lie below 0.2 (mean gap 0.025, largest 0.161).

[new] A direct quality test on our generators. To provide this stronger evidence, four new non-author raters, all professional annotators, compared density's image with pruning's image for the same request on SD3.5 and Klein (120 pairs per rater, stratified by reward gap).

  • They can see quality differences: on curated high-gap control pairs they chose the higher-reward image 98.8% of the time, and all four passed the attention and repeat checks.
  • No clear difference between density and pruning: over all requests, they preferred pruning's image in 53.1% of cases [49.6, 56.5], an interval that includes both 50% and the 51.3% reading above.
  • Off-prompt rate: no clear difference (density minus pruning, +2.7 points [−5.4, +10.8]).
  • They clearly see more variety: viewing one image at a time, they found density's outputs more varied in 74.5% of trials [65.1, 83.3] (reply to Reviewer 2HRr, A2).

Following your suggestion, we have added both results to Section 6.5 and Appendix I.2 (Table 35) of the revision.

Reviewer_2HRrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies progressive pruning: NN candidates are previewed at a midpoint, a reward model keeps the best kk, and one survivor is delivered. It argues that making the survivor set diverse barely helps, because the request returns a single image. The explanation is the champion constraint: if the preview champion also wins on final reward — a "locked" request — any selector that keeps the champion delivers it, whatever its objective, and the locked share is measured at 49–72%; a Gaussian model predicts that share from preview accuracy and pool size with no fitted parameter, matching 11 of 12 measured cases within 0.02. The proposed fix is champion-free selection, instantiated as exact max-sum and as density-aware selection, a single sort on reward minus log crowding in the whole pool; both raise output diversity over reward-only pruning at fixed budget while keeping 86–96% of the reward gain, and requiring the champion removes more than half of that gain.

Critical strengths
  • The paper identifies the right variable. Locking is formalized independently of objective, solver and features, then measured, then predicted out of sample with no fitted parameters across backbones, checkpoints and a second prompt bank, and replicated on a language model.
  • The champion intervention is a genuine control: with k=2k{=}2 the "required" version coincides with greedy max-sum, so only the constraint changes. The reward cost is not hidden — it is calibrated against a human tolerance, checked with two non-selecting judges, and shown to survive an oracle preview.
Critical weaknesses
  • The human evaluation measures a different question than the one the paper argues for. Raters judge eight-image strips side by side, which is a set-level simultaneous judgment — the very regime the paper says does not matter, and only two raters means the effective sample for rater variance is two.
  • The methodological novelty is thin, so the measurement must carry the paper. Lemma 1 is near-tautological, Proposition 1 is the classical probability that the argmax of two correlated Gaussians coincides, max-sum is known, density is fitness sharing, and "exact" max-sum enumerates 15 subsets. Headline effects are 4–8% relative and come from one configuration (midpoint, 6→2, ImageReward for preview and final).
Questions for authors
  1. Preview and final reward are both ImageReward, which inflates ρ\rho and hence the locked share. What is LL when the two rewards come from different models, and does the gain persist?
  2. Can you report a sequential single-image protocol rather than strips, plus a CI for 72.5% that accounts for the two-rater structure?
  3. At k=1k=1 the constraint is total and the comparison is density top-1 versus reward top-1. Does density beat best-of-NN there — the cheapest and most likely deployment?
4 comments
AuthorsOpenReview ↗

We are very grateful for your insightful and constructive feedback (1/3)

Dear Reviewer 2HRr,

We are very grateful for your insightful and constructive feedback. We are glad that you found that the paper identifies the right variable and that the champion intervention is a genuine control. Each of your three questions led to a new analysis; results marked [new] were produced for this discussion and are in the revised PDF, where all changes are marked in blue. Please find our detailed point-by-point responses below.

W1. The human evaluation measures a different question than the one the paper argues for. Raters judge eight-image strips side by side, which is a set-level simultaneous judgment — the very regime the paper says does not matter, and only two raters means the effective sample for rater variance is two.

R1. Thank you for raising this point. We would like to clarify that the strips show output diversity, not a kept set: each strip is the eight images delivered to eight independent requests of one prompt, one image per request, taken from the main-benchmark selections. This is exactly the paper's definition of output diversity. No rater saw candidates that survived together; showing the eight delivered images side by side is only a way to display them at once. The judgements also track DreamSim on these delivered images: on SD3.5, raters favour density's strip in 80–95% of judgements once it adds more than 0.005 (Appendix I).

On the number of raters, A2 gives each rater's own interval and an interval that resamples prompts and raters together. On side-by-side viewing, A2 also reports a [new] study with four more raters who saw the images one at a time, as you suggested.


W2. The methodological novelty is thin, so the measurement must carry the paper. Lemma 1 is near-tautological, Proposition 1 is the classical probability that the argmax of two correlated Gaussians coincides, max-sum is known, density is fitness sharing, and "exact" max-sum enumerates 15 subsets. Headline effects are 4–8% relative and come from one configuration (midpoint, 6→2, ImageReward for preview and final).

R2. Thank you for this careful assessment. Each proof is short; the value lies in the question the results answer: whether a more diverse kept set changes the one delivered output, which to our knowledge had not been measured. Specifically:

  1. A quantity in a new role. The locked share L is the probability that the preview picks the final winner; here it is also the share of requests no champion-keeping selector can change. It is 49–72% on our backbones and can be read from any pipeline's scores, whatever the objective, solver or features.
  2. Its cost to users. Selectors that keep the champion turn only 6–22% of their set-diversity gain into output diversity, against 26–66% for selectors free to drop it. So a method judged on its kept set can look much better than what users receive.
  3. A prediction with no fitted parameter. Proposition 1 gives L from preview accuracy and pool size, within 0.02 in 11 of 12 settings; A1 extends it to ρ = 0.25–0.35 (residuals ≤ 0.017).
  4. [new] Where the set gain goes. Exact max-sum raises set diversity, but about half of that gain (45–62%) only pushes the two kept images apart from each other, not away from other requests' images. A user who receives one image never sees this half. The candidate pool alone, before any selector runs, predicts this share with a mean absolute error of 0.025 (Appendix F.2).

The principle follows: on a locked request, a selector changes what the user receives only if it may drop the champion.

Effect size. The natural scale is what pruning removes. Output diversity (DreamSim):

BackboneNativePruningDensity
Klein0.18000.16710.1796
SD3.50.24950.22300.2414

Density keeps 86% and 90% of pruning's ImageReward gain on these two backbones. Across the four backbones, pruning removes 5–11% of native output diversity. That is the same order as what published diversity samplers add over native sampling at matched quality on non-distilled models (≈2–8%: Particle Guidance, CNO). Density recovers half to nearly all of it at the same budget, without changing the sampler.

Not one configuration.

  • Pool size and survivors: free selectors convert more of their set gain into output diversity than champion-keeping ones in all 24 (N, k) settings up to N=16, k=4 (Appendix G).
  • Prompt suites: both selectors beat pruning on two further suites, 12 of 12 (Appendix D.3).
  • Rewards: PickScore can select (Appendix D.2); different preview and final rewards are in A1.
  • Schedules: [new] a two-cut 8→4→2 schedule at budget 4 keeps the gain over pruning in 10 of 10 settings (+8 to +20 ×10⁻³ DreamSim); requiring the champion removes 30–46% of it (Appendix F.3).
  • k=1: see A3.
AuthorsOpenReview ↗

We are very grateful for your insightful and constructive feedback (2/3)

Q1. Preview and final reward are both ImageReward, which inflates ρ and hence the locked share. What is L when the two rewards come from different models, and does the gain persist?

A1. Thank you for this insightful question. [new] Yes, a shared reward does raise L, and Proposition 1 predicts by how much. To test this, we replayed every pairing of ImageReward and PickScore as preview and final reward on the five midpoint settings (Infinity has no preview PickScore).

Preview → final rewardLocked share L
ImageReward → ImageReward0.49–0.60
PickScore → PickScore0.45–0.54
ImageReward → PickScore0.24–0.31
PickScore → ImageReward0.25–0.29
Chance0.17

With no fit, Proposition 1 predicts all ten cross-reward values within 0.017, at ρ = 0.25–0.35, which extends the law to a new range. The largest same-reward residuals are 0.009 and 0.015. The revision adds the ten points to Figure 4 and the full table to Appendix F.3 (Table 27).

The gain persists. Free selectors beat pruning in 20 of 20 cross-reward comparisons, by +6.8 to +23.3 ×10⁻³ DreamSim. Density's gain over pruning is larger than in the shared-reward setting in nine of ten.

Requiring the champion still costs a large part of the gain. Free selectors beat their champion-keeping versions in 19 of 20 cross-reward comparisons, at their main-paper strengths. The requirement removes 37% of the gain on average (24–55% IR→Pick, 7–56% Pick→IR), against 52–75% when ImageReward scores both stages, as in the paper. The cost moves with L, as the lemma implies. This gives a design rule: the higher L is in a pipeline, the more it matters to let the selector drop the champion. We are grateful for this question, which led to a useful extension of the paper's main result.


Q2. Can you report a sequential single-image protocol rather than strips, plus a CI for 72.5% that accounts for the two-rater structure?

A2. Thanks for the great suggestion. [new] We report both below.

Intervals that account for the two raters.

SD3.5Klein
Strips where density's variety is preferred72.5%70.3%
Interval, prompts resampled (as in Appendix I)[65.5, 79.0][62.5, 77.5]
Interval, prompts and raters resampled[65.3, 79.5][62.0, 78.0]
Rater 1 / rater 272.5% / 72.5%71.5% / 69.0%
Difference between raters0.0 [−4.5, +5.0]+2.5 [−4.0, +9.0]

Each rater's own interval excludes 50%. Two blinded raters each saw the difference, and they barely disagreed (Appendix I.1, Table 34).

The sequential protocol you suggested. Four new non-author raters (professional annotators) rated 48 Appendix I prompts where the two rules deliver different images. Each trial shows the eight images delivered to eight independent requests one at a time, with no going back, then the other rule's eight for the same prompt, and asks which sequence showed more variety.

  • Result: raters chose density's sequence in 74.5% of trials [65.1, 83.3], resampling prompts and raters (rater variance now has three degrees of freedom): 76.6% on SD3.5 and 72.4% on Klein. Each rater's own interval is above 50%.
  • Agrees with the strips: on the same prompts the Appendix I strips gave 84.4% and 79.2%, so the two viewing modes agree within sampling error (difference −7.3 points [−15.9, +1.6]).
  • Either order: density's sequence is preferred whether shown first (79.7%) or second (69.3%).
  • Follows the metric: the share of decisive votes for density rises from 58% to 79% to 97% across terciles of density's DreamSim gain.

So the change in output diversity is visible whether the images are seen together or one at a time. We have added this study to Section 6.5 and Appendix I.2 of the revision.

AuthorsOpenReview ↗

We are very grateful for your insightful and constructive feedback (3/3)

Q3. At k=1 the constraint is total and the comparison is density top-1 versus reward top-1. Does density beat best-of-N there — the cheapest and most likely deployment?

A3. Thanks for this very practical question. [new] Yes: density beats reward top-1 at the same budget, and best-of-2 and best-of-3 at matched reward.

Against reward top-1, at the same budget (3.5 trajectories on the diffusion backbones, 2.55 on Infinity). Density top-1 wins on all six settings: +6.3 to +21.4 ×10⁻³ DreamSim and +4.9 to +9.9 rare-mode points, giving up 0.014 to 0.033 ImageReward (β = 1/4); all intervals exclude zero. Max-sum has no spread term at k=1 and reduces to reward top-1, so density is the only one of our two selectors that acts there (Section 6.4; Appendix E.2, Table 19).

Against best-of-N, at matched reward:

  • Best-of-2: density is above on all six settings, +10.7 to +25.9 ×10⁻³ DreamSim and +10.6 to +17.4 rare-mode points (3.5 trajectories against 2).
  • Best-of-3: above in rare-mode share on all five settings that reach its reward, and above or level in DreamSim.
  • Best-of-4: above on Infinity at 2.55 trajectories against 4 (+5.8 [+3.4, +8.1]) and below on FLUX.1 at 3.5 against 4 (−4.9 [−9.1, −0.9]). On Klein and SD3.5, best-of-4's reward is out of reach at k=1, since even reward top-1 falls short of it.

Why the paper uses k=2. k=1 saves half a trajectory by skipping the final comparison, and the high-reward end needs that comparison. At k=2 (budget 4), both selectors' curves pass above best-of-4 on every backbone at equal or higher reward.

Thanks again for the time and care you invested in reviewing our work; your suggestions have been very helpful. We hope our responses address your concerns, and we would be grateful if you would take them into account in your final evaluation. If any part of our responses remains unclear, please feel free to let us know, and we will be glad to clarify further.

Hi, thank you very much for taking the time to review our manuscript. Your valuable comments are greatly appreciated and have helped us improve the quality of our work.

As the author-reviewer discussion period is nearing its end, we would sincerely appreciate it if you could take a moment to review our rebuttal and let us know whether we have adequately addressed your concerns. This would also give us an opportunity to respond to any further questions or concerns you may have. Thank you again for your time and consideration!

Reviewer_uRCNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an important distinction between set diversity and output diversity in inference-time alignment for generative models. While existing diversity-aware selection methods typically aim to maintain a diverse set of candidates during generation, many practical inference pipelines ultimately return only a single output per request. Consequently, diversity among retained candidates does not necessarily translate into diversity among the outputs delivered to users.

The authors identify a phenomenon termed the champion constraint. Specifically, when the candidate with the highest intermediate reward also has the highest final reward, any selection strategy that preserves this candidate will necessarily return the same output, regardless of the diversity of the other retained candidates.

To address this limitation, the authors propose champion-free selection, which allows the highest-scoring intermediate candidate to be discarded. Two selection strategies are investigated: exact max-sum diversification, which jointly optimizes candidate reward and pairwise diversity, and density-aware selection, which penalizes candidates located in densely populated regions of the candidate pool.

Critical strengths

S1. The paper identifies a meaningful and underexplored distinction between set diversity and output diversity. In many practical applications, the user receives only one result rather than the entire set of candidates. Therefore, optimizing the diversity of retained candidates may be insufficient if the final selection process repeatedly favors the same type of output.The paper explicitly formulates output diversity as a property of the final output distribution across independent requests, rather than relying solely on diversity within candidate sets.

S2. The proposed champion-free selection strategy addresses the identified limitation. The proposed design is conceptually straightforward: rather than always preserving the candidate with the highest intermediate reward, the selector is allowed to discard it when doing so improves diversity. The density-aware formulation measures local crowding over the entire candidate pool instead of optimizing only the diversity of the retained subset. This provides a plausible mechanism for increasing the probability that candidates from less frequent modes become final outputs.

Critical weaknesses

W1. The practical interpretation of output diversity remains dependent on the choice of perceptual representation. The main diversity evaluation uses DreamSim, while the proposed density-aware selector relies on DINOv2 feature distances. Although these are reasonable representation choices, greater distance in a perceptual embedding space does not necessarily imply greater semantic diversity or better coverage of the different meaningful interpretations of a prompt. For example, variations in background, visual composition, color, or rendering style could increase perceptual diversity without substantially changing the generated semantic content.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

We truly appreciate your thoughtful and constructive comments (1/2)

Dear Reviewer uRCN,

We truly appreciate your thoughtful and constructive comments. We are glad that you found the distinction between set diversity and output diversity meaningful and underexplored. Following your suggestion, we have added a new blinded check and expanded discussions; results marked [new] were produced for this discussion and are in the revised PDF, where all changes are marked in blue. Please find our detailed response below.

W1. The practical interpretation of output diversity remains dependent on the choice of perceptual representation. The main diversity evaluation uses DreamSim, while the proposed density-aware selector relies on DINOv2 feature distances. Although these are reasonable representation choices, greater distance in a perceptual embedding space does not necessarily imply greater semantic diversity or better coverage of the different meaningful interpretations of a prompt. For example, variations in background, visual composition, color, or rendering style could increase perceptual diversity without substantially changing the generated semantic content.

Questions for authors: See weakness

R1. Thank you for raising this important point. We fully agree that perceptual and semantic diversity are different, and a new blinded check shows that density's added variety includes what the images depict. Section 3 defines output diversity as the distance between the images that two independent requests of one prompt receive. Below we report that check, explain why layout and style variety also matters, and show that the gains do not depend on the selector's own feature space.

[new] A blinded judge asked what varies, attribute by attribute. For each prompt, we showed a vision-language model (Claude Haiku 5.5) two sets of eight images: the images density delivers to eight separate requests, and the images pruning delivers. The judge was not told which method made which set. For each of seven attributes (subject, count, pose, composition, background, colour, style), it said which set varies more, or that the two are the same. We ran this on every main-benchmark prompt where the two sets differ (170 on SD3.5, 182 on Klein), twice per prompt with the sets swapped. A second pass asked about subject, count and pose only, so that the content answer cannot simply follow the judge's overall impression. The table gives how often the judge chose density's set when it saw a difference; 50% would mean no difference:

AttributeSD3.5Klein
Content: subject, count, pose75.5% [65.4, 84.6]71.6% [61.5, 80.6]
Layout: composition, background68.0% [59.4, 76.0]68.4% [59.7, 76.7]
Appearance: colour, style72.0% [62.7, 80.4]60.5% [49.6, 70.7]
Content, asked about alone77.0% [68.2, 84.9]64.5% [54.9, 74.2]

So on both backbones density adds variety in what the images depict, not only in layout or style, including when the judge is asked about content alone. Content differs between the two sets less often than layout does, as expected when a prompt fixes its subject, but when it differs, density's set is usually the more varied one. The judge is reliable on the checks we can run: it saw no difference on any of 30 sets that repeat the same images in a new order, and it agreed with the human raters in 68 of its 78 decisive calls where both raters chose the same set. In light of your suggestion, we have added this check to Section 6.5 and Appendix I.3 (Table 36) of the revision.

Layout and style variety also matters to users. A user who receives one image per request never sees a set; what they notice is whether repeated requests return the same picture, and a new background or composition already makes it a different picture. Blinded raters do notice: they prefer density's variety in 72.5% (SD3.5) and 70.3% (Klein) of comparisons, more often where the DreamSim gain is larger (Appendix I). [new] Four further raters, shown one image at a time, found density's outputs more varied in 74.5% of trials [65.1, 83.3].

AuthorsOpenReview ↗

We truly appreciate your thoughtful and constructive comments (2/2)

The gain is not measured in the selector's own space. The selector reads DINOv2-L preview features, while evaluation uses DreamSim on the final images. Since DreamSim also uses DINO features, Appendix D.4 re-embeds every delivered image with OpenCLIP ViT-g/14, which the selector does not use. OpenCLIP is trained to match images with captions, so it reflects what an image depicts more than its low-level appearance. The appendix also reports the Vendi score on a DreamSim kernel and LPIPS. Both selectors raise output diversity over pruning under all four measures on every backbone: all 32 intervals are above zero, OpenCLIP included.

The rare-mode gain does not depend on DreamSim. Density's rare-mode advantage over max-sum is defined by clustering. With clusters refit in OpenCLIP space it is +1.5 points, against +1.6 in DreamSim space. It is positive in 119 of 120 cluster definitions, and a cluster-free rarity measure gives the same sign in all 128 pooled comparisons, 90 with intervals above zero (Appendix D.6).

Thanks again for the time and care you invested in reviewing our work; your suggestions have been very helpful. We hope our responses address your concerns, and we would be grateful if you would take them into account in your final evaluation. If any part of our responses remains unclear, please feel free to let us know, and we will be glad to clarify further.

Hi, thank you very much for taking the time to review our manuscript. Your valuable comments are greatly appreciated and have helped us improve the quality of our work.

As the author-reviewer discussion period is nearing its end, we would sincerely appreciate it if you could take a moment to review our rebuttal and let us know whether we have adequately addressed your concerns. This would also give us an opportunity to respond to any further questions or concerns you may have. Thank you again for your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

We sincerely appreciate the time, effort, and thoughtful feedback

Dear Reviewers, AC, and SAC,

We thank the reviewers for their careful reviews. All three reviewers now recommend acceptance. Reviewer vN8W raised their rating after the rebuttal, writing that it "adequately addressed my major empirical concerns".

Strengths the reviewers highlighted:

  • An important, overlooked problem: "a meaningful and underexplored distinction" between set and output diversity (uRCN); "an important limitation of conventional diversity-aware selection" (vN8W).
  • A clear, predictive explanation: the champion constraint "provides a clear explanation" (vN8W); the paper "identifies the right variable" and predicts it "with no fitted parameters" (2HRr).
  • Rigorous evidence: “substantial evidence” (vN8W); the champion intervention is “a genuine control”, and the finding is “replicated on a language model” (2HRr).

The paper in brief:

Progressive selection previews each candidate partway through generation, finishes the best few and delivers the one with the highest final reward. The champion is the candidate with the best preview score. On 49–72% of requests (locked requests) it also wins at the end, so any selector that keeps it delivers the same image as reward-only pruning, however diverse the kept set. Selectors that keep the champion therefore turn only 6–22% of their set-diversity gain into output diversity (the variety across independent requests of one prompt), against 26–66% for selectors free to drop it. The resulting rule: a diversity-aware selector must be free to drop the champion. Our selector, density, which penalises crowding in the candidate pool, follows it and raises output diversity on four text-to-image models at the same budget.

How we addressed each concern:

Reviewer vN8W (rating raised)

  • Main concern: what does density add beyond simpler champion-free rules? We compared three such rules in all six settings, including the two the reviewer named. At equal reward, density delivers more output diversity and more rare modes (image types plain sampling seldom produces) than each, with all 36 intervals above zero (Table 30). The reviewer wrote that these comparisons "directly address my main concern".
  • Quality. Four professional annotators, who chose the higher-reward image in 98.8% of large-gap control pairs, found no clear quality difference between the two rules' images (53.1% [49.6, 56.5] for pruning; Section 6.5).
  • Beyond the Gaussian model. With no fitted parameter, our formula for the locked share predicts 11 of 12 measured values within 0.02 and the shares on a language model (App. F.3, H); density's rare-mode advantage over exact max-sum holds under 119 of 120 mode definitions (App. D.6).

Reviewer 2HRr

  • Human protocol. In our original experiment, each strip already showed one image from each of eight independent requests, so the original study already judged output diversity, not a kept set. In our new study asked by the reviewer, four professional annotators, shown one image at a time, found density's outputs more varied in 74.5% [65.1, 83.3] of trials, each annotator's interval above 50%; with raters resampled, the strip intervals also exclude 50% (Section 6.5).
  • Effect size and one configuration. Pruning removes 5–11% of plain sampling's output diversity; density restores half to nearly all of that loss while keeping 86–96% of pruning's reward gain (Table 1). The gains hold with two pruning stages (10 of 10, App. F.3) and on two further prompt suites (12 of 12, App. D.3).
  • Different preview and final rewards. Without refitting, the formula predicts the locked share in this setting too (0.24–0.31), and champion-free selectors beat pruning in all 20 comparisons (App. F.3).
  • One survivor versus best-of-N. With one survivor, density beats keeping the best preview in output diversity and rare modes in all six settings, within the reward tolerance, and beats best-of-2 at matched reward (App. E.2).

Reviewer uRCN

  • Single concern: does perceptual diversity reflect semantic diversity? A blinded vision-language judge chose density's images as more varied in what they depict (subject, count or pose) in 75.5% (SD3.5) and 71.6% (Klein) of content calls, both intervals above 50% (App. I.3). Under four diversity measures, including OpenCLIP, which the selector never uses, all 32 intervals are above zero (App. D.4).

Simple tools, a new result. As 2HRr and vN8W note, the tools are simple: a short proof and a single sort. What they deliver is new: the variable that decides whether diversity reaches the user (the locked share), a formula that predicts it with no fitted parameter across models, rewards and a language model, and a rule every diversity-aware selector can follow: be free to drop the champion (Section 6.3, App. F.3, H).

We believe the revision has resolved the concerns raised in the reviews, and thank the reviewers again for helping strengthen the paper.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.