From Set Diversity to Output Diversity: Progressive Selection for Inference-Time Alignment

Inference-time alignment often samples several candidates and prunes them progressively: a reward model scores a preview of each candidate partway through generation, and only the best few, the survivors, are finished. Such selection is known to reduce diversity. Existing diversity methods for selection make the kept set diverse, but when a user receives one sample per request, what matters is output diversity, not set diversity. We characterise how the two relate and find that one candidate, the *champion* whose preview scores best, largely decides whether a more diverse set changes what the user receives. Selectors that keep the champion turn only 6% to 22% of their gain in set diversity into output diversity, against 26% to 66% for selectors free to drop it, and this *champion constraint* binds on 49% to 72% of requests, a share we prove rises with preview quality and falls with pool size. We propose champion-free selection and instantiate it with exact max-sum diversification and with density-aware selection, a single sort that penalises crowding in the candidate pool. On four text-to-image models, both selectors raise output diversity over reward-only pruning at the same budget while keeping 86% to 96% of its reward gain, and requiring them to keep the champion removes more than half of that increase. Density also delivers rare modes 7% to 14% more often than max-sum, as our analysis predicts, and blinded raters prefer its variety in 72.5% of comparisons on SD3.5 and 70.3% on FLUX.2 Klein.