acceptodds
Under review as a conference paper at ICLR 2027

Explore Before You Optimize: Robust Initialization for Reward-Guided Generation

Abstract

Reward-guided test-time optimization improves generation quality but often reduces diversity by concentrating samples around a limited set of high-reward modes. Recent explore-then-refine methods encourage early exploration before reward-guided refinement, yet this process can be inefficient and the resulting diversity may still degrade during subsequent optimization. We address these limitations by constructing a robust, coverage-oriented initialization population. Building on guidance potential, we introduce a local robustness objective that favors latents located in broad low-potential neighborhoods, where the low-contraction property is more stable under reward-driven perturbations. We further propose a population-level semantic coverage objective that reduces redundancy in the predicted representation space, allowing a limited candidate set to span a broader range of semantic modes. Experiments across multiple benchmarks show that our method achieves competitive generation quality while consistently improving output diversity.

Primary area:
generative models
Keywords:
Generative Models, Diversity Preservation, Reward-Guided Test-Time Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XphyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The approach in the paper aims to preserve diversity in reward-guided text-to-image generation. It does so by selecting initial latents for neighborhood robustness and semantic coverage. The authors reports experiments with SDXL and SD3.5, noting improved diversity with competitive quality.

Critical strengths

I think the paper makes a useful and interesting point, which is that diverse initial noise samples do not necessarily lead to semantically diverse outputs. Additionally, during reward-guided refinement, diversity can actually diminish. The idea to treating initialization as a separate stage is thus interesting and a plausible way to address this issue. To me, the two proposed stages are also intuitive. And the paper actually does show that each stage contributes something.

Overall, an interesting and well written paper.

Critical weaknesses

While the approach in the paper is interesting, there are a few weaknesses that would need to be addressed before acceptance.

For example, the paper calls the measure “average pairwise DINOv2 similarity"; however, higher values are treated as better. Since higher similarity usually means less diversity, the paper should define the metric and its direction. This makes the main results thus a bit difficult to interpret for me.

For me, to reproduce it, the feature mapping used for semantic coverage would need to specified in more detail, e.g. identify parts such as the encoder, layer, and preprocessing, and clarify how it relates to the DINOv2 evaluation metric, etc.

Finally, the paper should clarify what neighborhood robustness and semantic coverage add to guidance-potential initialization. It also seems like the current ablation does not establish gains over the closest prior approach.

Minor points:

  • I found two spelling errors: "denoising trajectorie” and “Further analysis are provided”
  • Space missing between a few approaches and their citations, e.g. "DivGen(Parmar et al., 2026)", "pretrained distribution(Gao et al., 2023; Uehara et al., 2024)" etc.
Questions for authors
  • What exactly is the reported DINOv2 diversity metric? pairwise similarity, distance, or a transformed score such as one minus similarity?

  • What encoder, layer, and preprocessing define the feature mapping in the semantic coverage objective?

  • Could you provide variability estimates across the three runs (tables 1 and 2, etc.)?

  • Novelty appears somewhat incremental, i.e.the method combines local robustness with semantic-coverage selection. This is not necessarily bad but it would be great if the authors could further highlight what is the most unique contribution of the approach.

3 comments
AuthorsOpenReview ↗

We sincerely thank the reviewer for the careful review and thoughtful attention to the experimental details and practical aspects of our work, and we are more than happy to provide further clarifications and additional evidence to address these concerns.

Response to Weaknesses 1

We thank the reviewer for pointing out this inconsistency. This is indeed a typographical error in the description of the metric. Our intention was to evaluate generation diversity using the average pairwise DINOv2 distance, rather than similarity. Specifically, the pairwise distance is defined as \begin{equation} b_{ij}=1-\cos\left(f_{\mathrm{DINO}}(x^{(i)}),f_{\mathrm{DINO}}(x^{(j)})\right) \end{equation} where fDINO(⋅)f_{\mathrm{DINO}}(\cdot) denotes the DINOv2 feature representation. A higher value therefore indicates greater dissimilarity between generated samples and, consequently, higher diversity. The reported results were computed using this distance metric. We apologize for the misleading description and will correct the text in the revised manuscript.

=========================================================================

Response to Weaknesses 2

We instantiate the feature mapping P(⋅)P(\cdot) using a frozen DINOv2 ViT-B/14 encoder with publicly available pretrained weights. Specifically, we extract the normalized CLS token from the final Transformer layer as the global image representation. Input images are converted to RGB, resized to 224×224224\times224 using bicubic interpolation, and normalized using the ImageNet mean and standard deviation. The extracted features are then L2-normalized before constructing the semantic coverage matrix.

=========================================================================

Response to Weaknesses 3

Given the time and computational constraints during the rebuttal period, we conducted three additional independent runs for the experiments in Table 1 and report the results in the form of mean ± standard deviation (Mean ± Std) to provide a better assessment of the variability across different random seeds.

SDXL

MethodIR ↑PickScore ↑HPSv2 ↑GenEval ↑DINOv2 ↑
BoN1.108 ± 0.02122.451 ± 0.1240.291 ± 0.0040.629 ± 0.0120.592 ± 0.011
FKS1.179 ± 0.02821.893 ± 0.1370.294 ± 0.0050.631 ± 0.0140.557 ± 0.012
RBF1.142 ± 0.01922.315 ± 0.1080.299 ± 0.0040.623 ± 0.0110.567 ± 0.013
NTS0.978 ± 0.02421.602 ± 0.1190.295 ± 0.0050.585 ± 0.0130.575 ± 0.014
PSP1.218 ± 0.02222.594 ± 0.1310.295 ± 0.0040.642 ± 0.0150.611 ± 0.012
DivIn1.185 ± 0.02721.954 ± 0.1420.294 ± 0.0050.639 ± 0.0130.735 ± 0.014
CADS1.146 ± 0.02022.106 ± 0.1260.297 ± 0.0040.651 ± 0.0120.711 ± 0.013
DivGen1.151 ± 0.02521.768 ± 0.1180.292 ± 0.0050.635 ± 0.0140.781 ± 0.012
Ours1.216 ± 0.01922.637 ± 0.1150.309 ± 0.0040.679 ± 0.0110.777 ± 0.013

SD3.5

MethodIR ↑PickScore ↑HPSv2 ↑GenEval ↑DINOv2 ↑
BoN1.327 ± 0.02623.204 ± 0.1050.285 ± 0.0040.745 ± 0.0120.617 ± 0.010
FKS1.298 ± 0.02122.458 ± 0.1260.303 ± 0.0050.744 ± 0.0110.589 ± 0.013
RBF1.263 ± 0.02423.211 ± 0.1130.299 ± 0.0040.735 ± 0.0140.595 ± 0.011
NTS1.091 ± 0.01921.918 ± 0.1080.292 ± 0.0050.701 ± 0.0120.604 ± 0.012
PSP1.379 ± 0.02323.081 ± 0.1190.311 ± 0.0040.750 ± 0.0130.629 ± 0.011
DivIn1.318 ± 0.02022.571 ± 0.1320.298 ± 0.0050.751 ± 0.0140.697 ± 0.013
CADS1.311 ± 0.02222.701 ± 0.1140.304 ± 0.0040.760 ± 0.0120.713 ± 0.014
DivGen1.302 ± 0.02522.251 ± 0.1070.301 ± 0.0040.750 ± 0.0130.742 ± 0.012
Ours1.361 ± 0.01822.781 ± 0.0960.316 ± 0.0040.778 ± 0.0100.759 ± 0.011

=========================================================================

Response to Weaknesses 4

Our key contribution is to introduce a new perspective on diversity preservation by shifting its focus from auxiliary objectives during reward optimization to the design of the initial candidate population. Specifically, we aim to construct a population whose diversity is both efficient to establish and robust to subsequent reward-driven updates. To this end, we introduce a local robustness objective to identify candidates in broad low-potential regions, together with a population-level semantic coverage objective to reduce redundancy and preserve complementary semantic modes among the selected candidates.

Thank you for the clarifications, which addressed most of my original concerns. I would still appreciate variability estimates for Table 2 but otherwise I'm happy to increase my score.

AuthorsOpenReview ↗

Thank You for Your Helpful Feedback and Response

Thank you for your willingness to increase your score. We also appreciate your suggestion regarding the variability estimates in Table 2. Given the limited time remaining in the rebuttal period, we will conduct the corresponding experiments and include the variability estimates in the revised version of the paper. We wish you all the best!

Reviewer_BvyqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to improve the diversity of model outputs through a two-stage approach. First, it constructs a set of more favorable initial noise candidates before reward optimization, using guidance potential to favor low-potential regions. Second, it uses early predictions to select four candidates from a pool of eight, reducing potential redundancy. The experimental evaluation is clearly structured and logically coherent, reflecting solid engineering work. However, the additional computational overhead of the method may limit its value in practical deployment.

Critical strengths
  1. The paper is well written, with clear exposition and a coherent narrative.

  2. The experimental evaluation is systematic, covering the main results and visualizations, computational efficiency analysis, ablation studies, experiments assessing complementarity with reward optimization, and hyperparameter analysis.

Critical weaknesses
  1. My main concern is computational cost. Based on an approximate reading of Figure 4, the method requires roughly 5–6% more total generation FLOPs than PSP at the default configuration. On EvalMuse, Table 4 reports relative increases in HPS, EvalMuse, and DINOv2 scores of 4.08%, 1.32%, and 6.75% for SDXL, and 2.29%, 3.61%, and 7.50% for SD3.5, respectively. I am concerned that these improvements may not sufficiently justify the additional computational cost in practical deployment.

  2. I suggest adding preliminary experiments earlier in the paper to demonstrate three aspects: distributional collapse caused by optimization toward high-reward regions, the instability of existing early exploration strategies, and the relationship between lower guidance potential and stronger exploration. Such evidence would provide more convincing empirical support for the motivation.

  3. The ablation study evaluates Stage 1 and Stage 2 only as whole modules. Since Stage 1 combines multiple components, a more convincing analysis would separately ablate the individual components of its objective to clarify their respective contributions.

Questions for authors

If the authors can address the following concerns with additional evidence, I would consider raising my score.

  1. Can the authors provide a compute-matched comparison with PSP to clarify whether the reported gains justify the additional computational cost? In particular, do the advantages persist when both methods use the same total generation FLOPs?

  2. Can the authors provide preliminary empirical evidence demonstrating optimization-induced distributional collapse, the instability of existing early exploration strategies, and the relationship between lower guidance potential and stronger exploration?

  3. Can the authors separately ablate the individual components of the Stage 1 objective to clarify their respective contributions and whether their combination is necessary?

1 comment
AuthorsOpenReview ↗

We sincerely thank the reviewer for the careful review and positive comments. We hope our responses have addressed your concerns, and would greatly appreciate it if you could point out any remaining questions that require further clarification.

Response to Weaknesses 1

Our method improves generation quality and diversity with only a modest additional computational cost. As shown in Fig. 4, we compare our method with N=4N=4 against other methods using larger candidate pools (N=8,12,16N=8, 12, 16). Despite using substantially fewer candidate samples, our method still achieves notable improvements across all evaluation metrics. We believe this comparison provides a practical assessment of the compute-performance trade-off raised by the reviewer. Since the total computational cost also depends on the specific operations performed by each method, precisely matching the total FLOPs is difficult. We therefore use the candidate pool size NN as a practical measure to control the sampling budget.

=========================================================================

Response to Weaknesses 2

We would like to clarify that the three phenomena discussed by the reviewer have been extensively investigated in prior work and constitute established challenges in reward-guided diffusion generation. Previous studies have demonstrated that reward optimization can lead to mode collapse and reduced generation diversity (Jena et al., 2025; Zhang et al., 2024; Chen et al., 2026). Recent work has further studied the exploration-exploitation trade-off in diffusion inference-time scaling, highlighting the difficulty of evaluating early-stage candidates (Su et al., 2025). In addition, DivIn (Li et al., 2026) has established the importance of guidance-potential-aware initialization for improving generation diversity. Building on these findings, our work investigates how to improve the initialization process by accounting for the local robustness of guidance potential and population-level semantic coverage.

=========================================================================

Response to Weaknesses 3

The two components of our Stage 1 objective serve complementary purposes: neighborhood perturbation captures local variations in the guidance potential, while entropic risk aggregation accounts for unfavorable high-potential perturbations. Neighborhood perturbation extends the pointwise evaluation of the original DivIn objective to the surrounding latent space, allowing us to characterize the local behavior of the guidance potential. This is important because a low pointwise potential does not necessarily imply a low potential in its neighborhood. However, minimizing the expected neighborhood potential alone is insufficient, as candidates with the same average potential may exhibit substantially different local distributions. In particular, a candidate may have a low average potential while still containing unfavorable high-potential regions within its neighborhood. To account for such cases, we introduce entropic risk aggregation, which assigns exponentially larger weights to higher-potential perturbations. Following the reviewer's suggestion, we conduct additional ablation experiments to clarify the contributions of the two components.

Stage 1 ObjectiveSDXL HPS ↑SDXL EvalMuse ↑SDXL DINOv2 ↑SD3.5 HPS ↑SD3.5 EvalMuse ↑SD3.5 DINOv2 ↑
DivIn0.3023.1260.6420.3103.3330.626
Neighborhood Mean0.3043.1370.6450.3113.3430.628
Full Objective0.3063.1500.6480.3133.3540.631
Reviewer_osryOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a robust initialization strategy for reward-guided text-to-image generation to improve output diversity while maintaining generation quality. The method consists of two stages: (1) local robustness optimization, which uses a neighborhood-robust guidance potential and Langevin dynamics to identify initial latents with stable low-contraction properties, and (2) semantic coverage selection, which reduces redundancy among candidates in the predicted representation space.

Critical strengths
  • interesting motivation: The paper addresses an important problem in reward-guided generation: improving generation quality while preserving diversity. The idea of building a robust and diverse initial population before reward optimization is interesting.
Critical weaknesses

Weakness 1: Diversity Evaluation Is Weak and Misleading for a Paper Whose Core Contribution Is Diversity

  • The paper states in line 306 that DINOv2 is measured using the average pairwise similarity between generated samples. However, the results show that the proposed method achieves the highest DINOv2 scores, which the authors interpret as indicating greater diversity. Is this not contradicting with the reported metric definition ?

  • The proposed method evaluates diversity using only a single DINOv2-based metric, whereas baselines like DivIn and CADS employ multiple diversity metrics, including Vendi Score, in-batch similarity, recall, coverage... Can the paper provide additional evaluation on similar metrics?

Weakness 2: Algorithm concerns

  • Why do the authors use greedy selection instead of exhaustive search, given that selecting 4 out of 8 candidates involves only 70 possible subsets? How much time does it take to compute Eq. 11? Since these objectives can be computed from the pairwise inner products of the eight precomputed feature vectors, wouldn't evaluating all 70 subsets incur negligible computational overhead?

Weakness 3: Unclear whether robustness is really necessary.

  • The paper argues that U_c(z) can be low while U_c(z+\delta) can be high, where \delta represents changes caused by reward optimization. It then proposes R(z) to find regions where U_c stays low under small changes. However, the current ablation does not clearly show the benefit of this approach over the original U_c from DivIn. The authors could do an ablation similar to Table 3, replacing Stage 1 with DivIn initialization while keeping Stage 2, instead of simply removing Stage 1.
  • Could the authors provide evidence showing that U_c actually changes significantly under small reward-induced perturbations \delta ?
Questions for authors

See Weakness 1 and Weakness 2 and Weakness 3.

Minor questions

  • The methodology's objective function borrow the guidance potential U_c introduced by DivIn and claims that the proposed neighborhood-robust potential favors regions with more stable low-contraction properties. Could the authors provide empirical evidence comparing the correlations of the original U_c and the proposed robust potential with actual trajectory contraction to support this claim?
6 comments
AuthorsOpenReview ↗

We sincerely thank the reviewer for the careful evaluation of our work and the valuable suggestions. We hope that our responses and the ensuing discussion will help clarify the concerns raised.

Response to Weaknesses 1

(1) We thank the reviewer for pointing out this inconsistency. This is indeed a typographical error in the description of the metric. Our intention was to evaluate generation diversity using the average pairwise DINOv2 distance, rather than similarity. Specifically, the pairwise distance is defined as \begin{equation} b_{ij}=1-\cos\left(f_{\mathrm{DINO}}(x^{(i)}),f_{\mathrm{DINO}}(x^{(j)})\right) \end{equation} where fDINO(⋅)f_{\mathrm{DINO}}(\cdot) denotes the DINOv2 feature representation. A higher value therefore indicates greater dissimilarity between generated samples and, consequently, higher diversity. The reported results were computed using this distance metric. We apologize for the misleading description and will correct the text in the revised manuscript.

(2) We thank the reviewer for the suggestion regarding diversity evaluation. We note that most of the suggested aspects are either already covered by our existing evaluation or are not directly applicable to our task, while the Vendi Score provides a valuable additional perspective. Specifically, batch-wise similarity is already captured by the DINOv2 metric reported in our original evaluation. We have therefore supplemented our experiments with the Vendi Score to further assess generation diversity. Regarding Recall and Coverage, these metrics are primarily designed for class-conditional generation, where generated samples are compared against a predefined set of real images or class-specific reference distributions. Such a setting differs from ours, making these metrics less directly applicable to our evaluation protocol.

MethodGenEval SDXL Vendi ↑GenEval SD3.5 Vendi ↑EvalMuse SDXL Vendi ↑EvalMuse SD3.5 Vendi ↑
BoN1.806 ± 0.0141.837 ± 0.0131.752 ± 0.0161.786 ± 0.014
FKS1.861 ± 0.0161.886 ± 0.0121.798 ± 0.0131.824 ± 0.015
RBF1.838 ± 0.0111.869 ± 0.0151.781 ± 0.0151.809 ± 0.012
NTS1.779 ± 0.0181.821 ± 0.0141.724 ± 0.0191.761 ± 0.017
PSP1.853 ± 0.0131.891 ± 0.0161.792 ± 0.0121.831 ± 0.014
DivIn1.926 ± 0.0111.951 ± 0.0141.856 ± 0.0141.876 ± 0.013
CADS1.891 ± 0.0101.922 ± 0.0131.821 ± 0.0111.853 ± 0.016
DivGen1.923 ± 0.0121.969 ± 0.0171.843 ± 0.0151.887 ± 0.012
Ours1.938 ± 0.0131.964 ± 0.0121.864 ± 0.0121.902 ± 0.010

=========================================================================

Response to Weaknesses 2

We adopt greedy selection to improve the scalability of our method to larger candidate pools and varying selection budgets. Specifically, its computational complexity grows polynomially with the candidate pool size, whereas exhaustive search requires evaluating a combinatorial number of subsets. Although exhaustive search is also feasible with our current candidate pool, greedy selection offers greater scalability. For example, when selecting 4 candidates from a pool of 32, greedy selection requires 122 candidate-inclusion evaluations, compared with 35,960 subset evaluations for exhaustive search, reducing the number of objective evaluations by approximately 99.66%. We will clarify this design consideration in the revised manuscript.

AuthorsOpenReview ↗

More Response

Response to Weaknesses 3

(1) Following the reviewer’s suggestion, we conducted an additional ablation by replacing our Stage 1 with the original DivIn initialization while retaining Stage 2. All experiments follow the same settings as our previous experiments to ensure a fair comparison. The results show that DivIn + Stage 2 outperforms Stage 2 only, indicating that initialization based on the original DivIn guidance potential already improves generation quality and diversity. More importantly, Ours + Stage 2 consistently outperforms DivIn + Stage 2 under the same semantic coverage selection strategy, demonstrating the additional benefit of incorporating the local robustness objective Rρ,λ(z)R_{\rho,\lambda}(z) into Stage 1. These results establish the contribution of local robustness beyond the original DivIn initialization.

Stage 1Stage 2SDXL HPS ↑EvalMuse ↑DINOv2 ↑SD3.5 HPS ↑EvalMuse ↑DINOv2 ↑
××0.2903.0140.5880.3043.2180.562
×✓0.2943.0790.6270.3063.2730.609
DivIn✓0.3023.1260.6420.3103.3330.626
✓ (Ours)×0.3013.1320.6350.3093.3160.614
✓ (Ours)✓0.3063.1500.6480.3133.3540.631

(2) From a theoretical perspective, a low potential value at a given latent does not, by itself, imply low potential values in its neighborhood, nor does our formulation require small perturbations to cause substantial potential changes. Rather, the robustness objective explicitly accounts for potential variations in a local neighborhood. To provide an intuitive empirical assessment of this property and directly address the reviewer's concern, we conduct a controlled perturbation analysis using isotropic Gaussian noise to examine whether our initialization achieves lower neighborhood potential under small latent perturbations compared with the original DivIn initialization.

Experimental setup. We perturb each selected latent using Gaussian noise: \begin{equation} \delta \sim \mathcal{N}\left(0,\frac{\rho^2}{d}I_d\right), \qquad z'=z+\delta, \end{equation} where ρ\rho controls the perturbation magnitude. We consider five perturbation scales, ρ∈{0.05,0.1,0.2,0.3,0.4}\rho\in\{0.05,0.1,0.2,0.3,0.4\}, and sample 32 perturbations for each candidate. We evaluate the change in guidance potential: \begin{equation} \Delta U_c(z,\delta) = U_c(z+\delta)-U_c(z). \end{equation} We report two complementary metrics: (1) the mean potential increase, measuring the average change in guidance potential; and (2) the neighborhood potential variance, measuring its sensitivity to local perturbations.

Results and discussion. The results show that our robust initialization consistently exhibits smaller potential increases and lower neighborhood potential variance than DivIn across all perturbation scales. In particular, at the default perturbation scale ρ=0.2\rho=0.2, our method reduces the mean potential increase from 0.01488 to 0.00641 (a 56.9% reduction) and the neighborhood potential variance from 0.000568 to 0.000169 (a 70.2% reduction). A low value of Uc(z)U_c(z) does not necessarily imply that the potential remains low in the neighborhood of zz. By accounting for both the expected potential and its variability under local perturbations, our robust objective favors candidates with more stable neighborhood potential profiles. This provides empirical evidence for the benefit of incorporating local robustness beyond directly optimizing the original DivIn potential.

ρ\rhoDivIn ΔUc\Delta U_cOurs ΔUc\Delta U_cDivIn VarianceOurs Variance
0.050.000990.000390.0000350.000010
0.100.003860.001640.0001420.000042
0.200.014880.006410.0005680.000169
0.300.034120.014380.0013000.000385
0.400.059650.025710.0022990.000695

Thank you for the response.

I still have some remaining concerns:

Weakness 2. The main experiments use 8-choose-4, so the 32-choose-4 example does not directly justify using greedy selection in this setting. If scalability is the main reason, the authors need to show what practical benefits larger pools bring, at what cost, and run experiments at the scale used in their argument. The authors also need to compare exhaustive search with greedy selection in both the paper’s main setting and the 32-choose-4 example, in terms of runtime, objective value, and, if possible, downstream quality and diversity. “Reducing the number of objective evaluations by approximately 99.66%” is not sufficient to justify this choice if the greedy solution is too far from the exact optimum.

Weakness 3 and reliability of improvements. The method’s improvements over the strongest baselines (in the main paper) and DivIn + Stage 2 (in the rebuttal) appear very marginal. For prompts where your method achieves better scores in either quality or diversity, do the generated samples show any interesting characteristics compared with the baselines? Do these improvements or characteristics support the robustness claim, or do they simply reflect a better starting population?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.