acceptodds
Under review as a conference paper at ICLR 2027

Learning Sampling Parameters for Diffusion Models

Abstract

Text-to-image diffusion models expose many inference-time sampling parameters, including prompts, classifier-free guidance scales, and noise schedules. These parameters are typically manually chosen once and held fixed across prompts and denoising timesteps, despite different prompts and stages of generation benefiting from different parameter values. We introduce LeSAMP a framework for learning prompt-conditioned, timestep-varying sampling parameters while keeping the diffusion model frozen. We formulate parameter selection as a reinforcement learning problem: given a user prompt, a large language model is trained to emit schedules for the chosen sampling parameters. We optimize our model using rewards from human preference models and VLM-as-a-judge. We evaluate LeSAMP on multiple diffusion backbones and find that compared to baselines, LeSAMP has a win rate of up to 68.12% using human preference scores and 73.37% using VLM-as-a-judge. These gains are validated in a user study where we achieve win rates of up to 59.46% over previous baselines. Our results suggest that learned sampling-parameter policies provide a complementary approach to existing post-training methods for improving diffusion model outputs.

Primary area:
generative models
Keywords:
Diffusion Models, Large Language Models, Vision-Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bup4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose LeSAMP, a framework for learning prompt-conditioned, timestep-varying sampling parameters while keeping the diffusion model weights frozen. It fine-tunes an LLM, Qwen3, using GRPO to generate schedules for positive prompts, negative prompts, guidance scales, and noise levels. LeSAMP achieves higher preference win rates under both VLM-as-a-judge and HPSv3 evaluation than existing prompt-scheduling methods, including SAP and R2F.

Critical strengths
  • The proposed framework keeps the diffusion model frozen which avoids backpropagation through the denoising trajectory and provides a complementary approach to diffusion model fine-tuning.
  • The paper is well motivated and clearly written, addressing the important yet relatively less-explored problem of learning time-varying sampling parameters for image generation.
Critical weaknesses
  • The comparison with SAP [1] and R2F [2] does not account for the additional training budget, since both baselines are purely training-free, whereas LeSAMP requires LLM fine-tuning using reward models.
  • The improvements compared to the baselines (in Tables 2 and 3) appear modest. Given the additional training cost, stronger emprical evidence is needed to establish the practical significance of these improvements.

[1] Image generation from contextually-contradictory prompts (Huberman et al., CVPR 2026)
[2] Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance (Park et al., ICLR 2025)

Questions for authors
  1. In Figure 6, the learned CFG schedule remains constantly above the default CFG value throughout denoising. Could the improvement be explained by a higher overall guidance strength rather than by the time-varying schedule itself? How does the learned schedule compare with higher constant CFG values, such as one matching its average guidance strength?

  2. To what extent does the progression from shorter prompts to increasingly detailed prompts described in Ln322? Are the relevant instructions heuristic or motivated by prior works? How does performance change when these instructions are removed from the meta-prompt?

  3. How sensitive is performance to the GRPO group size? Could the authors provide an ablation using different numbers of sampled completions per prompt?

  4. How much do the learned CFG and negative-prompt schedules vary across input prompts?

2 comments

We thank the reviewer for their thoughtful review and suggestions. We address their questions and concerns below.

Q1 (learned CFG schedule vs. higher constant CFG).

We compare the LeSAMP policy that samples the learned schedule to different constant higher CFG schedules. We find that the learned schedule outperforms all of them.

CFG setting vs. default samplingTraining HoldoutWhoops-Hard ∪\cup ContraBench
LeSAMP67.7 ±\pm 5.263.2 ±\pm 6.1
Constant 6.057.0 ±\pm 5.255.7 ±\pm 6.2
Constant 7.051.7 ±\pm 5.651.8 ±\pm 6.8
Mean scheduled CFG50.7 ±\pm 5.555.7 ±\pm 6.6
Constant 8.548.2 ±\pm 6.053.2 ±\pm 6.8
Max scheduled CFG39.8 ±\pm 5.644.3 ±\pm 7.0
Learned schedule, reversed40.0 ±\pm 5.844.3 ±\pm 6.6

Every constant schedule performs much worse than the learned schedule, and some even lose to default SD 3.5. Reversing the schedule also loses to default sampling, so the timing of the guidance matters rather than just its level.

Q2 (progression from shorter to more detailed prompts).

The progression from shorter to longer prompts is learned, rather than part of the system prompt given to the policy during training or evaluation. Instead, the system prompt begins like the SAP system prompt, which asks to start from a placeholder or substitute concept and swap in the intended concept at a later step. There is no instruction to start short and add detail over time. Thus, we see that our learned policy departs from the SAP instructions after training.

Explicitly instructing the untrained model to make its prompts gradually longer and more detailed does not recover the gain: the trained policy still wins 63.3 ±\pm 5.5 / 65.0 ±\pm 6.9 against the untrained prompted model (see our reply to Reviewer eMaU, Q1).

Q3 (GRPO group size).

We train two policies with group sizes 4 and 8. Below, we show winrates against the default sampler at each group size. We observe that group size 4 is still enough to learn policies that outperform the baseline.

Group sizeTraining HoldoutWhoops-Hard ∪\cup ContraBench
866.0 ±\pm 5.558.9 ±\pm 6.9
464.2 ±\pm 5.861.1 ±\pm 6.7

Q4 (variation of CFG and negative-prompt schedules across prompts).

On SD 3.5, the policy learns a mostly universal negative-prompt schedule, cycling through generic photographic-quality terms (e.g., "blurry", "low resolution") at intermediate and late steps. With Flux, the negative prompt at the first and last denoising steps is mostly fixed, while intermediate steps adapt to the prompt content (e.g., "blurry faces" on portrait prompts). This provides evidence that some conditioning signals, like negative prompts, have different optimal values for different models. CFG schedule differs between prompts, but generally we see a monotonically increasing schedule, starting near 4 and ending around 8. More details about this are discussed in Section 4.1.

W1 / W2 (training cost and size of the gains).

We acknowledge that training is an additional cost. However, we also show that LeSAMP still performs better than the baseline even with smaller, more accessible policies with 1.7B and 4B parameters, which have win rates of 58.75–65.22% against default sampling (Table 4). At inference, LeSAMP adds a single generation from this LLM per image, whereas SAP and R2F query an API-based LLM for every prompt.

We also show that LeSAMP improves upon the baseline on public benchmarks, where it raises GenEval from 0.68 to 0.75, GenEval 2 (atom-level / prompt-level) from 64.0 / 17.0 to 69.4 / 22.9, and GenAI-Bench VQAScore from 0.74 to 0.79 (see our reply to Reviewer eMaU, Q4 for more details).

Thank you for the rebuttal. The CFG ablation and the clarification on prompt progression address my questions well. However, my main concern remains. The new results are reported against default sampling rather than SAP and R2F, whose margin in Tables 2 and 3 still appears modest. I will therefore maintain my rate as Weak Rejection.

Reviewer_eMaUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LeSAMP, a framework that uses GRPO to train an LLM to predict sampling parameters that vary across denoising stages, while keeping the text-to-image model frozen. These parameters include positive and negative prompts, CFG scale, and the noise schedule, with Gemini serving as the reward model during training. The framework represents the parameter schedules in JSON format. At inference time, the trained LLM generates a complete schedule from the user's prompt before sampling begins, and the diffusion model follows this schedule to generate an image.

Experiments on SD3.5 and FLUX.1 compare LeSAMP with SAP and R2F, reporting preference win rates on a self-collected set of user requests and on Whoops-Hard ∪ ContraBench. The reported results support the effectiveness of the approach, with additional experimental comparisons provided in the appendix.

Critical strengths
  1. The framework simplifies learned sampling control. It predicts complete schedules without intermediate visual conditioning or backpropagation through the frozen generator, potentially easing integration with existing models.

  2. Several evaluations support the empirical gains. Results span multiple backbones, inference seeds, human judgments, and another VLM evaluator. The authors also disclose weak noise-schedule results, limited joint-optimization gains, and HPSv3 reward hacking.

Critical weaknesses
  1. The paper offers limited incremental novelty, and the advantages over prior work are not sufficiently established. R2F and SAP already introduce stage-dependent prompting, while PromptLoop, which is not discussed in the manuscript, uses RL and supports precomputing prompts. LeSAMP’s incremental contribution is to jointly plan parameter values and switching points in a single LLM generation without intermediate visual feedback, extending this approach to multiple sampling parameters. This integration is potentially useful, but its added value remains unclear. The second claimed contribution, a unified action space, also appears primarily to be an engineering integration, with limited evidence of a distinctive design principle or targeted experiments validating its benefits.

  2. Evaluation coverage and metrics do not adequately establish the extent of the gains. Testing is limited to self-collected requests and specialized difficult prompts, without broader public benchmarks. Preference win rates also obscure improvement magnitude and whether gains come from adherence or image quality. Table 10 should be promoted to the main paper, as it suggests stronger improvements in alignment than quality.

Questions for authors
  1. What does the policy learn beyond simple scheduling heuristics? The reported behaviors—progressively adding prompt details, using largely generic negative prompts, and increasing CFG over time—appear amenable to concise natural-language instructions. Can an untrained LLM recover comparable gains when explicitly instructed to follow these learned strategies? If a substantial gap remains, what prompt-dependent decisions or failure cases explain the advantage of the RL-trained policy?

  2. In Table 7, the FLUX negative-prompt policy trained with HPSv3 achieves only a 42.50% HPSv3 win rate against the default sampler, despite a 65.79% Gemini win rate. Could you explain it?

  3. How well does the trained LLM policy generalize across diffusion backbones? Can a policy trained on one backbone improve generation on others without further training, or does deploying LeSAMP on each new diffusion model require separately fine-tuning the LLM?

  4. Could the authors evaluate LeSAMP on a broader range of established public benchmarks? Results covering more diverse prompts and generation capabilities would help establish whether the reported gains extend beyond the current user-request and challenging-prompt datasets.

1 comment

We thank the reviewer for their thoughtful review and suggestions. We address their questions and concerns below.

W1 (novelty).

Thank you for the pointer to PromptLoop [1]. We will cite it in the paper. LeSAMP differs from PromptLoop, SAP, and R2F, in a few ways:

  • What the policy has access to - PromptLoop trains a multimodal LLM on intermediate denoised latents, which its authors describe as “essential during training”. LeSAMP can be trained with a text-only LLM, with no access to intermediate latents at any stage. Its schedules can therefore be used with any black-box model whose sampler accepts per-step inputs (prompt, negative prompt, CFG scale, noise schedule), without access to intermediate latents or model weights.
  • Switch points - PromptLoop refines the prompt at fixed, evenly spaced timesteps at inference. LeSAMP learns the switch points and the number of segments per prompt.
  • Control type - LeSAMP also allows for learned scheduling of CFG, negative prompts and noise schedules, in addition to the positive prompt.
  • Learned vs hand-designed - SAP and R2F use hand-designed prompts to elicit specific behaviors. Table 1 shows that these are suboptimal. LeSAMP outperforms both SAP and R2F on both backbones and both datasets (62.7–73.4% win rate), and human evaluators also prefer LeSAMP over SAP (Table 2). This indicates that the learned schedules and conditioning values are an important part of our method.

Q1 (can instructions alone recover the learned strategies?).

Instructions alone cannot completely recover the learned strategies. We used Opus 5.5 to analyze all the sampled schedules from our evaluations and come up with a description of what the policy learned. Then, we gave the untrained base model that same description and compared it to the LeSAMP policy. The summarized description directed the untrained policy to gradually make every prompt longer, keep the same elements throughout each intermediate prompt, and use no placeholder concepts. The base model copies this structure closely, yet the trained policy still wins clearly on both datasets.

ComparisonTraining HoldoutWhoops-Hard ∪\cup ContraBench
LeSAMP vs. instructed base model63.3 ±\pm 5.565.0 ±\pm 6.9

This indicates that while the prompted untrained model is able to copy similar high-level patterns as the trained model, there are still other details that the untrained model is unable to follow just from prompted instructions. Some examples of suboptimal behavior from the prompted untrained model include rephrasing that drops critical details from the user prompt, invented attributes, and particularly on Whoops-Hard ∪\cup ContraBench, dropped contradictory elements. In addition, prompting the untrained model in this way is only possible after a trained policy reveals what patterns it has learned.

Q2 (Table 7, Flux negative prompt).

We find that learning a negative prompt for Flux that satisfies HPSv3 is a difficult task, likely due to Flux’s guidance distillation. Guidance distillation folds the unconditional pass into the model weights, offering less controllability from the negative prompt than a non-distilled model would. The higher Gemini win rate suggests it’s easier to learn a negative prompt that helps with prompt adherence, rather than the absolute aesthetics HPSv3 favors. The same split in reward model preference appears for the negative prompt policy trained with the Gemini judge in Table 3. This is consistent with our findings that HPSv3 prefers aesthetics over prompt adherence in Appendix B.2.

Q3 (transfer across backbones).

We evaluate a policy trained on SD 3.5 and apply it to Flux without any retraining. We observe positive gains from this, indicating that some learned policies for one model can be transferred to other models.

ComparisonTraining HoldoutWhoops-Hard ∪\cup ContraBench
LeSAMP (SD 3.5-trained →\to Flux) vs. default Flux72.5 ±\pm 5.162.9 ±\pm 7.2

Q4 (public benchmarks).

We evaluated SD 3.5 medium and a LeSAMP-trained policy with SD 3.5 medium on three public benchmarks. We observe that on each benchmark, the LeSAMP-trained policy outperforms the baseline.

BenchmarkSD 3.5LeSAMP
GenEval [2] (Overall)0.680.75
GenEval 2 [3] (atom-level / prompt-level)64.0 / 17.069.4 / 22.9
GenAI-Bench [4] (VQAScore)0.740.79

References

[1] Lee and Ye, "Plug-and-Play Prompt Refinement via Latent Feedback for Diffusion Model Alignment", CVPR 2026.

[2] Ghosh et al., "GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment", NeurIPS 2023.

[3] Kamath et al., "GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation", arXiv 2025.

[4] Li et al., "GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation", CVPR Workshops 2024.

Reviewer_fbVVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes LeSAMP, which keeps a text-to-image diffusion model frozen and trains an LLM policy with GRPO to emit, given a user prompt, piecewise schedules over the denoising trajectory for four sampling parameters: positive prompt, negative prompt, CFG scale, and noise schedule. The reward is a group-level pairwise win rate computed by a VLM judge (Gemini 3.5 Flash); HPSv3 is also explored as a reward but is found to be reward-hacked. On SD 3.5 and FLUX.1 [dev], LeSAMP is compared against default sampling parameters, SAP, and R2F, with win rates of up to 73.37% under the Gemini judge, 68.12% under HPSv3, and up to 59.46% in a 13-participant user study.

Critical strengths
  • The problem is well motivated: sampling parameters are typically fixed globally, while prior work shows that both the prompt and the denoising stage affect which values are optimal.
  • The method is simple and backbone-agnostic. The diffusion model stays frozen, each update only requires inference through the sampler, and small policies still yield gains.
  • The evaluation is fairly broad: two main backbones, an in-distribution set (User Requests) and an out-of-distribution set (Whoops-Hard ∪\cup ContraBench), two automatic judges, a user study, and additional backbones in Appendix A.
  • The baselines are treated fairly. Replacing the GPT-4o LLM in SAP and R2F with the stronger GPT-5.4 mini removes the concern that gains come from a stronger base LLM.
  • The authors are candid about reward hacking with HPSv3 and provide a judge–human agreement analysis (Table 9).
  • The Flow-GRPO comparison (Table 8) contains a genuinely interesting observation: the weight-updating method is stronger in distribution but degrades out of distribution.
Critical weaknesses

W1. Training data, privacy, and ethics are essentially undocumented.

  • The only description of the training data is one sentence in Appendix H ("user image generation requests from the web"). There is no source, size, license, deduplication, or filtering procedure, and no statement on whether the 240-prompt User Requests set will be released. Thus the main results are not reproducible.
  • The training prompts are scraped from the web ("user image generation requests from the web," Appendix H). The privacy or intellectual-property status of this data must be discussed. Prompts written by real users frequently contain personal preference, business names, brand names, and descriptions of real people or commissioned works, and the users who wrote them did not consent to their use as RL training data or to their reproduction in a paper. Moreover, every user prompt and every generated image is sent to a third-party closed API (Gemini) for scoring. This data transfer is not discussed.
  • The user study is not independent: the evaluators were employees of the authors' organization, participating "as part of their regular job requirements" (Appendix G). They have a stake in the method's success, so their preferences cannot be treated as an unbiased confirmation of the automatic metrics. An independent, blinded evaluator pool would be needed to support the user-study claims.

W2. Key controls are missing, so the source of the gains is unclear.

  • There is no baseline that runs the untrained Qwen3-32B with the same system prompt.
  • There is no comparison with standard static prompt upsampling / enhancement, which is the default component in many generation pipelines and the most direct competitor. Promptist is cited but not compared against directly. An alternative approach could also be directly using powerful commercial model, such as ChatGPT Atlas, to provide a more detailed and semantic rich prompt.
  • A test-time selection baseline is missing. LeSAMP's reward is simply "which of eight images does Gemini prefer," so the most direct alternative is to generate N images with default parameters and let the same judge pick one at inference time.
  • The time-varying ablation compares two separately trained policies, so the difference also reflects training variance rather than time variation alone.
  • In Table 8, LeSAMP wins only 40.00% head-to-head against Flow-GRPO on User Requests, i.e., it loses in distribution. The paper calls the two "complementary" but reports no experiment combining them, and Flow-GRPO's hyperparameters are not reported.

W3. Limited analysis*

  • The paper provides exactly one full example of how the positive prompt evolves over the denoising trajectory: Appendix I.1 (SD 3.5, segments at steps 0, 4, and 10). The three prompts are nearly identical; the step-0 prompt is already a complete rewrite, and later segments only rephrase and add a few details ("dense fog," "lit gas lamp," "19th-century clothing"). This example arguably undermines rather than supports the claim that time variation matters.
  • Figure 1 shows the policy outputting "a red and white car" before switching to "a gift wrapped junk car," i.e., a SAP-style placeholder concept. The text (Section 4, Appendix J) states that the trained policy does not do this. The overview figure therefore illustrates a behavior the paper itself says the method does not have.
  • The core qualitative claim—that the policy "outputs shorter prompts at the beginning and gradually adds detail"—is not quantified.
  • Failure case examples are not provided.
Questions for authors
  1. What is the source and size of the training prompts and will the training data and the User Requests set be released?
  2. Were the user-study evaluators blinded to which method produced each image?
  3. How does the untrained Qwen3-32B with the same system prompt perform against default sampling parameters, and how does LeSAMP compare with a static prompt-enhancement baseline (e.g., a commercial LLM rewriting the prompt once) and with best-of-N selection by the same Gemini judge?
  4. If the trained policy's last prompt is used for all 50 steps without switching, what is the win rate? This would show whether the gain comes from switching or just from the rewrite.
  5. What hyperparameters were used for Flow-GRPO in Table 8, and was combining it with LeSAMP tried?
  6. An ablation of switch-point positions and number of segments is needed.
  7. Does a policy trained on SD 3.5 retain its gains when applied to Flux, or could serve as a base for fine-tuning to accelerate?
1 comment

We thank the reviewer for their thoughtful review and suggestions. We address their questions and concerns below.

Q1 / W1 (training data and consent).

The original training set contained 24,000 user requests collected from the web, and the User Requests evaluation set of 240 held-out requests from the same source. Users who submitted the original requests explicitly agreed that their data could be used for model training and evaluation. Due to organizational restructuring, we unfortunately lost access to the original User Requests dataset and are no longer able to release it as originally planned.

However, to make our results reproducible, we re-ran our experiments on DiffusionDB [1], a public dataset of ~1.8M prompts. We evaluate on 200 held-out prompts from DiffusionDB, along with Whoops-Hard ∪\cup ContraBench. In addition, all experiments in this rebuttal use DiffusionDB for training. We observe results consistent with the paper (73.0% / 72.5% win rate against default sampling on each evaluation dataset, respectively). This result is expected, as both User Requests and DiffusionDB come from a similar distribution of user image generation requests.

Q2 (user study).

Yes, evaluators were blind to which method produced each image, and image order was randomized, so they could not favor our method.

Q3 / W2 (baselines).

We evaluate on all three suggested baselines and see that policies trained with LeSAMP outperform each of them. The table below shows LeSAMP's win rates against each suggested baseline:

ComparisonTraining HoldoutWhoops-Hard ∪\cup ContraBench
LeSAMP vs. untrained policy73.0 ±\pm 5.068.2 ±\pm 6.5
LeSAMP vs. one-shot rewrite (judge LLM)58.0 ±\pm 5.857.5 ±\pm 7.0
LeSAMP vs. best-of-8 (judge-selected)59.5 ±\pm 5.952.5 ±\pm 4.9

LeSAMP beats the one-shot rewrite even though the rewriter is the judging model itself, and beats best-of-8 selection while rendering one image instead of eight.

Q4 / W2 (switching vs. last prompt).

Below, we compare images generated by our policy with images generated from the policy’s last prompt in the schedule. We see that our policy consistently beats images generated from the last prompt.

ComparisonTraining HoldoutWhoops-Hard ∪\cup ContraBench
LeSAMP vs. last schedule prompt only52.0 ±\pm 1.752.7 ±\pm 2.0

Similar experiments supporting the use of learned scheduling as opposed to rule-based selection can be seen in our reply to Reviewer bup4 (Q1).

Q5 / W2 (Flow-GRPO).

For Flow-GRPO, we ran a hyperparameter sweep and found the best results with parameters recommended in the Flow-GRPO paper [2], with the same prompts, batch size and training budget as LeSAMP. We also evaluated combining Flow-GRPO and LeSAMP. Results from applying the trained LeSAMP policy, without retraining, to SD 3.5 trained with Flow-GRPO are shown below. We observe that a LeSAMP trained policy is indeed complementary to Flow-GRPO, as LeSAMP + Flow-GRPO wins over Flow-GRPO alone.

ComparisonTraining HoldoutWhoops-Hard ∪\cup ContraBench
LeSAMP + Flow-GRPO vs. Flow-GRPO67.2 ±\pm 5.164.3 ±\pm 6.7

Q6 (switch points and number of segments).

The policy chooses the number of segments and the switch points itself. Less restriction means a larger search space and a harder setting to learn in, but it gives the policy the most freedom to discover the best setup, and the goal is the best image rather than a fixed design. However, we still evaluate on ablated settings regarding switch points and number of segments. We observe LeSAMP has a slightly higher winrate and that the learned switch points do positively contribute to the policy's performance.

ComparisonTraining HoldoutWhoops-Hard ∪\cup ContraBench
LeSAMP vs. same prompt, evenly spaced switches52.2 ±\pm 2.853.6 ±\pm 3.1

Our reply to Q4 provides additional evidence that the learned number of segments contributes to LeSAMP’s performance, as collapsing the schedule to a single segment lowers the win rate.

Q7 (transfer to Flux).

We evaluate a policy trained on SD 3.5 and apply it to Flux without any retraining. We observe positive gains from this, indicating that some learned policies for one model can be transferred to other models.

ComparisonTraining HoldoutWhoops-Hard ∪\cup ContraBench
LeSAMP (SD 3.5 →\to Flux) vs. default Flux72.5 ±\pm 5.162.9 ±\pm 7.2

References

[1] Wang et al., "DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models", ACL 2023.

[2] Liu et al., "Flow-GRPO: Training Flow Matching Models via Online RL", NeurIPS 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.