- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose LeSAMP, a framework for learning prompt-conditioned, timestep-varying sampling parameters while keeping the diffusion model weights frozen. It fine-tunes an LLM, Qwen3, using GRPO to generate schedules for positive prompts, negative prompts, guidance scales, and noise levels. LeSAMP achieves higher preference win rates under both VLM-as-a-judge and HPSv3 evaluation than existing prompt-scheduling methods, including SAP and R2F.
Critical strengths
- The proposed framework keeps the diffusion model frozen which avoids backpropagation through the denoising trajectory and provides a complementary approach to diffusion model fine-tuning.
- The paper is well motivated and clearly written, addressing the important yet relatively less-explored problem of learning time-varying sampling parameters for image generation.
Critical weaknesses
- The comparison with SAP [1] and R2F [2] does not account for the additional training budget, since both baselines are purely training-free, whereas LeSAMP requires LLM fine-tuning using reward models.
- The improvements compared to the baselines (in Tables 2 and 3) appear modest. Given the additional training cost, stronger emprical evidence is needed to establish the practical significance of these improvements.
[1] Image generation from contextually-contradictory prompts (Huberman et al., CVPR 2026)
[2] Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance (Park et al., ICLR 2025)
Questions for authors
-
In Figure 6, the learned CFG schedule remains constantly above the default CFG value throughout denoising. Could the improvement be explained by a higher overall guidance strength rather than by the time-varying schedule itself? How does the learned schedule compare with higher constant CFG values, such as one matching its average guidance strength?
-
To what extent does the progression from shorter prompts to increasingly detailed prompts described in Ln322? Are the relevant instructions heuristic or motivated by prior works? How does performance change when these instructions are removed from the meta-prompt?
-
How sensitive is performance to the GRPO group size? Could the authors provide an ablation using different numbers of sampled completions per prompt?
-
How much do the learned CFG and negative-prompt schedules vary across input prompts?
2 comments
Thank you for the rebuttal. The CFG ablation and the clarification on prompt progression address my questions well. However, my main concern remains. The new results are reported against default sampling rather than SAP and R2F, whose margin in Tables 2 and 3 still appears modest. I will therefore maintain my rate as Weak Rejection.
We thank the reviewer for their thoughtful review and suggestions. We address their questions and concerns below.
Q1 (learned CFG schedule vs. higher constant CFG).
We compare the LeSAMP policy that samples the learned schedule to different constant higher CFG schedules. We find that the learned schedule outperforms all of them.
Every constant schedule performs much worse than the learned schedule, and some even lose to default SD 3.5. Reversing the schedule also loses to default sampling, so the timing of the guidance matters rather than just its level.
Q2 (progression from shorter to more detailed prompts).
The progression from shorter to longer prompts is learned, rather than part of the system prompt given to the policy during training or evaluation. Instead, the system prompt begins like the SAP system prompt, which asks to start from a placeholder or substitute concept and swap in the intended concept at a later step. There is no instruction to start short and add detail over time. Thus, we see that our learned policy departs from the SAP instructions after training.
Explicitly instructing the untrained model to make its prompts gradually longer and more detailed does not recover the gain: the trained policy still wins 63.3 ± 5.5 / 65.0 ± 6.9 against the untrained prompted model (see our reply to Reviewer eMaU, Q1).
Q3 (GRPO group size).
We train two policies with group sizes 4 and 8. Below, we show winrates against the default sampler at each group size. We observe that group size 4 is still enough to learn policies that outperform the baseline.
Q4 (variation of CFG and negative-prompt schedules across prompts).
On SD 3.5, the policy learns a mostly universal negative-prompt schedule, cycling through generic photographic-quality terms (e.g., "blurry", "low resolution") at intermediate and late steps. With Flux, the negative prompt at the first and last denoising steps is mostly fixed, while intermediate steps adapt to the prompt content (e.g., "blurry faces" on portrait prompts). This provides evidence that some conditioning signals, like negative prompts, have different optimal values for different models. CFG schedule differs between prompts, but generally we see a monotonically increasing schedule, starting near 4 and ending around 8. More details about this are discussed in Section 4.1.
W1 / W2 (training cost and size of the gains).
We acknowledge that training is an additional cost. However, we also show that LeSAMP still performs better than the baseline even with smaller, more accessible policies with 1.7B and 4B parameters, which have win rates of 58.75–65.22% against default sampling (Table 4). At inference, LeSAMP adds a single generation from this LLM per image, whereas SAP and R2F query an API-based LLM for every prompt.
We also show that LeSAMP improves upon the baseline on public benchmarks, where it raises GenEval from 0.68 to 0.75, GenEval 2 (atom-level / prompt-level) from 64.0 / 17.0 to 69.4 / 22.9, and GenAI-Bench VQAScore from 0.74 to 0.79 (see our reply to Reviewer eMaU, Q4 for more details).