acceptodds
Under review as a conference paper at ICLR 2027

ActDiff: Structured Action Search for Test-Time Alignment of Diffusion Models

Abstract

Diffusion models achieve strong generative performance, yet aligning their outputs with human preferences remains challenging. Test-time scaling provides a flexible alternative to fine-tuning by allocating additional inference compute to reward-guided search without retraining. However, existing approaches primarily explore stochastic noise perturbations, offering a narrow, unstructured search space that becomes increasingly restrictive for distilled few-step and deterministic samplers, where stochastic branching is limited or absent. We examine this limitation from an information-theoretic perspective, showing that reward improvement is constrained by the influence of the searched variable on the final output. Motivated by this analysis, we introduce ActDiff, a plug-and-play framework that searches over structured control conditions, such as prompt conditioning and guidance strength, using lightweight, state-dependent diagnostics from the denoising process to guide exploration. This enables more systematic and reward-relevant trajectory variations than noise-only search. ActDiff is gradient-free and compatible with diverse reward functions, search strategies, and diffusion samplers, covering both stochastic and deterministic sampling with standard and distilled few-step models. Experiments demonstrate that expanding the search space beyond noise consistently improves reward alignment and generation quality over noise-centric baselines, with particularly pronounced gains in distilled few-step settings, where noise-based exploration quickly saturates.

Primary area:
generative models
Keywords:
diffusion models, test-time scaling few-step diffusion, controllable generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RUWDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ActDiff, a test-time alignment framework that searches over prompt variants and classifier-free guidance (CFG) scales at each denoising step. The authors employ Monte Carlo Tree Search (MCTS) with a trajectory-feedback prior and provide an information-theoretic analysis of the proposed action space. Experiments across multiple diffusion backbones demonstrate improvements in target reward scores over existing search methods.

Critical strengths
  • The proposed structured action space provides an intuitive and practical alternative to noise-based search, particularly for few-step distilled models and deterministic samplers where intermediate stochastic branching is limited.

  • The method achieves promising improvements across several diffusion backbones and reward functions. Its compatibility with existing particle-based search algorithms further demonstrates the flexibility of the proposed framework.

Critical weaknesses
  • Proposed method requires substantial inference-time computation, as MCTS involves repeated denoising rollouts and reward-model evaluations for generated candidates. These repeated reward evaluations introduce additional computational overhead beyond the reported diffusion-model NFEs, potentially limiting practical applicability. Although the appendix provides a limited wall-clock profiling study, the main benchmark lacks end-to-end latency comparisons. Evaluating performance under matched wall-clock budgets, including reward evaluation costs and prompt generating, would better demonstrate the practical efficiency of the method.

  • Although the ablation studies demonstrate the benefits of jointly searching over prompt variants and CFG scales, it would be interesting to further investigate the effectiveness of step-wise action selection. For example, comparing ActDiff with a static search strategy that selects a single prompt–CFG pair and keeps it fixed throughout denoising could provide additional insights into the benefits of dynamically adjusting these controls.

Questions for authors
  • It would be interesting to investigate whether adjusting the CFG scale in existing methods, such as Best-of-N (BoN) and FK-Steering, also leads to performance improvements. If so, comparing these CFG-adjusted baselines with ActDiff under comparable computational budgets would help clarify the benefits of the proposed method.

  • Additionally, it would be interesting to evaluate whether reward overoptimization occurs by reporting complementary metrics, such as CLIPScore and Aesthetic Score. Such evaluations could help determine whether improvements in the optimized reward come at the expense of other aspects of image quality or prompt alignment.

1 comment

We thank Reviewer RUWD for the critical feedback and detailed comments. We address your questions point by point below.

W1: Wall-clock matched comparison

We thank the reviewer for this suggestion. We now report end-to-end wall-clock results on SiD under matched NFEs, including prompt rewriting, text encoding, denoising, decoding, and reward scoring; all methods were run on a single NVIDIA RTX A6000 GPU. Table 3 breaks down the time per prompt at 128 NFE. ActDiff's per-prompt time is comparable to that of DAS and lower than those of Best-of-N and FK-Steering. ActDiff uses fewer NFEs than the budget allows and calls the reward model far fewer times than Best-of-N and FK-Steering, which reduces both decoding and reward-scoring time. Prompt rewriting accounts for only a small fraction of the total time. Therefore, the additional processing adds no significant compute overhead compared with the strongest competitors.

Table 3. Time breakdown at 128 NFE on SiD (ImageReward as the search reward; seconds per prompt). Total also includes minor bookkeeping overhead not listed separately.

MethodActual NFEReward evaluationsRewriteText encodingDenoisingDecodingReward scoringTotal
ActDiff115.418.02.260.3247.714.459.0063.88
Best-of-N128.032.00.000.1356.227.4616.5980.45
FK-Steering128.096.00.000.1356.2321.5947.74125.74
DAS128.016.00.000.1352.543.697.2463.63

W2: Static search strategy

In response to both Reviewer hnox and Reviewer RUWD, Table 2 below reports results for a static strategy, which fixes a single prompt–guidance pair for the entire trajectory, and for random action selection from the same bank. Fix pair action is least favorable since a good trace typically needs changing prompt-guidance pair inside the trajectory

Table 2. Same-bank controls with a single initial noise (SiD, 4 steps, ImageReward as the search reward, n = 100).

Search rewardMethodImageReward ↑HPSv3 ↑HPSv2 ↑PickScore ↑Avg. NFE
ImageRewardActDiff1.552112.4040.312322.759115.4
ImageRewardRandom action sequences1.457512.2660.31222.755126.1
ImageRewardStatic prompt–guidance pair1.429712.2740.31022.603125.3

Q1: CFG-adjusted baselines

We have evaluated CFG-adjusted Best-of-N and FK-Steering on a fixed 100-prompt subset of our main benchmark, using SiD with ImageReward guidance and a budget of about 128 NFE. Under comparable compute, adding the same CFG grid to Best-of-N lowers ImageReward, because BoN cannot adjust CFG within a trajectory and therefore spends more of its budget on over-saturated samples. For FK-Steering, sharing the CFG bank across particles yields a modest ImageReward gain, but the result remains below ActDiff.

Table 5. Baselines given under augmented CFG bank . The CFG bank gives a modest improvement; however, the initial seed remains the dominant factor for particle-based algorithms.

MethodImageRewardHPSv3HPSv2PickScore
ActDiff1.552112.4040.31222.759
Best-of-N, CFG 1.01.506012.1300.31122.774
Best-of-N, seed × CFG grid1.473712.0670.31022.723
FK-Steering, CFG 1.0 (K = 32)1.463912.1330.33222.742
FK-Steering, CFG bank (K = 16)1.493412.1500.33322.711

Q2: Complementary metrics (CLIPScore and Aesthetic)

We compared ActDiff with the two most competitive baselines under ImageReward-guided search, following the settings of our main experiment. The results are mixed, but CLIPScore and LAION Aesthetic remain close across methods, and the differences in these two metrics are small. We acknowledge that ActDiff trails slightly in LAION Aesthetic. We attribute this to a reward trade-off within the model's capacity rather than to severe reward hacking.

Table 6. CLIPScore (ViT-B/32) and LAION Aesthetic of the returned images (ImageReward as the search reward; scored against the original prompt).

BackboneMethodCLIPScore ↑Aesthetic ↑
SiDActDiff0.8966.28
SiDBest-of-N0.9066.34
SiDFK-Steering0.8976.34
SenseFlow-LargeActDiff0.8886.16
SenseFlow-LargeBest-of-N0.8946.24
SenseFlow-LargeFK-Steering0.9046.17
FLUX.1-schnellActDiff0.8986.33
FLUX.1-schnellBest-of-N0.9066.27
FLUX.1-schnellFK-Steering0.9086.30
SD3.5-LargeActDiff0.9246.14
SD3.5-LargeBest-of-N0.9166.24
Reviewer_LK19OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ActDiff, a test-time search method for diffusion models that treats the prompt variant and the CFG scale at each denoising step as discrete actions. Starting from a fixed initial noise, MCTS searches over sequences of (prompt rewrite, guidance value) pairs, scores terminal images with a reward model against the original prompt, and returns the highest-reward image. The same action space is also inserted into particle-based methods (DAS, FK-Steering). Experiments on three 4-step distilled models and SD3.5-Large with ImageReward and HPSv3 as objectives compare against BoN, beam search, DTS, SoP, DAS and FK-Steering.

Critical strengths
  • The motivation is clear: few-step distilled models and ODE samplers leave little room for noise-based search, so searching over inference-time controls such as prompt wording and guidance scale is a natural direction.

  • The method is gradient-free and works with any reward model, including VLM-based rewards where gradient-based methods are impractical.

Critical weaknesses
  • ActDiff spends 136–240 NFE per image on the distilled models, equivalent to best-of-N over 30–60 seeds. In Fig. 4, at matched NFE, the advantage over BoN is small in most settings, and on SD3.5-Large it only appears above roughly 1000 NFE. It is unclear what the intended use case is: a 4-step distilled model is chosen for speed, and spending 100+ NFE per image removes that advantage. BoN is also fully parallel while MCTS is sequential, so the wall-clock gap is likely larger than the NFE gap suggests.

  • The experiments do not separate the search space from the search algorithm. The needed control is random selection of (prompt, CFG) at each step from the same banks, with the same seed and budget, keeping the best image. Without it, there is no evidence that MCTS or the trajectory-feedback prior contributes anything.

  • Table 4 suggests the rewards favor the distilled models themselves: the 28-step baseline scores below its own 4-step distilled baseline on both IR and HPSv3. HPSv2 varies only in the third decimal and PickScore stays between 22.5 and 23 for every method. I am therefore not convinced that the improvement reflects a real improvement in image quality.

  • The paper reports a large number of tables with little explanation of how they relate. For example, DAS on FLUX under IR-guided search is 1.488 in Table 4 and 1.555 in Table 8, and it is not stated what differs between these runs. No error bars or seeds are reported anywhere, so it is not possible to judge which of the reported differences are meaningful.

Questions for authors

Please refer to Weaknesses section.

1 comment

We thank Reviewer LK19 for the constructive feedback and detailed questions. We address your concerns point by point below.

W1.1 Motivation for test-time scaling of distilled models and intended use: The motivation is similar to that for studying test-time scaling of ODE samplers for flow-based models. We observe that distilled diffusion models inevitably lose image diversity, although previous work argues that diversity metrics such as LPIPS distance do not drop, or even rise, after distillation. Because this loss of diversity is visible in practice, searching over initial seeds alone does not fully exploit the model's capacity. We also respectfully argue that, although timestep distillation is primarily intended to speed up inference, it is still worthwhile to explore the distilled model's full capacity at a cost comparable to generating two or three images with CFG from the undistilled model.

W1.2 Parallel sampling: We acknowledge that BoN can be parallelized and therefore has an advantage in wall-clock time in practice. However, ActDiff can also be parallelized, by growing local MCTS trees from multiple seeds at once. We intentionally used a single seed in the main tables in order to focus on exploring the action space beyond the initial seed.

W2 Search space versus search algorithm: We have added exactly this control. Table 2 compares ActDiff with random selection of (prompt, CFG) at each step from the same banks, using the same initial seed and the same NFE budget and keeping the best image (n = 100 prompts). ActDiff achieves a higher ImageReward than random selection, which shows that search at all denoising points contributes beyond the action space itself.

Table 2. Same-bank controls with a single initial noise (SiD, 4 steps, ImageReward as the search reward, n = 100).

Search rewardMethodImageReward ↑HPSv3 ↑HPSv2 ↑PickScore ↑Avg. NFE
ImageRewardActDiff1.552112.4040.312322.759115.4
ImageRewardRandom action sequences1.457512.2660.31222.755126.1
ImageRewardStatic prompt–guidance pair1.429712.2740.31022.603125.3

W3 Reward metrics: We first point out that it is not uncommon for a distilled diffusion model to beat its teacher in overall image quality, at the cost of a loss in diversity that is hard to capture with quantitative metrics. Furthermore, the comparison between the teacher and the distilled model is not central to our work: we focus only on improvements to the same model under the same reward model, and all the reward models we use are widely adopted as benchmarks in the T2I/T2V community. We therefore respectfully argue that this criticism does not bear on our contribution. HPSv2 and PickScore serve as held-out metrics: because selection is driven by the main reward, we expect them to rise slightly on some images and drop on others. If they do not change much overall, or degrade only slightly, we take this as evidence that the search does not over-optimize the reward.

W4 Table discrepancies and uncertainty: We thank the reviewer for pointing this out, and we acknowledge both issues. The DAS numbers on FLUX should be identical in the two tables. In Table 8, the FLUX entries for both DAS and DAS + ActDiff were taken from runs that had completed only 80 of the 200 prompts; the correct values on the full prompt set are those reported in Table 4. In the revision, we will correct Table 8 and state the setting of each table explicitly. We also acknowledge that the paper reports no error bars: because search-based methods are expensive, each result comes from a single run. We did not have the compute to run multiple seeds for all experiments during the rebuttal period. However, all main-table results were computed on the same A100 cluster with fixed random seeds, so all methods are compared under the same conditions.

Reviewer_hnoxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ActDiff searches over prompt variants and guidance strengths during diffusion sampling, using MCTS with a trajectory-feedback prior or augmenting particle-based methods. The approach targets few-step and deterministic generators, where intermediate noise branching is limited. Experiments report strong optimized-reward scores across several backbones, supported by multi-reward evaluation, ablations, and human comparisons.

Critical strengths

The structured action interface is clear and practically useful. It exposes controls already available in frozen generators and accommodates black-box terminal rewards. Applying the interface to MCTS, DAS, and FK-Steering demonstrates flexibility beyond a single search algorithm.

The empirical evaluation is broad. ActDiff achieves the highest reported IR- and HPSv3-guided target-reward means across all three few-step backbones. Experiments on SD3.5-Large extend the evaluation to deterministic sampling, while control and prior ablations support the usefulness of both prompt and guidance choices.

Critical weaknesses

The implemented search configuration needs clarification. Algorithm 1 uses PUCT only after every action at a node has been tried. With four prompts and seven guidance values, the 30-simulation configuration in Table 16 would allow at most two subsequent root selections. Table 17 instead specifies 25 split simulations and two exploit replays without explaining their relationship to this algorithm. The separate four-prompt profiling study is useful, but its different configurations and exclusion of rewrite generation also leave the main benchmark’s end-to-end cost unresolved.

The experiments establish the usefulness of the controls more convincingly than the benefit of searching them separately at each step. The control-removal and prior ablations do not directly compare against choosing one prompt–guidance pair for the entire trajectory from the same bank, or randomly sampling action sequences at matched cost. These comparisons would isolate the contribution of sequential tree search from improvements due to better conditioning choices.

Original-task fidelity remains incompletely measured. Appendix K mentions filtering invalid or divergent rewrites without specifying a reproducible validation procedure or reporting constraint-preservation rates. Scoring against the original prompt does not enforce counts, negations, or relations. The human alignment results partially address this concern, but each setting has only 50 attempted comparisons, one evaluator per pair, and no reported valid-response counts or uncertainty intervals.

Questions for authors

Please address the points raised in the weaknesses above, particularly the implemented search configuration, comparisons using the same action bank, and validation of rewrite fidelity.

1 comment

We thank Reviewer hnox for the positive feedback and constructive questions. We address your questions point by point below.

W1.1 Search configuration:

We thank the reviewer for checking the search configuration carefully. The reviewer is right that the 30-simulation configuration did not benefit much from this prior. However, the main tables were not obtained with this configuration; it was used only for the budget-scaling curves in Figure 4. In addition, the description in Table 17 was incorrect and did not match our implementation, which follows Algorithm 1. In the revision, we have corrected Table 16 to state the configurations of the main tables and of Figure 4 separately, and we have fixed Table 17. To verify the usefulness of our PUCT prior, we also reran the PUCT ablation with 56 simulations on a 100-prompt subset of our benchmark, due to compute limits. Table 1 reports the results. The full prior achieves the highest ImageReward; removing either component of the prior, or replacing it with a uniform or random prior, lowers ImageReward.

Table 1. PUCT prior ablation with 56 simulations in total (SiD).

MethodImageReward↑HPSv2↑PickScore↑
Prior1.6790.31222.751
No Prior1.5860.31222.765
Only dd1.6190.31222.783
Only uu1.6250.31122.762
Uniform Prior1.6320.31222.725
Random Prior1.6150.30922.746

W1.2 Wall-clock cost:

We report end-to-end wall-clock results on SiD under matched NFEs, including prompt rewriting, text encoding, denoising, decoding, and reward scoring; all methods were run on a single NVIDIA RTX A6000 GPU. Table 3 breaks down the time per prompt at 128 NFE. ActDiff's per-prompt time is comparable to that of DAS and lower than those of Best-of-N and FK-Steering. ActDiff calls the reward model fewer times than Best-of-N and FK-Steering, which reduces both decoding and reward-scoring time. Prompt rewriting accounts for only a small fraction of the total time. Therefore, the additional processing adds no significant compute overhead compared with the strongest competitors.

**Table 3. Time breakdown at 128 NFE on SiD on 100 prompts

MethodActual NFEReward evaluationsRewriteText encodingDenoisingDecodingReward scoringTotal
ActDiff115.418.02.260.3247.714.459.0063.88
Best-of-N128.032.00.000.1356.227.4616.5980.45
FK-Steering128.096.00.000.1356.2321.5947.74125.74
DAS128.016.00.000.1352.543.697.2463.63

W2 Static and random selection strategies:

In response to all Reviewers’ concerns, we report Table 2 below for a static strategy, which fixes a single prompt–guidance pair for the entire simulation rollout, and for random action selection from the same bank. The random action sequences use the same initial noise and NFE budget as ActDiff. On a fixed 100-prompt subset of our main benchmark, using SiD with ImageReward guidance, tree-guided search outperforms both the random and the static rollout strategies. We further observe that even random action selection slightly outperforms the static strategy: many high-reward trajectories use a low CFG at the first step and a higher CFG at later steps, so a static rollout under-explores these regions. Nevertheless, we argue that the combination of the search space is more significant side of our contribution, searching is mainly for reaching the higher performance ceiling that we created.

Table 2. Same-bank controls with a single initial noise (SiD, 4 steps, ImageReward as the search reward, n = 100).

Search rewardMethodImageReward ↑HPSv3 ↑HPSv2 ↑PickScore ↑Avg. NFE
ImageRewardActDiff1.552112.4040.312322.759115.4
ImageRewardRandom action sequences1.457512.2660.31222.755126.1
ImageRewardStatic prompt–guidance pair1.429712.2740.31022.603125.3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.