- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
This paper tackles inference-time alignment for joint audio-visual generation. The paper introduces variance dependent particle search with Metropolis-Hastings (MH), which is training-free, reward-agnostic framework. Under the same computational cost, the proposed method beats BoN and EvoSearch across two generation models.
Critical strengths
- The quantitative comparison is conducted with two models. Comparison under different NFE is also shown in Figure 3.
- Human evaluation shows that the performance gain from the proposed method is valid both quantitatively and qualitatively.
Critical weaknesses
The critical ones are Weaknesses 1 and 2.
- The proposed method may not be scalable with respect to compute (e.g., NFEs). Figure 3 shows that the performance gain of the proposed method saturates with increasing NFEs. Inference-time alignment methods for diffusion models, such as Best-of-N [1], Heuristic Search [2, 3], and Evolutionary Search [4], show scalability with NFEs. This attribute may be important because if the model provider uses more computational budget, then the performance gain improves continuously. I understand that the authors show the comparison with Best-of-N and EvoSearch in Table 1, but under a fixed computational budget. Please discuss inference-time scalability with these prior works and compare the scalability with these methods.
- The wall-clock time is not provided in the paper. The evaluation of video and audio from the intermediate diffusion particles may cost a lot of time, but this cost is not reflected in NFEs. I wonder whether the proposed methods are more efficient than Best-of-N or EvoSearch in wall-clock time.
- The novelty seems to be limited, and the performance gain is also incremental compared with SMC. This method is a combination of SMC and MCMC, but these two and their combination are not novel algorithms [5]. The performance gain of the proposed method shown in Table 1 is not so large compared with Uniform and Uniform + MH.
[1] Ma et al. Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps. CVPR 2025.
[2] Zhang et al. Inference-time Scaling of Diffusion Models through Classical Search. ICLR 2026.
[3] Oshima et al. Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search. NeurIPS 2025.
[4] He et al. Scaling Image and Video Generation via Test-Time Evolutionary Search.
[5] Moral et al. Sequential monte carlo samplers. Journal of the Royal Statistical Society: Series B. (cited by the authors)
Questions for authors
Please see Critical Weaknesses and address concerns.
6 comments
Rebuttal (2/3)
W2. End-to-end wall-clock efficiency
To evaluate practical computational efficiency beyond NFE, we conducted additional CUDA-synchronized, end-to-end wall-clock experiments on LTX-2 using NVIDIA H100 80 GB GPU. Our measurements account for generation, intermediate reward evaluations, latent decoding, adaptive resampling, and MH refinement. All methods use the same reward evaluation suite for a fair comparison, and only one-time model loading is excluded.
Table 3. End-to-end runtime breakdown on LTX-2 (seconds per prompt).
| Method | NFE | Total (s) | Generation (s) | Reward eval. (s) | Decode (s) |
|---|---|---|---|---|---|
| Best-of-N | 600 | 258.8 | 193.2 | 60.8 | 0.8 |
| EvoSearch | 646 | 311.0 | 208.3 | 82.0 | 2.0 |
| SWIM (particle only) | 600 | 311.7 | 189.4 | 116.6 | 3.7 |
| SWIM (particle + MH) | 640 | 335.7 | 206.2 | 122.9 | 3.7 |
The results provide several observations:
1. Comparable generation costs. All methods share the same LTX-2 backbone, with generator execution requiring approximately 0.32 seconds per NFE.
2. Reward evaluation overhead is fully included. SWIM spends approximately 117–123 seconds evaluating intermediate particles and MH proposals, compared with 61 seconds for BoN and 82 seconds for EvoSearch. These evaluations enable more informed particle selection and refinement.
3. Decoding overhead is small. Latent decoding takes approximately 3.7 seconds for SWIM, representing roughly 1% of its total runtime.
4. Predictable MH refinement cost. MH refinement requires approximately 121–122 seconds across initial particle populations K = 4, 8, and 12. This cost is already included in the total runtime and overlaps with the generation, reward evaluation, and decoding components.
We further evaluate runtime scaling across different inference budgets.
Table 4. End-to-end runtime across inference budgets. Each entry reports wall-clock seconds (actual NFE).
| Approx. NFE | Best-of-N | EvoSearch | SWIM (particle only) | SWIM (particle + MH) |
|---|---|---|---|---|
| ~500 | 217 (500) | 243 (497) | 258 (500) | 281 (540) |
| ~700 | 304 (700) | 311 (646) | 366 (700) | 389 (740) |
| ~900 | 389 (900) | 448 (944) | 471 (900) | 498 (940) |
The measurements show approximately linear runtime scaling with NFE across all methods. The average wall-clock cost is approximately 0.43 s/NFE for BoN, 0.46 s/NFE for EvoSearch, 0.53 s/NFE for SWIM (particle only), and 0.54 s/NFE for SWIM (particle + MH). Since the shared generator accounts for approximately 0.32 s/NFE, the differences primarily reflect reward evaluation and refinement overhead. Importantly, SWIM preserves linear inference-time scaling rather than introducing a higher-order computational cost.
While SWIM performs additional intermediate reward evaluations, these translate into substantially stronger alignment quality. In our scaling experiments, SWIM reaches a VR-TA score of 0.99 at 690 NFE, compared with 0.87 for BoN at 700 NFE and 0.80 for DLBS at 864 NFE. In the multi-objective setting, SWIM also achieves approximately 2× higher JavisScore (0.11–0.13 versus 0.05–0.07) and higher TA-IB than BoN and EvoSearch.
Rebuttal (3/3)
W3. Methodological novelty and improvements over SMC
Sequential Monte Carlo (SMC) and Markov Chain Monte Carlo (MCMC) are established techniques, and their combination has been studied previously [5]. Our contribution is a principled, budget-aware inference-time alignment framework that determines how to allocate computation across particle exploration and MH refinement for joint audio-visual generation.
We highlight two key methodological contributions.
1. Variance-aware particle allocation under a fixed inference budget.
Unlike standard SMC methods that maintain a fixed particle population, we formulate particle allocation across denoising checkpoints as a constrained variance-minimization problem and derive a closed-form optimal allocation schedule. Our paper provides the derivation and proof, establishing that particle allocation should account for both reward-weight variability and the computational cost of each denoising segment.
Since the optimal allocation depends on quantities unavailable in advance, we develop an online approximation using variance estimates and adaptive budget allocation, as detailed in the paper. This provides a principled approach to distributing a fixed inference budget rather than uniformly increasing the number of particles.
The theoretical motivation is supported empirically: Figure 6 demonstrates lower estimator-variance surrogates and improved TV-IB compared with uniform allocation. Our additional adaptive-K versus fixed-K experiments further show that the gains persist from 200 to 2000 NFE, independently of MH refinement.
2. Budget-aware coupling of global exploration and local refinement.
While SMC and MCMC have been combined before, SWIM specifically addresses how to coordinate global exploration and local refinement under a limited inference budget. Particle search explores multiple denoising trajectories and identifies a promising candidate, which initializes a short MH chain using local noise-and-denoise proposals.
This division allows MH to refine an already strong candidate rather than spending additional computation on global exploration. Our matched-compute ablations in Figure 3 show that combining particle search with MH achieves higher mean JavisScore than either particle-only or MH-only search across the evaluated budgets. Figure 4 further demonstrates that stronger particle-based initialization improves subsequent MH refinement.
We additionally conducted component-wise ablations to quantify the contributions.
Table 5. Contribution of MH and adaptive allocation on LTX-2.
| Method | AV-IB | AVH | JS | CLAP |
|---|---|---|---|---|
| Uniform@600 (particle only) | 0.203 | 0.188 | 0.071 | 0.140 |
| Uniform+MH@560 | 0.210 | 0.197 | 0.106 | 0.133 |
| SWIM@560 (Ours) | 0.211 | 0.200 | 0.110 | 0.150 |
Table 6. Contribution of MH and adaptive allocation on JavisDiT++.
| Method | AV-IB | AVH | JS | CLAP |
|---|---|---|---|---|
| Uniform@600 (particle only) | 0.223 | 0.211 | 0.115 | 0.188 |
| Uniform+MH@560 | 0.241 | 0.230 | 0.165 | 0.189 |
| SWIM@560 (Ours) | 0.245 | 0.234 | 0.166 | 0.197 |
On LTX-2, introducing MH improves JavisScore from 0.071 to 0.106 (+49.3%), AV-IB from 0.203 to 0.210, and AVH from 0.188 to 0.197. Adaptive allocation further improves JavisScore to 0.110 and CLAP from 0.133 to 0.150 (+12.8%).
Similarly, on JavisDiT++, MH improves JavisScore from 0.115 to 0.165 (+43.5%), while adaptive allocation further improves AV-IB, AVH, and CLAP. These results demonstrate the complementary benefits of the two components.
3. Additional experimental validation.
Beyond the component ablations, we validate the framework across multiple settings:
- Generalization across resampling strategies: Our method is not tied to a specific SMC resampling strategy and can be applied across different resampling schemes. Table 2 demonstrates improvements over uniform allocation with systematic, multinomial, and Fleming–Viot resampling on image-generation tasks.
- Joint audio-visual generation: Table 1 and the human evaluation in Figure 2 demonstrate improvements across JavisDiT++ and LTX-2.
- Inference-time scalability: Our additional experiments demonstrate continued improvements with increasing NFE and stronger alignment than BoN, DLBS, and EvoSearch at medium and high budgets.
In summary, SWIM's novelty lies in its theoretically motivated, variance-aware particle allocation and the budget-constrained coordination of global particle exploration with local MH refinement. We believe together, these contributions enable effective multi-objective audio-visual alignment without modifying the pretrained generator.
Rebuttal (1/3)
We thank the reviewer for the thoughtful feedback. We address inference-time scalability, computational efficiency, and methodological novelty through additional scaling experiments and end-to-end runtime measurements including reward evaluation, decoding, and MH refinement.
W1. Inference-time scalability with increasing computational budgets
SWIM improves inference-time alignment across computational budgets, beyond the fixed-budget comparisons in Table 1. We compare SWIM against Best-of-N (BoN), Diffusion Latent Beam Search (DLBS), and EvoSearch over a wider budget range.
1. SWIM scales consistently with increasing inference-time compute.
We evaluate all methods using VideoReward text-alignment (VR-TA), from approximately 300 to 2300 NFE. Although SWIM targets multi-objective alignment, the baselines primarily target single-objective generation. We therefore restrict SWIM to VR-TA for a fair comparison. Results are averaged over 15 JavisBench-Mini prompts.
Table 1. Inference-time scaling using VR-TA as the common guiding reward. Each entry reports VR-TA (actual NFE).
SWIM achieves the strongest performance at medium and high budgets. It reaches 0.99 VR-TA at 690 NFE, compared with 0.80 for DLBS at 864 NFE and 0.87 for BoN at 700 NFE. At larger budgets, SWIM further improves to 1.18, demonstrating continued scaling through adaptive particle exploration and MH refinement.
2. Adaptive allocation continues to improve with increasing compute.
We isolate adaptive allocation by comparing adaptive-K against fixed-K particle search without MH, using identical VR-TA rewards and budgets on 15 randomly selected JavisBench-Mini prompts from 200 to 2000 NFE.
Table 2. Adaptive-K versus fixed-K scaling using VR-TA (15 randomly selected prompts).
Adaptive allocation maintains its advantage as compute increases, with gains growing from +0.0103 at 200 NFE to +0.0659 at 600 NFE and +0.0763 at 2000 NFE. This isolates its benefit independently of MH. Given the rebuttal timeframe, we plan to extend this preliminary experiment to more prompts and seeds in the revised version.
3. Comparison with the referenced inference-time scaling methods.
The referenced methods primarily target single-objective alignment. We therefore evaluate SWIM using only VR-TA despite its multi-objective design, finding competitive performance at low budgets and stronger results at medium and high budgets.
Ma et al., Inference-Time Scaling for Diffusion Models: We compare against BoN in both the main paper and scaling study. Ma et al. report BoN outperforming search-over-paths in text-conditioned evaluation (88.18 vs. 86.84), making it a strong representative baseline. SWIM achieves better scaling at medium and high budgets.
Zhang et al., Inference-Time Scaling of Diffusion Models through Classical Search: Their local annealed Langevin refinement requires reward gradients with respect to latents, whereas our black-box audio-visual rewards operate on decoded outputs without gradient access. Thus, this component is not directly applicable. Their gradient-free global search uses particle-based resampling, closely related to the SMC framework that SWIM extends with adaptive allocation and gradient-free MH refinement.
Oshima et al., Diffusion Latent Beam Search: We implement DLBS following the authors' formulation, including stochastic candidate expansion and lookahead scoring. Although designed for single-objective text-to-video alignment, we compare it using the same VR-TA reward. SWIM achieves 0.99 at 690 NFE versus DLBS's 0.80 at 864 NFE.
He et al., Scaling Image and Video Generation via Test-Time Evolutionary Search: We implement EvoSearch using its evolutionary checkpoint-search formulation in both the main experiments and single-objective scaling study. SWIM achieves higher VR-TA scores across evaluated budgets.
For the main paper's multi-objective comparisons, we incorporate Adaptive Reward Weighting (ARW) from [6] into BoN and EvoSearch, providing stronger baselines than a fixed, unnormalized sum of heterogeneous rewards.
[6] Jung et. al. Inference-Time Scaling for Joint Audio-Video Generation