acceptodds
Under review as a conference paper at ICLR 2027

Spend Where It Matters: Particle Search with MH Refinement for Budget-Constrained Audio-Visual Alignment

Abstract

Reward-guided inference-time alignment can improve joint audio-visual generation, but optimizing multiple alignment objectives requires many expensive model evaluations, making existing methods costly under limited compute. Particle-based search enables global exploration but incurs estimation error with a finite population, while Metropolis-Hastings (MH) can refine candidates locally but benefits from a strong initialization. We introduce Spend Where It Matters (\modelname), a two-stage, training-free, reward-agnostic framework that combines particle search with MH refinement under a fixed compute budget. In the first stage, we adaptively allocate particles across denoising checkpoints using online variance estimates, minimizing estimation variance while concentrating compute where it is most useful. This search provides a promising candidate for MH refinement, which uses local noise-and-denoise proposals on completed outputs to further improve reward. By starting from a strong candidate, MH can recover additional reward without requiring a larger particle population or costly global exploration. We show that, at matched compute, combining particle search with MH refinement achieves higher rewards than either approach alone. On JavisBench-Mini, at closely matched compute, \modelname achieves mean relative gains across seven metrics of over BoN and over EvoSearch with JavisDiT++, and over BoN and over EvoSearch with LTX-2. Beyond joint audio-visual generation, we also demonstrate the benefits of adaptive particle allocation on SD1.5 image generation, improving AestheticScore by and ImageReward by over uniform allocation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Visual Video Generation, Inference-Time Model, Metropolis-Hastings, Diffusion Model, Flow Matching, Image Generation, Compute Vision.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vpMHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper tackles inference-time alignment for joint audio-visual generation. The paper introduces variance dependent particle search with Metropolis-Hastings (MH), which is training-free, reward-agnostic framework. Under the same computational cost, the proposed method beats BoN and EvoSearch across two generation models.

Critical strengths
  • The quantitative comparison is conducted with two models. Comparison under different NFE is also shown in Figure 3.
  • Human evaluation shows that the performance gain from the proposed method is valid both quantitatively and qualitatively.
Critical weaknesses

The critical ones are Weaknesses 1 and 2.

  1. The proposed method may not be scalable with respect to compute (e.g., NFEs). Figure 3 shows that the performance gain of the proposed method saturates with increasing NFEs. Inference-time alignment methods for diffusion models, such as Best-of-N [1], Heuristic Search [2, 3], and Evolutionary Search [4], show scalability with NFEs. This attribute may be important because if the model provider uses more computational budget, then the performance gain improves continuously. I understand that the authors show the comparison with Best-of-N and EvoSearch in Table 1, but under a fixed computational budget. Please discuss inference-time scalability with these prior works and compare the scalability with these methods.
  2. The wall-clock time is not provided in the paper. The evaluation of video and audio from the intermediate diffusion particles may cost a lot of time, but this cost is not reflected in NFEs. I wonder whether the proposed methods are more efficient than Best-of-N or EvoSearch in wall-clock time.
  3. The novelty seems to be limited, and the performance gain is also incremental compared with SMC. This method is a combination of SMC and MCMC, but these two and their combination are not novel algorithms [5]. The performance gain of the proposed method shown in Table 1 is not so large compared with Uniform and Uniform + MH.

[1] Ma et al. Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps. CVPR 2025.
[2] Zhang et al. Inference-time Scaling of Diffusion Models through Classical Search. ICLR 2026.
[3] Oshima et al. Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search. NeurIPS 2025.
[4] He et al. Scaling Image and Video Generation via Test-Time Evolutionary Search.
[5] Moral et al. Sequential monte carlo samplers. Journal of the Royal Statistical Society: Series B. (cited by the authors)

Questions for authors

Please see Critical Weaknesses and address concerns.

6 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for the thoughtful feedback. We address inference-time scalability, computational efficiency, and methodological novelty through additional scaling experiments and end-to-end runtime measurements including reward evaluation, decoding, and MH refinement.

W1. Inference-time scalability with increasing computational budgets

SWIM improves inference-time alignment across computational budgets, beyond the fixed-budget comparisons in Table 1. We compare SWIM against Best-of-N (BoN), Diffusion Latent Beam Search (DLBS), and EvoSearch over a wider budget range.

1. SWIM scales consistently with increasing inference-time compute.

We evaluate all methods using VideoReward text-alignment (VR-TA), from approximately 300 to 2300 NFE. Although SWIM targets multi-objective alignment, the baselines primarily target single-objective generation. We therefore restrict SWIM to VR-TA for a fair comparison. Results are averaged over 15 JavisBench-Mini prompts.

Table 1. Inference-time scaling using VR-TA as the common guiding reward. Each entry reports VR-TA (actual NFE).

Approx. NFESWIM (Ours)DLBS [3]BoN [1]EvoSearch [4]
~3000.72 (340)0.72 (352)0.67 (300)0.55 (399)
~5000.85 (490)0.76 (544)0.86 (500)0.63 (532)
~7000.99 (690)0.80 (864)0.87 (700)0.69 (798)
~13001.05 (1190)0.97 (1232)0.94 (1400)0.79 (1197)
~20001.18 (2290)1.07 (1928)0.99 (2100)0.78 (1995)

SWIM achieves the strongest performance at medium and high budgets. It reaches 0.99 VR-TA at 690 NFE, compared with 0.80 for DLBS at 864 NFE and 0.87 for BoN at 700 NFE. At larger budgets, SWIM further improves to 1.18, demonstrating continued scaling through adaptive particle exploration and MH refinement.

2. Adaptive allocation continues to improve with increasing compute.

We isolate adaptive allocation by comparing adaptive-K against fixed-K particle search without MH, using identical VR-TA rewards and budgets on 15 randomly selected JavisBench-Mini prompts from 200 to 2000 NFE.

Table 2. Adaptive-K versus fixed-K scaling using VR-TA (15 randomly selected prompts).

NFEAdaptive-KFixed-KΔ (Adaptive − Fixed)
2000.94660.9363+0.0103
4001.12161.0880+0.0336
6001.16761.1017+0.0659
10001.26351.1950+0.0685
15001.30281.2337+0.0691
20001.30171.2254+0.0763

Adaptive allocation maintains its advantage as compute increases, with gains growing from +0.0103 at 200 NFE to +0.0659 at 600 NFE and +0.0763 at 2000 NFE. This isolates its benefit independently of MH. Given the rebuttal timeframe, we plan to extend this preliminary experiment to more prompts and seeds in the revised version.

3. Comparison with the referenced inference-time scaling methods.

The referenced methods primarily target single-objective alignment. We therefore evaluate SWIM using only VR-TA despite its multi-objective design, finding competitive performance at low budgets and stronger results at medium and high budgets.

  • Ma et al., Inference-Time Scaling for Diffusion Models: We compare against BoN in both the main paper and scaling study. Ma et al. report BoN outperforming search-over-paths in text-conditioned evaluation (88.18 vs. 86.84), making it a strong representative baseline. SWIM achieves better scaling at medium and high budgets.

  • Zhang et al., Inference-Time Scaling of Diffusion Models through Classical Search: Their local annealed Langevin refinement requires reward gradients with respect to latents, whereas our black-box audio-visual rewards operate on decoded outputs without gradient access. Thus, this component is not directly applicable. Their gradient-free global search uses particle-based resampling, closely related to the SMC framework that SWIM extends with adaptive allocation and gradient-free MH refinement.

  • Oshima et al., Diffusion Latent Beam Search: We implement DLBS following the authors' formulation, including stochastic candidate expansion and lookahead scoring. Although designed for single-objective text-to-video alignment, we compare it using the same VR-TA reward. SWIM achieves 0.99 at 690 NFE versus DLBS's 0.80 at 864 NFE.

  • He et al., Scaling Image and Video Generation via Test-Time Evolutionary Search: We implement EvoSearch using its evolutionary checkpoint-search formulation in both the main experiments and single-objective scaling study. SWIM achieves higher VR-TA scores across evaluated budgets.

For the main paper's multi-objective comparisons, we incorporate Adaptive Reward Weighting (ARW) from [6] into BoN and EvoSearch, providing stronger baselines than a fixed, unnormalized sum of heterogeneous rewards.

[6] Jung et. al. Inference-Time Scaling for Joint Audio-Video Generation

AuthorsOpenReview ↗

Rebuttal (2/3)

W2. End-to-end wall-clock efficiency

To evaluate practical computational efficiency beyond NFE, we conducted additional CUDA-synchronized, end-to-end wall-clock experiments on LTX-2 using NVIDIA H100 80 GB GPU. Our measurements account for generation, intermediate reward evaluations, latent decoding, adaptive resampling, and MH refinement. All methods use the same reward evaluation suite for a fair comparison, and only one-time model loading is excluded.

Table 3. End-to-end runtime breakdown on LTX-2 (seconds per prompt).

MethodNFETotal (s)Generation (s)Reward eval. (s)Decode (s)
Best-of-N600258.8193.260.80.8
EvoSearch646311.0208.382.02.0
SWIM (particle only)600311.7189.4116.63.7
SWIM (particle + MH)640335.7206.2122.93.7

The results provide several observations:

1. Comparable generation costs. All methods share the same LTX-2 backbone, with generator execution requiring approximately 0.32 seconds per NFE.

2. Reward evaluation overhead is fully included. SWIM spends approximately 117–123 seconds evaluating intermediate particles and MH proposals, compared with 61 seconds for BoN and 82 seconds for EvoSearch. These evaluations enable more informed particle selection and refinement.

3. Decoding overhead is small. Latent decoding takes approximately 3.7 seconds for SWIM, representing roughly 1% of its total runtime.

4. Predictable MH refinement cost. MH refinement requires approximately 121–122 seconds across initial particle populations K = 4, 8, and 12. This cost is already included in the total runtime and overlaps with the generation, reward evaluation, and decoding components.

We further evaluate runtime scaling across different inference budgets.

Table 4. End-to-end runtime across inference budgets. Each entry reports wall-clock seconds (actual NFE).

Approx. NFEBest-of-NEvoSearchSWIM (particle only)SWIM (particle + MH)
~500217 (500)243 (497)258 (500)281 (540)
~700304 (700)311 (646)366 (700)389 (740)
~900389 (900)448 (944)471 (900)498 (940)

The measurements show approximately linear runtime scaling with NFE across all methods. The average wall-clock cost is approximately 0.43 s/NFE for BoN, 0.46 s/NFE for EvoSearch, 0.53 s/NFE for SWIM (particle only), and 0.54 s/NFE for SWIM (particle + MH). Since the shared generator accounts for approximately 0.32 s/NFE, the differences primarily reflect reward evaluation and refinement overhead. Importantly, SWIM preserves linear inference-time scaling rather than introducing a higher-order computational cost.

While SWIM performs additional intermediate reward evaluations, these translate into substantially stronger alignment quality. In our scaling experiments, SWIM reaches a VR-TA score of 0.99 at 690 NFE, compared with 0.87 for BoN at 700 NFE and 0.80 for DLBS at 864 NFE. In the multi-objective setting, SWIM also achieves approximately 2× higher JavisScore (0.11–0.13 versus 0.05–0.07) and higher TA-IB than BoN and EvoSearch.

AuthorsOpenReview ↗

Rebuttal (3/3)

W3. Methodological novelty and improvements over SMC

Sequential Monte Carlo (SMC) and Markov Chain Monte Carlo (MCMC) are established techniques, and their combination has been studied previously [5]. Our contribution is a principled, budget-aware inference-time alignment framework that determines how to allocate computation across particle exploration and MH refinement for joint audio-visual generation.

We highlight two key methodological contributions.

1. Variance-aware particle allocation under a fixed inference budget.

Unlike standard SMC methods that maintain a fixed particle population, we formulate particle allocation across denoising checkpoints as a constrained variance-minimization problem and derive a closed-form optimal allocation schedule. Our paper provides the derivation and proof, establishing that particle allocation should account for both reward-weight variability and the computational cost of each denoising segment.

Since the optimal allocation depends on quantities unavailable in advance, we develop an online approximation using variance estimates and adaptive budget allocation, as detailed in the paper. This provides a principled approach to distributing a fixed inference budget rather than uniformly increasing the number of particles.

The theoretical motivation is supported empirically: Figure 6 demonstrates lower estimator-variance surrogates and improved TV-IB compared with uniform allocation. Our additional adaptive-K versus fixed-K experiments further show that the gains persist from 200 to 2000 NFE, independently of MH refinement.

2. Budget-aware coupling of global exploration and local refinement.

While SMC and MCMC have been combined before, SWIM specifically addresses how to coordinate global exploration and local refinement under a limited inference budget. Particle search explores multiple denoising trajectories and identifies a promising candidate, which initializes a short MH chain using local noise-and-denoise proposals.

This division allows MH to refine an already strong candidate rather than spending additional computation on global exploration. Our matched-compute ablations in Figure 3 show that combining particle search with MH achieves higher mean JavisScore than either particle-only or MH-only search across the evaluated budgets. Figure 4 further demonstrates that stronger particle-based initialization improves subsequent MH refinement.

We additionally conducted component-wise ablations to quantify the contributions.

Table 5. Contribution of MH and adaptive allocation on LTX-2.

MethodAV-IBAVHJSCLAP
Uniform@600 (particle only)0.2030.1880.0710.140
Uniform+MH@5600.2100.1970.1060.133
SWIM@560 (Ours)0.2110.2000.1100.150

Table 6. Contribution of MH and adaptive allocation on JavisDiT++.

MethodAV-IBAVHJSCLAP
Uniform@600 (particle only)0.2230.2110.1150.188
Uniform+MH@5600.2410.2300.1650.189
SWIM@560 (Ours)0.2450.2340.1660.197

On LTX-2, introducing MH improves JavisScore from 0.071 to 0.106 (+49.3%), AV-IB from 0.203 to 0.210, and AVH from 0.188 to 0.197. Adaptive allocation further improves JavisScore to 0.110 and CLAP from 0.133 to 0.150 (+12.8%).

Similarly, on JavisDiT++, MH improves JavisScore from 0.115 to 0.165 (+43.5%), while adaptive allocation further improves AV-IB, AVH, and CLAP. These results demonstrate the complementary benefits of the two components.

3. Additional experimental validation.

Beyond the component ablations, we validate the framework across multiple settings:

  • Generalization across resampling strategies: Our method is not tied to a specific SMC resampling strategy and can be applied across different resampling schemes. Table 2 demonstrates improvements over uniform allocation with systematic, multinomial, and Fleming–Viot resampling on image-generation tasks.
  • Joint audio-visual generation: Table 1 and the human evaluation in Figure 2 demonstrate improvements across JavisDiT++ and LTX-2.
  • Inference-time scalability: Our additional experiments demonstrate continued improvements with increasing NFE and stronger alignment than BoN, DLBS, and EvoSearch at medium and high budgets.

In summary, SWIM's novelty lies in its theoretically motivated, variance-aware particle allocation and the budget-constrained coordination of global particle exploration with local MH refinement. We believe together, these contributions enable effective multi-objective audio-visual alignment without modifying the pretrained generator.

Reviewer_SyW2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SWIM, a training-free inference-time alignment framework for joint audio-visual generation under a constrained computational budget. The proposed framework combines two complementary techniques: adaptive particle-based search for global exploration and Metropolis-Hastings refinement for improving the selected candidate.

In the first stage, the authors formulate particle allocation across denoising checkpoints as a constrained variance-minimization problem. They derive a closed-form allocation rule for a surrogate objective based on particle-weight dispersion and propose an online approximation using effective sample size estimates. In the second stage, the highest-reward candidate from particle search initializes a short MH refinement chain, where proposals are generated by re-noising and re-denoising the completed latent.

Critical strengths

1.Particle search and MH refinement have complementary properties. Particle search provides diverse candidate trajectories, while MH refinement can exploit a promising initialization.The proposed two-stage strategy is conceptually reasonable and does not require modifying the pretrained generator.

2.The paper evaluates two audio-visual backbones and extends the particle-allocation analysis to image generation. Figures 3–5 investigate the complementary effects of particle search and MH refinement, initialization strength, and proposal noise level. Table 2 further considers different particle resampling strategies. These experiments provide evidence that the combination is beneficial under the evaluated settings.

Critical weaknesses

W1. The independent contribution of adaptive particle allocation appears relatively modest. Although the paper reports substantial improvements over Best-of-N and EvoSearch, these comparisons evaluate the overall two-stage framework rather than isolating the proposed adaptive allocation mechanism. The more direct comparison is against Uniform+MH in Table 1. At the lower budget, the reported mean relative improvements across seven metrics are approximately 1.0% for JavisDiT++ and 1.2% for LTX-2. Some individual metrics also exhibit small regressions compared with Uniform+MH. For example, on JavisDiT++ at 360 NFE, TA-IB decreases from 0.1900 to 0.1894. These results do not invalidate the proposed adaptive allocation, but they suggest that much of the overall performance gain may arise from combining particle search and MH rather than the variance-aware schedule itself.

W2.The experiments primarily measure computational cost in generator forward evaluations. However, SWIM involves intermediate reward computation, potentially repeated latent decoding, adaptive resampling, and completed-sample evaluation during MH refinement. For joint audio-visual generation, these additional operations may be nontrivial. Since computational efficiency is central to the paper's motivation, end-to-end wall-clock measurements would strengthen the practical claims and clarify whether the reported advantages persist when all relevant inference costs are included.

Questions for authors

1.What fraction of improvement can be attributed independently to adaptive particle allocation, MH refinement, and the choice of reward functions used in the two stages? 2.Can the authors provide end-to-end runtime measurements that include reward-model evaluation, intermediate decoding, and MH proposal overhead?

4 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for the constructive feedback and thoughtful questions. We provide additional ablations isolating the contributions of adaptive particle allocation and MH refinement, along with end-to-end runtime measurements that account for reward evaluation, decoding, and MH overhead.

W1. Independent contribution of adaptive particle allocation

While combining particle search with MH refinement provides substantial improvements, adaptive allocation independently improves the utilization of a fixed inference budget by redistributing particles across denoising checkpoints according to estimated reward-weight variance.

We highlight four observations:

1. Adaptive allocation improves performance without increasing the inference budget.

Unlike approaches that improve performance by increasing the number of particles or denoising steps, our variance-aware allocation redistributes the same number of generator forward evaluations (NFE). This is particularly relevant for joint audio-visual generation, where individual model evaluations are expensive.

Importantly, the improvements are consistent across both backbones and become more pronounced for certain metrics and budgets. For instance, at 560 NFE, adaptive allocation improves CLAP from 0.1328 to 0.1498 (+12.8%) on LTX-2 and from 0.1894 to 0.1973 (+4.2%) on JavisDiT++. On LTX-2, JavisScore also improves from 0.1058 to 0.1102 (+4.2%). These improvements are obtained without increasing the total NFE.

2. Isolating the contribution of adaptive allocation.

We isolate adaptive allocation by comparing SWIM against Uniform+MH at matched NFE, with identical MH refinement and only the particle allocation strategy differing. On LTX-2 at 560 NFE, adaptive allocation improves CLAP by 12.8% (0.133 → 0.150) and JavisScore by 4.2% (0.106 → 0.110). At 360 NFE, it similarly improves CLAP by 3.0%, JavisScore by 2.2%, and AVH by 1.7%, with all remaining metrics non-degrading at both budgets. These consistent gains require no additional NFE, demonstrating the independent benefit of variance-aware allocation. Furthermore, Figure 6 shows lower estimator variance and higher TV-IB across budgets.

3. The gains from adaptive allocation persist as the inference budget increases.

To further examine whether the benefits of adaptive allocation persist at larger computational budgets, we conducted an additional scaling experiment comparing adaptive-K against fixed-K using VR-TA as the common guiding reward. We evaluate on a randomly selected subset of 15 prompts from JavisBench-Mini, extending the inference budget from 200 to 2000 NFE.

Table 1. Scaling behavior of adaptive-K versus fixed-K using VR-TA (15 randomly selected prompts). Both methods use the same inference budget and guiding reward.

NFEAdaptive-KFixed-KΔ (Adaptive − Fixed)
2000.94660.9363+0.0103
4001.12161.0880+0.0336
6001.16761.1017+0.0659
10001.26351.1950+0.0685
15001.30281.2337+0.0691
20001.30171.2254+0.0763

The gains from adaptive allocation persist as the inference budget increases, growing from +0.0103 at 200 NFE to +0.0763 at 2000 NFE. Since these experiments exclude MH refinement, they directly establish the independent benefit of variance-aware allocation. Although this preliminary experiment uses 15 randomly selected prompts, we plan to extend it to a larger prompt set and additional seeds in the revised version, given the limited rebuttal timeframe.

4. Adaptive allocation and MH refinement provide complementary improvements.

We additionally isolate MH refinement by comparing a fixed-population particle funnel with and without MH. For LTX-2, introducing MH improves JavisScore from 0.0706 to 0.1058 (+49.9%), AV-IB from 0.2027 to 0.2099 (+3.6%), and AVH from 0.1879 to 0.1974 (+5.1%). Adaptive allocation subsequently improves these results, including a substantial improvement in CLAP.

Thus, the two components address distinct aspects of budget-constrained alignment. Particle allocation improves how computational resources are distributed during exploration, and MH refinement improves the candidate obtained from that exploration. Their combination provides the strongest overall performance under a fixed inference budget.

AuthorsOpenReview ↗

Rebuttal (2/3)

W2. End-to-end computational cost and wall-clock efficiency

To evaluate computational efficiency beyond NFE, we conducted additional CUDA-synchronized, end-to-end wall-clock measurements on LTX-2 using NVIDIA H100 80 GB GPU, accounting for generator evaluations, intermediate reward-model evaluations, latent decoding, adaptive resampling, and MH refinement. Only the one-time model-loading cost is excluded.

For consistency, all compared methods use the same reward evaluation suite.

Table 2. End-to-end runtime breakdown on LTX-2 (seconds per prompt).

MethodNFETotal (s)Generation (s)Reward eval. (s)Decode (s)
Best-of-N600258.8193.260.80.8
EvoSearch646311.0208.382.02.0
SWIM (particle only)600311.7189.4116.63.7
SWIM (particle + MH)640335.7206.2122.93.7

The MH refinement stage takes approximately 122 seconds across the evaluated initial particle populations (K = 4, 8, 12). It is already accounted for in the total runtime as it is subsumed in the generation, reward evaluation, and decoding breakdown times.

We make the following observations:

  • Generation cost: Generator execution remains comparable across methods, at approximately 0.32 seconds per NFE.
  • Reward evaluation: Intermediate reward evaluations account for approximately 117–123 seconds for SWIM, reflecting its repeated evaluation of partially denoised candidates.
  • Decoding: Latent decoding contributes approximately 3.7 seconds, representing roughly 1% of SWIM's total runtime.
  • MH refinement: The refinement stage has a nearly constant runtime across initial particle populations and is explicitly included in the measurements.

We also evaluate how end-to-end runtime scales with the inference budget.

Table 3. End-to-end wall-clock runtime across compute budgets. Each entry reports seconds (actual NFE).

BudgetBest-of-NEvoSearchSWIM (particle only)SWIM (particle + MH)
~500 NFE217 (500)243 (497)258 (500)281 (540)
~700 NFE304 (700)311 (646)366 (700)389 (740)
~900 NFE389 (900)448 (944)471 (900)498 (940)

These results provide a more complete characterization of the inference cost and demonstrate that SWIM's runtime scales predictably with increasing compute budgets. The additional reward evaluations enable more informed candidate selection and refinement, translating the available compute into higher alignment quality.

Importantly, SWIM maintains its alignment-quality advantages across the evaluated budgets when all inference costs are accounted for. Thus, the improvements are not simply a consequence of omitting reward evaluation or decoding costs from the computational analysis.

We note, every method's wall-clock grows linearly in NFE with essentially the same slope — approximately 0.43 (Best-of-N), 0.46 (EvoSearch), 0.53(SWIM-particle only )and 0.54 (SWIM-particle + MH) seconds per NFE, all dominated by the shared ~0.32 s/NFE generation cost — so the runtime curves differ only by a near-constant offset, not in their growth rate. SWIM therefore does not increase the time complexity of inference (it stays linear, within ~20% of Best-of-N in absolute seconds, e.g. 498 vs 389 s at ~900 NFE); it converts essentially the same compute budget into substantially higher alignment quality — e.g. roughly 2× higher JS (≈0.11 vs 0.05–0.07) and higher TA-IB than Best-of-N/EvoSearch at matched budget.

AuthorsOpenReview ↗

Rebuttal (3/3)

Q1. What fraction of improvement can be attributed independently to adaptive particle allocation, MH refinement, and the choice of reward functions used in the two stages?

We isolate the contributions through a progressive ablation that starts with fixed-population particle search, adds MH refinement, and finally replaces uniform particle allocation with the proposed variance-aware schedule. We evaluate both LTX-2 and JavisDiT++.

Table 4. Component-wise ablation on LTX-2.

MethodTV-IBTA-IBAV-IBAVHJSCLAPVR-TA
Uniform@600 (particle only)0.3830.1680.2030.1880.0710.1400.741
Uniform+MH@5600.3820.1670.2100.1970.1060.1330.617
SWIM@560 (Ours)0.3820.1700.2110.2000.1100.1500.619

Table 5. Component-wise ablation on JavisDiT++.

MethodTV-IBTA-IBAV-IBAVHJSCLAPVR-TA
Uniform@600 (particle only)0.3830.1880.2230.2110.1150.1880.672
Uniform+MH@5600.3790.1890.2410.2300.1650.1890.533
SWIM@560 (Ours)0.3790.1900.2450.2340.1660.1970.530

These results establish the distinct contributions of each stage.

1. Particle search establishes strong initial alignment.

The particle-only baseline allocates its entire inference budget to particle exploration, allowing it to evaluate more particles than our particle+MH approach. While this achieves a higher VR-TA score (0.741 on LTX-2), the improvements do not extend to the other alignment objectives. In contrast, SWIM uses fewer particles but achieves substantially higher JavisScore, AV-IB, and AVH, while remaining competitive on text-alignment metrics. This demonstrates the benefit of allocating part of the budget to MH refinement rather than exclusively increasing particle exploration.

2. MH refinement substantially improves joint audio-visual alignment.

The second stage introduces a JavisScore-based reward to improve complementary alignment dimensions. On LTX-2, MH increases:

  • JavisScore: 0.071 → 0.106 (+49.3%)
  • AV-IB: 0.203 → 0.210 (+3.4%)
  • AVH: 0.188 → 0.197 (+4.8%)

Similarly, on JavisDiT++, MH improves:

  • JavisScore: 0.115 → 0.165 (+43.5%)
  • AV-IB: 0.223 → 0.241 (+8.1%)
  • AVH: 0.211 → 0.230 (+9.0%)

These results demonstrate the benefit of stage-wise rewards: particle search establishes initial alignment, while MH improves joint audio-visual consistency.

3. Adaptive particle allocation further improves results at unchanged NFE.

Replacing fixed allocation with the variance-aware schedule produces additional improvements across both models. CLAP increases from 0.133 to 0.150 (+12.8%) on LTX-2 and from 0.189 to 0.197 (+4.2%) on JavisDiT++. On LTX-2, JavisScore improves from 0.106 to 0.110 and AVH from 0.197 to 0.200. On JavisDiT++, AV-IB improves from 0.241 to 0.245 and AVH from 0.230 to 0.234.

These gains require no additional NFE, as adaptive allocation redistributes the existing particle budget. Furthermore, our adaptive-K versus fixed-K scaling experiment (Table 1 in W1), conducted on 15 randomly selected prompts, shows positive VR-TA gains across all tested budgets from 200 to 2000 NFE, confirming benefits independent of MH refinement.

Overall, the ablations demonstrate complementary contributions: particle search establishes initial alignment, MH improves joint audio-visual consistency through a complementary reward, and adaptive allocation improves search efficiency. Together, they enable stronger multi-objective alignment under a constrained inference budget.

Q2. Can the authors provide end-to-end runtime measurements that include reward-model evaluation, intermediate decoding, and MH proposal overhead?

As detailed in W2, we conducted CUDA-synchronized end-to-end runtime experiments on LTX-2 using an NVIDIA H100 80 GB GPU, including generator evaluations, intermediate reward evaluations, decoding, resampling, and MH refinement.

At approximately 640 NFE, SWIM requires 335.7 seconds per prompt, including 206.2 seconds of generation, 122.9 seconds of reward evaluation, and 3.7 seconds of decoding. MH refinement accounts for approximately 122 seconds of the total runtime, including its generation and evaluation operations.

We additionally evaluate runtime scaling across approximately 500, 700, and 900 NFE, with complete results in W2. These measurements characterize the tradeoff between end-to-end inference cost and alignment quality, including all relevant evaluation and refinement overheads.

Dear Reviewer,

The discussion period is nearing its end, and we haven’t yet seen a reply from you. We encourage you to respond to the authors’ comments, especially if you have further questions about points that may affect your assessment.

Thank you for taking the time to engage in the discussion.

Best,

Your AC

Reviewer_s8bJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the limitations of existing inference-time alignment methods, which require expensive model evaluations when optimizing multiple alignment objectives. The authors propose SWIM, a two-stage, training-free, reward-agnostic framework that combines particle search with Metropolis-Hastings (MH) refinement. Experimental results validate the effectiveness of the proposed approach and demonstrate state-of-the-art performance on audio-visual generation tasks.

Critical strengths
  • Formulates particle allocation across denoising checkpoints as a variance-minimization problem.
  • Proposes initializing the MH chain using the particles with the highest particle-stage reward.
  • Demonstrates consistent improvements over baselines on joint audio-visual generation tasks through experimental results.
Critical weaknesses
  • Limited technical novelty: The proposed method builds incrementally on existing approaches.
  • Marginal performance gains: The performance gain of the proposed method over the Uniform@400 baseline is limited.
  • Human study margins: The advantage of the proposed method over the baseline on LTX-2 in the human study is marginal.
Questions for authors

Baseline clarity: Which version of the LTX-2 baseline model used in the human study corresponds to the one presented in the main results?

5 comments
AuthorsOpenReview ↗

Rebuttal (1/4)

We thank the reviewer for their feedback. We would like to clarify the technical contributions of SWIM, its improvements over the uniform particle baseline, and the human evaluation results.

W1. Limited technical novelty

While SWIM builds on established sampling techniques, our contribution is a budget-aware framework for multi-objective audio-visual alignment, where multiple heterogeneous reward signals must be optimized without requiring reward gradients or additional model training.

We highlight three key technical contributions:

1. Variance-aware particle allocation under a fixed compute budget.

We formulate particle allocation across denoising checkpoints as a constrained variance-minimization problem and derive the optimal allocation rule, Km∗∝σm+1LmK_m^* \propto \frac{\sigma_{m+1}}{\sqrt{L_m}}. Since future reward-weight variance is unknown, we develop an online algorithm using ESS-based estimates and adaptive budget pacing to dynamically allocate particles while respecting the available NFE budget.

2. Combining global exploration with local refinement.

We use particle search to identify promising candidates and initialize a short MH refinement chain. This is particularly useful under limited compute, where local refinement can further improve a candidate without restarting generation from noise. Figures 3 and 4 demonstrate that the two stages complement each other, with SWIM outperforming particle-only and MH-only approaches under matched computational budgets.

3. Stage-wise optimization of multiple alignment objectives.

Joint audio-visual generation requires balancing text-video, text-audio, and audio-visual alignment. Rather than treating all reward signals identically throughout denoising, we progressively introduce alignment objectives during particle search and subsequently refine completed outputs using complementary rewards. This allows the two stages to focus on different aspects of alignment and provides better reliability and control over the entire alignment process.

Overall, our contribution lies in the formulation and coordination of these components under a shared compute budget. The variance-derived allocation rule, online scheduling algorithm, and stage-wise refinement strategy together provide a practical framework for improving joint audio-visual alignment across different generative backbones.

AuthorsOpenReview ↗

Rebuttal (2/4)

W2. Marginal performance gains over Uniform@400

The improvements over Uniform are substantial on metrics that measure joint audio-visual alignment. While Uniform achieves strong text-video alignment (VR-TA), it primarily optimizes this objective, with considerably lower performance on JavisScore, AVH, and AV-IB. SWIM addresses this by combining particle search with MH refinement to improve multiple alignment objectives.

From Table 1, we highlight:

1. LTX-2: Compared with Uniform@400, SWIM@360 improves JavisScore from 0.0686 to 0.1105 (+61.1%), AVH from 0.1874 to 0.2028 (+8.2%), and AV-IB from 0.2027 to 0.2111 (+4.1%).

2. JavisDiT++: SWIM@360 improves JavisScore from 0.1125 to 0.1695 (+50.7%), AVH from 0.2098 to 0.2357 (+12.3%), and AV-IB from 0.2218 to 0.2442 (+10.1%).

These gains persist at higher budgets. Compared with Uniform@600, SWIM@560 improves JavisScore by 56.1% on LTX-2 and 44.0% on JavisDiT++, alongside improvements in AVH (+6.7%/+10.8%) and AV-IB (+4.3%/+9.6%), respectively. All these improvements are achieved with 40 fewer NFE.

To further isolate the contributions of each component, we compare Uniform particle search, Uniform+MH, and full SWIM.

Table 1. Component-wise ablation on LTX-2.

MethodAV-IBAVHJSCLAPVR-TA
Uniform@6000.20270.18790.07060.14040.7414
Uniform+MH@5600.20990.19740.10580.13280.6165
SWIM@5600.21140.20040.11020.14980.6193

Table 2. Component-wise ablation on JavisDiT++.

MethodAV-IBAVHJSCLAPVR-TA
Uniform@6000.22330.21110.11490.18840.6716
Uniform+MH@5600.24080.23040.16490.18940.5325
SWIM@5600.24480.23400.16550.19730.5299

The results show that Uniform concentrates on VR-TA at the expense of other alignment objectives, whereas MH refinement substantially improves joint audio-visual alignment. Adaptive allocation further improves the results without increasing NFE. In particular, CLAP improves by 12.8% on LTX-2 and 4.2% on JavisDiT++ compared with Uniform+MH.

Overall, the results demonstrate that SWIM achieves stronger multi-objective alignment than Uniform across both backbones and compute budgets, rather than optimizing a single reward at the expense of other alignment metrics.

AuthorsOpenReview ↗

Rebuttal (3/4)

W3. Human evaluation margins on LTX-2

Although the preference margins on LTX-2 are smaller than those on JavisDiT++, SWIM receives higher preference shares than ARW-based EvoSearch across all six criteria on both backbones.

For LTX-2, we report the following results from Figure 2:

CriterionEvoSearchSWIM (Ours)
Text-Visual25.52%32.59%
Text-Audio23.43%26.70%
Audio-Visual23.04%26.57%
Visual Aesthetics35.47%36.39%
Audio Aesthetics29.06%32.98%
Overall37.30%42.93%

We highlight two observations:

1. SWIM is preferred across all alignment criteria.

On LTX-2, SWIM achieves higher preference shares for text-video (+7.07 percentage points), text-audio (+3.27), and audio-video alignment (+3.53). It also achieves a 5.63 percentage-point advantage in overall preference. The visual aesthetics preference shares are similar, indicating that the alignment improvements are achieved while maintaining comparable visual quality.

2. The preference improvements are consistent across both backbones.

On JavisDiT++, SWIM achieves substantially higher overall preference (65.59% vs. 24.29%), along with higher preference shares for text-video (51.96% vs. 26.18%), text-audio (44.94% vs. 22.54%), and audio-video alignment (41.43% vs. 25.64%). These improvements also extend to visual aesthetics (66.80% vs. 22.00%) and audio aesthetics (51.01% vs. 28.88%).

The human evaluation results are consistent with our quantitative findings. On LTX-2, SWIM improves JavisScore by 61.1% compared with Uniform@400 at a lower NFE budget, with corresponding improvements in AVH and AV-IB. Moreover, compared with ARW-based EvoSearch@532, SWIM@560 improves JavisScore from 0.1095 to 0.1102, AVH from 0.1949 to 0.2004, and AV-IB from 0.2109 to 0.2114.

Together, the human preference results and automatic metrics provide consistent evidence that SWIM improves audio-visual alignment on both backbones.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.