acceptodds
Under review as a conference paper at ICLR 2027

LatticeSMC: Where to Spend Inference-Time Compute in Chunked Sequence Generators

Abstract

Long-form generators for music, motion and video produce a sequence chunk by chunk, each chunk by iterative denoising, while the rewards that matter are defined on the whole sequence. Existing inference-time steering methods act on one axis of this process at a time, best-of-N at the end, Feynman-Kac steering across denoising steps, streaming pipelines across chunks, and have been compared under different return rules at unmatched compute. We introduce budget-matched chunked steering, the task of maximizing a sequence-level reward with a chunked generator under a fixed number of denoiser evaluations, and propose LatticeSMC, a sampler derived from a Feynman-Kac model on the two-dimensional lattice of chunk index and denoising step. Two exact telescoping results make the design derivable: for chunk-additive rewards the two axes carry identical weights, so resampling belongs wherever lookahead is cheapest; for terminal rewards any prefix score yields an exact intermediate potential, so a prefix-evaluable reward is a twist with no estimation and no extra denoiser evaluations. LatticeSMC resamples properly on these exact potentials, at chunk boundaries and, when the plug-in reward is free, within chunks, and returns either a draw from its particle approximation of the reward-tilted target or its best particle, and recovers Feynman-Kac steering as its within-chunk schedule. Under matched compute, on a music-to-dance diffusion model and a 40-second text-to-music model, it raises beat alignment from 0.234 to 0.441 (best-of-N: 0.354) and prompt adherence from 0.470 to 0.560 at 32 particles with held-out quality metrics at base, holds its lead on the long-range reward at six and eight chunks, and is preferred by human raters over every comparator in 60 to 77 percent of pairwise judgements. Two measurable properties of the reward settle what remains: how hard to commit at a boundary follows the information in its exact potential, which the effective-sample-size gate reads off and greedy pruning cannot, and whether content-axis lookahead pays follows the within-set predictability of future reward, measured before any particle is spent.

Primary area:
generative models
Keywords:
Inference-Time Scaling, Sequential Monte Carlo, Feynman-Kac Steering, Reward-Guided Generation, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_ThJoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of budget-matched inference-time steering for chunked sequence generators (e.g., autoregressive chunk diffusion and flow-matching models for motion and audio). Generating long sequences in chunks creates a two-clock process: denoising steps within a chunk and sequential chunk progression. The authors formulate this process as a Feynman-Kac model on a 2D lattice of chunk index and denoising step. They establish two core telescoping results: (1) for chunk-additive rewards, denoising-axis and chunk-axis potentials multiply to the identical target, meaning resampling should occur wherever reward evaluation is computationally cheapest; (2) for terminal rewards, prefix-score increments with a terminal correction telescope exactly to the reward-tilted target, yielding exact twists without learned estimation.

Based on these results, the authors propose LatticeSMC, a training-free sequential Monte Carlo sampler with ESS-gated resampling at chunk boundaries and optional within-chunk dense steps when plug-in scoring is free. They further provide a within-set predictability diagnostic explaining when lookahead compute is beneficial versus counterproductive.

Validation is carried out under strictly matched denoiser evaluations across two modalities: a music-to-dance diffusion model on AIST++ (evaluating beat alignment and long-range repetition) and a 40-second text-to-music flow-matching model (evaluating prompt adherence and motif recurrence). LatticeSMC substantially outperforms Best-of-N, greedy chunk pruning, and Feynman-Kac steering (e.g., beat alignment 0.441 vs. BoN 0.354; prompt adherence 0.560 vs. BoN 0.530 at 32 particles). Results are corroborated across model scales (4M to 38M), horizons up to 8 chunks, held-out non-steered quality metrics, and double-blind human preference evaluations.

Critical strengths
  1. Unifying Theoretical Foundation via Exact Telescoping Properties: The 2D Feynman-Kac lattice formulation is a seminal conceptual contribution. By establishing Proposition 1 (axis equivalence for chunk-additive rewards) and Proposition 2 (exact prefix-score twists for terminal rewards), the paper replaces ad-hoc heuristics with a principled framework where the target distribution is provably preserved without requiring learned estimation networks. This directly resolves where and how resampling should occur.
  2. Methodological Rigor and Diagnostic Clarity on Lookahead Compute: The paper establishes a rigorous experimental standard by evaluating methods under matched computational budgets and identical return rules (sampling draw vs. search argmax). Crucially, Section 4 introduces the within-set predictability metric \rho_k, uncovering why traditional learned regressors degrade SMC performance despite high cross-prompt R^2. This diagnostic insight prevents common pitfall practices in diffusion guidance.
Critical weaknesses
  1. Lack of Validation on Large-Scale Modern Diffusion/Video Models: While the paper motivates long-form chunked generation using streaming video and large foundation models, the experiments are conducted on relatively small, custom-trained motion diffusion models (4M and 38M parameters) and an audio flow-matching model. Validating whether the findings hold on multi-billion parameter open-weights video generators (e.g., Open-Sora, CogVideoX, or streaming video diffusion) would significantly strengthen its empirical impact across broader generative AI domains.
Questions for authors
  1. In Section 3.4 and Table 3, the authors show that for beat alignment normalized over sequence length, holding the per-boundary tilt fixed across horizons requires scaling the temperature as \alpha \propto 1/K. How should practitioners automatically adapt \alpha when dealing with hybrid rewards that combine a sequence-averaged component with an additive cumulative component?
  2. When applying LatticeSMC with search mode (argmax return), did the authors observe particle diversity collapse across early chunks when N is scaled up to 64 or 128? Does systematic resampling maintain sufficient genealogical diversity across long horizons (K >= 8)?
  3. Have the authors considered integrating gradient-based guidance (e.g., DPS or classifier guidance) alongside LatticeSMC resampling, and does the 2D lattice formulation accommodate continuous gradient drifts during the denoising steps?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ThJo

Thank you for the encouraging assessment. We ran an experiment for each of your three questions. Your "Section 3.4 and Table 3" are Section 4.4 and Table 2 in the revised PDF.

W1, large video models. The model and both propositions are stated for any chunked generator, and our pruning baseline is the selection rule of streaming video pipelines. A matched-budget study on a model such as CogVideoX (32 particles, forty conditions, several seeds, a video reward model in the loop) is more than we can run in the discussion period, so the Limitations now name it as the next testbed and we do not promise it here. On scale within the study: the music model is Stable Audio 3 Medium, with a 1.4B-parameter transformer (now named in Section 4.1), and every ordering on dance holds from 4M to 38M parameters.

Q1, hybrid rewards (new Appendix D.9, Table 13). Potentials multiply, so each component can carry its own temperature. We implemented the automatic rule (bisection on each alpha so that the component's ESS at a boundary equals N/4, from base samples) and ran it on beat alignment plus repetition.

  • The rule recovers the scaling of our horizon study: alpha for beat alignment is 0.018 at K = 4 and 0.011 at K = 8 (close to 1/K), and alpha for repetition is 0.029 and 0.031 (constant).
  • It does not raise the sum: +0.009 [-0.001, +0.023] at K = 4 and -0.041 [-0.096, +0.022] at K = 8 against one shared alpha. Per-component temperatures act as component weights and move reward from repetition to beat alignment.
  • Our recommendation is therefore one shared alpha for a hybrid reward, with separate alphas as the dial for a preference between components. With the shared alpha the boundary schedule leads chunk pruning on the sum at both horizons (+0.024* and +0.088*).

Q2, diversity at larger N and K (new Appendix D.10, Table 14). We logged parent indices and ran N = 64 at K = 4 and 8, and N = 128 at K = 4.

  • Surviving first-chunk ancestors double with N in every setting. On beat alignment at K = 8 and N = 64 the gated schedule keeps 16.3 after the last boundary against 2.9 for pruning (+13.4 [+11.4, +16.1]).
  • Lineages do concentrate where the reward is informative and the horizon long: on repetition at K = 8, 2.2 ancestors remain at N = 64 and 44% of sequences end on a single ancestor (52% under pruning).
  • This does not cap the reward. From N = 32 to 64 the argmax reward rises in all five settings (+0.028 to +0.098, every interval excluding zero), and N = 128 adds a further 0.020.
  • The return rule acts after the last chunk, so argmax return does not change these dynamics.

Q3, gradient guidance (new Appendix D.11, Table 15). Guidance changes the transition kernel, and both propositions are statements about potentials that hold for any kernel, so the lattice accommodates gradient drifts under every placement.

  • Guidance alone (DPS-style, prompt adherence): one guided sample reaches 0.641 against 0.560 for LatticeSMC at N = 32, at roughly a third of the wall-clock, with lower seed diversity (0.047 against 0.061 for the base model). It needs a reward that is differentiable through the decoder (18.7 GB for one sample), and larger step sizes raise the CLAP score through high-frequency energy.
  • Guidance as the kernel of the lattice (N = 4, 20 prompts): the potentials telescope exactly (deviation 1.4e-14). The guided boundary schedule reaches 0.651, against 0.625 for one guided sample on the same prompts (+0.025 [+0.013, +0.039]) and 0.647 for best-of-4 over guided chains (+0.004 [-0.004, +0.011], not separated). A population adds to guidance; the extra gain of resampling over best-of-N is small at this N.
Reviewer_fLieOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In this work, the authors address the problem of how to improve chunk-based music (and motion and video) diffusion/flow generation methods. To do so, the authors apply a sequential Monte Carlo methodology to help decide how to allot inference-time compute for the case of pretrained models that have an ability to perform length extension (continuations). The authors compare selection with a pool of candidates and scoring A) fully completing the sequence (long form generation) vs. B) generating per chunk and scoring after each chunk and C) additional during denoising within each chunk, along with testing reweighting during resampling. Experimental evaluation is performed for music-to-dance and text-to-music generation with objective and subjective evaluation. Findings show that B is preferred for music, no video experimentation is done.

Critical strengths

The work compares selection after full-sequence generation, after each chunk, and during denoising, with objective and subjective evaluation on music-to-dance and text-to-music generation. The finding that chunk-boundary selection is preferred for music gives a concrete comparison of where to allocate inference-time compute. I found no obvious technical errors or issues.

Critical weaknesses
  1. Practical motivation, positioning, and clarity of the contribution. I struggle with the overall motivation and positioning of the work along with the experimental design. Related work is heavily focused on SMC, and I struggle to clearly understand the contributions of the lattice Feynman-Kac model versus past work. There is no related work or background section, so it is hard to get more context about the intended research community for the contribution, which I assume is the sequential Monte Carlo area.

There are now many fast generation methods for music, including distribution matching distillation (DMD), its application to music (Presto), consistency trajectory models, adversarial post-training (ARC), and more that significantly speed up music diffusion models, including Stable Audio 3. While these methods are complementary, the paper should provide more background and commentary on them. It is noted in the appendix that Stable Audio 3 is actually used for the text-to-music experiments. The paper should also better contextualize whether the separately trained 4M- and 38M-parameter dance models are 'fast' distilled models and more prominently evaluate overall speed using something like real-time factors, even though there is some discussion and a plot. What I am looking for is a simple comparison of full-song quality versus speed against chunk-wise generation with the SMC method, which is hard to understand now.

Sentence-level phrasing and overall organization are fine. However, while I have worked on chunked diffusion generation methods for a significant amount of time, I find the terminology difficult to read. SMC terms such as potentials, twists, resampling, telescoping, and ESS-gated resampling, together with paper-specific terms such as lattice, axis equivalence, and commitment, could be defined more clearly and simply. The paper could first introduce and define the SMC terms, then introduce its added terms.

  1. Impact of the empirical results beyond simpler selection. The main finding appears to be that scoring and selecting candidates after generating each chunk of a longer song improves the reward. This methodology, however, adds substantial complexity and inference cost for scoring, and it is unclear how much additional benefit comes from the proposed SMC formulation rather than simply selecting promising prefixes more frequently. The top-1 pruning result outperforms the recommended LatticeSMC setting on multiple metrics. This makes the added value of the proposed formulation difficult to assess.
Questions for authors

How much additional benefit comes from the SMC formulation rather than selecting promising prefixes more frequently? The introduction states that the proposed method matches or outperforms every method on every reward, but top-1 pruning performs better on some metrics. How should 'matches' be interpreted here?

2 comments
AuthorsOpenReview ↗

Response to Reviewer fLie

Thank you for the careful reading. Your comments led to the largest changes in the revision, including two new experiments.

W1, positioning and background. The main text now has Section 2, Related Work. It names the audience, practitioners who scale inference-time compute on chunked music, motion and video generators, and states what is new: Feynman-Kac steering selects within one denoising pass, streaming pipelines select greedily across chunks, and the two had been neither related nor compared at equal compute. The lattice model contains both, shows when they are the same sampler (Proposition 1), gives exact chunk-level scores for whole-sequence rewards (Proposition 2), and compares the placements at matched budget.

W1, fast generation. Section 2 now discusses consistency trajectory models, DMD, Presto, ARC and Stable Audio 3 (extended in Appendix A). We agree they are complementary: distillation lowers the cost of one sample, steering decides how to spend several. Section 4.1 now names Stable Audio 3 Medium (8 steps, adversarial post-training) and states that the 4M and 38M dance models are standard 50-step diffusion models without distillation.

W1, speed and the full-song comparison (new Table 3). On one A6000:

  • Music, 40 s: one-pass base runs at RTF 0.06 and chunk-wise base at 0.08. At N = 32, best-of-N runs at RTF 2.3, chunk pruning and LatticeSMC's boundary schedule at 5.1, and Feynman-Kac steering at 18.4. The boundary schedule delivers the reward of Feynman-Kac steering at 0.28 of its wall-clock; 85% of its time is the decoder.
  • Dance, 20 s: every method at N = 32 runs faster than real time (RTF 0.43 to 0.44).
  • Full song against chunk-wise: Stable Audio 3 generates far more than 40 s in one pass. We run it chunk by chunk because placement across chunks is our object of study, and we now say so. With the same 1,024 denoiser evaluations, 128 decodes and 128 CLAP calls, one-pass best-of-128 reaches 0.552 and chunk-wise LatticeSMC 0.560 (paired +0.008 [+0.002, +0.014]; +0.013 [+0.007, +0.019] at equal wall-clock). The two base clips are equally adherent (0.469 and 0.470), so the gain comes from selecting at boundaries.
  • Where one pass wins, we report it: on motif recurrence, a reward that arrives with the last chunk, one-pass best-of-128 is ahead by 0.011*, and one-pass clips are smoother at the 10 s marks (the chunk-wise clips have higher PQ). Chunk-wise steering is the only route with a prefix to continue, a stream, or a sequence longer than the window.

W1, terminology. Section 2 defines potential, weight, tilt, resampling, ESS gating, twist, telescoping and Feynman-Kac model in plain terms, then our four terms (lattice, placement, axis equivalence, commitment). The introduction glosses potentials and resampling at first use.

W2 and your question: how much does the SMC formulation add over selecting prefixes more often, and what does "matches" mean? You are right about the sentence. It now reads "every method of the matched main comparison", and top-1 selection is a row of Table 1.

Top-1 selection is the zero-temperature corner of our boundary schedule, so we ran the experiment your question calls for: the ESS-gated sampler at a sharp tilt (new Table 11, N = 32, same conditions and budget).

  • At a chunk boundary, on rewards every boundary carries, the two coincide. Beat alignment 0.468 against 0.473 for top-1; prompt adherence 0.564 against 0.564. Here the formulation adds nothing over frequent prefix selection, and the paper now says so.
  • On rewards that arrive late, the same sampler keeps its lead. Repetition 0.944 against 0.850 (+0.094 [+0.066, +0.126]); motif recurrence 0.926 against 0.909 (+0.017 [+0.012, +0.022]). The gate does not fire at boundaries that carry no reward, so all 32 prefixes reach the first informative boundary; top-1 has committed by then. One setting serves all four rewards, where no single M does.
  • Resampling inside the chunk has no counterpart in prefix selection. At the same tilt it raises beat alignment to 0.528 (+0.056 [+0.038, +0.075] over top-1) and repetition to 0.991 (+0.141 [+0.112, +0.171]), at the same denoiser evaluations.
  • The formulation also supplies exact prefix potentials for terminal rewards (Proposition 2) and a sampling mode.

Thanks for the detailed response and revision. The new related work, definitions and speed table help address some of presentation concerns, and I appreciate the disclosure about the repeated segments.

However, the new results confirm my main concern. Table 11 shows SMC adds nothing over top-1 prefix selection when rewards arrive at every boundary, and on music the gain over one-pass best-of-128 is +0.008 at roughly 85x the base RTF. The cases where SMC helps (late-arriving rewards, within-chunk resampling) seem much narrower than the paper claims. The human eval also still compares against top-N/4 rather than top-1.

I'm lowering my score to 1. This is because of the new results, not the presentation, which has improved. A paper focused on when SMC beats greedy selection would be more convincing.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the reviewers for their time. A revised PDF is uploaded, with changed text in blue. The main text uses the tenth page, and sections are renumbered because of a new Section 2.

  1. New Section 2, Related Work (main text): positioning against prior inference-time steering; fast generators (consistency trajectory models, DMD, Presto, ARC, Stable Audio 3); plain definitions of the SMC terms, then of our own. Appendix A gains a paragraph on fast generation. (Reviewer fLie)
  2. Section 4.1 names the generators. The music model is Stable Audio 3 Medium at its shipped 8 steps; the dance models are 50-step diffusion models without distillation. (fLie)
  3. New Table 3 and a Speed paragraph (Section 4.4): seconds per sequence and real-time factors for every method on a dedicated GPU, with one-pass generation of the full 40 s as the reference. Details in new Appendix D.8 and Table 12. (fLie)
  4. New experiment on SMC versus frequent prefix selection (Table 11, Appendix D.7): the ESS-gated sampler at a sharp tilt against top-1 selection. Top-1 selection and the sharp-tilt rows are now in Table 1, the contribution statement reads "every method of the matched main comparison", and Section 4.4 has a paragraph on what the formulation adds. (fLie)
  5. New Appendix D.9 and Table 13: hybrid reward with per-component temperatures and an automatic ESS-target rule. (Reviewer ThJo)
  6. New Appendix D.10 and Table 14: lineage counts and reward at N = 64 and 128, K = 4 and 8. (ThJo)
  7. New Appendix D.11 and Table 15: gradient guidance alone and as the transition kernel of the lattice. (ThJo)
  8. A disclosure we add ourselves. While rebuilding the horizon tables we found that the K = 6 and K = 8 condition sets contain repeated music segments (31 of 40 and 11 of 16 are distinct). Appendix D.6 now states this and gives the comparisons with the distinct segment as the bootstrap unit: no difference changes sign, and every comparison quoted in the main text keeps its status.
  9. Layout. To make room, the crossover table and the twist table moved to Appendices D.4 and C.2; their numbers remain in the text. The Limitations now state the video scope and the music horizon precisely.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.