acceptodds
Under review as a conference paper at ICLR 2027

Parallel Causal Denoising: From Exactness to Batch Selection

Abstract

Parallel generation reduces waiting in sequence models, but denoisers restricted to current and earlier positions can distort relationships when dependent positions are generated together. For Gaussian data and a fixed optimal denoiser, we establish a necessary and sufficient condition for exact generation when noise levels decrease continuously. We also quantify the final generation error, revealing why correlation alone can select the wrong grouping: it ignores how much variation remains after using earlier context. We use this analysis to develop a batching method with a near-optimality guarantee. For a specified Gaussian family with weak correlations only between adjacent positions, its final distributional error (KL divergence) is at most 1% above the optimum at the same batch count. We extend selection to trained linear denoisers using their actual sampling updates at the chosen computational budget. Gaussian experiments support the selection guarantee and show a median 5.07-fold speedup in evaluating and selecting complete groupings over a baseline using numerical integration.

Primary area:
generative models
Keywords:
causal denoising, parallel generation, diffusion models, batch selection, Gaussian transport, distributional error

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LA1vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper focuses on flow matching using causal velocities and the error that arises from when denoising tokens in parallel in this setting. They derive explicit form of this error when the target is Gaussian and decompose the KL into a sum of errors that depend on the choice of batches during generation. They use these results to design better sampling schemes that minimize this error.

Critical strengths

The paper has interesting theoretical insights in the gaussian case on the effect of restricting the velocity to be causal. It shows (at least in the gaussian case) exactly why it does not in general generate the correct distribution and they use this to have a criterium to choose the generation order (causally).

Critical weaknesses

The setting is not very well explained (in which case we would want to have these kind of causal denoisers) and more importantly, the few cases mentioned in the introduction are not seriously considered in the experiments. The gaussian results are interesting but almost all the theoretical results are limited to this very toy-ish distribution, giving little insights as to what happens in the general case, and the empirical evidence does not complement this enough.

Questions for authors

Could you test your method in actual large-scale setting where training a standard (non-causal) denoiser is not possible, or not preferable for some reason ? How much of the theory transfers from the gaussian case to other more complex distribution like the ones encountered in real settings ?

Reviewer_J3weOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The author first analyzes the conditions under which parallel sampling in diffusion models (based on Gaussian noise) can accurately estimate the distribution of the training data. When these conditions are not met, the author finds that batch selection can be used to control the final accumulated distributional error; furthermore, by applying the chain rule for Kullback-Leibler (KL) divergence, this final error can be quantitatively estimated. The author proves that batch selection can keep the final error within 1% in a near-optimal manner, and provides small-scale experimental results to support these findings.

Critical strengths

The paper offers a unique contribution, providing a quantitative analysis of parallel generation error and introducing a new theory for batch selection.

Critical weaknesses
  1. The batch selection experiments are too simple to be convincing. Many implementations are repeated only three times and involve a very small scale. This raises serious doubts regarding their practical value in the context of diffusion models sampling.

  2. Figure 1 lacks sufficient explanation, making its underlying logic difficult to follow. It requires significant revision and reorganization.

Questions for authors
  1. Could the authors analyze the impact of different batch selection strategies on the final error? What does their KL chain look like? Please provide a visual representation in the form of a figure.

  2. Is it possible to include experiments on batch selection within the context of parallel diffusion samplers used for modeling high-dimensional data distributions? Specifically, how much sampling time or the number of function evaluations (NFE) can be saved while maintaining a comparable negative log-likelihood (NLL)?

Reviewer_gSZqOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies parallel generation with causal denoisers: when multiple sequence positions are denoised together, when does the resulting sampler remain exact, and how should batch boundaries be chosen when some error is unavoidable? For Gaussian targets and a fixed optimal causal denoiser, the paper derives a necessary and sufficient condition for exact parallel generation and decomposes the final KL error into per-batch costs. It then proposes a conditional-variance-weighted batch-selection method with a near-optimality guarantee for a specified weak-adjacent-correlation Gaussian family. The approach is further adapted to trained linear denoisers and evaluated with a nonlinear model on HAPT motion-sensor sequences.

Critical strengths

1.The paper is well organized and clearly explains the connection between causal parallel denoising, exactness conditions, and batch selection.

2.Increasing parallelism while preserving causal structure is an important problem for video, time-series generation, and diffusion language models.

3.The paper analyzes final distributional error over the complete generation process. The observation that conditional variances should complement conditional correlations when choosing batch boundaries is insightful.

Critical weaknesses

1.The exactness result assumes Gaussian targets, continuous time, a fixed optimal causal denoiser, and a particular synchronous batch update. The stronger 1% near-optimality result applies only to a specific weak-adjacent-correlation Gaussian covariance family. For trained linear models, the paper obtains an approximate selection criterion based on covariance costs instead of a comparable general near-optimality guarantee, and no theoretical guarantee is provided for nonlinear models. Thus, the theoretical insight is valuable, but its extrapolation to practical generative models remains limited.

2.The only real-data experiment uses 32-point HAPT accelerometer sequences, a fixed eight-batch configuration, and a finite candidate-partition pool. Response-based selection improves mean energy by only 0.223% over full validation search, without uncertainty analysis across models, test subjects, or sampling randomness. This evidence is currently insufficient to establish practical value for complex, high-dimensional generative models.

3.The 5.07x result concerns Gaussian selection preprocessing relative to numerical integration. The HAPT comparison of 3.701s versus 15.479s also measures selection cost only and excludes model loading and subsequent generation. The paper does not show an end-to-end quality-latency trade-off relative to sequential generation, equal-size blocking, or existing parallel decoding methods.

4.The experiments mainly compare correlation, a weighted score, Gaussian cost, equal batches, and exhaustive validation search. Although the paper discusses DEMASK, Block Diffusion, and Diffusion Forcing, it remains unclear how much benefit the proposed selection rule provides over compatible dependency-guided or dynamic-chunking methods.

Questions for authors

1.The 1% selection guarantee applies only to a specific weak-adjacent-correlation Gaussian family. Can the authors clarify which aspects of the analysis remain valid for general Gaussian covariance structures, trained nonlinear denoisers, and non-Gaussian sequence distributions? Are there empirical failure cases where the proposed bound or selection rule becomes unreliable?

2.In the HAPT experiment, response-based selection improves energy by only 0.223% over full validation search. Please report uncertainty estimates across models, subjects, selection contexts, and sampling randomness, and clarify whether this difference is statistically meaningful.

3.Can the authors evaluate the method on a more challenging sequential-generation setting, such as longer human-motion sequences, a larger time-series dataset, or a small-scale video, audio, or diffusion-language-model setting? If full-scale experiments are costly, a controlled subset or shorter benchmark would still help test whether the selection method scales beyond 32-point HAPT sequences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.