acceptodds
Under review as a conference paper at ICLR 2027

S2PD: Serial-to-Parallel Diffusion for Physically and Logically Consistent Video Generation

18 views

Abstract

Bidirectional video diffusion models denoise entire videos in parallel, yet when trained on effectively unlimited in-distribution data from procedural generators, continue to violate physical laws and simple symbolic rules. We introduce Serial- to-Parallel Diffusion (S2PD), which performs autoregressive diffusion at high noise before switching to parallel diffusion at low noise. The autoregressive phase provides the serial computation needed to coordinate interdependent events and produce valid state transitions while the parallel phase jointly refines the entire video and reduces sampling time relative to fully serial generation. We imple- ment S2PD with two architectures: a pixel-space diffusion transformer trained from scratch and a pretrained video model adapted through LoRA fine-tuning with causal attention. Across games, physical simulations, and real video, S2PD follows rules more reliably than matched bidirectional baselines and generates videos with greater temporal stability and sampling efficiency than other serial methods.

Primary area:
generative models
Keywords:
video generation, physical plausibility, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fHgwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Serial-to-Parallel Diffusion (S2PD), which performs autoregressive block-causal diffusion: it establishes video structure at high noise levels and switches to parallel diffusion at low noise levels to refine details. Under the experimental conditions set by the authors, this method achieves better physical and logical consistency than pure block-causal diffusion.

Critical strengths

The paper's final experimental results show that the authors' S2PD-based scheme indeed achieves better performance on multiple evaluation metrics than the pure block-causal method.

Critical weaknesses
  1. Simply put, the method adds parallel refinement during sampling on top of the block-causal scheme, and the Method section is too brief overall. The authors consistently emphasize that the serial approach achieves better physical consistency, but what they actually do is add parallel refinement during sampling to the serial method, which makes the source of the performance gains in Table 2 unclear. In addition, in describing video causality, the authors do not distinguish causality among tokens within a single frame from causality among tokens across different frames, which undermines the academic rigor of the paper.

  2. The experimental setup is unreasonable in places, and the experimental results contradict some of the claims. First, compared with the block-causal method, S2PD additionally changes the training to "train with multiple block sizes." This substantively introduces two variables (parallel diffusion and multiple block sizes) into the experiment at the same time, and it is unclear which operation actually improves the consistency metrics. Second, why are different datasets used to train the two architectures, DiT-B and Wan-5B-LoRA? Normally, network architecture and dataset should be independent; different architectures should be trained on the same dataset for the comparison to be more informative.

Questions for authors
  1. In fact, the effects of causality among tokens within a single video frame and among tokens across different frames on consistency should be considered separately, yet you seem to have completely ignored this issue? In the Table 6 experiments, are orderings such as raster and zigzag defined with respect to a single group of image tokens?

  2. Where is the experiment supporting this claim: "we demonstrate that scaling training data does not enable bidirectional diffusion models to model complex video"? All the experiments seems to be conducted under the same total training budget.

  3. The main contribution of this paper is adding parallel refinement during sampling after the block-causal scheme, but the overall motivation of the paper argues that the serial scheme has better physical consistency. Does this deviate from your actual contribution?

3 comments

We thank the reviewer for their feedback and first correct a fundamental misunderstanding in their Paper Summary, before addressing each concern in detail.

Our method does not “achieve better physical and logical consistency than pure block-causal diffusion”. We demonstrate that serial methods dramatically outperform bidirectional diffusion under matched training compute budgets. Then, we show our method retains the same physical and logical consistency as other serial methods while being 2x as fast.

W1 Part 1, Q3) We first establish that serial computation is essential for logically consistent video generation, then address its high runtime cost by replacing most serial steps with a parallel refinement stage. We demonstrate that this approach substantially accelerates sampling while preserving the benefits of serial computation, all within a single block diffusion model. We modify the abstract and switch the order of our first two contribution bullet points to make the flow more clear. We expand our Method section by adding Section 3.2 Baselines, dedicated to explaining our bidirectional and serial baselines. In Section 3.3 Training, we explicitly emphasize that Flex-Forcing introduced mixed-block-size training, which we adopt for all our S2PD configurations. We clarify that our contributions are establishing the importance of serial computation for coherent generation and the serial-to-parallel sampling scheme, not the mixed-block-size training procedure.

W1 Part 2, Q1) We are not entirely certain we understand the question. To avoid confusion, we almost completely removed the word "causal" from the paper, renamed the block-causal diffusion method (BC) to its paper name, Block Diffusion (BD). We focus our ablations on hyperparameters that affect the model's ability to capture dependencies in the data like token order and block size. Token order specifies the global ordering of tokens within the 3D spatiotemporal volume and is determined before partitioning the sequence into blocks. We hope this clarifies our approach and welcome further questions if we have not fully addressed the concern.

W2 Part 1) We acknowledge the confusion and train dedicated models with the block size mixture {64, N_v}, denoted Pin-64, for a controlled comparison. Table 2 now reports S2PD results using both Pin-64 and Pin-8 models. Pin-64 isolates the contribution of parallel diffusion, while Pin-8 captures the effect of reducing block size. We find Pin-8 is preferable for discrete datasets with complex rules, where the additional serial computation improves logical consistency. In contrast, Pin-64 is preferable for video datasets with high-frequency textures, where the need for serial computation is lower.

W2 Part 2, Q2) We add experiments comparing both architectures on the same dataset in Appendix A.1, Model Scaling. We find that bidirectional diffusion models improve only marginally with increased model scale and continue to substantially underperform serial methods. In contrast, Table 8 demonstrates that S2PD effectively leverages increased model capacity, reducing its CD-FVD score on Rubik’s Cube Real from 335.2 to 64.4.

Important changes to the paper are marked in purple. We hope all your concerns have been addressed. If your understanding and the clarity of the paper as well as the soundness of the experiments has improved, please consider raising your score. Thank you!

Thank you for the detailed response and additional experiments. We acknowledge that the statement in our Summary that S2PD achieves better physical and logical consistency than pure BC was inaccurate. We correct it using the paper’s own wording on page 2:

We introduce S2PD, which preserves the physical plausibility and logical consistency of serial generation while improving sampling efficiency and temporal stability.

Three concerns remain:

  1. Within-frame versus cross-frame relationships (Q1). Our question concerns whether causal relationships or dependencies among tokens within a frame should be distinguished from those across frames, and whether they affect consistency differently. The explanation of global token ordering and partitioning does not directly address this point. Please clarify the role of within-frame serial computation and how the existing experiments distinguish its contribution from that of cross-frame serial computation.

  2. Pin-64 training configuration. The revised paper specifies that Pin-64 is trained with block sizes {64, N_v}, but it remains unclear whether the BD baseline uses the same configuration. If BD is trained only with block size 64, both training and sampling differ, so the comparison does not fully isolate the contribution of parallel refinement. Please specify the block sizes used to train the BD baseline.

  3. Training-data scaling. Model scaling and training-data scaling are different questions. Our original Q2 concerned the latter, which the added model-scaling experiments do not directly address. Please point to the corresponding data-scaling experiments or qualify the conclusion to reflect the tested data and training budgets.

We thank the reviewer for their quick response and their additional questions.

C1) Our experiments (Table 5) show that within-frame seriality is primarily important for the discrete datasets, where a block size of 8 that does have within-frame seriality outperforms block size of 64 that does not have any within-frame seriality. On the other datasets, within-frame seriality does not appear to be important. Thus, we believe the optimal choice of how much serial computation to use depends on the dataset.

C2) BD is trained and sampled with a block size of 64, corresponding to one temporal chunk of four frames. S2PD with Pin-64 is trained with block sizes {64, Nᵥ} and uses a block size of 64 during its serial phase. We consider this a fair comparison of our sampling strategy.

To further isolate the contribution of our parallel refinement stage, we add a new ablation in Appendix C.1, Parallel Refinement Ablation, which compares sampling schemes using the same checkpoint. The results further support our arguments. Specifically, we compare serial-only and serial-to-parallel sampling with the same Pin-64 checkpoints. The serial-only sampling is the same sampling scheme used by cDF and BD.

On discrete and continuous datasets, serial-to-parallel sampling improves sampling speed with minimal impact on consistency. On video datasets, Pin-64 (serial-only) exhibits the same temporal flickering illustrated in Figure 7. With the checkpoint held fixed, serial-to-parallel sampling improves CD-FVD while achieving a 2× speedup. We show qualitative results in Figure 10 and report the quantitative results in Table 11.

C3) Figure 8 in Appendix A.1 directly addresses data scaling through training for longer on unique data. For our procedurally generated datasets, each epoch uses a different random seed and thus adds unique data to the training, scaling the dataset size. In Figure 8, extending training from epoch 10 to 20 doubles the training dataset size. We now explicitly clarify it in Section 4.1 and changed the title Appendix A.1 to Model and Data Scaling. The results clearly indicate that training-data scaling does not address the limitations of the bidirectional models. Serial methods, even at much lower training budgets, outperform bidirectional models.

Please let us know if this response satisfactorily addresses your concerns. Thank you!

Reviewer_D37dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that bidirectional video diffusion models break simple rules even with unlimited procedural data because they lack serial computation. The authors build 12 rule-checkable game and physics datasets, each with a parser that counts invalid transitions, and add four video datasets. They compare a bidirectional baseline with causal Diffusion Forcing (cDF), block-causal diffusion (BC), and their method, S2PD. S2PD denoises blocks autoregressively down to τ=0.6 and then refines the whole video jointly. Models are a 132.7M DiT-B trained from scratch and Wan-5B with LoRA.

Critical strengths

The benchmark is likely to be reused. Each frame is parsed into a symbolic state and checked against the rules, so rule following is measured directly, without FVD or pixel alignment. The residual scoring in Eq. 2 avoids re-penalizing inherited errors. The gap between serial and parallel generation is large and holds on every synthetic dataset, with training token budgets matched. On Chess, invalid transitions per rollout drop from 48.1 to 3.3–7.2. On the Wan video datasets at block size 64, S2PD beats cDF and BC on CD-FVD on all four datasets (e.g., MOVi-C 229.0 vs. 450.0/470.4) in about half the sampling time (6.8–7.1 s vs. 13.4–13.8 s). The same model sampled fully in parallel (τ=1.0) also matches the dedicated bidirectional model (Chess 48.7 vs. 48.1), so mixed block-size training costs nothing in parallel mode.

Critical weaknesses
  1. Main reservation. The first contribution claims S2PD keeps the rule-following of serial methods while sampling faster. On the game datasets, the paper's own tables do not show both at once. In Table 2, S2PD uses block size 8 (its numbers match Table 5, Flex block 8), while cDF and BC use block 64, so S2PD gets eight times as many serial steps. With the same block size of 64, S2PD makes many more errors (Table 5, Chess: 20.2 vs. 3.3 for cDF and 5.7 for BC). With block size 8, its errors get close (Chess 7.2), but it takes 10.4 s. cDF/BC times on DiT-B are not reported; fully serial sampling at block 64 takes 3.4 s (Table 4). Please report cDF/BC sampling times and train S2PD with {64, N_v} for a matched comparison.
  2. How many rollouts are evaluated per dataset? Please report variance. The same configuration gives different numbers in Tables 4 and 5 (Chess at block 64: 19.6 vs. 20.2), and this difference is as large as some gaps between serial methods in Table 2.
  3. What do the parsers report on ground-truth simulator videos? This matters most for Rubik's Cube 3D, the only discrete dataset where all serial methods stay high (Table 2: 30.5 / 34.6 / 24.9 for cDF / BC / S2PD). Without a ground-truth floor, model errors cannot be separated from parser errors.
Questions for authors

see the weaknesses above

1 comment

We thank the reviewer for their feedback and careful questions around evaluation. We note their kind words like “benchmark is likely to be reused”, before addressing each concern in detail.

W1) We train an additional set of models with configuration {64, N_v}, denoted Pin-64, and update Table 2 to report results for both Pin-64 and Pin-8. We note that utilizing spatial autoregression by reducing block size from 64 to 8 has modest improvements limited to only discrete datasets. Pin-64 outperforms Pin-8 on continuous and video datasets. We also add DiT-B sampling times for all serial methods to Table 3. Our method remains consistently 2× faster across model architectures.

W2) Table 4 previously used a sample size of 256, while all other tables used 1,024. We now consistently use 1,024 samples across all tables and explicitly state this in the text. We also update Table 2 to report both the mean and standard deviation.

W3) We agree that a ground-truth floor is necessary. We spend a great deal of effort explaining and finalizing our evaluation pipeline in Appendix B. In Table 9, we evaluate 1,024 ground-truth videos per dataset through the parsing and consistency-checking pipeline. All seven discrete datasets, including Rubik’s Cube 3D, have zero detected invalid transitions. The high errors on generated Rubik’s Cube videos therefore cannot be explained by a similarly high ground-truth evaluation floor.

Important changes to the paper are marked in purple. If you think our synthetic datasets and evaluation are sound and valuable to the community, please consider raising your score. Thank you!

Reviewer_X4tHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

S2PD combines autoregressive denoising at high noise levels with parallel refinement at low noise levels. Training with multiple block sizes supports both phases within one model. Experiments on games, physical simulations, and video datasets examine whether this schedule retains the consistency benefits of serial generation while reducing its sampling cost.

Critical strengths
  1. The hybrid schedule addresses a concrete limitation of serial video generation. It reserves sequential computation for establishing video structure and batches the remaining denoising over the full sequence. The implementation works with both a model trained from scratch and a pretrained Wan model adapted through LoRA.
  2. The quality and speed comparison is persuasive. At block size 64, Table 3 reports lower CD-FVD than both serial baselines on all four video datasets, at roughly half the sampling time. On MOVi-C, S2PD scores 229.0 in 6.8 seconds, compared with 450.0 in 13.7 seconds for cDF. This is a useful improvement over an already serial baseline.
  3. The evaluation examines consistency directly. The game rules and simulator dynamics provide interpretable checks on generated transitions. Comparisons use matched model sizes and approximately matched training FLOPs. The comparison between serial and bidirectional generation gives the empirical study value beyond the proposed sampling schedule.
Critical weaknesses
  1. The conclusion about data scaling is too broad. An unlimited procedural data source does not mean unlimited training. The reported runs use 100,000 optimizer steps, and Figure 4 does not vary model capacity or compute. These experiments show persistent errors under the tested budgets. They do not establish that additional data or compute cannot help bidirectional diffusion. The claim should be narrowed accordingly.
  2. The contribution of parallel refinement is not fully isolated. Table 2 mainly establishes the advantage of serial generation; S2PD is not uniformly better than the serial baselines. For example, cDF makes 3.3 invalid chess transitions per rollout versus 7.2 for S2PD. Table 4 varies cached context noise together with the transition threshold. A comparison holding block size and context noise fixed would better explain which gains come from parallel refinement.
  3. The consistency claims need to reflect the evaluators' scope. Appendix B states that the 3D cube evaluator neither tracks hidden colors across occlusions nor checks full-cube solvability. CD-FVD on real video also does not verify legal transitions. The authors describe manual checks of the evaluators, but do not quantify their error rates. Evaluation sample counts and uncertainty would help assess the smaller differences between serial methods.
Questions for authors
  1. At a fixed block size and cached context noise level, how does parallel refinement change consistency and sampling time relative to fully serial sampling?
  2. How many generated videos underlie each metric, and what do the manual checks reveal about missed violations and parsing errors?
1 comment

We thank the reviewer for their feedback and their clear, concise, and correct Paper Summary. We note their kind words on how the “comparison between serial and bidirectional generation gives the empirical study value beyond the proposed sampling schedule”, before addressing each concern in detail.

W1) We acknowledge that access to unlimited data does not imply unlimited training compute and have narrowed the claim in the conclusion accordingly. We add Appendix A.1, Model Scaling, where we investigate scaling using DiT-B as our smaller model and Wan-5B-LoRA as our larger model. We find that bidirectional diffusion models continue to dramatically underperform smaller serial methods. We hypothesize that this explains why modern bidirectional video generators perform so poorly at generating gymnastics videos.

W2, Q1) We found the question unclear but provide additional analysis to address the potential concern. We add Appendix C.2, Context Noise Level Sweep. In Table 11, we sweep the context renoising noise level for cDF and BC, and the transition noise level for S2PD, from 0.0 to 1.0 in increments of 0.2, using a block size of 64. We find that cDF and BC perform best at 0.2, while S2PD performs best at 0.6. With these optimized settings, the temporal stability improvement over other serial methods is reduced but still evident, while the speed advantage remains substantial.

W3) We add Appendix B.1, Evaluation Correctness, which details the development and validation of our evaluation pipeline. Our evaluation pipeline measures no false positives on 1024 ground truth videos per discrete dataset and a near-zero error floor on 1024 ground truth videos per continuous dataset. We also introduce our Evaluation Audit Interface, which enables systematic inspection of false positives and informs our regression test suite. Throughout the paper, we visualize detected errors with red overlays, allowing readers to directly inspect and verify our metrics. Logical inconsistencies are exceptionally difficult to detect through visual inspection alone, motivating our use of programmatic evaluation. While we observe no false negatives in our video gallery, we acknowledge that we cannot reliably estimate the false-negative rate at scale.

Q2) We now explicitly state in the text that all metrics are computed over 1,024 videos, while plots use a sample size of 256. In Chess, our favorite violation, depicted in Figure 9, occurs when models move a piece pinned to the king, an illegal move that amateur human players also frequently make. This points to a need to bring even more serial computation to bear to reliably follow complex rules during video generation. In Tetris, we observe that serial methods occasionally clear lines despite a remaining empty cell that should prevent the line from clearing.

Important changes to the paper are marked in purple. If you learned something about the importance of serial computation for video generation, please consider raising your score. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.