- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Serial-to-Parallel Diffusion (S2PD), which performs autoregressive block-causal diffusion: it establishes video structure at high noise levels and switches to parallel diffusion at low noise levels to refine details. Under the experimental conditions set by the authors, this method achieves better physical and logical consistency than pure block-causal diffusion.
Critical strengths
The paper's final experimental results show that the authors' S2PD-based scheme indeed achieves better performance on multiple evaluation metrics than the pure block-causal method.
Critical weaknesses
-
Simply put, the method adds parallel refinement during sampling on top of the block-causal scheme, and the Method section is too brief overall. The authors consistently emphasize that the serial approach achieves better physical consistency, but what they actually do is add parallel refinement during sampling to the serial method, which makes the source of the performance gains in Table 2 unclear. In addition, in describing video causality, the authors do not distinguish causality among tokens within a single frame from causality among tokens across different frames, which undermines the academic rigor of the paper.
-
The experimental setup is unreasonable in places, and the experimental results contradict some of the claims. First, compared with the block-causal method, S2PD additionally changes the training to "train with multiple block sizes." This substantively introduces two variables (parallel diffusion and multiple block sizes) into the experiment at the same time, and it is unclear which operation actually improves the consistency metrics. Second, why are different datasets used to train the two architectures, DiT-B and Wan-5B-LoRA? Normally, network architecture and dataset should be independent; different architectures should be trained on the same dataset for the comparison to be more informative.
Questions for authors
-
In fact, the effects of causality among tokens within a single video frame and among tokens across different frames on consistency should be considered separately, yet you seem to have completely ignored this issue? In the Table 6 experiments, are orderings such as raster and zigzag defined with respect to a single group of image tokens?
-
Where is the experiment supporting this claim: "we demonstrate that scaling training data does not enable bidirectional diffusion models to model complex video"? All the experiments seems to be conducted under the same total training budget.
-
The main contribution of this paper is adding parallel refinement during sampling after the block-causal scheme, but the overall motivation of the paper argues that the serial scheme has better physical consistency. Does this deviate from your actual contribution?
3 comments
Thank you for the detailed response and additional experiments. We acknowledge that the statement in our Summary that S2PD achieves better physical and logical consistency than pure BC was inaccurate. We correct it using the paper’s own wording on page 2:
We introduce S2PD, which preserves the physical plausibility and logical consistency of serial generation while improving sampling efficiency and temporal stability.
Three concerns remain:
-
Within-frame versus cross-frame relationships (Q1). Our question concerns whether causal relationships or dependencies among tokens within a frame should be distinguished from those across frames, and whether they affect consistency differently. The explanation of global token ordering and partitioning does not directly address this point. Please clarify the role of within-frame serial computation and how the existing experiments distinguish its contribution from that of cross-frame serial computation.
-
Pin-64 training configuration. The revised paper specifies that Pin-64 is trained with block sizes
{64, N_v}, but it remains unclear whether the BD baseline uses the same configuration. If BD is trained only with block size 64, both training and sampling differ, so the comparison does not fully isolate the contribution of parallel refinement. Please specify the block sizes used to train the BD baseline. -
Training-data scaling. Model scaling and training-data scaling are different questions. Our original Q2 concerned the latter, which the added model-scaling experiments do not directly address. Please point to the corresponding data-scaling experiments or qualify the conclusion to reflect the tested data and training budgets.
We thank the reviewer for their quick response and their additional questions.
C1) Our experiments (Table 5) show that within-frame seriality is primarily important for the discrete datasets, where a block size of 8 that does have within-frame seriality outperforms block size of 64 that does not have any within-frame seriality. On the other datasets, within-frame seriality does not appear to be important. Thus, we believe the optimal choice of how much serial computation to use depends on the dataset.
C2) BD is trained and sampled with a block size of 64, corresponding to one temporal chunk of four frames. S2PD with Pin-64 is trained with block sizes {64, Nᵥ} and uses a block size of 64 during its serial phase. We consider this a fair comparison of our sampling strategy.
To further isolate the contribution of our parallel refinement stage, we add a new ablation in Appendix C.1, Parallel Refinement Ablation, which compares sampling schemes using the same checkpoint. The results further support our arguments. Specifically, we compare serial-only and serial-to-parallel sampling with the same Pin-64 checkpoints. The serial-only sampling is the same sampling scheme used by cDF and BD.
On discrete and continuous datasets, serial-to-parallel sampling improves sampling speed with minimal impact on consistency. On video datasets, Pin-64 (serial-only) exhibits the same temporal flickering illustrated in Figure 7. With the checkpoint held fixed, serial-to-parallel sampling improves CD-FVD while achieving a 2× speedup. We show qualitative results in Figure 10 and report the quantitative results in Table 11.
C3) Figure 8 in Appendix A.1 directly addresses data scaling through training for longer on unique data. For our procedurally generated datasets, each epoch uses a different random seed and thus adds unique data to the training, scaling the dataset size. In Figure 8, extending training from epoch 10 to 20 doubles the training dataset size. We now explicitly clarify it in Section 4.1 and changed the title Appendix A.1 to Model and Data Scaling. The results clearly indicate that training-data scaling does not address the limitations of the bidirectional models. Serial methods, even at much lower training budgets, outperform bidirectional models.
Please let us know if this response satisfactorily addresses your concerns. Thank you!
We thank the reviewer for their feedback and first correct a fundamental misunderstanding in their Paper Summary, before addressing each concern in detail.
Our method does not “achieve better physical and logical consistency than pure block-causal diffusion”. We demonstrate that serial methods dramatically outperform bidirectional diffusion under matched training compute budgets. Then, we show our method retains the same physical and logical consistency as other serial methods while being 2x as fast.
W1 Part 1, Q3) We first establish that serial computation is essential for logically consistent video generation, then address its high runtime cost by replacing most serial steps with a parallel refinement stage. We demonstrate that this approach substantially accelerates sampling while preserving the benefits of serial computation, all within a single block diffusion model. We modify the abstract and switch the order of our first two contribution bullet points to make the flow more clear. We expand our Method section by adding Section 3.2 Baselines, dedicated to explaining our bidirectional and serial baselines. In Section 3.3 Training, we explicitly emphasize that Flex-Forcing introduced mixed-block-size training, which we adopt for all our S2PD configurations. We clarify that our contributions are establishing the importance of serial computation for coherent generation and the serial-to-parallel sampling scheme, not the mixed-block-size training procedure.
W1 Part 2, Q1) We are not entirely certain we understand the question. To avoid confusion, we almost completely removed the word "causal" from the paper, renamed the block-causal diffusion method (BC) to its paper name, Block Diffusion (BD). We focus our ablations on hyperparameters that affect the model's ability to capture dependencies in the data like token order and block size. Token order specifies the global ordering of tokens within the 3D spatiotemporal volume and is determined before partitioning the sequence into blocks. We hope this clarifies our approach and welcome further questions if we have not fully addressed the concern.
W2 Part 1) We acknowledge the confusion and train dedicated models with the block size mixture {64, N_v}, denoted Pin-64, for a controlled comparison. Table 2 now reports S2PD results using both Pin-64 and Pin-8 models. Pin-64 isolates the contribution of parallel diffusion, while Pin-8 captures the effect of reducing block size. We find Pin-8 is preferable for discrete datasets with complex rules, where the additional serial computation improves logical consistency. In contrast, Pin-64 is preferable for video datasets with high-frequency textures, where the need for serial computation is lower.
W2 Part 2, Q2) We add experiments comparing both architectures on the same dataset in Appendix A.1, Model Scaling. We find that bidirectional diffusion models improve only marginally with increased model scale and continue to substantially underperform serial methods. In contrast, Table 8 demonstrates that S2PD effectively leverages increased model capacity, reducing its CD-FVD score on Rubik’s Cube Real from 335.2 to 64.4.
Important changes to the paper are marked in purple. We hope all your concerns have been addressed. If your understanding and the clarity of the paper as well as the soundness of the experiments has improved, please consider raising your score. Thank you!