acceptodds
Under review as a conference paper at ICLR 2027

The Seriality Gap in Video Diffusion Models

13 views

Abstract

When one ball strikes another, then another, video models should predict the consequences of each bounce. In controlled experiments on multi-ball hard-sphere dynamics, we find that the performance of standard bidirectional video diffusion degrades as the causal chain lengthens, even when provided more denoising steps. In a length-matched control, where ball–ball interactions are absent, the degradation largely disappears, isolating dependent-event structure rather than video length as the cause. Across intervention studies, **methods that increase effective serial computation improve performance disproportionately**, including autoregressive/blockwise generation and architectural depth. This finding extends to both video prediction and simulator-state prediction, and also applies to both physical and algorithmic tasks. We identify this pattern as the **seriality gap**: a mismatch between tasks requiring growing serial computation and video diffusion models whose denoising loop does not provide scalable serial compute. We then prove that, for fully observed deterministic video prediction with an exact score, denoising steps do not add serial computation beyond the backbone, indicating a structural obstacle for video diffusion on serial reasoning and simulation tasks.

Primary area:
generative models
Keywords:
diffusion models, video prediction, world models, computational complexity, serial computation, inference-time scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Uk4xOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper diagnoses a seriality gap in bidirectional diffusion using hard-sphere and Turmite prediction in video and simulator-state representations. Longer event dependencies increase prediction difficulty, additional denoising steps do not close the measured gap, and task-aligned temporal factorization and greater backbone depth improve accuracy. For fully observed deterministic prediction, Proposition 4.1 shows that a TC0 exact-score network permits state prediction through one network evaluation, under the stated finite-precision and recovery assumptions.

Critical strengths
  1. The empirical gap survives substantial controls. At 49 frames and width 1536, hard-sphere rollout-5 error falls from 0.883 for bidirectional generation to 0.607 for AR, with a paired gap interval of [0.262, 0.290] (Tables 3–4). The five-ball separate-lane control nearly eliminates this local-error gap (p.36). Direct-state, extractor-sensitivity, and three-seed experiments further support robustness (Appendices B.4.2, C.1, F.3).

  2. Dependency and shared-weight tests strengthen the finding. At fixed event count, two-agent Turmites show decreasing bidirectional accuracy with dependency depth at all four tested counts; all eight within-seed bootstrap intervals are below zero (Table 17). A shared checkpoint trained on all four factorizations preserves AR > Block-5 > Block-15 > bidirectional generation (Appendix F.1).

Critical weaknesses
  1. The Turmite state comparison confounds temporal factorization with projection frequency. It supports Observation 3 on serial generation (§3.1, ll.310–317), but AR and blockwise generation apply argmax and feed back one-hot states after every block; bidirectional generation projects only after denoising the full future (Appendix C.2, ll.1127–1131). Smaller blocks also mean more frequent projection. The shared-checkpoint control retains this difference, leaving its contribution to the AR benefit unresolved (Appendix F.1).
Questions for authors
  1. How much of the Turmite AR/blockwise advantage comes from repeated discrete projection? Compare hard and soft prefix feedback under matched training to separate projection from temporal factorization.
  2. How does the exact-score result relate to transformer CoT expressivity results? Which computational limitation is shared with CoT, and what is specific to video diffusion? What does the theorem imply for intermediate visual/latent computation before the first output frame?
  3. Can a fixed visual model solve longer, unseen dependency chains by arbitrarily increasing internal steps, without retraining or lengthening the output video? Specify candidate intermediate visual states, per-step updates, and supervision. Can a purely visual CoT be learned from video alone, or does it require explicit symbolic states or supervision?
3 comments
AuthorsOpenReview ↗

Uk4x rebuttal (1/2)

We thank the reviewer for highlighting the strength of our experimental controls. We also welcome their questions connecting our findings to CoT expressivity and intermediate visual computation.

W1 / Q1

The Turmite state comparison confounds temporal factorization with projection frequency. [...] Smaller blocks also mean more frequent projection. The shared-checkpoint control retains this difference, leaving its contribution to the AR benefit unresolved (Appendix F.1).

How much of the Turmite AR/blockwise advantage comes from repeated discrete projection? Compare hard and soft prefix feedback under matched training to separate projection from temporal factorization.

We thank the reviewer for pointing out this subtlety. Below we show that repeated discrete projection is unnecessary for the observed serialization advantage.

To isolate the contribution of discrete projection, we re-evaluate the three shared checkpoints from Appendix F.1 with continuous predictions fed back between blocks. We hold each checkpoint and all other evaluation settings fixed, changing only the prefix feedback.

Predicted turnsARBlock-5Block-15Bidir
191.0000 / 1.00000.9965 / 0.99650.9244 / 0.92520.8621 / 0.8621
291.0000 / 1.00000.9973 / 0.99730.7949 / 0.79570.6286 / 0.6286
391.0000 / 1.00000.9989 / 0.99890.7752 / 0.77440.5149 / 0.5149

Final-grid cell accuracy ↑; hard one-hot / continuous prefix feedback. Hard-feedback evaluations reproduce all 12 previously reported scores exactly.

Removing discrete projection preserves AR > Block-5 > Block-15 > Bidir at every horizon. AR and Block-5 final-grid accuracies are unchanged; Block-15 changes by at most 0.0008, with paired 95% bootstrap confidence intervals over held-out examples including zero at all three horizons.

AuthorsOpenReview ↗

Uk4x rebuttal (2/2)

Q2

How does the exact-score result relate to transformer CoT expressivity results? Which computational limitation is shared with CoT, and what is specific to video diffusion?

Our paper does not study explicit scratchpad CoT before the first output frame, so we interpret “CoT” here as predicting intermediate frames autoregressively, as in our autoregressive models.

The connection is that our exact-score result shows that serial computation must come from the backbone rather than additional diffusion steps, under our assumptions of determinism, full observability, and an exact score.

Intuitively, autoregressive diffusion can distribute computation across successive frames for state prediction problems. This is analogous to how transformer CoT can go beyond TC⁰ through sufficiently long intermediate generation under the assumptions of Merrill and Sabharwal (2024). Thus, autoregressive generation can help bridge the Seriality Gap observed with bidirectional diffusion, even though each fixed-depth backbone evaluation remains in TC⁰ under the log-precision assumptions.

What does the theorem imply for intermediate visual/latent computation before the first output frame?

Under our assumptions of determinism, full observability, and an exact score, a single score-network evaluation already contains enough information to recover the complete future. Thus, the prediction burden rests on the backbone, not denoising steps.

Autoregressive frame generation provides serial computation across successive frames, but later frames cannot supply computation needed for earlier frames. Therefore, if producing the first frame already requires substantial computation, the model needs sufficient backbone capacity or additional computation before that frame, such as recurrence, looping, or scratchpad steps. This is why latent-frame autoregression is not a universal remedy (Lines 123–124).

Q3

Can a fixed visual model solve longer, unseen dependency chains by arbitrarily increasing internal steps, without retraining or lengthening the output video? Specify candidate intermediate visual states, per-step updates, and supervision. Can a purely visual CoT be learned from video alone, or does it require explicit symbolic states or supervision?

This question goes beyond the scope of our current paper: we study the Seriality Gap and interventions that reduce it, rather than develop a model that resolves arbitrarily long, unseen dependency chains. Nevertheless, we think this is an interesting direction.

For the first part, a natural candidate is a looped backbone that repeatedly updates a hidden latent state using shared weights. Additional loops would provide more computation without requiring more output frames. Looped language models provide evidence that this approach can support reasoning (Saunshi et al., 2025), and looping has also been explored for image and video generation (Goyal et al., 2026). However, these results do not establish that arbitrarily increasing internal steps will resolve longer, unseen dependency chains, especially in videos.

For the second part, we interpret a purely visual CoT as an internal visual scratchpad generated before predicting each output frame, without adding frames to the output video. The training videos would not directly supervise these scratchpad states; instead, the model could learn them end-to-end through the video-prediction objective. We are optimistic that useful intermediate computation can be learned this way. However, whether the learned scratchpad remains visually meaningful, rather than becoming an opaque latent representation, is an interesting empirical question beyond our current study.

The authors' reply address my major concerns. Thus, I maintain the positive score.

Reviewer_WLZqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper identifies the "Seriality Gap" in video diffusion models, revealing that bidirectional generation fundamentally degrades on long chains of dependent events and that increasing denoising steps fails to resolve this bottleneck. The authors theoretically prove that denoising loops do not add scalable serial computation beyond the backbone, and demonstrate that interventions increasing effective serial compute improve performance. These claims are rigorously validated through meticulously controlled testbeds that isolate serial dependency depth from visual complexity, alongside a novel local physical consistency metric that mitigates accumulated drift.

Critical strengths

1.Sharp and important question:Whether video diffusion models actually gain “reasoning” from denoising iterations is a central controversy. The controlled physics simulation cleanly separates visual complexity from dependency depth, and the experimental design is conceptually clean.

2.Clear mathematical structure in the theoretical result:Proposition 4.1 uses the single-step recovery property of the exact score to reduce the denoising loop to one score evaluation, arguing that a TC backbone cannot solve P-hard billiard-ball prediction via more denoising steps. The argument is internally coherent.

  1. Multi-task, multi-representation evidence:The same trend appears across hard-sphere video, direct simulator-state, Turmite video, and Turmite simulator-state; the gap disappears in single-ball and separate-lane controls, strengthening the claim that serial dependency is the driver.
Critical weaknesses
  1. “Visual simplicity” of the hard-sphere testbed is not “task simplicity.” The paper repeatedly emphasizes low visual complexity and deterministic dynamics, so failure must be due to serial dependency. But ball identity (color) and precise collision geometry remain a difficult state-estimation problem in pixel space. The color-identity violations and tunneling reported in Figure 4 also occur in AR models, despite less frequently. If part of the bidirectional failure comes from pixel-level state extraction rather than insufficient serial computation in denoising, the attribution to “seriality gap” is incomplete. The direct simulator-state experiment (§3.1) is offered as a response, but its model architecture (d_model=512, 8 layers) is entirely different from the video DiT, so it cannot directly rule out pixel-level factors.

  2. Tension between the formal statement in the theory section and the operational claim in the experiments. Proposition 4.1 in §4 is a statement about the exact score, but the abstract and §1 present it as the general conclusion that “denoising steps do not add scalable serial computation.” For approximate scores, finite precision, and training-distribution shift, this general conclusion is not proven. The limitations section acknowledges this, but the headline claim is not weakened accordingly.

Questions for authors
  1. Regarding Natural Video: Have you considered running a lightweight experiment on a slightly more complex, yet still controlled, physics benchmark (e.g., soft-body dynamics, fluid simulation, or cloth simulation) to see if the seriality gap persists when visual complexity and non-rigid deformations are introduced?
2 comments
AuthorsOpenReview ↗

WLZq rebuttal (1/2)

We appreciate the reviewer’s recognition of the question’s importance and the theory’s clarity, and their thoughtful comments about how to interpret and extend our findings.

W1

The paper repeatedly emphasizes low visual complexity and deterministic dynamics, so failure must be due to serial dependency. But ball identity (color) and precise collision geometry remain a difficult state-estimation problem in pixel space. [...] If part of the bidirectional failure comes from pixel-level state extraction rather than insufficient serial computation in denoising, the attribution to “seriality gap” is incomplete. The direct simulator-state experiment (§3.1) is offered as a response, but its model architecture (d_model=512, 8 layers) is entirely different from the video DiT, so it cannot directly rule out pixel-level factors.

The direct simulator-state models retain the same DiT architecture family, with two main architectural differences:

  1. The input consists of (x,y,vx,vy)(x,y,v_x,v_y) for each ball, with appropriate embeddings, rather than VAE latents.
  2. A smaller DiT backbone is used for the compact representation: 512 dimensions and 8 layers, compared with 768 dimensions and 30 layers for the video models. These backbones predict short horizons accurately even with bidirectional generation, so they are not simply incapable of solving the task.

These differences do not change the model family or the task’s sequential dependencies; both models are trained with denoising objectives. We therefore disagree with the characterization that the state model’s architecture is “entirely different” from the video DiT.

Moreover, the video Turmite experiments, which have much lower state-estimation ambiguity due to the discrete nature of the underlying states, provide strong evidence against state estimation as an explanation for the gap. Pixel-level state estimation cannot explain the gap across both video and direct simulator-state prediction, whereas the Seriality Gap hypothesis accounts for the shared pattern.

We nevertheless recognize that state-estimation errors can contribute to the video results and quantify state recoverability in Appendix B.2.

Q1

Have you considered running a lightweight experiment on a slightly more complex, yet still controlled, physics benchmark (e.g., soft-body dynamics, fluid simulation, or cloth simulation) to see if the seriality gap persists when visual complexity and non-rigid deformations are introduced?

We did consider similar extensions. The benchmarks the reviewer suggests are promising because they can combine long dependency chains with higher visual complexity and non-rigid dynamics while retaining simulator ground truth.

The central challenge in characterizing the Seriality Gap, as we do in our work, is to vary dependency depth while holding potential confounds such as video length and visual complexity fixed, and to define transition metrics that distinguish dynamics errors from visual state-estimation errors. We prioritized testbeds that make these controls precise, allowing us to test competing explanations and isolate the role of serial computation.

Our prediction is that the Seriality Gap will persist when visual complexity and non-rigid deformations are introduced, because accurate prediction will still require propagating long chains of dependent events.

Independent evidence already supports this prediction in these settings. S2PD (Hu et al., 2026) (please be cautious: reading the paper might reveal our author identities) explicitly evaluates deformable materials and fluids in MPMWorlds, alongside richer rigid-body simulations and real Rubik’s Cube videos while pointing to the same serial computation bottleneck that we characterize.

We report their published results for Bidir generation versus AR variants, including causal Diffusion Forcing (cDF), Block-Causal diffusion (BC), and Serial-to-Parallel Diffusion (S2PD), below:

DatasetBidirectional (Parallel)cDF (Serial)BC (Serial)S2PD (Serial)
MPMWorlds (deformable materials and fluids)88.743.243.942.5
Kubric MOVi-A (rigid bodies)252.8168.5141.2137.0
Kubric MOVi-C (realistic objects and backgrounds)512.4376.5240.3231.7
Rubik’s Cube Real (natural video)698.770.870.964.4

Table 1. CD-FVD ↓ (distributional video quality); Wan-5B-LoRA; serial block size 64. S2PD combines serial generation with parallel refinement. Source: Hu et al. (2026), Table 2.

Serial generation retains its advantage with non-rigid dynamics and richer appearance. These gains are consistent with our prediction, while our testbeds isolate the dependence on dependency depth through the controls the reviewer highlights.

We outline concrete experimental designs for these extensions in our response to Reviewer 1 (W1 / Q2).

AuthorsOpenReview ↗

WLZq rebuttal (2/2)

W2

Proposition 4.1 in §4 is a statement about the exact score, but the abstract and §1 present it as the general conclusion that “denoising steps do not add scalable serial computation.” For approximate scores, finite precision, and training-distribution shift, this general conclusion is not proven. The limitations section acknowledges this, but the headline claim is not weakened accordingly.

We understand the reviewer's concern and will revise the abstract and introduction to state the scope of the theoretical claim explicitly.

Abstract (Lines 39-40). Change

whose denoising loop does not provide scalable serial compute

to

whose denoising loop does not appear to provide scalable serial compute

Introduction (Line 116). Add

This result provides theoretical intuition for our observations, but does not establish that the same mechanism explains the behavior of learned models with approximate scores.

Figure 2 (Lines 95-96). Change

[...] but denoising does not supply scalable serial compute (Section 4).

to

[...] but denoising does not supply scalable serial compute under the exact-score assumptions of Section 4.

Reviewer_xk15OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a seriality gap in video diffusion models. In controlled multi-ball dynamics, bidirectional diffusion performs worse as the chain of dependent collisions grows, even with more denoising steps. A length-matched, non-interacting control suggests the decline is due to event dependencies, not video length. Autoregressive or blockwise generation and deeper architectures improve performance. The paper also proves that, under fully observed deterministic prediction with an exact score, denoising steps do not add serial computation beyond the backbone.

Critical strengths
  1. The paper tackles an interesting problem at the intersection of theory and practice, with implications for designing video prediction models.
  2. Its controlled experiments distinguish causal-chain complexity from video length and provide a plausible explanation for the observed performance gap.
Critical weaknesses
  1. The experiments focus on synthetic, deterministic hard-sphere dynamics. It remains unclear whether the findings extend to natural videos with richer appearance, uncertainty, and interactions.
  2. The interventions do not isolate serial computation from other changes: autoregressive or blockwise generation also changes the modeling structure, while increasing depth changes capacity and optimization.
  3. The theoretical result relies on strong assumptions, including full observability, deterministic dynamics, and an exact score. Its implications for practical models with approximate scores are unclear.
Questions for authors
  1. Autoregressive video generation often performs worse than bidirectional diffusion in practice. Why, then, does adding serial computation help in your experiments? What do you see as the main factors behind this difference?
  2. Do you expect the seriality gap to persist in natural-video prediction or more complex physical systems? What experiments could test whether the observed trend scales with longer causal chains and richer dynamics?
  3. Can you separate the effect of serial computation from changes in architecture, training, and capacity—for example, through more strictly compute- and parameter-matched comparisons? How robust is the theoretical conclusion when the score is approximate?
4 comments
AuthorsOpenReview ↗

xk15 rebuttal (1/2)

We appreciate the reviewer’s interest in our work at the intersection of theory and practice, and their thoughtful questions about the scope and implications of our findings.

W1 / Q2

It remains unclear whether the findings extend to natural videos with richer appearance, uncertainty, and interactions. [...] Do you expect the seriality gap to persist in natural-video prediction or more complex physical systems?

We do, because many natural videos and physical systems still do require predicting long chains of dependent events.

Richer appearance, partial observability, and dynamical uncertainty in natural videos and physical systems introduce additional challenges, but do not remove the need to predict chains of dependent events. Our findings suggest that increasing denoising steps alone does not supply the serial computation needed to resolve those dependencies (Lines 359–367).

To support the above argument, the advantages of autoregressive (AR) over bidirectional (Bidir) models have been independently verified on a different set of complex physical systems and real Rubik's Cube videos in S2PD (Hu et al., 2026), which points to the same serial computation bottleneck that we characterize (please be cautious: reading the paper might reveal our author identities).

We report their published results for Bidir generation versus AR variants, including causal Diffusion Forcing (cDF), Block-Causal diffusion (BC), and Serial-to-Parallel Diffusion (S2PD), below:

Dataset / MethodsBidirectional (Parallel)cDF (Serial)BC (Serial)S2PD (Serial)
Kubric MOVi-A (rigid bodies)252.8168.5141.2137.0
Kubric MOVi-C (realistic objects and backgrounds)512.4376.5240.3231.7
MPMWorlds (deformable materials and fluids)88.743.243.942.5
Rubik’s Cube Real (natural video)698.770.870.964.4

Table 1. CD-FVD ↓ (distributional video quality); Wan-5B-LoRA; serial block size 64. S2PD combines serial generation with parallel refinement. Source: Hu et al. (2026), Table 2.

Based on our theoretical argument, our results in both deterministic and stochastic systems (Appendix E), and the supporting evidence above, we can reasonably conclude that serial generation retains its advantage in these richer settings, including natural video, consistent with our prediction.

What experiments could test whether the observed trend scales with longer causal chains and richer dynamics?

Rigorously characterizing the Seriality Gap itself, as we do in our work, requires carefully controlled experiments. In particular, one would have to show that the gap grows with dependency depth while holding potential confounds such as video length and visual complexity fixed, and that additional denoising steps do not close it.

For instance, two possible extensions are:

  • MPMWorlds (Kovačič & Ellis, 2026): Construct longer chains of dependent interactions among deformable objects, alongside controls with the same number of interactions distributed across independent chains. Match video duration, object count, and rendering conditions, and compare states recovered from generated videos against simulator ground truth to measure physical prediction errors.
  • Rubik’s Cube Real (Hu et al., 2026): Use recordings annotated with cube states and turns to evaluate state consistency over longer move sequences. Condition on the prescribed turn sequence unless the evaluation accepts any legal continuation. For a shallow-dependency control, use turn-and-undo pairs—for instance, turn the right face and immediately undo it, then do the same with the top face. Each pair returns the cube to its starting configuration, preventing changes from accumulating across the full sequence. Match video duration, turn count, and recording conditions.

In each setting, we would compare bidirectional and blockwise generation, testing whether the gap grows with dependency depth and whether additional denoising steps close it.

AuthorsOpenReview ↗

xk15 rebuttal (2/2)

Q1

Autoregressive video generation often performs worse than bidirectional diffusion in practice. Why, then, does adding serial computation help in your experiments? What do you see as the main factors behind this difference?

The main trade-off is between the benefit of serial computation in AR generation and the cost of exposure bias (aka. train-test gap) introduced by teacher forcing.

The train-test gap arises because the model conditions on its own generated frames at test time but on ground-truth frames during training. This gap also widens due to error accumulation (aka. drift). It is a major obstacle to maintaining quality over long autoregressive rollouts (Self Forcing, Huang et al., 2025).

Why is it any different in our experiments? The difference is that bouncing balls have simple visuals and local dynamics, making each successive update relatively easy to model autoregressively and helping keep the train-test gap manageable. Therefore, in our experiments, the benefit of serial computation outweighs the cost of the train-test gap. Natural videos do not necessarily share the same trade-off. However, recent developments in bridging the train-test gap, such as Self Forcing (Huang et al., 2025), reduce this cost of AR generation.

W2 / Q3 (a)

The interventions do not isolate serial computation from other changes: autoregressive or blockwise generation also changes the modeling structure, while increasing depth changes capacity and optimization.

Can you separate the effect of serial computation from changes in architecture, training, and capacity—for example, through more strictly compute- and parameter-matched comparisons?

Yes, and that is precisely what our AR and blockwise comparisons are designed to test. Within tasks, models have:

  1. Comparable inference FLOPs with Bidir and a common training protocol. Our comparisons use the same training iterations, batch size, optimizer settings, and data within each horizon (Appendix B.2). With cached prefixes, AR uses fewer inference FLOPs than Bidir, so its advantage does not come from a larger inference-compute budget.
  2. Same backbone architecture and parameter count as Bidir. The only differences are the attention masks, context conditioning, and generation schedule—from one temporal latent frame at a time to all future frames jointly.

Moving from Bidir to AR is therefore a minimal structural change that adds serial computation by allowing later predictions to build on earlier results. Controlling away this change in modeling structure would remove the very intervention that tests our hypothesis.

We agree that, in Observation 5 (Line 269), changing depth or width also changes optimization behavior—for example, deeper models may be harder to optimize (Lines 274–276).

Therefore, to further control for optimization differences, we trained shared models with varying block sizes (Appendix F.1), holding parameters and optimization history fixed across inference modes. The serialization advantage persists under shared weights.

W3 / Q3 (b)

The theoretical result relies on strong assumptions, including full observability, deterministic dynamics, and an exact score. Its implications for practical models with approximate scores are unclear.

How robust is the theoretical conclusion when the score is approximate?

Our experiments show that the predictions motivated by our theory remain robust beyond its idealized assumptions:

  • Full observability: Hard-sphere bouncing ball experiments have imperfect state recovery from rendered videos.
  • Deterministic dynamics: The vibrating-wall ablation (Appendix E) shows that serialization retains its advantage on deterministic segments of stochastic trajectories.
  • Exact score: Our experiments use learned score approximations throughout.

Importantly, the theory motivates concrete predictions: a growing gap between serial and parallel models as serial complexity increases, and the inability of additional denoising steps to close it. These observations are made across our tasks and experimental settings (Observations 1-5). Related findings provide further support: PhaseLock (Han et al. 2026) reports limited benefits from additional denoising steps for physical consistency, while S2PD (Hu et al., 2026) demonstrates advantages of serialization, including on natural video.

Reviewer_xk15OpenReview ↗

Thanks for your rebuttal

Thanks for your reply, the trade-off makes sense. Diffusion show how like multiple token predictions, they predict multi-token but in one time (in autoregressive order). From this perspective, do u think the DLLM follow this serial gaps? i.e., multiple times autoregressive rollout beats one time diffusion prediction.

AuthorsOpenReview ↗

xk15 follow-up

We thank the reviewer for the continued discussion.

We interpret the reviewer’s question as asking whether Diffusion Large Language Models (DLLMs) might also exhibit the Seriality Gap. We hypothesize that they can on tasks with long chains of dependent computations, such as multi-step reasoning and coding, although our work does not establish this for language.

Autoregressively generated intermediate steps can provide a computational scratchpad in which later predictions condition on results computed earlier. This connects to our discussion of chain-of-thought (CoT) in our response to Reviewer Uk4x (Q2).

Existing language-model results provide suggestive parallels.

Block Diffusion (Arriola et al., ICLR 2025) combines autoregressive generation across blocks with diffusion within each block. Its OpenWebText and LM1B results show improved perplexity bounds as block size decreases, a trend similar to our Observation 3 (Lines 224–232):

ModelTokens per blockOpenWebText perplexity ↓LM1B perplexity ↓
Autoregressive117.5422.83
SEDD (Diffusion)—≤ 24.10≤ 32.68
MDLM (Diffusion)—≤ 22.98≤ 31.78
Block Diffusion16≤ 22.27≤ 30.60
Block Diffusion8≤ 21.68≤ 29.83
Block Diffusion4≤ 20.73≤ 28.23

Table 2. Training budgets were 524B tokens for OpenWebText and 65B tokens for LM1B; diffusion perplexities are upper bounds. Source: Block Diffusion, Tables 3–4.

These results are consistent with benefits from serialization, although they do not isolate serial computation.

We also find the distillation results in The Diffusion Duality (Sahoo et al., 2025) and Self-Distillation Through Time (Deschenaux & Gulcehre, 2025) interesting in relation to Observation 4 (Lines 260-267). In particular, with 16 sampling steps, distilled Duo achieves slightly better generative perplexity and similar entropy compared with the original model at 1,024 steps (75.24 versus 77.69 perplexity; 5.53 versus 5.55 entropy; Appendix D.5, Table 6). These results motivate studying the computational role of denoising iterations, particularly whether they provide serial computation that scales with task dependency depth.

An important qualification is that DLLMs use diverse corruption processes, token-update rules, and sampling schedules, making these design choices central to the question. For instance, LLaDA (Nie et al., 2025) supports both semi-autoregressive decoding and confidence-based remasking, allowing later predictions to build on previously accepted tokens. The key question is whether the update order aligns with task dependencies and supplies the required serial computation.

Our theorem applies to fully observed deterministic prediction with an exact score under Gaussian corruption. It motivates investigating whether the limited serial computation within each fixed-depth backbone evaluation creates a similar bottleneck in DLLMs. However, extending the theorem to discrete corruption processes and language modeling requires further analysis.

A direct test would vary dependency depth while controlling sequence length and other potential confounds, then compare autoregressive, joint, and blockwise generation across denoising-step budgets, including comparisons at matched total inference compute. We think this is a promising direction for future research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.