acceptodds
Under review as a conference paper at ICLR 2027

Sample Midway, Condition Deeper: One-Pass Autoregressive Video Generation

Abstract

Discrete-token video generators sample a clip as a grid of thousands of codes, and raster decoding runs one full transformer pass per code (174 s per UCF101 clip on an A100). Parallel decoders sample a group of codes per pass, but the codes in a group are drawn independently, so a choice that the condition leaves open, such as the phase of a pedaling motion, can be drawn differently at different positions. Before a code is drawn, every hidden state is a deterministic function of the condition, so only the drawn code itself can tell another position which option was taken, and this code can re-enter the network at the layer where it was drawn. We propose IntraAR, which samples successive token groups at successively deeper layers of one forward pass. After a group is sampled at layer , the embedding of each drawn code is added to its hidden state, the row is frozen as key–value memory, and the remaining layers predict the next groups conditioned on it. One pass thus performs sampling steps with an exact likelihood. We show that sampling a group earlier pays off when the information it passes to later groups exceeds the fitting error caused by its shallower prediction. On five video benchmarks, IntraAR lowers batch-1 latency by 24.5–41.5× relative to matched raster decoding while I3D-FVD changes by −2.26 to +1.83; on three ImageNet backbones, latency falls by 11.2–29.8× at FID increases of 0.14–0.28. Removing only the embedding of the drawn code, with identical groups and computation, raises UCF101 I3D-FVD from 195.6 to 269.6.

Primary area:
generative models
Keywords:
autoregressive video generation, transformer inference, parallel decoding, video generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_G118OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes intraAR, an image / video generating principle where sample deeper at network instead of multiple forward process like normal AR doing. The experiment shows a good potential of the current method, comparing with the 5 classical video generator baseline under 3 imagenet backbones, intraAR greatly reduces latency by 24.5 - 41.5 on video and 11.2 - 29.8 on images, with relative the same FVD and FID.

Critical strengths

Strengths:

  1. The most interesting part of this paper is the idea itself, elegant and well-motivated, with addressing the challenges currently like lossing coherence and hidden states problem, the intraAR solves this by simply sampling deepth.

  2. Well designed and comprehensive experiments results support the main claim strongly.

Critical weaknesses

Weakness:

  1. Like the thoery proved here, the current method i think is a good fit for the unguided generation. I may also want to see under the guided direction how far this method can push forward, like different CFG number or A guided raster vs. IntraAR pair.

  2. I think one of the limitation for this paper is that the number of sampling step is largely constrained by the depth. So what might happen for the longer or higher resolution videos. It would be btter to show a scaling experiment here.

I am happy to rasie my score later if these results are interesting

Questions for authors

Besides the weakness above, here is one questions i am thinking of:

How do quality and latency change for longer or higher-resolution clips?

3 comments
AuthorsOpenReview ↗

Response to Reviewer G118 (1/2): guided generation

(1/2) We are very grateful for your review. You put your finger on exactly the idea we care about: before a code is drawn every hidden state is a deterministic function of the condition, so coherence has to come from the drawn code itself, and the remaining depth of the same pass can condition on it. It means a lot to us that you found this elegant and well motivated, and that you read the matched experiments as strong support for the main claim. Your two requests, guidance and scale, were the two things missing from the evaluation, and both are now in the paper. This reply covers guidance; the next one covers longer and larger clips.

W1. How far does IntraAR go under guidance?

It keeps its matched behaviour at every scale we tested (four ImageNet and two UCF101 scales, including NAR's 1.25). Under identical classifier-free guidance, IntraAR stays within 0.12 to 0.23 FID of Raster on ImageNet and within 1.52 to 2.52 I3D-FVD on UCF101, and the latency reduction remains 14.4-14.5× and 40.2-40.3×.

How guidance works in one pass. Both branches (conditional and null-class) run as one batch of two through the same pass. At each commitment the two logits are combined as u + w(c − u), one code is drawn per position, and each branch writes that same code into its own memory row. This mirrors raster guidance, which feeds one sampled token to both branches. Training is unchanged, and Raster uses its native guided sampler with the same null-class pathway.

Results (three-seed means; ImageNet/LlamaGen-XL FID, UCF101/OmniLM I3D-FVD; CFG 1 rows are the original Tables 1 and 3):

ImageNet CFG11.522.53
Raster19.406.213.864.836.62
IntraAR19.546.343.984.996.85
UCF101 CFG11.251.5
Raster193.76179.64181.29
IntraAR195.59181.16183.81

Batch-1 latency with both branches: 3.01 s versus 0.21 s on ImageNet (CFG 2), 184.28 s versus 4.58 s on UCF101 (CFG 1.25).

What we read from this:

  • The two decoders trace the same fidelity-coverage curve. Both reach their best FID at w = 2 (Raster 3.86, IntraAR 3.98), and precision rises while recall falls with w in the same way (Table R5).
  • The gap is smallest near the best scale and widest at w = 3 (+0.23 FID, +2.52 I3D-FVD at 1.5). Strong guidance sharpens the distributions of the earliest groups, which IntraAR predicts with fewer layers and which every later group inherits. This is the same inheritance that makes the no-prefix control lose more sample quality than its NLL suggests (Sec. 5.3).
  • VideoMAE-FVD, which is more sensitive to motion, stays within one point of Raster at both UCF101 scales.
  • The second branch is almost free for IntraAR on ImageNet, where a single 256-row pass is launch-bound at batch 1, so the speedup even rises slightly (14.4× at w = 2 versus 14.0× unguided). On UCF101's 5,120 rows it costs a bit more, and the speedup goes from 41.5× to 40.3×.
  • Against the external guided recipes of Table 3 (FID 3.81 to 7.15 at 0.39 to 5.33 s), guided IntraAR reaches FID 3.98 at 0.21 s. On UCF101 at NAR's own scale of 1.25, IntraAR now has lower I3D-FVD than NAR (181.16 versus 184.53) at 0.59× its latency, which resolves the one Table 1 setting where NAR led thanks to its guidance.

You are right that the theory concerns the unguided model: guidance samples a different distribution, so Theorems 1 and 2 do not cover it. The revision says this explicitly and presents the guided results as empirical.

Where in the paper: Table R1 and the "Guidance" paragraph of Sec. 5.2; full metrics (sFID, IS, precision, recall, per-seed speedup) in Table R5 and App. G.1; Sec. 6 states the scope.

AuthorsOpenReview ↗

Response to Reviewer G118 (2/2): longer and higher-resolution clips

(2/2) This part answers W2 and your question on longer and higher-resolution clips.

W2 / Q. How do quality and latency change for longer or larger clips, given that depth limits the number of steps?

The one-pass gap to Raster grows with the number of codes per group, exactly as the depth bound predicts, and a second pass removes most of it at about the same latency. IntraAR stays the fastest decoder in every setting.

Setup. We retrained Raster and IntraAR (L/P/S = 24/8/16) from the UCF101 weights on 65-frame clips at 256² and on 17-frame clips at 384², using OmniLM's native 4× temporal and 8× spatial compression. Since one pass has at most L − P = 16 steps (Theorem 2), the groups grow with the grid: 320 codes per group in Table 1, 720 at 384², and 1,088 at 65 frames. A two-pass variant commits the first half of the grid in 16 groups and then treats those codes as observed frames, exactly as K600 treats its prefix, for 32 steps in total.

Results (I3D-FVD / batch-1 latency in s, three-seed means):

Clip (codes)Codes/group, 1 passRasterIntraAR, 1 passIntraAR, 2 passes
17 f, 256² (5,120)320193.76 / 174.04195.59 / 4.19194.62 / 4.27
17 f, 384² (11,520)720198.47 / 421.58205.81 / 11.31200.71 / 10.83
65 f, 256² (17,408)1,088341.95 / 676.85359.62 / 19.71348.91 / 18.24
  • Quality. The one-pass gap is +1.83, +7.34, and +17.67 I3D-FVD. It follows codes per group, which is the dependence floor of Theorem 2 at work. Halving the groups with a second pass recovers 53 to 69% of the gap (61 to 69% on the two enlarged grids). VideoMAE-FVD, computed on 16 frames spread over the whole clip, shows the same pattern.
  • Latency. Two passes cost 0.93 to 1.02× the one-pass latency. The second pass reruns the stack once more, but it also removes the attention between the two halves, which grows with the square of the grid, so the two effects nearly cancel. At 65 frames, two passes take 18.24 s against 676.85 s for Raster.
  • No drift at the end of long clips. On the last 17 frames of the 65-frame clips, I3D-FVD is 228.73 (Raster), 232.58 (two passes), and 239.84 (one pass), so the late frames of an IntraAR clip are as close to Raster as the clip as a whole.

In practice, one pass suffices for grids of the original size, and long grids should use a second pass, which keeps the latency of a single pass (Sec. 5.5 and Sec. 6).

Where in the paper: new Sec. 5.5 with Table R2; all metrics and peak memory in Table R6 and App. G.2; Sec. 6 (Limitations) states the bound and the remedy.

For the AC discussion, in brief. (i) Every comparison is matched: Raster and IntraAR start from the same weights and see the same tokens, so the speedups are not recipe effects. (ii) The mechanism is isolated: removing only the drawn code at identical computation raises UCF101 I3D-FVD from 195.6 to 269.6, and a new depth-matched raster baseline (Table R3) shows that reading codes at intermediate layers costs little once the draws are in view, so the gain comes from the write. (iii) The new evidence (guidance, longer clips, more decoders) agrees with the original claims, and where one pass reaches its limit, the paper now says so and gives the fix.

Your comments directly shaped Sec. 5.5 and the guided evaluation, and we thank you for them. You mentioned you would be happy to raise your score if these results were interesting; if they address your two questions, we would be very grateful if you would consider doing so and supporting the paper in the discussion.

Thank you so much for the detail rebuttal.

Most of my concerns are solved, and I think I will main my current score.

Reviewer_toR4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IntraAR, which generates successive groups of visual tokens at increasing depths within a single Transformer forward pass. Sampled token embeddings are written back as frozen memory, allowing deeper layers to condition on earlier draws. The paper provides a likelihood-based analysis and reports substantial latency reductions over matched raster decoding while maintaining similar generation quality.

Critical strengths
  1. The paper clearly explains the tension between parallel sampling and preserving dependencies, and connects this motivation to the proposed architecture.
  2. Intermediate sampling, explicit write-back, and conditioning at deeper layers form a coherent mechanism. The analysis provides a useful account of the trade-off between earlier predictions and the information they supply to subsequent groups.
  3. Experiments span five video datasets and three image-generation backbones, with useful mechanism ablations and clearly specified latency and throughput measurement protocols.
Critical weaknesses
  1. Incremental conceptual novelty. DnD-Transformer already performs autoregressive prediction along model depth and feeds generated codes back to condition deeper layers. IntraAR extends this idea from residual codes at one position to groups of spatial-temporal positions. This distinction is meaningful, but the significance of the additional architectural contributions could be established more clearly.
  2. Limited video-specific comparisons and scope. Despite the video-centric framing, video baselines are limited to Raster, PAR, and NAR, while broader decoder comparisons appear only on ImageNet. Evaluation on short clips also leaves scalability and temporal consistency on longer videos unclear.
  3. The method relies on intermediate categorical sampling and token-embedding write-back, and is evaluated exclusively on discrete-token generators. Its applicability to chunk-autoregressive diffusion or flow-based video models remains unclear, since generating a chunk generally requires iterative refinement. This narrows the demonstrated scope of the video-generation contribution and warrants a clearer discussion of the architectural limitations and potential extensions.
Questions for authors

see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer toR4 (1/2): relation to DnD and the role of each component

(1/2) Thank you for a careful and fair review. We were glad that you found the tension between parallel sampling and preserved dependencies clearly explained, that you saw intermediate sampling, write-back, and deeper conditioning as one coherent mechanism, and that you found the analysis of earlier predictions versus the information they pass on useful. Your three points identified exactly where the submission undersold or under-specified its contribution, and each led to a change in the paper. This reply covers W1; the next covers W2 and W3.

W1. Novelty relative to DnD-Transformer, and the significance of the additional components.

You are right that DnD also predicts along model depth and conditions deeper layers on generated codes, and the submission should have made the difference sharper. The revision does so in Sec. 2 and in a new App. G.6. The difference lies in which dependence the depth axis carries.

  • DnD places the d residual codes of one spatial position at heads along depth. This needs a residual multi-code tokenizer, and positions remain autoregressive, one pass each (256 passes for a 16×16 image). With the single-code tokenizers of all eight of our model/task pairs, DnD has one code per position and reduces to Raster, which is our matched reference.
  • IntraAR places different positions along depth. That removes the passes across positions, which are the actual cost on video grids of 1,024 to 17,408 codes, and it is what lets a UCF101 clip go from 174 s to 4.19 s.

Moving positions onto the depth axis creates problems that a single position does not have, and each component answers one of them:

  1. Positions committed first have no input code yet. They start from mask rows at native coordinates and share P prefix layers that see the whole grid (requirement R1).
  2. A draw has to reach other positions, not the same residual stream. It is written as a key-value row that later positions attend to (R2), and the write must keep the draw recoverable (Proposition 2).
  3. That row must keep meaning the same draw while other rows change, so it is frozen.
  4. The depth budget must be split between steps and fitting, which the total-correlation floor (Eq. 2), the early-draw identity (Eq. 8), and the depth bound (Theorem 2) quantify. These concern dependence between positions and have no counterpart in DnD.

Each component, measured separately (UCF101/OmniLM I3D-FVD; IntraAR 195.59 at 4.19 s; Table 5):

VariantWhat it removesI3D-FVDLatency (s)
No sample writethe drawn code269.574.15
No prefixshared context before the first draw249.263.43
Code-only write (new)the context of the draw213.474.18
Unfrozen rows (new)a fixed record of the draw201.485.10
Dense queriesskipping frozen rows195.375.13

Two things follow. The code alone closes 76% of the no-write gap on UCF101 and 83% on LlamaGen-XL (Table 4), and the remainder, larger on video, is the context that the hidden state adds (which object part or moving region a position belongs to). Freezing serves both requirements: unfrozen rows cost what dense queries cost and still lose quality. The no-write control, at identical groups, layers, and computation, isolates the drawn code itself, and a new depth-matched raster baseline (Table R3) shows that reading codes at intermediate layers is not what produces the quality. So the extra parts are not decorations on a DnD-style idea; each one is needed for depth to carry dependence between positions, and each is measured.

Where in the paper: Sec. 2 (DnD sentence), Sec. 5.3 ("Context, code, and a fixed record"), new rows in Tables 4 and 5, Table R3, App. G.3, G.4, and G.6.

AuthorsOpenReview ↗

Response to Reviewer toR4 (2/2): video comparisons, longer clips, and scope

(2/2) This part covers W2 and W3.

W2a. Video baselines were limited to Raster, PAR, and NAR.

This was a fair criticism. Table R4 adds the two decoder families that were missing on video, each trained from the same weights with the same added tokens, on the two OmniLM tasks whose 5,120-code grids are the longest in Table 1: MaskGIT-12 (masked decoding with re-masking, the "corrected in further passes" family of Sec. 2) and LPD-64 (locality-aware parallel decoding, the strongest grouped decoder in our ImageNet comparison). Both use CFG 1.25 on UCF101, as NAR does.

MethodUCF101 I3D / MAE / sK600 I3D / MAE / s
Raster193.76 / 145.43 / 174.0435.76 / 54.97 / 111.16
NAR184.53 / 138.19 / 7.8037.16 / 56.74 / 6.62
MaskGIT-12 (new)201.84 / 154.37 / 38.7640.83 / 64.12 / 28.31
LPD-64 (new)189.42 / 141.05 / 7.3138.47 / 59.02 / 5.86
IntraAR195.59 / 144.52 / 4.1936.72 / 55.15 / 3.81

IntraAR keeps the lowest latency, at least 1.5× below the faster added decoder. LPD has lower I3D-FVD on UCF101, where it uses guidance, while IntraAR is better on K600; MaskGIT-12 pays 12 full passes without a key-value cache and is behind on both. Under matched guidance IntraAR also has lower UCF101 I3D-FVD than NAR (181.16 versus 184.53; Table R1). The five-task comparison of Table 1 already covers PAR and NAR on BAIR, Sky, and Taichi; we chose the two longest grids for the new decoders because that is where the cost of extra passes, and hence the comparison, matters most.

W2b. Scalability and temporal consistency on longer videos.

New Sec. 5.5 retrains Raster and IntraAR on 65-frame clips and on 384² clips (Tables R2, R6). The one-pass gap grows with codes per group (+1.83 I3D-FVD at 320 codes, +7.34 at 720, +17.67 at 1,088), as the depth bound predicts, and a second pass recovers 53 to 69% of it at 0.93 to 1.02× the one-pass latency. For temporal consistency we report two measures. VideoMAE-FVD uses 16 frames spread over the whole clip with features trained for motion recognition, and it shows the same pattern as I3D-FVD. On the last 17 frames of the 65-frame clips, I3D-FVD is 228.73 for Raster and 232.58 and 239.84 for two- and one-pass IntraAR, so IntraAR's late frames do not drift further from Raster than the clip as a whole.

W3. Applicability to chunk-autoregressive diffusion or flow video models.

You are right that the demonstrated scope is discrete-token generators, and the revision now says so explicitly (Limitations in Sec. 6, new App. G.6). IntraAR needs two things at a commitment: a value drawn from an intermediate state in one step, and an embedding of that value that deeper layers can read. Discrete tokens give both through the vocabulary head and the input embedding.

  • Continuous tokens with a light per-token sampler (MAR) provide the embedding directly and the draw by running that sampler at the commitment layer, at one sampler call per commitment. We name this as the closest extension and do not claim it.
  • Chunk-autoregressive diffusion (e.g., Diffusion Forcing, CausVid) obtains a chunk only after several denoising passes of the whole network, so no complete draw exists at an intermediate layer. There, coupling within a chunk comes from the extra passes, the second family in our Sec. 2. The write of Eq. (3) becomes applicable once a chunk is produced in a single network evaluation, as in one-step distilled generators.

We think stating this boundary plainly is more useful to readers than a broad claim, and Sec. 6 now does.

If the additional decoders, the scale study, and the clarified scope resolve your concerns, we would be grateful if you would consider revisiting your score.

Reviewer_na61OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To avoid the independent sampling problem of parallel decoding, this paper proposes to sample tokens from intermediate layers and feed the sampled tokens into successively deeper layers as KV cache. In this way, later token groups can explicitly condition on the realized samples of earlier groups, while avoiding repeated full forward passes as in conventional autoregressive decoding.

Critical strengths

1, It proposes IntraAR samples token groups with KV cache and perserves the exact likelihood.

2, Detailed analysis on earlier draw and sampling steps.

3, Increased performance with reduced latency compared with raster decoding.

Critical weaknesses

1, Earlier tokens must be committed at intermediate layers and do not make use of the full power of the whole network. It might be unreasonable for images/videos, as the earlier tokens are not necessarily less important than later tokens. Why should we decode earlier tokens ahead of time? This could harm the performance.

2, The number of sequential conditioning stages is inherently bounded by the available layers.

3, Compared with speculative decoding, it decodes earlier tokens ahead of time, without verification.

4, Do you have a baseline model that could decode tokens at different layers? We could still use raster decoding, but we decode different tokens from different layers. Such a baseline would help confirm the benefits of sample-and-write-back conditioning.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer na61 (1/2): early commitment and a depth-matched raster baseline

(1/2) Thank you for your review and for a summary that captures the method precisely. We appreciated that you singled out the exact likelihood, the analysis of earlier draws and sampling steps, and the latency reduction. Your questions are the right ones to ask of a method that samples inside the network, and the baseline you proposed in W4 is now in the paper and turned out to be a clear piece of evidence for the mechanism. This reply covers W1 and W4, the next one W2 and W3.

W1. Earlier groups are committed at intermediate layers; why decode them ahead of time?

You are right that a group committed at layer ℓ_s is predicted with only ℓ_s layers, and that early positions are no less important than late ones. Being drawn early does not make a group less important; it makes its draw available to every later group. Eq. (8) states the trade-off exactly: an earlier draw lowers the loss when the information it passes on exceeds the extra fitting error of its shallower prediction. We measure this balance in three ways.

  1. Delaying the first draw (Table 6). Predicting G1 one layer later, together with G2, helps G1 by 0.09 to 0.25 nats/token but hurts G2 by 0.29 to 0.62 and all later groups by 0.12 to 0.23, in all three settings.
  2. More steps per pass (Fig. 7). With S = 16 the first group is committed at P + 1, seven layers earlier than with S = 2, yet quality improves monotonically with S (UCF101 I3D-FVD from 336.02 to 195.59).
  3. Your proposed baseline (new Table R3): a raster decoder that reads every code at the layer where IntraAR commits it, with all earlier draws in view. Early readout costs it only 0.05 to 0.06 nats/token, about a tenth of what removing the write costs at the same layers (0.40 on LlamaGen-XL, 0.82 on UCF101). So committing at an intermediate layer is cheap once earlier draws are available, and the draw is what matters.

Two design choices keep the earliest groups well informed. All positions first pass through P shared, bidirectional prefix layers that see the whole grid (20 of 36 layers for LlamaGen-XL, 8 of 24 for OmniLM), so G1 is drawn from states that already encode the layout. This is why IntraAR matches or improves on the depth-matched raster readout in NLL in both settings: a causal raster readout cannot see the rest of the grid. In addition, IntraAR's NLL stays within 0.04 nats/token of Raster on the three ImageNet backbones and, with video Raster NLL now measured, on UCF101 (5.78 versus 5.76).

W4. A raster baseline that decodes different tokens at different layers.

Thank you for this suggestion; it is now Table R3 (App. G.4), on LlamaGen-XL and UCF101, trained from the same weights with teacher forcing. Raster, exit at ℓ_s keeps raster order and causal attention, reads code i out at the commitment layer of its IntraAR group, and feeds the draw into the next pass at the input, so every draw is in view.

UCF101 / OmniLMPassesNLLI3D-FVDLatency (s)
Raster5,1205.76193.76174.04
Raster, exit at ℓ_s5,1205.82197.36174.67
IntraAR15.78195.594.19
No sample write16.60269.574.15

ImageNet/LlamaGen-XL FID: Raster 19.40, Raster exit at ℓ_s 19.83, IntraAR 19.54, no write 29.70 (Table R3).

How to read it. Decoding different codes at different layers does not by itself produce IntraAR's results, and it does not by itself hurt much: with the draws in view, the raster readout loses only 0.05 to 0.06 nats/token, while the no-write control, which reads at exactly the same layers without the draws, loses 0.40 to 0.82. What carries the quality is the realized draw, and IntraAR keeps it while needing one pass instead of 5,120. The contribution of the draw itself is isolated by the no-write control, which keeps IntraAR's groups, layers, and computation and removes only the drawn code.

Where in the paper: Table R3 and App. G.4; Sec. 5.3 ("Context, code, and a fixed record"); the existing evidence in Sec. 5.4 and App. D.4 and D.5.

AuthorsOpenReview ↗

Response to Reviewer na61 (2/2): the depth bound and verification

(2/2) This part covers W2 and W3.

W2. The number of sequential conditioning stages is bounded by the number of layers.

Correct, and the paper states this as Theorem 2: with distinct commitment layers, one pass has at most L − P sampling steps. The revision measures what the bound costs and how to relax it.

  • Within the bound, returns diminish quickly. With S = 8 steps, FID or FVD is already within 2 to 12% of its 16-step value (Fig. 7, App. D.5), so on the grids of the original experiments the 16 available steps are close to what is useful.
  • When the grid grows, the bound binds. New Sec. 5.5 retrains Raster and IntraAR on 384² clips and on 65-frame clips. With 16 steps, the groups grow and the one-pass gap to Raster grows with them: +1.83 I3D-FVD at 320 codes per group, +7.34 at 720, and +17.67 at 1,088, as the theorem predicts.
  • The bound is per pass. R passes give R(L − P) steps. A second pass, which reads the first pass's codes as observed frames, recovers 53 to 69% of the gap at 0.93 to 1.02× the one-pass latency, because it also removes attention between the two halves. At 65 frames that is 2 sequential passes instead of 17,408 (latency 18.24 s versus 676.85 s).

Sec. 6 (Limitations) now states the bound together with this remedy, so readers know when one pass suffices.

W3. Compared with speculative decoding, IntraAR decodes earlier tokens without verification.

You are right that IntraAR has no verification step. It does not need one, because nothing in it is an approximation of another model. Speculative decoding drafts tokens with a cheaper model and verifies them so that the output follows a fixed target distribution (Leviathan et al., 2023). IntraAR's draw at layer ℓ_s is a sample from IntraAR's own model: every conditional is a normalized categorical, the product over groups is an exact autoregressive distribution, and its likelihood is evaluated exactly in one teacher-forced pass (Lemma 1). There is no target to verify against. What verification protects in speculative decoding, the distance to the reference model, we measure directly: NLL within 0.04 nats/token of Raster, FID within 0.14 to 0.28, and I3D-FVD within −2.26 to +1.83.

To make the comparison concrete, Table R4 now runs speculative decoding with full verification (SpecDec: draft length 4, the VVS drafter of Table 3, full verification against the matched unguided Raster) on ImageNet/LlamaGen-XL:

ImageNet / LlamaGen-XLFIDsFIDLatency (s)
Raster19.408.872.72
SpecDec (full verification)19.438.912.92
VVS (partial verification, CFG 1.75; Table 3)5.387.483.50
IntraAR19.548.780.19

SpecDec reproduces Raster's quality, as exact verification guarantees, but at temperature 1 visual drafts are rarely accepted, so it runs at 1.07× Raster's latency. This matches Jang et al. (ICLR 2025), who report 0.93× speed for EAGLE-2 on LlamaGen-XL with random sampling. VVS accepts more drafts by skipping part of the verification and still does not beat Raster's latency in our setting. Verification thus preserves a target at the price of sequential passes, while IntraAR removes the passes and lets its own likelihood and sample metrics show how close it stays.

Where in the paper: Sec. 2 (speculative decoding sentence), Sec. 5.2 ("Guidance" paragraph, last sentence), Table R4 (bottom) and App. G.5; Sec. 5.5, Table R2, and Sec. 6 for W2.

We are grateful for questions that made the mechanism easier to see. If the new baselines and the scale study resolve your concerns, we would kindly ask you to consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and new results

We are grateful to all three reviewers for their time and for comments that were specific enough to act on directly. Every request led to a concrete change in the paper, and we believe the revision is clearly stronger for it.

What the reviewers found convincing. Reviewer G118 describes the idea as "elegant and well-motivated" and the experiments as supporting the main claim strongly. Reviewer toR4 notes that the paper "clearly explains the tension between parallel sampling and preserving dependencies" and that intermediate sampling, write-back, and deeper conditioning "form a coherent mechanism", with useful ablations and clearly specified latency protocols. Reviewer na61 highlights the exact likelihood, the analysis of earlier draws and sampling steps, and the latency reduction over raster decoding.

Main revisions (all new or changed text is blue in the revised PDF; new tables are numbered R1 to R6 so that the original numbering is unchanged):

  1. Guided matched pairs at four ImageNet and two UCF101 guidance scales (G118): Table R1 and "Guidance" in Sec. 5.2; all metrics in Table R5 and App. G.1.
  2. Longer and larger clips (G118, toR4, na61): new Sec. 5.5 with Table R2; all metrics, a two-pass variant, and a drift check on the last frames in Table R6 and App. G.2.
  3. Two more video decoders, MaskGIT and LPD, on UCF101 and K600, and speculative decoding with full verification on ImageNet (toR4, na61): Table R4 and App. G.5.
  4. A depth-matched raster baseline that reads every code at its IntraAR layer (na61): Table R3, Sec. 5.3, App. G.4.
  5. Two write and memory variants, code-only write (LlamaGen-XL, UCF101) and unfrozen rows (UCF101) (toR4): new rows in Tables 4 and 5, Sec. 5.3, App. G.3.
  6. Relation to DnD and the scope of the method for continuous-token and diffusion generators (toR4): Sec. 2, the Limitations paragraph of Sec. 6, and App. G.6.
  7. The AI-use statement now also discloses LLM assistance with drafting and editing the discussion-period text.

The method, the title, the theory, and the claims are unchanged. Each new experiment checks a statement the submission already makes.

New results at a glance

QuestionNew evidenceOutcome
Does the matched comparison hold under guidance?Raster and IntraAR at the same scale, both branches in one batchFID gap 0.12 to 0.23; I3D-FVD gap 1.52 to 2.52; speedup 14.4-14.5× (ImageNet) and 40.2-40.3× (UCF101)
What happens on longer or larger clips?65 frames at 256² and 17 frames at 384², Raster and IntraAR retrainedone-pass gap grows with codes per group, as Theorem 2 predicts (+1.83 to +17.67); a second pass recovers 53-69% of it at 0.93-1.02× the latency
Is IntraAR still the fastest against more decoders?MaskGIT-12 and LPD-64 on UCF101 and K600; SpecDec on ImageNetlowest latency on both video tasks; SpecDec matches Raster quality but runs at 1.07× Raster's latency
Does early readout, rather than the write, explain the results?Raster reading each code at its IntraAR layer (LlamaGen-XL, UCF101)early readout costs Raster 0.05-0.06 nats/token, a tenth of what removing the write costs at the same layers
Do the context in the write and freezing matter?code-only write, unfrozen rowscode alone closes 76% (UCF101) and 83% (LlamaGen-XL) of the no-write gap; unfrozen rows raise UCF101 I3D-FVD from 195.59 to 201.48

Two results sharpen statements the paper already makes. Under guidance, IntraAR at CFG 1.25 has lower I3D-FVD than NAR on UCF101 (181.16 versus 184.53), which removes the one Table 1 setting where NAR led because of its guidance. On 65-frame clips one pass is clearly behind Raster, and we now recommend two passes for such grids (Sec. 5.5 and Sec. 6).

Individual replies follow under each review. We would be glad to clarify anything further during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.