- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes intraAR, an image / video generating principle where sample deeper at network instead of multiple forward process like normal AR doing. The experiment shows a good potential of the current method, comparing with the 5 classical video generator baseline under 3 imagenet backbones, intraAR greatly reduces latency by 24.5 - 41.5 on video and 11.2 - 29.8 on images, with relative the same FVD and FID.
Critical strengths
Strengths:
-
The most interesting part of this paper is the idea itself, elegant and well-motivated, with addressing the challenges currently like lossing coherence and hidden states problem, the intraAR solves this by simply sampling deepth.
-
Well designed and comprehensive experiments results support the main claim strongly.
Critical weaknesses
Weakness:
-
Like the thoery proved here, the current method i think is a good fit for the unguided generation. I may also want to see under the guided direction how far this method can push forward, like different CFG number or A guided raster vs. IntraAR pair.
-
I think one of the limitation for this paper is that the number of sampling step is largely constrained by the depth. So what might happen for the longer or higher resolution videos. It would be btter to show a scaling experiment here.
I am happy to rasie my score later if these results are interesting
Questions for authors
Besides the weakness above, here is one questions i am thinking of:
How do quality and latency change for longer or higher-resolution clips?
3 comments
Response to Reviewer G118 (2/2): longer and higher-resolution clips
(2/2) This part answers W2 and your question on longer and higher-resolution clips.
W2 / Q. How do quality and latency change for longer or larger clips, given that depth limits the number of steps?
The one-pass gap to Raster grows with the number of codes per group, exactly as the depth bound predicts, and a second pass removes most of it at about the same latency. IntraAR stays the fastest decoder in every setting.
Setup. We retrained Raster and IntraAR (L/P/S = 24/8/16) from the UCF101 weights on 65-frame clips at 256² and on 17-frame clips at 384², using OmniLM's native 4× temporal and 8× spatial compression. Since one pass has at most L − P = 16 steps (Theorem 2), the groups grow with the grid: 320 codes per group in Table 1, 720 at 384², and 1,088 at 65 frames. A two-pass variant commits the first half of the grid in 16 groups and then treats those codes as observed frames, exactly as K600 treats its prefix, for 32 steps in total.
Results (I3D-FVD / batch-1 latency in s, three-seed means):
| Clip (codes) | Codes/group, 1 pass | Raster | IntraAR, 1 pass | IntraAR, 2 passes |
|---|---|---|---|---|
| 17 f, 256² (5,120) | 320 | 193.76 / 174.04 | 195.59 / 4.19 | 194.62 / 4.27 |
| 17 f, 384² (11,520) | 720 | 198.47 / 421.58 | 205.81 / 11.31 | 200.71 / 10.83 |
| 65 f, 256² (17,408) | 1,088 | 341.95 / 676.85 | 359.62 / 19.71 | 348.91 / 18.24 |
- Quality. The one-pass gap is +1.83, +7.34, and +17.67 I3D-FVD. It follows codes per group, which is the dependence floor of Theorem 2 at work. Halving the groups with a second pass recovers 53 to 69% of the gap (61 to 69% on the two enlarged grids). VideoMAE-FVD, computed on 16 frames spread over the whole clip, shows the same pattern.
- Latency. Two passes cost 0.93 to 1.02× the one-pass latency. The second pass reruns the stack once more, but it also removes the attention between the two halves, which grows with the square of the grid, so the two effects nearly cancel. At 65 frames, two passes take 18.24 s against 676.85 s for Raster.
- No drift at the end of long clips. On the last 17 frames of the 65-frame clips, I3D-FVD is 228.73 (Raster), 232.58 (two passes), and 239.84 (one pass), so the late frames of an IntraAR clip are as close to Raster as the clip as a whole.
In practice, one pass suffices for grids of the original size, and long grids should use a second pass, which keeps the latency of a single pass (Sec. 5.5 and Sec. 6).
Where in the paper: new Sec. 5.5 with Table R2; all metrics and peak memory in Table R6 and App. G.2; Sec. 6 (Limitations) states the bound and the remedy.
For the AC discussion, in brief. (i) Every comparison is matched: Raster and IntraAR start from the same weights and see the same tokens, so the speedups are not recipe effects. (ii) The mechanism is isolated: removing only the drawn code at identical computation raises UCF101 I3D-FVD from 195.6 to 269.6, and a new depth-matched raster baseline (Table R3) shows that reading codes at intermediate layers costs little once the draws are in view, so the gain comes from the write. (iii) The new evidence (guidance, longer clips, more decoders) agrees with the original claims, and where one pass reaches its limit, the paper now says so and gives the fix.
Your comments directly shaped Sec. 5.5 and the guided evaluation, and we thank you for them. You mentioned you would be happy to raise your score if these results were interesting; if they address your two questions, we would be very grateful if you would consider doing so and supporting the paper in the discussion.
Thank you so much for the detail rebuttal.
Most of my concerns are solved, and I think I will main my current score.
Response to Reviewer G118 (1/2): guided generation
(1/2) We are very grateful for your review. You put your finger on exactly the idea we care about: before a code is drawn every hidden state is a deterministic function of the condition, so coherence has to come from the drawn code itself, and the remaining depth of the same pass can condition on it. It means a lot to us that you found this elegant and well motivated, and that you read the matched experiments as strong support for the main claim. Your two requests, guidance and scale, were the two things missing from the evaluation, and both are now in the paper. This reply covers guidance; the next one covers longer and larger clips.
W1. How far does IntraAR go under guidance?
It keeps its matched behaviour at every scale we tested (four ImageNet and two UCF101 scales, including NAR's 1.25). Under identical classifier-free guidance, IntraAR stays within 0.12 to 0.23 FID of Raster on ImageNet and within 1.52 to 2.52 I3D-FVD on UCF101, and the latency reduction remains 14.4-14.5× and 40.2-40.3×.
How guidance works in one pass. Both branches (conditional and null-class) run as one batch of two through the same pass. At each commitment the two logits are combined as u + w(c − u), one code is drawn per position, and each branch writes that same code into its own memory row. This mirrors raster guidance, which feeds one sampled token to both branches. Training is unchanged, and Raster uses its native guided sampler with the same null-class pathway.
Results (three-seed means; ImageNet/LlamaGen-XL FID, UCF101/OmniLM I3D-FVD; CFG 1 rows are the original Tables 1 and 3):
Batch-1 latency with both branches: 3.01 s versus 0.21 s on ImageNet (CFG 2), 184.28 s versus 4.58 s on UCF101 (CFG 1.25).
What we read from this:
You are right that the theory concerns the unguided model: guidance samples a different distribution, so Theorems 1 and 2 do not cover it. The revision says this explicitly and presents the guided results as empirical.
Where in the paper: Table R1 and the "Guidance" paragraph of Sec. 5.2; full metrics (sFID, IS, precision, recall, per-seed speedup) in Table R5 and App. G.1; Sec. 6 states the scope.