acceptodds
Under review as a conference paper at ICLR 2027

VFlash-FM: Training for One-Pass Drafting in Video Language Models

Abstract

Speculative decoding reduces the serial cost of video language model generation, but drafting itself is constrained by the long visual context: autoregressive drafters pay for it at every proposed token, while one-pass parallel drafters lose most of a block to an early error. We introduce VFlash-FM, which improves one-pass drafting through training alone. Our key observation is that a discrete-flow path from a fully masked block to the target continuation defines a family of auxiliary prediction tasks: completing a block from the drafter's own partially revealed predictions, including its mistakes. Training the drafter on these self-generated partial states, with a step-aware flow objective and target-confidence weighting oriented toward accepted prefixes, improves the same network's predictions from all-mask inputs. The extra computation is paid once during training rather than at every decoding step, so deployment remains a single drafter forward over cached target features followed by target verification. Across five video benchmarks, VFlash-FM reaches macro decoding speedups of on LLaVA-OneVision-7B and on Qwen2.5-VL-7B, exceeding the published speedups of prior video drafting methods we compare against; on LLaVA, it improves over a non-flow drafting variant in both committed progress and speed across all tested verification configurations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, video understanding, multimodal, video llms

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8ZGmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VFlash-FM improves one-pass drafting through additional training. It learns from partially filled drafter states using step conditioning, a discrete flow-path loss, direct supervision, and CAT weighting, while retaining one draft forward pass per inference round. Experiments evaluate long-video description generation with chain and tree verification on LLaVA-OneVision and Qwen2.5-VL.

Critical strengths
  1. The paper is clearly written, and the training method and inference procedure are easy to follow. It clearly explains that the additional computation is spent during training, while inference still requires only one draft forward pass per round.

  2. The comparison with the non-flow baseline shows consistent improvements across the tested video datasets and verification configurations. The method accepts more draft tokens and achieves faster decoding, providing useful evidence that the overall training recipe works.

Critical weaknesses
  1. The method introduces partial-state training, step conditioning, and a new loss together, so it remains unclear which changes account for the gains. It would help to compare against partial-state training without the path loss, as well as a simpler baseline that only adjusts the supervision weights.

  2. The method adds extra forward passes, while the baseline is mainly matched by training epochs. Since the approach trades additional training computation for faster inference, a comparison under the same GPU-hour budget would help establish whether it still outperforms a baseline trained for longer.

  3. The evaluation covers two 7B target models, but does not include other model sizes or inference at larger batch sizes. It also focuses on long-form descriptions and excludes video prefill from timing. Further results would help clarify how the gains extend to other tasks, concurrent workloads, and end-to-end request latency.

Questions for authors

Please see the weaknesses.

3 comments

Thank you very much for your review and for your comments. We address your concerns one by one below.

The method introduces partial-state training, step conditioning, and a new loss together, so it remains unclear which changes account for the gains. It would help to compare against partial-state training without the path loss, as well as a simpler baseline that only adjusts the supervision weights.

Thank you for this comment. We report the comparison without the path loss that you ask for, and the closest matched run we have to a baseline that only adjusts the supervision weights. To keep them matched, both runs below were trained with the same compute setup: the same deduplicated list of 176.5k clips, the same drafter with 4 layers and 5 epochs. Both use block 16 and one drafting pass.

Partial statesStep conditioningPath lossChain: M / decodeTree 60: M / decode
Anoyes, with a constant inputyes5.90 / 4.21×7.72 / 4.10×
Byesyesno5.85 / 4.17×7.55 / 4.01×

Partial state training without the path loss. This is row B. It trains on partial states with the direct supervision only, CE + KL, and reaches 5.85. Row A has the path loss on the all mask state and never sees partial states, and reaches 5.90. So at one drafting pass the run with the path loss is slightly ahead, 5.90 against 5.85, and partial states do not add accepted tokens.

The full recipe at the same compute. We tried to train the full recipe, with all three changes together, at the same compute as rows A and B during the rebuttal period, but the time was not enough to reach results. We therefore cannot yet say how it compares with them, and we will report it in the final revision. Rows A and B are controls, not a comparison with the full recipe.

A baseline that only adjusts the supervision weights. On all mask states the path loss equals 10·CE at every logged step, because its mask term is numerically zero. Row A is therefore training with adjusted weights, 11·CE + KL, plus the step conditioning. It is the closest matched run to the baseline you describe, not exactly it, because of the step conditioning.

Step conditioning. Both rows contain the step conditioning module. In row A its input never changes, t is always 0 and h is always 1, because that run only sees all mask states. In row B it varies with the state. So these two runs cannot isolate the effect of the module.

Which change accounts for what. The clear effect appears with a second drafting pass. Chain arm, 750 clips:

Drafter1 pass: M / decode2 passes: M / decode
Row A, no partial states5.90 / 4.21×2.15 / 1.30×
Row B, partial states without the path loss5.85 / 4.17×6.44 / 3.85×

Partial state training is what makes a second pass possible, and it does not need the path loss for that: the drafter of row B gains 0.59 M from a second pass, while the drafter of row A collapses. On a 7B target the second pass costs more time than it saves, 3.85× against 4.17× at one pass, so this is not yet a wall clock gain. We are now working on making the second pass faster, so that it becomes a wall clock gain as well.

The method adds extra forward passes, while the baseline is mainly matched by training epochs. Since the approach trades additional training computation for faster inference, a comparison under the same GPU-hour budget would help establish whether it still outperforms a baseline trained for longer.

Thank you for this comment. You are right that the extra forward passes add training cost, and we measured it. The two runs below share the same deduplicated list of 176.5k clips, the same drafter with 4 layers and 5 epochs. They differ in whether training uses the extra forward passes.

Extra forward passes in trainingTraining GPU hoursChain: M / decodeTree 60: M / decode
Anone, all mask states only1,7565.90 / 4.21×7.72 / 4.10×
Brollouts to partial states2,0385.85 / 4.17×7.55 / 4.01×

Cost of the extra forward passes. For the same number of epochs the rollouts add about 16% GPU hours. It is less than the number of extra passes suggests, because a training step is dominated by the frozen target forward and by video decoding. The GPU hours count the whole job time, including restarts and the evaluations during training.

Same GPU hour budget. We did not have the time in the rebuttal period to train a run for longer, so we cannot show two runs with exactly the same budget. But rows A and B already answer the question in one direction. The run without the extra passes uses 16% fewer GPU hours and accepts at least as many tokens at one drafting pass. Training it for longer, up to the budget of row B, is not expected to make it worse. So at one drafting pass the extra forward passes do not pay for their cost.

What the extra cost buys. It buys the ability to use a second drafting pass, as the table in our first answer to you shows: with the rollouts a second pass adds 0.59 M, and without them the drafter collapses.

The evaluation covers two 7B target models, but does not include other model sizes or inference at larger batch sizes. It also focuses on long-form descriptions and excludes video prefill from timing. Further results would help clarify how the gains extend to other tasks, concurrent workloads, and end-to-end request latency.

Thank you for this comment. We added end to end measurements that include the target video prefill, and measurements on shorter outputs and on other instructions. Unless stated otherwise, the numbers are with chain verification on one GPU, on the 750 clips of the paper protocol, with the autoregressive baseline timed in the same process. End to end includes the target video prefill.

TargetPrefillMDecodeEnd to end
LLaVA OneVision 7B1.42 s5.493.93×2.31×
Qwen2.5 VL 7B2.41 s3.172.34×1.57×

Video prefill and end to end latency. The end to end speedup is lower than the decode speedup, and the whole difference comes from the prefill of the target, which cannot be avoided: the target has to read the video once, and that takes the same time with and without our drafter. The drafter adds no prefill of its own. On the LLaVA OneVision 7B target the time to the first token is 1.42 s in both cases. At 512 tokens the prefill is about a quarter of the autoregressive time and more than half of ours, so 3.93× per token becomes 2.31× end to end.

Other tasks. We tested shorter outputs and other instructions on the LLaVA OneVision 7B target.

SettingMean output tokensMDecodeEnd to end
Paper protocol, detailed description, cap 5124115.493.93×2.31×
Cap 2562474.242.98×1.75×
Cap 1281283.412.39×1.39×
Cap 64643.422.35×1.23×
A different long instruction, cap 5124055.473.92×2.27×
A short instruction602.731.96×1.19×

Shorter outputs: the end to end speedup follows the output length, 2.31× at 512 tokens, 1.75× at 256, 1.39× at 128 and 1.23× at 64. There are two causes. The prefill is a fixed cost, and acceptance is lower at the start of a response: M is about 3.4 over the first 64 to 128 tokens against 5.5 over 512.

Other instructions: a different long instruction, "Analyze the video, focusing on its cinematography and narrative structure.", gives practically the same acceptance and speed as the prompt of the paper, 5.47 and 3.92× against 5.49 and 3.93×. A short instruction, "Describe the video in two sentences.", gives 60 output tokens on average and behaves as that length predicts, 1.19× end to end.

So the gain depends on the length of the answer and not on the instruction. For tasks whose answers are only a few tokens long there is almost nothing left to accelerate.

Other model sizes. In prior work such as ParallelVLM, the larger targets are 32B and 72B models, and these sizes require a very large amount of compute for our method: every target needs its own drafter, and the frozen target runs in every training step. We started these experiments on a 32B target during the rebuttal period, but the time was not enough to complete them. We will report the results in the final revision.

Larger batch sizes. The results above are for one request at a time. For this answer we implemented a first batched version of our decoding and compared it with batched autoregressive decoding on the same GPU. All requests of a batch advance together by the shortest accepted prefix among them, and a request that accepted more keeps its extra tokens for its next block. Every committed token is still verified by the target. 160 clips, 32 from each benchmark, decoding only.

Batch sizeTokens per cycleSpeedup over batched autoregressive decoding
15.523.93×
24.743.14×
43.991.94×

The gain holds at batch sizes 2 and 4, becomes smaller as the batch grows, and with this version is lost at larger batch sizes. The main reason is that autoregressive decoding gets much cheaper per request in a batch, since one step takes about the same time for several requests as for one. We will add this analysis to the final revision.

Concurrent workloads. We ran one small test: 1 to 4 requests at the same time on one GPU, each in its own process, on 12 to 16 clips of the VideoDetailCaption benchmark. Each speedup is against autoregressive decoding measured under the same load.

Requests at the same timeDecodeEnd to end
13.07×1.98×
23.29×1.90×
33.47×2.21×
43.93×2.14×

The speedup stays at the same level when the GPU is shared. This is a small test, so we give it as a first indication only.

Reviewer_WR3tOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VFlash-FM, a speculative decoding method for Video Large Language Models (VideoLLMs) focusing on improving the training recipe (partial reveal of own predictions, step-aware discrete flow matching, and target-confidence weighting) for one-pass drafters. The VFlash-FM is evaluated across five video benchmarks against three baseline methods (SpecVLM, ParallelVLM, SPARROW) on two VideoLLMs (LLaVA-OneVision-7B, Qwen2.5-VL-7B).

Critical strengths
  • The paper is well-written and easy to follow
  • The proposed training recipe (VFlash-FM) is well motivated, and the experimental results are promising.
Critical weaknesses
  • The performance gain over baseline methods is clear, but the report is limited to decoding-time comparison.
  • The proposed method is evaluated on two VideoLLMs with the same parameter size (7B).
Questions for authors
  • A schematic illustration of the proposed method can help readers more easily understand the motivation and solution of the proposed VFlash-FM. Please consider adding one.
  • I expect that the compute bottleneck of VideoLLMs can come before decoding time. Could the authors provide supplementary evaluation on end-to-end latency improvement over baseline methods?
  • Please consider adding comparative experiments on VideoLLMs with parameter sizes other than 7B.
2 comments

Thank you very much for your review and for your comments. We address your concerns one by one below.

A schematic illustration of the proposed method can help readers more easily understand the motivation and solution of the proposed VFlash-FM. Please consider adding one.

Thank you for this suggestion. We agree that a schematic illustration will make the motivation and the solution much easier to follow, and we will add one to the final revision.

The figure will show one full decoding cycle. The drafter proposes a block of tokens in one pass, and the target model checks the block in a single pass and keeps the longest prefix it agrees with. The figure will also show how the drafter is trained, so that the difference from the DFlash+CAT baseline is visible at a glance.

I expect that the compute bottleneck of VideoLLMs can come before decoding time. Could the authors provide supplementary evaluation on end-to-end latency improvement over baseline methods?

Thank you for this question. You are right that a large part of the time is spent before decoding, when the target model reads the video. We measured it, and we now report end to end speedups that include this prefill. The numbers are for the drafter of the paper with chain verification on one GPU, on the 750 clips of the paper protocol, against autoregressive decoding with the same target, timed in the same process.

TargetPrefillDecodeEnd to end
LLaVA OneVision 7B1.42 s3.93×2.31×
Qwen2.5 VL 7B2.41 s2.34×1.57×

Where the time goes. On the LLaVA OneVision 7B target the 128 frames are about 25,000 prompt tokens, and reading them takes 1.42 s with and without our drafter. At 512 output tokens this is about a quarter of the autoregressive time and more than half of ours, so 3.93× in decoding becomes 2.31× end to end.

What our method adds before decoding. Nothing. The drafter reuses the features that the target has already computed, so it has no prefill of its own. The time to the first token is 1.42 s with and without it.

Baseline methods. The prefill of the target cannot be avoided by any speculative decoding method that keeps the target unchanged, so it is the same fixed cost for all of them. Before decoding starts, the methods differ only in what the draft side adds on top of it, and ours adds nothing. We also ran one baseline, SpecVLM, with its public code under the same conditions: the same GPU, the same prompt, 128 frames and a cap of 512 tokens. We ran it with two drafters: the 7B model drafting for itself, as in its paper, and a smaller drafter, LLaVA OneVision 0.5B.

Method, with the LLaVA OneVision 7B targetDecodeEnd to end
SpecVLM, public code, 0.5B drafter0.51×0.54×
SpecVLM, public code, 7B self drafting0.47×0.49×
The drafter of the paper, chain3.93×2.31×

The speedups of SpecVLM are measured against the autoregressive loop of its own code.

Please consider adding comparative experiments on VideoLLMs with parameter sizes other than 7B.

Thank you for this suggestion. We agree that other model sizes matter. In prior work such as ParallelVLM, the larger targets are 32B and 72B models, and these sizes require a very large amount of compute for our method: every target needs its own drafter, and the frozen target runs in every training step. We started these experiments on a 32B target during the rebuttal period, but the time was not enough to complete them. We will report the results in the final revision.

I appreciate the authors’ response. My concern on end-to-end performance gain is addressed, but other concerns remain unresolved. I understand that the rebuttal period is limited, thus am maintaining my score unchanged, but my recommendation is borderline.

Reviewer_JXh3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how training can improve one-pass parallel drafting for video language models. VFlash-FM reuses cached target-model features and employs a four-layer drafter to predict a block containing 15 proposal positions. During training, no-gradient rollouts create partially revealed states containing the drafter's own mistakes. Supervision combines time/step-size conditioning, a discrete-flow path loss, target cross-entropy and top-64 KL, and CAT prefix-confidence weighting. Deployment retains a single drafter forward followed by chain or tree verification. Evaluation uses fixed-prompt, long-form caption generation on five video collections. Macro decoding speedups, excluding multimodal prefill, reach 3.850× on LLaVA-OneVision-7B and 2.541× on Qwen2.5-VL-7B. On LLaVA, all nine verification configurations improve over a locally trained non-flow DFlash+CAT variant.

Critical strengths

The local comparison shows consistent gains without adding rollout passes at inference. In the chain comparison in Table 2, mean committed tokens per cycle increase from 4.76 to 5.50, and macro speedup rises from 3.472× to 3.773×. Appendix Table 4 also shows an increase from 3.486× to 3.850× with the same plain Tree-60 rule, so the gain is not merely an artifact of choosing different tree structures. Improvements of 6.8%–10.9% across all nine verification configurations are more persuasive than a single favorable operating point.

The connection between training and useful accepted prefixes is developed carefully. The method is motivated by consecutively correct prefixes rather than isolated token accuracy, and explicitly trains recovery from the drafter's own erroneous partial states. The all-mask loss decomposition, gradients for erroneous tokens, and accounting of additional training rollouts clarify what the recipe changes. Reusing target hidden states also avoids a separate drafter Transformer prefill over the long visual context, and results on a second target demonstrate deployment feasibility beyond one model.

Critical weaknesses

The ablation validates a combined training recipe without identifying the main source of improvement. Relative to DFlash+CAT, FM simultaneously adds partial states, the path loss, 22.5M conditioning parameters, and an average of 3.25 no-gradient forwards before each gradient-bearing forward. For training samples at the all-mask deployment condition, the objective becomes 11 CE + KL - 10 q(M), which already changes the balance of hard-label and soft-distribution supervision substantially. The appendix appropriately notes that rollout states do not follow the reference path distribution and that there is no theoretical guarantee of improved all-mask prediction. Simple controls matching loss weights, capacity, and training compute are therefore important for identifying the incremental contribution of flow/partial-state training. The slightly different LLaVA training manifests should ideally also be unified; Appendix F states that evaluation clips are excluded from training, so this difference is not evidence of evaluation leakage.

The evaluation mainly supports long-form caption decoding, with more limited evidence for general video tasks and cross-method superiority. All five benchmarks use the same detailed-description prompt, with 50 clips for each of three fixed seeds per benchmark, rather than the benchmarks' native QA or reasoning tasks. Timing starts after the first target token and excludes multimodal prefill, so the reported factors are not end-to-end request speedups. SpecVLM, ParallelVLM, and SPARROW results in Table 1 are taken from their respective papers, without harmonized timing and length statistics. The strongest evidence is therefore the local DFlash+CAT comparison; broader task coverage and absolute superiority over other methods still require matched-environment evaluation.

Questions for authors

On the same deduplicated training manifest, could the authors compare at least all-mask training with matched CE/KL weights, partial-state training with the original direct supervision, and the full recipe, while reporting training GPU-hours? A compute-matched DFlash+CAT control would also clarify the value of the extra rollouts.

What are the end-to-end speedups including prefill for native video QA, shorter responses, or varied instructions? Could at least one main video speculative-decoding baseline be reproduced with matched hardware, prompts, output caps, and timing boundaries?

What is the full-output token-by-token agreement rate of the chain and tree implementations against the same-precision autoregressive baseline? If numerical differences occur, can the first divergence positions and validation of split attention, tree positions, and cache updates be reported?

3 comments

Thank you very much for your review and for your comments. We address your concerns one by one below.

Setup shared by all numbers. We use the protocol of the paper: 5 benchmarks, 3 draws and 50 clips each, 750 clips in total, 128 frames, at most 512 new tokens, the LLaVA OneVision 7B target, bf16. Everything was measured again on a single GPU, with the autoregressive baseline timed in the same process. This GPU is a newer model than the one used in the paper, so the absolute speedups differ slightly from the tables of the paper. M, the number of accepted tokens per cycle, does not depend on hardware. Decode excludes prefill, end to end includes it.

On the same deduplicated training manifest, could the authors compare at least all-mask training with matched CE/KL weights, partial-state training with the original direct supervision, and the full recipe, while reporting training GPU-hours? A compute-matched DFlash+CAT control would also clarify the value of the extra rollouts.

Thank you for this question. We ran the parts of this decomposition that fit in the rebuttal period, and we answer each part separately. To keep the comparison matched, every row below was trained with the same compute setup: the same deduplicated list of 176.5k clips, the same drafter with 4 layers and 5 epochs. Both rows use block 16 and one drafting pass.

TrainingTraining GPU hoursChain: M / decodeTree 60: M / decode
AFlow objective on all mask states only, with the t and h conditioning1,7565.90 / 4.21×7.72 / 4.10×
BPartial states with direct supervision: CE + KL, no path loss2,0385.85 / 4.17×7.55 / 4.01×

All mask training with matched CE/KL weights. The closest run we have at this compute is row A. It trains on all mask states only. There the path loss equals 10·CE at every logged step, because its mask term is numerically zero, so the objective is 11·CE + KL. Row A has the same t and h conditioning parameters as the full recipe, so it is not a pure reweighting control. Its only difference from the full recipe is that it never sees partial states. It reaches 5.90.

Partial state training with the original direct supervision. This is row B, at 5.85. At one pass, partial states give no gain over all mask training.

The full recipe. Thank you for suggesting this comparison. We tried to run the full recipe at this compute during the rebuttal period, but the time was not enough to reach results. We will report it in the final revision.

Training GPU hours. They are in the table and count the whole job time, including restarts and the evaluations during training. Rollouts add about 16%, because a training step is dominated by the frozen target forward and by video decoding.

Compute matched DFlash+CAT control. Thank you for suggesting this control as well. A properly matched control needs all arms to be trained on the same hardware, and the time of the rebuttal period was not enough for that. We will train all arms on the same hardware and report this control in the final revision. The DFlash+CAT checkpoint of the submission is not a matched control, because it was trained with different compute and schedule, so its numbers in the paper should not be compared with rows A and B.

The value of the extra rollouts. At one pass we measure none: with the same compute, all mask training in row A is at least as good as partial state training in row B. Their value appears with a second drafting pass. Chain arm, 750 clips:

Drafter1 pass: M / decode2 passes: M / decode
Row A, all mask states only5.90 / 4.21×2.15 / 1.30×
Row B, partial states with direct supervision5.85 / 4.17×6.44 / 3.85×

Only the drafter trained on partial states can use a second pass: it gains 0.59 M, while the all mask drafter collapses. On a 7B target the second pass costs more time than it saves, 3.85× against 4.17× at one pass, so this is not yet a wall clock gain. We are now working on making the second pass faster, so that it becomes a wall clock gain as well.

What is the full-output token-by-token agreement rate of the chain and tree implementations against the same-precision autoregressive baseline? If numerical differences occur, can the first divergence positions and validation of split attention, tree positions, and cache updates be reported?

Thank you for this question. The outputs are not identical to the autoregressive baseline token by token. Over the 750 clips of the paper protocol, the median position of the first token that differs is token 74 with the chain and token 73 with the tree. Both are in bf16, against the baseline in the same precision.

Why this happens. Speculative decoding promises no loss in distribution: every committed token is the choice of the target model itself. It does not promise the same exact sequence, because that is limited by the hardware. Verifying several tokens in one forward changes the order of the floating point arithmetic compared with decoding one token at a time. In bf16 this changes the logits slightly, a near tie between two tokens can flip, and from that token on the two generations continue differently. This applies to every speculative decoding setup, not to ours in particular. We confirmed it with the target alone, with no drafter and no tree: when it rescores its own output in blocks, it diverges in the same way in bf16 and not at all in fp32.

Validation of split attention, tree positions and cache updates. We compared our split attention with the standard attention on real draft trees, branch by branch, and it adds no difference beyond the same rounding. The tree positions and the cache updates pass our structural checks.

What are the end-to-end speedups including prefill for native video QA, shorter responses, or varied instructions? Could at least one main video speculative-decoding baseline be reproduced with matched hardware, prompts, output caps, and timing boundaries?

Thank you for this question. We measured shorter responses and varied instructions, and we ran one baseline under matched conditions. We answer each part separately. Our numbers are for the drafter of the paper with chain verification, on one GPU, on the 750 clips of the paper protocol, against an autoregressive baseline timed in the same process. End to end includes prefill, decode does not.

SettingMean output tokensMDecodeEnd to end
Paper protocol, detailed description, cap 5124115.493.93×2.31×
Cap 2562474.242.98×1.75×
Cap 1281283.412.39×1.39×
Cap 64643.422.35×1.23×
A different long instruction, cap 5124055.473.92×2.27×
A short instruction602.731.96×1.19×

End to end speedup. The end to end speedup is lower than the speedup per token, and the whole difference comes from the prefill of the target, which cannot be avoided: the target has to read the 128 frames, about 25,000 prompt tokens, once, and that takes 1.42 s with or without our drafter. The drafter adds no prefill of its own: it reuses the features that the target has already computed. The time to the first token is 1.42 s in both cases. At 512 tokens the prefill is about a quarter of the autoregressive time and more than half of ours, so 3.93× per token becomes 2.31× end to end.

Shorter responses. The end to end speedup follows the output length: 2.31× at 512 tokens, 1.75× at 256, 1.39× at 128 and 1.23× at 64. There are two causes. The prefill is a fixed cost, and acceptance is lower at the start of a response: M is about 3.4 over the first 64 to 128 tokens against 5.5 over 512.

Varied instructions. A different long instruction, "Analyze the video, focusing on its cinematography and narrative structure.", gives practically the same acceptance and speed as the prompt of the paper: 5.47 and 3.92× against 5.49 and 3.93×. The short instruction, "Describe the video in two sentences.", behaves as its output length predicts.

A baseline run under matched conditions. We ran SpecVLM with its public code on the same GPU, with the same prompt, 128 frames and a cap of 512 tokens. We ran it with two drafters. The first is the setting its paper uses for a 7B target: the 7B model drafts for itself, with 90% of the video tokens pruned from the cache of the draft. The second is a smaller drafter, LLaVA OneVision 0.5B, with the same pruning. The speedups of SpecVLM are measured against the autoregressive loop of its own code, timed in the same process.

Method, with the LLaVA OneVision 7B targetTokens per verificationDecodeEnd to end
SpecVLM, public code, 0.5B drafter4.480.51×0.54×
SpecVLM, public code, 7B self drafting5.000.47×0.49×
The drafter of the paper, chain5.493.93×2.31×

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.