- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VFlash-FM improves one-pass drafting through additional training. It learns from partially filled drafter states using step conditioning, a discrete flow-path loss, direct supervision, and CAT weighting, while retaining one draft forward pass per inference round. Experiments evaluate long-video description generation with chain and tree verification on LLaVA-OneVision and Qwen2.5-VL.
Critical strengths
-
The paper is clearly written, and the training method and inference procedure are easy to follow. It clearly explains that the additional computation is spent during training, while inference still requires only one draft forward pass per round.
-
The comparison with the non-flow baseline shows consistent improvements across the tested video datasets and verification configurations. The method accepts more draft tokens and achieves faster decoding, providing useful evidence that the overall training recipe works.
Critical weaknesses
-
The method introduces partial-state training, step conditioning, and a new loss together, so it remains unclear which changes account for the gains. It would help to compare against partial-state training without the path loss, as well as a simpler baseline that only adjusts the supervision weights.
-
The method adds extra forward passes, while the baseline is mainly matched by training epochs. Since the approach trades additional training computation for faster inference, a comparison under the same GPU-hour budget would help establish whether it still outperforms a baseline trained for longer.
-
The evaluation covers two 7B target models, but does not include other model sizes or inference at larger batch sizes. It also focuses on long-form descriptions and excludes video prefill from timing. Further results would help clarify how the gains extend to other tasks, concurrent workloads, and end-to-end request latency.
Questions for authors
Please see the weaknesses.
3 comments
The method adds extra forward passes, while the baseline is mainly matched by training epochs. Since the approach trades additional training computation for faster inference, a comparison under the same GPU-hour budget would help establish whether it still outperforms a baseline trained for longer.
Thank you for this comment. You are right that the extra forward passes add training cost, and we measured it. The two runs below share the same deduplicated list of 176.5k clips, the same drafter with 4 layers and 5 epochs. They differ in whether training uses the extra forward passes.
| Extra forward passes in training | Training GPU hours | Chain: M / decode | Tree 60: M / decode | |
|---|---|---|---|---|
| A | none, all mask states only | 1,756 | 5.90 / 4.21× | 7.72 / 4.10× |
| B | rollouts to partial states | 2,038 | 5.85 / 4.17× | 7.55 / 4.01× |
Cost of the extra forward passes. For the same number of epochs the rollouts add about 16% GPU hours. It is less than the number of extra passes suggests, because a training step is dominated by the frozen target forward and by video decoding. The GPU hours count the whole job time, including restarts and the evaluations during training.
Same GPU hour budget. We did not have the time in the rebuttal period to train a run for longer, so we cannot show two runs with exactly the same budget. But rows A and B already answer the question in one direction. The run without the extra passes uses 16% fewer GPU hours and accepts at least as many tokens at one drafting pass. Training it for longer, up to the budget of row B, is not expected to make it worse. So at one drafting pass the extra forward passes do not pay for their cost.
What the extra cost buys. It buys the ability to use a second drafting pass, as the table in our first answer to you shows: with the rollouts a second pass adds 0.59 M, and without them the drafter collapses.
The evaluation covers two 7B target models, but does not include other model sizes or inference at larger batch sizes. It also focuses on long-form descriptions and excludes video prefill from timing. Further results would help clarify how the gains extend to other tasks, concurrent workloads, and end-to-end request latency.
Thank you for this comment. We added end to end measurements that include the target video prefill, and measurements on shorter outputs and on other instructions. Unless stated otherwise, the numbers are with chain verification on one GPU, on the 750 clips of the paper protocol, with the autoregressive baseline timed in the same process. End to end includes the target video prefill.
| Target | Prefill | M | Decode | End to end |
|---|---|---|---|---|
| LLaVA OneVision 7B | 1.42 s | 5.49 | 3.93× | 2.31× |
| Qwen2.5 VL 7B | 2.41 s | 3.17 | 2.34× | 1.57× |
Video prefill and end to end latency. The end to end speedup is lower than the decode speedup, and the whole difference comes from the prefill of the target, which cannot be avoided: the target has to read the video once, and that takes the same time with and without our drafter. The drafter adds no prefill of its own. On the LLaVA OneVision 7B target the time to the first token is 1.42 s in both cases. At 512 tokens the prefill is about a quarter of the autoregressive time and more than half of ours, so 3.93× per token becomes 2.31× end to end.
Other tasks. We tested shorter outputs and other instructions on the LLaVA OneVision 7B target.
| Setting | Mean output tokens | M | Decode | End to end |
|---|---|---|---|---|
| Paper protocol, detailed description, cap 512 | 411 | 5.49 | 3.93× | 2.31× |
| Cap 256 | 247 | 4.24 | 2.98× | 1.75× |
| Cap 128 | 128 | 3.41 | 2.39× | 1.39× |
| Cap 64 | 64 | 3.42 | 2.35× | 1.23× |
| A different long instruction, cap 512 | 405 | 5.47 | 3.92× | 2.27× |
| A short instruction | 60 | 2.73 | 1.96× | 1.19× |
Shorter outputs: the end to end speedup follows the output length, 2.31× at 512 tokens, 1.75× at 256, 1.39× at 128 and 1.23× at 64. There are two causes. The prefill is a fixed cost, and acceptance is lower at the start of a response: M is about 3.4 over the first 64 to 128 tokens against 5.5 over 512.
Other instructions: a different long instruction, "Analyze the video, focusing on its cinematography and narrative structure.", gives practically the same acceptance and speed as the prompt of the paper, 5.47 and 3.92× against 5.49 and 3.93×. A short instruction, "Describe the video in two sentences.", gives 60 output tokens on average and behaves as that length predicts, 1.19× end to end.
So the gain depends on the length of the answer and not on the instruction. For tasks whose answers are only a few tokens long there is almost nothing left to accelerate.
Other model sizes. In prior work such as ParallelVLM, the larger targets are 32B and 72B models, and these sizes require a very large amount of compute for our method: every target needs its own drafter, and the frozen target runs in every training step. We started these experiments on a 32B target during the rebuttal period, but the time was not enough to complete them. We will report the results in the final revision.
Larger batch sizes. The results above are for one request at a time. For this answer we implemented a first batched version of our decoding and compared it with batched autoregressive decoding on the same GPU. All requests of a batch advance together by the shortest accepted prefix among them, and a request that accepted more keeps its extra tokens for its next block. Every committed token is still verified by the target. 160 clips, 32 from each benchmark, decoding only.
| Batch size | Tokens per cycle | Speedup over batched autoregressive decoding |
|---|---|---|
| 1 | 5.52 | 3.93× |
| 2 | 4.74 | 3.14× |
| 4 | 3.99 | 1.94× |
The gain holds at batch sizes 2 and 4, becomes smaller as the batch grows, and with this version is lost at larger batch sizes. The main reason is that autoregressive decoding gets much cheaper per request in a batch, since one step takes about the same time for several requests as for one. We will add this analysis to the final revision.
Concurrent workloads. We ran one small test: 1 to 4 requests at the same time on one GPU, each in its own process, on 12 to 16 clips of the VideoDetailCaption benchmark. Each speedup is against autoregressive decoding measured under the same load.
| Requests at the same time | Decode | End to end |
|---|---|---|
| 1 | 3.07× | 1.98× |
| 2 | 3.29× | 1.90× |
| 3 | 3.47× | 2.21× |
| 4 | 3.93× | 2.14× |
The speedup stays at the same level when the GPU is shared. This is a small test, so we give it as a first indication only.
Thank you very much for your review and for your comments. We address your concerns one by one below.
Thank you for this comment. We report the comparison without the path loss that you ask for, and the closest matched run we have to a baseline that only adjusts the supervision weights. To keep them matched, both runs below were trained with the same compute setup: the same deduplicated list of 176.5k clips, the same drafter with 4 layers and 5 epochs. Both use block 16 and one drafting pass.
Partial state training without the path loss. This is row B. It trains on partial states with the direct supervision only, CE + KL, and reaches 5.85. Row A has the path loss on the all mask state and never sees partial states, and reaches 5.90. So at one drafting pass the run with the path loss is slightly ahead, 5.90 against 5.85, and partial states do not add accepted tokens.
The full recipe at the same compute. We tried to train the full recipe, with all three changes together, at the same compute as rows A and B during the rebuttal period, but the time was not enough to reach results. We therefore cannot yet say how it compares with them, and we will report it in the final revision. Rows A and B are controls, not a comparison with the full recipe.
A baseline that only adjusts the supervision weights. On all mask states the path loss equals 10·CE at every logged step, because its mask term is numerically zero. Row A is therefore training with adjusted weights, 11·CE + KL, plus the step conditioning. It is the closest matched run to the baseline you describe, not exactly it, because of the step conditioning.
Step conditioning. Both rows contain the step conditioning module. In row A its input never changes, t is always 0 and h is always 1, because that run only sees all mask states. In row B it varies with the state. So these two runs cannot isolate the effect of the module.
Which change accounts for what. The clear effect appears with a second drafting pass. Chain arm, 750 clips:
Partial state training is what makes a second pass possible, and it does not need the path loss for that: the drafter of row B gains 0.59 M from a second pass, while the drafter of row A collapses. On a 7B target the second pass costs more time than it saves, 3.85× against 4.17× at one pass, so this is not yet a wall clock gain. We are now working on making the second pass faster, so that it becomes a wall clock gain as well.