- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
PercepDraft accelerates generative visual perception by exploiting predictability along autoregressive and denoising trajectories while keeping the backbone frozen. For structured text outputs, it uses draft heads and backbone prefix verification. For dense outputs, a lightweight spatial network predicts future velocity offsets and per-horizon errors. The paper evaluates detection, panoptic segmentation, depth, and surface normals across SenseNova-Vision and additional task-compatible backbones. The results show that the proposed methods substantially reduce inference time compared with the original decoding procedures, while largely maintaining task performance.
Critical strengths
-
Substantial practical acceleration, particularly for dense perception. On SenseNova-Vision, depth latency decreases from 8.66 to 1.38 seconds and normal-estimation latency from 8.70 to 1.61 seconds, with comparable aggregate accuracy. Panoptic latency decreases from 12.1 to 5.91 seconds while PQ changes from 51.4 to 51.3. These measured wall-clock improvements are meaningful, and validation on additional backbones suggests that the approach is not restricted to one output interface.
-
A lightweight and practically motivated adaptation strategy. The method preserves backbone weights and introduces small auxiliary predictors rather than retraining the full perception model. The dense design addresses an important task-dependent trade-off: geometry tolerates aggressive approximation, whereas panoptic segmentation is more sensitive. The error-based horizon selection and endpoint check provide a plausible mechanism for adapting computation to this difference. The paper also reports failure cases and explicitly acknowledges that endpoint verification does not certify the entire trajectory.
Critical weaknesses
The incremental contribution of the proposed components is not sufficiently isolated from simpler alternatives. Constant-velocity drafting already achieves 3.84 depth AbsRel at 16.2 calls, versus 3.85 at 13.7 calls for the complete separately trained variant (Table 7), indicating that much of the headline depth acceleration is available without learned velocity prediction. Likewise, restricted-vocabulary parallel token heads reach 3.46 tokens/call versus 3.54 for PercepDraft at identical F1 (Table 8). These comparisons do not negate the method’s usefulness, but they leave its additional complexity and methodological contribution insufficiently justified.
Questions for authors
- Can you report matched-quality wall-clock comparisons using batched FlowCast verification? Please distinguish results from faithful implementations from those of the simplified adaptations
- Can you provide quality–latency comparisons for the velocity-source/horizon combinations at identical tolerances and comparable training settings? For the token branch, what wall-clock benefit remains over restricted-vocabulary parallel heads under the same verifier? How much dense acceleration remains relative to tuned reduced-step schedules, including the eight-step Marigold baseline already reported?
5 comments
Response to Reviewer e7mr (2/3)
[Q1] Wall-clock comparison with batched FlowCast, distinguishing faithful implementations from adaptations.
[A2] Thank you for the suggestion. We implemented the batched verification of FlowCast following the paper, measured its quality on the evaluation images of Table 1 and its latency against the reference on the same GPU, and the revision states for every baseline whether it follows the original implementation or is adapted to this backbone.
(1) FlowCast with batched verification as published. This version follows Algorithm 1 of FlowCast. It drafts all remaining states, verifies them in batched backbone calls, accepts the prefix under the velocity MSE threshold (including 0.07, the value used for BAGEL editing in the FlowCast paper), and corrects one step with the verified velocity after a rejection. At batch size one, the batched call reproduces the per-state call element-wise. Speedups are wall-clock ratios to the reference.
| Setting | NYUv2 depth ↓ | NYUv2 normals ↓ | Calls (depth / normals) ↓ | Speedup (depth / normals) ↑ |
|---|---|---|---|---|
| Reference (49 steps) | 4.05 | 13.88 | 98 | 1.0× |
| FlowCast, serial implementation in Table 1 | 3.92 | 13.85 | 100 / 102 | 1.0× |
| FlowCast, batched as published, ε=0.02 | 3.88 | – | 413 / – | 0.27× / – |
| FlowCast, batched as published, ε=0.04 | 4.11 | 13.96 | 320 / 395 | 0.34× / 0.28× |
| FlowCast, batched as published, ε=0.07 | 4.36 | 14.32 | 280 / 339 | 0.39× / 0.33× |
| FlowCast, batched as published, ε=0.12 | 4.63 | 14.87 | 251 / 296 | 0.43× / 0.37× |
| PercepDraft (Table 1) | 3.87 | 13.77 | 13.4 / 15.8 | 6.3× / 5.4× |
| Panoptic (COCO) | PQ ↑ | Calls ↓ | Speedup ↑ |
|---|---|---|---|
| Reference (49 steps) | 51.4 | 98 | 1.0× |
| FlowCast, serial implementation in Table 1 | 48.1 | 107.7 | 1.0× |
| FlowCast, batched as published, ε=0.02 | 51.7 | 1021 | 0.13× |
| FlowCast, batched as published, ε=0.07 | 51.1 | 723 | 0.18× |
| FlowCast, batched as published, ε=0.12 | 51.1 | 636 | 0.20× |
| PercepDraft (Table 1) | 51.3 | 38.7 | 2.0× |
Batched FlowCast is slower than the reference at every threshold, at 0.27–0.43× on geometry and 0.13–0.20× on panoptic segmentation, and at ε=0.07 it is also slower on KITTI and ScanNet (0.38× and 0.40×). It evaluates 2.6 to 10.4 times as many states per image as the reference, and a single SenseNova-Vision backbone call already runs near the arithmetic peak of the GPU, so packing K states into one call costs close to K separate calls; batching reduces the number of sequential calls but not the total computation. The serial implementation in Table 1 is therefore the more favorable of the two for FlowCast. This also shows that, on a backbone whose single call saturates the device, acceleration has to come from evaluating fewer states. PercepDraft verifies each segment once at its landing point and sets the segment length from the predicted errors, which is where its speedup comes from.
(2) Implementation of each baseline. Appendix C.2.1 of the revision states the implementation of each baseline as follows.
| Method | Implementation |
|---|---|
| FlowCast | Table 1 uses a serial implementation (drafted states verified one at a time, stopping at the first failure, relative ℓ2 residual on the velocity); this response adds the batched verification as published (all remaining states verified in a batch, velocity MSE) |
| DeepCache, TeaCache, TaylorSeer | Reimplemented on the MoT backbone of SenseNova-Vision with their published caching and reuse rules (the original implementations target U-Net or DiT backbones) |
| FastV, AgilePruner | Run with their official configurations |
| Medusa | Trained with the same data, number of steps and number of heads as PercepDraft's token drafter, predicting over the full vocabulary |
We thank the reviewer again for the suggestion; this comparison makes our evaluation more complete.
Revision. We add Table 16 (Appendix D) with the quality and latency of batched FlowCast, and Appendix C.2.1 states the implementation of each baseline.
Response to Reviewer e7mr (3/3)
[Q2] Velocity-source and horizon combinations at identical tolerances, the token branch versus restricted-vocabulary parallel heads, and the speedup remaining over tuned reduced-step schedules including the eight-step Marigold baseline.
[A3] Thank you for the questions, which we answer in turn.
(1) Velocity source and horizon rule at identical tolerances. The table in W1 (1) reports the quality, calls and latency of the four combinations at the same per-task tolerance and with the same training setting. PercepDraft (trajectory drafter with trusted horizon) reaches the best or near-best quality with the fewest calls on all three tasks. Against the simplest alternative, constant velocity with a fixed horizon, it uses one third fewer calls on depth, raising the speedup from 4.5× to 6.3×, and about half and 70% of the calls on normals and panoptic segmentation. The trusted horizon provides most of the reduction in calls, and the trajectory drafter's velocities further improve quality.
(2) Wall-clock gain of the token branch over restricted-vocabulary parallel heads. With the same verifier and runtime, PercepDraft's token branch takes 1.25 / 6.65 / 23.5 s per image on LVIS / VisDrone / Dense200, the restricted-vocabulary parallel heads 1.29 / 6.82 / 23.4 s, and the full-vocabulary Medusa heads 1.50 / 8.33 / 28.7 s. The restricted vocabulary and the chained heads are both part of PercepDraft's token drafter and together are 1.20–1.25× faster than the existing Medusa heads. On top of the restricted vocabulary, chaining raises the committed tokens per call from 3.46 to 3.54, giving a further 3% and 2.5% on LVIS and VisDrone, and is on par on Dense200.
(3) Remaining speedup over tuned reduced-step schedules. On the evaluation images of Table 1, we compare CFG at every step, CFG only for t>0.9, and no CFG between 6 and 49 steps, and also apply PercepDraft to the unguided samplers without a trajectory drafter, using constant-velocity drafts and landing-point verification only. CFG denotes classifier-free guidance, which the released SenseNova-Vision sampler applies at every step. Calls are backbone evaluations per image (two per step with CFG), and speedups are wall-clock ratios to the reference on the same GPU.
| Setting | NYUv2 depth ↓ | KITTI ↓ | NYUv2 normals ↓ | ScanNet ↓ | Calls (depth / normals) ↓ | Speedup (depth / normals) ↑ |
|---|---|---|---|---|---|---|
| Reference (CFG, 49 steps) | 4.05 | 6.23 | 13.88 | 12.69 | 98 | 1.0× |
| CFG, 24 steps | 4.16 | 7.62 | 15.63 | 13.68 | 48 | 2.0× |
| CFG, 12 steps | 6.45 | 20.16 | 22.94 | 21.07 | 24 | 3.7× / 3.8× |
| CFG for t>0.9, 49 steps | 4.06 | 6.30 | 13.85 | 12.67 | 65 | 1.5× / 1.4× |
| No CFG, 12 steps | 4.09 | 5.99 | 13.93 | 12.67 | 12 | 6.5× / 6.7× |
| No CFG, 6 steps | 3.94 | 5.83 | 13.79 | 12.55 | 6 | 10.7× / 11.0× |
| PercepDraft (Table 1) | 3.87 | 5.60 | 13.77 | 12.62 | 13.4 / 15.8 | 6.3× / 5.4× |
| No CFG, 12 steps + PercepDraft | 3.85 | 5.59 | 13.61 | 12.39 | 4 | 14.1× / 14.5× |
| No CFG, 6 steps + PercepDraft | 3.75 | 5.59 | 13.63 | 12.38 | 3 | 16.5× / 16.8× |
On panoptic segmentation, the reference (CFG, 49 steps) reaches 51.4 PQ and PercepDraft 51.3 PQ at 2.0×. The reduced-step schedules give the following PQ (speedup).
| Panoptic PQ ↑ (speedup ↑) | 20 steps | 16 steps | 8 steps |
|---|---|---|---|
| CFG | 48.1 (1.8×) | 46.0 (2.1×) | 36.9 (2.8×) |
| CFG for t>0.9 | 47.8 (2.2×) | 46.5 (2.5×) | 37.8 (3.1×) |
| No CFG | 49.6 (2.5×) | 49.6 (2.8×) | 47.4 (3.4×) |
| Marigold v1.1 normals | NYUv2 ↓ | ScanNet ↓ | Calls ↓ | Speedup ↑ |
|---|---|---|---|---|
| Reference (50 steps) | 16.98 | 15.37 | 50 | 1.0× |
| Uniform 8 steps | 16.53 | 15.07 | 8 | 4.6× |
| PercepDraft | 16.54 | 15.04 | 5.1 | 5.8× |
| Marigold v1.0 depth | NYUv2 ↓ | KITTI ↓ | Calls ↓ | Speedup ↑ |
|---|---|---|---|---|
| Reference (50-step DDIM) | 6.02 | 12.97 | 50 | 1.0× |
| 10-step DDIM | 6.15 | – | 10 | 4.0× |
| PercepDraft | 5.91 | 12.90 | 5.4 | 5.6× |
With CFG at every step, as released, the fastest reduced-step schedule that matches the reference is CFG for t>0.9 over 49 steps at 1.5×/1.4×, while PercepDraft reaches 6.3×/5.4× with lower error. On panoptic segmentation no reduced-step schedule reaches the reference quality. Without CFG few steps work on geometry but lose at least 1.8 PQ on panoptic segmentation; PercepDraft is orthogonal and reduces the 12- and 6-step unguided samplers to 4 and 3 calls with lower error (up to 16.8×). On Marigold, PercepDraft is 1.26× faster than the tuned 8-step schedule at comparable quality for v1.1 normals, and both faster and more accurate than 10-step DDIM for v1.0 depth. In neither case does PercepDraft require choosing a step count for the model or the task.
Revision. As in W1; we add Table 15 (Appendix D) with the reduced-step schedules, and the Marigold comparison is in Appendix B.1.
Response to Reviewer e7mr (1/3)
We sincerely thank the reviewer for the detailed review and for recognizing the substantial wall-clock acceleration on dense perception, the lightweight adaptation that keeps the backbone frozen, and the error-based horizon selection with endpoint verification that adapts computation to the different tolerance of geometry and panoptic segmentation. All changes are marked in blue in the revision.
[A1] Thank you for this careful observation. We reran the comparison under one unified setting, so that all rows now share the same tolerance and horizon rule (in Appendix C.2.2 the rows with and without the trusted horizon used different tolerances).
(1) The 2×2 comparison under a unified setting. All four combinations use the per-task tolerances of Table 1 (0.4 for geometry, 0.2 for panoptic segmentation) and the same images. Both fixed-horizon combinations draft 8 steps per segment, both trajectory-drafter combinations use the same trajectory drafter, and the trusted horizon of the constant-velocity row comes from an error head trained with the same data and number of steps. Calls are backbone evaluations per image (98 for the reference), and speedups are wall-clock ratios to the reference on the same GPU.
The speedups (depth / normals / panoptic) of the four rows are 4.5× / 3.2× / 1.5×, 4.6× / 3.3× / 1.6×, 6.2× / 5.4× / 1.9× and 6.3× / 5.4× / 2.0×.
The trusted horizon provides most of the reduction in calls (with the trajectory drafter, from 19.6/29.6/51.8 to 13.4/15.8/38.7 at essentially unchanged quality), and the trajectory drafter's velocities further lower the error and the calls (depth AbsRel 3.95 to 3.87; panoptic 50.9 to 51.3 PQ with 41.7 to 38.7 calls). Constant velocity with a fixed horizon, the alternative you mention, needs clearly more calls at the same tolerance (20.0/30.0/55.0 vs. 13.4/15.8/38.7), while PercepDraft also has lower depth error (3.87 vs. 3.93) and a higher speedup (6.3× vs. 4.5×). These unified results replace the corresponding rows of Table 7 and Table 3, with the tolerance and horizon rule stated.
(2) Landing-point verification makes even simple drafts effective. The constant-velocity row already benefits from PercepDraft's landing-point verification. The depth trajectory is nearly straight in its middle part (2.8% to 3.9% velocity change between adjacent steps, Table 11), so most constant-velocity drafts pass verification; the same training-free drafts also reduce the 16-step Pixel-Perfect Depth sampler from 16 to 5 calls with lower error. On top of this, the trusted horizon decides how far to advance from the predicted error of each segment, so one tolerance per task suffices, and the trajectory drafter's velocities make the landing point more accurate. Section 4.6 of the revision states the role of each component accordingly.
(3) Token branch. The restricted record vocabulary is part of PercepDraft's token drafter (Section 3.4), and the restricted-vocabulary parallel heads in Table 8 are an ablation of our own design without chained prediction heads, not an existing alternative. With the same verifier and runtime, the per-image latency (s) of the three token-drafting heads is as follows.
PercepDraft's token branch is 1.20–1.25× faster than the existing Medusa heads. Restricting each head to the 1718 token ids used by perception records accounts for most of this gain; chaining the heads, so that each head conditions on the predictions of the previous ones, raises the committed tokens per call from 3.46 to 3.54 and gives a further 3% and 2.5% on LVIS and VisDrone. The two are tied on Dense200, where the accepted length is limited by the last coordinate digit, which both kinds of heads find equally hard to predict.
Revision. The four velocity-source and horizon-rule rows of Table 3 (main text) and Table 7 (appendix) are replaced by the unified results with the tolerance and horizon rule stated, and the discussion in Section 4.6 is updated; Table 5 adds the latency of the restricted-vocabulary parallel heads.