- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RIVET, a variable-granularity execution framework for token-tree verification in speculative decoding. It combines structure-aware mask specialization, adaptive query tiling, and resource-aware KV partition scheduling to reduce redundant computation and improve GPU utilization. The method preserves exact tree-attention semantics through online-softmax merging. Experiments on Qwen3-8B and Llama-3.1-8B demonstrate improvements in both attention-kernel latency and end-to-end decoding performance.
Critical strengths
-
The paper identifies concrete inefficiencies in existing tree-attention implementations, including query padding, KV scheduling imbalance, and redundant mask checks. The proposed framework addresses these issues in a coherent manner while preserving exact attention semantics.
-
The implementation integrates with FlashInfer and SGLang, and the experiments demonstrate meaningful improvements over several representative attention backends. The evaluation includes both kernel-level and end-to-end measurements, with component ablations supporting the effectiveness of the proposed optimizations.
Critical weaknesses
-
The main components largely adapt established techniques, including mask specialization, adaptive query tiling, and KV partition scheduling. More importantly, Appendix C shows that the topology-compiled region path is 4.2–4.9% slower than a simpler prefix–suffix hybrid on branching trees, while providing only a marginal improvement on chain structures. This raises questions about when the more elaborate heavy-path decomposition is actually beneficial and whether it is necessary for the reported gains.
-
Although the kernel benchmarks cover batch sizes up to 32 and KV lengths up to 32K, the end-to-end experiments are restricted to batch sizes 8/16 and KV lengths 8K/16K on two 8B models using A800 GPUs. The generality of the reported improvements across longer contexts, larger batches, different hardware, and more diverse tree structures remains unclear.
-
Equation (2) introduces a profiled cost model incorporating CTA execution times, scheduling waves, and merge overhead. While the ablation in Figure 6 demonstrates the benefit of KV-split scheduling, it does not establish whether the proposed model outperforms simpler tuned fixed-split or KV-length-based policies. A more controlled comparison would help justify the additional scheduling complexity.
Questions for authors
-
Appendix C shows that topology-compiled regions are slower than the simpler prefix–suffix hybrid on branching trees. Under what conditions does heavy-path decomposition provide a clear advantage, and how much does it contribute when the other optimizations are held fixed?
-
Can the authors clarify the expected performance under larger batches, longer KV contexts, and more diverse tree topologies? How representative are the evaluated speculative decoding workloads?
-
How accurately does the scheduling cost model predict measured runtime, and how does it compare with simpler fixed-split or length-based scheduling policies under the same settings?
We thank the reviewer for their careful comments on Appendix C, larger-scale evaluation, and CTA latency; we respond to each point below.
Q1. Appendix C. Hybrid denotes our topology-compiled method and Region the native masked path. A regression is possible only for the deliberately constructed chain with nearly equal request lengths; no such case appeared in the 864 measured requests.
Q2. Larger batch and context. We extend the five-way attention-kernel comparison beyond the paper's B32/32K range to batches of 64 and 128 and KV lengths of 64K and 128K. The table reports median latency in milliseconds.
RIVET has the lowest latency in all four configurations, reducing it by 21.8–41.7% relative to the fastest baseline at each point.
Q3. CTA latency. For a fixed Q template and KV interval, isolated CTA execution time is stable across repeated runs. We profile the GPU duration of one native-causal attention kernel per invocation, with exactly one CTA, no KV split, and no merge. The table gives median / scheduling cost model predicted latency in microseconds over 150 runs after warmup:
Across these nine fixed-shape classes, scheduling cost model predicted latency is at most 0.09% above the median (coefficient of variation below 0.10%). A simpler fixed-split strategy would leave CTA workloads imbalanced, creating a long tail in execution time.