- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents FlashSampling, a fused sampling method based on Gumbel-Max that integrates categorical sampling into the LM-head matmul. It computes logits tile by tile without materializing the full tensor in HBM, reducing memory traffic and separate sampling kernel overhead. In multi-GPU settings, FlashSampling replaces the standard all-gather of logits with streaming writes to overlap GPU communication. The authors also integrate FlashSampling into vLLM and demonstrate up to 10% - 3% reduction in time per output token for various model with exact sampling results.
Critical strengths
-
The proposed method is simple and well motivated. By using the Gumbel-Max trick and tile-wise argmax, FlashSampling can fuse categorical sampling into the LM-head matmul without materializing the full logits tensor in HBM. The mathematical reasoning is clear and easy to follow.
-
The implementation and experimental results are good. The authors implement Triton kernels and integrate FlashSampling into vLLM. All benchmarks are evaluated on modern GPU architectures like H100, H200, B200, and B300, and demonstrates actual speedups in both kernel-level benchmarks and end-to-end LLM serving.
Critical weaknesses
-
The current implementation only supports unrestricted multinomial sampling, while commonly used decoding strategies such as top-k and top-p sampling are not supported as listed as future work. I’m not fully convinced how this method currently could be generalized and actually benefit the practical LLM serving especially for some reasoning models.
-
While the authors evaluate multi-GPU performance up to TP8 in microbenchmarks, the actual e2e vLLM experiments only cover TP1 and TP2. It would be helpful to see how the performance will change in real TP4 serving workloads, especially considering the proposed multi-GPU communication optimization.
Questions for authors
-
One is somehow mentioned in the weakness. Many model’s default setting does not incorporate unrestricted multinomial sampling. I am curious how commonly this sampling strategy is used in actual LLM serving and could the authors provide some discussion on the practical relevance of this setting?
-
The contribution 2 mentioned "while grouped, online, and distributed variants are exact in distribution by hierarchical factorization through group log-masses." However I think main text does not mention this variant and Algorithm 1 looks like it does not include the group log-masses. Have these variants actually been implemented and evaluated, or are they mainly extensions?
-
In Figure 5, the TPOT improvement for Qwen3-1.7B drops significantly at batch size 64, from 10.2% at batch size 32 to only 0.4%, while the other models do not show a similar drop. Could the authors explain this behavior?
3 comments
Thanks for your response, my concerns have been addressed. I will maintain my score.
Thanks for your prompt response!
Thank you for the review and for the three specific questions.
W1 and Q1 (top-k, top-p, and how common unrestricted sampling is). We agree that truncated sampling is the common default. The models in our own evaluation ship with it: Qwen3-1.7B's default is T=0.6, top-k=20, top-p=0.95. The general response describes the fused top-k-then-top-p path and its measurements: with k=20 and p=0.95 it takes 24.7% to 36.0% less kernel time than FI1 at B=1 to 256 on one H200.
Temperature-only sampling is still a real setting. vLLM's own request defaults are temperature=1.0, top_p=1.0, and top-k disabled, and the OpenAI chat completions API has no top-k parameter. vLLM does apply a model's
generation_config.jsonwhen the model ships one, which is why Qwen3 runs with truncation unless the request overrides it. Temperature-only sampling is also what one uses when samples must follow the model's own distribution.We also have one data point on a reasoning model. In the GSM8K check of Section 4.6, the vLLM baseline ran Qwen3-1.7B in thinking mode with its default truncation (T=0.6, top-k=20, top-p=0.95), while FlashSampling ran temperature-only at T=0.6. Accuracy was 89.6% against 89.4% (paired bootstrap, p=0.776). The vLLM baseline run temperature-only scored 88.4% (p=0.080 against the truncated baseline, p=0.181 against FlashSampling). On this task, dropping truncation did not change accuracy measurably. We will state these sampling parameters in Section 4.6 and add the temperature-only baseline, which is the matched comparison.
W2 (end-to-end results beyond TP=2). The paper's end-to-end runs use TP=1 and TP=2. At the kernel level the advantage over FI2 grows from 1.3 to 1.6x at TP=2 to 1.6 to 2.3x at TP=4 and 1.9 to 3.0x at TP=8 (Table 6).
Q2 (grouped, online, and distributed variants). The kernel we benchmark is Algorithm 1 throughout, on one GPU and under tensor parallelism. Ranks exchange each tile's perturbed maximum and its index through P2P writes, and Stage 2 takes the argmax over all tiles of all ranks. This is exact pathwise by Lemma D.5 and uses no log-masses. Algorithms I.2 to I.4, which use group log-masses, are extensions and are not part of the reported experiments. The online variant (I.3) targets memory-constrained settings (Appendix D.1), and the log-masses also give log Z when an application needs it (Appendix L). Lemma D.1 connects the two forms: a group's perturbed maximum is distributed as Gumbel(L_k, 1), so exchanging maxima is the log-mass scheme with the Gumbel draws reused. You are right that contribution 2 and the first paragraph of Section 4.3 describe the multi-GPU path in log-mass terms ("a local sample and a local log-mass"), which Algorithm 1 does not use. We will rewrite both to say that the implemented path exchanges perturbed maxima, and we will label I.2 to I.4 as extensions.
Q3 (Qwen3-1.7B at B=64). We do not trust this cell. Three things in the data point to a measurement problem and not to the kernel. Both systems land on almost the same TPOT at B=64, 2.14 ms and 2.13 ms (Table 7). The run-to-run deviation of the cell is 3.6%, against at most 0.8% in every other cell of Table 8. And FlashSampling's TPOT rises by 0.42 ms between B=32 and B=64, after rising by 0.03 ms between B=1 and B=32, while the fused kernel's own latency changes by about 10 µs from B=16 to B=64 (Table 6). We saw the same signature in a later sweep of this model on B200 with FI2 as the sampler: TPOT at concurrency 64 was 3.190 ms in a concurrent sweep and 2.109 ms when that sweep was rerun in isolation. Qwen3-1.7B has the shortest decode step of the four models, so it is the most exposed to host-side interference. We reran Qwen3-1.7B at B=64 on one H200, both systems, five runs each: 2.958 ms for vLLM and 2.739 ms for FlashSampling, a reduction of 7.4% with a standard deviation of 1.2 points, and between 4.8% and 7.6% in each paired run. With top-k=20 and top-p=0.95 on the same model and GPU the reduction at B=64 is 13.4% (general response). We will mark the cell as unreliable in the paper.