- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces FlexTile to adapt GEMM tiling and FlexLoad to share KV-cache reads across requests without changing their numerical computation. FlexBI improves the efficiency of batch-invariant LLM serving while preserving numerical consistency. Experiments in vLLM and SGLang show higher throughput and lower P99 time to first token than the corresponding BI baselines.
Critical strengths
- The paper identifies execution choices that can remain flexible under BI constraints and develops concrete optimizations for both GEMM and decode attention.
- The performance gains are demonstrated in both vLLM and SGLang, making the approach relevant to existing serving systems.
- The detailed ablations help explain where the speedups come from and bitwise checks support numerical consistency in the tested settings.
Critical weaknesses
- The paper explains the numerical contract and validation procedure, but provides limited detail on how PRESERVESREDUCER is implemented. Listing the actual admission rules and showing how a candidate tile configuration passes these checks would make FlexTile easier to reproduce.
- The operator-level ablations and controlled experiments support both components, but their contributions to the end-to-end gains are less clear. The paper could better distinguish how much FlexTile and FlexLoad each contribute to throughput and tail latency, and how they interact when used together.
Questions for authors
- Section 3.2 explains the fixed reducer and its numerical constraints. Could the authors provide the concrete checks or kernel-template restrictions used to implement PRESERVESREDUCER in Algorithm 1? A worked example of an admitted tile change would help clarify the implementation.
- Figures 4 and 5 provide useful evidence for both components. Could the authors quantify their individual contributions to the online throughput and P99 TTFT gains in Figure 3, for example by disabling one component at a time at a representative operating point?
Answers for Reviewer vqoM
Q1. How is PRESERVESREDUCER implemented?
Thank you for highlighting this reproducibility gap. FlexTile recovers adaptive output tiling by encoding the numerical reducer in a restricted kernel template and exposing physical scheduling choices separately. We have expanded Appendix B.3 with the kernel restrictions and an admitted tile transition (Figure 7).
Concretely, the loader requires a supported numerical contract and accepts exactly five schedule fields:
BLOCK_SIZE_M,BLOCK_SIZE_N,GROUP_SIZE_M,num_warps, andnum_stages. All must be positive integers, and the M/N tile dimensions must be powers of two. Extra fields, including a schedule-levelBLOCK_SIZE_K, are rejected. Thus, a candidate cannot override reduction parameters when adapting its output tile.The template takes BK from the contract (64 in the evaluated configuration), initializes an FP32 accumulator, and traverses K blocks in ascending order using
accumulator = tl.dot(a, b, accumulator). Each update consumes the preceding accumulator state. Bias is applied in FP32 before a single output cast, and split-K is fixed to one. These restrictions retain the per-output reduction chain while allowing independent outputs to be remapped.The updated artifact includes the loader, kernel template, dispatch catalog, and reproducible checks through
run-bi.sh --suite gemm. The checks enumerate the template's actual tile mapping forCOVERSONCEand compare output bits across schedules, batch positions, and dispatch boundaries.Q2. What are the individual end-to-end contributions?
Following your suggestion, the new RQ1 ablation quantifies each component’s contribution to online serving. The table presents representative results from Figure 4 for Qwen3-14B with vLLM. FlexTile only uses native attention, and FlexLoad only uses native GEMM tiling. The TPS values average three trace cohorts. Both components improve throughput and reduce P99 TTFT, and full FlexBI achieves the largest gains on both workloads. Together, they reduce execution cost while affecting subsequent batching and completion order, and their joint gain need not equal the sum of their individual gains.