acceptodds
Under review as a conference paper at ICLR 2027

FlexBI: Regaining Execution Flexibility for Batch-Invariant LLM Inference

Abstract

Batch-invariant (BI) inference preserves bitwise-identical results for each request regardless of its surrounding batch. Existing BI implementations couple this numerical guarantee to fixed physical schedules, limiting their ability to exploit changing concurrency and prefix sharing. FlexBI regains execution flexibility by separating a fixed request-local numerical contract from adaptive physical scheduling. Flextile regains dynamic GEMM output tiling by keeping each output's reduction trace invariant across tile changes. Flexload uses partition-invariant sharing to share identical physical key-value (KV) page loads while preserving each query's logical reduction. In online serving across two models and two datasets, FlexBI delivers 25–33% higher output throughput and 16–32% lower P99 time to first token than same-engine BI in vLLM and SGLang.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
batch-invariant inference, deterministic inference, large language model inference, kernel scheduling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vqoMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces FlexTile to adapt GEMM tiling and FlexLoad to share KV-cache reads across requests without changing their numerical computation. FlexBI improves the efficiency of batch-invariant LLM serving while preserving numerical consistency. Experiments in vLLM and SGLang show higher throughput and lower P99 time to first token than the corresponding BI baselines.

Critical strengths
  • The paper identifies execution choices that can remain flexible under BI constraints and develops concrete optimizations for both GEMM and decode attention.
  • The performance gains are demonstrated in both vLLM and SGLang, making the approach relevant to existing serving systems.
  • The detailed ablations help explain where the speedups come from and bitwise checks support numerical consistency in the tested settings.
Critical weaknesses
  • The paper explains the numerical contract and validation procedure, but provides limited detail on how PRESERVESREDUCER is implemented. Listing the actual admission rules and showing how a candidate tile configuration passes these checks would make FlexTile easier to reproduce.
  • The operator-level ablations and controlled experiments support both components, but their contributions to the end-to-end gains are less clear. The paper could better distinguish how much FlexTile and FlexLoad each contribute to throughput and tail latency, and how they interact when used together.
Questions for authors
  • Section 3.2 explains the fixed reducer and its numerical constraints. Could the authors provide the concrete checks or kernel-template restrictions used to implement PRESERVESREDUCER in Algorithm 1? A worked example of an admitted tile change would help clarify the implementation.
  • Figures 4 and 5 provide useful evidence for both components. Could the authors quantify their individual contributions to the online throughput and P99 TTFT gains in Figure 3, for example by disabling one component at a time at a representative operating point?
1 comment
AuthorsOpenReview ↗

Answers for Reviewer vqoM

Q1. How is PRESERVESREDUCER implemented?

Thank you for highlighting this reproducibility gap. FlexTile recovers adaptive output tiling by encoding the numerical reducer in a restricted kernel template and exposing physical scheduling choices separately. We have expanded Appendix B.3 with the kernel restrictions and an admitted tile transition (Figure 7).

Concretely, the loader requires a supported numerical contract and accepts exactly five schedule fields: BLOCK_SIZE_M, BLOCK_SIZE_N, GROUP_SIZE_M, num_warps, and num_stages. All must be positive integers, and the M/N tile dimensions must be powers of two. Extra fields, including a schedule-level BLOCK_SIZE_K, are rejected. Thus, a candidate cannot override reduction parameters when adapting its output tile.

The template takes BK from the contract (64 in the evaluated configuration), initializes an FP32 accumulator, and traverses K blocks in ascending order using accumulator = tl.dot(a, b, accumulator). Each update consumes the preceding accumulator state. Bias is applied in FP32 before a single output cast, and split-K is fixed to one. These restrictions retain the per-output reduction chain while allowing independent outputs to be remapped.

The updated artifact includes the loader, kernel template, dispatch catalog, and reproducible checks through run-bi.sh --suite gemm. The checks enumerate the template's actual tile mapping for COVERSONCE and compare output bits across schedules, batch positions, and dispatch boundaries.

Q2. What are the individual end-to-end contributions?

Following your suggestion, the new RQ1 ablation quantifies each component’s contribution to online serving. The table presents representative results from Figure 4 for Qwen3-14B with vLLM. FlexTile only uses native attention, and FlexLoad only uses native GEMM tiling. The TPS values average three trace cohorts. Both components improve throughput and reduce P99 TTFT, and full FlexBI achieves the largest gains on both workloads. Together, they reduce execution cost while affecting subsequent batching and completion order, and their joint gain need not equal the sum of their individual gains.

ConfigurationToolAgent 1.6 req/s: TPS / P99 TTFT (s)Conversation 1.2 req/s: TPS / P99 TTFT (s)
Native BI109.15 / 156.88107.21 / 209.26
FlexTile only124.87 / 142.50122.00 / 197.41
FlexLoad only134.59 / 132.61129.57 / 184.84
Full FlexBI144.27 / 127.49142.35 / 175.93
Reviewer_tEdvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper provides a flexible batch-invariant inference method for LLM serving. The authors report that the output throughput improved by 25-33% using online serving on datasets.

Critical strengths
  1. The results of online serving seem good using two models and two datasets. But not supportive enough for this paper.
Critical weaknesses
  1. The illustration of where the benefits come from is not clear enough. By separating request-local numerical scheduling from physical scheduling, why can flexible BI have higher throughput and lower TTFT? I can only guess that the gain is from higher concurrency or prefix caching reuse, but there is no evidence showing me the relationship.
Questions for authors

By employing this method to share flexload across multiple requests, is it safe for users, or does it lead to possible data leakage? It is also important to illustrate where the benefits in throughput come from.

1 comment
AuthorsOpenReview ↗

Answers for Reviewer tEdv

Weakness. Why does separating numerical and physical scheduling improve throughput and TTFT?

Thank you for this helpful question. Batch invariance fixes each output's numerical reduction, while FlexBI adapts output tiling and shared KV delivery. These freedoms reduce operator cost without changing output bits.

FlexTile avoids both oversized and undersized output tiles. Smaller tiles reduce wasted work at small batches, while wider tiles improve reuse when sufficient work is available. Small batches benefit from reduced tile size, while the wider GEMM benefits from a 128×256 tile at M=320–512. Figure 5(a) provides the corresponding engine-level attribution: its cache-free batch sweep removes prefix reuse, and the gain narrows around B=192–256 before widening as the fixed mapping becomes poorly matched again.

GEMM K×NActive rows MAdaptive tileFixed BI (µs)Adaptive (µs)Speedup
5120×5120116×128155.5257.332.71×
5120×51203232×128140.1061.232.29×
5120×51206464×128128.9771.391.81×
5120×5120128128×128118.28118.231.00×
5120×17408320128×256493.67411.701.20×
5120×17408512128×256669.49571.851.17×

FlexLoad removes repeated KV loads while preserving each query's reduction and private state. With 32 requests sharing a 16K prefix and each retaining a 128-token private suffix, switching on shared loading gives the following complete attention-call times on A100. The benefit scales with sharing: one group of eight among 32 requests gives 1.126–1.177×, versus 2.609–4.185× for four groups of eight.

Query heads / KV heads / head dimensionSharing off (µs)Sharing on (µs)SpeedupRequests per shared taskLogical KV blocks read saved
16 / 16 / 961536.39206.367.445×3296.12%
16 / 8 / 128826.75121.346.813×1693.02%
16 / 4 / 128441.0698.974.457×886.82%
20 / 4 / 128446.59116.293.840×886.82%
32 / 4 / 64255.7896.282.657×474.42%
32 / 4 / 128454.04165.462.744×474.42%

The online component ablation shows the corresponding serving gains. On Qwen3-14B with vLLM, ToolAgent at 1.6 requests/s yields the following representative results from Figure 4. Both components independently improve throughput and tail latency. Together, they raise throughput by 32.2% and reduce P99 TTFT by 18.7%. Lower operator cost enables faster engine steps, increasing completed work and releasing resources for waiting requests near service capacity.

ConfigurationOutput TPSP99 TTFT (s)
Native BI109.15156.88
FlexTile only124.87142.50
FlexLoad only134.59132.61
Full FlexBI144.27127.49

Question. Could sharing KV loads across requests leak user data?

No. FlexLoad shares the delivery of KV pages already mapped to participating requests by the cache manager, preserving each request’s visible KV positions. Existing systems such as PagedAttention [1] and SGLang’s RadixAttention [2] manage cross-request reuse through shared cache storage. FlexLoad operates on those established mappings and retains each request’s page table and attention masks.

The sharing decision depends on physical page identity, not text similarity. The planner requires an exact match of a leaf's complete ordered page vector. Requests A and B can therefore share loads from their common prefix, while A's private suffix remains outside B's computation. Each query retains independent softmax state and output.

Cache isolation remains enforceable before FlexLoad runs. For example, vLLM's cache_salt restricts prefix reuse to requests carrying the same salt, while SGLang's radix-cache namespace separates entries with different extra_key values (vLLM documentation, SGLang v0.5.16 implementation).

We have added a “KV access boundaries” discussion to Appendix B.1 (Framework integration), detailing how FlexLoad shares permitted reads while preserving the cache manager’s reuse eligibility and per-request access scope.

References:

[1] Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023.

[2] Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. 2024.

Reviewer_WCUpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Hello, authors! Thank you for the work and for nice prepared artifact package, it was really helpful for reading the paper. Below is my understanding of the paper.

Batch-invariant (BI) inference means a request's logits and KV cache are bitwise identical no matter what other requests are in the batch. Existing BI implementations (e.g. vLLM/SGLang BI modes following the Thinking Machines recipe) get this by fixing the physical schedule (fixed tiles, no split-K, one split in attention), which loses performance under changing concurrency and prefix sharing. FlexBI separates a fixed request-local numerical contract (operands, reduction order, precision, rounding points) from the physical work map, which can change every step. FlexTile keeps the GEMM reduction fixed but chooses output tile size, warps and stages per shape from an offline catalog. FlexLoad splits each request's KV into fixed leaves with a left-to-right merge, and lets queries with identical page vectors for a leaf share one KV load while keeping separate softmax state; a GPU planner rebuilds the groups every step and is CUDA Graph compatible.

Evaluation is with Qwen3-14B and StarCoder2-15B on Mooncake ToolAgent and Conversation traces, in vLLM and SGLang, against same-engine BI, with +25-33% output throughput and 16-32% lower P99 TTFT at the highest rates, operator-level speedups, small overhead, a bitwise test with zero mismatches, and H100 TP4 results in the appendix.

The artifact has vLLM and SGLang forks, native baselines rebuilt from clean upstream, and an online runner.

Critical strengths
  • the problem is real and the idea generalizes: BI matters for reproducibility and especially for RL where sampling and training must match bitwise, and today it costs a lot of speed. separating the request-local numerical contract from the physical work map makes it clear what really has to stay fixed, and shows the rest (output tiling, KV load sharing) can be regained. combining prefix KV sharing with BI is a useful step, these were usually seen as incompatible
  • careful engineering and evaluation setup: integrated into both vLLM and SGLang with CUDA Graph support, native baselines are verifiably clean upstream), arrivals and continuations are paired across systems with recorded hashes, real Mooncake traces, and the overhead is measured separately and small
Critical weaknesses

overall the idea is useful and the artifact is well prepared, but my main concern is attribution and verification: it is not clear how much of the reported gain comes from FlexTile / FlexLoad themselves, and the BI guarantee under KV sharing is not really tested end-to-end

  • the SGLang comparison includes more than FlexTile and FlexLoad. FlexBI-SGLang has a scheduler admission change for chunked prefill that native SGLang-BI does not have, and it also replaces prefill attention with its own kernel, while the paper says prefill stays on the native BI path. with long prompts and chunked prefill this directly affects TTFT, so the SGLang TTFT gains can't be attributed only to the proposed method
  • there is no end-to-end ablation of the core idea. in the operator test sharing alone can save at most about 1.28x of KV traffic, but FlexLoad shows more than that, so part of the gain likely comes from fixed 512-token splits instead of one split, which is already the decode recipe from Thinking Machines. the code has a switch to disable sharing, but all runs keep it on, and Conversation slices have little prefix sharing but similar gains to ToolAgent. also the bitwise test in App. C uses prompts shorter than one 512-token leaf, so no sharing group with more than one query can form there, and BI under real load sharing is not tested end-to-end
Questions for authors

clarification:

  • the formal condition traceij(s)=τij\mathrm{trace}_{ij}(s) = \tau_{ij}(I mean trace_ij(s) = tau_ij) is a definition, not something that is checked: in the code admission is matching of contract fields in the JSON catalog, COVERSONCE / PRESERVESREDUCER from Alg. 1 are not implemented, and bitwise probes are not in the package. so BI across tile changes relies on Triton keeping the reduction order for every BM/BN / warps / stages choice, which is only tested empirically. could you release the probes and say how many shapes/inputs they cover?
  • how much of the SGLang gain remains with the upstream scheduler and native prefill attention?
  • how much of the FlexLoad 1.38-1.50x comes from fixed 512-token splits vs sharing alone?
  • in vLLM, is a token's attention bitwise identical whether computed in prefill or decode (like after recompute preemption)?
  • why does native vLLM-BI fail the eager test and FlexBI pass it?
  • PROTOCOL.md says the earlier online numbers are not from this package. from ypur repo md file:

Boundary to earlier results The original local campaign prepared Qwen3-14B and StarCoder2 separately with their checkpoint-specific context filters. The package freezes the StarCoder2 source rows for both models to implement the shared-trace protocol stated in the manuscript. Earlier online numbers are not results from this exact shared input package. Re-run the package and use its new output before attributing published numbers to this protocol.

I just wondering if this is your coding agent artefact? so which protocol produced Fig. 3 and the abstract numbers, and could you report results from the packaged run?

Nevertheless suggestions:

  • end-to-end ablation: FlexTile only, FlexLoad without sharing, full system
  • discuss vLLM cascade attention / FlashInfer shared-prefix kernels
3 comments
AuthorsOpenReview ↗

Answers for Rerevier WCUp (1/3)

We sincerely thank you for recognizing the value of our work and for your detailed review. Your comments have helped us refine the component ablations and correct several descriptions. The answer may be rather lengthy as we wish to provide a quantitative clarification.

W1 / Q2. How do prefill and scheduler changes affect the SGLang comparison?

Thanks for raising this question. CanonicalAttention is part of FlexBI’s method, and we have added an ablation to quantify its contribution. We also ablate the scheduler adjustment separately. Removing it from FlexBI changes P99 TTFT by less than 1% at each tested point.

We have clarified this phase-specific integration in Section 3.4 and Appendix B.1. Both phases use the same numerical contract, while cross-request KV-load sharing is enabled only during decoding to reduce repeated KV reads in memory-bound phase.

The ablation below compares full FlexBI with native prefill and CanonicalPrefill. Each cell reports TPS / P99 TTFT (s), with percentages relative to Native BI. SGLang's native BI prefill traverses the full KV length for each query tile, introducing redundant work for causal attention. CanonicalPrefill avoids this work, and increases TPS by 5.90% and reduces P99 TTFT by 10.20% over native prefill. vLLM's native BI path already avoids this full-length causal traversal. Enabling CanonicalPrefill therefore yields only 0.08% higher TPS, with 2.12% higher P99 TTFT on average than FlexBI with native prefill. Most of the vLLM serving gains come from shared KV loading and dynamic GEMM tiling.

SGLang

ModelWorkload, req/sNative BIFlexBI with native prefillFlexBI with CanonicalPrefill
Qwen3-14BToolAgent, 1.6118.23 / 159.54140.43 (+18.78%) / 123.62 (−22.51%)147.88 (+25.08%) / 115.61 (−27.54%)
Qwen3-14BConversation, 1.2113.78 / 216.29136.12 (+19.63%) / 182.78 (−15.49%)143.34 (+25.98%) / 169.50 (−21.63%)
StarCoder2-15BToolAgent, 2.4163.31 / 82.51193.73 (+18.63%) / 67.45 (−18.25%)206.95 (+26.72%) / 56.40 (−31.64%)
StarCoder2-15BConversation, 1.6175.46 / 123.52206.44 (+17.66%) / 102.34 (−17.15%)219.21 (+24.93%) / 91.42 (−25.99%)

vLLM

ModelWorkload, req/sNative BIFlexBI with native prefillFlexBI with CanonicalPrefill
Qwen3-14BToolAgent, 1.6109.15 / 156.88144.27 (+32.18%) / 127.49 (−18.73%)145.31 (+33.13%) / 131.08 (−16.45%)
Qwen3-14BConversation, 1.2107.21 / 209.26142.35 (+32.78%) / 175.93 (−15.93%)141.87 (+32.33%) / 178.03 (−14.92%)
StarCoder2-15BToolAgent, 2.4150.64 / 93.36192.15 (+27.56%) / 69.72 (−25.32%)193.45 (+28.42%) / 71.70 (−23.20%)
StarCoder2-15BConversation, 1.6161.48 / 137.27205.68 (+27.37%) / 103.79 (−24.39%)204.14 (+26.42%) / 105.48 (−23.16%)

Scheduler control. A chunk-prefill-slot fix from the early offline setup remained in the SGLang integration after the framework upgrade. We have separated this setting from FlexBI. Percentages in the first column are relative to Native BI. Those in the second are relative to FlexBI with CanonicalPrefill reported above. Removing it from FlexBI changes P99 TTFT by less than 1% at each point.

ModelWorkload, req/sNative + scheduler fix: P99 TTFT (s)FlexBI without scheduler fix: P99 TTFT (s)
Qwen3-14BToolAgent, 1.6161.71 (+1.36%)116.21 (+0.52%)
Qwen3-14BConversation, 1.2213.71 (−1.19%)168.73 (−0.45%)
StarCoder2-15BToolAgent, 2.482.52 (+0.01%)56.96 (+0.99%)
StarCoder2-15BConversation, 1.6122.92 (−0.49%)90.75 (−0.73%)

W2. What are the individual end-to-end contributions of FlexTile and FlexLoad?

We have added the component comparison to RQ1. The table presents representative component comparisons from Figure 4.

ConfigurationToolAgent 1.6 req/s: TPSP99 TTFT (s)Conversation 1.2 req/s: TPSP99 TTFT (s)
Native BI109.15156.88107.21209.26
FlexTile only124.87142.50122.00197.41
FlexLoad only134.59132.61129.57184.84
Full FlexBI144.27127.49142.35175.93

W4. Is end-to-end BI preserved when multiple requests actually share KV loads?

Yes. We have added BI tests covering actual KV sharing, batch and cache changes, and prefill–decode consistency to Appendix C (Table 3). Both models and engines show zero bit mismatches.

The following representative results combine two 8,193-token targets per configuration.

ModelEngineRaw-logit values / bit mismatchesSelected KV values / bit mismatches
Qwen3-14BvLLM165,306,368/089,128,960/0
Qwen3-14BSGLang165,306,368/089,128,960/0
StarCoder2-15BvLLM53,477,376/044,564,480/0
StarCoder2-15BSGLang53,477,376/044,564,480/0
AuthorsOpenReview ↗

Answers for Rerevier WCUp (2/3)

Q1. How are FlexTile’s numerical contract and admission checks implemented and verified?

In the current Triton, output tiling is separated from the reduction primitive. Dot products use fixed MMA instructions, emitted in ascending K order, with each update consuming the preceding accumulator state. Changing BM/BN or warp allocation redistributes independent outputs while retaining each output’s MMA update sequence. With identical initialization and epilogue, this preserves output bits by induction. The bitwise probes (through run-bi.sh --suite gemm in the updated artifact) validate this mechanism in the generated kernels.

We have revised the method overview in Figure 2 and expanded Appendix B.3 with instruction-level reasoning, the admission checks in Algorithm 1, and tile-transition and page-sharing examples in Figures 7 and 8.

We also add the unified BI experiment in RQ5 and Appendix C (Table 3), providing the corresponding request-level validation.

Q4. Are prefill and decode bitwise consistent in vLLM, including after recomputation?

With CanonicalAttention enabled for prefill, we observed no cross-phase inconsistencies in the tested comparisons. We did observe bit-level differences between prefill without CanonicalAttention and Canonical decode for the same token after recomputation in vLLM. However, we would like to clarify that BI does not require cross-phase equality under fixed phase assignments. For example, vLLM-Ascend [1] requires outputs to be independent of batch size and request order, while excluding preemption and recomputation. The latest uploaded code provides switches to enable or disable CanonicalAttention during prefill in both vLLM and SGLang.

The unified BI test in Appendix C (Table 3) confirms bitwise prefill–decode consistency, including cold-cache recomputation.

With CanonicalPrefill, vLLM averages +0.08% TPS and +2.12% P99 TTFT versus FlexBI with native prefill. Cells give TPS / P99 TTFT (s), with percentages relative to native BI.

ModelWorkload, req/sNative BIFlexBI with native prefillFlexBI with CanonicalPrefill
Qwen3-14BToolAgent, 1.6109.15 / 156.88144.27 (+32.18%) / 127.49 (−18.73%)145.31 (+33.13%) / 131.08 (−16.45%)
Qwen3-14BConversation, 1.2107.21 / 209.26142.35 (+32.78%) / 175.93 (−15.93%)141.87 (+32.33%) / 178.03 (−14.92%)
StarCoder2-15BToolAgent, 2.4150.64 / 93.36192.15 (+27.56%) / 69.72 (−25.32%)193.45 (+28.42%) / 71.70 (−23.20%)
StarCoder2-15BConversation, 1.6161.48 / 137.27205.68 (+27.37%) / 103.79 (−24.39%)204.14 (+26.42%) / 105.48 (−23.16%)

References:

[1] vLLM-Ascend. Batch Invariance. https://docs.vllm.ai/projects/ascend/en/main/user_guide/feature_guide/batch_invariance.html

Q5. What explains the different eager-test results for native vLLM-BI and FlexBI?

The earlier check used an older vLLM RMSNorm implementation. Its reduction order changed with the total token count in the batch. All performance evaluations use the newer vLLM v0.26.0 baseline specified in the experimental setup. We have clarified the statements.

Q6. Do the paper’s results and the submitted artifact use the same trace protocol?

All online experiments reported in the paper use the unified traces described in the appendix. We use the same StarCoder2-filtered trace slices across models and systems. Workload selection, request order, and arrival times remain consistent across comparisons.

During early exploration, we sampled different traces to accommodate the models’ different context lengths, then unified the traces for the final experiments to avoid introducing unnecessary variables. The incorrect description arose during artifact preparation. A coding agent helped organize the submitted code using only part of the experimental records. It did not reflect the unified traces used for the reported results. We have corrected the documentation in the latest uploaded code.

Q7. How does FlexLoad differ from cascade attention and shared-prefix kernels?

Thank you for pointing out these related techniques. Existing shared-prefix attention methods such as cascade attention and shared-prefix kernels partition computation around the shared-prefix/private-suffix boundary. When this boundary changes with co-batches or cache state, the local softmax reductions and their merge structure can also change. Mathematical equivalence of these decompositions does not guarantee bitwise-identical results across batches. These methods do not by themselves provide BI under dynamic sharing.

FlexLoad separates numerical partitioning from physical grouping. Each query keeps fixed logical leaf boundaries, leaf computations, and merge order. At runtime, the planner groups query-leaf work only when the complete ordered physical-page vectors match. Each query retains its own softmax state and output. We have clarified this relationship in the related work section.

AuthorsOpenReview ↗

Answers for Reviewer WCUp (3/3)

Sorry for the disorder in the answers. This is because the number of characters for each answer is limited.

W3 / Q3. How much of FlexLoad’s speedup comes from partitioning versus shared loading?

The following ablation holds Canonical partitioning and the numerical contract fixed and switches only shared loading. Each cell reports the latency of Canonical without sharing divided by that with sharing. Columns give Hq/Hkv/D. Greater sharing coverage and larger groups expose more reusable KV reads. FlexLoad uses these opportunities to improve performance while keeping each query’s reduction unchanged.

Sharing topologyLogical KV read-saving opportunity16/16/9616/8/12816/4/12820/4/12832/4/6432/4/128
One group of 8, 24 independent21.71%1.153×1.177×1.159×1.158×1.148×1.126×
One group of 16, 16 independent46.51%1.487×1.342×1.311×1.308×1.268×1.242×
Four groups of 886.82%4.185×4.022×3.676×2.724×2.609×2.677×
Two groups of 1693.02%5.668×5.970×4.241×3.222×2.651×2.710×
One group of 3296.12%7.445×6.813×4.457×3.840×2.657×2.744×

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Overview and Revisions

We sincerely thank the ACs and all reviewers for their careful evaluation and constructive feedback. We appreciate the reviewers’ recognition of the value of recovering execution flexibility for batch-invariant inference and the encouraging serving results across two models and two workloads.

We have addressed the main concerns through the following additions:

  • Component contributions. We added end-to-end FlexTile-only and FlexLoad-only ablations to RQ1 (Figure 4). Our rebuttal further isolates shared loading under fixed Canonical partitioning and examines the contributions of CanonicalPrefill and the scheduler adjustment.
  • Method clarity and BI guarantees. We clarified the method overview in Figure 2 and expanded Appendix B.3 with instruction-level reasoning, explicit admission checks, and concrete tile-transition and page-sharing examples (Figures 7–8).
  • Broader BI validation. The new RQ5 and expanded Appendix C cover actual multi-request KV sharing, co-batch and cache changes, physical KV remapping, and prefill–decode consistency with cold-cache recomputation. All compared raw logits, final hidden states, and selected KV values match bit for bit.
  • We have also changed “Serving” to “Inference” in the title to more clearly reflect our operator-level contributions.

All changes are marked in blue in the revised manuscript. The main text includes one additional page to accommodate the clearer method illustration, component ablations, and expanded BI results. The revised illustration (Figure 2) clarifies the original method’s presentation while preserving the content it conveys. All experiments can be replicated using the submitted artifact.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.