acceptodds
Under review as a conference paper at ICLR 2027

TailorAttention: Accelerating LLM Attention with Latency-Aware Chunk Scheduling

Abstract

Modern LLM services handle heterogeneous workloads with sequence lengths varying across tasks and requests. Chunked attention distributes KV-cache chunks across cooperative thread arrays (CTAs), making partitioning and scheduling criti- cal to GPU utilization. Existing kernels, however, pipeline mainly within chunks, incurring boundary underutilization and phase-dependent CTA interference. Con- sequently, equal-sized chunks can have different latencies, making length-based scheduling unreliable and causing load imbalance and tail underutilization. We present TailorAttention, a kernel–scheduler co-design for chunked attention. Its Hopper- and Blackwell-optimized kernels enable inter-chunk pipelining and reduce output-layout conversion through transposed tensor-core dataflows, yielding predictable chunk latencies captured by affine models. Guided by these estimates, a millisecond-scale scheduler jointly selects potentially unequal chunk sizes and CTA assignments while accounting for reduction overhead, with a provable additive bound. Across GPUs, models, and workloads, TailorAttention improves attention- kernel performance by up to 4.96x and reduces end-to-end latency by 27.1% over FlashAttention-3/4 and FlashInfer

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
GPU Kernel, Hardware Acceleration, Systems for Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RSKVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies attention scheduling when requests have different sequence lengths. TailorAttention changes the kernel to reduce chunk-boundary overhead and make chunk latency more predictable. It then uses a latency model to choose chunk sizes and assign work to CTAs, including the cost of merging partial results. The paper compares with FlashAttention and FlashInfer on Hopper and Blackwell, includes kernel/scheduler ablations, and shows a serving improvement in SGLang.

Critical strengths
  • I like the connection between the kernel and scheduler. Making the kernel faster is useful by itself, but making its latency easier to predict also helps the scheduling decision. The discussion of V/O buffers and the transposed computation makes the idea fairly clear.

  • The evaluation covers several query lengths, GQA settings, and workload mixes, using strong kernel baselines. The kernel-only versus full-system comparison is helpful. I think there is already a useful contribution here, even though a same-kernel scheduling comparison would make the source of the gains clearer.

Critical weaknesses

There are one detail I could not follow in the serving setup. Section 5 says the weights are BF16, but Section 5.4 runs gpt-oss-120b on one H100. The official model card lists 117B parameters and MXFP4 MoE weights for single-GPU deployment, so a fully resident BF16 copy would not fit the stated memory. Is BF16 here only referring to attention? This may be a wording issue, but the actual formats and residency need to be clear. It would also be good to report numerical agreement with the reference attention implementation.

For the theory, my understanding is that Theorem 1 bounds the processing makespan T, not the full T+R objective. Please make the scope consistent in the surrounding text. I do not see this as invalidating the kernel results; it just needs a more precise claim.

Questions for authors
  1. What weight and KV formats are actually used for the single-H100 gpt-oss-120b run? How closely do the attention outputs match the reference implementation?

  2. Does the reported serving time include building and transferring the schedule? Please also clarify the scope of the bound. If you already have results for a length-based scheduler on the same TA kernel, they would be useful to see.

Reviewer_2hhmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies load imbalance and chunk-boundary overhead in attention for LLM requests with different sequence lengths. It combines kernels for Hopper and Blackwell with inter-chunk pipelining and transposed tensor-core computation. Affine latency models guide joint KV-chunk partitioning and CTA assignment, including reduction cost. The scheduler searches candidate completion times and provides an additive bound for the chunk-processing stage. The evaluation compares attention kernels with FlashAttention-3/4 and FlashInfer on H100 and B300. It also reports a 27.1% reduction in average turn latency against FlashAttention-3 in one SGLang multi-turn serving setup.

Critical strengths
  1. The paper addresses a practical efficiency problem. Requests with different KV lengths can leave GPU resources idle even when the attention computation is already highly optimized. Reducing chunk-boundary overhead and execution tails is a useful target for LLM serving. The reported kernel and end-to-end results suggest practical value.

  2. The kernel and scheduler designs have a clear connection. The separate V/O buffers and transposed computation aim to reduce both execution cost and latency variation. The scheduler then uses these latency estimates and accounts for reduction cost. This connects concrete hardware changes to the scheduling objective.

Critical weaknesses
  1. The performance breakdown is too coarse. Figure 4 separates the combined kernel optimizations from the full system, but does not distinguish the main kernel changes or show how much of the scheduling gain comes from better load balance versus reduction-cost control. A more detailed breakdown would make the contribution of these design choices clearer.

  2. The evaluation needs clearer analysis and presentation. Figure 3 is too crowded to read at normal page size. The results show gains across several workloads, but provide limited analysis of how sequence-length heterogeneity affects these gains or why TA is slower than FA-3 in some cases. This makes it difficult to identify the conditions where the method is most useful.

Questions for authors
  1. Which design choices account for the gains?

    Could you provide a finer breakdown of the gains from transposed tensor-core computation, inter-chunk pipelining, and accounting for reduction cost in scheduling? Controlled ablations where feasible, or timing of the main stages, would help separate their contributions.

  2. When does the scheduling approach help most?

    How does sequence-length heterogeneity affect the gains at a fixed batch size and total KV workload? Please also explain the slower cases in Figure 3, especially short prefill on Summarization and Agentic AI. For example, the plotted speedup over FA-3 is 0.88x for Agentic AI with q=128, head configuration (16:4), and batch size 32.

  3. Reporting inconsistencies need correction.

    Could the authors reconcile the FA-3/FI maximum speedups and the head/batch settings between Figure 3 and Section 5.1? Please also correct the B300 baseline label in Figure 4.

Reviewer_HYzrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents TailorAttention, a kernel–scheduler co-design for attention under heterogeneous sequence lengths, targeting decoding, speculative decoding, and short prefill. It combines transposed tensor-core computation with inter-chunk pipelining to reduce boundary overhead and make chunk latency more predictable. Using affine models for chunk-processing and reduction costs, it formulates joint chunk partitioning and CTA assignment as a MILP. A fast heuristic searches candidate processing-time budgets and selects schedules based on their total estimated latency. The theoretical analysis states an additive bound for the chunk-processing stage.

Critical strengths

I think the paper’s strongest idea is to treat chunk-latency predictability as a kernel design objective, then use that predictability to guide partitioning and CTA assignment. The proposed inter-chunk pipeline and cost model form a coherent approach to heterogeneous attention workloads, including the trade-off between better load balance and additional chunk and reduction overheads.

Critical weaknesses
  1. The novelty needs to be clearer. The transposed attention computation appears closely related to FlashMLA-ETAP. I would like the authors to explain the difference more clearly, as I currently find it hard to tell which parts of the kernel design are new.
  2. The comparison with related work is incomplete. LeanAttention also addresses unequal partitioning and load balancing. Although it is cited, it is not included in the experimental comparison. Without this comparison, it is difficult to judge how much the proposed scheduling approach adds beyond existing methods.
Questions for authors
  1. Optimization objective and guarantee. Section 4.1 minimizes J = T + R, but Theorem 1 only concerns T. Could you clarify what is guaranteed for the total objective J? For the same processing-time limit, different partitions may have different reduction costs, so I am not sure how constructing one greedy partition captures this trade-off.
  2. End-to-end configuration. How was gpt-oss-120b run on a single H100 if the model weights were BF16? Around 120B BF16 parameters would require about 240 GB for the weights alone.
  3. Speedup numbers in Figure 3. Section 5.1 reports 3.74× over FA-3 and 2.46× over FlashInfer, but Figure 3 appears to label them the other way around. Which baseline does each result correspond to, and how were these maximum speedups calculated?
1 comment
Area_Chair_fffaOpenReview ↗

Author-reviewer discussion period is ending soon

Dear Reviewer HYzr,

If you haven't done so, please read through authors' response and let authors know if you need further clarification or have more questions. Beginning 11/19, authors are not allowed to respond any more, so this is good time to review the response and ask more.

Thank you.

-AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.