acceptodds
Under review as a conference paper at ICLR 2027

Triceps: Tri-Component Separable Attention for Long-Context Prefill

Abstract

The complexity of full attention during prefill in Transformer-based large language models is the main computational bottleneck in applications that require handling long contexts, such as long-document question answering, repository-level code agents or long book summarization. Existing sparse attention methods combine fixed patterns by union (attention sinks, sliding windows, retrieved blocks, vertical and slash patterns), resulting in a mask denser than the mass it captures requires, wasting computation on blocks that carry almost no attention. We instead show that the attention matrix admits a training-free multiplicative tri-component separable approximation: a relative-position factor, a key factor and a query-normalization factor. Combined multiplicatively, the three factors induce a score threshold that a union of fixed patterns cannot express. We dynamically evaluate this approximation per head to route computation between three modes: an long convolution when the error measured on the last query rows falls below a threshold, an sparse mode for low-density masks, and delegation to dense or another sparse attention technique otherwise. We evaluate this method on RULER and LongReason with two open-weight long-context models, Qwen3-30B-A3B-Instruct-2507 and Llama-3.1-Nemotron-8B-UltraLong-4M-Instruct, and, at extended context lengths (128k and beyond), we observe speedups in Time-To-First-Token (TTFT) over dense attention while achieving a superior speed–accuracy trade-off compared to existing sparse methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM inference, Prefill, long-context, efficient attention, structured attention, training-free

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_xvCEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Triceps, a training-free prefill acceleration method for long-context LLMs that approximates the interior attention matrix (outside the exact sink and sliding-window Λ\Lambda-shape) as an elementwise product of three rank-1/Toeplitz matrices: Aint≈H⊙D⊙CA_{\text{int}} \approx H \odot D \odot C, capturing query normalization, relative-position slash diagonals, and key-wise vertical columns. At runtime, the authors fit DD and CC in the log-domain via ridge-regularized least squares on 10410^4 uniformly sampled pre-softmax logits, calibrate HH, and evaluate the approximation on the last k=64k = 64 query rows to route each head to either an O(Nlog⁡N)\mathcal{O}(N \log N) FFT convolution kernel, a multiplicative block-sparse Triton kernel, or a delegated sparse method (FlexPrefill). Experiments on Qwen3-30B-A3B and Nemotron-8B across RULER and LongReason (64k to 256k ctx) demonstrate end-to-end TTFT speedups at 128k--256k.

Critical strengths

Factorizing the interior vertical-slash pattern as H⊙D⊙CH \odot D \odot C by fitting a row-centered linear model in the log-score domain (Eq. 5) is a clean alternative to taking a rigid union of dominant columns and diagonals. In block-sparse mode, adding the fitted log-components (h~m+d~m−n+c~n+log⁡(m+1)≥θ\widetilde{h}_m + \widetilde{d}_{m-n} + \widetilde{c}_n + \log(m+1) \ge \theta) induces a smooth multiplicative threshold where strong key importance can compensate for weaker positional weight, yielding low deployed densities (11%--25% across paths, Tab. 2).

The systems integration in vLLM (w/ TP=2 up to 256k ctx) and the transparent FLOP/pseudocode breakdown in App. C (Tab. 6, Alg. 1--10) are also well executed. In addition, the exponential-family covariance identity in App. B.2 (Covpi,t(ei,:,v)\text{Cov}_{p_{i,t}}(e_{i,:}, v)) gives a nice theoretical explanation for why gating the convolution mode on output error EconvE_{\text{conv}} is much more reliable than using score-level R2R^2.

Critical weaknesses

My biggest concern is that the claim of a superior speed-accuracy trade-off over existing sparse methods is hard to reconcile w/ the accuracy drops in Tab. 1 and Tab. 4. On LongReason w/ Qwen3-30B-A3B (Tab. 1a), Triceps (τ=0.4\tau = 0.4) drops by 8.95 points below Dense at 64k (67.62% vs. 76.57%) and 5.79 points at 256k (67.51% vs. 73.30%), whereas FlexPrefill, PBS-Attn, and PRISM stay within ~1 point of Dense. Similarly, on RULER (Tab. 1b, Tab. 4), Triceps posts the lowest average accuracy among all online baselines in nearly every setting, w/ sharp drops on subtasks like Qwen3 cwe at 128k/256k (31.60% and 8.40% vs. PRISM's 64.10% and 26.90%) and Nemotron niah-multikey-1 at 64k (73.00% vs. Dense 86.00%). Because Tab. 1 only reports a single operating point (τ=0.4,kc=0.5\tau = 0.4, k_c = 0.5) without a Pareto curve, it is unclear if any TTFT speedup over FlexPrefill remains once τ\tau is tightened to match FlexPrefill's accuracy.

Second, although the O(Nlog⁡N)\mathcal{O}(N \log N) FFT convolution mode is pitched as a headline contribution, Tab. 2 shows it is selected for only 1.7%--4.4% of heads on Nemotron and 4.6%--13.5% on Qwen3, and its share actually shrinks as ctx length grows from 64k to 256k (e.g., from 13.5% down to 6.1% on Qwen3 LongReason). Meanwhile, the decomposition, O(Nd2)\mathcal{O}(N d^2) row-variance calibration, and routing pass incur substantial upfront overhead (39% of prefill GPU time at 64k, Fig. 2), making Triceps slower than FlexPrefill and PRISM at 64k across most settings in Tab. 5.

Third, Triceps relies heavily on FlexPrefill as its delegation backend for 10.5%--31.8% of heads (Sec. 5.1), which includes the retrieval heads where the rank-1/Toeplitz fit breaks down. Those delegated heads pay Triceps' full fitting and k=64k = 64 tail-row measurement overhead (TovhT_{\text{ovh}}), and then run FlexPrefill's own selection pass from scratch (line 426). There are also a couple of mismatches between the main text and App. C: Alg. 1 calls convolution calibration (Alg. 7, line 7) only after routing measurement (Alg. 5, line 4), meaning EconvE_{\text{conv}} in Alg. 5 is evaluated using the statistics-calibrated hh from Alg. 4 rather than the convolution-calibrated hh used at execution; and Alg. 6 introduces an undocumented cost model C\mathcal{C} that does not appear in Eq. 13.

Questions for authors
  1. Why does Qwen3-30B-A3B suffer such a steep accuracy drop on LongReason at 64k (67.62% vs. 76.01% for FlexPrefill) and 256k (67.51% vs. 72.29%), as well as on RULER cwe? If you lower τ\tau so that Triceps matches FlexPrefill's accuracy on Qwen3 LongReason and RULER, what is the resulting TTFT speedup?

  2. Given that the O(Nlog⁡N)\mathcal{O}(N \log N) convolution mode is only triggered on 1.7%--6.1% of heads at 256k (Tab. 2), what are the end-to-end TTFT and accuracy if you disable the convolution path entirely and only route between Triceps' block-sparse mode and delegation?

  3. For the 20%--32% of heads delegated to FlexPrefill, why not reuse the exact tail-row attention scores already computed during ROUTINGMEASUREMENT (on the last k=64k = 64 rows) instead of running FlexPrefill's selection pass again? Also, how does Triceps compare if delegation falls back to dense FlashAttention-2 rather than FlexPrefill?

  4. Can you clarify the ordering in Alg. 1 (lines 3--7)? Is OiconvO_i^{\text{conv}} inside ROUTINGMEASUREMENT (Alg. 5, line 4) computed using the statistics-calibrated hh from Alg. 4, and if so, why not use the exact row mass from Alg. 7 when measuring EconvE_{\text{conv}}? Also, how is the cost model C\mathcal{C} in Alg. 6 defined?

3 comments

We thank Reviewer xvCE for the insightful technical review and for highlighting the exponential-family covariance identity in Appendix B.2.

  • Q1 — Accuracy on Qwen3 LongReason / RULER cwe & Matched-Accuracy TTFT:

    RULER cwe and LongReason on the initial configuration (kc=0.5,τ=0.4k_c = 0.5, \tau = 0.4, 18.118.1\\% dropped mass) were over-pruned. As detailed in Global Comment 1 (Table A, Table 5), setting kc=0.2k_c = 0.2 and τ=0.5\tau = 0.5 bounds dropped mass to 9.59.5\\%. On Qwen3 LongReason, accuracy increases to 74.56%, 73.43%, and 72.04% at 64k, 128k, and 256k (matching Dense at 128k) while running faster than FlexPrefill at 128k (9.68s vs. 10.77s) and 256k (25.02s vs. 28.98s). On RULER, Triceps achieves the highest average accuracy among sparse baselines in 5 of 6 settings: Nemotron niah_multikey_1 (64k) improves from 73.00% to 87.00% (vs. Dense 86.00%), and Qwen3 cwe improves to 46.40% at 128k (vs. FlexPrefill 21.30%) and 13.60% at 256k (vs. FlexPrefill 10.19%). While PRISM achieves higher scores on cwe (64.10% / 26.90%), it comes at a higher computational cost (52.86s vs. 41.55s) and still underperforms Triceps (τ=0.5\tau=0.5) on the overall RULER average across all context lengths (Table A).

 

  • Q2 — 64k Overhead, Convolution Share vs. Context Length & Disabling Convolution:

    Kernel optimizations reduced upfront overhead from 3939\\% to 2222\\% at 64k and 55\\% at 256k (Global Comment 1, Figure 4), making Triceps faster than FlexPrefill and PRISM at 64k on Nemotron and faster than PRISM on Qwen3 (Table A). With the updated configuration, while we reduced the number of dropped blocks (Table 2), the convolution share rises to 9.89.8\\%\text{—}34.7\\% on LongReason and 4.84.8\\%\text{—}24.8\\% on RULER (decreasing mildly with NN as a single global Toeplitz operator spans a longer prefix). As shown in Global Comment 3 and Table B (Table 4), each convolution head executes ∼4×\sim 4\times faster than block-sparse at 256k (2.7 ms2.7\text{ ms} vs. 9.7—12.6 ms9.7\text{—}12.6\text{ ms}, Figure 4), and disabling convolution (Triceps No-Conv) increases 256k TTFT across all benchmarks (up to +6.2+6.2\\%) at comparable accuracy.

 

  • Q3 — Reusing the k=64k=64 Tail-Row Scores in FlexPrefill & Dense FA2 Fallback:

    • We do not reuse the k=64k=64 tail-row scores because we treat delegation as a modular black-box interface (supporting FlexPrefill, Dense FA2, PRISM, etc.) and keep FlexPrefill's reference kernel unmodified. Moreover, our routing kernel reduces block logsumexps on-the-fly in SRAM without materializing the 64×N64 \times N score matrix in HBM. Sharing the tail-row pass with FlexPrefill is feasible and would further reduce the delegation bar in Figure 4, although this represents 0.050.05\\% of FLOPs at 256k, which we consider negligible.
    • Dense FA2 fallback: Reported as Triceps FA2-Delegation in Global Comment 3 and Table B (Table 4): even when every delegated head runs exact dense FlashAttention-2, Triceps achieves 2.06× to 2.36×2.06\times\text{ to }2.36\times speedup over Dense on LongReason at 256k and 1.74× to 1.85×1.74\times\text{ to }1.85\times on RULER.

 

  • Q4 — Ordering in Algorithm 1/5 for EconvE_{\text{conv}} & Cost Model C\mathcal{C} in Algorithm 6:

    • EconvE_{\text{conv}} calibration: Inside ROUTINGMEASUREMENT (Alg. 5), EconvE_{\text{conv}} is evaluated with the exact convolution row-mass normalization of Equation 10 (Alg. 7), not the statistics-calibrated h\boldsymbol{h} from Alg. 4: on the k=64k = 64 tail rows, our fused Triton kernel accumulates both the unnormalized interior output and the exact row sum (d∗c)i(\boldsymbol{d} * \boldsymbol{c}) _ i directly in O(kNd)\mathcal{O}(kNd) time and normalizes by AΛ′1i+gi(d∗c)i\boldsymbol{A}' _ {\Lambda}\mathbf{1}_i + g_i(\boldsymbol{d}*\boldsymbol{c})_i. Full-sequence CALIBRATEHORIZONTAL_CONVOLUTION (Alg. 7, which costs 3 full-length FFTs across all NN rows) is called after DECIDEHEAD in Alg. 1 so that full-sequence FFTs are only paid on heads actually routed to conv.
    • Cost model C\mathcal{C}: Alg. 6 is a generalized form. In our code (ConstantCostModel), Cconv=0\mathcal{C} _ {\text{conv}} = 0, Csparse(ρ)=ρ\mathcal{C} _ {\text{sparse}}(\rho) = \rho, and Cdeleg=Cdense=ρ0=0.60\mathcal{C} _ {\text{deleg}} = \mathcal{C} _ {\text{dense}} = \rho_0 = 0.60, making arg⁡min⁡mV[m]\arg\min_m \mathcal{V}[m] in Algorithm 6 identical to Equation 13. We simplified Algorithm 6 in Appendix C to match Equation 13 directly.

Thank you for the detailed response and additional experiments. I have two remaining questions: i). Table A varies Triceps’ configuration while keeping the baselines at fixed operating points. Could you also sweep FlexPrefill’s budget on Qwen3 LongReason to show whether Triceps remains faster at matched accuracy? ii). Table B shows that disabling convolution can improve accuracy, while replacing delegation with exact attention can reduce it. Could you explain these changes and clarify how they support attributing the accuracy loss mainly to block selection? Varying kck_c while holding τ\tau fixed would help isolate this effect.

We thank Reviewer xvCE for the follow-up questions and for suggesting the fixed-τ\tau ablation over kck_c.

 

i) Matched-Accuracy Comparison with FlexPrefill on Qwen3 LongReason

On Qwen3-30B-A3B LongReason, Table A already provides a direct comparison at similar or higher accuracy against FlexPrefill (γ=0.95\gamma = 0.95):

  • 128k: Triceps (τ=0.5\tau = 0.5) achieves higher accuracy (73.4373.43\\% vs. 71.9171.91\\%, matching Dense FA2) and lower TTFT (9.68s9.68\text{s} vs. 10.77s10.77\text{s}, 1.11×1.11\times faster).

  • 256k: Triceps (τ=0.5\tau = 0.5) closely matches FlexPrefill (72.0472.04\\% vs. 72.2972.29\\%) while reducing TTFT from 28.98s28.98\text{s} to 25.02s25.02\text{s} (1.16×1.16\times faster).

Because each full LongReason evaluation on Qwen3-30B-A3B is computationally expensive, we focused our remaining GPU time before the discussion deadline on completing the fixed-τ\tau ablation over kck_c below.

 

ii) Isolating Block Selection (kck_c Sweep at Fixed τ=0.6\tau = 0.6) & Table B Variations

Following your suggestion, we evaluated Qwen3-30B-A3B on the full LongReason benchmark (794 prompts) holding τ=0.6\tau = 0.6 fixed (Econv≤0.6E_{\text{conv}} \le 0.6 unchanged) and varying only kc∈0.20,0.35,0.50k_c \in \\{0.20, 0.35, 0.50\\}, so only the block-sparse retained mass is affected:

Configuration (τ=0.6\tau = 0.6 fixed)Max MdropM_{\text{drop}}LR Acc (%) 64k / 128k / 256kLR TTFT (s) 64k / 128k / 256k
Triceps (kc=0.50,τ=0.6k_c = 0.50, \tau = 0.6)25.925.9\\%50.5050.50 / 58.1958.19 / 61.8461.843.403.40 / 7.687.68 / 19.2419.24
Triceps (kc=0.35,τ=0.6k_c = 0.35, \tau = 0.6)18.918.9\\%63.6063.60 / 68.5168.51 / 67.6367.633.713.71 / 8.548.54 / 21.5321.53
Triceps (kc=0.20,τ=0.6k_c = 0.20, \tau = 0.6) (Table A/B)11.311.3\\%74.4374.43 / 71.6671.66 / 71.5471.544.104.10 / 9.399.39 / 24.1024.10

Holding the convolution threshold fixed (τ=0.6\tau = 0.6), tightening kck_c from 0.50→0.35→0.200.50 \to 0.35 \to 0.20 reduces the maximum allowed dropped attention mass MdropM_{\text{drop}} from 25.925.9\\% to 11.311.3\\%, monotonically improving accuracy by +23.93+23.93 pts (64k), +13.47+13.47 pts (128k), and +9.70+9.70 pts (256k). In contrast, at fixed kc=0.20k_c = 0.20 in Table B, disabling the convolution heads (No-Conv vs. Full, redirecting heads with Econv≤τE_{\text{conv}} \le \tau to block-sparse) produces a mean absolute difference of only 0.730.73 percentage points across all 12 model/length columns, and replacing FlexPrefill delegation with exact dense attention (FA2-Delegation vs. Full, which only modifies the delegated heads that already retain 9595\\% mass under FlexPrefill while leaving the remaining block-sparse and convolution heads unchanged) yields a mean absolute difference of only 0.860.86 percentage points. Both variations are an order of magnitude smaller than the +9.70 to +23.93+9.70\text{ to }+23.93 point effect of kck_c, confirming that the accuracy drop in our initial configuration came from not retaining sufficient attention mass during block selection rather than from the convolution or delegation approximations.

Reviewer_QJ5gOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Triceps, a training-free method for accelerating long-context prefill by approximating attention as the product of relative-position, key-dependent, and query-normalization factors. The same decomposition supports both convolutional attention and block-sparse mask construction, with attention sinks and local windows handled separately. Based on output error and discarded attention mass measured on the final query rows, together with mask density, Triceps routes each head to convolution, block-sparse attention, or another attention method. The authors integrate Triceps into vLLM and evaluate accuracy and time-to-first-token on RULER and LongReason using Qwen3 and Nemotron at 64K–256K context lengths.

Critical strengths

The core method is well formalized, with a clear mathematical formulation of the tri-component attention approximation. And the evaluation is reasonably comprehensive, covering multiple models and long-context benchmarks with measurements of both accuracy and time-to-first-token.

Critical weaknesses
  1. The presentation lacks sufficient visual guidance and empirical motivation for the algorithm. The main text contains no figures before the experimental section, making the mathematical decomposition and routing procedure difficult to follow. Although the paper discusses known vertical–slash attention patterns, it provides little upfront empirical analysis or profiling to explain why these patterns support the proposed multiplicative decomposition or when convolution would be beneficial. Consequently, the transition from the problem to the algorithm feels abrupt. An overview diagram and a motivating analysis of attention patterns would improve readability and better support the design choices.

  2. The claimed speed–accuracy advantage is not convincingly established. On Qwen3 LongReason at 64K, Triceps is both less accurate than FlexPrefill (67.62 vs. 76.01) and slower (4.47 s vs. 4.06 s). At 256K, its approximately 3.4× TTFT speedup over dense attention comes with an accuracy drop of 5.79 percentage points (Tables 1 and 5). It remains unclear whether Triceps would retain a speed advantage over sparse baselines if they were tuned to tolerate similar accuracy loss. Accuracy–latency curves across multiple budgets are needed to establish whether Triceps provides a better trade-off at comparable quality.

Questions for authors
  1. Please specify the GPU model, inference precision, and timing batch configuration, and clarify whether Equation 13's cascade or Algorithm 6's cost-based sparse/delegation choice produced the reported runs.

  2. Can the authors report a focused accuracy–TTFT sweep on Qwen3 LongReason, where the loss is largest, with k_c fixed and baseline budgets adjusted over a comparable range? Specifically, how much speedup remains when Triceps is tuned to match baseline accuracy, and does it remain faster when sparse baselines are tuned to tolerate a similar accuracy loss? (Since we observe the accuracy drop in this workload)

  3. It's still unclear that how much does the conv branch contributes. What changes in accuracy and TTFT if the convolution branch is disabled while the remaining routing is retained?

3 comments

We thank Reviewer QJ5g for the thorough review and helpful suggestions on visual presentation and experimental details.

  • W1 — Visual Guidance & Upfront Empirical Motivation:

    As detailed in Global Comment 2, we added a top-level pipeline diagram in Figure 1 (Section 3) illustrating the AΛ+Aint\boldsymbol{A} _ {\Lambda} + \boldsymbol{A} _ {\text{int}} split, the H⊙D⊙C\boldsymbol{H} \odot \boldsymbol{D} \odot \boldsymbol{C} approximation, and the 64-row routing to each of the three modes. We also added an empirical 64k block-mask example in Figure 2, to visualize and motivate how our method constructs sparser masks compared against exact offline oracle masks and prior methods.

 

  • W2 & Q2 — Accuracy–TTFT Sweep on Qwen3 LongReason & 64k Latency:
    1. 64k latency: As discussed in Global Comment 1, optimizing our approximation and routing kernels (with the method unchanged) reduced upfront overhead from 3939\\% to 2222\\% at 64k and 55\\% at 256k (Figure 4), delivering speedups starting from 64k (Table A, Figure 5, Table 6).
    2. Accuracy–TTFT sweep: The updated results with kc=0.2k_c = 0.2 and τ∈0.5,0.6,0.7\tau \in \\{0.5, 0.6, 0.7\\} (Table A) show that at τ=0.5\tau = 0.5, Triceps closely approaches or outperforms prior methods at their reference budgets in accuracy while delivering up to 3.20×3.20\times speedup over Dense and 1.31×1.31\times over FlexPrefill (1.16× to 1.24×1.16\times\text{ to }1.24\times on Qwen3 LongReason at 256k across τ\tau), without needing to degrade the baselines to the initial submission's lower-accuracy regime.

 

  • Q1 — Hardware, Precision, Batching & Equation 13 vs. Algorithm 6:
    • Experimental setup: All runs use 2×2\times NVIDIA A100 80GB GPUs (TP=2), bfloat16 weights/KV-cache/kernels (with fp32 accumulation), and batch size = 1.
    • Eq. 13 vs. Alg. 6: Alg. 6 is a generalized form. Both produce the exact same routing: in our code (ConstantCostModel), Cconv=0\mathcal{C} _ {\text{conv}} = 0, Csparse(ρ)=ρ\mathcal{C} _ {\text{sparse}}(\rho) = \rho, and Cdeleg=Cdense=ρ0=0.60\mathcal{C} _ {\text{deleg}} = \mathcal{C}_{\text{dense}} = \rho_0 = 0.60, so arg⁡min⁡mV[m]\arg\min_m \mathcal{V}[m] in Algorithm 6 reduces identically to Equation 13's cascade. We simplified Algorithm 6 in Appendix C to match Equation 13 directly.

 

  • Q3 — Disabling the Convolution Branch: As detailed in Global Comment 3 and Table B (Table 4, Appendix A.5), disabling convolution (Triceps No-Conv) leaves accuracy comparable while increasing 256k TTFT across all four model/benchmark pairs: +6.2+6.2\\% on Nemotron LongReason (25.15s→26.70s25.15\text{s} \to 26.70\text{s}), +3.9+3.9\\% on Nemotron RULER (31.77s→33.00s31.77\text{s} \to 33.00\text{s}), and +1.4+1.4\\% on Qwen3 LongReason (24.10s→24.44s24.10\text{s} \to 24.44\text{s}) and RULER (40.48s→41.04s40.48\text{s} \to 41.04\text{s}, where Qwen3's smaller gap reflects its lower 2.22.2\\%\text{—}8.9\\% block-sparse density at 256k, Table 2).

Thanks for your response and additional efforts. For me, this work has mainly two shortages: 1. The method looks too complex, parameterized and needed carefully calibration, which makes it hard to be deployed in real scenarios. 2. The gain is not very significant, as in the Evaluation in Table 1, Triceps does not exhibit a significant advantage over the other methods in Table 1. After consideration, I think I will stay my current score.

We thank Reviewer QJ5g for reading our response and sharing these final thoughts. For completeness, we briefly clarify both points:

  1. Calibration & Deployment: As detailed in Point 1 of our follow-up response to Reviewer yhoF (Appendix A.1, Figure 6), kc=0.2k_c = 0.2 is fixed once by the theoretical dropped-mass bound, leaving a single user-facing threshold τ\tau (analogous to γ\gamma in FlexPrefill). We applied the same configuration unchanged across tasks, from 4 RULER subtasks on Nemotron-8B to the remaining 9 RULER subtasks and 794 LongReason prompts, and to another LLM architecture (MoE Qwen3-30B-A3B) without recalibration.

 

  1. Role of Table 1 vs. End-to-End TTFT Gains: The goal of sparse attention is to minimize the accuracy gap relative to dense attention (Table 1) while maximizing acceleration by constructing sparser masks (Figure 2, Table 2). The resulting gain is shown in Figure 5 (Table 6, Table A), where at comparable accuracy Triceps achieves up to 3.20× to 3.42×3.20\times\text{ to }3.42\times speedup over Dense and 1.31× to 1.50×1.31\times\text{ to }1.50\times over FlexPrefill. Moreover, as detailed in Point 3 of our follow-up response to Reviewer yhoF, these gains are measured on full-model end-to-end prefill TTFT in vLLM rather than on an isolated attention layer. This naturally dilutes the raw attention-layer speedup across the full model, as it measures the realistic wall-clock latency experienced by the user.
Reviewer_s3Z1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes approximating the attention scores, outside the sliding window and a fixed window at the beginning of context, by separating it into three multiplicative components: 1) a query specific normalizer 2) a score determined based on the key vector 3) a positional score depending on the distance between query and key. This approximation is motivated by a vertical slash pattern observed in earlier works on non-retrieval heads. Computing this approximation requires a convolution in order to compute the multiplication between the distance-based position scores and the other two scores.To adapt to different type of heads, the approximation is either applied directly (convolution mode) or over blocks (block sparse mode). In the latter mode, first a block with high attention mass is identified and dense attention is computed over that block. The decision between convolution or block sparse mode is taken based on how well the approximation performs over the last few queries. Similarly, if in the last few queries the block sparse discards a block with high attention score, the approximation is dropped for that query and full attention is used. Experiments show that a lot of queries successfully use block sparse while some use convolution and a small portion revert to full attention. The performance is usually close to SoTA on short context but there are significant gaps in longer contexts, both with the baseline and other sparse attention methods (like PRISM).

Critical strengths

The method used for approximating the pattern is interesting.

Critical weaknesses

From my understanding, choosing which mode to use is based on the gap between dense attention and approximation in the last few queries. However, this is impossible to compute without additional costs since each of the previous queries either compute dense attention or the approximate. This makes this impractical. It is still interesting that doing this, we can rely on block sparse for the majority of queries. But still this makes the method impractical.

In addition to above, the method does not work very well. The numbers show significant regret in longer context across model families.

Questions for authors
  1. How well can you fit the ridge regression for your approximations?
1 comment

We thank Reviewer s3Z1 for the feedback and address the two concerns and question below.

  • W1 — Practicality and Cost of Measuring the Gap on the Last Queries:

    We respectfully clarify a distinction between prefill and autoregressive decoding. Triceps accelerates prefill, where all NN query, key, and value vectors of a layer reside in GPU memory simultaneously. Computing the exact attention output $ \boldsymbol{ O }^{ \text{dense}} _ i$ on the last k=64k = 64 query rows does not require computing dense attention on the preceding N−kN - k queries: it only multiplies a 64×d64 \times d query slice against the N×dN \times d key and value matrices. Likewise, once the decomposition is fitted on 10410^4 interior samples, evaluating the approximation (Oiconv\boldsymbol{ O } ^ { \text{conv}} _ i and MdropM_{\text{drop}}) to measure the gap is also performed locally on only those k=64k = 64 rows before executing the chosen mode across all NN rows (see Figure 1).

    Computing this slice costs 4kNd=O(Nd)4kNd = \mathcal{O}(Nd) FLOPs per head: only 2k/N=0.052k/N = 0.05\\% of full causal attention FLOPs at N=256kN = 256\text{k}. As shown in Figure 4, the entire decomposition and 64-row routing pass takes 55\\% of prefill GPU time at 256k, and is included in every reported end-to-end TTFT (Table 6).

 

  • W2 — Long-Context Accuracy Across Model Families:

    The accuracy drop in the initial submission was induced by kc=0.5,τ=0.4k_c = 0.5, \tau = 0.4 (18.118.1\\% allowed dropped mass in block-sparse mode). Tightening the slope to kc=0.2k_c = 0.2 (Point 1 & Table A in Global Comment; Tables 1 & 5) eliminates this degradation across both Qwen3 and Nemotron: at τ=0.5\tau = 0.5, Triceps closely approaches or exceeds the accuracy of all sparse baselines on both LongReason and RULER while achieving up to 3.2×3.2\times end-to-end TTFT speedup over Dense (Figure 5, Table 6).

 

  • Q1 — How Well Can You Fit the Ridge Regression?

    We measured the coefficient of determination R2R^2 from the fit and classified all heads of Nemotron-8B (1,0241{,}024 heads) and Qwen3-30B-A3B (1,5361{,}536 heads) on the 1k–50k synthetic KV-retrieval benchmark of Wu et al. (2024):

    • Non-retrieval heads (87.387.3\\% on Nemotron, 89.189.1\\% on Qwen3) are well captured by the separable fit, with median R2=0.76R^2 = 0.76 on Nemotron and 0.680.68 on Qwen3; Triceps accelerates 82.782.7\\% and 70.770.7\\% of them via convolution or block-sparse attention.
    • Retrieval heads (12.712.7\\% on Nemotron, 10.910.9\\% on Qwen3) do not follow a matrix-wide vertical–slash pattern and concentrate in the low-R2R^2 tail (median R2=0.37R^2 = 0.37 and 0.330.33); our 64-row tail check detects this residual at runtime, delegating 70.670.6\\% and 64.464.4\\% of retrieval heads.

    These empirical results show the strong correlation between retrieval heads and fit quality, as well as how our routing dispatches them. For more details, please refer to Appendix A.3 (Figure 8). Figure 2 also provides an example of the mask obtained from the fit.

Reviewer_yhoFOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes Triceps, a training-free method for accelerating long-context prefill. It fits a tri-component separable attention approximation consisting of relative-position, key-dependent, and query-normalization factors. Based on runtime quality measurements, each attention head is routed to convolution, block-sparse attention, or delegation to another attention method, which is FlexPrefill in the main experiments.

Critical strengths

The paper provides extensive experiments across different datasets, models, and context lengths, evaluating both accuracy and efficiency. I also appreciate the detailed breakdown of routing shares and the computational costs of the different paths, which helps clarify how the method behaves in practice.

Critical weaknesses

W1: The routing breakdown shows that most head-calls use block-sparse attention, while only a small fraction use convolution, even fewer than use delegation. This leaves me unconvinced about how much the convolution path contributes to the reported gains. The tri-component approximation also guides block selection, but since many existing methods already use block-sparse attention, I would like clearer evidence of what this particular approximation adds over existing selection mechanisms. Component-wise ablations would help establish where the gains actually come from.

W2: The method also incurs noticeable accuracy losses on RULER and LongReason, while its efficiency advantage over existing sparse methods mainly appears at 128k and longer contexts. This makes me question whether the accuracy–efficiency trade-off is actually better. Comparisons at matched accuracy, or more complete accuracy–latency curves, would be more convincing than the current operating points.

Overall, my concern is that the approximation and routing overhead offsets part of the benefit of sparse computation, while the resulting attention computation still does not preserve task-relevant information accurately enough. The current results do not clearly establish whether the accuracy loss comes from convolution, block selection, or delegation, nor whether the additional speed justifies that loss.

Questions for authors

Please refer to Weaknesses.

5 comments

We thank Reviewer yhoF for appreciating our experimental scope and routing breakdowns.

  • W1 — Contribution of the Convolution Path & Component Ablations (No-Conv, FA2-Delegation):

    With the updated configuration, the convolution path handles 9.89.8\\%\text{ to }34.7\\% of heads on LongReason and 4.84.8\\%\text{ to }24.8\\% on RULER (Table 2). At 256k with Nemotron, a convolution head executes in 2.7 ms2.7\text{ ms}, compared to 9.7—12.6 ms9.7\text{—}12.6\text{ ms} for block-sparse (∼4×\sim 4\times slower) and 36.0—39.4 ms36.0\text{—}39.4\text{ ms} for delegation (∼14×\sim 14\times slower). Convolution routes the head into a faster attention computation path (Figure 4).

    In Table B (Global Comment; Table 4 in the paper), disabling convolution (Triceps No-Conv) leaves accuracy nearly unchanged while increasing 256k TTFT across all benchmarks: from 25.15s25.15\text{s} to 26.70s26.70\text{s} (+6.2+6.2\\%) on Nemotron LongReason, 31.77s31.77\text{s} to 33.00s33.00\text{s} (+3.9+3.9\\%) on Nemotron RULER, 24.10s24.10\text{s} to 24.44s24.44\text{s} (+1.4+1.4\\%) on Qwen3 LongReason, and 40.48s40.48\text{s} to 41.04s41.04\text{s} (+1.4+1.4\\%) on Qwen3 RULER (where Qwen3's smaller delta reflects its lower 2.22.2\\%\text{—}8.9\\% block-sparse density at 256k, Table 2). Furthermore, replacing FlexPrefill delegation with exact dense FlashAttention-2 (Triceps FA2-Delegation) achieves a 2.06× to 2.36×2.06\times\text{ to }2.36\times TTFT speedup over Dense on LongReason at 256k and 1.74× to 1.85×1.74\times\text{ to }1.85\times on RULER.

  • W1 — What the Tri-Component Approximation Adds to Block Selection:

    Please refer to Global Comment 2 (Figure 2), which discusses an empirical example of how our method achieves significantly sparser masks than prior methods; across all routed block-sparse heads in the full models (Table 2), this translates into 2.22.2\\%\text{—}16.9\\% block-sparse density on Qwen3 (2.22.2\\%\text{—}9.3\\% at 256k) and 7.27.2\\%\text{—}18.7\\% on Nemotron.

 

  • W2 — Accuracy–Efficiency Trade-off & Matched-Accuracy Comparisons:

    As discussed in Global Comment 1 (Table A), we report three configurations τ∈0.5,0.6,0.7\tau \in \\{0.5, 0.6, 0.7\\} at kc=0.2k_c = 0.2, alongside reduced kernel overhead (down from 3939\\% to 2222\\% at 64k and 55\\% at 256k, Figure 4, with the method unchanged). At τ=0.5\tau = 0.5, Triceps closely approaches or exceeds the accuracy of all sparse baselines while already delivering lower TTFT across all 128k–256k settings and on Nemotron at 64k (Figure 5, up to 1.31×1.31\times faster than FlexPrefill at τ=0.5\tau=0.5 and 1.50×1.50\times at τ=0.7\tau=0.7). Together, Tables A and B show that the initial accuracy loss came from an overly permissive block-selection threshold (kc=0.5→0.2k_c = 0.5 \to 0.2), rather than from convolution (No-Conv) or delegation (FA2-Delegation).

Thank you for the authors’ efforts in addressing my concerns. I am glad to see that the updated configuration substantially improves the accuracy, resolving one of my major concerns. However, I still have a few questions:

  1. How was the updated configuration determined? Is there any theoretical guidance for choosing kck_c and τ\tau, or were they selected through empirical search? If the latter, what data and selection criteria were used? More importantly, should we expect the selected configuration to generalize to other workloads?

  2. The contribution of convolution remains unclear to me. Although the updated configuration increases the proportion of convolution-based routing, the No-Conv ablation still achieves comparable or even better accuracy, while incurring only a modest TTFT overhead of approximately 1.4%–6.2% at 256k. This makes me question whether convolution provides sufficient practical benefits to justify its inclusion. Could the authors better clarify the role of convolution and the scenarios in which its benefits outweigh the potential accuracy trade-off?

  3. Reducing mask density is an intermediate result; TTFT is the ultimate metric of interest. I acknowledge that the proposed method selects fewer blocks, which is a positive result. However, I still have concerns about the overall TTFT improvements, particularly the baseline comparison. To my understanding, the FlexPrefill paper reports speedups starting at 16k context length and achieves up to 6.2× speedup over dense attention at 256k. In contrast, the results reported by the authors show less than 2× speedup for FlexPrefill at 256k. This is a substantial discrepancy. Could the authors explain the possible reasons for this gap, such as differences in hardware, models, or measurement protocols, and clarify whether the reported results are directly comparable?

I would be willing to raise my score if the authors could satisfactorily address these remaining questions.

We thank Reviewer yhoF for the constructive engagement and for noting that the updated configuration resolves the accuracy concern. We address your three remaining questions below:

 

1. How the Configuration (kc=0.2,τ∈0.5,0.6,0.7k_c = 0.2, \tau \in \\{0.5, 0.6, 0.7\\}) Was Determined & Generalization

We selected the configurations via a small empirical grid search whose range was bounded by theoretical guidance:

  • Theoretical guidance on the search space: τ\tau and kck_c directly control two post-softmax error bounds (Section 3.3, Appendix B.2):
    1. Convolution error threshold (τ<1\tau < 1): EconvE_{\text{conv}} takes the maximum relative ℓ2\ell_2 output error ∣Oiconv−Oidense∣2/∣Oidense∣2\\|\boldsymbol{O}^{\text{conv}} _i - \boldsymbol{O}^{\text{dense}} _i\\| _2 / \\|\boldsymbol{O}^{\text{dense}} _i\\| _2 over the last k=64k = 64 query rows (Equation 15, Appendix B.2). In causal attention, these rows attend across the full span of the c\boldsymbol{c} and d\boldsymbol{d} components and are thus most exposed to outliers from the fit. Completely erasing a head's output on a row (Oiconv=0\boldsymbol{O} ^ {\text{conv}} _ i = \mathbf{0}) yields a relative error of exactly 11, making τ=1\tau = 1 the natural upper bound where the approximation is no better than zero. By the law of cosines, requiring Econv≤τ<1E _{\text{conv}} \le \tau < 1 guarantees that every measured row stays positively aligned with Oidense\boldsymbol{O} ^ {\text{dense}} _i, with a cosine similarity of at least cos⁡θ≥1−τ2\cos\theta \ge \sqrt{1 - \tau^2} (0.87 to 0.710.87 \text{ to } 0.71 for τ∈0.5,0.6,0.7\tau \in \\{0.5, 0.6, 0.7\\}).

    2. Block-sparse dropped-mass slope (kc=0.2k_c = 0.2): 1−exp⁡(−kcτ)1 - \exp(-k_c \tau) bounds the discarded interior attention mass in block-sparse mode (MdropM_{\text{drop}}). Standard cumulative-mass methods discard 55\\%\text{ to }10\\% of attention mass (e.g., FlexPrefill γ=0.95\gamma = 0.95, PBS-Attn 0.900.90). The initial submission's kc=0.5,τ=0.4k_c = 0.5, \tau = 0.4 allowed 1−exp⁡(−0.20)=18.11 - \exp(-0.20) = 18.1\\% discarded mass, nearly 2×2\times looser than standard practice. Setting kc=0.2k_c = 0.2 bounds MdropM_{\text{drop}} to the standard 9.59.5\\%\text{ to }13.1\\% range across τ∈0.5,0.6,0.7\tau \in \\{0.5, 0.6, 0.7\\}.

 

  • Empirical grid search & out-of-distribution generalization: Within these bounds, we profiled the (τ,kc)(\tau, k_c) grid on only 4 RULER subtasks on Nemotron-8B (Appendix A.1, Figure 6). We selected kc=0.2,τ∈0.5,0.6,0.7k_c = 0.2, \tau \in \\{0.5, 0.6, 0.7\\} and then applied it unchanged to the remaining 9 RULER subtasks, to the entire LongReason benchmark (794 prompts), and to a completely different model family and architecture (Qwen3-30B-A3B, a 48-layer MoE), confirming that the configuration generalizes across models and tasks.

 

2. Clarifying the Role and Practical Benefit of Convolution

We agree that up to 256k under TP=2, the majority of Triceps' end-to-end speedup comes from the reduced attention density achieved by the multiplicative block-sparse selector (Aint≈H⊙D⊙C\boldsymbol{A} _ {\text{int}} \approx \boldsymbol{H} \odot \boldsymbol{D} \odot \boldsymbol{C}). However, because h,d,c\boldsymbol{h}, \boldsymbol{d}, \boldsymbol{c} and the 64-row gate are already computed to build the block mask, the convolution path adds nearly zero overhead (checking Econv≤τE_{\text{conv}} \le \tau). As stated, Triceps computes a convolution head at 256k on Nemotron ∼4×\sim 4\times faster than block-sparse (Figure 4). Yet the observed end-to-end gains are modest, which we attribute to two reasons:

  1. Amdahl's law on full-model TTFT: At 256k and τ=0.6\tau = 0.6, convolution is selected on 9.79.7\\%\text{—}17.2\\% of heads (Table 2); thus, the ∼4 to 14×\sim 4\text{ to }14\times per-head speedup over block-sparse and delegation kernels is diluted over the end-to-end TTFT by the remaining block-sparse and delegated heads as well as non-attention layers (e.g., FFN, MoE).

  2. Cross-GPU load-balancing under TP=2 (Section 6, Limitations; Figure 7, Appendix A.2): With static TP=2 head partitioning (splitting heads 0—150\text{—}15 and 16—3116\text{—}31 across the two GPUs) and a per-layer AllReduce barrier, each layer's wall-clock latency is determined by the critical path (the slower GPU). As Figure 7 visually shows, the three execution modes are unevenly distributed across heads within a layer: whenever convolution heads fall on the faster GPU while the other GPU executes more block-sparse or delegated heads, their time savings are hidden behind the AllReduce synchronization barrier.

The main benefit of the convolution path is its asymptotic O(Nlog⁡N)\mathcal{O}(N \log N) scaling compared with O(ρN2)\mathcal{O}(\rho N^2) for the block-sparse path. Between 64k and 256k (4×4\times context length), per-head convolution time grows by only 2.7×2.7\times (1.0 ms→2.7 ms1.0\text{ ms} \to 2.7\text{ ms}), whereas block-sparse time grows by ∼14×\sim 14\times (0.7—0.8 ms→9.7—12.6 ms0.7\text{—}0.8\text{ ms} \to 9.7\text{—}12.6\text{ ms}, Figure 4).

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of Revisions, Accuracy–Latency Curves, and Component Ablations

We thank all four reviewers for their constructive feedback. We have updated the manuscript and conducted targeted ablations.

First we address the shared questions across reviews:

  1. Accuracy–Latency Trade-off & Multi-Budget Evaluation (Reviewers yhoF, s3Z1, QJ5g, xvCE):

In the initial submission, the tested configuration (kc=0.5,τ=0.4k_c = 0.5, \tau = 0.4) allowed up to 1−exp⁡(−0.5×0.4)=18.11 - \exp(-0.5 \times 0.4) = 18.1\\% discarded tail-row attention mass in block-sparse mode, which was overly permissive for some of the tasks. Setting kc=0.2k_c = 0.2 and sweeping τ∈0.5,0.6,0.7\tau \in \\{0.5, 0.6, 0.7\\} tightens the block-sparse drop budget to 9.59.5\\%\text{—}13.1\\% while raising the convolution admission threshold τ\tau, simultaneously improving accuracy and routing a larger share of heads to the fast convolution path (updated Tables 1, 2, 5, 6 and Figures 3–7).

Independently, we optimized our approximation and routing kernels without changing the method itself, reducing upfront overhead to 2222\\% of prefill GPU time at 64k, 1010\\% at 128k, and 55\\% at 256k (Figure 4). Triceps now delivers end-to-end TTFT speedups starting at 64k. At τ=0.5\tau = 0.5, it closely approaches or exceeds the accuracy of all sparse baselines at their reference budgets while already achieving lower TTFT across all 128k–256k settings and on Nemotron at 64k (Table A, Figure 5, Table 6).


2. Visual Motivation & Block-Mask Comparison (Reviewers yhoF, s3Z1, QJ5g, xvCE):

We added a pipeline diagram in Figure 1 (Section 3) to depict a top-level overview of our method. We also included a 64k block-mask example comparing our method against exact offline oracle masks (9595\\% to 8080\\% mass) and prior methods in Figure 2 (Section 4). In our tri-component decomposition Aint≈H⊙D⊙C\boldsymbol{A}_{\text{int}} \approx \boldsymbol{H} \odot \boldsymbol{D} \odot \boldsymbol{C}, C\boldsymbol{C} captures global keys (vertical heavy-hitters), D\boldsymbol{D} captures relative-distance patterns induced by RoPE (slashes), and H\boldsymbol{H} directly estimates for each query row the number of selected blocks before softmax. Figure 2 illustrates for one head how combining all three yields sparser masks (8.28.2\\% density with τ=0.5\tau = 0.5) than prior heuristics because it selects blocks when H\boldsymbol{H}, D\boldsymbol{D} and C\boldsymbol{C} constructively reinforce (intersection) and discards blocks that one or two of those components cancel. As a result, our method selects localized segments of verticals and slashes, whereas MInference (53.853.8\\%) and FlexPrefill (17.217.2\\%) extrapolate full-length verticals and diagonals from a query subset and take their union. PRISM (25.525.5\\%) takes the union of low- and high-frequency masks without cross-frequency cancellation, and block mean-pooled heuristics (PRISM, and PBS-Attn 27.227.2\\% after intra-segment key permutation) overshoot mask density.


3. Component Ablations — Convolution Branch & Delegation Target (Reviewers yhoF, QJ5g, xvCE):

Triceps' three execution modes play complementary roles: convolution executes regular vertical–slash heads directly from H⊙D⊙C\boldsymbol{H} \odot \boldsymbol{D} \odot \boldsymbol{C} in O(Nlog⁡N)\mathcal{O}(N \log N) (2.7 ms2.7\text{ ms}/head on Nemotron at 256k), block-sparse uses the fitted decomposition as an ultra-sparse block selector (2.22.2\\%\text{—}18.7\\% density, 9.7—12.6 ms9.7\text{—}12.6\text{ ms}/head, ∼4×\sim 4\times slower than convolution), and delegation (36.0—39.4 ms36.0\text{—}39.4\text{ ms}/head, ∼14×\sim 14\times slower, Figure 4) preserves accuracy when needed (e.g. retrieval heads Appendix A.3). Table B (Table 4, Appendix A.5) isolates both design choices at kc=0.2,τ=0.6k_c = 0.2, \tau = 0.6: (i) disabling convolution (Triceps No-Conv), which redirects convolution heads to block-sparse and increases end-to-end 256k TTFT, measured over the full prefill, by up to 6.26.2\\% (25.15s→26.70s25.15\text{s} \to 26.70\text{s} on Nemotron LongReason; 31.77s→33.00s31.77\text{s} \to 33.00\text{s} on RULER) at comparable accuracy; and (ii) replacing FlexPrefill delegation with exact dense FlashAttention-2 (Triceps FA2-Delegation), which still achieves a 1.74× to 2.36×1.74\times\text{ to }2.36\times speedup over Dense at 256k, confirming that the acceleration comes from Triceps' convolution and block-sparse modes.

Table A: Accuracy (%) and End-to-End TTFT (s) Across Operating Points (kc=0.2k_c = 0.2, TP=2)

ModelMethodLR Acc 64k / 128k / 256kLR TTFT (s) 64k / 128k / 256kRULER Acc 64k / 128k / 256kRULER TTFT (s) 64k / 128k / 256k
Qwen3-30B-A3BDense (FA2)76.57 / 73.43 / 73.306.16 / 22.67 / 80.0692.85 / 89.46 / 84.826.11 / 22.57 / 79.99
FlexPrefill76.01 / 71.91 / 72.294.06 / 10.77 / 28.9889.61 / 86.15 / 84.575.57 / 16.51 / 45.82
PBS-Attn75.94 / 72.41 / 73.684.98 / 13.47 / 35.1989.73 / 85.06 / 82.045.63 / 16.88 / 48.39
PRISM75.81 / 74.06 / 73.304.32 / 12.66 / 33.8290.54 / 86.43 / 82.965.74 / 17.99 / 52.86
Triceps (τ=0.5\tau=0.5)74.56 / 73.43 / 72.044.18 / 9.68 / 25.0290.58 / 86.79 / 83.925.72 / 15.26 / 41.55
Triceps (τ=0.6\tau=0.6)74.43 / 71.66 / 71.544.10 / 9.39 / 24.1090.13 / 86.70 / 83.415.58 / 14.91 / 40.48
Triceps (τ=0.7\tau=0.7)70.65 / 70.40 / 68.393.99 / 9.21 / 23.4290.95 / 85.82 / 82.755.43 / 14.49 / 39.43
Nemotron-8BDense (FA2)45.97 / 45.66 / 45.105.25 / 17.48 / 63.7975.74 / 71.58 / 65.545.23 / 17.43 / 63.79
FlexPrefill45.80 / 44.96 / 44.204.54 / 11.55 / 35.5973.38 / 70.64 / 62.164.90 / 13.24 / 41.81
PBS-Attn44.30 / 43.83 / 43.804.91 / 12.57 / 36.0472.49 / 71.06 / 61.745.12 / 14.55 / 44.77
PRISM45.50 / 44.90 / 45.004.38 / 12.15 / 37.2173.58 / 71.17 / 62.695.12 / 15.09 / 49.10
Triceps (τ=0.5\tau=0.5)45.97 / 45.97 / 43.704.18 / 9.97 / 27.1674.62 / 72.44 / 63.114.66 / 11.78 / 33.71
Triceps (τ=0.6\tau=0.6)45.21 / 44.21 / 44.214.03 / 9.48 / 25.1574.54 / 71.30 / 61.974.55 / 11.32 / 31.77
Triceps (τ=0.7\tau=0.7)44.71 / 43.70 / 41.813.91 / 8.88 / 23.6973.54 / 71.15 / 62.764.41 / 10.81 / 30.13

Table B: Component Ablations (No-Conv and FA2-Delegation at kc=0.2,τ=0.6k_c = 0.2, \tau = 0.6)

ModelVariantLR Acc (%) 64k / 128k / 256kLR TTFT (s) 64k / 128k / 256kRULER Acc (%) 64k / 128k / 256kRULER TTFT (s) 64k / 128k / 256k
Nemotron-8BDense (FA2)45.97 / 45.66 / 45.105.25 / 17.48 / 63.7975.74 / 71.58 / 65.545.23 / 17.43 / 63.79
Triceps (Full)45.21 / 44.21 / 44.214.03 / 9.48 / 25.1574.54 / 71.30 / 61.974.55 / 11.32 / 31.77
Triceps (No-Conv)45.21 / 44.58 / 43.834.06 / 9.73 / 26.7074.12 / 72.19 / 64.184.57 / 11.68 / 33.00
Triceps (FA2-Delegation)45.34 / 44.71 / 42.204.10 / 10.54 / 30.9574.19 / 70.90 / 61.744.46 / 11.69 / 34.51
Qwen3-30B-A3BDense (FA2)76.57 / 73.43 / 73.306.16 / 22.67 / 80.0692.85 / 89.46 / 84.826.11 / 22.57 / 79.99
Triceps (Full)74.43 / 71.66 / 71.544.10 / 9.39 / 24.1090.13 / 86.70 / 83.415.58 / 14.91 / 40.48
Triceps (No-Conv)73.93 / 72.42 / 69.654.01 / 9.33 / 24.4490.55 / 86.02 / 83.655.55 / 15.06 / 41.04
Triceps (FA2-Delegation)72.50 / 71.16 / 67.764.27 / 11.34 / 33.8990.05 / 86.51 / 83.685.26 / 15.34 / 45.98
Area_Chair_LG9hOpenReview ↗

Response to the rebuttal.

Dear reviewers,

The authors have submitted rebuttal responses. Please read through their replies and update your review as you see fit. Thank you for your work.

AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.