- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces INTRA, which establishes cross-layer information propagation paths by alternating between "Scatter" and "Gather" operations on statically sparse groups. It also features a FlashAttention-style kernel designed to handle non-contiguous token access. The authors analyze coverage and connectivity within this grouped framework and, by employing LoRA adaptation, evaluate the approach’s quality and efficiency across image generation, language tasks, and video generation.
Critical strengths
-
Static linking avoids input-dependent pattern search, while structured indexing and token loading further realize the sparse pattern into an executable GPU kernel. The paper considers both communication architecture and practical runtime overhead.
-
In the 2048×2048 FLUX.1-dev experiments, generation time was reduced from 66.78 seconds to 43.72 seconds, an improvement of approximately 1.53×, and outperformed the image quality metrics of the compared CLEAR configuration. This supports the practical value of the method in the tested settings.
-
Ablation of Scatter/Gather combination, number of groups, switching interval, and global layer configuration provides support for design choices. The authors also explicitly distinguish between theoretical connectivity and task quality and disclose the differences between retrieval and timing configurations.
Critical weaknesses
-
The authors explicitly state that the block-level retrieval configuration in Table 4 differs from the token-level timing configuration. While this disclosure is both necessary and valuable, it cannot substitute for performance benchmarking based on the actual retrieval configuration used. Consequently, the existing experiments separately support the claims that "one configuration offers superior retrieval capability" and "another configuration yields lower attention latency," yet no evidence has been established to show that both hold true simultaneously. Furthermore, the operator latency and SCROLLS scores presented in Table 3 require a more direct alignment of configurations and a joint evaluation. This represents a significant gap in the evidence supporting the conclusions regarding the efficiency-quality trade-offs in the language domain.
-
In Table 8, the computational costs of the various methods differ substantially. For instance, at a 16K context length, the reported computational costs for INTRA, XAttention, and MInference are approximately 12.5%, 28%, and 58% of the dense baseline, respectively. While varying computational budgets do not inherently constitute an unfair comparison, the fact that INTRA underperforms in five out of the six SCROLLS tasks makes it difficult to discern whether the observed outcomes stem from superior methodological design or simply from selecting an operating point that requires less computation at the cost of lower quality. A comparison that controls for quality, or an analysis across a range of computational budgets, is necessary to determine the true advantages of the proposed trade-offs.
Questions for authors
-
How much speedup does the actual configuration that generates the paper quality score achieve? The authors have stated that NIAH retrieval and timing use different configurations. Please directly report the actual latency of the block-level configuration that achieved the 0.923 NIAH score; also clarify whether the SCROLLS scores in Table 3 correspond to the configuration used in the speed measurement in the table.
-
Were the MInference and XAttention latency in Table 3 obtained independently under the same conditions as INTRA? Please clarify whether these data were directly measured or converted from literature results, and explain whether all three used the same input shape, precision, and timing range, including their respective necessary online overhead. This directly determines whether "14.94×, 2.23×" constitute a valid direct comparison.
4 comments
W2 — Computation budget versus quality
Reviewer — Critical Weakness 2
In Table 8, the computational costs of the various methods differ substantially. For instance, at a 16K context length, the reported computational costs for INTRA, XAttention, and MInference are approximately 12.5%, 28%, and 58% of the dense baseline, respectively. While varying computational budgets do not inherently constitute an unfair comparison, the fact that INTRA underperforms in five out of the six SCROLLS tasks makes it difficult to discern whether the observed outcomes stem from superior methodological design or simply from selecting an operating point that requires less computation at the cost of lower quality. A comparison that controls for quality, or an analysis across a range of computational budgets, is necessary to determine the true advantages of the proposed trade-offs.
We agree that Table 3 does not prove a quality-matched SCROLLS advantage over XAttention or MInference. But the submitted ablations show that connection placement matters beyond the number of retained entries:
| Source | Attention allocation | Overall retained density | Mean NIAH |
|---|---|---|---|
| Table 5 | Windows in four selected layers | 0.069 | 0.028 |
| Table 5 | Four scatter layers | 0.123 | 0.923 |
| Table 5 | Dense attention in four selected layers | 0.185 | 1.000 |
| Table 16 | Wider local windows | 0.234 | 0.222 |
Table 5 fixes the four selected layer positions and adaptation budget. Table 16 compares different allocations. These are not matched-FLOPs experiments. A new 16K panel provides an additional matched-training-budget control:
| Method (16K, 460 examples) | Score |
|---|---|
| INTRA Aligned density | 0.6699 |
| MoBA | 0.2560 |
The two settings use the same reported density (0.1255), rank-16 LoRA, learning rate, 1,200 steps, data, and evaluator. This supports the value of the tested connectivity allocation. It is not a new SCROLLS result or proof of matched-quality latency dominance.
For images, submitted Table 2 offers a directly measured quality–latency comparison:
| 2048×2048 FLUX.1-dev | Generation time (s) | FID ↓ |
|---|---|---|
| CLEAR (r=8) | 47.81 | 32.46 |
| INTRA | 43.72 | 15.67 |
Q2 — Provenance and correction of Table 3(a)
Reviewer — Question 2
Were the MInference and XAttention latency in Table 3 obtained independently under the same conditions as INTRA? Please clarify whether these data were directly measured or converted from literature results, and explain whether all three used the same input shape, precision, and timing range, including their respective necessary online overhead. This directly determines whether “14.94×, 2.23×” constitute a valid direct comparison.
The XAttention and MInference latencies in Table 3(a) were literature-derived estimates. Specifically, we divided our measured FlashAttention-2 latency by the normalized speedup ratios reported in Figure 4 of XAttention. For example, at 8K, the estimated latencies were 2.86/1.5 = 1.91 ms for XAttention and 2.86/0.2 = 14.30 ms for MInference.
We have now independently benchmarked Dense, XAttention, MInference, and INTRA on the same A100. For transparency, we report the absolute latency of every method, including both INTRA's token-level G=8 operator and the newly measured block-level G=8 operator used for SCROLLS. Both have nominal 1/8 grouped attention density, but use different masks.
The submitted INTRA operator used token-level Scatter with G=8, while SCROLLS quality used block-level Scatter with G=8 and 256-token blocks. Both have nominal 1/8 grouped attention density. We have now measured the actual SCROLLS block-level G=8 operator using our kernel.
| Attention-operator latency (ms) | 8K | 16K | 32K |
|---|---|---|---|
| Dense | 5.37 | 21.00 | 82.58 |
| XAttention (S=16) | 3.57 | 8.42 | 26.64 |
| MInference (v0.1.6) | 39.54 | 52.27 | 72.00 |
| INTRA, token-level G=8 | 0.51 | 1.75 | 6.5 |
| INTRA, SCROLLS block-level G=8 | 0.50 | 1.8 | 6.6 |
Dear Reviewer, thank you for your helpful comments! Please let us know if you have any further questions or concerns that we can address before the discussion period ends.
Response to Reviewer
We thank the reviewer for their careful evaluation and constructive feedback. We address each of the reviewer's concerns below.
W1 / Q1 — Latency of the quality-evaluated configuration
Reviewer — Critical Weakness 1
Reviewer — Question 1
Thank you for pointing this out. We did not clearly distinguish the NIAH, SCROLLS, and token-level kernel settings. NIAH uses four block-level Scatter layers (block size 256, G=2) and 28 window layers. This is the configuration that obtains 0.923 mean NIAH accuracy in Table 4(a). Table 4(b) reports the sum of 32 attention-layer latencies for this schedule and we also measured full-model prefill speedup with the NIAH configuration on A100:
Thus, the original NIAH configuration has both a measured quality result at 4K–32K and a measured prefill speedup within that range.
SCROLLS uses block-level Scatter with G=8, rather than NIAH’s G=2. Table 3(a) originally timed a token-level G=8 operator. The two settings have the same nominal 1/8 group sparsity, but different attention masks. We now additionally measure the SCROLLS block-level G=8 operator with our kernel and it matches with the table reported speendup. We will state these distinctions clearly in the revised tables and text.