acceptodds
Under review as a conference paper at ICLR 2027

SparseTransfer: Does Calibration-Free Sparse Attention Generalize Across Domains?

Abstract

Training-free sparse attention methods are increasingly evaluated as if they were universal: a mechanism validated in one domain is often assumed to transfer to others without recalibration. We test this assumption directly by porting two independently-published, training-free block selection mechanisms EntropyInfer, an entropy-guided adaptive budgeting method, and a cumulative-energy-based thresholding method into a shared attention framework (XAttention), configuring each to match XAttention's own tuned baseline operating density, and evaluating both against that baseline across two domains: long-context language modeling (RULER, Llama-3.1-8B, context lengths 4K–128K) and long-video question answering (Video-MME, Qwen2-VL-7B, 900 videos). We observe a striking, consistent asymmetry: both mechanisms transfer far more faithfully to video-understanding (97–101 of baseline accuracy) than to language (77–99.6 of baseline, with the largest gap at the longest context length), and this pattern holds across both tested mechanisms, with the cumulative-energy method consistently more robust than EntropyInfer in both domains. We verify our findings are not artifacts of implementation error through multiple independent checks, including direct inspection of the patched selection function and measured selection density. Our results suggest that "calibration-free" claims in the sparse attention literature are domain-contingent rather than universal, and that block-selection quality, not merely computational budget is a meaningful, underexplored axis of transfer failure

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, cross-domain generalization, training-free methods, long-context language models, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RJQpOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to investigate how well training-free sparse attention methods transfer their performance across domains. The authors evaluate two modifications of XAttention, based on EntropyInfer and DynSparse, on RULER and Video-MME. The results show that both methods underperform the original XAttention on RULER, with EntropyInfer showing the largest degradation. On Video-MME, the results are mixed: EntropyInfer performs slightly worse than XAttention, while DynSparse achieves comparable performance.

Critical strengths

None

Critical weaknesses
  • The paper looks like low-effort work. The paper is built around a single main finding, which is repeated multiple times throughout the manuscript without adding much new analysis. This makes the paper look artificially extended. Additionally, the legend in Figure 3 and the presentation of Tables 3 and 4 look poorly edited and give the impression that they were generated without careful human review.

  • The experimental setup does not convincingly address the main research question. The paper claims to investigate whether training-free sparse attention methods transfer their performance across domains. However, the authors only test modifications of XAttention, specifically replacing its block-selection mechanism with those from EntropyInfer and DynSparse, rather than evaluating the original methods. Moreover, both modifications perform worse than XAttention on RULER, while on Video-MME, EntropyInfer performs slightly worse and DynSparse achieves similar accuracy. This raises questions about the validity of the experimental setup and whether it is suitable for answering the main research question.

Questions for authors

None

Reviewer_8HfXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether training-free sparse attention mechanisms retain their effectiveness when transferred without modification across implementations and domains. The authors integrate two existing mechanisms, EntropyInfer and DynSparse, into as a replacement for XAttention's block-selection interface and compare them against XAttention's tuned baseline. Experiments cover long-context language modelling and long-video understanding. The authors find performance is retrained in video and not language modelling. They claim this as evidence that calibration-free transfer is domain-dependent.

Critical strengths
  • Explores training free drop in replacements for XAttention. As far as my knowledge this is not yet explored.
  • An interesting finding that the performance on video understanding is unaffected while language modelling performance drops.
Critical weaknesses
  1. Weak motivation and unclear research question. The Introduction and Related Work do not adequately motivate why sparse attention mechanisms should transfer across architectures and domains without recalibration, particularly given the intuition underlying the No Free Lunch theorem that no single method is optimal across all settings. Although the paper motivates sparse attention through the computational expense of attention, it does not clearly specify whether the relevant bottleneck is inference computation, GPU memory requirements, or training and calibration costs, nor how the proposed cross-domain transfer evaluation addresses these concerns.

  2. Limited evidence for generalisability. The experimental setup (Section 3.1) evaluates only two selection mechanisms across two domains, using one model per domain, making it difficult to determine whether the observed asymmetry is a general phenomenon or specific to these configurations. Additional model families, architectures or domains would provide stronger support for the broader claims.

  3. Insufficient explanation of the observed domain asymmetry. Although Tables 1–2 and Section 4.3 demonstrate different accuracy recovery between language and video, the paper provides limited insight into why these sparse attention mechanisms behave differently across domains. Analysing attention maps, block-selection patterns, probes could help explain what XAttention captures in each domain and why alternative selection mechanisms achieve nearly identical performance on video.

Questions for authors
  1. Baseline selection (Section 3.1): Why was XAttention chosen as the fixed implementation framework, and why are comparisons restricted to its tuned baseline rather than including full attention or other relevant sparse attention approaches?

  2. Domain asymmetry (Tables 1–2, Section 4.3): What differences in attention or block-selection patterns explain why EntropyInfer and DynSparse perform substantially better relative to XAttention on video than on text? Could representative attention maps or block-selection analyses help explain this behaviour?

  3. Motivation for transfer (Introduction, Section 2): What is the practical justification for expecting sparse attention mechanisms to transfer across architectures, model scales and domains without recalibration, given their different characteristics and hyperparameter requirements?

  4. Presentation (Section 4, Figure 2 and Tables 3–4): Figure 2 and Tables 3–4 are not clearly referenced or discussed in the accompanying text. Could the authors integrate these results into the discussion and clarify their relevance to the main findings?

Reviewer_jgxBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SparseTransfer, an empirical study of whether training-free sparse attention mechanisms generalize across implementations and domains. The authors integrate EntropyInfer and DynSparse into XAttention and evaluate them on long-context language tasks and video understanding. The results show larger accuracy degradation on language tasks, particularly for EntropyInfer, suggesting that transferring selection mechanisms may require additional validation and adaptation.

Critical strengths

The paper investigates a relevant practical question: whether sparse attention methods remain effective outside their original evaluation settings. Using a shared attention framework is a useful attempt to control implementation differences when comparing selection mechanisms. The paper provides results across six context lengths, detailed task-level breakdowns, and the full Video-MME benchmark. The appendix also discloses budget adjustments and evaluation limitations.

Critical weaknesses
  • The evaluation scope is limited. Only two transferred mechanisms are tested, with XAttention serving as the baseline. Each domain uses one model and one benchmark. This provides preliminary evidence, but is insufficient to support broad conclusions about the cross-domain generalization of calibration-free sparse attention methods.

  • The findings remain largely descriptive. The paper reports performance differences but provides limited analysis of why transfer succeeds or fails. Task-level results reveal substantial variation, including severe degradation on some retrieval tasks and improvements on variable tracking, but these patterns are not explained through attention statistics, retained attention mass, or missed-block analysis. Deeper mechanism analysis would make the findings more informative for future method design.

  • The calibration-free transfer claim needs clarification. Appendix C states that EntropyInfer’s budget was recalibrated to approximately 20% after the initial implementation produced a density of around 2%. Although this adjustment may be reasonable, the reported results evaluate a budget-adjusted version. The authors should clearly distinguish direct transfer without retuning from transfer after budget adaptation.

Questions for authors
  1. Does the observed language–video asymmetry hold for additional selection mechanisms, model families, and benchmarks beyond the two mechanisms and two model–benchmark pairs tested here?
  2. Could the authors provide mechanism-level analysis explaining why transfer succeeds or fails, such as retained attention mass or missed-block patterns? What explains the different effects on retrieval and variable-tracking tasks?
  3. Could the authors separately report EntropyInfer’s results before and after budget recalibration? How should the adjustment described in Appendix C be reconciled with the claim of calibration-free transfer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.