acceptodds
Under review as a conference paper at ICLR 2027

HyLin: Modality-Aware Hybrid Attention for Efficient Long-Context Vision–Language Models

Abstract

While large vision-language models scale to ultra-long sequences, the quadratic complexity of softmax attention becomes a severe bottleneck for both multimodal understanding and video generation. Existing inter-layer hybrids retain expensive full-attention layers, uniform intra-layer hybrids ignore modality-specific structure, and sparse video methods often preserve the full KV cache. We argue that efficient attention must combine global context compression with fine-grained local retrieval. Our teacher-attention analysis reveals substantial heterogeneity across modalities, heads, and depth: visual rows are often diffuse and compressible, whereas text rows favor local retrieval, and heterogeneous cross-modal mass makes universal replacement unsafe. We therefore propose \model, a Hybrid Linear architecture centered on Hybrid Linear-coupled Window Attention (HLWA). In replaceable VLM layers, HLWA routes visual queries through a constant-state global Linear Attention stream and applies Sliding Window Attention (SWA) directly to the original text queries, keys, and values before restoring multimodal row order. Theory-guided calibration and downstream sensitivity identify high-risk layers, with 25% retained as full attention. A four-stage curriculum performs operator alignment, layer selection, vision-language distillation, and supervised fine-tuning. For video DiTs, we instantiate the same global-linear/local-exact principle with a distinct bidirectional operator. On the reported benchmarks, \model-7B closely tracks its full-attention teacher while delivering prefill and decode speedup together with memory reduction at 64K tokens; the video-DiT variant accelerates HunyuanVideo-13B by with competitive VBench quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid attention, linear attention, vision-language models, long-context modeling, video generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TbK3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HyLin, which converts a pretrained VLM (Qwen2.5-VL-3B/7B) into a hybrid model. In 75% of the layers, attention is replaced by HLWA: visual tokens are mixed by Gated DeltaNet, text tokens use softmax attention over the text prefix only, and there is no cross-modal attention within these layers. The remaining 25% keep full attention. They are selected by combining operator error with single-layer benchmark sensitivity, and the converted model is recovered through layer-wise calibration, KD, and SFT. The paper reports near-teacher accuracy on image and video benchmarks, 2.1×/3.6× prefill/decode speedups and 3.4× memory reduction at 64K tokens, and an extension to HunyuanVideo-13B with a 1.94× speedup.

Critical strengths
  1. The motivation is sound. Visual tokens dominate long-video and high-resolution inputs, so allocating computation by modality is a reasonable direction.
  2. The design is practical. For text-only inputs, HLWA reduces to full attention, so language ability is largely preserved, and removing visual KV caches from 75% of the layers yields real decode-time and memory savings.
  3. The evaluation is broad, covering two model scales, image and video understanding, several linear mixers, positional-encoding variants, different FA ratios, and multiple layer-selection strategies.
Critical weaknesses
  1. Limited novelty. Isolating modalities and confining cross-modal interaction to sparse full-attention layers resembles interleaved cross-modal designs (e.g., Flamingo, LLaMA-3.2-Vision, mPLUG-Owl3), which are not discussed. The 75% GDN / 25% softmax hybrid recipe with distillation is also close to InfiniteVL, which is cited but not compared against experimentally; concurrent work MOSAIC (arXiv 2607.09029) follows a similar direction and merits discussion.
  2. Layer selection is under-analyzed. The benchmark signal measures only single-layer replacement, ignoring interactions when 27 of 36 layers are replaced together. The effect of the selection benchmarks on later stages is not studied, the selection ablations use a simplified direct-SFT protocol instead of the full pipeline, and the selection benchmarks (e.g., MMMU, DocVQA, VideoMME) overlap with the test sets.
  3. The benefit of visual linearization is not isolated. Table 1a is confounded: the proposed variant uses full-prefix text attention, while the baselines use sliding windows. A control with modality isolation but visual softmax attention is missing. Capability-level analysis is also lacking: dense-text tasks degrade consistently (InfoVQA -2.3 at both scales), and grounding/hallucination are not evaluated.
  4. Accuracy and efficiency are measured separately. Accuracy comes mostly from short-context benchmarks, while speedups come from a separate 64K profiling run, and the longest-context benchmark shows the largest drop (LongVideoBench -1.5). Joint accuracy–latency results on long-context benchmarks, plus the token composition used in profiling, are needed.
Questions for authors
  1. How does the proposed method differ from, and compare experimentally against, InfiniteVL and interleaved cross-modal architectures?
  2. How sensitive is the final model, after the full four-stage pipeline, to the choice of selection benchmarks? Are the selection splits disjoint from the reported test sets?
  3. Can the authors report a control with modality isolation but softmax attention on the visual side, as well as grounding and hallucination results?
  4. Can the authors report accuracy and measured latency/memory jointly on long-context benchmarks, and clarify the token composition and settings used in the 64K profiling?
1 comment

Thank you for the supplementary experiment. After rebuttal, I tend to view HyLin as an example of the existing FA → hybrid transformation paradigm, rather than a new transformation route. This type of work usually follows the same process:

  1. Select linear/sparse candidate operators (GDN, Mamba2, SWA, KDA, JetBlock, etc.);
  2. Use a certain scoring rule (PPL/KL/CE Loss/Benchmark-Accuracy) to retain a few FA layers;
  3. Use (Blockwise/Global) Distillation or SFT to restore performance.

Puzzle、Jet-Nemotron、Distill-then-Replace、InfiniteVL, Most of them belong to this route. In this context, although the paper proposes the design of modal routing, it is more like an operator selection within the transformation layer rather than a new transformation framework. Tab. R2/R3 indicates that this selection is helpful under matching settings, which is useful, but still biased towards increments.

I suggest the author to compare and sort out the innovation and contribution of this method to the academic community.

Reviewer_QXd7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HyLin, a modality-aware hybrid attention framework for efficient long-context vision-language models. Its core module, HLWA, routes visual queries through Gated DeltaNet linear attention while retaining causal softmax attention over the full text prefix for text queries. Full-attention layers are selected based on calibrated operator error and downstream task sensitivity. The authors also extend the general principle of combining global linear mixing with local attention to video diffusion Transformers. Experiments demonstrate practical inference speedups, reduced memory usage, and task performance that is broadly comparable to the full-attention teacher.

Critical strengths

1 The method is well motivated. Distinguishing the computational requirements of visual and text queries, together with selective layer retention, is a reasonable improvement over uniform attention replacement.

2 The empirical evaluation is broad. The paper evaluates image understanding, video understanding, and video generation, with ablations, efficiency measurements, and detailed training configurations.

Critical weaknesses

1 The theoretical analysis does not establish the reliability of multi-layer replacement. The error analysis primarily concerns single-layer replacement under fixed teacher inputs. The derived bounds do not directly constrain the error of the learned Gated DeltaNet operator or guarantee stable error propagation when multiple layers are replaced. The effectiveness of layer selection therefore remains primarily empirically supported.

2 Performance preservation is not fully disentangled from additional training. HyLin undergoes multi-stage distillation and fine-tuning, but the continued-SFT control for the full-attention teacher covers only MME and OCRBench on the 3B model. Meanwhile, the 7B model drops from 58.6 to 54.1 on MMMU and from 2347 to 2243 on MME, indicating that performance preservation is not uniformly strong across tasks.

3 The scope of the efficiency claims needs clearer qualification. Retained full-attention layers and full-prefix text attention preserve quadratic terms in the overall computational complexity. The reported speedups therefore demonstrate practical gains under particular context lengths and modality compositions, rather than an asymptotic reduction in the complexity of the complete architecture. Furthermore, the video-generation extension uses a different operator, and its independent novelty and generalizability could be established more clearly.

Questions for authors

1 What is the core algorithmic novelty of HyLin over existing inter-layer hybrids, intra-layer hybrids, and sparse video attention methods? What precisely unifies the two operator designs?

2 How well do single-layer replacement errors and downstream sensitivity predict the final performance after simultaneous multi-layer replacement? Can the authors provide empirical evidence?

3 Can the authors provide matched full-attention continued-SFT controls across the main benchmarks for both 3B and 7B models to disentangle the effects of attention replacement from additional training?

4 How do the retained full-attention layers and full-prefix text attention affect efficiency as the proportion of text tokens increases? Under which context lengths and modality compositions do the reported speedups hold?

5 For video generation, HyLin improves VBench Final by only 0.53 points over the faster SVG2 baseline. How do the authors justify this quality gain relative to the additional inference time, and does the advantage persist across a broader range of generation scenarios?

Reviewer_NjrHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HyLin, a hybrid linear architecture for efficient long-context VLMs built on a modality-routed hybrid attention module (HLWA). HLWA processes quries from visual inputs with Gated DeltaNet and text queries with causal softmax attention over the full text prifix tokens, while 25% of layers are kept as full attention based on operator error and downstream sensitivity. HyLin is started from Qwen2.5-VL through a four-stage training pipeline. The authors apply the same core idea of HyLin to Video DiTs and achieves performance close to its original full-attention model with speedup and memory savings on both VLM and video DiTs.

Critical strengths

The paper includes diverse and extensive experiments, especially ablations, which would give useful insights to other researchers in this field. Results on video generation, not only on QA tasks, make this even more convincing.

Critical weaknesses
  • I have some concerns about a way of writing this paper. I think the paper would be much better if the authors read through the whole paper again and revise it. Below are some examples, and there are also other parts that are not reader-friendly.
    • The last paragraph of the introduction is overloaded. It contains both the two findings and the explanation of the proposed method. It may be better to split it into two paragraphs.
    • From the view of paper structure, related work and preliminaries are placed n the appendix, but I think they should come before the method section. Since the paper proposes a new combination of existing methods, the related work needs to be clarified first. There are several such points, but in particular, in the first paragraph of Sec. 2.1, what are the "teacher diagnostics"? This analysis is not introduced or described anywhere before.
    • The same concept is called by different names, such as behaviors, structures, and interaction pattern... This is confusing. The authors should use one term and define it when it first appears. For example, the first sentence of Sec. 2 says "... exhibit different structures," which is the first mention of "structures," but it is unclear what these structures refer to. It should at least mention this so that readers can quickly understand it.
  • The only evidence for the findings is the qualitative illustration in Fig. 2(a). I think the authors should show a quantitative analysis supporting that visual rows are diffuse.
  • The experiments use only Qwen2.5-VL-3B/7B as the backbone. This model is somewhat old, and it is necessary to show the trend (or any changes) on recent models. Also, all compared methods are from 2025 or earlier.
  • Visual token efficiency is getting a lot of attention in the paper, so I think the paper should compare with existing methods such as LongVU (Shen et al., 2025), LLaVA-Mini (Zhang et al., 2025c), VL-Cache (Tu et al., 2025), and more in terms of how much efficiency is improved. The authors only need to measure HyLin with the evaluation protocols of these papers and compare the results in a table, so I think this is doable.
Questions for authors
  • In page 1, lines 42-48, which size of Qwen2.5-VL is used for the "91 GB of memory"? Also, what is the setting of the "short video clip" (length, resolution, and frame rate)? These numbers need more details.
  • What about the audio modality? Do you think the proposed method can also help?
  • Please see the weakness section. I would like to hear from the authors and I hope the rebuttal can resolve my concerns.
1 comment
Reviewer_NjrHOpenReview ↗

Thanks

Thank you for the detailed response and the additional experiments. I will maintain my score, as my concern about the writing quality of the paper still remains. That said, I would like to note that my overall assessment of this paper is not entirely negative. If a borderline option were available, I would choose it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.