- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes HyLin, which converts a pretrained VLM (Qwen2.5-VL-3B/7B) into a hybrid model. In 75% of the layers, attention is replaced by HLWA: visual tokens are mixed by Gated DeltaNet, text tokens use softmax attention over the text prefix only, and there is no cross-modal attention within these layers. The remaining 25% keep full attention. They are selected by combining operator error with single-layer benchmark sensitivity, and the converted model is recovered through layer-wise calibration, KD, and SFT. The paper reports near-teacher accuracy on image and video benchmarks, 2.1×/3.6× prefill/decode speedups and 3.4× memory reduction at 64K tokens, and an extension to HunyuanVideo-13B with a 1.94× speedup.
Critical strengths
- The motivation is sound. Visual tokens dominate long-video and high-resolution inputs, so allocating computation by modality is a reasonable direction.
- The design is practical. For text-only inputs, HLWA reduces to full attention, so language ability is largely preserved, and removing visual KV caches from 75% of the layers yields real decode-time and memory savings.
- The evaluation is broad, covering two model scales, image and video understanding, several linear mixers, positional-encoding variants, different FA ratios, and multiple layer-selection strategies.
Critical weaknesses
- Limited novelty. Isolating modalities and confining cross-modal interaction to sparse full-attention layers resembles interleaved cross-modal designs (e.g., Flamingo, LLaMA-3.2-Vision, mPLUG-Owl3), which are not discussed. The 75% GDN / 25% softmax hybrid recipe with distillation is also close to InfiniteVL, which is cited but not compared against experimentally; concurrent work MOSAIC (arXiv 2607.09029) follows a similar direction and merits discussion.
- Layer selection is under-analyzed. The benchmark signal measures only single-layer replacement, ignoring interactions when 27 of 36 layers are replaced together. The effect of the selection benchmarks on later stages is not studied, the selection ablations use a simplified direct-SFT protocol instead of the full pipeline, and the selection benchmarks (e.g., MMMU, DocVQA, VideoMME) overlap with the test sets.
- The benefit of visual linearization is not isolated. Table 1a is confounded: the proposed variant uses full-prefix text attention, while the baselines use sliding windows. A control with modality isolation but visual softmax attention is missing. Capability-level analysis is also lacking: dense-text tasks degrade consistently (InfoVQA -2.3 at both scales), and grounding/hallucination are not evaluated.
- Accuracy and efficiency are measured separately. Accuracy comes mostly from short-context benchmarks, while speedups come from a separate 64K profiling run, and the longest-context benchmark shows the largest drop (LongVideoBench -1.5). Joint accuracy–latency results on long-context benchmarks, plus the token composition used in profiling, are needed.
Questions for authors
- How does the proposed method differ from, and compare experimentally against, InfiniteVL and interleaved cross-modal architectures?
- How sensitive is the final model, after the full four-stage pipeline, to the choice of selection benchmarks? Are the selection splits disjoint from the reported test sets?
- Can the authors report a control with modality isolation but softmax attention on the visual side, as well as grounding and hallucination results?
- Can the authors report accuracy and measured latency/memory jointly on long-context benchmarks, and clarify the token composition and settings used in the 64K profiling?
Thank you for the supplementary experiment. After rebuttal, I tend to view HyLin as an example of the existing FA → hybrid transformation paradigm, rather than a new transformation route. This type of work usually follows the same process:
Puzzle、Jet-Nemotron、Distill-then-Replace、InfiniteVL, Most of them belong to this route. In this context, although the paper proposes the design of modal routing, it is more like an operator selection within the transformation layer rather than a new transformation framework. Tab. R2/R3 indicates that this selection is helpful under matching settings, which is useful, but still biased towards increments.
I suggest the author to compare and sort out the innovation and contribution of this method to the academic community.