- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes VisionWeave, a framework for learning content-adaptive visual representations in multimodal large language models. The method combines a gated spatial pooler and a granularity router to adaptively select fine- or coarse-grained visual tokens, using a three-stage self-distillation strategy for training. Experiments on two Qwen models across eight benchmarks show that VisionWeave can substantially reduce visual tokens while largely preserving model performance. The authors also demonstrate practical efficiency improvements through SGLang deployment. Overall, the motivation is reasonable, and the experimental results are encouraging. My main concerns relate to the baseline comparisons, the flexibility of the granularity allocation mechanism, and its generalization to other MLLM architectures. Further clarification of these aspects would strengthen the paper.
Critical strengths
-
The paper addresses an important limitation of existing visual token compression methods, where fixed compression ratios can lead to information loss on detail-sensitive tasks. The motivation for content-adaptive visual granularity is reasonable and relevant to efficient MLLMs.
-
The proposed method combines a gated spatial pooler with a learned granularity router to construct mixed-granularity visual representations. The three-stage self-distillation strategy and integration with SGLang make the overall framework relatively complete.
-
The experiments cover eight benchmarks across two model scales, with additional evaluations on different input resolutions and video frame budgets. The results show favorable quality–efficiency trade-offs, and the SGLang experiments provide evidence of practical throughput and latency improvements.
Critical weaknesses
-
Recent studies have pointed out that the importance of visual regions may depend not only on image content but also on the specific task or question. In VisionWeave, the Granularity Router mainly relies on ViT features to allocate fine- and coarse-grained representations, and this design has shown good results in content-adaptive visual compression. However, different questions about the same image may require attention to different local details. I would appreciate further discussion on whether the current content-driven routing mechanism can adequately handle such task-dependent requirements, particularly how it compares with approaches that incorporate textual conditioning for task-specific fine-grained visual understanding.
-
The paper mainly compares VisionWeave with FastV†, VisionZip, and downsampling, which are representative baselines. However, I have two questions about the comparisons. First, since the main contribution of this work is learned content-adaptive visual representations, it would be helpful to include comparisons with more closely related methods, such as ViCO, which also explores adaptive visual granularity. While the paper discusses these methods conceptually, quantitative comparisons could further clarify the advantages of VisionWeave. Second, to support SGLang, the authors modify FastV by moving token selection from inside the LLM to after the vision encoder (FastV†). Since this change may affect the original method's behavior, I would appreciate some clarification on how well FastV† represents the performance of the original FastV.
-
While the paper introduces the concept of Elastic Visual Representation Weaving, the current implementation supports only two fixed granularity levels: four fine-grained tokens or one coarse-grained token per spatial block. This binary design simplifies training and deployment, but may limit flexibility when handling complex visual content, as some regions might benefit from intermediate levels of granularity. Additionally, the experiments are conducted on two Qwen models with different parameter scales but from the same model family. It would be helpful if the authors could discuss how well the proposed method might generalize to other MLLM architectures, particularly those with different visual encoders, token organization schemes, or positional encoding mechanisms.
Questions for authors
Please refer to the Critical Weaknesses section.
5 comments
Reply to reviewer HLgk Part2
Reply to Weakness 2(b): How Faithfully FastV† Represents FastV
Thank you for raising this concern.
We also include VisionZip as a reference in our main experiments: prior evaluations have found it to match or outperform the original FastV [3, 4]. They both remain competitive on datasets such as RealWorldQA, but lack adaptivity and exhibit poor cross-domain robustness—limitations inherent to their designs. On Qwen3.8-27B (Table 2), both lose only ~1% on RealWorldQA but over 47% on ScreenSpot-V2. We therefore believe the evaluated FastV variant sufficiently supports VisionWeave’s core advantages: adaptivity and robust cross-task performance.
Second, supporting the original FastV on Qwen3.8-27B in SGLang requires substantial effort, with no reference implementation in the original paper or community to our knowledge. When an algorithm is inherently difficult to deploy in a practical inference engine, this in itself is a major drawback. The specific challenges are
-
Hybrid attention. Qwen3.8-27B’s first three layers use linear attention without explicit attention weights. Even pruning after the fourth, full-attention layer requires bypassing FlashAttention to reconstruct attention weights, introducing overhead.
-
Layer-dependent KV caches. Full sequences in shallow layers and compressed sequences in deeper layers require distinct logical lengths and cache mappings. Supporting this demands changes to KV indexing, attention metadata, and paged-cache management while ensuring correct per-layer reads and updates throughout decoding—the central engine-level challenge.
-
Execution and scheduling compatibility. Engines typically prepare layouts before the forward pass, whereas FastV determines retained indices only after shallow-layer execution. Switching layouts and metadata between layers complicates compatibility with chunked prefill, continuous batching, and CUDA Graphs.
Therefore, integrating the original FastV into SGLang is practically infeasible without substantial engine-level redesign. In our paper, to ensure a fair comparison, we modified its design and invested engineering effort to make it deployable in SGLang, so that the comparison is conducted within the same inference stack. We hope this discussion addresses the concern.
[3] VisionZip: Longer is Better but Not Necessary in Vision Language Models [4] LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit
Reply to Weakness 3(a): Supporting Intermediate Granularity Levels
...supports only two fixed granularity levels...
Yes. As discussed in Appendix G, extending beyond two granularity levels is definitely a promising direction. However, introducing additional granularity levels is expected to create further challenges for large-scale training, particularly in terms of the supporting infrastructure. Given that current frontier MLLMs generally operate at a single, fixed visual granularity, our two-level design, which dynamically adapts granularity to visual content, already represents a substantial advance.
Our prioritization is therefore guided by expected practical benefits. We will first focus on scaling the two-level design as a native architectural capability of larger MLLMs through pre-training, thereby supporting broader downstream applications. Building on this foundation, we will further explore more aggressive forms of multi-granularity compression in future work.
Reply to Weakness 3(b): Generalization Beyond the Qwen Model Family
.. discuss how well the proposed method might generalize to other MLLM architectures...
Qwen3.8-27B adopts a hybrid of full and linear attention, together with a native-resolution ViT encoder and a pixel-unshuffle merger, making it representative of current frontier MLLM architectures. Variants such as MiMo and StepFun use sliding-window attention instead of linear attention; since sliding-window attention shares relevant properties with full attention, we do not expect this difference to pose substantial challenges for VisionWeave. Earlier models such as InternVL employ tile-based visual encoding, whose regular block structure is relatively straightforward to accommodate and may even simplify integration with VisionWeave; however, this design has become less common among frontier models. In addition, Kimi-K3 removes positional encoding (NoPE), eliminating the need for positional interpolation and further reducing adaptation complexity of VisionWeave.
Overall, we expect VisionWeave to be compatible with a broad range of frontier MLLM architectures.
═══════════════════════════════════════════════════════════════════════
We appreciate your insightful feedback. We hope the above discussion has further addressed your remaining concerns, and would be grateful if you could consider raising your score further.
Follow-up on Discussion
Dear reviewer, thank you again for your time.
We would like to kindly follow up on our rebuttal, as the discussion period is approaching its deadline. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns.
Thanks very much for your consideration.
Reply to reviewer HLgk Part1
Thank you for your thoughtful review and valuable feedback. We appreciate your recognition of our content-adaptive motivation, the completeness of the proposed framework, the favorable trade-offs across diverse benchmarks, and the practical improvements demonstrated through SGLang deployment. We provide detailed responses to your concerns below.
Reply to Weakness 1: Task-Dependent Routing and Textual Conditioning
Thank you for this insightful question. When designing VisionWeave, we deliberately chose to route solely according to the information density of different visual regions, independently of the question. And we view the two forms of adaptivity as complementary and best implemented at different levels: question-independent adaptivity at the native architecture level and question-dependent adaptivity at the agentic level.
As a robust and efficient native architecture, VisionWeave is designed to support diverse queries over the same visual input. As outlined in Appendix G, we are actively exploring how to integrate this capability into pre-training. Directly supporting question-dependent adaptivity at the architecture level would limit representation reuse across queries and could discard details needed by other queries or later reasoning steps.
Agentic control instead retains access to the original visual input, allowing granularity to be adjusted iteratively as new sub-questions arise. For example, Qwen3.8-Omni-Flash [1, 2] can autonomously select video frame rates or image resolutions according to the user’s question, which is particularly valuable for long-horizon reasoning requiring repeated retrieval of detailed visual evidence.
The two designs are therefore complementary: VisionWeave provides an efficient native model architecture, while agentic control adds question-dependent adaptivity on top.
[1] Qwen3.8-Omni: Towards Native Omni-Modal Agents
[2] Introducing agentic video understanding with Gemini, https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-agentic-video-in-gemini/
Reply to Weakness 2(a): Quantitative Comparison with ViCO
We agree that ViCO is closely related. However, ViCO cannot accommodate representations of different granularities under a unified positional encoding, and must therefore partition images into tiles for processing. This means a quantitative comparison within Qwen would require retraining with a tile-based encoder and abandoning native-resolution processing, which would in turn incur a substantial performance loss on the already well-pretrained frontier model itself. Moreover, reproducing its training pipeline would also require substantial computational and engineering effort, especially given the tens of thousands of GPU-hours already required for VisionWeave at the 27B scale.
More fundamentally, figure 14 directly illustrates their distinction: ViCO shows limited tile-level adaptivity, while VisionWeave performs clear content-adaptive routing. For the final version, we will use the additional pages to move the related-work discussion and Figure 14 into the main text, making this comparison more accessible. We hope this addresses your concern.