acceptodds
Under review as a conference paper at ICLR 2027

VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs

Abstract

Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure.Overcoming these limitations calls for foundation models that learn, end to end, where—and at what granularity—to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It combines two components: a gated spatial pooler constructs coarse-grained representations alongside native fine-grained representations within a shared MRoPE coordinate, while a granularity router learns their content-adaptive allocation. Through self-distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, VisionWeave adaptively adjusts token savings to visual content, saving 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, versus only 88% performance preserved for token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency–quality trade-offs across diverse tasks, resolutions and video frames. When deployed on SGLang serving engine, our method achieves a 2.30× end-to-end throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%.Together, we believe these results position elastic visual weaving as a promising native capability for next-generation multimodal foundation models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLMs, Token Effiency

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HLgkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VisionWeave, a framework for learning content-adaptive visual representations in multimodal large language models. The method combines a gated spatial pooler and a granularity router to adaptively select fine- or coarse-grained visual tokens, using a three-stage self-distillation strategy for training. Experiments on two Qwen models across eight benchmarks show that VisionWeave can substantially reduce visual tokens while largely preserving model performance. The authors also demonstrate practical efficiency improvements through SGLang deployment. Overall, the motivation is reasonable, and the experimental results are encouraging. My main concerns relate to the baseline comparisons, the flexibility of the granularity allocation mechanism, and its generalization to other MLLM architectures. Further clarification of these aspects would strengthen the paper.

Critical strengths
  1. The paper addresses an important limitation of existing visual token compression methods, where fixed compression ratios can lead to information loss on detail-sensitive tasks. The motivation for content-adaptive visual granularity is reasonable and relevant to efficient MLLMs.

  2. The proposed method combines a gated spatial pooler with a learned granularity router to construct mixed-granularity visual representations. The three-stage self-distillation strategy and integration with SGLang make the overall framework relatively complete.

  3. The experiments cover eight benchmarks across two model scales, with additional evaluations on different input resolutions and video frame budgets. The results show favorable quality–efficiency trade-offs, and the SGLang experiments provide evidence of practical throughput and latency improvements.

Critical weaknesses
  1. Recent studies have pointed out that the importance of visual regions may depend not only on image content but also on the specific task or question. In VisionWeave, the Granularity Router mainly relies on ViT features to allocate fine- and coarse-grained representations, and this design has shown good results in content-adaptive visual compression. However, different questions about the same image may require attention to different local details. I would appreciate further discussion on whether the current content-driven routing mechanism can adequately handle such task-dependent requirements, particularly how it compares with approaches that incorporate textual conditioning for task-specific fine-grained visual understanding.

  2. The paper mainly compares VisionWeave with FastV†, VisionZip, and downsampling, which are representative baselines. However, I have two questions about the comparisons. First, since the main contribution of this work is learned content-adaptive visual representations, it would be helpful to include comparisons with more closely related methods, such as ViCO, which also explores adaptive visual granularity. While the paper discusses these methods conceptually, quantitative comparisons could further clarify the advantages of VisionWeave. Second, to support SGLang, the authors modify FastV by moving token selection from inside the LLM to after the vision encoder (FastV†). Since this change may affect the original method's behavior, I would appreciate some clarification on how well FastV† represents the performance of the original FastV.

  3. While the paper introduces the concept of Elastic Visual Representation Weaving, the current implementation supports only two fixed granularity levels: four fine-grained tokens or one coarse-grained token per spatial block. This binary design simplifies training and deployment, but may limit flexibility when handling complex visual content, as some regions might benefit from intermediate levels of granularity. Additionally, the experiments are conducted on two Qwen models with different parameter scales but from the same model family. It would be helpful if the authors could discuss how well the proposed method might generalize to other MLLM architectures, particularly those with different visual encoders, token organization schemes, or positional encoding mechanisms.

Questions for authors

Please refer to the Critical Weaknesses section.

5 comments
AuthorsOpenReview ↗

Reply to reviewer HLgk Part1

Thank you for your thoughtful review and valuable feedback. We appreciate your recognition of our content-adaptive motivation, the completeness of the proposed framework, the favorable trade-offs across diverse benchmarks, and the practical improvements demonstrated through SGLang deployment. We provide detailed responses to your concerns below.

Reply to Weakness 1: Task-Dependent Routing and Textual Conditioning

Recent studies have pointed out that the importance of visual regions may depend not only on image content but also on the specific task or question...

Thank you for this insightful question. When designing VisionWeave, we deliberately chose to route solely according to the information density of different visual regions, independently of the question. And we view the two forms of adaptivity as complementary and best implemented at different levels: question-independent adaptivity at the native architecture level and question-dependent adaptivity at the agentic level.

As a robust and efficient native architecture, VisionWeave is designed to support diverse queries over the same visual input. As outlined in Appendix G, we are actively exploring how to integrate this capability into pre-training. Directly supporting question-dependent adaptivity at the architecture level would limit representation reuse across queries and could discard details needed by other queries or later reasoning steps.

Agentic control instead retains access to the original visual input, allowing granularity to be adjusted iteratively as new sub-questions arise. For example, Qwen3.8-Omni-Flash [1, 2] can autonomously select video frame rates or image resolutions according to the user’s question, which is particularly valuable for long-horizon reasoning requiring repeated retrieval of detailed visual evidence.

The two designs are therefore complementary: VisionWeave provides an efficient native model architecture, while agentic control adds question-dependent adaptivity on top.

[1] Qwen3.8-Omni: Towards Native Omni-Modal Agents

[2] Introducing agentic video understanding with Gemini, https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-agentic-video-in-gemini/

Reply to Weakness 2(a): Quantitative Comparison with ViCO

We agree that ViCO is closely related. However, ViCO cannot accommodate representations of different granularities under a unified positional encoding, and must therefore partition images into tiles for processing. This means a quantitative comparison within Qwen would require retraining with a tile-based encoder and abandoning native-resolution processing, which would in turn incur a substantial performance loss on the already well-pretrained frontier model itself. Moreover, reproducing its training pipeline would also require substantial computational and engineering effort, especially given the tens of thousands of GPU-hours already required for VisionWeave at the 27B scale.

More fundamentally, figure 14 directly illustrates their distinction: ViCO shows limited tile-level adaptivity, while VisionWeave performs clear content-adaptive routing. For the final version, we will use the additional pages to move the related-work discussion and Figure 14 into the main text, making this comparison more accessible. We hope this addresses your concern.

AuthorsOpenReview ↗

Reply to reviewer HLgk Part2

Reply to Weakness 2(b): How Faithfully FastV† Represents FastV

Thank you for raising this concern.

We also include VisionZip as a reference in our main experiments: prior evaluations have found it to match or outperform the original FastV [3, 4]. They both remain competitive on datasets such as RealWorldQA, but lack adaptivity and exhibit poor cross-domain robustness—limitations inherent to their designs. On Qwen3.8-27B (Table 2), both lose only ~1% on RealWorldQA but over 47% on ScreenSpot-V2. We therefore believe the evaluated FastV variant sufficiently supports VisionWeave’s core advantages: adaptivity and robust cross-task performance.

Second, supporting the original FastV on Qwen3.8-27B in SGLang requires substantial effort, with no reference implementation in the original paper or community to our knowledge. When an algorithm is inherently difficult to deploy in a practical inference engine, this in itself is a major drawback. The specific challenges are

  1. Hybrid attention. Qwen3.8-27B’s first three layers use linear attention without explicit attention weights. Even pruning after the fourth, full-attention layer requires bypassing FlashAttention to reconstruct attention weights, introducing overhead.

  2. Layer-dependent KV caches. Full sequences in shallow layers and compressed sequences in deeper layers require distinct logical lengths and cache mappings. Supporting this demands changes to KV indexing, attention metadata, and paged-cache management while ensuring correct per-layer reads and updates throughout decoding—the central engine-level challenge.

  3. Execution and scheduling compatibility. Engines typically prepare layouts before the forward pass, whereas FastV determines retained indices only after shallow-layer execution. Switching layouts and metadata between layers complicates compatibility with chunked prefill, continuous batching, and CUDA Graphs.

Therefore, integrating the original FastV into SGLang is practically infeasible without substantial engine-level redesign. In our paper, to ensure a fair comparison, we modified its design and invested engineering effort to make it deployable in SGLang, so that the comparison is conducted within the same inference stack. We hope this discussion addresses the concern.

[3] VisionZip: Longer is Better but Not Necessary in Vision Language Models [4] LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit

Reply to Weakness 3(a): Supporting Intermediate Granularity Levels

...supports only two fixed granularity levels...

Yes. As discussed in Appendix G, extending beyond two granularity levels is definitely a promising direction. However, introducing additional granularity levels is expected to create further challenges for large-scale training, particularly in terms of the supporting infrastructure. Given that current frontier MLLMs generally operate at a single, fixed visual granularity, our two-level design, which dynamically adapts granularity to visual content, already represents a substantial advance.

Our prioritization is therefore guided by expected practical benefits. We will first focus on scaling the two-level design as a native architectural capability of larger MLLMs through pre-training, thereby supporting broader downstream applications. Building on this foundation, we will further explore more aggressive forms of multi-granularity compression in future work.

Reply to Weakness 3(b): Generalization Beyond the Qwen Model Family

.. discuss how well the proposed method might generalize to other MLLM architectures...

Qwen3.8-27B adopts a hybrid of full and linear attention, together with a native-resolution ViT encoder and a pixel-unshuffle merger, making it representative of current frontier MLLM architectures. Variants such as MiMo and StepFun use sliding-window attention instead of linear attention; since sliding-window attention shares relevant properties with full attention, we do not expect this difference to pose substantial challenges for VisionWeave. Earlier models such as InternVL employ tile-based visual encoding, whose regular block structure is relatively straightforward to accommodate and may even simplify integration with VisionWeave; however, this design has become less common among frontier models. In addition, Kimi-K3 removes positional encoding (NoPE), eliminating the need for positional interpolation and further reducing adaptation complexity of VisionWeave.

Overall, we expect VisionWeave to be compatible with a broad range of frontier MLLM architectures.

═══════════════════════════════════════════════════════════════════════

We appreciate your insightful feedback. We hope the above discussion has further addressed your remaining concerns, and would be grateful if you could consider raising your score further.

AuthorsOpenReview ↗

Follow-up on Discussion

Dear reviewer, thank you again for your time.

We would like to kindly follow up on our rebuttal, as the discussion period is approaching its deadline. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns.

Thanks very much for your consideration.

Reviewer_fEojOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VisionWeave adds block-level elastic visual granularity to native-resolution MLLMs (Qwen3.5-4B, Qwen3.8-27B). For each 4×4 ViT-patch block, a learned router either keeps the four fine tokens or replaces them with one coarse token from a gated spatial pooler. The pooler sits between the ViT and the native merger, and both paths share the merger, so they live in the same embedding space. Training is three-stage self-distillation from a frozen native teacher: all-coarse pooler, soft-mix router, then hard-routing LLM fine-tuning. SGLang integration uses doubled MRoPE coordinates and halved frequencies for half-integer positions, plus encoder-stage layout metadata so sequence lengths are fixed before LLM scheduling. Evaluation covers eight benchmarks (natural image, document, GUI grounding, hallucination, video) against two training-free baselines and input downsampling, plus a 27B serving study: 43.0% average token savings for a 1.08% average drop, and 2.30× throughput on a long-video workload.

Critical strengths
  1. Complete and actually deployable. Half-integer RoPE and encoder-side layout metadata (Sec. 2.4, App. E.7) keep dynamic batching and CUDA graphs intact; Table 3 gives 2.30× throughput and −54.4% mean TTFT at 27B on long video. Why it matters: most token-reduction work stops at offline evaluation, and doing this at 27B inside a production serving engine is the paper's real value.
  2. Quality holds at scale and is tunable at inference. −0.90% average at 4B (35.8% savings), −1.08% at 27B (43.0%); a single threshold t trades savings for quality without retraining (Fig. 7: 27B Video-MME from 54.2% to 66.0% savings for −0.37); savings rise with resolution (Fig. 4). Why it matters: eight very different tasks at the 27B scale, not only the 7B regime.
Critical weaknesses
  1. The evidence does not separate routing from training and in-domain data. Table 2 compares at mismatched savings; matched, the 4B margin is 0.4–0.7 points, with FastV† ahead on DocVQA, InfoVQA, Video-MME and LongVideoBench. The all-coarse model in Fig. 6 (no router, no LLM fine-tuning) is already within ~1 point of native on video, yet no all-coarse + Stage 3 control is run, and shuffling the scores costs only 1.3–1.5 points. Why it matters: without a control that runs the same distillation pipeline but bypasses the router, it is unclear how much comes from routing and how much from fine-tuning on compressed inputs plus grounding data.
  2. Efficiency is measured on the friendliest workload, with no matched-token baseline. Table 3 uses one setting (27B, long video, ~53% savings, concurrency 4), where the ~170 s TTFT already signals severe queuing. No image or document workload, no breakdown of TTFT into encoder, router and prefill, no router/pooler cost. Why it matters: 2.30× shows that fewer tokens are faster, not that VisionWeave beats simpler alternatives under production-representative conditions.
Questions for authors

(A) Answers that would change my rating

  1. Can you add an "all-coarse + Stage 3 LLM fine-tuning" control — same pipeline, router bypassed — at matched savings? If it is clearly weaker across task types, the routing contribution is established, and I would raise Soundness to 3 and the rating to 3. Can you report all aggregates (Table 2 means, the 98.9% headline, the Table 7 margins) both with and without ScreenSpotV2, and show 2. the grounding gain is not mainly the training data — by ablating UGround, or by fine-tuning the baselines on the same UGround data? If it holds under either control, the attribution concern is resolved and I would raise my score.
  2. Can you provide router/pooler parameters, FLOPs and ViT-side latency; a TTFT breakdown into encoder, router and prefill; serving results on an image or document workload; and a serving comparison against input downsampling at matched token counts? Which attention map and layer does FastV† use after the encoder, and how was the downsampling resolution chosen to hit 50% savings? (B) Suggestions, not affecting my rating Move Table 7 into the main text and temper the abstract ("native capability", "first effort") to match it. Report per-benchmark results for t=1 (all-fine) and add reasoning benchmarks (MMMU, MathVista) to confirm Stage 3 causes no regressions. Ablate the router: last-layer features only, local attention only, a linear probe on pooled features. Report variance over seeds, and consider releasing code and weights given the ~30K GPU-hour cost. Move the related work (Appendix C) into the paper and compare explicitly with APT, which also adapts granularity to visual content but at the ViT input rather than the merger stage.
6 comments
AuthorsOpenReview ↗

Reply to Reviewer fEoj (1/4)

Thank you for the effort you invested in evaluating our work. We appreciate your recognition that VisionWeave is complete and deployable in a production serving engine, and that it preserves quality at scale. We provide detailed responses to your concerns below.

Reply to Weakness 1:

Table 2 compares at mismatched savings; matched, the 4B margin is 0.4–0.7 points, with FastV† ahead on DocVQA, InfoVQA, Video-MME and LongVideoBench. The all-coarse model in Fig. 6 is already within ~1 point of native on video, yet no all-coarse + Stage 3 control is run, and shuffling the scores costs only 1.3–1.5 points.

First, we respectfully point out two biased characterizations of our results.

(1) Regarding “the all-coarse model in Fig. 6 is already within ~1 point of native on video,” we point out that the all-coarse variant trails native by 4.05 points on VideoOCR (60.46 vs. 64.51).

(2) Regarding "shuffling the scores costs only 1.3–1.5 points", the quoted 1.3–1.5-point loss in Appendix E.3 excludes ScreenSpotV2 dataset. Across all eight benchmarks, average losses of 2.31 points (4B) and 5.36 points (27B) cannot be overlooked.

Second, we clarify adaptive token savings as a core contribution of VisionWeave: it adapts the compression ratio across tasks to preserve native-level performance, whereas FastV lacks this capability. The matched-budget setting is a controlled analysis in which FastV is evaluated with the per-task compression ratios determined by VisionWeave; without VisionWeave's router, FastV has no such mechanism. Thus, FastV's slight lead on a few datasets under these VisionWeave-derived ratios does not negate VisionWeave's advantage. Moreover, FastV's robustness issues—drastic degradation on grounding tasks and poor robustness across input resolutions (Figure 4)—cannot be overlooked.

Reply to Question 1: Part1

To address your following questions, we trained three variants for comparison with Qwen-3.8-27B: VisionWeave, the “all-coarse + Stage 3 LLM fine-tuning” control, and fine-tuned FastV. All three variants used the same training settings as in the main paper. To keep computational costs manageable, we trained each variant for 200 steps, averaging approximately 1,100 A100 GPU-hours per variant. The results below are evaluated under matched budgets.

Can you add an "all-coarse + Stage 3 LLM fine-tuning" control — same pipeline, router bypassed — at matched savings? If it is clearly weaker across task types, the routing contribution is established, and I would raise Soundness to 3 and the rating to 3.

Yes, we have conducted this comparison. VisionWeave generallythe "all-coarse + Stage 3 LLM fine-tuning" control across a broad range of benchmarks, with the largest gap on grounding tasks, where the all-coarse variant trails its VisionWeave counterpart by 22.88 points.

Matched budgetALL-coarse + Stage3 LLM fine-tuningVisionWeave
RWQA76.7377.39
DocVQA90.9590.99
InfoVQA71.2671.31
ScreenSpot-v265.6488.52
HallusionBench73.8773.52
VideoOCR70.7270.67
Video-MME70.8971.11
LongVideoBench65.9766.42
MathVista83.1084.10
Avg74.3577.11

Reply to Question 1: Part 2

Can you report all aggregates (Table 2 means, the 98.9% headline, the Table 7 margins) both with and without ScreenSpotV2 ...

Excluding ScreenSpotV2, VisionWeave retains 99.2% of native performance while saving 41.2% of visual tokens on 27B, and retains 99.0% while saving 37.4% on 4B. Both models maintain their advantage over FastV† and VisionZip, with or without ScreenSpotV2.

The recomputed Table 2 aggregates below show that VisionWeave preserves near-native performance through adaptive token savings under both with and without ScreenSpotV2 settings.

Table 2 aggregates.

ModelMethodScore (with)Score (without)Token savings (with, %)Token savings (without, %)
4BNative73.2570.8800
4BDownsampling67.8766.0750 (fixed)50 (fixed)
4BFastV†66.8666.0550 (fixed)50 (fixed)
4BVisionZip65.9765.3450 (fixed)50 (fixed)
4BVisionWeave72.5970.1835.8 (adaptive)37.4 (adaptive)
27BNative77.6575.2500
27BDownsampling72.5269.8750 (fixed)50 (fixed)
27BFastV†67.3269.8650 (fixed)50 (fixed)
27BVisionZip66.8169.3250 (fixed)50 (fixed)
27BVisionWeave76.7374.6343.0(adaptive)41.2 (adaptive)

We report the Table 7 aggregates as below. VisionWeave achieves higher mean scores than both FastV† and VisionZip at both model scales, regardless of whether ScreenSpotV2 is included.

Matched budgetMethodWith ScreenSpotV2Without ScreenSpotV2
4BVisionWeave72.5970.18
4BFastV†72.20 (−0.39)70.04 (−0.14)
4BVisionZip71.90 (−0.68)69.88 (−0.30)
27BVisionWeave76.7374.63
27BFastV†70.10 (−6.63)73.86 (−0.77)
27BVisionZip70.33 (−6.40)73.85 (−0.78)
AuthorsOpenReview ↗

Reply to Reviewer fEoj (2/4)

Reply to Question 1: Part 3

.... by fine-tuning the baselines on the same UGround data? If it holds under either control, the attribution concern is resolved and I would raise my score.

Following your suggestions, although fine-tuning improves FastV’s performance on ScreenSpot-v2, it still trails VisionWeave by a considerable margin of 12.58. FastV also underperforms by 3.3 points on MathVista, the reasoning benchmark you recommended, with no observed trend toward improvement of fine-tuning. These results provide strong evidence for VisionWeave’s robustness advantage.

Matched budgetFastV without finetuneFastV with finetuneVisionWeave
RWQA78.1777.3977.39
DocVQA88.2689.6490.99
InfoVQA71.2970.6671.31
ScreenSpot-v243.7975.9488.52
HallusionBench72.5474.1473.52
VideoOCR69.7970.5670.67
Video-MME71.1571.5971.11
LongVideoBench65.8265.8266.42
MathVista81.0080.8084.10
Avg.71.3175.1777.11

Reply to Weakness 2 & Question 2: Part1

Can you provide router/pooler parameters, FLOPs and ViT-side latency;

The additional modules, including router and pooler, contain only 102.30M parameters, approximately 0.38% of the 27B backbone.

ModuleParametersDtype
Router cross-attention63.792MBF16
Router head0.00461MFP32
Total Router63.797MMixed
Pooler38.507MBF16

In terms of FLOPs, consider one image with 1024 visual tokens.

  • Router: 0.169 TFLOPs.
  • Pooler: 0.067 TFLOPs.
  • Total router + pooler: 0.236 TFLOPs,
  • ViT transformer blocks: 5.455 TFLOPs.

The ViT-side latency results are provided in our response to the next question.

Reply to Weakness 2 & Question 2: Part2

Can you provide ... a TTFT breakdown into encoder, router and prefill; serving results on an image or document workload; and a serving comparison against input downsampling at matched token counts?

We answer these questions on DocVQA with an input budget of 1,024 visual tokens per image, client concurrency 128, and a server running-request limit of 128. Both use the same topology with two encoder replicas and language TP=2. The measured router and pooler latency are only 7.00 ms and 2.35 ms, respectively. VisionWeave improves throughput by 27.9% and reduces mean TTFT and TPOT by 19.1% and 27.4%, respectively.

However, the reported stage latencies cannot be summed to obtain client TTFT. Encoder and LLM scheduling are handled independently and automatically by the serving engine for each method, and the differences in token savings can substantially affect scheduling outcomes. For example, VisionWeave's token savings enable larger LLM prefill batches (Mean Prefill Batch: 10.02 vs. 7.44). Although its mean prefill batch wall time is slightly longer (1.94 vs. 1.84 s), it processes more requests per batch thereby achieving higher throughput and lower TTFT. Accordingly, practical serving systems are typically evaluated using end-to-end throughput, TTFT, and TPOT, which are the primary metrics in our paper. These end-to-end results demonstrate that VisionWeave's algorithmic benefits translate into practical serving improvements.

We also provide matched-budget downsampling results below. Downsampling achieves higher throughput than VisionWeave, but this comparison is for reference only: unlike VisionWeave, it does not perform dynamic compression and incurs larger loss.

MethodThroughput (req/min)ViT / image (ms)Router / image (ms)Pooler / image (ms)Prefill wall mean (s)Mean Prefill bszMean TTFT (s)Mean TPOT (ms)
Native EPD225.99216.50N/AN/A1.847.4422.87357.20
VisionWeave289.00 (1.28×)229.907.002.351.9410.0218.50 (↓19.1%)259.29 (↓27.4%)
Matched Downsampled390.96 (1.73×)98.18N/AN/A1.5010.1512.58224.42
AuthorsOpenReview ↗

Reply to Reviewer fEoj (3/4)

Reply to Weakness 2 & Question 2: Part3

Which attention map and layer does FastV† use after the encoder, and how was the downsampling resolution chosen to hit 50% savings?

We use the final vision-encoder layer’s attention map to score all visual tokens and retain the top K. This choice is motivated by deployment constraints: implementing the original FastV’s pruning after the second LLM layer in SGLang for Qwen3.8-27B is practically infeasible. The main challenges are

  1. Hybrid attention. The first three layers of Qwen3.8-27B use linear attention without explicit attention weights. Even pruning after the fourth, full-attention layer requires bypassing FlashAttention to reconstruct these weights, adding overhead.

  2. Layer-dependent KV caches. Full sequences in shallow layers and compressed sequences in deeper layers require distinct logical lengths and cache mappings. Supporting this demands changes to KV indexing, attention metadata, and paged-cache management while ensuring correct per-layer reads and updates throughout decoding—the central engine-level challenge.

  3. Execution and scheduling. Engines typically prepare layouts before the forward pass, whereas FastV determines retained indices only after shallow-layer execution. Switching layouts and metadata between layers complicates compatibility with chunked prefill, continuous batching, and CUDA Graphs.

These barriers motivated FastV† as a practical, infrastructure-compatible adaptation. We also include VisionZip, which performs comparably to the original FastV in prior work. Its similar performance to FastV† in our evaluation provides additional support for the validity of this adaptation.

For downsampling, we cap the maximum pixel count per image and let SGLang automatically handle resizing. We checked the resulting average visual token counts for each task to verify the intended reduction.

Reply to Suggestions: Part 1

Move Table 7 into the main text and temper the abstract ("native capability", "first effort") to match it.

We will move Table 7 into the main text and use the additional page available for the revision to bring the related work section into the main paper.

Our abstract explicitly defines the native capability: “Overcoming these limitations calls for foundation models that learn, end to end, where—and at what granularity—to allocate visual representations, a native capability we term elastic visual representation weaving.” The Introduction (line96-line102) further specifies three criteria for “native,” each supported by extensive evaluations of the architecture’s adaptivity and robustness. We also dedicate a paragraph in the Introduction (lines 87–96) to explicitly citing prior work on dynamic representational granularity: “Recent work has explored content-adaptive visual representations...”

If “first effort to establish elastic visual representation weaving as a native capability of frontier MLLMs through large-scale training” remains ambiguous, please let us know; we will removing this claim.

Reply to Suggestions: Part 2

Report per-benchmark results for t=1 (all-fine) and add reasoning benchmarks (MMMU, MathVista) to confirm Stage 3 causes no regressions.

We have reported all-fine performance in Figure 7 and, in our previous response, provided a MathVista comparison within controlled varients. Here, we present results for the Qwen3.8-27B and Qwen3.5-4B VisionWeave models used in the paper’s main experiments. In both cases, VisionWeave substantially outperforms the baseline while achieving accuracy close to that of the native model.

Model SizeMethodsMathVistaToken Savings
27BNative Model (Upper bound)84.90%0%
27BVisionWeave84.30%41.64%
27BFastV81.00%41.64%
4BNative Model (Upper bound)81.10%0%
4BVisionWeave80.30%39.44%
4BFastV76.60%39.44%

Reply to Suggestions: Part 3

Ablate the router: last-layer features only, local attention only, a linear probe on pooled features.

As discussed in Appendix G, ablating these router variants would require computational resources far beyond our budget. Nevertheless, we believe our extensive training and evaluation sufficiently demonstrate VisionWeave’s advantages, while further ablations could help refine the architecture in future work.

Reviewer_PmrhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presented VisionWeave, a method to achieve elastic visual representation weaving that balances between the representation granularity and inference budget in serving. It contains a gated spatial pooler that builds both coarse and fine-grained embeddings, and a granularity router to allocate the granularity adaptively to the content. In experiments, this method consistently achieved gains across benchmarks and VLMs, and the deployment with SFLang serving engine further showed its advantages of the inference throughput.

Critical strengths
  • The proposed gated spatial pooler and granularity router are designed in a learnable way, which gives the adaptability of the model to diverse visual scenes through end-to-end training.

  • Substantial technical details are presented, including both the model design, training recipes, and model inference / serving details. This makes the paper a technically sound submission.

Critical weaknesses
  • The router is limited to the binary choices by heuristic threshold. Is it possible to extend the router to multiple granularities by using mixture-of-experts (MoE)? This is intuitively more reasonable since the granularity of visual scenes is never binary.

  • The three-stage training seems to make the approach hard to reuse in practice, especially when the training cost is prohibitive on large-scale data. It would be better to compare with a baseline of end-to-end one-stage training and let users decide if the three-stage training should be adopted or not.

Questions for authors

N/A

3 comments

Thank you for your thoughtful review. We appreciate your recognition of the adaptability enabled by our learnable gated spatial pooler and granularity router, as well as the substantial technical details covering model design, training, inference, and serving. We provide detailed responses to your concerns below.

Reply to Weakness 1:

The router is limited to the binary choices by heuristic threshold. Is it possible to extend the router to multiple granularities by using mixture-of-experts (MoE)? This is intuitively more reasonable since the granularity of visual scenes is never binary.

Yes. As discussed in Appendix G, extending beyond two granularity levels is definitely a promising direction. However, introducing additional granularity levels is expected to create further challenges for large-scale training, particularly in terms of the supporting infrastructure. Given that current frontier MLLMs generally operate at a single, fixed visual granularity, our two-level design, which dynamically adapts granularity to visual content, already represents a substantial advance.

Our prioritization is therefore guided by expected practical benefits. We will first focus on scaling the two-level design as a native architectural capability of larger MLLMs through pre-training, thereby supporting broader downstream applications. Building on this foundation, we will further explore more aggressive forms of multi-granularity compression in future work.

Reply to Weakness 2: Training Reusability and the One-Stage Training Baseline

The three-stage training seems to make the approach hard to reuse in practice, especially when the training cost is prohibitive on large-scale data. It would be better to compare with a baseline of end-to-end one-stage training and let users decide if the three-stage training should be adopted or not.

Actually, we initially tried MoE-style, single-stage training, but observed training instability: the router could abruptly collapse to selecting a single granularity. This poses a particular challenge when scaling training to larger models. We therefore adopted the current three-stage recipe, which substantially improved training stability and enabled us to successfully scale to 27B on the first attempt. Figure 15 shows the stable training dynamics of this run.

We also agree simplifying training stages remains an important topic. Appendix G discusses integrating VisionWeave into base-model pre-training, which we are actively pursuing as a path toward unified, single-stage training. Realizing this at pre-training scale requires substantial work by a large engineering team and is beyond the scope of the present research paper.

═══════════════════════════════════════════════════════════════════════

We appreciate your insightful feedback. We hope the above discussion has further addressed your remaining concerns, and would be grateful if you could consider raising your score further.

AuthorsOpenReview ↗

Follow-up on Rebuttal

Dear reviewer, thank you again for your time.

We would like to kindly follow up on our rebuttal, as the discussion period is approaching its deadline. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns.

Thanks very much for your consideration.

Thanks for the clarification, my initial concerns are well addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.