- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Weight Space Implicit Representation Encoding to address the heavy prefilling stage in video understanding of LLMs. Specifically, it combines a question-conditioned bidirectional Mamba compressor to produce carrier tokens and uses hypernetworks that use the compressed decoder states and full-resolution features to generate LoRA parameters at selected layers of the LLM. The added modules are trained with context distillation and answer cross-entropy. Experiments evaluate QA accuracy across a number of benchmarks, latency at different video lengths, component ablations, and test a needle-in-a-haystack task.
Critical strengths
-
This paper targets an important practical problem: the latency and memory cost of processing long videos with video LLMs. The motivation is clear, and Figure 1 gives a valuable picture of how prefill time grows with video length and where some existing methods run out of memory.
-
The combination of compressed context and video-conditioned weight updates, with efficient linear attention and hypernetwork, respectively, provides an interesting way to adapt a frozen video LLM while keeping its original parameters fixed. The distillation from the full-context model together with cross-entropy supervision gives a straightforward training objective.
-
The experiments cover many commonly used benchmarks and relevant methods. Tables 1 and Table 4 show the proposed methods achieves competitive accuracy among the efficient models. I appreciate the paired update analysis in Table 2 and the compressor ablations in Table 3, which provide useful evidence for the added components.
-
The timing results in Table 5 are very direct to support the main claims. Figure 4 is also a bonus additional experiment, showing that the proposed method can preserve and retrieve hidden information effectively.
Critical weaknesses
-
The evaluation seems limited to multiple-choice QA. There are no reported results on free-form QA, detailed video descriptions or summarization. I think these are common practical uses of video LLMs, and I expect to see how well the method would preserve the detail and completeness needed for these outputs, and how the output quality of the original LLM would be affected.
-
I think the analysis of the key components and design choices is still limited. Table 3 mainly ablates the compressor, and Table 2 switches the generated update on and off, which are relatively shallow. For example, I expect more in-depth studies on the training objectives, the hypernetwork design and compressor network selection, or the number and placement of adapted layers and LoRA rank.
-
I think the needle in a haystack experiment in the Appendix is highly valuable and could be pushed further. This is a very challenging and convincing task for examining the compression and retrieval abilities. Shorter appearances, smaller visual details, or multiple similar needles with questions about their order would make this a more informative stress test.
Questions for authors
I hope the authors can respond to the weaknesses above. I also have one question:
In Figure 4, how does the hypernetwork generate the update when k=0? The method uses carrier token states, but this setting removes all visual tokens from the decoder input sequence.
(Does not affect my rating) I think Figure 3 in the main paper is not very informative, especially given how much space it takes up. I suggest moving it to the appendix or replacing it with a more informative visualization.
4 comments
Response 2/2
- Adapted layers and LoRA rank.
We investigate the effect of adaptation depth by generating LoRA updates for 3, 6, or 9 decoder layers, selected uniformly across the decoder depth. We then fix the number and placement of adapted layers to the original 9-layer configuration and vary the LoRA rank between 16, 32, and 64. All variants use the same hypernetwork architecture and training objective.
| Configurtion | Video-MME | MVBench |
|---|---|---|
| 3 adapted layers, rank 64 | 65.1 | 61.8 |
| 6 adapted layers, rank 64 | 65.4 | 62.1 |
| 9 adapted layers, rank 64 (full WIRE) | 66.2 | 62.8 |
| 9 adapted layers, rank 32 | 66.8 | 62.4 |
Together, these experiments examine the training objective, the architecture used to generate the updates, the number of adapted layers, LoRA rank, and compressor network selection. They complement the existing ablations by providing a more granular analysis of the individual design choices in WIRE. We will add this analysis in the revised version.
I think the needle in a haystack experiment in the Appendix is highly valuable and could be pushed further. This is a very challenging and convincing task for examining the compression and retrieval abilities. Shorter appearances, smaller visual details, or multiple similar needles with questions about their order would make this a more informative stress test.
We agree that the NIAH experiment is a valuable stress test of WIRE's long-context retrieval capabilities and that further variations could make it even more informative. In our existing evaluation, we test retrieval in 2,048-frame videos at k=6, placing the target at five different temporal positions. Full WIRE achieves 75%, 67%, 75%, 75%, and 75% accuracy at positions 0%, 25%, 50%, 75%, and 100%, respectively. We further compare against an independently trained compressor-only model, which achieves 50%, 50%, 65%, 60%, and 50%, and a variant that removes the full-resolution features Ft from the hypernetwork input while retaining the generated LoRA updates, which achieves 45%, 40%, 60%, 50%, and 55%. These results show that WIRE's advantage is consistent across retrieval positions and that both the generated updates and access to full-resolution features contribute to retrieval performance. We agree that shorter target appearances, smaller visual details, and multiple similar targets requiring temporal-order reasoning would provide useful additional stress tests. These extensions would help characterize the limits of WIRE beyond the current single-target retrieval setting, and we will discuss them as directions for future work.
In Figure 4, how does the hypernetwork generate the update when k=0? The method uses carrier token states, but this setting removes all visual tokens from the decoder input sequence.
When (k=0), all visual carrier tokens are removed from the decoder input sequence, so the hypernetwork does not receive carrier-token states. Instead, it generates the LoRA updates using the remaining inputs: the decoder hidden states from the question/text tokens and the full-resolution visual features (F_t), which are provided directly to the hypernetwork through its cross-attention pathway. Thus, even when (k=0), the hypernetwork can access visual information without inserting visual tokens into the decoder sequence. We will clarify this distinction in Figure 4 and the accompanying description.
(Does not affect my rating) I think Figure 3 in the main paper is not very informative, especially given how much space it takes up. I suggest moving it to the appendix or replacing it with a more informative visualization.
We will revise this figure in the revised version
We will incorporate all additional results and explanations presented in the rebuttal into the revised manuscript. Thank you again for your support and recognition of our work. We would be happy to provide any further clarification or address any remaining concerns. If we have fully addressed your concerns, we kindly ask you to consider reassessing your score.
Thank you again for your time and effort.
We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.
We would be happy to address any remaining concerns or provide further clarification. If our responses have addressed your remaining concerns, we would greatly appreciate it if you could consider strengthening your support for the paper.
Final Rating
I would like to thank the authors for their detailed rebuttal, especially the additional experiments, which address most of my concerns. After carefully reviewing the responses from all reviewers and the revised manuscript, I am happy to raise my rating. I would encourage the authors to further strengthen the newly added experiments, given the additional time available before the final version is due.
Response 1/2
We appreciate the reviewer’s thorough evaluation and valuable suggestions. We respond below to all concerns and questions raised, addressing each point in detail.
We agree that evaluating WIRE on free-form video understanding tasks is important. To address this, we conducted an additional paired evaluation comparing the original Qwen3-VL-8B model with WIRE (k=16) on 100 videos. We sampled 100 videos from the Video-MME benchmark, covering short, medium, and long-duration videos. For each video, we constructed open-ended detailed-description, and summarization prompts and evaluated both models using identical inputs. For each video, we evaluated detailed video description, and summarization using identical prompts for both models. Outputs were anonymized and evaluated by GPT-5.6 as an LLM judge, using a 1 - 5 scale for correctness, completeness, and factual consistency.
The results suggest that WIRE retains much of the original model's free-form generation quality despite substantially reducing the visual token budget. We will include these results and clarify that, while WIRE preserves much of the original model's free-form behavior, some loss in descriptive detail remains under aggressive visual token compression.
We agree that the original ablation study does not sufficiently explore several important design choices in WIRE. We therefore extend the analysis with additional controlled experiments, keeping the backbone, training data, visual token budget ((k=16)), and evaluation protocol fixed. We distinguish these experiments from the existing ablations as follows.
Existing ablations. Table 3 studies individual components of the Mamba2 compressor, including question conditioning, bidirectional scanning, and selective scanning. Table 2 evaluates the overall contribution of the generated updates. Our additional experiments further examine the direct question input to the hypernetwork, compare against an independently trained compressor-only baseline, and evaluate simple pooling-based compression methods. While these experiments establish the contribution of the main components, they do not fully characterize the design choices within the update-generation pathway.
Training objective: We compare three objectives: (i) the original KL distillation loss used in WIRE, (ii) answer-token cross-entropy using the ground-truth answers, and (iii) a combination of KL distillation and answer-token cross-entropy. All other training settings remain unchanged.
Hypernetwork architecture: We replace the current hypernetwork with a two-layer MLP. We first aggregate the same decoder hidden states, question representation, and full-resolution visual features (F_t) used by the original hypernetwork into a fixed-dimensional representation. The MLP then maps this representation to the LoRA matrices for the same adapted decoder layers. The input information, LoRA rank, output parameterization, and training objective remain unchanged, isolating the effect of the hypernetwork architecture.