acceptodds
Under review as a conference paper at ICLR 2027

WIRE: Weight Space Implicit Representation Encoding of Frozen Video LLMs

Abstract

Video language models represent a video as a long sequence of visual embeddings in a decoder's input. Every visual embedding is processed by every decoder layer during prefill, stored in the key value cache, and attended by every generated token, so a long video raises prefill, memory, and generation cost at once. We present WIRE (Weight Space Implicit Representation Encoding), which reduces this recurring cost by moving part of the video conditioned computation of a frozen video language model from its input activations into a temporary low rank update of its weights. WIRE keeps a short explicit visual context, so the update acts as a correction rather than a replacement of the video. WIRE first compresses the full resolution embeddings of each frame into a small set of carrier tokens using a question conditioned bidirectional selective scan, and physically contracts the decoder input so every decoder layer, cache entry, and decoding step operates on the shorter sequence. Hypernetworks at selected decoder layers read the compressed forward pass together with a readout of the full resolution features removed by contraction and generate LoRA updates to the attention and feedforward projections. The updates are merged after prefill, so autoregressive decoding runs the unmodified frozen backbone. On Qwen3-VL-8B, WIRE with 16 carrier tokens per frame contracts the visual stream by , reduces decoder prefill time by at 256 frames and at 1024 frames, shrinks the key value cache proportionally, and keeps per token decoding latency approximately constant. Across nine benchmark settings at 256 frames, it obtains the highest average accuracy among efficient video language models we evaluate under a common 256 frame budget. A paired update analysis toggles only the generated update while holding the compressed visual context fixed. Applying the generated update raises whole benchmark accuracy on Video-MME, MVBench, and LongVideoBench by up to points, with the largest gains on compression sensitive capabilities such as counting and action count, establishing that the generated update does task relevant work beyond the carrier tokens.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video llms, video understanding, multimodal, token compression

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_2UjmOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Weight Space Implicit Representation Encoding to address the heavy prefilling stage in video understanding of LLMs. Specifically, it combines a question-conditioned bidirectional Mamba compressor to produce carrier tokens and uses hypernetworks that use the compressed decoder states and full-resolution features to generate LoRA parameters at selected layers of the LLM. The added modules are trained with context distillation and answer cross-entropy. Experiments evaluate QA accuracy across a number of benchmarks, latency at different video lengths, component ablations, and test a needle-in-a-haystack task.

Critical strengths
  1. This paper targets an important practical problem: the latency and memory cost of processing long videos with video LLMs. The motivation is clear, and Figure 1 gives a valuable picture of how prefill time grows with video length and where some existing methods run out of memory.

  2. The combination of compressed context and video-conditioned weight updates, with efficient linear attention and hypernetwork, respectively, provides an interesting way to adapt a frozen video LLM while keeping its original parameters fixed. The distillation from the full-context model together with cross-entropy supervision gives a straightforward training objective.

  3. The experiments cover many commonly used benchmarks and relevant methods. Tables 1 and Table 4 show the proposed methods achieves competitive accuracy among the efficient models. I appreciate the paired update analysis in Table 2 and the compressor ablations in Table 3, which provide useful evidence for the added components.

  4. The timing results in Table 5 are very direct to support the main claims. Figure 4 is also a bonus additional experiment, showing that the proposed method can preserve and retrieve hidden information effectively.

Critical weaknesses
  1. The evaluation seems limited to multiple-choice QA. There are no reported results on free-form QA, detailed video descriptions or summarization. I think these are common practical uses of video LLMs, and I expect to see how well the method would preserve the detail and completeness needed for these outputs, and how the output quality of the original LLM would be affected.

  2. I think the analysis of the key components and design choices is still limited. Table 3 mainly ablates the compressor, and Table 2 switches the generated update on and off, which are relatively shallow. For example, I expect more in-depth studies on the training objectives, the hypernetwork design and compressor network selection, or the number and placement of adapted layers and LoRA rank.

  3. I think the needle in a haystack experiment in the Appendix is highly valuable and could be pushed further. This is a very challenging and convincing task for examining the compression and retrieval abilities. Shorter appearances, smaller visual details, or multiple similar needles with questions about their order would make this a more informative stress test.

Questions for authors

I hope the authors can respond to the weaknesses above. I also have one question:

In Figure 4, how does the hypernetwork generate the update when k=0? The method uses carrier token states, but this setting removes all visual tokens from the decoder input sequence.

(Does not affect my rating) I think Figure 3 in the main paper is not very informative, especially given how much space it takes up. I suggest moving it to the appendix or replacing it with a more informative visualization.

4 comments
AuthorsOpenReview ↗

Response 1/2

We appreciate the reviewer’s thorough evaluation and valuable suggestions. We respond below to all concerns and questions raised, addressing each point in detail.

The evaluation seems limited to multiple-choice QA. There are no reported results on free-form QA, detailed video descriptions or summarization. I think these are common practical uses of video LLMs, and I expect to see how well the method would preserve the detail and completeness needed for these outputs, and how the output quality of the original LLM would be affected.

We agree that evaluating WIRE on free-form video understanding tasks is important. To address this, we conducted an additional paired evaluation comparing the original Qwen3-VL-8B model with WIRE (k=16) on 100 videos. We sampled 100 videos from the Video-MME benchmark, covering short, medium, and long-duration videos. For each video, we constructed open-ended detailed-description, and summarization prompts and evaluated both models using identical inputs. For each video, we evaluated detailed video description, and summarization using identical prompts for both models. Outputs were anonymized and evaluated by GPT-5.6 as an LLM judge, using a 1 - 5 scale for correctness, completeness, and factual consistency.

TaskMetricQwen3-VL-8BWIRE (k=16)
SummarizationCoverage of key events4.104.02
Factual consistency4.244.08
Detailed descriptionCompleteness4.013.96
Factual consistency4.053.94

The results suggest that WIRE retains much of the original model's free-form generation quality despite substantially reducing the visual token budget. We will include these results and clarify that, while WIRE preserves much of the original model's free-form behavior, some loss in descriptive detail remains under aggressive visual token compression.

I think the analysis of the key components and design choices is still limited. Table 3 mainly ablates the compressor, and Table 2 switches the generated update on and off, which are relatively shallow. For example, I expect more in-depth studies on the training objectives, the hypernetwork design and compressor network selection, or the number and placement of adapted layers and LoRA rank.

We agree that the original ablation study does not sufficiently explore several important design choices in WIRE. We therefore extend the analysis with additional controlled experiments, keeping the backbone, training data, visual token budget ((k=16)), and evaluation protocol fixed. We distinguish these experiments from the existing ablations as follows.

Existing ablations. Table 3 studies individual components of the Mamba2 compressor, including question conditioning, bidirectional scanning, and selective scanning. Table 2 evaluates the overall contribution of the generated updates. Our additional experiments further examine the direct question input to the hypernetwork, compare against an independently trained compressor-only baseline, and evaluate simple pooling-based compression methods. While these experiments establish the contribution of the main components, they do not fully characterize the design choices within the update-generation pathway.

  1. Training objective and hypernetwork architecture.

Training objective: We compare three objectives: (i) the original KL distillation loss used in WIRE, (ii) answer-token cross-entropy using the ground-truth answers, and (iii) a combination of KL distillation and answer-token cross-entropy. All other training settings remain unchanged.

Hypernetwork architecture: We replace the current hypernetwork with a two-layer MLP. We first aggregate the same decoder hidden states, question representation, and full-resolution visual features (F_t) used by the original hypernetwork into a fixed-dimensional representation. The MLP then maps this representation to the LoRA matrices for the same adapted decoder layers. The input information, LoRA rank, output parameterization, and training objective remain unchanged, isolating the effect of the hypernetwork architecture.

ConfigurationVideo-MMEMVBench
Full WIRE (KL distillation)66.262.8
Answer-token CE only64.161.4
Combined CE + KL65.962.8
Two-layer MLP update generator64.461.3
AuthorsOpenReview ↗

Response 2/2

  1. Adapted layers and LoRA rank.

We investigate the effect of adaptation depth by generating LoRA updates for 3, 6, or 9 decoder layers, selected uniformly across the decoder depth. We then fix the number and placement of adapted layers to the original 9-layer configuration and vary the LoRA rank between 16, 32, and 64. All variants use the same hypernetwork architecture and training objective.

ConfigurtionVideo-MMEMVBench
3 adapted layers, rank 6465.161.8
6 adapted layers, rank 6465.462.1
9 adapted layers, rank 64 (full WIRE)66.262.8
9 adapted layers, rank 3266.862.4

Together, these experiments examine the training objective, the architecture used to generate the updates, the number of adapted layers, LoRA rank, and compressor network selection. They complement the existing ablations by providing a more granular analysis of the individual design choices in WIRE. We will add this analysis in the revised version.

I think the needle in a haystack experiment in the Appendix is highly valuable and could be pushed further. This is a very challenging and convincing task for examining the compression and retrieval abilities. Shorter appearances, smaller visual details, or multiple similar needles with questions about their order would make this a more informative stress test.

We agree that the NIAH experiment is a valuable stress test of WIRE's long-context retrieval capabilities and that further variations could make it even more informative. In our existing evaluation, we test retrieval in 2,048-frame videos at k=6, placing the target at five different temporal positions. Full WIRE achieves 75%, 67%, 75%, 75%, and 75% accuracy at positions 0%, 25%, 50%, 75%, and 100%, respectively. We further compare against an independently trained compressor-only model, which achieves 50%, 50%, 65%, 60%, and 50%, and a variant that removes the full-resolution features Ft from the hypernetwork input while retaining the generated LoRA updates, which achieves 45%, 40%, 60%, 50%, and 55%. These results show that WIRE's advantage is consistent across retrieval positions and that both the generated updates and access to full-resolution features contribute to retrieval performance. We agree that shorter target appearances, smaller visual details, and multiple similar targets requiring temporal-order reasoning would provide useful additional stress tests. These extensions would help characterize the limits of WIRE beyond the current single-target retrieval setting, and we will discuss them as directions for future work.

In Figure 4, how does the hypernetwork generate the update when k=0? The method uses carrier token states, but this setting removes all visual tokens from the decoder input sequence.

When (k=0), all visual carrier tokens are removed from the decoder input sequence, so the hypernetwork does not receive carrier-token states. Instead, it generates the LoRA updates using the remaining inputs: the decoder hidden states from the question/text tokens and the full-resolution visual features (F_t), which are provided directly to the hypernetwork through its cross-attention pathway. Thus, even when (k=0), the hypernetwork can access visual information without inserting visual tokens into the decoder sequence. We will clarify this distinction in Figure 4 and the accompanying description.

(Does not affect my rating) I think Figure 3 in the main paper is not very informative, especially given how much space it takes up. I suggest moving it to the appendix or replacing it with a more informative visualization.

We will revise this figure in the revised version

We will incorporate all additional results and explanations presented in the rebuttal into the revised manuscript. Thank you again for your support and recognition of our work. We would be happy to provide any further clarification or address any remaining concerns. If we have fully addressed your concerns, we kindly ask you to consider reassessing your score.

Thank you again for your time and effort.

We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.

We would be happy to address any remaining concerns or provide further clarification. If our responses have addressed your remaining concerns, we would greatly appreciate it if you could consider strengthening your support for the paper.

Reviewer_2UjmOpenReview ↗

Final Rating

I would like to thank the authors for their detailed rebuttal, especially the additional experiments, which address most of my concerns. After carefully reviewing the responses from all reviewers and the revised manuscript, I am happy to raise my rating. I would encourage the authors to further strengthen the newly added experiments, given the additional time available before the final version is due.

Reviewer_fojcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WIRE combines a short visual input with temporary input-conditioned weight updates for a frozen video language model. A question-conditioned bidirectional Mamba2 compressor replaces full-resolution visual tokens with carrier tokens. Hypernetworks at selected decoder layers use compressed hidden states, a readout of the removed visual features, and the question state to generate low-rank updates for attention and MLP projections. The updates are applied during prefill and merged before decoding. The system is trained by context distillation from the same backbone with uncompressed input. On Qwen3-VL-8B, the 16-token configuration reduces the visual stream by 3.5 times and decoder prefill time by 3.35 times at 256 frames. Benchmark comparisons and paired update toggles evaluate accuracy and the additional contribution of the generated updates.

Critical strengths
  • Combining physical sequence contraction with a generated weight correction is an interesting way to reduce repeated decoder work. The timing table includes the compressor, hypernetwork, and merge costs: at 256 frames, prefill decreases from 369 to 110 ms and TTFT from 550 to 323 ms for k=16. This provides a concrete practical benefit beyond a nominal token-reduction ratio.
  • The fixed-context paired analysis is useful. Holding the carrier tokens and question fixed while toggling the update gives whole-benchmark gains of +3.96 on Video-MME, +3.22 on MVBench, and +4.39 on LongVideoBench. This shows that the generated update contributes beyond the explicit carrier context in the trained system.
Critical weaknesses
  • The evidence does not adequately isolate the proposed weight pathway from a stronger trained compression system. The paired toggle removes a component from a jointly trained model; it is not a comparison with an independently trained carrier-only model using the same backbone, data, and distillation objective. The main efficient-model comparisons retain different backbones and training data, as the authors acknowledge. Moreover, the main model averages 59.1 versus 65.0 for the uncompressed backbone (Table 1). A matched carrier-only baseline is important for deciding whether this accuracy-efficiency trade-off justifies the added 0.98B trainable parameters.
  • Task-relevant improvement is not yet evidence that the main k=16 update recovers video-specific information removed by contraction. The paper acknowledges this limitation, and Appendix C makes it especially important: the weight-only k=0 model shows a video-mismatch contrast at 256 frames, but the analogous contrast is null for k=6 through k=16 in the needle task. This does not disprove a useful weight pathway, but it leaves the stronger recovery interpretation unsupported on the main benchmark setting. The text should consistently distinguish the paired accuracy gain from evidence of video-specific encoding.
Questions for authors
  1. Does the weight pathway add value over a compression model trained to work without it? Turning off the generated update while holding carrier tokens fixed is a useful intervention, but the carrier compressor was trained jointly with that update and may have learned to rely on it. The intervention therefore measures dependence within the trained system, rather than the best performance available from carrier tokens alone. Could the authors compare with an independently trained carrier-only system using the same frozen backbone, training examples, and distillation objective?

  2. What evidence shows that the generated update carries information specific to the target video? The paired update gains show that the update is useful, but a generic task correction or a question-conditioned calibration could also improve accuracy. Appendix C includes a mismatched-video contrast for the weight-only model, yet reports a null analogous contrast for the main carrier-present models on the needle task.

3 comments

We thank the reviewer for their thoughtful assessment and constructive feedback. Below, we address each of the major and minor concerns, along with all questions raised.

The evidence does not adequately isolate the proposed weight pathway from a stronger trained compression system. The paired toggle removes a component from a jointly trained model; it is not a comparison with an independently trained carrier-only model using the same backbone, data, and distillation objective. The main efficient-model comparisons retain different backbones and training data, as the authors acknowledge. Moreover, the main model averages 59.1 versus 65.0 for the uncompressed backbone (Table 1). A matched carrier-only baseline is important for deciding whether this accuracy-efficiency trade-off justifies the added 0.98B trainable parameters.

We address this concern with an independently trained compressor-only baseline, rather than disabling the hypernetwork in a jointly trained WIRE model. Specifically, we train a BIMBA variant adapted to Qwen3-VL-8B, using the same backbone, training data, and distillation objective as WIRE, but without the hypernetwork or generated LoRA. This provides a direct comparison between carrier-based visual compression and the full WIRE method.

ModelVideoMMEMVBench
Independently trained compressor-only (BIMBA-style), k=1662.160.2
Full WIRE, k=1666.262.8

Full WIRE improves over the independently trained compressor-only baseline by 4.1 points on Video-MME and 2.6 points on MVBench. This comparison isolates the additional benefit of the generated weight updates beyond what can be achieved by a separately trained compression system. We also evaluate simpler compression methods under the same backbone and token budget, including uniform token selection, spatial average pooling, and spatial bilinear pooling. Their results are 60.6/59.2, 61.2/59.8, and 61.5/59.8 on Video-MME/MVBench, respectively. WIRE outperforms all of them.

Task-relevant improvement is not yet evidence that the main k=16 update recovers video-specific information removed by contraction. The paper acknowledges this limitation, and Appendix C makes it especially important: the weight-only k=0 model shows a video-mismatch contrast at 256 frames, but the analogous contrast is null for k=6 through k=16 in the needle task. This does not disprove a useful weight pathway, but it leaves the stronger recovery interpretation unsupported on the main benchmark setting. The text should consistently distinguish the paired accuracy gain from evidence of video-specific encoding.

We agree that the accuracy improvement from the generated updates should be distinguished from evidence that they recover video-specific information discarded during contraction. Our results establish the former, but do not conclusively establish the latter at the main (k=16) setting. As discussed in our response to Reviewer 1, we conducted additional NIAH ablations at (k=6) and 2048 frames. The full WIRE model achieves 75%, 67%, 75%, 75%, and 75% accuracy across retrieval positions (0%, 25%, 50%, 75%, and 100%). Removing the hypernetwork and LoRA updates reduces accuracy to 50%, 50%, 65%, 60%, and 50%. Furthermore, removing the full-resolution features (F_t) from the hypernetwork input, while retaining the generated updates, reduces accuracy to 45%, 40%, 60%, 50%, and 55%. These results suggest that the full-resolution features contribute useful information to the weight pathway in this long-context retrieval setting.

However, we acknowledge that the video-mismatch contrast observed for the weight-only ((k=0)) model at 256 frames does not persist in the same way for (k=6) through (k=16) in the needle retrieval experiment. Thus, these experiments do not establish that the updates recover discarded video-specific information in the main (k=16) setting.

Our independently trained compressor-only comparison provides complementary evidence for the utility of the weight pathway: it achieves 62.1 on Video-MME and 60.2 on MVBench, compared with 66.2 and 62.8 for full WIRE. This establishes a task-performance improvement beyond carrier compression alone, but does not by itself identify the source of that improvement as video-specific recovery. We will revise the paper to make this distinction explicit. We will describe the generated updates as providing question-conditioned weight-space adaptation that improves performance beyond carrier compression alone

We will incorporate all these results and explanations presented in the rebuttal into the revised manuscript. We would be happy to provide any further clarification or address any remaining concerns. If we have fully addressed your concerns, we kindly ask you to consider reassessing your score.

Thank you again for your time and effort.

We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.

We would be happy to address any remaining concerns or provide further clarification. If our responses have addressed your remaining concerns, we would greatly appreciate it if you could consider strengthening your support for the paper.

Thanks the author for providing the additional evidence! My concerns have been addressed and I will raise my score.

Reviewer_jQXgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WIRE, a framework for efficient long-video understanding that combines visual token compression with input-conditioned weight adaptation. Unlike conventional token compression methods that rely only on the retained visual tokens, WIRE uses two complementary paths to preserve video information.

First, a question-conditioned bidirectional selective scan compresses each frame into a small set of carrier tokens, reducing the input sequence before the first decoder layer. Second, hypernetworks use both the compressed representations and the removed full-resolution visual features to generate temporary LoRA updates for selected attention and feedforward layers. The updates are merged into the frozen backbone, avoiding additional hypernetwork computation during autoregressive decoding. The compressor and hypernetworks are trained through distillation from the uncompressed frozen model.

Experiments on Qwen3-VL-8B show a 3.5× reduction in visual tokens and up to 4.14× faster decoder prefill. WIRE achieves the highest average accuracy among the evaluated efficient video language models under a common 256-frame setting. Notably, paired ablations with identical compressed visual contexts show that the generated LoRA updates improve benchmark accuracy by 3.2–4.4 percentage points, demonstrating their contribution beyond token compression alone.

Critical strengths

The main strengths of this paper are its practical efficiency gains and the ablation studies supporting the proposed architecture. The experiments provide evidence for both the computational benefits and the contribution of the generated LoRA updates.

1. Computational efficiency with competitive performance

WIRE reduces visual tokens from 56 to 16 per frame before the decoder, lowering prefill computation and KV-cache usage. It achieves up to 4.14× faster decoder prefill and the highest average accuracy among the evaluated efficient video language models under a common 256-frame budget.

2. Effectiveness of generated LoRA updates

Paired ablations with identical compressed visual contexts show 3.22–4.39 percentage point improvements from the generated LoRA updates across three benchmarks. Additional gains on action counting and subtitle grounding suggest that the updates are useful for tasks sensitive to visual compression.

3. Component-level ablation studies

Table 3 evaluates the main compressor components. Removing question conditioning at test time reduces accuracy by 1.9 points, while retrained variants without bidirectional scanning or selective scanning show drops of 1.1 and 1.5 points, respectively. These results provide supporting evidence for the compressor design.

Critical weaknesses

Major Weaknesses

1. Limited methodological novelty

The authors acknowledge that the compressor is BIMBA-like and do not claim it as a primary contribution (Sec. 2). Since BIMBA also reduces the visual tokens passed to the decoder, the main novelty of WIRE lies in its generated LoRA pathway. However, Frames2LoRA (Suri et al., 2026, arXiv:2606.04351) has already explored generating video-conditioned LoRA adapters for a frozen VLM, but is not discussed in the paper. Overall, WIRE appears to mainly combine existing visual token compression and hypernetwork-based LoRA techniques, without a clearly established methodological contribution beyond their combination.

2. Unclear role of the generated LoRA updates

Section 5.2 claims that the generated LoRA updates recover visual information lost during compression. However, Table 2 only shows accuracy improvements from enabling LoRA, while disabling it introduces a train–test mismatch since the compressor and hypernetwork are trained jointly, so the paired difference may overstate the update's contribution. The hypernetwork also receives the question representation (Eq. 12), so the gains could come from better processing of retained features or answer calibration.

Appendix C shows no additional needle-retrieval gains from LoRA when carrier tokens are present, although some settings are near ceiling accuracy. The effect is also absent at k=6 with 2048 frames, where accuracy is around 73%. The mismatched-video control is also limited to k=0. A separately trained compressor-only baseline, mismatched-video updates at k=16, and an ablation of the full-resolution readout would help clarify the actual role of LoRA.

3. Missing same-backbone compression baselines

WIRE (k=16) achieves an average accuracy of 59.1, compared to 65.0 for the uncompressed Qwen3-VL-8B (Table 1). Although WIRE outperforms the evaluated efficient models, these models use different backbones and training settings, making it difficult to isolate the benefit of WIRE's architecture.

The paper lacks a comparison with simpler compression methods using the same Qwen3-VL-8B backbone and visual token budget, such as spatial pooling or token selection. The bilinear-only variant in Table 3 still uses generated LoRA updates, so it does not address this issue.

Minor Weaknesses

1. Limited inference efficiency evaluation

The latency experiments are conducted only at batch size one (Table 5). Since each input requires its own generated LoRA updates, it is unclear how well the method scales to batched or concurrent inference. The additional memory overhead from the compressor and hypernetwork is also not quantified alongside the KV-cache savings.

2. Incomplete ablation studies

In Table 3, question conditioning is removed only at test time, while the other compressor variants are retrained. This makes the results difficult to compare directly. The reported improvements are also relatively small, with no variance estimates.

Questions for authors

Experimental Comparisons

1. Same-backbone compression baseline

How does WIRE compare with simple spatial pooling or token selection methods using the same Qwen3-VL-8B backbone and visual token budget?

2. Separately trained compressor-only baseline

Would it be possible to compare WIRE with a compressor-only model trained using the same distillation objective? This would help clarify the effect of disabling LoRA in a jointly trained model.

3. Video specificity of generated LoRA

Are results available at k=16 when the LoRA update is generated from a different video while keeping the question and carrier tokens unchanged?

4. Contribution of the full-resolution readout

Could the authors clarify the contribution of the full-resolution readout c(ℓ)c^{(\ell)} (Eq. 12), for example through an ablation without this input?

Clarifications

5. Relation to Prior Work

Could the authors clarify how WIRE differs from existing visual token compression and video-conditioned LoRA approaches, particularly the benefit of combining compressed carrier tokens with generated weight updates?

6. Paired update analysis

Which checkpoints, including the values of k, were used for Table 2? Could the authors also provide the overall benchmark accuracies for both the "Disc." and "Appl." settings, along with the complete subset results?

7. Inference overhead

Table 5 notes that per-example updates cannot be batched across examples. How does this limitation affect throughput under concurrent serving, and would the reported efficiency gains still hold in this setting?

8 comments
AuthorsOpenReview ↗

Response 1/3

We thank the reviewer for a careful reading and for concrete, actionable requests. Below, we answer every major and minor weakness and each question.

Major 1 - Methodological novelty; relation to BIMBA and Frames2LoRA (also Q5)

We agree that the compressor is BIMBA-like and do not claim it as a contribution (Sec. 2). Our contribution is the combination, and in particular three design choices that differ from prior video-conditioned LoRA work (a) Generator input. Frames2LoRA (Suri et al., 2026) maps frozen video encodings to an adapter so that a small VLM (SmolVLM2, 0.5B/2.2B) answers with zero visual tokens. WIRE's generators instead read the live forward pass of the frozen 8B decoder (the residual stream at nine depths) on the compressed input, together with a cross-attention readout of the full-resolution rows the decoder never sees. (b) Role of the update. Frames2LoRA's adapter replaces the visual input. WIRE's update complements a compressed input: the decoder keeps k tokens per frame and the update acts as a learned correction of the compressed decoder (for k = 16 it is largely question-conditioned rather than video-specific; see Major 2). This is what makes WIRE competitive on long-video benchmarks at 8B scale, where a zero-token adapter falls far behind. Our own k = 0 model (Appendix C) is the WIRE analogue of the Frames2LoRA setting. (c) Training signal. WIRE is trained by distillation against the same backbone with the full visual sequence (KL on the answer tokens), so no labels are used and the target is the uncompressed model itself. We will cite Frames2LoRA as concurrent work and states these differences with BIMBA, we will expand that paragraph with the comparison above.

Major 2 - Role of the generated LoRA updates

(a) Train test mismatch when disabling the update

(b) The hypernetwork also receives the question representation (Eq. 12), so the gains could come from better processing of retained features or answer calibration.

To further examine the role of video information in the generated updates, we compare against a compressor-only variant trained without the hypernetwork and LoRA updates. This provides a matched evaluation of whether the generated updates offer benefits beyond the compressed visual tokens. We note that Table 3 already examines the effect of question conditioning in the compressor, but this does not isolate the direct question input to the hypernetwork in Eq. 12. To address this concern, we additionally train a WIRE variant without providing the question representation to the hypernetwork, while keeping the question-conditioned compressor and the rest of the architecture unchanged. This allows us to assess how much the generated updates benefit from direct question conditioning, independently of the compressor's question-aware token selection.

ModelVideoMMEMVBench
Compressor-only (no hypernetwork / LoRA), K=1662.160.2
WIRE without question input to hypernetwork (trained from scratch)65.362.0
WIRE (compressor + hypernetwork + LoRA), K=1666.262.8

These results show that the generated updates provide consistent gains beyond compression alone, while removing direct question conditioning from the hypernetwork leads to a moderate performance drop, indicating that both visual information and question-conditioned adaptation contribute to WIRE's improvements.

(c) Needle retrieval (Appendix C)

We agree that the original NIAH results do not conclusively establish the contribution of LoRA when carrier tokens are present. For example, at (k=6) with 2048 frames, accuracy is 75%, 67%, 75%, 75%, and 75% at retrieval positions of 0%, 25%, 50%, 75%, and 100%, respectively. To further investigate this, we conduct two additional NIAH ablations. First, we evaluate a separately trained compressor-only model without the hypernetwork or LoRA updates, achieving 50%, 50%, 65%, 60%, and 50% across the five retrieval positions. Second, we remove the full-resolution features (F_t) from the hypernetwork input while retaining the hypernetwork and generated LoRA updates. This variant achieves 45%, 40%, 60%, 50%, and 55%. Both ablations show a substantial drop in retrieval accuracy compared with the original WIRE results. In particular, the drop when removing the full-resolution readout suggests that these features provide important video information to the hypernetwork, while the compressor-only results highlight the additional contribution of the generated updates beyond the carrier tokens alone.

AuthorsOpenReview ↗

Response 2/3

Major 3 - Same-backbone compression baselines, and simpler compression methods

We agree that comparisons across different backbones and training settings make it difficult to isolate the architectural contribution of WIRE. As discussed in our response to Major 2, we already provide controlled comparisons using the same Qwen3-VL-8B backbone and training setup. In particular, our compressor-only baseline, which removes the hypernetwork and LoRA updates, is essentially a BIMBA-style approach adapted to the Qwen3-VL-8B backbone. WIRE outperforms this baseline, showing the benefit of the generated weight updates beyond token compression alone. We also evaluate a WIRE variant without direct question input to the hypernetwork, further examining the contribution of question-conditioned adaptation. The corresponding results are provided in our response to Major 2. Regarding architecture generality, WIRE's weight-space adaptation mechanism is designed to be architecture-agnostic: it generates updates for a frozen backbone without requiring changes to its architecture or updates to its pretrained weights. The implementation must be adapted to the target model's layer structure, but the approach is not inherently tied to Qwen3-VL. We will clarify this point and acknowledge cross-backbone evaluation as an important direction for future work.

Additionally, we compare WIRE with simpler compression methods using the same Qwen3-VL-8B backbone and a visual token budget of (k=16). We evaluate uniform token selection, spatial average pooling, and spatial bilinear pooling. These baselines compress the visual features to 16 tokens per frame without using the WIRE hypernetwork or generated LoRA updates. All methods follow the same training and evaluation setup.

Compression MethodVideo-MMEMVBench
Uniform Token Selection60.659.2
Spatial Average Pooling61.259.8
Spatial Bilinear Pooling61.559.8
Wire66.262.8

WIRE outperforms the strongest simple compression baseline, spatial bilinear pooling, by 4.7 points on Video-MME and 3.0 points on MVBench. These results suggest that the improvement comes not only from reducing the visual token count, but also from WIRE's weight-space adaptation.

Minor 1 - Batched and concurrent inference; memory overhead

We conducted additional experiments to evaluate WIRE under batched and concurrent inference, including the case where every request has a different generated LoRA update. Batched decoding with per-request updates. Although updates generated for different requests cannot be merged into a single shared weight matrix, they can be applied independently using batched low-rank matrix products. We implement this using unmerged LoRA updates (rank 64, across nine layers and both attention and MLP projections), following the general approach used in multi-LoRA serving systems such as S-LoRA [1] and Punica [2] . We measured decode throughput on a single NVIDIA B200 (192 GB) at 256 input frames, with a different LoRA update for every request. We compare against the uncompressed Qwen3-VL-8B baseline. The ViT is excluded because it is identical in both methods. The reported throughput and memory are measured during GPU decoding.

BatchsizeQwen3-VL-8B (tok/s)WIRE, k=16 (tok/s)Qwen3-VL-8B peak memory (GB)WIRE peak memory (GB)
112011617.818.9
436939022.221.0
16803108439.329.1
321033163462.240.0
6411982175107.961.8
9612672424153.783.5
128OOM2591OOM105.4

The throughput advantage grows with batch size: WIRE achieves 2,591 tokens/s at batch size 128, while the uncompressed baseline runs out of memory. At batch size 96, WIRE reaches 2,424 tokens/s compared with 1,267 tokens/s for the baseline. This is also reflected in memory usage. While the compressor and hypernetwork add 0.98B parameters (1.97 GB in BF16), WIRE reduces the KV cache from approximately 1.49 GB to 0.59 GB per request. Consequently, the memory overhead is offset from batch size four onward, allowing substantially higher concurrency.

AuthorsOpenReview ↗

Response 3/3

Overall, these experiments show that per-request generated LoRA updates do not prevent efficient batched inference. WIRE maintains a small decoding overhead at batch size one and achieves increasingly higher throughput and concurrency at larger batch sizes.

We will incorporate all additional results and explanations presented in the rebuttal into the revised manuscript. We would be happy to provide any further clarification or address any remaining concerns. If we have fully addressed your concerns, we kindly ask you to consider reassessing your score.

[1] : https://arxiv.org/abs/2311.03285

[2] : https://arxiv.org/abs/2310.18547

Comments on the paper

1 comment

We thank all reviewers for their careful and constructive feedback. Below we summarize the additional experiments, corrections and revisions made during the discussion. Reviewer jQXg's follow-up questions led to several corrections, and these supersede the corresponding numbers in our earlier responses to Reviewers fojc and 2Ujm.

1. Same-backbone baselines (jQXg, fojc). All rows use Qwen3-VL-8B, the same training data, the same distillation objective and k=16 tokens per frame.

ModelVideo-MMEMVBench
Uniform token selection60.659.2
Spatial average pooling61.259.8
Spatial bilinear pooling61.559.8
Compressor-only, trained separately (BIMBA-style, no hypernetwork/LoRA)62.160.2
Compressor + shared (non-generated) LoRA, same placement and rank63.661.8
WIRE without question input to the hypernetwork65.362.0
WIRE66.262.8

A static LoRA recovers part of the gain over compression alone (+1.5 / +1.6). The generated update adds a further +2.6 / +1.0 over standard LoRA adaptation.

2. Needle retrieval, corrected (jQXg; supersedes the numbers in our responses to fojc and 2Ujm). Our first responses reported a preliminary subset with 20 needles per position. We re-ran everything on the full Appendix C protocol: 5 positions × 24 needles, with an identical paired item list for every variant. Setting: k=6, 2048 frames.

Variant0%25%50%75%100%Mean
WIRE75.066.775.075.075.073.3
Compressor-only (trained separately)54.250.062.558.354.255.8
WIRE without F_t (test-time ablation)50.045.858.354.250.051.7

The F_t variant removes the full-resolution readout at inference time and is not retrained. It shows how much the trained hypernetwork relies on F_t. 3. Training objective, corrected (jQXg; supersedes the labels in our response to 2Ujm). All main results use Eq. (17): KL distillation to the uncompressed backbone (τ = 1) plus answer-token cross-entropy with λ = 0.1. Our earlier statement that WIRE is trained "without labels" was incorrect. At k=16:

ObjectiveVideo-MMEMVBench
KL + 0.1 CE (Eq. 17, main results)66.262.8
KL only (λ = 0)66.062.3
KL + 1.0 CE65.962.8
CE only (no distillation)64.161.4

4. Inference efficiency (jQXg). Measured on a single B200 at 256 frames and k=16, with a separate generated LoRA for every request.

  • Decode throughput, per-request unmerged LoRA:
    • Batch 96: 2,424 tok/s for WIRE versus 1,267 tok/s for the baseline.
    • Batch 128: 2,591 tok/s for WIRE; the baseline runs out of memory.
  • Memory:
    • The compressor and hypernetwork add 1.97 GB (0.98B parameters in BF16).
    • The KV cache shrinks from 1.49 GB to 0.59 GB per request, so the added weights are paid back from batch size 4 onward.
  • End-to-end, covering ViT, compressor, hypernetwork, prefill and decode:
    • Per-request prefill falls from 402 ms to 273 ms; the compressor and hypernetwork add about 20 ms of that.
    • At each method's maximum batch size, WIRE serves 1.65× more requests/s with 128 output tokens and 1.93× more with 512.

5. Relation to prior work (jQXg). The revision cites Frames2LoRA (Suri et al., 2026) as concurrent work and states three differences.

  • Generator input. WIRE's generator reads the live forward pass of the frozen 8B decoder plus a full-resolution readout, not frozen encoder features.
  • Role of the update. WIRE's update corrects a shortened visual input; it does not replace the visual input entirely.
  • Training signal. WIRE uses self-distillation from the uncompressed backbone.

We have incorporated all additional results and explanations presented in the rebuttal into the revised manuscript. We would be happy to provide any further clarification or address any remaining concerns. If we have fully addressed your concerns, we kindly ask you to consider reassessing your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.