acceptodds
Under review as a conference paper at ICLR 2027

Compute Once, Update Many: Sharing Computation for Efficient Video-LLM Understanding

Abstract

Video-language models spend substantial computation repeatedly processing visual tokens during language decoder prefill. Existing acceleration methods often reduce this cost by removing or merging tokens, but retaining a token's individual representation need not require computing its update independently. We introduce a carrier–receiver computation sharing method that reduces repeated computation while updating each visual token's own hidden state. Selected visual carriers compute Attention and feed-forward network (FFN) updates, which receivers add to their own residual states. All token positions and key/value entries are retained, and non-visual tokens are processed normally. Input-dependent assignments identify which tokens share updates, while calibrated layer-wise ratios control the amount of sharing. Across three model families, our method maintains accuracy close to dense inference on MVBench, VideoMME, and LongVideoBench, while delivering measured decoder prefill speedups of – at 32 frames and consistent acceleration from 8 to 64 frames. Beyond standalone acceleration, our method consistently outperforms further token reduction alone when combined with four visual token reduction methods, improving accuracy by up to percentage points at matched decoder prefill FLOPs across the three benchmarks. These results show that sharing updates offers an effective way to reduce decoder computation while keeping visual tokens available for subsequent attention.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Efficient Inference, Computation Sharing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vY5GOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper proposes carrier–receiver computation sharing to accelerate the language decoder prefill stage of video-language models. The method computes Attention and FFN updates only for selected carrier tokens, while receiver tokens reuse the corresponding carrier updates. The author reports accuracy close to dense inference, with relative efficiency gains.

Critical strengths
  1. The paper distinguishes between retaining a token’s representation and independently computing its update. Sharing updates while preserving separate residual states provides a potential alternative to token removal.

  2. The experiments cover three model families and three established video understanding benchmarks.

Critical weaknesses
  1. My primary concern is the completeness of the paper and the scope of the work. The presentation is difficult to follow, particularly because the method lacks sufficient motivation to establish the need for carrier–receiver sharing. In addition, there is substantial unused space around the tables on pages 7 and 8, and the main text does not reach nine pages. Taken together, these issues considerably weaken my impression of the manuscript’s completeness.

  2. The evidence for the carrier assignment strategy is limited. The method assumes that similarity in projected hidden features identifies tokens whose operator updates can be shared. However, input similarity does not necessarily imply update similarity. The paper would benefit from providing evidence and observation on this point.

  3. Part of the experiment is incomplete. For example, accuracy is evaluated at 32 frames, while the experiments at other frame counts report latency only. Consequently, the results do not establish whether quality preservation remains consistent as the visual sequence grows.

Questions for authors
  1. What evidence supports the assumption that tokens with similar hidden states require similar Attention or FFN updates?

  2. How does carrier–receiver sharing compare with independently optimized visual token skipping methods at matched computational budgets or measured latency? The zero-update ablation uses a schedule calibrated for sharing, which may not provide a fair comparison with skipping.

5 comments
AuthorsOpenReview ↗

Response (1/3)

Dear Reviewer vY5G,

We thank you for the detailed review. Based on your feedback, we have conducted additional experiments and revised the manuscript to improve clarity.

W1: Motivation, presentation, and completeness of the manuscript.

1. Clarifying motivation: Our original motivation, as described in the Introduction, is to reduce redundant Attention and FFN computation during decoder prefill. We pursue this goal by exploiting redundancy in operator updates: visual tokens can retain distinct information while requiring similar updates. Carrier–receiver sharing therefore separates retaining a token's representation from independently computing its update. Carriers compute the original operator updates, and receivers add the assigned updates to their own residual states. This reduces repeated computation while preserving individual token representations, positions, and K/V entries. The additional analysis in W2 and Q1 below provides direct evidence linking input similarity to update similarity, further supporting the rationale for carrier–receiver sharing.

2. Improving presentation: We have conducted new experiments on scaling, generalization, and sharing mechanics. We have incorporated the core findings into the main text and appendix, and revised the manuscript layout to make it more compact and easier to read.

AuthorsOpenReview ↗

Response (2/3)

W2 & Q1: Evidence supporting the assumption that input similarity implies update similarity.

To verify that input feature similarity indicates update similarity, we conducted two analyses on Qwen3-VL-8B using 70 representative videos: 40 from MVBench covering all tasks and 30 from VideoMME covering short, medium, and long durations.

1. Correlation between Input Similarity and Update Similarity

We measured Spearman rank correlations between input-feature cosine similarity and operator-update cosine similarity across sampled actual carrier–receiver pairs, using inputs and updates from the same shared layer. We computed correlations for each video and shared layer, averaged across layers within each video, and then averaged across videos separately for each benchmark.

Table A: Correlation between Input Similarity and Update Similarity

DatasetOperatorSpearman Correlation
MVBenchAttention0.7052
VideoMMEAttention0.6985
MVBenchFFN0.8939
VideoMMEFFN0.8422

Taking FFN as an example, every token in a layer is processed by the same function:

f(x)=Wdown[SiLU(Wgatex)⊙Wupx].f(x)=W_{\mathrm{down}}\left[\mathrm{SiLU}(W_{\mathrm{gate}}x)\odot W_{\mathrm{up}}x\right].

The same function ff is applied to every token. Within a bounded input neighborhood,

∥f(xi)−f(xc)∥2≤L∥xi−xc∥2,\lVert f(x_i)-f(x_c)\rVert_2 \leq L\lVert x_i-x_c\rVert_2,

where LL measures how much the FFN can amplify input differences. When this amplification is moderate, similar inputs produce similar updates, motivating computation sharing.

A concrete example occurs in MVBench video action_antonym:027. At FFN layer 34, receiver token 336 is assigned to carrier token 547. Their input cosine similarity is 0.9500, and the cosine similarity between the receiver's independently computed reference update and the carrier's update is 0.9449. The carrier-to-receiver update norm ratio is 0.9840, showing that the updates are close in both direction and magnitude. Copying the carrier update yields a normalized squared error of 0.1087, compared with 1 for a zero update.

A similar observation holds for Attention. In the same video, at Attention layer 23, receiver token 265 is assigned to carrier token 256. The receiver's independently computed update and the carrier's update have a cosine similarity of 0.9502, and the carrier's update is only 1.1% larger in magnitude. Sharing this update yields a normalized squared approximation error of 0.1008.

The consistent positive correlations support the premise that similar inputs yield similar updates and validate our design.

2. Update Approximation Error

We measured the distance between each receiver's true dense update and its assigned shared update. Keeping the selected carrier set unchanged, we compared our input-feature-based matching with random assignment from the same pool of valid carriers.

Table B: Update Approximation Error (Our Matching vs. Random Matching)

DatasetOperatorMatch Error (Ours)Random ErrorRelative Error Reduction
MVBenchAttention0.6890.99530.8%
VideoMMEAttention0.6950.99630.2%
MVBenchFFN0.3831.40572.7%
VideoMMEFFN0.5031.49366.3%

Compared to random assignment within the same carrier pool, our matching strategy consistently lowers the approximation error, providing empirical support for our assignment mechanism. This discussion is now in Section 4.6.

AuthorsOpenReview ↗

Response (3/3)

W3: More Experiments.

To verify quality preservation on longer visual sequences, we evaluated accuracy at 128 and 256 frames on Qwen3-VL-8B and Qwen2-VL-7B.

Table C: Accuracy (Dense vs. Ours)

ModelDatasetFramesDense Acc.Ours Acc.Δ (pp)
Qwen3-VL-8BVideoMME12867.8166.63-1.19
Qwen3-VL-8BVideoMME25669.9668.89-1.07
Qwen3-VL-8BLongVideoBench12865.0064.85-0.15
Qwen3-VL-8BLongVideoBench25666.1265.82-0.30
Qwen2-VL-7BVideoMME12860.8159.63-1.19
Qwen2-VL-7BVideoMME25661.9360.78-1.15
Qwen2-VL-7BLongVideoBench12856.2556.39+0.15
Qwen2-VL-7BLongVideoBench25656.3256.69+0.37

Accuracy remains stable as frame count increases. Beyond these frame-scaling results, we conducted additional experiments to further evaluate our method, including calibration on STAR, scaling to a larger model, and analysis of Attention sharing efficiency. Please see the Global Response for a summary of these experiments and the corresponding manuscript updates.

Q2: Fair comparison with independently optimized visual token skipping methods.

Our original zero-update ablation used a schedule calibrated specifically for sharing. To compare against independently optimized skipping, we evaluated against Op-Skip (Luo Z, et al., 2026), a representative visual skipping method. Since an off-the-shelf implementation for our specific datasets and models was unavailable, we adapted their official codebase for Qwen3-VL-8B and Qwen2-VL-7B, utilized their official operator risk assessment for optimal layer selection, and aligned their FLOPs budget with ours.

Table D: Share vs. Independently Optimized Op-Skip at Matched FLOPs

ModelDatasetDense Acc.Op-Skip Acc.OursΔ (Ours vs Skip)Matched FLOPs (Ours / Op-Skip)
Qwen3-VL-8BMVBench66.60%66.08%66.63%+0.55 pp58.19% / 58.49%
Qwen3-VL-8BVideoMME62.48%59.81%61.81%+2.00 pp58.23% / 58.53%
Qwen2-VL-7BMVBench65.03%62.23%64.90%+2.68 pp56.59% / 56.55%
Qwen2-VL-7BVideoMME56.11%53.11%55.44%+2.33 pp56.68% / 56.65%

Our method consistently achieves higher accuracy than Op-Skip under equivalent computational budgets. This demonstrates the benefit of preserving individual residual states and sharing updates over simply skipping operators entirely.

Reviewer_McdHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a carrier–receiver computation-sharing framework to accelerate Video-LLM decoder prefill. Instead of pruning visual tokens, the method allows selected carrier tokens to compute Attention and FFN updates, which are then shared with receiver tokens while preserving their individual hidden states and key/value entries. The authors further introduce operator-specific carrier selection strategies and calibrated layer-wise sharing ratios to balance accuracy and efficiency.

Critical strengths
  1. The method reduces redundant computation while preserving individual token representations and all key/value entries, maintaining visual information for subsequent attention.
  2. The method adopts different carrier selection strategies for Attention and FFN, appropriately accounting for their distinct computational characteristics.
Critical weaknesses
  1. The method relies heavily on layer-wise sharing ratios calibrated on a small dataset. Its effectiveness may not generalize well to videos with different motion patterns, visual complexity, or task requirements.
  2. Attention sharing provides relatively limited FLOPs reduction but introduces additional carrier selection and gather/scatter overhead. Its actual latency benefits remain insufficiently evaluated.
Questions for authors

N/A

1 comment

Dear Reviewer McdH,

We thank you for your constructive feedback and the recognition of our core design. Below, we address your comments:

W1: Generalizability of the sharing ratios calibrated on a small dataset.

Regarding whether sharing ratios calibrated on a small dataset generalize, our experiments indicate they adapt well to unseen distributions. We originally applied ratios calibrated on 24 videos from VideoChat2-IT directly to MVBench, VideoMME, and LongVideoBench without dataset-specific tuning. The stable accuracy indicates consistent performance across varying contexts.

To further validate this, we conducted an additional calibration using 24 videos from the STAR dataset (Wu B, et al., 2024), which is built on Charades videos and introduces a shift in video source, scene content, time span, and reasoning tasks relative to VideoChat2-IT.

Table A: Schedule calibrated on the STAR dataset (Target S=0.5S=0.5)

Calibration SetOperatorB1B2B3B4B5B6
STAR (24 videos)Attention0.1251.0001.0000.5000.1250.125
FFN1.0000.7501.0000.1250.1250.125

As shown in Table 11 (Appendix B.2), the STAR-calibrated schedule maintains a similar layer-wise structure to our default VideoChat2-IT schedule. We evaluated this new schedule on the fixed 500-example subsets of MVBench and VideoMME:

Table B: Accuracy of the STAR-calibrated schedule

Calibration SetMVBench Acc.Δ (pp)VideoMME Acc.Δ (pp)
Dense66.60%—60.20%—
STAR (24 videos)66.60%0.0060.00%-0.20

These results show that the layer-wise redundancy pattern and downstream performance remain consistent despite the shift in calibration data. This supports that sharing ratios reflect intrinsic architectural properties rather than calibration data specifics.

W2: Limited FLOPs reduction of Attention sharing and its gather/scatter overhead.

While Attention sharing yields a modest FLOPs reduction on short sequences compared to FFN sharing, the computational cost of dense Attention grows quadratically with sequence length. Analytically, the FLOPs ratio of the Dense Attention branch to the Dense FFN branch is:

FAttentionFFFN=4Nd2+4Nddkv+4N2d6Ndm=2(d+dkv+N)3m\frac{F_{\mathrm{Attention}}}{F_{\mathrm{FFN}}} = \frac{4Nd^2+4Nd d_{\mathrm{kv}}+4N^2d}{6Ndm} = \frac{2(d+d_{\mathrm{kv}}+N)}{3m}

Here, NN is the total token count, dd is the hidden dimension, dkvd_{\mathrm{kv}} is the K/V projection width, and mm is the FFN intermediate width. For a fixed model, this ratio grows linearly with NN, and Attention computation exceeds FFN computation once N>32m−d−dkvN > \frac{3}{2}m-d-d_{\mathrm{kv}}. Therefore, mitigating Attention computation becomes essential for longer videos. We profiled the latency breakdown from 8 to 256 frames on Qwen3-VL-8B on 30 VideoMME videos. Reduction is the median per-input decrease relative to FFN-only sharing:

Table C: Measured Net Latency and the Additional Benefit of Attention (ms)

FramesDenseFFN-onlyAttn + FFN (Ours)Net Latency Reduction
8160.1133.8129.33.29%
32683.9528.5489.47.43%
1282810.72139.61936.09.54%
2564860.53907.13454.211.60%

Table D: Attention Routing Overhead during Decoder Prefill (Qwen3-VL-8B)

FramesRouting Overhead (ms)Proportion of Total Latency
323.140.64%
644.030.53%
1286.700.35%
25611.270.33%

The routing overhead accounts for only 0.3% to 0.6% of the total prefill latency. The net latency reduction from Attention sharing increases steadily, from 3.29% at 8 frames to 11.60% at 256 frames, showing that the computational savings offset the gather/scatter overhead. Table 6 in the main manuscript also shows that incorporating Attention sharing maintains accuracy at shorter frame counts.

Reviewer_DXYoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a carrier–receiver computation-sharing method to reduce the prefill cost of video LLM inference. The method computes updates for selected carrier tokens and shares them with receiver tokens while preserving their individual residual states. The method can also be combined with visual token reduction approaches. Experiments using Qwen3-VL-8B, Qwen2-VL-7B, and LLaVA-OneVision-7B on MVBench, VideoMME, and LongVideoBench show reduced prefill computation and latency, with competitive accuracy.

Critical strengths
  1. Simple yet effective. The proposed computation-sharing mechanism reduces redundant attention and FFN computation with a simple sharing-receiving flow, while the computation reduction and the accuracy are competitive. It can be combined with existing token reduction methods, making it potentially useful in practical applications.
  2. Consistent gains under matched compute budgets. Across baselines with matching computation budget, the proposed method generally achieves better accuracy than further reducing the number of tokens. The proposed method have the potential to mitigate the accuracy loss caused by token reduction.
Critical weaknesses
  1. Evaluation focuses on shot video. The latency experiments cover up to 64 frames. This corresponds to only 8 seconds for FPS=8. The scalability of the method for longer videos is unclear. Evaluations with more frames would help establish whether the reported speedups and accuracy retention hold for longer videos.
  2. Limited evaluation at larger model scales. The experiments are limited to models around 7B–8B parameters. It remains unclear whether the proposed method generalizes effectively to larger video LLMs, at least 30B-scale.
  3. Analysis of benchmark-dependent accuracy changes. The results show different sensitivity to computation sharing across benchmarks, particularly between MVBench and VideoMME. However, the paper does not investigate the reasons for these differences, making it difficult to understand which types of video understanding tasks or video inputs are most affected by the method.
Questions for authors

Why is MVBench relatively robust to computation sharing, while VideoMME exhibits larger accuracy degradation at higher sharing levels? Can the authors provide statistics of the two benchmarks to show the video data distribution, or provide case-level analysis?

2 comments
AuthorsOpenReview ↗

Response (1/2)

Dear Reviewer DXYo,

We thank you for your review and the recognition of our computation-sharing mechanism. Below, we address your concerns:

W1: Scalability for longer videos.

Regarding scalability, our original evaluations utilize "uniformly sampled frames" spanning the entire video duration. VideoMME and LongVideoBench already evaluate long-context reasoning over videos spanning minutes. To further validate our method on denser visual sequences, we scaled our experiments to 128 and 256 frames. As frame density increases, spatiotemporal redundancy naturally rises. Our carrier-receiver mechanism explicitly exploits this redundancy by allowing similar tokens to share updates while retaining their individual residual states and positions.

Table A: Measured Decoder Prefill Latency (Qwen3-VL-8B, Qwen2-VL-7B)

ModelFramesVisual TokensDense LatencyOurs LatencySpeedup
Qwen3-VL-8B1287,1682217.4 ms1536.5 ms1.443×
Qwen3-VL-8B25614,3366020.1 ms4292.9 ms1.402×
Qwen2-VL-7B1289,2162,248.2 ms1,561.9 ms1.439×
Qwen2-VL-7B25618,4325,467.8 ms3,881.4 ms1.409×

Table B: Accuracy (Dense vs. Ours)

ModelDatasetFramesDense Acc.Ours Acc.Δ (pp)
Qwen3-VL-8BVideoMME12867.8166.63-1.19
Qwen3-VL-8BVideoMME25669.9668.89-1.07
Qwen3-VL-8BLongVideoBench12865.0064.85-0.15
Qwen3-VL-8BLongVideoBench25666.1265.82-0.30
Qwen2-VL-7BVideoMME12860.8159.63-1.19
Qwen2-VL-7BVideoMME25661.9360.78-1.15
Qwen2-VL-7BLongVideoBench12856.2556.39+0.15
Qwen2-VL-7BLongVideoBench25656.3256.69+0.37

Table C: Sharing vs. Token-only at Matched FLOPs (Qwen3-VL-8B, VideoMME with VidCom²)

FramesBase (VidCom²)+OursToken-only (Matched FLOPs)Δ (+Ours vs Token-only)
12859.3358.4156.48+1.93 pp
25661.6761.1558.19+2.96 pp

These results demonstrate that sharing updates reduces computation in dense sequences, yielding a consistent decoder prefill speedup of approximately 1.4× while maintaining stable accuracy.

W2: Generalization to larger model scales.

To verify generalizability to larger parameter scales, we evaluated Qwen3-VL-32B-Instruct using the identical procedure. Due to computational and time constraints, we focused on MVBench at 32 frames and VideoMME at 256 frames. The inherent spatiotemporal redundancy in videos and the structural mechanics of Attention and FFN remain consistent across model sizes.

Table D: Scalability to 30B-scale Models (Qwen3-VL-32B-Instruct)

DatasetFramesDense Acc.Ours Acc.Δ (pp)Decoder prefill FLOPs(% of dense)
MVBench3270.8070.98+0.1856.91%
VideoMME25674.5273.74-0.7854.79%

These results confirm that our framework generalizes effectively to larger video LLMs, showing that carrier-receiver sharing leverages universal visual redundancy rather than specific model capacities.

AuthorsOpenReview ↗

Response (2/2)

W3 & Q1: Benchmark-dependent sensitivity and difference between MVBench and VideoMME.

The differing sensitivity between MVBench and VideoMME stems from their distinct temporal information density. MVBench consists of short clips focused on localized actions, whereas VideoMME evaluates long-context narrative reasoning.

Table E: Video Duration and Temporal Sampling Statistics under 32-Frame Sampling

BenchmarkMedian input durationMedian sampling interval
MVBench10.65 s0.34 s
VideoMME487.94 s15.74 s

Densely sampled short clips provide repeated observations of the same event, offering high redundancy. In longer videos, sampled frames cover more diverse scenes, placing greater information demands on individual observations under the same frame budget. The following case analysis illustrates how different sharing levels affect specific questions:

MVBench Case — "Character Order", Question #133 (The 28.96-second clip is represented by 32 frames, averaging 0.93 s apart.)

Table F: MVBench Case Study

QuestionGround truthDenseS=0.5S=0.625
What letters did the person show in order?B.cehiB.cehi ✓B.cehi ✓B.cehi ✓

VideoMME Case — Two questions from the same video (Video 776, Rugby Match) (The 51.49-minute video is represented by 32 frames, averaging 99.66 s apart.)

Table G: VideoMME Case Study

Question and categoryGround truthDenseS=0.5S=0.625
Competition and winner (Info Synopsis, 776-1)D.New Zealand wins the Hong Kong Sevens final.D (Correct ✓)D (Correct ✓)D (Correct ✓)
Fiji's score at the end of the first half (OCR, 776-3)D.7D.7(Correct ✓)D.7 (Correct ✓)A.1(Wrong ×)

These examples show that temporally concentrated evidence tolerates sharing well, while precise information tied to specific moments is more sensitive. The default S=0.5S=0.5 preserves all three answers. We have added this analysis to Appendix F (Figure 4).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to the Area Chair and Reviewers

We sincerely thank all the reviewers for taking the time to review our manuscript and for providing thoughtful and constructive feedback. The changes summarized below are marked in blue in the revised manuscript.

  • We clarified the motivation for carrier–receiver sharing in the Introduction and added analyses of input–update similarity and update approximation error in Section 4.6 (Table 5). (vY5G, W1 and W2/Q1)
  • We added 128- and 256-frame accuracy results to Table 1 and corresponding decoder prefill latency measurements to Appendix B.6 (Table 15). (DXYo, W1; vY5G, W3)
  • We clarified frame sampling and accuracy rounding in Section 4.1 and the scope of decoder prefill timing in Section 4.5.
  • We added a comparison with Op-Skip at closely matched decoder prefill FLOPs to the main text (Section 4.7, Table 8). (vY5G, Q2)
  • We added a calibration experiment using STAR as an alternative data source in Appendix B.2 (Table 11). (McdH, W1)
  • We added an evaluation of Qwen3-VL-32B-Instruct on MVBench and VideoMME in Appendix D (Table 17). (DXYo, W2)
  • We added measurements of the latency benefit and routing overhead of Attention sharing in Appendix E (Tables 18–19). (McdH, W2)
  • We added case studies from MVBench and VideoMME in Appendix F (Figure 4), illustrating how different questions respond to increased sharing. (DXYo, W3/Q1)
  • We have reorganized the structure of the paper and optimized the layout of the figures and tables. (vY5G, W1)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.