acceptodds
Under review as a conference paper at ICLR 2027

NEOPrune: Non-rEconstructible Orthogonal Token Pruning for Omni Modal Models

Abstract

Omni-modal large language models (OmniLLMs) process large numbers of audio and video tokens, incurring substantial computational and memory costs. Existing token pruning methods primarily use token importance or pairwise similarity, leaving the collective replaceability of layer-wise token updates underexplored. We introduce NEOPrune, a training-free two-stage method that estimates replaceability by reconstructing orthogonalized layer-wise updates, focusing on information newly introduced at each layer. In the Pre-LLM stage, it combines within-modality reconstruction with encoder attention; in the Inner-LLM stage, it estimates video-update replaceability conditioned on audio and combines it with text-query attention. Across four audio–visual benchmarks and multiple OmniLLMs, NEOPrune generally achieves higher accuracy than evaluated pruning baselines at matched token-retention ratios, while retaining near-full-token performance with only 35–45% of audio–visual tokens.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
OmniLLM, Multimodal Large Language Model, Token Pruning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_vF4BOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NEOPrune, a training-free token pruning method for omni-modal large language models (OmniLLMs). It estimates token replaceability through the reconstruction error of layer-wise orthogonal updates. Before the LLM, it combines within-modality reconstruction with encoder attention to select audio and video tokens. Inside the LLM, it progressively prunes video tokens using audio-conditioned reconstruction and text-query attention. Experiments cover four audio-visual benchmarks, primarily using Qwen2.5-Omni, with additional Qwen3-Omni results on AVUT.

Critical strengths
  1. The paper is readable and clearly presented. The two-stage framework and reconstruction-based selection criteria are generally easy to follow.

  2. The motivation is reasonably clear. Assessing whether several tokens can collectively explain another token provides a sensible approach to identifying redundancy beyond pairwise similarity.

Critical weaknesses
  1. The model coverage remains limited. Although Appendix A.8 includes Qwen3-Omni-30B-A3B-Instruct, its evaluation is restricted to AVUT, without corresponding efficiency measurements. The authors should extend this evaluation across the other benchmarks and report latency and memory usage to establish broader applicability.

  2. The baseline comparison is insufficiently comprehensive. Although FastV and DyCoke are included, representative methods such as VisionZip [1], VidCom2 [2], HoliTom [3], and OmniSIFT [4] are missing. Comparisons should assess both accuracy and efficiency under clearly specified, comparable budgets.

  3. The task coverage is too narrow. The evaluation focuses on audio-visual QA without assessing temporal grounding or caption generation. Experiments on LongVALE [6] and AVSCapBench [7] would help establish whether compression preserves temporal evidence and detailed audio-visual information.

  4. The modest accuracy gains come with higher latency, leaving the practical advantage unclear. In Table 2, at 35% retention, NEOPrune achieves 46.50% accuracy versus 45.30% for OmniZip and 45.55% for DASH, gains of only 1.20 and 0.95 percentage points. However, its latency increases to 5.02 s from 4.40 s and 4.56 s, respectively. Its prefill time is also 25.0% and 21.3% higher. This pattern persists in Appendix Table 6: average latency over 30 videos is 20.4% and 22.5% higher than the two baselines. Comparisons at matched accuracy or latency, together with a breakdown of reconstruction overhead, are needed to establish a convincing accuracy–efficiency advantage.

[1] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. VisionZip: Longer is Better but Not Necessary in Vision Language Models. CVPR, 2025.

[2] Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models. EMNLP, 2025.

[3] Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. HoliTom: Holistic Token Merging for Fast Video Large Language Models. NeurIPS, 2025.

[4] Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, Yuanxing Zhang, Jiaheng Liu, Qiang Liu, Pengfei Wan, and Liang Wang. OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models. ICML, 2026.

[5] Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos. CVPR, 2025.

[6] Yanghai Wang, Jiahao Wang, Jiafu Tang, Yuanxing Zhang, Zhe Cao, Hanyan Bian, Zijie Zhang, Weiliang Luo, Zhiyu Pan, Zixuan Dong, Jiaheng Liu, and Zhaoxiang Zhang. AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning. arXiv:2607.12820, 2026.

Questions for authors

Could the authors provide broader model, baseline, and task evaluations, together with efficiency comparisons at matched accuracy, to demonstrate the general applicability and practical advantages of the proposed method?

7 comments

We thank the reviewer for the constructive feedback and for recognizing the motivation behind our reconstruction-based approach. We appreciate the call for a more rigorous evaluation of NEOPrune’s broader applicability and practical accuracy–efficiency trade-offs. We are currently conducting additional experiments to address these concerns. As an initial update, we plan to share verified results on additional datasets and comparisons with relevant existing methods under clearly specified, comparable budgets as soon as they become available. We will provide further results and analysis as the ongoing experiments are completed. Thank you for your patience and for the suggestions that are helping us strengthen the empirical evaluation.

AuthorsOpenReview ↗

Response to Reviewer vF4B — Part 1 of 4

1. Broader evaluation and efficiency on Qwen3-Omni-30B (W1 & Q)

W1: The evaluation of Qwen3-Omni-30B-A3B-Instruct is restricted to AVUT and lacks corresponding efficiency measurements. Broader benchmark coverage and efficiency results are needed to assess the applicability and practical benefits of NEOPrune on the larger model.

Q: “Could the authors provide broader model, baseline, and task evaluations, together with efficiency comparisons at matched accuracy, to demonstrate the general applicability and practical advantages of the proposed method?”

Response:

We agree that broader model evaluation should include both accuracy and practical inference costs. Using Qwen3-Omni-30B-A3B-Instruct and the same experimental setup as in the paper, we extended the accuracy evaluation beyond AVUT to ShortVid. The AVUT evaluation for Qwen3-Omni used 1,729 QA pairs. Table 1 reports accuracy on both benchmarks at 45% and 35% AV token retention, with newly added VidCom2 results at both ratios and a full-token (100% retention) reference. Table 2 provides inference latency and LLM prefill measurements on AVUT.

Table 1. Accuracy of Qwen3-Omni-30B-A3B on AVUT and ShortVid

MethodAV token retention (%)AVUT accuracy (%)ShortVid accuracy (%)
Full10077.374.9
NEOPrune4576.574.7
NEOPrune3575.874.5
OmniZip4575.774.6
OmniZip3574.673.8
DASH4575.274.7
DASH3574.273.4
VidCom24576.274.1
VidCom23575.473.6

Bold accuracy values indicate the highest observed accuracy among the compression methods at each retention ratio.

The full-token reference achieves 77.3% accuracy on AVUT and 74.9% on ShortVid. At 45% retention, NEOPrune achieves 76.5% and 74.7%, respectively, corresponding to differences of 0.8 and 0.2 percentage points from full-token inference. At 35% retention, it achieves 75.8% and 74.5%, with differences of 1.5 and 0.4 percentage points.

At 35% retention, NEOPrune’s observed accuracy is 1.2, 1.6, and 0.4 percentage points higher than OmniZip, DASH, and VidCom2, respectively, on AVUT. On ShortVid, it is 0.7, 1.1, and 0.9 percentage points higher than OmniZip, DASH, and VidCom2, respectively. At 45% retention, NEOPrune also has higher AVUT accuracy than all three baselines, exceeding VidCom2 by 0.3 percentage points, and matches DASH at 74.7% on ShortVid. These results extend the evidence for the larger model to a second benchmark.

For the efficiency evaluation on AVUT, all methods used the same 60 inputs, and each input was measured three times. Model inference latency and LLM prefill are reported as the mean ±\pm standard deviation across the three measurements, in seconds and milliseconds, respectively. Table 2 reports results at 45% and 35% retention.

Table 2. Accuracy and inference efficiency on AVUT at 45% and 35% retention

MethodAV token retention (%)Accuracy (%)Model inference latency (s)LLM prefill (ms)
Full10077.31.777±0.0051.777 \pm 0.0051,360.9±9.41,360.9 \pm 9.4
NEOPrune4576.51.563±0.0081.563 \pm 0.0081,102.4±8.01,102.4 \pm 8.0
NEOPrune3575.81.475±0.0051.475 \pm 0.0051,040.6±3.81,040.6 \pm 3.8
OmniZip4575.71.440±0.0011.440 \pm 0.0011,024.6±5.61,024.6 \pm 5.6
OmniZip3574.61.376±0.0031.376 \pm 0.0031,008.3±10.31,008.3 \pm 10.3
DASH4575.21.464±0.0051.464 \pm 0.0051,043.4±6.61,043.4 \pm 6.6
DASH3574.21.411±0.0021.411 \pm 0.0021,028.4±4.51,028.4 \pm 4.5
VidCom24576.21.433±0.0031.433 \pm 0.0031,021.2±0.91,021.2 \pm 0.9
VidCom23575.41.403±0.0151.403 \pm 0.0151,012.6±14.01,012.6 \pm 14.0

On AVUT, NEOPrune achieves the highest accuracy among the compared compression methods at both retention ratios: 76.5% at 45% retention and 75.8% at 35% retention. Its inference latencies are 1.563 s and 1.475 s, respectively, compared with 1.433–1.464 s and 1.376–1.411 s for the baselines. NEOPrune thus offers higher accuracy at broadly comparable inference latency, with an additional 64–130 ms in these settings. Both configurations also have lower inference latency and LLM prefill time than the full-token reference shown in Table 2.

We will incorporate the additional ShortVid accuracy results, full-token accuracy references for both benchmarks, and AVUT efficiency measurements into the revised manuscript. These additions broaden the accuracy evaluation of Qwen3-Omni-30B-A3B and quantify its inference costs on AVUT.

AuthorsOpenReview ↗

Response to Reviewer vF4B — Part 2 of 4

2. Baseline comparisons and accuracy–efficiency analysis (W2, W4 & Q)

W2 & W4: Comparisons with representative token compression methods are insufficient, and NEOPrune’s accuracy gains at matched token retention come with higher latency, leaving its practical advantage unclear. Additional baselines under comparable budgets, efficiency comparisons at matched accuracy or latency, and an analysis of the additional cost are needed.

Q: “Could the authors provide broader model, baseline, and task evaluations, together with efficiency comparisons at matched accuracy, to demonstrate the general applicability and practical advantages of the proposed method?”

Response:

We agree with the reviewer that the range of baselines should be expanded and that accuracy should be evaluated together with actual inference latency. We therefore additionally evaluated VidCom2, one of the methods suggested by the reviewer, and compared it with the existing baselines at matched token retention and at similar accuracy or latency.

OmniSIFT [1] requires additional training. As our study focuses on training-free token compression, we restrict our baseline comparisons to training-free methods.

The additional experiments used Qwen2.5-Omni-7B and the same evaluation protocols as in the paper. Table 3 compares NEOPrune, OmniZip, DASH, and VidCom2 on four benchmarks at AV token retention ratios of 45% and 35%, alongside a full-token reference.

Table 3. Accuracy on four benchmarks with the additional baseline

MethodAV token retention (%)AVUT (%)WorldSense (%)ShortVid (%)VideoMME (%)
Full10064.546.870.566.0
NEOPrune4564.246.770.366.4
NEOPrune3563.346.569.766.5
OmniZip4563.045.969.966.3
OmniZip3561.045.369.066.1
DASH4563.545.969.966.4
DASH3561.545.669.466.7
VidCom24563.046.268.566.2
VidCom23561.645.765.866.3

Bold values indicate the highest accuracy among the compression methods at each retention ratio.

NEOPrune achieved higher observed accuracy than VidCom2 on all four benchmarks at both retention ratios. At 45% retention, the gains on AVUT, WorldSense, ShortVid, and VideoMME were 1.2, 0.5, 1.8, and 0.2 percentage points, respectively; at 35%, they were 1.7, 0.8, 3.9, and 0.2 percentage points. These results provide additional evidence of NEOPrune’s ability to preserve accuracy beyond the existing comparisons with OmniZip and DASH.

Reviewer_6KvLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NEOPrune is a training-free, two-stage token-pruning method for audio–video OmniLLMs. Its central idea extends reconstruction-based "replaceability" (APET, Ma et al. 2026) from token states to layer-wise orthogonal updates: each attention/FFN update is split into components parallel and orthogonal to the normalized module input, and a token's orthogonal update is reconstructed by ridge regression from other tokens' orthogonal updates—large residuals mean "hard to replace," so the token is kept. The Pre-LLM stage reconstructs within-modality (spatial 3×3 neighbors for video, ±4 temporal neighbors for audio) and fuses the error with encoder attention; the Inner-LLM stage reconstructs each video token's orthogonal update from temporally-aligned audio tokens and fuses with text-query attention for progressive, per-layer pruning. Across four benchmarks (WorldSense, AVUT, VideoMME, ShortVid) and three models (Qwen2.5-Omni-7B/3B, Qwen3-Omni-30B), NEOPrune keeps 35–45% of tokens while staying close to full-token accuracy and generally beating OmniZip/DASH at matched budgets, with up to 3.25× faster prefill and ~33% lower peak memory versus full-token inference.

Critical strengths
  1. Consistent generalization. The method wins across 3 models × 4 benchmarks, including +1.3/+1.6pp over DASH on the larger Qwen3-Omni-30B (Table 5). This breadth is the strongest evidence that the method—if not every component—works, and lowers the risk of a single-configuration artifact.
  2. A genuine mechanism test. A.2 scales down the orthogonal component using the parallel-component norm as reference and shows a larger accuracy drop (Audio 2.9 vs 5.9pp; All 1.7 vs 3.7pp), cleanly supporting the premise that the orthogonal component is what preserves performance—evidence beyond "the end metric went up." A.7's correct-answer log-prob (+0.0128 vs OmniZip −0.0343 / DASH −0.0584) is a reasonable intermediate check.
Critical weaknesses

W1 (most serious). The headline contribution—two-stage, audio-guided cross-modal reconstruction—is never ablated. Table 4 varies only the Pre-LLM reconstruction target and explicitly states "the Inner-LLM configuration and the chunk-budget allocation rule are kept fixed." So the paper cannot say whether Stage-2 gains come from cross-modal reconstruction, from text-query attention, or simply from pruning more video tokens across extra layers. The current evidence supports "the full method works," not "each component earns its keep," which directly undercuts the attribution of contribution #2. The minimal ablation (holding final retention, LLM layers, and budget fixed) is: (a) Stage-1 only; (b) Stage-1 + Stage-2 with text-query attention only; (c) Stage-1 + Stage-2 with audio-guided reconstruction only; (d) the full method.

W2. Main-result gains are small and carry no uncertainty estimate. On Qwen2.5-Omni-7B the improvements over the nearest baselines are AVUT 63.5→64.2 (+0.7), ShortVid 69.9→70.3 (+0.4), WorldSense 45.9→46.7 (+0.8), retention 99.4%→99.9%, with no seeds, std, bootstrap CI, paired significance, or per-sample win/loss. "Generally achieves higher accuracy" is therefore under-supported. I scope this to the Qwen2.5-Omni results (Qwen3-30B is larger, +1.3/+1.6pp) and do not claim the gains are noise—only that sampling fluctuation cannot be ruled out. For a deterministic training-free method, the right ask is a paired bootstrap plus per-sample win/loss and CIs, with decoding/temperature/tie-breaking stated.

W3. At matched retention the method is slower than the real baselines, and this is under-reported. Table 6 @35%: NEOPrune 375ms / 2.73× vs OmniZip 297ms / 3.44× and DASH 296ms / 3.45×; @45%: 505ms / 2.02× vs ~386ms / ~2.64×—roughly +26–30% prefill for +0.8–1.2pp. The paper does not hide this (Table 2 and A.9 note the extra cost), but the narrative leads with the 3.25× speedup over Full rather than an accuracy–latency Pareto against OmniZip/DASH, and the timing is not broken down (preprocessing / scoring-pruning / prefill / decode / memory), so one cannot tell whether the overhead comes from ridge scoring or from Inner-LLM dynamic pruning.

Questions for authors
  1. Can you provide the Stage-1-only / Stage-2-text-query-only / Stage-2-reconstruction-only / full-method ablation, with final retention, layers, and budget held fixed? How much does Stage-2 add on its own? (W1)
  2. Can you report a paired bootstrap, per-sample win/loss, and CIs on the Qwen2.5-Omni results, and state the decoding/tie-breaking protocol? (W2)
  3. Can you show an accuracy–latency Pareto against OmniZip/DASH with timing broken into preprocessing / scoring-pruning / prefill / end-to-end / peak memory, so the source of the overhead is clear? (W3)

If Q1 shows a significant Stage-2 effect and Q3's Pareto is acceptable, I would consider raising my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6KvL - Part 1 of 4

We appreciate the reviewer’s careful assessment and constructive suggestions. We agree that controlled ablations of Stage 2 and its components, uncertainty estimates for the reported accuracy gains, and accuracy–latency comparisons with a detailed runtime breakdown are needed to strengthen the paper. Below, we present additional experiments and analyses addressing these concerns and the corresponding questions.

1. Stage 1 and Stage 2 ablation (W1 & Q1)

W1: The contribution of Stage 2 is unclear because its components are not separately ablated. Controlled comparisons are needed to distinguish the effects of audio-guided reconstruction, text-query attention, and progressive pruning.

Q1: “Can you provide the Stage-1-only / Stage-2-text-query-only / Stage-2-reconstruction-only / full-method ablation, with final retention, layers, and budget held fixed? How much does Stage-2 add on its own? (W1)”

Response:

We agree with the reviewer that the existing ablation does not isolate the contributions of Stage 2 and its individual components. We therefore added the requested comparison of four configurations.

Using Qwen2.5-Omni-7B and the same evaluation settings as in the paper, we evaluated all four configurations on the full AVUT dataset at a final AV token retention ratio of 35%. The final number of retained AV tokens was matched across the four configurations for each input. For Stage-1 only, the entire token removal budget was allocated to Stage 1, with no pruning in Stage 2. The three configurations using Stage 2 retained the same pruning layers and token budgets as in the paper, varying only the scoring signals used in Stage 2.

Table 1. Stage 2 component ablation

ConfigurationAccuracy (%)Difference vs. Stage-1 only (pp)
Stage-1 only61.9—
Stage-1 + Stage-2-text-query-only63.0+1.1
Stage-1 + Stage-2-reconstruction-only62.9+1.0
Full63.3+1.4

Adding Stage 2 in the Full configuration improved accuracy by 1.4 percentage points over Stage-1 only. Full also achieved 0.3 and 0.4 percentage-point higher point estimates than text-query-only and reconstruction-only, respectively. This suggests that the two signals may play complementary roles in token selection.

To assess the difference between Full and Stage-1 only, we performed 100,000 video-level paired bootstrap resamples. In each resample, videos were sampled with replacement while retaining all questions and both configurations’ predictions for each selected video. We calculated the accuracy difference across all sampled questions in each resample. The 95% confidence interval, defined by the 2.5th and 97.5th percentiles of the resulting distribution, was [+0.34, +2.34] pp. This indicates a statistically significant accuracy improvement for the Full configuration with Stage 2 over Stage-1 only in this evaluation setting.

We will add these ablation results and comparison settings to the revised manuscript to clarify the contributions of Stage 2 and its scoring components.

AuthorsOpenReview ↗

Response to Reviewer 6KvL — Part 2 of 4

2. Uncertainty in accuracy gains (W2 & Q2)

W2: The small accuracy gains on Qwen2.5-Omni lack uncertainty estimates. Paired bootstrap confidence intervals and per-sample win/loss statistics are needed, together with the decoding and tie-breaking protocol.

Q2: “Can you report a paired bootstrap, per-sample win/loss, and CIs on the Qwen2.5-Omni results, and state the decoding/tie-breaking protocol? (W2)”

Response:

We agree with the reviewer that small accuracy differences should be accompanied by uncertainty estimates. We therefore compared the stored question-level predictions of NEOPrune and DASH using Qwen2.5-Omni-7B at a nominal AV token retention ratio of 45%.

For each benchmark, we performed 100,000 video-level paired bootstrap resamples. Videos were sampled with replacement, including all questions and both methods’ predictions for each selected video. Accuracy was calculated as the total number of correct answers divided by the total number of questions. In each resample, we computed the accuracy difference, NEOPrune − DASH, and used the 2.5th and 97.5th percentiles of the resulting distribution as the 95% confidence interval.

Table 2. Accuracy comparison and uncertainty

BenchmarkNEOPrune (%)DASH (%)Win / Loss95% CI of difference (pp)
AVUT64.263.566 / 54[−0.6, +2.0]
ShortVid70.369.916 / 12[−0.6, +1.4]
WorldSense46.745.9106 / 80[−0.1, +1.7]
VideoMME66.466.480 / 80[−0.9, +0.9]

Win denotes questions answered correctly only by NEOPrune; Loss denotes questions answered correctly only by DASH. The confidence intervals are for the accuracy difference, NEOPrune − DASH.

NEOPrune achieved higher accuracy than DASH on three benchmarks and matched DASH on VideoMME. However, these numerical advantages were not established as statistically significant differences.

The decoding, answer selection, and tie-breaking rules were as follows.

  • Decoding: Text responses from NEOPrune, OmniZip, and DASH in our direct evaluations were all generated using greedy decoding. No stochastic sampling or beam search was used, and temperature was not applicable.
  • Answer selection: We stripped leading and trailing whitespace from each generated response and converted its first character to uppercase, treating it as the option label to compare against the ground-truth label. Empty responses or responses not beginning with a valid option label were counted as incorrect.
  • Ties in generation scores: If multiple tokens shared the highest next-token score, we selected the first token in vocabulary order.
  • Ties in NEOPrune pruning scores: During rank transformation, tokens with equal raw scores were assigned lower ranks if they appeared earlier in the input. When final combined scores were tied, Stage 1 preferentially retained earlier tokens, whereas Stage 2 preferentially removed earlier tokens.
  • Ties in DASH pruning scores: We followed the official method’s score-based selection procedure without adding a separate tie-breaking priority.

We will add these statistics and evaluation protocols to the revised manuscript. We will also revise the performance comparisons in the main text to report the observed accuracy differences together with their uncertainty more clearly.

AuthorsOpenReview ↗

Response to Reviewer 6KvL — Part 3 of 4

3. Accuracy–latency trade-off and runtime breakdown (W3 & Q3)

W3: NEOPrune is slower than the baselines at matched retention. An accuracy–latency comparison and a detailed runtime breakdown are needed to clarify the trade-off and identify the source of the overhead.

Q3: “Can you show an accuracy–latency Pareto against OmniZip/DASH with timing broken into preprocessing / scoring-pruning / prefill / end-to-end / peak memory, so the source of the overhead is clear? (W3)”

Response:

We agree with the reviewer that accuracy and latency should be compared jointly and that the sources of additional latency should be clearly explained. Following a failure of the workstation used for the original experiments, we remeasured the efficiency of all methods on the same replacement workstation equipped with an NVIDIA RTX PRO 6000 Blackwell GPU, using identical software and inference settings, and will replace the efficiency tables in the paper with these results.

The following table compares accuracy, latency, and memory on WorldSense. Latency denotes model inference time, excluding CPU preprocessing.

Table 3. Accuracy, latency, and memory on WorldSense

MethodAV token retention (%)Accuracy (%)Model inference latency (s)LLM prefill (ms)Max. peak VRAM (GiB)
NEOPrune4546.71.388±0.0041.388 \pm 0.004543.9±1.67543.9 \pm 1.6731.78
NEOPrune3546.51.220±0.004\boldsymbol{1.220 \pm 0.004}401.1±1.05\boldsymbol{401.1 \pm 1.05}29.93
OmniZip4545.91.259±0.003\boldsymbol{1.259 \pm 0.003}427.3±0.38\boldsymbol{427.3 \pm 0.38}31.44
OmniZip3545.31.144±0.0021.144 \pm 0.002328.9±0.76328.9 \pm 0.7629.32
DASH4545.91.285±0.003\boldsymbol{1.285 \pm 0.003}429.7±0.25\boldsymbol{429.7 \pm 0.25}38.79
DASH3545.61.085±0.0041.085 \pm 0.004294.9±0.45294.9 \pm 0.4538.79

At matched retention, NEOPrune incurs higher latency than OmniZip and DASH because part of the token pruning is performed within the LLM. However, NEOPrune preserves accuracy at lower token retention ratios, allowing it to operate at a smaller token budget for comparable or higher accuracy.

For example, NEOPrune at 35% retention achieves 46.5% accuracy with a latency of 1.220 s, whereas OmniZip and DASH at 45% retention achieve 45.9% accuracy with latencies of 1.259 s and 1.285 s, respectively. Thus, with a 10-percentage-point lower token budget, NEOPrune achieves 0.6 percentage points higher accuracy while reducing latency by 3.1% and 5.1%, respectively. Its LLM prefill time and maximum peak VRAM are also lower than those of both baselines at 45% retention.

These results indicate that although NEOPrune has higher computational overhead at matched retention, its stronger accuracy retention enables a more favorable accuracy–latency operating point at a lower token budget.

Comments on the paper

10 comments
AuthorsOpenReview ↗

Response to Reviewer DdFN - Part 1 of 7

We thank the reviewer for the careful and constructive feedback. In the following, we provide a comprehensive response to your review comments. We agree that comparisons at comparable wall-clock latency, full-token reference results for the larger model, uncertainty in accuracy differences, and sensitivity to configuration choices should be strengthened. We present additional experiments and analyses addressing the concerns and questions below.

1. Latency-matched accuracy comparison (W1 & Q1)

W1: “At a fixed token budget the method is slower than the baselines it is compared with, which contradicts the paper's efficiency framing.”

Q1: “Please add a throughput-matched comparison: at the same end-to-end latency as OmniZip at 35% (1.142 s) and at 45% (1.259 s), what token budget and accuracy does NEOPrune achieve?”

We agree with the reviewer that practical deployment efficiency should be assessed by comparing accuracy at similar wall-clock latency, in addition to comparing methods at the same token retention ratio. Although NEOPrune incurs additional computational cost at a fixed token retention ratio, it can preserve accuracy at lower retention ratios. It is therefore important to compare the accuracy achieved at similar or shorter inference latency.

Following the failure of the workstation used in the original experiments, we repeated the efficiency measurements for all methods on the same replacement workstation equipped with an NVIDIA RTX PRO 6000 Blackwell GPU, using identical evaluation settings. The table below reports results on WorldSense using Qwen2.5-Omni-7B. Model inference latency denotes the inference time for a single answer-generation call, excluding CPU preprocessing. Latency and LLM prefill are reported as the mean and standard deviation across three repeated measurements. In the revised manuscript, we will update the existing efficiency tables with the results remeasured under the same conditions and add accuracy comparisons at similar inference latency. We will also clearly distinguish the speedup on the single longest sample from latency averaged across the evaluated samples.

Table Q1. Accuracy, latency, and memory on WorldSense

MethodAV token retention (%)Accuracy (%)Model inference latency (s)LLM prefill (ms)Max. peak VRAM (GiB)
NEOPrune4546.71.388±0.0041.388 \pm 0.004543.9±1.67543.9 \pm 1.6731.78
NEOPrune3546.51.220±0.004\boldsymbol{1.220 \pm 0.004}401.1±1.05\boldsymbol{401.1 \pm 1.05}29.93
NEOPrune3046.21.148±0.003\boldsymbol{1.148 \pm 0.003}340.2±1.90\boldsymbol{340.2 \pm 1.90}29.93
OmniZip4545.91.259±0.0031.259 \pm 0.003427.3±0.38427.3 \pm 0.3831.44
OmniZip3545.31.144±0.0021.144 \pm 0.002328.9±0.76328.9 \pm 0.7629.32
DASH4545.91.285±0.0031.285 \pm 0.003429.7±0.25429.7 \pm 0.2538.79
DASH3545.61.085±0.0041.085 \pm 0.004294.9±0.45294.9 \pm 0.4538.79

The comparisons corresponding to the two OmniZip settings requested by the reviewer are as follows:

At similar latency to OmniZip at 35% retention: NEOPrune at 30% retention achieves 46.2% accuracy with a latency of 1.148 s. Compared with 45.3% accuracy at 1.144 s for OmniZip at 35% retention, NEOPrune achieves 0.9 percentage points higher accuracy at nearly identical latency.

At similar or shorter latency than OmniZip at 45% retention: NEOPrune at 35% retention achieves 46.5% accuracy with a latency of 1.220 s. Compared with 45.9% accuracy at 1.259 s for OmniZip at 45% retention, NEOPrune achieves 0.6 percentage points higher accuracy with shorter latency.

Additionally, NEOPrune at 35% retention achieves 0.6 percentage points higher accuracy with shorter latency than DASH at 45% retention. At this setting, NEOPrune also has lower LLM prefill time and maximum peak VRAM than both OmniZip and DASH at 45% retention. These comparisons therefore show benefits in prefill time and memory usage as well as accuracy.

These results show that, even after accounting for its additional computational cost, NEOPrune can achieve higher accuracy than the baselines at similar or shorter wall-clock latency by using a lower token retention ratio.

AuthorsOpenReview ↗

Response to Reviewer DdFN - Part 2 of 7

2. Sources of additional latency (W1 & Q2)

W1: “At a fixed token budget the method is slower than the baselines it is compared with, which contradicts the paper's efficiency framing.”

Q2: “Where exactly does the extra latency come from — the orthogonal-update scoring, the ridge regression, or the multi-signal fusion at every pruned layer? Is the selection cost amortized across a batch, and can it be precomputed or approximated?”

We agree with the reviewer that the sources of additional latency should be clearly explained. We analyzed runtime and memory usage on WorldSense using Qwen2.5-Omni-7B at an AV token retention ratio of 45%.

Table Q2.1. Runtime and memory usage at 45% AV token retention

MethodCPU preprocessing (ms)Encoder + pre-LLM (ms)LLM prefill (ms)Total prefill (ms)Latency (s)Peak VRAM (avg. / max.) (GiB)
NEOPrune1276±251276 \pm 25809.9±7.6809.9 \pm 7.6543.9±1.7543.9 \pm 1.71353.8±6.21353.8 \pm 6.21.388±0.0041.388 \pm 0.00423.81 / 31.78
OmniZip1281±381281 \pm 38804.0±5.9804.0 \pm 5.9427.3±0.4427.3 \pm 0.41231.3±6.01231.3 \pm 6.01.259±0.0031.259 \pm 0.00323.70 / 31.44
DASH1281±311281 \pm 31817.8±5.6817.8 \pm 5.6429.7±0.2429.7 \pm 0.21247.5±5.51247.5 \pm 5.51.285±0.0031.285 \pm 0.00324.50 / 38.79

Total prefill is the sum of Encoder + pre-LLM and LLM prefill. Encoder + pre-LLM includes Stage 1 scoring/pruning for NEOPrune and the compression step for each baseline, whereas NEOPrune’s Stage 2 scoring/pruning is included in LLM prefill. Latency denotes model inference time for a single answer-generation call. It was measured separately from prefill and excludes CPU preprocessing. The reported timings are averages, and ±\pm indicates the standard deviation across three measurements.

All three methods use the same preprocessing, with similar CPU preprocessing times of approximately 1,280 ms. The maximum difference in Encoder + pre-LLM time was also about 14 ms. In contrast, NEOPrune’s LLM prefill was 116.6 ms longer than OmniZip’s and 114.2 ms longer than DASH’s, indicating that the additional cost occurs primarily during LLM prefill.

This difference is related to the pruning structure. OmniZip and DASH compress tokens to the final budget before entering the LLM. NEOPrune instead enters the LLM with 55.5% of the AV tokens retained and progressively prunes them at layers 15–25 to reach a final retention ratio of 45.0%. It therefore processes more tokens in the early and intermediate LLM layers. Compared with OmniZip, NEOPrune received approximately 23.5% more tokens on average at the first LLM layer, and the sum of token counts processed across all 28 layers was 17.5% higher.

Average peak VRAM ranged from 23.7 to 24.5 GiB across the three methods. NEOPrune’s maximum peak VRAM was 31.78 GiB, close to OmniZip’s 31.44 GiB and lower than DASH’s 38.79 GiB.

AuthorsOpenReview ↗

Response to Reviewer DdFN - Part 3 of 7

2. Sources of additional latency (W1 & Q2) - Continued

To further explain the scoring/pruning cost, we report an operation-level breakdown for NEOPrune, profiled separately on the same inputs. These operations are included in the corresponding intervals in Table Q2.1 and should not be added again to total prefill.

Table Q2.2. NEOPrune scoring/pruning time at 45% AV token retention

OperationStage 1 (ms)Stage 2 (ms)Total (ms)
Orthogonal-update scoring9.91.111.0
Ridge regression24.212.937.0
Multi-signal fusion0.41.01.4
Other: attention signals, token selection/removal, etc.17.814.932.7
Total52.230.082.2

Values are rounded to one decimal place, so the displayed components may differ slightly from their totals.

The total scoring/pruning cost was 82.2 ms, comprising 37.0 ms for ridge regression, 11.0 ms for orthogonal-update scoring, and 1.4 ms for multi-signal fusion, along with the other operations shown above. Stage 2 scoring/pruning took approximately 2.5–2.9 ms at each pruned layer, totaling 30.0 ms across the 11 layers.

These results show that the additional cost at matched retention is related not only to score computation but also to the two-stage pruning structure, which processes more tokens in the layers preceding Inner-LLM pruning. In particular, Stage 2 scoring/pruning takes approximately 30 ms, accounting for part of the approximately 114–117 ms difference in LLM prefill relative to the baselines.

Regarding batching, scores are computed from each input’s hidden states. The ridge regression problems within each input are processed in batched operations at each stage, but the unit of each problem differs between Stage 1 and Stage 2. Stage 1 batches token-level ridge regression problems. Stage 2 formulates chunk-level ridge regression problems that reconstruct video from audio within each chunk and processes them in batched operations. The selection cost is incurred for each input and is not amortized across multiple inputs in a batch. Because Stage 1 is independent of the question, its scores and outputs can be cached per video and reused across multiple questions about that video. Stage 2 depends on the question and must be computed at inference time. We will evaluate the practical latency savings from this cache reuse in future work.

As a possible approximation, scoring cost could be reduced by computing scores at fewer layers. We will evaluate the resulting accuracy–latency trade-off in future work.

We will incorporate this runtime breakdown and clarify the sources of additional latency, the scope of batched computation, and the possible precomputation and approximation strategies in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.