acceptodds
Under review as a conference paper at ICLR 2027

OmniPartition: Query-Guided Region Refinement for Audio–Visual Token Pruning

10 views

Abstract

Omnimodal large language models (OmniLLMs) jointly process audio, video, and text through a shared decoder sequence, but the resulting long multimodal context makes prefill increasingly expensive. Training-free token reduction offers an effective solution, yet existing methods often overlook the spatial and temporal structure of multimodal inputs or rely on fixed modality allocations and auxiliary models. We introduce OmniPartition, a training-free framework for query-guided, structure-aware audio–visual token pruning. OmniPartition first performs pruning after the first decoder block, where contextualized representations provide a reliable query-affinity signal. It then partitions video and audio tokens along their native spatial and temporal coordinates and adaptively refines these regions according to their query-weighted representation error. A shared integer budget is allocated through the marginal gain of each refinement, allowing the method to jointly determine which regions to preserve and how many tokens to retain without predefined modality ratios or temporal windows. The resulting compact sequence can be directly processed by standard FlashAttention. Across six audio–visual benchmarks and multiple OmniLLM backbones, OmniPartition consistently preserves performance under substantial token reduction. On Qwen2.5-Omni-7B, it retains 95.27% of full-token performance at 20% retention while achieving a 3.34 prefill speedup.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM, VLM, OmniLLM, Token Compress

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AChEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OmniPartition, a training-free method for audio–visual token pruning. After the first decoder block, it combines query relevance with spatial and temporal structure, adaptively refining regions to allocate a shared audio–visual token budget. Experiments cover six benchmarks and multiple models. On Qwen2.5-Omni-7B, a 20% layer-average token budget preserves 95.27% of full-token performance on average across the five benchmarks in Table 3, while achieving a 3.34× prefill speedup on WorldSense, including pruning overhead.

Critical strengths
  1. The method is practically useful. It requires neither training nor an auxiliary model, combines query guidance with structural preservation and shared-budget allocation, and remains compatible with standard FlashAttention.

  2. Results are consistent under aggressive compression. At a 20% budget, the method outperforms OmniZip and OmniSelect on all five benchmarks in Table 3. Results across multiple models and measured speedups that include pruning overhead further support its practical value.

Critical weaknesses
  1. Key baseline comparisons are missing. The paper discusses OmniDrop and OmniScope but provides no direct comparisons. The authors’ uniform-sampling baseline is also competitive: at a 45% budget, its average relative performance across the two reported datasets is nearly identical to OmniPartition’s, yet a comprehensive accuracy–latency comparison is missing. The method’s practical advantage therefore remains insufficiently established.

  2. The contributions of the core design choices are not isolated. The “w/o RE-PA” ablation changes both budget allocation and within-region token selection, making it difficult to separate the benefits of shared allocation from structural coverage. Moreover, the region error is computed relative to a weighted mean direction, whereas the retained token is selected by maximum query affinity. Reducing the former does not guarantee a reduction in the representation error of the actual retained token; targeted validation is needed.

Questions for authors
  1. Could you provide accuracy and latency comparisons against OmniDrop, OmniScope, and uniform sampling under consistent budgets and timing conditions, particularly the OmniDrop measurements mentioned in the introduction?

  2. Could you isolate the contributions by replacing only the shared-budget allocation while keeping region partitioning and representative selection unchanged, and then replacing only error-driven partitioning under the same budget?

  3. Does the region error computed around the weighted mean reliably predict the representation error of the actual retained token? Is there empirical evidence supporting this surrogate?

  4. Please specify the timing samples, input lengths, batch size, numerical precision, and synchronization procedure. Why does OmniSelect’s AudioCLIP overhead vary substantially across retention rates?

4 comments
AuthorsOpenReview ↗

Response to Reviewer AChE: Additional Baselines

We sincerely thank the reviewer for the constructive feedback and for recognizing the practical value of OmniPartition, particularly its training-free design, compatibility with FlashAttention, and effectiveness under aggressive audio–visual token compression. We address the concerns below with additional comparisons, controlled ablations, and theoretical clarification.

1. Baseline Comparisons and Efficiency

We thank the reviewer for requesting more comprehensive comparisons. We conduct additional experiments with OmniDrop, OmniScope, and Uniform Stride on Qwen2.5-Omni-7B under matched layer-average token retention rates.

As shown in Tables R1–R2, Uniform Stride remains competitive under moderate compression, particularly on DailyOmni and OmniVideoBench. However, the additional WorldSense results reveal its limitations, even at 45% retention (42.69 vs. 46.15). The performance gap widens under aggressive compression. At 20% retention, OmniPartition achieves higher mean performance retention than all three additional baselines(OmniDrop, Uniform Stride, OmniScope).

Table R1. Accuracy comparisons at 45% and 30% retention.

MethodRetainedOmni VideoBenchDailyOmniWorldSenseMean Ret. (%)
Full Tokens100%34.6064.1646.82100.00%
Uniform Stride45%34.1060.6542.6994.75%
OmniPartition45%34.0060.8246.1597.21%
Uniform Stride30%32.7058.9840.8391.23%
OmniPartition30%34.1059.7344.9695.89%

Table R2. Accuracy comparisons at 20% and 10% retention.

MethodRet.OVBWorldSenseDailyOmniShortVidAVUTMean Ret. (%)
Full Tokens100%34.6046.8264.1670.5064.59100.00%
OmniDrop20%33.7044.7056.4068.8060.2394.32%
Uniform Stride20%33.3039.9755.3166.6058.5490.58%
OmniScope20%33.2043.8855.8266.4060.1192.78%
OmniPartition20%34.6044.5857.1469.5060.3895.27%
OmniDrop10%30.1040.6153.4764.2054.2086.41%
Uniform Stride10%29.9036.3847.7060.3051.1580.64%
OmniScope10%29.7038.5249.0463.6054.9083.95%
OmniPartition10%30.3040.7053.6365.6055.3187.35%

Efficiency Analysis. As shown in Table R3, Uniform Stride achieves the lowest latency. At 20% retention, OmniPartition achieves a 3.34× speedup, compared with 1.43× for OmniDrop and 0.27× for OmniScope, which incur additional overhead from repeated pruning and attention computation, and an auxiliary CLIP model, respectively.

Table R3. Prefill efficiency on Qwen2.5-Omni-7B H100.

MethodRet.Peak Mem. (GiB)LLM PrefillExtra attn.PruneTotalSpeedup
Full Tokens100%22.80397.85––397.851.00×
Uniform Stride20%18.1289.20—14.14103.343.85×
OmniDrop20%18.1991.76157.8529.32278.931.43×
OmniScope20%19.3492.32–1375.541467.860.27×
OmniPartition (Triton)20%18.1777.77—41.40119.173.34×
Uniform Stride10%18.0163.72—14.8378.555.06×
OmniDrop10%18.1766.47159.0633.01258.541.54×
OmniScope10%18.8760.75–1294.641355.390.29×
OmniPartition (Triton)10%18.0949.75—37.8187.564.54×

These results clarify the trade-off between selection overhead and retained information: Uniform Stride favors computational simplicity, while OmniPartition achieves stronger performance preservation under aggressive compression with substantially lower latency than OmniDrop.

AuthorsOpenReview ↗

Response to Reviewer AChE: Controlled Ablations

2. Component-wise Ablations

We thank the reviewer for suggesting more controlled ablations. To distinguish the contributions of shared allocation and region partitioning, we introduce two additional variants while maintaining identical evaluation settings.

Table R4. Ablations on Qwen2.5-Omni-7B with 128 frames.

VariantRet.WorldSenseDailyOmniOVBShortVidMean Ret. (%)
Full Tokens100%46.8264.1634.6070.50100.00%
w/o Shared Allocation30%44.1959.1233.6069.2095.45%
w/o Region Partitioning30%43.2558.1533.8068.8094.57%
OmniPartition30%44.9659.7334.1069.7096.64%
w/o Shared Allocation20%43.5156.8133.7068.4093.97%
w/o Region Partitioning20%41.3355.4733.3064.8090.72%
OmniPartition20%44.5857.1434.6069.5095.71%

w/o Shared Allocation. We replace the global allocation procedure with predefined frame- and segment-specific token quotas. Native-coordinate partitioning, representation-error-based split gains, and affinity-based representative selection remain unchanged. Each frame or audio segment maintains an independent refinement heap until its assigned quota is reached. This isolates the contribution of adaptive shared allocation while preserving the remaining components.

w/o Region Partitioning. We remove native-coordinate recursive partitioning while retaining query-affinity-based representative selection and shared allocation. Each video frame or audio segment forms an initial unit, where representatives are added in descending query-affinity order. Tokens are assigned to their nearest representatives in feature space, and the resulting groups are evaluated using the same representation error in Eq. (8). A shared heap allocates additional representatives according to the reduction in aggregate representation error. This variant evaluates the contribution of native-coordinate partitioning under adaptive shared allocation.

As shown in Table R4, removing shared allocation reduces mean performance retention from 96.64% to 95.45%, while removing region partitioning reduces it to 94.57%. The larger performance degradation at 20% retention further demonstrates the importance of shared allocation and native-coordinate partitioning under aggressive compression.

AuthorsOpenReview ↗

Response to Reviewer AChE: Surrogate Validation

3. Surrogate Representation Error

We sincerely thank you for raising this important question. We first clarify the relationship between the surrogate representation error in Eq. (8) and the actual error associated with the retained token in Eq. (7), followed by empirical evidence supporting the surrogate.

Theoretical Relationship. For a region CC, let xi=hi/∣hi∣2x_i=h_i/\\|h_i\\|_2, sC=∑i∈Cwixis_C=\sum_{i\in C}w_ix_i, WC=∑i∈CwiW_C=\sum_{i\in C}w_i, and μC=sC/∣sC∣2\mu_C=s_C/\\|s_C\\|_2. The representation error associated with the retained token iC∗i_C^* is

Eret(C)=∑i∈Cwi(1−xi⊤xiC∗)=WC−sC⊤xiC∗.\begin{aligned} E_{\mathrm{ret}}(C) &=\sum_{i\in C}w_i(1-x_i^\top x_{i_C^*})\\\\ &=W_C-s_C^\top x_{i_C^*}. \end{aligned}

In contrast, the surrogate representation error in Eq. $ 8$ is

D(C)=WC−∣sC∣2.D(C)=W_C-\\|s_C\\|_2.

Since ∣xiC∗∣2=1\\|x_{i_C^*}\\|_2=1, the Cauchy–Schwarz inequality gives

sC⊤xiC∗≤∣sC∣2.s_C^\top x_{i_C^*}\leq \\|s_C\\|_2.

Consequently,

D(C)≤Eret(C),D(C)\leq E_{\mathrm{ret}}(C),

with the gap given by

Eret(C)−D(C)=∣sC∣2−sC⊤xiC∗=∣sC∣2(1−μC⊤xiC∗)≥0.\begin{aligned} E_{\mathrm{ret}}(C)-D(C) &=\\|s_C\\|_2-s_C^\top x_{i_C^*}\\\\ &=\\|s_C\\|_2(1-\mu_C^\top x_{i_C^*})\geq 0. \end{aligned}

Thus, D(C)D(C) represents the minimum representation error over all unit-norm directions, providing a continuous relaxation of the discrete retained-token error. When token representations within a region are sufficiently similar, the retained token is close to the weighted mean direction, narrowing the gap between the surrogate and actual errors.

Empirical Evidence. We evaluate how reliably the region error computed around the weighted mean, D(C)D(C), predicts the representation error of the actual retained token, Eret(C)E_{\mathrm{ret}}(C), across 324,387 final regions from 100 randomly sampled WorldSense examples.

Table R5. Pearson and Spearman correlations between the region error computed around the weighted mean D(C)D(C) and the representation error of the actual retained token Eret(C)E_{\mathrm{ret}}(C).

RepresentativePearson ↑Spearman ↑Regions
Highest Affinity0.9810.992324,387

As shown in Table R5, the high Pearson (0.981) and Spearman (0.992) correlations provide strong empirical evidence that the region error computed around the weighted mean reliably predicts the representation error of the actual retained token, supporting its use in guiding region refinement.

Preservation of Task-Relevant Evidence. We additionally compare OmniPartition's highest-affinity representative with the original token that minimizes weighted within-region distortion:

iCopt=arg⁡min⁡j∈C∑i∈Cwi(1−xi⊤xj).i_C^{\mathrm{opt}}=\arg\min_{j\in C}\sum_{i\in C}w_i(1-x_i^\top x_j).

Table R6. Comparison of highest-affinity and minimum-distortion representative selection on Qwen2.5-Omni-7B at 30% retention.

VariantWorldSenseDailyOmniOVBShortVidMean Ret. (%)
OmniPartition44.9659.7334.1069.7096.64%
Min-Distortion Representative44.2957.7333.2069.1094.64%

As shown in Table R6, OmniPartition consistently outperforms minimum-distortion representative selection across all four benchmarks, improving mean performance retention by 2.00 percentage points. These results indicate that minimizing representation error alone does not necessarily preserve task-relevant information as effectively as selecting the highest-affinity token.

AuthorsOpenReview ↗

Response to Reviewer AChE: Efficiency Measurements

4. Efficiency Measurement Protocol and AudioCLIP Overhead

We thank the reviewer for requesting further details on the efficiency measurements.

Measurement Protocol. We evaluate the same 100 WorldSense samples after one warm-up run, using a batch size of 1 and up to 128 video frames per sample. The original audiovisual durations range from 37 to 580 s (mean 157.1 s; median 101 s). Frame sampling yields 74–128 frames per example (mean 119.2).

Qwen2.5-Omni runs in BF16, while OmniSelect's AudioCLIP uses FP32, following its default configuration. Latency is measured using CUDA events, with synchronization at each measurement boundary. GPU peak memory statistics are reset after preprocessing for each sample. We report mean latency and the maximum peak memory across all samples. CPU preprocessing and autoregressive decoding are excluded.

AudioCLIP Overhead. We apologize for a data entry error affecting OmniSelect's 20% retention results in the original Table 5. We re-measured this setting under the same experimental conditions and obtained consistent results across repeated runs. The updated latency breakdown is reported in Table R7, with AudioCLIP latency corrected from 2242.06 ms to 1353.96 ms.

The corrected measurement resolves the anomalous value in Table 5. OmniSelect's substantial AudioCLIP overhead remains the main contributor to its total prefill latency, while OmniPartition avoids the auxiliary model and achieves substantially lower latency.

Table R7. Corrected prefill efficiency at 20% retention on Qwen2.5-Omni-7B (H100). Latencies are in milliseconds. AudioCLIP latency is shown in parentheses and included in Total.

MethodRet.Peak Mem. (GiB)LLM PrefillExtra attn.(ms)Prune (ms)Total (ms)Speedup
OmniSelect20%18.8073.2012.3415.06 (+1353.96)1454.570.27×
OmniPartition (Triton)20%18.1777.77–41.40119.173.34×
Reviewer_NpWuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniPartition is a training-free method for audio-visual token pruning in OmniLLMs. It computes query affinity after the first decoder block, partitions visual and audio tokens along their native spatial and temporal coordinates, and allocates a shared token budget through iterative region refinement. Refinement is guided by a query-weighted feature-distortion objective, while each final region retains its highest-affinity original token. The paper evaluates the method across six benchmarks and multiple model families, with component ablations, implementation details, and measured prefill efficiency including pruning overhead.

Critical strengths

1.The method combines query relevance, structural coverage, and budget allocation in a coherent framework. Recursive partitioning preserves spatial locality and temporal continuity, while a shared refinement procedure allocates tokens across audio and video without prescribing a fixed modality ratio. Table 4 provides useful component evidence: removing query-affinity weighting or adaptive refinement reduces performance, and random representative selection performs substantially worse.

2.The empirical evaluation includes multiple benchmarks, model families, and aggressive compression settings. At 20% retention, Table 3 reports 95.27% mean performance retention across its five benchmarks on Qwen2.5-Omni-7B. Table 5 additionally reports a 3.34× prefill speedup on WorldSense, including pruning overhead. The cross-model results and implementation-level measurements provide evidence that the approach offers practical benefits beyond token-count reduction alone.

Critical weaknesses

1.The comparison set does not fully establish the method’s position relative to closely related audio-visual compression approaches. OmniFocus and DASH are relevant omitted comparisons. Moreover, Appendix I shows that uniform structural sampling is already a strong baseline, but its detailed evaluation covers only two datasets and it is absent from the main efficiency comparison. This leaves uncertainty about the additional quality–latency benefit of adaptive refinement over a simple coverage-preserving strategy. Comparisons should account for pruning position through the layer-average budget and report measured latency.

2.The refinement objective and the actual retained representation are different. Equation 8 measures distortion relative to the normalized weighted mean of a region, whereas Equation 7 retains the original token with the highest query affinity. Consequently, the non-increasing distortion property in Equation 9 applies to the mean-based proxy, rather than directly to the features retained for subsequent decoder layers. The random-selection ablation supports query-aware selection, but does not establish how closely the proxy tracks the error of the selected representative. This limits the explanatory strength of the representation-error formulation, although it does not invalidate the empirical results.

Questions for authors

1.How does OmniPartition compare with uniform structural sampling across the remaining benchmarks and in measured prefill latency? A comparison with a closely related method such as OmniFocus or DASH, using a matched layer-average budget, would also help establish the additional benefit of adaptive refinement. 2.How well does the mean-based distortion in Equation 8 correlate with the distortion induced by retaining the highest-affinity token in Equation 7? Is there evidence comparing the current representative with an original token selected to minimize weighted within-region distortion? 3.Could the authors distinguish encoder-attention extraction requirements from decoder FlashAttention compatibility in Section 4.1? Requiring encoder attention or an auxiliary AudioCLIP model does not by itself preclude FlashAttention in the decoder. Please clarify the attention implementations used for each component in the timing comparisons.

3 comments
AuthorsOpenReview ↗

Response to Reviewer NpWu: Additional Baseline Comparisons

We thank you for the constructive feedback and the recognition of OmniPartition's contributions. We address the concerns regarding baseline comparisons, the relationship between the surrogate objective and retained representations, and FlashAttention compatibility below.

1. Additional Baseline Comparisons and Efficiency

We sincerely thank you for suggesting broader comparisons. We conduct additional experiments with Uniform Stride and DASH on Qwen2.5-Omni-7B across five benchmarks under matched layer-average retention.

Comparison with Uniform Stride and DASH. Uniform Stride remains competitive at moderate retention ratios but degrades substantially on WorldSense and under aggressive compression. In contrast, OmniPartition achieves higher mean performance retention at both retention ratios, with larger gains over Uniform Stride under aggressive compression.

Table R1. Comparison with Uniform Stride and DASH at 20% and 10% retention.

MethodRet.OVBWorldSenseDailyOmniShortVidAVUTMean Ret. (%)
Full Tokens100%34.6046.8264.1670.564.59100.00%
DASH20%32.8042.9753.3067.555.8890.38%
Uniform Stride20%33.3039.9755.3166.658.5490.58%
OmniPartition20%34.6044.5857.1469.560.3895.27%
DASH10%30.9039.7545.5364.252.2583.43%
Uniform Stride10%29.9036.3847.7060.351.1580.64%
OmniPartition10%30.3040.7053.6365.655.3187.35%

Comparison with OmniFocus. We further evaluate OmniFocus on four benchmarks at 20% and 10% retention. OmniPartition consistently outperforms OmniFocus across all evaluated benchmarks, achieving higher mean performance retention at both compression levels.

Table R2. Comparison with OmniFocus at 20% and 10% retention.

MethodRet.OVBDailyOmniShortVidAVUTBenchMean Ret. (%)
OmniFocus20%30.8051.8066.5055.2587.40%
OmniPartition20%34.6057.1469.5060.3895.28%
OmniFocus10%29.6046.9561.8048.2180.26%
OmniPartition10%30.3053.6365.6055.3187.46%

Efficiency Comparison. We further evaluate end-to-end decoder prefill latency on the same WorldSense samples using an NVIDIA H100 GPU, including pruning overhead. Uniform Stride incurs minimal selection overhead and achieves higher speedups, whereas DASH exhibits substantial pruning overhead that offsets much of its computational savings. OmniPartition achieves a more favorable balance between task performance and inference efficiency, retaining substantially more task performance than Uniform Stride while achieving 3.34× and 4.54× speedups at 20% and 10% retention, respectively.

These findings clarify the quality–latency trade-off: Uniform Stride offers lower selection overhead and faster inference, whereas OmniPartition consistently preserves performance, particularly at low token-retention ratios.

Table R3. Prefill efficiency comparison on Qwen2.5-Omni-7B.

MethodRet.Peak Mem. (GiB)LLM Prefill (ms)Extra attn. (ms)Prune (ms)Total (ms)Speedup
Uniform Stride20%18.1289.20–14.14103.343.85×
DASH20%33.91203.94–161.84365.781.09×
OmniFocus20%18.5274.82–58.26133.082.99×
OmniPartition (Triton)20%18.1777.77–41.40119.173.34×
Uniform Stride10%18.0163.72–14.8378.555.06×
DASH10%33.91189.38–147.87337.251.18×
OmniFocus10%18.2350.84–75.74126.583.14×
OmniPartition (Triton)10%18.0949.75–37.8187.564.54×
AuthorsOpenReview ↗

Response to Reviewer NpWu: Distortion Analysis

2. Relationship Between Surrogate and Retained-Token Distortion

We sincerely thank you for this constructive suggestion. We provide empirical evidence and theoretical analysis to clarify the relationship between the mean-based distortion in Eq. (8) and the distortion induced by retaining the highest-affinity token in Eq. (7).

Correlation Analysis. We measure the Pearson and Spearman correlations between the mean-based distortion D(C)D(C) and the actual retained-token distortion Eret(C)E_{\mathrm{ret}}(C) across 324,387 final regions from 100 randomly sampled WorldSense examples.

Table R4. Correlation between mean-based and retained-token distortion.

RepresentativePearson ↑Spearman ↑Regions
Highest Affinity0.9810.992324,387

As shown in Table R4, the high Pearson (0.981) and Spearman (0.992) correlations provide strong empirical evidence that the region error computed around the weighted mean reliably predicts the representation error of the actual retained token, supporting its use in guiding region refinement.

Representative Selection. We further compare the current representative (the highest-affinity token in Eq. (7)) with an original token selected to minimize weighted within-region distortion, while keeping region partitioning and token allocation unchanged.

Table R5. Comparison of highest-affinity and minimum-distortion representatives.

VariantWorldSenseDailyOmniOVBShortVidMean Ret. (%)
OmniPartition44.9659.7334.1069.7096.64%
Min-Distortion Representative44.2957.7333.2069.1094.64%

The minimum-distortion representative is selected as

iCopt=arg⁡min⁡j∈C∑i∈Cwi(1−xi⊤xj).i_C^{\mathrm{opt}}=\arg\min_{j\in C}\sum_{i\in C}w_i(1-x_i^\top x_j).

The current representative outperforms the minimum-distortion alternative across all four benchmarks, improving mean performance retention by 2.00 percentage points. This suggests that minimizing weighted within-region distortion does not necessarily preserve query-relevant information as effectively as highest-affinity selection.

Theoretical Relationship. We further clarify that Eq. (8) provides a lower bound on the actual retained-token distortion. For a region CC, let xi=hi/∣hi∣2x_i=h_i/\\|h_i\\|_2, sC=∑i∈Cwixis_C=\sum_{i\in C}w_ix_i, WC=∑i∈CwiW_C=\sum_{i\in C}w_i, and μC=sC/∣sC∣2\mu_C=s_C/\\|s_C\\|_2. The representation error associated with the retained token iC∗i_C^* is

Eret(C)=∑i∈Cwi(1−xi⊤xiC∗)=WC−sC⊤xiC∗.\begin{aligned} E_{\mathrm{ret}}(C) &=\sum_{i\in C}w_i(1-x_i^\top x_{i_C^*})\\\\ &=W_C-s_C^\top x_{i_C^*}. \end{aligned}

In contrast, Eq. (8) defines

D(C)=∑i∈Cwi(1−xi⊤μC)=WC−∣sC∣2.D(C)=\sum_{i\in C}w_i\bigl(1-x_i^\top\mu_C\bigr) =W_C-\\|s_C\\|_2.

Since ∣xiC∗∣2=1\\|x_{i_C^*}\\|_2=1, the Cauchy–Schwarz inequality gives

sC⊤xiC∗≤∣sC∣2.s_C^\top x_{i_C^*}\leq \\|s_C\\|_2.

Consequently,

Eret(C)−D(C)=∣sC∣2−sC⊤xiC∗=∣sC∣2(1−μC⊤xiC∗)≥0.\begin{aligned} E_{\mathrm{ret}}(C)-D(C) &=\\|s_C\\|_2-s_C^\top x_{i_C^*}\\\\ &=\\|s_C\\|_2(1-\mu_C^\top x_{i_C^*})\geq 0. \end{aligned}

Thus, D(C)D(C) provides a lower bound on the retained-token distortion, with the gap determined by the alignment between the retained token and the weighted mean direction. This gap decreases as regional token representations become more consistent. OmniPartition uses the mean-based distortion to guide region refinement while retaining the highest-affinity token to preserve query-relevant information.

AuthorsOpenReview ↗

Response to Reviewer NpWu: FlashAttention Clarification

3. FlashAttention Compatibility and Attention Extraction

We thank the reviewer for pointing out this distinction. Both the audio encoder and the language model decoder use FlashAttention-2 in our Qwen2.5-Omni implementation.

OmniZip and OmniSelect. Both methods require explicit attention scores from the audio encoder. Although the encoder uses FlashAttention-2 for its forward computation, FlashAttention-2 does not return attention matrices. Therefore, the required scores are separately recomputed from projected queries and keys using PyTorch operations outside the FlashAttention kernels. This introduces additional latency and peak memory overhead, particularly for long sequences such as VideoMME with 512 sampled frames. The additional latency is reported under “Extra attn.” in Table 5.

OmniSelect and AudioCLIP. OmniSelect additionally employs an auxiliary AudioCLIP model for cross-modal relevance estimation. AudioCLIP primarily uses convolutional operations and PyTorch nn.MultiheadAttention in its text encoder, without explicitly enabling FlashAttention-2. Its additional inference overhead is included under “Prune” in Table 5.

We will revise Section 4.1 to clarify that both the audio encoder and LLM decoder use FlashAttention. However, OmniZip and OmniSelect additionally compute explicit attention scores using PyTorch, incurring substantially higher latency and peak memory usage for long sequences and diminishing the efficiency benefits of FlashAttention.

Reviewer_f3EPOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OmniPartition, a training-free method for audio–visual token pruning in omni-modal LLMs. After the first decoder block, it selects query-text anchors and uses their similarity to audio–visual tokens to compute relevance weights. Video frames and audio segments are organized into spatial and temporal regions, which are recursively divided according to a query-weighted representation-error criterion under a shared token budget. Each final region retains its highest-affinity original token. The paper evaluates task performance, component ablations, and prefill efficiency across multiple models and benchmarks.

Critical strengths
  1. The framework addresses a practical compression problem with a coherent implementation. It combines query relevance, spatial/temporal coverage, and shared audio–visual budget allocation without additional training. The implementation reports pruning overhead explicitly and provides a fused kernel to reduce the cost of region evaluation.

  2. The empirical evaluation supports the usefulness of the overall design. Experiments cover Qwen2.5-Omni-3B/7B, VITA-1.5, and Baichuan-Omni, with relevant baselines including OmniZip. The component ablations provide evidence that query weighting and representative selection matter. On WorldSense with Qwen2.5-Omni-7B, the reported 20% retention setting achieves a 3.34× prefill speedup including pruning overhead; this measurement excludes subsequent decoding.

Critical weaknesses
  1. The connection between the optimization criterion and the retained information needs stronger justification. The method combines similarity-based weighting with greedy region refinement, but the analysis does not sufficiently explain when this combination preserves the evidence needed for the query. In particular, Eq. (8) evaluates a region using its weighted mean feature direction, whereas Eq. (7) retains the original token with the highest query affinity. These representatives need not coincide. The non-increasing surrogate distortion established in Eq. (9) therefore does not, by itself, establish improved representation by the actual retained token. The limiting feature configurations in Eqs. (10)–(11) provide useful intuition, but leave the alignment between the surrogate, token selection, and task-relevant evidence insufficiently characterized. An analysis or targeted diagnostic of this alignment would provide a more convincing explanation of why the method works and where it may fail.

  2. The attribution of the gains to the proposed mechanisms remains incomplete. Table 4 contains useful ablations, but the w/o RE-PA variant simultaneously replaces adaptive regional refinement with preset frame/segment budgets and affinity-based selection within those units. It therefore changes both spatial partitioning and budget allocation. Controlled comparisons that hold one factor fixed while varying the other would better establish their individual contributions. The paper already compares against OmniZip [1] in accuracy, latency, and qualitative examples. Building on those results, a more systematic comparison of the allocation and selection mechanisms at matched compression budgets would help explain which cases benefit specifically from query-guided region refinement. This would strengthen the methodological contribution beyond demonstrating the effectiveness of the combined pipeline.

The comparison should also include OmniSIFT [3], which combines spatio-temporal video pruning with vision-guided audio selection. Section 4.1 explicitly excludes it because its selectors require training. While this distinction is relevant, a comparison on the same backbone and inputs at matched compression budgets would help position the proposed approach against a directly related learned alternative. Training cost and added parameters should be reported separately, alongside task quality and inference latency including compression overhead. A methodological comparison should explain the trade-offs between learned modality-asymmetric selection and query-guided refinement under a shared budget.

Overall assessment. I recommend clear rejection. The results demonstrate practical compression benefits, but I find the explanatory depth and attribution of the central methodological contribution insufficient for acceptance. In particular, the current analysis does not adequately connect the region-refinement objective to the actual retained representatives, and the experiments do not fully separate the proposed mechanisms. The additional JavisGPT evaluation is a suggestion for broader validation; the principal reasons for my recommendation are the methodological concerns above.

References

[1] Keda Tao et al. OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models. CVPR 2026. Paper; official implementation.

[2] Kai Liu et al. JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation. NeurIPS 2025. Paper; official implementation.

[3] Yue Ding et al. OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models. arXiv:2602.04804, 2026. Paper.

Questions for authors
  1. Under what conditions does reducing the centroid-based distortion in Eq. (8) improve representation by the highest-affinity token selected in Eq. (7)? Could the authors provide an analysis or empirical diagnostic relating these two objectives to preservation of task-relevant evidence?

  2. Can the effects of regional partitioning and shared-budget allocation be separated, for example by holding one fixed while varying the other? How do the resulting mechanisms explain the observed differences from OmniZip beyond the existing aggregate scores and illustrative cases?

  3. Could the authors include OmniSIFT at matched compression budgets on the same backbone and inputs, reporting task performance, added parameters, training cost, and inference latency including compression overhead? Given the training requirement acknowledged in Section 4.1, what do the results reveal about the trade-offs between its learned modality-asymmetric selection and the proposed shared-budget region refinement?

  4. How readily do the first-layer affinity calculation and native-coordinate partitioning transfer to a model with a different audio–visual fusion design, such as JavisGPT [2]? What components can be used unchanged, and what adaptations would be needed?

Suggestion for broader evaluation. JavisGPT supports audio–video comprehension and generation and would be a useful additional setting for testing transfer. A matched-budget evaluation on its audio–visual understanding pathway could complement the existing multi-model results. Exploring generation would require specifying whether and how prunable audio–visual conditioning representations reach the generator, followed by evaluation of generation quality, audio–video synchrony, and total latency. This is a suggested extension beyond the current understanding benchmarks; a comparable architecture could serve the same purpose.

6 comments
AuthorsOpenReview ↗

Response to Reviewer f3EP: Surrogate Error Analysis

We sincerely thank you for recognizing the practical relevance of OmniPartition, its training-free design, and its empirical efficiency. We appreciate the concerns regarding the relationship between the surrogate objective and retained-token representations, the individual contributions of partitioning and allocation, and the comparison with OmniSIFT. We provide additional theoretical clarification and controlled experiments below.

1. Surrogate Representation Error

We appreciate the concern regarding the relationship between the centroid-based distortion in Eq. (8) and the representation by the highest-affinity token selected in Eq. (7). We provide theoretical analysis and empirical diagnostics to clarify their relationship and the preservation of task-relevant evidence.

Theoretical Relationship. For a region CC, let xi=hi/∣hi∣2x_i=h_i/\\|h_i\\|_2, sC=∑i∈Cwixis_C=\sum_{i\in C}w_ix_i, WC=∑i∈CwiW_C=\sum_{i\in C}w_i, and μC=sC/∣sC∣2\mu_C=s_C/\\|s_C\\|_2. The actual retained-token error is

Eret(C)=∑i∈Cwi(1−xi⊤xiC∗)=WC−sC⊤xiC∗.\begin{aligned} E_{\mathrm{ret}}(C) &=\sum_{i\in C}w_i(1-x_i^\top x_{i_C^*})\\\\ &=W_C-s_C^\top x_{i_C^*}. \end{aligned}

whereas the surrogate error is

D(C)=∑i∈Cwi(1−xi⊤μC)=WC−∣sC∣2.D(C)=\sum_{i\in C}w_i\bigl(1-x_i^\top\mu_C\bigr) =W_C-\\|s_C\\|_2.

Their difference satisfies

Eret(C)−D(C)=∣sC∣2−sC⊤xiC∗=∣sC∣2(1−μC⊤xiC∗)≥0.\begin{aligned} E_{\mathrm{ret}}(C)-D(C) &=\\|s_C\\|_2-s_C^\top x_{i_C^*}\\\\ &=\\|s_C\\|_2(1-\mu_C^\top x_{i_C^*})\geq 0. \end{aligned}

Thus, D(C)D(C) is a lower bound on the retained-token representation error, with their difference determined by the alignment between the selected token and the weighted mean direction. Specifically, when μC⊤xiC∗≥1−ϵ\mu_C^\top x_{i_C^*}\geq 1-\epsilon, we have Eret(C)−D(C)≤ϵ∣sC∣2E_{\mathrm{ret}}(C)-D(C)\leq\epsilon|s_C|_2. Region refinement reduces within-region feature variation, which can improve this alignment. However, reducing the surrogate error does not guarantee a monotonic decrease in the retained-token error, as the latter also depends on representative alignment. We therefore empirically examine their relationship.

Empirical Alignment. We measure the correlation between the surrogate error D(C)D(C) and the actual retained-token error Eret(C)E_{\mathrm{ret}}(C) across final regions from 100 randomly sampled WorldSense examples. The high Pearson (0.981) and Spearman (0.992) correlations demonstrate strong empirical agreement between the centroid-based distortion and retained-token representation error across regions.

Table R1. Pearson and Spearman correlations between the centroid-based distortion D(C)D(C) in Eq. (8) and the representation error Eret(C)E_{\mathrm{ret}}(C) induced by the highest-affinity token selected in Eq. (7), evaluated over 324,387 final regions from 100 WorldSense samples.

RepresentativePearson ↑Spearman ↑Regions
Highest Affinity0.9810.992324,387

Preservation of Task-Relevant Evidence. We further compare highest-affinity selection with the original token minimizing weighted within-region distortion:

iCopt=arg⁡min⁡j∈C∑i∈Cwi(1−xi⊤xj).i_C^{\mathrm{opt}}=\arg\min_{j\in C}\sum_{i\in C}w_i(1-x_i^\top x_j).

Table R2. Comparison of highest-affinity and minimum-distortion representative selection on Qwen2.5-Omni-7B at 30% retention.

VariantWorldSenseDailyOmniOVBShortVidMean Ret. (%)
OmniPartition44.9659.7334.1069.7096.64%
Min-Distortion Representative44.2957.7333.2069.1094.64%

As shown in Table R2, OmniPartition, which selects the highest-affinity token within each region, consistently outperforms the minimum-distortion alternative across all four benchmarks, achieving a 2.00-percentage-point improvement in mean performance retention. This suggests that minimizing representation error alone does not necessarily preserve task-relevant evidence as effectively as query-guided representative selection.

AuthorsOpenReview ↗

Response to Reviewer f3EP: Controlled Ablations

2. Controlled Ablations of Region Partitioning and Shared Allocation

We sincerely thank you for the suggestion. To isolate the contributions of region partitioning and shared allocation, we introduce two controlled ablations under identical evaluation settings.

Table R3. Controlled ablations on Qwen2.5-Omni-7B with 128 frames and 30% retention.

VariantRet.WorldSenseDailyOmniOVBShortVidMean Ret. (%)
Full Tokens100%46.8264.1634.6070.50100.00%
w/o Shared Allocation30%44.1959.1233.6069.2095.45%
w/o Region Partitioning30%43.2558.1533.8068.8094.57%
OmniPartition (Triton)30%44.9659.7334.1069.7096.64%
w/o Shared Allocation20%43.5156.8133.7068.4093.97%
w/o Region Partitioning20%41.3355.4733.3064.8090.72%
OmniPartition (Triton)20%44.5857.1434.6069.5095.71%

w/o Shared Allocation. We replace global allocation with predefined token quotas for individual video frames and audio segments, while retaining native-coordinate partitioning, region refinement based on representation error, and highest-affinity token selection. Each unit is refined independently until its assigned quota is reached. This variant isolates the contribution of shared allocation.

w/o Region Partitioning. We remove native-coordinate recursive partitioning while retaining query-affinity-based representative selection and shared allocation. Each video frame or audio segment forms an initial unit, where representatives are added in descending query-affinity order. Tokens are assigned to their nearest representatives in feature space, and the resulting groups are evaluated using the same representation error in Eq. (8). A shared heap allocates additional representatives according to the reduction in aggregate representation error. This variant evaluates the contribution of native-coordinate partitioning under adaptive shared allocation.

As shown in Table R3, removing either shared allocation or region partitioning consistently reduces mean performance retention, with more pronounced degradation at lower retention ratios. These results support the contributions of region partitioning and shared allocation. The larger degradation without region partitioning under aggressive compression highlights the importance of spatial and temporal coverage, while the shared-allocation ablation demonstrates the benefit of adapting token retention across audio–visual units.

Comparison with OmniZip. Unlike OmniZip's audio-guided selection, OmniPartition refines spatially and temporally localized regions according to query-weighted representation error. The controlled ablations show that both explicit regional coverage and adaptive allocation contribute to the observed improvements, particularly at lower retention ratios.

AuthorsOpenReview ↗

Response to Reviewer f3EP: OmniSIFT Comparison

3. Comparison with OmniSIFT

We appreciate the reviewer's suggestion to include OmniSIFT as a learned compression baseline. Unlike OmniPartition's training-free, query-guided region refinement, OmniSIFT employs trained modality-asymmetric selectors and a fine-tuned Qwen2.5-Omni-7B checkpoint. To ensure a fair comparison, we apply OmniPartition to the same publicly released OmniSIFT-7B checkpoint and evaluate both methods at matched token retention ratios.

As shown in Table R4, OmniPartition achieves higher mean performance retention than OmniSIFT at both 20% and 10% retention.

Table R4. Comparison on the OmniSIFT-7B backbone at 20% and 10% retention.

MethodRet.OVBWorldSenseDailyOmniShortVidBenchAVUTBenchMean Ret. (%)
Full Tokens100%35.8050.8573.7771.8070.47100.00%
OmniSIFT20%36.8048.1769.1769.8064.7196.06%
OmniPartition20%35.8048.5869.5971.1065.9296.49%
OmniSIFT10%35.7044.8860.9167.6059.8689.93%
OmniPartition10%36.2045.0561.4968.2060.6190.81%

Inference Efficiency. We further compare inference efficiency on Qwen2.5-Omni-7B using the same retention ratios. Total latency includes decoder prefill and compression overhead, excluding subsequent autoregressive decoding.

As shown in Table R5, OmniPartition achieves higher prefill speedups at 20% and 10% retention, reaching 3.22× and 4.99× compared with OmniSIFT's 2.66× and 3.49×. OmniPartition also exhibits lower measured peak GPU memory usage across all retention ratios.

**Table R5. Prefill efficiency on OmniSIFT-7B. **

MethodRet.Peak Mem. (GiB)LLM Prefill (ms)Prune (ms)Total (ms)Speedup
Full Tokens100%22.81399.29—399.291.00×
OmniSIFT20%21.9493.1156.74149.842.66×
OmniPartition (Triton)20%18.1778.5345.40123.933.22×
OmniSIFT10%21.9463.7950.74114.533.49×
OmniPartition (Triton)10%18.0947.0032.9779.974.99×

Training Requirements and Methodological Differences. Token pruning aims to reduce the computational and memory costs of OmniLLM inference, particularly in resource-constrained settings. However, OmniSIFT requires training model-specific modality-asymmetric selectors with approximately 4.85M additional parameters, introducing training costs and limiting its applicability across different backbones. In contrast, OmniPartition requires neither additional training nor learnable parameters, while achieving comparable or higher task accuracy with lower prefill latency and memory consumption. These results highlight the practical advantages of training-free, query-guided region refinement under a shared allocation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Rebuttal Updates

Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

We sincerely thank all reviewers for their careful evaluations, constructive feedback, and thoughtful suggestions. During the rebuttal period, we conducted extensive additional experiments and theoretical analyses to address the reviewers' concerns and further strengthen the empirical and methodological foundations of OmniPartition. We believe our current carefuly revised manuscript can address the revewers concerns.

We appreciate the reviewers' recognition of OmniPartition's training-free design, query-guided region refinement, spatial and temporal coverage, and adaptive allocation of a shared audio–visual token budget (Reviewers f3EP, NpWu, and AChE). The reviewers also acknowledged its practical efficiency, comprehensive evaluation across multiple model families, and effectiveness under aggressive token compression, particularly the 95.27% mean performance retention and 3.34× prefill speedup at 20% retention on Qwen2.5-Omni-7B.

The reviewers raised valuable concerns regarding the relationship between the surrogate representation error and retained-token distortion (Reviewers f3EP, NpWu, AChE), the individual contributions of region partitioning and shared allocation (Reviewers f3EP, AChE), and the need for broader baseline comparisons and efficiency evaluation (Reviewers NpWu, AChE). Additional suggestions addressed comparison with learned compression methods and generalization to different architectures (f3EP), as well as FlashAttention compatibility and measurement reproducibility (NpWu, AChE).

In response to these valuable comments, we conducted extensive additional analyses and experiments to further strengthen our work, as summarized below:

  • [Reviewers f3EP, NpWu, AChE] Surrogate Validation. We provided additional theoretical analysis clarifying the relationship between surrogate distortion and actual retained-token error. Empirical results show strong correlations (Pearson 0.981, Spearman 0.992), while highest-affinity selection outperforms minimum-distortion selection by 2.00 percentage points in mean performance retention.

  • [Reviewers f3EP, AChE] Controlled Ablations. We introduced separate ablations for region partitioning and shared allocation. Removing either component consistently reduces performance, confirming their individual contributions, particularly under aggressive compression.

  • [Reviewers NpWu, AChE] Additional Baselines. We expanded comparisons to Uniform Stride, DASH, OmniFocus, OmniDrop, and OmniScope. OmniPartition achieves higher mean performance retention under matched token budgets, while maintaining substantial prefill speedups of 3.34× and 4.54× at 20% and 10% retention, respectively.

  • [Reviewer f3EP] OmniSIFT Comparison. We evaluated OmniPartition against OmniSIFT on the same checkpoint. OmniPartition achieves higher mean performance retention and prefill speedups (3.22× vs. 2.66× at 20% retention), without requiring additional training or learnable parameters.

  • [Reviewer f3EP] Generalization to JavisGPT. We extended OmniPartition to JavisGPT, demonstrating its applicability to a different audio–visual fusion architecture without retraining. It preserves 101.22% of full-token performance at 45% retention and 97.29% even at 10% retention.

  • [Reviewers NpWu, AChE] Implementation and Efficiency Clarifications. We clarified FlashAttention usage and the additional attention-extraction overhead of existing methods, detailed the measurement protocol, and corrected an AudioCLIP latency reporting error to ensure accurate and reproducible efficiency comparisons.

These updates are temporarily highlighted in "blue" for facilitating checking.

We believe these additional results and clarifications substantially strengthen the theoretical justification, experimental comparisons, and generalizability of OmniPartition. We sincerely appreciate the reviewers' constructive engagement throughout the rebuttal process and their consideration of our additional evidence. Their feedback has been invaluable in improving the rigor and clarity of our work.

Many thanks,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.