- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OmniPartition, a training-free method for audio–visual token pruning. After the first decoder block, it combines query relevance with spatial and temporal structure, adaptively refining regions to allocate a shared audio–visual token budget. Experiments cover six benchmarks and multiple models. On Qwen2.5-Omni-7B, a 20% layer-average token budget preserves 95.27% of full-token performance on average across the five benchmarks in Table 3, while achieving a 3.34× prefill speedup on WorldSense, including pruning overhead.
Critical strengths
-
The method is practically useful. It requires neither training nor an auxiliary model, combines query guidance with structural preservation and shared-budget allocation, and remains compatible with standard FlashAttention.
-
Results are consistent under aggressive compression. At a 20% budget, the method outperforms OmniZip and OmniSelect on all five benchmarks in Table 3. Results across multiple models and measured speedups that include pruning overhead further support its practical value.
Critical weaknesses
-
Key baseline comparisons are missing. The paper discusses OmniDrop and OmniScope but provides no direct comparisons. The authors’ uniform-sampling baseline is also competitive: at a 45% budget, its average relative performance across the two reported datasets is nearly identical to OmniPartition’s, yet a comprehensive accuracy–latency comparison is missing. The method’s practical advantage therefore remains insufficiently established.
-
The contributions of the core design choices are not isolated. The “w/o RE-PA” ablation changes both budget allocation and within-region token selection, making it difficult to separate the benefits of shared allocation from structural coverage. Moreover, the region error is computed relative to a weighted mean direction, whereas the retained token is selected by maximum query affinity. Reducing the former does not guarantee a reduction in the representation error of the actual retained token; targeted validation is needed.
Questions for authors
-
Could you provide accuracy and latency comparisons against OmniDrop, OmniScope, and uniform sampling under consistent budgets and timing conditions, particularly the OmniDrop measurements mentioned in the introduction?
-
Could you isolate the contributions by replacing only the shared-budget allocation while keeping region partitioning and representative selection unchanged, and then replacing only error-driven partitioning under the same budget?
-
Does the region error computed around the weighted mean reliably predict the representation error of the actual retained token? Is there empirical evidence supporting this surrogate?
-
Please specify the timing samples, input lengths, batch size, numerical precision, and synchronization procedure. Why does OmniSelect’s AudioCLIP overhead vary substantially across retention rates?
4 comments
Response to Reviewer AChE: Controlled Ablations
2. Component-wise Ablations
We thank the reviewer for suggesting more controlled ablations. To distinguish the contributions of shared allocation and region partitioning, we introduce two additional variants while maintaining identical evaluation settings.
Table R4. Ablations on Qwen2.5-Omni-7B with 128 frames.
| Variant | Ret. | WorldSense | DailyOmni | OVB | ShortVid | Mean Ret. (%) |
|---|---|---|---|---|---|---|
| Full Tokens | 100% | 46.82 | 64.16 | 34.60 | 70.50 | 100.00% |
| w/o Shared Allocation | 30% | 44.19 | 59.12 | 33.60 | 69.20 | 95.45% |
| w/o Region Partitioning | 30% | 43.25 | 58.15 | 33.80 | 68.80 | 94.57% |
| OmniPartition | 30% | 44.96 | 59.73 | 34.10 | 69.70 | 96.64% |
| w/o Shared Allocation | 20% | 43.51 | 56.81 | 33.70 | 68.40 | 93.97% |
| w/o Region Partitioning | 20% | 41.33 | 55.47 | 33.30 | 64.80 | 90.72% |
| OmniPartition | 20% | 44.58 | 57.14 | 34.60 | 69.50 | 95.71% |
w/o Shared Allocation. We replace the global allocation procedure with predefined frame- and segment-specific token quotas. Native-coordinate partitioning, representation-error-based split gains, and affinity-based representative selection remain unchanged. Each frame or audio segment maintains an independent refinement heap until its assigned quota is reached. This isolates the contribution of adaptive shared allocation while preserving the remaining components.
w/o Region Partitioning. We remove native-coordinate recursive partitioning while retaining query-affinity-based representative selection and shared allocation. Each video frame or audio segment forms an initial unit, where representatives are added in descending query-affinity order. Tokens are assigned to their nearest representatives in feature space, and the resulting groups are evaluated using the same representation error in Eq. (8). A shared heap allocates additional representatives according to the reduction in aggregate representation error. This variant evaluates the contribution of native-coordinate partitioning under adaptive shared allocation.
As shown in Table R4, removing shared allocation reduces mean performance retention from 96.64% to 95.45%, while removing region partitioning reduces it to 94.57%. The larger performance degradation at 20% retention further demonstrates the importance of shared allocation and native-coordinate partitioning under aggressive compression.
Response to Reviewer AChE: Surrogate Validation
3. Surrogate Representation Error
We sincerely thank you for raising this important question. We first clarify the relationship between the surrogate representation error in Eq. (8) and the actual error associated with the retained token in Eq. (7), followed by empirical evidence supporting the surrogate.
Theoretical Relationship. For a region , let , , , and . The representation error associated with the retained token is
In contrast, the surrogate representation error in Eq. $ 8$ is
Since , the Cauchy–Schwarz inequality gives
Consequently,
with the gap given by
Thus, represents the minimum representation error over all unit-norm directions, providing a continuous relaxation of the discrete retained-token error. When token representations within a region are sufficiently similar, the retained token is close to the weighted mean direction, narrowing the gap between the surrogate and actual errors.
Empirical Evidence. We evaluate how reliably the region error computed around the weighted mean, , predicts the representation error of the actual retained token, , across 324,387 final regions from 100 randomly sampled WorldSense examples.
Table R5. Pearson and Spearman correlations between the region error computed around the weighted mean and the representation error of the actual retained token .
| Representative | Pearson ↑ | Spearman ↑ | Regions |
|---|---|---|---|
| Highest Affinity | 0.981 | 0.992 | 324,387 |
As shown in Table R5, the high Pearson (0.981) and Spearman (0.992) correlations provide strong empirical evidence that the region error computed around the weighted mean reliably predicts the representation error of the actual retained token, supporting its use in guiding region refinement.
Preservation of Task-Relevant Evidence. We additionally compare OmniPartition's highest-affinity representative with the original token that minimizes weighted within-region distortion:
Table R6. Comparison of highest-affinity and minimum-distortion representative selection on Qwen2.5-Omni-7B at 30% retention.
| Variant | WorldSense | DailyOmni | OVB | ShortVid | Mean Ret. (%) |
|---|---|---|---|---|---|
| OmniPartition | 44.96 | 59.73 | 34.10 | 69.70 | 96.64% |
| Min-Distortion Representative | 44.29 | 57.73 | 33.20 | 69.10 | 94.64% |
As shown in Table R6, OmniPartition consistently outperforms minimum-distortion representative selection across all four benchmarks, improving mean performance retention by 2.00 percentage points. These results indicate that minimizing representation error alone does not necessarily preserve task-relevant information as effectively as selecting the highest-affinity token.
Response to Reviewer AChE: Efficiency Measurements
4. Efficiency Measurement Protocol and AudioCLIP Overhead
We thank the reviewer for requesting further details on the efficiency measurements.
Measurement Protocol. We evaluate the same 100 WorldSense samples after one warm-up run, using a batch size of 1 and up to 128 video frames per sample. The original audiovisual durations range from 37 to 580 s (mean 157.1 s; median 101 s). Frame sampling yields 74–128 frames per example (mean 119.2).
Qwen2.5-Omni runs in BF16, while OmniSelect's AudioCLIP uses FP32, following its default configuration. Latency is measured using CUDA events, with synchronization at each measurement boundary. GPU peak memory statistics are reset after preprocessing for each sample. We report mean latency and the maximum peak memory across all samples. CPU preprocessing and autoregressive decoding are excluded.
AudioCLIP Overhead. We apologize for a data entry error affecting OmniSelect's 20% retention results in the original Table 5. We re-measured this setting under the same experimental conditions and obtained consistent results across repeated runs. The updated latency breakdown is reported in Table R7, with AudioCLIP latency corrected from 2242.06 ms to 1353.96 ms.
The corrected measurement resolves the anomalous value in Table 5. OmniSelect's substantial AudioCLIP overhead remains the main contributor to its total prefill latency, while OmniPartition avoids the auxiliary model and achieves substantially lower latency.
Table R7. Corrected prefill efficiency at 20% retention on Qwen2.5-Omni-7B (H100). Latencies are in milliseconds. AudioCLIP latency is shown in parentheses and included in Total.
| Method | Ret. | Peak Mem. (GiB) | LLM Prefill | Extra attn.(ms) | Prune (ms) | Total (ms) | Speedup |
|---|---|---|---|---|---|---|---|
| OmniSelect | 20% | 18.80 | 73.20 | 12.34 | 15.06 (+1353.96) | 1454.57 | 0.27× |
| OmniPartition (Triton) | 20% | 18.17 | 77.77 | – | 41.40 | 119.17 | 3.34× |
Response to Reviewer AChE: Additional Baselines
We sincerely thank the reviewer for the constructive feedback and for recognizing the practical value of OmniPartition, particularly its training-free design, compatibility with FlashAttention, and effectiveness under aggressive audio–visual token compression. We address the concerns below with additional comparisons, controlled ablations, and theoretical clarification.
1. Baseline Comparisons and Efficiency
We thank the reviewer for requesting more comprehensive comparisons. We conduct additional experiments with OmniDrop, OmniScope, and Uniform Stride on Qwen2.5-Omni-7B under matched layer-average token retention rates.
As shown in Tables R1–R2, Uniform Stride remains competitive under moderate compression, particularly on DailyOmni and OmniVideoBench. However, the additional WorldSense results reveal its limitations, even at 45% retention (42.69 vs. 46.15). The performance gap widens under aggressive compression. At 20% retention, OmniPartition achieves higher mean performance retention than all three additional baselines(OmniDrop, Uniform Stride, OmniScope).
Table R1. Accuracy comparisons at 45% and 30% retention.
Table R2. Accuracy comparisons at 20% and 10% retention.
Efficiency Analysis. As shown in Table R3, Uniform Stride achieves the lowest latency. At 20% retention, OmniPartition achieves a 3.34× speedup, compared with 1.43× for OmniDrop and 0.27× for OmniScope, which incur additional overhead from repeated pruning and attention computation, and an auxiliary CLIP model, respectively.
Table R3. Prefill efficiency on Qwen2.5-Omni-7B H100.
These results clarify the trade-off between selection overhead and retained information: Uniform Stride favors computational simplicity, while OmniPartition achieves stronger performance preservation under aggressive compression with substantially lower latency than OmniDrop.