- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies audio-visual token compression in omni-modal large language models and extends a previous image-domain idea, "first generate labels with evolutionary search, then train a compressor", to audio and video. In the offline stage, the authors split the video into temporal segments according to changes in visual semantics, and the audio follows the same segmentation. Within each segment, tokens are grouped and evolutionary search selects the token to keep in each group, where the criterion is whether the frozen model can reproduce its own original answer after compression; modality importance labels are obtained by masking the video or the audio separately. In the online stage, the authors train a lightweight compressor with these labels, weighting the supervision of different temporal segments during training, and at inference combine token-level and modality-level scores to select the retained audio and video tokens in a single pass before the LLM. Experiments show that the method approaches the full model at low retention ratios, still achieves clear speedups after accounting for the compressor overhead, and can be applied without retraining to a smaller model of the same family.
Critical strengths
- A good accuracy–efficiency trade-off. At lower retention ratios the method matches or exceeds existing methods at higher retention ratios, in some settings even surpassing the full model, and the reported speedups include the compressor's own overhead.
- Shows that temporal structure is necessary for audio-visual labeling. The ablations show a large drop when temporal segmentation is removed, indicating that directly transferring image-domain labeling to audio-visual inputs is insufficient and that distinguishing events in different temporal segments during search is effective.
- Labels defined by the target model's output loss. The labels do not rely on indirect signals such as attention or similarity; instead, candidate subsets are evaluated by the loss of the target OmniLLM. The idea is clear and provides a meaningful contrast to heuristic methods.
Critical weaknesses
- The core motivation is inconsistent with the experimental results. The paper attributes the common problem of existing methods to reliance on heuristic signals and not optimizing for the final output, yet OmniSIFT [1] trains an audio scorer and is itself a learned method. More importantly, in Table 4 the variant without TGTC is the closest to directly applying EvoComp [2] to audio-visual inputs, and it obtains only 55.0 and 50.3 on DailyOmni at 45% and 30% retention, below the training-free OmniZip [3] (60.0, 56.0) and OmniSelect [4] (60.7, 57.4); it only overtakes them after temporal structure is added. Hence the gains come mainly from the temporal structure rather than from "optimizing for the output" itself. The core narrative should be revised, and a control that isolates the value of output-driven labels should be added.
- Limited novelty over EvoComp [2]: all three additions have established counterparts. The first is temporal segmentation: FastVID [5], HoliTom [6] and MMG-Vid [7] all segment videos adaptively based on content, and TGTC only replaces the segmentation signal with topic distributions. The second, TWR, is essentially a weighted auxiliary classification loss, and neither it nor TGTC has been separated from simpler alternatives (W3). The third, query-dependent modality budgeting, is already done by OmniSelect [4], while MLI here brings only 0.3 and 0.6 points. Overall, the work reads more like an engineering extension of EvoComp to audio and video, and I suggest stating the new contributions accurately.
- The gains of neither TWR nor TGTC are separated from simpler alternatives. TWR in Eq. (6) is an auxiliary BCE loss with weight 0.3, and "w/o TWR" in Table 4 removes the entire auxiliary loss rather than only the temporal weights, so the 6.5 and 7.5 point drops may come from the extra classification supervision or a change in loss scale rather than the temporal weighting itself. TGTC sets the threshold at the 90th percentile within each video, so about 10% of adjacent frame pairs always become boundaries regardless of content, while the existing controls are only no segmentation and DPC-KNN. I suggest adding controls with constant window weights, shuffled window weights, and uniform segmentation with the same number of segments or FastVID [5]/HoliTom [6]-style feature-based segmentation.
- The supervision target is reproducing the teacher's own answer, which is inconsistent with the claim of "optimizing for correctness" and with QA-based evaluation. The paper criticizes existing methods for "not directly optimizing for the correctness of the final output" (l.53), but the labels here optimize for the model reproducing its own answer, and for LongVALE the target is even a model-generated segment description (Appendix C.1), whereas all four evaluation benchmarks are multiple-choice QA. The paper neither examines the relationship between the description-reproduction loss and QA accuracy nor checks whether the labels change with the question.
- Training data and offline cost are not disclosed. The paper does not report the size of the LongVALE subset used, the number of labeled samples, the total GPU hours for teacher generation, topic modeling and evolutionary search, the number of compressor parameters, or whether training videos are deduplicated against evaluation videos. The roughly 192 candidate evaluations per sample are only part of the cost and do not allow estimating the total. I found no evidence of data leakage, but this information is necessary to assess cost and reproducibility.
References
[1] Yue Ding, Yiyan Ji, Jungang Li, et al. OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-Modal Large Language Models. In ICML, 2026.
[2] Jiafei Song, Fengwei Zhou, Jin Qu, et al. EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling. In CVPR, 2026.
[3] Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Jian Liu, Huan Wang. OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models. In CVPR, 2026.
[4] Morunliu Yang, Ruotao Xu, Le Li, et al. OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-Modal Large Language Models. arXiv:2605.18041, 2026.
[5] Leqi Shen, Guoqiang Gong, Tao He, et al. FastVID: Dynamic Density Pruning for Fast Video Large Language Models. In NeurIPS, 2025.
[6] Kele Shao, Keda Tao, et al. HoliTom: Holistic Token Merging for Fast Video Large Language Models. In NeurIPS, 2025.
[7] Junpeng Ma, et al. MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs. In AAAI, 2026.
Questions for authors
- With the same temporal segmentation and grouping, if evolutionary search and learning are replaced by simple heuristic rules (e.g., selecting the token closest to the group mean, or using OmniZip [3]'s scores), how large is the gap to the full method? This would directly show the value of output-driven labels and is the experiment most likely to change my assessment.
- Could you add two controls: (a) TWR with constant and shuffled weights while keeping the total weight unchanged; (b) replacing TGTC with uniform segmentation of the same number of segments or FastVID [5]-style feature-based segmentation, with everything else unchanged?
- Were the variants in Table 4 re-tuned? What are the means and variances of the full model and the variants over multiple random seeds?
- On held-out QA samples with ground-truth answers, is the ranking of candidate subsets by the teacher-answer loss consistent with answer accuracy? How do the description targets of LongVALE yield question-dependent labels?
- Please report the size of the training set, how samples were selected and deduplicated against the evaluation sets, the GPU hours of each offline stage, and the number of compressor parameters. Based on the computation saved per request, how many requests are needed to amortize the offline cost?
2 comments
Q5: How do the description targets of LongVALE yield question-dependent labels?
A5: We would like to clarify that LongVALE provides description-conditioned supervision, rather than directly generating question-dependent labels. Specifically, during evolutionary labeling, the output loss is computed over the description tokens, encouraging the search to retain audio-video tokens relevant to the description.
During compressor training, we include the description as text input for LongVALE samples, whereas for Video-Holmes samples, we provide the corresponding question as text input. We first train the compressor on LongVALE and then further train it on Video-Holmes. In this process, LongVALE serves as a pretraining stage for learning general audio-video token importance, while Video-Holmes provides question-conditioned adaptation.
Q6: Were the variants in Table 4 re-tuned? What are the means and variances of the full model and the variants over multiple random seeds?
A6: We thank the reviewer for raising this concern. All variants in Table 4 are trained using the same hyperparameters and training procedure as the full EvoComp-2 model, without variant-specific re-tuning. Due to the computational cost of label generating and compressor training, we evaluate each configuration using a single training run. Nevertheless, the ablation results show consistent performance degradation when key components, particularly topic-guided temporal clustering and temporal-window reweighting, are removed at both 45% and 30% token retention ratios.
Furthermore, our additional controlled experiments in A1 and A2 provide complementary evidence for the effectiveness of evolutionary labeling and temporal-window reweighting.
Q7: Please report the size of the training set, how samples were selected and deduplicated against the evaluation sets, the GPU hours of each offline stage, and the number of compressor parameters.
A7: We construct supervision using 17,144 video segments from LongVALE, selected with durations between 10 and 200 seconds, and all 3,388 samples from Video-Holmes (1,551 training and 1,837 test samples), totaling 20,532 samples. The evaluation benchmarks in Tables 1–3 are sourced from different datasets (WorldSense, ShortVid-Bench, AVUT, and DailyOmni). Therefore, we did not perform additional cross-dataset deduplication against the evaluation sets.
The offline computational costs are summarized below:
| Stage | GPUs | Time |
|---|---|---|
| Target Response Generation | 24 × A100 | 3.2 h |
| Evolutionary Label Construction | 24 × A100 | 15.3 h |
| Topic Model Training | 1 × A100 | 0.5 h |
| Compressor Training | 2 × A100 | 47 h |
The number of parameters of the trained compressor is 223M.
Q8: Based on the computation saved per request, how many requests are needed to amortize the offline cost?
A8: 1) The number of requests required to amortize the offline cost depends heavily on video duration, as longer videos generally involve more audio-video tokens and offer greater computational savings per request. Therefore, a fixed break-even request count would not be representative of different deployment scenarios.
2) The offline cost reported in A7 can potentially be reduced in practical applications. Our current training uses a relatively broad dataset to support generalization across different tasks. For a specific application, a smaller, task-specific training dataset may be sufficient, potentially reducing the costs of supervision construction and compressor training.
3) Our cross-model transfer results in Table 3 demonstrate that the trained compressor can be reused across OmniLLMs within the same model family, avoiding repeated offline costs.
Response to Reviewer cdpw
We thank the reviewer for the detailed evaluation and constructive suggestions, as well as for acknowledging EvoComp-2's competitive accuracy–efficiency trade-offs. We address your concerns below.
Q1: With the same temporal segmentation and grouping, if evolutionary search and learning are replaced by simple heuristic rules, how large is the gap to the full method?
A1: To isolate the contribution of evolutionary search, we conduct experiments using the same topic-guided temporal clustering and semantic grouping as EvoComp-2, while replacing evolutionary search with a simple heuristic (EvoComp-2 w/ Heuristic Selection): selecting the token with the smallest L2 distance to the mean representation of each semantic group. We regenerate the token-retention labels for the entire training dataset and train the compressor with the same training procedure as EvoComp-2. The results on DailyOmni are shown below, together with the ablation results from Table 4.
It can be seen that replacing evolutionary search with heuristic selection leads to substantial accuracy degradation.
Q2: Could you add the control: TWR with constant and shuffled weights while keeping the total weight unchanged?
A2: To further investigate the effectiveness of TWR, we conduct experiments in which the temporal weights are replaced with a constant (EvoComp-2 w/ Constant TWR Weights), while keeping all other training settings unchanged. The results on DailyOmni are shown below, alongside the ablation results from Table 4.
The constant-weight variant performs similarly to the variant without TWR. These results suggest that the improvement primarily comes from adaptively emphasizing informative temporal regions, rather than simply introducing an additional loss term.
Q3: Could you add the control: replacing TGTC with uniform segmentation of the same number of segments or FastVID [5]-style feature-based segmentation, with everything else unchanged?
A3: To assess the effectiveness of TGTC, we conduct experiments on 1,551 samples from the Video-Holmes training set, comparing the inference accuracy of evolutionary search-generated token masks under different temporal segmentation strategies.
Specifically, we consider two alternatives: 1) Uniform Segmentation evenly partitions the frame sequence into approximately 10% as many intervals as frames, matching the number of intervals produced by TGTC; and 2) Visual-Feature-Based Segmentation, following FastVID (Shen et al., 2025), identifies temporal boundaries based on cosine similarity between consecutive frame representations, selecting the seven lowest-similarity positions and any additional positions with similarity below 0.9. All subsequent semantic grouping and evolutionary search procedures remain unchanged.
Topic-guided segmentation achieves the highest overall accuracy. The advantage is more pronounced on Timeline Analysis (TA), which requires reconstructing the sequence of events in a video. These findings support the effectiveness of TGTC in identifying meaningful semantic transitions.
Q4: On held-out QA samples with ground-truth answers, is the ranking of candidate subsets by the teacher-answer loss consistent with answer accuracy?
A4: To investigate the relationship between teacher-answer loss and downstream accuracy, we analyze the evolutionary search results from A3 on 1,551 Video-Holmes training samples. For each segmentation strategy, we record the teacher-answer loss of the best mask identified by evolutionary search and compare its average loss with the corresponding QA accuracy.
The three strategies exhibit consistent rankings between average teacher-answer loss and downstream QA accuracy: lower average loss corresponds to higher accuracy. In particular, our topic-guided segmentation achieves both the lowest average loss and the highest accuracy.