acceptodds
Under review as a conference paper at ICLR 2027

EvoComp-2: Learning to Compress Audio-Video Tokens for Omnimodal Large Language Models via Topic-Guided Evolutionary Labeling

Abstract

Omnimodal Large Language Models (OmniLLMs) achieve strong performance on audio-video tasks but incur substantial computational overhead due to the large number of audio and video tokens. EvoComp proposes to adopt evolutionary search to construct token-retention supervision and train a lightweight compressor for visual token compression. In this work, we extend EvoComp to OmniLLMs and propose EvoComp-2 for efficient audio-video token compression. Extending EvoComp from static visual inputs to synchronized audio-video sequences introduces new challenges arising from temporal redundancy and query-dependent modality importance. To address them, EvoComp-2 introduces topic-guided temporal clustering that adapts a topic model to video frames and partitions audio-video sequences into semantically coherent intervals before evolutionary labeling, enabling more structured and efficient token selection. We further construct modality-level importance supervision to capture the relative contribution of audio and video to different queries. For compressor training, we introduce a temporal-window reweighting loss to emphasize information-rich temporal regions and a modality-level importance loss to learn query-dependent modality preference, together with the GHM loss inherited from EvoComp. Extensive experiments demonstrate that EvoComp-2 achieves favorable accuracy-efficiency trade-offs. On WorldSense, EvoComp-2 preserves 98.7% of the original accuracy while retaining only 35% of the audio-video tokens. In terms of inference efficiency, it achieves a 1.6x end-to-end speedup on DailyOmni with 3.3x token compression.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Video Token Compression, Topic Model, Evolutionary Search, Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cdpwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies audio-visual token compression in omni-modal large language models and extends a previous image-domain idea, "first generate labels with evolutionary search, then train a compressor", to audio and video. In the offline stage, the authors split the video into temporal segments according to changes in visual semantics, and the audio follows the same segmentation. Within each segment, tokens are grouped and evolutionary search selects the token to keep in each group, where the criterion is whether the frozen model can reproduce its own original answer after compression; modality importance labels are obtained by masking the video or the audio separately. In the online stage, the authors train a lightweight compressor with these labels, weighting the supervision of different temporal segments during training, and at inference combine token-level and modality-level scores to select the retained audio and video tokens in a single pass before the LLM. Experiments show that the method approaches the full model at low retention ratios, still achieves clear speedups after accounting for the compressor overhead, and can be applied without retraining to a smaller model of the same family.

Critical strengths
  1. A good accuracy–efficiency trade-off. At lower retention ratios the method matches or exceeds existing methods at higher retention ratios, in some settings even surpassing the full model, and the reported speedups include the compressor's own overhead.
  2. Shows that temporal structure is necessary for audio-visual labeling. The ablations show a large drop when temporal segmentation is removed, indicating that directly transferring image-domain labeling to audio-visual inputs is insufficient and that distinguishing events in different temporal segments during search is effective.
  3. Labels defined by the target model's output loss. The labels do not rely on indirect signals such as attention or similarity; instead, candidate subsets are evaluated by the loss of the target OmniLLM. The idea is clear and provides a meaningful contrast to heuristic methods.
Critical weaknesses
  1. The core motivation is inconsistent with the experimental results. The paper attributes the common problem of existing methods to reliance on heuristic signals and not optimizing for the final output, yet OmniSIFT [1] trains an audio scorer and is itself a learned method. More importantly, in Table 4 the variant without TGTC is the closest to directly applying EvoComp [2] to audio-visual inputs, and it obtains only 55.0 and 50.3 on DailyOmni at 45% and 30% retention, below the training-free OmniZip [3] (60.0, 56.0) and OmniSelect [4] (60.7, 57.4); it only overtakes them after temporal structure is added. Hence the gains come mainly from the temporal structure rather than from "optimizing for the output" itself. The core narrative should be revised, and a control that isolates the value of output-driven labels should be added.
  2. Limited novelty over EvoComp [2]: all three additions have established counterparts. The first is temporal segmentation: FastVID [5], HoliTom [6] and MMG-Vid [7] all segment videos adaptively based on content, and TGTC only replaces the segmentation signal with topic distributions. The second, TWR, is essentially a weighted auxiliary classification loss, and neither it nor TGTC has been separated from simpler alternatives (W3). The third, query-dependent modality budgeting, is already done by OmniSelect [4], while MLI here brings only 0.3 and 0.6 points. Overall, the work reads more like an engineering extension of EvoComp to audio and video, and I suggest stating the new contributions accurately.
  3. The gains of neither TWR nor TGTC are separated from simpler alternatives. TWR in Eq. (6) is an auxiliary BCE loss with weight 0.3, and "w/o TWR" in Table 4 removes the entire auxiliary loss rather than only the temporal weights, so the 6.5 and 7.5 point drops may come from the extra classification supervision or a change in loss scale rather than the temporal weighting itself. TGTC sets the threshold at the 90th percentile within each video, so about 10% of adjacent frame pairs always become boundaries regardless of content, while the existing controls are only no segmentation and DPC-KNN. I suggest adding controls with constant window weights, shuffled window weights, and uniform segmentation with the same number of segments or FastVID [5]/HoliTom [6]-style feature-based segmentation.
  4. The supervision target is reproducing the teacher's own answer, which is inconsistent with the claim of "optimizing for correctness" and with QA-based evaluation. The paper criticizes existing methods for "not directly optimizing for the correctness of the final output" (l.53), but the labels here optimize for the model reproducing its own answer, and for LongVALE the target is even a model-generated segment description (Appendix C.1), whereas all four evaluation benchmarks are multiple-choice QA. The paper neither examines the relationship between the description-reproduction loss and QA accuracy nor checks whether the labels change with the question.
  5. Training data and offline cost are not disclosed. The paper does not report the size of the LongVALE subset used, the number of labeled samples, the total GPU hours for teacher generation, topic modeling and evolutionary search, the number of compressor parameters, or whether training videos are deduplicated against evaluation videos. The roughly 192 candidate evaluations per sample are only part of the cost and do not allow estimating the total. I found no evidence of data leakage, but this information is necessary to assess cost and reproducibility.

References

[1] Yue Ding, Yiyan Ji, Jungang Li, et al. OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-Modal Large Language Models. In ICML, 2026.
[2] Jiafei Song, Fengwei Zhou, Jin Qu, et al. EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling. In CVPR, 2026.
[3] Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Jian Liu, Huan Wang. OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models. In CVPR, 2026.
[4] Morunliu Yang, Ruotao Xu, Le Li, et al. OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-Modal Large Language Models. arXiv:2605.18041, 2026.
[5] Leqi Shen, Guoqiang Gong, Tao He, et al. FastVID: Dynamic Density Pruning for Fast Video Large Language Models. In NeurIPS, 2025.
[6] Kele Shao, Keda Tao, et al. HoliTom: Holistic Token Merging for Fast Video Large Language Models. In NeurIPS, 2025.
[7] Junpeng Ma, et al. MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs. In AAAI, 2026.

Questions for authors
  1. With the same temporal segmentation and grouping, if evolutionary search and learning are replaced by simple heuristic rules (e.g., selecting the token closest to the group mean, or using OmniZip [3]'s scores), how large is the gap to the full method? This would directly show the value of output-driven labels and is the experiment most likely to change my assessment.
  2. Could you add two controls: (a) TWR with constant and shuffled weights while keeping the total weight unchanged; (b) replacing TGTC with uniform segmentation of the same number of segments or FastVID [5]-style feature-based segmentation, with everything else unchanged?
  3. Were the variants in Table 4 re-tuned? What are the means and variances of the full model and the variants over multiple random seeds?
  4. On held-out QA samples with ground-truth answers, is the ranking of candidate subsets by the teacher-answer loss consistent with answer accuracy? How do the description targets of LongVALE yield question-dependent labels?
  5. Please report the size of the training set, how samples were selected and deduplicated against the evaluation sets, the GPU hours of each offline stage, and the number of compressor parameters. Based on the computation saved per request, how many requests are needed to amortize the offline cost?
2 comments
AuthorsOpenReview ↗

Response to Reviewer cdpw

We thank the reviewer for the detailed evaluation and constructive suggestions, as well as for acknowledging EvoComp-2's competitive accuracy–efficiency trade-offs. We address your concerns below.


Q1: With the same temporal segmentation and grouping, if evolutionary search and learning are replaced by simple heuristic rules, how large is the gap to the full method?

A1: To isolate the contribution of evolutionary search, we conduct experiments using the same topic-guided temporal clustering and semantic grouping as EvoComp-2, while replacing evolutionary search with a simple heuristic (EvoComp-2 w/ Heuristic Selection): selecting the token with the smallest L2 distance to the mean representation of each semantic group. We regenerate the token-retention labels for the entire training dataset and train the compressor with the same training procedure as EvoComp-2. The results on DailyOmni are shown below, together with the ablation results from Table 4.

Method45% Retention30% Retention
EvoComp-2 w/o TGTC55.050.3
EvoComp-2 w/ DPC-KNN52.147.1
EvoComp-2 w/ Heuristic Selection53.350.2
EvoComp-261.759.4

It can be seen that replacing evolutionary search with heuristic selection leads to substantial accuracy degradation.


Q2: Could you add the control: TWR with constant and shuffled weights while keeping the total weight unchanged?

A2: To further investigate the effectiveness of TWR, we conduct experiments in which the temporal weights are replaced with a constant (EvoComp-2 w/ Constant TWR Weights), while keeping all other training settings unchanged. The results on DailyOmni are shown below, alongside the ablation results from Table 4.

Method45% Retention30% Retention
EvoComp-2 w/o TWR55.251.9
EvoComp-2 w/ Constant TWR Weights56.952.3
EvoComp-261.759.4

The constant-weight variant performs similarly to the variant without TWR. These results suggest that the improvement primarily comes from adaptively emphasizing informative temporal regions, rather than simply introducing an additional loss term.


Q3: Could you add the control: replacing TGTC with uniform segmentation of the same number of segments or FastVID [5]-style feature-based segmentation, with everything else unchanged?

A3: To assess the effectiveness of TGTC, we conduct experiments on 1,551 samples from the Video-Holmes training set, comparing the inference accuracy of evolutionary search-generated token masks under different temporal segmentation strategies.

Specifically, we consider two alternatives: 1) Uniform Segmentation evenly partitions the frame sequence into approximately 10% as many intervals as frames, matching the number of intervals produced by TGTC; and 2) Visual-Feature-Based Segmentation, following FastVID (Shen et al., 2025), identifies temporal boundaries based on cosine similarity between consecutive frame representations, selecting the seven lowest-similarity positions and any additional positions with similarity below 0.9. All subsequent semantic grouping and evolutionary search procedures remain unchanged.

Segmentation MethodOverall AccuracyTimeline Analysis (TA)
Full Tokens45.0040.59
Uniform Segmentation44.1743.53
Visual-Feature-Based Segmentation43.8442.35
Topic-Guided Segmentation (Ours)44.5545.29

Topic-guided segmentation achieves the highest overall accuracy. The advantage is more pronounced on Timeline Analysis (TA), which requires reconstructing the sequence of events in a video. These findings support the effectiveness of TGTC in identifying meaningful semantic transitions.


Q4: On held-out QA samples with ground-truth answers, is the ranking of candidate subsets by the teacher-answer loss consistent with answer accuracy?

A4: To investigate the relationship between teacher-answer loss and downstream accuracy, we analyze the evolutionary search results from A3 on 1,551 Video-Holmes training samples. For each segmentation strategy, we record the teacher-answer loss of the best mask identified by evolutionary search and compare its average loss with the corresponding QA accuracy.

Segmentation MethodAverage Teacher-Answer Loss ↓QA Accuracy ↑
Uniform Segmentation0.36875544.17
Visual-Feature-Based Segmentation0.37077743.84
Topic-Guided Segmentation (Ours)0.36861844.55

The three strategies exhibit consistent rankings between average teacher-answer loss and downstream QA accuracy: lower average loss corresponds to higher accuracy. In particular, our topic-guided segmentation achieves both the lowest average loss and the highest accuracy.


Q5: How do the description targets of LongVALE yield question-dependent labels?

A5: We would like to clarify that LongVALE provides description-conditioned supervision, rather than directly generating question-dependent labels. Specifically, during evolutionary labeling, the output loss is computed over the description tokens, encouraging the search to retain audio-video tokens relevant to the description.

During compressor training, we include the description as text input for LongVALE samples, whereas for Video-Holmes samples, we provide the corresponding question as text input. We first train the compressor on LongVALE and then further train it on Video-Holmes. In this process, LongVALE serves as a pretraining stage for learning general audio-video token importance, while Video-Holmes provides question-conditioned adaptation.


Q6: Were the variants in Table 4 re-tuned? What are the means and variances of the full model and the variants over multiple random seeds?

A6: We thank the reviewer for raising this concern. All variants in Table 4 are trained using the same hyperparameters and training procedure as the full EvoComp-2 model, without variant-specific re-tuning. Due to the computational cost of label generating and compressor training, we evaluate each configuration using a single training run. Nevertheless, the ablation results show consistent performance degradation when key components, particularly topic-guided temporal clustering and temporal-window reweighting, are removed at both 45% and 30% token retention ratios.

Furthermore, our additional controlled experiments in A1 and A2 provide complementary evidence for the effectiveness of evolutionary labeling and temporal-window reweighting.


Q7: Please report the size of the training set, how samples were selected and deduplicated against the evaluation sets, the GPU hours of each offline stage, and the number of compressor parameters.

A7: We construct supervision using 17,144 video segments from LongVALE, selected with durations between 10 and 200 seconds, and all 3,388 samples from Video-Holmes (1,551 training and 1,837 test samples), totaling 20,532 samples. The evaluation benchmarks in Tables 1–3 are sourced from different datasets (WorldSense, ShortVid-Bench, AVUT, and DailyOmni). Therefore, we did not perform additional cross-dataset deduplication against the evaluation sets.

The offline computational costs are summarized below:

StageGPUsTime
Target Response Generation24 × A1003.2 h
Evolutionary Label Construction24 × A10015.3 h
Topic Model Training1 × A1000.5 h
Compressor Training2 × A10047 h

The number of parameters of the trained compressor is 223M.


Q8: Based on the computation saved per request, how many requests are needed to amortize the offline cost?

A8: 1) The number of requests required to amortize the offline cost depends heavily on video duration, as longer videos generally involve more audio-video tokens and offer greater computational savings per request. Therefore, a fixed break-even request count would not be representative of different deployment scenarios.

2) The offline cost reported in A7 can potentially be reduced in practical applications. Our current training uses a relatively broad dataset to support generalization across different tasks. For a specific application, a smaller, task-specific training dataset may be sufficient, potentially reducing the costs of supervision construction and compressor training.

3) Our cross-model transfer results in Table 3 demonstrate that the trained compressor can be reused across OmniLLMs within the same model family, avoiding repeated offline costs.


Reviewer_KgxrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvoComp-2, an audio-video token compression framework for OmniLLMs. It combines topic-guided temporal clustering and evolutionary search to generate token-level supervision, and trains a lightweight compressor with temporal and modality-level importance modeling. Experiments demonstrate promising accuracy-efficiency trade-offs on several audio-video understanding benchmarks.

Critical strengths
  1. Audio-video token compression is important for efficient OmniLLM inference. The proposed approach is lightweight and does not require modifying the underlying OmniLLM.
  2. Using evolutionary search guided by the target model's output loss to construct token-level supervision is an interesting approach. The topic-guided clustering further considers temporal redundancy.
  3. The method achieves competitive performance across several benchmarks, with up to 1.6× end-to-end inference speedup.
Critical weaknesses

For the method:

  1. Lack of direct evaluation of evolutionary supervision. Topic-guided evolutionary search generates the supervision labels and potentially determines the compressor's performance ceiling. However, the paper does not evaluate the downstream performance of the search-generated token masks directly, making it difficult to assess the quality of the supervision and the gap between the search and the learned compressor.
  2. Oversimplified modality importance modeling. During inference, all tokens from the same modality are multiplied by a shared importance score. This ignores temporal variations in modality importance. Furthermore, the binary modality-level supervision may be inadequate when both modalities are essential.
  3. If only 1 token is retained in one interval, the ASR performance will be heavily affected.

For the experiments: 1.All experiments focus on multiple-choice QA, without evaluating generation tasks such as video captioning. 2. Experiments only use Qwen2.5-Omni-7B and 3B, which belong to the same, relatively old model family. Cross-model transfer within this family is insufficient to demonstrate general applicability. 3. Insufficient training data details. Although Appendix C describes the use of LongVALE and Video-Holmes, the exact training sample counts and selection criteria remain unclear.

Questions for authors

See the weakness section~

5 comments
AuthorsOpenReview ↗

Response to Reviewer Kgxr

We appreciate the reviewer's constructive feedback and recognition of EvoComp-2's lightweight architecture and promising accuracy–efficiency trade-offs. We address the reviewer's concerns below.


Q1: Although Appendix C describes the use of LongVALE and Video-Holmes, the exact training sample counts and selection criteria remain unclear.

A1: For LongVALE, we extract video segments with durations between 10 and 200 seconds, yielding 17,144 training samples. For Video-Holmes, we use all 3,388 samples, including 1,551 from the training set and 1,837 from the test set. In total, our supervision dataset contains 20,532 samples.


Q2: The paper does not evaluate the search-generated token masks directly, making it difficult to assess the quality of the supervision and the gap between the search and the learned compressor.

A2: To directly evaluate the quality of search-generated masks and the performance gap between evolutionary search and the trained compressor, we conduct experiments on 1,551 samples from the Video-Holmes training set. We compare three token selection strategies: 1) Evolutionary Search directly uses the token masks generated by our evolutionary search; 2) Compressor + Group-Constrained Selection retains topic-guided temporal clustering and semantic grouping, but selects the token with the highest retention score predicted by the trained compressor within each semantic group; 3) Compressor Directly selects the top 35% of audio-video tokens based on the trained compressor's predicted retention scores. All three strategies retain 35% of the audio-video tokens, following the semantic group selection procedure described in Appendix C.1.

MethodAccuracy
Full Tokens45.00
Evolutionary Search44.55
Compressor + Group-Constrained Selection43.84
Compressor42.23

The results demonstrate that evolutionary search generates high-quality token masks that largely preserve the original model's performance. Moreover, the group-constrained selection results indicate that the trained compressor learns informative token retention scores. Although the last one achieves slightly lower accuracy, it provides a more lightweight and efficient inference strategy at inference time.


Q3: Oversimplified modality importance modeling.

A3: We would like to clarify that modality-level importance learning is designed to provide a coarse-grained adjustment to the token-level retention scores, rather than modeling modality importance across temporal intervals. In EvoComp-2, fine-grained temporal importance is captured through topic-guided evolutionary labeling and temporal-window reweighting. Modality-level importance learning complements this process by estimating the overall query-dependent preference between audio and video. At inference, modality-level scores serve as soft adjustments to token-level scores. Hence, informative tokens from both modalities can still be preserved.

While binary modality-level supervision does not explicitly capture temporal variations in modality preference, it complements the fine-grained token-level supervision with a lightweight global adjustment. The ablation results in Table 4 further demonstrate the effectiveness of this complementary design.


Q4: If only 1 token is retained in one interval, the ASR performance will be heavily affected.

A4: We would like to clarify a potential misunderstanding. During evolutionary labeling, we first partition audio-video tokens into temporal intervals and then perform semantic grouping separately for each modality within each interval. Each interval contains multiple semantic groups, and evolutionary search selects one token from each optimized group, rather than retaining only one token per temporal interval.


Q5: All experiments focus on multiple-choice QA, without evaluating generation tasks such as video captioning.

A5: We initially focused on multiple-choice QA benchmarks for more direct and reproducible accuracy evaluation, as video captioning typically relies on LLM-based evaluation, which may introduce additional variability. Nevertheless, to provide a more comprehensive evaluation, we conducted experiments on the video-SALMONN-2 captioning benchmark, using GPT-4.1 as the evaluator. We report three metrics: Missed Event, Incorrect Event, and Hallucination Event (lower is better). The results at a 30% audio-video token retention ratio are shown below.

MethodRetained RatioMissed Event ↓Incorrect Event ↓Hallucination Event ↓
Full Tokens100%40.110.38.0
Random Selection30%49.111.68.4
OmniSelect30%48.410.98.7
EvoComp-230%47.410.68.0

EvoComp-2 achieves the best performance among the compressed methods across all three metrics. These results demonstrate that EvoComp-2 also achieves competitive performance on video captioning tasks, extending its effectiveness beyond multiple-choice QA benchmarks.


Q6: Cross-model transfer within Qwen family is insufficient to demonstrate general applicability.

A6: We agree that our current experiments primarily demonstrate cross-model transfer from Qwen2.5-Omni-7B to Qwen2.5-Omni-3B. We acknowledge that the statement "can generalize across OmniLLMs of different scales" in Line 430 is overly broad. We will revise it to "can generalize across OmniLLMs of different scales within the same model family" to more accurately reflect our experimental findings.


Thanks for your detailed reponses. However, I still have concerns on Q3, Q4 and Q6. I will retain my rating. Q3: I think the modality importance modeling is very important, and the design is too simplified. Q4: No ASR results are shown. For example, a movie shot with two people speaking all the time, could your compression technique ecover the ASR result? Q6. Thanks for your reply but I still think Qwen2.5-omni experiments are limited.

Reviewer_tfeFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvoComp-2 extends evolutionary token-retention labeling to audio-video inputs. A neural topic model partitions the temporal sequence; semantic groups within intervals constrain an output-loss-based search for retention masks. A lightweight compressor is trained using GHM, temporal-window reweighting, and modality-importance supervision from separate modality-zeroing tests. At inference, token and modality scores determine joint audio-video retention. Results cover four benchmarks, transfer from Qwen2.5-Omni-7B to 3B, and inference latency. The main proposed increment is structured temporal supervision for efficient audio-video compression.

Critical strengths
  1. Practical accuracy–efficiency results. WorldSense accuracy is 46.2 at 35% token retention versus 46.8 with full tokens, preserving approximately 98.7% of the original accuracy. On DailyOmni, reported end-to-end latency decreases from 2.17 s to 1.36 s at approximately 3.3-fold compression. Measuring actual latency, including compressor overhead in prefill, strengthens the practical contribution.
  2. A sensible separation of offline search and deployment. Expensive supervision construction is performed offline, while deployment requires one compressor pass without fine-tuning the main model or modality encoders. Multiple benchmarks and retention ratios give the approach a useful empirical foundation.
Critical weaknesses
  1. An important learned competitor is omitted and the prior-work distinction is inaccurate. Section 2 groups OmniSIFT with methods relying on heuristic signals, but OmniSIFT explicitly uses end-to-end optimization through a differentiable straight-through estimator and evaluates Qwen2.5-Omni-7B. It is cited but absent from Tables 1–3 without a clear explanation. This omission matters because the paper's contribution is partly justified by a contrast between output-oriented learning and heuristic selection. The current results do not establish the advantage over a directly relevant learned alternative.
  2. The distinctive value of topic-guided supervision is not isolated. The inherited EvoComp framework already supplies evolutionary labeling, vocabulary grouping, a lightweight compressor, and GHM. Replacing global grouping with interval-wise grouping changes locality, group structure, and search space together; the DPC-KNN variant also changes selection by using cluster centers. Matched uniform or feature-based temporal partitions are missing. Table 4 therefore does not establish that topic modeling, rather than temporal locality, causes the gain. Moreover, Appendix C.1 uses teacher-generated answers as search targets: minimizing their loss preserves teacher outputs, not necessarily answer correctness. These distinctions matter to both the novelty claim and the proposed explanation of what the labels learn.
Questions for authors
  1. Baseline clarification: Why is OmniSIFT excluded, and can its accuracy/latency trade-off be compared with identical input processing, backbone, and retention budgets?
  2. Decisive ablation: With interval count, group budget, evolutionary evaluations, and compressor training fixed, how does topic segmentation compare with uniform and visual-feature-based boundaries? Does the claimed topic-specific advantage remain?
  3. Supervision clarification: How does using teacher-generated targets instead of original or verified answers affect label quality and downstream correctness? Can the claim be limited to teacher-output preservation if this distinction has not been evaluated?
  4. Practical cost: What are the exact supervision/training-set sizes and total offline GPU costs? These are needed to assess whether the additional machinery is justified relative to simpler learned compression.
2 comments
AuthorsOpenReview ↗

Response to Reviewer tfeF

We sincerely thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the practical accuracy–efficiency benefits of EvoComp-2 and its lightweight, deployment-friendly design. We address your questions and concerns below.


Q1: Why is OmniSIFT excluded, and can its accuracy/latency trade-off be compared with identical input processing, backbone, and retention budgets?

A1: 1) OmniSIFT jointly fine-tunes its audio token selection module VGAS and the Qwen2.5-Omni LLM decoder, whereas all methods in our Tables 1–3 use the original, non-fine-tuned Qwen2.5-Omni backbones. Indeed, the full-token performance differs substantially between the two experimental settings (49.7 vs. 46.8 on WorldSense and 72.2 vs. 64.2 on DailyOmni). Therefore, directly comparing the reported compression results would conflate differences in backbone training with compression effectiveness.

2) To provide a more comprehensive comparison, we directly transferred our trained compressor to the publicly released OmniSIFT-7B model and evaluated it on OmniVideoBench and DailyOmni. The results are summarized below, alongside those reported in OmniSIFT under the same token retention ratios.

MethodOmniVideoBench 35%OmniVideoBench 25%DailyOmni 35%DailyOmni 25%
Full Tokens35.635.672.272.2
Random33.432.666.365.2
DyCoke34.434.167.964.7
OmniZip35.134.167.766.2
OmniSIFT35.635.473.272.5
EvoComp-2 (transferred)35.434.670.869.7

EvoComp-2 consistently outperforms all compared methods except OmniSIFT, despite being transferred to the new backbone without additional training. OmniSIFT achieves higher accuracy through joint optimization of its VGAS module and LLM decoder.

3) EvoComp-2 requires only forward evaluations of the frozen target LLM during label construction, and its compressor is trained independently of the LLM backbone, without backpropagating through it. This decoupled training paradigm offers greater flexibility for deployment with frozen LLMs and potentially better training-memory scalability to larger OmniLLMs.


Q2: Section 2 groups OmniSIFT with methods relying on heuristic signals, but OmniSIFT explicitly uses end-to-end optimization.

A2: We thank the reviewer for pointing out this imprecise characterization. OmniSIFT uses heuristic saliency scores for visual token selection (STVP), followed by learning-based audio token selection (VGAS) conditioned on the retained visual tokens. We acknowledge that broadly categorizing OmniSIFT as heuristic-based was inaccurate and will revise the Related Work accordingly. Unlike OmniSIFT's sequential visual-first and vision-guided audio selection, EvoComp-2 jointly optimizes audio-video token selection.


Q3: How does topic segmentation compare with uniform and visual-feature-based boundaries?

A3: To evaluate different temporal segmentation strategies, we conduct experiments on 1,551 samples from the Video-Holmes training set, directly comparing the inference accuracy of their evolutionary search-generated token masks.

We consider two alternatives: 1) Uniform Segmentation evenly divides the frame sequence into approximately 10% as many intervals as frames, matching EvoComp-2's interval count determined by the 90th percentile of Bhattacharyya distances; and 2) Visual-Feature-Based Segmentation, following FastVID (Shen et al., 2025), computes cosine similarity between consecutive frame representations and identifies temporal boundaries at the seven lowest-similarity positions, as well as any additional positions where the similarity falls below 0.9. All subsequent semantic grouping and evolutionary search procedures remain unchanged.

Segmentation MethodOverall AccuracyTimeline Analysis (TA)
Full Tokens45.0040.59
Uniform Segmentation44.1743.53
Visual-Feature-Based Segmentation43.8442.35
Topic-Guided Segmentation (Ours)44.5545.29

Our method achieves the highest overall accuracy. The advantage is more pronounced on Timeline Analysis (TA), which requires reconstructing the sequence of events in a video. This provides stronger evidence that topic-guided segmentation effectively captures semantic transitions between distinct events.


Q4: How does using teacher-generated targets instead of original or verified answers affect label quality and downstream correctness? Can the claim be limited to teacher-output preservation if this distinction has not been evaluated?

A4: We use responses generated by the target OmniLLM as supervision targets for three reasons.

1) The target OmniLLM was not specifically trained on LongVALE or Video-Holmes, and its outputs (e.g., video descriptions on LongVALE and reasoning traces on Video-Holmes) may differ substantially from the reference annotations even with full audio-video tokens. Using the original annotations could therefore conflate pre-existing differences between model outputs and references with information loss caused by token compression.

2) Our primary objective is to accelerate inference while preserving the original model's output behavior as much as possible. Teacher-generated responses provide a model-consistent supervision signal for this objective.

3) This approach does not require human-annotated reference responses, allowing us to leverage a broader range of training data.

We agree that teacher-output preservation does not necessarily imply answer correctness. We acknowledge that the phrase "correctness of the final OmniLLM output" in the Introduction was imprecise and will revise it to accurately reflect our output-loss-based supervision objective.


Q5: What are the exact supervision/training-set sizes and total offline GPU costs?

A5: 1) We constructed supervision using 17,144 samples from LongVALE and all 3,388 samples from Video-Holmes (1,551 training and 1,837 test samples), totaling 20,532 samples. The offline computational costs are summarized below:

StageGPUsTime
Target Response Generation24 × A1003.2 h
Evolutionary Label Construction24 × A10015.3 h
Topic Model Training1 × A1000.5 h
Compressor Training2 × A10047 h

These costs are incurred offline and do not affect inference latency.

2) Unlike training-free methods, our learning-based approach can leverage additional task-specific training data to improve its adaptation to target tasks.

3) We evaluate the transferability of the compressor in Table 3 and A1. It still achieves strong performance without retraining.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.