acceptodds
Withdrawn from ICLR 2027

AudioRouter: Audio-Guided Visual Compression for Streaming Video Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming video language models require efficient visual compression to preserve task-relevant evidence under causal constraints, where future observations are unavailable. Existing approaches mainly rely on visual signals for token selection, which can miss event-related information under aggressive compression. Meanwhile, audio-visual models typically inject audio representations into the LLM, requiring audio-capable architectures. We propose AudioRouter, an audio-guided visual compression framework based on a simple principle: audio determines where to look, while vision determines what the LLM sees. Instead of introducing audio tokens into the LLM, AudioRouter uses synchronized audio to condition learnable queries and generate a routing distribution over native visual tokens. The compressed representation is formulated as , where audio controls the routing weights while the aggregated values remain purely visual. This preserves the standard visual-token interface and enables causal streaming inference. Experiments on long-video and streaming benchmarks show that AudioRouter maintains performance close to the full 196-token representation with only 64 tokens per frame, retaining about one-third of the visual tokens while substantially reducing LLM inference cost. https://anonymous.4open.science/r/AudioRouter-1F96.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Language Models, Audio-Guided Visual Compression, Visual Token Routing, Multimodal Representation Learning, Efficient Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_EcP4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AudioRouter uses synchronized audio to guide visual token compression for streaming video LLMs. Causal audio features and temporal context condition learnable queries. The queries produce routing weights over native visual tokens, yielding Z=PVZ=PV without a separate audio input stream to the LLM. Only the bottleneck is trained, using knowledge distillation and audio-visual alignment on external video data. With 64 tokens per frame, the paper reports 60.19 on VideoMME, 72.40 on StreamingBench RTVU, and 68.4 on MLVU. It also reports lower memory usage and LLM prefill latency than full-token inference.

Critical strengths
  1. Practical Design. AudioRouter uses audio cues while preserving the standard visual-token interface of a frozen VLM. Each audio window ends at the corresponding frame timestamp, making the routing causal. The design offers a practical way to use synchronized audio without an audio-capable LLM.
  2. Measured Efficiency Gains. The bottleneck has about 2.0M trainable parameters. Table 5 reports a reduction in peak memory from 58.22 GB to 29.70 GB and in LLM prefill time from 10.93 s to 1.85 s. The TTFT improvement is smaller, from 23.06 s to 20.88 s, but the memory and prefill savings are substantial.
Critical weaknesses

W1. Unresolved Evaluation Inconsistencies. The reported scores do not match the stated evaluation protocol. The full VideoMME benchmark has 900 questions in each duration group. Under the question-level aggregation in Appendix C, Overall should therefore equal the mean of Short, Medium, and Long, up to rounding. AudioRouter reports 60.19, whereas its three scores average to 59.57. Similar discrepancies occur at other token budgets in Table 6. The official RTVU annotations contain 128 CR questions, which cannot produce the reported 88.89% accuracy in a single evaluation of the complete category. The discrepancies suggest an undisclosed subset, a different aggregation procedure, or reporting errors. They do not establish the actual evaluation size. Until the question sets and scoring procedures are clarified, the claimed gains over Full Vision and prior compressors cannot be reliably assessed.

The training description also limits reproducibility. The manuscript omits the training-set size, the source of KD questions, and key loss and routing hyperparameters. A separate checkpoint is trained for each benchmark, but the validation and checkpoint-selection procedures are not reported. The broken training-data reference in Sec. 4.1 leaves these details unresolved.

W2. Insufficient Evidence For The Contribution Of Audio. Appendix B describes bypassing BEATs for A∅A_{\varnothing} and shifting audio features for At−1A_{t-1}. It does not state whether either control is trained separately. If audio is removed only at inference, the drop from 60.19 to 49.81 measures sensitivity to removing a learned input. The drop does not establish an advantage over a vision-only bottleneck trained with the same data, token budget, and KD objective. A matched training comparison is needed to isolate the contribution of audio conditioning.

The routing analysis provides limited additional support. If Table 3 uses H(pref,pˉ)H(p_{\mathrm{ref}},\bar p) with natural logarithms, uniform routing has CE ln⁡196≈5.28\ln 196\approx5.28, below both reported values of 10.47 and 8.70. The CE definition and a uniform baseline are needed to interpret the improvement. The qualitative evidence also contains inconsistent labels. The same driveway scene is labeled Burglary in Figure 2 and Arrest in Figure 4. Two other repeated surveillance scenes also receive different answers. The mismatches make the examples harder to interpret as evidence of task-relevant routing.

Overall. I recommend rejection because the main comparisons cannot be verified under the stated protocol, and the audio contribution remains insufficiently isolated. Consistent evaluation results and a matched training control could change my assessment.

Questions for authors
  1. Report the evaluated question counts and sample IDs for each benchmark and ablation. Explain the aggregation discrepancies in W1 and identify which baseline results were rerun. Provide AudioRouter and Full Vision results on identical question sets with the same frame sampling and scoring protocol.
  2. Clarify whether A∅A_{\varnothing} and At−1A_{t-1} were trained separately and whether A∅A_{\varnothing} retains temporal context. Compare AudioRouter with a vision-only bottleneck trained on the same data with the same token budget and KD objective.
  3. Specify the training-set size, KD question source, λAV\lambda_{\mathrm{AV}}, routing and InfoNCE temperatures, negative-sampling rules, and queue size. Explain how the benchmark-specific checkpoints and training schedules were selected, including any validation data used.
  4. Define the CE direction and log base in Table 3, report the number of evaluated videos, and include uniform routing as a reference. Resolve the conflicting question-answer labels in Figures 2 and 4.
Reviewer_uJYgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Motivation: Motivated by the need for additional information to guide visual token selection or compression in streaming video understanding, this paper uses audio as the guidance signal.

Methods: The method uses a separate audio stream to encode audio and extract event-relevant information through learnable queries for visual token compression. The VLM is frozen, and only the learnable queries, projection layers, and related modules are trained. Training combines a KL-divergence loss for visual knowledge preservation with a contrastive objective for audio-visual alignment.

Contributions: The paper design an audio-conditioned visual compression architecture for streaming video understanding that performs well with approximately one-third of the full visual-token budget.

Critical strengths
  1. Compatibility: The method uses a seperate audio stream instead of modifying the whole architecture, which preserving the original visual-token interface. It therefore incorporates audio conditioning without adding a separate audio-token stream to the LLM or updating the pretrained VLM parameters.

  2. Computation: The method reduce the token budget from 196 per frame to 64 per frame while maintaining competitive accuracy

Critical weaknesses
  1. Motivation: The motivation for using audio to guide visual compression is not sufficiently convincing. When audio provides little useful information, as in silent videos or scenes with little variation in sound, it offers limited guidance on which visual information should be retained.

  2. Method: The paper presents audio as a guide for selecting and compressing visual information. However, the training objectives appear to encourage the compressed tokens to capture both visual and audio information. This raises the question of whether the performance gains come from better visual selection or from additional audio information encoded in the compressed representation.

  3. Experiments: The streaming evaluation is limited to RTVU, leaving the method insufficiently tested across streaming benchmark. Comparisons with recent methods, such as InfiniPot-V[1] and DSCache[2], are also missing. The current results therefore do not provide sufficient evidence of the method’s effectiveness or its competitiveness with recent approaches.

[1] https://arxiv.org/pdf/2506.15745

[2] https://arxiv.org/pdf/2605.01858v1

Questions for authors
  1. Could the authors clarify why audio can effectively guide visual token compression, given that it contains less information than visual input?
  2. Figure 2 suggests that audio guidance helps the model focus on task-relevant information more effectively than visual input alone. However, it is a bit wierd that audio guidance helps the model identify purely visual attributes, such as texture and color, in the examples shown.
  3. The experiments would benefit from comparisons with recent methods(mentioned in weaknesses), as well as evaluation on OVO-Bench. The current evaluation relies more heavily on offline benchmarks than on streaming benchmarks. Demonstrating the method’s effectiveness across a broader range of streaming settings would make the results more convincing.
Reviewer_vZE6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose AudioRouter, where synchronized audio to condition learnable queries and generate a routing distribution over native visual tokens.The compressed representation is formulated as Zt=PtVt, where audio controls the routing weights while the aggregated values remain native visual values. Experiments on VideoMME, MLVU, and RTVU show competitive video understanding performance with only 64 visual tokens per frame. However, concerns remain regarding technical novelty, audio-visual spatial correspondence, and experimental validation.

Critical strengths
  1. The main idea is reasonable. Using synchronized audio to guide visual evidence selection offers a useful direction beyond compression based solely on visual saliency or token redundancy.
  2. The experiments are promising. The paper evaluates several benchmarks and includes controlled backbone comparisons, audio-conditioning and token-budget ablations, and efficiency measurements.
Critical weaknesses
  1. Novelty issue. AudioRouter modulates learnable queries with audio features, computes attention weights over visual tokens, and aggregates visual features. This formulation closely resembles standard audio-conditioned cross-attention or query-based resampling. The main contribution appears to be incorporating audio conditioning into an existing visual compression paradigm, rather than introducing a fundamentally new compression mechanism. The paper should clarify its advantages over conventional audio-visual cross-attention, Q-Former-style modules, and audio-conditioned resamplers.
  2. Moreover, audio-guided visual selection has been explored in prior work, such as Listen to Look (Gao et al., 2020), while Vista-LLM has investigated query-guided visual token compression. The paper does not sufficiently clarify the fundamental methodological advances beyond prior works. More direct comparisons with these related approaches are needed to substantiate its novelty.
  3. Routing visualizations and improved agreement with the full-vision teacher support the claim that audio changes visual selection. However, the teacher reference is based on correct-answer Gradient×Activation importance, rather than annotated sound-source locations. These results therefore do not directly establish audio-visual spatial grounding or show when audio guidance selects more useful evidence.
  4. The reported reductions in prefill latency (10.93 s to 1.85 s) and peak memory (58.22 GB to 29.70 GB) are substantial. However, Table 5 reports a smaller TTFT improvement, from 23.06 s to 20.88 s. It is supposed to clarify the reason.
Questions for authors

Please refer to the weakness.

Reviewer_Lv2UOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AudioRouter addresses visual compression for streaming video language models, where future observations and questions may be unavailable. Synchronized audio and temporal context condition learnable queries, which route and aggregate native visual tokens through Z=PVZ=PV. This preserves the existing visual-token interface without introducing a separate audio-token stream. The manuscript describes training with teacher-output distillation and audio–visual alignment while freezing the underlying models. Experiments on VideoMME, MLVU, and StreamingBench RTVU report competitive accuracy using 64 rather than 196 tokens per frame. Objective and token-budget ablations, routing visualizations, and efficiency measurements are provided to support the design.

Critical strengths

A clear and practically motivated compression design. Separating audio-conditioned routing from native visual-value aggregation provides a modular way to use synchronized audio without modifying the language-model interface. The inspected implementation supports this structure and freezes the underlying models. Attention to practical efficiency. The paper reports memory consumption, prefill latency, and TTFT separately. This is useful because the substantial prefill improvement in Table 5 is accompanied by a more modest TTFT reduction, allowing readers to distinguish component-level savings from user-facing latency.

Critical weaknesses

Table 2 reports a substantial improvement from 49.81% for vision-only compression to 60.19% with current audio. Establishing that this difference comes specifically from audio is central to the paper’s contribution. However, the inspected vision-only branch also bypasses temporal conditioning and the conditioning MLP, so the implemented comparison changes more than the audio input. The routing analysis offers useful qualitative evidence, but its interpretation also needs clarification. Table 3 establishes improvement relative to the vision-only control without a uniform-routing reference, while Figures 2 and 4 contain inconsistent question–answer labels for three shared examples. Confirmed: These differences exist in the inspected implementation and figures. Still unclear: Whether the submitted ablations used this exact configuration, and whether the figure mismatch is confined to layout or affects the teacher attribution targets. These points limit the current support for the proposed mechanism. 2. The reported experimental protocol and reproduction materials are not yet sufficiently aligned. The manuscript states that training uses external data and reserves the evaluation benchmarks for testing. The repository guide instead documents benchmark-specific training partitions and associates the main VideoMME and RTVU scores with internal validation subsets of 540 and 500 questions. The evaluation scripts apply the corresponding subset-filtering mechanism. These scores are arithmetically consistent with the documented subsets; the concern is their relationship to the manuscript’s protocol and the baseline comparisons. There is also a mismatch between the stated KD-plus-AV objective and the recommended phase4 configuration, which uses answer cross-entropy. Other training branches exist, so the historical configuration producing the reported checkpoints remains uncertain. Although the release provides substantial implementation detail, the supplied archive lacks the actual split files, training manifests, and prediction logs needed to resolve these questions. A clear mapping from each reported result to its configuration and checkpoint would materially improve reproducibility and confidence in the conclusions.

Questions for authors

● Which configurations implement the manuscript’s KD-plus-AV objective and its ablations? Were the “only” variants initialized from QA-trained checkpoints, and did the vision-only comparison retain the same temporal conditioning and training budget? ● Are the inconsistencies between Figures 2 and 4 limited to presentation? Please clarify the teacher attribution targets and how the routing metrics should be interpreted relative to a uniform-routing reference. ● Was the reported RTVU evaluation performed using the inspected path that combines sampling grids across questions before computing temporal features? If so, how is this reconciled with the stated append-only streaming protocol? Please provide the executable configurations and required data-split artifacts for the reported evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.