acceptodds
Under review as a conference paper at ICLR 2027

Listen First, Look as Needed: Uncertainty-Aware Asymmetric Routing for Audio-Visual Recognition

Abstract

Efficient audio-visual recognition should preserve accuracy while activating vision only when required. Existing static, visual-centric dynamic, and audio-preview methods allocate fixed budgets, route after visual processing begins, or use audio to rank visual segments, but do not calibrate audio-only acceptance before activating vision. We propose Uncertainty-Aware Asymmetric Routing (UAAR), a “listen first, look as needed” framework separating pre-visual acceptance from progressive multimodal verification. An empirical selective-risk criterion governs zero-frame exits, and an entropy-based reliability test disables audio guidance under degradation. Unresolved samples acquire sparse clips and autoregressively accumulate cross-modal evidence under three-, six-, or nine-frame budgets. On Kinetics-Sounds and AVE, UAAR achieves 83.35% and 85.82% accuracy while reducing activated computation by 57.3% and 53.4%, respectively, relative to full nine-frame inference. Under progressive audio corruption, UAAR suppresses unsafe audio-only exits and adaptively requests additional visual evidence. Code will be made publicly available upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Visual Recognition, Multimodal Learning, Uncertainty Quantification, Efficient Inference, Adaptive Computation, Asymmetric Routing, Cross-Modal Fusion, Early Exiting, Dynamic Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8TEJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Uncertainty-Aware Asymmetric Routing (UAAR) for efficient audio-visual event classification. Audio is always processed by a lightweight encoder, and visual clips are encoded only when needed. Inference follows three stages. First, an entropy-based reliability test, which decides whether the audio is unreliable and thus three uniformly spaced visual clips are used. Second, an audio-only exit, whose threshold is selected after training on a held-out calibration split. Third, for the remaining samples, up to three clips are chosen by audio feature saliency and fused sequentially. A recurrent gated fusion module exposes a prediction after each clip and stops once fused entropy falls below a threshold. On Kinetics-Sounds and AVE, UAAR-dynamic reaches 83.35% and 85.82% accuracy. Relative to full nine-frame inference, this reduces compute by 57.3% and 53.4%, at a cost of 1.29 and 1.00 accuracy points.

Critical strengths
  • The motivation is clear and practical. Audio is cheap, vision dominates sensing cost, and deciding whether to look before encoding any frame is a sensible design principle.

  • The method is simple, modular, and easy to deploy. Thresholds are set after training, so the operating point can be adjusted without retraining.

  • Under progressive audio corruption, the router behaves sensibly: audio-only exits fall and the visual budget grows.

Critical weaknesses
  • UAAR brings limited novelty as the method is a combination of known components. Each building block of UAAR is established: 1) Confidence-gated early exiting and classifier cascades are standard dynamic-inference techniques. 2) Using audio to choose which visual clips to encode was introduced by SCSampler (Korbar et al., ICCV 2019) and Listen to Look (Gao et al., CVPR 2020). 3) Audio-first modality skipping is already done by HCMS and AdaMML. 4) Gated recurrent fusion with deep supervision at each step is a routine design. The contribution is therefore incremental. The savings come mainly from skipping vision when audio is confident, which is a well-known idea.

  • The acceptance score is the standard maximum-softmax-probability confidence, so the calibration step amounts to selecting a confidence threshold on held-out data. Appendix B confirms that a conventional confidence threshold tuned to the same budget reaches exactly the same operating point (2.1916 GFLOPs, 83.35%). The calibration also misses its own target: with ε = 0.05, the test selective error is 5.78%.

  • The threshold h_deaf = 3.0 is close to the maximum possible entropy (ln 33 ≈ 3.50 on KS). It triggers on only 2.45% of clean samples. Removing it changes accuracy by 0.18 points on clean data and 0.36 points under the strongest corruption, both within one standard error. In the mismatch test (Appendix C), it fails to catch any of the confidently wrong audio exits, all of which are incorrect.

  • I noticed inconsistencies in how the baselines appear in the manuscipt. For example, the main table reports the HCMS configuration with the lowest test accuracy (83.11% at 4.40 GFLOPs). The 0.25-budget configuration in Appendix A reaches 84.03% at 2.50 GFLOPs, more accurate than UAAR-dynamic at similar cost. Also, AdaMML reaches 85.62%, above even the full-compute UAAR-static, but is excluded from the main comparison.

  • I also noticed some inconsistencies in terms of compute figures across the datasets. On AVE, UAAR-dynamic uses fewer frames than on KS but reports higher GFLOPs. UAAR-static also costs 5.94 vs 5.13 GFLOPs with the same architecture. Did I miss something? Furthermore, parameter counts also change across datasets without explanation. What drives this change?

  • I think the paper is also missing some important related works. SCSampler (Korbar et al., ICCV 2019) already uses audio to select salient clips for efficient recognition. Fast yet Safe (Jazbec et al., NeurIPS 2024) and SAFE-KD (Salim Khazem, SAFE-KD: Risk-Controlled Early-Exit Distillation for Vision Backbones, 2026) calibrate early-exit thresholds to a target selective risk with formal finite-sample guarantees, which UAAR’s empirical rule lacks.

  • The "w/o Audio Prior” ablation removes zero-frame exits and saliency-based sampling together, and it achieves higher accuracy (83.90% vs 83.35%).

  • The paper motivates continuous, resource-constrained sensing, but experiments use only trimmed 10-second clips from two small datasets. AVE, a localization benchmark, is repurposed for classification. It would then make sense to test UAAR on larger or long-video benchmarks (e.g., VGGSound, ActivityNet).

Overall, the current version of the manuscript, given all the weaknesses I have reported above, falls short of the ICLR bar. I recommend rejection.

Questions for authors

See also previous section.

  • Sec. 4.1 says h_safe is selected on the calibration set, while Sec. 3.3 and Appendix H say the validation set. Can the authors resolve this inconsistency?

  • Appendix H refers to a “finite-sample-corrected” selective risk, whereas Eq. 3 and Sec. 2 describe an uncorrected empirical criterion. Which one is correct?

Reviewer_PLbkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Uncertainty-Aware Asymmetric Routing (UAAR), an audio-first approach to efficient audio-visual recognition. The method first determines whether an audio-only prediction can be accepted, then progressively processes visual clips when additional evidence is needed. It combines an empirically calibrated audio-only exit, an entropy-based fallback, and progressive fusion with 0/3/6/9-frame budgets. On Kinetics-Sounds and AVE, UAAR achieves 83.35% and 85.82% accuracy, respectively, while reducing computation by 57.3% and 53.4% compared with full nine-frame inference.

Critical strengths
  • The paper addresses a practical problem in efficient audio-visual recognition: deciding whether visual processing is needed at all, rather than simply reducing its cost. The progressive 0/3/6/9-frame design is sensible, and the reported results show a promising trade-off between recognition accuracy and computation.
  • The experimental evaluation includes controlled baseline adaptations, component ablations, and robustness analysis under audio corruption. The authors also provide detailed routing statistics and acknowledge important limitations, including failures under semantic audio-visual mismatch.
Critical weaknesses
  • The methodological novelty appears limited relative to existing work. Audio-driven visual selection was previously explored in Listen to Look (Gao et al., CVPR 2020), while adaptive modality activation was investigated in AdaMML (Panda et al., ICCV 2021) and HCMS (Weng et al., TOMM 2023). Moreover, held-out confidence-based selective prediction is well established (Geifman and El-Yaniv, NeurIPS 2017). Appendix B shows that budget-matched confidence routing achieves exactly the same accuracy and computation as UAAR, making the distinct contribution of the proposed calibration mechanism unclear.
  • The audio-only exit remains vulnerable to confidently incorrect audio predictions. On Kinetics-Sounds, the observed selective error is 5.78%, slightly above the empirical target of 5%. This is not a violation of a theoretical guarantee, since the authors explicitly do not claim one. A more concerning result appears in Appendix C, where clean audio is replaced with audio from a different class. All 467 audio-only exits are incorrect in this setting, and overall accuracy drops to 47.67%. Disabling zero-frame exits improves accuracy to 61.57%, though at a higher computational cost. Admittedly, this is a synthetic stress test rather than a realistic distribution-shift benchmark. Still, it highlights a weakness of relying on audio confidence to decide whether visual evidence can be safely skipped.
  • The ablation studies do not sufficiently isolate the contributions of the proposed routing components. In Table 2, removing the audio prior simultaneously disables audio-only exits and replaces audio-guided visual sampling with uniform sampling, making it difficult to determine which mechanism contributes to the reported improvements. Interestingly, removing the audio prior increases accuracy from 83.35% to 83.90%, albeit with higher computation. Separate budget-matched ablations of audio-guided sampling, zero-frame acceptance, and the two fallback functions would better establish their individual effectiveness.
  • The experimental evidence remains insufficient to establish robust efficiency advantages. All main comparisons rely on single training runs, while efficiency is measured through GFLOPs without actual inference latency, energy consumption, or decoding overhead. Furthermore, the robustness evaluation primarily considers synthetic Gaussian perturbations, without testing realistic background noise, reverberation, or audio-visual misalignment. These limitations make it difficult to assess whether the reported accuracy–efficiency trade-offs generalize to practical deployment.
Questions for authors
  • Appendix B shows that budget-matched confidence routing matches UAAR in both accuracy and computation. What practical benefit does selective-error calibration provide over a validation-tuned confidence threshold? Is there evidence that it produces more stable routing behavior across different data splits or distribution shifts?
  • Could the authors provide budget-matched ablations that separately evaluate audio-only acceptance, audio-guided temporal sampling, the entropy-based acceptance veto, and the uniform-sampling fallback? The current ablations make it difficult to attribute the gains to individual components.
  • Equation (2) defines selective error as the empirical ratio of incorrect accepted predictions, but Appendix H refers to a finite-sample correction. Could the authors clarify the correction used? Also, Section 4.1 states that the Kinetics-Sounds fusion threshold is selected using the calibration set, whereas Appendix H refers to the validation split. Which split was actually used?
Reviewer_EdZyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes UAAR, an audio-first framework for efficient audio-visual recognition. It uses audio confidence and predictive entropy to decide whether to exit without vision or acquire visual evidence. Unresolved samples progressively process three-frame clips, with gated fusion and intermediate stopping under a maximum nine-frame budget. Audio-exit thresholds are selected on held-out data, and high-entropy audio triggers uniform visual sampling. Experiments on Kinetics-Sounds and AVE report reduced computation with some loss in accuracy.

Critical strengths
  1. The goal of avoiding unnecessary visual computation is practically relevant.
  2. The paper reports component ablations, audio-corruption experiments, and an audio-video mismatch stress test.
Critical weaknesses
  1. Confidence and entropy cannot reliably detect confident errors or tell whether audio alone is enough to skip vision; the semantic-mismatch test in Appendix C already shows this. Calibration on the same dataset says little about reliability under distribution shift, so cross-distribution evaluation is needed.
  2. The 3/6/9-frame schedule on ten-second clips is too limited to show generalizability. Nine-frame inference is a weak reference, not a full-video one. The evaluation should include longer videos, larger frame budgets, and fixed-FPS sampling.
  3. Lower GFLOPs don't mean faster inference. Static visual encoding can run in parallel, while progressive acquisition adds sequential steps and routing overhead. The paper should report measured end-to-end latency and throughput against an efficiently batched baseline.
  4. The contribution of each component is not well supported. Budget-matched confidence routing exactly matches UAAR (Table A2). The calibration description is inconsistent about finite-sample correction and data splits (Eqs. 2-3, Sec. 4.1, Appendix H). The ablations do not separate the effects of acceptance, sampling, and fallback (Table 2). The small gains are not checked over multiple training seeds (Appendix J, Table A7), and the dynamic baselines are not well budget-matched (Table A1, Fig. 3).
  5. The drawing and layout of Figure 1 need improvement. The AI Use Statement should be moved out of the main text if allowed, and the space used for more experiments and explanation of the method.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.