acceptodds
Under review as a conference paper at ICLR 2027

Cue-to-Context Subsumption Learning for Audio-Visual Generalized Zero-Shot Learning

Abstract

Audio-Visual Generalized Zero-Shot Learning (AV-GZSL) aims to recognize seen and unseen events by aligning audio-visual samples with textual categories. Although multimodal inputs provide rich information, the representations of full samples are often affected by changing backgrounds and surrounding contexts, making shared event cues difficult to capture across scenes. Existing alignment objectives specify which textual category a full sample should match, but do not explicitly indicate which audio and visual cues should support that match. We observe that recognizable localized cues can recur across different contexts, while each full sample presents these cues under a particular context. Based on this observation, we propose Cue-to-Context Subsumption Learning (CCSL). CCSL first extracts localized visual regions and auditory segments as box inputs. It then maps the box and full inputs into Gaussian distributions using a shared distribution mapper. A directional subsumption objective relates their means and variances, guiding the full representation with localized event cues while retaining information from the complete context. Experiments on VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL demonstrate consistent improvements over strong baselines. CCSL improves HM by 5.75 to 7.84 percentage points over the strongest baselines across the three benchmarks and achieves 30.63% unseen accuracy on VGGSound-GZSL.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Generalized Zero-Shot Learning, Audio-Visual Learning, Subsumption Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RX9POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Cue-to-Context Subsumption Learning for Audio-Visual Generalized Zero-Shot Learning. The method aims to reduce the influence of varying backgrounds and contexts by using localized audio-visual event cues to guide full-sample representations. Specifically, CCSL extracts visual regions and auditory segments as localized box inputs and maps both box and full inputs into Gaussian distributions through a shared distribution mapper. A directional KL-based subsumption objective is introduced to encourage mean alignment and constrain their relative variances. The framework also incorporates category classification, box-to-text alignment, and variance regularization losses. Experiments on VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL demonstrate consistent improvements over the compared baselines, with harmonic mean gains of 5.75–7.84 percentage points under CLIP and CLAP features. Ablation studies further examine the contributions of localized boxes and individual learning objectives.

Critical strengths
  1. Consistent and substantial empirical improvements. The proposed method achieves notable improvements in harmonic mean accuracy across three AV-GZSL benchmarks, particularly for unseen categories. The gains are also observed under different feature configurations, suggesting that the approach is not limited to a specific pretrained backbone. I consider this consistent empirical improvement the strongest reason to support acceptance.
  2. Intuitive motivation and meaningful experimental validation. The idea of using localized event cues to guide full-sample representations is intuitive and addresses a relevant challenge in AV-GZSL. The authors also provide useful ablation studies showing that simply adding localized boxes as training samples or replacing the proposed objective with MSE alignment does not yield comparable performance. These results provide evidence that explicitly modeling the relationship between localized and full representations is beneficial. Additionally, box extraction is only required during training, which avoids introducing an extra box-extraction step at inference time.
Critical weaknesses
  1. Limited justification of the proposed subsumption relationship. Although the directional KL-based objective enforces a specific relationship between the means and variances of the box and full distributions, the connection between this constraint and the claimed semantic subsumption remains insufficiently justified. In particular, maintaining a relative variance ratio does not necessarily guarantee that the box distribution captures a meaningful range of contextual variations. A more detailed explanation of the variance behavior would strengthen the methodological contribution.
  2. Insufficient direct validation of context robustness. The paper is motivated by the challenge of background and context variations, but the experiments mainly focus on standard benchmark performance. A direct evaluation under controlled background changes would provide stronger evidence for the central motivation. Furthermore, comparisons with simpler objectives, such as mean-only alignment or an unscaled KL loss, would help clarify whether the improvements specifically arise from the proposed directional subsumption formulation.These limitations reduce my confidence in the interpretation of the proposed mechanism, although they do not outweigh the consistent empirical improvements reported in the paper.
Questions for authors
  1. Could the authors provide additional comparisons with simpler alternatives, such as mean-only alignment between box and full representations, an unscaled KL objective, or reverse KL divergence? These comparisons would help clarify whether the performance gains primarily come from the proposed directional variance constraint or from more general localized feature alignment.
  2. The proposed objective encourages a specific variance ratio between the box and full distributions. However, when the means are aligned and the target variance ratio is satisfied, both variances could potentially shrink together without increasing the subsumption loss. Could the authors clarify how this behavior is avoided in practice and whether the learned variances capture meaningful contextual variations?
  3. Since the main motivation is to reduce the influence of varying backgrounds and contexts, could the authors provide additional evidence of robustness under background changes, such as controlled context-shift experiments or performance analysis across different background conditions? Such results would help establish a clearer connection between the proposed mechanism and its intended benefit.
Reviewer_PNRnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Cue-to-Context Subsumption Learning (CCSL) for audio-visual generalized zero-shot classification. Foundation models extract localized visual and auditory cues, which guide full-sample representations through a Gaussian, variance-scaled KL objective. Experiments report improvements on three established benchmarks with several feature configurations.

Critical strengths
  • The paper is well-written by humans with consistent benchmark gains. CCSL improves HM across the evaluated datasets and feature configurations, with substantial gains in unseen-category recognition.
  • The extensive experiments and ablations examine cue modalities, box augmentation, and training objectives, providing evidence that the proposed combination is effective within the evaluated setting.
Critical weaknesses
  • Insufficiently established relevance to contemporary approaches. The evaluation focuses on established video-classification benchmarks without comparing against audio-video-capable multimodal large language models or demonstrating a compelling accuracy–cost advantage. Such models, such as [1], have already been evaluated for VGGSound recognition, albeit under different protocols. Recent specialized methods such as AHSE [2] are also cited but omitted from Table 1. Consequently, the practical need for the proposed pipeline remains insufficiently justified.
  • The methodological advance is not sufficiently distinguished from existing alignment approaches. The cue-to-context hierarchy is closely related to prior compositional entailment work [3]. The main technical addition is a variance-scaled Gaussian KL objective, but comparisons with MSE do not establish whether the gains specifically require directional subsumption rather than ordinary distribution alignment.
  • The broader generalization claim lacks direct evidence. The benchmarks demonstrate recognition of held-out categories, but do not directly test the background and context changes motivating the method. It therefore remains unclear whether CCSL learns more context-robust representations or primarily improves discrimination within the existing benchmark protocols.

[1] Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation

[2] On Aligning Hierarchical Standardized Embedding for Audio-visual Generalized Zero-shot Learning.

[3] Compositional Entailment Learning for Hyperbolic Vision-Language Models. ICLR 2025

Questions for authors
  • Under the same candidate-label set and evaluation protocol, how does CCSL compare with a representative audio-video multimodal large language model and with AHSE? What measurable accuracy or computational advantage justifies the additional cue-extraction and training stages?
  • With the same cues and architecture, how does the proposed objective compare with ordinary KL alignment without variance scaling and with reversed KL direction? What evidence specifically supports the subsumption interpretation?
  • Under controlled changes to visual backgrounds or background audio that preserve the target event, does CCSL retain its advantage over the global-only baseline? Such evidence would directly test the paper’s central motivation.
Reviewer_KMXaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a cue-to-context perspective for audio-visual generalized zero-shot learning (AV-GZSL), in which localized audio and visual cues provide references for learning representations of full samples across different contexts while preserving contextual information. Based on this perspective, the authors propose Cue-to-Context Subsumption Learning (CCSL), which represents localized cues and full samples as Gaussian distributions and introduces a subsumption objective to align their means and constrain the variance of the full-sample distribution relative to that of the cue distribution.

Critical strengths

The paper investigates audio-visual generalized zero-shot learning from the perspectives of localized cues and distribution alignment, offering a potentially valuable approach to improving generalization to unseen classes.

Critical weaknesses
  1. Insufficient comparison with recent methods. The evaluation includes too few state-of-the-art methods published in 2026, making it difficult to convincingly establish the superiority of the proposed approach over the latest advances.

  2. Unaccounted computational overhead. The extraction of localized cues relies on multiple pretrained models, introducing additional computational costs. These costs should be quantified and taken into account when evaluating the overall efficiency and practicality of the proposed method.

Questions for authors

Please refer to the weaknesses listed above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.