- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes AAVS (Adaptive Attribute Distribution and Visual Structure Alignment), a semantic condition learning framework for generative Zero-Shot Learning (ZSL). It aims to address two limitations of standard class-level attributes: the lack of intra-class diversity and structural inconsistency with the visual space. The Adaptive Attribute Distribution (AAD) component captures semantic diversity by predicting attribute variance per dimension, calibrated by inter-class discriminability. Visual Structure Alignment (VSA) projects attributes into the visual space and aligns them with visual prototypes via a prototype alignment loss. The resulting semantic conditions are combined with a generator to synthesize visual features for unseen classes.
Critical strengths
The paper clearly identifies two core limitations of static class-level semantic conditions in Generative Zero-Shot Learning (ZSL): the lack of semantic diversity to capture intra-class variations, and the cross-domain structural inconsistency caused by the semantic-visual gap.
The proposed Adaptive Attribute Distribution (AAD) mechanism learns variation patterns entirely within the shared attribute space, constructing diverse semantic conditions without relying on visual features or instance-level annotations of unseen classes, which shows good logical consistency under strict ZSL settings.
Extensive testing was conducted on three standard benchmark datasets (AWA2, CUB, and SUN), achieving an average harmonic mean (H) of 72.4% in Generalized Zero-Shot Learning (GZSL), demonstrating the engineering effectiveness of the method.
Critical weaknesses
The core technical contribution is relatively thin. The AAD module merely introduces Gaussian distributions to sample attribute variations, while the VSA module relies on conventional Singular Value Decomposition (SVD) to remove dominant directions followed by simple contrastive alignment. This combination of techniques is highly foundational in the fields of feature enhancement and manifold alignment, lacking breakthrough theoretical depth or algorithmic innovation.
The AAD module forcefully assumes that attributes with higher discriminability have smaller intra-class variation variance, utilizing a negative correlation mapping via Equations 3 and 4. This strong assumption is not universally valid in the complex real visual world, as certain attributes serving as key classification criteria may still exhibit massive natural variations among different individuals within the same class. The paper lacks biological justification or rigorous mathematical proof for this mechanism, making it appear as a constructed heuristic.
At a time when generative models have highly evolved, the paper still selects the 2019 f-VAEGAN as the core generator. Although the authors mention the application of Diffusion Models in ZSL, they do not attempt to integrate AAVS with modern diffusion model backbones, nor do they provide in-depth comparisons with the latest frontier models based on Large Language Models or Diffusion Models, significantly weakening the paper's academic competitiveness and temporal relevance.
The VSA module refines prototypes by removing the dominant eigenvector of the seen-class prototype matrix. However, the visual distribution of unseen classes is highly likely to depend on these shared underlying structures that are treated as biases. Overly suppressing the common direction, as shown in the appendix experiments where a high value leads to a drop in unseen class accuracy, could destroy the critical cross-domain transferable knowledge relied upon by zero-shot learning, and the paper fails to theoretically guarantee the safety of this operation for unseen classes.
Questions for authors
Can the authors provide more in-depth theoretical derivations or extensive empirical evidence to prove the rigor and universality of the assumption in Equations 3 and 4—that highly discriminative attributes must have minimal variability—across various fine-grained visual datasets such as CUB?
Given that f-VAEGAN is relatively outdated, if the conditional generation framework of AAVS is seamlessly migrated to current Diffusion Models, can AAD and VSA still independently bring significant performance gains independent of a stronger generator?
When using SVD to extract and eliminate the dominant common direction , how does the model distinguish between dataset-specific biases unique to seen classes and critical cross-domain shared fundamental visual features? How can it be mechanistically ensured that the calculated based on seen classes will not cause destructive compression on the feature space of unseen classes?
13 comments
Response to Reviewer gzPV [Part II]
Dear Reviewer gzPV,
W2: "The AAD module forcefully assumes that attributes with higher discriminability have smaller intra-class variation variance, ... ”
and
Q1: "Can the authors provide more in-depth theoretical derivations or extensive empirical evidence to prove ... that highly discriminative attributes must have minimal variability”
R2: Thank you for raising this important question. We address this concern from the following perspectives:
(1) Clarifying the underlying motivation.
AAD does not assume that attribute discriminability is negatively correlated with the actual intra-class variability of real-world attributes. Our underlying motivation is that modeling intra-class variations can provide more diverse semantic conditions and thereby benefit generative ZSL.
(2) Interpreting the learned attribute variations.
As shown in Figure 3(a) of the paper (p. 8), we analyze the learned variances across attribute dimensions. We observe that attributes with higher discriminability tend to have smaller learned variances, whereas less discriminative attributes exhibit larger variances.
One possible interpretation is that the learned adaptive distributions reflect the role of different attributes in inter-class distinction. More discriminative attributes receive smaller perturbations to preserve their class-related semantics and maintain inter-class distinguishability. In contrast, less discriminative attributes receive larger perturbations to introduce semantic diversity and capture intra-class variations.
(3) Empirical validation of this interpretation.
To further examine this interpretation, we conduct additional comparative experiments using different attribute-discriminability-based calibration strategies. The results show that assigning larger perturbations to more discriminative attributes (Positive) degrades model performance, whereas our proposed negative-correlation calibration strategy achieves better results, as shown in Table R1.
Table R1. Comparison of different attribute perturbation calibration strategies on AWA2, CUB, and SUN. Acc, U, S, H and Avg. H are reported (%).
| AWA2 | CUB | SUN | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Strategy | Acc | U | S | H | Acc | U | S | H | Acc | U | S | H | Avg. H | |
| Uniform | 84.8 | 78.8 | 85.6 | 81.9 | 78.6 | 72.7 | 79.2 | 76.0 | 74.2 | 58.8 | 53.2 | 55.9 | 71.3 | |
| Positive | 80.9 | 77.7 | 81.5 | 79.6 | 74.8 | 67.1 | 80.0 | 73.0 | 73.8 | 60.8 | 50.2 | 55.0 | 69.2 (↓2.1) | |
| Negative (our) | 85.4 | 82.1 | 85.0 | 83.5 | 79.2 | 74.2 | 81.0 | 77.4 | 75.8 | 58.9 | 54.0 | 56.3 | 72.4 (↑1.1) |
This suggests that excessively perturbing highly discriminative attributes may undermine class-related semantics. Restricting their perturbations while allowing larger variations in less discriminative attributes can better balance intra-class semantic diversity and inter-class distinguishability. These results support our interpretation of the learned intra-class variations from the perspective of inter-class discriminability.
Please see Part III for the remaining response.
Response to Reviewer gzPV [Part III]
Dear Reviewer gzPV,
W3: "At a time when generative models have highly evolved, the paper still selects the 2019 f-VAEGAN as the core generator. ... ”
and
Q2: "..., can AAD and VSA still independently bring significant performance gains independent of a stronger generator?”
R3: Thank you for raising this concern. We address it from two aspects.
(1) Reasons for adopting f-VAEGAN as the primary generator.
Our contribution focuses on addressing two fundamental issues in generative ZSL: insufficient intra-class semantic diversity and inconsistent inter-class structures. Accordingly, we propose AAD and VSA as targeted solutions.
We adopt f-VAEGAN as the primary generator to ensure fair comparisons with existing methods and directly validate our proposed solutions. It serves as the common generator for a large proportion of state-of-the-art generative ZSL approaches.
(2) Additional experiments with a diffusion-based generator.
To examine the compatibility of AAVS with stronger generative backbones, we integrate it into DDGAN [1], a diffusion-based generative model, and evaluate it on AWA2, CUB, and SUN. As shown in Table R2, AAVS can also improve the performance of diffusion-based generators, demonstrating its generalizability beyond f-VAEGAN.
Table R2. Evaluation of AAVS with Denoising Diffusion GAN (DDGAN) on AWA2, CUB, and SUN. Acc, U, S, H, and Avg. H are reported (%). Results are averaged over three random seeds and reported as mean ± standard deviation.
| Method | AWA2 | CUB | SUN | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc | U | S | H | Acc | U | S | H | Acc | U | S | H | Avg. H | |
| DDGAN (Base) | |||||||||||||
| Base + VSA | |||||||||||||
| Base + AAD | |||||||||||||
| Base + AAVS | |||||||||||||
| (vs. Base) | +6.97 | +10.90 | +3.77 | +7.93 | +2.33 | +3.93 | +1.10 | +2.57 | +0.77 | +0.50 | +2.23 | +1.53 | +4.02 |
As shown, AAD and VSA independently improve the average H by 3.24% and 1.09%, respectively. Their combination yields a 4.02% gain, from 67.64% to 71.66%. These results demonstrate the effectiveness of both modules with diffusion-based generators and highlight AAVS's potential as a plug-in framework for enhancing diverse generative backbones.
The DDGAN-based framework is briefly introduced below. It consists of a generator , a feature discriminator , and a transition discriminator . DDGAN models the reverse diffusion process using conditional GANs rather than a Gaussian approximation. Given a noisy feature , timestep , a semantic condition , and random noise , the generator predicts a clean feature, from which a reverse transition is sampled:
DDGAN employs two discriminators with complementary roles: the feature discriminator distinguishes real clean features from generated features ; the transition discriminator distinguishes real reverse transitions from generated ones , encouraging realistic denoising transitions. Both discriminators are conditioned on the semantic condition . During inference, visual features are synthesized from Gaussian noise through successive reverse transitions conditioned on .
Revision Location: Revisions appear in Section 4.8 (p. 10), Appendix A.1 (p. 14), Table 4 (p. 10).
Please see Part IV for the remaining response.
Reference:
[R3] Z. Xiao, K. Kreis, and A. Vahdat. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. International Conference on Learning Representations (ICLR), 2022.
Response to Reviewer gzPV [Part I]
Dear Reviewer gzPV,
Thank you for your detailed review. We appreciate your recognizing our motivation, the design of AAD, and the empirical effectiveness of AAVS.
We understand your main concerns regarding (1) the technical contributions of our AAD and VSA modules; (2) the justification and generality of discriminability-guided variation calibration; (3) whether migrating AAVS to diffusion models would yield further gains; (4) whether prototype refinement removes information needed for unseen-class transfer. We address these concerns below.
R1: Thank you for your question. We would like to clarify our contributions in two aspects:
(1) A sound and generalizable motivation.
Our motivation is that improving intra-class semantic diversity and inter-class structural consistency can lead to better semantic conditions for generative ZSL. We systematically analyze these two aspects, demonstrate their importance for reliable feature generation, and provide empirical evidence supporting the soundness and generalizability of this motivation.
Building on this understanding, we develop targeted methods to effectively leverage these properties and validate their effectiveness through extensive experiments. We believe that this work provides a valuable perspective on understanding and improving semantic conditions in generative ZSL, with the potential to inspire future research in this direction.
(2) An innovative methodological approach.
Although the resulting framework is relatively simple, realizing our motivation requires overcoming several challenging issues. In particular, it is challenging to effectively leverage intra-class semantic diversity and inter-class structural consistency while ensuring generalization to unseen classes. Previous methods have attempted to introduce instance-level semantic enhancement using visual features [R1, R2]. However, such semantic enhancement cannot be applied to unseen classes without visual features, limiting its applicability in zero-shot learning.
We find that improving intra-class diversity and inter-class structural consistency entirely within the shared semantic space enables natural generalization to unseen classes. Based on this insight, we design AAD to model attribute distributions and VSA to incorporate visual inter-class structures. This coherent design provides an innovative methodological perspective for generative ZSL.
Reference:
[R1] S. Chen, W. Hou, Z. Hong, et al. Evolving Semantic Prototype Improves Generative Zero-Shot Learning. Proceedings of the International Conference on Machine Learning (ICML), pp. 4611–4622, 2023.
[R2] W. Hou, S. Chen, S. Chen, et al. Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23627–23637, 2024.