acceptodds
Under review as a conference paper at ICLR 2027

Adversarial Dependence Minimization

Abstract

Minimally redundant representations are typically learned by minimizing feature covariance. However, covariance-based methods fail to eliminate all dependencies/redundancies, as linearly uncorrelated variables can still exhibit nonlinear relationships. To address this, we introduce ADM, a differentiable algorithm that minimizes statistical dependence between feature dimensions through an adversarial game: auxiliary networks identify dependencies, while the encoder removes them. We prove that mutual independence is achieved at the global optimum, empirically verify convergence, and study three potential applications: extending PCA to nonlinear decorrelation, improving generalization in image classification, and preventing dimensional collapse in self-supervised learning. By promoting statistically independent representations, ADM paves the way for learning more robust, compressed, and generalizable representations across diverse applications.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, statistical independence, decorrelation, redundancy, PCA, ICA, SSL, generalization, factorial

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GCX4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an adversarial framework for learning mutually independent representations by minimizing nonlinear dependence among features.

Critical strengths

The main strength of this paper lies in its novel connection between statistical independence and representation learning through an adversarial framework. By formulating dependence minimization as a predictive game between an encoder and adversarial networks, the authors provide an intuitive and principled approach to learning mutually independent representations. The formulation is conceptually appealing and offers a flexible mechanism for incorporating nonlinear dependence minimization into existing representation learning objectives.

Critical weaknesses

While the proposed method effectively reduces nonlinear dependence, the paper does not convincingly demonstrate why such independence is desirable for representation learning. In particular, improved independence does not necessarily translate into better downstream performance, as illustrated by the ImageNet experiments.

Questions for authors

The authors attribute the relatively low linear probing accuracy of ADM on ImageNet to a potential compression-accessibility tradeoff, suggesting that semantic information may remain encoded in the representations but require more expressive nonlinear decoders. Could the authors provide nonlinear probing results to support this explanation?

1 comment

Thank you for recognizing ADM as a principled and flexible way to add nonlinear dependence minimization to existing objectives. As suggested, we ran nonlinear probing on ImageNet.

(W1)

While the proposed method effectively reduces nonlinear dependence, the paper does not convincingly demonstrate why such independence is desirable for representation learning. In particular, improved independence does not necessarily translate into better downstream performance, as illustrated by the ImageNet experiments.

We agree that independence is not desirable for every task, and studying when it could help is part of our contribution. ADM provably targets mutual independence, and we study it on tasks where it helps (Clevr-4, PICA) and where it does not (SSL). On Clevr-4, ADM raises kNN accuracy on taxonomies not used for training from 25.0 to 81.8 (texture) and from 16.4 to 98.9 (color), with shape unchanged (99.9), and PICA recovers independent factors (App. A).

Our SSL results are negative for ADM, but we observe an inverse relation between dependence minimization and linear accuracy that we think is insightful for the SSL community. Indeed, DINO outperforms Barlow Twins despite having more dependent features, and Barlow Twins outperforms ADM despite having more redundant ones. This contradicts the common assumption that less redundant SSL representations perform better, and suggests that view alignment matters more. We therefore do not recommend ADM for SSL and will present this section accordingly (see also our responses to Reviewers snsg and S9ZS).

(Q1)

The authors attribute the relatively low linear probing accuracy of ADM on ImageNet to a potential compression-accessibility tradeoff, suggesting that semantic information may remain encoded in the representations but require more expressive nonlinear decoders. Could the authors provide nonlinear probing results to support this explanation?

Following the reviewer’s suggestion, we trained a two-layer MLP probe on frozen ADM features, using the same augmentations and regularization as in the linear evaluation. After 100 epochs, it reaches 67.8% top-1 (vs. 65.3% linear) with no sign of overfitting yet. Some semantic information is thus only accessible nonlinearly, but accuracy remains below the linear accuracy of Barlow Twins and VICReg (73.2%), suggesting that invariance and dependence minimization objectives may also conflict. We will add these results and update the discussion.

Reviewer_S9ZSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an algorithm that minimizes statistical dependence between feature dimensions, called ADM. The method pitches an adversarial game with a minimax solution that must have mutually statistically independent features in an embedding z. They demonstrate applications like PCA with nonlinear (de)correlation, redundancy minimization, and dimensional collapse in SSL.

Critical strengths
  • The problem is crisply written, well-motivated, and easy to follow, and the examples make the formulation concrete.
  • The ADM regularizer is simple to implement, which makes the approach practical to adopt, and the range of motivating applications strengthens the case for its broader relevance; the differentiability and scalability makes it very compelling.
Critical weaknesses
  • Table 3 does demonstrate that DINO has the highest accuracy, but these results indicate that more dependence corresponds to better accuracy (which DINO leads in both regards). The goal of reducing dCor^2 is technically auxiliary to accuracy in this example.
  • Redundancy minimization appears closely related to out-of-distribution generalization, particularly under spurious correlations [1,2]. The current dataset in 6.2 is reasonable, but there are established benchmarks built specifically around spurious correlations and shifted test distributions, such as Waterbirds, CelebA, or Colored MNIST. Evaluating on one or more of these would strengthen the paper’s claims and make it easier to position against existing OOD work. [1] https://arxiv.org/abs/2006.07710 [2] https://arxiv.org/pdf/1911.08731
Questions for authors
  • In app I.3 "We trained two-layer dependency predictors with no probes." The probes appear to be the key component distinguishing ADM from Predictability Minimization. Could you clarify how the setting for this experiment relates to the paper's main results incl Theorem 4.1 which assumes standardized probes?
1 comment

Thank you for recognizing the broad relevance of ADM.

(W1 and Q1)

these results indicate that more dependence corresponds to better accuracy (which DINO leads in both regards). The goal of reducing dCor^2 is technically auxiliary to accuracy in this example.

In app I.3 "We trained two-layer dependency predictors with no probes." The probes appear to be the key component distinguishing ADM from Predictability Minimization. Could you clarify how the setting for this experiment relates to the paper's main results incl Theorem 4.1 which assumes standardized probes?

This is a good observation. We agree that dCor2dCor^2 is only a proxy for accuracy in SSL, and we indeed observe an inverse relation between dependence minimization and linear accuracy, which we find an interesting insight for the community, as it is commonly assumed that reducing redundancy further would boost performance. Our results instead suggest invariance (i.e., view alignment) is the key factor.

Without probes, ADM reduces to predictability minimization: the encoder minimizes ∑iVar(E[z~i∣z−i])\sum_i \mathrm{Var}(\mathbb E[\tilde z_i\mid z_{-i}]), which is zero IFF each ziz_i is mean-independent of z−iz_{-i}. Theorem 4.1 therefore does not provide guarantees for this experiment, although dCor2dCor^2 still dropped to 0.022 (vs. 0.128 and 0.130 for Barlow Twins and VICReg).

A variant with probes did not improve downstream accuracy either, so we kept the simplest formulation. We will clarify this in the paper and present SSL more clearly as a negative result.

(W2)

Redundancy minimization appears closely related to out-of-distribution generalization, particularly under spurious correlations [1,2]. (…) Evaluating on one or more of these would strengthen the paper’s claims and make it easier to position against existing OOD work.

Thank you for this great suggestion. Redundancy and spurious correlations are indeed related, but ADM as formulated would not help on these benchmarks: during training, core and spurious attributes are strongly dependent, since both predict the label. ADM would thus penalize encoding both cues, since this duplicates the label information. As it favours neither, the simplicity bias [1] would make it keep the spurious one, as most methods do.

But ADM could be adapted by making two changes: (1) Conditioning on the label by providing it to predictors and probes, and by standardizing per class. Then the inner objective would become Ey[mCor2(zi,z−i∣y)]E_y [\mathrm{mCor}^2(z_i, z_{-i} \mid y)], which is zero IFF the features are independent given the label yy (Theorem 4.1 applied per class). And (2) splitting the representation into groups, each with its own head and classification loss, and applying ADM between groups, so that the groups are independent given yy and each group predicts the class.

With those two changes, a cue that is not in one-to-one correspondence with the label (e.g., the background in Waterbirds) can be used by only one group, forcing others to encode different evidence (e.g., the bird). Indeed, in this case, the core and spurious attributes are independent given the label by construction, so one head per cue would be an exact solution. Group labels would then be needed only on validation data to select the robust head, rather than during training as in [2]. Since this requires a modified formulation, new theory, and tuning, we believe it deserves its own paper, and we will describe it in the “Open Questions and Future Directions” appendix as guidance for researchers with more expertise in this subject.

Reviewer_snsgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Adversarial Dependence Minimization (ADM) to yield minimally redundant, independent representations. This is accomplished through an adversarial game using auxiliary networks. The encoder is thereby encouraged to make representations independent. The authors use a simple example x1 = unif[-a, a] and x2 =x1^2 to illustrate that nonlinear relationships are not captured by measures of linear dependence. The authors explore 3 applications: nonlinear pca, nonlinear decorrelation, and learning features that generalize beyond label supervision.

The paper effectively argues that the dual total correlation (DTC) must be zero in order for variables to be independent. This is realized by their proposed architecture. Predictor networks each reconstruct missing values of a latent variable given knowledge of the rest of the latent variable this is done with an MSE objective. The encoder is encouraged to maximize this prediction error, yielding disentanglement.

The authors extend their method to PCA to extract nonlinearly independent variables. They demonstrate its success with a simple 3-variable example. They also apply their method to classifier generalization and SSL.

Their experiments with measuring dCor on ResNet18 (compared with linear decorrelation) suggest that the method produces independent representations. The paper indicates the method can boost generalization of classifiers in a limited setting, and that it can help prevent representation collapse in SSL.

Critical strengths

This paper is very clearly written and presents a genuinely interesting idea.

There is rich reproducibility information

The method appears promising for disentanglement; it would be nice to see some comparisons with other methods.

Critical weaknesses

The method appears very similar to NashAE [1]. The authors should clarify the distinction and include a discussion in the paper.

The experiments are scant and not very convincing in terms of showcasing the practical use of the method. For example, the method appears to be performing disentanglement, and there are many strong baselines such as beta-TCVAE, FactorVAE, beta-VAE, etc. The disentanglement performance of the method could be compared with that of these methods. Moreover, these traditional disentanglement methods could be used as baselines in the generalization and SSL experiments, but they are not.

The paper requires an ablation study on the choices of the method, particularly use of the "probe" networks and the mean/variance regularization strengths. There should also be an SSL ablation where only the mean/variance regularizations are used, no decorrelation.

[1] Yeats, Eric, et al. "Nashae: Disentangling representations through adversarial covariance minimization." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

Questions for authors

Could the SSL benefit be due to the fact that ADM explicitly controls the variance of the latent features? This should be “disentangled” from the independence promotion effect of ADM. An ablation study is needed.

If the downstream performance of the SSL-learned features is worse than Barlow Twins (the most similar approach as indicated in the paper), why do ADM? Why should one value having decorrelated features in this case?

How is this method different from NashAE [1]? It also disentangles using predictor networks and either an MSE or covariance minimization objective.

[1] Yeats, Eric, et al. "Nashae: Disentangling representations through adversarial covariance minimization." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

2 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you for the kind words on the writing style and for showing interest in the idea. Below, we address the reviewer's concerns one by one.

(Q1 / W3.b)

There should also be an SSL ablation where only the mean/variance regularizations are used, no decorrelation. Could the SSL benefit be due to the fact that ADM explicitly controls the variance of the latent features? This should be “disentangled” from the independence promotion effect of ADM.

Great intuition!

In self-supervised learning (SSL), VICReg already provided the result of using only a standard deviation regularization (in its Table 4): this variant achieves around 57.5% linear evaluation accuracy on ImageNet, which is already high but significantly lower than with the decorrelation term.

However, if the reviewer meant that we should “disentangle” the effect in the classifier generalization setting, this concern is also valid: the classifier could have benefited from maintaining the standard deviation above a given threshold without requiring ADM. We provide the ablation results hereunder:

shapetexturecolorcountdCor^2
Classifier100.025.016.436.10.409
Classifier + ADM99.981.898.936.00.038
Classifier + std. deviation reg. (only)100.020.813.935.90.408

The table shows that the standard deviation term alone leads to no performance improvement compared to the baseline. Standard deviation regularization is thus not sufficient, and the improved generalization is due to ADM’s redundancy minimization effect. We will add this result to the appendix.

(Q3 / W1)

The method appears very similar to NashAE [1]. The authors should clarify the distinction and include a discussion in the paper. How is this method different from NashAE [1]? It also disentangles using predictor networks and either an MSE or covariance minimization objective.

Thank you for sharing this work. We will discuss NashAE in the related work.

Its predictors minimize an MSE, and its encoder minimizes the covariance between each latent variable and its prediction. With optimal predictors, this covariance equals Var⁡(E[zi∣z−i])\operatorname{Var}\left(\mathbb{E}\left[z_i \mid z_{-i}\right]\right), which vanishes as soon as each ziz_i is mean-independent of the others. NashAE thus follows Predictability Minimization (Schmidhuber, 1992) and can at best reach mean independence. For example, for zz uniform on the unit circle, E[z2∣z1]=0\mathbb{E}\left[z_2 \mid z_1\right]=0, so NashAE's predictors detect nothing, whereas ADM's probe ψ(z2)=z22\psi\left(z_2\right)=z_2^2 is predicted exactly by 1−z121-z_1^2. Probes combined with standardization are what make the optimum be mutual independence (Theorem 4.1).

Besides probes, our paper also formally proves the connection to independence and provides an implementation that scales to hundreds of embedding dimensions.

Schmidhuber, Jürgen. "Learning factorial codes by predictability minimization." Neural computation 4.6 (1992): 863-879.

AuthorsOpenReview ↗

Rebuttal Part 2

(W2)

The experiments are scant and not very convincing in terms of showcasing the practical use of the method. For example, the method appears to be performing disentanglement, and there are many strong baselines such as beta-TCVAE, FactorVAE, beta-VAE, etc. The disentanglement performance of the method could be compared with that of these methods. Moreover, these traditional disentanglement methods could be used as baselines in the generalization and SSL experiments, but they are not.

β\beta-TCVAE and FactorVAE are indeed the closest baselines, since they penalize total correlation, i.e., mutual dependence. Both, however, are tied to the VAE framework. β\beta -TCVAE estimates total correlation from the Gaussian posterior q(z∣x)q(z \mid x), and β\beta-VAE acts through the KL divergence to the prior. Neither therefore applies to the deterministic encoders without decoders and used in our classification and SSL experiments. For the latter, models are trained at ImageNet scale with a ResNet-50, where we are not aware of any disentanglement autoencoder being applied. Only FactorVAE's estimator transfers: a discriminator between samples of q(z)q(z) and dimension-wise permuted samples, i.e., the approach of Brakel & Bengio (2017), whose instability we discuss in the related work. We will add it as a baseline on Clevr-4.

We purposely did not evaluate disentanglement: disentangled factors may be dependent, which makes the problem ill-defined (Locatello et al., 2019) and different from independence, our objective. ADM nonetheless contains Predictability Minimization as a special case (identity probes), so it can recover what NashAE recovers.

Our contribution is a method that provably targets mutual independence and scales to deep encoders, with applications showing where independence helps and where it does not (SSL). PICA and NLPICA (App. A.1, A.2) recover independent latent factors, which decorrelation or Predicability Minimization alone cannot guarantee. On Clevr-4, ADM raises the kNN accuracy on taxonomies not used for training from 25.0 to 81.8 (texture) and from 16.4 to 98.9 (color), while keeping shape at 99.9, which is significant. We will move the PICA results to the main paper.

(W3.a)

The paper requires an ablation study on the choices of the method, particularly use of the "probe" networks and the mean/variance regularization strengths.

Probe networks are required by theory to achieve full statistical independence, and in general, discarding them could lead to the encoder fooling the predictors despite dependence (see the answer to Weakness 1). We observed this limitation empirically: discarding probes when training on the PCA generalization is not sufficient to recover the product distribution found in Appendix A.1.

However, in practice, mean independence may be sufficient for some other applications, in which case probes can be discarded. Our results on SSL are an example where even mean independence already seems to conflict with the invariance loss term (as discussed in the answer to Question 2).

Regarding the standard deviation hinge loss coefficient, we found the exact hyperparameter value to have minimal impact as long as the coefficient is high enough (so that the encoder can not cheat). Here are the results for the classifier generalization setting on Clevr-4:

CoefficientAvg KNN accuracy (over the 4 taxonomies)dCor^2
0.25numerically unstable/
0.591.20.036
1.093.20.035
2.0 (default)93.50.038
4.093.60.035
8.093.60.041

All values above 0.5 perform similarly, all within random-seed variation levels. We will add this analysis to Appendix F.

(Q2)

If the downstream performance of the SSL-learned features is worse than Barlow Twins, why do ADM? Why should one value having decorrelated features in this case?

This is a very relevant question.

The results suggest that minimizing redundancy actually hurts performance in self-supervised learning, thus going against the common belief that optimal SSL representations should be minimally redundant. (As a side note, we observe that even BarlowTwins and VICReg do not push the linear correlation coefficient too high in practice, to avoid compromising invariance.)

Indeed, the highest-performing method, DINO, has higher accuracy than BarlowTwins despite a larger correlation. Similarly, BarlowTwins outperforms ADM despite its representations being more redundant. Therefore, ADM may not be the right method to use for SSL.

While discarding this application from the paper could have made its publication easier, we deliberately kept it, as we believe these observations provide valuable insights to the SSL community.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal and planned changes

We thank the reviewers for their personal and constructive feedback. Reviewers found ADM novel and principled (GCX4), a genuinely interesting idea (snsg), and simple to implement, differentiable, and scalable (S9ZS). The main shared concern is the SSL application, where ADM does not reach the accuracy of existing methods.

Our main contribution is the introduction of a differentiable and scalable adversarial game whose optimum is provably mutual independence (Theorem 4.1), which linear decorrelation and Predictability Minimization (including NashAE) do not guarantee. The applications are a first study of where independence helps and where it does not. ADM recovers independent factors in PICA and NLPICA, and improves transfer to taxonomies not used for training on Clevr-4 (kNN accuracy from 25.0 to 81.8 on texture and from 16.4 to 98.9 on color). In SSL, the results are negative but insightful for the community: as Reviewer S9ZS pointed out, we observe an inverse relation between dependence minimization and linear accuracy, against the common assumption that less redundant representations perform better.

The discussion with Reviewer S9ZS also showed how ADM can be adapted beyond the settings studied here. Conditioning the probes and predictors on the label would make the class-conditional density factorize at the optimum, which could be used for out-of-distribution detection. Applying ADM between groups of dimensions with separate classifier heads would further push each head to rely on different evidence, which could make ADM applicable to spurious-correlation benchmarks such as Waterbirds. We see this paper as a first step, and we believe ADM can be adapted to many more settings.

We will update the paper as follows:

  • reframe the applications as a first study of where independence helps and where it does not, and present SSL as a negative result, with the MLP-probing results and a clarification of the setting without probes
  • move PICA and NLPICA to the main text
  • discuss NashAE in the related work
  • add the hinge ablations to the appendix, and FactorVAE’s discriminator as a baseline on Clevr-4
  • describe the conditional and group-wise variants in the “Open Questions and Future Directions” appendix

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.