- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an adversarial framework for learning mutually independent representations by minimizing nonlinear dependence among features.
Critical strengths
The main strength of this paper lies in its novel connection between statistical independence and representation learning through an adversarial framework. By formulating dependence minimization as a predictive game between an encoder and adversarial networks, the authors provide an intuitive and principled approach to learning mutually independent representations. The formulation is conceptually appealing and offers a flexible mechanism for incorporating nonlinear dependence minimization into existing representation learning objectives.
Critical weaknesses
While the proposed method effectively reduces nonlinear dependence, the paper does not convincingly demonstrate why such independence is desirable for representation learning. In particular, improved independence does not necessarily translate into better downstream performance, as illustrated by the ImageNet experiments.
Questions for authors
The authors attribute the relatively low linear probing accuracy of ADM on ImageNet to a potential compression-accessibility tradeoff, suggesting that semantic information may remain encoded in the representations but require more expressive nonlinear decoders. Could the authors provide nonlinear probing results to support this explanation?
Thank you for recognizing ADM as a principled and flexible way to add nonlinear dependence minimization to existing objectives. As suggested, we ran nonlinear probing on ImageNet.
We agree that independence is not desirable for every task, and studying when it could help is part of our contribution. ADM provably targets mutual independence, and we study it on tasks where it helps (Clevr-4, PICA) and where it does not (SSL). On Clevr-4, ADM raises kNN accuracy on taxonomies not used for training from 25.0 to 81.8 (texture) and from 16.4 to 98.9 (color), with shape unchanged (99.9), and PICA recovers independent factors (App. A).
Our SSL results are negative for ADM, but we observe an inverse relation between dependence minimization and linear accuracy that we think is insightful for the SSL community. Indeed, DINO outperforms Barlow Twins despite having more dependent features, and Barlow Twins outperforms ADM despite having more redundant ones. This contradicts the common assumption that less redundant SSL representations perform better, and suggests that view alignment matters more. We therefore do not recommend ADM for SSL and will present this section accordingly (see also our responses to Reviewers snsg and S9ZS).
Following the reviewer’s suggestion, we trained a two-layer MLP probe on frozen ADM features, using the same augmentations and regularization as in the linear evaluation. After 100 epochs, it reaches 67.8% top-1 (vs. 65.3% linear) with no sign of overfitting yet. Some semantic information is thus only accessible nonlinearly, but accuracy remains below the linear accuracy of Barlow Twins and VICReg (73.2%), suggesting that invariance and dependence minimization objectives may also conflict. We will add these results and update the discussion.