acceptodds
Under review as a conference paper at ICLR 2027

Variance Fingerprints: Component-Wise Identification of Content Factors by Augmentation Design

Abstract

Augmentation-based self-supervised learning can recover the content shared across views without separating its underlying factors into individual coordinates. We ask whether augmentations can be designed to make those factors identifiable. Our approach allows content factors to fluctuate across augmented views while preserving their mean for each source instance. Each instance receives a fixed, recorded set of augmentation strengths that controls how much its factors fluctuate. When factors respond differently to changes in these strengths, their patterns of variation provide information that distinguishes them. Under suitable decoder and support assumptions, we prove that these differences identify individual factors in a conditional Gaussian model, up to permutation, scaling, and translation. We then develop a learner that uses the recorded strengths to estimate these patterns jointly across instances, enabling learning from few views per instance. Synthetic experiments demonstrate near-perfect factor recovery through nonlinear mixing. Controlled dSprites experiments show improved coordinate alignment using augmentations that target known factors, even without telling the learner which augmentation corresponds to which representation coordinate. These results suggest a route to component identification in self-supervised learning: augmentation design can supply identifying information beyond shared content alone.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
identifiability, self-supervised learning, nonlinear ICA, disentanglement, data augmentation, causal representation learning, content-style separation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NmhnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a new principle of identification in SSL algorithms they term Variance Fingerprints. They observe that if variances of augmentations are changed independently and the ratio of the change is unique then the relevant factor can be identified. This extends existing SSL identification proofs, which typically show recovery up to affine transformations, to recovery up to (block) permutation and scaling. They then introduce a learning algorithm to estimate the proposed model and apply it to synthetic datasets.

Critical strengths

The formal exposition of the paper appears to be coherent.

Critical weaknesses

The theoretical novelty appears limited. The central identification mechanism is the classical observation that independent Gaussian sources become identifiable when their conditional variance profiles vary non-proportionally across an auxiliary variable. This principle is well established in nonstationary ICA and auxiliary-variable nonlinear ICA (e.g., Pham & Cardoso). The proposed variance fingerprints instantiate this mechanism by making the auxiliary variation experimentally controlled. It remains unclear what fundamentally new insight the theoretical result contributes beyond this specialization of existing identification principles.

This particular novelty is also the central weakness of the paper, since it highly limits the usefulness of the setup. The authors propose to employ designed augmentation variances to steer the model to learn specific factors. However, training the model requires access and detailed control over these factors. What then is gained by subsequently recovering these factors through representation learning? Consequently, the experiments that are presented are not convincing in the sense that they use highly simplified setups with strongly controlled dataset augmentations.

Another problem is the writing, which in many places is hard to understand. Many statements are vague and employ unnecessary jargon that make reading the paper challenging. Additionally, many statements are not sufficiently explained, and variables are left unintroduced - already in the introduction, equation 1, what is alpha? This is never defined, leaving the reader to figure out alone what the authors mean to say. For publication a major rewrite of the paper would be required.

Questions for authors

How could this method conceivably be applied to any real-world use case?

Can you present evidence that the algorithm learns useful factors on non-synthetic datasets where the ground truth factors cannot be controlled?

3 comments
AuthorsOpenReview ↗

Author response

Thank you for the review. The paper was very hard to read. We spent considerable effort on restructuring and revising the text. Though it remains challenging in some places, we believe the current version does a much better job of communicating our ideas and findings.

We also agree that the identification mechanism is classical. Our contribution is a way to create the auxiliary variation it needs in SSL, where the data usually supply none, and the paper now presents this more clearly as a proof of concept with known factors.

Responses follow in the order of your comments. Page numbers refer to the revised PDF.

The theoretical novelty appears limited. The central identification mechanism is the classical observation that independent Gaussian sources become identifiable when their conditional variance profiles vary non-proportionally ... (e.g., Pham & Cardoso). ... It remains unclear what fundamentally new insight the theoretical result contributes ...

  • We agree, and the paper now states: "We therefore claim no new identification mechanism." Section 6 cites Matsuoka & Kawamoto and Pham & Cardoso and notes that repeated fingerprints are the nonlinear counterpart of proportional profiles (p. 10).
  • The new insight is where the variation comes from. The Discussion opens: "Auxiliary-variable identification requires an observed variable under which the factors' distributions change, and SSL data often come with none. We show that the augmentation design can create one" (p. 10). With one augmentation distribution for every instance, as in standard pipelines, the instance index does not separate components (Lemma 1, p. 3).
  • For nonlinear decoders, Theorem 1 also identifies a single zero fingerprint and blocks of repeated fingerprints, with the style law unspecified. Our reply to Reviewer TBhh gives details.
  • One correction to the summary: the strengths need not vary independently. One shared strength with distinct slopes suffices (Theorem 1(i); Figure 2A, p. 7).

... training the model requires access and detailed control over these factors. What then is gained by subsequently recovering these factors through representation learning? ... highly simplified setups with strongly controlled dataset augmentations.

  • Training uses no factor values. It needs the ability to set and record each augmentation's strength per instance, and when BB is fitted the learner is not told which augmentation acts on which coordinate (Section 4, p. 5).
  • What is gained is an encoder that maps new, unlabeled images to coordinates aligned with individual factors: MCC 0.9310.931 on held-out instances (Table 1, p. 8). Applying the same augmentations without strength variation leaves the coordinates mixed (MCC 0.8370.837, inside the range of random rotations).
  • The controlled setups are what let us test the claim. With translations along the diagonals the code takes the sum/difference frame, which post-hoc ICA does not return (Table 2, p. 9), and the gain remains at fixed augmentation energy (Table 3, p. 10).

Another problem is the writing ... variables are left unintroduced - already in the introduction, equation 1, what is alpha? ... a major rewrite of the paper would be required.

  • We agree, and we rewrote every section for readability. We took greater care to define all symbols and terms and to avoid jargon as much as possible. In our biased opinion, it reads much better now!
  • Equation 1 is now followed by: "A decoder ff generates each view from content αir∈Rn\alpha_{ir}\in\mathbb R^n and style βir\beta_{ir}", and then "content αir\alpha_{ir} consists of the factors that the augmentations perturb around a stable mean for each instance, and style βir\beta_{ir} consists of per-view nuisance factors" (p. 2).
  • If any passage is still unclear, please let us know which.

How could this method conceivably be applied to any real-world use case? Can you present evidence ... on non-synthetic datasets where the ground truth factors cannot be controlled?

  • We do not claim to have a full-scale practical application. This is a proof of concept with known factors (Discussion and Limitations, p. 10). A full-scale SSL implementation would require further research, in particular on finding augmentations that act through one factor and on evaluating alignment without ground truth. However, we believe that this controlled work is an important foundation for such future work.
  • One avenue, named in the discussion section, is domains with documented factors of variation, such as medical imaging, where generative models could supply such augmentations (p. 10). For instance, a generative model could edit an image along a documented factor by an amount that is set per instance and recorded.

Thank you again. Your comments on clarity changed the paper throughout.

I thank the authors for their reply which addressed some of my concerns. Unfortunately, I don't find the results interesting enough to recommend publication at ICLR and will maintain my rating.

Thank you for your reply. You wrote that our response addressed some of your concerns. Could you tell us which concerns remain open? We would be grateful for the chance to address them.

Reviewer_QZNoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates component-wise identifiability in augmentation-based self-supervised learning. The authors observe that existing SSL methods may recover shared content without identifying individual latent factors, since different invertible mixtures can preserve the same information. They propose designing instance-specific augmentations that induce distinct conditional variance responses across latent factors, termed variance fingerprints. Under a conditional Gaussian model and additional assumptions on the decoder and population support, the authors establish component-wise identifiability up to permutation, scaling, and translation when the variance fingerprints are distinct. They also develop a learning objective combining Gaussian instance discrimination and variance matching. Experiments on synthetic data, dSprites, and Shapes3D demonstrate improved coordinate alignment and factor recovery under controlled augmentation settings.

Critical strengths
  1. The paper addresses a well-defined theoretical question. The distinction between recovering a shared content representation and identifying individual latent factors is important for understanding the limitations of self-supervised representation learning.

  2. The proposed variance fingerprint framework has a clear mathematical motivation. The identification theorem characterizes the conditions under which individual components can be recovered and when ambiguity remains within groups of factors.

  3. The synthetic experiments provide encouraging evidence for the proposed identification mechanism, including nearly perfect factor recovery under nonlinear mixing. The controlled image experiments further demonstrate improved coordinate alignment through augmentation design.

  4. The authors provide useful analyses of learned variance responses, held-out instances, and different augmentation strengths. They also explicitly acknowledge several limitations of the theoretical assumptions and experimental settings.

Critical weaknesses
  1. The generalizability of the proposed framework is unclear. The theoretical results depend on carefully specified assumptions, particularly that latent factors exhibit distinct and predictable conditional variance responses to augmentation strengths. The experiments largely construct settings where these assumptions are expected to hold. While this demonstrates the validity of the proposed approach under favorable conditions, it remains unclear how broadly these conditions arise in practical representation learning problems.

  2. The approach requires substantial prior knowledge about the underlying factors. In the image experiments, augmentations are specifically designed to manipulate known generative factors such as position, scale, and hue. However, in realistic SSL settings, the relevant latent factors may be unknown, correlated, or difficult to manipulate independently. It is unclear how the required augmentation design could be constructed in such cases, or how much of the factor identification problem is already addressed by the prior knowledge used to design the augmentations.

  3. The paper does not sufficiently characterize when the method succeeds or fails under departures from its assumptions. The identification theorem relies on conditional Gaussian distributions, independent latent components, and specific conditions on the decoder and population support. Although these assumptions are clearly stated, the experiments provide limited evidence about robustness to violations. In particular, correlated augmentation effects, nonlinear variance responses, and weakly separated fingerprints could substantially affect identification. A more systematic sensitivity analysis would help establish the practical boundaries of the proposed framework.

Questions for authors

My question is mainly on the robustness of results when assumption partially fail. How does the method perform when an augmentation affects multiple correlated latent factors or when conditional variances do not follow the assumed log-linear relationship? Can the authors provide experiments that systematically relax these assumptions?

How does the proposed method compare with existing augmentation-aware SSL and disentanglement approaches under matched training and augmentation settings? Such comparisons would help clarify the additional contribution of variance fingerprint modeling.

3 comments
AuthorsOpenReview ↗

Author response

Thank you for your clear review and for your suggestions. In response to your comments, we added a new robustness sweep where we relax various assumptions made in the theory section to study how the results degrade. We found that identification is quite robust to intentional breakdown of our assumptions in synthetic tests (results summarized below). We also heavily revised the paper to clarify our claims and the limitations of our approach.

Responses follow in the order of your comments. Page numbers refer to the revised PDF.

The generalizability of the proposed framework is unclear. ... The experiments largely construct settings where these assumptions are expected to hold. ... it remains unclear how broadly these conditions arise in practical representation learning problems.

  • The design applies wherever an augmentation's strength can be set and recorded per instance. How broadly such augmentations exist in practice is open, and the paper now more clearly presents itself as a controlled proof of concept for our design idea (Discussion and Limitations, p. 10).
  • The experiments no longer only match the assumptions. The sweep below violates them one at a time, and the image runs depart from them in five listed ways and still align (Section 5.2; Appendix D.1).

The approach requires substantial prior knowledge about the underlying factors. ... how much of the factor identification problem is already addressed by the prior knowledge used to design the augmentations.

  • We concede that the prior choice of augmentations addresses the factor identification problem. However, every nonlinear identification result relies on supplied structure, usually an auxiliary variable, and that variable determines which components are identified. Our augmentations play this role, and here the structure is selected by the designer rather than incidental to the data. In some sense, this is a firmer footing, since you are not limited to variables that were collected alongside your primary variables. The deeper problem is that the factor interventions available through augmentations may be somewhat limited. However, we believe that this controlled study sets the groundwork for future research along those lines.
  • We added a new experiment with augmentations along the diagonals. Our code aligns with x+yx+y and x−yx-y, while post-hoc ICA of a code trained without strength variation returns xx and yy (Table 2, p. 9). Connecting to the previous point, this is a feature not a bug since there is no "true" coordinate frame.

The paper does not sufficiently characterize when the method succeeds or fails under departures from its assumptions. ... correlated augmentation effects, nonlinear variance responses, and weakly separated fingerprints ... (also Q1)

The sweep is in Section 5.1 (p. 7) and Appendix E.8 (Table E.13), on nonlinear synthetic data at ω=0.5\omega=0.5. In view MCC:

  • Weakly separated fingerprints. No loss up to cosine 0.90.9 between two fingerprints. When they coincide, only that pair loses alignment (0.8960.896, others 0.9940.994), as Theorem 1(iii) predicts.
  • Correlated augmentation effects. With leakage of each strength into the other log-variances: 0.9950.995 at leakage 0.10.1, 0.9820.982 at 0.40.4, and 0.7850.785 at leakage 11, where all fingerprints are equal.
  • Augmentations that move several factors together. The code aligns with the directions the augmentations perturb (frame view MCC 0.9120.912 to 1.0001.000).
  • Nonlinear variance responses. Saturating responses: ≥0.996\ge0.996 even when the linear part explains 75% of the log-variance variation. Non-monotone responses: 0.9960.996 at 47%, declining to 0.7680.768 when the linear part explains under 1%.
  • Non-Gaussian noise. Uniform: 0.9940.994. Student-tt with 2.52.5 degrees of freedom: 0.8980.898, with Amari distance 0.0130.013.
  • Dependent factors. The theorem needs independence only within an instance. With strongly correlated instance means, our Amari distance stays at most 0.0530.053, while post-hoc FastICA reaches 0.4100.410 (chance 0.420.42).

How does the proposed method compare with existing augmentation-aware SSL and disentanglement approaches under matched training and augmentation settings?

  • We trained InfoNCE, SimCLR, VICReg, the augmentation-aware SIE, EquiMod and AugSelf, a β\beta-VAE and GCL on the same views, augmentation families and strength table, with published defaults. The strongest, the β\beta-VAE, reaches MCC 0.7640.764 at ω=0.5\omega=0.5 against our 0.9320.932 (Section 5.2, p. 9; Table 3). After post-hoc FastICA, the best baseline (SIE) reaches 0.8730.873 (Appendix Table E.11).

Thank you again for your careful reading!

We wanted to follow up briefly before the discussion period closes. Your review asked for experiments that relax the assumptions and for comparisons under matched training and augmentation settings. The revision adds a sweep that violates several assumptions one at a time (Section 5.1; Appendix Table E.13) and additional baselines trained in matched conditions (Table 3; Appendix Table E.11). We would be grateful to hear whether these address your concerns, and whether any remain that we can address before Nov 18.

Reviewer_QZNoOpenReview ↗

Thank you

Thank you for the thoughtful response and additional experiments. I appreciate the authors' efforts to address my comments.

The additional results strengthen the paper. However, I still have some reservations about its generalizability, especially when the underlying factors are unknown and suitable augmentations are difficult to design. Given the remaining questions about broader applicability, I will respectfully maintain my original rating.

Reviewer_TBhhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper uses augmentation-induced variance fingerprints to identify content factors within a conditional-Gaussian comparison class. Distinct fingerprints yield component-wise identification. Repeated fingerprints yield block identification. A corresponding learner is evaluated on synthetic data, dSprites, and Shapes3D.

Critical strengths

The identification theorem appears sounding and precisely characterizes component and block ambiguities, including a unique zero fingerprint. Author also provides careful ablations and negative controls support the usefulness of variance pooling and transparently expose limitations. The work is generally solid.

Critical weaknesses
  1. The theoretical advance is still a bit unclear. The connection to auxiliary-variable ICA is acknowledged, but I’m not quite convinced how much Theorem 1 adds. Could you explain or isolate what existing results do not cover under comparable assumptions, particularly the zero/repeated fingerprints and nonparametric style block?

  2. The most distinctive setting needs stronger experimental support. I know this is maybe a bit more in detail. The nonlinear experiments that learn BB only use true B=IB=I in Appendix E.1, and the auxiliary style supervision has not been shown. For me, the most useful additions would be recovery with a dense true BB or a shared strength with distinct slopes, plus a separate ablation removing the style loss. Consistent recovery in these settings would make me considerably more positive.

  3. The empirical gains are not yet sufficiently isolated. Could post-hoc ICA on the ω=0\omega=0 dSprites representation close the gap? I would also like to see an ω\omega sweep with fixed expected perturbation energy, since increasing ω\omega changes both variance heterogeneity and augmentation magnitude. A clear advantage over that baseline, together with gains that survive the magnitude control, would be very beneficial to close this gap.

I am actually hesitating between 2/3, therefore, I am very looking forward to see how the authors address my above concerns.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Author response

Thank you for a careful review and helpful suggestions about how to further test our design. We ran the experiments you suggested and found that identification clearly holds in each setting. Your Post-hoc ICA suggestion was particularly clarifying. It nearly matches our MCC values when predicting x and y factors on dSprites. We ran an additional test where we augment along the diagonals x+y and x-y. Our design picks up the diagonal coordinates and post-hoc ICA still picks up x and y. These individual components happen to be independent in dSprites, which is exactly what ICA needs. This illustrates a main point that we have tried to clarify in our revision: the augmentations choose the content factors. This is a feature not a bug, since there's no "true" coordinate frame. Moreover, all non-linear identification methods must supply some structure, usually an auxiliary variable that happens to accompany the data. Here the designer selects that structure through the augmentations.

Responses follow in the order of your comments. Page numbers refer to the revised PDF.

The theoretical advance is still a bit unclear. ... Could you explain or isolate what existing results do not cover under comparable assumptions, particularly the zero/repeated fingerprints and nonparametric style block?

  • We agree that the mechanism is that of auxiliary-variable nonlinear ICA, and the paper now says so: "We therefore claim no new identification mechanism" (Section 6, p. 10).
  • The advance is a bridge from that theory into SSL. Lemma 1 shows that the standard pipeline, with one augmentation distribution for every instance, cannot identify components (p. 3). Theorem 1 shows what recorded per-instance strengths add: components for distinct fingerprints and blocks for repeated ones (p. 4).
  • On what existing results cover: with (i,si)(i,s_i) as the auxiliary variable, distinct nonzero fingerprints meet the variability condition of iVAE (Khemakhem et al., 2020, Thm. 2) and GCL (Hyvärinen et al., 2019, Thm. 1), so those results apply (Theorem 1(i)). They do not state the cases you name: a single zero fingerprint, where that condition fails; repeated fingerprints, identified as blocks; and an unspecified style law. Theorem 1 also needs only injective Borel decoders.
  • Before the theorem we added: "Theorem 1 also covers rank-deficient designs, leaves the style law unspecified, and needs only injective Borel decoders" (p. 4). The paragraph after it explains each case (p. 5).

The nonlinear experiments that learn BB only use true B=IB=I ... the most useful additions would be recovery with a dense true BB or a shared strength with distinct slopes, plus a separate ablation removing the style loss.

  • Dense true BB. When the true BB is a random dense matrix that the learner fits from zero, view MCC is 0.997±0.0010.997\pm0.001 at ω=0.5\omega=0.5, against 0.9970.997 with the true BB supplied (Section 5.1, p. 7; Appendix Table E.4).
  • Shared strength with distinct slopes. The learner recovers the factors, more slowly: Amari distance 0.0720.072 at ω=0.5\omega=0.5 against 0.0200.020 for B=I4B=I_4, on energy-matched data (p. 7; Appendix Table E.9).
  • Style loss. Removing the regression of the style auxiliary changes view MCC by at most 0.0010.001 at ω>0\omega>0 (p. 7; Appendix Table E.4). The image learner has no style loss (Appendix D.2).

Could post-hoc ICA on the ω=0\omega=0 dSprites representation close the gap? ... an ω\omega sweep with fixed expected perturbation energy ...

  • Post-hoc ICA. With augmentations along the image axes it nearly does: FastICA of the ω=0\omega=0 code reaches MCC 0.926±0.0220.926\pm0.022 against our 0.932±0.0010.932\pm0.001. Section 5.2 now reports this.
  • As mentioned above, we added augmentations that translate along the diagonals. Our code then aligns with the sum/difference frame, above its rotation reference in 5 of 5 seeds, while FastICA of the ω=0\omega=0 code still returns the position frame (0 of 5; Table 2, p. 9). On synthetic data with strongly dependent instance means, our Amari distance stays at most 0.0530.053, while FastICA reaches 0.4100.410 (chance 0.420.42; p. 7).
  • Fixed energy. At the ω=0\omega=0 energy, the ω=0.5\omega=0.5 run reaches MCC 0.9440.944 against 0.8370.837 at ω=0\omega=0, and the ω=0\omega=0 run at the ω=0.5\omega=0.5 energy reaches 0.8510.851 (p. 9; Table 3, p. 10). The gain therefore comes from variation of the strengths across instances. The loss at ω=1\omega=1 is largely an energy effect: at the ω=0\omega=0 energy, the ω=1\omega=1 run reaches 0.8340.834, against 0.5790.579 without matching.

The robustness sweep described in our reply to Reviewer QZNo also covers the settings you singled out, including weakly separated and coinciding fingerprints.

Thank you for reading our replies and updating your score. Please let us know if there are any remaining concerns we can address.

Thanks for your rebuttal and my concerns are addressed. According to my estimation of the general quality of the paper, I raise from 2 to 3 and I think it reflects the true quality of this paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author revision note and overview of response to reviewers

We thank the three reviewers for their time and for comments that improved the paper substantially. This comment summarizes the revision; details are in the individual replies. Page numbers refer to the revised PDF.

The contribution, as the revision now states it. We agree with the reviewers that the identification mechanism is that of auxiliary-variable nonlinear ICA, and the paper claims no new mechanism (Section 6, p. 10). The paper is a bridge from that theory into SSL. Lemma 1 shows that the standard pipeline, with one augmentation distribution for every instance, cannot identify individual content components (p. 3). Theorem 1 shows what recorded per-instance augmentation strengths add: components for distinct fingerprints and blocks for repeated ones, including cases where the usual rank condition fails (pp. 4 to 5).

Prior knowledge (QZNo, Nmhn). Every nonlinear identification result relies on supplied structure, usually an auxiliary variable that happens to accompany the data, and that variable determines which components are identified. Here the designer selects the structure through the augmentations. The learner receives no factor values.

Readability (Nmhn). The paper was hard to read. We rewrote every section and defined each symbol and term at first use.

New results (TBhh, QZNo).

  • The augmentations choose the factors. With augmentations along the image axes, post-hoc ICA nearly matches our code on dSprites (MCC 0.9260.926 against 0.9320.932). With augmentations along the diagonals, our code aligns with x+yx+y and x−yx-y in 5 of 5 seeds, and post-hoc ICA still returns xx and yy (Table 2, p. 9). No coordinate frame is the true one, and the design lets the practitioner choose it.
  • The gain is not an effect of larger augmentations. At fixed augmentation energy, strength variation raises MCC from 0.8370.837 to 0.9440.944 (Table 3, p. 10).
  • Requested settings. The learner recovers the factors with a dense random BB fitted from zero, with one shared strength, and without the style loss (Section 5.1, p. 7).
  • Robustness. A sweep that relaxes one assumption at a time shows recovery declining gradually (p. 7; Appendix E.8).
  • Baselines. Eight methods trained on the same views and strengths reach at most MCC 0.764 against our 0.932 (Table 3; Appendix Table E.11).

Scope. The paper is a controlled proof of concept with known factors. A full-scale SSL implementation requires further research. One avenue is domains with documented factors of variation, such as medical imaging, where generative models could supply the augmentations (Discussion, p. 10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.