- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates which downstream semantic tasks remain recoverable from representations learned through same-instance self-supervised learning (SSL). The authors introduce captured posterior energy, (B(F)), as a task-dependent measure of the information accessible within the linear function space of a learned representation. Under centered and whitened representations, they establish theoretical connections between recoverability, directional class-distance-normalized variance (CDNV), linear-probe and centroid alignment, few-shot nearest-centroid classification, and multitask centroid geometry. They further analyze a canonical two-view SSL objective and show that its population optimum selects the leading eigenfunctions of a two-view operator, allowing task recoverability to be expressed through spectral overlap with the downstream posterior. Experiments on CelebA, dSprites, and 3DShapes, involving W-MSE, VICReg, Barlow Twins, and I-JEPA representations, examine the predicted relationships between recoverability, spectral structure, representation geometry, and few-shot transfer. Overall, the paper seeks to provide a unified theoretical explanation of which downstream tasks survive SSL and how their retained information influences downstream learning.
Critical strengths
-
A meaningful and well-motivated theoretical question. The paper addresses an important issue in SSL: although different views of the same instance are encouraged to have consistent representations, this does not guarantee that all downstream semantic attributes are equally preserved. The task-dependent formulation is useful for understanding why a pretrained representation may transfer well to some tasks but poorly to others. The attempt to connect task recoverability, representation geometry, and few-shot transfer within one framework is conceptually appealing.
-
Clear mathematical formulation and reasonably broad empirical analysis. The paper develops explicit mathematical relationships linking captured posterior energy to directional CDNV and few-shot nearest-centroid performance under well-defined assumptions. The experiments cover several established SSL approaches and both natural and synthetic datasets. The manuscript is generally well organized, and the theoretical statements and corresponding empirical analyses are presented clearly. These are valuable contributions even though the interpretation and generality of the findings require further clarification.
Critical weaknesses
- The novelty and distinction from closely related spectral SSL theories are not sufficiently established.
The connection between augmentation-induced invariance, spectral properties of representations, and downstream generalization has been studied in prior work. In particular:
Cabannes et al., The SSL Interplay: Augmentations, Inductive Bias, and Generalization (ICML 2023, arXiv:2302.02774), analyze the interplay between augmentation, architectural inductive bias, training, and downstream generalization.
Johnson et al., Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions (ICLR 2023, arXiv:2210.01883), characterize approximately view-invariant functions through a positive-pair Markov operator and its eigenfunctions, and connect the resulting representations to downstream linear prediction and generalization bounds.
The latter work appears particularly relevant to the current paper's characterization of downstream task recoverability through projections onto leading eigenfunctions of a two-view operator. While the present submission additionally connects this spectral perspective to directional CDNV, few-shot nearest-centroid bounds, and multitask centroid geometry, the distinction between the existing spectral characterization and the new theoretical contribution should be made more explicit.
In addition, for balanced binary labels, captured posterior energy is equivalent to the explained variance of an optimal population linear least-squares probe. Likewise, the exact relationship between (B(F)) and directional CDNV follows from the covariance decomposition under whitening. This does not invalidate the theoretical contribution, but a more careful discussion is needed to clarify which findings represent genuinely new insights and which unify or reformulate established results.
- The experimental evidence does not sufficiently establish the proposed mechanism of task survival under SSL.
The central explanation is that SSL preferentially preserves cross-view-stable modes, and that downstream tasks survive according to their overlap with the selected spectral subspace. However, the current experiments primarily measure recoverability and related geometric or classification properties in already learned representations.
In particular, the strong agreement between predicted and observed directional CDNV in Fig. 4 largely verifies an identity that holds for any centered, whitened representation with balanced binary labels, independently of SSL. Similarly, the spectral reconstruction experiment in Fig. 7 estimates spectral directions from pretrained feature representations, which provides useful evidence about their retained subspaces but does not by itself demonstrate that the proposed spectral mechanism explains how these representations were learned.
A more direct experiment would manipulate the augmentation distribution in a controlled setting. For example, augmentations could selectively disrupt information relevant to a particular downstream task while preserving information relevant to other tasks. The theory would then predict that the affected task's recoverability (B_r) should decrease, while recoverability for unaffected tasks should remain relatively stable. Demonstrating this behavior across augmentation settings would provide stronger evidence for the paper's central claim.
The analysis also relies heavily on idealized population-optimal, centered and whitened representations, whereas practical SSL algorithms have additional architectural and optimization characteristics. The empirical implications of these differences deserve clearer discussion.
Questions for authors
Q1. Novelty relative to existing spectral SSL theories. Could the authors clarify the main theoretical advances beyond Cabannes et al. (ICML 2023, arXiv:2302.02774) and Johnson et al. (ICLR 2023, arXiv:2210.01883)? In particular, how does the spectral characterization of task recoverability differ from existing analyses based on approximately view-invariant eigenfunctions and downstream linear prediction? Which results are new consequences of the proposed framework, rather than reformulations of existing spectral or linear-prediction results?
Q2. Direct evidence for augmentation-dependent task survival. Could the authors provide a controlled experiment investigating how task recoverability changes when the augmentation distribution selectively removes task-relevant information? For example, if an augmentation disrupts color-related information but preserves shape-related information, does the measured (B_r) decrease selectively for color-dependent tasks while remaining stable for shape-dependent tasks? Such evidence would more directly validate the proposed explanation of why particular tasks survive SSL.
Q3. Applicability to other representative SSL approaches. While the paper evaluates several SSL methods, including VICReg, Barlow Twins, and I-JEPA, could the authors clarify how the theoretical conclusions are expected to extend to other widely adopted approaches such as MoCo, DINO/DINOv2, or SwAV? In particular, which properties of their training objectives are necessary for the proposed spectral characterization, and which aspects may not be covered by the current analysis? This clarification could help distinguish the generality of the theoretical framework from the empirical applicability of the recoverability measure.
Q4. Scope of task recoverability. The current definition of (B(F)) characterizes recoverability within the linear span of the representation. Could the authors clarify the implications for tasks that may be recoverable through nonlinear downstream predictors but have limited linear recoverability? Additionally, is there a principled extension to multiclass classification, such as ImageNet-style recognition, that could establish the practical relevance of the proposed task-level characterization beyond balanced binary attributes?
4 comments
Continued Response to Reviewer TYE2 (Part 2)
(W2.1) Reviewer: The experiments establish relationships between recoverability, directional geometry, and downstream performance, but provide limited direct evidence that SSL preserves tasks by selecting cross-view-stable spectral modes. In particular, Fig. 4 validates an identity that holds for any whitened representation, while Fig. 7 reconstructs spectral structure from already learned features rather than explaining how that structure emerged. A more direct test of the proposed learning mechanism would strengthen the central claim.
Response: We thank the reviewer for this careful observation. We agree that Figs. 4 and 7 serve different purposes from directly validating the proposed SSL learning mechanism, and we appreciate the opportunity to clarify their roles.
Fig. 4 was designed to empirically validate the exact relationship between recoverability and directional CDNV, using independent data splits to verify that the predicted geometric relationship holds in practice. Fig. 7 was intended to characterize the semantic information retained in the spectral subspace estimated from learned representations, rather than explain how that subspace was learned. While neither experiment alone establishes the proposed SSL learning mechanism, both are important for empirically validating the predictions of our theoretical results (Proposition 3.1 and Corollary 4.2, respectively).
Importantly, Fig. 5 provides complementary evidence for the practical relevance of recoverability: it shows that is strongly associated with empirical few-shot NCC error across downstream tasks, representation ranks, and support sizes. Unlike the relationship with directional CDNV, this association with actual classification performance is not an algebraic identity.
We also appreciate the reviewer's constructive suggestion to directly manipulate the augmentation distribution. We have conducted a new controlled augmentation intervention experiment, which tests whether changing the cross-view stability of a semantic factor produces the predicted change in its recoverability. Please see the new Fig. 8 in our revised manuscript and our detailed response to the augmentation experiment below addressing W2.2.
(W2.2 + Q2) Reviewer: Controlled augmentation experiment: Can the theory predict how selectively disrupting task-relevant information through augmentations affects semantic recoverability and downstream transfer?
Response: We thank the reviewer for this valuable suggestion. We have added a controlled experiment that directly tests how changing the augmentation distribution determines which semantic factors are preserved by SSL.
We generate synthetic images (see Fig. 8(a)) with three independent binary factors: shape (circle/triangle), color (red/blue), and background texture (stripes/dots). To generate two views, each factor is independently resampled with a specified probability. We fix the resampling probabilities for shape and texture at and , respectively, while varying the probability for color.
Our theory predicts which factors should be retained. The corresponding operator eigenvalues are , , and (please check our new Appendix B.4 for details). For a two-dimensional representation, the top-2 subspace should switch from shape and color to shape and texture at .
To test this prediction, we train lightweight CNN encoders using the whitening-based cross-view alignment objective studied in our theoretical framework (Eqn. 2), across seven augmentation settings and three seeds. All 21 runs recover the predicted leading spectral modes. Evaluating on the same clean, held-out images, we find that color recoverability drops from approximately 1 to 0 around the predicted transition, between and , while shape recoverability remains approximately 1 throughout. Consistently, color 20-shot NCC error rises from 0% to 50%, whereas shape classification remains nearly perfect.
These results provide direct evidence for our central claim (and reviewer’s comment): the augmentation distribution determines which semantic factors enter the SSL-selected subspace, and their spectral recoverability predicts the resulting downstream transfer behavior. We have incorporated this experiment into Section 5.2 (Fig. 8) and Appendix B.4 of the revised manuscript.
Continued Response to Reviewer TYE2 (Part 3)
(W2.3) Reviewer: The analysis relies on idealized population-optimal, centered and whitened representations, whereas practical SSL algorithms have additional architectural and optimization characteristics. The differences require clearer discussion.
Response: We thank the reviewer for highlighting this important distinction. Our geometric identities and few-shot guarantees apply to any centered, whitened representation, whereas the spectral-selection characterization assumes a population-optimal SSL surrogate. We intentionally keep the theoretical framework architecture-agnostic to isolate the role of augmentation-induced spectral structure. Our experiments with diverse pretrained SSL encoders examine how these predictions translate to practical settings, while acknowledging that architectural and optimization effects are not explicitly modeled. We have clarified this scope in the revised manuscript (Sections 5.1 and 6).
(Q3) Reviewer: Applicability to other representative SSL approaches. Which assumptions are necessary for the spectral characterization, and how far do the recoverability-based geometric and few-shot conclusions extend to methods such as MoCo, DINO/DINOv2, and SwAV?
Response: We thank the reviewer for this question. We clarify that the spectral characterization and the recoverability-based guarantees have different scopes.
The spectral characterization is objective-dependent. It is derived for the whitening-based cross-view alignment objective studied in Eq. 2, and extends to the covariance-regularized and prediction-based surrogates analyzed in Appendices C–D. Since MoCo, DINO/DINOv2, and SwAV employ different training objectives, their spectral selection mechanisms are not formally covered by our current analysis.
In contrast, , its exact relationship with directional CDNV, and our few-shot NCC guarantee (Thm. 3.4) apply to any centered, whitened representation, independently of how it was learned. To demonstrate this broader applicability, we additionally evaluated DINO, SwAV, MoCo v3, and two DINOv2 encoders on all 40 CelebA attributes. Across these five encoders, predicted and observed directional CDNV agree closely (median observed-to-predicted ratio: 1.007–1.014), and recoverability strongly correlates with few-shot NCC error (Spearman correlation of approximately −0.99).
We have added these results in Figs. 18–19 (Appendix B.5), demonstrating that the geometric and few-shot implications of recoverability extend empirically beyond the SSL objectives covered by our spectral analysis.
(Q4) Reviewer: How does (B(F)), which measures linear recoverability, relate to tasks accessible through nonlinear predictors? Can the framework be extended to multiclass classification, such as ImageNet-style recognition, to demonstrate relevance beyond balanced binary tasks?
Response: We thank the reviewer for raising these points. On nonlinear recoverability, please see our response to reviewer YrP9 (W2). On multiclass classification, we extend semantic recoverability by treating each class as a one-vs-rest binary task. For equally likely classes, let and define the standardized binary label . The class-specific recoverability is , and multiclass recoverability is defined as . This recovers the original binary definition when .
We evaluate this extension on 20 randomly selected ImageNet-1K classes using a frozen ImageNet-pretrained VICReg encoder. Across five representation ranks ranging from 16 to 256, we estimate recoverability on training images and evaluate 20-way classification on 1,000 held-out images. At every rank, classwise recoverability positively correlates with held-out NCC recall (Spearman's ranging from 0.705 to 0.795). Across ranks, multiclass recoverability increases from 0.591 to 0.812, accompanied by improvements in NCC accuracy (78.1% to 92.7%) and linear-probe accuracy (88.0% to 93.6%).
These results provide empirical evidence that our task-level recoverability measure remains informative beyond balanced binary classification. We have added the multiclass formulation and results in Appendix B.6 (Table 3 and Fig. 20).
A Gentle Follow-up and Thanks for Your Feedback
We thank the reviewer again for their constructive feedback.
We understand that this is a busy time for reviewers, with several papers to discuss. With the discussion period ending soon, we would greatly appreciate any feedback on our rebuttal, particularly if any concerns remain unresolved. We would be happy to provide further clarification.
Thank you again for your time and consideration.
Author Response to Reviewer TYE2
We thank the reviewer for their thoughtful and constructive feedback.
Response: We thank the reviewer for highlighting these closely related works. We agree that the spectral characterization of SSL and its connection to downstream linear prediction have been studied previously. Our work introduces a new framework that unifies recoverability, transferability, and multi-task geometry.
Specifically, our framework provides the following important conclusions:
Which tasks can be learned in a sample-efficient manner? Johnson et al. (2023) establish guarantees for downstream linear prediction. Our Theorem 3.4 instead provides a direct m-shot nearest-centroid classification bound in terms of task recoverability B(F), representation dimension r, and support size m. Together with the exact directional CDNV identity (Prop. 3.1), this characterizes how retained semantic information and finite-sample centroid estimation jointly affect downstream classification.
How are multiple correlated tasks represented together? Luthra et al. (2026) study multitask geometry and establish near-orthogonality of semantic directions for independent tasks. Our Theorem 3.5 extends this analysis to the more realistic setting of dependent tasks, explicitly characterizing how label correlations and shared semantic information influence decision-axis alignment and joint-class centroid geometry. This allows us to reason about how multiple semantic attributes coexist within the same learned representation.
The central distinction is therefore not the spectral characterization itself, but the new geometric and few-shot consequences we derive from task-specific posterior overlap with the selected subspace. These results connect spectral selection to directional neural collapse, finite-shot classification, and the joint organization of correlated semantic tasks.
Revision to the manuscript. To clarify this distinction, we have revised the Introduction section (L46 and contribution (iii) (L65)) and expanded the discussion on both suggested works in Related Work (L95).
Response: We thank the reviewer for this observation. We have made this clarification more explicit in the revised manuscript (refer to L189 after Prop. 3.1).
As detailed in our response to W1.1, our contribution is not these identities in isolation, but the theoretical framework they enable. By connecting recoverability to the spectral subspace selected by SSL, we characterize when favorable task geometry emerges from a label-free objective. Building on this connection, we derive a direct few-shot NCC guarantee (Thm. 3.4) and characterize the joint geometry of dependent downstream tasks (Thm. 3.5).