acceptodds
Under review as a conference paper at ICLR 2027

Which Tasks Survive Self-Supervised Learning?

Abstract

Same-instance self-supervised learning (SSL) learns representations by enforcing consistency across two views of the same underlying instance. This principle alone, however, does not determine which downstream tasks remain recoverable from the learned representation. We study this question through *semantic recoverability*, defined as the amount of a task's posterior score captured by the represented function space. We show that, for centered and whitened representations, recoverability exactly determines directional class-distance normalized variance (CDNV), controls few-shot nearest-centroid classification, and governs the strength of task-relevant semantic directions; the population linear probe and centroid axis coincide, and multiple well-recovered tasks approach a factorial centroid geometry. We then analyze a canonical two-view SSL objective and show that its population optimum spans the leading cross view-stable modes of the associated two-view operator. This yields a closed-form spectral characterization of semantic recoverability: a downstream task is preserved to the extent that its posterior lies in the selected spectral subspace. We validate these predictions on synthetic and real datasets across several SSL methods, testing the predicted relationships among recoverability, directional geometry, spectral structure, and few-shot transfer. Together, these results give a task-level account of what information survives same-instance SSL and how the retained information appears in downstream geometry and transfer.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self supervised learning, representation learning, neural collapse

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TYE2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates which downstream semantic tasks remain recoverable from representations learned through same-instance self-supervised learning (SSL). The authors introduce captured posterior energy, (B(F)), as a task-dependent measure of the information accessible within the linear function space of a learned representation. Under centered and whitened representations, they establish theoretical connections between recoverability, directional class-distance-normalized variance (CDNV), linear-probe and centroid alignment, few-shot nearest-centroid classification, and multitask centroid geometry. They further analyze a canonical two-view SSL objective and show that its population optimum selects the leading eigenfunctions of a two-view operator, allowing task recoverability to be expressed through spectral overlap with the downstream posterior. Experiments on CelebA, dSprites, and 3DShapes, involving W-MSE, VICReg, Barlow Twins, and I-JEPA representations, examine the predicted relationships between recoverability, spectral structure, representation geometry, and few-shot transfer. Overall, the paper seeks to provide a unified theoretical explanation of which downstream tasks survive SSL and how their retained information influences downstream learning.

Critical strengths
  1. A meaningful and well-motivated theoretical question. The paper addresses an important issue in SSL: although different views of the same instance are encouraged to have consistent representations, this does not guarantee that all downstream semantic attributes are equally preserved. The task-dependent formulation is useful for understanding why a pretrained representation may transfer well to some tasks but poorly to others. The attempt to connect task recoverability, representation geometry, and few-shot transfer within one framework is conceptually appealing.

  2. Clear mathematical formulation and reasonably broad empirical analysis. The paper develops explicit mathematical relationships linking captured posterior energy to directional CDNV and few-shot nearest-centroid performance under well-defined assumptions. The experiments cover several established SSL approaches and both natural and synthetic datasets. The manuscript is generally well organized, and the theoretical statements and corresponding empirical analyses are presented clearly. These are valuable contributions even though the interpretation and generality of the findings require further clarification.

Critical weaknesses
  1. The novelty and distinction from closely related spectral SSL theories are not sufficiently established.

The connection between augmentation-induced invariance, spectral properties of representations, and downstream generalization has been studied in prior work. In particular:

Cabannes et al., The SSL Interplay: Augmentations, Inductive Bias, and Generalization (ICML 2023, arXiv:2302.02774), analyze the interplay between augmentation, architectural inductive bias, training, and downstream generalization.

Johnson et al., Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions (ICLR 2023, arXiv:2210.01883), characterize approximately view-invariant functions through a positive-pair Markov operator and its eigenfunctions, and connect the resulting representations to downstream linear prediction and generalization bounds.

The latter work appears particularly relevant to the current paper's characterization of downstream task recoverability through projections onto leading eigenfunctions of a two-view operator. While the present submission additionally connects this spectral perspective to directional CDNV, few-shot nearest-centroid bounds, and multitask centroid geometry, the distinction between the existing spectral characterization and the new theoretical contribution should be made more explicit.

In addition, for balanced binary labels, captured posterior energy is equivalent to the explained variance of an optimal population linear least-squares probe. Likewise, the exact relationship between (B(F)) and directional CDNV follows from the covariance decomposition under whitening. This does not invalidate the theoretical contribution, but a more careful discussion is needed to clarify which findings represent genuinely new insights and which unify or reformulate established results.

  1. The experimental evidence does not sufficiently establish the proposed mechanism of task survival under SSL.

The central explanation is that SSL preferentially preserves cross-view-stable modes, and that downstream tasks survive according to their overlap with the selected spectral subspace. However, the current experiments primarily measure recoverability and related geometric or classification properties in already learned representations.

In particular, the strong agreement between predicted and observed directional CDNV in Fig. 4 largely verifies an identity that holds for any centered, whitened representation with balanced binary labels, independently of SSL. Similarly, the spectral reconstruction experiment in Fig. 7 estimates spectral directions from pretrained feature representations, which provides useful evidence about their retained subspaces but does not by itself demonstrate that the proposed spectral mechanism explains how these representations were learned.

A more direct experiment would manipulate the augmentation distribution in a controlled setting. For example, augmentations could selectively disrupt information relevant to a particular downstream task while preserving information relevant to other tasks. The theory would then predict that the affected task's recoverability (B_r) should decrease, while recoverability for unaffected tasks should remain relatively stable. Demonstrating this behavior across augmentation settings would provide stronger evidence for the paper's central claim.

The analysis also relies heavily on idealized population-optimal, centered and whitened representations, whereas practical SSL algorithms have additional architectural and optimization characteristics. The empirical implications of these differences deserve clearer discussion.

Questions for authors

Q1. Novelty relative to existing spectral SSL theories. Could the authors clarify the main theoretical advances beyond Cabannes et al. (ICML 2023, arXiv:2302.02774) and Johnson et al. (ICLR 2023, arXiv:2210.01883)? In particular, how does the spectral characterization of task recoverability differ from existing analyses based on approximately view-invariant eigenfunctions and downstream linear prediction? Which results are new consequences of the proposed framework, rather than reformulations of existing spectral or linear-prediction results?

Q2. Direct evidence for augmentation-dependent task survival. Could the authors provide a controlled experiment investigating how task recoverability changes when the augmentation distribution selectively removes task-relevant information? For example, if an augmentation disrupts color-related information but preserves shape-related information, does the measured (B_r) decrease selectively for color-dependent tasks while remaining stable for shape-dependent tasks? Such evidence would more directly validate the proposed explanation of why particular tasks survive SSL.

Q3. Applicability to other representative SSL approaches. While the paper evaluates several SSL methods, including VICReg, Barlow Twins, and I-JEPA, could the authors clarify how the theoretical conclusions are expected to extend to other widely adopted approaches such as MoCo, DINO/DINOv2, or SwAV? In particular, which properties of their training objectives are necessary for the proposed spectral characterization, and which aspects may not be covered by the current analysis? This clarification could help distinguish the generality of the theoretical framework from the empirical applicability of the recoverability measure.

Q4. Scope of task recoverability. The current definition of (B(F)) characterizes recoverability within the linear span of the representation. Could the authors clarify the implications for tasks that may be recoverable through nonlinear downstream predictors but have limited linear recoverability? Additionally, is there a principled extension to multiclass classification, such as ImageNet-style recognition, that could establish the practical relevance of the proposed task-level characterization beyond balanced binary attributes?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer TYE2

We thank the reviewer for their thoughtful and constructive feedback.

(W1.1 + Q1) Reviewer: Prior work, particularly Johnson et al. (2023) and Cabannes et al. (2023), already connects augmentation-induced spectral structure to downstream prediction and generalization. What additional theoretical conclusions does our framework establish beyond these existing connections?

Response: We thank the reviewer for highlighting these closely related works. We agree that the spectral characterization of SSL and its connection to downstream linear prediction have been studied previously. Our work introduces a new framework that unifies recoverability, transferability, and multi-task geometry.

Specifically, our framework provides the following important conclusions:

  1. Which tasks can be learned in a sample-efficient manner? Johnson et al. (2023) establish guarantees for downstream linear prediction. Our Theorem 3.4 instead provides a direct mm-shot nearest-centroid classification bound in terms of task recoverability B(F)B(F), representation dimension rr, and support size mm. Together with the exact directional CDNV identity (Prop. 3.1), this characterizes how retained semantic information and finite-sample centroid estimation jointly affect downstream classification.

  2. How are multiple correlated tasks represented together? Luthra et al. (2026) study multitask geometry and establish near-orthogonality of semantic directions for independent tasks. Our Theorem 3.5 extends this analysis to the more realistic setting of dependent tasks, explicitly characterizing how label correlations and shared semantic information influence decision-axis alignment and joint-class centroid geometry. This allows us to reason about how multiple semantic attributes coexist within the same learned representation.

The central distinction is therefore not the spectral characterization itself, but the new geometric and few-shot consequences we derive from task-specific posterior overlap with the selected subspace. These results connect spectral selection to directional neural collapse, finite-shot classification, and the joint organization of correlated semantic tasks.

Revision to the manuscript. To clarify this distinction, we have revised the Introduction section (L46 and contribution (iii) (L65)) and expanded the discussion on both suggested works in Related Work (L95).

(W1.2) Reviewer: The interpretation of B(F)B(F) through linear least-squares regression and its exact relationship with directional CDNV follow from standard identities under whitening. Need clearer discussion on which results are new contributions and which are reformulations that enable the broader framework?

Response: We thank the reviewer for this observation. We have made this clarification more explicit in the revised manuscript (refer to L189 after Prop. 3.1).

As detailed in our response to W1.1, our contribution is not these identities in isolation, but the theoretical framework they enable. By connecting recoverability to the spectral subspace selected by SSL, we characterize when favorable task geometry emerges from a label-free objective. Building on this connection, we derive a direct few-shot NCC guarantee (Thm. 3.4) and characterize the joint geometry of dependent downstream tasks (Thm. 3.5).

AuthorsOpenReview ↗

Continued Response to Reviewer TYE2 (Part 2)

(W2.1) Reviewer: The experiments establish relationships between recoverability, directional geometry, and downstream performance, but provide limited direct evidence that SSL preserves tasks by selecting cross-view-stable spectral modes. In particular, Fig. 4 validates an identity that holds for any whitened representation, while Fig. 7 reconstructs spectral structure from already learned features rather than explaining how that structure emerged. A more direct test of the proposed learning mechanism would strengthen the central claim.

Response: We thank the reviewer for this careful observation. We agree that Figs. 4 and 7 serve different purposes from directly validating the proposed SSL learning mechanism, and we appreciate the opportunity to clarify their roles.

Fig. 4 was designed to empirically validate the exact relationship between recoverability and directional CDNV, using independent data splits to verify that the predicted geometric relationship holds in practice. Fig. 7 was intended to characterize the semantic information retained in the spectral subspace estimated from learned representations, rather than explain how that subspace was learned. While neither experiment alone establishes the proposed SSL learning mechanism, both are important for empirically validating the predictions of our theoretical results (Proposition 3.1 and Corollary 4.2, respectively).

Importantly, Fig. 5 provides complementary evidence for the practical relevance of recoverability: it shows that BrB_r is strongly associated with empirical few-shot NCC error across downstream tasks, representation ranks, and support sizes. Unlike the relationship with directional CDNV, this association with actual classification performance is not an algebraic identity.

We also appreciate the reviewer's constructive suggestion to directly manipulate the augmentation distribution. We have conducted a new controlled augmentation intervention experiment, which tests whether changing the cross-view stability of a semantic factor produces the predicted change in its recoverability. Please see the new Fig. 8 in our revised manuscript and our detailed response to the augmentation experiment below addressing W2.2.

(W2.2 + Q2) Reviewer: Controlled augmentation experiment: Can the theory predict how selectively disrupting task-relevant information through augmentations affects semantic recoverability and downstream transfer?

Response: We thank the reviewer for this valuable suggestion. We have added a controlled experiment that directly tests how changing the augmentation distribution determines which semantic factors are preserved by SSL.

We generate synthetic images (see Fig. 8(a)) with three independent binary factors: shape (circle/triangle), color (red/blue), and background texture (stripes/dots). To generate two views, each factor is independently resampled with a specified probability. We fix the resampling probabilities for shape and texture at 0.020.02 and 0.350.35, respectively, while varying the probability pp for color.

Our theory predicts which factors should be retained. The corresponding operator eigenvalues are λshape=0.9604\lambda_{\mathrm{shape}}=0.9604, λtexture=0.4225\lambda_{\mathrm{texture}}=0.4225, and λcolor=(1−p)2\lambda_{\mathrm{color}}=(1-p)^2 (please check our new Appendix B.4 for details). For a two-dimensional representation, the top-2 subspace should switch from shape and color to shape and texture at p=0.35p=0.35.

To test this prediction, we train lightweight CNN encoders using the whitening-based cross-view alignment objective studied in our theoretical framework (Eqn. 2), across seven augmentation settings and three seeds. All 21 runs recover the predicted leading spectral modes. Evaluating on the same clean, held-out images, we find that color recoverability BcolorB^{\mathrm{color}} drops from approximately 1 to 0 around the predicted transition, between p=0.33p=0.33 and p=0.37p=0.37, while shape recoverability remains approximately 1 throughout. Consistently, color 20-shot NCC error rises from 0% to 50%, whereas shape classification remains nearly perfect.

These results provide direct evidence for our central claim (and reviewer’s comment): the augmentation distribution determines which semantic factors enter the SSL-selected subspace, and their spectral recoverability predicts the resulting downstream transfer behavior. We have incorporated this experiment into Section 5.2 (Fig. 8) and Appendix B.4 of the revised manuscript.

AuthorsOpenReview ↗

Continued Response to Reviewer TYE2 (Part 3)

(W2.3) Reviewer: The analysis relies on idealized population-optimal, centered and whitened representations, whereas practical SSL algorithms have additional architectural and optimization characteristics. The differences require clearer discussion.

Response: We thank the reviewer for highlighting this important distinction. Our geometric identities and few-shot guarantees apply to any centered, whitened representation, whereas the spectral-selection characterization assumes a population-optimal SSL surrogate. We intentionally keep the theoretical framework architecture-agnostic to isolate the role of augmentation-induced spectral structure. Our experiments with diverse pretrained SSL encoders examine how these predictions translate to practical settings, while acknowledging that architectural and optimization effects are not explicitly modeled. We have clarified this scope in the revised manuscript (Sections 5.1 and 6).

(Q3) Reviewer: Applicability to other representative SSL approaches. Which assumptions are necessary for the spectral characterization, and how far do the recoverability-based geometric and few-shot conclusions extend to methods such as MoCo, DINO/DINOv2, and SwAV?

Response: We thank the reviewer for this question. We clarify that the spectral characterization and the recoverability-based guarantees have different scopes.

The spectral characterization is objective-dependent. It is derived for the whitening-based cross-view alignment objective studied in Eq. 2, and extends to the covariance-regularized and prediction-based surrogates analyzed in Appendices C–D. Since MoCo, DINO/DINOv2, and SwAV employ different training objectives, their spectral selection mechanisms are not formally covered by our current analysis.

In contrast, B(F)B(F), its exact relationship with directional CDNV, and our few-shot NCC guarantee (Thm. 3.4) apply to any centered, whitened representation, independently of how it was learned. To demonstrate this broader applicability, we additionally evaluated DINO, SwAV, MoCo v3, and two DINOv2 encoders on all 40 CelebA attributes. Across these five encoders, predicted and observed directional CDNV agree closely (median observed-to-predicted ratio: 1.007–1.014), and recoverability strongly correlates with few-shot NCC error (Spearman correlation of approximately −0.99).

We have added these results in Figs. 18–19 (Appendix B.5), demonstrating that the geometric and few-shot implications of recoverability extend empirically beyond the SSL objectives covered by our spectral analysis.

(Q4) Reviewer: How does (B(F)), which measures linear recoverability, relate to tasks accessible through nonlinear predictors? Can the framework be extended to multiclass classification, such as ImageNet-style recognition, to demonstrate relevance beyond balanced binary tasks?

Response: We thank the reviewer for raising these points. On nonlinear recoverability, please see our response to reviewer YrP9 (W2). On multiclass classification, we extend semantic recoverability by treating each class as a one-vs-rest binary task. For KK equally likely classes, let p=1/Kp=1/K and define the standardized binary label Zk=1Y=k−pp(1−p)Z_k = \frac{\mathbf{1}\\{Y=k\\} - p}{\sqrt{p(1-p)}}. The class-specific recoverability is Bk(F)=∣∣ΠSFE[Zk∣X]∣∣L2(PX)2B_k(F)=||\Pi_{S_F}\mathbb{E}[Z_k\mid X]||^2_{L^2(P_X)}, and multiclass recoverability is defined as Bmulti(F)=1K∑k=1KBk(F)B_{\mathrm{multi}}(F)=\frac{1}{K}\sum_{k=1}^{K}B_k(F). This recovers the original binary definition when K=2K=2.

We evaluate this extension on 20 randomly selected ImageNet-1K classes using a frozen ImageNet-pretrained VICReg encoder. Across five representation ranks ranging from 16 to 256, we estimate recoverability on training images and evaluate 20-way classification on 1,000 held-out images. At every rank, classwise recoverability positively correlates with held-out NCC recall (Spearman's ρ\rho ranging from 0.705 to 0.795). Across ranks, multiclass recoverability increases from 0.591 to 0.812, accompanied by improvements in NCC accuracy (78.1% to 92.7%) and linear-probe accuracy (88.0% to 93.6%).

These results provide empirical evidence that our task-level recoverability measure remains informative beyond balanced binary classification. We have added the multiclass formulation and results in Appendix B.6 (Table 3 and Fig. 20).

AuthorsOpenReview ↗

A Gentle Follow-up and Thanks for Your Feedback

We thank the reviewer again for their constructive feedback.

We understand that this is a busy time for reviewers, with several papers to discuss. With the discussion period ending soon, we would greatly appreciate any feedback on our rebuttal, particularly if any concerns remain unresolved. We would be happy to provide further clarification.

Thank you again for your time and consideration.

Reviewer_jDHeOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies which downstream tasks remain recoverable from self-supervised representations. It defines captured posterior energy as the portion of a task's posterior score contained in the function space spanned by the representation. Under centering and whitening, this quantity determines directional CDNV and controls few-shot nearest-centroid classification and multitask centroid geometry. For a canonical two-view SSL objective, the paper identifies recoverability with the task posterior's overlap with the leading spectral subspace of the two-view operator. Experiments evaluate these relationships on CelebA, dSprites, and 3DShapes using several SSL encoders.

Critical strengths

I find the paper's central question meaningful and the overall framework appealing. A representation can preserve some downstream distinctions while losing others, and captured posterior energy provides a clear way to express this task dependence. Connecting this quantity to directional geometry, few-shot classification, and the subspace selected by an SSL objective gives a useful account of how these pieces fit together. The distinction between task-relevant directional structure and global collapse is particularly helpful.

The empirical evaluation provides support beyond controlled synthetic examples. It includes real CelebA images, several SSL methods, and ImageNet-pretrained encoders. The split-sample comparison in Figure 4 estimates recoverability and directional CDNV on separate subsets, which makes the agreement more informative than measuring both on the same samples. Figure 6 also supplements selected visual examples with results over all eligible attribute triplets. My overall assessment is positive; the concerns below primarily concern the additional insight over prior work and the practical reach of the conclusions.

Critical weaknesses
  1. Earlier work connects directional CDNV to few-shot transfer and multitask near-orthogonality (Luthra et al., 2026), while spectral analyses characterize the subspace selected by SSL (Zhai et al., 2025). I appreciate that this paper connects these perspectives and extends the multitask analysis to dependent labels and joint-centroid geometry (Theorem 3.5). What concrete conclusion does this formulation provide beyond the earlier results?

  2. Estimating recoverability requires target labels (Section 5.1), and the analysis uses large data subsets (Appendix A). This supports the theoretical relationships, but leaves open how stable the estimate would be with only a few target labels. Could different small support sets give substantially different estimates or even change the ranking of representations? This matters if the quantity is meant to inform few-shot transfer rather than only describe it with access to many labels.

  3. The dimension-dependent term in Theorem 3.4 can be large at small support sizes, although Table 1 reports informative bounds at selected ranks. That comparison uses whitened, rank-selected features for the proposed bound and full-dimensional features normalized to unit length for prior bounds. This difference is disclosed, but a comparison with matched preprocessing and rank selection, where the assumptions permit, would help separate the benefit of the new bound from that of the feature transformation.

  4. In the image-classification setting studied here, a model can make predictions once it has a small labeled support set, without ongoing feedback from a person. What additional decision would recoverability help us make? For example, could it reduce the labeling or trial and error needed to choose a representation, compared with simply evaluating candidate models on the available labels? I understand that the contribution is mainly explanatory, but the practical implications for transfer would be clearer if the authors identified a concrete use beyond describing performance after the labels are available.

Questions for authors
  1. What additional prediction does the recoverability formulation provide, particularly for correlated tasks?
  2. How stable are recoverability estimates across small labeled support sets, and are they reliable enough to compare representations?
  3. How much of the improvement in Table 1 remains with matched preprocessing and rank selection?
  4. What decision can recoverability improve over directly evaluating candidate models on the available labels, and could this reduce labeling or model-selection effort?
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer jDHe

We thank the reviewer for their thoughtful and constructive feedback.

(W1 + Q1) Reviewer: What new theoretical insights does the proposed framework offer beyond existing results (Luthra et al., 2026; Zhai et al., 2025) on directional CDNV, few-shot transfer, and spectral SSL analysis?

Response: We thank the reviewer for recognizing the connection between spectral representation learning and directional geometry. We agree that prior work establishes important components of this picture. Our main contribution is a unified, task-dependent characterization of when favorable downstream geometry and few-shot transfer emerge from a label-free SSL objective.

Specifically, prior directional-CDNV analyses relate small V~F\widetilde{V}_F to downstream transfer, but do not establish when SSL should produce such geometry. We address this through captured posterior energy B(F)B(F), which exactly determines directional CDNV under whitening (Prop. 3.1) and yields a direct few-shot NCC guarantee (Thm. 3.4). Crucially, for the population W-MSE objective, we characterize B(F⋆)=BrB(F^\star)=B_r through the overlap between the task posterior and the leading cross-view spectral modes (Cor. 4.2). This connects augmentation stability, task recoverability, directional geometry, and few-shot performance, allowing us to predict which tasks are preserved or discarded as augmentation statistics change (see new Fig. 8).

Unlike the independence-based near-orthogonality analysis of Luthra et al. (2026), our Theorem 3.5 accommodates dependent labels. Specifically, its proof shows that when two tasks are highly recoverable, the alignment of their decision directions approaches the correlation between their labels. Thus, independent tasks develop nearly orthogonal directions, while correlated tasks can retain aligned directions. The theorem additionally quantifies how joint-class centroids approach a factorial hyperrectangle.

We provide new empirical evidence by evaluating this relationship across all 780 pairs of CelebA attributes and five SSL encoders. Fig. 12(b) (Appendix) shows that recoverability-weighted alignment tracks label correlation with regression slopes of 0.77–0.85 for trained SSL encoders, compared with 0.39 for random initialization.

(W2 + Q2) Reviewer: How reliable are recoverability estimates in low-shot settings, and can they consistently predict downstream performance or representation rankings?

Response: We thank the reviewer for raising this important concern. We have added two new complementary experiments (refer to Fig. 12(a) and Fig. 13) to verify the reliability of few-label recoverability estimates.

We evaluate recoverability estimation across five SSL encoder settings and all 40 CelebA attributes, varying the labeled support size m∈5,10,20,50,100m\in\\{5,10,20,50,100\\} and rank r∈16,32,64,128r\in\\{16,32,64,128\\}. Using a fixed whitening transformation and a bias-corrected estimator, we compare estimates against a large, disjoint reference set over 200 random support draws. As shown in Fig. 13, RMSE consistently decreases with mm and increases with rr. At a practical value of m set to 100, RMSE is approximately 0.07–0.11.

We additionally investigate whether few-label recoverability estimates predict downstream task difficulty within a fixed encoder. Using VICReg-CelebA at r=16r=16, we compare recoverability-based task rankings with held-out NCC accuracy across 100 random support draws. As shown in Fig. 12(a), the mean Spearman correlation across all 40 attributes increases from 0.27 at 5 shots to 0.93 at 100 shots.

These results show that recoverability becomes substantially more reliable with moderate label budgets, but also confirm the reviewer's concern that estimates can be unstable in the extremely low-shot regime. Additionally, assessing ranking stability across different encoders remains an interesting direction for future work. We have clarified this in our revised manuscript (L450).

AuthorsOpenReview ↗

Continued response to Reviewer jDHe

(W3 + Q3) Reviewer: How much of the improvement comes from the proposed bound rather than whitening and rank selection? How does the dimension-dependent term affect its usefulness in low-shot settings?

Response: We thank the reviewer for this constructive suggestion. We have conducted a matched comparison to isolate the improvement due to our bound from that due to feature preprocessing. Specifically, we evaluate all prior bounds on the same whitened top-rr features used by our method, selecting the rank that minimizes each bound separately for each shot count mm. The revised Table 2 reports both the original and matched settings.

The matched comparison confirms that preprocessing substantially improves the earlier bounds. Importantly, however, our bound remains consistently tighter even after controlling for these differences. It outperforms the strongest prior bound by 1.3–1.5× for m≥5m\geq 5 and is the only non-vacuous bound at m=1m=1 (0.490 versus 1.13 for the strongest baseline). These results demonstrate that the improvement is not solely attributable to whitening or rank selection, but also reflects the tighter guarantee provided by our analysis.

On the dimension-dependent term: We agree that our bound can become vacuous when r≫mr\gg m. This dependence reflects the tradeoff between capturing additional semantic information and the increased difficulty of estimating class centroids in higher dimensions with limited labels. Our bound explicitly accounts for both effects through BrB_r and the finite-sample estimation terms, allowing it to capture how this tradeoff changes with the number of shots.

We have incorporated this matched comparison and its discussion into Section 5.2 of the revised main text, with detailed numerical results in Table 2, to make this important distinction more explicit.

(W4 + Q4) Reviewer: On the practical utility of recoverability. What actionable decisions can recoverability support in few-shot transfer beyond explaining downstream performance, particularly when labeled data are limited?

Response: We thank the reviewer for raising this important question. While our primary contribution is explanatory, recoverability can also help identify which downstream tasks are best preserved by a fixed SSL encoder, without requiring a separate labeled validation set.

To investigate this, we conducted an additional experiment using VICReg trained on CelebA. We estimate BrB_r using only m∈5,10,50,100m\in\\{5,10,50,100\\} labeled examples per class and compare it with held-out few-shot NCC accuracy across 40 attributes. At rank r=16r=16, the mean Spearman correlation between estimated recoverability and NCC task rankings increases from 0.27 at 5 shots to 0.87 at 50 shots and 0.93 at 100 shots. These results demonstrate that recoverability can provide a useful indication of task suitability without additional validation labels, although estimation variability remains substantial in the extremely low-shot regime.

We have incorporated these results, together with representative attribute-level comparisons, in the revised appendix (Fig. 12(a) and Table 1). While this establishes utility for assessing tasks within a fixed representation, directly evaluating whether recoverability can reduce labeling costs for selecting among candidate encoders remains an interesting direction for future work.

AuthorsOpenReview ↗

A Gentle Follow-up and Thanks for Your Feedback

We thank the reviewer again for their constructive feedback.

We understand that this is a busy time for reviewers, with several papers to discuss. With the discussion period ending soon, we would greatly appreciate any feedback on our rebuttal, particularly if any concerns remain unresolved. We would be happy to provide further clarification.

Thank you again for your time and consideration.

Reviewer_YrP9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to answer the research question: Which downstream tasks survive same-instance SSL, and what geometry and few-shot behavior follow from their recoverability? To this purpose, they introduce the captured posterior energy, B(F)B(F), to measure semantic recoverability. In addition, the paper also theoretically explains why semantic recoverability matters and when SSL makes a task recoverable.

Critical strengths
  1. The topic that this paper focuses on is very meaningful in the community

  2. The presentation is clear, and the mathematical proof seems correct

Critical weaknesses
  1. According to the formulation in the paper, it only considers classification downstream tasks, which is hard to generalize to real-world scenarios. Therefore, I feel the title is too ambitious relative to the contribution of this paper.

  2. The assumption B(F)=E[YF(X)]2B(F)=\mathbb{E}[YF(X)]^2 may ignore the nonlinear case. In the paper, if B(F)B(F) is larger, the recoverability is stronger. However, if we take the instance: if F(x)=ZF(x)=Z and Y=1Y=1 when Z2>aZ^2>a, otherwise -1, then obviously B(F)=E[YF(X)]2=0B(F)=\mathbb{E}[YF(X)]^2=0 if the distribution of ZZ is symmetric to 0. I think this case can be easily solved with a classifier, but with B(F)=0B(F)=0, which means B(F)B(F) may not be able to cover the nonlinear case.

Questions for authors
  1. In 2.1, the paper mentions that the P(Y=1)=P(Y=−1)=1/2P(Y=1)=P(Y=-1)=1/2, which means the data is balanced for binary classificaiton. However, the assumption is too strong in the real world, even for classification tasks. Does the conclusion of the theory still hold if this assumption is violated?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer YrP9

We thank the reviewer for their thoughtful and constructive feedback.

(W1) Reviewer: Given the theoretical analysis is limited to only classification downstream tasks, the title feels too ambitious.

Response: We thank the reviewer for raising this concern. The goal of the paper is not limited to predicting classification performance, but to understand what concepts live in the representation space learned by SSL algorithms, under what conditions they can be recovered with few labeled samples, and how they are geometrically organized in the embedding space. Binary classification provides a mathematically tractable setting for studying these questions, while allowing us to examine multiple semantic attributes and their joint structure.

We believe the current title reflects this broader motivating goal, but we are happy to consider alternatives, such as replacing "tasks" with "concepts" or "semantic tasks" if it can be confused with broader downstream applications. We have replaced “downstream tasks” with “semantic tasks” in our central question for further clarification (refer to L52).

(W2) Reviewer: Can B(F)B(F) characterize task information when downstream labels are recoverable only by nonlinear functions?

Response: We agree with the reviewer that B(F)B(F) does not capture arbitrary nonlinear recoverability. Indeed, in the reviewer's example, the label is perfectly determined by the representation, yet B(F)=0B(F)=0 due to symmetry. This illustrates an important distinction underlying our work: preserving task information is not the same as organizing it into a form that supports simple, sample-efficient downstream learning. Our focus is therefore on linear recoverability, which captures a fundamental aspect of representation quality relevant to linear probing and few-shot nearest-centroid classification. In this setting, B(F)B(F) quantifies the alignment between representations and downstream labels. We have clarified this motivation and the scope of our results in the revised manuscript in Section 6.

(Q1) Reviewer: In 2.1, the paper mentions that the P(Y=1)=P(Y=−1)=1/2P(Y=1)=P(Y=-1)=1/2, which means the data is balanced for binary classification. However, the assumption is too strong in the real world, even for classification tasks. Does the conclusion of the theory still hold if this assumption is violated?

Response: We thank the reviewer for asking this question. Our central geometric conclusions can be extended to arbitrary binary class priors π∈(0,1)\pi\in(0,1), with appropriate adjustments for class imbalance.

Specifically, by standardizing the labels and whitening the representations under the corresponding class distribution, we define prior-adjusted recoverability Bπ=π(1−π)∣μ+−μ−∣2B_\pi=\pi(1-\pi)|\mu_+-\mu_-|^2. We show that the prior-weighted directional CDNV satisfies the same exact relationship, V~π=(1−Bπ)/(2Bπ)\widetilde V_\pi=(1-B_\pi)/(2B_\pi), recovering Prop. 3.1 when π=1/2\pi=1/2. Moreover, the population MSE probe remains aligned with the centroid difference (see Appendix B.2).

We evaluate this extension on six randomly selected CelebA attributes across five SSL encoders, varying class priors from 0.1 to 0.9. As shown in the new Fig. 14, predicted and held-out prior-weighted directional CDNV agree closely (median observed/predicted ratio: 0.996–1.008), whereas the original unweighted CDNV deviates under imbalance.

These results demonstrate that the recoverability–geometry relationship is not restricted to balanced classification. In Section 6, we have added a new clarification that the original few-shot NCC bound (Thm. 3.4) does not extend unchanged to arbitrary class priors.

AuthorsOpenReview ↗

A Gentle Follow-up and Thanks for Your Feedback

We thank the reviewer again for their constructive feedback.

We understand that this is a busy time for reviewers, with several papers to discuss. With the discussion period ending soon, we would greatly appreciate any feedback on our rebuttal, particularly if any concerns remain unresolved. We would be happy to provide further clarification.

Thank you again for your time and consideration.

Thanks for the response. I will keep the score as 3.

Thank you for your continued support of our work!

Best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.