acceptodds
Under review as a conference paper at ICLR 2027

Alignment, Not Isotropy: Sharp Regret Bounds and a Task-Covariance Correction for Spectral Representation Learning

Abstract

Spectral features can remain optimal under strongly uneven task preferences when they retain the directions most useful to the tasks. We show how transfer depends on this alignment. In a local-task model, expected probing gain depends on the task prior through its covariance . For features, the spectral choice selects the leading eigenspace of , while task-optimal features select that of , where records dependence between views. We derive regret bounds that account for alignment, matching worst-case lower bounds for a flat leading spectrum, and bounds using the leading directions with a spectral-tail term. A spectral gap makes the regret bound quadratic in small anisotropy. We also bound the imbalance from random preference patterns and from averaging independent tasks with an isotropic population covariance. With known , changing one term of the spectral contrastive loss selects task-optimal features. We propose a diagnostic using a labelled task bank and test correction without retraining. Correction reduces synthetic held-out regret from to with tasks when less task-relevant directions dominate. On CIFAR-100, empirical regret falls by – on held-out fine-label tasks using the same images. Correction can nevertheless hurt with a small labelled task bank.

Primary area:
learning theory
Keywords:
representation learning, self-supervised learning, contrastive learning, spectral methods, transfer learning, multi-task learning, learning theory, information theory, minimax rates

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RmyCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the transferability of spectral representations under anisotropic downstream task distributions. The authors show that transfer performance depends on the alignment between spectral features and task covariance, rather than task isotropy alone. They derive alignment-dependent regret bounds, matching worst-case lower bounds, and improved guarantees under spectral gaps. Based on this analysis, they propose a task-covariance-aware correction to the spectral contrastive objective and a diagnostic procedure for determining when correction is needed. Experiments on synthetic data and CIFAR-100 demonstrate reductions in relative regret through task-informed feature selection.

Critical strengths
  1. The paper provides a theoretical analysis of spectral representation learning beyond conventional isotropy assumptions. The alignment-based characterization explains when spectral features remain optimal under anisotropic task distributions.

  2. The paper derives alignment-dependent regret bounds and establishes improved guarantees under spectral-gap assumptions. The synthetic experiments illustrate how task alignment and spectral properties affect transfer regret.

Critical weaknesses
  1. Limited applicability of the theoretical guarantees. The main theoretical results rely on finite-alphabet distributions, local-attribute tasks, and population-level representations. Although extensions are discussed in the appendix, finite-sample guarantees for practical deep representation learning remain largely unaddressed.

  2. Limited novelty and validation of the proposed correction. The correction for frozen encoders is equivalent to classical Fisher discriminant analysis, as acknowledged by the authors. Furthermore, the proposed task-covariance-aware spectral contrastive loss is not evaluated through actual encoder training.

  3. Insufficient evaluation of downstream transfer. The CIFAR-100 experiments use a single pretrained encoder and task split, with correction and evaluation performed on the same images. Moreover, the results report relative probing regret rather than downstream classification accuracy, making it difficult to assess the practical significance of the improvements.

  4. Insufficient validation of the proposed diagnostic procedure. Algorithm 1 frequently recommends unnecessary corrections in the synthetic experiments, particularly when task covariance estimates are inaccurate. The CIFAR-100 experiments do not evaluate this algorithm directly, instead relying on empirical task-bank regret to decide when correction is needed.

  5. Overly complicated presentation and unclear organization. The paper introduces numerous theoretical results, auxiliary definitions, and experimental settings without clearly distinguishing the central contributions from relatively straightforward consequences of standard spectral analysis. The extensive technical details and fragmented experimental presentation make the overall narrative difficult to follow.

Questions for authors
  1. How do the theoretical guarantees extend to practical deep representation learning with finite training samples?

  2. What are the main advantages of the proposed correction over classical Fisher discriminant analysis, and can the corrected loss be evaluated through actual encoder training?

  3. Can the authors provide downstream classification results on unseen images with multiple encoders and task splits?

  4. Can the authors directly evaluate Algorithm 1 on real-world datasets and investigate its reliability under limited task information?

  5. Could the authors clarify the main theoretical contributions beyond standard spectral analysis and improve the organization to better distinguish the central results from auxiliary findings?

Reviewer_PQqHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the transferability of spectral representations under anisotropic downstream task preferences. It shows that the task prior determines average linear-probing gain through ΛB=BΛB⊤\Lambda_B=B\Lambda B^\top, and that the task-optimal kk-dimensional representation corresponds to its leading eigenspace. The authors derive alignment-dependent regret bounds, matching worst-case lower bounds for flat leading spectra, quadratic bounds under spectral gaps, and localized bounds incorporating spectral-tail information. They also analyze task-covariance concentration and propose a task-weighted spectral contrastive loss and a diagnostic/correction procedure. Synthetic experiments examine these theoretical predictions, while a CIFAR-100 experiment evaluates post-hoc correction within a frozen encoder.

Critical strengths
  1. Meaningful theoretical characterization beyond isotropy. The reduction to ΛB=BΛB⊤\Lambda_B=B\Lambda B^\top and the alignment-dependent regret bounds clarify why spectral features can remain optimal under strongly anisotropic tasks. The matching worst-case bounds, quadratic spectral-gap rate, and explicit tail terms strengthen this result.

  2. Careful examination of the theory and its limitations. The synthetic experiments isolate the effects of alignment, task count, and preference patterns. The authors also transparently report harmful corrections with small task banks and unnecessary correction triggers, providing useful evidence about the method's limitations.

Critical weaknesses
  1. The incremental novelty relative to existing spectral and task-aware methods needs clearer justification. The task-optimal eigenspace follows standard variational arguments, and the frozen-encoder correction is equivalent to pooled Fisher LDA. The distinct contribution of the regret guarantees relative to prior weak-symmetry analyses, particularly Tang and Han (2026), should be more clearly established.

  2. The proposed correction and diagnostic lack convincing practical validation. The corrected loss is never trained, while CIFAR-100 uses one frozen encoder, one task split, and shared images, without uncertainty estimates or downstream accuracy; it also bypasses Algorithm 1's theoretical decision rule. The diagnostic frequently proposes unnecessary corrections and can harm performance with small task banks.

Questions for authors
  1. What are the precise improvements over Tang and Han (2026)'s weak-symmetry guarantees? In particular, which results go beyond changing the evaluation criterion from error exponents to relative probing regret?

  2. Can Algorithm 1's actual plug-in decision rule be evaluated on CIFAR-100, including unnecessary correction rates and resulting regret? Could split-bank stability checks or confidence bounds improve its reliability?

  3. Can the authors provide a small-scale end-to-end training experiment using the corrected loss LΛ\mathcal{L}_{\Lambda}, comparing its performance with ordinary spectral training and post-hoc correction?

Reviewer_QQurOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the transferability of spectral representations under non-isotropic downstream task distributions. The authors argue that transfer performance depends on the alignment between learned representations and task preferences, rather than task isotropy alone. They introduce task covariance to characterize the optimal representation and derive alignment-dependent regret bounds, including matching worst-case lower bounds and improved bounds under a spectral gap. Based on these theoretical results, the paper proposes a task-covariance correction method to improve spectral representations, either by modifying the contrastive learning objective or by correcting frozen features. Experiments on synthetic data and CIFAR-100 demonstrate that the proposed correction can reduce transfer regret, although its effectiveness depends on the size of the labelled task bank.

Critical strengths
  1. The paper provides a rigorous theoretical analysis of spectral representation learning beyond the isotropy assumption, with several regret bounds and matching lower bounds.

  2. The proposed task-covariance correction is simple and can improve frozen representations without retraining the encoder.

Critical weaknesses
  1. The theoretical analysis relies on finite alphabets and local-task assumptions, which are quite restrictive compared to practical representation learning settings. It is unclear whether the derived regret bounds still hold for representations learned by modern deep neural networks.

  2. The paper proposes a task-covariance-corrected contrastive loss and theoretically shows that it can recover task-optimal features. However, the proposed loss is not evaluated in the experiments. The experiments only perform post-hoc correction on frozen features, which does not demonstrate whether the proposed training objective actually improves representation learning.

  3. The empirical evaluation is limited. The real-world experiments only use the CIFAR-100 dataset. Moreover, the improvements are measured by task-signal regret rather than downstream classification accuracy. It is unclear whether these improvements translate into better performance on unseen images or other datasets.

Questions for authors
  1. The theoretical analysis focuses on spectral contrastive learning, but the CIFAR-100 experiments use SimCLR with NT-Xent loss. The spectral directions are selected from the frozen SimCLR features rather than directly learned using the analyzed objective. How do the theoretical guarantees apply to this experimental setting? Can the authors provide further justification for this connection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.