- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed to use a (variant of a) PCA anomaly detector in the representation learned by some foundation model. Because these models are large and flexible, they potentially are able to reconstruct all types of signals, including anomalies, and therefore some adaptations are needed to tune the anomaly detector to the new task at hand. This paper proposes to enhance the PCA by two additional contributions: a scaling of the PCA features (patch-adaptive normality transform) and an additional contrastive loss to adapt the PC components.
The paper is very extensive (including appendices 43 pages!), including numerous datasets and foundation models. I focused on the main text of the paper.
Critical strengths
Good idea to use a PCA-based anomaly detector in the representation of foundation models. (A reference to an anomaly detection approach using PCA in a normal feature space would be nice)
Critical weaknesses
-
A question that I have is that the initial features of the representation on which the PCA is applied, is not per se well scaled. PCA is sensitive to the scaling of features, and will focus on the large-variance directions. In step two of the approach, these PCA features may be rescaled, but in step one the PCA still depends on the relative scaling of the original representation. The contrastive learning often seem to solve it always (as I can see in Table 2): is this PCA initialisation really needed?
-
Another question that I have is the optimization of hyperparameters, like the number of PCA components, or N, or \lambda. If you optimize for the best reconstruction of normal data, then you probably end up with the original problem (that you make the model as complex as possible such that all data, including anomalies, can be perfectly reproduced). Or do you need supervised anomaly data in order to optimize the hyperparameters? This is not clear to me.
Questions for authors
These are the two questions I had in the 'Critical weaknesses':
- how essential is the PCA initialization?
- how are the hyperparameters optimized; do you need supervised information?
5 comments
Response to Reviewer MwGp (Part 2/3)
W2 / Q2: How are hyperparameters optimized? Do you need supervised information? Dose this optimization lead to overfitting to anomalies?
We first clarify that NCL has three main hyperparameters:
- Number of principal directions (for the PCA subspace in NormSpace)
- Number of training patches (few-shot normal patches)
- Fusion weight (learnable per-channel weights in Equation (9))
Then we elaborate in detail on how the hyperparameters are optimized without the supervision information. All hyperparameters are set in a purely unsupervised manner, with no access to anomaly labels during training. Specifically:
- (number of PCA components): We fix across all experiments. This choice is motivated by two considerations:
- Theoretical guidance (Theorems 4, Appendix B.2.2): The top- approximation error is bounded by , and empirically the singular value spectrum of TSFM patch features decays rapidly (most datasets have an effective rank where is the feature dimension). Retaining components typically captures main variance in the normal training patches.
- Sensitivity analysis (Appendix P, "Sensitivity Analysis for Singular Vectors"): We evaluated across three TSFMs (ROSE, TMM, UniTS) on ten datasets. As reported by the results in Appendix P, the performance of our method is relatively stable in the range , with delivering good performance while remaining computationally modest.
- (number of training patches): We obtain training patches from five normal subsequences across all experiments. This extreme few-shot setting is motivated by:
- Theorem 2 (Appendix B.1): The subspace estimation error decreases as . With five samples and typical patch counts (16–32 patches per sample), the total number of patches is 80–160, which, as Theorem 2 shows, is sufficient for stable subspace estimation.
- Empirical validation (Appendix O, "Utility of Few-Shot Normal Samples"): We evaluate sample sizes ranging from 1 to 20 on three TSFMs across ten datasets. Performance improves with sample size but exhibits diminishing returns beyond five samples. We choose only five samples as a practical default that balances performance and data efficiency.
- (fusion weights): These are learnable parameters optimized via backpropagation to minimize the loss on the few-shot normal training samples. These are learnable parameters optimized via backpropagation to minimize the loss on the few-shot normal training samples. They are initialized to be 0.2 (equal weighting) and adapted during training through gradient descent. Specifically, at each training epoch, we compute the gradients where is the total loss (Equation 11) evaluated on normal samples, and update via: \begin{equation} \boldsymbol{\lambda} \leftarrow \boldsymbol{\lambda} - \eta \frac{\partial \mathcal{L}}{\partial \boldsymbol{\lambda}}, \end{equation} where is the learning rate. No anomaly data is involved in this optimization.
Another concern you raised is: "If you optimize for the best reconstruction of normal data, then you probably end up with the original problem (that you make the model as complex as possible such that all data, including anomalies, can be perfectly reproduced)." This concern is addressed by two mechanisms:
- The TSFM is frozen. We do not optimize the TSFM's parameters, which means we do not increase the model's capacity to fit arbitrary patterns. The TSFM's broad pattern-modeling capability (which causes it to reconstruct anomalies well) remains unchanged. NCL only learns a low-dimensional projection and re-weighting of the TSFM's frozen features.
- NCL constrains the feature space to a low-dimensional normal subspace. The PCA step explicitly restricts the model to operate within a -dimensional subspace ( for typical TSFMs). This reduces the effective capacity relative to the full TSFM feature space. The learnable transformation (PatchNT) has only parameters (the linear heads for and ), which is tiny compared to the frozen TSFM (millions of parameters). The contrastive constraint (ConNC) further regularizes this transformation to form compact normal manifolds.
In summary, optimizing for normality learning within a low-dimensional PCA subspace does not increase the model's ability to fit anomalies, instead, it constrains the model to the normal subspace, which is precisely the goal. The frozen TSFM provides the initial broad representations, and NCL "narrows" them to normality.
Response to Reviewer MwGp (Part 3/3)
Additional Clarifications
Reference to PCA-based anomaly detection: Thank you for this suggestion. In the Related Work section of the revised submission, we have added several references to classic PCA-based anomaly detection methods to provide better connections to prior related studies, and at the same time, we also positioned PCA-based anomaly detection in the context of TSFMs by clarifying how NCL is related to the PCA detector via . is a variant of our method NCL that is equivalent to using PCA on top of TSFM's representation space for anomaly detection. This variant retains the sample efficiency and interpretability of PCA while leveraging the powerful representational capacity of frozen TSFMs. The subsequent PatchNT and ConNC in NCL components further augment this simple variant with adaptive transformation and geometric regularization, enabling greater discriminative power for anomaly detection than vanilla TSFMs.
Response to Reviewer MwGp (Part 1/3)
Dear Reviewer MwGp,
We sincerely thank you for the positive assessment and the insightful questions. Below we address each concern in detail.
Feature Scaling and PCA Sensitivity. We agree with you that PCA is sensitive to feature scaling and naturally focuses on high-variance directions. In our case, the patch features are extracted from frozen TSFMs that have been pre-trained on large-scale corpora. Two properties mitigate the scaling issue:
Essentiality of PCA Initialization. The driving force behind the consistent improvement of our contrastive learning (as shown in Table 2) lies in its effectiveness in further enhancing the discriminability of features. However, this does not mean that PCA initialization is unimportant. They simultaneously enhance the anomaly detection capability of the entire module from different aspects. Below, we further extract the comparative data from Table 2 to illustrate the significance of PCA initialization. We conducted three sets of comparisons simultaneously, including Vanilla vs. NCLN, NCLP vs. NCLNP, and NCLPC vs. NCLNPC. These variants mean:
Vanilla vs. NCLN:
NCLP vs. NCLNP:
NCLPC vs. NCLNPC:
From the pairwise comparison results, we can clearly find that PCA initialization consistently improve anomaly detection performance. The PCA initialization is essential in the sense that it provides a sample-efficient starting point for identifying the normal subspace. Without it, the learnable transformation (as NCLP) must discover these directions from scratch using only the reconstruction/forecasting loss, which is less efficient. The PCA initialization provides a "warm start" by concentrating the transformation on the directions that already capture the most normal variation. The contrastive loss then refines these directions into compact, discriminative manifolds.