- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes Null-space analysis method for OOD detection. The methods operates by minimizng null space projections for ID data such taht whenever OOD data comes during test, the corresponding projection is larger and easy to detect.
Critical strengths
- The method is simple as it does not rely on auxiliary or curated outlier datasets.
- The result shows that FPR95 gains over conventional baselines is substantial.
Critical weaknesses
- Null space analysis has ben already established research.
- Lack of rigorous evaluation on other OOD benchmarks. Also, important post hoc OOD baselines are missing such as Energy, Mahalanobis, KNN, ViM, ASH, LogitNorm, Adagrad etc.
- Presnetation is poor and hard to follow. Too much of related works section provided.
- No algorithm or codes are provided maiking the claims hard to reproduce.
- No statistical reporting of mean and std has been done on the results raising concerns about stability of the reported values.
- Lack of generality- not applied to other architectures; only tested in WRN.
Questions for authors
- How does NuSA differ from NECO and ViM, both conceptually and empirically?
- Can authors report the histograms of NuSA scores for ID and OOD data?
3 comments
“Null space analysis has ben already established research.”
In regards to the novelty of NuSA, it is difficult to process the “established research” without the inclusion of references. We are unaware of any other work in null space analysis of neural network weights for outlier detection.
Thank you authors for the rebuttal. Several of my concerns still remains unanswered (W2, W4, W5, W6, Q1), and there has not been any update to the paper to reflect the changes as advised in the comments. Due to severe presentation issues and lack of supporting empirical evidences, I do not intend to increase my score.
“How does NuSA differ from NECO and ViM, both conceptually and empirically?”
NuSA is related to but distinct from both NECO and ViM. ViM and NECO are both post-hoc methods. Further, ViM and NECO rely on the PCA of intermediate feature representations of in distribution (ID) samples. ViM analyzes residual magnitudes in the subspace that is orthogonal to the principal subspace associated with the eigendecomposition of the ID feature covariance. They select the top eigenvectors, ranked by eigenvalue, to define the principal subspace. NECO measures a similar value but instead projects directly onto the principal subspace. On the other hand, NuSA is a training time regularization method based on exact computation of the null space of the final layer’s weight matrix. To make the distinction clearer, PCA identifies directions of significant variation in the feature representations, whereas row space and null space analysis identify directions visible or invisible, respectively, to the selected transformation. Our work examines the effectiveness of using the NuSA score as an outlier detector when the model has been trained using the NuSA penalty.