- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies post-hoc OoD detection from the perspective of mode connectivity. Starting from two independently trained endpoint models, the authors learn a low-loss connecting path, sample several models along the path, and analyze their parameter variation with a low-dimensional SVD/PCA-style decomposition. The main empirical observation is that dominant path directions are relatively stable for both InD accuracy and OoD detection, whereas some lower-variance directions can be highly OoD-sensitive. Based on this observation, the paper constructs a single “principal mode” by aggregating only the top singular directions, and evaluates it with logit-, feature-, and gradient-based OoD detectors. On ImageNet-1K with ResNet-50, the resulting model generally improves OoD detection over the endpoint models and often over endpoint ensembling, with little loss in InD accuracy.
Critical strengths
- The paper identifies an interesting empirical phenomenon: OoD behavior can vary substantially along a low-InD-loss connecting path, and this variation appears concentrated in relatively low-variance parameter directions. This is a useful extension of prior work that studied variation across isolated modes.
- The proposed principal mode is a single network at inference time, so it can be paired with existing OoD scoring methods without ensemble-time overhead. The ImageNet experiments show that this construction can improve OoD detection while largely preserving InD accuracy.
Critical weaknesses
- The novelty appears incremental once placed in the broader literature. Mode connectivity and learned low-loss curves are established; low-dimensional weight-space subspaces based on PCA/SVD have been studied in SWAG/Subspace Inference and neural-network-subspace work; single-model weight aggregation is closely related to SWA/model-merging methods; and prior OoD work (e.g., Fang et al., 2024) already showed large OoD variation across independently trained modes. The paper’s genuinely new part is therefore narrower: the empirical link between path-relative low-variance directions and OoD sensitivity, plus the specific principal-mode construction. I do not think the current related-work discussion or experiments sufficiently isolate how much is gained specifically from this new ingredient.
- The principal-mode construction is insufficiently justified. After a non-standard centering step, the method directly forms ω* = (1/s) Σ_j σ_j u_j, without adding back a mean/reference model and without constraining ω* to lie on the learned low-loss path. This is not standard PCA reconstruction, and the fact that the resulting vector has the correct dimensionality does not by itself explain why it should be a well-performing neural network. The theoretical result is conditional and explicitly does not guarantee that rapid spectral decay alone makes ω* close to the sampled low-loss models.
Questions for authors
Can the authors justify the principal-mode formula more directly and provide decisive ablations against natural alternatives: (i) standard rank-s PCA reconstruction with the mean/reference model restored, (ii) the subspace/path midpoint, (iii) SWA/weight averaging, and (iv) a reference model plus only the retained principal-direction residual? Please also explain why the chosen centering operation is preferable for characterizing variation across sampled models.