acceptodds
Withdrawn from ICLR 2027

Exploiting Mode Connectivity for Out-of-Distribution Detection via Low-Dimensional Subspace Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Detecting Out-of-Distribution (OoD) from In-Distribution (InD) samples is fundamental to trustworthy deep learning. To characterize InD-OoD disparities, existing detection methods largely focus on outputs from a single local optimum, or mode, which provides only a limited and potentially noisy view of the underlying InD-OoD data-model relationship. To move beyond this local perspective, we study OoD detection through the lens of mode connectivity, where two independently trained modes can be connected by a low-loss path on InD data, suggesting that this path potentially encodes richer InD and OoD information than any isolated mode alone. We first reveal that models sampled along a low-loss connecting path exhibit nearly consistent InD accuracy but markedly different OoD detection performance, implying that OoD discriminability is not uniformly preserved along the low-loss path. Based on this finding, we propose a low-dimensional subspace modeling approach over the connecting path. Our analysis shows that low-variance principal directions are closely associated with unstable OoD behavior, whereas dominant directions capture stable structure from the path. By removing the former and aggregating the latter, we construct a principal mode, which is compatible with existing detection methods, delivers stronger OoD detection than even mode ensembling, and maintains competitive InD recognition. Our study highlights low-dimensional parameter subspace as a new and effective perspective for understanding and improving OoD detection.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
out-of-distribution detection, mode connectivity, principal component analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KjSwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-hoc OoD detection from the perspective of mode connectivity. Starting from two independently trained endpoint models, the authors learn a low-loss connecting path, sample several models along the path, and analyze their parameter variation with a low-dimensional SVD/PCA-style decomposition. The main empirical observation is that dominant path directions are relatively stable for both InD accuracy and OoD detection, whereas some lower-variance directions can be highly OoD-sensitive. Based on this observation, the paper constructs a single “principal mode” by aggregating only the top singular directions, and evaluates it with logit-, feature-, and gradient-based OoD detectors. On ImageNet-1K with ResNet-50, the resulting model generally improves OoD detection over the endpoint models and often over endpoint ensembling, with little loss in InD accuracy.

Critical strengths
  1. The paper identifies an interesting empirical phenomenon: OoD behavior can vary substantially along a low-InD-loss connecting path, and this variation appears concentrated in relatively low-variance parameter directions. This is a useful extension of prior work that studied variation across isolated modes.
  2. The proposed principal mode is a single network at inference time, so it can be paired with existing OoD scoring methods without ensemble-time overhead. The ImageNet experiments show that this construction can improve OoD detection while largely preserving InD accuracy.
Critical weaknesses
  1. The novelty appears incremental once placed in the broader literature. Mode connectivity and learned low-loss curves are established; low-dimensional weight-space subspaces based on PCA/SVD have been studied in SWAG/Subspace Inference and neural-network-subspace work; single-model weight aggregation is closely related to SWA/model-merging methods; and prior OoD work (e.g., Fang et al., 2024) already showed large OoD variation across independently trained modes. The paper’s genuinely new part is therefore narrower: the empirical link between path-relative low-variance directions and OoD sensitivity, plus the specific principal-mode construction. I do not think the current related-work discussion or experiments sufficiently isolate how much is gained specifically from this new ingredient.
  2. The principal-mode construction is insufficiently justified. After a non-standard centering step, the method directly forms ω* = (1/s) Σ_j σ_j u_j, without adding back a mean/reference model and without constraining ω* to lie on the learned low-loss path. This is not standard PCA reconstruction, and the fact that the resulting vector has the correct dimensionality does not by itself explain why it should be a well-performing neural network. The theoretical result is conditional and explicitly does not guarantee that rapid spectral decay alone makes ω* close to the sampled low-loss models.
Questions for authors

Can the authors justify the principal-mode formula more directly and provide decisive ablations against natural alternatives: (i) standard rank-s PCA reconstruction with the mean/reference model restored, (ii) the subspace/path midpoint, (iii) SWA/weight averaging, and (iv) a reference model plus only the retained principal-direction residual? Please also explain why the chosen centering operation is preferable for characterizing variation across sampled models.

Reviewer_L9WwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that ood detection methods are limited because they rely on a single trained model, and proposes using the low-loss path that connects two independently trained models instead. The authors first show that models sampled along such a path (a quadratic Bézier curve between two ResNet50s fitted on ImageNet-1K) have nearly identical in-distribution accuracy but ood detection performance that varies significantly. They then run PCA/SVD over the sampled models’ parameters. Perturbing along the two dominant singular directions barely affects behavior, while perturbing along a lower variance third direction hurts ood detection, so they build a single “principal mode” from only the top two singular directions, weighted by their singular values. Across Energy, KPCA, and GradPCA detectors, this principal mode beats both endpoint models and their ensemble on ood detection, with in-distribution accuracy staying competitive to single models and no extra inference cost.

Critical strengths
  1. A clear, novel empirical observation.

Models along a low-loss path keep nearly constant in-distribution accuracy, but their ood detection varies. This framing is new, as prior work on multiple modes for ood detection treated modes as isolated rather than looking along connecting paths. The observation is useful and the authors show it is replicated across ood datasets, paths, and mode pairs.

  1. Practical and easy to adopt.

The output is a single standard network, so it works with existing post-hoc detectors without changes and adds no inference cost, unlike ensembling. The authors also show directly why the obvious alternatives fall short: ensembling gains nothing beyond the two endpoints, and naive weight averaging collapses.

Critical weaknesses
  1. The principal-mode construction lacks a clear justification.

ω* centers each sampled model by its own mean weight, omits the mean, rescales by 1/s and depends on a sign convention, so it amounts to a combination of the path’s weights with coefficients set by no stated criterion. Notably, plain weight averaging of path models collapses even with batch-norm recalibration (App. D.2), yet ω* works (itself a weighted combination of these models), and neither the paper nor the appendix explains this or compares against standard PCA reconstruction and alternative sign choices.

  1. The evaluation omits key baselines and selects hyperparameters on test data.

The comparison covers only the endpoints and their ensemble, leaving out single-model alternatives that would show whether the SVD construction is needed: the path midpoint or best path model, standard PCA reconstruction, SWA, and averaging after permutation alignment. In addition, the choice of s is presented via ood test-set performance (Fig. 4) without a held-out validation split. Although a rationale is given, no threshold is fixed a priori, and performance is sensitive to this choice.

Questions for authors
  1. Plain weight averaging of path models collapses even with batch-norm recalibration, yet ω* is also a weighted combination of these models and works well. What explains this difference, and how do the 1/s scaling, the sign convention and the omitted mean each affect performance?
  2. How does ω* compare against alternative single-model baselines such as the path midpoint, the best path model (selected on held-out data), standard PCA reconstruction with the mean restored, SWA and endpoint averaging after permutation alignment?
  3. Fig. 3 only tests positive steps along u₁, u₂ and u₃ without any control, so do random directions orthogonal to the path or negative steps along u₃ produce the same ood degradation?
  4. Since ω* may alter the scale of the batch-norm affine and final layer parameters and hence the logit magnitude, do the Energy gains persist when the temperature is tuned for the endpoint models or when logit norms are matched?

Minor: Figure 1 lacks axis labels/descriptions, so it is unclear what the plotted landscape represents, even with the figure caption.

Reviewer_DYvpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new angle for solving OOD detection, which considers the low-loss path in the parameter space. The paper shows that the model performance may vary across the path. Based on this finding, the paper proposes using the loss path to improve the performance of OOD detection. Specifically, the method forms a new model by aggregating the principal vectors of PCA from the sampled model along the defined path. The formed model can be used to improve different OOD scores. The experiments are conducted on ImageNet-1K OOD detection benchmark with Resnet50.

Critical strengths

I think this is the first paper that focuses on finding a way to aggregate different models to improve the model performance

Critical weaknesses
  1. From my perspective, the paper aims to find or form a good model to improve the performance of OOD detection. Many training-based methods have been proposed for this purpose. Generally, many try to train a model to improve OOD detection and use designed or existing scores, such as energy score, to separate ID and OOD. More related methods can be found in the surveys, such as [1]. However, there is no comparison or discussion with the methods in this scope.

  2. The method appears to require a lot of computing resources, since multiple models trained on the same dataset may be involved. I think the discussion of application and deployment in the real world, especially the cost of deployment, is essential. In the current version, I do not see how this method can be used in the real world.

  3. The model sampling is unclear. In addition, I do not find an ablation study with different ϕ\phi.

[1] Yang, Jingkang, et al. "Generalized out-of-distribution detection: A survey." International Journal of Computer Vision 132.12 (2024): 5635-5662.

Questions for authors

If Bézier curves can be used for ϕ\phi, can spline curves with different basis functions also be considered?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.