acceptodds
Under review as a conference paper at ICLR 2027

A Pre-Registered Invariance Audit of Post-Hoc Out-of-Distribution Detectors under Exact Network Symmetries

Abstract

We present, to our knowledge, the first pre-registered invariance test for post hoc OOD detectors: is detector behaviour a property of the network function or of its coordinates? Two exact symmetries rescale or permute the penultimate representation without changing predictions. Before training, we registered predictions for 28 constants across 21 detectors; across six backbones, 13 of 16 testable rows matched, and the three misses are reported as falsifications. Whether rescaling reaches the penultimate feature depends on the classifier head, not the backbone. Rebuilt references gave ΔAUROC ≈ 10⁻⁸; frozen ones shifted AUROC by up to 0.64, and frozen unit-indexed masks by up to 0.63 under permutation. Algebra fixes whether a detector moves and how far its score shifts, with closed forms matched for two detectors, and approximated for a third, across backbones and a second seed; AUROC loss and its direction depend on checkpoint and OOD set. A post hoc ImageNet-1K replication, on two architectures and five OOD sets, reproduces the refit–frozen split. ViM moved in float32 but was ranking-invariant in float64. As practical guidance, we recommend logit-only scores, or references rebuilt on the deployed checkpoint, and avoiding constants fixed in absolute units, as in NCI and GMM, which move regardless. Predictions, manifests, results and a claim-checking script are released; registrations are verifiable from git ancestry.

Primary area:
interpretability and explainable AI
Keywords:
out-of-distribution detection, invariance, network symmetries, reparameterisation, pre-registration, ReLU rescaling, permutation symmetry, audit

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_syNeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a pre-registered invariance audit of post-hoc out-of-distribution (OOD) detectors under exact network symmetries. The authors investigate whether the behavior of OOD detectors depends on the network function or on its internal representation. They apply two transformations that preserve the network predictions: 1) rescaling the penultimate features while inversely scaling the classifier weights, and 2) permuting the feature coordinates with corresponding changes to the classifier weights.

The authors pre-register predictions for 28 constants across 21 detectors and report that 13 out of 16 evaluated detector-symmetry pairs match their predictions. The main finding is that logit-only detectors remain invariant, while feature-based detectors can change significantly depending on whether their internal references are rebuilt or kept frozen. For example, rebuilding the references produces almost no AUROC change, while frozen references can cause large changes even though the network predictions remain the same.

The authors also analyze the three failed predictions, which they attribute to numerical precision, and provide algebraic explanations for how several detector scores change under these transformations. Additional post-hoc experiments on ImageNet-1K are reported to support the findings.

Critical strengths

None

Critical weaknesses
  • Limited experimental coverage: Although the authors pre-register predictions for 28 constants across 21 detectors, only 16 detector-symmetry pairs are included in the main evaluation. The experiments are also limited to legacy CNN architectures such as VGG-16 and ResNet-18, and permutation symmetry is evaluated on substantially fewer combinations than rescaling. Even the additional ImageNet experiments only use VGG-16 and ResNet-50 with a smaller set of detectors. Since the main contribution is a systematic invariance audit of OOD detectors, the limited and uneven experimental coverage makes it difficult to generalize the conclusions to a broader range of detectors and modern architectures.

  • Limited novelty and unclear pre-registration evaluation: The main invariance principle has already been observed in prior work, as the authors acknowledge in Section 1 and Section 2. The main novelty therefore comes from the pre-registered validation and broader empirical analysis. However, Table 1 reports 13 out of 16 correct predictions, while Appendix C states that strictly applying the registered tolerance of (10^{-6}) results in only 11 out of 16. Although the authors explain that some measurements are close to the threshold, it is unclear whether this interpretation was determined before the experiments. Since pre-registration is one of the primary contributions, this ambiguity raises concerns about the validity of the main confirmatory result.

Questions for authors
  • Clarification of the pre-registered evaluation: Table 1 reports 13 out of 16 correct predictions, while Appendix C states that strictly applying the registered tolerance of (10^{-6}) results in only 11 out of 16. Could the authors clarify whether the interpretation of near-threshold measurements was determined before the experiments? If not, why should 13/16 be considered the primary confirmatory result instead of 11/16? Since pre-registration is one of the main contributions, this distinction is important for assessing the validity of the results.

  • Clarification of the experimental generalizability: Although the authors pre-register predictions across 21 detectors, only 16 detector-symmetry pairs are included in the main evaluation, and most experiments rely on legacy CNN architectures. Could the authors clarify to what extent the findings can be generalized to modern architectures such as Vision Transformers? In particular, which conclusions are theoretically guaranteed under the tested symmetries, and which depend on the specific architectures and implementations evaluated?

  • Clarification of the novelty: The authors acknowledge that the main invariance principle was already observed in prior work. Could the authors clarify what new insights this study provides beyond the previously identified rule, apart from the pre-registered evaluation and expanded experimental coverage? This would help me better understand the novelty of the paper.

3 comments
AuthorsOpenReview ↗

Response to Reviewer syNe

Thank you for the three clear questions.

  1. Near-threshold cells. No. The registration defines INVARIANT as “ΔAUROC < 1e-6 and Δscore < 1e-6” and contains no rule for near-threshold cells. The reading of MSP and refit ReAct was made after seeing the data, so the confirmatory result is 11/16. The 13/16 reading will be labelled post-hoc.

  2. Generalisation. Algebraic claims hold for any architecture in which the symmetry reaches the detector input: logit-only scores and rebuilt references are invariant; frozen scalar references move under rescaling; frozen unit-indexed references move under permutation; the closed forms apply. Reachability is head-dependent. In a standard Vision Transformer the final LayerNorm scale and shift give an exact rescaling of the feature the detectors read. We expect the same refit-versus-frozen split but have not tested it. Magnitude and direction of AUROC change are stated only for the networks tested. A post-hoc ImageNet-1K replication (VGG-16-BN, ResNet-50, five OOD sets) reproduces the split.

  3. Novelty beyond the known rule.

  • Reachability is decided by the head, not the backbone.
  • Closed forms predict exact score movement.
  • Ranking invariance does not imply operating-point invariance (GradNorm ranking stays; transferred threshold moves from accepting none to accepting all).
  • Scalar and unit-indexed references fail under different symmetries, so both tests are needed.
  • These symmetries occur in deployment (quantisation equalisation and model merging).

Thank you for the reply. I think the paper may show incremental novelty but still requires significant revision due to the framing and organization of the paper in its current state. I keep my score.

AuthorsOpenReview ↗

Thank you for the follow-up

Thank you so much for your continued engagement. We accept that the framing and organisation need significant revision. In the revised version, the introduction will state the question directly (whether a detector's output is unchanged under reparameterisations that leave the network's predictions unchanged), lead with the deployment setting in which these transformations occur, and organise the results around the four findings listed in our reply rather than by detector.

Reviewer_BDNrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work presents a pre-registered invariance audit of post-hoc OOD detectors. Rather than proposing a new detector, it asks whether a detector’s behavior is a property of the function computed by the network or of the arbitrary parameter coordinates used to represent it.

Critical strengths
  1. The addressed topic is somehow interesting.
Critical weaknesses
  1. The central research question, whether detector behaviour is a property of the network function or of its coordinates (lines 014-015), is not sufficiently justified. The paper repeatedly contrasts “function” and “coordinates,” but network coordinates (parameters) directly determine the network function, and the network function in turn determines the representations extracted from inputs. The two are therefore not independent objects that can be cleanly separated in the way the research question suggests. This motivating question appears conceptually confused and weakens the paper’s framing.

  2. What this paper actually explores is whether a detector’s output is invariant under exact function-preserving changes of parameterization (gauge transformations). Nevertheless, the significance and insights of the gauge-invariance exploration are not adequately developed. The paper shows that frozen references can move under exact function-preserving reparameterizations while rebuilt references do not, and it recommends logit-only scores, rebuilding references on the deployed checkpoint, and avoiding absolute constants. However, the paper does not establish why gauge invariance is an important property for OOD detection beyond implementation correctness. It is unclear in what respects such gauge sensitivity benefits research. Nor does it show that gauge-invariant detectors are better detectors: logit-only MSP and Energy are invariant in the audit, yet they are often weaker than feature-based detectors such as KNN, ViM, and ReAct in OOD detection performance. What new insights can the paper offer to inform the design of stronger OOD detection methods?

Questions for authors
  1. Could the authors clarify how the function-versus-coordinates dichotomy is meaningful given that parameters determine the function, and justify why this framing is necessary rather than simply testing invariance under gauge transformations?

  2. Beyond implementation correctness, what practical insights does gauge invariance provide for OOD detection, and does it imply that gauge-invariant detectors are preferable or lead to better detector design?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BDNr

Thank you for the expert review. We answer both questions directly.

  1. Framing. Parameters determine the function; they are not independent objects. The claim is precisely the one you describe: whether a detector’s output is invariant under exact gauge transformations that leave the function unchanged. We will use that wording throughout.

  2. Practical content.

  • These transformations occur in deployment (cross-layer equalisation; permutation alignment for merging). Frozen references break silently: DICE AUROC shifts up to 0.64 under rescaling, unit-indexed masks up to 0.63 under permutation, predictions unchanged.
  • Gauge invariance does not require weaker detectors. KNN, ReAct and DICE remain invariant (ΔAUROC ≈ 10⁻⁸) when references are rebuilt on the deployed checkpoint. The guidance is to rebuild or transform references covariantly.
  • AUROC can hide operating-point failure. GradNorm ranking is invariant to eight decimals, yet a transferred threshold moves true-positive rate from 0.0 to 1.0 across the registered scales.
  • Design rule: constants (ridge, threshold, percentile, covariance floor) must be expressed relative to the features or rebuilt on the deployed checkpoint; unit-indexed objects must be permuted with the features. GMM’s absolute floor (reg_covar = 1e-6) violates this and moves regardless of refitting.

Gauge invariance is a reliability property, not a measure of detection accuracy. We will state this explicitly.

Thanks for the responses. The framing needs significant revision, and the importance of introducing Gauge invariance needs more clear and convincing elaboration. I will maintain my score.

AuthorsOpenReview ↗

Thank you for the follow-up

We truly appreciate the reviewer's continued engagement and agree that our current framing does not sufficiently substantiate the role of gauge invariance. To address this, we will substantially revise the relevant sections and provide a concrete deployment-time evaluation. Specifically, we will apply gauge transformations, including cross-layer equalisation and permutation alignment, to an already trained network. We will then demonstrate that the model's predictions are preserved under these transformations, whereas detectors that rely on frozen reference statistics change their scores and decisions. This comparison will give a direct, operational illustration of why detector references must be rebuilt or transformed together with the network.

Reviewer_jeACOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents the first pre-registered invariance audit of post-hoc OOD detectors, testing whether their behavior reflects the network’s function or merely its parameterization. By applying exact rescaling and permutation symmetries that preserve predictions across six backbones, it shows that logit-only detectors and rebuilt feature references are largely invariant, whereas frozen references can change AUROC by up to 0.64. Thirteen of sixteen testable predictions match the results, with the remaining falsifications traced to finite-precision effects rather than failures of the underlying rule.

Critical strengths

This paper combines broad empirical coverage with precise closed-form analyses, controlled refit-versus-frozen ablations, and a post-hoc ImageNet replication. Finally, its transparent reporting of falsifications, numerical artifacts, and reproducibility materials strengthens both the credibility and practical value of its conclusions.

Critical weaknesses
  1. The OOD detection method proposed in this paper is almost exactly the same as "Sharp Minima Can Generalize for Deep Nets". The only difference is that it is used for a different task. Therefore, the innovation of this method is a major issue.

  2. According to the appendix, the result should be 11/16 when strictly following the pre-registered criteria. Why is 13/16 used as the official result in the main text? If borderline cases can be judged through additional interpretation, what is the significance of the pre-registered criteria? Is there a standard basis for such additional interpretation?

  3. The paper does not sufficiently explain why OOD detectors must satisfy function-preserving reparameterization invariance. Some OOD methods inherently rely on features to compute reference statistics, indicating that this dependence is a prerequisite of these methods.

  4. The core mathematical principles presented in this paper have already appeared in previous work, so it is unclear what the substantive novelty of this paper is.

  5. Table 2 only reports the absolute changes in AUROC, but the directions of these changes are unclear.

  6. The Float64 experiment for ViM changes multiple variables simultaneously, so additional controlled experiments are needed.

  7. The paper initially pre-registered 21 OOD detectors, but only 15 were eventually evaluated. Some methods could not be evaluated due to interface or parameter issues. Why were these issues not fixed?

Questions for authors

See Weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer jeAC

Thank you for noting the closed forms and transparent reporting.

Relation to Dinh et al. (2017). We propose no new detector. Dinh et al. used the rescaling symmetry to show sharpness is not a property of the function. We use the same symmetry plus permutation as a test harness for existing OOD detectors. What is new is:

  • the head condition for reachability (stock ResNet-18 exposes no S1 at the detector feature; a two-layer head on the same trunk does);
  • the rebuilt-versus-frozen split (rebuilt at 10⁻⁸, frozen up to 0.64);
  • closed forms for score movement (|c²−1| for SHE, |c−1| for frozen DICE and GradNorm);
  • distinct failures under rescaling (scalar references) versus permutation (unit-indexed masks).

11/16 versus 13/16. Under the registered tolerance the result is 11/16. See the general comment; we will report 11/16 as the confirmatory result.

Feature-based methods must depend on features. The issue is a reference computed under one parameterisation being reused under another. Rebuilt references are invariant.

Table 2 will report signed ΔAUROC. We will add a controlled float64 ViM run that changes only precision. We audited the published implementations at a pinned commit, unmodified; patching them ourselves would have meant auditing our own variants, and changing the code after registration. Every registered detector (21) will receive a one-line status row; excluded detectors and reasons remain in Section 4.1 (e.g., SCALE absent from the pinned OpenOOD commit).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers. Three shared points are accepted.

The confirmatory result is 11/16. Registration commit fe8efff defines INVARIANT as “ΔAUROC < 1e-6 and Δscore < 1e-6” and states that a detector meeting only the first “is recorded in its own category, not as a pass.” MSP (score drift to 1.5×10⁻⁶) and one refit-ReAct cell (ΔAUROC = 2.9×10⁻⁶) therefore do not count as passes. The 13/16 reading was made after seeing the data; it will be reported only as a labelled post-hoc interpretation.

Framing will be corrected: the question is whether a detector’s output is invariant under exact function-preserving reparameterisations (gauge transformations).

These symmetries appear in deployment. Cross-layer equalisation (Nagel et al., ICCV 2019) applies a per-channel form of S1; permutation alignment for merging (Ainsworth et al., ICLR 2023) applies S2. A reference fitted before either step and reused after it is stale. In our audit, frozen references shifted AUROC by up to 0.64 under scalar rescaling while every model prediction stayed unchanged. This motivation will be added to the introduction.

We propose no new detector. The contribution is a falsifiable test that separates implementation choices safe under these symmetries from those that are not.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.