- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a pre-registered invariance audit of post-hoc out-of-distribution (OOD) detectors under exact network symmetries. The authors investigate whether the behavior of OOD detectors depends on the network function or on its internal representation. They apply two transformations that preserve the network predictions: 1) rescaling the penultimate features while inversely scaling the classifier weights, and 2) permuting the feature coordinates with corresponding changes to the classifier weights.
The authors pre-register predictions for 28 constants across 21 detectors and report that 13 out of 16 evaluated detector-symmetry pairs match their predictions. The main finding is that logit-only detectors remain invariant, while feature-based detectors can change significantly depending on whether their internal references are rebuilt or kept frozen. For example, rebuilding the references produces almost no AUROC change, while frozen references can cause large changes even though the network predictions remain the same.
The authors also analyze the three failed predictions, which they attribute to numerical precision, and provide algebraic explanations for how several detector scores change under these transformations. Additional post-hoc experiments on ImageNet-1K are reported to support the findings.
Critical strengths
None
Critical weaknesses
-
Limited experimental coverage: Although the authors pre-register predictions for 28 constants across 21 detectors, only 16 detector-symmetry pairs are included in the main evaluation. The experiments are also limited to legacy CNN architectures such as VGG-16 and ResNet-18, and permutation symmetry is evaluated on substantially fewer combinations than rescaling. Even the additional ImageNet experiments only use VGG-16 and ResNet-50 with a smaller set of detectors. Since the main contribution is a systematic invariance audit of OOD detectors, the limited and uneven experimental coverage makes it difficult to generalize the conclusions to a broader range of detectors and modern architectures.
-
Limited novelty and unclear pre-registration evaluation: The main invariance principle has already been observed in prior work, as the authors acknowledge in Section 1 and Section 2. The main novelty therefore comes from the pre-registered validation and broader empirical analysis. However, Table 1 reports 13 out of 16 correct predictions, while Appendix C states that strictly applying the registered tolerance of (10^{-6}) results in only 11 out of 16. Although the authors explain that some measurements are close to the threshold, it is unclear whether this interpretation was determined before the experiments. Since pre-registration is one of the primary contributions, this ambiguity raises concerns about the validity of the main confirmatory result.
Questions for authors
-
Clarification of the pre-registered evaluation: Table 1 reports 13 out of 16 correct predictions, while Appendix C states that strictly applying the registered tolerance of (10^{-6}) results in only 11 out of 16. Could the authors clarify whether the interpretation of near-threshold measurements was determined before the experiments? If not, why should 13/16 be considered the primary confirmatory result instead of 11/16? Since pre-registration is one of the main contributions, this distinction is important for assessing the validity of the results.
-
Clarification of the experimental generalizability: Although the authors pre-register predictions across 21 detectors, only 16 detector-symmetry pairs are included in the main evaluation, and most experiments rely on legacy CNN architectures. Could the authors clarify to what extent the findings can be generalized to modern architectures such as Vision Transformers? In particular, which conclusions are theoretically guaranteed under the tested symmetries, and which depend on the specific architectures and implementations evaluated?
-
Clarification of the novelty: The authors acknowledge that the main invariance principle was already observed in prior work. Could the authors clarify what new insights this study provides beyond the previously identified rule, apart from the pre-registered evaluation and expanded experimental coverage? This would help me better understand the novelty of the paper.
3 comments
Thank you for the reply. I think the paper may show incremental novelty but still requires significant revision due to the framing and organization of the paper in its current state. I keep my score.
Thank you for the follow-up
Thank you so much for your continued engagement. We accept that the framing and organisation need significant revision. In the revised version, the introduction will state the question directly (whether a detector's output is unchanged under reparameterisations that leave the network's predictions unchanged), lead with the deployment setting in which these transformations occur, and organise the results around the four findings listed in our reply rather than by detector.
Response to Reviewer syNe
Thank you for the three clear questions.
Near-threshold cells. No. The registration defines INVARIANT as “ΔAUROC < 1e-6 and Δscore < 1e-6” and contains no rule for near-threshold cells. The reading of MSP and refit ReAct was made after seeing the data, so the confirmatory result is 11/16. The 13/16 reading will be labelled post-hoc.
Generalisation. Algebraic claims hold for any architecture in which the symmetry reaches the detector input: logit-only scores and rebuilt references are invariant; frozen scalar references move under rescaling; frozen unit-indexed references move under permutation; the closed forms apply. Reachability is head-dependent. In a standard Vision Transformer the final LayerNorm scale and shift give an exact rescaling of the feature the detectors read. We expect the same refit-versus-frozen split but have not tested it. Magnitude and direction of AUROC change are stated only for the networks tested. A post-hoc ImageNet-1K replication (VGG-16-BN, ResNet-50, five OOD sets) reproduces the split.
Novelty beyond the known rule.