acceptodds
Under review as a conference paper at ICLR 2027

Distance Does Not Predict Competence: A Two-Axis Decomposition of Out-of-Distribution Failure

Abstract

Out-of-distribution generalization is usually measured in a single aggregate metric, which assumes that all directions away from the training data have identical effects on extrapolation. We show they do not. We introduce a two-axis evaluation protocol that separates the directions that are governed by a model's imposed structure from those governed by the law it has to learn. Invariant architectures generalize exactly along their built-in symmetries but fail as badly as generic Neural Networks when the law itself must be extrapolated, and the representation that is most accurate in distribution can be the least accurate out-of-distribution. We show that distance from the training data does not predict competence. At matched distance, errors differ by three to four orders of magnitude depending only on the direction taken, and a model given the most distant test set can outperform the same model given closer test sets. On distant queries whose answers the recovered law already provides, six standard uncertainty and distance metrics rank these answerable points as the least answerable. Being far from the data and being undetermined by it are distinct conditions. The shape of out-of-distribution failure, whether a model diverges or flattens, follows whether its representation is bounded, and the pattern holds in Transformer sequence models without hand-built invariances. We argue that a model's competence should be predicted by whether it computes a representation structurally equivalent to the generating law, which training data combined with the chosen representation can signal, and not by taking into account its proximity to the training set. Reliable abstention requires knowing what the hypothesis class determines, a signal that distance metrics lack.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
out-of-distribution generalization, extrapolation, distribution shift, uncertainty estimation, deep ensembles, out-of-distribution detection, equivariance, symbolic regression, representation learning, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T1kyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores the generalization properties of neural networks when extrapolating to data that differ from the train set along different directions. They investigate this setting under a variety of symmetry conditions present in the training data generating process.

Critical strengths
  • The paper tackles a relevant problem regarding the geometry of OOD generalization in neural networks
Critical weaknesses
  • The paper is very clearly heavily written by Claude. While this alone is not disqualifying as a weakness, several AI prose issues (e.g. the introduction of definite descriptions like "The LayerNorm Model" on L062 without introducing their referents first, claudish sentence structures like L082-084 that bloat the paper, etc...) make the paper very difficult to interact with. This severely inhibits my ability to engage with the content of the paper.
Questions for authors

N/A

1 comment

Thank you for the review. We take the point on the prose. The paper discloses that generative AI was used for drafting and polishing, and the responsibility for how it reads is ours. Your two examples are real defects:

  • L062 refers to "the LayerNorm model" before §4 introduces the four architectures.
  • L082-084 puts the paper's main reading, a forward reference to §8 and the propagation rule of a companion paper into one dense passage.

In the revised PDF, now uploaded, we rewrote the introduction and §2 so that every model and term is introduced before it is used, with one claim per sentence, and shortened dense passages in §4 to §10. Changes are marked in blue.

Since the prose kept you from the content, a plain summary may help. A model can leave its training data in different directions. Along some, an architectural constraint such as rotation invariance fixes the answer. Along others, the model must extrapolate a law it learned from data. We measure the two separately and find:

  1. Invariant models are exact along their symmetry and fail as badly as generic networks along the law (§3).
  2. Two test sets at the same distance from the training data can differ in error by three to four orders of magnitude, and the most distant set can be the easiest (§4).
  3. Whether a model diverges or flattens outside the data follows whether its internal representation is bounded (§5).
  4. Ensemble disagreement and five other standard scores rank the queries whose answers the data already fix as the least answerable (§6).

We would be grateful if you would look at the content again with the revised text and engage with the paper.

Reviewer_rMHXOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies out-of-distribution (OOD) generalization through the lens of task structure. This means taking data that obeys known structural constraints (e.g., a symmetry) and studying the extent to which a trained neural network respects this structure when extrapolating to OOD data. The paper studies this both in terms of generic models and models with architectural constraints that make them more aligned with the data. The primary example of this is architectural adaptations that make a model naturally invariant to a symmetry in the data. One of the primary messages of the paper is that models that can generalize to one type of OOD shift (because of architectural adaptation for example), may completely fail on another type of distribution shift. The paper’s goal is to correct a perceived assumption that OOD performance can be captured by a single quantity. The paper also explores (i) whether distance from the training distribution predicts OOD performance (it does not), (ii) how asymptotic properties of an architecture impact OOD performance (asymptotic properties of the architecture need to align with asymptotic properties of the model), and finally (iii) whether model confidence can be used to predict OOD performance (it cannot). All of these experiments are run with synthetic and physics motivated problems.

Critical strengths
  • An interesting problem and effective experimental design: Understanding OOD generalization through controlled experiments is a valuable research direction. Designing experiments to disentangle different types of generalization also seems particularly useful (e.g., incorporating certain OOD priors into the model architecture, while leaving others to be learned). Though the reviewer believes that by focusing on small-scale synthetic experiments, the paper misses out on some qualitatively different phenomena that may only appear for large, complex training sets, he is also sympathetic with the need to start with more interpretable tasks where OOD directions are more obvious.
Critical weaknesses
  • A single quantity captures out-of-distribution performance: The reviewer would dispute one of the central premises of this work: that there is a belief that out-of-distribution performance can be captured by a single quantity. This reviewer cannot remember this being claimed anywhere in print or in verbal conversation. Indeed, works such as [1] explicitly look at different ways of moving data out of distribution and find varying effects depending on which is chosen. The only example that the reviewer can think of that tends toward this idea is the accuracy-on-the-line phenomenon [2]. However, this concerns training at large-scale and is not relevant to the synthetic experiments in this paper.
  • Clarity: This paper is very hard to read. There are quite a few instances where vague claims are made that are never subsequently explained. Here is an example “The claim predicts its own exception, and it is testable.” In this case, neither the claim, the exception, nor in what sense it is testable are ever directly explained. It may be that in many of these instances, the reader can infer what is meant, but this is a lot of work and makes for slow and laborious reading. In other cases, notation is mentioned that is only defined later (e.g., dd on line 064). Finally, the reader was confused by certain basic terms. For instance, what does it mean for a representation to be “bounded”? The reviewer’s interpretation is that this means that the architectures tends to be asymptotically bounded as input size grows, but this may be incorrect. Either way, the meaning of this term should be defined.
  • The small-scale experiments probably exclude the interesting parts of OOD generalization: Many of the most interesting results in OOD generalization tend to only happen at large model/training set sizes. The reviewer is therefore somewhat skeptical that the small-scale experiments that were run in this paper are sufficient to convincingly prove the main points. At the very least, the conclusions of this work should be explicitly confined to small data/network settings.

[1] Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations." arXiv preprint arXiv:1903.12261 (2019). [2] Miller, John P., et al. "Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization." International conference on machine learning. PMLR, 2021.

Questions for authors
  • What does ‘law’ mean in this paper?
  • In the simple settings of this paper, the notion of distance between training examples is mostly straightforward. In more realistic problems, this is not so obvious. How do these results generalize to settings where it is not clear what the correct notion of distance is (e.g., computer vision, language)?
  • What does it mean for a representation to be “bounded”?
  • What does ‘the axes come apart mean’ in (1) in the introduction?
  • What is dd in (ii) in the introduction?
  • I don’t understand what the symbolic search is baseline is. Can more information be provided?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer rMHX, part 1 of 2 (weaknesses)

Thank you for a careful review. We answer the weaknesses here and the questions in a second comment.

W1: the premise. We agree that nobody claims in print that OOD performance is a single quantity, and our wording overstated the point. What we meant is narrower, and the revised introduction says it precisely:

  • A reporting practice. Benchmarks summarize a shifted set by an aggregate. ImageNet-C reports per-corruption errors, but its headline metric, mCE, averages over corruption types and severities (Hendrycks & Dietterich, 2019). Our protocol splits the directions differently, by what governs the target along each one: an imposed group or the learned law. Because the split is defined by the architecture's imposed structure, it says in advance along which directions a given architecture is exact.
  • An explicit premise. Distance- and density-based OOD detectors assume that distance from the training data signals failure (Lee et al., 2018; Sun et al., 2022). §4 tests that premise.

The revision cites Hendrycks & Dietterich (2019) as the closest precedent for direction-resolved evaluation. On accuracy-on-the-line, Miller et al. (2021) themselves report shifts where the correlation is weak, including some CIFAR-10-C shifts and Camelyon17-WILDS, and Teney et al. (2022) find an inverse correlation on Camelyon17-WILDS. Instantiation III is a controlled case of that inversion: the RBF model is the most accurate in distribution and the least accurate on the law axis in all 24 batches. We think this connection holds at small scale too, and Related work now makes it.

W2: clarity. We accept this. The examples you give are fixed as follows.

  • "The claim predicts its own exception, and it is testable" (§4). The claim is the reading of §8: competence follows whether the model's representation is structurally equivalent to the law. If that is right, distance measured in the model's own feature space should track error when that space carries the law, and fail when the representation saturates. We tested this. The feature-space scores recover the ordering for three architectures and fail on the tanh model. The sentence is replaced by this explanation.
  • L064 used the input dimension dd before §4 defined it. It is now defined in place.
  • "Bounded" is defined in our answer to Q3. The definition is now in the introduction and §5, with the bounds in Appendix J.

We also rewrote the introduction so that every term is defined before its first use.

W3: scale. We agree the conclusions should be confined explicitly, and the abstract and Limitations now say so. Our claims are of two kinds.

  • Counterexamples. The dissociations and the inversion are controlled cases. One case where distance orders outcomes backwards shows that distance is not a reliable predictor in general. It says nothing about how often this happens at scale.
  • Structural arguments. Proposition 1 and the piecewise-affine divergence of §5 depend on the architecture class, so they hold at any width and depth.

Whether large models trained on large data come to represent the law, and so remove the law-axis failure, is open, and our experiments do not address it. Limitations now says this.

AuthorsOpenReview ↗

Reply to Reviewer rMHX, part 2 of 2 (questions)

Q1: what "law" means.** The law is the scalar function gg in f(x)=g(ρ(x))f(x) = g(\rho(x)) of §2, the part of the target the model must learn from data: g(r)=r−2g(r) = r^{-2} in Instantiation I, cos⁡2θ\cos^2\theta in II, and the Lennard-Jones potential V(r)V(r) in III and §4. "Generating law" means the true data-generating mechanism. The term is now defined in the second paragraph of §1.

Q2: distance where the right metric is unclear. This is close to our conclusion. In vision and language there is no metric known to be correct, and §4 shows why the choice matters: distance tracks competence only to the extent that the space it is measured in already encodes the law. A learned feature space is the natural candidate. §4 shows it works for three of four architectures and fails where the representation saturates, and §6 shows it fails on determined queries. What transfers to realistic settings is the protocol. A known group action labels the symmetry axis exactly (rotations and translations in vision, permutation in sets and sequences, as in §7), and a held-out range of a labeled factor gives the law axis. A molecular instance for example.

Q3: what "bounded" means. Your reading is close, with one correction: bounded in the magnitude of the input, not in the number of inputs. We call a representation bounded when the features the output head reads stay in a bounded set however large the input becomes, sup⁡x∥ϕ(x)∥<∞\sup_x \lVert\phi(x)\rVert < \infty.

  • tanh: every coordinate lies in (−1,1)(-1, 1).
  • LayerNorm of width mm with gain γ\gamma and bias β\beta: ∥ϕ(x)∥≤m max⁡i∣γi∣+∥β∥\lVert\phi(x)\rVert \le \sqrt{m}\,\max_i |\gamma_i| + \lVert\beta\rVert for every input.
  • Unnormalized ReLU: eventually affine along every ray (Hein et al., 2019), so ∥ϕ(tx)∥\lVert\phi(tx)\rVert grows linearly in tt unless the slope vanishes, and the prediction diverges.
  • Pre-LN Transformer: both nonlinear branches read LayerNorm outputs and are bounded. The residual stream also carries the linear input embedding, which is unbounded. In the task of §5 it reaches the output only through the mean of the particle embeddings, which has zero expectation over uniformly distributed angles and does not grow with the target. This caveat is now in Appendix J, with a pointer from §5.

Q4: "the axes come apart". The error along the symmetry axis and the error along the law axis are separate quantities, and one does not predict the other. The invariant model of Table 1 is exact along the symmetry axis and as wrong as the generic network on the law axis. A single OOD number averages the two, so it depends on the mix of directions in the test set. Item (i) now says it this way.

Q5: item (ii) of the introduction. Item (ii) summarizes §4, the paper's central negative result:

  • Two test sets at the same nearest-neighbour distance from the training data, one compressing the tetrahedron and one expanding it, have errors that differ by 744× to 6787×.
  • The most distant test set gives the LayerNorm model an error of 0.28, the nearest OOD set an error of 3679.
  • When the input dimension grows from 2 to 32, the catastrophic direction becomes the nearer of the two, and the error ratio holds.
  • Feature-space distance recovers the ordering for three of the four architectures.

The revised introduction names the four architectures before this item and splits it into shorter sentences.

Q6: the symbolic search baseline. The revision calls it by its standard name, Symbolic Regression (SR), with references. It is a reference point: it shows the target is recoverable from the raw data and what exact extrapolation looks like. We do not offer it as a competing method (Limitations). The procedure is standard bottom-up enumerative synthesis.

  • Grammar. The raw input coordinates and the constant 1; unary operators sq, neg, abs, sqrt(|·|), sin, tanh, exp, log(|·|) and inv; binary operators add, sub, mul and div. No operator mentions the radius, so the search must build x2+y2x^2 + y^2 itself.
  • Enumeration. At each level every unary operator is applied to the pool and every binary operator to pairs from it, expanding the 420 smallest expressions. Instantiation I uses three levels. For the four-coordinate input of Instantiation II we use two full levels followed by a unary-only level, to stay within memory.
  • Deduplication. Two expressions whose outputs agree on a probe set of inputs, after removing scale and offset, count as one, and the smaller is kept.
  • Selection. Fit a⋅e+ba \cdot e + b by least squares and minimize nlog⁡MSE+∣e∣log⁡nn \log \mathrm{MSE} + |e| \log n, fit error plus description length.

In Instantiation I it returns inv(add(sq(x), sq(y))), that is 1/(x2+y2)1/(x^2 + y^2). In Instantiation III we did not run the search. The grammar can express the Lennard-Jones potential, but reaching it takes about six levels, more than the enumeration holds in memory, so Table 1 leaves that entry empty. This description was only in the supplementary code (symbolic_search.py); it is now in Appendix H.

Reviewer_KspuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors decompose out-of-distribution failure across two axes - symmetry and law. Through this framework, they make the point that distance is not a reliable metric for assessing degree of failure. The authors start by explaining this on a rotation invariant model, continue to show their insights on different neural networks to strengthen their claims. They also show that representational boundedness can explain ood failure and further reinforce that models can be confidently wrong on OOD inputs.

Critical strengths

This work contains two strong contributions that can be of utility to the community:

  1. Using representational boundedness to understand how neural networks behave when faced with samples that are ood.

  2. Showing that two shifts are the same distance can cause produce very different errors, thereby making the point of the paper.

Beyond these two contributions, this work takes a step towards breaking down out of distribution failure - how one can, in the future, quantify failure beyond simple accuracy on shifted sets. I think it further pushes the community to think about breaking down ood failure. I believe this work, while limited in novelty, is a good empirical and insightful paper that pushes towards better understanding how models fail under distributional shifts.

Critical weaknesses
  1. The authors show that representational/feature space distance does track error on distributional shifts. Would the authors consider toning down their message regarding distance?

  2. While the synthetic and toy-like settings are a limitation, could the authors kindly explain how on would go about expanding on their analysis through more realistic settings? For future work.

Questions for authors

I have any relevant questions above.

1 comment

Thank you for the review and for the two contributions you single out.

W1: toning down the distance message. Yes. The paper's own feature-space results qualify the headline, and the abstract states the claim more broadly than §4 supports. The evidence, by space:

  • Input space. At matched nearest-neighbour distance, error differs by 744× to 6787× across four architectures. The most distant regime gives the LayerNorm model an error of 0.28, against 3679 for the nearest OOD regime, in all 20 model-seed combinations.
  • Feature space. The neighbour score on the pooled representation orders the twelve regimes at ρ ≥ 0.92 for the ReLU, LayerNorm and Transformer models (Table 7). It fails on the tanh model, which saturates the most, and there it ranks the catastrophic compressed regime as less OOD than the benign expanded one.
  • Determined queries (§6). On the radial chirp, feature-space Mahalanobis and feature-space kNN score AUC 0.31 for telling novel from determined queries, below chance in all 8 batches, like the four other scores.

So the claim we can defend is this: input-space distance does not predict competence, and feature-space distance predicts it only where the representation still carries the law. In the revision:

  • The abstract, the introduction and the title of §4 now say "input-space distance", and the abstract states the feature-space result next to the headline, in addition to §4 and Limitations.
  • For the camera-ready, we propose the title "Input-Space Distance Does Not Predict Competence: A Two-Axis Decomposition of Out-of-Distribution Failure".

W2: extending the analysis to realistic settings. We have not run this yet; it is the next step we plan. The protocol needs two things: a group action known in advance, which labels the symmetry axis exactly, and reference labels beyond the training range on the law axis. Real data can supply both. The most direct extension is a molecular benchmark such as rMD17:

  • Symmetry axis. Rotate held-out molecules. The energy is rotation-invariant by physics, so this axis needs no new labels.
  • Law axis. Hold out configurations by a physical coordinate, for example compressed bond lengths or the highest-energy configurations, and use the dataset's reference energies as labels.
  • Models. An equivariant potential (NequIP or MACE), SchNet, and a plain Neural Network on coordinates.

From Instantiation III we expect the equivariant and SchNet models to be exact under rotation while the plain network degrades, and all three to degrade on the law axis. Whether SchNet degrades the most depends on whether its RBF centres cover the held-out range, which this design tests directly. The same recipe applies to vision, with a group-equivariant CNN (Cohen & Welling, 2016) and a held-out range of a labeled generative factor, and to sequences, as §7 does with permutation. A short version is now in Limitations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank the three reviewers. The reviews agree on four points, and the revised PDF, now uploaded, addresses each. Every change is marked in blue.

  • Scope of the distance claim (Kspu, rMHX). The paper refutes input-space distance. Feature-space distance recovers the ordering for three of the four architectures of §4. It fails on the saturating tanh model, and on the determined queries of §6, where feature-space Mahalanobis and kNN both score AUC 0.31. The abstract, the introduction and the title of §4 now say "input-space distance", and the abstract states the feature-space result next to the headline claim.
  • The premise (rMHX). We reworded it. Instead of attributing a belief to the field, the introduction describes a reporting practice and an explicit premise of the OOD-detection literature, and Related work cites Hendrycks & Dietterich (2019), Miller et al. (2021) and Teney et al. (2022).
  • Clarity (rMHX, T1ky). We rewrote the introduction and §2 so that every term is defined before its first use: law, symmetry axis, law axis, bounded representation, the four architectures, and the symbolic search. The sentences the reviewers quote (L062, L064, L082-084, "The claim predicts its own exception") are replaced. The grammar and selection criterion of the symbolic search moved from the supplementary code into Appendix H, and the bounds behind "bounded" into Appendix J.
  • Scale (Kspu, rMHX). The abstract and Limitations now confine the conclusions to the settings tested, and Limitations describes how the protocol transfers to a real interatomic-potential benchmark.

No experiment and no reported number changed. The symbolic search is now called Symbolic Regression (SR) throughout, with references, and the revision uses the tenth page allowed during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.