Distance Does Not Predict Competence: A Two-Axis Decomposition of Out-of-Distribution Failure

Out-of-distribution generalization is usually measured in a single aggregate metric, which assumes that all directions away from the training data have identical effects on extrapolation. We show they do not. We introduce a two-axis evaluation protocol that separates the directions that are governed by a model's imposed structure from those governed by the law it has to learn. Invariant architectures generalize exactly along their built-in symmetries but fail as badly as generic Neural Networks when the law itself must be extrapolated, and the representation that is most accurate in distribution can be the least accurate out-of-distribution. We show that distance from the training data does not predict competence. At matched distance, errors differ by three to four orders of magnitude depending only on the direction taken, and a model given the most distant test set can outperform the same model given closer test sets. On distant queries whose answers the recovered law already provides, six standard uncertainty and distance metrics rank these answerable points as the least answerable. Being far from the data and being undetermined by it are distinct conditions. The shape of out-of-distribution failure, whether a model diverges or flattens, follows whether its representation is bounded, and the pattern holds in Transformer sequence models without hand-built invariances. We argue that a model's competence should be predicted by whether it computes a representation structurally equivalent to the generating law, which training data combined with the chosen representation can signal, and not by taking into account its proximity to the training set. Reliable abstention requires knowing what the hypothesis class determines, a signal that distance metrics lack.