acceptodds
Under review as a conference paper at ICLR 2027

The Recursive Manifold Hypothesis for Hierarchical Data Laws

Abstract

Many real-world distributions are organized at more than one resolution. A coarse conditional law captures structure shared by its descendants; conditioning more finely reveals additional variation, while the resulting laws remain related through their common parent. We call this the recursive manifold hypothesis. We formalize the hypothesis as multiscale conditional refinement of probability laws across successive coarse-to-fine relations. The energy distance metric between probability distributions compares the laws model-independently and allows us to decompose their variation exactly. Each parent–child relation has three coordinates: absolute law variation \(V\), the fraction \(F\) revealed by the children, and parent organization \(A\) relative to the expected organization of a random parent map with the same group sizes. The product \(R=FA\) yields a sufficient population criterion for one specified parent–child relation. We estimate this profile from pooled energy-distance U-statistics and quantify uncertainty in \(R\) by an observation jackknife. Experiments center on ImageNet-1k and Amazon Reviews 2023 and include pushforward laws from DINOv2-L and Qwen3-Embedding-4B. Across image, text, and spiking-speech datasets, 13 of 14 relation–representation pairs reveal additional variation that the supplied parent organizes better than a random map with the same group sizes. ImageNet-1k, Amazon Reviews, FGVC-Aircraft, and ImageNet-R satisfy the criterion on two successive relations. Spiking Heidelberg Digits distinguishes structural strength across attributes: speakers are strongly organized within words, whereas the language attribute provides little organization of word laws. Excluding singleton branches preserves these empirical findings. These results suggest objectives and models that respect the hierarchical organization in data laws.

Primary area:
learning theory
Keywords:
manifold hypothesis, recursive manifold hypothesis, hierarchical data laws, multiscale conditional refinement, conditional distributions, energy distance, nonparametric statistics, pretrained representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F6inOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose the recursive manifold hypothesis which says data laws have coarse-to-fine hierarchical organizations. they introduce a model-independent framework using energy distance to decompose variation across parent-child relations. they define a population criterion based on innovation fraction and adjusted parent organization. evaluate this across image text and spiking-speech datasets demonstrating that real-world representations like dinov2 exhibit structured refinement across multiple scales.

Critical strengths

extending the manifold hypothesis to explicitly model relational and multiscale structure using energy distance is a rigorous approach. avoids the need to fit local density models. empirical validation is broad showing that the structured refinement profile holds across vision language and audio modalities

Critical weaknesses

The reliance on pre-specified hierarchies like wordnet or fixed metadata limits the scope. unclear how a representation learner might organically discover or induce these recursive manifolds without heavy top-down supervision. connection to practical representation learning with the ppca demonstration feels somewhat bolted on and disconnected from the model-free energy distance theory presented in the first half. going through the intro i was also under the impression that this is a trivial combination of the manifold and the data has hierarchies hypotheses, no need to call this some new thing

Questions for authors

could this energy-distance decomposition be used during training to encourage learning representations that fit the hypothesis instead of just evaluating them post-hoc? did you measure how sensitive the refinement profile is to the intrinsic dimensionality of the chosen pushforward representation?

Reviewer_o6GcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper presents a hypothesis regarding the organization of real-world data distributions, evaluated both on raw inputs and on representations of pretrained networks. Going beyond the view that data lie on several disjoint manifolds corresponding to categories (the union-of-manifolds hypothesis), it proposes that each category itself, the "parent", is a mixture of child subcategories with their own statistics, where the parent captures structure shared by its children while the children introduce additional variation. This rule can be applied recursively to each child in a tree-like hierarchy.

The paper then builds a probabilistic formalism based on energy distance, which yields four main measurements that are not independent: V is the absolute variation among the child laws at a given level, F is the fraction of this variation revealed by the refinement (i.e., not already resolved by the parents), and A measures how much better the given parent map organizes the children than the average size-matched random parent map. Their product R=FAR = F A serves as the sufficient criterion for structured refinement, with uncertainty quantified by a jackknife.

The paper provides a set of experiments with hierarchical categorizations, including CIFAR-100, ImageNet-1k, ImageNet-R, FGVC-Aircraft, Amazon Reviews and Spiking Heidelberg Digits, and measures these relations both on raw input representations and on features of pretrained models (DINOv2, Qwen3), reporting that 13 of 14 relation-representation pairs satisfy the criterion R>0R > 0 (lower 95% interval endpoint above zero).

Critical strengths

The paper reads well and is clear to follow (beyond a minor notation issue mentioned in the weaknesses). The motivation is clear and interesting, and the probabilistic approach, comparing complete conditional distributions through energy distance, seems to have real potential for revealing this structure and other interesting phenomena. The approach is model-free and tuning-free: the conditional laws are compared directly through pairwise distances between samples, without fitting a density or any other parametric model. With the Euclidean metric, energy distance is a true metric on distributions, so in principle it can detect effects beyond first moments, and it has no hyperparameters that need to be tuned to a specific dataset or representation. The structure of the experiments and the chosen datasets (with subcategories given by released metadata or deterministic rules rather than hand-picked) provide a broad and diverse test bed across images, text and spiking speech, and allow a real examination of the method.

Critical weaknesses

The paper's main contribution beyond the conceptual idea is the introduction of new measurements. I will focus on AA, which seems to be the important measure of how well the parent law explains the variance and, to my understanding, also determines the sign of the criterion RR: since F>0F > 0 for any distinct child laws (and in all reported cases), and since R=FAR = FA, the condition R>0R > 0 is equivalent to A>0A > 0.

This criterion, being better than the average size-matched random parent map, seems far too weak, since roughly half of random regroupings themselves exceed this average. When measuring the accuracy of discrete categorical classification, the usual requirement is that the score of the true class (e.g., its logit) is higher than that of every other class. It seems reasonable to demand, analogously, that each child law PcP_c is closer in energy distance to its own parent law PvP_v than to the law PwP_w of any other parent ww, i.e. D(Pc,Pv−c)<D(Pc,Pw)D(P_c, P_v^{-c}) < D(P_c, P_w) for every ww other than vv, where by Pv−cP_v^{-c} we mean the parent law with cc itself excluded. Reporting the fraction of children satisfying this condition would clarify the strength of the proposed hierarchical organization. This fraction can be read as an "accuracy" of the hierarchy, which is easier to comprehend than a distance whose units are not completely clear.

The main, bigger concern is that the authors compare AA of the trained representations with AA at the input level (Tables 4 and 6). In 4 of the 5 vision relations, AA is larger, sometimes substantially, at the input level (e.g., CIFAR-100: 0.346 for 8-by-8 luminance means vs. 0.171 for DINOv2). This suggests that the measurement does not capture semantic organization as claimed, but rather low-level input statistics, which carry no real semantic meaning. For comparison, the untangling hypothesis of neural manifolds (linear separability of concepts) fails at the input level and holds much better in learned representations. For text (Amazon Reviews), the learned Qwen3 representation does give a higher AA than hashed counts, although at the fine category/middle relation the gap (0.444 vs. 0.323, Table 4) is comparable in size to the vision gaps.

Some other works on hierarchical organization should be discussed and compared. The main one is "The Geometry of Categorical and Hierarchical Concepts in Large Language Models" (Park et al., ICLR 2025), which has very direct relations to this work: it also evaluates WordNet hierarchies in the representations of large pretrained language models, and proposes a precise geometric model of how hierarchical structure is reflected in the representation space.

More minor notes:

  1. Figure 1 appears before the title, which is probably a LaTeX typo.
  2. Sections 2.2-2.3 define several quantities with simple relations to one another (e.g., $\kappa_j $, bj=1−tjb_j = 1 - t_j and tjt_j), which makes it hard to see which measurements are independent. It would be clearer to define fewer quantities, moving those needed only for intermediate steps to the appendix. A table summarizing the notation, how each quantity is measured and what it represents would also improve clarity.
  3. The proof-of-concept PPCA demonstration is presented in the discussion (Section 7.3) and supports the abstract's concluding claim about learning, but appears only in the appendix. If this is a main point, the results should probably appear in the main text (even though I understand that the page limit might have played a role).
Questions for authors

More controls are needed to show that the suggested quantities capture a genuinely semantic phenomenon.

  1. Please run the experiment discussed in the weaknesses, comparing each child with its closest competing parent rather than with the average random parent map, and report the results formulated as an "accuracy" measure.
  2. How do AA and the other measurements (including the one suggested in 1) change across the depth of the network, from the input through the intermediate layers of DINOv2 and Qwen3? Do they systematically improve with depth? How do they compare between strong models and weaker models?
  3. What happens for a network with random weights? Are AA and FF close to their random-parent levels (A=0A = 0, i.e. F=tjF = t_j), or does AA remain positive as for the raw inputs?
  4. Can the hierarchical organization be visualized in some cases, for example by projecting the representations of a parent category to a lower dimension and coloring its different subcategories?
  5. Please discuss the differences and similarities with Park ICLR 2025.
Reviewer_8AttOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper hypothesizes that the probability laws governing real-world data are organized into a hierarchical, coarse-to-fine tree structure: a parent law captures shared invariants, while each descendant level introduces variations that remain constrained by this parent framework. The authors propose a criterion to validate this hypothesis and compute it across various data hierarchies.

Critical strengths

The proposed hypothesis is intuitive and makes good sense, as do the metrics introduced to evaluate it.

Critical weaknesses

However, the fundamental question remains: what is the practical utility of this finding? Even if the hypothesis is empirically validated across several real-world datasets, what are its actionable implications? Although the authors briefly touch upon potential downstream consequences, the paper leaves the reader without a concrete answer. This constitutes the main weakness of the manuscript: the authors need to clearly demonstrate a specific, practical use case or application for this hypothesis.

Questions for authors

See the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.