acceptodds
Withdrawn from ICLR 2027

Universal Sample-Based Embeddings of Higher-Order Objects

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural networks operate naturally on finite-dimensional vectors, yet many problems require reasoning about higher-order objects such as distributions, functions, and graphs. Such objects capture relationships within a ground set and rarely admit an explicit compact representation. Moreover, we typically want a compact representation not of a single higher-order object but of an entire family of such objects, including members unseen during training. While a few approaches exist for particular object types, none is designed to work across object types. We propose a unified framework that represents any higher-order object as a fixed-size embedding vector learned over a family of such objects. An encoder maps an object, seen only through a finite set of samples, to an embedding, and a domain-specific decoder supplies the training signal by modeling the object conditioned on that embedding. Since the encoder works only from samples, one architecture serves different object types. Because the decoder is a model already suited to single instances of the object type, the framework inherits that model's native operations and inductive biases. We instantiate the framework on distributions, functions, and graphs. In each case we show that a single trained model represents an entire family and generalizes to members unseen during training. Furthermore, we demonstrate the downstream utility of these embeddings: interpolating distributions in embedding space, solving differential equations in embedding space, and predicting unseen edges from a graph's embedding.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, higher-order objects, embeddings, amortized inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mExyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a unified neural network framework designed to learn representations of higher-order objects, such as probability distributions, functions, and graphs.

Critical strengths
  1. The figures are well presented. The figures are clearly plotted, making the experimental results easier to understand.

  2. The experimental setup is clearly described. The authors provide detailed descriptions of the experimental settings, including the datasets, model architectures, and hyperparameters, which helps readers understand the evaluation and supports reproducibility.

Critical weaknesses
  1. The research gap and methodological novelty are unclear. The authors should better explain the limitations of existing methods, why the considered problem is challenging, and what new contributions the proposed framework offers beyond existing encoder–decoder approaches.

  2. The experimental validation is insufficient. Comparisons with representative existing methods and standard DeepSets or Set Transformer baselines are missing.

Questions for authors
  1. Presentation and clarity. The paper is difficult to follow, as several key concepts and the underlying problem setting are not clearly introduced. For example, the definition of "higher-order objects" is rather vague. The authors should provide clearer definitions and a more systematic introduction to the considered learning tasks.

  2. Research motivation: What is the specific research gap addressed by this paper? Why is the considered problem challenging, and what limitations of existing approaches motivate the proposed framework?

  3. Methodological novelty: What are the fundamental differences between the proposed framework and existing approaches such as DeepSets-based encoder–decoder models?

  4. Experimental comparisons: Why are representative existing methods not included as baselines? What evidence supports the effectiveness of the proposed multi-stage encoder compared with standard DeepSets or Set Transformer architectures?

  5. Theoretical justification The paper lacks theoretical analysis of the proposed framework. Could the authors clarify whether any approximation or generalization guarantees can be established? In particular, what justifies the claimed universality, and under what conditions can finite-dimensional embeddings accurately represent the considered objects?

  6. Universality and theoretical justification: What exactly does "universal" mean in the proposed framework? Is it intended to imply universal approximation capability, or only applicability to different object types? What evidence supports the intended claim?

Reviewer_7EFEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a unified framework for learning fixed-dimensional representations of higher-order objects, including probability distributions, functions, and graphs. The authors demonstrate reconstruction of a family of two-dimensional ring-shaped distributions, reconstruction and operator prediction for polynomial and ODE solution families, and graph geometry and adjacency reconstruction on random trees, scale-free graphs, and weighted gridworlds.

Critical strengths
  1. reconstruction performance looks promising 2, The paper provides a coherent framework for constructing finite-dimensional embeddings from sample-based observations of objects that do not naturally admit fixed-dimensional representations. The use of a common encoder-decoder training procedure across distributions, functions, and graphs is conceptually attractive.
Critical weaknesses
  1. core framework substantially overlaps with existing encoder-decoder methods, particularly Neural Processes and distribution-embedding methods. While applying a common formulation across different object types is interesting, the paper does not sufficiently establish which methodological components are fundamentally new.
  2. Despite evaluating three distinct object types, the experimental comparisons are relatively limited. For example, the function experiments lack direct comparisons with Neural Processes or existing function-encoding approaches. Similarly, the graph experiments report reconstruction metrics without comparisons against established graph-embedding or edge-prediction baselines. These missing comparisons make it difficult to assess the practical advantages of the proposed framework.
  3. The graph experiments primarily consider small graphs with 10 or 20 vertices. Table 4 shows that reconstruction performance deteriorates substantially as graph size increases, particularly for scale-free graphs. Furthermore, the number of sampled edges may be large relative to the number of distinct edges, raising questions about the framework's ability to reconstruct genuinely unobserved graph structures.
Questions for authors
  1. What is the fundamental methodological contribution beyond combining a permutation-invariant set encoder with an object-specific conditional decoder? How does the proposed framework differ operationally from Neural Processes and existing distribution embedding methods?
  2. What fraction of distinct graph edges is already present in the encoder's input? Can the authors report performance on genuinely unobserved edges and evaluate substantially smaller edge-observation budgets?
Reviewer_kAbKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a generic recipe for learning fixed-size embeddings of "higher-order objects" (distributions, functions, graphs) over a family of such objects. An object is observed only through a finite (weighted) sample set; a permutation- and cardinality-invariant set encoder (multi-stage Set Transformer, App. C) maps the set to an embedding θ, and a domain-specific decoder — the model one would normally fit to a single object — is conditioned on θ and supplies the loss (Fig. 1). Each sample set is split in half: one half is embedded, the other scores the decoder. Instantiations: (i) distributions, with normalizing-flow or flow-matching decoders on a 2-D "donut" family (Table 1, App. D); (ii) functions, with a cross-attention pointwise decoder on polynomials, damped oscillators (state dim 2–32) and Lotka–Volterra ODEs, plus MLP "operators" acting on embeddings (Table 2–3); (iii) graphs, with the ALLO Laplacian objective lifted to a family by predicting its dual variables from θ, on random trees, scale-free graphs (10/20 vertices) and a weighted 4×4 gridworld (Table 4, 13). The paper claims this is the first framework general enough to represent and learn across families of arbitrary higher-order objects, and shows embedding interpolation/arithmetic, ODE solving in embedding space, and edge prediction.

Critical strengths
  1. S1 — A concrete, previously untried instantiation on graphs (Sec. 5, App. F). Replacing ALLO's per-graph dual-variable matrix with a network predicted from θ turns a single-graph spectral objective into a family-level one, and the evaluation is honest: threshold-free ROC-AUC/AP plus two reconstruction heads, with failure modes (hubs, over-prediction by the medians rule) analysed rather than hidden (Table 4, 13). This is the part of the paper that is clearly new and is why the recommendation is a weak rather than a clear rejection.
  2. S2 — Careful analysis of what the embedding encodes (Sec. 3, App. D.2–D.3). Four principal components explain 99.5% of embedding variance and align one-to-one with the donut's generative parameters (|r| ≥ 0.95); interpolation stays within the family; parallelogram arithmetic holds to 3–4% relative error against a 3.9% sampling-noise floor and breaks exactly where the family is multiplicative (radius × width). The sample-count sweep (Tables 5–6) shows saturation by n≈128. This is consequential because it substantiates the "reusable compact representation" claim with measurements rather than pictures.
Critical weaknesses
  1. W1 — The central novelty claim is contradicted by prior work that the paper mischaracterizes [Fundamental to the stated contribution; fixable by repositioning].
    • Issue and location: Abstract, Sec. 1, Sec. 7 ("the first framework general enough to learn compact representations of higher-order objects"); Sec. 1 on Neural Processes ("learn distributions over functions ... expose no representation of an individual function"); Sec. 6/App. B on GDEs ("applied only to continuous distributions in computational biology").
    • Gap: Generative Distribution Embeddings (Fishman et al., NeurIPS 2025; arXiv May 2025, outside the ICLR 2027 two-month contemporaneous window) already define the distribution instantiation in full generality: a distributionally invariant set encoder paired with any conditional generative model trained with that model's own plug-in loss, with theory (distributional invariance, W2 geometry) that this paper lacks, and their implementation recipe recommends multi-pooling encoders whose pooled output is fed back to the element embeddings — structurally the multi-stage encoder of App. C. Conditional Neural Processes (Garnelo et al., ICML 2018) are exactly the function instantiation: permutation-invariant encoding of (x, y) context pairs into a fixed-size deterministic representation, a decoder queried at target x, and training by random context/target splits; attention-based decoders over the context representation are Attentive/Transformer NPs. Using an MLP on such representations as an operator is likewise established. What remains as new is the observation that the same template applies across object types plus the graph instantiation (S1).
    • Impact: Contribution is judged against the paper's own framing. With GDE and CNP correctly positioned, the paper is a cross-domain demonstration of a known amortized inference template, not a new framework, which caps Informativeness and the rating.
    • Suggested remedy: Withdraw "first"; state precisely what differs from GDE (if anything beyond swapping the decoder family) and from CNP/ANP; move the graph instantiation and the cross-domain analysis to the centre of the contribution.
  2. W2 — The evaluation cannot support "a unified framework matches specialized methods" because two of three domains have no baselines and all families are toys [Fixable].
    • Issue and location: Tables 2–3 (functions) and Table 4/13 (graphs) report only the method's own errors. Table 1 compares only to KDE and a method-of-moments fit, not to Neural Statistician or GDE, the closest prior methods. All families are synthetic and small: 2-D rings, cubic polynomials, ODEs with ≤ 6 parameters, graphs with 10–20 vertices; 3 seeds.
    • Gap: The ODE "solution operator" maps 6 clean parameters to an embedding; the obvious baseline — an MLP (parameters, t) ↦ z(t) trained directly — is not run, so it is unknown whether the embedding adds anything. On functions, a CNP/ANP with identical encoder capacity is the natural control for the embedding column. On graphs, Sec. 2 says the embedding "reliably separates edges from non-edges", but scale-free n=20 gives ROC-AUC 0.847, AP 0.625 and medians-head F1 0.331 (Table 13), and the gap to the degree oracle (F1 0.613) shows the learned head, not just the geometry, is the bottleneck.
    • Impact: Without controls, the headline claims about inherited capabilities and generality rest on absolute numbers that have no reference point, and the paper gives no evidence that generality does not cost accuracy relative to specialized models.
    • Suggested remedy: Add CNP (and ideally ANP) on the Table 2 families, GDE or Neural Statistician on Table 1, and the direct parametric MLP on the ODE operator task; soften the graph language to match Table 13; one non-synthetic family in any domain would materially strengthen the paper.
Questions for authors
  1. (Clarification, drives W1) Beyond allowing non-generative decoders, what in the framework or the encoder is not already specified by GDE (Fishman et al. 2025, incl. their App. A.4 multi-pooling encoder recipe)? Please state the difference concretely.
  2. (Clarification, drives W1/W2) How does the function instantiation differ from a Conditional Neural Process with an attention decoder? Can a CNP/ANP baseline with matched encoder be reported on the Table 2 families, and a direct MLP (parameters, t) ↦ z(t) baseline on the ODE operator task?

Comments on the paper

1 comment

We thank the reviewers for their time and valuable feedback. After careful consideration, we have decided to withdraw the paper to address the comments and further improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.