acceptodds
Under review as a conference paper at ICLR 2027

Interpretable Interaction Dictionaries for Mixture Recovery in Discrete Data

Abstract

Unsupervised representation learning typically maps observations into spaces where latent structure becomes easier to recover, but the resulting coordinates are often difficult to relate to observable properties of the data distribution. We study an alternative for discrete data in which representation coordinates are explicit binary interaction moments. Rather than seeking only a set of statistics sufficient for a downstream recovery task, we aim to construct a structured and human- interpretable representation whose higher-order coordinates remain traceable to simpler dependencies in the data. We instantiate this principle with ODIN, which uses an information-geometric criterion to select a nested hierarchy of interaction atoms, and apply the resulting representation to mixtures of product-Bernoulli distributions. Mixture parameters are estimated by constrained moment matching in the learned moment space. For any fixed dictionary, we show that full column rank of the lifted-moment Jacobian provides a local identifiability certificate, and that enriching the dictionary cannot decrease its rank. This separates the statistical role of dictionary construction from the geometric conditions governing recovery. We evaluate how the learned interaction representation affects parameter recovery and Jacobian geometry on synthetic mixtures, and assess its usefulness on real- world binary clustering benchmarks

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Unsupervised learning, mixtures of distributions, Information geometry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4wNcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

Unsupervised representation learning can be considered as an unsupervised latent mixture model without any ground truth latent factors known, where dictionary learning (DL) is important branch of it. This paper claimed they focuses on a special DL case that estimation the mixture of product-Bernoulli mixture (binary latent class) model. Their proposed ODIN select a set of variable subsets with a greed method, and layer search that ranks the candidate subset by KL improvement. Then fits the mixture weights and means by least square. Their theory results show full column rank of Jacobian of this moment implies local identifiability. They provided synthetic and real experiments.

Critical strengths
  1. This is true research problem that the reviewer enjoys the time spent on reading, regardless of the reviewer’s final judgement of significance or contribution.

  2. The paper clearly separates the topic of which moments to use, and fixed set of moments implies local recovery.

  3. The paper is transparent about their limitation: random dictionary can also reach full rank. And the proxy is not reliable at higher order.

  4. Results on mushrooms clearly improvement against baseline, although is not best on the KL metric.

Critical weaknesses
  1. The reviewer believes the problem is classical that this paper is not well positioning itself against known results. For binary data, each atom moment is an off-diagonal entry of the order-S tensor. This work is a partially observed CP decomposition for binary latent class model. The aforementioned type of model has classical identifiability results, the Kruskal condition of three-way CPD. The related work is not sufficiently discussed. For example, Allman 2009 https://arxiv.org/pdf/0809.5032 corollary 2, and Lyu 2025 https://arxiv.org/pdf/2506.09165 corollary 8 seems are global identifiability results. The second work seems also has condition that easy to satisfy, “If N(µ) ≥ 2m − 1, then µ is identifiable”.

  2. The problem framing is not tightly matching this work. The introduction motivates through causal representation learning. Nonlinear ICA, and SAE with mechanism interpretability is not strongly related to this work. Besides, this is not a classical dictionary learning where there is unsupervised matrix factorization X = AS with both factors unknown. But here the dictionary was not basis learned from sparse recovery from, let’s say overcomplete dictionary learning or SAE for interpretability. This makes the problem framed and story is unclear and subtle. As a reviewer, it’s hard to catch neither the theoretical nor practical significance of this work. From the reviewer point of view, the really interesting problem should be: “which subsets of moments imply the identifiability, hopefully global” was not addressed.

  3. The selection method is not fully justified, and the identifiability was only local. The algorithm 1 was titled information-based dictionary learning but it’s essentially a greedy selection for monomial subset, selected by KL gain. The greedy has no submodular guarantee. The D-optimal selection (log det J^T J) would be more natural rule and should be included as a baseline. Besides, correct me if I’m wrong, as the Jacobian entries are polynomial, full rank at one random point implies generic local identifiability, which the paper does not using this.

  4. Results side: “Figure 5: Recovery on the synthetic product-Bernoulli mixture” shows that Song-RBF is better when number of samples n is small, where EM is better n is large.

Questions for authors
  1. Could the author clarify that how does the paper contributions against:

Allman 2009 https://arxiv.org/pdf/0809.5032

Lyu 2025 https://arxiv.org/pdf/2506.09165

Chen 2019 https://arxiv.org/pdf/1803.06521

Gordon 2021 https://proceedings.mlr.press/v134/gordon21a/gordon21a.pdf.

This helps to address the contribution and significance concern.

  1. How does ODIN compares with:

random dictionary, the full order <=3 dictionary (which seems already full rank at Table 3), the greedy D-optimal, or E-optimal, etc,

in terms of recovery error? And what is the real algorithm used in such experiment, algorithm 1 or J_S? Does the EM has restart and ten selected? This helps to address the concern that why we need ODIN to do selection.

5 comments

Positioning. We agree with the reviewer's reformulation: each atom is one entry of a moment tensor of the mixture, and ODIN fits the mixture to a subset of these entries. The four works below use all entries up to an order fixed by mm, or the full joint distribution.

What none of them does. All four take statistics fixed by mm as input and return the mixture. ODIN reverses this: it learns the statistics from data, without assuming a mixture, uses them as the representation, and recovers the mixture from them. This gives:

  1. A representation. The atoms are explicit co-activation events, chosen for the dependence they carry. They describe the data whether or not the mixture model holds, and serve both to estimate and to describe the components.
  2. A criterion for arbitrary dictionaries. The four works certify one fixed set of statistics. Our Jacobian condition applies to any subset of moments, including one selected from data, and is checked on the instance.
  3. The low-order regime. The global results require orders or numbers of variables growing with mm; ODIN certifies dictionaries of order at most 3. For m=4m=4, d=6d=6, Gordon et al. need 3m−3=93m-3=9 separated variables and Lyu & Yang 2m−1=72m-1=7 separable ones, so neither applies; Allman et al. apply generically but non-constructively. ODIN's order-3 dictionary has a full-rank Jacobian and is estimated directly.

The price is a weaker guarantee: local, generic once verified at one point, without finite-sample bounds. A global characterization for sparse subsets, the problem the reviewer highlights, remains open.

Allman et al. (2009) prove generic identifiability for d≥2⌈log⁡2m⌉+1d \ge 2\lceil\log_2 m\rceil+1 (Corollary 5) by applying Kruskal's theorem to the full joint distribution of three groups of variables, i.e. the complete atom lattice. The result is non-constructive. ODIN uses a sparse sub-lattice with an estimator; our genericity argument uses their one-point technique.

Chen & Moitra (2019) learn binary product mixtures in total variation, not in parameter space, since the parameters need not be identifiable. ODIN targets the parameters, because it interprets the components.

Gordon et al. (2021), the closest, identify the parameters from multilinear moments (our atoms) of order up to 3m−33m-3. Their guarantee is stronger, but their moment set is fixed and grows with mm.

Lyu & Yang (2025) remove the linear independence condition, which fails for Bernoulli components once m≥3m\ge 3, and allow nonparametric components. They use joint distributions of blocks of 2m−12m-1 separable variables, which require d≥2m−1d\ge 2m-1; ODIN applies below this threshold.

Generic identifiability. The reviewer is correct, and we add this argument. The entries of JDJ_D are polynomial in (π,μ)(\pi,\mu). If JDJ_D has full column rank at one point, a maximal minor is a nonzero polynomial, so JDJ_D has full rank outside a measure-zero set, and the parameters are generically locally identifiable from DD and from any superset of DD. One rank check at the estimate certifies this.

Selection rule. Algorithm 1 is a greedy, hierarchical selection by refined-information gain, without submodularity guarantee; we state this and rename it a selection procedure. D- and E-optimal rules need a pilot estimate, since JDJ_D depends on the unknown parameter; ODIN selects from data statistics before any fit.

Dictionary comparison. On Vote (d=16d=16) and SPECT (d=22d=22), with unknown true parameters, we compare the Jacobian at a common pilot estimate, post-hoc fit and clustering accuracy (Vote / SPECT):

DictionaryAtomsRank (of 33/45)σmin⁡\sigma_{\min}KLACC
All, order ≤ 2136/25332/440/02.642/6.9810.892/0.536
ODIN333/98533/450.101/0.0762.668/6.9220.892/0.618
Greedy D-opt.333/98533/450.545/0.1722.645/6.9670.888/0.554
Greedy E-opt.333/98533/450.617/0.1742.642/6.9530.892/0.554
All, order ≤ 3696/1,79333/450.650/0.1752.642/6.9280.892/0.536

Order 2 is never enough: the rank is 2d2d, one short of the 2d+12d+1 parameters. D- and E-optimal selection condition better than ODIN, as expected, since they maximize conditioning at the pilot. ODIN's criterion shows in the fit and partitions. On SPECT, it has the lowest KL among equal-size dictionaries and the highest accuracy. On Vote, its accuracy matches the best and its fit is slightly lower. Order-matched random dictionaries reach a similar fit.

Reviewer_4wNcOpenReview ↗

Response to the rebuttal 1

I thank the authors for the detailed response. I preciate that they honestly accept several of my points directly: the CP/moment-tensor reformulation, the lack of a submodularity guarantee, and the generic local identifiability argument. They also ran a substantial set of new experiments. I appreciate both the attitude and effort, and I would like to be able to argue for this paper. But to do that I need a concrete reason, and the points below are still open.

I want to first list what I feel resolved:

Generic local identifiability: The authors accept my argument and add it.

Greedy selection without a submodularity guarantee: The authors acknowledge it.

Positioning against Chen (2019) and Lyu (2025): The authors' clarifications address my question

Reformulation as a partially observed moment tensor: The authors agree with it.

Reviewer_4wNcOpenReview ↗

Response to the rebuttal 2

Main concerns that are still blocking me from champion this work:

There is no evidence shows that ODIN's selection step helps.

I think it's the selection step that makes ODIN different from other moment methods. Then you need strong evidence for above step. The algorithm is choosing atoms one at a time by refined-information gained from current dictionary. But meanwhile, the appendix says "the ODIN dictionaries used in our experiments are constructed using the tractable interaction score J_S", and "depends on the current dictionary D, whereas J(S) depends only on the marginals indexed by subsets of S." So the experiments tells me that score does not change when adding atoms and step by step greedy seems not what was run. Please correct me if I'm wrong.

Then for the problem that 'what does ODIN's data-driven selection contribute?' I see two way the paper could sell it:

a. better recovery:

The rebuttal does not support this track and author tends to go for a compact, interpretable dictionary.

b. better interpretability:

The component are computed from the fitted (π, μ). The interpretability make sense only if those parameters are identifiable and stable. This goes back to the identifiability and conditioning.

For m=2, existing results already works like Gorden, and Lyu. So it already give identifiability at same moment order.

Reviewer_ah8uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ODIN, a framework for recovering binary latent class models (mixtures of product-Bernoulli distributions) from an interpretable moment representation. The motivation is that learned embeddings can support latent-variable recovery, but their coordinates are not directly related to observable properties of the data. ODIN instead represents observations through explicit co-activation events of subsets of variables and estimates the mixture parameters by matching the corresponding moments. The method builds the dictionary of interaction atoms hierarchically using a proxy of refined information, lifts observations to these coordinates, and solves a constrained least-squares moment-matching problem. Experiments are conducted on a synthetic product-Bernoulli mixture and on three UCI binary clustering benchmarks.

Critical strengths

The problem setting is relevant and well-motivated: performing latent-variable recovery in a representation whose coordinates remain explicit statistics of the data is useful because learned embeddings rarely relate to observable properties of the data. The high-level design is sensible (use co-activation frequencies directly as representation coordinates and recover the mixture through an intuitive, analyzable moment map) and limitations are discussed in detail.

Critical weaknesses

The interpretability of the recovered components does not come from ODIN. To interpret a component, the paper reports how likely each selected combination of variables is to be jointly active in that component, and how much more likely this is than in the other components. These numbers are computed directly from the estimated mixture weights and Bernoulli means, so any fit of the same product-Bernoulli model provides them. This includes standard EM, which is the maximum-likelihood estimator under correct specification. What ODIN adds is the choice of which combinations to inspect, not the interpretability of the components. The authors should compare against EM fitted to the same model, with the same atoms ranked afterwards, and show whether ODIN yields components that are more accurate, more stable, or more informative.

The paper does not show that its data-driven atom selection is useful. In practice, atoms are selected with a proxy that, by the authors’ own diagnostics, agrees poorly with the motivating refined-information criterion for higher-order atoms. Moreover, the comparison with random dictionaries uses only six variables. The authors should compare ODIN with all low-order atoms and with random dictionaries of the same size, on problems with more variables.

Minor: the appendix has presentation issues. Table 10 should be reformatted for better readability. Appendix H contains an empty subsection heading (H.3). The “Appendix Organization” paragraph is incomplete, omitting Appendix E and Appendices H–L. Section 6.1 refers to Appendix J.2 for an identifiability argument based on a three-view grouping, but J.2 contains only the parameter-counting precondition. Section 6.2 refers to Appendix G for preprocessing details that are in Appendix F.

Questions for authors

ODIN versus EM: Suppose the same product-Bernoulli mixture is fitted with standard EM, and the same atom activations and enrichment scores are then computed from the EM estimates. Do the recovered components, their atom-based descriptions, or the recovery error differ from those obtained with ODIN? If they do not, what does the ODIN estimator contribute beyond selecting which atoms to inspect?

Usefulness of atom selection: Does information-based selection improve recovery compared with simpler choices of atoms? Specifically, how does ODIN compare in recovery error with (i) a dictionary containing all low-order atoms and (ii) random dictionaries of the same size, on problems with more variables than the six-variable synthetic setting? Where exact refined information is computable, does using it instead of the proxy change the selected dictionary or the recovery error?

2 comments

We thank the reviewer for the precise reading, including the appendix.

ODIN versus EM. We agree with the reviewer's decomposition. Within a product-Bernoulli component, the activation probability of an atom SS is ∏i∈Sμki\prod_{i\in S}\mu_{ki}, so atom-level descriptions are functions of the fitted (π,μ)(\pi,\mu), and any estimator of the same model can produce them. What ODIN contributes is the representation: which of the 2d2^d possible co-activation events to use. This choice is made from the observed distribution, before any mixture is assumed, and the same atoms serve for estimation and for description.

Does the ODIN estimator also differ from EM? We fitted both independently and compared enrichment rankings over the same atoms:

ComparisonVoteSPECT
Enrichment correlation (Spearman)0.9950.777
Top-atom overlap (Jaccard)1.0000.181

On Vote, the two estimators give the same component descriptions. On SPECT, they differ. The reason is the estimation target. EM maximizes the likelihood of the full distribution; ODIN minimizes the distance between empirical and model moments on the selected dictionary only. Under correct specification, both are consistent and EM is efficient, so we do not claim ODIN is more accurate there. Under misspecification, they converge to different projections: EM to the KL projection, ODIN to the mixture that best reproduces the selected interactions. The agreement on Vote and the divergence on SPECT are consistent with Vote being close to a two-component product-Bernoulli mixture and SPECT being further from it. When the model is misspecified, ODIN is therefore not equivalent to EM followed by post-hoc ranking.

Usefulness of information-guided selection. ODIN does not take a dictionary size as input: it decides how many interactions of each order are needed. A random dictionary with the same size and order profile therefore already inherits part of ODIN's selection. With this caveat, we compared equal-budget dictionaries on Vote (d=16d=16) and SPECT (d=22d=22). These experiments measure clustering, not parameter-recovery error.

DictionaryAtoms (Vote / SPECT)Vote ACCSPECT ACC
ODIN333 / 9850.8920.618
Random, order-matched (mean of 10 runs)333 / 9850.8920.584
All atoms up to order 3696 / 1,7930.8920.536

On Vote, the three dictionaries give the same partition: when components are separated by low-order statistics, any sufficiently rich dictionary recovers them, and ODIN does so with half the atoms. On SPECT, ODIN is slightly above the order-matched random dictionary and above the complete order-3 dictionary, with 45% fewer atoms.

Proxy versus exact refined information. At order 2, the proxy and exact refined information agree closely (Pearson 0.96–0.98, top-1 agreement 93–100% for d≤6d\le 6), so the selected pairs are essentially the same. At order 3, agreement drops (Pearson 0.40–0.46), so the selected triplets differ. A second result bounds the effect on recovery: at d=6d=6 with a fixed triplet budget, triplets chosen by refined information and random triplets reach comparable parameter error (0.0551 vs 0.0539 with eight triplets). The choice among triplets, and hence the proxy's order-3 errors, has limited effect on recovery error; what matters is that triplets are included.

Presentation. All corrections are made in the revised PDF: Table 10 is reformatted, the empty heading H.3 is removed, the Appendix Organization paragraph lists every appendix, and Section 6.2 points to Appendix F. Section 6.1 overstated the content of Appendix J.2, which contains only the parameter-counting precondition; it now cites the generic identifiability result of Allman, Matias and Rhodes (2009) for latent class models, which is based on the three-view grouping.

Revision. We will add the three comparisons above to the appendix. The Discussion will state that the benefit of information-guided selection is a compact, interpretable dictionary, not a systematic gain in accuracy or recovery at a fixed order profile.

I thank the authors for the response and the new experiments. Several of my points are now resolved:

  1. Interpretability: The authors agree that atom-level descriptions come from the fitted parameters, so any estimator of the same model can produce them.
  2. Scope of claims: The Discussion will state that the benefit of selection is a compact dictionary, not a systematic gain in accuracy or recovery at a fixed order profile.
  3. Presentation: All the appendix issues are fixed.

What remains open:

  1. ODIN versus EM: On SPECT, the low top-atom overlap (Jaccard 0.181) shows that the two estimators differ, not that ODIN is better. Showing that would require either a criterion favoring the moment projection over the KL projection or an external comparison such as accuracy or held-out fit. It would also require a variability estimate, such as bootstrap overlap.
  2. Selection: On Vote, ODIN matches order-matched random dictionaries. On SPECT, the gain (0.618 vs 0.584) has no reported variance. At order 3, refined-information and random triplets give comparable parameter error (0.0551 vs 0.0539). This suggests that the order profile matters but the choice of atoms within each order does not.
  3. Scale: The new comparisons on larger problems (Vote, SPECT) measure only clustering accuracy.

Overall, the problem is relevant, and the design is sensible. However, its interpretability is available from any fit of the same model, its estimator has no demonstrated advantage over EM, and its selection shows no clear advantage over random selection at a fixed order profile. I therefore keep my score.

Reviewer_tybjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies interpretable representations for discrete data using explicit binary co-activation moments. ODIN builds a hierarchical dictionary of interaction atoms, and a constrained moment-matching procedure uses these moments to estimate a mixture of product-Bernoulli distributions. For a fixed dictionary, the paper establishes a sufficient Jacobian-rank condition for local identifiability and shows that adding atoms cannot reduce the rank. Experiments examine synthetic component recovery, dictionary geometry, and clustering on three real datasets. The main contribution is a finite, inspectable representation that connects observable interactions with mixture recovery, rather than a guarantee that the selection procedure always recovers the true latent structure.

Critical strengths
  1. An explicit and useful interpretation of the representation. Each coordinate corresponds to a concrete co-activation event, and the hierarchy records how higher-order interactions are assembled. The component summaries in Appendix K illustrate how the same atoms used for estimation can also describe fitted clusters. This makes the representation easier to inspect and gives the method value beyond clustering accuracy alone.

  2. A clear separation between representation construction and recovery conditions. The paper distinguishes dictionary selection, local identifiability, and numerical optimization. The Jacobian analysis provides a useful diagnostic for a given dictionary, while the experiments include comparisons with EM, matched-budget random dictionaries, and an alternative inversion procedure. I also appreciate the explicit discussion of model misspecification and the limitations of local guarantees.

Critical weaknesses
  1. The specific benefits of the proposed dictionary selection could be further clarified. While the hierarchical, information-guided construction is intuitive and provides an interpretable representation, its advantages over simpler interaction selection strategies are not entirely clear. The paper includes relevant comparisons with random dictionaries, but a more explicit discussion of when and why ODIN is preferable would help strengthen the contribution.

  2. The experimental validation could be somewhat broader. The empirical results are encouraging, particularly on the synthetic recovery task and several real-world clustering benchmarks. Nevertheless, the recovery experiments focus on a relatively limited set of mixture configurations. Additional evidence across more diverse settings would further strengthen confidence in the robustness and general applicability of the approach.

Questions for authors

Could the authors elaborate on the main practical advantages of the information-guided hierarchical dictionary over simpler interaction selection strategies? In particular, is its primary benefit the interpretability and organization of the selected interactions, or does it also tend to improve recovery stability?

How sensitive is the recovery performance to the underlying mixture configuration, such as the number of components or the degree of separation between them? Any additional evidence or discussion would help clarify the robustness of the proposed approach.

1 comment

We thank the reviewer for the careful reading, and for recognizing both the interpretability of the representation and the separation between dictionary construction and recovery conditions.

Scope of the guarantees. The summary notes that the paper does not guarantee "that the selection procedure always recovers the true latent structure." We agree, and the paper does not claim it. Our result is local and sufficient: for a fixed dictionary DD, full column rank of JD(η)J_D(\eta) implies local identifiability, and adding atoms cannot reduce this rank. As the reviewer notes, the paper discusses the limitations of these local guarantees explicitly, and our claims are stated within them. If the soundness score reflects a different concern, we would be grateful to know it so we can address it directly.

Q1. What does information-guided selection add?

Co-activation frequency is driven by marginal rates: two frequent features co-occur often even when they are independent. Refined information instead scores the dependence that lower-order atoms leave unexplained. We compared selection rules on SPECT (d=22d=22) at a matched budget:

DictionaryAtomsPost-hoc KL ↓
ODIN9856.922
Frequency, top-9859857.085
Random, order-matched (mean of 10 runs)9856.931
All atoms up to order 31,7936.928

Frequency selection is clearly worse (+0.16 nats), consistent with the mechanism above. ODIN matches the complete order-3 dictionary with 45% fewer atoms. The random baseline copies the number of atoms per order chosen by ODIN, so it already inherits part of ODIN's selection.

The same holds for recovery. At d=6d=6 with a fixed triplet budget, triplets chosen by refined information and random triplets reach comparable parameter error (0.0551 vs 0.0539 with eight triplets), so information-guided selection does not systematically improve recovery. Its benefit is that it sets the size and order profile of the dictionary and identifies which interactions carry residual dependence. This yields a compact, interpretable representation, rather than a systematic gain at a fixed profile.

Q2. Sensitivity to the mixture configuration.

We ran 160 additional fits: m∈{2,3}m\in\{2,3\}, four separations δ\delta from 0.02 to 0.20, n∈{103,104}n\in\{10^3,10^4\}, 10 draws each. The table reports the median parameter-recovery error over draws, in units of 10−310^{-3}, for the two extreme separations:

δ\deltam=2m=2, n=103n=10^3m=2m=2, n=104n=10^4m=3m=3, n=103n=10^3m=3m=3, n=104n=10^4
0.0226.326.417.112.0
0.202.160.2015.970.630

Two regimes appear. When components are well separated, an increase in nn reduces the error by about a factor of ten: recovery is limited by sampling noise. When components nearly coincide, the error stops decreasing with nn (m=2m=2: 26.3 vs 26.4) and remains of the order of δ\delta itself. The error is then dominated by bias from weak identifiability, not by sampling variance. The local theory predicts this degradation: as δ→0\delta \to 0, the components coincide, the mixing weight between them is no longer identified, and the smallest singular value of JDJ_D tends to zero. More data cannot compensate for this loss of rank.

Revision. We will add both tables and the full sensitivity grid to the appendix. The Discussion will state that the benefit of information-guided selection is a compact, interpretable dictionary, not a systematic gain in accuracy or recovery at a fixed order profile.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.