- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Unsupervised representation learning can be considered as an unsupervised latent mixture model without any ground truth latent factors known, where dictionary learning (DL) is important branch of it. This paper claimed they focuses on a special DL case that estimation the mixture of product-Bernoulli mixture (binary latent class) model. Their proposed ODIN select a set of variable subsets with a greed method, and layer search that ranks the candidate subset by KL improvement. Then fits the mixture weights and means by least square. Their theory results show full column rank of Jacobian of this moment implies local identifiability. They provided synthetic and real experiments.
Critical strengths
-
This is true research problem that the reviewer enjoys the time spent on reading, regardless of the reviewer’s final judgement of significance or contribution.
-
The paper clearly separates the topic of which moments to use, and fixed set of moments implies local recovery.
-
The paper is transparent about their limitation: random dictionary can also reach full rank. And the proxy is not reliable at higher order.
-
Results on mushrooms clearly improvement against baseline, although is not best on the KL metric.
Critical weaknesses
-
The reviewer believes the problem is classical that this paper is not well positioning itself against known results. For binary data, each atom moment is an off-diagonal entry of the order-S tensor. This work is a partially observed CP decomposition for binary latent class model. The aforementioned type of model has classical identifiability results, the Kruskal condition of three-way CPD. The related work is not sufficiently discussed. For example, Allman 2009 https://arxiv.org/pdf/0809.5032 corollary 2, and Lyu 2025 https://arxiv.org/pdf/2506.09165 corollary 8 seems are global identifiability results. The second work seems also has condition that easy to satisfy, “If N(µ) ≥ 2m − 1, then µ is identifiable”.
-
The problem framing is not tightly matching this work. The introduction motivates through causal representation learning. Nonlinear ICA, and SAE with mechanism interpretability is not strongly related to this work. Besides, this is not a classical dictionary learning where there is unsupervised matrix factorization X = AS with both factors unknown. But here the dictionary was not basis learned from sparse recovery from, let’s say overcomplete dictionary learning or SAE for interpretability. This makes the problem framed and story is unclear and subtle. As a reviewer, it’s hard to catch neither the theoretical nor practical significance of this work. From the reviewer point of view, the really interesting problem should be: “which subsets of moments imply the identifiability, hopefully global” was not addressed.
-
The selection method is not fully justified, and the identifiability was only local. The algorithm 1 was titled information-based dictionary learning but it’s essentially a greedy selection for monomial subset, selected by KL gain. The greedy has no submodular guarantee. The D-optimal selection (log det J^T J) would be more natural rule and should be included as a baseline. Besides, correct me if I’m wrong, as the Jacobian entries are polynomial, full rank at one random point implies generic local identifiability, which the paper does not using this.
-
Results side: “Figure 5: Recovery on the synthetic product-Bernoulli mixture” shows that Song-RBF is better when number of samples n is small, where EM is better n is large.
Questions for authors
- Could the author clarify that how does the paper contributions against:
Allman 2009 https://arxiv.org/pdf/0809.5032
Lyu 2025 https://arxiv.org/pdf/2506.09165
Chen 2019 https://arxiv.org/pdf/1803.06521
Gordon 2021 https://proceedings.mlr.press/v134/gordon21a/gordon21a.pdf.
This helps to address the contribution and significance concern.
- How does ODIN compares with:
random dictionary, the full order <=3 dictionary (which seems already full rank at Table 3), the greedy D-optimal, or E-optimal, etc,
in terms of recovery error? And what is the real algorithm used in such experiment, algorithm 1 or J_S? Does the EM has restart and ten selected? This helps to address the concern that why we need ODIN to do selection.
5 comments
Response to the rebuttal 1
I thank the authors for the detailed response. I preciate that they honestly accept several of my points directly: the CP/moment-tensor reformulation, the lack of a submodularity guarantee, and the generic local identifiability argument. They also ran a substantial set of new experiments. I appreciate both the attitude and effort, and I would like to be able to argue for this paper. But to do that I need a concrete reason, and the points below are still open.
I want to first list what I feel resolved:
Generic local identifiability: The authors accept my argument and add it.
Greedy selection without a submodularity guarantee: The authors acknowledge it.
Positioning against Chen (2019) and Lyu (2025): The authors' clarifications address my question
Reformulation as a partially observed moment tensor: The authors agree with it.
Response to the rebuttal 2
Main concerns that are still blocking me from champion this work:
There is no evidence shows that ODIN's selection step helps.
I think it's the selection step that makes ODIN different from other moment methods. Then you need strong evidence for above step. The algorithm is choosing atoms one at a time by refined-information gained from current dictionary. But meanwhile, the appendix says "the ODIN dictionaries used in our experiments are constructed using the tractable interaction score J_S", and "depends on the current dictionary D, whereas J(S) depends only on the marginals indexed by subsets of S." So the experiments tells me that score does not change when adding atoms and step by step greedy seems not what was run. Please correct me if I'm wrong.
Then for the problem that 'what does ODIN's data-driven selection contribute?' I see two way the paper could sell it:
a. better recovery:
The rebuttal does not support this track and author tends to go for a compact, interpretable dictionary.
b. better interpretability:
The component are computed from the fitted (π, μ). The interpretability make sense only if those parameters are identifiable and stable. This goes back to the identifiability and conditioning.
For m=2, existing results already works like Gorden, and Lyu. So it already give identifiability at same moment order.
Positioning. We agree with the reviewer's reformulation: each atom is one entry of a moment tensor of the mixture, and ODIN fits the mixture to a subset of these entries. The four works below use all entries up to an order fixed by m, or the full joint distribution.
What none of them does. All four take statistics fixed by m as input and return the mixture. ODIN reverses this: it learns the statistics from data, without assuming a mixture, uses them as the representation, and recovers the mixture from them. This gives:
The price is a weaker guarantee: local, generic once verified at one point, without finite-sample bounds. A global characterization for sparse subsets, the problem the reviewer highlights, remains open.
Allman et al. (2009) prove generic identifiability for d≥2⌈log2m⌉+1 (Corollary 5) by applying Kruskal's theorem to the full joint distribution of three groups of variables, i.e. the complete atom lattice. The result is non-constructive. ODIN uses a sparse sub-lattice with an estimator; our genericity argument uses their one-point technique.
Chen & Moitra (2019) learn binary product mixtures in total variation, not in parameter space, since the parameters need not be identifiable. ODIN targets the parameters, because it interprets the components.
Gordon et al. (2021), the closest, identify the parameters from multilinear moments (our atoms) of order up to 3m−3. Their guarantee is stronger, but their moment set is fixed and grows with m.
Lyu & Yang (2025) remove the linear independence condition, which fails for Bernoulli components once m≥3, and allow nonparametric components. They use joint distributions of blocks of 2m−1 separable variables, which require d≥2m−1; ODIN applies below this threshold.
Generic identifiability. The reviewer is correct, and we add this argument. The entries of JD are polynomial in (π,μ). If JD has full column rank at one point, a maximal minor is a nonzero polynomial, so JD has full rank outside a measure-zero set, and the parameters are generically locally identifiable from D and from any superset of D. One rank check at the estimate certifies this.
Selection rule. Algorithm 1 is a greedy, hierarchical selection by refined-information gain, without submodularity guarantee; we state this and rename it a selection procedure. D- and E-optimal rules need a pilot estimate, since JD depends on the unknown parameter; ODIN selects from data statistics before any fit.
Dictionary comparison. On Vote (d=16) and SPECT (d=22), with unknown true parameters, we compare the Jacobian at a common pilot estimate, post-hoc fit and clustering accuracy (Vote / SPECT):
Order 2 is never enough: the rank is 2d, one short of the 2d+1 parameters. D- and E-optimal selection condition better than ODIN, as expected, since they maximize conditioning at the pilot. ODIN's criterion shows in the fit and partitions. On SPECT, it has the lowest KL among equal-size dictionaries and the highest accuracy. On Vote, its accuracy matches the best and its fit is slightly lower. Order-matched random dictionaries reach a similar fit.