- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors break down determining whether an overcomplete dictionary is useful in terms of telling us something about the data it was derived from into three separate (classically well-established) problems of identifiability, finite-sample estimation, and recovery. For each of the three problems, they introduce measures that could inform us of the degree to which the dictionary is sufficiently informative.
Critical strengths
- The authors are looking to address a useful problem in the field and identify a relevant separation of the issues that are otherwise often conflated in the name of interpretability via sparsity.
Critical weaknesses
- Far too much of the text is direct (or minimally edited) LLM output; for a paper that intends to establish a framework for understanding (as opposed to pushing SoTA), the readers would greatly benefit from the authors spending more time "humanizing" the text and doing away with a number of LLM cliches scattered through the paper.
- E.g., in the abstract: Experiments are organized by what each dataset can actually identify. Controlled support-law sweeps, overcomplete recovery studies, and bottleneck audits show that worst-pair geometry, statistical identifiability, optimizer success, and operational recoverability can move independently.
- Likewise, lines 40-47 aren't very clear; What does it mean for identifiability to be a "population statement" and how does the "claim hierarchy" rank the three aspects being studied, i.e., identifiability, estimation, and recovery? As I understand, the authors only provide their interpretation of those terms, and not an explicit hierarchy regarding necessity in this section? Readability of the paper would benefit immensely from stating the intuitions why identifiability precludes finite sample estimation precludes recovery in the Introduction and abstract themselves.
- As currently framed, the contributions seem rather incremental, and the authors don't do a good job of impressing on the reader exactly why their framing is different and better than what previously studied versions already accomplish.
- To that point, I'm not sure I fully agree with lines 48-53 and think it is too strong of a claim. There has been literature in sparse approximation, compressed sensing, and dictionary learning that explicitly replaces worst-case guarantees with guarantees conditioned on a distribution over sparse supports?
Questions for authors
- While interesting as a concept (and likely mathematically correct), I fail to see what the exact takeaway of the work is.
- Could the authors explain what they establish in terms of how sparse feature learning and interpretability should work beyond saying that identifiability, finite-sample estimation, and recovery have different observables?
- It would be really useful to see an end-to-end application of the framework, even if it is on a simplistic toy-model to drive home the point, and even just help with exposition of the ideas. At present, the framework simply feels like three separate issues (that are historically well-studied in the signal processing and TCS communities) put together for anyone familiar with those areas, and fails to come across as more than a theoretical construct for those unaware of these specific issues.
4 comments
We hope the above clarifications and the additional experiments in the revised draft sufficiently addressed your concerns. If you are satisfied, we kindly request you to consider updating the score to reflect the newly added results and discussion. We remain committed to addressing any remaining points you may have during the discussion phase.
We hope the above clarifications and the additional experiments in the revised draft sufficiently addressed your concerns. If you are satisfied, we kindly request you to consider updating the score to reflect the newly added results and discussion. We remain committed to addressing any remaining points you may have during the discussion phase.
As the discussion deadline is approaching, we would greatly appreciate your valuable feedback on our revisions.
We clarify the central scientific contribution: a unified, quantitative framework connecting feature identifiability, finite-sample estimation, and support-conditioned recovery. A new end-to-end overcomplete experiment demonstrates how these guarantees translate into dictionary estimation and coefficient decoding, establishing practical criteria for evaluating sparse feature recovery beyond reconstruction accuracy.
We have thoroughly revised the manuscript in response to the reviewer's insightful comments and uploaded the updated version.
We thank the reviewer for asking for the scientific payoff and an end-to-end demonstration. Our central conclusion is that claiming a learned sparse representation recovers generating features requires evidence that the activation law identifies their directions, finite samples estimate them stably, and realized supports permit coefficient decoding. We prove conditions for each transition and test their interaction in one generative model.
Q1: What does the framework establish beyond three distinct notions?
1. When can feature directions be learned? Under the stated non-Gaussianity and tensor-uniqueness conditions, fourth-order activation statistics determine the dictionary up to signed permutation (Theorem 3.9). Gaussian subspaces instead admit rotationally equivalent dictionaries (Proposition 3.10). This distinguishes data-determined features from arbitrary decompositions.
2. When does identifiability translate into estimation? Theorem G.6 links population uniqueness to finite-sample recovery through global basin separation and local inverse conditioning. Under its stated assumptions, Theorem G.8 bounds direction error by 4(εn+βdep)/γ⋆, separating sampling error from cross-cumulant dependence bias. Proposition G.9 supplies an observable test of diagonal-cumulant model adequacy. These results quantify sampling, conditioning, and source-model limitations.
3. When do estimated directions yield recoverable features? For a fixed dictionary, Proposition 3.1 establishes Rν(W)≥Cν(W) under any support distribution ν. Its certificate accounts for interactions within the active support and competition from inactive atoms. Theorem H.1 characterizes favorable local geometry beyond the universal Welch-based certification endpoint. Global coherence alone cannot determine realized recovery.
The connection is consequential, not terminological. More samples cannot resolve rotational nonidentifiability; accurate dictionary estimation cannot repair decoding failures caused by hazardous supports; and successful decoding with a supplied dictionary cannot establish that the dictionary was identified from data. Each failure has a distinct condition and diagnostic. Classical tensor uniqueness and ERC supply foundations; our contributions include support-law certification, dependence-aware estimation, and their connection to operational recovery.
Q2: End-to-end application
We added an overcomplete experiment with d=8, K=12, sparse Bernoulli–Exponential sources, fourth-cumulant dictionary estimation, and held-out basis-pursuit (BP) decoding across five independent runs.
Non-Gaussian sources yield improving direction estimates; the Gaussian control does not. At 100k samples, estimated-dictionary NMSE (0.0434) nearly matches the oracle (0.0427), while natural-support ERC (0.497) is below observed BP success (0.851). This traces one model from activation statistics through estimation to decoding and isolates the conservatism of support-wide certification. Theorem G.6 analyzes an ideal global estimator; the experiment uses a practical cumulant estimator.
We then hold the true dictionary fixed and vary only the held-out support law. Supports are selected by ERC margin; BP success is evaluated separately on fresh coefficients.
The ERC trend is imposed by the support intervention; the BP outcomes are independently measured. With dictionary geometry unchanged, support-wide certification falls from 1 to 0, yet BP still succeeds on 87.1% of coefficient realizations at the endpoint. This shows that accurate directions do not guarantee decoding on every support, while failed support-wide certification does not preclude successful coefficient recovery.
Practical takeaway. Evaluate sparse features through verifiable claims about activation-law identification, finite-sample direction accuracy and model fit, and decoding on realized supports, including inactive competitors. Reconstruction accuracy alone cannot certify these properties. The end-to-end study demonstrates why each matters.