acceptodds
Under review as a conference paper at ICLR 2027

Estimating the Intrinsic Dimension of Neural Manifolds from Variational Autoencoders

Abstract

Neural population activity is often conceptualized as lying on low-dimensional manifolds. Yet estimating the intrinsic dimension of these manifolds is difficult in noisy, curved neural data, and existing estimators are often not reliable enough to serve as a scientific yardstick. We therefore investigate a method based on fitting a variational autoencoder (VAE) to the neural data and estimating dimension from the effective rank of the decoder Jacobian. We explain the effectiveness of this approach by linking the VAE to local probabilistic principal component analysis, showing that the variational objective penalizes decoders that use unnecessary latent dimensions. In neuroscience-inspired simulations with highly noisy and curved data, our method accurately recovers intrinsic dimension where alternative methods fail. Applied to population recordings from the mouse head direction system, it identifies two intrinsic dimensions. Further analysis suggests that one of these dimensions may reflect gain modulation, even in the absence of explicit experimental manipulation. Applied to population recordings from a grid cell module in mouse medial entorhinal cortex, our method identifies 4 dimensions, related to x and y position, head direction, speed, and gain. Since VAE models are common in neuroscience, our method can be applied to previously fit models. We demonstrate this on a VAE fit to juvenile male zebra finch songs over weeks of vocal practice, identifying an increase in dimensionality over time.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Dimensionality, Generative model, Neural manifold

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_R2jZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper estimates intrinsic dimension from the effective rank of a VAE decoder Jacobian, evaluated at posterior means. A local PPCA analysis explains why VAE training can suppress unnecessary decoder directions.
  • Simulations examine recovery under different noise levels, tuning smoothness, and true dimensions. Applications to head-direction recordings, grid-cell activity, and birdsong illustrate its use on biological data and previously fitted VAEs.
Critical strengths
  • The simulations address noisy, curved manifolds where neighborhood estimators struggle. Comparisons on observations and learned representations help identify the benefit of the VAE-based approach.
  • The real-data analyses include useful surrogate controls and examine relationships with behavioral variables and gain modulation. Reusing existing fitted VAEs is also practically appealing.
Critical weaknesses
  • The estimated dimension is not stable across similarly predictive architectures: Table 2 reports 2.1–4.7 for the same head-direction data.
  • The covariance correction does not resolve this ambiguity. Estimates still range from 2.0 to 3.5, leaving the specific two-dimensional conclusion dependent on model choice.
  • Exact Jacobian rank is invariant to invertible latent reparameterization; the proposed 95% effective-rank estimator is not. The exact-rank argument therefore does not establish robustness of the quantity actually reported.
  • The local Gaussian analysis explains a regularization mechanism, but does not establish consistent dimension recovery for the nonlinear Poisson models used in the neural applications.
  • The proposed architecture correction introduces a neighborhood choice—1% of the encoded data. Its sensitivity needs examination, especially since neighborhood-scale dependence is a central criticism of competing estimators.
Questions for authors
  • How should an architecture be selected when models with comparable validation performance give substantially different dimension estimates?
  • Across training seeds and the tested spectral thresholds, how stable are the biological conclusions for architectures with similar validation performance?
1 comment

We thank the reviewer for acknowledging the usefulness of our method to the neuroscience community. Here we address the critiques.

1. The estimated dimension is not stable across similarly predictive architectures.

We added Appendix A.8, which explains why the Jacobian effective rank depends on network architecture: greater depth penalizes extra Jacobian dimensions more strongly but risks posterior collapse. Based on this, we propose a practical criterion for selecting the depth, and show that held-out ELBO cannot serve this purpose (Figure A.6A); see point 4 for details. Thus, the noted instability is a property of the VAE that can be dealt with in practice.

2. The exact-rank argument does not strictly apply to the effective rank.

We agree; we acknowledged this before and now do so with more emphasis. The invariance of the exact Jacobian rank does not imply invariance of the proposed effective-rank estimator. Our goal was to highlight one way in which the VAE's usefulness for ID estimation is less affected by its known shortcomings. It is not a proof of the reliability of the method, which is most convincingly shown by the empirical experiments.

3. The local Gaussian analysis explains a regularization mechanism, but does not establish consistent dimension recovery for the nonlinear Poisson model.

We agree that the local Gaussian analysis is not a proof of consistent dimension recovery for nonlinear Poisson models. Its purpose is to explain the regularization mechanism underlying VAE-ID, while the accuracy of dimension recovery is established empirically through our neuroscience-inspired simulations with known ground-truth dimensions. We have clarified this distinction in the revised manuscript (end of Section 3.2).

4. Dependence on the neighborhood choice and the 95% threshold.

We agree that every ID estimator, including ours, depends on experimenter-selected choices. Prior VAE-based methods are no exception: Zheng (2022), for instance, defines active dimensions as those with posterior variance below 0.05. We have therefore toned down our criticism that neighborhood-based methods need an arbitrary neighborhood. The real question is not whether a method has such choices, but whether they can be made reliably.

  • Neighborhood-based methods: the choice cannot be made reliably (Figure A.2E–F). At high noise, CorrInt and ABID overestimate the dimension at every neighborhood size we tested. At low noise, CorrInt comes close to the true value only at the smallest size, while ABID overestimates throughout. Because the size that works changes with the noise level, it cannot be chosen without knowing the ground truth.
  • VAE-ID: the choices can be made reliably.
    • Spectral threshold. A single 95% threshold is accurate across all simulated conditions (Figure 2, Table 1), and our conclusions on real data hold at 97% and 99% (Appendix A.6).
    • Network architecture, which we now explain and control. Greater depth penalizes extra Jacobian dimensions more strongly, but too much depth leads to posterior collapse (Appendix A.8). Dai (2018) observed that their estimator needed enough layers. However, they did not connect this to posterior collapse, which was discussed only later (Dai, 2020), and they gave no way to choose the depth. Building on this understanding, we propose a practical rule for selecting depth, and show that held-out ELBO cannot serve this purpose (Figure A.6A).
    • No neighborhood correction. We removed the neighborhood-based covariance correction, so it adds no choice of its own.

Summary. We appreciate the reviewer's careful examination of the theoretical assumptions and methodological reliability of VAE-ID. In response, we have substantially expanded our understanding of when the estimator works and what determines its accuracy. In particular, our new analysis of network depth and posterior collapse explains why similarly predictive architectures can produce different ID estimates, and motivates a practical criterion for architecture selection. We have also removed the previous neighborhood-based correction, examined threshold sensitivity, and clarified the scope of our theoretical arguments. The revised manuscript therefore offers a more complete methodology for reliable ID estimation, together with a clearer account of its theoretical foundations and limitations. We believe these advances are particularly important for neuroscience, where dimensionality estimates are increasingly used to draw scientific conclusions about neural representations, calling for practically reliable, well-tested methods. We hope our additions directly address the reviewer's primary concerns and that they warrant reconsideration of the original assessment.

Reviewer_4tBcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the intrinsic dimension of neural manifolds. The authors proposed using the Jacobian of the decoder as the intrinsic dimesnion. They theoretically proved that the Jacobian of the decoder is a proper intrinsic dimension estimator and validated the argument on synthetic and toy datasets.

Critical strengths

N/A

Critical weaknesses
  • I found this paper is not well written. The organization does not follow the standard structure and is hard to follow. Why is the related work a subsection of introduction? Section 2 mixes different kinds of material: Section 2.1 defines the estimator; Sections 2.2 and 2.3 are theory derivations.

  • In section 2.3, the authors explain why VAE does not overfit from a local PPCA perspective. I think the authors regard this as an important part of the contribution. However, Dai et al. 2018 has already proved this. Specifically, if the deocder mean is affine, the decoder variance is λI\lambda I and the encoder is arbitrary complex, VAE collapse to PPCA.

  • The experiments are applied on synthetic or toy datasets. In addition, the paper compares only against generic intrinsic-dimension estimators. Table 1 benchmarks CorrInt, ABID, MLE, TwoNN and Local PCA, all of which operate on the raw observations or on latent means; none of them is a VAE-based estimator. The prior VAE-based approaches that the paper itself surveys in Section 1.1 — and on which its own Section 2.3 derivation directly builds — are never evaluated numerically anywhere in the paper (Zheng et al. 2022, Rolinek et al. 2019, Kumar and Poole 2020)

  • The authors claim that ``We introduced a framework for extracting manifold intrinsic dimension from trained VAEs, explained how VAE objective is conducive to ID estimation, and demonstrated its utility in rodent extracellular population recordings and bird song spectrograms." This is an over claim. In Dai et al. 2018, they proved that a nonlinear VAE can identify the number of intrinsic dimension by simply counting the number of nonzero columns in the weight of the first decoder layer (see table 1 and figure 2 of the paper). The encoder covariance statistics can also be used to identify the intrinsic dimension easily (see figure 3 of the paper).

Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Connections with robust pca and the role of emergent sparsity in variational autoencoder models. Journal of Machine Learning Research, 19(41):1–42, 2018.

Yijia Zheng, Tong He, Yixuan Qiu, and David P Wipf. Learning manifold dimensions with conditional variational autoencoders. Advances in Neural Information Processing Systems, 35:34709–34721, 2022.

Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue PCA directions (by accident). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12406–12415, 2019.

Abhishek Kumar and Ben Poole. On implicit regularization in β-VAEs. In International Conference on Machine Learning, pp. 5480–5490. PMLR, 2020

Questions for authors

See weakness

1 comment

We thank the reviewer for the constructive feedback. Our updated manuscript provides new evidence for the unique contribution of our work and its importance for neuroscience.

1. Our unique contributions are four-fold:

  1. We turn existing VAE theory into a practical ID estimator that is accurate on neural-data-inspired simulations. It improves substantially on non-VAE-based and prior VAE-based methods, making its results on real data more credible.
  2. We apply it to neural data and derive novel scientific insights.
  3. We develop a framework to interpret the dimensions.
  4. We discuss the architectural dependency of the estimator and propose a practical criterion for choosing the architecture, which is lacking in previous VAE-based methods.

We do not regard the mathematical derivation as our unique contribution, and we modified our language to make clear that we are drawing on prior work. We keep it in the paper to motivate the estimator.

2. The prior VAE-based methods measure different quantities.

Dai (2018) and Zheng (2022) both boil down to variants of counting nonzero columns of the decoder weights or Jacobian, which is very different from counting independent Jacobian directions, as we propose. This leads to large differences in accuracy. We now include explicit comparisons showing that the prior VAE-based estimators perform poorly empirically (Figure A.1; Table 1):

EstimatorMean absolute error ↓R2R^2 ↑
Active columns (Dai, 2018)6.2−35
Active dimensions (Zheng, 2022)1.2−1.2
VAE-ID (ours)0.40.59

They are therefore not reliable enough for neuroscience use. We also include theoretical arguments for why prior methods are suboptimal for ID estimation (Appendix A.4). In brief, prior methods are derived assuming a diagonal posterior covariance. The resulting penalty on the Jacobian is on its column norms, not on its eigenspectrum. It is easy to construct cases where additional dimensions in the Jacobian are not penalized by the objective under a diagonal posterior.

3. Interpreting the latent dimensions.

Our contribution for relating the latent dimensions to interpretable variables is original. Its results are already novel and raise new neuroscientific questions, highlighting its usefulness for the wider community.

4. Organization of the paper.

The paper is organized so that neuroscience readers interested in applying the tool, but less interested in the theory, can skip the Methods section and go directly to the results. This may be less familiar to a machine learning audience. Some explanations aim to provide intuition for readers with a neuroscience background rather than to establish new results. Following the reviewer's suggestion, we moved the related work to a separate section.

5. Scope of the experiments.

Our experiments are not limited to "synthetic or toy datasets". In addition to systematic neuroscience-inspired simulations, we apply our method to three established experimental systems of substantial interest in neuroscience:

  • the head-direction system and entorhinal grid cells, central to understanding spatial navigation and continuous attractors;
  • zebra finch vocalizations, a widely studied model of motor learning and development.

These address meaningful questions about the dimensionality and interpretation of neural and behavioral manifolds.

Significance for neuroscience. Estimating the intrinsic dimension of neural manifolds is a fundamental problem, yet existing methods such as CorrInt and ABID are often unreliable in practice. Our expanded experiments further show that prior VAE-based approaches (Dai, 2018; Zheng, 2022) are substantially less accurate than VAE-ID. We fully acknowledge that the connection between VAEs and dimension pruning is not new, nor do we claim novelty in the underlying mathematical derivation. Our contribution is to turn these theoretical insights into a substantially more accurate estimator of manifold intrinsic dimension, supported by systematic validation and the additional ability to interpret the learned latent dimensions.

The distinction is crucial: knowing that VAEs can prune unnecessary dimensions does not mean that previous methods have successfully extracted the intrinsic dimension of complex neural data. We believe it would be a disservice to the neuroscience community if a method that demonstrably improves upon existing alternatives were dismissed primarily because it builds on known theoretical insights. Scientific value should be judged not only by the originality of the underlying mathematical ingredients, but also by whether they lead to effective methodology and novel findings. We hope our added experiments and clarifications make clear that this is precisely what our work contributes, and warrant a reconsideration of the original assessment.

Reviewer_wj1vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method to estimate the intrinsic dimension (ID) of neural population activity from a trained variational autoencoder (VAE). At each encoded data point, the local ID is defined as the number of singular directions of the decoder Jacobian that account for 95% of the sum of its squared singular values. The authors linearize the decoder locally and relate the VAE objective to probabilistic PCA. They show that this objective penalizes large singular values of the Jacobian, so that redundant latent directions are suppressed.

The method is validated in two steps. In simulations (spike data generated from smooth tuning curves on a dd-torus, with d≤4d \le 4 and Gaussian or Poisson noise), the estimator is much more accurate than neighborhood-based estimators applied to the observations. It is also more accurate than two of these estimators, CorrInt and ABID, applied to the latent codes of the VAE, but the difference is smaller (mean absolute error of 0.40 vs. 0.55). On real data, the estimator finds two dimensions for head-direction cells in the post-subiculum, which the authors interpret as head direction and a shared gain, and four dimensions for a grid module in the medial entorhinal cortex. When applied to published VAE models of zebra finch songs, the estimated ID increases with age in four of the five birds.

Critical strengths
  1. The paper shows convincingly that standard ID estimators fail on noisy spike data, and it proposes a practical alternative. With Poisson noise and low tuning smoothness, the neighborhood-based estimators that are commonly used in neuroscience overestimate the ID by several dimensions. In contrast, the VAE-based estimate remains close to the ground truth and improves when more data are available. Since the ID is more and more often reported as a summary of population activity, this result is useful for the community, even if the estimator does not outperform every alternative.

  2. The Jacobian makes it possible to relate the estimated dimensions to variables. The authors align the local tangent directions with the derivatives of candidate tuning models. In this way, they can ask which variables span the manifold, and not only how many dimensions it has. Counting active latent units or using neighborhood estimators does not give this information. In my opinion, this is the most original part of the paper.

Critical weaknesses

My recommendation is based on two weaknesses. I read the paper as a potential user of the method, not as a specialist in dimension estimation. Both points could be addressed with limited additional analyses (Questions 1 to 3).

  1. The results on real data depend on modeling choices, and their interpretation goes beyond the evidence. The paper criticizes existing estimators because their output depends on a free parameter (the size of the neighborhood). As a user, I would expect evidence that the proposed method does not have the same problem, but the paper rather suggests the opposite. On the head-direction data, six network architectures that fit the data equally well give estimates between 2.1 and 4.7 dimensions (Table 2), and the architecture used in the paper was selected on simulations, not on these data. The value of two dimensions also depends on a 95% threshold that was tuned on the same simulations. Finally, none of the estimates in the paper, in simulation or on real data, is reported with its variability across training runs, so it is not possible to judge which differences are meaningful. Without these checks, I cannot tell whether the "two dimensions" are a property of the data or a consequence of the chosen settings.

The biological interpretation could also be more cautious. From the point of view of dynamical systems, a two-dimensional cloud of activity does not necessarily contradict a one-dimensional ring attractor. The attractor is the set where the dynamics settle, while the recorded activity may also contain fluctuations around this set, for example in the amplitude of the activity bump. A noisy limit-cycle oscillator is a well-known example. Its attractor is a one-dimensional cycle, but its recorded states fill a two-dimensional annulus, because the fluctuations in amplitude decay only gradually while the fluctuations in phase persist. In the same way, the "gain" interpretation, the attribution of the four dimensions found in the entorhinal cortex to five variables, and the trends in bird song are presented with more confidence than the analyses support.

  1. It is not clear what the method adds to existing approaches that read the dimension from a VAE. The theoretical argument explains why a VAE tends to switch off unnecessary latent dimensions, which is helpful, but it does not justify the specific counting rule (the 95% threshold). Moreover, a closely related result (Zheng et al., NeurIPS 2022) already shows that VAEs can recover the dimension of the data by counting the latent dimensions that are actually used by the model. The paper cites this work but does not compare the two approaches. Without this comparison, the reader cannot know whether the proposed estimator is more accurate than this simpler option or only different from it.
Questions for authors

The questions are given in order of importance. The answer to each of them could change my assessment.

  1. Are the estimates stable across training runs? Could you report the variability over at least five training seeds for all estimates obtained with VAE-ID, both in simulation and on real data? For the head-direction data, which support the main biological claim, could you also report, for each architecture of Table 2, the average estimate and its distribution across time bins? If, after the covariance correction, the average estimates are close to two, with a spread across seeds and architectures clearly smaller than one dimension, my main concern would be resolved.

  2. How does VAE-ID compare with the number of active latent dimensions (Zheng et al., 2022)? Could you report both quantities for the same trained VAEs, in the simulations and on the head-direction data? A clear improvement in accuracy, or a clear explanation of what the Jacobian adds, would strengthen the contribution.

  3. Does the method recover the scenario that is proposed for the head-direction data? For a simulated ring with a weak multiplicative gain, with the same noise level and number of neurons as in the head-direction recordings, does VAE-ID find two dimensions? How does the result change with the noise variance of the decoder in this particular case?

  4. Is the second dimension really a multiplicative gain? Does a surrogate with an additive shared fluctuation also give two dimensions? Is the second decoder direction better aligned with the direction predicted by a multiplicative gain (a scaling of the current activity pattern) than with the direction predicted by an additive change (the same increase of all firing rates)?

  5. Given the distinction between the attractor and the activity that it generates, would you rephrase the statement that the result goes against the view of the head-direction circuit as a one-dimensional ring attractor (l. 343-344)? Does the second coordinate behave as a stable direction, with fluctuations that relax quickly, or as a neutral direction, with fluctuations that persist as along head direction? Only the second case would indicate a two-dimensional attractor.

Additional remarks:

  • Eqs. 1 and 2 define the negative ELBO.
  • The distribution of the local ID across time bins is shown only for the head-direction and entorhinal data (Fig. 3A and 5A). It would be useful to show this distribution also where only averages are reported, in particular for the surrogate data (Fig. 3C) and for bird song (Fig. 6). This would help to distinguish errors of the estimator at individual time bins from real changes in dimension, for example between behavioral states.
2 comments

We thank the reviewer for acknowledging the usefulness of our method to the neuroscience community and the originality of our method for relating the dimensions to variables of interest. Here we address the critical weaknesses and the questions.

1. Results on real data depend on modeling choices, which makes them hard to interpret.

We agree that every ID estimator, including ours, depends on experimenter-selected choices. Prior VAE-based methods are no exception: Zheng (2022), for instance, defines active dimensions as those with posterior variance below 0.05. We have therefore toned down our criticism that neighborhood-based methods need an arbitrary neighborhood. The real question is not whether a method has such choices, but whether they can be made reliably.

  • Neighborhood-based methods: the choice cannot be made reliably (Figure A.2E–F). At high noise, CorrInt and ABID overestimate the dimension at every neighborhood size we tested. At low noise, CorrInt comes close to the true value only at the smallest size, while ABID overestimates throughout. Because the size that works changes with the noise level, it cannot be chosen without knowing the ground truth.
  • VAE-ID: the choices can be made reliably.
    • Spectral threshold. A single 95% threshold is accurate across all simulated conditions (Figure 2, Table 1), and our conclusions on real data hold at 97% and 99% (Appendix A.6).
    • Network architecture, which we now explain and control. Greater depth penalizes extra Jacobian dimensions more strongly, but too much depth leads to posterior collapse (Appendix A.8). Dai (2018) observed that their estimator needed enough layers. However, they did not connect this to posterior collapse, which was discussed only later (Dai, 2020), and they gave no way to choose the depth. Building on this understanding, we propose a practical rule for selecting depth, and show that held-out ELBO cannot serve this purpose (Figure A.6A).
    • No neighborhood correction. We removed the neighborhood-based covariance correction, so it adds no choice of its own.

2. Unclear what our method adds to existing VAE-based ID estimators.

We now include explicit comparisons showing that the prior VAE-based estimators perform poorly empirically (Figure A.1; Table 1):

EstimatorMean absolute error ↓R2R^2 ↑
Active columns (Dai, 2018)6.2−35
Active dimensions (Zheng, 2022)1.2−1.2
VAE-ID (ours)0.40.59

They are therefore not reliable enough for neuroscience use. We also include theoretical arguments for why prior methods are suboptimal for ID estimation (Appendix A.4). In brief, prior methods are derived assuming a diagonal posterior covariance. The resulting penalty on the Jacobian is on its column norms, not on its eigenspectrum. So it is easy to construct cases where additional dimensions in the Jacobian are not penalized by the objective under a diagonal posterior.

3. Robustness to training seeds.

Across HD, MEC and a 3D Poisson simulation, the per-fit mean ID varies by at most 0.83 across five seeds at the 95% threshold (Table 2), so stochastic training is not a major source of variability.

4. Interpretation of the 2D HD result.

Since our method does not concern dynamics, we do not make claims about the attractor. Our claim concerns only the population activity point cloud and the inferred underlying rate manifold. However, generic noisy fluctuation around a ring attractor does not produce our 2D result (Appendix A.7). We simulated a ring attractor network with recurrent noise, driven by angular-velocity modulation of the weights. Its activity is estimated to be 1D. Thus our 2D result reveals genuine structure in the neural activity.

5. Multiplicative or additive gain.

We tested an additive gain model on the HD and MEC data. However, the Jacobian of the tuning model with respect to the additive gain is too similar to that for the multiplicative gain (cosine similarity ≥ 0.94). We report this in the paper and remain agnostic to the form of the gain.

6. Additional remarks.

  • The distribution of local ID is now shown for all data (Figure A.7 for bird song).
  • Equations are now marked as nELBO.

Summary. We have substantially strengthened both aspects of the work. The new comparisons establish that VAE-ID offers a considerable improvement over previous VAE-based estimators, while the additional analyses of threshold sensitivity, training variability, architecture selection, and head-direction dynamics strengthen the reliability and interpretation of our results. We have also clarified the limits of our biological conclusions. We believe these revisions directly address the two critical weaknesses and strengthen the case for VAE-ID as a useful scientific tool. We hope they warrant reconsideration of the original assessment.

I thank the authors for their responses. I will take them into account in my final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.