acceptodds
Under review as a conference paper at ICLR 2027

Spatially Aware Nonlinear Dimension Reduction Improves the Efficiency of Spatial Domain Detection in Spatial Multi-Omics Studies

Abstract

Spatial multi-omics technologies enable the joint characterization of multiple molecular modalities while preserving spatial context, but integrating heterogeneous nonlinear measurements with complex spatial dependencies remains challenging. Here, we introduce SpaLVM, a Bayesian latent variable model that jointly captures cross-modality variation and spatial dependencies through Gaussian process priors, enabling nonlinear dimensionality reduction of spatial multi-omics data. We further develop a sparse, structured variational inference scheme with inducing points for efficient and scalable computation, achieving an approximately 10-fold speedup over the original SpaLVM implementation. Across simulated and real datasets spanning diverse technologies and tissue structures, SpaLVM consistently outperforms existing spatial single-modal, spatial multimodal, non-spatial single-modal, and non-spatial multimodal methods in dimensionality reduction and spatial domain detection. SpaLVM provides a flexible and scalable framework for nonlinear integration and spatial domain detection in spatial multi-omics data.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
spatial multi-omics, variational inference, dimension reduction, spatial domain detection, spatial transcriptomics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FHRLOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a model for spatial multiomics data that uses Gaussian process priors on a Bayesian latent variable model. This is the first such formulation. Empirical benchmarking using simulated and real data against nine existing methods demonstrates convincingly that the proposed model works well.

Critical strengths

The paper does an excellent job of framing the problem, laying out the state-of-the-art methods, and describing its contribution to the field. The model is described in detail, as are the experimental methods.

The empirical results are very strong, with the proposed method outperforming every other method in nearly every setting.

Critical weaknesses

None

Questions for authors

Why does your method exhibit perfect performance according to all three performance measures on Simulation 1, scenarios 1-3? The text points out that SpaLVM ranks first, but doesn't explore why (or even mention that) its performance is perfect.

It seems strange that COSMOS gets a better LISI score even though its clusters are erroneous. Does this point to a problem with your evaluation setup?

2 comments
AuthorsOpenReview ↗

Response to Question 1

Response to Question 1: We thank the reviewer for these insightful comments. We agree that the perfect performance of SpaLVM across all three evaluation metrics in Simulation 1 (Scenarios 1–3) deserves further explanation. This performance can be partially attributed to the nonlinear mapping, which provides greater flexibility in capturing complex relationships between the latent representations and observed multi-omics data, compared with the linear mapping used in SMOPCA. This advantage is also consistently observed in our comparisons between SpaLVM and SMOPCA across different experiments.

In addition, the performance of SpaLVM benefits from its ability to jointly model spatial dependencies and multiple modalities, as demonstrated by the ablation experiments in Tables S3–S5 and Figures S29–S32. The additional comparisons of different spatial regularization strategies in Figures S33–S36 further demonstrate the benefits of our original formulation in Equation (15).

Together, these model components enable SpaLVM to effectively capture nonlinear multi-omics relationships while preserving spatial structure, thereby explaining its perfect performance in these simulation scenarios.

AuthorsOpenReview ↗

Response to Question 2

Response to Question 2: We thank the reviewer for these insightful comments. We agree that a better LISI score does not necessarily indicate more accurate clustering. The LISI (Local Inverse Simpson’s Index) metric was originally introduced by Korsunsky et al. in “Fast, sensitive and accurate integration of single-cell data with Harmony” (Nature Methods, 2019) and has since been adopted in various studies, including SpatialPCA and SMOPCA.

In our study, LISI is calculated based on cell locations and predicted cluster labels to quantify the diversity of predicted labels within local spatial neighborhoods. A lower LISI score indicates greater local spatial consistency (i.e., smoother spatial distributions) of the predicted labels. However, greater spatial smoothness does not necessarily translate into better clustering accuracy, as excessive spatial smoothness may obscure genuine spatial heterogeneity and lead to erroneous cluster assignments. Therefore, the LISI score alone is insufficient to assess clustering accuracy and should be interpreted alongside complementary biological and histological evidence, such as gene/protein marker expression patterns and H&E tissue morphology. Thus, the better LISI score obtained by COSMOS does not contradict its erroneous clustering results; rather, it highlights the distinction between spatial consistency and clustering accuracy. We have revised the manuscript to clarify the interpretation and limitations of the LISI metric in Appendix C.3, “Evaluation Metrics” (Lines: 1441-1445).

Reviewer_C9CjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpaLVM, a probabilistic model for nonlinear dimensionality reduction and spatial domain detection in paired spatial multi-omics data. The model combines a spatial Gaussian process prior over shared latent variables with modality-specific Gaussian process mappings from the latent space to observed features. The authors develop structured variational inference with inducing variables for both the spatial and mapping components, and optimize a composite objective constructed from modality-specific variational bounds. Spatial domains are obtained by applying K-means to the inferred latent means. Experiments cover synthetic data, semi-synthetic benchmarks constructed from seven single-cell datasets, and two real spatial multi-omics datasets. The reported results are promising, particularly the clustering performance and acceleration over the authors’ fixed-point implementation. My main concerns concern the attribution of these improvements to the proposed methodological choices, the limited validation on natural tissue structures, and the incomplete evaluation of the sparse approximation’s accuracy-cost trade-off.

Critical strengths
  1. The model clearly separates spatial dependence among latent states from nonlinear relationships between those states and molecular measurements.
  2. The sparse spatial parameterization addresses an important computational bottleneck.
  3. The evaluation includes nine baselines spanning spatial and non-spatial, single-modality and multimodal approaches.
Critical weaknesses
  1. Spatial multimodal latent-variable modeling is already represented by methods such as SMOPCA and SpaVAE, while nonlinear shared GP latent spaces have substantial prior literature, including "Multi-view Learning as a Nonparametric Nonlinear Inter-Battery Factor Analysis." The manuscript discusses several relevant predecessors, but the experiments do not clearly establish which proposed component explains the gains. The current ablations remove modalities or spatial dependence; they do not directly test nonlinear versus linear mappings under otherwise matched conditions. Comparisons with separate linear methods are informative, but also change other modeling and implementation choices.
  2. Equation (15) explicitly optimizes a weighted sum of modality-specific bounds rather than the standard joint-likelihood ELBO. With equal weights, the spatial KL has coefficient MM relative to the summed data-fit terms, instead of coefficient one. The manuscript explains this deliberate choice, but does not demonstrate its empirical advantage or characterize its effect on spatial smoothing. Moreover, equal modality weights do not equalize modality contributions because each HmH_m sums over its features. The behavior under substantially different feature counts or noise levels remains unclear.
  3. Validation on naturally occurring tissue structure is limited. Only two real spatial datasets are evaluated, and only mouse brain provides reference domain labels. The seven datasets in Simulation Study II supply real molecular measurements, but their cells are assigned to predefined spatial patterns; they therefore do not constitute seven additional natural spatial tissue benchmarks. These results leave uncertainty about boundary preservation, small domains, and spatially intermixed populations.
  4. The speed improvement over SpaLVM* is convincing, but the two implementations use different variational parameterizations. Runtime alone does not establish whether the sparse version preserves representation quality. The manuscript lacks a corresponding accuracy comparison and sensitivity analysis over inducing-point counts. Furthermore, the real datasets contain only 1,949 and 2,491 observations, with 1,024 spatial inducing locations. The complexity analysis supports improved scaling under fixed inducing-point counts, but larger-NN runtime and memory measurements are needed to substantiate practical scalability.
Questions for authors
  1. Could the authors compare the ARD-RBF mapping with a linear mapping kernel while keeping the spatial prior, preprocessing, latent dimensionality, and optimization protocol fixed? How much of the improvement remains attributable specifically to nonlinearity?
  2. How does Equation (15) compare empirically with the standard joint-likelihood ELBO? Does the stronger relative spatial regularization improve agreement with reference labels, increase spatial smoothness, or both? How sensitive are the results to modality weighting when feature counts and noise levels differ?
  3. Could the authors report clustering accuracy, runtime, and peak memory for SpaLVM and SpaLVM* under comparable convergence criteria? A sweep over spatial inducing locations JJ, mapping inducing inputs TT, and a modest range of dataset sizes would clarify the accuracy–cost trade-off.
6 comments
AuthorsOpenReview ↗

Response to Weakness 1 (part 1)

Response to Weakness 1 (part 1): We thank the reviewer for these helpful comments and constructive suggestions. The relationship between GPLVM (and GPLVM related extensions) and SpaLVM is analogous to that between Probabilistic PCA, SpatialPCA, and SMOPCA, as well as that between scVI, TotalVI, and SpaVAE. In addition, we proposed a novel sparse spatial inducing-variable parameterization (Section 2.2.2) to greatly improve the computational efficiency. Standard spatial GP models face severe computational bottlenecks when applied to large-scale spatial multi-omics data. In SpaLVM, we proposed a sparse spatial inducing-variable variational parameterization method. By utilizing smaller number of spatial inducing locations and whitened inducing variables, we retain the exact GP conditional p(x:l∣vl,S)p(x_{:l}|v_l,S) (i.e., the true Gaussian distribution of the latent representation given the whitened inducing variables vlv_l ​and spatial coordinates SS; Section 2.2.2) and reduce the spatial computational complexity and memory (Appendix A.11).

AuthorsOpenReview ↗

Response to Weakness 1 (part 2) and Question 1

Response to Weakness 1 (part 2) and Question 1: We thank the reviewer for these helpful comments and constructive suggestions. We would like to clarify that the existing comparison between SMOPCA and SpaLVM already serves as an ablation study of linear versus nonlinear mappings. Specifically, SpaLVM employs an ARD-RBF kernel to induce a nonlinear mapping from latent variables to observations. Replacing the ARD-RBF kernel with a linear kernel reduces this mapping to a linear form, resulting in a linear latent variable model. Thus, SMOPCA can be viewed as the linear counterpart of SpaLVM, and the performance differences between the two methods, evaluated using the same datasets, preprocessing procedures, and evaluation metrics, provide a direct assessment of the benefits of nonlinear mappings.

The differences in optimization strategies between the two methods arise naturally from their respective model structures. SMOPCA has a relatively simple model structure, allowing its parameters to be estimated through maximum likelihood estimation (MLE), with some parameters admitting partially closed-form solutions, without requiring variational inference. In contrast, the nonlinear ARD-RBF mapping in SpaLVM makes the marginal likelihood analytically intractable. Consequently, SpaLVM requires variational inference for the spatial latent variables and sparse variational inference with inducing points for the mapping Gaussian processes (GPs).

Therefore, imposing an identical optimization strategy on the two methods would either be impractical for SpaLVM or unnecessarily complicate SMOPCA, and evaluating each method using an optimization framework appropriate to its model structure provides a more principled and fair comparison.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2

Response to Weakness 2 and Question 2: We thank the reviewer for this constructive suggestion. Existing methods, such as SpaVAE and SpatialGlue, only support equal weighting across modalities, and the contribution of each modality to the objective function is therefore influenced by its number of features. We provide a flexible weighting scheme that allows users to specify modality-specific weights according to their needs. As shown in Equation (15), ∑mwm\sum_m w_m helps maintain the relative contribution of the spatial/prior regularizer when the number of modalities increases.

Furthermore, following the reviewer’s suggestion and inspired by the classical β\beta-VAE framework, we introduced an alternative formulation in Equation (16) (Lines: 258-269), providing users with additional flexibility in controlling the balance between the data-fit terms and the spatial KL regularizer. We compared our original formulation (Equation (15)) with the alternative formulation (Equation (16) with (\beta=1)) in both simulation studies and real data experiments (Figures S33–S36). The new results show that the original formulation achieves slightly better performance.

Reviewer_zyfFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this study, the authors developed a new Bayesian latent variable model, called SpaLVM, for nonlinear integration of spatial multiomics data to do the dimension reduction and spatial domain detection. It jointly captures cross-modality variation and spatial dependencies. Their design of sparse variational inference framework largely speeds up the implementation. The authors showed their method outperforms existing nine methods across diverse datasets including simulation and real datasets.

Critical strengths

The method and model design is novel and well motivated. The claims are well supported by the experiments. The experiments well cover the data types and the results are rigorously evaluated across replications.

Critical weaknesses
  1. There is no overview of the model framework or architecture design.
  2. The ablation study only considers the data related components such as the modality and spatial prior but no model related components.
Questions for authors
  1. It would be helpful to have a figure to show the overview of the model/architecture
  2. CellCharter is well used in the literature for spatial domain detection, so I wonder if the authors could explain why they didn’t include it in the benchmarking or perform experiments to include it.
  3. For ablation study, could the authors add more ablation on the model components? This will show how or why the model can achieve good performance.
  4. For Figure 4, I wonder if the red ones suggest the markers for each cluster? If so, for the top panel, the cluster 0 has no good markers. It seems like the genes on red/blue with higher values were selected in the figure. I suggest only considering the highly expressed genes (red) as markers to show. Additionally, it would be helpful to compare the maker genes between two modalities, which might provide some insights about the good performance or why the two modalities are required from a biological perspective. For the figure legend on color, it states “color shows the mean expression”, but since the range is from -2 to 2, I doubt it’s z-score or similarly normalized values rather than mean expression (unexpecting values less than 0). The authors should double check and confirm.
5 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

Response to Weakness 1 and Question 1: We thank the reviewer for this valuable suggestion and fully agree that an overview figure would help improve the clarity and accessibility of the proposed method. Given the limited time available during the rebuttal period, we have prioritized conducting additional experiments to more comprehensively demonstrate the advantages of our proposed method. We will incorporate a schematic overview of the proposed model into the final version of the manuscript.

AuthorsOpenReview ↗

Response to Question 2

Response to Question 2: We thank the reviewer for these helpful comments and constructive suggestions. Following the reviewer’s recommendations, we have included CellCharter in all experiments and updated all the related figures and tables accordingly.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 3

Response to Weakness 2 and Question 3: We thank the reviewer for these helpful comments and constructive suggestions. In response, we conducted additional ablation experiments to investigate the effects of different spatial prior regularization strategies on model performance (Figures S33–S36).

Specifically, inspired by the classical β\beta-VAE framework, we added an alternative formulation in Equation (16), which provides users with greater flexibility in controlling the relative contribution of the data-fit terms and the spatial KL regularizer. We compared the original formulation (Equation (15)) with the alternative formulation (Equation (16) with \beta = 1) in both simulation studies and real data experiments (Figures S33–S36). The results indicate that the original formulation generally achieves slightly better performance.

Together with our previous experiments, we have systematically investigated several key components and design choices of SpaLVM, including:

  • (1) incorporating spatial information versus ignoring spatial information (Tables S3–S5 and Figures S29–S32);
  • (2) jointly modeling multiple modalities versus modeling each modality independently (Tables S3–S5 and Figures S29–S32);
  • (3) nonlinear mapping versus linear transformation (SpaLVM versus SMOPCA in all experiments);
  • (4) different configurations and hyperparameter settings of the spatial KL regularizer (Figures S33–S36);
  • (5) the performance of different variational parameterizations: the sparse spatial inducing-variable variational parameterization (SpaLVM) and the fixed-point variational parameterization (SpaLVM*) (Figures S33–S36).

These ablation studies provide additional insights into the contributions of different model components and help explain the performance of SpaLVM.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Acknowledgment and General Response

We would like to express our sincere gratitude for the opportunity to revise our manuscript and respond to the reviewers’ comments. We greatly appreciate the constructive feedbacks provided by the reviewers, which has helped us improve the manuscript and further strengthen its novelty and significance. We have carefully revised the manuscript in light of their comments and provided detailed, point-by-point responses to all the reviewers’ concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.