- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposed LION (Local Influence Inference Network), a model for spatial data that aims to make predictions based on spatial bags of features that represent sets of spatial observations, and provide a ranking of the most influential observations that most influenced the predictions. Performance is evaluated on 3 variations of MNIST generated using LION itself, 3 modified H&E pathology datasets, and one generated spatial transcriptomic datasets, each generated or modified to provide exact ground-truth rankings of local influence. Further results on real biological data, including Visium as well as Xenium and TCR sequencing data are provided.
Critical strengths
Fewer parameters: As stated in the results, LION performs similarly or competitively with baseline methods, such as GNNs and DeepSet models, including substantially more parameters. The benefits of this less parameterised model are convincing in terms of model complexity.
Additional ablations: The paper reports a wide range of additional ablations that clarify the main results, including the effects of different model components, and further exploratory and descriptive results on real spatial transcriptomics datasets.
Biological relevance: The main setup, aiming to predict spatial labels and evaluate the influence of local observation is relevant for the chosen domain of spatial transcriptomics data analysis.
Critical weaknesses
Readability: The abstract uses very technical language, for example there would likely be a way to describe the proposed methods in a manner more understandable to a lay person. It introduces variables, such as G + 4, the number of trainable parameters, without defining G.. Further, the abstract describes results on “Dense” or “morphology-interaction” tasks, without specifying what these setups refer to. Similarly, technical terms such as reactive or bystander fraction, amongst other concepts, could benefit from further clarification. Similarly, when first reading the manuscript and reaching line 58, I remain unsure what the mentioned coefficients and sign reversal refers to as it only becomes clearer much later in the text. Overall, by providing further clarifications, defining all introduced variables, and explaining technical terminology whenever possible there is great potential to improve the readability of the text for a general ML audience.
Mathematical definitions: The presentation of the mathematical results has many imprecise parts and could be improved. Def 1(i) does not detail for which i the prediction should hold, likely for each i \in I, but it would be nice to specify. Similarly, it would be useful to clearly define Sg and Sh in line 117, to define s^ in line 128 and so on.Def 1 (ii) does not clarify how the learnt vector relates to the prediction function or if it needs to relate at all. Given how much of the other concepts in the following section are not given as definitions, I wonder if this problem statement should be a definition itself without further formalisation. Overall, the theoretical results, specifically the proofs and text in the appendix, could benefit from clearer formalisation to distinguish new theorems and proofs from definitions or general text.
Appendices: While it is useful further materials are included, the appendices are referenced frequently in the main text, including when describing the main empirical results. Overall, this appears as if the main text is not self-contained in itself, but rather requires substantial additional reading, making it difficult to follow and review. For example, the main results on the real biological data are only briefly discussed in Section 6.5, but almost exclusively contained in the appendices. Clarifying and presenting the results to be more concise and self-contained in the main text and focusing the appendix on the most relevant additional content would greatly improve the presentation.
LLM-generated text: Having read the full manuscript, I believe that substantial parts of the text read as if they have been LLM-generated, including large parts of the appendices, and potentially some of the theoretical results, making it difficult to follow the full text. I believe the presentation could benefit from further manual edits to the text to clarify and explain the key results.
Uncertainty evaluation: Due to the lack of uncertainty evaluation reported in the main text it is difficult to assess whether there are critical differences in performance between models, or clear differences in results, say for the main results in Figure 3, 4, 5, or 8. Including relevant confidence intervals for all of the results would greatly strengthen the evaluation. For example, because the results in Figure 4 are averaged after cross validation, it would be to at least report standard deviation estimates as detailed in the full results in the Appendix in Table 5.
Baseline comparisons: LION is compared to only the best rather than all evaluated baseline methods across the results in the main text. For many evaluated datasets there does not seem to be notable differences between LION and this best baseline. Overall, based on the relatively high number of ties reported, it is difficult to assess how significant or relevant an advancement the proposed model is making over the most competitive baseline methods, such as GNN+dist. The full results in Table 5 seem to support that overall there are no significant performance differences between LION and GNN+dist on many of the biological datasets.
Limited relevance of the MNIST evaluation: The biggest performance differences are seen on sparse, dense, or correlated variations of MNIST, which model digits, and are not directly relevant to biological applications. Hence, while LION seems to perform well on this simple image dataset, this does not provide clear evidence that LION would necessarily outperform baseline methods at applications to spatial transcriptomics or histopathology data.
Biological interpretation: The scope, relevance, and impact of the biological results in Section 6.5 is only briefly summarised, and likely deserves further clarification. To address a general ML audience, or even a specialised audience interested in spatial transcriptomics, more detailed explanations and clarifications would be beneficial in the main text.
Missing experimental details: Despite the extensive further materials, certain data processing steps, such as the choice of parameters for Leiden clustering (line 1442), the gene sets used to annotate the cell types, a reference to exactly how the datasets were generated, and the number of generated spatial samples are not explained in appendix K.
Questions for authors
General questions:
- Does influence always decay over distance? It seems to be worth discussing this spatial assumption as a potential limitation of the proposed setup.
- Does the learned ranking as defined in Def1 not also need to relate to the learned predictions?
- Is it not a bit circular to evaluate LION on LION-generated MNIST examples?
Main suggestions:
- Could you provide a more self-contained and concise summary of your main results in the main text rather than relying heavily on additional appendices?
- Could you report the full results in the main text with suitable uncertainty estimates?
- How confident are you that LION will generally outperform alternative predictive models at tasks in spatial biology in particular? Can you comment on for which types of biological datasets and tasks you expect LION to perform well and for which applications it might fail?
3 comments
Response to Reviewer GK34 (2/2): evidence beyond the model form, biology and missing details
Spatial-MNIST. This benchmark is intentionally LION-form: it tests Theorem 1 under correct specification. Evidence beyond the model's own form comes from three sources. First, the spatial transcriptomics rule and the three H&E rules are not LION-form, and LION's Spearman correlation on them is 0.577 to 0.698 (Table 8). Second, in the interaction-strength sweep of Appendix V, as within-instance interactions grow to 83% of the instance-score variance, LION's Spearman changes by only −0.003, whereas each distance-equipped baseline declines by 0.12 to 0.15. Third, a new semi-synthetic benchmark on real Xenium expression and cell geometry (Appendix W) has 360 features, 20 of them active, and four label rules outside LION's generative family: linear, saturating, pairwise interaction, and an annular distance profile. Under every rule, LION's signed Spearman correlation is 0.24 to 0.27, compared with at most 0.085 for six baselines, and its AUROC for separating active from null features is 0.70 to 0.76, compared with at most 0.64.
Biological interpretation. Section 6.5 now ends with a paragraph that explains what the two tissue studies show. In both, the models are trained only on the presence of a cell type, and the biological readouts use information the models never see. In the tumors, LION recovers more TCR-defined reactive T cells, the subset most relevant to antitumor immunity, than models with similar accuracy, and its top-ranked tumor-cell genes are enriched for predicted HLA-binding peptides. In myocarditis, the learned ordering associates mitochondrial respiration in cardiomyocytes with the absence of T cells and macrophages but not of B cells.
Does influence always decay with distance? No, and LION's kernel is monotone non-increasing, so it cannot represent an influence that peaks away from the focal cell. We tested this directly with an annular rule whose influence peaks at 30 µm (Appendix W). LION's ranking recovery under this rule (Spearman 0.273) matched that under the monotone linear rule (0.259), so the misspecified distance profile did not degrade the feature ranking in this setting, although the learned kernel does not reproduce the peak. We discuss this, together with interaction-dominated targets, region-dependent rankings, and overlapping neighborhoods, in a new paragraph of Section 6.4, "When LION is appropriate."
Missing details. Appendix K now states the protocol used to generate the VisiumHD data (10x Genomics user guide CG000685), the marker genes for the cell types used in our analyses, the Leiden clustering parameters (resolution 1.0, 15 nearest neighbors), and the number of spatial samples: two mice, each profiled on one VisiumHD tissue section.
Dear Reviewer GK34,
Thank you again for your thoughtful review. As the discussion period is drawing to a close, we wanted to gently follow up on our response and revision, which rewrites the abstract and introduction and adds Definition 1, Table 1, and the new discussion in Section 6.4. We would greatly appreciate it if you could let us know whether these changes address your concerns, and we are happy to clarify anything further.
Best regards, The Authors
Response to Reviewer GK34 (1/2): ranked object, presentation and uncertainty
We thank the reviewer for the detailed feedback. We first clarify the object being ranked. LION ranks features, such as genes, rather than observations: each feature g receives one learned coefficient Sg, and the ranking is the ordering of these coefficients. The observations in a bag, such as the cells around a focal location, contribute to the bag prediction through a distance kernel, and the same coefficients that define the ranking also define that prediction, f(B)=f(B;S^,φ). Our primary goal is therefore recovery of the true feature ranking, which we evaluate on benchmarks with known ground truth. Predictive accuracy is a secondary requirement, and we now describe LION's prediction as competitive rather than superior.
Readability and definitions. We revised the abstract and introduction for a general ML audience. The abstract no longer uses undefined symbols or benchmark names and defines reactive and bystander T cells; the introduction now states what the coefficients are before the reversal result; and Section 6.1 defines each benchmark name. Definition 1 now states the index range, defines Sg∗, and makes S^ the feature score vector of the predictor, f(B;S^,φ); s^ is defined where it first appears.
Self-containment. The length of the appendix reflects the scope of the paper. We introduce a new problem, recovering a shared feature ranking from bag-level labels, together with a model whose ranking is identifiable. Supporting this requires both theory, namely identifiability, interval estimation, and their proofs, and an evaluation broad enough to show that LION works beyond its own model form: benchmarks with known rankings on synthetic, histology, and spatial transcriptomics data, component ablations, robustness and capacity analyses, and two biological applications. We placed this material in the appendix so that the main text could present the problem, the method, and the main results. The revision adds four analyses in the same way. In Appendix P, a four-step ladder fitted on identical folds attributes most of LION's ranking advantage over DW-logistic to the learned feature normalization and distance kernel. In Appendix D.5, simulated coverage of nominal 95% intervals for the feature contrasts is 0.93 to 0.95 under correct specification. In Appendix L, under four-fold spatial block cross-validation of the Xenium study, LION reaches AUROC 0.80 versus 0.74 for the strongest baseline. In Appendix W, on a semi-synthetic benchmark with real Xenium expression, LION recovers the feature ranking best under four label rules outside its model form. To make the main text readable on its own, Section 6.1 now contains a protocol table that lists, for the main analyses, the data, splits, seeds, selection rule, and where the results are reported, and it defines each benchmark shown in the figures. Section 6.5 now ends with a paragraph explaining what the biological results show.
Writing. In preparing the submission, we used an LLM only to polish the language. The problem formulation, method, theory, experiments, and analyses are our own, and the length of the paper comes from the experimental details and results we report, not from LLM-generated text. In this revision, we have also revised the language to make it clearer and more direct, most substantially in the abstract and introduction.
Uncertainty and baselines. Figure 4 now shows error bars (±1 standard deviation over the 25 runs, 5 folds × 5 seeds): for AUROC and AUPRC, on LION and on the best distance-equipped baseline, and for Spearman correlation, on LION. Table 8 now reports the standard deviation of LION's Spearman correlation on every benchmark:
A standard deviation of 0.000 denotes a value below 0.0005. Table 7 reports every distance-equipped baseline with its standard deviation, and the caption of Figure 4 now points to it.