acceptodds
Under review as a conference paper at ICLR 2027

Molecularly Guided Tissue Segmentation and Recognition in Spatial Transcriptomics

Abstract

Spatial transcriptomics reveals molecular organization within tissue, but translating these patterns into tissue boundaries and class labels remains challenging. We introduce Molecularly Oriented Spatial Segmentation and Tagging (MOSST), a framework that uses molecular evidence to guide both region proposal and tissue recognition. Spatial gene-expression patterns provide box-and-point prompts for SAM3 on H&E and FICTURE maps, supplemented by H&E proposals where molecular signal is weak. Candidate masks are selected without tissue-region annotations. GPT-6 then classifies each candidate using H&E and a Cell Map displaying annotated cell types within and around its boundary. We evaluate MOSST on ten lung tissue microarrays with 75 confidence-annotated ground-truth regions. Evaluation of all selected candidates yields 37/75 regions both localized and correctly classified under one-to-one matching at intersection-over-union ≥ 0.10. In a separate recognition experiment, candidates are paired to ground-truth regions by geometric overlap and held fixed. Adding Cell Map to H&E increases correct classifications from 33/63 to 42/63 and also outperforms the tested cell-composition text representation. These findings support combining molecularly guided region proposals with candidate-aligned cell-type maps for tissue recognition. Precise boundary delineation and generalization beyond the evaluated cohort require further validation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
spatial transcriptomics, multimodal learning, tissue segmentation, vision-language models, computational pathology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_xen5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MOSST, a training-free pipeline that segments and names tissue regions in H&E images of spatial transcriptomics (Xenium) samples. For each tissue microarray (TMA), genes are clustered by spatial co-expression into modules, and the top-scoring regions of each module give box-and-point prompts to SAM3, applied to the H&E image and to a FICTURE factor map. An H&E-only branch adds candidates where FICTURE signal is weak, and the candidates are selected by SAM3 score, prompt stability, and overlap without using tissue-region annotations. GPT-6 then assigns each candidate one of twelve tissue classes from H&E and a Cell Map, which shows biologist-annotated cell types around the candidate contour. On ten lung TMAs with 75 annotated regions, 37/75 regions are both localized and correctly classified at IoU ≥ 0.10 (13/75 at IoU ≥ 0.50). In a separate experiment with candidates held fixed, adding the Cell Map to H&E raises correct classifications from 33/63 to 42/63, and it outperforms a cell-composition text input (35/63). The authors state that precise boundary delineation and generalization beyond the evaluated cohort need further validation.

Critical strengths
  1. The pipeline needs no training. The selection parameters were set on a separate Visium HD dataset and then fixed for the ten Xenium TMAs.

  2. The paper describes implementation details such as the SAM3 checkpoint, the selection procedure, and the metrics in the appendix, and states that prompts and code are provided in the supplement.

Critical weaknesses
  1. The practical value of the setting is unclear, and the Cell Map result is hard to interpret. The method needs measured spatial transcriptomics: the GeneMap and the FICTURE map both come from the transcript counts, and the H&E branch that fills in weak-signal areas is defined by the absence of FICTURE signal, so it cannot run without transcriptomics either. The best performance also needs the Cell Map, which is built from biologist-provided cell-type annotations. In the fixed-candidate experiment, adding it raises correct classifications from 33/63 to 42/63 (9 of the 42 correct predictions, about 21%), but these annotations already hint at the tissue structure, and the authors acknowledge that the experiment does not separate spatial arrangement from cell-type identity. Producing both inputs is likely to cost more than drawing the tissue regions directly on H&E, yet the paper does not say in which situation obtaining tissue names and outlines this way is worthwhile. This matters because the whole contribution rests on these inputs being worth their cost, and the paper does not show that.

  2. The evaluation is too weak to support the claims. The abstract reports only 37/75 at IoU ≥ 0.10, where a 10% overlap counts as a match. At IoU ≥ 0.50 the number drops to 13/75, so boundary accuracy is not adequately validated. The study also uses only ten lung tissue microarrays (TMAs), 75 ground-truth regions (63 in the recognition experiment), and a single cohort, so generalization to other tissues or datasets is not shown. This matters because with this scale and this criterion the reported numbers cannot show that the method works reliably.

Questions for authors
  1. How stable are the results when GPT-6 is called repeatedly? The requests do not set temperature, top-p, or a seed, and each input condition uses only one response per candidate. Could you report the variation of the main results (e.g., 37/75 and 42/63) across repeated runs?

  2. What happens if the cell-type annotations for the Cell Map are predicted by an off-the-shelf model instead of using the biologist-provided annotations?

  3. In what practical situation is this method better than annotating the tissue regions directly? How does it compare with existing spatial-domain methods?

1 comment
AuthorsOpenReview ↗

Response to Reviewer xen5

We thank Reviewer xen5 for a thoughtful review, and in particular for raising practical value and stability. Both led to new experiments in the revision; point-by-point replies follow.

W1/Q3. Practical value and cost of the inputs. Thank you for raising this; we agree that practicality is central for adoption.

  • No extra data. MOSST uses only what an imaging-based spatial transcriptomics experiment already produces.
  • No training, and a bounded cost. MOSST needs no training and makes at most 40 recognition calls per core.
  • No expert annotation needed. The fully annotation-free version replaces expert cell types with the FICTURE factor map, which is learned from transcripts alone. End to end, it finds and correctly names:
    • 51.5% of structures and lesions, against 50.0% with expert cell types;
    • 57.9% of well-defined structures, against 55.3%.

The molecular information also improves segmentation itself: molecular prompts find 71.2% of regions, against 60.6% for the same number of grid prompts.

W2. Evaluation strength. We have strengthened the evaluation in two ways.

  • Stricter metric. End-to-end results now use one-to-one matching, and we additionally report panoptic quality at IoU > 0.5 (SQ and PQ, as in PanNuke and CoNIC); IoU ≥ 0.1 is no longer used. 55.3% of well-defined structures are found and correctly named, and matched regions have a mean IoU of 0.87.
  • Larger scale. On a public Xenium pulmonary fibrosis cohort (Vannan et al., 2025; GEO GSE250346; 31 held-out samples, 441 pathologist-annotated regions, parameters fixed beforehand), MOSST finds 55.6% of structures and lesions, against 46.1% for grid prompts (paired +9.4 points, 95% interval [+2.5, +15.6]). With pathologist outlines as masks, GPT-6.1 names 93.5% of structures correctly at the coarse class level.

Q1. Stability of repeated GPT calls. Thank you for this suggestion.

  • We repeated the full recognition step on 400 candidates with identical inputs (GeneMap-only candidate set).
    • The top-1 class is identical for 91.8% of candidates, and the coarse class for 94.2%.
    • Found and correct on the 66 regions is 54.5% vs. 56.1% across the two runs.
  • We also ran open-weight vision–language models locally, with the same inputs and prompt, on 52 found regions. The correct class ranks first for 44.2% with Gemma-4-31B, 28.8% with Qwen3-VL-32B and 11.5% with MedGemma-27B, against 75.0% for GPT-6.1. The pipeline accepts any recogniser.

Q2. Predicted instead of biologist-provided cell types. We have run this comparison on the same 52 found regions (found with GeneMap-only candidates). Correct class first:

InputCorrect first
H&E alone61.5
+ CellTypist cell types (public lung-fibrosis atlas)67.3
+ FICTURE map (annotation-free)73.1
+ expert cell types75.0

End to end on the final candidates, the fully annotation-free pipeline finds and correctly names 51.5% of the 66 regions, against 50.0% with expert cell types.

We hope these additions address the reviewer's questions, and we are happy to discuss any further points. If the concerns are resolved, we would be grateful if the assessment could reflect this.

Reviewer_7MCfOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript introduces MOSST, a computational framework that integrates spatial transcriptomics-derived molecular patterns, foundation segmentation models (SAM3), and vision-language models for tissue-region localization and recognition. The key idea is to use molecular signals to generate candidate regions and then use multimodal representations, including H&E images and cell-type maps, for tissue classification. The problem is interesting and timely, as bridging spatial molecular measurements with interpretable tissue structures is an important challenge in spatial omics analysis. The authors provide an end-to-end workflow and evaluate it on lung tissue microarrays.

Critical strengths

1.The authors tackle an important problem: how to connect spatial molecular information with interpretable tissue structures. Unlike typical spatial domain detection methods, their framework aims to localize and recognize actual biological tissue regions. This has potential applications in spatial omics and computational pathology.

2.The authors combines spatial gene expression, histological images, and cell-type annotations, and they build on foundation segmentation and vision-language models. Overall, this gives a practical workflow for localizing and recognizing tissue regions.

3.The authors compare different molecular and histological inputs. Notably, when they add Cell Map information to H&E, tissue recognition accuracy on fixed candidates goes from 33/63 to 42/63. This shows that cell-type information can genuinely complement histological images, and the authors back it up with empirical evidence.

Critical weaknesses
  1. The framework mostly integrates together existing tools, including spatial gene-expression analysis, SAM3, and vision-language models, rather than proposing a genuinely new computational method. The integration is useful in practice, but the methodological novelty is thin. It's largely a combination of established components.

  2. The experiments show that adding Cell Map improves recognition accuracy, but it's not clear whether this gain comes mainly from spatial molecular organization or simply from the extra cell-type annotation. The authors need more careful ablation studies to separate these two factors and to show what spatial transcriptomics actually contributes on its own.

  3. The manuscript presents MOSST as a tissue segmentation and recognition framework, but the boundary agreement is fairly weak. Only 13 of 75 annotated regions are correctly localized and classified at IoU ≥ 0.5. This makes you wonder whether the method is really doing accurate tissue segmentation, or whether it's mostly doing rough region localization and recognition.

Questions for authors
  1. I think the main concern is that the proposed framework appears to be primarily an integration of existing components rather than a new computational methodology. The proposed pipeline combines GeneMap/FICTURE for molecular representations, SAM3 for segmentation and GPT-6 for tissue recognition. While the integration is interesting, most individual components are existing foundation models or established spatial transcriptomics tools. The manuscript itself states that the contribution is the use of molecular evidence within a segmentation-recognition workflow rather than the cascade itself. Therefore, it remains unclear what fundamental methodological advance MOSST provides beyond a carefully designed application pipeline. The authors should clarify: (1) What is the novel algorithmic contribution? (2) Which component cannot be replaced by existing approaches without losing performance? (3) Does MOSST introduce a new representation, optimization objective, or learning framework? At present, the contribution appears closer to an engineering integration of existing models.

  2. Another major weakness is that the contribution of molecular information itself is not convincingly demonstrated. The recognition experiments show that adding Cell Map improves performance: H&E alone: 33/63; H&E + Cell Map: 42/63 However, the improvement from FICTURE/molecular factor information appears limited: H&E + FICTURE: 37/63; H&E + both maps: 37/63. This raises the question: Is the improvement driven by spatial transcriptomics, or simply by providing cell-type annotations to the vision-language model? The manuscript should include stronger ablation analyses.

  3. The manuscript describes MOSST as a segmentation framework, but the quantitative evidence for accurate segmentation is limited. The reported results show: 37/75 correct matches at IoU ≥ 0.1; 13/75 correct matches at IoU ≥ 0.5. The relatively low IoU ≥ 0.5 performance suggests that the method may be better characterized as region proposal/localization rather than accurate tissue segmentation. The authors should clarify: (1) Is precise boundary delineation an objective of MOSST? (2) How does MOSST compare with existing segmentation methods quantitatively? and (3)Are errors mainly due to segmentation inaccuracies or recognition errors? More detailed segmentation metrics (Dice, mIoU, boundary accuracy) would strengthen the claims.

  4. The evaluation is performed on: ten lung TMAs and 75 annotated tissue regions. Although the dataset is carefully annotated, this scale is relatively limited for evaluating a general tissue recognition framework. The manuscript also acknowledges: rare tissue classes, lack of external validation, no independent test cohort. For a framework intended to combine foundation models and spatial omics, broader validation would be important. Suggested experiments: additional tissue types, external cohorts, cross-platform evaluation, cross-laboratory validation.

  5. The manuscript uses several different denominators: all candidate masks, accessible GT regions, and representative candidates. Although the authors carefully explain these choices, the evaluation remains difficult to interpret. For example, recognition performance is evaluated only after candidate selection and GT-based pairing. Therefore, the recognition results do not evaluate the complete autonomous pipeline. The authors should provide a clearer primary metric reflecting the end-to-end task. Given an unseen spatial transcriptomics sample, how accurately can MOSST recover and annotate tissue regions without any human intervention?

  6. Misalignment between paper narrative and experimental results, and insufficient highlighting of original contributions. The title claims “Molecularly Guided Tissue Segmentation and Recognition”. However, experimental results demonstrate that molecular information is mostly leveraged to guide segmentation candidate generation. The final recognition performance is largely affected by prompts and externally provided Cell Map annotations. This creates confusion for readers regarding the actual role of molecular information in the downstream recognition step. Furthermore, the original contributions of this work are not sufficiently emphasized, as novel designs are interleaved among invocations of multiple off the shelf modules. It is suggested to revise the writing, clearly distinguish reused existing tools from the original designs proposed in this paper in the abstract, introduction, related work and method sections, and better highlight the unique contributions of this work.

  7. Lack of sensitivity analysis for thresholds and hyperparameters. Multiple empirically-set hyperparameters are adopted in this pipeline, including thresholds for constructing GeneMap modules, generating spatial prompts and overlap thresholds for candidate selection. Nevertheless, sensitivity experiments for these hyperparameters are absent in the manuscript. As a result, the robustness of the chosen hyperparameter set remains unclear, and it is unknown how performance varies with threshold perturbations. It is recommended to add sensitivity analysis for key hyperparameters.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7MCf

We thank Reviewer 7MCf for a thorough review and many concrete suggestions. Each one led to a new analysis in the revision; point-by-point replies follow.

Q1/W1. Methodological novelty. We have made the contributions explicit in the abstract and introduction:

  1. Segmentation of irregular, biologically meaningful structures from spatial transcriptomics and hematoxylin and eosin (H&E) images, instead of tiles.
  2. Spatial-data-guided prompting from gene co-expression peaks and FICTURE expression-factor regions, with two prompt forms on two images and agreement-weighted budgeted selection.
  3. Input construction that lets a pretrained vision–language model identify tissue structures from molecular maps.

Each component is ablated on the 66 regions:

  • Prompt forms: one form gives 57.6%; both forms with agreement weighting give 71.2%.
  • Images: H&E masks only 60.6%; FICTURE masks only 62.1%.
  • Grid prompts: 60.6% alone; 65.2% with FICTURE regions added.
  • Prompt sources: GeneMap only 74.2%; FICTURE regions only 37.9%. On these small cores, GeneMap peaks already reach most structures. FICTURE-region prompts add the small structures that dominate dense external sections.

The pipeline works with other segmenters. With GeneMap prompts, clean-up and selection fixed, the share of regions found is:

SegmenterFound
SAM374.2
SAM 2.166.7
SAM60.6
MedSAM256.1
PathoSAM50.0

MOSST needs no training. Revised PDF: Introduction; Table 2; Appendix (segmentation models).

Q2/W2. Spatial transcriptomics or cell-type annotation? Thank you for this important question. We separated the two effects:

  • Proposal: molecular prompts find 71.2% of regions, against 60.6% for grid prompts.
  • Recognition, on 52 fixed regions (correct class ranked first):
InputCorrect first
H&E61.5
+ Cell Map75.0
Shuffled legend51.9
Shuffled cell arrangement76.9
+ FICTURE map (no annotation)73.1
+ CellTypist cell types67.3
  • The gain holds without expert annotation: the annotation-free pipeline reaches 51.5% end to end, against 50.0% with expert cell types.

Q3/W3. Segmentation accuracy and source of errors.

Stage (66 regions)Retained
A good mask exists in the candidate pool89.4%
The mask is among the 40 selected68.2%
Found (tolerant matching)71.2%
Matched at IoU > 0.5, any class68.2%
Also correctly named (fine / coarse)50.0% / 60.6%

Among the 50 found regions, the correct class ranks first for 72.0% and in the top three for 92.0%. Most naming errors are between related classes, such as venule vs. arteriole.

Q4/W4. Scale. We added a public Xenium pulmonary fibrosis cohort as an external test (Vannan et al., 2025; GEO GSE250346): 31 held-out samples, 441 regions, with parameters fixed beforehand. MOSST finds 55.6% of structures and lesions, against 46.1% for grid prompts (paired +9.4 points, 95% interval [+2.5, +15.6]). Details are in the general response.

Q5/W5. Primary end-to-end metric. We agree and have revised the evaluation.

  • The primary end-to-end metric is the share of regions that are found and correctly named, with masks and regions matched one-to-one. We also report panoptic RQ, SQ and PQ at IoU > 0.5, as in PanNuke and CoNIC, and a class-aware Dice (Appendix G). IoU ≥ 0.1 is no longer used.
  • Results: 55.3% of well-defined structures (PQ 0.456) and 50.0% of the 66 regions (PQ 0.408).
  • Annotation is not exhaustive, so unmatched candidates in unannotated tissue are not counted as false positives. Revised PDF: Table 3; Appendix G.

Q6/W6. Narrative. The method section now states where molecular information enters. In proposal, it enters through GeneMap and FICTURE-region prompts and the FICTURE map. In recognition, it enters through the Cell Map or FICTURE map. The reused tools (SAM3, FICTURE, GPT-6.1) are named as such.

Q7/W7. Sensitivity (measured with GeneMap prompts).

  • Clean-up radius (none to 64 px): 71.2–77.3% found. Without clean-up, mean best IoU drops by 0.05.
  • Thirteen alternative FICTURE maps: 66.7–71.2%, against 74.2% for the joint K = 14 map.
  • Fifteen GeneMap threshold sets: 53.0–75.8%, against 74.2%. Only sets covering the top 2–3% of bins lose more than 8 points.
  • Sixty-six upstream variants (bin size, smoothing, gene filters, module count, merging, point placement, FICTURE rendering): 40.9–75.8%. None improves by more than 1.6 points, and fixing four modules is the largest loss.
  • Candidate budget of 20/40/50/80 per core: 46.7/66.7/73.3/82.7% of all 75 regions. Revised PDF: Section 5.6; Appendix F.

We hope these results address the reviewer's concerns, and we welcome further questions. If the concerns are resolved, we would be grateful if the assessment could reflect this.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We sincerely thank all three reviewers for their time and for careful, constructive reviews. We rebuilt the segmentation stage and the evaluation, and answered each requested control with a new experiment. Compared with the submission, the method adds FICTURE-region prompts, point-only masks as a second prompt form, and selection weighted by agreement between the two forms. The evaluation now uses re-annotated outlines, a stricter one-to-one found criterion with panoptic quality at IoU > 0.5, boundary metrics, an equal-budget grid baseline and an external cohort; because annotation and metrics changed, we report results under the new protocol rather than as differences from the submitted numbers. The individual replies address each point in detail.

Unless stated otherwise, numbers come from the final pipeline on the ten Xenium cores, with at most 40 candidates per core. Found and recognition rates are percentages of regions; IoU, SQ, PQ and Dice are fractions.

Contributions

  1. Segmentation of irregular, biologically meaningful tissue structures from spatial transcriptomics and hematoxylin and eosin (H&E) images, instead of equal-sized tiles, without training or annotation.
  2. Spatial-data-guided prompting. Prompts come from gene co-expression peaks and from FICTURE expression-factor regions, use two prompt forms on two images, and pass through agreement-weighted, budgeted selection. This finds 71.2% of regions, against 60.6% for an equal number of grid prompts under the same rules.
  3. Input construction that lets a pretrained vision–language model identify tissue structures from spatial transcriptomics: Cell Maps, expression-factor maps and automatically typed cells, tested with label-shuffled and view controls.

Registration and annotation. H&E and Xenium are aligned by the full-slide affine registration exported from Xenium Explorer. Nucleus centroids fall on H&E nuclei within 0–6 pixels (0–2.6 µm) on every core (Appendix A). Cell-type labels were re-annotated and region outlines were refined; results on the original outlines are also reported.

Main results

Regions (n)FoundMean best IoUFound & correctPQ
All except alveoli (66)71.20.63050.00.408
Well-defined structures (38)81.60.71755.30.456
Lesions (28)57.10.51242.90.342
Alveoli, reported separately (9)33.30.36033.30.219
Annotation-free MOSST, 66 regions71.20.63051.50.419
Annotation-free MOSST, 38 structures81.60.71757.90.482

Regions matched at IoU > 0.5 to a correctly named mask have a mean IoU (SQ) of 0.80–0.87 in every group except alveoli. A fully annotation-free MOSST, which uses the FICTURE factor map in place of expert cell types, performs on par with the expert-annotated version.

External cohort: a public Xenium pulmonary fibrosis cohort (Vannan et al., 2025; GEO GSE250346).

  • We ran MOSST on 31 held-out samples (441 regions) with all parameters fixed beforehand. The FICTURE-region prompt parameters were set on 10 separate development samples. External region outlines were refined to tissue boundaries, and nine mislabelled regions were removed.
  • The external cores are 2–4× larger than ours. The per-core pipeline therefore runs on core-sized windows, with a candidate budget proportional to tissue area.
  • Structures and lesions (180): MOSST finds 55.6%, against 46.1% for grid prompts under the same rules (paired +9.4 points, 95% interval [+2.5, +15.6]; 13 samples up, 3 down).
  • Structures: 66.7% vs. 55.6% (+11.1 [+3.3, +18.3]); venules: 61% vs. 36%. End to end with the same GPT-6.1 recogniser, MOSST finds and names 57.3% of the 150 structures and lesions whose class is in the label set at the level of related classes, against 47.3% for grid prompts (+10.0 points, 95% interval [0.0, +18.0]; exact class 44.0% vs. 39.3%).
  • GeneMap peaks and FICTURE-region prompts are complementary:
PromptsOur cohortExternal
GeneMap only74.234.4
FICTURE regions only37.959.4
Both (MOSST)71.255.6
  • With the pathologist outlines as masks, GPT-6.1 with the same prompt names 93.5% of 46 structures correctly at the coarse class level (67.4% fine).

New experiments in the revision

  • Ablation of every component, and a candidate budget curve.
  • Recognition input controls: H&E only, Cell Map only, shuffled legend, shuffled cell arrangement, FICTURE map, and CellTypist cell types. Also three crop views.
  • Eight pathology encoders, three open-weight VLMs, five promptable segmenters, and a full repeat of the recognition step.
  • A fully annotation-free pipeline.
  • Sensitivity to clean-up, FICTURE maps, GeneMap thresholds and upstream parameters.

We are grateful for the reviewers' suggestions, which made the paper clearer and the evidence considerably stronger. We would be glad to answer any further questions during the discussion period.

AuthorsOpenReview ↗

Response to Reviewer WDSh

We thank Reviewer WDSh for a detailed and constructive review. We ran every suggested experiment and added it to the revision; point-by-point replies follow.

W1. Molecular prompts vs. a simple grid. Both arms used the same number of prompts, the same SAM3 settings, selection rule and budget, and no size limit.

  • MOSST finds 71.2% of the 66 regions; grid prompts find 60.6%.
  • End to end, with the same GPT-6.1 recogniser and identical inputs, grid candidates are found and correctly named for 42.4% of the regions, against 50.0% for MOSST. PQ is 0.328 vs. 0.408, and well-defined structures reach 47.4% vs. 55.3%.
  • Before selection, MOSST's candidate pool already holds a good mask (IoU ≥ 0.5) for 89.4% of the regions. Revised PDF: Section 5.2, Table 2; Section 5.3.

W2. Boundary agreement and expert assessment. Cell-type labels were re-annotated and region outlines were refined.

  • The original expert outlines were drawn before MOSST was developed, and only for structures the annotators could identify with confidence, so they are an independent reference. On them, the same candidates reach a mean best IoU of 0.448 (Table 1, last column). Our assessment compares predicted masks with expert-drawn outlines.
  • The main results use the refined outlines. On these, the median best IoU is 0.778 over all 66 regions (0.830 for the 38 well-defined structures, 0.569 for the 28 lesions).
  • On the found regions (47 of 66), Dice is 0.898, boundary F-score at 20 µm 0.862 and mean surface distance 11.3 µm; for the 31 found well-defined structures 0.904, 0.882 and 10.0 µm, and for the 16 found lesions 0.887, 0.825 and 13.7 µm. Changes in local cell composition or FICTURE factors may provide additional cues for delineating boundaries visible on H&E.
  • Matched regions (IoU > 0.5, correctly named) have a mean IoU (SQ) of 0.80–0.87 on the refined outlines in every group except alveoli. Revised PDF: Section 5.1, Table 1, Appendix G.

W3. Dependence on biologist-provided cell types.

  • On a fixed set of 52 regions found with GeneMap-only candidates, we compared what each input contributes:
    • H&E alone: 61.5% correct-first.
    • Cell types predicted by CellTypist (model trained on a public lung-fibrosis atlas): 67.3%.
    • Expert labels: 75.0%.
  • Fully annotation-free MOSST:
    • The FICTURE factor map, learned from transcripts alone, gives 73.1% on the same regions.
    • End to end on the final candidates, this version finds and correctly names 51.5% of the 66 regions (50.0% with expert cell types) and 57.9% of well-defined structures (55.3%). PQ is 0.419 vs. 0.408. Revised PDF: Table 4; Table 3, lower block.

W4. Does the cell arrangement matter? Thank you for proposing this control, which we ran as suggested. Within each request, cell-type labels were permuted among the cells inside the target and, separately, outside it, so every dot keeps its position and the composition is unchanged (87% of dots change type). Correct-first is 76.9%, against 75.0% with the original Cell Map (47 of 52 predictions unchanged). Permuting the colour legend instead, which keeps the arrangement but scrambles identities, lowers it to 51.9%, below H&E alone (61.5%). The recogniser therefore uses which cell types are present in and around the target; their fine arrangement adds little. Revised PDF: Section 5.4, Table 4.

W5. Pathology image encoders.

  • We ran eight image–text encoders (PLIP, BiomedCLIP, QuiltNet, PathGen-CLIP, KEEP, CONCH, MUSK, TITAN) zero-shot on the same 52 regions, with three crop views. The best are CONCH (42.3%) and MUSK (36.5%), on the outlined view.
  • GPT-6.1 on the same views reaches 61.5% with H&E alone and 75.0% with H&E + Cell Map (class-macro AUROC 0.905 and 0.972). Revised PDF: Section 5.4; Appendix Table 12.

W6. Patients, tissues and rare classes. We appreciate this concern and added an independent cohort.

  • Appendix D gives per-class results with the number of regions. Alveoli, which have no outer boundary, are reported separately.
  • External cohort: a public Xenium pulmonary fibrosis cohort with pathologist-annotated regions (Vannan et al., 2025; GEO GSE250346):
    • Setup: 31 held-out samples, with parameters fixed on 10 separate development samples. External region outlines were refined to tissue boundaries, and nine mislabelled regions were removed.
    • Structures and lesions: MOSST finds 55.6%, against 46.1% for grid prompts (paired +9.4 points, 95% interval [+2.5, +15.6]).
    • Structures alone: 66.7% vs. 55.6%.
    • End to end (same recogniser): 57.3% vs. 47.3% found and named at the level of related classes; 44.0% vs. 39.3% exact class.
    • With pathologist outlines as masks, GPT-6.1 names 93.5% of 46 structures correctly at the coarse class level (67.4% fine).

We hope these results address the reviewer's concerns, and we are happy to provide further analyses during the discussion. If the concerns are resolved, we would be grateful if the assessment could reflect this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.