acceptodds
Under review as a conference paper at ICLR 2027

ViO-Align: A Decoupled Vision-Omics Alignment Method for In Situ Cell Phenotyping

Abstract

Routine hematoxylin and eosin (H&E) slides preserve both cellular morphology and surrounding tissue architecture, providing a practical basis for phenotyping cells in their native microenvironment. However, conventional vision models are mainly trained with morphology-defined labels, while molecularly supervised approaches often operate on cropped or prelocalized cells and therefore do not directly connect molecular supervision with raw-patch in situ phenotyping. ViO-Align is introduced as a decoupled vision–omics framework that transfers single-cell molecular supervision to H&E-only in situ cell phenotyping. In Stage I, matched H&E cell crops and transcriptomic profiles are aligned through morphology-conditioned gene gating to learn molecularly guided single-cell representations. In Stage II, cells are localized from uncropped H&E patches using count-guided flow, while center-masked microenvironment aggregation captures complementary local tissue context. Cross-stage alignment transfers Stage-I molecular supervision to raw-patch representations, and downstream phenotyping explicitly combines target-cell morphology with the corresponding microenvironment representation. Only H&E images are required at inference. Across four established cell phenotyping benchmarks, ViO-Align improves macro-F1 by 4.67 absolute points on average over the strongest strictly test-disjoint baseline on each dataset, while maintaining competitive cell localization performance. To more directly evaluate molecularly guided representation learning, five molecularly defined cell-state recognition tasks are further considered. ViO-Align achieves the highest AUROC on all five tasks and improves over the strongest vision baseline by up to 3.86 absolute AUROC points. The learned H&E representations also transfer to patient-disjoint breast cancer diagnosis. Overall, ViO-Align achieves strong and consistent performance across cell phenotyping, molecular-state recognition, and diagnostic transfer, demonstrating robust generalization across cellular and tissue-level tasks. Code is provided in the supplementary material to facilitate reproducibility.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Computational pathology, Spatial transcriptomics, Cell phenotyping, Cell localization, Multimodal learning, Molecular supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yX2JOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ViO-Align, which transfers single-cell molecular supervision from Xenium spatial transcriptomics to H&E-only in situ cell phenotyping. Stage I performs cross-modal contrastive alignment via morphology-conditioned gene gating (MGG), and Stage II bridges the molecular supervision to raw-patch localization and phenotyping through registered cell identities. ViO-Align gains 4.67 macro-F1 points on average over four phenotyping benchmarks and attains the highest AUROC on all five molecular cell-state tasks.

Critical strengths
  • Clear motivation and well-positioned framework. The paper distills existing gaps into supervision-, context-, and inference-level mismatches (Figure 1). Its cell-level alignment → raw-patch transfer → H&E-only inference design connects them sensibly, with practical value.

  • Complete method with thorough ablations. MGG, CFL, MEA, and cross-stage alignment each play a distinct role, and ablations cover all key components. Removing MGG drops Stage-I Top-1 retrieval from 92.88% to 33.14% (Table 4).

  • Rigorous evaluation protocol. The strictly test-disjoint principle is enforced, excluding baselines whose weights overlap the test set. HoVer-Net and CellViT++ are re-evaluated under identical splits and evaluator (Section 4.1).

  • Comprehensive tasks with consistent gains. Beyond the four benchmarks (average macro-F1 +4.67), five molecularly defined cell-state tasks directly test molecularly guided representations. ViO-Align attains the highest AUROC on all five (Table 3).

  • Compact model. ViO-Align uses only 19.6M parameters, far smaller than baselines such as HoVer-Net (54.7M). This is an advantage in the accuracy–efficiency trade-off.

Critical weaknesses
  • (Primary weakness) The core idea is straightforward, but the writing is over-packaged and hard to read. Stripped of stacked self-coined terms and module names, the paper is essentially cross-modal pretraining on Xenium pairs plus identity-bridged transfer to raw patches. Key details are pushed to the appendix, leaving the training procedure unreconstructable from the main text.

  • The critical no-transcriptomic-supervision control is missing. Stage-I alignment is never replaced by visual self-supervision or ImageNet initialization under an identical budget. The alternative explanation—gains mainly from large-scale upstream pretraining—thus cannot be ruled out.

  • Upstream–downstream tissue overlap is an undiscussed confound. The 15 upstream Xenium datasets include breast cancer samples, while downstream BreaKHis and three molecular-state tasks are also breast cancer. The paper never discusses or removes this confound.

Questions for authors
  • Most critical: please provide the no-transcriptomic control—identical pipeline and budget, with Stage-I alignment replaced by visual self-supervision or ImageNet initialization—and report the downstream results.

  • What is downstream performance after excluding breast-cancer or same-organ upstream data, and how large is the contribution of tissue overlap?

  • Is the anomalous CellViT++ result on MoNuSAC (Fd = 46.15) due to class mapping or the evaluation pipeline? Do the 300 epochs in Table 10 and the "at most 100 epochs" in the text denote different phases, and what are the missing λ_center/λ_count values?

8 comments

We thank you for the detailed and constructive feedback, including the recognition of our framework's motivation and practical H&E-only inference setting. We understand that the key concerns are whether the two-stage procedure is sufficiently clear and reproducible, whether its gains truly come from transcriptomic supervision rather than additional image pretraining, whether same-organ pretraining affects downstream results, and whether the CellViT++ comparison and training specifications are correct. We have therefore reorganized the method description, added a matched-schedule H&E-only control and breast-exclusion analyses, corrected the CellViT++ evaluation, and made the training schedules and loss weights explicit. We address these points individually below.

W1 — "Overly verbose and difficult to read" / "Key details are relegated to the appendix"

We thank you for acknowledging the motivation of the framework while pointing out that excessive terminology and appendix-dependent implementation details obscure the actual training procedure. We understand that the central concern is whether a reader can reconstruct the sequence from single-cell molecular alignment to raw-patch transfer and H&E-only inference without piecing together scattered descriptions. We have therefore simplified the manuscript's organization, brought essential training specifications into the main text, and aligned the ablation discussion with the two supervision-transfer steps. To make the sequence reconstructable at a glance, we first summarize the inputs, supervision, and learned role of each training phase below, followed by the corresponding manuscript revisions.

Overview of the three training phases, including their inputs, supervision, and learned roles.

PhaseInput and supervisionLearned role
Stage IRegistered H&E cell crops and paired RNA; within- and cross-modal contrastive objectivesMolecularly guided cell-image encoder
Stage IIUncropped H&E patches, registered centers, and fixed Stage-I references; localization and identity-matched alignmentPatch localization and contextual cell features
DownstreamH&E patches with cell locations and phenotype labelsJoint localization and phenotype prediction

(1) Simplified organization. The revised Introduction focuses on cell-crop molecular supervision versus raw-patch in situ inference. The Method follows the actual execution order—framework overview (Section 3), Stage-I molecular representation learning (3.1), identity-matched Stage-II transfer (3.2), and H&E-only downstream phenotyping (3.3)—instead of introducing the framework through an expanding set of module labels. MGG, CFL, and MEA remain only where necessary to specify mechanisms and ablations.

(2) Key details moved into the main text. Section 3 now explains the Stage-I/II inputs, objectives, fixed teacher references, physical-cell-ID positives, and the transition from registered to predicted centers at inference. Section 4.1 separately specifies the 50 / 300 / up-to-100-epoch schedules, optimizers, and complete Stage-II loss weights (5, 20, 20, 0.5, 0.1). Target-construction algorithms remain in the Appendix.

(3) Experiments now follow the two transfers. Section 4.3 / Table 5 combines two independently defined controls that test (i) the RNA-derived Stage-I teacher and (ii) the Stage-II alignment objective. These directly correspond to the two mechanisms described in Section 3.

Revision in the manuscript: Sections 1, 3, 4.1, and 4.3 have been rewritten/expanded for clarity and reproducibility; Figures 1–2 are retained, not claimed as newly redesigned.

AuthorsOpenReview ↗

W2 — "The critical no-transcriptomic-supervision control is missing"

We thank you for requesting the direct control needed to distinguish the contribution of transcriptomic supervision from that of additional upstream visual pretraining. We agree that the relevant comparison must preserve the two-stage architecture and training schedules while changing the source of Stage-I teacher supervision. To this end, we have trained a matched-epoch H&E-only self-supervised teacher and compared it with the molecularly supervised teacher under identical Stage-II and downstream procedures. The control design and its downstream results are detailed below.

(1) Exactly what is changed. Stage I uses the same Xenium H&E cell crops, data splits, randomly initialized visual encoder, augmentations, optimizer, batch size, and 50 epochs, but learns only from two-view image NT-Xent. We remove the RNA encoder, MGG, RNA-view objective, and H&E–RNA contrastive objective. This produces a purely visual Stage-I teacher. Both variants then train a newly initialized Stage II with the same uncropped patches, localization targets, identity-matched NCE objective, and 300-epoch schedule, followed by identical supervised downstream fine-tuning. Thus the control changes the source of teacher supervision, not the transfer architecture; matched epochs do not necessarily imply identical total compute.

(2) Downstream results (FdF_d / phenotype FavgF_{\mathrm{avg}}, %; merged main Table 5).

DatasetH&E-only FdF_dFull FdF_dΔFd\Delta F_dH&E-only FavgF_{\mathrm{avg}}Full FavgF_{\mathrm{avg}}ΔFavg\Delta F_{\mathrm{avg}}
CoNSeP74.1676.18+2.0257.8364.22+6.39
GLySAC84.2587.07+2.8258.1668.68+10.52
MoNuSAC82.1583.96+1.8167.6873.68+6.00
PanNuke80.1682.22+2.0651.8653.61+1.75
Average80.1882.36+2.1858.8865.05+6.17

Under matched Stage-I and Stage-II training schedules, transcriptomic supervision yields +2.18 pp average detection F1 and +6.17 pp average phenotype macro-F1. The larger phenotyping gains provide direct evidence that the molecular signal contributes beyond additional H&E-only pretraining.

Revision in the manuscript: The controlled experiment and its results appear in the merged Table 5 (Section 4.3), with exact image-only NT-Xent, encoder, and training-protocol details in Appendix "Transcriptomics-Free H&E-only Control."

AuthorsOpenReview ↗

W3 — "Upstream–downstream tissue overlap is an undiscussed confound"

We thank you for highlighting the possibility that shared tissue types between upstream pretraining and downstream evaluation could confound the interpretation of transfer performance. We understand that this requires distinguishing overlap in biological samples from overlap at the organ level, and then testing whether same-organ pretraining materially influences the relevant downstream results. We therefore clarify the sample-disjoint data setting and compare full, breast-only, and leave-breast-out pretraining on four breast molecular-state tasks. The scope and findings of this analysis are described below.

(1) Sample overlap versus organ overlap. The Xenium pretraining cohort and downstream datasets contain distinct biological samples. Some tasks share the tissue type, which is different from sharing patients or slides. We now state this distinction explicitly.

(2) Breast-exclusion check. We compared Full, Breast-only, and Leave-Breast-Out pretraining on four breast molecular-state tasks:

Pretraining sourcesFour-task mean AUROC
Full0.6742
Breast-only0.6624
Leave-Breast-Out0.6702

Removing all breast-derived pretraining data changes the average AUROC by −0.0040, compared with Full. These results show limited sensitivity of these four molecular-state evaluations to same-organ pretraining. They do not refer to a separate BreaKHis tissue-exclusion run.

Revision in the manuscript: The Appendix "Impact of Breast-Derived Upstream Pretraining Data" includes this controlled comparison; the corresponding discussion distinguishes same-organ from sample-level overlap.

Reviewer_eAv3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a cell centre detection and classification framework utilizing an approach that aligns spatial transcriptomics representations with a vision-only ResNet, and combine it with a Vision Transformer+Count-guided Flow Localisation (ViT+CFL) approach for cell centre detection and classification.

Critical strengths

1.⁠ ⁠The topic is clearly relevant to the sub-field, since better cell phenotyping is an active area of research and has societal impact, specifically in clinical usage. For these specific reasons, the work is also relevant to the ICLR conference track the authors have submitted to. 2.⁠ ⁠The authors propose an approach to infuse cell phenotyping frameworks using a ResNet that has been pre-aligned with spatial transcriptomics using the MGG module. The aligned ResNet is used inside a vision-only Cell Phenotyping network, making spatial transcriptomics supervised cell detection and classification possible. 3.⁠ ⁠The authors compare against other vision-only cell detection and classification frameworks and report improvements.

Critical weaknesses

There are a few predominant weaknesses in the manuscript:

1.⁠ ⁠A complex multi-step training pipeline with many moving parts, with only a few of them motivated: The approach has two pre-training stages. First, we are to align a vision model of cell images with the gene transcripts (using a gated contrastive loss objective), then include it inside the ViT+CFL pipeline. Furthermore, to incorporate the ResNet information, the ViT+CFL is further pre-trained using the align loss. Is this loss term needed? This is not at all clear. While some ablations have been presented, the gene-aligned ResNet should be tested in isolation by taking any off-the-shelf detector and simply running the pre-trained ResNet for classification tasks to see if the MGG-aligned ResNet really helps. The current ablation only removes one part out of very complicated machinery, and it is not really clear what the real gain from the ResNet is. 2.⁠ ⁠No collapse prevention in the “Morphology-conditioned Gene Gating, MGG” module: While the authors do not report collapse in the ResNet pre-training, what is conceivable is a collapse such that the gene encoder produces all constants and, therefore, loses all gene information. When this happens, it is obvious to the reviewer that the loss can continue to decrease simply because the gating module leaks image representation information to the other side of the contrastive loss term. In other words, once the gene encoder collapses, the gated z_tx essentially is a function of z_img. The authors do not address this risk. 3.⁠ ⁠A near-complete lack of error bars, extending no confidence to the results presented. 4.⁠ ⁠The authors cite two related sources, SpatioCell and sCellST, but not GHIST (https://www.nature.com/articles/s41592-025-02795-z), which could be a line item for comparison in Table 3. sCellST (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12858858/), which they cite, could also have been a comparative line item in Tables 1 and 3.

Questions for authors

Will you please report on how you avoid collapse or if different hyperparameters led to collapse?

5 comments

We thank you for the thoughtful assessment and for recognizing the relevance of molecularly guided cell phenotyping. We understand that the main questions concern which stage of the framework produces the observed gains, whether the gene encoder genuinely retains RNA information rather than relying on image-conditioned gating, and whether the evaluation provides sufficient uncertainty estimates and relevant comparisons. In response, we clarify the roles of Stage I and Stage II, provide a Stage-I-only representation evaluation and a no-alignment ablation, analyze RNA recoverability and training stability, and add statistical-variability results together with GHIST and sCellST comparisons. Our detailed responses follow.

W1 — "Is this additional alignment loss necessary?" / "The gene-aligned ResNet should be tested in isolation"

We thank you for asking whether the additional Stage-II alignment loss is necessary and whether the molecularly aligned Stage-I encoder is useful on its own. We understand that these questions aim to disentangle two sources of improvement: molecular information learned from registered H&E–RNA cell pairs and the subsequent transfer of that information to cell representations in uncropped patches. To address them, we clarify why the two stages serve different purposes, isolate the contribution of the Stage-II alignment loss, and evaluate frozen Stage-I features independently of the Stage-II detector and contextual branch. To distinguish the roles of the two pretraining stages and downstream fine-tuning, we first summarize their inputs, supervision, and learned roles below, followed by the supporting evidence.

Overview of the three training phases, including their inputs, supervision, and learned roles.

PhaseInput and supervisionLearned role
Stage IRegistered H&E cell crops and paired RNA; within- and cross-modal contrastive objectivesMolecularly guided cell-image encoder
Stage IIUncropped H&E patches, registered centers, and fixed Stage-I references; localization and identity-matched alignmentPatch localization and contextual cell features
DownstreamH&E patches with cell locations and phenotype labelsJoint localization and phenotype prediction

(1) Why two stages? Stage I learns molecularly guided image features from registered cell crops and RNA; Stage II transfers these features to cell-specific representations within raw patches, while learning cell localization. Stage-I reference features are fixed during Stage-II pretraining and matched by physical cell ID. At inference, the image-side Stage-I encoder supplies target-cell morphology, while Stage II supplies location and contextual information. We now state this division in Section 3 and Sections 3.1–3.3.

(2) Is Stage-II alignment needed? Yes. Removing only Lalign\mathcal L_{\mathrm{align}} reduces the four-dataset average detection F1 from 82.36 to 80.56 and macro-F1 from 65.05 to 63.05. The original localization-specific ablation independently records +1.54/+2.46 pp on Stage-II validation/unseen pre-fine-tuning data. The merged Table 5 in Section 4.3 contains the full-task ablation alongside the H&E-only control; the Appendix retains the upstream localization study.

(3) Is the Stage-I encoder useful independently of Stage II? Revised Table 1 evaluates frozen Stage-I features at ground-truth cell locations, with no Stage-II detector or context branch. The original ResNet variant reaches 77.65% mean classification accuracy, compared with 56.62% for Volta and 69.57% for AutoCell; Table 1 also includes sCellST, while Table 4 separately compares UNI2-H and CTransPath before and after Stage-I molecular alignment. These frozen-feature experiments isolate Stage-I representation quality, whereas Table 2 evaluates complete detection-and-classification pipelines. The ground-truth-crop protocol is not equivalent to classification after an off-the-shelf detector. The revised manuscript makes the distinction between these two protocols explicit.

AuthorsOpenReview ↗

W2 — "The gene encoder could ... collapse to constant representations"

We thank you for identifying a plausible representation-collapse failure mode in morphology-conditioned gene gating. The concern is that the gene encoder might become nearly constant while MGG still allows the contrastive objective to decrease by passing through image-derived information; a low training loss alone would therefore not establish that transcriptomic information is retained. To investigate this mechanism directly, we analyze RNA representations before image-conditioned gating, measure their recoverability across checkpoints, ablate the RNA-view objective, and test sensitivity to training hyperparameters. We present these diagnostic results below.

(1) Checkpoint-level RNA recoverability. Using the same Stage-I pretraining data and gene-input configuration as the main experiments, we evaluated pre-gating gene representations across 10 checkpoints (epochs 5–50). A held-out linear readout predicting RNA PCA10 achieved R2=0.557R^2=0.557–0.6260.626, versus −0.011-0.011 for a constant-feature control. These results show that the evaluated pre-gating RNA representations retain substantial transcript-dependent variation rather than collapsing to constant features.

(2) Which term prevents loss of RNA information? In a controlled 10-epoch comparison, retaining the RNA-view contrastive objective gives R2=0.604R^2=0.604, whereas removing it gives R2=−0.161R^2=-0.161. The RNA-view term is thus an independent transcriptomic learning signal, in addition to the image-conditioned alignment term.

(3) Hyperparameter sensitivity. The tested learning-rate/batch-size settings yield the following pre-gating RNA readouts:

Learning rateBatchR2R^2Constant collapse
10−410^{-4}320.574No
2×10−42\times10^{-4}320.604No
4×10−44\times10^{-4}320.501No
2×10−42\times10^{-4}160.554No

Revision in the manuscript: Section 3.1 now explains the role of RNA-view consistency, and the new Appendix subsection "Gene-Encoder Recoverability and Training Stability" reports the checkpoint, objective-ablation, and hyperparameter diagnostics. These diagnostics use the same Stage-I data and gene-input configuration as the main experiments, with controlled changes to the training objective or hyperparameters where specified. Existing raw-gene attribution across six tissue contexts provides complementary input-sensitivity evidence.

AuthorsOpenReview ↗

W3 — "A near-complete lack of error bars"

We thank the reviewer for emphasizing the importance of variability estimates. We agree that uncertainty should be supported by actual evaluation records and that results taken from the literature must be distinguished from independently re-evaluated experiments. We have therefore organized the available variability estimates in the revised Appendix while keeping the main result tables concise.

(1) Where are the uncertainty results reported? Appendix D.1, "Variability Summaries for Main Experimental Results," now contains six detailed tables: Stage-I cell-level classification (main Table 1), complete localization and phenotyping (Table 2), foundation-backbone transfer (Table 4), the H&E-only and w/o Alignment controls (two separate Appendix tables corresponding to the merged main Table 5), and the MGG ablation (Table 6). These tables provide the supplied mean ± variability values where available, with dataset-level breakdowns and available difference estimates. We do not assign an uncertainty value where none was provided, including the H&E-only ΔFavg\Delta F_{\mathrm{avg}} column.

(2) How are literature baselines handled? Main Table 2 labels methods as Re-eval. or Reported. Its corresponding table in Appendix D.1 retains Reported methods as published point estimates without error bars; ±\pm values appear only for independently re-evaluated methods and ViO-Align. We do not infer repeated-run variability for literature-only results or imply identical upstream pretraining.

Revision in the manuscript: Section 4.1 directs readers to Appendix D.1, "Variability Summaries for Main Experimental Results", for the six detailed uncertainty tables. The main tables retain their point estimates for readability; the Appendix explicitly distinguishes independently evaluated variability from reported-only results.

Reviewer_X3SMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper presents a model for phenotyping H&E slides. The main premise of the proposed model is that the representation of cells can be supervised with molecular data obtained from spatial transcriptomics samples. The approach trains models in two stages. The first stage learns a fused cross-modal representation of morphology and gene expression, and the second stage uses this representation to guide the learning of image-based only representations of cells.
Critical strengths
  • The idea of merging or fusing transcriptomics and image-based representations is interesting.
  • The model architecture and training strategy seem well designed.
Critical weaknesses
  • The main drawback of the experimental study is that the baselines are not strong and comprehensive. Other foundation models for pathology could have been used as a reference. And the other models have different training regimes that my have different training datasets (amounts and types of supervision). The main results that the author compare against come from another paper (CP-Mamba). Efforts to train other models under the same conditions are very limited.
  • A control experiment that trains the same architecture of stage 2 without the supervision of the fused multimodal representation seems to be missing. The ablations report performance without MGG, which is a different evaluation.
  • The technical novelty is limited. Most of the components are standard deep learning building blocks, and the applicability or extensibility to other applications is difficult.
  • The writing is too intricate and it is hard to follow. The writing is technically correct, but globally confusing. The motivations of the work are not well explained and the decisions of the design require prior knowledge that is not properly introduced.
Questions for authors

Can you comment and address the comments in the weaknesses?

6 comments

We thank you for the careful evaluation and for recognizing the potential of combining transcriptomic and image-based representations. We understand that the central concerns are whether our performance gains remain convincing against stronger and more fairly evaluated baselines, whether cross-stage molecular guidance is necessary, and whether the contribution and its extensibility are explained clearly enough. To address these points, we have expanded and re-evaluated the baseline comparisons, added a targeted Stage-II alignment ablation, examined molecular interpretability and Stage-I backbone extensibility, and substantially clarified the manuscript's motivation and training procedure. We respond to each concern in detail below.

W1 — "The baselines are not strong and comprehensive"

We thank you for raising the important question of whether the reported gains are supported by sufficiently strong and comparable baselines. We understand that this concern involves not only the range of methods included, but also differences in training regimes, the provenance of published results, and the relevance of pathology foundation models. To address it, we expand the complete-task baseline set, independently re-evaluate methods where compatible protocols are available, and separately test the extensibility of Stage-I molecular supervision to additional pretrained encoders. The corresponding comparisons and protocol clarifications are detailed below.

(1) Stronger, task-matched comparisons. Our primary task is joint localization and phenotyping from uncropped H&E patches, rather than cell-representation classification alone. We have added DeNuC (MICCAI 2026)[1], a recent UNI2-H-based detection-and-classification pipeline, alongside CNN (DIST, DSF-CNN, SONNET, HoVer-Net), graph (CGT, SENC), state-space (CP-Mamba), and foundation-model (CellViT++) methods.

Reference

[1] Zijiang Yang, Chen Kuang, and Dongmei Fu. DeNuC: Decoupling Nuclei Detection and Classification in Histopathology. In Medical Image Computing and Computer Assisted Intervention (MICCAI), LNCS 16891, pp. 118–127, Springer, 2026. Paper.

(2) "Different training regimes" and results inherited from CP-Mamba. We independently re-evaluated SONNET, HoVer-Net, CellViT++, and DeNuC, using the available implementations and weights with aligned downstream splits, class definitions, detection matching, and evaluation metrics. Methods that could not be reproduced under a compatible protocol retain their published numbers; the revised main Table 2 explicitly labels every result `Re-eval.` or `Reported`, instead of presenting all values as equally controlled.

(3) "Other pathology foundation models." In addition to complete-task baselines, we tested Stage-I molecular supervision with UNI2-H and CTransPath under the same frozen-feature linear-probing protocol. Average classification accuracy increases from 77.58 to 78.85 (+1.27 pp) for UNI2-H and from 75.74 to 78.18 (+2.45 pp) for CTransPath (revised Table 4). These are Stage-I backbone experiments, distinct from complete Stage-II pipeline comparisons.

Under the expanded full-task comparison, ViO-Align obtains the highest detection F1 and phenotype macro-F1 on all four benchmarks. Its macro-F1 gain over the strongest eligible baseline is:

DatasetStrongest eligible baseline (macro-F1)ViO-AlignGain (pp)
GLySACDeNuC 63.1168.68+5.57
CoNSePHoVer-Net 63.1264.22+1.10
MoNuSACDeNuC 72.9673.68+0.72
PanNukeCP-Mamba 52.8753.61+0.74
Average gain+2.03

Revision in the manuscript: Section 4.1 specifies baseline provenance and the common evaluation protocol; Table 1 contains the original Stage-I representation comparison, Table 2 the complete-task baselines, and Table 4 the separate UNI2-H/CTransPath backbone-transfer study. The main-text and abstract numerical comparisons have been updated to the stronger baseline set.

AuthorsOpenReview ↗

W2 — "A control experiment ... without the supervision of the fused multimodal representation ... is missing"

We thank you for highlighting the need to directly verify whether cross-stage molecular guidance contributes to downstream cell localization and phenotyping. We understand that the key concern is whether the molecularly informed representations learned in Stage I provide additional benefits to Stage II, beyond those obtained from its localization objectives alone. To address this, we clarify the distinction between the existing MGG ablation and the requested cross-stage alignment control, and extend our evaluation to all four downstream benchmarks. These analyses allow us to isolate the contribution of cross-stage alignment while keeping the Stage-I molecular teacher and localization objectives unchanged. Our detailed responses are provided below.

(1) Distinguishing the Stage-I and Stage-II controls. MGG governs Stage-I molecular representation learning. The requested control instead retains the molecularly trained Stage-I encoder and all Stage-II localization objectives, while setting the Stage-II alignment weight to zero (λalign=0\lambda_{\mathrm{align}}=0).

(2) Localization and complete-task evidence. The original upstream localization ablation already showed +1.54 pp detection F1 on Stage-II validation (85.84 → 87.38) and +2.46 pp on unseen data before fine-tuning (78.14 → 80.60). We now extend the controlled ablation to all four complete downstream localization-and-phenotyping tasks, also summarized in the merged Table 5:

DatasetFdF_d w/o / fullFavgF_{\mathrm{avg}} w/o / fullMacro-F1 gain (pp)
CoNSeP74.27 / 76.1863.15 / 64.22+1.07
PanNuke80.53 / 82.2251.56 / 53.61+2.05
MoNuSAC82.15 / 83.9672.12 / 73.68+1.56
GLySAC85.28 / 87.0765.37 / 68.68+3.31
Average80.56 / 82.3663.05 / 65.05+2.00

Cross-stage alignment improves average detection F1 by +1.80 pp and phenotype macro-F1 by +2.00 pp, even though both variants use the same Stage-I molecular teacher.

Revision in the manuscript: Section 3.2 explains identity-matched cross-stage alignment; Section 4.3 / Table 5 report this ablation alongside the transcriptomics-free control; Appendix "Joint Localization–Alignment Training" retains the original localization-stage comparisons.

AuthorsOpenReview ↗

W3 — "The technical novelty is limited" / "extensibility ... is unclear"

We thank the reviewer for the important comments on the positioning of our contributions and the extensibility of our method. We recognize that the central concerns are whether ViO-Align offers a clearly defined methodological contribution beyond combining standard network components, and whether its effectiveness depends on a particular visual backbone. To address these concerns, we clarify the two interconnected supervision transfer mechanisms at the core of ViO-Align, provide additional gene-level interpretability evidence for MGG, and evaluate the transferability of Stage-I molecular alignment across different visual backbones using UNI2-H and CTransPath. Our clarifications and additional evidence are presented below.

(1) What the framework contributes. ViO-Align is built around two linked supervision transfers, rather than a new convolution, ViT, or attention operator. Stage I aligns registered Xenium H&E–RNA cell pairs to learn molecularly guided cell-image representations. Stage II uses registered physical-cell identity to establish cross-stage positives between these references and cell-specific contextual features in uncropped H&E patches. Molecular guidance can therefore reach the complete in situ localization-and-phenotyping task, while inference remains H&E-only. MGG, cell localization, and contextual aggregation implement complementary parts of this learning strategy.

(2) "Technical novelty": interpretable molecular guidance through MGG. Beyond cross-modal matching, morphology-conditioned gene gating (MGG) modulates latent transcriptomic factors according to the paired cell's visual morphology.

To examine which molecular inputs contribute to this alignment, we provide raw-gene gradient attribution in Appendix D, "Biological Plausibility of Image-Conditioned Gene Gating." Specifically, we propagate gradients of the image–transcriptomics similarity score through MGG and the transcriptomic encoder back to individual input genes, then aggregate absolute attribution magnitudes across cells.

The Appendix table and figure report biologically relevant genes across six tissue contexts, including S100B/PTN in brain glioblastoma, MUC12/CLCA1 in normal colon, TAT/CYP-family genes in liver, and MS4A1/BANK1 in reactive tonsil. This analysis links the alignment objective to biologically plausible, tissue-associated molecular signals. Importantly, MGG gates latent factors, not individual genes; gene-level interpretation comes from the end-to-end attribution analysis.

(3) "Extensibility ... is unclear": backbone extensibility tested on two pretrained encoders. We applied the same Stage-I molecular alignment to UNI2-H and CTransPath, comparing frozen-feature linear-probing performance before and after alignment:

Stage-I backboneOriginal accuracyAfter molecular alignmentGain
UNI2-H77.5878.85+1.27 pp
CTransPath75.7478.18+2.45 pp

The gains on both encoders show that Stage-I molecular supervision can improve different pretrained pathology representations, rather than relying on the original ResNet backbone. These tests assess Stage-I backbone extensibility; the full Stage-II pipeline evaluations continue to use the original configuration.

Revision in the manuscript: Section 1 retains the original four-part Contributions structure and adds the backbone evidence to the MGG contribution; Sections 3.1–3.3 clarify the two supervision transfers and the role of MGG; new Table 4 separately reports UNI2-H/CTransPath backbone transfer; and Appendix D, "Biological Plausibility of Image-Conditioned Gene Gating," presents the raw-gene attribution method, six-tissue gene-importance table, and corresponding figure.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response to Reviewers

We sincerely thank all three reviewers for their careful evaluation and constructive feedback. We appreciate their recognition of the following strengths:

  • Significance: "The topic is clearly relevant to the sub-field" and "better cell phenotyping is an active area of research and has societal impact" — eAv3; "Clear motivation and well-positioned framework" — yX2J.
  • Methodology: "The idea of merging or fusing transcriptomics and image-based representations is interesting" and "The model architecture and training strategy seem well designed" — X3SM; "Its cell-level alignment → raw-patch transfer → H&E-only inference design connects them sensibly, with practical value" — yX2J.
  • Evaluation and Performance: "The authors compare against other vision-only cell detection and classification frameworks and report improvements" — eAv3; "Complete method with thorough ablations", "Rigorous evaluation protocol", and "Comprehensive tasks with consistent gains" — yX2J.
  • Applicability and Efficiency: "Compact model" and "This is an advantage in the accuracy–efficiency trade-off" — yX2J; "relevant to the ICLR conference track" — eAv3.

We addressed the reviewers' concerns through controlled experiments, stronger comparisons, and methodological clarifications. Our major revisions are summarized below:

  • Effectiveness of molecular supervision transfer (X3SM, yX2J): We added a matched-budget H&E-only control that retains the two-stage pipeline but replaces the RNA-guided Stage-I teacher with a visual-only teacher. RNA supervision improves average phenotype macro-F1 by 6.17 pp and detection F1 by 2.18 pp across four benchmarks. This directly supports the effectiveness of single-cell RNA supervision for in situ cell localization and phenotyping in uncropped H&E patches, beyond matched visual-only pretraining.

  • Necessity of cross-stage alignment (X3SM, eAv3): Removing only Stage-II alignment while retaining the molecularly trained Stage-I encoder and localization objectives reduces average phenotype macro-F1 by 2.00 pp and detection F1 by 1.80 pp. This verifies the contribution of identity-guided alignment in bringing molecular supervision to raw-patch in situ phenotyping.

  • Stronger baselines and extensibility (X3SM, eAv3, yX2J): We added DeNuC, GHIST, and sCellST and re-evaluated SONNET, HoVer-Net, CellViT++, and DeNuC using aligned protocols. ViO-Align achieves the highest detection F1 and phenotype macro-F1 on all four benchmarks, with an average 2.03 pp macro-F1 advantage over the strongest eligible baselines. Stage-I molecular alignment also improves UNI2-H and CTransPath linear-probing accuracy by 1.27 and 2.45 pp, supporting the effectiveness and architectural extensibility of our training paradigm beyond ResNet.

  • Gene-encoder stability and biological plausibility (eAv3, X3SM): We verified that no gene-encoder representation collapse was observed under the evaluated conditions. Pre-gating RNA recoverability across checkpoints, objective ablations, and hyperparameter analyses confirms retained transcript-dependent information (held-out RNA PCA10 R2=0.557R^2=0.557–0.6260.626). We also clarified the existing gene-level attribution analysis across six tissue contexts, supporting biological plausibility.

  • Robustness to upstream tissue composition (yX2J): We compared Full, Breast-only, and Leave-Breast-Out pretraining. Excluding breast-derived upstream samples changes mean AUROC across four breast molecular-state tasks only from 0.6742 to 0.6702. This supports robustness to upstream tissue composition in these evaluations and shows that performance does not depend solely on same-organ pretraining.

  • Clarity and reproducibility (X3SM, eAv3, yX2J): We reorganized the Introduction and Methodology around cell-to-patch molecular supervision transfer, clarified Stage-I/Stage-II roles and H&E-only inference, specified training schedules, loss weights, and evaluation protocols, and corrected the CellViT++ result through detector fine-tuning.

Together, these results support that ViO-Align effectively leverages single-cell RNA supervision for accurate in situ cell localization and phenotyping directly from uncropped H&E patches, without molecular measurements at inference. Further analyses support the effectiveness, extensibility, and robustness of the training paradigm.

We thank the reviewers again for their constructive suggestions and welcome further discussion.

Reviewer_eAv3OpenReview ↗

Change to weak accept

I had preset criteria in confidential comments that would lead to me changing from weak reject to weak accept. The authors have addressed these adequately and I believe the manuscript now can be considered acceptable.

AuthorsOpenReview ↗

Thanks for the Updated Recommendation & a Gentle Follow-up Regarding It

Dear Reviewer eAv3,

Thank you very much for revisiting our rebuttal and for updating your recommendation to weak accept. We truly appreciate your constructive feedback.

As a gentle note, we noticed that the "Rating" in your "Official Review" still shows "2: Weak rejection". We are not sure whether this indicates any remaining concerns on your side. If you have any further questions, please feel free to let us know, and we will respond promptly.

Thanks again for your time and support!

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.