acceptodds
Under review as a conference paper at ICLR 2027

Beyond Appearance: Protein-Grounded Relational Reasoning for Structural Pathological Event Recognition

Abstract

Modern computational pathology has achieved remarkable progress in cancer diagnosis, yet most models still make decisions primarily from histological appearance. This paradigm becomes insufficient for structural pathological events, where visually similar tissue environments can imply opposite diagnoses simply because their constituents are organized differently. We term this ambiguity histological isomerism: a model may recognize the right constituents yet still reach the wrong diagnosis if it fails to understand how they constitute a pathological event. We present FRAME, which moves computational pathology beyond appearance toward biologically grounded structural reasoning. FRAME learns molecularly grounded H&E representations from limited paired H&E-mIF data, organizes local evidence into anatomical targets and their surrounding constituents, and reasons about how these constituents are configured relative to target boundaries. mIF is required only during training, while inference remains H&E-only. With only hundreds to thousands of task-specific training samples, FRAME outperforms large-scale pathology foundation-model baselines by 2.5-5.1 points on general cancer diagnosis and grading. With only dozens of pathologist-corrected ROIs, it further outperforms the strongest pathology foundation-model baselines by an average of 12.5 points on composition-matched histological isomerism hard subsets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computational pathology, Pathology diagnosis, Representation learning, Reliable diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jNswOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes FRAME, a pathology framework for recognizing structural pathological events beyond appearance-based modeling. FRAME uses paired H&E–mIF data during training to learn protein-grounded spatial representations, then performs constituent parsing and boundary-centric relational reasoning to model how tissue components are organized around anatomical targets. At inference, the method requires only H&E images. The framework is evaluated on molecular grounding, structural-event recognition, and slide-level cancer diagnosis/grading tasks, showing improvements over several pathology foundation models and conventional MIL/segmentation baselines.

Critical strengths
  1. Well-motivated problem formulation. The paper clearly identifies the limitation of appearance-centric pathology models and motivates the need to model target-relative spatial organization for structural pathological events.
Critical weaknesses
  1. Unclear ViT encoder. The paper only mentions a ViT encoder, but does not specify the exact backbone (e.g., UNI, CONCH, DINOv3 ViT) or whether it is frozen or fine-tuned.
  2. Missing molecular-grounding data scale. The authors list five internal H&E–mIF datasets, but do not report the number of patients, WSIs, or paired patches. This weakens the claimed data efficiency compared with large-scale PFMs.
  3. Unclear fairness of PFM comparisons. Table 1 compares FRAME directly with UNI, CONCH, and CHIEF, although FRAME is a task-specific framework with molecular grounding, constituent parsing, and structural reasoning. Without a unified frozen-encoder + same-MIL-head protocol, these results do not show that FRAME learns better tile representations.
  4. Insufficient details for Table 6 baselines. The tile encoder used for MIL-based methods and the aggregation strategy used for PFM-based methods are not clearly specified.
  5. Ambiguous training paradigm. Based on the current description, FRAME appears closer to task-specific molecularly supervised adaptation/fine-tuning of an existing ViT backbone than to pretraining a new general-purpose pathology foundation model. The paper should clarify this distinction.
  6. Figures 1 and 2 are unclear.
Questions for authors

Please respond to the Critical Weaknesses. If the authors fully address my concerns, I will increase my score.

6 comments
AuthorsOpenReview ↗

Response to W1–W2

Sincerely thank you for careful review ! We organize our responses into W1–W6, presented in three response windows (W1–W2, W3-W4, and W5-W6). Supplementary figures are collected in the revised Appendix (starting from p. 23) and are also available through this anonymous link: https://anonymous.4open.science/r/FRAME_r3/. Apologize for any inconvenience.

W1. Backbone Specification and Training Strategy

Apologize for the lack of clarity. We use CTransPath as the pathology-pretrained backbone, selected for its relatively lightweight architecture and manageable training and deployment costs. Its parameters remain frozen throughout FRAME training.

About frozen or fine-tuned. Our design considers the difference between the two modalities: H&E provides high-dimensional morphological information, whereas mIF provides spatially aligned continuous protein-expression signals. We therefore retain the pretrained morphological representations and introduce a trainable Spatial Adapter (Sec. 3.2, Eq. 2) to facilitate local feature adaptation and molecular information fusion, rather than directly fine-tuning the entire backbone with protein-recovery supervision.

To validate this design choice, we additionally compare direct CTransPath fine-tuning against our frozen-backbone + Spatial Adapter design using the protein-recovery setting in Sec. 4.2.1 (Fig. 3; Appendix Table 4). Direct fine-tuning reduces PCC, Spearman, AUROC, and AP by 0.174, 0.167, 0.208, and 0.146, respectively. These results support retaining the pretrained backbone while learning molecular adaptation through the Spatial Adapter under our protein-grounding setting.

We further provide a direct visual comparison between protein recovery using the fine-tuned CTransPath and our original design in Appendix Fig. 12 (revised Appendix, p. 24; anonymous supplementary link).

W2. Molecular-Grounding Data Scale and Composition

Thank you for this helpful suggestion. Based on Appendix A.2, we provide a more transparent account of the scale and composition of our paired H&E–mIF training data.

  • Data scale and cohort composition. We collected four patients per cancer type, representing high, intermediate, low, and normal conditions, with two same-section H&E–mIF pairs per patient. The five cohorts comprise 20 patients, 40 paired samples, and approximately 7.58 million spatially registered patches.
CohortPatientsPairsRegistered patchesChannelsChannel names
Rectum7mIF48~1.74M7DAPI, SMA, S100, CD31, FAP, D2-40, CKPAN
Kidney7mIF48~1.28M7DAPI, CD20, CD23, CKpan, FOXP3, CD8, S100
Prostate9mIF48~1.60M9CKpan, P53, Ki-67, P504s, PSA, ERG, AR, P63, S100
Gastric15mIF48~1.37M15CD138, MPO, D2-40, S100, CD34, CKpan, P53, Ki-67, HER2, CEA, CDX2, MUC5AC, MUC6, CLDN18.2, Syn
Liver9mIF48~1.59M9P53, Ki-67, S100, Arginase-1, AFP, CK7, CK19, GPC3, CKpan
  • Biological basis of panel selection. The panels were designed to capture both organ-specific molecular characteristics and cellular constituents shared across organs. In particular, CKpan and S100 are included across all five cohorts, providing shared molecular evidence for epithelial and neural/adipocyte-associated constituents, respectively. Other markers, such as SMA, CD31, FAP, and D2-40, provide complementary evidence for additional cellular constituents and anatomical structures. To make these biological correspondences clearer, we provide a pathologist-guided summary below and an accompanying schematic in Fig. 9 (revised Appendix, p. 23; anonymous link).
Shared cellular constituentsProtein markersAssociated anatomical structures
Smooth muscle cellsSMASmooth muscle
Vascular endothelial cellsCD31Capillaries, veins, arteries
Nerve-associated cellsS100 (SOX10)Nerves
AdipocytesS100 (Perilipin-1)Adipose tissue
Fibroblast-associated cellsFAPFibrous stroma
Lymphatic endothelial cellsD2-40Lymphatic vessels

These paired samples are used specifically for training-time molecular grounding, providing dense, spatially aligned protein supervision while retaining a frozen pathology-pretrained backbone and H&E-only inference (see W1). Thus, FRAME does not rely on large-scale pretraining of a new foundation model; instead, it exploits limited paired molecular data to obtain dense biological supervision for downstream structural reasoning.

AuthorsOpenReview ↗

Response to W3-W4

W3. Fairness of PFM Comparisons

We apologize for the lack of clarity. Table 1 aims to evaluate the complete diagnostic capabilities of different methods under identical H&E WSI splits and task-specific labels (Sec. 4.1; Appendix A.4).

We fully agree with your suggestion of a more controlled comparison using frozen encoders and an identical MIL head. To this end, we freeze the encoders used in the Morphology Encoding stage and compare them with frozen HIPT, CTransPath, UNI, and CONCH encoders under a shared ABMIL head. CHIEF is not separately included because it uses CTransPath as its encoder.

Backbone / feature representationMIL headPANDA AUCCPTAC AUCTCGA-CRC AUC
HIPTABMIL0.8320.7150.837
CTransPathABMIL0.9210.7630.842
UNIABMIL0.9370.8220.857
CONCHABMIL0.9410.8290.862
FRAMEABMIL0.9560.8440.883

Under this unified aggregation protocol, FRAME consistently outperforms the strongest PFM baseline (CONCH) by 1.5, 1.5, and 2.1 percentage points on PANDA, CPTAC, and TCGA-CRC, respectively. These results further support the effectiveness of FRAME's molecularly grounded representations beyond the choice of WSI aggregation architecture.

W4. Implementation Details of Baselines

Thank you for pointing out this missing information. For conventional MIL baselines, we use ImageNet-pretrained ResNet50 as the tile encoder, followed by each method's native MIL aggregator. For patch-level PFM baselines, we use their frozen pretrained encoders with ABMIL for WSI-level prediction, whereas models with native WSI-level architectures retain their original aggregation strategies.

To further clarify the implementation protocol, we expand Appendix A.4:

Benchmarking protocol. Our objective in Table 6 is to evaluate complete WSI-level diagnostic capabilities rather than isolated tile representations. We therefore prioritize implementation fidelity:

  1. Complete WSI pipelines: For methods with publicly available end-to-end implementations, we retain their original encoders and aggregation architectures, including TEA-Graph, Patch-GCN, ZoomMIL, HIPT, and CHIEF.
  2. Partial implementations: For methods providing only patch encoders or WSI aggregators, we retain the original components and use standardized CLAM/TRIDENT-based WSI preprocessing and feature extraction to complete the pipeline.
  3. Paper-based reimplementation: For methods without publicly available implementations, we extract 1024-dimensional tile features using an ImageNet-pretrained ResNet50 following the CLAM preprocessing pipeline, and implement the original aggregation architecture based on the published descriptions.
AuthorsOpenReview ↗

Response to W5-W6

W5. FRAME's Training Paradigm and Methodological Positioning

Thank you for raising this important point! Your comment prompted us to clarify both FRAME's training paradigm and its methodological contribution.

FRAME extends morphological representation into biologically grounded structural reasoning by explicitly modeling tissue constituents and their target-relative relationships (Sec. 2.1, Line 122).

For an intuitive analogy, in visual scene understanding, detecting a person and a bicycle identifies what is present, but distinguishing "riding a bicycle" from "standing beside a bicycle" requires modeling how they are spatially related. Similarly, identifying tumor cells and a nerve is insufficient to determine perineural invasion (PNI); the diagnosis depends on how tumor cells are organized relative to the nerve boundary.

(1) What is FRAME and how is it trained?

FRAME progressively transforms H&E morphology through three stages: Morphology Encoding learns protein-grounded spatial representations; Constituent Parsing identifies Anatomical Targets and their surrounding Constituent Attributes; and Relation Reasoning models local constituent–boundary interactions and integrates them into global contour configurations for Structural Pathological Event Recognition.

Training paradigm. We retain a frozen CTransPath backbone, without additional large-scale pretraining or backbone fine-tuning. Five internal cohorts comprise 20 patients and 40 paired H&E–mIF samples (8 pairs per cancer type, ~7.58M registered patches), providing dense, spatially aligned, multi-channel protein supervision. Internal Parsing objectives enable the model to learn target masks and constituent attributes as intermediate representations, without requiring exhaustive manually annotated constituent maps. Dataset-provided task-specific labels supervise final event prediction. Importantly, mIF is used only during training, while inference requires H&E images alone.

(2) Why does FRAME matter?

  • Molecular supervision. A limited number of spatially registered H&E–mIF pairs provides dense token-level protein supervision, enabling the model to learn biologically grounded representations of epithelial, stromal, vascular, and nerve-associated constituents while reducing reliance on exhaustive manual semantic annotations.

  • Structural reasoning. General-purpose pathology foundation models learn rich morphological and contextual representations, but their conventional prediction pipelines do not explicitly model the event-defining spatial relationships between tissue constituents and specific anatomical targets. FRAME addresses this gap through Constituent Parsing and Boundary-centric Relation Reasoning, enabling the model to recognize not only what constituents are present, but also how they are organized to form pathological events.

To further facilitate understanding, we provide a pathologist-guided H&E–mIF biological correspondence schematic (Fig. 9; see W2) and visualizations on ORION-CRC illustrating constituent recognition and structural reasoning under molecularly supervised training (Appendix Fig. 10; anonymous link).

W6. Clarification of Figures 1 and 2

Thank you for highlighting this issue. Your feedback, together with the preceding comments (W1–W5), helped us provide sufficient pathological context and biological foundations.

We have made three complementary revisions:

  • Pathological motivation (Fig. 1). We replace the abstract illustrations of structural pathological events with representative H&E sections of PNI invasion, selected and annotated by pathologists, with enlarged views provided in Appendix Fig. 11.
  • Biological and methodological context (Fig. 2). We supplement the framework explanation with a pathologist-guided H&E–mIF correspondence schematic (Fig. 9) and ORION-CRC visualizations illustrating constituent recognition and structural reasoning (Fig. 10). These additions clarify the biological basis and information flow underlying FRAME.
  • Clinical relevance (Related Work). Following your suggestion, we expand the discussion of histological isomerism with representative clinical prevalence data for PNI, LVI, and EMVI across multiple malignancies, establishing the clinical relevance of the structural events addressed by FRAME.

The revised figures and expanded discussion are included in the updated manuscript and Appendix (pp. 23–24), and are available through our anonymous supplementary link.

Reviewer_vyiMOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a specific challenge in pathological image recognition, that is, what is present is insufficient to determine what event has formed. This question asks us to distinguish pathological tissues that have similar tissue composition but different spatial distributions, thereby recognizing pathological events with spatial dependency. The authors propose the FRAME framework. Supervised by multi-channel mIF protein signals, the framework trains a sub-module to predict protein expression from H&E-stained images, alongside an image segmentation module. Finally, it performs relation-aware prediction by modeling relationships based on the boundaries of the identified tissue components.

Critical strengths

The perspective proposed in this paper is valuable, and it addresses a problem that is currently relatively overlooked but potentially important. Experimental results demonstrate that FRAME achieves SOTA performance across multiple datasets when compared to various baselines. Furthermore, not requiring extra mIF image input at the inference stage is an advantage, which can reduce the limitations for real-world deployment.

The insight of this paper is valuable, and the method also achieves good experimental results. I did not find any fatal problems. However, since this paper focuses on detection in medical imaging, generalization in real-world scenarios will be very important. The current analysis of the method's generalization is insufficient, so I tentatively scored 2.

If the authors can provide additional experiments in the rebuttal to address my concerns, I would be glad to raise my score.

Critical weaknesses

I have a few concerns as follows.

(1) The first is how FRAME specifically performs across different tissue compositions and the events they dominate. The paper mentions a composition vector that involves various tissue constituents such as ptumor, ptarget, pstroma, and pvessel, yet FRAME's specific performance on these constituents remains unclear, while only overall metrics are reported. Through such an analysis, we could simulate how the model might perform when encountering the same types of events in previously unseen tissue environments.

(2) Second, regarding the mIF supervision channels used in the "Morphology Encoding" stage, Figure 2 shows seven channels. It is unclear whether these channels contribute equally, or if the model relies primarily on a subset of them. Clarifying this would provide insight into the efficiency of the FRAME training process.

(3) Finally, the authors give the definition of hard samples in Section 4.2.3 (experiments are in Figure 5). Although the main text gives the selection principle of similar tissue composition and opposite labels, I didn't find the construction details in the whole paper. And the authors didn't clarify why some of the samples are much harder. I agree that the criteria for partitioning can be flexible. Yet a clear partition protocol must be given, along with showing the gaps between hard and non-hard samples on some dimensions.

Questions for authors
  1. What is the performance across different tissues and what is the generalization performance? Please report the results on different organs or tissues separately, and analyze whether the tissue distribution of the training data affects the results.

  2. Which mIF channels bring gains? Can a few channels alone achieve performance close to the full panel, or is there selectivity with respect to different events? This should be clarified through experiments.

  3. How common is histological isomerism in actual diseases or certain conditions? Here, I suggest adding some representative diseases, proportions, or real cases in the introduction or related work. It is useful to convince readers from different backgrounds.

  4. What exactly is the Parsed target in Figure 2, and how is it obtained? Please add an explanation, and what is its relationship with the subsequent relation module?

6 comments
AuthorsOpenReview ↗

Response to Q1&W1–Q2&W2

Thank you for recognizing of FRAME ! We organize our responses into five parts, presented in three response windows (Q1&W1–Q2&W2, W3, and Q3-Q4). Supplementary figures are collected in the revised Appendix (starting from p. 23) and are also available through this anonymous link: https://anonymous.4open.science/r/FRAME_r2/. Apologize for any inconvenience.

Q1 & W1. Constituent-Level Performance and Cross-Organ Generalization

Thank you for this insightful question.

(A) Constituent-Level Performance. We evaluate 4 major constituents (Tumor, Stroma, Blood vessel, and Nerve-associated) across liver, rectal, and prostate H&E subsets of PAIP 2021, transferring mIF supervision between organs. The results are reported below (Dice, %).

mIF sourceH&E test organTumorStromaBlood vesselNerve-associated
LiverLiver96.3396.4290.7891.59
LiverRectum93.4894.9689.3788.69
LiverProstate94.0894.4389.2489.62
RectumLiver94.8995.0689.4790.27
RectumRectum96.2296.7891.1989.94
RectumProstate95.1395.4590.2590.71
ProstateLiver94.1294.6589.1489.86
ProstateRectum95.7495.8290.3889.08
ProstateProstate96.6896.5691.2691.34

Rows indicate the mIF supervision source, while columns indicate the H&E evaluation organ. Bold values denote same-organ evaluation.

The mean Dice across four constituents is 93.76% under same-organ evaluation and 92.25% under cross-organ evaluation, a decrease of only 1.51 percentage points. The decreases are 1.84, 1.53, 1.44, and 1.25 points for Tumor, Stroma, Blood vessel, and Nerve-associated structures, respectively. Notably, cross-organ nerve recognition remains stable; for example, Rectum-to-Prostate transfer achieves 90.71%, exceeding the same-organ result of 89.94%.

(B) Cross-Organ Generalization of Molecular Grounding. We further evaluate whether molecular grounding learned from different organs transfers to prostate adenocarcinoma diagnosis on PANDA. As summarized below, models trained with liver, rectal, kidney, and gastric mIF supervision achieve AUCs of 96.08%, 97.90%, 96.77%, and 94.49%, respectively, compared with 98.14% under prostate-specific supervision.

mIF sourcePANDA AUC (%)Δ (pp)
Liver96.08−2.06
Rectum97.90−0.24
Kidney96.77−1.37
Gastric94.49−3.65
Prostate (same-organ)98.140.00

The cross-organ settings yield a mean AUC of 96.31%, only 1.83 percentage points below same-organ supervision. Although transferability varies across source tissues, all settings retain strong diagnostic performance.

(C) Qualitative visualization. To facilitate intuitive assessment, we provide representative constituent-level predictions from both in-domain (same-organ) and cross-organ settings, alongside their ground-truth annotations. These visualizations are included at page 23 of the revised manuscript and are available through the following anonymous link: [ https://anonymous.4open.science/r/FRAME_r2/ ].

Q2 & W2. Complementary and Event-specific Roles of mIF Channels

Thank you for this meaningful question.

  • Biological rationale. The seven channels provide complementary rather than interchangeable biological evidence. CKPAN identifies epithelial components, SMA supports smooth muscle recognition, CD31 marks vascular endothelium, S100 provides evidence associated with neural structures and adipocytes, FAP supports fibroblastic stroma recognition, and D2-40 identifies lymphatic endothelium, while DAPI serves as a nuclear counterstain. Their contributions are therefore event-dependent: S100 is particularly relevant to perineural invasion, whereas CD31 and D2-40 support vascular and lymphatic structure recognition, respectively. These markers are interpreted jointly with H&E morphology rather than as independent semantic labels (Secs. 3.2–3.3).

    To clarify these biological correspondences, we provide a pathologist-guided summary of shared cellular constituents, molecular markers, and anatomical structures in Fig. 9 (revised Appendix, p. 23; anonymous link).

mIF sourceNeural Dice (%)Δ (pp)
Liver88.69−1.25
Prostate89.08−0.86
Kidney88.47−1.47
Gastric90.06+0.12
Rectum (same-organ)89.940.00
  • Experimental validation. To evaluate whether marker-specific biological evidence transfers across organs, we assess S100-guided neural recognition in rectal H&E images using supervision from different tumor-specific mIF panels. Cross-panel Dice scores range from 88.47% to 90.06%, with a mean of 89.08%, only 0.87 percentage points below same-organ supervision (89.94%).
AuthorsOpenReview ↗

Response to W3

W3. Construction and Characterization of Histological-Isomerism Hard Subsets

Thank you for raising this important concern. We have further clarified the construction protocol and characterized the difficulty of the selected cases.

(1) Clinical motivation and construction protocol.

To our knowledge, no publicly available benchmark specifically evaluates composition-matched ROIs with opposite diagnostic labels. We therefore construct a pathologist-curated challenge subset for deployment-oriented evaluation. FRAME serves only as a candidate proposal mechanism: its Attribute Head AA (Sec. 3.3) predicts constituent probability maps to identify potentially composition-matched ROIs, without using event predictions or confidence scores for selection. Three pathologists (one associate chief physician and two attending physicians) subsequently review the candidates to determine final inclusion. To examine potential model-dependent selection bias, we independently repeat candidate screening using UN-SAM, obtaining 91.4% candidate-selection agreement with FRAME.

(2) Matching protocol and sample statistics.

For each ROI, we derive a constituent composition vector rir_i from the predicted constituent probability maps, summarizing the proportions of tumor, target, stroma, vessel, and other relevant constituents. ROIs with opposite diagnostic labels are paired when their Euclidean composition distance satisfies ∥ri−rj∥2<ϵ\|r_i-r_j\|_2<\epsilon, with a fixed threshold of ϵ=0.143\epsilon=0.143 (Sec. 4.2.3). Matched candidates undergo pathological review and subsequent deduplication.

DatasetCandidate ROIsMatched pairsFinal hard ROIs
TLSD214124187
PIND194113169
CAMELYON1617798151
PAIP 2021264166246
Total849501753

Pathologists retain 88.7% (753/849) of candidate ROIs. Individual ROIs may participate in multiple matched pairs; thus, the final column reports unique ROIs after deduplication.

(3) Why are these cases challenging?

To further assess the difficulty of the selected cases, we compared UNI feature embeddings of composition-matched, opposite-label ROI pairs against organ- and ROI-size-matched random opposite-label controls. The matched pairs exhibit a substantially higher mean cosine similarity (0.92 vs. 0.63, a difference of 0.29), indicating that diagnostically opposite cases are also much closer in learned morphological feature space, making them more challenging for appearance-based recognition. To further illustrate the pathological significance and visual difficulty of these hard cases, we provide representative pathologist-reviewed H&E case of PNI (Fig. 11), showing tumor cells adjacent to versus invading nerve structures. These examples make the distinction between similar tissue composition and different target-relative configurations more intuitive (revised Appendix, p. 24; anonymous supplementary link).

AuthorsOpenReview ↗

Response to Q3-Q4

Q3. Clinical Relevance and Prevalence of Histological Isomerism

Thank you for this valuable suggestion. We expand the pathological motivation and add representative clinical evidence and visual examples to the revised manuscript.

  • Definition and scope. Histological isomerism is not an independent disease entity or diagnostic category, but a conceptual formulation of diagnostic ambiguity in structural pathological events. As discussed in the Introduction (Lines 97–100) and Related Work (Sec. 2.3), existing appearance-centric pathology models lack explicit modeling of the anatomical relationships that define these events. It describes cases where similar tissue compositions yield different diagnoses due to different target-relative configurations (Sec. 3.1, Eq. 1). Representative examples include perineural invasion (PNI), lymphovascular invasion (LVI), and extramural vascular invasion (EMVI).

  • Clinical prevalence. These structural events are prevalent across multiple malignancies. A systematic review of 58 studies involving 22,900 colorectal cancer patients reported an 18.2% PNI incidence [1]. A meta-analysis of 7,004 gastric cancer patients reported a 35.9% PNI incidence [2]. For vascular invasion, LVI was reported in 26.3% of 139,026 colorectal cancer patients with documented status [3], while a meta-analysis of 7,262 rectal cancer patients reported a pooled 26% EMVI prevalence [4]. These findings demonstrate that the underlying structural pathological events represent clinically relevant diagnostic challenges across multiple cancer types.

  • Pathological illustrations. We further recognize the importance of making this concept accessible to readers without specialized pathological backgrounds. We have therefore added a pathologist-guided H&E–mIF biological correspondence schematic (Fig. 9) and representative pathologist-reviewed H&E cases of PNI (Fig. 11). These examples illustrate how similar tissue constituents can form different spatial configurations and lead to different pathological diagnoses (revised Appendix, pp. 23–24; anonymous supplementary link).

[1] DOI: 10.1097/PAS.0000000000000518

[2] DOI: 10.1136/jclinpath-2019-206372

[3] DOI: 10.1016/j.ijsu.2016.08.528

[4] DOI: 10.3748/wjg.v22.i4.1721

Q4. Definition and Role of the Parsed Target

Sorry for the confusion.

A Parsed Target in Fig. 2 denotes a model-predicted anatomical instance, represented by its target mask and target token, rather than an externally provided annotation.

Morphology Encoding generates a soft target prior SS from spatial morphology features and protein cues (Sec. 3.2, Eq. 2, Lines 199–210). This prior initializes target queries but is not a final target mask. The actual target masks MlM^l are progressively predicted by the Mask Prediction module within Coupled Constituent Parsing (Sec. 3.3, Eq. 3, Lines 216–229), yielding CLC^L, MLM^L, and TLT^L. The Attribute Head further predicts non-exclusive constituent attributes AA from CLC^L (Lines 230–235).

These outputs provide complementary inputs to Relation Reasoning: the refined target mask Mˉt\bar{M}_t defines the contour from which boundary anchor tokens Bt0B_t^0 are sampled, while AA and CLC^L identify the surrounding constituents and their spatial evidence. The Relation Builder then models their target-relative relationships (Eq. 4). Finally, the target token TLT^L serves as a query to read the resulting boundary configuration for structural-event prediction (Eq. 5; Sec. 3.4, Lines 242–265).

Reviewer_VHCcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FRAME, a framework for structural pathological event recognition that focuses on how tissue constituents are spatially organized around anatomical targets. The authors introduce the concept of histological isomerism, referring to cases where similar tissue compositions can lead to different diagnoses due to their spatial configurations. FRAME consists of three stages: protein-grounded morphology encoding using paired H&E and mIF data, coupled constituent parsing, and boundary-centric relational reasoning. The model uses mIF supervision during training but requires only H&E images at inference. Experiments on cancer diagnosis, lesion parsing, and structural-event recognition show improvements over several pathology foundation models and segmentation baselines.

Critical strengths
  • The paper addresses an interesting and clinically relevant problem. For pathological events such as perineural or vascular invasion, the spatial relationships between tissue components can be more important than their individual appearances. The motivation is reasonable and clearly presented.

  • The boundary-centric reasoning design seems to be a useful aspect of the method. Using anatomical boundaries as the reference for modeling surrounding tissue relationships is more directly connected to the diagnostic task than generic spatial aggregation.

Critical weaknesses
  • The fairness of the comparisons with pathology foundation models is not fully clear. FRAME uses several sources of supervision, including paired H&E–mIF data, target masks, constituent attributes, and task-specific labels. However, the main comparisons in Table 1 are primarily against general-purpose pathology foundation models such as UNI, CHIEF, and CONCH. It is not clear whether these baselines receive comparable task-specific supervision or adaptation. The large improvements reported in Table 1 are interesting, but they may partly reflect the additional molecular and structural supervision available to FRAME.

  • The paper also emphasizes good performance with limited training samples. I think the total supervision budget should be reported more clearly, including the amount of paired mIF data and parsing annotations used before the final task-specific training. H&E-only inference is useful, but it does not make the training conditions equivalent. Comparisons under more closely matched supervision settings would help establish the advantage of the proposed architecture.

  • More details are needed on the construction of the histological isomerism hard subsets. An important part of the paper is the evaluation on composition-matched cases, where tissue environments with similar constituent compositions have opposite diagnostic labels. FRAME shows quite large improvements on these subsets. However, the results depend heavily on how these samples are selected. For example, how are the constituent composition vectors obtained? Are they derived from independent annotations or model-predicted constituent maps? The matching threshold and the number of selected samples are also important for assessing the difficulty and representativeness of these subsets. Similar constituent proportions do not necessarily mean that the samples are otherwise comparable in morphology or other diagnostic features. Without a more detailed characterization of the selected cases, I am not fully convinced that the gains can be attributed specifically to resolving histological isomerism.

  • It is still difficult to separate the benefit of boundary geometry from that of the proposed relational reasoning architecture. The boundary-centric ablation is useful. On the histological isomerism task, the reported accuracy increases from 74.6% with composition-based representation and 83.2% with center-based representation to 95.7% with boundary-based modeling. This clearly suggests that boundary information matters. My question is whether the full relation reasoning module is necessary to obtain these gains. Boundary-based features already provide richer spatial information, including distance, orientation, and inside/outside relationships. A simpler geometry-aware predictor using the same features would be a useful comparison. The current experiments support the importance of target-relative geometry, but do not completely isolate the additional contribution of the relation interaction and contour transformer modules.

  • The generalizability of the protein-grounding stage needs further investigation. The protein-grounding module is trained using internal paired H&E–mIF data with tumor-specific marker panels. The reported protein prediction results are promising, and the auxiliary/fusion ablation suggests that protein-guided features are useful. Still, the extent to which this grounding transfers across cancer types, tissues, and different marker panels seem unclear. This is especially relevant because the paper presents FRAME as a general framework for structural pathological event recognition. Different anatomical targets may require different molecular markers, and not all markers are equally predictable from H&E morphology. More information about the size and composition of the mIF training data, along with cross-domain or cross-panel evaluation, would help assess how broadly the approach can be applied.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to W1–W2

Sincerely thank you for careful review ! We organize our responses into W1–W5, presented in three response windows (W1–W2, W3-W4, and W5). Supplementary figures are collected in the revised Appendix (starting from p. 23) and are also available through this anonymous link: https://anonymous.4open.science/r/FRAME_r1/. Apologize for any inconvenience.

W1. Benchmark Fairness

Apologize for the lack of clarity. For Table 1, all methods use the same dataset-provided WSIs and task-specific labels under identical downstream evaluation protocols.

Regarding structural supervision, target masks and constituent attributes are model-predicted intermediate outputs, not additional manually annotated supervision sources.

Specifically, Morphology Encoding predicts a soft target prior (SS) from spatial morphology features and protein cues (Sec. 3.2, Eq. 2, Lines 199–210). The actual target masks (MlM^l) are progressively predicted by the Mask Prediction module within Coupled Constituent Parsing (Sec. 3.3, Eq. 3, Lines 216–229), while the Attribute Head predicts non-exclusive constituent attributes (AA) from the learned spatial features (CLC^L) (Lines 230–235). No additional manually annotated target masks or constituent-attribute maps are provided in the Table 1 classification setting.

Regarding molecular supervision, it is a part of FRAME. Protein-grounded spatial tokens are learned from limited paired H&E–mIF data (8 pairs each cancer) during Morphology Encoding. The existing Component Contribution ablation (Sec. 4.2.4, Lines 445–450) examines the contributions of these components in structural-event recognition.

We fully agree that a corresponding evaluation on the general cancer classification tasks in Table 1 is also important. To directly address this concern, we additionally evaluate a FRAME variant that bypasses Morphology Encoding (ME) and uses the native frozen CTransPath features instead. The results are as follows:

MethodPANDA DiagnosisPANDA GradingCPTAC DiagnosisCPTAC GradingTCGA DiagnosisTCGA Grading
Best PFM0.9530.9440.8420.8130.8960.878
FRAME w/o ME0.9640.9480.8560.8290.9140.907
Full FRAME0.9810.9730.8670.8580.9370.929

Without ME, FRAME exceeds the best PFM results by 0.4–2.9 AUC points across all six evaluations. Incorporating the full ME stage further improves AUC by 1.1–2.9 points over this variant.

W2. Total Supervision Budget

Thank you for this thoughtful suggestion. We clarify the molecular and parsing supervision separately.

  • Paired H&E–mIF data. We expand Appendix A.2 to report the complete composition of our 5 internal tumor-specific cohorts. In total, the molecular grounding stage uses 20 patients, 40 same-section H&E–mIF pairs, and approximately 7.58 million spatially registered patches. The corresponding marker panels are detailed in Appendix A.2 (line 796-800). These spatially aligned measurements provide token-level molecular supervision during Morphology Encoding.
CohortPatientsPairsPatchesChannelsChannel names
Rectum7mIF48~1.74M7DAPI, SMA, S100, CD31, FAP, D2-40, CKPAN
Kidney7mIF48~1.28M7DAPI, CD20, CD23, CKpan, FOXP3, CD8, S100
Prostate9mIF48~1.60M9CKpan, P53, Ki-67, P504s, PSA, ERG, AR, P63, S100
Gastric15mIF48~1.37M15CD138, MPO, D2-40, S100, CD34, CKpan, P53, Ki-67, HER2, CEA, CDX2, MUC5AC, MUC6, CLDN18.2, Syn
Liver9mIF48~1.59M9P53, Ki-67, S100, Arginase-1, AFP, CK7, CK19, GPC3, CKpan
  • Parsing annotation budge. As clarified in W1, general cancer classification in Table 1 uses only dataset-provided H&E WSIs and task-specific labels, without additional parsing annotations or physician-corrected ROIs. For constituent parsing benchmarks, we use the corresponding public dataset annotations as ground truth. Physician-corrected ROI annotations are separately investigated in the annotation-efficiency experiments (Sec. 4.2.4, Lines 469–474; Appendix A.7.3).

Following your suggestion, we additionally evaluate FRAME with zero physician-corrected ROIs, extending the original annotation budget to 0–128:

ROIsH&E onlyw/o mIFFull
00.700.750.82
80.720.780.86
160.780.840.91
320.830.890.95
640.870.930.96
1280.900.950.97

Without physician-corrected ROIs, FRAME achieves 0.82 AUC, compared with 0.75 for FRAME without mIF supervision and 0.70 for the H&E-only baseline. The advantage persists across all annotation budgets, demonstrating that molecular grounding provides complementary supervision and reduces reliance on additional expert-corrected ROI annotations.

AuthorsOpenReview ↗

Response to W3-W4

W3. Construction and Validity of Histological-Isomerism Hard Subsets

Thank you for raising this important concern.

(1) How were the hard subsets constructed?

  • Motivation. To our knowledge, no publicly available benchmark specifically evaluates composition-matched tissue regions with opposite labels. We construct a pathologist-curated challenge subset for deployment-oriented evaluation. FRAME's Attribute Head only estimates constituent composition and nominates potentially composition-matched ROIs. Three pathologists (one associate chief physician and two attending physicians) review candidates for final inclusion. Institutional ethics approval information will be disclosed in the final Ethics Statement under double-blind review.

  • Candidate selection and annotation provenance. Candidate ROIs are identified using constituent probability maps predicted by FRAME's Attribute Head AA (Sec. 3.3). Their composition vectors summarize the spatial proportions of tumor, target, stroma, vessel, and other relevant constituents (Sec. 4.2.3). UN-SAM screening yields 91.4% candidate-selection agreement.

  • Matching criteria and sample counts. Oppositely labeled ROIs are matched when ∥ri−rj∥2<ϵ\|r_i-r_j\|_2<\epsilon, with ϵ=0.143\epsilon=0.143. Matched candidates are subsequently deduplicated after pathological review.

DatasetCandidate ROIsMatched pairsFinal hard ROIs
TLSD214124187
PIND194113169
CAMELYON1617798151
PAIP 2021264166246
Total849501753

Pathologists retain 88.7% (753/849) of candidate ROIs. Individual ROIs may participate in multiple matched pairs; thus, the final column counts unique ROIs rather than pair endpoints.

(2) Why does this evaluation capture histological isomerism, and does FRAME benefit from structural reasoning?

  • Pathological rationale. As defined in Sec. 3.1 (Eq. 1), histological isomerism describes similar constituent compositions yielding different diagnoses due to different target-relative configurations. For example, tumor cells adjacent to or invading a nerve may involve similar constituents but represent different pathological events. This perspective is consistent with studies of cellular neighborhoods [1] and spatial tumor microregions [2].

  • Subset validity. We agree that compositional similarity alone does not guarantee morphological equivalence. To further assess whether the selected cases are also similar in appearance beyond compositional matching, we compared UNI feature embeddings of composition-matched, opposite-label ROI pairs against organ- and ROI-size-matched random opposite-label pairs. The former exhibit a substantially higher mean cosine similarity (0.92 vs. 0.63), supporting greater similarity in learned morphological feature space.

  • Structural benefit. FRAME achieves 95.7%, 97.6%, 94.7%, and 81.7% accuracy on TLSD, PIND, CAMELYON16, and PAIP 2021, respectively, outperforming the strongest PFM baselines by 12.5 percentage points on average (Sec. 4.2.3, Fig. 5). Composition-only, center-based, and boundary-centric representations achieve 74.6%, 83.2%, and 95.7%, respectively (Sec. 4.2.4, Appendix Table 7), supporting explicit target-relative geometry.

[1] DOI:10.1016/j.cell.2020.07.005, Cell, 2020

[2] DOI:10.1038/s41586-024-08087-4, Nature, 2024

W4. Distinguishing Boundary Geometry from Relational Reasoning

Following your suggestion, we introduce a Geometry-aware MLP using the same constituent features and target-relative geometric descriptors as FRAME (Sec. 3.4). It comprises two fully connected layers (256 and 128 dimensions, GELU), mean pooling, and a linear classifier, without Relation Interaction or the Contour Transformer. We add each module separately to isolate its contribution.

VariantRelation InteractionContour TransformerIsomerism Acc. (%)PAIP AUCTLSD AUC
Geometry-aware MLP——88.80.9390.944
+ Relation Interaction✓—91.40.9510.952
+ Contour Transformer—✓92.00.9470.959
Full FRAME✓✓95.70.9580.969

The Geometry-aware MLP achieves 88.8% accuracy, confirming the value of boundary geometry alone. Full FRAME improves isomerism accuracy by 6.9 percentage points, with PAIP and TLSD AUC gains of 0.019 and 0.025, respectively. Both modules independently improve the baseline, while their combination performs best, demonstrating the benefits of local constituent–boundary interactions and global contour modeling.

AuthorsOpenReview ↗

Response to W5

W5. Generalizability of Protein Grounding Across Organs and Marker Panels

Thank you for highlighting this important aspect.

  • mIF training data. Following your suggestion, we organized the paired H&E–mIF collection by selecting four patients per cancer type, representing high, intermediate, low, and normal conditions, with two paired samples per patient. The complete cohort statistics, marker panels, and registered patch counts are reported in W2 (Total Supervision Budget).

  • Biological rationale. Although organ-specific morphology and molecular marker profiles are prevalent (e.g., CDX2 for intestinal epithelial differentiation and CK19 for biliary epithelial phenotyping), different organs share fundamental cellular constituents and anatomical structures. These shared biological components provide potential cross-organ molecular anchors for protein-grounded representation learning. For example, smooth muscle cells, vascular endothelial cells, nerve-associated cells, fibroblasts, adipocytes, and lymphatic endothelial cells are found across multiple tissue types and can be characterized using corresponding protein markers. This shared cellular and structural basis motivates the transferability of molecular grounding beyond its tissue of origin [3]. We summarize these cross-organ correspondences below and provide a pathologist-guided schematic in Fig. 9 (revised Appendix, p. 23; anonymous link), illustrating how shared molecular markers provide biological evidence for recognizing cellular constituents and anatomical structures from H&E morphology.

Shared cellular constituentsProtein markersAssociated anatomical structures
Smooth muscle cellsSMASmooth muscle
Vascular endothelial cellsCD31Capillaries, veins, arteries
Nerve-associated cellsS100 (SOX10)Nerves
AdipocytesS100 (Perilipin-1)Adipose tissue
Fibroblast-associated cellsFAPFibrous stroma
Lymphatic endothelial cellsD2-40Lymphatic vessels
  • Cross-organ transfer. We transfer molecular grounding learned from different cancer types to prostate adenocarcinoma diagnosis on PANDA. As shown below, cross-organ supervision achieves AUCs of 94.49%–97.90%, with a mean of 96.31%, only 1.83 percentage points below prostate-specific supervision (98.14%). Although the transferability varies across source tissues, all cross-organ settings retain strong diagnostic performance.
mIF sourcePANDA AUC (%)Δ (pp)
Liver96.08−2.06
Rectum97.90−0.24
Kidney96.77−1.37
Gastric94.49−3.65
Prostate (same-organ)98.140.00
  • Cross-panel robustness. We further evaluate S100-guided neural recognition in rectal H&E images using mIF supervision from different tumor-specific panels. Cross-panel Dice scores remain between 88.47% and 90.06%, with a mean of 89.08%, only 0.87 percentage points below same-organ supervision (89.94%).
mIF sourceNeural Dice (%)Δ (pp)
Liver88.69−1.25
Prostate89.08−0.86
Kidney88.47−1.47
Gastric90.06+0.12
Rectum (same-organ)89.940.00
  • Qualitative evidence. To complement the quantitative transfer results, we provide representative constituent-level predictions from both same-organ (upper part) and cross-organ (lower part) settings, together with their corresponding ground-truth annotations. These visualizations show that the major tissue constituents remain accurately recognized even when molecular grounding is transferred across organs, providing intuitive support for the generalizability reported above. The examples are included in the revised Appendix (Fig.10, page 23) and are also available through the anonymous supplementary link: [ https://anonymous.4open.science/r/FRAME_r1/ ].

[3] DOI: s41586-025-09053-4, Nature, 2025.

AuthorsOpenReview ↗

With Sincere Gratitude for Your Thoughtful Review — We Would Truly Value Your Feedback

Dear Reviewer VHCc,

Thank you again for your exceptionally thoughtful and constructive feedback, which has greatly helped us strengthen our work! We would sincerely appreciate knowing whether our rebuttal has addressed your concerns and whether you might consider revisiting your evaluation in light of our additional evidence. If any questions remain, we would be more than happy to provide further clarification.

We truly value your insights and look forward to your feedback!

Comments on the paper

1 comment
Area_Chair_Tt5aOpenReview ↗

Reminder – Authors' rebuttal posted, please weigh in

Dear Reviewers,

I'd like to draw your attention to the current state of the discussion: we have passed the midpoint of the discussion period, and the authors' rebuttal is now available.

At this point, I would like to ask each of you to re-engage with the paper and the ongoing discussion. Specifically, please review the authors' replies, assess whether your individual concerns and questions have been satisfactorily resolved, and share any additional feedback you may have with the authors.

Your continued input will be essential for the committee to make a sound and timely decision.

Thank you in advance for your engagement.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.