- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DeepSite-MRI, a public benchmark for diagnosis of deep-seated intracranial tumors from preoperative 3D T1-CE MRI. The dataset contains 249 biopsy-confirmed cases spanning Glioma, PCNSL, Germinoma, and a Rare category. To reduce the cost of spatial annotation, the authors propose a Two-Stage spatial Prior pipeline (TSP), which first uses a vision-language model for slice-wise coarse tumor localization and then applies a subject-specific voxel-wise MLP for refinement.
The generated spatial priors are validated through an independent expert study and are also compared with MedSAM3 on a 50-case subset with expert tumor-core delineations. The paper further benchmarks ten 3D CNN and Transformer architectures under no-prior, coarse-prior, and refined-prior settings, and analyzes performance as a function of training-set size. The best reported four-class classification accuracy is 78.9%.
Critical strengths
The strongest contribution is the dataset itself. Deep-seated intracranial tumors are underrepresented in existing public MRI benchmarks, and biopsy-confirmed pathological labels are particularly valuable in this setting. The paper also makes a serious effort to validate the automatically generated spatial priors rather than treating them as ground truth without verification.
I also found the expert validation relatively strong. The evaluation uses three independent board-certified neurosurgeons, and the reported clinical acceptability and inter-rater agreement are high. In addition, the comparison against expert-delineated tumor cores on 50 cases provides a useful quantitative check of the TSP localization quality. The broad evaluation across ten 3D backbones is also helpful for establishing initial benchmark baselines.
Critical weaknesses
My main technical concern is the formulation of the second stage of TSP. The paper states that the coarse prior systematically overestimates lesion extent and includes surrounding non-tumor tissue, and that the MLP refinement corrects boundaries and suppresses background noise. However, the final refined prior is defined as the voxel-wise union of the coarse prior and the MLP output. Under this definition, false-positive voxels already present in the coarse prior cannot be removed. Moreover, the appendix states that the MLP is trained exclusively on high-confidence coarse-prior voxels described as positive samples, but the construction of negative training samples is not clearly specified. This makes it difficult to understand how the reported refinement actually performs tumor/background discrimination.
My second concern is the robustness of the benchmark conclusions under the severe class imbalance and limited domain diversity. Of the 249 cases, only 18 are Germinoma and 10 are in the heterogeneous Rare category, leading to very large fold-wise variation in minority-class recall. In addition, 232/249 scans come from the same Philips Ingenia CX system. The dataset is still useful, but the current results are better viewed as an initial benchmark than as evidence of robust multi-center generalization or reliable four-class clinical diagnosis.
The data-efficiency conclusions appear somewhat stronger than the presented scaling experiment supports. Lower performance with fewer training cases demonstrates sensitivity to data scale, but does not by itself establish a fundamental architectural deficiency or isolate self-attention as the cause.
Questions for authors
Main-text-driven
Q1: The coarse prior is described as overestimating tumor extent and including surrounding edema/background, but the refined prior is defined as the union of the coarse prior and the MLP prediction. Under this formulation, how can the refinement remove false-positive regions already present in the coarse prior? Please clarify the exact implementation of the refinement stage and whether Eq. (2) represents the actual inference procedure.
Q2: The dataset is highly imbalanced, with only 18 Germinoma and 10 Rare cases. Could the authors additionally report macro-F1, balanced accuracy, or macro-AUROC for the main benchmark models? The current overall accuracy can be dominated by Glioma and PCNSL performance and may not adequately reflect minority-class behavior.
Q3: The “Rare” class contains only ten cases and is described as comprising several low-frequency pathologies. Could the authors report the actual pathological composition of this class and explain the clinical rationale for treating these heterogeneous diagnoses as one classification category?
Appendix-driven
Q4: The appendix states that the voxel-wise MLP is trained exclusively on high-confidence coarse-prior voxels, which are described as positive samples. How are negative/background samples constructed, what loss is used, and how is class balance handled during subject-specific MLP training?
Q5: The main text states that Qwen3-VL-Thinking was selected from several candidate VLMs and refers to Appendix C for the comparison, but I could not find this VLM-selection experiment in Appendix C. Could the authors provide the candidate-model comparison and clarify the selection criterion, including whether the 50-case expert-validation subset was used in model or prompt selection?
6 comments
Response to Reviewer KbC9 Comments (part 2)
:
MLP training incorporates both positive and negative pseudo-labels. Positive pseudo-labels are derived from relatively reliable regions within the coarse prior, while negative pseudo-labels are obtained from surrounding brain tissue. Class imbalance is addressed through a 1:1 sampling ratio of positive and negative samples. The MLP is trained separately for each patient using binary cross-entropy loss and L2 regularisation, without requiring additional manual voxel-level annotations.
The trained MLP outputs are used to generate the candidate region and to apply probability weighting to voxels within the union region. We will provide these training details in the revised manuscript, clearly distinguishing pseudo-label construction from the generation of the final spatial prior.
This pipeline provides a reproducible and replaceable spatial prior baseline for the benchmark, facilitating comparisons of the classification improvements achieved by different spatial prior methods under a consistent evaluation setting.
:
The selection of the VLM was primarily guided by its ability to identify lesion locations in the original MRI scans and support subsequent 3D spatial prior construction. During preliminary trials, we tested Kimi, Qwen2.5-VL, and GPT-5. Under the experimental settings used at the time, the first two models did not return usable tumour localisation results, while the GPT-5 trial workflow produced regenerated MRI images rather than the requested bounding-box coordinates. These observations were used to identify a suitable localisation component for our pipeline and do not constitute a rigorous ranking of model performance.
The current benchmark employs Qwen3-VL-Thinking. The 50 expert-validated cases were used to evaluate the outputs of the subsequent pipeline and were not involved in VLM selection or prompt refinement. We will correct the appendix references and provide traceable details of the preliminary trial settings and observations, clearly distinguishing exploratory trials from formal quantitative comparisons.
The core contribution of this work lies in the biopsy-confirmed dataset and its unified evaluation benchmark. The localisation component is replaceable, allowing the research community to investigate improved methods under consistent data and evaluation conditions. The value of the dataset does not depend on the current VLM achieving optimal performance.
:
Multi-centre data sources and scanner diversity represent two distinct dimensions. Currently, 232 out of 249 cases (93.2%) were acquired using the Philips Ingenia CX, and the existing results primarily characterise model performance under this acquisition distribution. We will rename the subsection "Scanner Distribution", explicitly highlight this proportion in the main text, and retain the cross-tabulation of scanner models and pathological diagnoses.
The dataset provides biopsy-confirmed labels for deep-seated intracranial lesions, spatial priors, and a unified evaluation protocol, supporting method comparisons within the cohort. Cross-scanner and cross-centre generalisation will be treated as separate evaluation objectives in future dataset expansions, rather than being inferred from the current cross-validation results.
:
We thank the reviewer for highlighting the need to clarify the scope of our experimental interpretations. The primary purpose of the data scaling experiments is to quantify the performance gains achieved by increasing the number of training samples in this task, providing a reference for future dataset expansion and research on data-efficient learning.
We agree that our original phrasing overstated the causal inference. The scaling experiment shows that performance is sensitive to training set size, but we acknowledge that this observation alone cannot establish a fundamental architectural deficiency nor isolate self-attention as the cause. We will therefore revise the manuscript to soften these claims: the relevant sentences will only describe the finding as an empirical sensitivity to data scale without explicitly stating the effect to specific architectural components. We believe this revision aligns the conclusions more accurately with the evidence presented.
Response to author
I appreciate the authors' detailed rebuttal, which addresses most of my concerns. However, the consistency between the clarified refinement method and the reported experimental results remains uncertain. Therefore, I will maintain my current rating of 2
Response to Reviewer KbC9 Comments (part 1)
We thank you for your detailed review and constructive suggestions on the methodology and evaluation of DeepSite-MRI. Your comments help us clarify the spatial prior pipeline and the scope of the benchmark conclusions. We therefore provide further details on prior refinement, MLP training, class grouping, and VLM selection, supplement the minority-class evaluation, and clarify the scope of our generalisation claims and the interpretation of the data scaling experiments.
Union Operation and Background Suppression (Q1,W1):
Equation (2) describes only the union operation that determines the candidate region. The complete inference process also includes probability weighting to determine the prior intensity of each voxel within this region. We will provide the full procedure in the revised manuscript as follows:
(1) Positive and negative pseudo-labels are sampled from relatively reliable regions within the coarse prior and surrounding brain tissue, respectively, to train a patient-specific MLP.
(2) The MLP predictions are thresholded, and connected regions intersecting with the coarse prior are retained to obtain the candidate region (M).
(3) The union of (M) and the coarse prior (C) is then weighted by the MLP-predicted probability (p(v)):
B=C∪M,S(v)=1B(v)⋅(v).
The resulting continuous-valued prior (S) is concatenated with the MRI for classification.
Therefore, although low-confidence voxels within the coarse prior remain in the candidate region, their prior responses are reduced. Here, "background suppression" refers to attenuating these responses rather than removing existing voxels. We will narrow the scope of this claim accordingly.
Class Imbalance and Minority-Class Evaluation (Q2):
We have additionally reported balanced accuracy, macro F1, and macro AUROC (OvR) for three representative models using the Refined Prior. These metrics assign equal weight to each class, ensuring that minority and majority classes contribute equally to the overall evaluation. The results are presented below:
The class imbalance also reflects the challenges of collecting such data. Cases of deep-seated intracranial tumours with both preoperative MRI and biopsy-confirmed diagnoses are limited, and certain pathological types are particularly difficult to accumulate. By publicly releasing DeepSite-MRI, we aim to provide a shared resource for the research community to investigate minority-class recognition, few-shot learning, and data-efficient methods, while continuing to expand the dataset in future releases.
Rare Category Composition and Grouping Rationale (Q3):
The “Rare” category comprises 7 cases of brain metastasis, 1 case of brain abscess, 1 case of melanoma, and 1 case of noninfectious granuloma, totaling 10 cases. Treating these low-frequency diagnoses as separate classes would pose challenges in the following two respects.
From a clinical perspective, the three main categories have distinct pathological characteristics and diagnostic and treatment considerations, so we retain them as separate classes to study preoperative differential diagnosis. The remaining lesions are also part of the differential diagnosis in actual biopsy cohorts. Retaining these cases allows us to evaluate the model’s ability to distinguish the three main diagnostic categories from other lesions. Rare represents “other diagnoses” in the current benchmark; it does not imply that these diseases share the same treatment approach, nor can it replace a specific pathological diagnosis following biopsy.
From a technical perspective, each of the remaining diagnoses has only 1–7 cases, with three classes containing just one case each. When the sole case of a class is assigned to the validation set, no samples of that class remain in the training set, making stable five-fold cross-validation difficult. Moreover, the very small number of training samples makes it difficult to adequately capture within-class imaging variability, limiting the model’s ability to learn generalizable class-specific features.
We therefore grouped these low-frequency cases into Rare to balance the feasibility of evaluation at the current dataset size with coverage of cases encountered in clinical practice. As more data accumulate, we will further assess the feasibility of using more fine-grained categories.