acceptodds
Under review as a conference paper at ICLR 2027

DeepSite-MRI: A Public Benchmark for Deep-Seated Intracranial Tumor Diagnosis

Abstract

Deep-seated intracranial tumors (thalamus, basal ganglia, brainstem) are adjacent to critical functional areas, making direct surgical resection typically infeasible; clinicians rely on stereotactic biopsy to determine subtypes for treatment planning. However, existing brain tumor datasets focus on superficial tumors, and no public benchmark with biopsy-confirmed labels exists for deep-seated tumors. To bridge this gap, we present DeepSite-MRI, the first such benchmark, comprising 249 biopsy-confirmed cases across four subtypes (Glioma, PCNSL, Germinoma, Rare), together with a Two-Stage spatial Prior pipeline (TSP) that combines VLM-guided coarse localization with MLP-based refinement, reducing per-case annotation time from approximately 90 minutes to approximately 3 minutes (96% clinical acceptance rate); this paradigm is transferable to other data-scarce medical imaging domains. Meanwhile, we systematically evaluate 10 mainstream architectures spanning CNNs and Transformers to characterize the difficulty and research value of this benchmark. Results show that even well-established methods that perform strongly on large-scale brain tumor benchmarks underperform on DeepSite-MRI (best accuracy 78.90%), which not only demonstrates the potential for further research in this direction, but also indicates that the benchmark can serve as a foundation for data-constrained learning and clinical translation. Code and data will be released upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Brain MRI, 3D Medical Image, Deep-Seated Brain Tumors Datasets, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KbC9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DeepSite-MRI, a public benchmark for diagnosis of deep-seated intracranial tumors from preoperative 3D T1-CE MRI. The dataset contains 249 biopsy-confirmed cases spanning Glioma, PCNSL, Germinoma, and a Rare category. To reduce the cost of spatial annotation, the authors propose a Two-Stage spatial Prior pipeline (TSP), which first uses a vision-language model for slice-wise coarse tumor localization and then applies a subject-specific voxel-wise MLP for refinement.

The generated spatial priors are validated through an independent expert study and are also compared with MedSAM3 on a 50-case subset with expert tumor-core delineations. The paper further benchmarks ten 3D CNN and Transformer architectures under no-prior, coarse-prior, and refined-prior settings, and analyzes performance as a function of training-set size. The best reported four-class classification accuracy is 78.9%.

Critical strengths

The strongest contribution is the dataset itself. Deep-seated intracranial tumors are underrepresented in existing public MRI benchmarks, and biopsy-confirmed pathological labels are particularly valuable in this setting. The paper also makes a serious effort to validate the automatically generated spatial priors rather than treating them as ground truth without verification.

I also found the expert validation relatively strong. The evaluation uses three independent board-certified neurosurgeons, and the reported clinical acceptability and inter-rater agreement are high. In addition, the comparison against expert-delineated tumor cores on 50 cases provides a useful quantitative check of the TSP localization quality. The broad evaluation across ten 3D backbones is also helpful for establishing initial benchmark baselines.

Critical weaknesses

My main technical concern is the formulation of the second stage of TSP. The paper states that the coarse prior systematically overestimates lesion extent and includes surrounding non-tumor tissue, and that the MLP refinement corrects boundaries and suppresses background noise. However, the final refined prior is defined as the voxel-wise union of the coarse prior and the MLP output. Under this definition, false-positive voxels already present in the coarse prior cannot be removed. Moreover, the appendix states that the MLP is trained exclusively on high-confidence coarse-prior voxels described as positive samples, but the construction of negative training samples is not clearly specified. This makes it difficult to understand how the reported refinement actually performs tumor/background discrimination.

My second concern is the robustness of the benchmark conclusions under the severe class imbalance and limited domain diversity. Of the 249 cases, only 18 are Germinoma and 10 are in the heterogeneous Rare category, leading to very large fold-wise variation in minority-class recall. In addition, 232/249 scans come from the same Philips Ingenia CX system. The dataset is still useful, but the current results are better viewed as an initial benchmark than as evidence of robust multi-center generalization or reliable four-class clinical diagnosis.

The data-efficiency conclusions appear somewhat stronger than the presented scaling experiment supports. Lower performance with fewer training cases demonstrates sensitivity to data scale, but does not by itself establish a fundamental architectural deficiency or isolate self-attention as the cause.

Questions for authors

Main-text-driven

Q1: The coarse prior is described as overestimating tumor extent and including surrounding edema/background, but the refined prior is defined as the union of the coarse prior and the MLP prediction. Under this formulation, how can the refinement remove false-positive regions already present in the coarse prior? Please clarify the exact implementation of the refinement stage and whether Eq. (2) represents the actual inference procedure.

Q2: The dataset is highly imbalanced, with only 18 Germinoma and 10 Rare cases. Could the authors additionally report macro-F1, balanced accuracy, or macro-AUROC for the main benchmark models? The current overall accuracy can be dominated by Glioma and PCNSL performance and may not adequately reflect minority-class behavior.

Q3: The “Rare” class contains only ten cases and is described as comprising several low-frequency pathologies. Could the authors report the actual pathological composition of this class and explain the clinical rationale for treating these heterogeneous diagnoses as one classification category?

Appendix-driven

Q4: The appendix states that the voxel-wise MLP is trained exclusively on high-confidence coarse-prior voxels, which are described as positive samples. How are negative/background samples constructed, what loss is used, and how is class balance handled during subject-specific MLP training?

Q5: The main text states that Qwen3-VL-Thinking was selected from several candidate VLMs and refers to Appendix C for the comparison, but I could not find this VLM-selection experiment in Appendix C. Could the authors provide the candidate-model comparison and clarify the selection criterion, including whether the 50-case expert-validation subset was used in model or prompt selection?

6 comments
AuthorsOpenReview ↗

Response to Reviewer KbC9 Comments (part 1)

We thank you for your detailed review and constructive suggestions on the methodology and evaluation of DeepSite-MRI. Your comments help us clarify the spatial prior pipeline and the scope of the benchmark conclusions. We therefore provide further details on prior refinement, MLP training, class grouping, and VLM selection, supplement the minority-class evaluation, and clarify the scope of our generalisation claims and the interpretation of the data scaling experiments.

Union Operation and Background Suppression{\color{#24577a}\text{Union Operation and Background Suppression}} (Q1,W1){\color{#247a78}\text{(Q1,W1)}}:

Equation (2) describes only the union operation that determines the candidate region. The complete inference process also includes probability weighting to determine the prior intensity of each voxel within this region. We will provide the full procedure in the revised manuscript as follows:

(1) Positive and negative pseudo-labels are sampled from relatively reliable regions within the coarse prior and surrounding brain tissue, respectively, to train a patient-specific MLP.

(2) The MLP predictions are thresholded, and connected regions intersecting with the coarse prior are retained to obtain the candidate region (M).

(3) The union of (M) and the coarse prior (C) is then weighted by the MLP-predicted probability (p(v)):

B=C∪M,S(v)=1B(v)⋅(v).B=C\cup M,\qquad S(v)=\mathbf{1}_{B}(v)\cdotp(v).

The resulting continuous-valued prior (S) is concatenated with the MRI for classification.

Therefore, although low-confidence voxels within the coarse prior remain in the candidate region, their prior responses are reduced. Here, "background suppression" refers to attenuating these responses rather than removing existing voxels. We will narrow the scope of this claim accordingly.

Class Imbalance and Minority-Class Evaluation{\color{#24577a}\text{Class Imbalance and Minority-Class Evaluation}} (Q2){\color{#247a78}\text{(Q2)}}:

We have additionally reported balanced accuracy, macro F1, and macro AUROC (OvR) for three representative models using the Refined Prior. These metrics assign equal weight to each class, ensuring that minority and majority classes contribute equally to the overall evaluation. The results are presented below:

Model (Refined Prior)Balanced AccuracyMacro F1Macro AUROC (OvR)
ConvNeXt3D57.2%58.9%0.81
UniFormer53.3%55.2%0.79
ResNet3D51.3%52.8%0.77

The class imbalance also reflects the challenges of collecting such data. Cases of deep-seated intracranial tumours with both preoperative MRI and biopsy-confirmed diagnoses are limited, and certain pathological types are particularly difficult to accumulate. By publicly releasing DeepSite-MRI, we aim to provide a shared resource for the research community to investigate minority-class recognition, few-shot learning, and data-efficient methods, while continuing to expand the dataset in future releases.

Rare Category Composition and Grouping Rationale{\color{#24577a}\text{Rare Category Composition and Grouping Rationale}} (Q3){\color{#247a78}\text{(Q3)}}:

The “Rare” category comprises 7 cases of brain metastasis, 1 case of brain abscess, 1 case of melanoma, and 1 case of noninfectious granuloma, totaling 10 cases. Treating these low-frequency diagnoses as separate classes would pose challenges in the following two respects.

From a clinical perspective, the three main categories have distinct pathological characteristics and diagnostic and treatment considerations, so we retain them as separate classes to study preoperative differential diagnosis. The remaining lesions are also part of the differential diagnosis in actual biopsy cohorts. Retaining these cases allows us to evaluate the model’s ability to distinguish the three main diagnostic categories from other lesions. Rare represents “other diagnoses” in the current benchmark; it does not imply that these diseases share the same treatment approach, nor can it replace a specific pathological diagnosis following biopsy.

From a technical perspective, each of the remaining diagnoses has only 1–7 cases, with three classes containing just one case each. When the sole case of a class is assigned to the validation set, no samples of that class remain in the training set, making stable five-fold cross-validation difficult. Moreover, the very small number of training samples makes it difficult to adequately capture within-class imaging variability, limiting the model’s ability to learn generalizable class-specific features.

We therefore grouped these low-frequency cases into Rare to balance the feasibility of evaluation at the current dataset size with coverage of cases encountered in clinical practice. As more data accumulate, we will further assess the feasibility of using more fine-grained categories.

AuthorsOpenReview ↗

Response to Reviewer KbC9 Comments (part 2)

MLP Training and Pseudo-Label Construction{\color{#24577a}\text{MLP Training and Pseudo-Label Construction}} (Q4){\color{#247a78}\text{(Q4)}}:

MLP training incorporates both positive and negative pseudo-labels. Positive pseudo-labels are derived from relatively reliable regions within the coarse prior, while negative pseudo-labels are obtained from surrounding brain tissue. Class imbalance is addressed through a 1:1 sampling ratio of positive and negative samples. The MLP is trained separately for each patient using binary cross-entropy loss and L2 regularisation, without requiring additional manual voxel-level annotations.

The trained MLP outputs are used to generate the candidate region and to apply probability weighting to voxels within the union region. We will provide these training details in the revised manuscript, clearly distinguishing pseudo-label construction from the generation of the final spatial prior.

This pipeline provides a reproducible and replaceable spatial prior baseline for the benchmark, facilitating comparisons of the classification improvements achieved by different spatial prior methods under a consistent evaluation setting.

VLM Selection and Validation Independence{\color{#24577a}\text{VLM Selection and Validation Independence}} (Q5){\color{#247a78}\text{(Q5)}}:

The selection of the VLM was primarily guided by its ability to identify lesion locations in the original MRI scans and support subsequent 3D spatial prior construction. During preliminary trials, we tested Kimi, Qwen2.5-VL, and GPT-5. Under the experimental settings used at the time, the first two models did not return usable tumour localisation results, while the GPT-5 trial workflow produced regenerated MRI images rather than the requested bounding-box coordinates. These observations were used to identify a suitable localisation component for our pipeline and do not constitute a rigorous ranking of model performance.

The current benchmark employs Qwen3-VL-Thinking. The 50 expert-validated cases were used to evaluate the outputs of the subsequent pipeline and were not involved in VLM selection or prompt refinement. We will correct the appendix references and provide traceable details of the preliminary trial settings and observations, clearly distinguishing exploratory trials from formal quantitative comparisons.

The core contribution of this work lies in the biopsy-confirmed dataset and its unified evaluation benchmark. The localisation component is replaceable, allowing the research community to investigate improved methods under consistent data and evaluation conditions. The value of the dataset does not depend on the current VLM achieving optimal performance.

Scanner Concentration and Generalisation Limits{\color{#24577a}\text{Scanner Concentration and Generalisation Limits}} (W2){\color{#247a78}\text{(W2)}}:

Multi-centre data sources and scanner diversity represent two distinct dimensions. Currently, 232 out of 249 cases (93.2%) were acquired using the Philips Ingenia CX, and the existing results primarily characterise model performance under this acquisition distribution. We will rename the subsection "Scanner Distribution", explicitly highlight this proportion in the main text, and retain the cross-tabulation of scanner models and pathological diagnoses.

The dataset provides biopsy-confirmed labels for deep-seated intracranial lesions, spatial priors, and a unified evaluation protocol, supporting method comparisons within the cohort. Cross-scanner and cross-centre generalisation will be treated as separate evaluation objectives in future dataset expansions, rather than being inferred from the current cross-validation results.

Interpretation of Data Scaling Results{\color{#24577a}\text{Interpretation of Data Scaling Results}} (W3){\color{#247a78}\text{(W3)}}:

We thank the reviewer for highlighting the need to clarify the scope of our experimental interpretations. The primary purpose of the data scaling experiments is to quantify the performance gains achieved by increasing the number of training samples in this task, providing a reference for future dataset expansion and research on data-efficient learning.

We agree that our original phrasing overstated the causal inference. The scaling experiment shows that performance is sensitive to training set size, but we acknowledge that this observation alone cannot establish a fundamental architectural deficiency nor isolate self-attention as the cause. We will therefore revise the manuscript to soften these claims: the relevant sentences will only describe the finding as an empirical sensitivity to data scale without explicitly stating the effect to specific architectural components. We believe this revision aligns the conclusions more accurately with the evidence presented.

Reviewer_KbC9OpenReview ↗

Response to author

I appreciate the authors' detailed rebuttal, which addresses most of my concerns. However, the consistency between the clarified refinement method and the reported experimental results remains uncertain. Therefore, I will maintain my current rating of 2

Reviewer_8D4UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DeepSite-MRI, a dataset of 249 biopsy-confirmed deep-seated intracranial tumor cases covering four categories: Glioma, PCNSL, Germinoma, and Rare. It also proposes TSP, a two-stage pipeline combining VLM-based localization with subject-specific MLP refinement to generate spatial priors. These priors are concatenated with MRI volumes for downstream classification. Experiments evaluate ten architectures under three prior conditions.

Critical strengths
  1. This paper addresses a clinically relevant problem involving deep-seated intracranial tumors, and collecting MRI data paired with biopsy-confirmed diagnoses represents a valuable effort.

  2. Evaluating ten architectures under Without Prior, Coarse Prior, and Refined Prior conditions provides useful baseline results and evidence that spatial priors can improve classification accuracy.

Critical weaknesses
  1. The authors repeatedly emphasize the annotation time reduction from 90 to approximately 3 minutes. Voxel-wise delineation is not required for case-level classification. For segmentation, the paper has not demonstrated that annotations produced within 3 minutes achieve accuracy comparable to manual delineation. Please give a detailed explanation.

  2. The rationale for the four-class categorization is insufficiently explained in this paper. Although distinguishing pathological tumor types has clinical value, the paper should justify the specific grouping into Glioma, PCNSL, Germinoma, and Rare.

  3. Scanner diversity is emphasized disproportionately to the actual acquisition distribution. The heading "Scanner Diversity" at L198 and the emphasis on three manufacturers and four MRI systems present acquisition diversity as a dataset strength. However, 232/249 cases (93.2%) originate from Philips Ingenia CX. Although the limitation is acknowledged, the presentation should more accurately reflect this strong concentration.

  4. For Block 2, taking the union of the coarse prior and the MLP prediction preserves every voxel in the coarse prior, including incorrectly included background. This appears inconsistent with the claim of “correcting boundary deviations and suppressing background noise” in Appendix A.2, L733–735. Please explain how it removes or suppresses over-segmented regions.

  5. The extent of segmentation annotation is unclear. Tab. 1 lists DeepSite-MRI as a 249-case dataset supporting both classification and segmentation, whereas Appendix B explicitly describes expert-delineated tumor cores for only 50 cases. The table should distinguish these annotation levels rather than leave the impression that all 249 cases have equivalent segmentation references.

  6. The reported improvements are inconsistent with Table 4. L893 reports an accuracy gain of 2.50 percentage points for ResNet, whereas Table 4 shows an increase from 67.27% to 74.61%, corresponding to 7.34 percentage points. Based on Table 4, the minimum gain is 3.24 percentage points for I3D, and the average gain is approximately 4.71 rather than 4.76.

Questions for authors
  1. What does the approximately 3-minute annotation time include? Does it cover expert review and correction?

  2. What clinical rationale motivates this specific four-class grouping?

  3. For block 2, if it preserves the entire coarse region, how does the refinement suppress background that was incorrectly included?

  4. Please explain the experimental results in detail.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8D4U Comments (part1)

We thank you for your careful review and constructive feedback on the DeepSite-MRI benchmark and its annotation pipeline. Your comments help us clarify the intended role of the spatial priors and the scope of the dataset and evaluation. We therefore provide further details on the timing measurements, class definitions, prior refinement, scanner distribution, and annotation coverage, and address the reported numerical inconsistencies.

Annotation Efficiency and Spatial Prior Utility{\color{#24577a}\text{Annotation Efficiency and Spatial Prior Utility}} (Q1,W1){\color{#247a78}\text{(Q1,W1)}}:

1)The approximately 3-minute duration refers to the automated processing time for a single raw MRI scan with TSP deployed on a single RTX 4090 GPU, covering preprocessing, model inference, and spatial prior generation. It excludes subsequent expert review and manual corrections.

2)Detailed manual segmentation and TSP-generated spatial priors serve different purposes, so directly comparing their segmentation accuracy does not measure the contribution of this work. Our contribution is the automatic generation of spatial cues that aid classification, rather than achieving the boundary accuracy of detailed manual segmentation. The utility of these spatial cues is assessed through expert evaluations and downstream classification experiments.

Rationale for Four-Class Categorization{\color{#24577a}\text{Rationale for Four-Class Categorization}} (Q2,W2){\color{#247a78}\text{(Q2,W2)}}:

We grouped deep-seated intracranial lesions into four categories for the following reasons.

The collection of such data is constrained by lesion location, the requirement for biopsy confirmation, and the completeness of preoperative MRI data. For example, thalamic tumors have been reported to account for approximately 5% of all brain tumors, but this proportion applies only to thalamic tumors (https://link.springer.com/article/10.1186/s12883-015-0487-x ). In the current cohort of 249 cases, Glioma, PCNSL, and Germinoma are the relatively major categories, with 149, 72, and 18 cases, respectively. The remaining cases include 7 cases of brain metastasis, 1 case of brain abscess, 1 case of melanoma, and 1 case of noninfectious granuloma, totaling 10 cases. Treating these low-frequency diagnoses as separate classes would pose challenges in the following two respects.

From a clinical perspective, the three main categories have distinct pathological characteristics and diagnostic and treatment considerations, so we retain them as separate classes to study preoperative differential diagnosis. The remaining lesions are also part of the differential diagnosis in actual biopsy cohorts. Retaining these cases allows us to evaluate the model’s ability to distinguish the three main diagnostic categories from other lesions. Rare represents “other diagnoses” in the current benchmark; it does not imply that these diseases share the same treatment approach, nor can it replace a specific pathological diagnosis following biopsy.

From a technical perspective, each of the remaining diagnoses has only 1–7 cases, with three classes containing just one case each. When the sole case of a class is assigned to the validation set, no samples of that class remain in the training set, making stable five-fold cross-validation difficult. Moreover, the very small number of training samples makes it difficult to adequately capture within-class imaging variability, limiting the model’s ability to learn generalizable class-specific features.

We therefore grouped these low-frequency cases into Rare to balance the feasibility of evaluation at the current dataset size with coverage of cases encountered in clinical practice. As more data accumulate, we will further assess the feasibility of using more fine-grained categories.

AuthorsOpenReview ↗

Response to Reviewer 8D4U Comments (part2)

Scanner Distribution and Benchmark Scope{\color{#24577a}\text{Scanner Distribution and Benchmark Scope}} (W3){\color{#247a78}\text{(W3)}}:

The dataset includes MRI scans acquired from three manufacturers and four scaner models, we will rename the subsection "Scanner Distribution", and retain the cross-tabulation of scanner models and pathological diagnoses in Table 7.

The scanner information is intended to help users understand the dataset's provenance and scope of applicability, while the current cross-validation results support benchmark comparisons within the cohort.

Prior Refinement and Background Suppression{\color{#24577a}\text{Prior Refinement and Background Suppression}} (Q3,W4){\color{#247a78}\text{(Q3,W4)}}:

1)During the complete inference process of Block 2, the Union operation preserves the full candidate region, which is subsequently subjected to MLP-based probability weighting. In the binary candidate mask, both the lesion and erroneously included background voxels are assigned a value of 1. After weighting, the prior value at each voxel becomes the tumour probability predicted by the MLP. Consequently, regions within the coarse prior that are identified by the MLP as likely background receive lower prior responses. Here, "suppression" refers to reducing the prior intensity at these locations rather than removing voxels. We will clarify this distinction in the revised manuscript.

2)The Refined Prior serves as a spatial prior for classification rather than a segmentation mask of the tumour core. Its candidate region may include some peritumoural tissue, with probability weighting assigning different prior intensities to different locations. The resulting prior is then concatenated with the MRI for classification.

3)Based on the results in the existing Table 4, the Refined Prior improves mean accuracy by approximately 2.35 percentage points compared with the Coarse Prior. The present clarification makes an existing step explicit in the methodological description; it does not introduce a new refinement operation or change the pipeline used to obtain the reported results.

Annotation Levels and Segmentation Coverage{\color{#24577a}\text{Annotation Levels and Segmentation Coverage}} (W5){\color{#247a78}\text{(W5)}}:

The dataset provides three distinct levels of information:

●249 cases with biopsy-confirmed pathological classification labels;

●249 cases with TSP-generated spatial priors;

●50 cases with expert voxel-level delineations of the tumour core as reference annotations.

These three levels of information support case-level classification, spatial prior research, and segmentation evaluation on the manually annotated subset, respectively. We will explicitly report the coverage of each annotation level in Table 1 and specify the segmentation task as the "50-case expert-annotated subset", enabling users to accurately identify the appropriate training signals and evaluation references.

Corrections to Reported Accuracy Gains{\color{#24577a}\text{Corrections to Reported Accuracy Gains}} (Q4,W6){\color{#247a78}\text{(Q4,W6)}}:

We thank the reviewer for their careful examination. We will correct all typographical errors in the revised manuscript. These corrections do not affect the conclusions supported by our findings.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 8D4U,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Reviewer_Vti1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce DeepSite-MRI, a dataset of 249 biopsy-confirmed cases of deep-seated intracranial tumors spanning glioma, primary central nervous system lymphoma, germinoma, and other rare pathologies. I consider the dataset a foundational contribution. The authors also propose TSP, a two-stage annotation pipeline that combines VLM-based coarse localization with patient-specific MLP refinement to generate spatial priors, substantially reducing annotation time. Evaluations of 10 mainstream architectures, including CNNs and Transformers, demonstrate the dataset’s potential to support research on deep-seated intracranial tumors.

Critical strengths

Building and publicly releasing a dataset requires substantial effort and provides significant research value. DeepSite-MRI and its biopsy-confirmed labels are valuable resources for advancing this field, and I appreciate this contribution. The experimental coverage is comprehensive. The authors compare 10 architectures under three conditions, without priors, with coarse priors, and with refined priors, using five-fold cross-validation and varying training set sizes to assess model performance.

Critical weaknesses

My support for acceptance is conditional on the authors ensuring that the dataset will actually be publicly released. This is my primary concern. Additionally, the reported 2.50 percentage-point improvement for ResNet in Appendix E may be a typographical error, as Table 4 indicates an improvement of 7.34 percentage points. Please verify this discrepancy.

Questions for authors
  1. Can the authors commit to publicly releasing the dataset?

  2. How was the reduction in annotation time from 90 to 3 minutes per case measured? Were both approaches evaluated using the same annotation objectives and quality standards? Does the 3-minute estimate include preprocessing, model execution, expert review, and any necessary corrections? Please clarify these details in the paper.

  3. The expert evaluation reports that 48 of 50 annotations were acceptable without further correction, while seven cases received an Accuracy Score (AS) below 4. What were the Clinical Acceptability (CA) ratings for these seven cases? Please explain how the two assessments relate.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Vti1 Comments

We thank you for recognising the value of the DeepSite-MRI dataset and the completeness of our experimental evaluation. Public release is essential to realising the contributions of this work. We therefore explicitly reaffirm our commitment to making the dataset publicly available and provide further clarification regarding the scope of the timing measurements, expert evaluation, and numerical corrections.

Commitment to Public Data Release{\color{#24577a}\text{Commitment to Public Data Release}} (Q1){\color{#247a78}\text{(Q1)}}:

We explicitly commit to publicly releasing the current version of DeepSite-MRI, comprising 249 cases, together with the accompanying code upon acceptance of the paper. We will also provide dataset documentation, details of annotation coverage, and the evaluation settings required to reproduce our experiments, enabling the research community to conduct independent studies and compare methods.

Meanwhile, we are continuing to collect additional cases towards a future version comprising over 1,000 cases, which we also plan to release publicly. The release of the current version and its subsequent expansion will proceed independently. Our long-term goal is to establish a public data resource that can continuously support research on deep-seated intracranial tumours.

Annotation Time Measurement and Comparability{\color{#24577a}\text{Annotation Time Measurement and Comparability}} (Q2){\color{#247a78}\text{(Q2)}}:

1)The approximately 3-minute duration refers to the automated processing time for a single raw MRI scan with TSP deployed on a single RTX 4090 GPU, covering preprocessing, model inference, and spatial prior generation. It excludes subsequent expert review and manual corrections. The 90-minute duration for detailed manual delineation was measured directly.

2)The comparison of these two durations is based on essentially the same annotation objectives and quality standards. However, we do not require the spatial priors generated by TSP to match the tumor contours exactly.

Relationship Between AS and CA{\color{#24577a}\text{Relationship Between AS and CA}} (Q3){\color{#247a78}\text{(Q3)}}:

AS assesses the quality of spatial localisation, whereas CA evaluates whether the spatial prior can be used directly for its intended research purpose without manual modification. The case-level AS is calculated as the mean score assigned by three experts, while CA is determined by a majority vote of their independent assessments. Therefore, a mean AS below 4 and CA = Yes can occur simultaneously: a mean score below the threshold does not necessarily indicate that the majority of experts consider manual modification necessary.

Specifically, among the seven cases with AS < 4, five were rated CA = Yes by the majority of experts, while the remaining two were rated CA = No. All other 43 cases received CA = Yes. Thus, the overall acceptance rate of 48/50 cases is consistent with the seven cases having AS < 4, as detailed below:

Case GroupCA = YesCA = NoTotal
AS < 4527
AS ≥ 443043
Total48250

We will provide the individual expert scores for these seven cases and the cross-tabulation above, and explicitly clarify how both metrics are aggregated in the main text and Appendix C. CA = Yes will continue to strictly indicate that the spatial prior can be used for its specified research purpose without further manual modification. These additions are intended to improve the transparency and verifiability of the evaluation results, while the evaluation criteria remain unchanged.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer Vti1,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

The issues I am concerned about have been emphasized by the author in their response. I have no further comments. Publicly releasing such a dataset is a very meaningful work, and I appreciate it. I have already improved my score and wish you good luck.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer Vti1,

Thank you for your careful review and for taking the time to consider our responses. We appreciate your recognition of the dataset’s value and your support for our work. We remain committed to publicly releasing the dataset and accompanying code upon acceptance to support future research.

Regards from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.