acceptodds
Under review as a conference paper at ICLR 2027

Super-Generalist: Generalist–Specialist Synergy for Accurate 3D Computed Tomography Diagnosis and Lesion Grounding across Diverse Diseases

Abstract

Vision–language models offer a promising generalist approach to medical diagnosis, providing broad disease coverage and zero-shot diagnostic capabilities. In 3D computed tomography (CT), however, these generalist models often lack fine-grained anatomical and lesion awareness for reliable diagnosis and spatial interpretability. In contrast, supervised specialist models achieve strong performance on specific tasks but typically lack generalization across diseases and anatomies. In this work, we present SuG, a Super-Generalist framework for 3D CT diagnosis that unifies generalist vision–language learning with specialist objectives, enabling zero-shot multi-disease diagnosis and lesion grounding alongside specialist-competitive diagnostic performance. We perform specialist-enhanced vision-language alignment in SuG by incorporating spatial priors from multiple segmentation experts, including anatomy, class-specific lesion and class-agnostic lesion segmentors that capture lesions beyond anatomies annotated during training. To improve lesion grounding capability, we leverage lesion masks as spatial priors to calibrate text-conditioned visual attention, encouraging disease-related semantics to focus on clinically relevant regions. We evaluate SuG on extensive chest and abdominal CT benchmarks, including CT-RATE, Merlin, MedVL-CT69K, and several in-house tumor datasets. SuG achieves state-of-the-art performance across a wide range of disease diagnosis tasks and surpasses specialist models on several critical tumor diagnosis benchmarks. Furthermore, SuG demonstrates strong lesion grounding capability, including robust generalization to lesion types lacking class-specific supervision. Code and models will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical vision-language learning, 3D computed tomography, zero-shot disease diagnosis, lesion grounding

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_dejCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SuG uses anatomy and lesion segmentation labels together with medical reports to train a model for CT diagnosis and lesion localization. It combines anatomy features at different scales and uses lesion labels to guide attention. The experiments cover chest and abdominal diseases, lesion localization, and performance under different training and testing conditions.

Critical strengths
  • The main idea is well motivated. Anatomy labels and lesion labels give the model useful information at different levels. The full model improves lesion localization over the specialist-only variant. It also performs better on an external kidney dataset without using kidney tumor or cyst segmentation labels during training. Together with the diagnosis results, this supports the value of combining specialist knowledge in a broader model.
  • The evaluation covers many practical settings. The authors test joint chest and abdomen training, reduced segmentation labels, changes to report parsing, corrupted reports, and report generation. These tests give useful evidence that the method works under different conditions, even though they do not explain every interaction between its parts.
Critical weaknesses
  • Some reported results are inconsistent. The pancreatic and gastric cancer results differ between the lesion diagnosis tables and the detailed disease results in the appendix, even though the category mapping appears to match. The authors should explain whether the prompts, test cases, or model checkpoints differ. The liver category also mixes cysts with different types of tumors, so its match to the specialist's training labels needs clarification. These issues make some comparisons harder to interpret, but they do not by themselves invalidate the broader diagnosis and localization results.
  • Some claims go beyond the comparisons shown. The statistical tests use VLWS even where OpenVocabCT or nnUNet-fuse reports stronger results in the relevant comparison. Those tests therefore do not establish an improvement over the strongest alternatives. Also, showing that attention overlaps a lesion supports localization, but does not show that the model's diagnostic reasoning is reliable.
Questions for authors
  • Why do the pancreatic and gastric cancer results differ across the lesion diagnosis and detailed disease tables? Does the liver specialist use the same disease definitions as the combined liver category?
  • Could you compare against the strongest relevant baselines in the statistical tests, especially nnUNet-fuse for the abdominal lesion comparison? Otherwise, please limit the claims to the comparisons that were actually tested.
3 comments
AuthorsOpenReview ↗

Response to Reviewer dejC (Part 1 of 3)

We appreciate your recognition of our motivation and comprehensive evaluation. We also thank the reviewer for the valuable suggestions, which have helped improve the quality of our manuscript. Here we address your concerns point by point below.

1. Concerns regarding inconsistent results in appendix (W1, Q1)

Thank you for pointing this out. The four-category lesion evaluation in Tables 5/A26 and the per-disease evaluation in Table A27 follow different task definitions: the former focuses on organ-specific tumor categories, whereas the latter evaluates individual diseases. For the corresponding pairs of "pancreatic tumor" and "pancreatic cancer", as well as "gastric tumor" and "gastric cancer", on MedVL-CT69K, we use the same SuG model checkpoint, test cases, and ground-truth labels. The apparent differences in the reported results are attributable to the different task-specific text prompts.

Task definitions and evaluation consistency. The categories in the four-category lesion diagnosis task can encompass multiple pathological subtypes rather than being restricted to a single cancer type. For example, the pancreatic tumor annotations used for specialist pretraining also cover pancreatic neuroendocrine tumors (PNETs) and solid pseudopapillary tumors (SPTs). Therefore, the "pancreatic tumor" category is broader than the specific diagnosis of "pancreatic cancer". We maintain a consistent prompting strategy for the four-category lesion diagnosis task across LesionSegAbdomen and MedVL-CT69K. Although the pancreatic and gastric tumor categories in MedVL-CT69K happen to correspond exclusively to pancreatic cancer and gastric cancer, respectively, we do not modify the task-specific prompts accordingly. All competing vision–language methods follow the same evaluation protocol within each task to ensure fair comparisons.

Task-specific prompts. The specific positive and negative prompts are provided below (Table R5). The prompts used in Table A27 additionally incorporate imaging descriptions or disease-related terms, making them different from those used in Tables 5/A26. Consequently, the same test cases receive different prediction scores, leading to differences in the reported diagnostic metrics.

Coverage of liver specialist annotations. Regarding the liver tumor category, the lesion annotations used for specialist training cover multiple pathological subtypes rather than being limited to a single cancer type. The combined liver tumor category in MedVL-CT69K includes liver cysts, liver cancer, and liver metastases, all of which are covered by the specialist training labels. Moreover, the specialist training data also include other liver lesions, such as hemangiomas and focal nodular hyperplasia (FNH).

For clarity, we have added a brief explanation in the revised Appendix C.4.1 of the task-specific prompting protocols and the correspondence between the combined liver-category labels and the specialist training annotations.

Evaluation TaskTablePositive PromptNegative Prompt
Pancreatic tumor5/A26Pancreatic tumor or pancreatic cancer.No tumor is observed in the pancreas.
Pancreatic cancerA27Space-occupying lesion, pancreatic cancer.No pancreatic cancer is observed.
Gastric tumor5/A26Gastric cancer, gastric tumor.No tumor is observed in the stomach.
Gastric cancerA27Gastric wall thickening, Gastric cancer.No gastric cancer is observed.

Table R5: Task-specific prompts for pancreatic and gastric lesion diagnosis.

AuthorsOpenReview ↗

Response to Reviewer dejC (Part 2 of 3)

2. Concerns regarding statistical significance analysis (W2, Q2)

Thank you for pointing this out. We have now included direct statistical comparisons against the strongest relevant baselines, particularly nnUNet-fuse for the abdominal lesion diagnosis task, as highlighted by the reviewer. We have carefully re-examined the baseline selection in Tables A18–A24 and recalculated the corresponding statistical significance results.

Baseline selection criteria and selected comparators. In the revised analysis, we select the strongest non-SuG baseline based on the highest average performance under each dataset and evaluation setting, and consistently compare all metrics within each table against this baseline. Specifically, the selected baselines are VLWS for A18, HCFNet for A19, OpenVocabCT for A20, HCFNet for A21 and A22, OpenVocabCT for A23, and nnUNet-fuse for A24. Among these, the baseline selections for Tables A20, A23, and A24 differ from those in the original manuscript. The revised statistical significance results for these three tables are presented below.

Results. These revised results demonstrate that SuG continues to achieve statistically significant improvements on the majority of metrics when compared against the strongest baselines under the corresponding evaluation settings. In particular, on LesionSegAbdomen, SuG achieves statistically significant improvements over nnUNet-fuse in average AUC and average specificity, while the difference in average sensitivity does not reach statistical significance. We have updated the corresponding statistical significance results in revised Appendix Tables A20, A23 and A24 to reflect these comparisons.

MetricOpenVocabCTSuG (Improvement)p-value
AUC83.285.1 (+1.89)***
F159.360.7 (+1.42)**

Table A20 (updated): Bootstrap significance analysis on Merlin for unbalanced metrics.

MetricOpenVocabCTSuG (Improvement)p-value
AUC89.793.7 (+3.99)***
Sensitivity83.486.8 (+3.38)*
Specificity84.288.0 (+3.74)***

Table A23 (updated): Bootstrap significance analysis on MedVL-CT69K for the four tumor categories.

MetricnnUNet-fuseSuG (Improvement)p-value
AUC94.796.1 (+1.41)*
Sensitivity89.289.4 (+0.20)
Specificity89.091.7 (+2.71)***

Table A24 (updated): Bootstrap significance analysis on LesionSegAbdomen.

AuthorsOpenReview ↗

Response to Reviewer dejC (Part 3 of 3)

3. Concern regarding the interpretation of lesion grounding (W2)

We thank the reviewer for this important clarification. We agree that spatial overlap between attention maps and lesion masks, while providing evidence of lesion localization, does not by itself establish causal faithfulness or the reliability of the model’s diagnostic reasoning.

Our grounding evaluation is intended to assess whether text-conditioned attention maps are spatially aligned with lesion annotations. The lesion-guided calibration loss explicitly supervises these maps using lesion masks, and the consistent improvements in AUC, AUPR, and Cohen’s d in Table 6 demonstrate improved lesion-focused localization and spatial interpretability. These grounding results are complementary to the diagnostic classification results, rather than evidence that the final diagnostic decision is causally determined by the attended lesion region.

For clarity, we have revised potentially overstrong wording in the manuscript. For example, we have replaced “trustworthy/reliable/faithful grounding” and “diagnostic traceability” with “improved lesion localization” and “spatial interpretability.” Establishing the causal faithfulness of localized evidence to diagnostic decisions is an important direction for future work.

Reviewer_E61fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Super-Generalist (SuG), a unified framework for 3D CT diagnosis and lesion grounding. It combines specialist-enhanced vision-language learning with lesion-guided attention calibration to improve broad disease recognition and fine-grained lesion understanding. The paper evaluates SuG on multiple chest and abdominal CT benchmarks, including general disease diagnosis, tumor-centric diagnosis, and lesion grounding. The experimental results are promising overall, although several claims and statistical reporting details require clarification.

Critical strengths
  1. SuG addresses the limited diagnostic coverage of specialist models and the insufficient capability of generalist medical VLMs by integrating specialist-enhanced vision-language learning with lesion-guided attention calibration in a unified framework.

  2. The paper provides comprehensive evaluation across diverse diagnostic tasks and datasets, covering general disease diagnosis, fine-grained diagnosis, and lesion grounding. This broad evaluation provides a relatively comprehensive assessment of the proposed framework beyond a single dataset or task, with both internal and external validation.

Critical weaknesses

Please identify one of two critical weakness of the paper that support rejections. For each, explain briefly why it is consequential to your overall assessment. The goal is not to enumerate every negative point, but to identify and substantiate the considerations that most strongly determine your recommendation. Enter "None" if no material weakness can be identified.

  1. Some performance claims are inconsistent with the reported results. In Section 3.3, the authors state that “SuG achieves superior performance across all categories.” However, according to Table 5, SuG does not outperform OpenVocabCT on the Pancreas category. This result appears inconsistent with the stated claim.

  2. The baseline selection in the statistical significance analysis appears incorrect for some comparisons. In Appendix G, “Statistical Significance Analysis”, the authors state that “Overall, the improvements of SuG over the second-best method are statistically significant on the vast majority of metrics.” However, in Table 5, OpenVocabCT achieves an AUC of 89.7 on MedVL-CT69K, exceeding the 88.1 reported for VLWS. Nevertheless, Table A23 compares SuG against VLWS rather than the OpenVocabCT. A similar baseline-selection issue appears in Table A24. The authors should verify these comparisons and report significance tests against the appropriate second-best baseline.

  3. Issues on Precision and F1. Tables A28 and A29 report per-disease Precision, ACC, F1, and AUC for CT-RATE and RAD-ChestCT. Under the standard definition, F1=2PR/(P+R). Since recall cannot exceed 1, the F1 score must satisfy F1 < 2P/(1+P)). However, several reported per-disease F1 scores exceed this upper bound given the corresponding Precision values. The authors should explicitly specify the computation and averaging conventions used for Precision and F1, and verify the affected entries if these metrics are intended to follow the standard definitions.

Questions for authors

none

2 comments
AuthorsOpenReview ↗

Response to Reviewer E61f (Part 1 of 2)

Thank you for recognizing the significance of our work and the comprehensiveness of our evaluation. We appreciate your constructive feedback, which has helped us improve the rigor of our manuscript. Here we address your concerns point by point below.

1. Concerns regarding inconsistent performance claim in Table 5 (W1)

We apologize for the imprecise statement. As you correctly noted, SuG does not achieve the best performance on the Pancreas category of MedVL-CT69K. We have revised the text in the updated manuscript to: “SuG achieves the best average performance and outperforms prior methods in most categories.”

2. Concerns regarding statistical significance analysis (W2)

Thank you for pointing this out. The reviewer is correct regarding Tables A23 and A24: the original statistical comparisons against VLWS do not establish statistically significant improvements over the strongest baselines for the corresponding tasks. We have carefully re-examined the baseline selection in Tables A18–A24 and recalculated the corresponding statistical significance results.

Baseline selection criteria and selected comparators. In the revised analysis, we select the strongest non-SuG baseline based on the highest average performance under each dataset and evaluation setting, and consistently compare all metrics within each table against this baseline. Specifically, the selected baselines are VLWS for A18, HCFNet for A19, OpenVocabCT for A20, HCFNet for A21 and A22, OpenVocabCT for A23, and nnUNet-fuse for A24. Among these, the baseline selections for Tables A20, A23, and A24 differ from those in the original manuscript. The revised statistical significance results for these three tables are presented below.

Results. These revised results demonstrate that SuG continues to achieve statistically significant improvements on the majority of metrics when compared against the strongest baselines under the corresponding evaluation settings. The only exception is sensitivity on LesionSegAbdomen, for which the improvement over nnUNet-fuse is not statistically significant. We have updated the corresponding statistical significance results in Appendix Table A20, A23 and A24 to reflect these comparisons.

MetricOpenVocabCTSuG (Improvement)p-value
AUC83.285.1 (+1.89)***
F159.360.7 (+1.42)**

Table A20 (updated): Bootstrap significance analysis on Merlin for unbalanced metrics.

MetricOpenVocabCTSuG (Improvement)p-value
AUC89.793.7 (+3.99)***
Sensitivity83.486.8 (+3.38)*
Specificity84.288.0 (+3.74)***

Table A23 (updated): Bootstrap significance analysis on MedVL-CT69K for the four tumor categories.

MetricnnUNet-fuseSuG (Improvement)p-value
AUC94.796.1 (+1.41)*
Sensitivity89.289.4 (+0.20)
Specificity89.091.7 (+2.71)***

Table A24 (updated): Bootstrap significance analysis on LesionSegAbdomen.

AuthorsOpenReview ↗

Response to Reviewer E61f (Part 2 of 2)

3. Concerns regarding Precision and F1 (W3)

Thank you for identifying this issue. Following the evaluation protocol adopted in fVLM [R1], the Precision reported in Tables A28 and A29 refers to the precision of the positive disease class, whereas the F1 score is the Weighted F1, computed as the support-weighted average of the F1 scores for the positive and negative classes. Specifically, we use precision_score(y, pred) and f1_score(y, pred, average="weighted") from sklearn.metrics, respectively. The metrics are defined as follows:

P+=TPTP+FP,F1+=2TP2TP+FP+FN,F1−=2TN2TN+FP+FN.P_+ = \frac{TP}{TP+FP}, \qquad F1_+ = \frac{2TP}{2TP+FP+FN}, \qquad F1_- = \frac{2TN}{2TN+FP+FN}.

F1weighted=n+F1++n−F1−n++n−,n+=TP+FN,n−=TN+FP.F1_{\mathrm{weighted}} = \frac{n_+F1_+ + n_-F1_-}{n_+ + n_-}, \qquad n_+ = TP+FN, \qquad n_- = TN+FP.

Here, TP, FP, TN, and FN denote the numbers of true positives, false positives, true negatives, and false negatives, respectively, while n+n_+ and n−n_- represent the numbers of ground-truth positive and negative samples. Thus, the Weighted F1 is the average of the positive-class and negative-class F1 scores, weighted by their respective sample proportions.

The upper bound highlighted in the review, F1+≤2P+/(1+P+)F1_+ \leq 2P_+/(1+P_+), applies when Precision and F1 are computed for the same positive class and therefore does not directly constrain the Weighted F1 reported in our tables. We acknowledge that the original manuscript did not sufficiently clarify this distinction. To address this issue, we have explicitly relabeled the corresponding table columns as "Weighted F1" in updated Table 4 and Appendix Tables A28 and A29, provided the metric definitions and the reference to fVLM, and verified the affected entries to ensure consistency with the stated evaluation protocol.

[R1] Large-scale and fine-grained vision-language pre-training for enhanced ct image understanding, ICLR 2025

Reviewer_Nu8uOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SuG, a Super-Generalist framework capable of diagnosing multiple diseases from CT scans. Through specialist-enhanced vision–language alignment, which aligns text with image features guided by multiple segmentor, SuG improves diagnostic performance and enables zero-shot diagnosis. By calibrating text-conditioned attention with lesion masks, it also achieves precise lesion localization. This framework achieves leading results on multiple CT datasets, demonstrating the effectiveness of the proposed method.

Critical strengths
  • The idea of combining generalist and specialist model is well motivated. SuG keeps the broad coverage and zero-shot ability of CLIP-style models while gaining the lesion-level precision of specialist.
  • The three-stage training strategy is sound. The segmentation, contrastive and calibration losses complement each other well.
  • SuG achieves the best average results on multiple CT benchmarks, which demonstrates the effectiveness of the method.
  • Pooling organ-specific tokens with predicted masks and normalizing across scales is effective, and the ablation clearly supports it.
Critical weaknesses
  • The paper does not specify the model structure of encoder, decoder, segmentor model structure and size, and use pretrained weights or not.
  • The inference pipeline for different diagnostic tasks is not clearly explained. The method part only focuses on training.
Questions for authors
  • What architectures are used for the encoder, decoder and segmentor? Are they initialized from pretrained weights?
  • Could the authors describe the inference pipeline for different diagnostic tasks? Which components are used at inference time?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Nu8u

Thank you for recognizing the soundness and effectiveness of our work, as well as for your constructive comments and suggestions. Here we address your concerns point by point below.

1. Question about architecture details (W1, Q1)

Thank you for your question. Below, we provide the architectural specifications, model sizes, and weight-initialization details of SuG.

Visual encoder and decoder. SuG employs an nnU-Net-style 3D U-Net for the abdominal visual branch, with six encoder stages comprising 32, 64, 128, 256, 320, and 320 channels, respectively. The decoder progressively restores spatial resolution through skip connections.

Segmentation heads. The three segmentation tasks share the same encoder and decoder in both the abdominal and chest settings, with task-specific segmentation heads. For the abdominal setting, the three segmentation heads produce 37 channels for organ segmentation, 5 channels for class-specific lesion segmentation, and 2 channels for class-agnostic lesion segmentation, all including the background class. These outputs correspond to 36 anatomical structures, four lesion categories (colorectal, gastric, liver, and pancreatic lesions), and class-agnostic lesion foreground, respectively. For the chest setting, the organ segmentation head produces 5 channels (4 anatomical structures plus background), while the class-specific (lung tumor) and class-agnostic lesion segmentation heads each produce 2 channels (foreground and background).

Text encoder. We adopt a BERT-base architecture with 12 transformer layers, a hidden dimension of 768, and 12 attention heads. The [CLS] token is further projected into a 256-dimensional embedding for image–text matching.

Model size. SuG adopts the same overall architecture across MedVL-CT69K, CT-RATE, and Merlin, with dataset-specific variations in the number of segmentation output channels and text vocabulary size. Additionally, Merlin incorporates global image–text projection layers. These differences result in total parameter counts of 134.01M, 141.21M, and 141.51M for MedVL-CT69K, CT-RATE, and Merlin, respectively.

Weight initialization. Stage 1 trains the specialist visual branch from randomly initialized weights. In Stage 2, we initialize the visual branch from the Stage 1 checkpoint, while the text encoder is initialized from different pretrained BERT checkpoints depending on the dataset: bert-base-chinese for MedVL-CT69K and bert-base-uncased for Merlin and CT-RATE. Finally, in Stage 3, we initialize the model from Stage 2 checkpoint.

We have added these architectural and initialization details to Section C.2.2 in the revised Appendix.

2. Question about inference details (W2, Q2)

Thank you for pointing this out. In Section 3.1, we have already described the computation of diagnostic scores and localization heatmaps, while Appendix D.2 has already outlined the organ-level and global inference modes for Merlin. For clarity, we provide a more comprehensive explanation below.

Diagnosis across diverse diseases. For organ-related diseases, the CT volume is first processed by the visual encoder and segmentation decoder to obtain multi-scale features and predicted organ masks. The predicted masks are then used to extract and aggregate features from the corresponding organs. We compute the cosine similarities between each organ-level visual representation and the positive and negative disease prompt embeddings. The resulting similarities are normalized with a softmax function to obtain diagnostic scores. Both general disease diagnosis and diagnosis of the four lesion categories follow this pipeline, differing only in the queried disease prompts and corresponding organs. For diseases in Merlin that are not associated with a single organ, global visual representations are instead matched with the corresponding disease prompts.

Lesion grounding across diverse diseases. For lesion grounding, localization heatmaps are generated by computing the cosine similarity between positive disease prompt embeddings and spatial visual tokens.

Components used at inference time. For disease diagnosis and lesion grounding, SuG uses the visual encoder, segmentation decoder, organ-segmentation head, and text encoder.

We have consolidated these details in Section C.2.4 in the revised Appendix and explicitly specify the components used for each task to facilitate understanding and reproducibility.

Thanks for the response of the author. This has clearly addressed my concerns. I now fully understand the model architecture, initialization scheme, and how inference operates across different tasks.

Reviewer_4t5qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper integrates anatomical and lesion-related information, learned by specialized segmentation models, into a general-purpose vision-language model. The goal is to enable the model not only to identify the presence of diseases in CT scans but also to pinpoint the exact locations of lesions with greater accuracy. To achieve this, the authors employ techniques such as segmentation-based supervision, the alignment of anatomical regions with text, and the use of lesion masks to guide the model's attention. Experimental results demonstrate significant performance improvements in the diagnosis of various diseases and specific types of tumors; furthermore, the model’s capability was validated through transfer learning on lesions for which no corresponding category-specific segmentation annotations were available.

Critical strengths
  1. The research addresses a problem of practical significance, and the overall approach is sound. The authors note that while general-purpose vision-language models can identify many diseases, they often struggle to pinpoint the exact location of lesions; conversely, specialized segmentation models offer strong spatial awareness but are constrained by the need for disease-specific annotations. The authors propose a method that integrates the strengths of both.
  2. The experimental evaluation is comprehensive and reflects a substantial body of work. The authors tested their approach not only on abdominal CT scans but also on chest CT scans and external datasets, covering tasks such as multi-disease diagnosis, tumor diagnosis, and lesion localization. These results demonstrate that the method is effective beyond a single specific task.
Critical weaknesses
  1. The ablation study is rather simplistic and fails to provide a clear explanation for why each mechanism is effective. The authors primarily employ a cumulative, step-by-step approach to adding modules. While this demonstrates that performance shifts upon the inclusion of new components, it does not adequately elucidate the individual contributions of these components or the nature of their interrelationships. In particular, there is a lack of targeted comparison to determine whether class-specific and class-agnostic attention are truly complementary, or whether mask supervision conditioned on text is superior to standard mask supervision.

  2. In the actual training process described in the paper, the segmentation losses associated with the specialist models are incorporated into Stages 2 and 3. Consequently, the current experiments cannot definitively distinguish whether the generalist model has genuinely internalized the knowledge acquired by the specialist during its pre-training phase, or if the observed performance gains are simply a result of the model continuously receiving additional voxel-level segmentation supervision.

Questions for authors
  1. Could the authors further verify the independent contributions of each mechanism and their interrelationships through more targeted controlled experiments?

  2. How do the authors distinguish whether the final performance gains stem from the transfer of specialist knowledge acquired during Stage 1, or from the voxel-level segmentation supervision introduced in subsequent stages?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4t5q (Part 1 of 2)

Thanks for your insightful feedback. We appreciate the positive assessment of the soundness of our work and the substantial effort underlying it, as well as the constructive suggestions for further strengthening the manuscript. Here we address your concerns point by point below.

1. Concerns regarding the cumulative ablation design (W1, Q1)

Thank you for this constructive suggestion. We agree that the cumulative ablation in the original Table 7 does not fully isolate the contribution of each mechanism. We therefore add targeted leave-one-component-out controlled ablations. Starting from the full SuG configuration, we remove the specified component or component group and retrain every variant under the identical data split, optimization protocol, and evaluation procedure.

Class-specific and class-agnostic lesion attention. We conduct a 2×22\times2 ablation by independently enabling or disabling the two attention mechanisms (Table R1). Full SuG achieves the best performance across all evaluation settings, and adding either mechanism to the other consistently improves performance. Without class-agnostic attention, class-specific attention slightly lowers general-disease AUC (87.4 to 87.3), likely by biasing attention toward focal, predefined lesions rather than diffuse abnormalities. In contrast, class-specific attention is particularly beneficial for annotated lesion-centric diagnosis, whereas class-agnostic attention contributes more strongly to broad disease recognition and generalization to unannotated localized lesions. These results indicate distinct yet complementary benefits.

Text-conditioned lesion-mask supervision. Notably, in SuG, text-conditioned lesion-mask supervision is built upon, rather than an alternative to, standard lesion-mask supervision: the CA lesion mask predicted by the specialist decoder serves as the spatial target for learning class-agnostic attention. It therefore cannot be independently trained after removing standard supervision. We instead evaluate its additional contribution by removing only text-conditioned supervision while retaining standard lesion-mask supervision and all other settings (Table R2). This removal decreases AUC from 89.5 to 87.4 for general diseases, from 87.6 to 84.4 for annotated lesion-centric diseases, and from 89.2 to 84.8 for unannotated localized lesion diseases, demonstrating that explicit text–lesion alignment provides substantial benefits beyond standard lesion-mask supervision.

Multi-scale feature aggregation and normalization. Because ℓ2 normalization is applied to the multi-scale features, we jointly remove multi-scale feature aggregation and ℓ2 normalization from the full SuG model to evaluate the contribution of this feature-construction module (Table R3). This removal consistently reduces performance by 3.1, 8.2, and 4.1 points across the three disease settings, confirming the importance of multi-scale feature aggregation and ℓ2 normalization.

All of these newly added results have been incorporated into the revised Appendix Section H, Tables A32, A33, and A34. Thank you again for this constructive suggestion.

Class-specific lesion attentionClass-agnostic lesion attentiongeneral diseasesfour annotated lesion-centric diseasesunannotated localized lesion diseases
SuG✓✓89.587.689.2
SuG w/o Class-agnostic✓✗87.387.186.1
SuG w/o Class-specific✗✓88.585.588.2
SuG w/o Class-specific and Class-agnostic✗✗87.484.484.8

Table R1. Ablation of class-specific and class-agnostic lesion attention.

general diseasesfour annotated lesion-centric diseasesunannotated localized lesion diseases
SuG89.587.689.2
SuG w/o text-conditioned lesion mask supervision87.484.484.8

Table R2. Ablation of text-conditioned lesion-mask supervision.

general diseasesfour annotated lesion-centric diseasesunannotated localized lesion diseases
SuG89.587.689.2
SuG w/o Multi-scale feat. and ℓ2 norm86.479.485.1

Table R3. Ablation of multi-scale feature aggregation and normalization.

AuthorsOpenReview ↗

Response to Reviewer 4t5q (Part 2 of 2)

2. Concerns regarding specialist knowledge transfer versus continued segmentation supervision (W2, Q2)

Thank you for raising this point. To distinguish the effects of Stage-1 lesion-specialist pretraining and subsequent lesion segmentation supervision, we compare full SuG with two variants: one without Stage-1 pretraining and one without Stage-2/3 lesion segmentation supervision (Table R4). For the latter, we freeze the Stage-1 specialist checkpoint as a teacher to generate stable class-agnostic (CA) lesion masks for class-agnostic attention, while training the Stage-2/3 model without lesion segmentation losses. This is necessary because, as the vision encoder continues to evolve under vision-language learning, the pretrained decoder would otherwise produce unreliable CA masks. We do not evaluate a variant without lesion supervision in all stages, since Stage-3 text-conditioned supervision requires meaningful CA masks from a trained lesion specialist.

Table R4 shows that removing Stage-2/3 lesion segmentation supervision reduces AUC from 89.5/87.6/89.2 to 88.2/87.5/86.4, while removing Stage-1 pretraining reduces it to 86.3/87.3/83.5. Thus, both Stage-1 lesion-specialist pretraining and the continued Stage-2/3 lesion segmentation supervision contribute to final performance; Stage-1 pretraining is particularly important for broad diagnosis and generalization to unannotated lesions. We have added this ablation result to the revised Appendix Section H, Table A35.

general diseasesfour annotated lesion-centric diseasesunannotated localized lesion diseases
SuG89.587.689.2
SuG w/o Stage-2/3 lesion segmentation supervision88.287.586.4
SuG w/o Stage-1 lesion-specialist pretraining86.387.383.5

Table R4. Ablation of lesion-specialist pretraining and subsequent lesion segmentation supervision.

I appreciate the author’s response; however, it does not address my primary concern. The core issue of knowledge transfer remains insufficiently validated. While the additional experiments demonstrate the contribution of "Stage 1" expert pre-training to the final model's performance, there is a lack of direct empirical evidence confirming that the generalist model has truly acquired and retained the knowledge from the expert models, or that this knowledge can function independently in downstream tasks. Furthermore, I remain concerned about the reproducibility of this work, as several implementation details regarding the interactions between various components are not clearly specified. Specifically, the paper lacks a clear description of the dimensional alignment between multi-scale visual features and text embeddings; the selection and mapping of "class-specific" versus "class-agnostic" lesion masks used for attention supervision; and the joint optimization process that integrates specialist and generalist datasets. These ambiguities make it extremely difficult to independently replicate the exact model architecture and training pipeline. Consequently, I will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviewer feedback and our revisions

We thank the reviewers for their insightful and constructive feedback, which has greatly helped us improve the quality of our manuscript.

We are pleased that reviewers found our work:

  • Solid Motivation and Significance: Reviewers found that SuG “addresses a problem of practical significance” and “integrates the strengths of both” generalist and specialist models (Reviewer 4t5q). They further described the idea as “well motivated” (Reviewer Nu8u, dejC) and noted that it addresses the limited coverage of specialist models and capability of generalist medical VLMs (Reviewer E61f).
  • Sound Technical Design: Reviewers considered the approach “sound” (Reviewer 4t5q) and the three-stage strategy sound, with segmentation, contrastive, and calibration losses that “complement each other well” (Reviewer Nu8u). They also found the organ-token pooling and multi-scale normalization effective (Reviewer Nu8u).
  • Comprehensive Evaluation across Diverse Settings: Reviewers highlighted the “comprehensive” evaluation across abdominal and chest CT, as well as external datasets (Reviewer 4t5q), diverse tasks and datasets (Reviewer E61f), and “many practical settings” (Reviewer dejC).
  • Strong Empirical Effectiveness and Generalizability: Reviewers found that SuG is effective “beyond a single specific task” (Reviewer 4t5q) and “achieves the best average results on multiple CT benchmarks” (Reviewer Nu8u).

Meanwhile, we have provided additional experiments and clarifications to address the following primary concerns:

  • Component-wise Ablation (Reviewer 4t5q): We added leave-one-component-out and 2 × 2 attention ablations (Tables R1–R3), showing that class-specific attention, class-agnostic attention, text-conditioned lesion-mask supervision, and multi-scale feature aggregation each make distinct and complementary contributions.
  • Specialist Pretraining and Knowledge Integration (Reviewer 4t5q): We isolated the contributions of Stage-1 pretraining and subsequent segmentation supervision (Table R4). In the follow-up discussion, Reviewer 4t5q acknowledged the performance benefits of Stage-1 pretraining. We further demonstrated its contribution to grounding annotated and unannotated lesions (Tables R6–R7) and clarified that SuG does not claim independent knowledge retention without specialist components.
  • Architecture and Inference Specification (Reviewer Nu8u): We clarified architecture, initialization, and inference details (Appendix C.2.2–C.2.4). In the follow-up discussion, Reviewer Nu8u confirmed that our response “has clearly addressed my concerns.” And we further specified feature alignment, mask supervision, and joint optimization in response to Reviewer 4t5q.
  • Claim Precision and Statistical Comparison (Reviewers E61f, dejC): We revised the overstrong performance claim and recomputed significance tests against the strongest task-specific baselines (Tables A20, A23–A24), confirming significant improvements on most relevant metrics.
  • Metric Definition and Evaluation Consistency (Reviewers E61f, dejC): We clarified weighted-F1 computation, task-specific prompts, and label definitions (Table 4; Tables A28–A29; Appendix C.4.1), resolving the apparent metric and cross-table inconsistencies.
  • Interpretation of Lesion Grounding (Reviewer dejC): We revised the grounding claims to distinguish improved localization and spatial interpretability from causal diagnostic faithfulness.

We have revised the manuscript accordingly, and all changes are highlighted in blue. If you have any further questions or concerns, we would be glad to discuss them. We would also be grateful if you could kindly consider updating your score in light of our responses and the additional results. Updates reflecting the follow-up discussion are marked in italics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.