- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether an inexpensive probe of a frozen 3D CT encoder can predict performance after full vision-language fine-tuning. I understand the practical motivation behind reducing the cost of encoder selection, and the paper is reasonably complete in terms of implementation and experiments. The introduction of MeasureVQA, which evaluates size, density, location, texture, and disease separately, is also a contribution worth acknowledging.
However, I do not find the paper convincing overall. The writing, particularly in the abstract and introduction, reads as immature and insufficiently professional, with repeated claims that are stated more strongly than the evidence supports. The organization is also problematic, and multiple essential appendices referenced in the main text are absent from the main PDF.
More importantly, the paper does not provide a sufficiently reasonable justification for focusing on representations before multimodal interaction with the LLM. The observed differences between encoders may reflect their specific compatibility with the fixed aggregation, projector, and LLM rather than their intrinsic downstream value. The evaluation also contains strong target circularity because the probe and downstream tasks are largely constructed from the same underlying targets.
Overall, I consider this a relatively ordinary paper that, despite its reasonable level of experimental completeness, does not currently meet the ICLR acceptance standard. I would reconsider my assessment if the authors can convincingly address the two central concerns below.
Critical strengths
-
The paper addresses a practical efficiency problem and provides a reasonably complete empirical study. If a cheap probe can reliably eliminate weak encoder candidates before full VLM training, it could reduce the cost of developing 3D medical vision-language systems. The authors evaluate the idea across multiple encoder variants, language-model backbones, readout configurations, and clinical capabilities.
-
MeasureVQA is a useful capability-level evaluation resource. Separating size, density, location, texture, and disease provides more interpretable information than relying only on aggregate report-generation scores. The dataset construction, image-removal controls, and analysis of position-aware probing are worthwhile components of the paper.
Critical weaknesses
-
The motivation for focusing specifically on the image encoder is not sufficiently convincing. In an end-to-end VLM/MLLM, multimodal interaction between visual representations and the LLM may constitute a more fundamental bottleneck than the image encoder alone. The paper does not adequately explain why representations measured before this interaction should reliably predict final system performance. Moreover, differences obtained by replacing encoders within a fixed pipeline may reflect encoder-specific compatibility with the chosen aggregation method, projector, or LLM rather than intrinsic encoder quality. Therefore, the current evidence does not establish that the encoder is really a meaningful intervention point for forecasting downstream performance.
-
The probe and downstream tasks exhibit strong target circularity. MeasureVQA questions are constructed by discretizing the same mask- and Hounsfield-unit-derived quantities that the probe is directly trained to predict. Similarly, the disease probe uses report-derived disease labels that are closely aligned with those evaluated by CE-F1. A strong correlation is therefore partly expected by construction because the probe and downstream evaluation measure access to essentially the same underlying targets. This demonstrates task-specific feature accessibility, but it does not establish that CheapCT can forecast performance on genuinely distinct downstream generation or reasoning objectives.
Questions for authors
To keep the discussion focused, I have only two assessment-critical questions corresponding directly to my main weaknesses. Please see the weaknesses section.
2 comments
Reply
Your response is basically convincing. Even though I think your approach is not elegant enough, there doesn’t seem to be much room for improvement within the current methodological framework. Considering the relationship between the quality levels and score distributions of different papers, and making a comparative judgment, I think that in this case I am willing to give this paper a more positive assessment. However, I should note that I don’t think this topic belongs to a track like AI4S; there is still a certain boundary between science and medical imaging. Anyway, I have raised my score.
Thank you for acknowledging the practical motivation and MeasureVQA's capability-level evaluation.
Our contribution is a systematic empirical validation of inexpensive probing for encoder selection in 3D-CT vision-language models. Building on existing probing methods, we compare readout designs and retain token-position information for spatial attributes.
Additional encoder experiments. We added FlexiCT (May 2026), HLIP (May 2025), and ASAP (May 2026), which did not inform readout design or selection. Forecasts were fixed before the corresponding downstream training. Across 30 encoder–target–backbone combinations, 26 predicted ranks are within one position of the observed rank. Among twelve configurations, selection regret is below 0.004 in eight of ten evaluated target–backbone combinations. These results support screening additional candidates, followed by downstream confirmation. We added the full score and rank comparisons in Prediction on additional encoders.
Q1. Why are representations before multimodal interaction a meaningful selection target? Could the results reflect pipeline compatibility?
Response: Encoder choice strongly influences downstream performance in the evaluated 3D-CT pipelines. This motivates our focus on inexpensive encoder screening before multimodal training. We evaluate the utility of visual representations, including encoder and aggregation, within specified downstream pipelines. Each representation receives separately trained projector and LoRA adapters under a common protocol. Cross-backbone agreement and the additional-encoder results above support this screening use. Pipeline compatibility contributes to the utility relevant to this decision; our results do not isolate universal intrinsic encoder quality or establish the encoder as the largest bottleneck. We clarified separate projector/LoRA training in the Method's Backbones and discussed encoder selection and pipeline scope in Discussion and Limitations.
Q2. Does shared supervision make the correlation expected, and what does it establish about distinct generation or reasoning objectives?
Response: Target alignment is intentional: we ask whether relative accessibility of a clinical capability under a small readout predicts encoder rankings after multimodal training and text generation. Shared targets make a relationship plausible, but do not guarantee that ranking survives adaptation, as Location's readout sensitivity and the added-encoder prediction errors illustrate. In additional scoring analyses on Llama-3.1-8B, disjoint disease-label subsets retain a median Spearman correlation of 0.90; disjoint scoring attributes yield median correlations of 0.62–0.84 across the four measurement capabilities. These are scoring-only checks: training supervision and checkpoint selection remain unchanged. We interpret the findings as evidence for capability-specific screening, with prediction of distinct generation and reasoning objectives remaining future work. We revised the Method's Measuring forecast agreement to explain the purpose of target alignment and state that the evaluation concerns the same clinical capabilities. These scoring analyses are provided here as supplementary evidence.
Q3. How have you improved the writing and aligned the contribution statements with the evidence?
Response: We revised the Abstract, Introduction's study overview and three contribution statements, and Conclusion to align claims with the evaluated settings. We scoped headline correlations to the original CT-RATE setting, clarified measurement targets versus report-derived disease labels, and highlighted the added-encoder forecasts. These sections emphasize initial screening and downstream confirmation. In Discussion and Limitations, the two limitation paragraphs are explicitly labeled.
Q4. Can the essential appendices be made easier to access from the main PDF?
Response: Thank you for this helpful suggestion. In the revised manuscript, we include the appendices after the references in the same PDF to make the supporting material easier to access. We have also checked the appendix cross-references.