acceptodds
Under review as a conference paper at ICLR 2027

When Do Cheap Probes Predict Expensive Training? Probing 3D-CT Encoders for Text Generation

Abstract

Building a 3D CT vision language model begins with a choice of which image encoder to build on. Today that choice is made by fine-tuning every candidate through the full language model and comparing downstream scores, an enormously expensive search. A cheap probe on the encoder's representation promises a way out, but whether it forecasts the expensive outcome has never been tested. We test this with CheapCT on report generation and on MeasureVQA, a new VQA dataset we build. MeasureVQA scores the outcome one capability at a time, its answers measured from segmentation masks and Hounsfield units. Report generation scores the whole report at once and reflects mostly disease. CheapCT forecasts expensive training across every capability. The forecast survives changing the probe readout and the language-model backbone. The rank agreement between CheapCT and fine-tuning stays high throughout, from ρ = 0.90 to 0.97. Used to choose an encoder, CheapCT picks one nearly as good as the best while fine-tuning a single candidate, at orders of magnitude less compute. We release the code and MeasureVQA at https://anonymous.4open.science/r/CheapCT-anon-33B5j.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Medical Report Generation, 3D CT, Probing Representation, 3D CT vision-language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K1RNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether an inexpensive probe of a frozen 3D CT encoder can predict performance after full vision-language fine-tuning. I understand the practical motivation behind reducing the cost of encoder selection, and the paper is reasonably complete in terms of implementation and experiments. The introduction of MeasureVQA, which evaluates size, density, location, texture, and disease separately, is also a contribution worth acknowledging.

However, I do not find the paper convincing overall. The writing, particularly in the abstract and introduction, reads as immature and insufficiently professional, with repeated claims that are stated more strongly than the evidence supports. The organization is also problematic, and multiple essential appendices referenced in the main text are absent from the main PDF.

More importantly, the paper does not provide a sufficiently reasonable justification for focusing on representations before multimodal interaction with the LLM. The observed differences between encoders may reflect their specific compatibility with the fixed aggregation, projector, and LLM rather than their intrinsic downstream value. The evaluation also contains strong target circularity because the probe and downstream tasks are largely constructed from the same underlying targets.

Overall, I consider this a relatively ordinary paper that, despite its reasonable level of experimental completeness, does not currently meet the ICLR acceptance standard. I would reconsider my assessment if the authors can convincingly address the two central concerns below.

Critical strengths
  1. The paper addresses a practical efficiency problem and provides a reasonably complete empirical study. If a cheap probe can reliably eliminate weak encoder candidates before full VLM training, it could reduce the cost of developing 3D medical vision-language systems. The authors evaluate the idea across multiple encoder variants, language-model backbones, readout configurations, and clinical capabilities.

  2. MeasureVQA is a useful capability-level evaluation resource. Separating size, density, location, texture, and disease provides more interpretable information than relying only on aggregate report-generation scores. The dataset construction, image-removal controls, and analysis of position-aware probing are worthwhile components of the paper.

Critical weaknesses
  1. The motivation for focusing specifically on the image encoder is not sufficiently convincing. In an end-to-end VLM/MLLM, multimodal interaction between visual representations and the LLM may constitute a more fundamental bottleneck than the image encoder alone. The paper does not adequately explain why representations measured before this interaction should reliably predict final system performance. Moreover, differences obtained by replacing encoders within a fixed pipeline may reflect encoder-specific compatibility with the chosen aggregation method, projector, or LLM rather than intrinsic encoder quality. Therefore, the current evidence does not establish that the encoder is really a meaningful intervention point for forecasting downstream performance.

  2. The probe and downstream tasks exhibit strong target circularity. MeasureVQA questions are constructed by discretizing the same mask- and Hounsfield-unit-derived quantities that the probe is directly trained to predict. Similarly, the disease probe uses report-derived disease labels that are closely aligned with those evaluated by CE-F1. A strong correlation is therefore partly expected by construction because the probe and downstream evaluation measure access to essentially the same underlying targets. This demonstrates task-specific feature accessibility, but it does not establish that CheapCT can forecast performance on genuinely distinct downstream generation or reasoning objectives.

Questions for authors

To keep the discussion focused, I have only two assessment-critical questions corresponding directly to my main weaknesses. Please see the weaknesses section.

2 comments

Thank you for acknowledging the practical motivation and MeasureVQA's capability-level evaluation.

Our contribution is a systematic empirical validation of inexpensive probing for encoder selection in 3D-CT vision-language models. Building on existing probing methods, we compare readout designs and retain token-position information for spatial attributes.

Additional encoder experiments. We added FlexiCT (May 2026), HLIP (May 2025), and ASAP (May 2026), which did not inform readout design or selection. Forecasts were fixed before the corresponding downstream training. Across 30 encoder–target–backbone combinations, 26 predicted ranks are within one position of the observed rank. Among twelve configurations, selection regret is below 0.004 in eight of ten evaluated target–backbone combinations. These results support screening additional candidates, followed by downstream confirmation. We added the full score and rank comparisons in Prediction on additional encoders.

Q1. Why are representations before multimodal interaction a meaningful selection target? Could the results reflect pipeline compatibility?

Response: Encoder choice strongly influences downstream performance in the evaluated 3D-CT pipelines. This motivates our focus on inexpensive encoder screening before multimodal training. We evaluate the utility of visual representations, including encoder and aggregation, within specified downstream pipelines. Each representation receives separately trained projector and LoRA adapters under a common protocol. Cross-backbone agreement and the additional-encoder results above support this screening use. Pipeline compatibility contributes to the utility relevant to this decision; our results do not isolate universal intrinsic encoder quality or establish the encoder as the largest bottleneck. We clarified separate projector/LoRA training in the Method's Backbones and discussed encoder selection and pipeline scope in Discussion and Limitations.

Q2. Does shared supervision make the correlation expected, and what does it establish about distinct generation or reasoning objectives?

Response: Target alignment is intentional: we ask whether relative accessibility of a clinical capability under a small readout predicts encoder rankings after multimodal training and text generation. Shared targets make a relationship plausible, but do not guarantee that ranking survives adaptation, as Location's readout sensitivity and the added-encoder prediction errors illustrate. In additional scoring analyses on Llama-3.1-8B, disjoint disease-label subsets retain a median Spearman correlation of 0.90; disjoint scoring attributes yield median correlations of 0.62–0.84 across the four measurement capabilities. These are scoring-only checks: training supervision and checkpoint selection remain unchanged. We interpret the findings as evidence for capability-specific screening, with prediction of distinct generation and reasoning objectives remaining future work. We revised the Method's Measuring forecast agreement to explain the purpose of target alignment and state that the evaluation concerns the same clinical capabilities. These scoring analyses are provided here as supplementary evidence.

Q3. How have you improved the writing and aligned the contribution statements with the evidence?

Response: We revised the Abstract, Introduction's study overview and three contribution statements, and Conclusion to align claims with the evaluated settings. We scoped headline correlations to the original CT-RATE setting, clarified measurement targets versus report-derived disease labels, and highlighted the added-encoder forecasts. These sections emphasize initial screening and downstream confirmation. In Discussion and Limitations, the two limitation paragraphs are explicitly labeled.

Q4. Can the essential appendices be made easier to access from the main PDF?

Response: Thank you for this helpful suggestion. In the revised manuscript, we include the appendices after the references in the same PDF to make the supporting material easier to access. We have also checked the appendix cross-references.

Reviewer_K1RNOpenReview ↗

Reply

Your response is basically convincing. Even though I think your approach is not elegant enough, there doesn’t seem to be much room for improvement within the current methodological framework. Considering the relationship between the quality levels and score distributions of different papers, and making a comparative judgment, I think that in this case I am willing to give this paper a more positive assessment. However, I should note that I don’t think this topic belongs to a track like AI4S; there is still a certain boundary between science and medical imaging. Anyway, I have raised my score.

Reviewer_zuJsOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether lightweight representation probes can predict the text-generation performance of pretrained 3D-CT encoders after expensive vision-language model fine-tuning, thereby reducing the model-selection cost of fine-tuning candidate encoders individually. The authors propose CheapCT, which probes the visual tokens of encoders using a lightweight position-aware attention readout, and construct MeasureVQA to assess CT understanding along five capabilities: size, density, location, texture, and disease. Some measurement answers are generated programmatically from segmentation masks and Hounsfield units. The authors report that the dataset covers 70,548 volumes from CT-RATE and Merlin, approximately 1.92 million discrete questions, and 450,000 numerical questions.

The main validation compares nine representation configurations derived from six base encoders and their corresponding fine-tuning results on CT-RATE. Table 2 shows Spearman correlations of 0.90 between CheapCT and downstream CE-F1, and 0.93–0.97 for four measurement capabilities. The authors also test different LLM backbones, readout architectures, image noise/mismatch, and transfer to Merlin. The paper's contribution is mainly a predictive empirical question in medical imaging, a capability-decomposed dataset, and a corresponding model-selection method.

Critical strengths
  1. The problem is important and the evaluation directly matches the objective. The paper does not only report probe accuracy, but relates probe rankings to rankings after full fine-tuning: Table 2 reports a correlation of 0.90 for CE-F1 and 0.93–0.97 for measurement capabilities, and the authors attempt to rule out purely linguistic shortcuts by using patient-mismatched images. This is a more valuable contribution than representation probing alone.

  2. Capability decomposition reveals failure modes that an aggregate score can easily conceal. MeasureVQA separates spatial location and measurement attributes from overall report quality. Figure 4 shows that the correlation between encoder rankings for location and size is only approximately −0.02, while disease, density, and texture are strongly coupled. This provides interpretable evidence that probes must be constructed for the specific downstream capability.

Critical weaknesses
  1. There are too few effectively independent samples for model-selection correlations, and extrapolation claims go beyond the evidence. The nine observed points come from only six base encoders, with some points being GridAvg/ORCA aggregation variants of the same encoder; ViSD+CT-CLIP also shares components with other candidates (§4.2, p.5). Thus, a permutation test over nine points cannot directly be treated as evidence of generalization across nine independent encoder families. Although report generation is tested with four LLM backbones, MeasureVQA is tested with only two; the GREEN correlation for Gemma-3-27B is 0.50, and MedGemma's Location correlation is 0.70 (Figure 3, p.7). Merlin includes only four configurations from two base encoders and shows clear overestimation by the location probe (§5.6 and Figure 6). Leave-out validation at the encoder-family level is needed, instead of summarizing 0.90–0.97 as applying across all settings.

  2. The GREEN results and the interpretation of image dependence need clarification. Table 2 reports a GREEN correlation of 0.97, while Figure 5 reports 0.78. The supplementary Table 10 also reports 0.97 for Llama-3.1-8B. It is unclear why these values differ. In addition, supplementary Table 9 shows that GREEN scores are often close to their shuffle baselines. For example, COLIPRI-ORCA scores 0.35 with real images and 0.31 with shuffled images. The corresponding CE-F1 scores are 0.49 and 0.27. This suggests that GREEN is less sensitive to image-specific information. The paper should explain how this affects the interpretation of the reported correlations and the use of GREEN for encoder selection.

Questions for authors
  1. Could the authors perform leave-one-family-out validation or clustered bootstrap at the base encoder family level? The nine configurations are correlated, so I think validation at the family level would be more convincing. Please also report ranking confidence intervals, selection regret, and variation across fine-tuning seeds for each capability. It would be helpful to include new encoder families that were not used during readout design or selection.

  2. Could the authors explain the difference between the GREEN correlations in Table 2 and Figure 5? Table 2 and supplementary Table 10 report 0.97, while Figure 5 reports 0.78. Are these results based on different settings or calculations?

  3. Supplementary Table 9 shows that GREEN scores are often close to their shuffle baselines. How should we interpret the high correlation between CheapCT and GREEN in this case? Does the correlation mainly reflect differences in encoder ranking, even when the absolute image-dependent improvement is small?

  4. Could the authors clarify the relationship between R² and tertile accuracy in the evaluation? Appendix C.3 uses R² for probe checkpoint selection, while the main results report tertile accuracy. How are the continuous probe predictions converted into the reported classification results?

1 comment

Thank you for the feedback. Our contribution is a systematic validation of probing for 3D-CT encoder selection.

Q1a. How do you address related configurations and new encoders?

Response: We added FlexiCT (May 2026), HLIP (May 2025), and ASAP (May 2026), expanding six encoder types/nine configurations to nine types/twelve configurations. These encoders did not inform readout design or selection. Using the fixed readout and mappings fitted to the original nine configurations, we recorded forecasts before downstream training. Across 30 encoder–target–backbone combinations, 26 ranks are within one position. All three CE-F1 ranks are recovered on Llama-3.1-8B; its CE-F1 ρ rises from 0.90 to 0.96. Size, density, and texture reach 0.94–0.99 on both backbones. ASAP's Location is forecast first but observed fifth and sixth on Llama-3.1-8B and MedGemma-27B, respectively. In a separate retrospective readout-selection check, all six family-held-out folds select the headline, with 92.5% pooled pairwise agreement on comparisons involving excluded configurations. Configurations remain related; we removed the permutation-test claim from the main text. We added Prediction on additional encoders and the readout check in the selection-protocol appendix. MedGemma-27B measurement results are complete for the added encoders. The corresponding report-generation experiments are ongoing, and we will add the results once complete.

Q1b. How stable are rankings across seeds?

Response: On Llama-3.1-8B, we tested seeds 2026, 2027, and 2028 for fVLM, ViSD+CT-CLIP, and COLIPRI-ORCA on CE-F1 and Size, plus GREEN for the first two. Size uses common questions. SDs range from 0.0014 to 0.0134. Larger CE-F1 and Size gaps retain their ordering across observed seed pairings. For Size, COLIPRI-ORCA and fVLM differ by 0.0041 on average, retaining their mean-score ordering in seven of nine pairings. We added the SD bound and ranking stability to Selection quality. The appendix's central 95% correlation ranges over 200 patient splits measure partition sensitivity. Formal rank confidence intervals, family-clustered bootstrap intervals, and seed coverage of every capability are not provided. Close candidates require downstream confirmation.

Q1c. What is the selection regret?

Response: Regret is the best observed score minus the probe-selected score within the candidate set. For the original nine configurations on Llama-3.1-8B, selection retains 95–100% of the best score across five clinical capabilities. With twelve configurations, regret is below 0.004 in eight of ten target–backbone combinations. MedGemma-27B Size loses only 0.0008 despite ranking second. Location regrets are 0.067 and 0.107 on Llama-3.1-8B and MedGemma-27B, respectively. We added the definition and original results to Selection quality, and expanded regret to Prediction on additional encoders.

Q1d. Which settings support the headline correlations?

Response: The Abstract's ρ=0.90–0.97 is now scoped to the original nine CT-RATE configurations with Llama-3.1-8B. The Method distinguishes four report-generation backbones from two for measurement VQA. Forecast reliability in Discussion and Limitations addresses weaker GREEN on Gemma-3-27B, less consistent Location on MedGemma-27B and Merlin, and downstream confirmation. Merlin has only four configurations from two base encoders.

Q2–Q3. Why did GREEN correlations differ, and what does the shuffle floor imply?

Response: We corrected the outdated figure to GREEN ρ=0.97, matching both tables, and synchronized BLEU-4/ROUGE-L using the shared validation CE-F1-selected checkpoint. For the original nine CT-RATE configurations with Llama-3.1-8B, mean GREEN is 0.309 with real images and 0.289 under shuffle. Probe correlations are 0.97 and 0.83, respectively, and 0.93 with their difference. This supports score prediction, while the mean image gain of 0.020 calls for a separate grounding assessment. We revised The floor control and its appendix explanations, removing the claim that GREEN collapses like CE-F1.

Q4. How do R² checkpoint selection and tertile accuracy relate?

Response: Measurement probes use masked MSE; validation R² selects the checkpoint. We restore predictions to the original measurement scale and apply the same training-fixed tertile cutpoints to predictions and ground truth. Accuracy measures matching buckets on valid samples, averaged over attributes within each capability and then three seeds. Binary clinical VQA retains its specified thresholds. We clarified Measuring forecast agreement in the Method and added the formula and aggregation details to the appendix's Probe readout and training.

Reviewer_Nsb5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The goal of the paper is to study whether inexpensive representation probes can guide encoder selection for 3D CT vision-language models. The authors introduce CheapCT and MeasureVQA, which evaluates size, density, location, texture, and disease using image-derived measurements and report-derived disease labels. Experiments compare nine encoder/aggregation variants on CT-RATE, with four language backbones for report generation and two for measurement VQA, followed by an evaluation on Merlin. Probe scores predict downstream performance well in the main CT-RATE setting, and the readout and image-shuffling controls help explain when this relationship holds.

Critical strengths
  1. The paper addresses a practical problem: choosing a CT encoder without repeatedly training an expensive vision-language model. The comparison across representations and language backbones is substantial, and the reported selection performance makes the finding useful beyond a correlation analysis.

  2. MeasureVQA provides a useful complement to report-level evaluation. The measurement definitions, quality filters, and shortcut checks are documented in the supplement. I also appreciate the readout comparison and shuffled-image controls, which show why location needs positional information and why a plausible report is not necessarily grounded in the correct scan.

Critical weaknesses
  1. The selection protocol needs a cleaner separation between tuning and evaluation. Appendix C.2 reports each downstream metric at its own best validation checkpoint, while C.3 selects probe checkpoints on validation and identifies the strongest readout from the same study. An untouched evaluation set is not described. With only nine related representation variants, it is unclear how much of the reported selection performance carries over to new patients or encoder families.

  2. The forecast is less general than the conclusion suggests. GREEN correlation falls to 0.50 for Gemma-3-27B, and the Merlin results include location failures and report scores close to the shuffle floor. These cases are discussed, but the floor check requires an already trained downstream model. It therefore does not yet establish how a practitioner can recognize an unreliable forecast before paying for downstream training.

Questions for authors
  1. Can you clarify whether any patients were held out from both checkpoint selection and the reported forecast evaluation? How stable are the encoder ranking and selection regret when related encoder/aggregation variants are grouped together or one encoder family is left out?

I2. s there a criterion available before downstream training that identifies the unreliable Merlin or GREEN settings? If not, which narrower set of tasks and training conditions do you consider supported by the results?

2 comments

Thank you for recognizing the practical value of encoder screening and capability-level evaluation. Our contribution is a systematic empirical validation of existing probing methods for prioritizing 3D-CT encoder candidates before multimodal training.

Q1. How are probe early stopping, downstream checkpoint selection, and forecast evaluation separated? Were any patients reserved for evaluation?

Response: We corrected the appendix's checkpoint description: CE-F1 and GREEN use a shared checkpoint selected by validation CE-F1, while each MeasureVQA run uses its best validation-accuracy checkpoint. The original evaluation reused validation patients for selection and scoring. We added a patient-level hold-out evaluation. Retrained probes use a development subset of training patients for early stopping. The 1,304 validation patients are split equally into A for downstream checkpoint selection and B for scoring both models. All reconstructions of a patient remain together. Across 200 splits, median Llama-3.1-8B CE-F1 correlation remains ρ=0.90\rho=0.90; measurement medians on both backbones differ from the original correlations by at most 0.05. This retrospective check reuses the validation cohort that informed the original study. We added the protocol and results to the appendix's Selection protocol and patient-level hold-out evaluation, corrected Implementation details, and added a pointer in the Method.

Q2. How do related configurations affect ranking and selection regret, and does prediction extend to additional encoder families?

Response: We added three recent encoders, FlexiCT (May 2026), HLIP (May 2025), and ASAP (May 2026), expanding six encoder types and nine configurations to nine types and twelve configurations. These encoders did not inform readout design or selection. Using the fixed original readout and mappings fitted to the original nine configurations, we recorded forecasts before downstream training. Across 30 evaluated encoder–target–backbone combinations, 26 predicted ranks are within one position of the observed rank. All three new-encoder CE-F1 ranks are recovered on Llama-3.1-8B. Selection regret, the best observed score minus the probe-selected candidate's score, is below 0.004 in eight of ten evaluated target–backbone combinations. Location has larger regrets of 0.067 and 0.107. A separate retrospective readout-selection check excludes one of six encoder families per fold. All folds select the headline readout, with 92.5% pooled ranking agreement on comparisons involving excluded configurations. This family check concerns readout selection; the regret results concern the full candidate sets. Expanded correlations still describe related configurations. We added forecast comparisons and expanded correlations to Prediction on additional encoders, the regret definition to Selection quality, and the readout check to the selection-protocol appendix. The corresponding experiments are ongoing, and we will add the results once they are complete.

Q3. Can unreliable GREEN or Merlin forecasts be identified before expensive downstream training?

Response: Our study does not establish a validated criterion for detecting these failures before downstream training. CheapCT provides an inexpensive first step for prioritizing encoder candidates, followed by training and evaluation of shortlisted candidates. Noise and shuffle tests assess whether trained models use the correct scan. We revised the Conclusion to distinguish initial screening from downstream confirmation and clarified the role of trained-model grounding checks in Limitations: Forecast reliability within Discussion and Limitations.

Q4. Which tasks and training conditions are supported by the evidence?

Response: The strongest evidence supports CE-F1 and size, density, and texture screening on CT-RATE under the evaluated training protocols. Report generation covers four language-model backbones and measurement VQA two. GREEN is less consistent on Gemma-3-27B, and Location on MedGemma-27B and Merlin requires additional validation. Merlin's small candidate set also limits broader generalization. We clarified these boundaries in Discussion and Limitations and revised the Abstract, Introduction, and Conclusion to scope the findings to the evaluated settings and emphasize downstream confirmation.

Thank you for the response and the additional analyses. The patient-level split addressed my concern about using the same patients for checkpoint selection and scoring. Since it reuses the original validation cohort, I view it as a robustness check within that cohort. The reported forecasts for the three additional encoders also give me more confidence in CheapCT as a screening tool.

My remaining concerns are the higher selection regret for location and the difficulty of identifying unreliable forecasts before downstream training. The revised framing makes sense to me. Using the probe to shortlist encoders, then confirming the choice through downstream evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.