acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation

Abstract

Automated radiology report generation from 3D CT volumes often suffers from incomplete abnormality coverage. We provide empirical evidence that this limitation stems from a representational bottleneck: contrastive 3D CT embeddings encode discriminative abnormality signals, yet exhibit dimensional concentration, with CT-CLIP concentrating 90% of its variance in only 2 of 512 principal components. Corroborating this, scaling the language model yields no measurable improvement, suggesting that the bottleneck lies in the visual representation rather than the generator. This bottleneck limits both generation and retrieval. Image-based similarity often fails to isolate fine-grained abnormalities. We propose AdaRAG-CT, an adaptive augmentation framework that compensates for this visual bottleneck by introducing supplementary textual information through controlled retrieval and selectively integrating it during generation. On the CT-RATE benchmark, AdaRAG-CT reaches a Clinical F1 of 0.480. Ablations show that relevant retrieved context improves clinical efficacy, while adaptive triggering balances clinical and language quality. Code is available at https://anonymous.4open.science/r/AdaRAG-CT-anonymous-5756.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
3D CT report generation, retrieval-augmented generation, adaptive retrieval, representational bottleneck, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W526OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates dimensional concentration in contrastive 3D CT embeddings and proposes AdaRAG-CT for radiology report generation. The framework combines organ-indexed sentence retrieval with a learned retrieval token that selectively triggers context injection and sentence regeneration. Training mixes oracle and realistic retrieval contexts. On CT-RATE, the 8B model improves Clinical F1 from 0.455 to 0.480 over its retrieval-free baseline.

Critical strengths

The paper examines multiple CT encoders through complementary analyses of embedding geometry, abnormality classification, and retrieval quality. The framework is clearly described and improves both Clinical F1 and several textual metrics over its baseline. Comparisons across retrieval pipelines and injection strategies provide useful insights into how retrieved context affects report generation.

Critical weaknesses

The central bottleneck hypothesis is more strongly stated than the evidence supports, and the experiments do not clearly establish that retrieval recovers missing patient-specific information. The methodological advance over existing adaptive RAG approaches and the benefit of adaptive triggering remain insufficiently demonstrated. These limitations weaken both the conceptual contribution and the interpretation of the reported gains.

Questions for authors
  1. The lower-variance directions retain useful abnormality classification signals, suggesting that information may be present but inadequately exploited by the similarity metric or decoder. Moreover, the generation model uses max-pooled CT-CLIP features, whereas the headline dimensional-collapse result concerns average pooling. Whitening, principal-component removal, or learned similarity metrics would help distinguish representational anisotropy from information loss. The weaker performance of the 70B model alone also does not rule out optimization or adaptation issues.

  2. Learning a retrieval token closely follows Self-RAG, while generating a tentative sentence and regenerating it after retrieval resembles FLARE. Direct comparisons under the same settings are needed to establish the methodological contribution. In Table 5, fixed-interval retrieval achieves a higher Clinical F1 than adaptive retrieval (0.494 versus 0.480). Although adaptive retrieval improves textual metrics, its advantage should be evaluated under matched retrieval frequency and computational budgets.

  3. It remains unclear whether retrieved text provides reliable patient-specific information. Since the query sentence is generated from the same limited visual representation, retrieval may reinforce an initially incorrect prediction rather than correct it. Reports from other patients do not establish that an abnormality is present in the current case. Random retrieval, organ-matched but abnormality-mismatched retrieval, and analyses of error correction versus amplification would clarify this issue. The 70% oracle-context training ratio also requires ablation.

  4. Clinical F1 measures the presence of 18 abnormalities but does not assess location, laterality, size, or severity. Improvements in this metric therefore do not substantiate the claim that retrieval recovers fine-grained details missing from the visual representation. Attribute-level factual evaluation or expert assessment, together with examples showing changes in omissions and hallucinations before and after retrieval, would better support this claim.

  5. The main comparison uses results from original publications, while the proposed method is evaluated on a filtered validation subset. The unified evaluation mentioned in the manuscript is not included in the provided PDF. Furthermore, the retrieval-free baseline already achieves 0.455 Clinical F1, exceeding CT-Agent’s reported 0.420. Comparisons on identical samples with the same evaluation pipeline, alongside separate controls for baseline design, retrieval training, and inference-time retrieval, are needed to attribute the gains accurately.

Reviewer_cp2GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies possible reasons for the current low clinical accuracy of structured radiology reports generated from 3D CT data by vision-language models, by directly analyzing the representation space of 3D CT embeddings, which suggested that most of the variance is concentrated in a few dimensions and thus that existing embeddings are insufficient for fine-grained observations. AdaRAG-CT is therefore proposed to add supplementary textual context, which was shown to improve Clinical F1 and other efficacy metrics, on the CT-RATE validation dataset.

Three contributions are claimed in the Introduction, on identifying a representation bottleneck in CT embeddings, designing AdaRAG-CT to include additional information through RAG, and validating AdaRAG-CT on CT-RATE. These claims are largely justified if with some caveats as discussed later in the review.

Critical strengths
  • Systematic analysis of potential bottlenecks in CT-based report generation is timely
Critical weaknesses
  • Correlation between principal component-based participation ratio and information capacity is not well-established for the visual embeddings
  • Partial reliance on visual embeddings for adaptive retrieval despite their claimed deficiencies
Questions for authors
  1. In the Introduction section and Section 5.1, CT-RATE is referenced without sufficient background. This might be included in supplementary material for context - including whether clinical implementation would be expected to have access to data other than the 3D CT volumes in diagnosis - especially since the main results are on Clinical F1 score (Table 4).

  2. In Section 3, the first claimed contribution on the identification of a (visual) representational bottleneck is justified by analysis on linear probing on representations (Table 2), as well as arguments on the low effective dimensionality.

A major assertion is that "An embedding can achieve high binary AUC (e.g., 'pleural effusion present vs. absent') using just one or two directions, yet may not reliably preserve the fine-grained semantic detail required for report generation, such as laterality, severity, and co-occurring abnormalities" (Line 157), and that the principal components (PCs) outside dim90dim_{90} is "long-tail noise rather than signal" (Line 153). However, it is also noted that linear probing on the top two PCs (90% of variance) of zero-shot CT-CLIP achieves a mean AUC of 0.552, whereas the bottom half of PCs ([far] less than 10% of variance) achieves an AUC of 0.746.

Given this, the main claim that the "the visual channel is too limited" (i.e. there is a representational bottleneck, Line 161) may be contested, since there does not appear to be a clear correlation between effective dimensionality (by explained variance) and relevant information content (by classification performance as reflected by AUC), which is not actually addressed either by the comparison to the effective dimensionality of ImageNet. Given this, the relationship between embedding effective dimensionality and (fine-grained) information content should be better justified.

  1. Related to the above, a direct analysis of the various 3D CT contrastive embeddings as to the supposed fine-grained semantic detail (e.g. laterality, severity, co-occurring abnormalities) could in principle be performed, by linear probing of those features (as extracted from the ground truth structured reports or otherwise - i.e. the label sets in Section 3.2) as done for the main abnormality classes (Table 2). This would more convincingly support the central claims on a representational bottleneck in the visual channel, and thus might be strongly considered.

  2. Additionally, the training of only linear probes, rather than more-complex classifiers, may also be justified since abnormality information that is not linearly accessible, may still be present. Additional analysis with non-linear models for the metrics reported in Table 2 may thus be valuable supplementary information.

  3. In Section 3, an analysis of possible (text) decoder bottleneck was done by replacing the 8B LLM with a 70B variant, which instead decreased Clinical F1 from 0.455 to 0.405. While this does superficially support the assertion of the decoder not being a bottleneck, the significant reduction in Clinical F1 score may be worthy of comment, since the increased LLM/decoder size should not result in such a reduction. Further comparison with <8B LLMs may also be appropriate.

  4. In Section 4, the AdaRAG-CT framework is proposed to augment the VLM with adaptive retrieval-augmented generation. However, there appears to be an in-principle objection on the proposed adaptive retrieval procedure (Section 4.4).

The retrieval (by organ) is for the most similar training images using image embedding cosine similarity. However, this would appear to inherit the claimed representational embedding concerns, in that there is no guarantee that the most similar images by cosine similarity would actually be similar in the fine-grained features that existing representations are claimed to be weaker with. The use of the image embeddings for retrieval might thus be further justified.

  1. In Section 5.2 (and Section 2), parameter and other details for the MLLM CT-Agent are not well-described. This is relevant as CT-Agent is the closest-performing method, from the various metrics.
Reviewer_6AeVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses incomplete abnormality coverage in 3D chest CT report generation on CT-RATE and makes two main contributions:

  • a diagnostic analysis (Section 3): the authors probe four frozen contrastive encoders (CT-CLIP zero-shot, CT-CLIP VocabFine, FVLM, and ViSD-Boost), showing that image-based similarity often fails to isolate fine-grained abnormalities.

  • the AdaRAG-CT method (Section 4), built on a LLaVA-style base model with one global CT-CLIP token and four ViSD-Boost organ tokens, adding three components:

  1. n organ-indexed database of ~572K training-report sentences embedded with a contrastively tuned CXR-BERT;
  2. a Self-RAG-style token ([RAG]) supervised at high-perplexity sentences (>70th percentile) and trained with an oracle/real context mixture (p_oracle = 0.7);
  3. an inference loop where on [RAG], the model drafts the next sentence, uses it as a retrieval query, rolls the draft back, and regenerates.

A main results the authors report the followings: AdaRAG-CT reaches Clinical F1 = 0.480 (8B) and 0.426 (70B) on the CT-RATE validation set (Table 4). Fixed-interval injection with N = 5 reaches a higher Clinical F1 (0.494) but lower BLEU-4 and ROUGE-L (Table 5).

Critical strengths
  1. the systematic spectral, probing, and retrieval analysis of four 3D CT contrastive encoders with bootstrap intervals (Tables 1–3) provides concrete, reusable insights into anisotropy and organ-level retrieval asymmetries.
  2. the authors transparently acknowledge key limitations (inability to isolate retrieval value from training distribution proximity, false trigger firing on template sentences, and Clinical F1's blindness to severity and laterality).
Critical weaknesses

. The central thesis is unsupported and partially contradicted by the data. Equating low PCA effective dimensionality with information poverty is not totally correct, as PCA is unsupervised (label-agnostic) and the paper's results reported in section 3.1 (AUC 0.746 vs. 0.552) sugest diagnostic signal resides in low-variance directions. Furthermore, 70B model degradation may reflect training confounds rather than a visual capacity ceiling.

  • The paper reports some small and unquantified empirical gains with unverified mechanisms. The net gain from inference-time retrieval is modest (+0.018 Clinical F1), it lacks multi-seed confidence intervals, it suffers from validation-peak checkpoint selection, and it is evaluated using a binary extraction metric that cannot confirm the claimed delivery of laterality, severity, or co-occurrence details.

Concerning trhcnical soundess, I have the following concerns:

  • the diagnostic argument conflates variance concentration with information poverty. The paper repeatedly asserts, from the abstract onwards, that "PCA shows that most variance is concentrated in only a few dimensions," treating this as evidence that the visual channel is "too limited to begin with" (Section 3.1). PCA, however, is unsupervised: it measures variance, not class separability. Leading directions may reflect non-diagnostic intra-class variability (patient anatomy, scanner model, acquisition protocol) rather than abnormality content. The probe on the lower half of the spectrum (0.746) substantially outperforms the probe on the top-2 PCs (0.552). If this reflects the average pool embedding, the lower half also matches or exceeds the full embedding (0.739). The diagnostic signal therefore resides almost entirely in low-variance directions. The space is not information-poor; its dominant variance is misaligned with the actual semantics The actual dimensionality does not predict probe performance across models. CT-CLIP avg-pool has the lowest dimensional footprint (dim90 = 2.0 zero-shot, 1.0 VocabFine) with an AUC of 0.739. Among organ encoders, FVLM has dim90 ≈ 4.0–6.0 (0.823 AUC) while ViSD-Boost has dim90 ≈ 5.0–9.3 (0.853 AUC). Conversely, CT-CLIP max-pool exhibits a drastically higher dimensionality (dim90 ≈ 103.6) yet a lower AUC (0.669). This lack of monotonic alignment shows that spectral variance distribution does not dictate diagnostic separability. The generator receives embeddings through a learned 2-layer MLP (Eq. 1), which can actually rescale low-variance directions. Variance concentration in raw embeddings therefore is not very informative on what the LLM can extract. A single top-2 versus lower-half comparison is not very general. The analysis should be done by probe-AUC curves across top-k and bottom-k components, with random k-dimensional subsets as a control. Additionally, the text must state whether embeddings were centered and normalized prior to PCA, as L2-normalized vectors with a large mean offset artificially inflate the first principal component. For cosine retrieval (Section 3.2), variance concentration does matter, but key baseline controls (e.g., post-hoc whitening or zeroing dominant components before retrieval) are missing.

  • Several claims are somewhat unsupported. The paper attributes the gap relative to ImageNet CLIP to natural images "encouraging the encoder to use many dimensions," without citation or experiment. The comparison is uncontrolled across dataset, pre-training scale, modality, and class count. The second proposed factor (that a global contrastive loss is too coarse) is similarly untested. The claim that probes "confirm that the contrastive objective successfully encodes diagnostic signals" holds for pleural effusion (0.911) or cardiomegaly (0.891), but breaks down for focal findings like lung nodule (0.619) or pulmonary fibrotic sequelae (0.621). This per-label heterogeneity is a more direct argument for a fine-grained bottleneck than PCA, and the paper should try to explain or at least to discuss this. Given severe class imbalance, AUPRC should accompany AUC, and a non-linear probe would establish an information upper bound. The 70B model performs worse than the 8B model (0.405 vs. 0.455 Clinical F1; recall drops from 0.469 to 0.373). While Section 6 speculates that stronger language priors in 70B induce confident hallucinations, no empirical ablations are provided to get a better understanding of this phenomenon. Table 3 lacks random or prevalence baselines. In Text2Text retrieval, the query is the model's own draft, a function of the visual tokens. The database consists of training reports on which the model was trained. The claim that the textual channel "supplies information absent from the visual input" (Section 5.2) is unproven.

  • The evaluation is not completely coinvincing. Clinical Efficacy measures classifier extraction, not text quality: Evaluation relies on a RadBERT-based classifier mapping reports to 18 binary labels. Extraction errors directly propagate into precision and recall. Ground-truth labels in CT-RATE appear to be classifier-derived from reports. If so, evaluation compares two classifier outputs rather than predictions against expert human annotation. Binary mapping discards localization (e.g., right-upper vs. left-lower lobe nodule), severity (mild vs. massive effusion), hedged language ("suspicious for", "consistent with"), and hallucinations outside the 18-label set. The paper asserts that retrieval supplies laterality, severity, and co-occurrence details. No reported metric verifies this. The adoption of standard NLP metrics is not adequate: BLEU, ROUGE-L, and METEOR measure surface overlap. LLaMA Score is unvalidated against radiologist judgment for CT, it comes from the same model family as the generator, and shows tiny unquantified differences (7.30 vs. 7.75). The authors should report entity- and relation-level metrics specific to radiology (RadGraph F1 / RadGraph-XL), RaTEScore, and GREEN (or RadCliQ). In addition, a radiologist reader study on a stratified subset would really help to support claims regarding laterality and severity.

  • Experimental framework can be improved. Tables 4 and 5 report no confidence intervals or training-seed variance, making gains of +0.025 and +0.018 statistically unquantifiable. Test-set bootstrapping (as in Fig. 2) does not capture training-seed variance. The reported 8B AdaRAG-CT metrics coincide with the step-2000 peak of the Figure 2 validation curves (Clinical F1 ≈ 0.48, BLEU-4 ≈ 0.24). Because Figure 2 uses the same validation set as Table 4 with no separate test split, selecting the peak overestimates performance. At later checkpoints, Clinical F1 settles around 0.46–0.47, reducing the net gain over the base model (0.455) to ~0.01. Parameters (N, 70th-percentile threshold, p_oracle, λ, K_coarse, K_fine) appear to be tuned on the evaluation split. Fixed-interval N = 5 (0.494) and N = 7 (0.482) both beat adaptive RAG (0.480) on Clinical F1. Furthermore, of the +0.025 overall gain over the base model (0.455 → 0.480), +0.007 is achieved merely by training with the RAG loss without retrieval (0.462 in Table 5), leaving only +0.018 attributable to inference-time retrieval. The claim that "relevant retrieved context improves clinical efficacy" lacks supporting ablations (no random-context, shuffled-context, or inference-time oracle conditions). Baseline numbers in Table 4 are copied from original publications that differ in volume subsets, weighting schemes (weighted vs. macro F1), and task framing. AdaRAG-CT's ROUGE-L (0.354) and METEOR (0.246) trail CT-CHAT (0.584) and CT-Agent (0.425) by large margins, yet this gap is not addressed. Computational costs are not even mentioned.

Questions for authors

I think that the criticasl concerns outlined above are very critical and tghat there are too many modifications to be made specifically in order to address them. I can change my assessment only if answers to such concerns will actually be provided in a satisfactory way.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.