acceptodds
Under review as a conference paper at ICLR 2027

Scaling 3D Visual Grounding in Abdominal CT

Abstract

Visual grounding models can enhance radiology workflows by linking clinical report findings to their corresponding image regions. This is particularly valuable for 3D CT, where findings often occupy a tiny fraction of the volume. Despite the promise of this application, progress has been limited. Training 3D grounding models requires large sets of paired phrases and regions, and building such datasets is expensive, requiring radiologists to annotate images by hand. We posit that this supervision is already created implicitly during routine reporting, as radiologists frequently place 2D annotations (e.g., distance measurement, arrows) on key images to make measurements and to support report interpretation. We introduce an automated pipeline that converts these routine clinical annotations into large-scale phrase–region supervision for 3D visual grounding. The pipeline links each annotation to the corresponding finding in the report through metadata matching, then uses a promptable 3D segmentation model to convert the 2D annotation into a volumetric mask. This produces phrase–mask–volume datasets without requiring additional radiologist annotation. Applied to a single institution's clinical picture archiving and communication system (PACS), our approach generated 105K phrase–mask–volume triplets from 59K abdominal CT exams. We also introduce two benchmarks for evaluating visual grounding in abdominal CT, LocusBench-Onc and LocusBench-ED, which comprise 240 oncology and 260 emergency-department radiologist-reviewed phrase–mask–volume triplets, respectively, with the latter spanning 13 distinct finding categories such as appendicitis, hematoma, and hernia. We further introduce LocusCT, a 3D visual grounding model trained on our 105K phrase–mask–volume dataset, which achieves hit rates of 0.725 on LocusBench-Onc and 0.773 on LocusBench-ED, substantially outperforming comparator models. These results show that routine annotations recorded in PACS can provide a scalable and previously unused source of supervision for 3D visual grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Visual Grounding, Computed Tomography (CT), Abdominal, Radiology, Weak Supervision, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_g2QHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a pipeline for constructing large-scale 3D abdominal CT visual grounding data from routine clinical annotations. Report phrases are matched to radiologists’ measurement lines and arrows using measurements and imaging metadata, and SAM2CT converts these annotations into volumetric masks. The authors construct approximately 105K phrase–mask–volume triplets, train LocusCT, and introduce LocusBench, comprising 500 radiologist-reviewed examples covering oncology and emergency-department findings. Validation includes zero-shot comparisons, training two architectures on the constructed data, ablations, and an initial external evaluation on Merlin.

Critical strengths
  1. A practical and scalable source of clinically grounded supervision. Obtaining volumetric annotations for individual findings is expensive. The proposed pipeline leverages annotations already produced during routine care, reducing the need for additional manual delineation when constructing training data. The resulting 105K triplets and improvements with increasing training data provide convincing evidence that this supervision source is useful. This is the paper’s principal contribution.

  2. An expert-reviewed benchmark for abdominal finding grounding. LocusBench extends evaluation beyond predominantly anatomical or chest-focused datasets to oncology and emergency-department abdominal findings. Both phrase–annotation correspondence and segmentation quality are reviewed, with problematic examples corrected and re-reviewed. Together with experiments on two architectures and data-scaling ablations, this provides meaningful validation of the constructed supervision.

Critical weaknesses
  1. Limited baseline coverage and insufficient comparison under matched supervision. The zero-shot evaluation includes only VoxTell, SegVol, and BiomedParse-v2. Several methods evaluated on ReXGroundingCT could also serve as relevant baselines, but their applicability is neither evaluated nor discussed. Broader comparisons, including training or fine-tuning representative methods on the same data, would help establish their relative performance under comparable supervision.

  2. Limited independent external validation. The training data and LocusBench originate from the same institution, and benchmark examples are selected through the same annotation pipeline. The evaluation therefore primarily represents positive findings with retained clinical annotations and matchable report descriptions. The Merlin experiment includes only 78 findings, relies on radiologist ratings rather than independently annotated segmentation masks, and lacks corresponding baseline evaluations. This provides preliminary evidence of transfer but leaves uncertainty about generalization across institutions and reporting styles.

Questions for authors

See weakness

1 comment

Thank you for the thoughtful review, and for recognizing the routine-care supervision as a practical and scalable contribution, the value of LocusBench, and the validation across two architectures and data scales.

W1. Limited baseline coverage and comparison under matched supervision.

We selected VoxTell, SegVol, and BiomedParse-v2 as the strongest publicly available, text-promptable 3D segmentation models that can be applied to abdominal CT without retraining. Many methods evaluated on ReXGroundingCT are not publicly available or were developed for chest CT; we will add a discussion of their applicability to the Related Work.

To address comparison under matched supervision, we fine-tuned pretrained VoxTell, the strongest zero-shot baseline, on the same training data (10 epochs, lr = 5e-6) and evaluated it under the same protocol:

ModelOnc DiceOnc Hit RateED DiceED Hit Rate
VoxTell (zero-shot)0.1170.1880.1180.209
LocusCT (from scratch)0.4790.7330.4710.765
LocusCT (VoxTell initialization)0.4580.6960.4840.810

Training on our data raises VoxTell's hit rate from 0.188 to 0.696 on LocusBench-Onc and from 0.209 to 0.810 on LocusBench-ED, a nearly 4× gain under identical architecture, prompts, and evaluation. Fine-tuning from pretrained weights and training from scratch perform similarly, with each leading on one benchmark. We retain LocusCT (from scratch) as our primary model because its performance is more balanced across the two benchmarks. We will add this comparison to the paper.

We also tested whether the zero-shot gap reflects prompt format rather than grounding ability by prompting all baselines with short category labels (≤ 4 words) instead of full report sentences. This improves them only modestly (best: 0.221 on Onc), so prompt format does not explain the gap (full table in our response to Reviewer rwjo).

W2. Limited independent external validation.

We agree, and built LocusBench-Merlin to address each point you raised (details in the comment to all reviewers, "All Reviewers: Expanded External Validation"):

  • Different institution and reporting style. Findings come from the external Merlin test set, with grounding queries taken directly from its reports.
  • Independent of our pipeline. Merlin contains no GSPS annotations, so no case passed through our annotation pipeline, and findings are not restricted to our categories.
  • Segmentation masks and baselines. 154 findings across 86 studies, with reference masks from three radiologists and all three baselines under the same protocol.

Reference masks were initialized from LocusCT predictions, which radiologists could edit or redraw; they relocated the finding entirely in 48 of 154 cases. Because some anchoring cannot be ruled out, we treat LocusCT's absolute hit rate as an upper bound and its margin over baselines as the main result.

LocusCT localizes 68.8% of findings versus 26.0% for the best baseline (VoxTell). On findings within our existing categories, its hit rate (0.779) is in line with LocusBench-Onc and -ED (0.733, 0.765), indicating that LocusBench results are not an artifact of pipeline selection. On finding types none of our benchmarks cover (e.g., fractures, fluid collections), LocusCT still reaches 0.597, more than twice the best baseline (0.247). We will also state the GSPS-matching selection step explicitly in the Limitations, framing LocusBench results as performance on pipeline-matchable findings.

Reviewer_rwjoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies 3D visual grounding in abdominal CT, focusing on the scarcity of high-quality paired image–text–region data. The authors propose an automated pipeline for constructing weakly supervised data by matching lesion descriptions in radiology reports to routine clinical GSPS annotations and using SAM2CT to generate 3D segmentation masks, yielding approximately 105,000 training samples. They also introduce two radiologist-reviewed benchmarks, LocusBench-Onc and LocusBench-ED, containing 240 and 260 finding-level samples, respectively; the latter covers 13 categories of emergency-related findings. In addition, they train LocusCT using the VoxTell architecture. Experiments show that LocusCT substantially outperforms the zero-shot performance of the comparison models on both test sets and demonstrates some generalization to the external Merlin dataset.

Critical strengths

S1:The paper proposes a scalable approach to constructing weakly supervised data with practical clinical value.

The main contribution is the use of GSPS annotations already produced by radiologists during routine image interpretation to convert information distributed across images and reports into correspondences suitable for training 3D visual grounding models. This approach can reduce the need for additional manual annotation while retaining fine-grained lesion descriptions from real clinical reports. The resulting dataset of approximately 105,000 phrase–mask–volume triplets is substantial in scale and covers a variety of clinical findings.

Compared with prior work that mines lesion data from clinical measurements, this paper further establishes correspondences between natural language descriptions and 3D masks, allowing the data to be used directly for lesion grounding with open-ended text queries. The data scaling experiments also indicate that this source of weak supervision provides useful information for model training. Thus, beyond its value in data engineering, the work offers a practically meaningful research direction for constructing large-scale medical vision–language supervision.

S2:The paper establishes physician-reviewed benchmarks for visual grounding in abdominal CT and validates the approach from multiple perspectives.

LocusBench covers oncology-related findings and 13 categories of emergency lesions, extending the coverage of fine-grained clinical lesions in existing 3D medical visual grounding evaluations. Physicians review the correspondence between text and annotations, and problematic 3D segmentations are revised and reviewed again, providing a clearly defined process for manual quality control.

In addition to comparisons with existing zero-shot methods, the paper examines the effects of model architecture, data scale, and training data filtering strategies. The external Merlin evaluation provides preliminary evidence from another data source and reports separate rating levels for suitability for localization and for clinical measurement. Together, these experiments provide a relatively comprehensive validation and are an important basis for recommending acceptance.

Critical weaknesses

W1:The selection process for the evaluation set may limit how well it represents visual grounding tasks in real clinical practice.

LocusBench test samples are first generated through the automated GSPS–report matching pipeline and then reviewed and revised by physicians. Consequently, lesions included in the evaluation set must satisfy certain conditions for clinical annotation and text matching. Findings that are not explicitly marked, lack measurement information, or cannot be matched using the metadata rules may be underrepresented. This construction process can improve the reliability of the benchmark annotations, but it may also bias the evaluation distribution toward lesions for which spatial correspondences are easier to establish.

The external Merlin test includes only findings with measurement information, with six samples per category and 78 in total, and relies primarily on subjective ratings from a single radiologist. This experiment provides encouraging preliminary evidence of transfer, but the small number of samples per category is insufficient for stable estimates of between-category differences, and the single-reader design does not allow inter-reader variability to be estimated. Aggregate performance under class-balanced sampling also cannot be directly interpreted as the overall success rate under the case distribution encountered in clinical practice.

W2:The reference space and Hit computation used in the main performance comparisons require further verification.

In the supplementary materials, the LocusBench entry point for pretrained VoxTell uses the original images and masks, whereas the corresponding entry point for the model trained from scratch uses resampled images and masks. The evaluation function computes Dice directly on the reference grid loaded by each entry point. Model-specific input preprocessing is reasonable, but the discrete representation of the final reference masks, the cropped field of view, and the sample sets must remain comparable, or the impact of any differences should be assessed through sensitivity analysis. Saving predictions mapped back to the original space does not, by itself, establish that scoring was performed in the original reference space.

Another issue is that the supplementary implementation computes Dice as ((2|P\cap G|+1)/(|P|+|G|+1)). When the prediction is empty and the reference mask contains only 1–9 voxels, the score can still reach 0.1 and thus be counted as a Hit. An empty prediction paired with an empty reference mask also receives a perfect score. The practical impact depends on whether these cases occur in the final evaluation and, if so, how many there are. These issues affect the accurate interpretation of performance differences and require confirmation through the final configurations, statistics on the samples actually included, and recomputed results.

Questions for authors

1.Cross-model evaluation configurations

The default entry points in the supplementary evaluation code suggest that pretrained VoxTell and VoxTell trained from scratch may use images and reference masks at the original resolution and at a resampled resolution, respectively. Could the authors confirm the final evaluation configurations used for Tables 1 and 2 and explain whether differences in scoring space affect the reference representation of small lesions and the resulting Dice scores? If such differences exist, has the comparability of the cross-model performance results been verified?

2.Dice smoothing and Hit Rate

The Dice smoothing term in the supplementary implementation may allow an empty prediction to reach the Hit threshold when the reference mask contains only a few voxels. Please confirm whether any reference masks on the final evaluation grid are empty or contain only 1–9 voxels, and whether any cases with zero overlap were actually counted as Hits.

3.Training conditions in the data scaling experiments

Please provide the actual number of training steps, effective batch size, early stopping rules, and final checkpoint selection for each data scale in Table 3, and explain how these settings correspond to the supplementary scripts. This information would help assess the combined effects of data volume and training budget reflected in the current scaling experiments.

1 comment

Thank you for the detailed and careful review, including the close reading of our supplementary code, and for recognizing the scalability and clinical value of the weak supervision, LocusBench, and our multi-perspective validation.

W1. Selection bias in the evaluation set.

We agree this is an important limitation, and will state the GSPS-matching selection step explicitly in the Limitations, framing LocusBench results as performance on pipeline-matchable findings. To measure its effect, we built LocusBench-Merlin (see the comment to all reviewers, "All Reviewers: Expanded External Validation"), which addresses each concern you raised:

  • No pipeline selection. Merlin contains no GSPS annotations, so no case passed through our pipeline, and no measurement or slice reference was required.
  • Not class-balanced. We extracted all positive, clinically actionable findings from randomly sampled studies, rather than a fixed number per category.
  • Quantitative evaluation. 154 findings across 86 studies, with reference masks from three radiologists and all three baselines under the same protocol.

On findings within our existing categories (n=77), LocusCT's hit rate (0.779) is in line with LocusBench-Onc and -ED (0.733, 0.765), indicating that pipeline selection does not substantially inflate our results. On finding types outside all our benchmarks (n=77), LocusCT reaches 0.597, more than twice the best baseline (0.247). We note that each mask was reviewed by one radiologist, so inter-reader variability still cannot be estimated, and we list this as a limitation.

W2 / Q1–Q2. Scoring space and Dice smoothing.

Thank you for identifying both issues. As you observed, the submitted LocusCT results were scored on the resampled grid, while the pretrained comparators were scored at native resolution. We have updated LocusCT's inference so that the model produces a probability map on its resampled grid, which is then resampled to the native resolution before thresholding. All models are now scored on the same cases against the same native-resolution reference masks. We have also removed the smoothing term from all Dice computations. No reference mask on the native grid is empty or contains fewer than 10 voxels (our smallest mask is 20 voxels), so no zero-overlap prediction could be counted as a Hit.

LocusCT hit rates before and after this change:

ModelOnc (submitted)Onc (native)ED (submitted)ED (native)
LocusCT0.7250.7330.7730.765

Hit rate changes by less than 0.01 on both benchmarks, so our conclusions are unchanged. We will update all numbers in the manuscript, and the released code will use a single native-resolution scoring path for every model.

Q3. Training conditions in the data-scaling experiments.

AblationDataTrain tripletsInput channelsSteps per epochSelected epochTotal training steps
Data scale10%10,03431,2544357,684
Data scale30%30,10333,76241165,528
Input channels30%30,10313,76241165,528
Training sourceCurated only23,02632,87838117,998
LocusCT (full)100%100,344312,54321301,032

All models used an effective batch size of 8 (2 per GPU × 4 GPUs), with incomplete final batches dropped. Each run continued for 3 epochs past its best validation Dice, at which point training stopped and the checkpoint with the highest validation Dice was selected. Because each run trained until validation Dice plateaued, smaller data scales were not cut short: they trained for more epochs (41–43) than the full model (21) before converging. The lower performance at smaller scales therefore reflects less data rather than an insufficient training budget. We will add this table to the paper, and the released scripts will include the exact configuration for each run.

Reviewer_GMtrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies 3D visual grounding in abdominal CT and proposes a pipeline for constructing large-scale training data from routine clinical reports and DICOM GSPS annotations.

  • The pipeline extracts positive findings from reports, matches them to GSPS annotations using measurement and slice information, and generates volumetric masks with SAM2CT. This produces approximately 105K phrase–mask–volume triplets for training.
    • The authors also introduce LocusBench, a radiologist-verified benchmark covering oncology and emergency findings, and train a VoxTell-based model called LocusCT.
    • The results show substantial improvements over existing zero-shot methods on both benchmarks.
Critical strengths
  1. The paper addresses a clinically relevant problem. Leveraging annotations already available in routine clinical workflows is a practical way to alleviate the shortage of large-scale supervision for 3D medical visual grounding.
  2. LocusBench covers both oncology and diverse emergency findings, with radiologist verification providing a useful benchmark for evaluation.
  3. The reported performance gains are substantial, and the data-scaling experiments provide further evidence of the usefulness of the constructed training data.
Critical weaknesses
  • The baseline comparisons are not sufficiently controlled.
    • Existing methods are evaluated zero-shot, while LocusCT is trained on 105K task-specific samples.
      • It is therefore unclear how much of the improvement comes from the proposed supervision strategy rather than additional in-domain training.
  • The benchmark may not be representative of general clinical findings.
    • LocusBench is constructed from cases with matching GSPS annotations, which may favor findings that are easier to measure or localize. This limits how broadly the reported results can be interpreted.
  • The evidence for generalization is still limited.
    • The external evaluation on Merlin includes only 78 findings assessed by a single radiologist. Without quantitative ground truth or baseline comparisons on this dataset, it is difficult to assess the robustness of the reported improvements.
Questions for authors
  1. Could the authors provide a more controlled comparison, such as fine-tuning pretrained VoxTell on the same training data and under the same evaluation protocol?
    • It would also be useful to compare supervision using the original report phrases against simpler finding-category descriptions to better understand where the improvements come from.
  2. Please refer the weaknesses.
2 comments

Thank you for the careful review, and for recognizing the clinical relevance of reusing routine annotations, the value of LocusBench, and the data-scaling evidence. We address your three concerns below. The third, and part of the second, are addressed by LocusBench-Merlin, a new external benchmark described in the comment to all reviewers, "All Reviewers: Expanded External Validation."

W1 / Q1. Baseline comparisons are not sufficiently controlled.

We agree the zero-shot comparison alone cannot separate the supervision strategy from in-domain training. In our setting, however, the two are closely linked: our contribution is a way to obtain in-domain grounding supervision at scale without dedicated annotation, and no comparable in-domain training set exists for these comparators. Two results in the paper already isolate the effect of the data itself. First, LocusCT uses the VoxTell architecture but is trained from random initialization, so its performance comes entirely from our weak labels. Second, hit rate on LocusBench-Onc rises from 0.458 to 0.550 to 0.733 with 10%, 30%, and 100% of the training triplets, with the same architecture and protocol.

Following your suggestion, we also fine-tuned pretrained VoxTell on the same training data (10 epochs, lr = 5e-6) and evaluated it under the same protocol:

ModelOnc DiceOnc Hit RateED DiceED Hit Rate
VoxTell (zero-shot)0.1170.1880.1180.209
LocusCT (from scratch)0.4790.7330.4710.765
LocusCT (VoxTell initialization)0.4580.6960.4840.810

This experiment directly answers where the improvement comes from. Training on our data raises VoxTell's hit rate from 0.188 to 0.696 on LocusBench-Onc and from 0.209 to 0.810 on LocusBench-ED (Dice: 0.117 to 0.458 and 0.118 to 0.484), a nearly 4× gain under identical architecture, prompts, and evaluation.

We retain LocusCT (from scratch) as our primary model because its performance is more balanced across the two benchmarks, and note that fine-tuning a pretrained model reaches comparable performance in fewer epochs, a promising direction for applying our data to other architectures. We will add this comparison to the paper.

Report phrases vs. category descriptions. We did not train a model on category-level supervision, and agree this would be an informative ablation that we will note as future work. We deliberately trained on full report phrases to preserve as much report information as possible: condensing a phrase often removes details, such as anatomical location or size, needed to identify which finding is meant.

W2. The benchmark may not be representative of general clinical findings.

We agree, and will state the GSPS-matching selection step explicitly in the Limitations. LocusBench-Merlin was built to measure this directly. Merlin contains no GSPS annotations, so no case passed through our pipeline, and findings were extracted from report Impressions with no requirement for a measurement or slice reference, directly addressing the concern that LocusBench favors findings that are easy to measure. On findings in our existing categories (n=77), LocusCT's hit rate (0.779) is in line with LocusBench-Onc and -ED (0.733, 0.765), indicating that pipeline selection does not substantially inflate our results.

On coverage, we agree many finding types fall outside LocusBench. The oncology benchmark spans all oncologic findings, and the 13 ED categories were selected by radiologists as most useful for visual grounding. To go beyond them, half of LocusBench-Merlin (n=77) consists of finding types none of our benchmarks cover (e.g., fractures, fluid collections). LocusCT reaches 0.597 on these, more than twice the best baseline (0.247).

W3. The evidence for generalization is still limited.

We agree the original Merlin study was limited, and LocusBench-Merlin replaces it as our primary external evaluation. It has 154 findings across 86 studies, reference masks from three radiologists, and all three baselines under the same protocol. LocusCT localizes 68.8% of findings versus 26.0% for the best baseline (VoxTell), a 2.6× margin, and leads in every finding group.

The fine-tuning comparison addresses my concern about the baselines.

My main concern now is that the external reference masks were initialized from LocusCT predictions, with only one reader reviewing each case. This could favor LocusCT.

I prefer to keep my original rating.

Reviewer_3vmtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses routine clinical annotations stored as DICOM GSPS objects (measurement lines and arrows placed by radiologists during interpretation) to create weak supervision for 3D visual grounding in abdominal CT. An LLM extracts single-finding sentences and matching metadata from reports. GSPS objects are linked to these sentences using measurements or slice references and converted to volumetric masks with SAM2CT. The resulting dataset contains 105K phrase-mask-volume triplets from 65K volumes (59K exams) at one institution. The authors also introduce LocusBench, two radiologist-verified test sets with 500 findings in total, and LocusCT, which achieves hit rates of 0.725 and 0.773, compared with at most 0.209 for the zero-shot baselines.

Critical strengths

• The paper makes good use of a supervision signal that is already produced during clinical care but is usually discarded. Linking PACS annotations to report phrases through metadata and converting them to 3D masks yields 105K finding-level triplets without dedicated annotation. I found the per-stage yields and failure-mode analysis particularly useful: they make clear how much data is lost at each step and where the pipeline can be improved. • The data-scale ablation directly supports the paper's main claim. Hit rate on LocusBench-Onc increases from 0.458 to 0.550 to 0.725 with 10%, 30%, and 100% of the training triplets. It is also informative that the full, noisier dataset outperforms the curated subset, suggesting that scale can outweigh some label noise in this setting. • LocusBench is a useful resource in its own right: it contains 500 expert-verified findings and covers both oncology cases and 13 emergency-department categories. The quality control and error analysis is also solid. First-pass review quantifies the weak-label quality (mean scores of 4.27/4.32 out of 5), and the authors report per-category failures such as diverticulitis (hit rate 0.35) as well as the bimodal Dice distributions.

Critical weaknesses

• The main evaluation is restricted to cases that the weak-labeling pipeline already handled successfully, which limits what the reported numbers say about deployment. All LocusBench cases were first matched and segmented by the pipeline before radiologist review. Findings without routine annotations, without a matchable report sentence (15.2% of reports), or without a successful GSPS match (57.4% of GSPS objects are unmatched) cannot enter the benchmark. The test cases therefore come from a distribution close to the successfully matched portion of the training data. This does not affect the quality of the radiologist-verified labels, but it may make the 0.725/0.773 hit rates optimistic for arbitrary report findings. The limitations mention findings that radiologists leave unannotated, but not this additional selection step. A small evaluation on report findings sampled regardless of pipeline success would help quantify the effect. The Merlin study is useful initial evidence, but 78 findings rated by one radiologist, without masks or a comparator under the same protocol, are not enough to establish cross-institution generalization.

Questions for authors

What prompts and preprocessing did the zero-shot comparators receive? Please state whether SegVol and BiomedParse-v2 were prompted with the full report sentence or a category label, and ideally report both settings. This determines whether the zero-shot hit rates at or below 0.21 measure grounding ability or vocabulary mismatch.

Will a complete, runnable code release be provided? The current supplementary is missing the training/, utils/, and train modules imported by its own entry points, as well as the scripts/embedding/ files named in requirements.txt. Please also state in the paper: the text encoder used by LocusCT, the train/validation split sizes, and the criterion used for model selection and early stopping.

1 comment

Thank you for the thoughtful review, and for highlighting the value of reusing routine clinical annotations, the data-scale ablation, and LocusBench as a resource.

W1. LocusBench is restricted to cases the weak-labeling pipeline handled successfully.

We agree this is an important point, and that our Limitations should state this selection step explicitly. To measure its effect, we built LocusBench-Merlin (details in the comment to all reviewers, "All Reviewers: Expanded External Validation"): 154 findings from the external Merlin test set, with reference masks from three radiologists and comparisons against all three baselines. Merlin contains no GSPS annotations, so no case passed through our pipeline, and findings are not restricted to our predefined categories.

On LocusBench-Merlin findings within our existing categories (ED/Onc, n=77), LocusCT's hit rate is 0.779, in line with its LocusBench-Onc and LocusBench-ED results (0.733 and 0.765). This suggests the LocusBench results are not substantially inflated by pipeline selection. On finding types outside all our existing benchmarks (n=77), LocusCT reaches 0.597, versus 0.247 for the best baseline. We will state the GSPS-matching selection step in the Limitations, framing LocusBench results as performance on pipeline-matchable findings and LocusBench-Merlin as performance on findings sampled independently of the pipeline. LocusBench-Merlin replaces the original single-reader Merlin study as our primary external evaluation.

Q1. What prompts and preprocessing did the zero-shot comparators receive?

(a) Prompts. In the submission, all comparators and LocusCT were prompted with the same verbatim report sentence. These sentences contain content the comparators were not trained to interpret (e.g., measurements, de-identification tags, references to prior exams), so part of the performance gap could reflect vocabulary mismatch. At the same time, our findings require more specificity to localize (e.g., anatomical location) than the comparators' class vocabularies provide. To separate these effects, we used an LLM to generate a short label (≤ 4 words) from each sentence, typically an anatomical location plus a finding category (e.g., "periumbilical ventral hernia", "esophagogastric junction mass"), and reran all comparators with these labels:

ModelOnc (Sentence)Onc (Label)ED (Sentence)ED (Label)
SegVol0.0440.106 (+0.062)0.0370.078 (+0.041)
BiomedParse-v20.1580.180 (+0.022)0.1820.190 (+0.008)
VoxTell0.1880.221 (+0.033)0.2090.217 (+0.008)

Short labels improve every comparator, but only modestly. The best label-prompted result (0.221) remains far below LocusCT (0.733 Onc, 0.765 ED), which indicates model capability is the primary reason for the performance gap. We will add this table to the Appendix and report the label-prompted results as the main comparator results; LocusBench-Merlin uses the same label prompts for all baselines.

(b) Preprocessing. All comparators received the same CT volumes as LocusCT, preprocessed following each model's official repository and paper. We will list the per-model preprocessing steps in the Appendix.

Q2. Will a complete, runnable code release be provided?

Yes. We thank the reviewer for checking the supplementary closely and apologize that it was incomplete. The full release will include the missing modules (training/, utils/, the train module, and scripts/embedding/), along with complete instructions to reproduce training and evaluation.

Q3. LocusCT training details.

We will add the following to the paper:

  • Text encoder: Qwen3-Embedding-4B (frozen).
  • Train set: 100,344 triplets (49,450 patients).
  • Validation set: 2,537 triplets (1,503 patients).
  • Model selection: Models were trained until validation Dice had no increase after 4 epochs (epoch 21 was selected for LocusCT).

Comments on the paper

2 comments
AuthorsOpenReview ↗

All Reviewers: Expanded External Evaluation

Thank you to all reviewers for their thoughtful reviews. All reviewers raised concerns about the limited external validation of LocusCT, and we agree these concerns were valid. In response, we built LocusBench-Merlin, a new external benchmark with radiologist-approved reference masks, sampled independently of our weak-labeling pipeline and not restricted to predefined finding categories. LocusCT localizes 68.8% of findings versus 26.0% for the strongest baseline, and outperforms every baseline both on finding types covered by our existing benchmarks and on types none of them cover.

The shared concerns were:

  1. Pipeline selection. Every LocusBench case passed GSPS–report matching, so findings without a routine annotation, a matchable report sentence, or a successful match could not enter the benchmark.
  2. Limited coverage. LocusBench-ED and the original Merlin study cover 13 predefined categories.
  3. Limited evaluation. The original Merlin study had 78 findings rated by one radiologist, with no reference masks or baselines.

LocusBench-Merlin addresses each: no case passed through our pipeline (1), half its findings fall outside all existing benchmark categories (2), and it has reference masks from three radiologists and three baselines (3).

Original Merlin study. A radiologist rated LocusCT's predictions for 78 report phrases (6 per ED category) as useful (score ≥ 3 of 5) in over 75% of cases. The task mirrors clinical use: given a finding documented in a prior report, the model localizes it and the radiologist judges whether it helped them find it. We view this as initial evidence of transfer, but agree it is limited: single reader, 13 categories, no reference masks, and no model comparison.

LocusBench-Merlin.

  • Findings. From randomly sampled Merlin test studies, Qwen3.8-27B extracted all positive, clinically actionable Impression findings and their supporting Findings sentence(s), used as the grounding query. Merlin has no GSPS annotations, so no case passed through our pipeline.
  • Exclusions. We excluded organ-level findings (e.g., hepatomegaly), which organ segmentation models already handle, and findings describing many lesions, keeping only findings that identify a single object so each has one correct location. No finding was excluded because a model failed to localize it.
  • Reference masks. Radiologists started from LocusCT's prediction, which they could edit or delete and redraw. Each mask was reviewed by 1 of 3 radiologists.
  • Scale. 154 findings across 86 studies. ED/Onc (n=77): oncologic findings or one of the 13 ED categories. Other (n=77): all remaining types (e.g., fractures, fluid collections), covered by none of our existing evaluations. Groups were assigned manually; the equal split was not intentional.

Results. Hit rate = fraction of targets with Dice ≥ 0.1. LocusCT is queried with the full supporting sentence(s), its intended use. Following reviewer feedback, baselines are queried with shortened phrases (≤ 4 words), their best format on LocusBench-Onc/ED.

ModelOverall (n=154)ED/Onc (n=77)Other (n=77)
LocusCT0.6880.7790.597
VoxTell0.2600.2730.247
SegVol0.0520.0520.052
BiomedParse-v20.1040.1040.104

LocusCT localizes 68.8% of findings, 2.6× the strongest baseline. On ED/Onc findings, its hit rate (77.9%) is in line with LocusBench-Onc and -ED (73.3%, 76.5%) even though no case passed through our pipeline, indicating LocusBench results are not an artifact of pipeline selection. On Other findings, LocusCT still localizes 59.7%, more than twice the best baseline (24.7%). This is lower than on ED/Onc, as expected outside the training categories, but LocusCT leads by a wide margin in every group.

Remaining limitations. Reference masks were initialized from LocusCT predictions, which may anchor radiologists on which structure is the finding, not just its boundaries. Radiologists relocated the finding in 48 of 154 cases, so predictions were not accepted uncritically, but anchoring cannot be ruled out; we treat the absolute hit rate as an upper bound and the margin over baselines as the main result. Also other limitations remain: one reader per mask, LLM-extracted findings, Dice ≥ 0.1 does not penalize over-segmentation, and single-institution abdominal CT.

Changes to the manuscript.

  • Add LocusBench-Merlin to Section 5 as our primary external evaluation; move the original reader study to the appendix.
  • Report hit rates at stricter Dice thresholds in the appendix.
  • Clarify the GSPS-matching selection step in Limitations, framing LocusBench as performance on pipeline-matchable findings and LocusBench-Merlin as performance on findings sampled independently of it.
  • Add the limitations above to the Limitations section.
AuthorsOpenReview ↗

All Reviewers: Code, Model, and Benchmark Release

To support reproducibility, we will release the following upon publication:

  • Code: complete, runnable code for LocusCT training and evaluation, including the modules missing from the submitted supplementary and the exact configuration for every run reported in the paper.
  • Model: trained LocusCT weights.
  • LocusBench: grounding queries and radiologist-approved reference masks for all three benchmarks: LocusBench-Onc (240 findings), LocusBench-ED (260), and LocusBench-Merlin (154), totaling 654 findings. LocusBench-Merlin is built on the publicly available Merlin dataset; LocusBench-Onc and LocusBench-ED will be released as de-identified volumes, subject to institutional data-sharing approval.

The training data are drawn from institutional clinical records and cannot be released, but the pipeline can be applied to any institution's PACS data that include GSPS annotations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.