acceptodds
Withdrawn from ICLR 2027

RadGenome-Anatomy: A Large-Scale Anatomy-Labeled Chest Radiograph Dataset via Physically Grounded Volumetric Projection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Anatomical structure labels for radiographs are essential for medical image segmentation and a broad range of downstream diagnostic tasks. However, annotating anatomy directly on 2D radiographs is intrinsically ambiguous, as 3D anatomical structures are projected onto a single 2D plane where boundaries may overlap, be occluded, or appear only partially visible. These ambiguities complicate the construction of datasets with broad anatomical coverage and consistent labels. To address these limitations, we introduce ***RadGenome-Anatomy***, the largest anatomy-labeled chest radiograph dataset, containing over **10 million** segmentation masks across **210** anatomical structures in studies. It is constructed by projecting large-scale 3D anatomical masks from CT volumes into 2D radiographic space through canonical radiographic geometry. This shifts annotation from directly tracing uncertain 2D boundaries to defining anatomy in volumetric space, where structures that overlap or become partially invisible in radiographs remain spatially separable. As a result, each 2D mask represents the physically grounded projected footprint of a volumetrically defined structure. The scale and broad anatomical coverage of *RadGenome-Anatomy*, including structures that are overlapping, partially visible, or difficult to delineate directly, enable research on geometric measurements as explicit evidence for radiographic interpretation. We demonstrate this by training *XAnatomy* to predict structure-specific masks and derive clinically relevant measurements, achieving diagnostic accuracies of , , and for cardiomegaly, kyphosis, and scoliosis, respectively.

Primary area:
datasets and benchmarks
Keywords:
Image Segmentation, Anatomy Dataset, Radiographic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BgWXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RadGenome-Anatomy, a dataset for anatomy segmentation on chest X-rays. It is built from RadGenome-ChestCT, which provides organ-level segmentation masks. The authors project the 3D CT volumes and their masks into synthetic PA and LL radiographs. They benchmark several segmentation models on the dataset and demonstrate how predicted masks can support anatomical measurements for assessing cardiomegaly, scoliosis, and thoracic kyphosis.

Critical strengths
  1. The dataset addresses a meaningful annotation challenge by providing geometrically consistent masks for overlapping anatomical structures that are difficult to annotate directly on chest X-rays.
  2. The real-image evaluation and measurement-based applications provide initial evidence that the dataset can support anatomy segmentation.
Critical weaknesses
  1. Limited novelty. The projection and measurement pipeline follows prior work such as PAX-Ray++. The main contribution is expanded dataset scale and anatomical coverage.
  2. Insufficient label quality validation. The manual correction process is unclear, and direct expert assessment of the projected labels is lacking. Real-image validation mainly covers the lungs and heart.
  3. Insufficient evidence of practical value. The comparison with VLMs does not isolate the dataset’s contribution, as the segmentation-based pipeline uses task-specific geometric rules. The paper does not establish whether the dataset improves performance over strong chest X-ray segmentation baselines or adds value beyond existing training data.
Questions for authors
  1. ChestCT provides segmentation masks for 197 categories, whereas this dataset contains 210 classes. What accounts for this difference, and how were the additional or modified labels generated?
  2. Who performed the manual corrections shown in Figure 1? Were clinicians involved, and what proportion of the data was reviewed? How was label quality assessed?

Minor presentation issues include a repeated sentence in the Introduction (lines 42–44), an awkward paragraph break around line 182.

Reviewer_tJVUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses a dataset of 3D segmentations and CT scans to generate 2D anatomy segmentation labels by rendering 2D digitally reconstructed radiographs (DRR) and projecting the 3D anatomy masks to the 2D DRRs. The author's benchmark 19 segmentation models on the DRRs and evaluate zero shot generalization to real radiography for lung and heart segmentation. The authors also demonstrate using the trained segmentation model to compute the cardiothoracic ratio and the kyphosis and scoliosis Cobb angles and compare performance to zero-shot VLMs.

Critical strengths

The authors make a strong case to justify the clinical relevance of explicit 2D segmentations to perform interpretable measurements for certain clinically relevant measures as opposed to relying entirely on VLM analysis of medical images.

Critical weaknesses

The novelty is minimal and overclaimed. Using CT volumes and 3D annotations to generate 2D segmentation labels by rendering DRRs is not a "new annotation paradigm". The authors acknowledge that prior work has already demonstrated this approach, such as Seibold et al. (2023). The contribution is only in the scale of the dataset.

The sim-to-real gap is not sufficiently addressed. DRRs are different in appearance to real 2D radiographs. It appears that the authors are using a very simple imaging model, where the DRRs are created with a parallel beam projection rather than the cone-beam geometry of a real X-ray imager, and do not include effects like scatter, detector noise, beam hardening, and other effects that have been modeled in prior work on generating high quality DRRs. The authors also claim that their annotation method "replaces" the need for 2D annotation, but only support this claim through experiments on lung and heart annotations, which are large coarse structures.

Questions for authors

The authors should consider adding a baseline with the XAnatomy architecture trained on a dataset with manual 2D annotations, e.g. a held out split or cross-dataset to support their claim that their annotation method "replaces" the need for 2D annotation.

The authors should also consider validation on real radiographs for smaller or more difficult structures beyond lungs/heart.

Could the authors include more detail on the XAnatomy architecture and training method?

Reviewer_y9efOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper generates large amount of DRRs (PA and LAT) and corresponding 2D segmentations by projecting the CT Segmentation dataset proposed earlier (RadGenome-Chest CT) consisting of 210 class labels (left/right are different classes including individual ribs). This DRR dataset is used to benchmark 19 2D segmentation architectures on the validation subset of the same CT dataset-projected DRRs. On the real X-ray transfer performance front, a zero-shot transfer of Lung and Heart segmentation performance on three real CXR datasets is tested. Additionally, the segmentation model is used to measure CTR, kyphosis and scoliosis angle and then use these measurements to predict cardiomegaly and kyphosis and scoliosis grades. The segmentation and then geometric measurements are ultimately used to perform coarse classification (for. e.g mild, moderate and severe or cardiomegaly present or absent).

Critical strengths
  • Architecture and anatomy coverage Benchmarking: the papers benchmarks 19 segmentation architectures acroos architecture families giving the community a broad starting point.
  • Attempt at real-world validation: the authors evaluate a model trained on DRRs only on real X-rays, which is the right question to ask for a (somewhat) synthetic dataset (but it is not new: Seibold 2022,2023 did this).
Critical weaknesses

Real-world evaluation is zero-shot only; linear probing and fine-tuning are missing. Based on the zero-shot results, the paper positions the model as a "generalist", out-of-the-box segmenter for real X-rays. Given the domain-shift gap, the realistic use of such a model is as a starting point that is cheaply fine-tuned on a real dataset. The paper does not show that this beats training from scratch or is comparable to other methods(beyond zero-shot evaluation against VLMs), so the evaluation falls short of supporting the dataset's practical value. Given the reduced performance reported in the paper and the obvious domain gap that DRRs do not model, the performance is bound to reduce (in this case, significantly).

more rigor needed to justify the contribution. Using X-ray projections generated from large scale CT Datasets to obtain pseudolabels to leverage X-ray anatomy segmentation has been shown to be useful earlier, as also mentioned in the paper (Seibold 2022, 2023). The contribution here is benchmarking across architectures, so the experimental rigor and transparency matter more. I would have liked a more rigorous experiment design including external CT dataset cohort, more anatomy coverage in real-world evaluation, additional baseline comparison for real-world evaluation, fine-tuning experiments, and better ground truths. The real world evaluation is the exact same setup as in Seibold 2023 that only test the three labels (lung and heart) for CTR and vertebra for scoliosis angles out of the 210 anatomy labels. Hence, the thoroughness of the real-world evaluation section is lacking.

Documentation, rigor and reproducibility are insufficient.

  • Manual correction. It is mentioned only in passing in figure 1. The text does not say how many cases or which organs were corrected, or how substantial the corrections were.
  • Geometric measurements. Clinically relevant metrics such as CTR and scoliosis angles are hard to measure automatically, especially for abnormal anatomy, which is the subgroup of most interest. The paper acknowledges this only qualitatively. It does not report how many cases admitted automatic measurement, what compromises were needed, or what quality control was performed.
Questions for authors

(S) Please add linear-probing and fine-tuning experiments on real radiograph datasets, compared with training from scratch and with existing methods, to show the value of RadGenome-Anatomy as a pre-training resource.

(S) Please explore going beyond CTR and spine angles to show the value of XAnatomy to "derive clinically relevant measurements" from the generated masks for the other anatomies as well. The current clinical geometric measurement evaluation is also only at coarse level of mild, moderate, severe rather than precise evaluation.

(C) How many 3D masks were manually corrected, for which organs, by whom, and how large were the corrections?

(C) For CTR, kyphosis and scoliosis, how many cases allowed automatic measurement? What compromises were needed, particularly for abnormal anatomy, and was any quality control performed on the measurements?

(C) Was the aspect ratio of the images maintained? The appendix figure suggests the images were squashed to a square. This affects downstream geometric measurement.

(C) Describe the XAnatomy Architecture? Is it the same as the best performing architecture in the benchmark?

Figure and Table Readability Table 2 (dataset stats) is largely redundant and poor use of space. It could have been two or three sentences instead. Figure 2 is hard to read, labels are far too small to read. Only a few inner-ring and outer sectors are readable and many of these sectors do not have labels at all. The readability issue of labels also occur in figure 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.