acceptodds
Under review as a conference paper at ICLR 2027

DART: Diffusion for Anatomical Repair with Text

Abstract

Current state-of-the-art segmentation models still produce anatomically implausible shapes, such as broken vessels and fragmented glands, on the structures that are hardest to segment. Learned shape representations capture what structures look like and can help correct these errors, but most cover a fixed set of anatomical classes. Extending this coverage across classes and imaging modalities is costly because it often requires new voxel-level annotations. We propose Diffusion for Anatomical Reshaping with Text (DART), which learns a shared shape representation from segmentation probabilities. It uses anatomical descriptions to correct these probabilities without receiving medical images. Across 13 public benchmarks, absolute Dice gains reach 31% for individual gallbladder cases in CT and 10% for individual prostate cases in MRI. Using existing segmentation predictions, we scale a single shape representation from 50 to 377 anatomical classes without new voxel-level annotations for correction training. The shared representation also corrects four lesion classes absent from training and transfers from CT to MRI without retraining. Corrected labels also improve segmentation models, raising one model’s Dice from 57.1% to 67.5%. The same pipeline supports annotations for 45,000 CT scans. These results suggest that sharing shape information within and across classes can reduce the need for new voxel-level annotations when extending segmentation to more anatomical classes and imaging modalities. Code and models will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Shape prior, diffusion model, segmentation, generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vGk9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DART, a text-conditioned diffusion model for correcting medical segmentation predictions. DART takes a single-class probability map and an anatomical description, adds noise to an intermediate timestep, and then applies reverse diffusion to obtain a corrected mask. The model is trained on predicted probability maps from CT-family datasets, with a vocabulary of 377 anatomical classes. The experiments cover different upstream segmentation models, transfer to MRI and unseen classes, and downstream training with corrected labels. The paper also presents DART-45K, an annotation resource produced through a pipeline combining specialist models, DART correction, and DART-Seg.

Critical strengths

(1)The method addresses a useful problem. A correction model that works on segmentation probabilities can be applied after different upstream models without requiring the original medical image. The automatic CT experiments and MRI results provide useful evidence for this design. For example, Appendix C reports an improvement from 74.1 to 82.3 Dice for MRI pancreas segmentation.(2)The broad anatomical coverage and the use of multiple controls strengthen the evaluation. In the 90-class TotalSegmentator comparison, mean Dice increases from 73.6 to 75.6, and DART performs best on 77 classes. The identity, volume-matched identity, pure-noise, and traditional postprocessing controls help check whether the improvements can be explained by preprocessing or simple shape operations. The inclusion of connectivity and boundary metrics is also helpful.(3)The corrected labels improve downstream segmentation models. All fifteen configurations in Table 8 improve when trained with DART-corrected labels. The gain is therefore not limited to one student architecture. The annotation pipeline and the audit of 10,000 scans also represent a substantial effort and could be useful to the community.

Critical weaknesses

Major concerns (1)DART and the single-pass U-Net baseline are not given equivalent input information. Section 4 and Appendix M state that DART receives soft probability maps, while the single-pass U-Net receives thresholded masks. Thresholding removes uncertainty information that remains available to DART. Using the same backbone and text conditioning does not remove this difference. The comparison therefore cannot show how much of the improvement comes from diffusion or partial noising, rather than from retaining the original probabilities. A strong single-pass model trained with probability inputs is an important missing baseline. (2)The text-versus-index ablation mixes the effect of conditioning with class coverage. Table 6 reports 74.20 Dice for text conditioning and 69.10 for index conditioning. However, Appendix J explicitly states that the evaluation includes classes outside the index model’s vocabulary. This shows an advantage of text for representing unseen classes, but it does not isolate the benefit of anatomical semantics. The mismatch and paraphrase experiments do not fully answer this question either. A model that mainly uses the text embedding as a class identifier could also perform worse with a wrong description and remain stable under similar paraphrases. (3)I could not trace how the training probability maps were generated or how parameters were chosen for unseen classes. Appendices A and B describe the datasets, but it is difficult to reconstruct which upstream model and checkpoint generated the training probabilities for each anatomical class. The specialist models listed in Appendix N describe the 45K annotation pipeline and do not directly resolve this question about DART training. There is also an unclear example in Section 3.3: the hippocampus provides the noise level for a hippocampal tumor, while Figure 4a lists the hippocampus itself as an unseen task. The source of this parameter should be explained. (4)The vocabulary-scaling control only evaluates the original 50 classes. In Table 5, the 50-class and 377-class models are both tested on the same original 50 classes, obtaining 73.85 and 74.20 Dice. This supports the claim that expanding the vocabulary does not reduce the old-class mean. It does not show how the added classes perform, whether low-frequency classes benefit, or whether some old classes become worse despite the stable average. Other experiments do evaluate more classes, but the controlled vocabulary experiment still gives limited evidence about what is gained by sharing one model across 377 classes. (5)Preservation of abnormal anatomy needs more direct evaluation. The KiTS23 and lesion-transfer results are useful, but an improvement in whole-organ Dice does not necessarily mean that a small abnormal region is preserved. For example, the reported kidney Dice improves from 91.8 to 95.1, without a separate measure of how much of the tumor region remains covered by the corrected kidney mask. Since DART learns a distribution of predicted shapes, it may favor common shapes during correction. The central uncertainty is when correction removes a genuine segmentation error and when it might suppress uncommon but valid anatomy.

(6)The overview in Figure 2 looks clearly AI-generated. It should be redrawn. (7)The baseline comparisons are less complete in the automatic and MRI settings. The most complete 90-class comparison uses MedSAM predictions obtained with reference-derived bounding boxes. The paper also includes automatic-input experiments, but Figure 3 aggregates selected structures, and the external tables do not provide the same complete comparison against learned correction methods. Appendix M indicates that additional external baselines were evaluated. Reporting these results would help determine whether DART remains better than other correction methods under the transfer settings, beyond simply improving over the original input.

Questions for authors

(1)Could the authors train the single-pass U-Net with soft probability inputs, using the same backbone, text conditioning, subject splits, and a comparable training budget? Does DART still show a consistent advantage? A DART variant trained and evaluated with binary inputs would further help separate the effect of input information from the correction procedure.

(2)What is the text-versus-index difference when evaluation is restricted to classes covered by both models, with matched training examples? Please report unseen classes separately. Could the authors also compare class-name conditioning with full anatomical descriptions under matched training conditions, to test whether the shape and location descriptions provide additional benefit?

(3)Please provide the actual training, validation, and test counts, together with the upstream model and checkpoint used to generate the training probabilities for each class. Please also specify the crop margin and the source of (t^*) for every unseen task. In particular, how is the hippocampus noise level obtained in the example in Section 3.3?

(4)For the classes with reference annotations, could the authors report results separately for the original and added classes, and for frequent and rare classes? Which original classes become worse after expanding the vocabulary? For the positive-transfer claim, a small comparison with separately trained class groups would be more informative.

(5)On the existing abnormal-anatomy cases, what fraction of cases becomes worse after correction, and how large are these decreases? Could the authors report coverage of the annotated abnormal region by the corrected organ mask, together with representative failure cases? Repeated sampling on a small subset would also help establish whether these findings are stable.

(6)Please replace Figure 2 with a redrawn technical overview of the method.

(7)Please provide the external baseline results by dataset and upstream model, including paired differences against the strongest correction baseline and uncertainty on those differences. On representative automatic CT and MRI tasks, does DART retain its advantage over the probability-input baseline requested in Question 1?

10 comments
AuthorsOpenReview ↗

Response to Reviewer vGk9 (1/6): Matched Probability Inputs and Binary-Input DART

Thank you for your detailed and constructive review. We appreciate your recognition of DART's usefulness as an image-independent correction framework, its broad anatomical coverage and controls, and the improvements obtained when downstream models train on DART-corrected labels. Your questions have helped us identify important comparisons and reporting details to strengthen.

We respond to each of your seven concerns and corresponding questions below. In particular, we have completed new probability-matched and binary-input experiments, an additional 327-class evaluation, and case-level tumor analysis. We distinguish completed evidence from analyses that will be added in the revision.

Q1. Probability inputs versus binary-mask baselines

Concern: DART receives soft probability maps, whereas the original single-pass U-Net receives thresholded masks. Does the advantage persist when input information is matched?

Question 1: Could the authors train the single-pass U-Net with soft probability inputs, using the same backbone, text conditioning, subject splits, and a comparable training budget? Does DART still show a consistent advantage? A DART variant trained and evaluated with binary inputs would further help separate the effect of input information from the correction procedure.

Thank you for identifying this legitimate confound. We agree that the original U-Net comparison did not isolate the advantage of retaining soft probabilities from that of DART's correction procedure. Following your suggestion, we conducted two complementary experiments.

Probability-matched U-Net. We trained a single-pass U-Net with the same backbone, soft probability-map inputs, anatomical text conditioning, training data, subject splits, and training settings as DART. We evaluated on approximately 500 cases from five datasets—TotalSegmentator, AMOS22, FLARE22, PDDCA, and MSD Pancreas—with upstream predictions generated by TotalSegmentator and MedSAM.

Table R1. Mean Dice across the five-dataset matched comparison.

MethodMean Dice (%)Gain (points)
Original predictions79.3—
Probability-matched single-pass U-Net82.1+2.8
DART85.6+6.3

DART exceeds the probability-matched U-Net by 3.5 Dice points. We also tested the same TotalSegmentator cases highlighted in the manuscript: DART outperformed the matched U-Net on 88 of 90 organs. We will include the full per-organ comparison in the revision.

Binary-input DART. We also trained and evaluated a DART variant using thresholded binary inputs rather than the full probability maps. Mean Dice remained within approximately 1–2% between the two DART variants. However, retaining probabilities improved normalized surface Dice (NSD) by up to 20% on selected organs, showing that the additional uncertainty information is particularly useful for boundary refinement.

Together, these experiments establish that probability information helps boundary quality but does not explain DART's entire advantage over the single-pass model. We do not claim that they separately isolate partial noising, iterative denoising, and the diffusion training objective; rather, they isolate the probability-input confound and support the advantage of DART's overall diffusion-based correction procedure. We will add both experiments, with per-organ Dice and NSD results, to the revision.

AuthorsOpenReview ↗

Response to Reviewer vGk9 (2/6): Vocabulary Coverage and Anatomical Text Semantics

Q2. Text conditioning, class coverage, and anatomical semantics

Concern: Table 6 mixes text conditioning with class coverage, because the index-conditioned model cannot represent classes outside its training vocabulary. Prompt mismatch and paraphrase tests do not fully isolate descriptive semantics.

Question 2: What is the text-versus-index difference when evaluation is restricted to classes covered by both models, with matched training examples? Please report unseen classes separately. Could the authors also compare class-name conditioning with full anatomical descriptions under matched training conditions, to test whether the shape and location descriptions provide additional benefit?

Thank you for this important methodological distinction. We agree that Table 6 measures a combination of vocabulary extensibility and anatomical conditioning, and that its overall difference cannot be attributed solely to the semantic content of descriptions.

Table 6 evaluates TotalSegmentator, AMOS22, and WORD cases, including classes outside the learned index vocabulary:

ModelMean Dice (%)Gain (points)
Original predictions68.71—
Index-conditioned69.10+0.39
Text-conditioned DART74.20+5.49

The 5.10-point gap is not a pure estimate of semantic understanding. Instead, it demonstrates a practical advantage of an extensible conditioning interface: a frozen text-conditioned model can receive a description of a new class, whereas an index embedding has no learned entry for it. We do not claim that detailed text necessarily improves Dice on familiar classes.

We have also conducted a preliminary class-name-only versus full-description comparison. On the tested subset, full anatomical descriptions did not provide a substantial improvement over class names alone. We will report this observation transparently; it suggests that representations of anatomical identity may account for much of the observed conditioning benefit.

At the same time, Figure 4(a) documents correction on seven transfer tasks excluded from DART training, including four lesion classes and six MRI tasks. For example, hippocampus improves from 78.0% to 81.4%, hippocampal tumor from 73.0% to 78.1%, and pancreatic cyst from 55.2% to 59.1%. These findings demonstrate zero-shot transfer, not a causal decomposition of the text embedding.

We additionally tested an incorrect full anatomical description on the held-out hippocampus task. Dice fell from approximately 81.4% with the correct description to 76.3% with the incorrect one (a 5.1-point decrease). Appendix E similarly finds near-invariance to correct paraphrases (−0.02 points on average) but a 2.07-point decrease under mismatched descriptions. These tests establish sensitivity to conditioning, including outside the training vocabulary, but changing a whole description also changes class identity.

We agree that the specific question about shape and location semantics remains unresolved by the current experiments. In the revision we will add (i) a text-versus-index evaluation restricted to classes seen by both models with matched training examples, (ii) separate seen- and unseen-class results, and (iii) an expanded class-name-only versus full-description analysis. We will additionally preserve the class name while perturbing individual words, shape descriptions, and location cues, on both seen and unseen classes.

We appreciate the suggestion and will clarify that vocabulary coverage, prompt sensitivity, and independently measured semantic contribution are distinct claims.

AuthorsOpenReview ↗

Response to Reviewer vGk9 (3/6): Training Provenance, Data Splits, and Noise Levels

Q3. Training probability maps, reproducibility, and noise-level transfer

Concern: The manuscript does not make it possible to trace the upstream models/checkpoints used to generate training probability maps, the dataset splits, or the source of noise levels on unseen tasks. Section 3.3 appears inconsistent with the unseen hippocampus task in Figure 4(a).

Question 3: Please provide the actual training, validation, and test counts, together with the upstream model and checkpoint used to generate the training probabilities for each class. Please also specify the crop margin and the source of t* for every unseen task. In particular, how is the hippocampus noise level obtained in the example in Section 3.3?

Thank you for identifying these reproducibility gaps. We agree that listing the datasets is not sufficient to reconstruct which upstream prediction model and checkpoint generated each class's training probabilities.

Training-probability provenance. We initially considered approximately 25–30 segmentation models, drawing on reported performance and independent evaluations including Touchstone (Bassi et al., NeurIPS 2024), as well as published results for TotalSegmentator, AtlasNet, MOOSE, CADS, and other systems. We prioritized strong class-specific upstream performance rather than relying on one universal segmentation model. As Appendix A reports, DART was trained on predicted probability maps from 3,066 volumes across eight public datasets.

We will add a reproducibility table giving, for every class, the source dataset, upstream model and exact checkpoint, and will distinguish those training-prediction sources from the specialist models employed later in the separate DART-45K annotation pipeline (Appendix N). We will also report exact subject-level training/validation/test counts and numerical preprocessing crop margins, replacing the current approximate description of dataset-specific partitions.

How t* was chosen. For classes with validation calibration, t* was selected once from the specified candidate noise levels on validation data, then fixed across test cases and upstream models. For an unseen class, we reused the calibrated noise level of the closest anatomically similar class. The anatomical analogue was selected based on structural similarity; this was a nearest-anatomy matching decision, not an additional optimization on target-task test labels. We will provide a mapping of each unseen task to its source class and fixed transferred t* so that this procedure is fully reproducible. We will not describe the matching step as an automatically learned parameter predictor.

Hippocampus clarification. We agree that Section 3.3 is inaccurate as written. Figure 4(a) states that the hippocampus was not in the DART training vocabulary, so the manuscript's example should not imply that a trained hippocampus class supplied a calibrated t* for hippocampal tumor. For these held-out tasks, the levels were instead transferred from anatomically similar classes with existing calibrations, without choosing t* using their test annotations. We will correct the example and explicitly identify the actual source class and numerical t* for both hippocampus and hippocampal tumor in the revision.

These details are not yet fully specified in the current manuscript; we do not want to invent checkpoint IDs, split counts, crop margins, or the particular source class in this reply. We appreciate the request and will supply the exact provenance and parameter mapping in the revised appendix.

Reviewer_tjS8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DART uses a text-conditioned 3D diffusion model to correct medical image segmentation predictions. It learns a shared anatomical shape representation from segmentation probability maps and uses anatomical descriptions to guide correction without accessing the original medical image. The model is trained across 377 anatomical classes and evaluated on 13 public benchmarks, including transfer from CT to MRI and to lesion classes not seen during training. The paper also studies vocabulary scaling, text conditioning, different upstream segmentation models, and the use of DART-corrected masks as training labels for downstream segmentation models.

Critical strengths
  1. DART uses one shared model across 377 anatomical classes and transfers from CT to MRI and to unseen lesion classes without retraining.
  2. The corrected masks are useful beyond post-processing. Models trained on DART-corrected labels improve across a wide range of segmentation architectures, showing that the learned corrections can provide useful supervision for downstream training.
Critical weaknesses
  1. DART receives the full probability map, while the main learned baselines operate on thresholded masks. The single-pass U-Net is therefore not a fully matched control, and the effect of diffusion is mixed with the extra information carried by the probability input.
  2. Table 6 does not cleanly isolate the role of anatomical text. The index-conditioned model cannot represent classes outside its training vocabulary, so the gap also reflects vocabulary coverage. The paraphrase and mismatch tests show that DART responds to the description, but they do not establish how much cross-class transfer comes from the semantic content of the text.
  3. Figure 2 has poorly aligned arrows and connectors, while Figures 1 and 2 also show small text, uneven sharpness, and inconsistent visual texture across different elements.
Questions for authors
  1. How do you separate the effect of diffusion and partial noising from the extra information retained in the probability-map input, given that the single-pass U-Net uses thresholded masks?
  2. What evidence in the current experiments supports attributing cross-class transfer to the semantic content of the anatomical descriptions, beyond the vocabulary-coverage advantage of text conditioning?
8 comments
AuthorsOpenReview ↗

Response to Reviewer tjS8 (1/3): New Probability-Matched Experiment Addresses the Main Soundness Concern

Thank you for the thoughtful and constructive review. We especially appreciate your recognizing that DART uses one shared model across 377 anatomical classes with transfer from CT to MRI and unseen lesions, and that its corrected masks provide useful supervision for downstream segmentation architectures.

Your criticisms identify important distinctions that we agree should be clearer. We have completed a new, probability-matched control addressing the main soundness concern, performed an additional description-mismatch experiment on an unseen class, and will strengthen the analysis of descriptive semantics and the figures in the revision.

Most importantly, when the single-pass U-Net receives the same soft probabilities and anatomical text as DART, DART still achieves 85.6% mean Dice versus 82.1% across five datasets (~500 cases). On an additional TotalSegmentator comparison, DART outperforms the matched U-Net on 88 of 90 organs. We provide the point-by-point response below.

Q1. Separating diffusion-based correction from the probability-input advantage

"DART receives the full probability map, while the main learned baselines operate on thresholded masks. The single-pass U-Net is therefore not a fully matched control, and the effect of diffusion is mixed with the extra information carried by the probability input."

Thank you for identifying this confound. We agree that the original comparison did not fully distinguish the contribution of soft probability inputs from that of DART's diffusion-based refinement. Following your suggestion, we conducted a new controlled experiment designed specifically to address this issue.

We evaluated DART against a newly trained, probability-matched single-pass U-Net on five datasets: TotalSegmentator, AMOS22, FLARE22, PDDCA, and MSD Pancreas, covering approximately 500 cases. Upstream predictions came from TotalSegmentator and MedSAM. Crucially, the U-Net received the same soft probability maps and anatomical text conditioning as DART, and used the same training data and training settings.

Table R1. Mean Dice across five datasets with matched probability-map input.

MethodMean Dice (%)Gain (points)
Original predictions79.3—
Soft-input single-pass U-Net82.1+2.8
DART85.6+6.3

DART retains a 3.5-point advantage over the probability-matched U-Net. The U-Net's +2.8-point gain confirms that soft probabilities contain useful information for correction, but DART's +6.3-point gain shows that access to this information alone does not account for DART's performance.

To assess consistency beyond the aggregate, we also evaluated the probability-matched U-Net on the same TotalSegmentator cases emphasized in the manuscript. DART outperformed it on 88 of 90 anatomical organs. We will include the complete per-organ comparison alongside Table R1 in the revision.

This experiment directly removes the input-information asymmetry raised in the review. We do not claim that it isolates partial noising from iterative denoising or the diffusion training objective separately; it establishes the advantage of DART's overall diffusion-based correction procedure over a single-pass alternative with matched probability inputs and text conditioning.

We appreciate this suggestion: the added experiment makes our methodological comparison substantially more rigorous.

AuthorsOpenReview ↗

Response to Reviewer tjS8 (2/3): Cross-Class Transfer, Text Conditioning, and Semantic Analysis

Q2. Anatomical descriptions, vocabulary coverage, and cross-class transfer

"Table 6 does not cleanly isolate the role of anatomical text. The index-conditioned model cannot represent classes outside its training vocabulary, so the gap also reflects vocabulary coverage. The paraphrase and mismatch tests show that DART responds to the description, but they do not establish how much cross-class transfer comes from the semantic content of the text."

Thank you for this important distinction. We agree that vocabulary coverage and the semantic contribution of anatomical descriptions must not be conflated. Extending beyond a fixed training vocabulary is itself a central motivation for DART's text-conditioned design, while isolating shape/location semantics is a separate scientific question.

1. What Table 6 establishes

Replacing the text embedding with a learned integer-index class embedding, with other model components fixed, produces the following results on TotalSegmentator, AMOS22, and WORD test cases, including classes outside the index model's vocabulary:

ModelMean Dice (%)Gain (points)
Original predictions68.71—
Index-conditioned69.10+0.39
Text-conditioned DART74.20+5.49

We agree that the 5.10-point gap cannot be interpreted as a pure measurement of descriptive semantics: the index-conditioned comparator has no learned embedding for unseen classes. Rather, Table 6 measures a practical capability of the framework—text prompts allow a single frozen model to address classes outside its learned index vocabulary without adding parameters or retraining. We do not claim that text must outperform indices on familiar, in-vocabulary classes.

2. Figure 4 demonstrates actual zero-shot cross-class transfer

All seven classes in Figure 4(a) were excluded from DART training. Six are evaluated on MRI, although DART was trained on CT-derived segmentation probabilities; four are lesion classes. The same frozen DART model improves Dice on every task:

Unseen classInput Dice+ DARTGain
Pancreatic cyst55.259.1+3.9
Pancreatic lesion59.263.0+3.8
Left atrium65.568.4+2.9
Hippocampal tumor73.078.1+5.1
Brain tumor75.477.5+2.1
Hippocampus78.081.4+3.4
Prostate zone78.682.1+3.5

Thus, DART's ability to accept descriptions of unseen structures is accompanied by measured correction improvements, including cross-modality transfer. This demonstrates transfer, though not by itself which semantic features cause that transfer.

3. Evidence that descriptions affect predictions on unseen classes

Appendix E holds the input predictions, noise level, random seed, and model weights fixed while varying the prompt. Semantically equivalent paraphrases change Dice by only −0.02 points on average; mismatched anatomical descriptions reduce it by 2.07 points.

We also completed an additional mismatch experiment on hippocampus, a class unseen during DART training. The correct description yields 81.4% Dice (versus 78.0% upstream); replacing the entire description with an incorrect one reduces Dice to approximately 76.3%, a 5.1-point drop. This demonstrates prompt-dependent correction behavior even outside the training vocabulary. Because both class identity and descriptors change, we do not present this as proof of shape/location semantics independently of the class name.

4. Additional semantic isolation in the revision

We agree that our current experiments do not fully establish how much cross-class transfer comes from descriptive semantics beyond vocabulary coverage. We will add controlled prompt perturbations that preserve the correct class name while independently changing (i) single descriptive words, (ii) shape cues, and (iii) location cues, along with full mismatches as a reference. We will evaluate these on seen and unseen classes with the same inputs, noise levels, and model weights.

In summary, Table 6 establishes the practical advantage of an extensible vocabulary, Figure 4 establishes zero-shot transfer, and Appendix E plus the new held-out hippocampus test establish sensitivity to anatomical conditioning. The additional tests will specifically investigate the unresolved contribution of individual descriptive semantics. Thank you for pushing us to distinguish these claims more precisely.

AuthorsOpenReview ↗

Response to Reviewer tjS8 (3/3): Figure Revisions and Responses to Both Questions

Q3. Figure clarity and visual presentation

"Figure 2 has poorly aligned arrows and connectors, while Figures 1 and 2 also show small text, uneven sharpness, and inconsistent visual texture across different elements."

Thank you for this helpful feedback. We agree and will redraw Figures 1 and 2, correct the alignment of arrows and connectors, improve text readability, and ensure consistent resolution, typography, and visual formatting. We will replace the existing figures in the revised manuscript.

Direct answers to Questions for Authors

Question 1. How do you separate diffusion and partial noising from the additional information in probability maps?

Following your suggestion, we completed a probability-matched comparison across TotalSegmentator, AMOS22, FLARE22, PDDCA, and MSD Pancreas (~500 cases). The single-pass U-Net receives the same soft probability maps, anatomical text conditioning, training data, and training settings as DART. Original predictions achieve 79.3% Dice, the matched U-Net 82.1%, and DART 85.6% (Table R1 in our first response). DART therefore retains a 3.5-point advantage with the probability-input confound removed. It also outperforms the matched U-Net on 88/90 organs in the additional TotalSegmentator comparison.

This isolates the effect of having richer input information from the advantage of DART's overall diffusion-based correction procedure. It does not separately establish the causal contribution of partial noising versus iterative denoising; we will make that distinction explicit.

Question 2. What currently supports semantic cross-class transfer beyond vocabulary coverage?

We agree that no experiment in the current manuscript fully isolates the independent contribution of anatomical shape/location semantics beyond vocabulary coverage and class identity. Figure 4 demonstrates improvements on seven held-out classes (six MRI and four lesion tasks); Table 6 establishes the practical benefit of text conditioning beyond a fixed index vocabulary. Appendix E shows near-invariance to correct paraphrases (−0.02 Dice points) and sensitivity to fully mismatched descriptions (−2.07 points).

In a new experiment on the unseen hippocampus class, replacing the correct description with an incorrect one lowers Dice from 81.4% to approximately 76.3%. This extends the evidence of conditioning sensitivity to a held-out class, but because the entire prompt was replaced, it does not isolate individual semantic descriptors.

We will therefore add experiments that keep the class name fixed while perturbing individual words, anatomical shape descriptions, and location cues on seen and unseen classes. These will more directly test the contribution of descriptive semantics. Thank you for identifying this missing control.

Closing

We appreciate the reviewer's recognition of DART's zero-shot transfer and improved downstream training labels. Your feedback has motivated a completed probability-matched benchmark, an additional held-out-class description test, and specific revisions to the semantic analysis and figures. We hope these responses clarify which claims are already established and which will be tested more directly in the revision, and we would appreciate reconsideration of the rating in light of the new controlled results. If you have any follow up questions, please let us know and we would be happy to answer.

Reviewer_QXpFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DART is a text-condition 3D diffusion model, which is used to modify medical segmentation model. The input of DART is the segmentation map, and they add the name, shape and text of structures as prior, and utilize noise-denoise diffusion process to correct the predicted segmentation maps. In 13 public benchmarks, DART obtains the best Dice. Most importantly, the processed labels can also be used to train a better model. They also release DART-45K dataset, which contains 45000 unlabeled CT.

Critical strengths

The motivation is very clear. In introduction section, they show the results of Touchstone/TotalSegmentator to demonstrate the unequal quality of segmentation, which can be easily validated. The experiments contain 13 different public benchmarks with 3 automatic segmentation models and 15 downstream structures, which is so comprehensive. The limitation is described clearly. In appendix H, they report some failed cases and give explanation, such as the small changes of bones.

Critical weaknesses

They did not compare their method with the latest general segmentation models. There are several foundation models, such as VISTA3D and SegVol, but they only compare with traditional methods and MedSAM. The core parameters should be changed on each single class, which is not convenient. Each organ needs a noise level during validating. The running time and memory cost should be added in experiment section. One single segmentation map cost 0.59 seconds, and one CT with 50 organs will cost 30 second. Each organ will run one diffusion sampling. So I think the time cost should be discussed clearly. Experiments show that the more accurate the upstream prediction, the smaller the gain from DART. Please explain this phenomenon.

Questions for authors

DART only utilizes segmentation maps as input, but I think you can use images and segmentation map together as inputs because the image can be easily obtained. Could you please make the hyperparameters being learned automatically, instead of manually changing them? More questions can be found in weakness above.

9 comments
AuthorsOpenReview ↗

Response to Reviewer QXpF (1/4): Overview and Foundation-Model Comparisons

Thank you for the constructive review and for recognizing that "the motivation is very clear," that our experiments are "so comprehensive," and that the limitations are described clearly. We appreciate your thoughtful suggestions regarding foundation-model comparisons, noise-level selection, computational efficiency, and image conditioning.

We agree that these considerations deserve further clarification and will strengthen the revised manuscript. In particular:

  1. We will extend our comparisons to include VISTA3D and SegVol while clarifying our performance-driven baseline selection.
  2. We clarify that noise levels are selected only once per anatomical class, with no per-case or dataset-specific retuning. Figure 7 already demonstrates generalization across external datasets and imaging modalities with these fixed values.
  3. We will make runtime and memory requirements more prominent in the experimental section.
  4. We provide quantitative evidence from Appendix D explaining why correction gains decrease as upstream segmentation accuracy improves.
  5. We explain how operating on segmentation probabilities without image intensities enables zero-shot transfer from CT to MRI.

We address each concern below.


Q1. Missing comparisons with recent foundation models

"They did not compare their method with the latest general segmentation models. There are several foundation models, such as VISTA3D and SegVol, but they only compare with traditional methods and MedSAM."

Thank you for this constructive criticism! We agree that comparisons with recent foundation models would strengthen our evaluation, and we will add VISTA3D and SegVol to our comparisons in the revision, evaluating their predictions before and after DART on applicable benchmark classes.

We would also like to clarify the rationale behind our original baseline selection.

Our upstream segmentation models were selected primarily based on demonstrated segmentation performance, rather than simply their designation as foundation models or their recency. In fact, stronger upstream predictions provide a more rigorous test of DART, as they contain fewer errors and consequently leave less room for improvement.

Recent independent studies support this approach:

  1. Jaubert et al. (MIDL 2026) report that TotalSegmentator outperforms VISTA3D by 12.5 Dice points (92.8% vs. 80.3%) on the 117-class TotalSegmentator benchmark [1].

  2. Rokuss et al. (CVPR 2026) demonstrate that TotalSegmentator outperforms SegVol on abdominal organs (67.64% vs. 52.50%) and liver/vessels (49.48% vs. 41.34%). Their newer VoxTell model also substantially outperforms SegVol across 11 zero-shot benchmarks (70.85% vs. 30.35% mean Dice), illustrating the substantial differences in performance among foundation models across different tasks [2].

  3. MedSAM was selected as an extensively evaluated medical foundation model. Ma et al. (Nature Communications, 2024) evaluated MedSAM across 146 internal and external validation tasks (86 internal and 60 external), demonstrating broad generalization across anatomical structures, pathological conditions, and imaging modalities [3].

These independent results illustrate that supporting more anatomical classes or being designated a foundation model does not necessarily imply superior segmentation accuracy. We therefore prioritized demonstrated performance on relevant anatomical tasks over model recency or vocabulary size alone.

Importantly, our existing experiments already evaluate MedSAM, TotalSegmentator, AtlasNet, and WholeBodySeg. Across the three automatic segmentation models, DART improves mean Dice by 2.47, 1.04, and 3.77 points, respectively (Appendix D). These results demonstrate improvements across different upstream architectures and starting accuracies.

Furthermore, DART is a segmentation correction framework rather than a standalone segmentation model. Accordingly, our primary methodological comparisons involve eleven correction baselines, including SegRefiner, Post-DAE, and a single-pass U-Net with the same backbone but without diffusion. These directly assess DART's contribution as a correction method, rather than evaluating the segmentation capabilities of the upstream models alone.

In summary, we appreciate your suggestion and will expand the foundation-model comparisons to include VISTA3D and SegVol. Our original baseline selection was motivated by independently demonstrated segmentation performance, and our existing experiments establish DART's effectiveness across multiple upstream models and against eleven competing correction methods.

AuthorsOpenReview ↗

Response to Reviewer QXpF (2/4): References, Noise-Level Selection, and Runtime

References for Q1

[1] Jaubert et al. "Breaking the Memory Barrier: Efficient Multi-Class 3D Segmentation for Hundreds of Classes." MIDL, 2026. https://proceedings.mlr.press/v315/jaubert26a.html

[2] Rokuss et al. "VoxTell: Free-Text Promptable Universal 3D Medical Image Segmentation." CVPR, 2026. https://arxiv.org/abs/2511.11450

[3] Ma et al. "Segment anything in medical images." Nature Communications, 2024. https://doi.org/10.1038/s41467-024-44824-z

Q2. Per-class noise-level selection and hyperparameter tuning

"The core parameters should be changed on each single class, which is not convenient. Each organ needs a noise level during validating."

Thank you for raising this important practical consideration! We would like to clarify that DART does not require per-case or per-scan hyperparameter tuning. The noise level is selected only once per anatomical class on a validation split and subsequently held fixed for all test cases. Moreover, many anatomical classes share the same selected noise levels, and the per-class values are provided with our implementation to facilitate reuse without repeated calibration.

The selection process is automatic and reproducible: we evaluate a small, predefined set of candidate noise levels on validation data, select the best-performing value for each class, and fix that value for inference. No test annotations are used for this selection.

Class-specific noise levels are motivated by variations in anatomical shape, structural complexity, and segmentation error characteristics. Compact solid organs generally require smaller corrections, while vessels, glands, and hollow organs may benefit from greater reshaping. A single universal noise level would not necessarily provide the appropriate balance between preserving accurate structures and correcting errors across all anatomies.

Importantly, Figure 7 already demonstrates the scalability of this approach. The noise levels selected once on the original validation data are reused across independent external datasets, anatomical structures, and imaging modalities without any dataset-specific validation or recalibration.

DART consistently improves mean Dice across these external benchmarks, including:

  1. Pancreas (MRI): 74.1% → 82.3% (+8.2 points).
  2. Left atrium (MRI): 80.2% → 84.8% (+4.6 points).
  3. Prostate (MRI): 87.6% → 91.9% (+4.3 points).
  4. Hippocampus (MRI): 86.0% → 89.8% (+3.8 points).
  5. Kidney with tumor (KiTS23, CT): 91.8% → 95.1% (+3.3 points).

These results are especially relevant because the same noise-level selection is reused across independent datasets, including MRI datasets, despite DART being trained on CT-derived segmentation probabilities.

The selected values also remain unchanged across different upstream segmentation models, as demonstrated in Appendix D. This further supports the practicality of the calibration procedure.

Therefore, noise-level selection is a one-time, reproducible calibration rather than a recurring optimization requirement. Figure 7 provides direct evidence that fixed, reusable noise levels already scale across diverse datasets and imaging modalities.

Q3. Runtime and memory costs

"The running time and memory cost should be added in experiment section. One single segmentation map cost 0.59 seconds, and one CT with 50 organs will cost 30 second. Each organ will run one diffusion sampling."

Thank you for this helpful suggestion! We agree that reporting computational requirements more prominently would improve the paper's clarity and reproducibility. We will expand the experimental section to explicitly discuss inference time, peak GPU memory requirements, and the corresponding hardware specifications.

DART requires approximately 0.59 seconds per segmentation probability map on one AMD MI250X GPU at 128³ resolution, corresponding to approximately 29.5 seconds for 50 anatomical classes when processed sequentially. This represents the additional correction overhead after upstream segmentation.

The per-map runtime and hardware are already reported in Appendix L. We will make these details more visible in the main experimental section and add peak-memory measurements to provide a clearer account of the computational requirements.

AuthorsOpenReview ↗

Response to Reviewer QXpF (3/4): Correction Gains and Cross-Modality Transfer

Q4. Why do more accurate upstream predictions yield smaller improvements?

"Experiments show that the more accurate the upstream prediction, the smaller the gain from DART. Please explain this phenomenon."

Thank you for highlighting this observation! This behavior is expected: when upstream segmentation accuracy is already high, fewer errors remain for DART to correct, naturally limiting the achievable improvement in Dice.

Our experiments directly support this explanation. Appendix D reports strong negative correlations between initial Dice and DART's correction gain for TotalSegmentator (Spearman's ρ = −0.89, p < 0.001) and AtlasNet (ρ = −0.68, p = 0.008).

For example, DART improves duodenum segmentation by 4.07 Dice points from an initial score of 60.44% using TotalSegmentator, compared with a 0.95-point improvement from an initial score of 72.75% using AtlasNet.

This relationship extends beyond the duodenum: the largest improvements are generally observed on structures such as adrenal glands, esophagus, and gallbladder, where baseline segmentation is more challenging, whereas organs such as the liver and spleen, already approaching high Dice scores, offer less room for correction.

These results indicate that DART's improvements are closely related to the remaining segmentation errors. We will clarify this relationship in the revised manuscript and explicitly reference the quantitative analysis in Appendix D.

Q5. Why does DART use segmentation probabilities without the original medical images?

"DART only utilizes segmentation maps as input, but I think you can use images and segmentation map together as inputs because the image can be easily obtained."

Thank you for this thoughtful suggestion! We agree that incorporating original medical images could provide additional information for segmentation correction. However, our decision to operate exclusively on segmentation probabilities, without image intensities, is deliberate and central to DART's cross-modality generalization.

By learning anatomical shape priors directly from segmentation probability maps, DART avoids dependence on modality-specific image appearance, such as CT Hounsfield units or MRI intensity distributions. Incorporating CT images during training could introduce reliance on CT-specific characteristics, potentially limiting zero-shot generalization to other imaging modalities.

Importantly, Figure 7 provides direct experimental evidence supporting our design. Although DART is trained only on CT-derived segmentation probabilities, it successfully generalizes to MRI segmentation without MRI-specific training, model adaptation, or dataset-specific noise-level recalibration.

Specifically, using predictions from TotalSegmentator MRI:

  1. Pancreas (MRI): 74.1% → 82.3% (+8.2 Dice points).
  2. Left atrium (MRI): 80.2% → 84.8% (+4.6 points).
  3. Prostate (MRI): 87.6% → 91.9% (+4.3 points).
  4. Hippocampus (MRI): 86.0% → 89.8% (+3.8 points).

These results demonstrate that anatomical shape information learned from CT-derived segmentation probabilities can transfer to MRI despite substantial differences in underlying image appearance.

Furthermore, operating independently of medical image intensities allows DART to serve as a general-purpose correction framework for predictions from different upstream segmentation architectures without requiring modality-specific image preprocessing or retraining.

We acknowledge that incorporating image information could improve correction in cases where the predicted segmentation lacks sufficient anatomical information, particularly when image boundaries contain evidence not recoverable from the probability map alone.

However, incorporating image intensities introduces an additional modality-dependent input and would require evaluating whether the resulting accuracy improvements outweigh potential reductions in cross-modality generalization.

Our goal is to develop a generalizable anatomical correction framework that transfers across imaging modalities and segmentation architectures. The zero-shot CT-to-MRI results in Figure 7 support the effectiveness of this deliberate design choice.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2): The Scalability Gap in Anatomical Segmentation Correction

Dear reviewers and Area Chair,

Thank you for your thoughtful reviews and the additional experiments your suggestions have motivated. We would like to highlight the broader problem DART aims to address:

How can we scale anatomical segmentation correction from a handful of structures to hundreds of anatomical classes, while enabling transfer to previously unseen structures?

01 | Segmentation is not solved across human anatomy

Modern medical image segmentation models have made remarkable progress, with systems such as TotalSegmentator covering more than 100 anatomical structures. However, segmentation quality remains highly uneven across the human body.

While large organs such as the liver and spleen often achieve excellent accuracy, more challenging structures—including the adrenal glands, duodenum, gallbladder, esophagus, and complex vascular anatomy—frequently exhibit segmentation errors. Even TotalSegmentator documents failures involving missing portions of the colon and iliac arteries, as well as confusion between neighboring vertebrae and ribs [1].

These errors matter because segmentation is not only about voxel overlap. Anatomical plausibility, connectivity, topology, and boundary integrity are important for downstream applications.

The ability to segment hundreds of structures does not mean we can reliably correct all of them.

02 | The scalability gap in existing correction methods

Several important methods have explored segmentation refinement and anatomical shape priors.

Post-DAE [2] introduced denoising autoencoders for anatomically plausible segmentation correction, with experiments on lung and cardiac structures. SegRefiner [3] introduced model-agnostic refinement through discrete diffusion, demonstrating improvements on general-vision segmentation tasks. More recently, RefineCatDiff [4] investigated categorical diffusion refinement across BTCV, BraTS-2020, and ISIC-2018.

These works establish the value of learned correction. However, their demonstrated anatomical coverage remains relatively limited compared with the hundreds of structures targeted by modern segmentation systems. While some methods are explicitly model-agnostic, they do not establish a single shared, text-conditioned 3D anatomical shape representation with demonstrated correction across hundreds of classes and transfer to unseen structures.

This motivated our central question: Can we learn one generalizable anatomical shape prior instead of developing separate correction representations for different anatomical structures?

References

[1] Wasserthal et al. TotalSegmentator: Robust Segmentation of 104 Anatomic Structures in CT Images. Radiology: Artificial Intelligence, 2023.

[2] Larrazabal et al. Post-DAE: Anatomically Plausible Segmentation via Post-Processing With Denoising Autoencoders. IEEE Transactions on Medical Imaging, 2020.

[3] Wang et al. SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process. NeurIPS, 2023.

[4] Liu et al. RefineCatDiff: Toward High-Quality Medical Image Segmentation via a Categorical Diffusion Refinement Framework. Advanced Intelligent Systems, 2025.

AuthorsOpenReview ↗

General Response (2/2): DART's Scalability, Experimental Results, and Future Impact

03 | DART: a shared shape representation for 377 anatomical classes

DART addresses this question through a single text-conditioned 3D diffusion model trained on segmentation probability maps. It learns anatomical shape regularities independently of the original medical-image intensities, allowing the same correction framework to operate across different upstream models.

We demonstrate that this representation can scale from 50 to 377 anatomical classes, while preserving mean performance on the original 50 classes (73.85% versus 74.20%). The additional 327 classes achieve approximately 78.1% mean Dice using the expanded model.

Importantly, DART is not only scalable but also effective. Across 13 public benchmarks, we demonstrate segmentation improvements on challenging anatomical structures. The new experiments requested during rebuttal further strengthen this finding:

  • Probability-matched U-Net: DART achieves 85.6% versus 82.1% mean Dice across five datasets (~500 cases).
  • SegVol: DART achieves 75.62% versus 71.10% for the strongest competing correction baseline.
  • VISTA3D: DART achieves 87.41% versus 85.70% for the strongest competing correction baseline.

Beyond improvements on familiar classes, the same frozen model demonstrates transfer to seven held-out tasks, including four unseen lesion classes, and generalizes from CT-derived training predictions to MRI without retraining.

We believe these findings demonstrate an important capability: anatomical correction need not be restricted to a small, fixed vocabulary of structures or one particular upstream segmentation model.

04 | What scalable anatomical correction could enable

We envision anatomical shape correction becoming a reusable component of medical imaging AI, rather than a specialized model developed separately for each anatomical structure.

A broadly transferable shape representation could support future research in text-prompted diffusion, cross-class anatomical knowledge transfer, correction of automatically generated annotations, and training segmentation models using improved pseudo-labels.

Alongside our paper, we plan to publicly release the 377-class DART diffusion correction model, a separate segmentation foundation model, and DART-45K, containing annotations for approximately 45,000 CT scans.

We hope these resources will provide a foundation for further investigation into how anatomical representations can be shared across structures, architectures, and imaging modalities.

We also recognize the limitations highlighted by the reviewers, including guarantees of pathological-region preservation for unseen abnormalities and the distinction between anatomical class identity and detailed descriptive semantics. We will clarify these claims and provide additional reproducibility documentation in the revision.

05 | Closing

DART's central contribution is not simply improving Dice on individual organs. It is demonstrating that a shared, text-conditioned 3D anatomical correction model can scale to hundreds of structures while retaining the ability to transfer across models, anatomical classes, and imaging modalities.

We believe this represents an important step toward making segmentation correction a scalable and reusable capability in medical imaging AI.

We respectfully ask the reviewers and Area Chair to consider this broader contribution, our additional controlled experiments, and the potential research value of the planned public releases when evaluating the paper.

Thank you again for your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.