Box2Lesion: Training-Free Spatial Control by Inverting Frozen Diffusion Models

Diffusion models have shown strong capability for controllable generation of medical images with lesions. However, in many lesion-synthesis settings, the generator produces an image from spatial guidance without explicitly providing a precise mask of the realized lesion. We study whether a frozen conditional diffusion model can provide a denoising-based objective for controllable lesion generation, without training an additional segmentation or mask-generation network. To address this question, we introduce Box2Lesion, a training-free framework that reuses a frozen lesion synthesis model to recover a refined binary lesion mask from a healthy source image and a coarse bounding box. The box is first used as a coarse mask condition to generate a provisional lesion image. We then optimize only a compact $8\times 8$ residual representation using the conditional denoising behavior of the same frozen model. A source-aware objective encourages the candidate mask to match the box-conditioned denoising response on the provisional lesion image while remaining close to the unconditioned response on the corresponding healthy source. Stochastic mask optimization together with a topology-constrained projection enables refinement of the lesion mask while maintaining spatial coherence without imposing a predefined lesion shape. The recovered mask is then fed back into the same frozen generator to produce paired synthetic lesion images and masks without additional training. We evaluate Box2Lesion on LIDC-IDRI lung CT and PI-CAI prostate MRI using nnU-Net and SwinUNETR. Across both datasets and backbones, Box2Lesion augmentation improves all reported metrics over the real-data baseline and achieves the best result in the dataset–backbone–metric comparisons. These results show that spatial information already encoded in a pretrained lesion generator can be recovered at test time and reused for controllable synthesis and downstream supervision.