acceptodds
Withdrawn from ICLR 2027

Box2Lesion: Training-Free Spatial Control by Inverting Frozen Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models have shown strong capability for controllable generation of medical images with lesions. However, in many lesion-synthesis settings, the generator produces an image from spatial guidance without explicitly providing a precise mask of the realized lesion. We study whether a frozen conditional diffusion model can provide a denoising-based objective for controllable lesion generation, without training an additional segmentation or mask-generation network. To address this question, we introduce Box2Lesion, a training-free framework that reuses a frozen lesion synthesis model to recover a refined binary lesion mask from a healthy source image and a coarse bounding box. The box is first used as a coarse mask condition to generate a provisional lesion image. We then optimize only a compact residual representation using the conditional denoising behavior of the same frozen model. A source-aware objective encourages the candidate mask to match the box-conditioned denoising response on the provisional lesion image while remaining close to the unconditioned response on the corresponding healthy source. Stochastic mask optimization together with a topology-constrained projection enables refinement of the lesion mask while maintaining spatial coherence without imposing a predefined lesion shape. The recovered mask is then fed back into the same frozen generator to produce paired synthetic lesion images and masks without additional training. We evaluate Box2Lesion on LIDC-IDRI lung CT and PI-CAI prostate MRI using nnU-Net and SwinUNETR. Across both datasets and backbones, Box2Lesion augmentation improves all reported metrics over the real-data baseline and achieves the best result in the dataset–backbone–metric comparisons. These results show that spatial information already encoded in a pretrained lesion generator can be recovered at test time and reused for controllable synthesis and downstream supervision.

Primary area:
generative models
Keywords:
Diffusion Models, Training-Free Learning, Controllable Generation, Medical Image Synthesis, Lesion Synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1FRXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tries to explore methods for generating lesion masks by combining inverse optimization and topological constraints with an annealing strategy that integrates stochastic exploration and deterministic optimization; the authors try to validate approach through downstream segmentation experiments across multiple datasets.

Critical strengths

Generating masks with appropriate shapes and locations for lesions is an intriguing area of ​​exploration and holds clinical importance; the motivation behind the paper is commendable.

Critical weaknesses
  1. The methodological design raises concerns; specifically, certain operations—such as the use of truncation thresholds in the algorithms—lack a solid theoretical basis, making the method susceptible to dataset-specific overfitting and raising doubts about its generalizability. Furthermore, the design fails to address the mechanism of high-density collapse inherent in diffusion models, and the authors make several overblown claims.

  2. The reliability and stability of the experiments are questionable. The paper fails to report results from repeated trials or provide error analyses for the segmentation stage; clearly quantifying the errors and uncertainty associated with the segmentation process is essential to establishing the paper's robustness.

Questions for authors

Two issues require a direct response from the authors:

  1. Regarding methodology, is there a theoretical mechanism supporting the initialization of ∣I0−IH∣\vert{}I_0 - I_H\vert{}? Given that one of the primary goals of synthesizing lesions (pathologies) is to address the long-tailed distribution problem, the current mechanism appears to facilitate easy exploration of mask regions associated with high probability density; meanwhile, the deformations and similar features incorporated into the design seem to stem largely from the authors' subjective intent. The authors need to clearly distinguish between their motivations, assumptions, and theoretical evidence.

  2. The experiments lack corresponding error analysis and an assessment of uncertainty regarding the downstream segmentation model; it remains unclear to what extent the intrinsic uncertainty of the downstream segmentation model affects the evaluation of the proposed method's effectiveness.

Reviewer_ev5eOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Box2Lesion aims to recover binary lesion masks from coarse bounding boxes using a frozen, mask-conditioned lesion generator. It first generates a provisional lesion image and then optimizes a low-resolution mask using the generator’s denoising errors to identify the lesion region. The optimization includes topology constraints and regularization to produce plausible masks, which are then used to generate synthetic image–mask pairs. The method is evaluated through downstream segmentation experiments using nnU-Net and SwinUNETR on the LIDC-IDRI and PI-CAI datasets, comparing its performance with real-only training and existing synthetic data generation approaches. Additional experiments investigate mask recovery accuracy, the contribution of different components, and the effect of the regularization parameter.

Critical strengths

I think the main strength of this paper is its interesting and clearly formulated idea of recovering a lesion mask by inverting the conditioning process of a frozen diffusion model. The proposed approach is relatively simple and uses a low-dimensional mask representation together with denoising errors to identify the lesion region, without relying on cross-attention maps. Another strength is the controlled recovery experiment, where the ground-truth editing mask is hidden from the method and used only for evaluation.

Critical weaknesses
  • Limited evidence for the mask recovery approach: While the optimization idea is interesting, the results provide limited evidence that the diffusion objective meaningfully improves lesion mask recovery beyond the initial image-difference prior. The improvements in Dice are relatively small, and the recovered masks show limited agreement with the ground truth. The downstream segmentation gains are also modest and based on single runs, and no standard deviations, making it difficult to assess their reliability.

  • Missing comparisons with existing segmentation methods: The method is not compared with foundation models such as MedSAM, which can already generate segmentation masks from bounding-box prompts. Such a comparison would help clarify whether Box2Lesion offers a meaningful advantage over existing approaches. It would also be useful to compare the downstream results against synthetic data generated using simpler conditioning strategies, to determine whether the improvements come from the proposed mask recovery process or simply from adding more synthetic training data.

  • The segmentation results obtained directly from the optimization show relatively low agreement with the ground-truth masks, with Dice scores ranging from around 0.36 to 0.53. On PI-CAI, the recovered masks also perform slightly worse than the original bounding boxes in terms of IoU. This raises questions about how accurately the method recovers the true lesion boundaries and how much useful spatial information the diffusion-based optimization actually captures.

Questions for authors

Given the relatively low agreement between the optimized masks and the ground truth, how do the authors explain the improvements in downstream segmentation when training on synthetic data generated using these masks? Could the authors clarify whether the downstream models are learning meaningful lesion boundaries despite the limited accuracy of the recovered masks, or whether the improvements mainly come from the additional synthetic training data?

Reviewer_n8GCOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce a new way to generate masks based on bounding boxes via a frozen diffusion model. The recovered mask is reused to generate paired synthetic lesion images and masks for segmentation training. Experiments on lung CT and prostate MRI show modest, consistent improvements over real-only training and the compared augmentation strategies.

Critical strengths
  1. The authors introduce a novel way to utilize bounding boxes information for a mask-conditioned diffusion network.
Critical weaknesses
  1. Since the pipeline already trains a mask-conditioned ControlNet using lesion annotations, bounding-box supervision could be derived from those masks. A box-conditioned generator with a mask-prediction component is a relevant alternative; comparison would clarify the benefit of the proposed frozen-model approach.
  2. The method assumes a healthy candidate region and bounding box, but does not clearly explain how their locations and sizes are obtained.
  3. Representing the optimized residual on an 8×88\times8 grid may limit recovery of fine boundaries and irregular lesion shapes. An ablation over grid resolutions would help assess this trade-off.
  4. Dice gains over real-only training range from approximately 0.3% to 2.3%. Repeated runs and uncertainty estimates would help establish whether these improvements are reliable and justify the additional test-time optimization cost.
Questions for authors

See the weakness section for the suggested additional experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.