acceptodds
Under review as a conference paper at ICLR 2027

Scientific Image and Mask Generation with Adaptive Patching

Abstract

Building annotated datasets of high-resolution scientific images faces two bottlenecks: computationally expensive image synthesis and labor-intensive pixel-wise annotation. We present a unified framework that addresses both through a shared, spatially adaptive latent representation for image–mask synthesis. An edge-guided quadtree allocates finer patches to structurally complex regions, and a shared codec maps variable-size patches into a common latent space. Conditioned on reference layouts, a rectified flow model generates latent features from which image and semantic mask branches decode paired outputs. The mask branch is trained using simulator-provided labels with the image codec frozen, avoiding manual annotation for its training supervision. On concrete X-ray CT data, our framework generates images at resolutions up to 8192 8192, achieving a global FID of 18.04 and a native-resolution patch-FID of 22.50. At 4096 4096, it achieves global and patch FID scores of 17.98 and 40.98, compared with 46.84 and 73.73 for the evaluated class-conditioned dense rectified-flow baseline, alongside approximately 2.3 higher generative-model training throughput. Evaluation against manual annotations on 12 generated images yields a three-class macro Dice of 0.736. These results establish the feasibility of combining efficient high-resolution generation and semantic decoding within a single framework for scientific image–mask synthesis.

Primary area:
generative models
Keywords:
rectified flow, adaptive tokenization, quadtree representation, high-resolution image synthesis, image-mask synthesis, scientific imaging

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_WYyqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

QuaDiT proposes a spatially adaptive representation for high-resolution scientific image–mask synthesis. An edge-guided quadtree assigns a fixed budget of variable-sized patches, which a shared autoencoder maps to uniform latent tokens. A geometry-aware diffusion transformer generates tokens conditioned on layouts sampled from training images. Image and semantic heads decode shared features, with the semantic head trained using simulator-provided labels. Experiments demonstrate improved concrete CT image generation and reduced generative-model training cost at 4K, image synthesis at 8K, and additional results on synthetic hydrogel images. Quantitative mask evaluation is conducted on encoded simulator images.

Critical strengths
  • A practically consequential improvement in high-resolution generation efficiency: At 4K, QuaDiT uses 1,501 tokens versus 4,096 for dense DiT-RF, achieving 2.25× training throughput and reducing peak memory.

  • Substantial image-quality gains supported by useful diagnostics: The main baseline shares the modern rectified-flow recipe, making this stronger evidence than comparison only with the original DDPM recipe. KID supports the same ranking, and native-patch evaluation examines detail obscured by whole-image resizing.

Critical weaknesses
  • This paper is mainly focused on Scientific Image and Mask Generation but after reading the paper, I think the method can be used to generate common images. It is unclear if the QuadDiT method can be applied to regular images such as LAION-5B High-Res dataset.

  • Lack of comparison with recent pixel-space diffusion models such as PixNeRD, PixelGen, etc. To me, it is unclear how does this latent-based diffusion model compared to recent SOTA pixel-space diffusion methods. I would consider training those method on the 2K and 4K settings of S8C to have a fair comparison with the proposed method.

  • The method seems like a combinations of existing methods. I do not see new and interesting components in both quadtree image representation, autoencoder and DiT architectures. They are very similar to previous works

Questions for authors
  • Additional explanations would strengthen the paper but I am not convinced yet about the novelty aspect of the proposed architecture.
  • I would like to see comparison with recent pixel space diffusion on common high resolution image dataset.
Reviewer_oPt2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QuaDiT, a framework for high-resolution scientific image–mask synthesis using spatially adaptive latent tokens. It combines edge-guided quadtree partitioning, a shared autoencoder for variable-size patches, and a geometry-aware diffusion transformer. A simulator-supervised semantic head further enables paired image–mask generation. Experiments on concrete CT and Hydrogel datasets demonstrate improvements in generation efficiency and image quality over the evaluated baselines.

Critical strengths

1.The proposed spatially adaptive latent representation provides a meaningful approach to reducing the computational cost of high-resolution image generation. Combining budgeted quadtree partitioning with a shared autoencoder and geometry-aware diffusion transformer is technically well motivated. 2.The experiments demonstrate substantial improvements in generation quality and computational efficiency on high-resolution concrete CT images. The evaluation includes multiple image resolutions, generation metrics, computational measurements, and an additional Hydrogel dataset.

Critical weaknesses
  1. The reported segmentation accuracy is evaluated on simulator-generated images rather than QuaDiT-generated CT images. Without quantitative evaluation in the target domain, the semantic reliability of the synthesized image–mask pairs remains insufficiently validated.
  2. QuaDiT and the dense DiT baseline differ in their autoencoders, token allocations, positional encodings, and layout conditioning. Without controlled ablations, the contribution of edge-guided adaptive patching to the reported improvements remains unclear.
  3. Appendix C states that QuaDiT uses latent normalization statistics estimated from 200 validation images, whereas the dense DiT baseline uses training-set statistics. This introduces an inconsistency in the evaluation protocol, and its impact on the reported results remains unexamined.
Questions for authors
  1. Could the authors provide quantitative evidence of the semantic accuracy of masks associated with QuaDiT-generated CT images, or evaluate their utility for downstream segmentation on real CT data?
  2. Could the authors compare edge-guided patching with alternative partitioning strategies under the same token budget, autoencoder, and diffusion training configuration to isolate its contribution?
  3. Could the authors clarify whether the 200 validation images used for latent normalization overlap with the FID evaluation set, and report the results when normalization statistics are estimated exclusively from training data?
Reviewer_CLbQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a framework for scientific image and segmentation mask synthesis. Firstly an edge-guided quadtree processes variable-size patches into a fixed sized token. Then an autoencoder maps those into a uniform-directional latent token. A geometry-aware transformer trained with rectified flow generates tokens using reference patch layouts. Moreover, a mask head and image head learn to reconstruct these from a shared decoder’s representations. Evidence for mask reliability and downstream scientific utility seems to be more limited from a broader angle.

Critical strengths
  1. The fixed token budget lets a standard DiT be used with only the positional encoding replaced. Each token carries the same values, and so cost differences come from token count.

  2. The mask head reads frozen decoder features. It does not need the re-encoding of generated images, and replaces manual labels with simulator supervision.

Critical weaknesses
  1. QuaDiT receives structural clues from real-image layouts that the baselines lack, while several other model components also differ. This makes unclear how adaptive matching itself makes this improvement.
  2. Generated-mask accuracy and usefulness are not validated substantially. Strong results on simulated images do not establish that masks for newly generated images are accurate.
Questions for authors

-How do results change if dense baselines receive equivalent structural information, or if QuaDiT uses content-independent layouts with the same token count?

-Can you compare QuaDiT with a uniform-layout UniAE variant and a dense baseline using a comparable token budget?

-Which layouts and how many generated and reference images were used to calculate Hydrogel FID in Table 1?

-What are the five segmentation classes, their individual Dice scores on S8D, and the results of a simple segmentation baseline? -Could you clarify the 12-image manual evaluation mentioned in the OpenReview abstract and compare mask quality against a simulator-trained segmenter applied to generated images?

-How do the results change when latent normalization statistics are computed only from training images?

-Why were only three of the 46 specimens used, and is there evidence of generalization to a specimen excluded from training?

Reviewer_hJwrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

QuaDiT combines fixed-budget quadtree tokenization, a shared autoencoder, and a geometry-aware rectified-flow transformer for high-resolution scientific image-mask synthesis. A simulator-supervised semantic head decodes masks from shared image-decoder features. Experiments demonstrate improved CT generation quality and training efficiency, including native 8K generation, with an additional evaluation on synthetic Hydrogel data. Semantic validation primarily uses simulations; the submission-page abstract additionally reports manual evaluation on 12 generated images.

Critical strengths
  1. The CT results are the strongest part of the paper. With training-set layouts, QuaDiT reaches 4K global/patch FID of 22.64/38.51, compared with 46.84/73.73 for DiT-RF, while training approximately 2.25× faster and using 54.6% less peak allocated memory (Tables 1-2). The baseline already uses a modern rectified-flow recipe, which makes this comparison informative. The extension to native 8K generation adds practical value.

  2. The shared decoder lets the mask head work directly on generated features without re-encoding the image. Native-resolution patch metrics and the memorization audit are also useful checks beyond the headline FID scores.

Critical weaknesses
  1. It remains unclear which parts of the design account for the improvement. The codec, token allocation, and geometry encoding all change together, and QuaDiT receives an image-derived layout that supplies structural information unavailable to the class-conditioned dense baseline. Since the CT preprocessing already allocates fewer tokens to the background, a simple ROI-aware control would be particularly informative. The current results establish an advantage of the complete system, but provide limited evidence about the contribution of adaptive allocation itself. Appendix C raises a separate concern: QuaDiT uses normalization statistics from 200 validation images, whereas the baseline uses training statistics. The effect of this difference has not been measured.

  2. The descriptions of mask evaluation need to be reconciled. The submission-page abstract reports three-class macro Dice of 0.736 on 12 manually annotated generated images, while the provided manuscript’s conclusion says that evaluation on real/generated CT remains qualitative. The reported manual result is relevant, but its selection and annotation procedures are missing from the manuscript, making its significance difficult to assess.

Questions for authors
  1. How do the main results change with normalization estimated exclusively from training data?

  2. Is there a control using the same codec and a comparable token budget? An ROI-aware allocation without edge-based refinement would help distinguish the benefit of representing the background coarsely from the benefit of adapting to structures within the specimen.

  3. Please clarify the manual evaluation mentioned on the submission page. How were the 12 images selected and annotated, how do the three evaluated classes map to the five supervised classes, and what are the per-class scores?

  4. Which layout sources and checkpoints were used for the headline CT scores and the numerical Hydrogel results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.