acceptodds
Withdrawn from ICLR 2027

Partition Everything: Whole-Image Segmentation by Palette-Conditioned Flow Matching

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Decomposing an image into its constituent objects and parts provides a fundamental representation for scene understanding and image editing. Yet whole-image segmentation remains cumbersome with promptable models such as SAM, requiring dense point prompting followed by heuristic merging and suppression of overlapping, duplicate masks. We introduce Partition Everything Model (PEM), which formulates whole-image partitioning as palette-conditioned image generation. Our rectified-flow model learns to repaint each region with the mean color of a supplied continuous palette field, representing the partition as an image without prescribing semantic labels or the number of regions. To distinguish regions at different spatial scales, we condition the model on two complementary palettes and jointly predict their corresponding color maps, combining broad spatial variation with local color contrast. Together, these maps provide a six-dimensional representation from which we extract an exhaustive, non-overlapping partition. Furthermore, we introduce annotated-region conditioning to learn from incomplete annotations while predicting complete image partitions at inference. Across seven benchmarks containing 12,138 images and 117,300 instances, PEM outperforms SAM3 by 15.57 percentage points in instance-weighted mIoU (55.27% vs. 39.70%), while predicting all regions jointly in a single forward pass.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image partitioning, whole image segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HpfKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to address the whole image segmentation problem. While SAM prompts a dense grid of points and collects the resulting masks, which might cause overlapped or missing masks, whole image segmentation methods use generative models to generate color images as partition so that every pixel can receive a non-overlapping mask. The question is how to get the colored partition images from the GT masks to train the flow matching model. To avoid the permutation problem, previous work uses centroid of a fixed color field to assign each object a color (UniGS), or supervise the similarity within each object (Gen2seg). This paper proposes to use two fixed color fields, one high frequency and one low frequency, and take their mean within the object mask as the object target color. The low frequency field is to separate large objects and the high frequency field is to separate small objects. To utilize existing object segmentation datasets, this paper introduces mask-conditioning generation during training. At inference, the mask is the whole image. The qualitative and quantitative results show that this paper achieves very strong performance.

Critical strengths
  1. The qualitative results of across artwork in Figure 1 is very impressive.
  2. The idea of using two fixed spatial color fields is simple yet effective. Low-frequency map can separate large objects while fall short at small objects. High-frequency map can separate small objects but the large objects receive close-to-grey color.
Critical weaknesses
  1. This paper uses FLUX2 as backbone while gen2seg uses SD2. It is unclear whether the performance gain mainly come from a newer backbone. This paper lacks an important ablation test. Without this test, I cannot give accept.
  2. The name "palettes" really confuses me in abstract and introduction. I didn't realize it is a fixed spatial color field until I looked at Figure 2, which is only mentioned in the end of introduction. I thought it was some discrete color set, so I was really confused.
  3. The fixed spatial color fields are hand-designed. It might need some ablation tests to show the effectiveness of the current design. For example, how about learned spatial embeddings, or other frequency bands, which might perform similarly or better?
Questions for authors

Does the number of palettes matter? What if we use 3 or 4?

Reviewer_RqSrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper models whole-image segmentation as palette-conditioned generation. Specifically, diffusion model initialized from FLUX repaints each region with its mean color under two complementary palettes, and a watershed-based readout recovers a non-overlapping partition. Annotated-region conditioning enables training on mostly partial annotations. The method reports large gains over SAM-family and gen2seg baselines on multiple benchmarks.

Critical strengths
  1. The complementary low- and high-frequency palettes are a simple and well-motivated design.
Critical weaknesses
  1. Missing baselines for an existing task. The task is essentially class-agnostic panoptic/entity segmentation, but Mask2Former-style and entity segmentation methods (e.g., CropFormer [1]) are not compared. Query-based methods already succeed in similar tasks in one pass at far lower cost. Motivating the work against SAM's automatic mask generation is a strawman.

  2. Generative formulation not justified. As shown in Tab.5, single-step inference performs best, so the model reduces to deterministic regression. Whether the gains come from flow matching or simply from a large pretrained backbone is untested, and no latency or memory comparison is given.

  3. Unfair evaluation. PEM is trained on splits of several evaluation benchmarks, with author-created annotations, while all baselines are zero-shot. In-domain and out-of-domain results should be reported separately.

  4. Under-defined metric. The mask-selection rule for overlapping SAM candidates is unspecified, unassigned stuff regions penalize SAM by design, and a single-granularity metric conflicts with the claim of not prescribing granularity.

[1] High-Quality Entity Segmentation

Questions for authors
  1. How does a class-agnostic Mask2Former or CropFormer trained on the same data perform?

  2. What are the oracle mask-selection results for SAM/SAM3, and the in-/out-of-domain breakdown?

Reviewer_GTQfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Partition Everything Model (PEM), a framework for whole-image segmentation that formulates image partitioning as palette-conditioned flow matching. Specifically, the authors propose a palette-based representation that jointly predicts two complementary color maps using low- and high-frequency palettes, enabling the model to partition an image without explicitly specifying the number of instances. An annotated-region conditioning mechanism is further introduced to enable training with partially annotated data. Experiments across seven benchmarks demonstrate promising results. Nevertheless, the scarcity of the training data (only 3,154 images) in this work remains the main concern.

Critical strengths
  1. This paper proposes an rectified-flow-based paradigm for whole-image segmentation. Unlike the SAM family, which relies on dense point prompting to generate numerous candidate masks, followed by several post-processes, PEM formulates whole-image partitioning as the joint prediction of two color maps. This approach eliminates the need to explicitly specify the number of instances or mask queries, offering a relatively simple and elegant design, although its technical novelty is somewhat limited.

  2. The region conditioning mechanism effectively addresses the challenge of learning from partially annotated data. This allows a large amount of existing segmentation data to be directly utilized for training while also reducing annotation costs.

  3. The paper provides extensive qualitative visualizations, including model predictions and results from different ablation variants. These visualizations help readers better understand the model's capabilities and performance.

Critical weaknesses
  1. The paper is titled Partition Everything Model (PEM), which suggests a rather ambitious and foundational scope. However, I find the training dataset insufficiently scaled, consisting of only 3,154 images. I am also curious why the authors constructed a new training dataset by sampling a small subset of images from five existing sources, rather than directly using all available training samples.

  2. I also have concerns about the experimental evaluation. The proposed task is closely related to panoptic segmentation, with one major difference being that panoptic segmentation additionally requires predicting semantic categories. However, there appears to be a considerable performance gap between PEM and state-of-the-art panoptic segmentation methods. For example, HyperSeg [1] has already achieved 77.2% mIoU on COCO. Given the broad scope implied by the name Partition Everything Model, I believe comparisons with specialized models on the evaluated benchmarks would also be valuable.

[1] HyperSeg: Hybrid Segmentation Assistant with Fine-grained Visual Perceiver. CVPR-2025.

  1. The use of complementary low- and high-frequency palettes can be regarded as the main methodological contribution of this work. However, I find this design relatively straightforward, which somewhat limits the methodological novelty of the paper.

  2. The appendix provides extensive qualitative comparisons among different methods. From these visualizations, gen2seg appears to produce more fine-grained segmentation results in some cases, such as distinguishing different parts of a person (e.g., the head and torso). Moreover, PEM does not consistently produce the best results across all examples. For instance, in the fifth row of Figure 11, PEM fails to distinguish individual fruits in a basket. I am also curious about what the authors consider the ideal partition for certain examples. For instance, in the last two rows of Figure 12, it is unclear how the thin structures should ideally be separated. Including the ground-truth annotations in these visualizations would help readers better understand the expected segmentation granularity and assess the quality of the predictions.

  3. Given the broad applicability implied by the name “Partition Everything Model”, releasing the pretrained weights, inference code, and training code would be important for facilitating reproducibility and benefiting the research community.

Questions for authors
  1. The name Partition Everything Model (PEM) suggests a model with broad applicability, yet the training dataset is surprisingly small. Why did the authors not scale up the training data? Does the proposed method demonstrate favorable scaling behavior as more training data becomes available? I understand that scaling up the training dataset is not feasible within the short rebuttal period, but I still consider this a critical weakness of the paper.

  2. Based on the qualitative results in the appendix, PEM demonstrates impressive capabilities in partitioning densely distributed instances and objects with complex shapes. I am curious why such a small training dataset is sufficient to achieve these results. Is this primarily due to the strong pretrained representations of FLUX.2 Klein, or is the partitioning task itself relatively easy to learn? Table 2 suggests that the key factor behind the performance improvement is the introduction of the palette-based representation, which leads to a substantial gain. I would appreciate further discussion or analysis of what enables the model to acquire these capabilities with such limited training data.

  3. The annotated-region conditioning mechanism introduces a potential train–test distribution mismatch. During training, the model is conditioned primarily on partial annotation masks, whereas at inference, the conditioning mask is set to all ones. How does the model handle this discrepancy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.