- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PartFactor, an open-vocabulary part segmentation framework that separates part recognition from foreground prediction. The method first trains an object-conditioned part decoder and then learns a lightweight visual foreground head while keeping the decoder frozen. A background-logit reparameterization is introduced to make foreground probability independent of part scores. Experiments demonstrate substantial improvements in unseen-part mIoU over HOPS, with gains of 10.12 and 23.79 points on PartImageNet and PartImageNet-OOD, respectively.
Critical strengths
- The paper identifies foreground rejection as a distinct failure mode in open-vocabulary part segmentation. The motivation is clear, and the proposed probability factorization provides a simple and interpretable way to address this problem.
- The experimental results are strong. PartFactor achieves substantial improvements in unseen-part segmentation on both benchmarks while requiring fewer task-trainable parameters and achieving faster inference than HOPS.
- The paper provides fairly comprehensive experimental analysis, including fixed-decoder ablations, comparisons at matched background false-positive rates, and detailed error analysis. These experiments help explain how the proposed approach recovers correctly recognized parts that would otherwise be classified as background.
Critical weaknesses
- The methodological novelty appears somewhat limited. Separating foreground probability from conditional class predictions is a relatively straightforward idea, and the proposed background-logit reparameterization follows naturally from this factorization. The paper could better clarify what distinguishes this formulation from existing approaches that model foreground and class predictions separately.
- Although the fixed-decoder experiments support the effectiveness of foreground decoupling, it remains difficult to determine how much this mechanism contributes to the overall improvements over HOPS. PartFactor also uses different pretrained encoders, a new decoder architecture, and a different training pipeline. These differences make it difficult to attribute the large performance gains specifically to the proposed decoupling strategy.
- The effectiveness of the approach appears to depend on the underlying decoder. Applying foreground adaptation to HOPS and PartCATSeg leads to only marginal improvements, which raises questions about its generalizability. In addition, the gains are mainly concentrated in larger parts, while performance on small parts remains limited.
Questions for authors
- Could the authors provide additional architecture-level ablations or comparisons using matched pretrained encoders to better isolate the contribution of foreground decoupling from other architectural changes?
- The experiments on HOPS and PartCATSeg suggest that foreground adaptation is not consistently beneficial across different decoders. Could the authors further clarify under what conditions the proposed approach is expected to be effective, and whether its effectiveness can be predicted from the initial decoder's foreground rejection behavior?
- On PartImageNet-OOD, the original-background visual-plus-score head already achieves 55.42% unseen mIoU, compared with 56.11% for the proposed score-decoupled head. This relatively small difference suggests that much of the improvement may come from introducing spatial visual foreground modeling rather than score decoupling itself. Could the authors further clarify the relative contributions of these two components?
If the authors can address most of my concerns, I would be willing to raise my score.
7 comments
Response to Reviewer UVdM (2/5)
Original comment:
Although the fixed-decoder experiments support the effectiveness of foreground decoupling, it remains difficult to determine how much this mechanism contributes to the overall improvements over HOPS.
: We now measure the foreground contribution directly on fixed, background-supervised decoders and distinguish it from complete-system performance.
The revised default trains each decoder with part and background supervision from its first update. We then freeze the decoder and learn foreground support.
Across the four benchmarks, unseen mIoU improves as follows:
| Benchmark | Joint training only | PartFactor | Foreground gain |
|---|---|---|---|
| PartImageNet | 48.68 | 51.52 | +2.83 |
| PartImageNet-OOD | 53.48 | 56.88 | +3.40 |
| Pascal-Part-116 | 22.47 | 32.24 | +9.77 |
| ADE20K-Part-234 | 7.36 | 11.24 | +3.88 |
Each pair shares the same decoder and conditional part distribution on the decoder grid. Gains are computed from unrounded mIoU values.
The PI and OOD gains have paired-image 95% intervals of [1.76, 4.65] and [3.16, 3.66] points. These measure evaluation-image variation with trained model weights fixed.
We further examine whether the foreground contribution persists across different learned representations. Three complete PI decoder trainings with independent initializations yield unseen-mIoU improvements of 2.83, 5.59, and 3.58 points after foreground adaptation, averaging (Appendix S, Table 42). Separately, three foreground-head initializations on the fixed default decoder reach 51.77 ± 0.27% unseen mIoU on PI and 56.80 ± 0.07% on OOD (Appendix Table 36). These experiments distinguish variability in the learned decoder from variability in the foreground intervention itself. In both settings, the foreground gain remains positive.
These paired experiments measure the contribution of the foreground stage without changing the decoder's conditional part representation.
Original comment:
PartFactor also uses different pretrained encoders, a new decoder architecture, and a different training pipeline. These differences make it difficult to attribute the large performance gains specifically to the proposed decoupling strategy.
: We investigate representation quality through separate controls and use fixed-decoder experiments to isolate foreground adaptation from those differences.
To examine the effect of pretrained representations, we replace the default encoders with HOPS's DINOv2-S/14 and CAT-Seg-initialized CLIP. Foreground adaptation still improves unseen PI mIoU from 39.44% to 41.29% on the same frozen decoder (Appendix Table 29).
Separately, the architecture ablations reveal where the representation gains come from. Removing the DINO structural pathway reduces conditional unseen-part accuracy from 87.82% to 71.97%. Removing object context causes a much larger performance loss than simplifying the decoder's iterative refinement (Appendix Tables 31 and 32).
These experiments distinguish two contributions.
The structural and semantic representations determine which part identities the model can recognize. The foreground stage determines which of those identities survive the final prediction.
The complete-system comparison with HOPS includes both effects, while the paired fixed-decoder experiments isolate the foreground intervention. This distinction is now explicit in the revised manuscript.
Together, W2.1 and W2.2 provide a more precise attribution of PartFactor's performance: recognition quality establishes the available part identities, and foreground specialization contributes additional recovery while retaining the trained part decoder.
Original comment:
The effectiveness of the approach appears to depend on the underlying decoder. Applying foreground adaptation to HOPS and PartCATSeg leads to only marginal improvements, which raises questions about its generalizability.
: The revised experiments show that foreground recovery depends on the error structure of the starting decoder.
Response to Reviewer UVdM (3/5)
A foreground head can recover correctly ranked parts that are rejected as background. Its effect is therefore determined partly by how many such predictions exist and partly by the errors introduced when they are accepted.
This helps explain the external-decoder results.
On OOD, retrained HOPS has zero measured correct-part rejection at its original operating point (Appendix Table 27). A new foreground head has little of this particular error to recover. A source-calibrated global bias reaches 32.65% unseen mIoU, slightly above the spatial-head alternatives (Appendix Table 22).
PartCATSeg presents a different case. Its fixed 20,000-update checkpoint has 6.08% correct-part rejection on PI (Appendix Table 27). The separate foreground-head experiment uses the earlier 8,000-update decoder, with unseen mIoU of 35.69% before adaptation and 35.15–36.04% after (Appendix Table 23). These experiments show that having correct identities available for recovery does not by itself ensure a net improvement. The learned foreground decision must also avoid losing existing correct labels or introducing new errors.
These observations establish an important distinction: correct-part rejection identifies a recovery opportunity, while the eventual performance gain depends on the selectivity of the learned foreground decision.
The external decoder experiments therefore help explain when targeted foreground learning is useful and why its effect varies across models.
Original comment:
In addition, the gains are mainly concentrated in larger parts, while performance on small parts remains limited.
: We have repeated the area analysis using the revised background-supervised models and find measurable improvements across part sizes, including small regions.
On PI, mean image–part IoU for regions smaller than 1,000 pixels rises from 9.21% to 12.04%. On OOD, 1,086 such regions improve from 15.04% to 19.83%, with median IoU rising from 3.01% to 11.93% (Appendix Table 33).
The smallest PI regions retain a zero median, indicating that fine boundaries and ambiguous part identities remain difficult. However, foreground specialization produces measurable improvements even in this challenging size range.
The external decoder and part-size experiments therefore clarify when targeted foreground learning is useful. They also identify where better recognition and fine-scale evidence are the more appropriate next steps.
Original question:
Could the authors provide additional architecture-level ablations or comparisons using matched pretrained encoders to better isolate the contribution of foreground decoupling from other architectural changes?
: We have added both the matched-initialization control and independently retrained architectural ablations.
First, we replace PartFactor's default pretrained encoders with HOPS's DINOv2-S/14 and CAT-Seg-initialized CLIP, removing DINOv3-L, SigLIP 2, and ResNet-50.
The jointly trained PartFactor decoder reaches 39.44% unseen PI mIoU, increasing to 41.29% after foreground adaptation with the decoder fixed (Appendix Table 29).
Second, we retrain architecture replacements under a common PI schedule (Appendix Table 31).
The full architecture reaches 47.06% unseen mIoU. Removing object context reduces unseen mIoU to 0.00%, while replacing the three refinement rounds with a single image–text matching step retains 46.83%.
This result suggests that object-conditioned evidence is especially important for recognition, whereas additional iterative refinement contributes relatively little under the tested schedule.
The structural-feature ablation provides another view. Removing DINOv3 reduces conditional unseen-part accuracy from 87.82% to 71.97%, accompanied by an unseen-mIoU decrease from 47.06% to 30.05% (Appendix Table 32).
These controls reveal which parts of the representation supply useful part identities. The matched-decoder foreground experiment then measures how those identities can be retained without relearning the representation.
Response to Reviewer UVdM (1/5)
We sincerely thank you for the thoughtful review and for recognizing the meaningful failure diagnosis, strong empirical results, and controlled analysis of PartFactor. We especially appreciate your questions about methodological novelty, the contribution of pretrained representations, and the conditions under which foreground adaptation is useful.
We have incorporated your feedback and marked the key revisions addressing your comments in orange in the revised manuscript.
These questions go directly to what we wanted to understand: when a model misses a part, how much of the correct prediction is already present, and what is the simplest intervention that can recover it?
The revised manuscript now includes experiments with normally background-supervised decoders, matched pretrained initializations, independently retrained architecture controls, and additional tests of foreground evidence and score suppression. These experiments provide a clearer account of both the benefits and the limits of foreground learning.
Critical Weaknesses
Weakness W1. Limited methodological novelty
W1.1. The simplicity of the factorization and reparameterization
Response: PartFactor uses a simple mathematical construction to identify a measurable source of segmentation error and turn it into a controlled learning problem.
The underlying probability factorization is standard. Its usefulness here comes from separating two errors that produce the same visible outcome.
A pixel may disappear because the decoder prefers the wrong part identity. Alternatively, it may prefer the correct identity yet lose the final comparison with background.
Equation (3) shows why this happens, and Equation (4) defines the event as correct-part rejection. Measuring it tells us which correct predictions are already available but discarded.
We then design a direct intervention: freeze the decoder, preserve the conditional part distribution on its prediction grid, and learn only foreground support. This makes it possible to measure how much segmentation performance can be recovered without relearning part identity.
The experiment remains meaningful after ordinary background supervision. On the joint-trained PI decoder, conditional unseen-part accuracy remains 85.90%, while correct-part rejection falls from 18.94% to 17.10% after visual foreground adaptation (Appendix Table 27).
The mathematical simplicity is important to the experimental design. By changing only the foreground decision, we can identify what that decision contributes without introducing further changes to part recognition.
W1.2. Distinction from existing approaches
Response: Our contribution connects a precisely measured failure mode, a controlled recovery experiment, and an economical intervention whose effects and limits can be established.
The matched-background-error experiments provide a further test. At a 2% background-FPR budget, correct unseen-part recall increases from \color{#073}\textbf{62.19\\% to 70.95\\% on PI} and \color{#073}\textbf{74.86\\% to 80.60\\% on OOD} (Section 5.3 and Appendix P).
Thus, the learned foreground model improves the spatial selection of correctly named parts, beyond shifting a single global threshold.
The intervention is deliberately small. The foreground head reuses existing features and trains only 22,177 parameters on PI. This allows us to investigate the source of an error without making the recognition model more complicated.
The revised Related Work also discusses the relevant approaches to separating recognition, spatial grouping, and objectness, including ZegFormer, MaskCLIP, SAN, FC-CLIP, and OVRCOAT.
The resulting contribution connects a specific failure mode, a measurable recovery opportunity, and a computationally inexpensive intervention. It provides a way to decide when additional capacity should improve recognition and when it should improve foreground support.