acceptodds
Under review as a conference paper at ICLR 2027

Calibration-Guided State Selection and Class-Conditioned Fusion for Frozen Diffusion Segmentation

Abstract

Diffusion representations vary substantially across network layers and denoising timesteps. Prior work exploits this variation through fixed states, task-level selection, shared aggregation, or pixel-adaptive rules, yet these approaches do not explicitly model class-dependent reliability after downstream decoding. We find that a state with strong average performance is not uniformly reliable across semantic classes. Based on this finding, we propose Classwise State Routing (CSR), a label-calibrated, training-free framework for diffusion segmentation. Using a small labeled calibration set, CSR estimates classwise reliability over the joint layer–timestep space to guide compact state-region selection and class-conditioned response fusion before the frozen decoder. No model parameter is updated, and all routing decisions remain fixed during evaluation. Controlled ablations with the selected state pools held fixed show that class-conditioned weighting improves over a class-agnostic shared-weighting baseline on all four evaluated datasets. Across six backend-task settings and four benchmarks, the complete pipeline yields improvements in 10 of 12 open-vocabulary semantic segmentation (OVSS) comparisons and all 12 category-agnostic comparisons, with maximum gains of 12.79 and 6.88 mIoU percentage points. The procedure can also be reapplied to alternative diffusion backbones after backbone-specific recalibration.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Models, Training-Free Segmentation, Classwise State Routing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AfH5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Classwise State Routing (CSR), a label-calibrated framework for frozen diffusion segmentation. Motivated by the observation that classes often prefer different layer–timestep states from the strongest shared state, CSR estimates downstream classwise reliability, selects a compact state pool through complementary class coverage, and performs fixed class-conditioned response fusion followed by concentration calibration. No model parameters are updated. Experiments across four datasets and six backend–task settings show improvements in 10 of 12 OVSS comparisons and all 12 category-agnostic comparisons.

Critical strengths

S1. The idea is fresh and well motivated, with a convincing connection between observation and design. Figure 2 shows that reliable states occupy a compact region, yet class preferences remain heterogeneous within it: 17 of 19 Cityscapes classes prefer a state other than the strongest shared state. Using classwise downstream reliability rather than a single global ranking is a sensible direction, and separating shared-pool selection from class-conditioned fusion translates this insight into a practical design. The compact pool addresses the cost of evaluating many states at inference, while Table 2 isolates the benefit of class conditioning under a fixed pool.

S2. Extensive analyses substantiate the method beyond the main performance table. The paper examines calibration-set size, subset reselection, diffusion-noise sensitivity, alternative backbones, and computational cost. In particular, subset reselection retains positive mIoU gains despite changes in the selected states, and the backbone experiments demonstrate that the procedure remains useful after recalibration. The explicit separation of pooling, class-conditioned weighting, response calibration, and decoder calibration makes the reported gains easier to interpret and supports the practical credibility of the framework.

Critical weaknesses

W1. The calibration budget lacks an actionable selection criterion. Using different image counts across datasets is reasonable, and the required subsets are small. However, the paper does not explain how a practitioner should choose that count. Figure 4 shows non-monotonic performance as the budget increases, so more calibration images do not straightforwardly identify a better operating point. An analysis of whether held-out calibration performance, reliability-estimate stability, or diminishing coverage gains can identify a suitable budget would make the method more reproducible and usable without consulting evaluation labels.

W2. The scope of class-conditioned generalization remains limited to calibration-supported categories. All benchmark classes are represented in the OVSS calibration subsets, while unsupported classes fall back to shared reliability. The experiments therefore establish the effectiveness of class-conditioned routing for supported categories, but do not show how much benefit remains for novel categories, an important aspect of the open-vocabulary setting. A held-out-class experiment comparing class-conditioned routing for supported classes with the shared fallback for unsupported classes would clarify this boundary.

Questions for authors
  1. How were the calibration budgets of 50/30/100/50 images chosen? Have you investigated a stopping or budget-selection criterion using only calibration data disjoint from evaluation, such as performance on a held-out calibration subset, reliability-estimate stability, or diminishing class-coverage gains? Can such a criterion identify an effective operating point despite the non-monotonic trend in Figure 4, without updating model parameters?
  2. How does CSR perform when some evaluation categories are excluded from calibration? Please distinguish performance on supported and unsupported categories, and compare the shared-reliability fallback with the original baseline and uniform state fusion.
Reviewer_NjVgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that the diffusion state—defined by a timestep and a layer index—that performs best on average across classes may differ from the optimal state for each individual class. Motivated by this finding, CSR uses a labeled calibration dataset to determine a state-selection and fusion strategy offline, then performs open-vocabulary semantic segmentation (OVSS) through class-conditioned response fusion.

Critical strengths
  • The analysis of downstream segmentation reliability provides concrete evidence that the most reliable states for individual classes can differ from the state with the highest average reliability. The quantitative and qualitative OVSS results support the effectiveness of the proposed approach.
  • Furthermore, the authors provide thorough ablation studies (Section 4.4 and Appendices C.1, C.4) demonstrating the method's robustness across different backbones and calibration set configurations.
Critical weaknesses

In Figures S3, S5, and S7, CSR appears to produce false-positive predictions when compared against the ground-truth masks.

Questions for authors
  1. Could the authors quantify the false-positive behavior observed in these examples and explain its potential causes? Does this behavior reflect a limitation of CSR?
  2. What is the computational cost of offline calibration, including candidate-state evaluation and state-pool selection?
Reviewer_NxAMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Classwise State Routing (CSR) for segmentation with frozen diffusion models. The observation is that reliability of a diffusion state (a layer-timestep pair) varies across semantic classes, so a single shared state is suboptimal. Using a small labeled calibration set (30 to 100 images), the method scores all states through the full frozen pipeline, selects a compact pool (3 timesteps x 5 consecutive layers via anchored greedy search on a class-coverage objective), then fuses the pooled responses with fixed per-class softmax weights plus a peak-RMS channel rescaling. A scalar variant without the class axis handles category-agnostic segmentation. No parameter is trained; everything is frozen after calibration. Experiments plug CSR into three OVSS backends (DiffSegmenter, iSeg, Seg4Diff) and three category-agnostic backends (DiffSeg, DiffCut, Seg4Diff) on SD3, over VOC, Cityscapes, PC59 and ADE20K, with ablations separating pooling, shared weighting and class conditioning, calibration-size and subset-stability studies, and backbone transfer to FLUX.1-dev and SD3.5.

Critical strengths
  1. The classwise reliability observation is well documented (Fig. 2: 17 of 19 Cityscapes classes prefer a non-shared state, mean gap 3.38 IoU) and the experimental protocol is unusually careful for this literature: calibration strictly disjoint from evaluation, paired comparisons with shared noise and preprocessing, seed and calibration-subset resampling, and honest reporting of negative cells. This matters for me because many papers in the training-free segmentation space are much sloppier on exactly these points.

  2. Breadth of validation. Six backend-task combinations, four datasets, and recalibrated transfer to two other backbones give reasonable evidence the recipe is reusable, not a single-pipeline trick.

Critical weaknesses
  1. The comparison setting is tilted. CSR consumes labeled images plus a large per-dataset configuration search (Tables S2, S4 to S7: κ, ρ, K and even the sign of α change per dataset and backend), while the baselines it is added to use no labels. The budget-matched control is missing: spend the same 30 to 100 labeled images on a simple alternative (linear probe on frozen features, or tuning the baseline’s own hyperparameters). Without it one cannot tell whether the routing machinery matters or any use of this supervision would do.

  2. The contribution attributable to the central idea is small. In Table 2, uniform fusion over the selected pool captures most of the gain; class conditioning adds only 0.24 to 1.76 mIoU, and shared-reliability weighting often underperforms uniform. Subset resampling yields state pools with Jaccard overlap as low as 0.20, so no stable routing structure is identified. What remains is a grid-search-plus-softmax recipe at 3.6x inference cost, which also hurts two of three OVSS backends on VOC.

  3. All comparisons are self-paired (backend vs backend+CSR); the pipelines are never positioned against the stronger training-free OVSS field. CLIP-based methods (SCLIP, ClearCLIP, ProxyCLIP, NACLIP, CaR) are absent entirely, though their published numbers on these benchmarks are generally at or above the best CSR-augmented results, with zero labels and far lower cost. OVDiff and FreeDA are cited but not evaluated; GoCA appears only in an appendix under an uncontrolled protocol. The paper shows CSR improves three backends, not that the resulting pipelines are competitive for the task.

  4. The open-vocabulary claim is untested, and the mechanism degrades exactly where it is most needed. Calibration covers every evaluated category on all four benchmarks; unseen classes fall back to shared reliability, and this path is never evaluated (no held-out-class experiment). Meanwhile on ADE20K, 150 classes share 50 calibration images, so per-class support is tiny and shrinkage (κ=45) pulls most class estimates back to the shared score; the paper itself notes shared pooling accounts for most of the ADE20K gain. The classwise mechanism thus works only on small closed vocabularies and vanishes as class count grows, which contradicts the OVSS framing.

  5. The true calibration cost is not reported. Offline calibration runs the full pipeline for 1,200 states times the calibration set (60,000 forward-plus-decode passes on ADE20K), plus the (κ, ρ, K, α) grid. Only online latency appears in Table 4; no wall-clock calibration time is given, which flatters the lightweight framing.

Questions for authors
  1. Add a supervision-matched baseline using the same calibration images (e.g. linear probe on frozen SD3 features). How much of the gain survives?

  2. Run a held-out-class calibration experiment: calibrate without a subset of classes and report IoU on those classes. Also report the ADE20K per-class support distribution and how far the classwise weights actually deviate from uniform.

  3. Place the best CSR-augmented results next to current training-free OVSS methods (ProxyCLIP, NACLIP, OVDiff, FreeDA) under one protocol. If they remain below zero-label CLIP baselines, what is the practical case for labeled calibration at 3.6x inference cost?

  4. Report wall-clock calibration time per dataset.

  5. For Cityscapes iSeg, Table S4 mentions a post-hoc diagnostic region on the full 28x24 grid differing from the frozen configuration. Which region produced the main Table 1 numbers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.