Calibration-Guided State Selection and Class-Conditioned Fusion for Frozen Diffusion Segmentation

Diffusion representations vary substantially across network layers and denoising timesteps. Prior work exploits this variation through fixed states, task-level selection, shared aggregation, or pixel-adaptive rules, yet these approaches do not explicitly model class-dependent reliability after downstream decoding. We find that a state with strong average performance is not uniformly reliable across semantic classes. Based on this finding, we propose Classwise State Routing (CSR), a label-calibrated, training-free framework for diffusion segmentation. Using a small labeled calibration set, CSR estimates classwise reliability over the joint layer–timestep space to guide compact state-region selection and class-conditioned response fusion before the frozen decoder. No model parameter is updated, and all routing decisions remain fixed during evaluation. Controlled ablations with the selected state pools held fixed show that class-conditioned weighting improves over a class-agnostic shared-weighting baseline on all four evaluated datasets. Across six backend-task settings and four benchmarks, the complete pipeline yields improvements in 10 of 12 open-vocabulary semantic segmentation (OVSS) comparisons and all 12 category-agnostic comparisons, with maximum gains of 12.79 and 6.88 mIoU percentage points. The procedure can also be reapplied to alternative diffusion backbones after backbone-specific recalibration.