- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Mixture of Prototype Experts (MoPE) for single-domain generalized object detection. The method organizes visual conditions into photometric, textural, and spatial prototypes, and employs multi-label preference optimization to guide expert routing based on multimodal annotations. Multi-style augmentation further diversifies the source distribution. Experiments across weather and artistic domain shifts demonstrate improvements over existing methods.
Critical strengths
-
The figures are well designed and clearly illustrate the motivation, overall architecture, and key components, making the proposed framework easy to understand.
-
The manuscript is clearly written and logically structured. The abstract and introduction effectively distinguish the proposed approach from existing methods, while the methodology provides coherent explanations and smooth transitions between components.
-
The integration of semantic prototype preferences with sparse expert routing is an interesting idea. In particular, using multi-label preference optimization to guide expert selection provides a potentially useful alternative to conventional task-driven MoE routing.
Critical weaknesses
-
The proposed Multi-Style Augmentation (MSA) relies on FLUX.2 to generate diverse weather and illumination conditions. However, visually convincing synthetic images do not necessarily follow the distributions of real-world environments. In my previous experiments, I observed that training an object detector on rainy images generated by pix2pix resulted in worse performance on real rainy images than training with daytime images alone. Although this observation involves a different generation model, it highlights a potential limitation of synthetic weather augmentation. The manuscript does not sufficiently investigate whether the generated distributions accurately capture the visual degradation encountered in real unseen domains. While Appendix G provides qualitative checks of annotation consistency, these do not establish the distributional realism or transferability of the generated samples.
-
The central assumption of MoPE is that experts corresponding to the semantic attributes identified by the multimodal model should be preferred during routing. However, a semantically relevant expert is not necessarily the most effective expert for object detection. MPO primarily encourages the router to select experts consistent with the multimodal annotations, but does not explicitly ensure that these experts learn more discriminative or domain-robust features. In my understanding, the framework essentially performs input-dependent feature transformations through lightweight convolutional and attention-based experts, followed by weighted aggregation with a shared expert. It remains unclear whether these experts acquire genuinely complementary detection capabilities or whether the improvements mainly arise from the increased model capacity and diversified training data. The routing visualizations demonstrate semantic consistency but provide limited evidence of actual expert specialization.
-
The framework manually defines 28 prototypes distributed across P3, P4, and P5, with 4 photometric, 8 textural, and 16 spatial prototypes, respectively. This design is primarily motivated by assumptions about the hierarchical characteristics of YOLO backbone features. However, the manuscript does not provide sufficient experimental evidence to justify the number, semantic definitions, or stage assignments of these prototypes. Although Appendix E investigates the number of activated experts, this is different from validating the predefined prototype taxonomy itself. Comparisons with alternative prototype configurations, different stage assignments, and expert selection strategies are needed to establish whether the proposed design is necessary and effective.
Questions for authors
-
How do the authors assess the distribution gap between FLUX.2-generated images and real-world weather conditions? Could unrealistic synthetic distributions negatively affect generalization to real unseen domains?
-
Why should semantically relevant experts necessarily be more effective for detection?
-
Do different prototype experts learn genuinely specialized and complementary features? Can the authors provide quantitative feature-level analysis or controlled expert-swapping experiments beyond routing activation visualizations?
-
What motivates the specific allocation of 4, 8, and 16 prototypes to P3, P4, and P5? How sensitive is performance to the prototype taxonomy, expert counts, and semantic-to-stage assignments?
8 comments
Response to SusM — W2, Q2 & Q3: Expert Specialization and Sources of Performance Improvement (Part 1/2)
We sincerely thank the reviewer for these thoughtful and important concerns. We agree that semantic relevance alone does not necessarily imply superior detection capability, and that routing visualizations alone are insufficient to establish meaningful expert specialization. We would like to clarify the underlying mechanism and provide additional experimental evidence addressing these concerns.
(1) Why should semantically relevant experts be more effective for detection?
Our method does not assume semantically relevant experts are inherently more effective for detection. Instead, their specialization is encouraged and progressively learned during training.
The predefined prototypes serve as semantic anchors rather than representing pre-existing expert capabilities. Initially, the experts are independently parameterized without predefined detection specializations. During training, MPO encourages the router to assign higher selection probabilities to experts whose prototypes match the visual attributes of the input image. Since MoPE employs sparse Top-K routing, the selected experts participate in feature computation and receive task-driven gradients from the detection loss.
Consequently, different experts are exposed to different visual conditions through preference-guided routing, encouraging them to gradually develop condition-specific feature representations. For example, the daytime expert is more frequently activated for daytime images and consequently receives more detection supervision from such scenes.
Importantly, MPO guides expert selection, while the detection objective drives the learning of discriminative features. Thus, semantic specialization is encouraged through the interaction between preference-guided routing and detection optimization, rather than being an assumption imposed by the prototype labels.
This mechanism is supported by Fig. 3, which demonstrates semantically differentiated routing across unseen domains. Furthermore, Table 4 reports an improvement from 29.00% to 35.97% without MSA. With MSA, the target-domain average [email protected] improves from 40.30% to 44.21% when MPO is introduced.
(2) Controlled expert-swapping experiments: Do experts learn genuinely specialized capabilities?
Following the reviewer's suggestion, we conducted additional controlled expert-swapping experiments to quantitatively investigate the functional differentiation of the learned experts beyond routing activation visualizations.
Specifically, we conducted controlled interventions on the trained MoPE models by altering only the routing-to-expert index correspondence during inference, keeping all learned model parameters unchanged, without any retraining or fine-tuning.
At P3, we swapped the illumination expert corresponding to each evaluation domain with the unknown expert. At P4, we consistently applied the following fixed swaps across all domains: realistic ↔ fog, rain ↔ artistic, and clear ↔ snow.
We evaluated three settings: P3-only swapping, P4-only swapping, and joint P3+P4 swapping, across the source domain and four unseen target domains using three random seeds.
The expert-swapping results, their interpretation, and the capacity-matched comparison are presented in Part 2/2 below.
Response to SusM — W2, Q2 & Q3: Expert Specialization and Sources of Performance Improvement (Part 2/2)
Continuing our response to W2, Q2, and Q3, we present the expert-swapping results for the interventions described in Part 1/2, followed by their interpretation and the capacity-matched comparison.
The results are summarized below. Each entry reports [email protected] / mAP@[0.5:0.95] (%, mean ± standard deviation).
| Method | Day Clear (Source) | Night Clear | Dusk Rainy | Night Rainy | Day Foggy | Target Avg. |
|---|---|---|---|---|---|---|
| Original MoPE | 62.77±0.21 / 38.55±0.16 | 51.30±0.53 / 29.70±0.28 | 47.87±0.47 / 28.42±0.29 | 32.70±0.66 / 19.22±0.27 | 44.97±0.12 / 28.04±0.11 | 44.21±0.38 / 26.34±0.21 |
| P3 Swapping | 61.57±0.34 / 37.80±0.15 | 49.56±0.43 / 28.72±0.22 | 46.02±1.12 / 27.42±0.68 | 29.04±0.50 / 17.17±0.29 | 43.42±0.29 / 27.08±0.18 | 42.01±0.32 / 25.10±0.18 |
| P4 Swapping | 61.08±0.41 / 37.11±0.37 | 48.71±0.72 / 27.98±0.53 | 46.03±0.92 / 27.15±0.64 | 30.46±1.20 / 17.72±0.67 | 43.64±0.25 / 26.86±0.30 | 42.21±0.73 / 24.93±0.52 |
| P3+P4 Swapping | 60.30±0.42 / 36.76±0.42 | 47.82±1.01 / 27.55±0.60 | 44.66±0.81 / 26.51±0.37 | 27.95±0.69 / 16.46±0.34 | 42.94±0.23 / 26.55±0.38 | 40.84±0.60 / 24.27±0.39 |
Consistent performance degradation. All three swapping strategies reduce mean detection performance across the source domain and all four unseen target domains. Specifically, P3-only and P4-only swapping reduce the average target-domain [email protected] by 2.20 and 2.00 percentage points, respectively, while joint swapping results in a larger decrease of 3.37 points (44.21% → 40.84%).
Consistent degradation under stricter evaluation metrics. The corresponding reductions in mAP@[0.5:0.95] are 1.24, 1.41, and 2.07 points, respectively. Under joint swapping, the average mAP@[0.5:0.95] decreases from 26.34% to 24.27%. Notably, Night-Rainy exhibits the largest [email protected] degradation under joint swapping, decreasing from 32.70% to 27.95% (−4.75 points).
(3) Evidence of functional expert differentiation.
The key observation is that only the routing-to-expert assignments are modified, while all learned model parameters remain unchanged. If different experts had learned nearly identical feature transformations, exchanging their routing assignments would be expected to have little effect on detection performance. However, the consistent performance degradation observed across different domains suggests that the learned experts have developed functionally differentiated behaviors and that their original routing correspondences play a meaningful role in detection.
Together with the semantically differentiated routing patterns in Fig. 3, these results provide further quantitative support for the effectiveness of preference-guided training in encouraging condition-dependent expert specialization. The observed performance differences also suggest that semantic routing alignment is accompanied by meaningful functional differences among the learned experts, rather than merely reflecting the predefined prototype labels.
(4) Disentangling expert design from augmentation and model capacity.
To further address the concern that improvements may primarily result from additional model capacity or diversified training data, we conducted a capacity-matched comparison with an expanded YOLO-ESMoE baseline using the same MSA-augmented data, 4/8/16 experts, and Top-K settings of 1/2/3.
The expanded ESMoE contains 55.704M parameters, compared with 55.645M for MoPE. Their inference costs are 112.9 GFLOPs and 112.1 GFLOPs, respectively. Despite this closely matched capacity, MoPE achieves 44.21% vs. 41.97% [email protected] and 26.34% vs. 24.96% mAP@[0.5:0.95], consistently outperforming the expanded baseline across all four target domains.
These results demonstrate that MoPE's improvements cannot be explained solely by data augmentation or increased model capacity.
Overall, the additional experiments strengthen the evidence that preference-guided expert routing contributes to meaningful, condition-dependent functional differentiation and improved detection generalization. While these results do not fully establish the independence or optimality of individual expert representations, they provide quantitative support beyond semantic routing consistency.
We sincerely appreciate the reviewer's constructive feedback and will incorporate these additional experimental results into the revised manuscript.
Response to SusM — W1 & Q1: Distribution Realism and Transferability of MSA
We thank the reviewer for this insightful comment and for sharing the observation regarding negative transfer from pix2pix-generated rainy images. We agree that visually plausible synthetic images do not necessarily match the distribution of real-world weather conditions, and that generative augmentation may sometimes introduce unrealistic artifacts.
(1) Distribution gap assessment and motivation of MSA.
Regarding distribution gap assessment, our current evaluation focuses on annotation consistency and downstream transferability rather than explicitly measuring statistical divergence between synthetic and real weather distributions.
However, we would like to emphasize an important distinction between distribution fidelity and downstream task effectiveness. The primary objective of MSA is not to precisely reproduce the probability distribution of real weather conditions, but to introduce diverse, label-preserving appearance variations that improve the detector's robustness to unseen domains. Exact distribution matching is not a prerequisite for augmentation to be beneficial.
Under the SDGOD setting, target-domain data and statistics are unavailable during training. Therefore, MSA aims to enrich source-domain visual diversity rather than explicitly approximate any particular target distribution. Its fixed prompt bank is constructed from representative photometric and textural prototypes, without using target-domain images, annotations, or statistics. These generated variations also provide diverse observations for prototype preference supervision.
Regarding annotation quality, Appendix G provides qualitative consistency checks between original and generated images, showing that object content, scene geometry, and bounding-box alignment are generally preserved in the inspected examples. We acknowledge that these checks assess annotation consistency rather than establishing an exact match with real-world weather distributions.
(2) Experimental evidence of transferability and potential negative transfer.
Our existing ablation results provide direct evidence of such benefits. As shown in Table 4, introducing MSA alone, without MPO, improves the average [email protected] across four real-world unseen weather domains from 29.00% to 40.30% (+11.30 percentage points). Importantly, improvements are observed consistently across all four domains, including Dusk-Rainy (+15.33 points) and Night-Rainy (+16.43 points). These results demonstrate that the generated variations contribute positively to detection performance on real unseen-domain images.
Moreover, several recent studies have independently demonstrated the effectiveness of diffusion-generated training data for object detection, including InstaGen (CVPR 2024), GeoDiffusion (ICLR 2024), and ODGEN (NeurIPS 2024). These findings further support the potential of modern generative approaches to improve downstream detection performance across different settings, without requiring generated images to perfectly reproduce real-world data distributions.
We further clarify that MSA is not designed or claimed to match target-domain distributions. Instead, it introduces diverse visual variations through a fixed prototype-based prompt bank while preserving object content and geometry, without using target-domain statistics. The reviewer's pix2pix experience highlights a legitimate risk of synthetic augmentation, but our experimental results indicate that MSA does not exhibit negative transfer in the evaluated domains.
(3) Limitations and clarification.
We acknowledge that these results do not establish an exact statistical match between synthetic and real weather distributions or guarantee improvements under every possible unseen condition. Nevertheless, the consistent improvements on real unseen-domain benchmarks provide direct empirical evidence of the task-level transferability of MSA, which is the primary objective of our augmentation strategy.
We sincerely appreciate the reviewer's constructive suggestion and will explicitly clarify the distinction between distributional realism, annotation consistency, and downstream transferability in the revised manuscript.