acceptodds
Withdrawn from ICLR 2027

DuetMoE: Coupling Inter- and Intra-Subgroup Robustness for Fair Medical Image Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As medical AI expands across diverse healthcare settings worldwide, equitable performance across patient populations is becoming essential to trustworthy clinical use. Fairness in medical image analysis is often evaluated through average performance across predefined subgroups, yet similar subgroup averages can conceal substantial variation among individual patients. Therefore, a reliable medical AI requires addressing two complementary objectives: inter-subgroup fairness, which reduces performance disparities across groups, and intra-subgroup robustness, which protects poorly served patients within each group. To jointly address these objectives, we propose DuetMoE, a subgroup-aware mixture-of-experts framework that couples group-level adaptation with patient-specific clinical guidance, enabling more reliable medical image analysis for individual patients. For settings without linked clinical records, we further introduce DuetMoE+, which retains subgroup-routed experts and incorporates a KL-constrained distributionally robust objective to address intra-subgroup robustness. We evaluate our methods on PI-CAI, radiotherapy, and Harvard-FairSeg. Across four evaluation settings, our methods lead in overall or equity-scaled performance, reduce the inter-subgroup mean Dice gap by up to 30.7 percent, and raise intra-subgroup 25th-percentile Dice by up to 11.5 points over the strongest reported baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
medical image analysis, fairness, MoE, multimodal learning, distributionally robust optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VubKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DuetMoE, a mixture-of-experts framework for fair medical image analysis. The authors consider two aspects of fairness: inter-subgroup fairness and intra-subgroup robustness. DuetMoE uses subgroup-specific routers and patient-specific clinical information to guide expert selection and feature processing. For datasets without clinical records, DuetMoE+ combines subgroup routing with KL-based distributionally robust optimization. Experiments are conducted on PI-CAI, radiotherapy, and Harvard-FairSeg datasets.

Critical strengths
  • The paper studies a meaningful problem in medical image analysis. Considering both inter-subgroup fairness and intra-subgroup robustness is reasonable, since good average performance cannot guarantee good performance for every patient.

  • The overall framework is well motivated. Combining subgroup information with patient-specific clinical context is an interesting direction for personalized medical image analysis.

  • The experiments cover different imaging modalities and clinical tasks, including prostate cancer detection, radiotherapy segmentation, and retinal segmentation. Both multimodal and unimodal settings are considered.

  • The experimental evaluation is relatively comprehensive. The authors compare with several fairness and segmentation baselines, and provide subgroup-level results, patient-level performance distributions, and ablation studies.

  • The proposed methods achieve improvements in several settings, especially in the worst-performing subgroups and low-performance patient groups. These results show some practical potential of the framework.

Critical weaknesses
  1. Limited scalability and flexibility of subgroup-specific routing. This is my main concern. Fairness can be evaluated under many different sensitive attributes, such as age, sex, race, and institution. However, the proposed method requires a predefined subgroup partition and constructs a separate router for each subgroup. If the fairness attribute changes from institution to age or sex, it seems that the routers need to be reconstructed or retrained. Moreover, in real applications, multiple sensitive attributes may need to be considered simultaneously. Constructing routers for all combinations of attributes will introduce more parameters and data sparsity problems, especially for small subgroups. The method also requires subgroup labels during inference. Therefore, I think this design is difficult to scale to general fairness settings, and its practical applicability is limited. The paper does not sufficiently discuss or evaluate this issue.

  2. Limited methodological novelty. The proposed framework mainly combines several existing techniques. Subgroup-specific routing is inherited from dMoE, clinical information is incorporated through standard cross-attention mechanisms, and the intra-subgroup optimization is based on existing KL-DRO. The two proposed variants also use different mechanisms to improve intra-subgroup robustness, rather than showing a unified solution. I think the paper does not clearly explain what new technical insight is introduced beyond combining these existing components. More importantly, the necessity of subgroup-specific MoE routing is not sufficiently justified. A simpler shared router conditioned on subgroup embeddings, or other conditional feature adaptation methods, may achieve similar results. The current experiments do not clearly show why the proposed routing design is necessary.

Questions for authors

See the weakness.

Reviewer_n5FfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Summary: This paper considers the challenge of fair medical image segmentation. It tries to both (1) introduce an expanded definition of fairness that considers two factors (minimize performance discrepancy across subgroup AND protect poorly served patients within each subgroup) and (2) introduce a new methodology DuetMoE(+) to tackle this expanded definition of fairness. The DuetMoE extends previous fair MoE method (dMoE) to exploit auxiliary patient metadata where available. Where auxiliary metadata is not available, it instead applies DRO within each subgroup and denotes this DuetMoE+. The results show some improvement on equality-scaled version of various segmentation metrics (across subgroup subgroup) and bottom % performance (within-subgroup evaluation).

Critical strengths

Strength: S1: The clinical metadata conditioning used in DuetMoE is sensible.

S2: DRO is a sensible way to robustify the distribution of predictions.

S3: Most of the evaluations show improved performance.

Critical weaknesses

Weaknesses:

W1: Problem definition. The specific expanded definition of fairness chosen for consideration and evaluation is not sufficiently justified and defensible. The paper chooses standard group fairness + within-group worst cases. However:

W1.1: The big classic textbook dichotomy in fairness notions are group and individual fairness. But these are not the ones chosen. So why choose the two considered here?

W2.2: The latter “bottom percentile” definition used is much more related to the literature on tail risk minimization, and distributionally robust optimization than fairness per-se. So the position the paper has taken very much blurs the boundary between typical fairness notions and distributional robustness notions.

W1.3: Assuming we agree that group fairness and some notion of distributional/tail risk robustness are an expanded notion of fairness to optimize, then why should the latter tail risk robustness be subgroup conditional, rather than the overall tail-risk across the whole population? For example, if visible subgroup=race, but the specific instances which are failing are due to a latent factor of age, which orthogonally cross-cuts racial subgroups. Then it could make more sense to want to optimize both the subgroup fairness and the global tail risk.

W1.4: Hierarchical DRO and others (Fairness Gerrymandering, Hidden Stratification) already consider multiple granularities of fairness. So, the current combination of macro and micro fairness notions isn’t a fundamentally novel perspective on fairness.

W2: Method/narrative disjointness: DuetMoE and DuetMoE+ contributions are essentially totally different methods, despite the paper trying to position them as being tightly related. DuetMoE expands prior dMoE routing with additional clinical metadata conditioning. DuetMoE+ differently expands dMoE with the distributional robustness loss. So they are essentially completely different extensions built on the original dMoE work.

W3: Methodology justification

W3.1: Assuming we accept wanting to optimize both group fairness and bottom percentile robustness, then why apply tail risk optimization within subgroup rather than globally for DuetMoE+? (See also W1.3).

W3.2: DuetMoE+ uses KL-DRO to optimize the tail risk. However, the link between train risk and test risk is not guaranteed. Roughly speaking, test set mistakes that are due to training underfitting are likely to improve by additional optimization on related examples in training. However, test set mistakes due to overfitting could be worsened by additional optimization on related examples in training. There is nothing provided to distinguish these two cases to make sure we are taking the right intervention. This is a known weakness of DRO type methods. However, this issue is likely to be much worse in the tail risk optimization case considered here than in the typical GroupDRO used in the group fairness literature. In realistic data with outliers annotation errors, etc the outliers/errors will be in the tail risk distribution. So the KL-DRO will spend more optimization effort memorizing the data points that are already noisy and counterproductive.

W4: Method novelty. DuetMoE+ is of limited novelty since it’s vanilla dMoE+standard KL-DRO.

W5: Evaluation issues & limitations:

W5.1: A major pitfall in fairness studies is that specific choice of validation metric for model selection can have a bigger impact than the fairness intervention itself [Zong, Medfair, ICLR23]. For reliable quantitative comparison, the validation criterion should reflect the fairness metric being evaluated, and model/hyperparameter/checkpoint selection should be done the same way across competitor methods in qualitative comparison. However, this seems to not be the case in the current evaluation. The validation protocol differs in ad-hoc ways across experiments (App B.3), is often fixed iteration based rather than validation criteria based, and is not fully specified for other experiments. Some competitors results are taken from other papers with different HPO, so are not quantitatively comparable. App Table 12 illustrates this issue for one hyperparameter in this paper. You can pretty much minimize or maximise any metric in the table by different choice of radius hyperparameter.

W5.2: ES metrics quantify between-group disparity reasonably consistently across the paper. However, the within-group evaluation is not uniform across all the tasks. Particularly PC-AI Tab 5+6 use of “Sensitivity at 90% specificity” seems to not be a tail risk measure of the kind the paper claims to optimize. Why not report some kind of bottom quartile to better reflect tail risk (as already done in others such as Tab 12, etc)?

Questions for authors

W5.2: Why sensitivity rather than quartile related tail risk evalution metric?

Reviewer_vKp6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the need to consider not only overall accuracy but also performance disparities across subgroups when applying models across different hospitals and patient populations. The authors propose DuetMoE to jointly address inter-subgroup fairness and intra-subgroup robustness. The former concerns differences in average performance across subgroups, while the latter focuses on prediction reliability for poorly served patients within each subgroup. DuetMoE is a mixture-of-experts framework that combines subgroup adaptation with patient-specific guidance. A patient’s subgroup label selects the corresponding router, which then selects experts from a shared expert pool to process image features. For settings without linked clinical records, the authors introduce DuetMoE+, which retains subgroup-specific routing and employs within-subgroup KL-constrained distributionally robust optimization. This increases the training weights of high-loss patients to improve predictions for poorly served patients within each subgroup.

Critical strengths
  1. Well-motivated problem formulation: Many existing studies in medical image analysis focus on improving average performance, whereas reliable medical models must perform consistently well across different groups. The problem addressed by this paper is therefore highly relevant to current research.
  2. Dynamic retrieval of clinical guidance: After routing, each expert in DuetMoE uses the patient’s image features assigned to it to query that patient’s clinical memory again. This makes the clinical guidance both patient-specific and expert-specific.
  3. Addressing difficult cases: DuetMoE+ combines subgroup adaptation with training that emphasizes difficult cases within each subgroup. Subgroup-specific routing adapts the model to different patient groups, while within-subgroup KL-DRO gives greater training weight to high-loss patients within each group.
Critical weaknesses
  1. Dependence on accurate subgroup labels at inference: DuetMoE requires linked clinical records, and both variants need to know each patient’s subgroup membership at inference. These requirements make practical deployment the method’s main limitation.
  2. Architectural complexity: DuetMoE uses a frozen 8B language model, multiple clinical attention mechanisms, and mixture-of-experts modules. However, the paper does not systematically compare computational costs, such as parameter counts, GPU memory usage, and inference speed.
  3. Inconsistent fairness improvements: On PI-CAI, DuetMoE achieves the highest overall PI score, but its ES-PI is lower than that of MoME. Its PI score on PCNN, the lowest-performing institution, is also lower than MoME’s. This shows that improved overall performance does not automatically translate into better inter-subgroup fairness. The theoretical analysis characterizes properties of the training loss, and the authors explicitly acknowledge that it does not guarantee improvements for every patient.
  4. Stability of experimental results: The T1 test subgroup in the RT dataset contains only 11 cases. Such a small sample makes it difficult to obtain stable results.
Questions for authors

Since this paper primarily focuses on fairness in medical applications, I consider dependence on accurate subgroup labels at inference and inconsistent fairness improvements to be the two most important issues that need to be addressed.

  1. Dependence on accurate subgroup labels at inference: DuetMoE requires linked clinical records, and both variants need to know each patient’s subgroup membership at inference. These requirements make practical deployment the method’s main limitation.
  2. Inconsistent fairness improvements: On PI-CAI, DuetMoE achieves the highest overall PI score, but its ES-PI is lower than that of MoME. Its PI score on PCNN, the lowest-performing institution, is also lower than MoME’s. This shows that improved overall performance does not automatically translate into better inter-subgroup fairness. The theoretical analysis characterizes properties of the training loss, and the authors explicitly acknowledge that it does not guarantee improvements for every patient.

If the author can address these two issues, I would be inclined to increase the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.