acceptodds
Desk rejected at ICLR 2027

Uncertainty-Aware Evidential Prototype Reconstruction for Robust Multimodal Federated Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Federated Learning (MMFL) facilitates privacy-preserving collaboration across distributed multimodal data, yet existing methods universally assume clean labels. Real-world federated ecosystems, however, exhibit label noise compounded by modality heterogeneity, with structural patterns unique to multimodal settings: semantic mismatch across modalities, asymmetric corruption affecting individual modalities, and consistent label errors where coherent modalities collectively indicate incorrect labels. We propose FedEPR, a phased reconstruction framework grounded in evidential uncertainty quantification. By reformulating classification within Dempster-Shafer theory, our approach models epistemic uncertainty via Dirichlet distributions, enabling principled detection of inter-modal disagreement through a modal conflict factor. A globally-guided prototype calibration mechanism further anchors local representations to server-aggregated semantics, counteracting prototype drift under Non-IID distributions, while modality-decoupled evidential aggregation adaptively handles missing modalities and suppresses high-variance gradient updates from noisy clients. Extensive experiments demonstrate that FedEPR outperforms state-of-the-art methods by over 12% under severe noise conditions.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Multimodal Federated Learning, Evidential Uncertainty, Label Noise, Modality Conflict

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_KGc6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies multimodal federated learning with unreliable supervision. It considers three kinds of corruption: the two modalities show different classes, one modality is degraded, or the label is wrong while both inputs agree.

The proposed method, FedEPR, has four parts. Each modality has an evidential head that outputs class beliefs and an uncertainty score. Dempster–Shafer fusion combines the two modalities and measures how much they conflict. Each client mixes its class prototypes with global prototypes from the server. A fixed sequence of rules then sets the training label: it uses one modality's prediction, the fused prediction, or the original label, depending on conflict and uncertainty. The server aggregates each modality's model separately and gives more weight to clients with higher total evidence.

Experiments use AV-MNIST, UPMC Food-101 and CMU-MOSEI with injected noise. The paper also tests naturally noisy data (raw UPMC, a low-agreement CMU subset, WebVision) and reports ablations and a human audit of 1,500 samples.

Critical strengths
  • A useful problem framing: Separating disagreement between modalities, degradation of one modality and label error is a sensible way to think about unreliable supervision in multimodal FL, and the setting (non-IID data with unimodal clients) is realistic.

  • Evaluation beyond the designed noise: The paper adds naturally noisy data, a human audit, and noise processes outside its generator, and it states its limitations openly (confidently wrong clients, both modalities degraded, overlap between generator and rules).

Critical weaknesses

The reported gains are not attributable to the label-reconstruction mechanism the paper is about

  • Ablation: In Table 3, the variant without reconstruction reaches 91.82 / 80.47 / 43.56, which already exceeds every baseline in Table 1 at the same setting (best: 89.17 / 74.56 / 42.63). Reconstruction adds 3.65, 4.87 and 2.59 points, a minority of the total gain over FedAvg. Most of the improvement therefore comes from the evidential heads, DS fusion and prototypes, which the baselines do not have. No results at 0% noise are given, so it is unknown whether the gap exists without any corruption.

  • Noise definition: Under the 40/30/30 mixture, only 30% of corrupted samples have a wrong label; "60% noise" means 18% label noise. For semantic mismatch and asymmetric corruption the observed label is correct, yet the first two branches of Eq. (6) replace it with a modality's prediction without consulting it. In these cases, reconstruction can at best reproduce the given label. The benefit there more likely comes from masking the unreliable branch and down-weighting the sample, which is not tested against a baseline that keeps labels and only masks or re-weights.

  • Aggregation: Evidence-weighted aggregation is presented as a component but is not ablated in Table 3. Since evidence is learned from noisy labels, a client that memorizes its noise gains weight; the paper notes this but does not measure it.

  • Reliance on evidential uncertainty: All routing decisions depend on u = C/S after 20 warm-up rounds on noisy labels, and on eight thresholds and weights. Figure 6 shows a spread of about six points over the τ and λ_KL grid. How these are selected, and whether the validation data are clean, is not stated in the main text.

The comparison and positioning do not match the problem studied

  • Baselines: All methods in Tables 1 and 2 are missing-modality or generic FL methods with no mechanism for noisy supervision. Noisy-label FL methods appear only in the appendix, adapted branch by branch. No baseline combines multimodal fusion with noise handling, such as evidential multi-view learning under label noise (TMNR, Xu et al., IJCAI 2024) or conflict-aware evidential fusion (ECML, Xu et al., AAAI 2024) trained with FedAvg, or a standard sample-selection method applied across the two modality branches.

  • Generator overlap: The controlled generator produces exactly the three patterns the rule is built to detect, as the paper acknowledges. The gains on natural noise are much smaller (about 4 points), and these are the more informative results.

  • Related work. The claim to "pioneer" the study of multimodal label noise should be tempered. Exchanging class centroids between server and clients to correct noisy labels is already established (Yang et al., IEEE Intelligent Systems 2022), and federated prototypes more generally (FedProto, Tan et al., AAAI 2022) are not cited. Evidential multi-view learning has been studied under label noise and under view conflict (TMNR; ECML). Confidence-based client weighting in multimodal FL with incomplete modalities has also been proposed (FLISM, arXiv:2405.11828). The contribution is a combination of these elements in a federated setting, and it should be positioned as such.

Questions for authors

What are the results at 0% noise for all methods? Do the baselines use the same encoders and fusion as FedEPR?

With reconstruction disabled but the auxiliary-loss mask and sample weights kept (and the reverse), how do the results change? For mismatch and asymmetric corruption, how often does reconstruction change a correct label?

What is the effect of replacing evidence-weighted aggregation with sample-size weighting? How do the weights behave for clients with the highest noise rates?

How are the thresholds chosen, and is the validation set clean?

Please compare with a multimodal noise-robust baseline under the same protocol (e.g., TMNR or ECML with FedAvg), and move the noisy-label FL baselines into the main comparison.

Please report per-seed results for Table 1.

14 comments
AuthorsOpenReview ↗

Response to Reviewer KGc6

Thank you for the detailed feedback. We respond to each point below.

Weaknesses

[W1] Reconstruction versus the full pipeline

To separate these effects, we varied reconstruction and loss handling independently at 40% corruption and alpha = 0.5, keeping our evidential/prototype pipeline fixed. "Mask + weights" denotes the auxiliary mask and sample weight in Eq. (10).

Reconstruct targetsMask + weightsAV-MNISTFood-101CMU-MOSEI
✗✗89.5478.1241.47
✓✗93.8283.1544.73
✗✓91.8280.4743.56
✓✓95.4785.3446.15

With masking and weighting enabled, we gain 3.65 / 4.87 / 2.59 points by reconstructing targets. Without them, we gain 4.28 / 5.03 / 3.26. Adding mask and weights to reconstruction gives a further 1.65 / 2.19 / 1.42 points. The clean-data control is reported in [Q1] of Part 2.

[W2] Meaning of the noise rate

Here, the corruption rate counts affected examples. Flipped labels account for 30% of the 40/30/30 mixture. At 60% training corruption, the nominal annotation-error fraction is 0.60×0.30=0.180.60\times0.30=0.18.

Following B.2, we now use "training corruption rate" in our headings.

For mismatch and asymmetric corruption, our generator preserves the original annotations. In the single-pattern runs (Table 31), reconstruction improves over keeping labels by 3.41 / 2.36 points on AV-MNIST and 4.11 / 3.01 on Food-101. The mask applies to auxiliary losses, while our fused loss still updates both encoders (C.16).

[W3] Evidence-weighted aggregation

To isolate aggregation, we kept local training fixed and compared the rules at 40% corruption and alpha = 0.5 (Table 32).

AggregationAV-MNISTFood-101CMU-MOSEI
Sample-size weighting94.08 ± 0.2383.26 ± 0.2744.72 ± 0.38
Evidence weighting (ours)95.47 ± 0.1785.34 ± 0.2146.15 ± 0.31

Evidence weighting adds 1.39 / 2.08 / 1.43 points over sample-size weighting. In Table 33, high-noise clients receive evidence weight relative to sample share of 0.76 / 0.74 / 0.72 under wide noise spread and 1.02 / 1.04 / 1.06 under fixed class permutations. Sample-size weighting gives a ratio of one.

[W4] Parameter and validation selection

For selection, we use staged single-factor sweeps on clean validation data: 25 configurations, each run with three seeds. Baselines receive the same budget for their own hyperparameters. Figure 6 is a separate post-selection sensitivity check. We also tested fixed defaults and noisy validation (Table 35).

DatasetClean validationFixed defaultsNoisy validation
AV-MNIST95.4795.1294.58
Food-10185.3484.9683.72
CMU-MOSEI46.1545.8444.93

Fixed defaults stay within 0.38 points of clean-validation selection. With noisy validation, our gains over observed-label training are 2.76 / 3.25 / 1.37. The revised appendix gives the search ranges. Our warm-up setting remains 20 of 100 rounds. The human audit is used only for analysis.

[W5] Relevant baselines

Following your suggestion, we added TMNR/ECML with FedAvg and attention-fusion variants of FedCorr/FedNoRo/FedLN. At 40% corruption, our gains over the strongest added baseline, TMNR, are 4.64 / 8.92 / 2.59 points. The comparison appears in [Q5] of Part 2, Figure 7 and Table 34.

[W6] Generator overlap

We also tested naturally noisy data without synthetic corruption (Tables 2 and 23). We improve over FedMobile by 4.26 points on UPMC Raw, 4.38 on CMU Low-agreement and 3.23/3.78 on IID/non-IID WebVision. The revised evaluation reports natural and synthetic noise separately.

[W7] Prior work

Thank you for pointing out these connections. We replaced "pioneer" with a description of our reconstruction contribution in heterogeneous bimodal FL. The revision credits Yang et al. [1] for centroid exchange and globally guided pseudo-labeling, FedProto [2] for federated prototypes, TMNR/ECML [3, 4] for noisy-label/conflictive multi-view learning, and FLISM [5] for confidence-aware aggregation.

References

  • [1] Yang, S., Park, H., Byun, J., Kim, C. Robust federated learning with noisy labels. IEEE Intelligent Systems, 2022.
  • [2] Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q., Jiang, J., Zhang, C. FedProto: Federated prototype learning across heterogeneous clients. AAAI 2022.
  • [3] Xu, C., Zhang, Y., Guan, Z., Zhao, W. Trusted multi-view learning with label noise. IJCAI 2024.
  • [4] Xu, C., Si, J., Guan, Z., Zhao, W., Wu, Y., Gao, X. Reliable conflictive multi-view learning. AAAI 2024.
  • [5] Orzikulova, A., Kwak, J., Shin, J., Lee, S.-J. Federated learning for time-series healthcare sensing with incomplete modalities. arXiv:2405.11828, 2024.
AuthorsOpenReview ↗

Response to Reviewer KGc6

Questions

[Q1] Results without injected corruption

To examine this, we evaluated all methods from Table 1 at 0% injected corruption (Table 34).

MethodAV-MNISTFood-101CMU-MOSEI
FedAvg96.8286.4749.83
FedProx97.0887.1550.42
FedMSplit97.4688.3452.17
CreamFL97.7289.2653.46
PmcmFL97.9490.1854.58
MPCPFL98.1590.8754.93
FedMobile98.0790.6254.71
FedEPR98.7391.3255.62

At 0%, we exceed the strongest listed baseline, MPCPFL, by 0.58 / 0.45 / 0.69 points. At 40%, our margins over the same method are 6.91 / 12.69 / 3.87. We also evaluated the added noisy-label and multi-view baselines at both rates.

Our encoder settings and comparison budgets appear in B.3 and B.4, respectively. The original noisy-label adaptations average predictions, while our method uses evidential DS fusion. The revised appendix specifies the encoders and the added fusion variants.

[Q2] Labels versus masks and weights

The controls in Table 29 let us separate these effects. With mask and weights enabled, reconstruction adds 3.65 / 4.87 / 2.59 points ([W1], Part 1). With reconstruction in place, mask and weights add 1.65 / 2.19 / 1.42. Our mismatch/asymmetric corruption preserves the original annotations. A reconstruction branch can also return the original class. For initially correct labels, preservation follows Eq. (12),

Pr⁡(y′=y∣y~=y)=1−a,\Pr(y'=y\mid\tilde y=y)=1-a,

where y~\tilde y, yy and y′y' denote the observed, clean and reconstructed labels.

[Q3] Weights of the noisiest clients

We checked these weights directly (Table 33). Relative to their sample share, the noisiest clients receive lower weights under wide noise spread (0.76 / 0.74 / 0.72) and slightly higher weights under fixed class permutations (1.02 / 1.04 / 1.06). The accuracy comparison in [W3] of Part 1 (Table 32) changes only aggregation.

[Q4] Threshold selection

For the controlled benchmarks, we use clean validation data and staged single-factor sweeps. Accuracy stays within 0.38 points of the original selection with fixed defaults and within 1.62 points with noisy validation. The results are in the table in [W4] of Part 1 and Table 35. Appendix B.4 lists all five candidate sets and the 75-run budget per method. Figure 6 is a separate post-selection sensitivity check. For WebVision, we use training data only for selection and reserve the human audit for analysis.

[Q5] TMNR/ECML and main-table baselines

Here are the added comparisons at 40% corruption and alpha = 0.5 (Table 34). Average-fusion rows reuse Appendix C.8; attention changes only prediction fusion.

MethodFusionAV-MNISTFood-101CMU-MOSEI
FedCorrAverage84.7269.8338.56
Attention88.2473.8641.18
FedNoRoAverage86.3569.4738.87
Attention88.7374.2841.64
FedLNAverage86.1870.1539.24
Attention87.5472.6440.53
TMNR + FedAvgNative90.8376.4243.56
ECML + FedAvgNative90.1775.8343.12
FedEPROriginal95.4785.3446.15

Our gains over TMNR, the strongest added baseline at this corruption rate, are 4.64 / 8.92 / 2.59 points.

[Q6] Per-seed results

We provide the individual accuracies underlying Table 1 for seeds 42, 43 and 44 in Part 3 below. They also appear in Table 10 (Appendix C.1).

AuthorsOpenReview ↗

Response to Reviewer KGc6

[Q6] Per-seed results

Here are the individual Table 1 accuracies for seeds 42, 43 and 44 at each training corruption rate, using alpha = 0.5 and 30% unimodal clients. These are also listed in Table 10 (Appendix C.1).

AV-MNIST

MethodSeed20%40%60%
FedAvg4289.0281.8570.83
4389.4681.2069.75
4489.5481.9670.68
FedProx4290.3682.6172.19
4389.8982.7171.24
4490.2982.0372.06
FedMSplit4291.4784.7673.72
4391.8284.1574.47
4491.8784.6874.59
CreamFL4292.5486.3978.54
4392.5986.3177.86
4492.2285.8478.65
PmcmFL4293.8087.4377.47
4393.7586.9377.36
4493.4687.3676.71
MPCPFL4294.4588.6778.09
4394.2088.2878.19
4494.4988.7377.48
FedMobile4293.6589.3379.80
4393.9389.2779.22
4493.9788.9179.90
FedEPR4298.3195.5591.90
4398.1095.5992.29
4498.2895.2792.35

Food-101

MethodSeed20%40%60%
FedAvg4274.4565.1852.85
4373.8765.0653.92
4474.3764.3154.09
FedProx4275.6766.2355.13
4375.1566.1255.28
4475.5965.4454.17
FedMSplit4277.0768.6957.55
4377.5068.0858.41
4477.5768.7958.55
CreamFL4278.3770.9061.65
4378.8270.9860.86
4478.7670.3461.78
PmcmFL4280.0273.5064.72
4380.3773.4264.01
4480.4272.9264.83
MPCPFL4281.6972.8667.13
4382.0672.7867.02
4482.0172.3166.34
FedMobile4280.8274.6968.43
4380.4974.7668.54
4480.8874.2367.78
FedEPR4289.2485.4980.48
4389.2885.1079.90
4488.9985.4380.40

CMU-MOSEI

MethodSeed20%40%60%
FedAvg4241.6735.6028.82
4342.4934.6527.64
4442.3835.4728.65
FedProx4243.3336.3828.59
4342.5936.5129.64
4443.2335.6229.49
FedMSplit4245.5137.9531.97
4345.6138.6631.83
4444.9038.7730.94
CreamFL4246.9840.3333.49
4346.4239.6833.62
4447.0640.4432.70
PmcmFL4247.7941.6434.61
4348.2940.9333.74
4448.3741.5434.49
MPCPFL4248.9442.5335.31
4348.8641.8734.63
4448.3942.4435.42
FedMobile4247.5042.7936.05
4348.0042.8835.23
4448.0842.2235.94
FedEPR4253.4046.2940.64
4353.4646.3739.93
4452.9845.7940.54
Reviewer_Evw5OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FedEPR, an uncertainty-aware evidential prototype reconstruction framework for robust multimodal federated learning under noisy supervision and heterogeneous client data. The method combines Dirichlet evidential learning and Dempster–Shafer fusion to estimate modality uncertainty and cross-modal conflicts, while global prototype calibration provides semantic guidance for label reconstruction. A conflict-aware reconstruction strategy dynamically corrects unreliable labels, and modality-specific evidence-weighted aggregation improves federated model collaboration. The authors evaluate FedEPR on multiple multimodal benchmarks, including AV-MNIST, UPMC Food-101, CMU-MOSEI, and WebVision. Extensive experiments covering structured and natural noise, non-IID distributions, missing modalities, and ablation studies demonstrate consistent improvements over existing baselines.

Critical strengths
  1. The paper addresses the practical challenges of noisy supervision and heterogeneous modalities in multimodal federated learning. By integrating evidential uncertainty estimation, cross-modal conflict analysis, and prototype-guided label reconstruction, the proposed framework provides a coherent and interpretable approach to improving robustness under different corruption patterns. 2.The paper presents extensive experiments across multiple multimodal benchmarks, including both synthetic and naturally occurring noise. The evaluation covers non-IID distributions, missing modalities, ablation studies, and sensitivity analyses, demonstrating consistent improvements over existing baselines. The thorough experimental evaluation and clear presentation strengthen the credibility of the proposed approach and positively influence my overall assessment.
Critical weaknesses
  1. The proposed framework largely integrates established techniques, including Dirichlet evidential learning, Dempster–Shafer fusion, and prototype calibration. Although their combination is well motivated, the core label reconstruction mechanism primarily relies on manually designed conditional rules. The paper does not sufficiently demonstrate why these specific rules offer fundamental advantages over simpler uncertainty-aware reconstruction strategies, which somewhat limits its methodological contribution. 2.The theoretical analysis assumes that stochastic gradient variance is upper-bounded by a quantity inversely proportional to accumulated evidence. However, this relationship is not directly established by Dirichlet evidential learning. While the authors provide supporting correlation analyses, these are limited to a single dataset and training configuration. Furthermore, the independent benefit of evidence-weighted aggregation over existing noise-aware aggregation methods remains insufficiently demonstrated, weakening the justification for this key component.
Questions for authors
  1. The proposed framework combines several established techniques, including Dirichlet evidential learning, Dempster–Shafer fusion, global prototype calibration, and evidence-weighted aggregation. Although their integration is well motivated, the core label reconstruction mechanism in Equation (6) mainly relies on manually designed conditional rules. The authors should further clarify its distinctive technical contributions compared with existing uncertainty-aware pseudo-labeling methods or simpler modality-selection strategies in the introduction.
  2. Although Table 3 demonstrates the overall effectiveness of dynamic label reconstruction, the individual contributions of the three reconstruction branches in Equation (6) remain unclear. It would be helpful to provide more fine-grained ablation studies by disabling each branch individually, with appropriate fallback handling, or to report branch-specific activation frequencies and correction accuracy.
  3. Equation (11) assigns larger aggregation weights to clients with stronger accumulated evidence, implicitly assuming that higher confidence indicates more reliable model updates. However, this assumption may not always hold, particularly under severe label noise or non-IID data distributions, where models can become highly confident in incorrect predictions. Although Appendix C.18 reports correlations between accumulated evidence, gradient variance, and residual label error, these empirical results are limited to 17 client–modality pairs on AV-MNIST under 40% structured corruption at a single training round. The authors are encouraged to provide targeted ablation studies comparing the proposed strategy with standard FedAvg and existing noise-aware aggregation methods under heterogeneous noise conditions.
  4. In Appendix C.8, existing noisy-label federated learning methods, including FedCorr, FedNoRo, and FedLN, are adapted to multimodal settings by independently processing each modality and simply averaging their predictions. However, this adaptation does not explicitly exploit cross-modal interactions, which are central to the proposed FedEPR framework. These baselines may not fully reflect the potential of existing methods in multimodal settings. The authors are encouraged to justify the fairness of this adaptation strategy and maybe consider stronger multimodal fusion alternatives to ensure a more convincing comparison.
  5. Appendix C.11 describes P3 and P4 as input-degradation processes rather than label-flipping processes. However, Table 21 reports relatively high correction precision and substantial fractions of changed labels under these settings. The authors should explicitly define these metrics and explain how successful label correction is evaluated when the original labels have not been modified.
  6. In Appendix A.8, the authors assume that \mathbb{E}|\epsilon_k^m|^2 \leq \sigma_m^2/\bar e_k^m, implying that larger accumulated evidence corresponds to a smaller upper bound on stochastic gradient variance. However, Dirichlet evidential learning only establishes that uncertainty decreases as total evidence increases, i.e., u_k^m=C/S_k^m, without establishing a direct mathematical relationship between evidence strength and gradient variance. For instance, it is possible that \bar e_A>\bar e_B while \mathbb{E}|\epsilon_A|^2>\mathbb{E}|\epsilon_B|^2, which does not contradict the evidential formulation.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Evw5

Thank you for the thoughtful review. Below, we respond to your concerns about reconstruction and aggregation.

Weaknesses

[W1] Contribution of the conditional rules

The main distinction is how we choose supervision: our rule uses conflict, uncertainty and prototype alignment to decide when to change a label and which prediction to use. Under strong conflict, we check a source's alignment with a calibrated class reference as well as its uncertainty. Under reliable agreement, we consider the fused prediction. Otherwise, we keep the observed label.

To examine these choices, we varied the target policy or aggregation rule while keeping the remaining components and unimodal-client handling fixed. We use 40% corruption and alpha = 0.5; the results below also appear in Tables 30 and 32.

ComponentVariantAV-MNISTFood-101CMU-MOSEI
TargetObserved labels91.82 ± 0.2280.47 ± 0.2743.56 ± 0.36
Unconditional fused targets90.35 ± 0.3178.26 ± 0.3641.83 ± 0.47
Uncertainty-gap only92.74 ± 0.2581.53 ± 0.3043.92 ± 0.40
Conflict only93.86 ± 0.2383.17 ± 0.2844.68 ± 0.38
Conflict + uncertainty94.53 ± 0.2084.26 ± 0.2545.37 ± 0.35
AggregationSample-size weighting94.08 ± 0.2383.26 ± 0.2744.72 ± 0.38
Full methodFedEPR95.47 ± 0.1785.34 ± 0.2146.15 ± 0.31

Our full rule improves over observed-label training by 3.65 / 4.87 / 2.59 points. Replacing every label with its fused prediction reduces accuracy below observed-label training on all three datasets.

[W2] Evidence and gradient variance

Thanks for asking how evidence enters the variance bound. Our analysis assumes the evidence-variance relation (Appendix A.8). For positive accumulated evidence and stochastic errors that are mean-zero and uncorrelated conditional on the weights, we obtain

\quad\Longrightarrow\quad \mathbb{E}\Big\\|\sum_k w_k^m\epsilon_k^m\Big\\|^2 \leq\sigma_m^2\sum_k\frac{(w_k^m)^2}{\bar e_k^m}.$$ Minimizing this bound subject to $\sum_k w_k^m=1$ gives $w_k^m=\bar e_k^m/\sum_j\bar e_j^m$, as in Eq. (31). In Appendix C.18 (**Table 28**), we measured a correlation of -0.91 between evidence and gradient variance across 17 client-modality pairs. Using mean evidence per sample gives a correlation of -0.83. The revised text states this assumption alongside Eq. (11). > # **Questions** > ## [Q1] Difference from simpler pseudo-labeling The difference is in when we accept a model-generated target. Let $\hat y_i^s=\arg\max_c b_{i,c}^s$. Unconditional fused pseudo-labeling uses $\hat y_i^{\mathrm{fuse}}$ for every sample. Our bimodal rule follows the ordered branches of Eq. (6), $$y'_i=\begin{cases} \hat y_i^{m_i^{\mathrm{proto}}}, & \text{high conflict},\\\\ \hat y_i^{m_i^{\mathrm{unc}}}, & \text{large uncertainty gap},\\\\ \hat y_i^{\mathrm{fuse}}, & \text{reliable consensus contradicting the label},\\\\ \tilde y_i, & \text{otherwise}. \end{cases}$$ Here $m_i^{\mathrm{proto}}$ balances calibrated prototype alignment and uncertainty, while $m_i^{\mathrm{unc}}$ selects the less uncertain modality. Consensus correction requires low conflict and low fused uncertainty. The revised introduction makes this distinction explicit. > ## [Q2] Branch-specific ablations Thanks for asking about the individual branches. To see what each one adds, we tested the uncertainty and conflict branches separately, then added uncertainty and consensus to the conflict branch in sequence. In **the table in [W1]** (also **Table 30**), comparing "Conflict only" with "Conflict + uncertainty" gives gains of 0.67 / 1.09 / 0.69 points. Moving from "Conflict + uncertainty" to the full rule adds consensus and brings a further 0.94 / 1.08 / 0.78 points. > ## [Q3] Aggregation under heterogeneous noise In fact, the two noise settings lead to different weighting behavior. With local training fixed, evidence weighting improves accuracy by 1.39 / 2.08 / 1.43 points over sample-size weighting (**the last two rows in [W1]**; **Table 32**). **Table 33** reports the client weights: for high-noise clients, evidence weight relative to sample share is 0.76 / 0.74 / 0.72 under wide noise spread and 1.02 / 1.04 / 1.06 under fixed class permutations. Since sample-size weighting gives a ratio of one, evidence weighting downweights these clients in the first setting and slightly upweights them in the second, where their predictions are systematically incorrect.
AuthorsOpenReview ↗

Questions (continued)

[Q4] Stronger multimodal fusion

We agree that the fusion choice deserves a direct comparison. Appendix C.8 (Table 17) uses branch-wise correction with average prediction fusion. Our added attention variant changes only inference fusion: it takes a softmax over inverse branch entropies, giving lower-entropy predictions more weight. Training and aggregation stay the same. At 40% corruption and alpha = 0.5, this improves FedCorr, FedNoRo and FedLN (Table 34). The strongest adaptation, FedNoRo with attention, reaches 88.73 / 74.28 / 41.64 on AV-MNIST / Food-101 / CMU-MOSEI. Our gains remain 6.74 / 11.06 / 4.51 points.

[Q5] Correction under input degradation

Thanks for pointing out this distinction. A reconstruction branch can return the original class. We now separate the two cases in Table 22. For P1/P2, correction precision measures actual class changes that match the clean label. For P3/P4, the labels start correct, so we report preservation among triggered samples (86.74% / 75.32%) and reconstruction trigger rate (34.2% / 37.5%). Triggering means entering a non-fallback branch, even if its output is unchanged. AUROC in these input-degradation cases measures detection of degraded inputs. The values are unchanged; the revised headers make their meaning explicit.

[Q6] Scope of the variance assumption

Here, it helps to separate the definition from the assumption: u=C/Su=C/S defines evidential uncertainty, while the variance relation is a separate assumption (Appendix A.8; see Part 1). The bound also accounts for residual reconstruction bias. With evidence weights βkm=eˉkm/∑jeˉjm\beta_k^m=\bar e_k^m/\sum_j\bar e_j^m, Eq. (33) gives

=\mathbb{E}_k\pi_k^m+ \frac{\operatorname{Cov}_k(\bar e_k^m,\pi_k^m)}{\mathbb{E}_k\bar e_k^m}.$$ Here $\pi_k^m$ is residual label error and the averages are over participating clients. Confidently incorrect clients can make the covariance positive, increasing the bias bound in Eq. (32).
AuthorsOpenReview ↗

To Reviewer Evw5

We appreciate your thoughtful comments and patience during the discussion. We have updated the rebuttal verison of our manuscript to clarify the aggregation assumption and include matched reconstruction and aggregation controls in Appendices A.8, C.20, and C.22.

For ease of reference, revisions most directly related to your comments are marked in teal. Shared additions are cross-referenced in the appendix guide. Changes to figures and tables are indicated in their captions.

We thank the authors for their detailed responses and additional experiments, which have addressed most of our concerns.

One remaining concern relates to Q6. Appendix A.8 assumes an inverse relationship between accumulated evidence and the variance of client stochastic update errors. However, Appendix C.18 (Eq. 39 and Table 29) evaluates within-client per-example gradient variance, which does not directly verify this assumption.

Overall, we appreciate the authors' efforts to clarify the theoretical analysis and strengthen the experimental evaluation. We have no further questions and will consider our discussions on my final assessment.

Reviewer_u1LmOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces FedEPR, a novel framework for multimodal federated learning designed to handle noisy supervision and client data heterogeneity. To address noise patterns FedEPR leverages modality specific Dirichlet evidence to model epistemic uncertainty, Dempster-Shafer theory to expose cross-modal conflict and globally calibrated class prototypes. Further at server side, modality-decoupled evidence-weighted aggregation down-weights corrupted client updates. Results across existing bimodal benchmarks and real-world web datasets show that FedEPR outperforms existing baselines with minimal communication overhead relative to standard FedAvg.

Critical strengths
  • Usage of Dirichlet distribution to model epistemic uncertainty as a reliability proxy rather than using softmax scores.
  • Leveraging cross-modal conflict to dynamically route label reconstruction.
  • Usage of accumulated evidence as reliability adjusted weights for server level aggregation.
  • Outperforms existing baselines under heavy synthetic noise and natural noisy web datasets.
Critical weaknesses
  • The current proposed approach is suited for bimodal settings. Extending the framework to three or more modalities cannot determine which subset/coalition of modalities is reliable.
  • In Table 1, while the authors consider noise rates composed of synthetic mixtures of three multimodal corruption patterns, it will be good to show the effects of individual corruption patterns.
  • In the cases of simultaneous modality corruption, the performance drop is significant (Table 20).
  • The gains on naturally noisy web datasets like UPMC Food-101 and Webvision are less when compared with synthetic noise settings.
  • Under non-IID settings, the accuracy of FedEPR drops across all benchmarks.
Questions for authors
  • Have the authors considered other noise distributions in the form of per- client adversarial noise, arbitrary sensor failures ?
  • How do you handle when a client becomes overconfident while overfitted to the corrupted training data ?
  • Further have the authors considered the setting of training with clean data and evaluation on corrupted test data ? This setting can simulate the scenario of encountering corrupted data during inference. Currently FedEPR’s training time target reconstruction doesn’t directly apply to mitigate test-time diverse modality failures.
4 comments
AuthorsOpenReview ↗

Response to Reviewer u1Lm

Thank you for the insightful review! We address your comments and questions below.

Weaknesses

[W1] More than two modalities

You raise a useful question about extending the rule. Our reconstruction rule is designed for bimodal clients. With three or more modalities, correlated errors can make an agreeing coalition unreliable. We discuss higher-order source selection in Appendix E and now state the bimodal scope explicitly in the introduction.

[W2] Individual corruption patterns

To examine the patterns individually, we tested mismatch, asymmetric corruption and label flips at 40% corruption and alpha = 0.5 (Table 31). Keeping our pipeline fixed, reconstruction improves over observed-label training by 3.41 / 2.36 / 6.91 points on AV-MNIST and 4.11 / 3.01 / 8.89 on Food-101, respectively. The largest gains occur under label flips, where reconstruction directly corrects annotations.

[W3] Simultaneous corruption

We also tested joint modality degradation (P4, Table 21). In this setting, we obtain 90.86 on AV-MNIST and 78.92 on Food-101, compared with 87.34 and 72.53 for FedMobile. Both sources are degraded, leaving less reliable evidence for reconstruction than under single-modality degradation.

[W4] Natural noise

We agree that naturally noisy data provide a useful test here. On these data, we improve over FedMobile by 4.26 points on UPMC (88.47 vs. 84.21, Table 2) and 3.78 on non-IID WebVision (64.52 vs. 60.74, Table 23). The 1,500-sample audit is used only for analysis.

[W5] Non-IID data

Here, we use global calibration for class-skewed local prototypes, while retaining local and global estimation errors in Eq. (13). On WebVision (Table 23), our accuracy changes from 68.61 to 64.52 under non-IID partitioning, compared with 65.38 to 60.74 for FedMobile. Our gains remain 3.23 and 3.78 points, respectively.

Questions

[Q1] Other noise distributions

We also examined noise beyond the training mixture. Appendix C.11 (Table 21) covers instance-dependent flips, adjacent-class flips and single/joint degradation. The client-weight diagnostics in Table 33 cover a wider noise spread and fixed class permutations.

[Q2] Overconfident clients

To examine this directly, we measured high-noise clients' evidence weight relative to their sample share (Table 33). Under wide noise spread, their relative weights fall to 0.76 / 0.74 / 0.72 on AV-MNIST / Food-101 / CMU-MOSEI. Under fixed class permutations, they rise to 1.02 / 1.04 / 1.06, slightly upweighting clients with systematic incorrect supervision. Eq. (11) assigns weights using accumulated confidence. A fixed class permutation preserves a learnable input-label mapping, so a client can fit incorrect labels confidently and still receive a relatively high weight.

[Q3] Corrupted test inputs

To focus on test-time corruption, we trained with 0% injected corruption and tested degraded or missing inputs, with FedMobile evaluated under the same conditions (Table 36). A/B denote image/audio for AV-MNIST and image/text for Food-101.

DatasetTest inputFedMobileFedEPR
AV-MNISTClean98.0798.73
A degraded91.8494.36
B degraded90.5693.28
Both degraded79.6384.57
A missing85.7288.45
B missing83.4686.37
Food-101Clean90.6291.32
A degraded80.4783.76
B degraded78.3481.92
Both degraded65.8369.45
A missing73.5676.83
B missing70.2473.52

At inference, our model predicts from the available modality opinions, while target reconstruction takes place during training.

AuthorsOpenReview ↗

To Reviewer u1Lm

Thank you for your careful review and patience. We have updated the rebuttal version of our manuscript, adding results on individual corruption patterns, heterogeneous client noise, and clean training followed by corrupted test inputs in Appendices C.21, C.22, and C.25.

To help you locate the changes, revisions primarily addressing your comments are highlighted in blue. The appendix guide also points to shared additions addressing more than one review. Updates to figures and tables are marked in their captions.

Reviewer_u1LmOpenReview ↗

Response to author rebuttal

The authors have provided a detailed rebuttal for the associated questions and points mentioned under weakness section. I would like to raise the final rating to acceptance.

Dear Reviewer u1Lm,

Thank you for your positive reassessment. Your detailed questions during the review process helped us clarify important aspects of our method. Best wishes!

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.