- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies multimodal federated learning with unreliable supervision. It considers three kinds of corruption: the two modalities show different classes, one modality is degraded, or the label is wrong while both inputs agree.
The proposed method, FedEPR, has four parts. Each modality has an evidential head that outputs class beliefs and an uncertainty score. Dempster–Shafer fusion combines the two modalities and measures how much they conflict. Each client mixes its class prototypes with global prototypes from the server. A fixed sequence of rules then sets the training label: it uses one modality's prediction, the fused prediction, or the original label, depending on conflict and uncertainty. The server aggregates each modality's model separately and gives more weight to clients with higher total evidence.
Experiments use AV-MNIST, UPMC Food-101 and CMU-MOSEI with injected noise. The paper also tests naturally noisy data (raw UPMC, a low-agreement CMU subset, WebVision) and reports ablations and a human audit of 1,500 samples.
Critical strengths
-
A useful problem framing: Separating disagreement between modalities, degradation of one modality and label error is a sensible way to think about unreliable supervision in multimodal FL, and the setting (non-IID data with unimodal clients) is realistic.
-
Evaluation beyond the designed noise: The paper adds naturally noisy data, a human audit, and noise processes outside its generator, and it states its limitations openly (confidently wrong clients, both modalities degraded, overlap between generator and rules).
Critical weaknesses
The reported gains are not attributable to the label-reconstruction mechanism the paper is about
-
Ablation: In Table 3, the variant without reconstruction reaches 91.82 / 80.47 / 43.56, which already exceeds every baseline in Table 1 at the same setting (best: 89.17 / 74.56 / 42.63). Reconstruction adds 3.65, 4.87 and 2.59 points, a minority of the total gain over FedAvg. Most of the improvement therefore comes from the evidential heads, DS fusion and prototypes, which the baselines do not have. No results at 0% noise are given, so it is unknown whether the gap exists without any corruption.
-
Noise definition: Under the 40/30/30 mixture, only 30% of corrupted samples have a wrong label; "60% noise" means 18% label noise. For semantic mismatch and asymmetric corruption the observed label is correct, yet the first two branches of Eq. (6) replace it with a modality's prediction without consulting it. In these cases, reconstruction can at best reproduce the given label. The benefit there more likely comes from masking the unreliable branch and down-weighting the sample, which is not tested against a baseline that keeps labels and only masks or re-weights.
-
Aggregation: Evidence-weighted aggregation is presented as a component but is not ablated in Table 3. Since evidence is learned from noisy labels, a client that memorizes its noise gains weight; the paper notes this but does not measure it.
-
Reliance on evidential uncertainty: All routing decisions depend on u = C/S after 20 warm-up rounds on noisy labels, and on eight thresholds and weights. Figure 6 shows a spread of about six points over the τ and λ_KL grid. How these are selected, and whether the validation data are clean, is not stated in the main text.
The comparison and positioning do not match the problem studied
-
Baselines: All methods in Tables 1 and 2 are missing-modality or generic FL methods with no mechanism for noisy supervision. Noisy-label FL methods appear only in the appendix, adapted branch by branch. No baseline combines multimodal fusion with noise handling, such as evidential multi-view learning under label noise (TMNR, Xu et al., IJCAI 2024) or conflict-aware evidential fusion (ECML, Xu et al., AAAI 2024) trained with FedAvg, or a standard sample-selection method applied across the two modality branches.
-
Generator overlap: The controlled generator produces exactly the three patterns the rule is built to detect, as the paper acknowledges. The gains on natural noise are much smaller (about 4 points), and these are the more informative results.
-
Related work. The claim to "pioneer" the study of multimodal label noise should be tempered. Exchanging class centroids between server and clients to correct noisy labels is already established (Yang et al., IEEE Intelligent Systems 2022), and federated prototypes more generally (FedProto, Tan et al., AAAI 2022) are not cited. Evidential multi-view learning has been studied under label noise and under view conflict (TMNR; ECML). Confidence-based client weighting in multimodal FL with incomplete modalities has also been proposed (FLISM, arXiv:2405.11828). The contribution is a combination of these elements in a federated setting, and it should be positioned as such.
Questions for authors
What are the results at 0% noise for all methods? Do the baselines use the same encoders and fusion as FedEPR?
With reconstruction disabled but the auxiliary-loss mask and sample weights kept (and the reverse), how do the results change? For mismatch and asymmetric corruption, how often does reconstruction change a correct label?
What is the effect of replacing evidence-weighted aggregation with sample-size weighting? How do the weights behave for clients with the highest noise rates?
How are the thresholds chosen, and is the validation set clean?
Please compare with a multimodal noise-robust baseline under the same protocol (e.g., TMNR or ECML with FedAvg), and move the noisy-label FL baselines into the main comparison.
Please report per-seed results for Table 1.
14 comments
Response to Reviewer KGc6
Questions
[Q1] Results without injected corruption
To examine this, we evaluated all methods from Table 1 at 0% injected corruption (Table 34).
| Method | AV-MNIST | Food-101 | CMU-MOSEI |
|---|---|---|---|
| FedAvg | 96.82 | 86.47 | 49.83 |
| FedProx | 97.08 | 87.15 | 50.42 |
| FedMSplit | 97.46 | 88.34 | 52.17 |
| CreamFL | 97.72 | 89.26 | 53.46 |
| PmcmFL | 97.94 | 90.18 | 54.58 |
| MPCPFL | 98.15 | 90.87 | 54.93 |
| FedMobile | 98.07 | 90.62 | 54.71 |
| FedEPR | 98.73 | 91.32 | 55.62 |
At 0%, we exceed the strongest listed baseline, MPCPFL, by 0.58 / 0.45 / 0.69 points. At 40%, our margins over the same method are 6.91 / 12.69 / 3.87. We also evaluated the added noisy-label and multi-view baselines at both rates.
Our encoder settings and comparison budgets appear in B.3 and B.4, respectively. The original noisy-label adaptations average predictions, while our method uses evidential DS fusion. The revised appendix specifies the encoders and the added fusion variants.
[Q2] Labels versus masks and weights
The controls in Table 29 let us separate these effects. With mask and weights enabled, reconstruction adds 3.65 / 4.87 / 2.59 points ([W1], Part 1). With reconstruction in place, mask and weights add 1.65 / 2.19 / 1.42. Our mismatch/asymmetric corruption preserves the original annotations. A reconstruction branch can also return the original class. For initially correct labels, preservation follows Eq. (12),
where , and denote the observed, clean and reconstructed labels.
[Q3] Weights of the noisiest clients
We checked these weights directly (Table 33). Relative to their sample share, the noisiest clients receive lower weights under wide noise spread (0.76 / 0.74 / 0.72) and slightly higher weights under fixed class permutations (1.02 / 1.04 / 1.06). The accuracy comparison in [W3] of Part 1 (Table 32) changes only aggregation.
[Q4] Threshold selection
For the controlled benchmarks, we use clean validation data and staged single-factor sweeps. Accuracy stays within 0.38 points of the original selection with fixed defaults and within 1.62 points with noisy validation. The results are in the table in [W4] of Part 1 and Table 35. Appendix B.4 lists all five candidate sets and the 75-run budget per method. Figure 6 is a separate post-selection sensitivity check. For WebVision, we use training data only for selection and reserve the human audit for analysis.
[Q5] TMNR/ECML and main-table baselines
Here are the added comparisons at 40% corruption and alpha = 0.5 (Table 34). Average-fusion rows reuse Appendix C.8; attention changes only prediction fusion.
| Method | Fusion | AV-MNIST | Food-101 | CMU-MOSEI |
|---|---|---|---|---|
| FedCorr | Average | 84.72 | 69.83 | 38.56 |
| Attention | 88.24 | 73.86 | 41.18 | |
| FedNoRo | Average | 86.35 | 69.47 | 38.87 |
| Attention | 88.73 | 74.28 | 41.64 | |
| FedLN | Average | 86.18 | 70.15 | 39.24 |
| Attention | 87.54 | 72.64 | 40.53 | |
| TMNR + FedAvg | Native | 90.83 | 76.42 | 43.56 |
| ECML + FedAvg | Native | 90.17 | 75.83 | 43.12 |
| FedEPR | Original | 95.47 | 85.34 | 46.15 |
Our gains over TMNR, the strongest added baseline at this corruption rate, are 4.64 / 8.92 / 2.59 points.
[Q6] Per-seed results
We provide the individual accuracies underlying Table 1 for seeds 42, 43 and 44 in Part 3 below. They also appear in Table 10 (Appendix C.1).
Response to Reviewer KGc6
[Q6] Per-seed results
Here are the individual Table 1 accuracies for seeds 42, 43 and 44 at each training corruption rate, using alpha = 0.5 and 30% unimodal clients. These are also listed in Table 10 (Appendix C.1).
AV-MNIST
| Method | Seed | 20% | 40% | 60% |
|---|---|---|---|---|
| FedAvg | 42 | 89.02 | 81.85 | 70.83 |
| 43 | 89.46 | 81.20 | 69.75 | |
| 44 | 89.54 | 81.96 | 70.68 | |
| FedProx | 42 | 90.36 | 82.61 | 72.19 |
| 43 | 89.89 | 82.71 | 71.24 | |
| 44 | 90.29 | 82.03 | 72.06 | |
| FedMSplit | 42 | 91.47 | 84.76 | 73.72 |
| 43 | 91.82 | 84.15 | 74.47 | |
| 44 | 91.87 | 84.68 | 74.59 | |
| CreamFL | 42 | 92.54 | 86.39 | 78.54 |
| 43 | 92.59 | 86.31 | 77.86 | |
| 44 | 92.22 | 85.84 | 78.65 | |
| PmcmFL | 42 | 93.80 | 87.43 | 77.47 |
| 43 | 93.75 | 86.93 | 77.36 | |
| 44 | 93.46 | 87.36 | 76.71 | |
| MPCPFL | 42 | 94.45 | 88.67 | 78.09 |
| 43 | 94.20 | 88.28 | 78.19 | |
| 44 | 94.49 | 88.73 | 77.48 | |
| FedMobile | 42 | 93.65 | 89.33 | 79.80 |
| 43 | 93.93 | 89.27 | 79.22 | |
| 44 | 93.97 | 88.91 | 79.90 | |
| FedEPR | 42 | 98.31 | 95.55 | 91.90 |
| 43 | 98.10 | 95.59 | 92.29 | |
| 44 | 98.28 | 95.27 | 92.35 |
Food-101
| Method | Seed | 20% | 40% | 60% |
|---|---|---|---|---|
| FedAvg | 42 | 74.45 | 65.18 | 52.85 |
| 43 | 73.87 | 65.06 | 53.92 | |
| 44 | 74.37 | 64.31 | 54.09 | |
| FedProx | 42 | 75.67 | 66.23 | 55.13 |
| 43 | 75.15 | 66.12 | 55.28 | |
| 44 | 75.59 | 65.44 | 54.17 | |
| FedMSplit | 42 | 77.07 | 68.69 | 57.55 |
| 43 | 77.50 | 68.08 | 58.41 | |
| 44 | 77.57 | 68.79 | 58.55 | |
| CreamFL | 42 | 78.37 | 70.90 | 61.65 |
| 43 | 78.82 | 70.98 | 60.86 | |
| 44 | 78.76 | 70.34 | 61.78 | |
| PmcmFL | 42 | 80.02 | 73.50 | 64.72 |
| 43 | 80.37 | 73.42 | 64.01 | |
| 44 | 80.42 | 72.92 | 64.83 | |
| MPCPFL | 42 | 81.69 | 72.86 | 67.13 |
| 43 | 82.06 | 72.78 | 67.02 | |
| 44 | 82.01 | 72.31 | 66.34 | |
| FedMobile | 42 | 80.82 | 74.69 | 68.43 |
| 43 | 80.49 | 74.76 | 68.54 | |
| 44 | 80.88 | 74.23 | 67.78 | |
| FedEPR | 42 | 89.24 | 85.49 | 80.48 |
| 43 | 89.28 | 85.10 | 79.90 | |
| 44 | 88.99 | 85.43 | 80.40 |
CMU-MOSEI
| Method | Seed | 20% | 40% | 60% |
|---|---|---|---|---|
| FedAvg | 42 | 41.67 | 35.60 | 28.82 |
| 43 | 42.49 | 34.65 | 27.64 | |
| 44 | 42.38 | 35.47 | 28.65 | |
| FedProx | 42 | 43.33 | 36.38 | 28.59 |
| 43 | 42.59 | 36.51 | 29.64 | |
| 44 | 43.23 | 35.62 | 29.49 | |
| FedMSplit | 42 | 45.51 | 37.95 | 31.97 |
| 43 | 45.61 | 38.66 | 31.83 | |
| 44 | 44.90 | 38.77 | 30.94 | |
| CreamFL | 42 | 46.98 | 40.33 | 33.49 |
| 43 | 46.42 | 39.68 | 33.62 | |
| 44 | 47.06 | 40.44 | 32.70 | |
| PmcmFL | 42 | 47.79 | 41.64 | 34.61 |
| 43 | 48.29 | 40.93 | 33.74 | |
| 44 | 48.37 | 41.54 | 34.49 | |
| MPCPFL | 42 | 48.94 | 42.53 | 35.31 |
| 43 | 48.86 | 41.87 | 34.63 | |
| 44 | 48.39 | 42.44 | 35.42 | |
| FedMobile | 42 | 47.50 | 42.79 | 36.05 |
| 43 | 48.00 | 42.88 | 35.23 | |
| 44 | 48.08 | 42.22 | 35.94 | |
| FedEPR | 42 | 53.40 | 46.29 | 40.64 |
| 43 | 53.46 | 46.37 | 39.93 | |
| 44 | 52.98 | 45.79 | 40.54 |
Response to Reviewer KGc6
Thank you for the detailed feedback. We respond to each point below.
To separate these effects, we varied reconstruction and loss handling independently at 40% corruption and alpha = 0.5, keeping our evidential/prototype pipeline fixed. "Mask + weights" denotes the auxiliary mask and sample weight in Eq. (10).
With masking and weighting enabled, we gain 3.65 / 4.87 / 2.59 points by reconstructing targets. Without them, we gain 4.28 / 5.03 / 3.26. Adding mask and weights to reconstruction gives a further 1.65 / 2.19 / 1.42 points. The clean-data control is reported in [Q1] of Part 2.
Here, the corruption rate counts affected examples. Flipped labels account for 30% of the 40/30/30 mixture. At 60% training corruption, the nominal annotation-error fraction is 0.60×0.30=0.18.
Following B.2, we now use "training corruption rate" in our headings.
For mismatch and asymmetric corruption, our generator preserves the original annotations. In the single-pattern runs (Table 31), reconstruction improves over keeping labels by 3.41 / 2.36 points on AV-MNIST and 4.11 / 3.01 on Food-101. The mask applies to auxiliary losses, while our fused loss still updates both encoders (C.16).
To isolate aggregation, we kept local training fixed and compared the rules at 40% corruption and alpha = 0.5 (Table 32).
Evidence weighting adds 1.39 / 2.08 / 1.43 points over sample-size weighting. In Table 33, high-noise clients receive evidence weight relative to sample share of 0.76 / 0.74 / 0.72 under wide noise spread and 1.02 / 1.04 / 1.06 under fixed class permutations. Sample-size weighting gives a ratio of one.
For selection, we use staged single-factor sweeps on clean validation data: 25 configurations, each run with three seeds. Baselines receive the same budget for their own hyperparameters. Figure 6 is a separate post-selection sensitivity check. We also tested fixed defaults and noisy validation (Table 35).
Fixed defaults stay within 0.38 points of clean-validation selection. With noisy validation, our gains over observed-label training are 2.76 / 3.25 / 1.37. The revised appendix gives the search ranges. Our warm-up setting remains 20 of 100 rounds. The human audit is used only for analysis.
Following your suggestion, we added TMNR/ECML with FedAvg and attention-fusion variants of FedCorr/FedNoRo/FedLN. At 40% corruption, our gains over the strongest added baseline, TMNR, are 4.64 / 8.92 / 2.59 points. The comparison appears in [Q5] of Part 2, Figure 7 and Table 34.
We also tested naturally noisy data without synthetic corruption (Tables 2 and 23). We improve over FedMobile by 4.26 points on UPMC Raw, 4.38 on CMU Low-agreement and 3.23/3.78 on IID/non-IID WebVision. The revised evaluation reports natural and synthetic noise separately.
Thank you for pointing out these connections. We replaced "pioneer" with a description of our reconstruction contribution in heterogeneous bimodal FL. The revision credits Yang et al. [1] for centroid exchange and globally guided pseudo-labeling, FedProto [2] for federated prototypes, TMNR/ECML [3, 4] for noisy-label/conflictive multi-view learning, and FLISM [5] for confidence-aware aggregation.
References