- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DualP-EGFL, a framework for multimodal federated learning under jointly missing modalities and labels. The method constructs modality–label evidence graphs from verified observations and introduces a two-level verification mechanism to improve the reliability of cross-modal knowledge transfer. Specifically, edge-level verification evaluates structural compatibility and predictive utility, while sample-level verification employs OOD gating and Top-K path aggregation to selectively provide supervision for missing labels. Unsupported labels are regularized to maintain low evidential confidence. The authors evaluate the proposed method on MM-IMDb, MELD, and synthetic datasets, including a 100-client federated setting. Experimental results demonstrate improvements in hidden-label prediction over existing baselines, supported by component ablations and transfer reliability analyses.
Critical strengths
-
The paper addresses the challenging and practically relevant setting of multimodal federated learning with jointly missing modalities and labels. By highlighting the difficulty of verifying cross-modal knowledge transfer under incomplete supervision, the work identifies an important problem that is not sufficiently addressed by existing approaches.
-
The proposed method achieves notable improvements over existing baselines, particularly in the 100-client MM-IMDb setting, where supported-hidden Macro-AUPRC improves by 7.65 percentage points over PePSY. The additional component ablations and transfer reliability analyses provide useful insights into the proposed framework, supporting its potential effectiveness in recovering missing supervision.
Critical weaknesses
-
Although reliable cross-modal knowledge transfer is the central motivation of the paper, the experimental results raise serious concerns about the effectiveness of the proposed verification mechanism. In the 100-client MM-IMDb setting, the false-support rate reaches 100% with 99.99% transfer coverage, indicating that the method fails to reject unsupported transfers. Conversely, the low false-support rate of 0.68% on MELD is achieved with only 3.76% coverage. These results reveal a substantial trade-off between transfer reliability and coverage, undermining the paper's central claim of reliable knowledge transfer.
-
The proposed framework primarily combines established techniques, including prototype learning, Procrustes alignment, uncertainty estimation, OOD detection, and Top-K aggregation. Although their integration is motivated by the dual-partial setting, the paper does not sufficiently establish what fundamentally new mechanism emerges from this combination. Moreover, the ablation studies do not adequately justify several central design choices, particularly target-side predictive utility verification and Top-K path aggregation. This makes it difficult to determine whether the complexity of the proposed framework is necessary or whether simpler alternatives could achieve comparable performance.
Questions for authors
- DualP-EGFL involves multiple interconnected components, including signed modality–label evidence graph construction, edge-level transfer verification, sample-level OOD gating, and evidential knowledge transfer. However, the paper lacks an overall architecture or workflow diagram illustrating how these components interact within the federated learning framework. As a result, readers may find it difficult to quickly understand the complete training procedure and the information exchanged between clients and the server.
- A central contribution of this paper is to prevent unreliable cross-modal knowledge transfer through edge-level and sample-level verification. However, Section 5.4 reports that the 100-client MM-IMDb experiment achieves 99.99% transfer coverage with a 100% false-support rate (FSR). This indicates that the proposed verification mechanism fails to reject unsupported transfers in the large-scale setting, despite the substantial improvements in prediction performance. Such a result directly challenges the paper's central claim of reliable knowledge transfer.
- On MELD, DualP-EGFL achieves a low FSR of 0.68%, but its transfer coverage is only 3.76%, meaning that almost all hidden entries are rejected from receiving transferred supervision. In contrast, the six-client MM-IMDb setting achieves 75.41% coverage but still suffers from a 55.01% FSR. These results raise concerns about whether the proposed mechanism provides a practically useful balance between rejecting unreliable transfers and recovering missing supervision. The authors should report coverage–reliability trade-off curves across datasets and demonstrate meaningful improvements at comparable coverage levels.
- Equation (13) defines path reliability as (g_j=r_{\mathrm{pair}}\exp[-\min(d_j^+,d_j^-)/\tau_j]), which depends on the distance to the nearest positive or negative prototype rather than the separation between them. Consequently, a sample with (d_j^+\approx d_j^-\approx0) may receive high reliability even though Equation (10) produces (p_j^{\mathrm{tr}}\approx0.5), indicating little discriminative evidence. This suggests that the gating mechanism may confuse proximity to the prototype distribution with confidence in the transferred prediction. The authors should justify this design and compare it with margin-aware or uncertainty-aware applicability scores.
- The paper uses false-support rate (FSR) as a key metric to demonstrate the reliability of its knowledge transfer mechanism. However, it is unclear how the ground-truth support status of hidden entries is independently established, particularly on real-world datasets such as MM-IMDb and MELD. Since a hidden entry may have an unknown label but still be supported by useful cross-modal evidence, support cannot be determined solely from label observability or the model's own gating decision. The authors should explicitly describe how supported and unsupported targets are identified for evaluation and demonstrate that this definition is independent of the proposed verification mechanism.
- Proposition 1 establishes the concentration bound (|\hat{u}-u^\star|\leq\frac{5}{2}\sqrt{\log(8/\delta)/(2n_{\min})}), assuming that the calibration samples are independent of the predictor being evaluated. However, the proposed framework constructs cross-modal prototypes using verified observations and subsequently evaluates transfer utility on verified target-modality samples. The paper does not clearly specify whether these procedures use disjoint data subsets. If the same observations are reused for prototype construction and utility verification, the independence assumption may be violated, and the concentration bound would not directly apply. The authors should clarify whether held-out calibration or cross-fitting is implemented.
- The proposed bound is (|\hat{u}-u^\star|\leq\frac{5}{2}\sqrt{\log(8/\delta)/(2n_{\min})}), where (n_{\min}) denotes the smaller number of verified positive and negative calibration samples. At (\delta=0.05), the bound exceeds 1 when (n_{\min}=10), making it vacuous since the transfer utility is bounded within ([0,1]). Achieving a bound below 0.1 would require approximately 1,586 verified samples from each class. This requirement may be difficult to satisfy in the very dual-partial setting considered by the paper. The authors should report the actual calibration sample counts and quantify how often their theoretical guarantee is non-vacuous.
- The theorem assumes that supported targets have at least (K) paths satisfying (r_j\geq r_{\mathrm{in}}), (d_j\leq d_{\mathrm{in}}), and (\tau_j\geq\underline{\tau}), while unsupported targets satisfy (r_j\leq r_{\mathrm{out}}), (d_j\geq d_{\mathrm{out}}), and (\tau_j\leq\overline{\tau}). Under these assumptions, a separating threshold exists when (Kr_{\mathrm{out}}e^{-d_{\mathrm{out}}/\overline{\tau}}<\tau_g\leq Kr_{\mathrm{in}}e^{-d_{\mathrm{in}}/\underline{\tau}}). However, the theorem does not explain why the learned evidence graph should satisfy these separation conditions, particularly under severe modality and label missingness. The authors should provide empirical evidence of such separation and discuss how often the conditions hold in practice.
Response to Reviewer kGY9
W1: Reliability-Coverage Failure
Response: Our mechanism achieves its design goal—controllable rejection of unreliable transfers—which prior methods lack. Utility verification reduces MM-IMDb-100 coverage from 99.90% to 5.41%, cutting FSR from 99.83% to 13.15% and teacher error from 26.50% to 16.34%. At 5% coverage: FSR 0.71% (IMDb-6), 10.14% (IMDb-100), 5.04% (Synthetic). Despite conservative rejection, DualP-EGFL achieves +10.41 pp gain over FedAvg on MM-IMDb-100 [95% CI: +9.18, +11.64].
Extreme points clarified: MELD 0% coverage reflects insufficient cross-modal evidence, not failure; Synthetic low coverage reflects correct rejection under adversarial signals. Our contribution is threefold: (1) formulate when to transfer as an explicit decision, (2) provide dual-level verification for control, and (3) expose the coverage-reliability trade-off prior methods ignore.
W2: Established Components and Insufficient Ablations
Response: Revised ablations isolate utility checking, Top-K vs unrestricted sum, applicability type (distance/margin/entropy), OOD gating, and graph distillation. MM-IMDb-100 without utility: AUPRC 54.06% vs 53.73%; Synthetic unrestricted sum: 98.20% vs 98.07%. Revised claim: This is an integration study with explicit rejection trade-offs, not a proof each component improves prediction. Results in Table 2 and Appendix Table 8.
Q1: Overall Workflow
Response: Framework figure (revised manuscript Page 2) shows complete workflow and information exchanges. We clarify this remains centralized simulation, not secure distributed deployment.
Q2: MM-IMDb-100 Accepts Unsupported Transfers
Response: With utility verification: Coverage 5.41%, FSR 13.15%, Teacher Error 16.34%. Without verification: Coverage 99.90%, FSR 99.83%, Teacher Error 26.50%. Dual-level verification reduces false acceptance by 86.68 pp and teacher error by 10.16 pp.
Q3: Curves at Similar Coverage
Response: We provide curves at 6 target coverage levels with achieved coverage, FSR, teacher error, and Brier score. Complete per-seed curves in supplementary CSVs. Results in Appendix Table 7.
Q4: Prototype Proximity vs Discriminative Confidence
Response: Confidence-aware applicability (margin/entropy) improves teacher correctness by 9.79-13.42 pp and Brier score by 0.042-0.063 at similar coverage on MM-IMDb-6, validating this concern. However, both alternatives reject everything (0% coverage) on MM-IMDb-100, MELD, and Synthetic at default thresholds, requiring dataset-specific retuning. We chose distance-based applicability for stable generalization but acknowledge incorporating discriminative confidence as promising future work. Results in Appendix Table 11.
Q5: Independent Support Ground Truth
Response: Support truth is computed independently from available modalities, protocol allowed-transfer tensor, and OOD flags—not from predictions or gate outputs. Real-data flags do not physically perturb features; these are protocol support metrics, not natural OOD detection measures. MM-IMDb-6 default: protocol precision 96.83±2.12%, teacher error 29.84±6.78%, confirming these are distinct metrics.
Q6: Independence Assumption in Proposition 1
Response: Revised runner reserves disjoint optimization (70%), development (15%), audit (15%) splits. Normalization uses optimization rows only; final calibration performed once after fitting. Row counts: MM-IMDb-6 (1,008/216/216), MM-IMDb-100 (8,400/1,800/1,800), MELD-6 (~5,122/1,096/1,099), Synthetic (1,134/243/243). Addresses direct reuse but not correlated observations. Claim is pointwise for audit-independent frozen predictor, not simultaneous certification.
Q7: Vacuous Concentration Bound
Response: At δ=0.05, bound radius ~1.26 at n_min=10 (vacuous). Radius <0.1 requires ≥1,586 samples/class. Median n_min: 13.5 (Synthetic) to 111.5 (MELD-6). No pair achieves radius <0.1 across datasets. Results do not support tight guarantees in sparse regimes. Analysis in Appendix Table 14.
Q8: Empirical Support-Separation
Response: Training does not guarantee Theorem 1 separation. For K=2: supported samples have ≥2 paths in 11.78-73.86% of cases; unsupported samples have ≥2 paths in 1.01-74.14%. None of 19 runs achieve perfect separation. Theorem remains explanatory conditional result, not empirical gating justification. Results in Appendix Table 15.