acceptodds
Under review as a conference paper at ICLR 2027

Learning Cross-Modal Generative Priors for Multimodal Federated Learning with Missing Modalities

Abstract

Multimodal federated learning often involves clients that observe only one modality, while genuine image–text supervision is concentrated at paired clients. We present FedCGP, a framework that learns conditional feature priors from paired observations and uses them to regularize incomplete clients. Its central idea is to train the representation map actually used during local learning: fitting a denoiser on corrupted targets alone does not ensure useful outputs when queries are instead seeded by independent Gaussian noise. Paired clients therefore combine real-pair alignment, conditional denoising, and contrastive guidance through a single-step query. Incomplete clients keep the prior fixed, detach its outputs, and update their observed encoders alongside real supervised objectives. Block-specific aggregation preserves paired-only prior learning while balancing paired and incomplete contributions to shared representations. We analyze how guidance changes normalized features and how auxiliary gradients interact with real-task learning. Experiments on Flickr30k and MS-COCO combine federation-level evaluation with controlled generator and query-path studies. In the frozen-feature study, generated-to-real supervision substantially improves denoiser correspondence, while deterministic translation remains a strong alternative. Replacing the trained single-step query with a different query path at fixed weights causes a large drop in correspondence, showing that the query procedure is part of the learned interface. These results highlight query-aligned learning and selective update ownership as key design choices for transferring paired supervision under missing modalities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Federated Learning, Missing Modalities, Cross-Modal Generative Priors, Representation Learning, Knowledge Transfer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ULpBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper considers multimodal federated learning when some clients have only image data or text data, rather than paired image-text data. The authors propose FedCGP, which learns a conditional generative model from clients with paired data and uses it to generate features for clients with missing modalities. These generated features serve as additional supervision during local training. The method also uses a training strategy that matches how the generated features are used at test time, along with separate aggregation rules for different parts of the model.

The authors evaluate FedCGP on Flickr30K and MS-COCO under different client settings. The results show better retrieval performance than the baselines considered in the paper. They also compare different generator designs and query strategies, and study the effects of client heterogeneity, client participation, and computational cost. The experiments suggest that how the generated features are trained and used has an important effect on the final performance.

Critical strengths
  1. The paper considers an important issue in conditional feature generation.

The denoiser is trained on corrupted real features, but incomplete clients query it using Gaussian noise. The paper recognizes this difference and designs the training procedure accordingly. This is a reasonable choice, since the generated features need to be useful for the clients that actually use them. The division between paired and incomplete clients is also clear.

  1. The experiments cover more than just the final retrieval results.

The authors test the method on two datasets and consider different client settings. They also compare different generators and query strategies. The ablation results help clarify which parts of the method matter. In particular, query guidance improves the generated features, although the deterministic MLP also performs well. This comparison is useful for understanding the benefits of the proposed approach.

Critical weaknesses
  1. The baseline comparisons are not compute-matched.

FedCGP uses 12 local epochs per round, while FedCola uses 5, although both use 30 communication rounds (Appendix E). The baseline results are also taken from prior work rather than obtained under the same training setup. This makes it difficult to separate the effect of the proposed method from that of additional local training.

The gains over FedTaste in the default setting are 2.15 points on Flickr30K and 2.92 points on MS-COCO. The paper does not report results across multiple independent runs for the main federation experiments, so it is unclear how much these results vary. Comparisons with matched training budgets and multiple seeds would make the results more convincing.

  1. The contribution of the generative prior is not fully clear.

Table 3 uses Eq. (11), which directly aligns generated features with real target-modality features. The actual method uses Eq. (4), which aligns observed representations with generated ones. Since incomplete clients do not have access to the real target features, the results in Table 3 do not directly validate the guidance objective used in FedCGP. Table 5 does show an improvement from adding query guidance to the full system, but it does not resolve this difference between the two objectives.

The deterministic MLP also achieves better generated-to-real correspondence than the guided denoiser in Table 3. Although Appendix F.1 reports better overall performance for FedCGP than for the deterministic alternative, the comparison does not clearly show where the difference comes from. Appendix D.4 raises a similar question: correctly conditioned generated targets provide no additional classification benefit in the receiver-side pilot. This pilot is limited, so the result should not be taken as evidence that generated features are ineffective. Still, the paper does not clearly show whether the denoising-based prior offers an advantage over simpler conditional feature translation.

Questions for authors

Q1. Could the authors compare FedCGP with FedCola and FedTaste under the same local training budget? FedCGP uses 12 local epochs per round, compared with 5 for FedCola. Are the gains over FedTaste consistent across multiple independent runs?

Q2. Appendix F.1 already compares FedCGP with a deterministic-generator alternative. Could the authors clarify how closely the two setups are matched in model capacity, training objective, and optimization budget? Given the results in Table 3, it is still unclear what the denoising-based prior adds over deterministic feature translation.

Q3. In Appendix D.4, correctly conditioned generated targets do not improve classification accuracy in the receiver-side pilot. Does the same hold when the full model is trained? Could the authors report results on CIFAR100 and AG_NEWS with and without generated targets?

3 comments
AuthorsOpenReview ↗

Matched-budget results, the deterministic comparator, and receiver benefits

Thank you for distinguishing frozen-feature correspondence from the prior's usefulness during federated training. We added separate controls for these questions. References below refer to the revised manuscript.

Q1. Matched training budgets and independent runs

We reran FedCola, a system MLP, and FedCGP on a common Flickr split and evaluation pipeline, with shared encoder/task-head initialization and participation protocols, 30 rounds, and batch size 128. Results are retrieval composite scores, reported as mean ± sample SD over three independent training/participation seeds on a fixed split.

MethodPaired weight β5 local epochs12 local epochs
FedCola189.14 ± 0.2584.90 ± 0.61
MLP698.89 ± 0.7793.76 ± 0.84
FedCGP6103.07 ± 0.59100.70 ± 0.58
FedCGP1101.81 ± 0.6398.63 ± 0.71

FedCGP remains ahead of FedCola at both epoch budgets and at common β = 1. These controls match rounds and local epochs, not FLOPs or runtime. Sections 5.1–5.2 and Tables 3 and A14 report the comparisons; Appendix F.5 specifies the protocol.

We did not include FedTaste in these controlled reruns because its implementation has not been publicly released. Its reported results also use 50 communication rounds, whereas our FedCola/FedCGP comparisons use 30. Importing its published scores would therefore not provide a matched-budget comparison. We retain FedTaste as a literature reference, but do not claim that the original 2.15/2.92-point margins establish a matched-budget or multi-seed advantage. The new runs address variability relative to FedCola and the MLP; repeatability of the advantage over FedTaste remains untested.

Q2. What does the prior add over deterministic translation?

We now distinguish the historical system MLP, the original 394,496-parameter frozen-feature MLP, and the new capacity-matched system MLP. The new MLP uses two directional 384→768→384 GELU mappings with normalized outputs: 1,181,952 parameters versus 1,185,536 for the denoiser.

Both system variants retain Eq. (4) guidance, real-task losses, receiver training, aggregation, and the same round/epoch budget. They use the same encoder/head learning rate (1e-4), prior/MLP learning rate (1e-3), decay, and gradient clipping. MLP fitting detaches real conditions and targets and follows the denoising branch's fitting-loss coefficient and warm-up schedule. Appendix F.5 specifies these controls.

The architecture-specific fitting objectives remain different: target-feature MSE for the MLP and noise prediction for FedCGP. FedCGP exceeds this MLP by 4.18/6.94 points at 5/12 epochs in Table 3. This supports the complete prior design in this setting, rather than isolating a benefit of stochasticity alone.

We also directly tested Eq. (4) and Eq. (11) on the same frozen cache with equal fitting budgets. Evaluation identities were excluded from upstream training and head fitting. C@1 (%) below is mean ± SD over three head-fitting seeds.

Training objectiveImage → textText → image
Denoising only0.33 ± 0.040.26 ± 0.04
Denoising + Eq. (4)22.40 ± 0.3823.18 ± 0.42
Denoising + Eq. (11)24.41 ± 0.4225.78 ± 0.45
MLP MSE + Eq. (4)27.73 ± 0.5029.30 ± 0.51

Eq. (4) improves correspondence, but the MLP still achieves higher frozen-feature C@1. We retain this finding. C@1 tests the map at fixed encoders; Table 3 tests its use during representation learning and aggregation. The system result therefore does not imply that the denoiser is a better translator under every metric. Section 5.3, Appendix F.6, and Table A9 present this distinction and the direct diagnostic.

Q3. Classification with and without generated targets

For each of three parent seeds, receiver branches share a round-10 checkpoint and participation through round 30. Paired prior learning, β = 6, and 12 local epochs remain fixed. Branching precedes the first receiver auxiliary update, and full federated optimization continues in all branches.

Receiver conditionCIFAR100 accuracy (%)AG_NEWS accuracy (%)
Correct condition86.30 ± 0.2189.17 ± 0.14
Auxiliary off85.16 ± 0.2388.57 ± 0.15
Self-target84.88 ± 0.2788.61 ± 0.17
Shuffled target85.28 ± 0.2988.33 ± 0.18

Correct targets improve global task-head accuracies by 1.14/0.60 percentage points over auxiliary off and outperform both target controls. Section 5.4 and Table 6 report these results. The negative pilot remains in Appendix D.4: its 100-step normalization/head-only updates test a more restricted setting.

Table A13 additionally removes the entire prior from round 1 at the same β = 6: retrieval decreases from 100.70 ± 0.58 to 97.20 ± 0.68. This separates the total prior contribution from receiver-only use.

Thank you for the detailed rebuttal and the additional controlled experiments. I appreciate the authors' efforts to address my concerns. The new evidence addresses a substantial portion of my original concerns. I am therefore inclined to raise my rating from 2 to 3, while retaining some reservations about the strength and generality of the empirical conclusions.

Thank you very much for your careful review and helpful feedback. We are glad that the additional experiments addressed many of your concerns. Your suggestions have helped us improve our paper, and we will extend the experiments to examine robustness and generality across a wider range of settings. We sincerely appreciate your time and consideration.

Reviewer_kektOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FedCGP, a federated cross-modal generation prior framework for client-level modality-incomplete scenarios. Its core design is as follows: paired clients with real image-text pairs learn the bidirectional conditional generation prior; incomplete clients with only single modality will fix the received prior and obtain auxiliary targets through detached Gaussian-seeded single-step queries, which are used to regularize the encoders of the local observed modality; the server only aggregates the uploads of paired clients for the prior, and weights the aggregation of shared representation blocks according to paired/incomplete contributions. The paper also presents local gradient analysis, explaining the alignment relationship between the generation target and the gradients of the real task, and conducts federated retrieval experiments, feature freezing diagnosis, query path replacement, and component ablation on Flickr30k and MS-COCO.

Critical strengths

Client-level modality incompleteness is a real problem in multi-modal federated learning. The paper clearly distinguishes between "clients with paired evidence" and "clients with only single-modal observations", and proposes the separation of prior learning and prior usage permissions. This perspective is more detailed than simply using modal missing interpolation.

Critical weaknesses

(1) FedCola reports 30 communication rounds with five local epochs; FedCGP uses 30 rounds with 12 local epochs at every client and batch size 128. Such an experimental setup may lead to an unfair comparison of the results. (2) The paper proposes that incomplete clients use detached to generate targets for regularizing the observed encoder. Do these gains mainly come from the retrieval supervision enhancement provided by paired clients, rather than from incomplete clients actually benefiting from the generation target? (3) The description in Figure 1, "missing-view example illustrates client-level modality incompleteness", is rather brief. The main text should provide a more intuitive explanation for the paired/incomplete data flow.

Questions for authors

Please see the weakness section.

1 comment
AuthorsOpenReview ↗

Training-budget controls and the contribution of incomplete-client training

Thank you for your comments. We added controlled experiments for training budget and receiver benefit, and clarified the paired/incomplete data flow in Figure 1. References below refer to the revised manuscript.

  1. Training-budget fairness

We agree that the original 12-versus-5 local-epoch comparison leaves a budget confound. We reran the Flickr setting with a common split and evaluation pipeline, shared encoder/task-head initialization and participation protocols, 30 rounds, and batch size 128. Results are retrieval composite scores, reported as mean ± sample SD over three independent training/participation seeds on a fixed split.

MethodPaired weight β5 local epochs12 local epochs
FedCola189.14 ± 0.2584.90 ± 0.61
FedCGP1101.81 ± 0.6398.63 ± 0.71
FedCGP6103.07 ± 0.59100.70 ± 0.58

FedCGP remains ahead at both epoch budgets, including when β is also matched. This addresses the local-epoch confound, although matching rounds and epochs does not imply equal FLOPs or runtime. Sections 5.1–5.2 separate these results from the original benchmarks; Tables 3 and A14 report the comparisons, and Appendix F.5 specifies the protocol.

  1. Are the gains attributable only to paired-client supervision?

The receiver control retains paired prior learning, β = 6, encoder updates, and aggregation in every branch. Only the receiver intervention changes. Within each of three parent seeds, branches share the round-10 checkpoint and subsequent participation, then continue full training to round 30 with 12 local epochs. Receiver regularization starts after round 10, so the intervention precedes its first use.

Receiver conditionRetrieval scoreCIFAR100 (%)AG_NEWS (%)
Correct condition100.70 ± 0.5886.30 ± 0.2189.17 ± 0.14
Auxiliary off98.80 ± 0.6585.16 ± 0.2388.57 ± 0.15
Self-target99.56 ± 0.7184.88 ± 0.2788.61 ± 0.17
Shuffled target98.04 ± 0.7485.28 ± 0.2988.33 ± 0.18

Correct conditioning improves CIFAR100/AG_NEWS accuracy by 1.14/0.60 percentage points and retrieval by 1.90 points over auxiliary off. It also outperforms self-targets and shuffled targets. Because the paired-learning procedure and weighting are retained, these results support an additional contribution from receiver-side conditioning. Classification is evaluated using the global task heads associated with incomplete-client tasks, rather than per-client accuracy.

Section 5.4 and Table 6 report this experiment; Section 5.5 discusses its relationship to paired supervision. Table A13 separately evaluates removal of the entire prior from the start of training.

  1. Clearer paired/incomplete data flow

We expanded the Figure 1 caption. A paired client observes real image–text pairs and trains the conditional prior. An image-only client has images and real class labels, but no paired text. It freezes the received prior, queries it using an observed image feature and Gaussian noise, and uses the detached complementary feature as an auxiliary target alongside its real classification loss. The text-only case is symmetric.

Only paired clients upload prior updates. Incomplete clients contribute updates to compatible representation blocks, allowing their real observations to participate in federated learning. Generated targets transfer paired information without being treated as new genuine pairs. Generation is used during local training; retrieval evaluation uses real-observation representations without generation.

The revised caption introduces this data flow, with the full procedure in Sections 3.1–3.4 and Appendix A.

Reviewer_5CXsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces FedCGP, a federated learning framework that accommodates both paired image–text clients and unimodal classification clients. Paired clients train a conditional denoiser to generate representations through a guided, single-step process initialized with Gaussian noise, while unimodal clients leverage detached synthetic targets to regularize their encoders. The server aggregates prior updates exclusively from paired clients and employs role-specific weighting for representation parameters. Experiments on Flickr30k and MS-COCO evaluate retrieval performance, generator configurations, query mechanisms, and local computational costs.

Critical strengths
  • The method clearly distinguishes training with paired data from regularization using generated targets on unimodal clients. Its explicit gradient flow and parameter-specific aggregation strategy make the training procedure easier to understand.
  • The query-path analysis shows that effective denoising does not necessarily translate into useful Gaussian-initialized queries. Deterministic baselines and reported negative results on the receiver side further clarify the method's limitations.
Critical weaknesses
  • The main comparison uses unequal local training budgets for different methods.

  • The main diagnostic relies on a guidance objective that differs from the one used in federated training. Specifically, Equation 11 aligns generated features with real target features, whereas Equation 4 aligns them with observed conditions. As a result, it remains unclear how the diagnostic findings translate to the full federated setting, particularly given that the restricted receiver-side pilot shows no further improvement in accuracy.

Questions for authors
  • How does FedCGP compare with FedCola under the same local-epoch budget and evaluation pipeline? Please report variation across federation seeds for a representative setting.

  • Does a prior trained with Equation 4 improve generated-to-real correspondence? Comparing it directly with Equation 11 would test whether the diagnostic explains the actual method.

  • Do incomplete clients improve on CIFAR100 and AG_NEWS under full-system training? Prior-free, self-target, and shuffled-target controls should use the same paired-source weighting.

3 comments
AuthorsOpenReview ↗

Matched-budget comparisons and direct tests of guidance and receiver benefits

Thank you for identifying the controls needed to assess the method. We added matched-budget comparisons, a direct test of Eq. (4), and receiver interventions under full federated training. Section and table references below refer to the revised manuscript.

  1. Comparison with FedCola under the same local-epoch budget

We reran FedCola and FedCGP on the same Flickr split and evaluation pipeline, with shared encoder/task-head initialization and participation protocols. Both use 30 rounds and batch size 128. Results are retrieval composite scores, reported as mean ± sample SD over three independent training/participation seeds on a fixed split.

MethodPaired weight β5 local epochs12 local epochs
FedCola189.14 ± 0.2584.90 ± 0.61
FedCGP1101.81 ± 0.6398.63 ± 0.71
FedCGP6103.07 ± 0.59100.70 ± 0.58

FedCGP remains ahead at both epoch budgets, including at the same paired-source weight. Thus, the advantage persists after controlling for the two differences in local epochs and aggregation weight. These comparisons match update schedules, not FLOPs or runtime.

Sections 5.1–5.2 distinguish these controlled runs from the original reported benchmarks. Tables 3 and A14 report the default- and common-weight comparisons; Appendix F.5 specifies the shared protocol.

  1. Does Eq. (4) improve generated-to-real correspondence?

Yes. We agree that Eq. (11) alone cannot validate the deployed guidance objective. We therefore fitted the two variants separately on the same frozen cache with equal fitting budgets and evaluated generated queries against real target-modality features. Evaluation identities were excluded from upstream encoder training and head fitting. C@1 (%) below is mean ± SD over three head-fitting seeds.

Training objectiveImage → textText → image
Denoising only0.33 ± 0.040.26 ± 0.04
Denoising + Eq. (4)22.40 ± 0.3823.18 ± 0.42
Denoising + Eq. (11)24.41 ± 0.4225.78 ± 0.45

Eq. (4) improves correspondence in both directions. This directly tests the objective used in federation, while Eq. (11) remains a separate diagnostic. The experiment is described in Section 5.3 and Appendix F.6, with results in Table A9.

  1. Do incomplete-client tasks benefit during full training?

We tested receiver interventions at β = 6 and 12 local epochs. For each of three parent seeds, branches share the round-10 checkpoint and subsequent participation, then continue full optimization to round 30. Paired prior learning and aggregation remain active in every branch. Since receiver regularization starts after round 10, branching precedes its first update.

Receiver conditionCIFAR100 accuracy (%)AG_NEWS accuracy (%)
Correct condition86.30 ± 0.2189.17 ± 0.14
Auxiliary off85.16 ± 0.2388.57 ± 0.15
Self-target84.88 ± 0.2788.61 ± 0.17
Shuffled target85.28 ± 0.2988.33 ± 0.18

Correct targets improve the two global task-head accuracies by 1.14 and 0.60 percentage points over auxiliary off and outperform both target controls. Self-target uses the detached observed feature; shuffled-target permutes generated targets across examples while retaining positive-index assignments.

We also tested a prior-free model trained from round 1 with the same β = 6. Its retrieval score is 97.20 ± 0.68 versus 100.70 ± 0.58 for FedCGP. This separates removal of the entire prior from disabling only receiver-side use.

Section 5.4 and Table 6 report the receiver controls; Appendix F.8 and Table A13 report the prior-free comparison. The negative pilot remains in Appendix D.4: it updated only final normalization and classification heads for 100 steps, whereas the new controls continue full federated training.

Thank authors for the reponse. It addressed most of my concernt. Thus I changed my ratings.

Thank you very much for your time and helpful feedback throughout the discussion. We are very pleased that the additional experiments and clarifications addressed most of your concerns. Your suggestions have helped us improve our paper, and we sincerely appreciate the care you put into reviewing our work.

Comments on the paper

1 comment

Dear reviewer kekt,

The authors have responded to your concerns. Please review their responses as soon as possible and provide any necessary feedback. Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.