- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper considers multimodal federated learning when some clients have only image data or text data, rather than paired image-text data. The authors propose FedCGP, which learns a conditional generative model from clients with paired data and uses it to generate features for clients with missing modalities. These generated features serve as additional supervision during local training. The method also uses a training strategy that matches how the generated features are used at test time, along with separate aggregation rules for different parts of the model.
The authors evaluate FedCGP on Flickr30K and MS-COCO under different client settings. The results show better retrieval performance than the baselines considered in the paper. They also compare different generator designs and query strategies, and study the effects of client heterogeneity, client participation, and computational cost. The experiments suggest that how the generated features are trained and used has an important effect on the final performance.
Critical strengths
- The paper considers an important issue in conditional feature generation.
The denoiser is trained on corrupted real features, but incomplete clients query it using Gaussian noise. The paper recognizes this difference and designs the training procedure accordingly. This is a reasonable choice, since the generated features need to be useful for the clients that actually use them. The division between paired and incomplete clients is also clear.
- The experiments cover more than just the final retrieval results.
The authors test the method on two datasets and consider different client settings. They also compare different generators and query strategies. The ablation results help clarify which parts of the method matter. In particular, query guidance improves the generated features, although the deterministic MLP also performs well. This comparison is useful for understanding the benefits of the proposed approach.
Critical weaknesses
- The baseline comparisons are not compute-matched.
FedCGP uses 12 local epochs per round, while FedCola uses 5, although both use 30 communication rounds (Appendix E). The baseline results are also taken from prior work rather than obtained under the same training setup. This makes it difficult to separate the effect of the proposed method from that of additional local training.
The gains over FedTaste in the default setting are 2.15 points on Flickr30K and 2.92 points on MS-COCO. The paper does not report results across multiple independent runs for the main federation experiments, so it is unclear how much these results vary. Comparisons with matched training budgets and multiple seeds would make the results more convincing.
- The contribution of the generative prior is not fully clear.
Table 3 uses Eq. (11), which directly aligns generated features with real target-modality features. The actual method uses Eq. (4), which aligns observed representations with generated ones. Since incomplete clients do not have access to the real target features, the results in Table 3 do not directly validate the guidance objective used in FedCGP. Table 5 does show an improvement from adding query guidance to the full system, but it does not resolve this difference between the two objectives.
The deterministic MLP also achieves better generated-to-real correspondence than the guided denoiser in Table 3. Although Appendix F.1 reports better overall performance for FedCGP than for the deterministic alternative, the comparison does not clearly show where the difference comes from. Appendix D.4 raises a similar question: correctly conditioned generated targets provide no additional classification benefit in the receiver-side pilot. This pilot is limited, so the result should not be taken as evidence that generated features are ineffective. Still, the paper does not clearly show whether the denoising-based prior offers an advantage over simpler conditional feature translation.
Questions for authors
Q1. Could the authors compare FedCGP with FedCola and FedTaste under the same local training budget? FedCGP uses 12 local epochs per round, compared with 5 for FedCola. Are the gains over FedTaste consistent across multiple independent runs?
Q2. Appendix F.1 already compares FedCGP with a deterministic-generator alternative. Could the authors clarify how closely the two setups are matched in model capacity, training objective, and optimization budget? Given the results in Table 3, it is still unclear what the denoising-based prior adds over deterministic feature translation.
Q3. In Appendix D.4, correctly conditioned generated targets do not improve classification accuracy in the receiver-side pilot. Does the same hold when the full model is trained? Could the authors report results on CIFAR100 and AG_NEWS with and without generated targets?
3 comments
Thank you for the detailed rebuttal and the additional controlled experiments. I appreciate the authors' efforts to address my concerns. The new evidence addresses a substantial portion of my original concerns. I am therefore inclined to raise my rating from 2 to 3, while retaining some reservations about the strength and generality of the empirical conclusions.
Thank you very much for your careful review and helpful feedback. We are glad that the additional experiments addressed many of your concerns. Your suggestions have helped us improve our paper, and we will extend the experiments to examine robustness and generality across a wider range of settings. We sincerely appreciate your time and consideration.
Matched-budget results, the deterministic comparator, and receiver benefits
Thank you for distinguishing frozen-feature correspondence from the prior's usefulness during federated training. We added separate controls for these questions. References below refer to the revised manuscript.
Q1. Matched training budgets and independent runs
We reran FedCola, a system MLP, and FedCGP on a common Flickr split and evaluation pipeline, with shared encoder/task-head initialization and participation protocols, 30 rounds, and batch size 128. Results are retrieval composite scores, reported as mean ± sample SD over three independent training/participation seeds on a fixed split.
FedCGP remains ahead of FedCola at both epoch budgets and at common β = 1. These controls match rounds and local epochs, not FLOPs or runtime. Sections 5.1–5.2 and Tables 3 and A14 report the comparisons; Appendix F.5 specifies the protocol.
We did not include FedTaste in these controlled reruns because its implementation has not been publicly released. Its reported results also use 50 communication rounds, whereas our FedCola/FedCGP comparisons use 30. Importing its published scores would therefore not provide a matched-budget comparison. We retain FedTaste as a literature reference, but do not claim that the original 2.15/2.92-point margins establish a matched-budget or multi-seed advantage. The new runs address variability relative to FedCola and the MLP; repeatability of the advantage over FedTaste remains untested.
Q2. What does the prior add over deterministic translation?
We now distinguish the historical system MLP, the original 394,496-parameter frozen-feature MLP, and the new capacity-matched system MLP. The new MLP uses two directional 384→768→384 GELU mappings with normalized outputs: 1,181,952 parameters versus 1,185,536 for the denoiser.
Both system variants retain Eq. (4) guidance, real-task losses, receiver training, aggregation, and the same round/epoch budget. They use the same encoder/head learning rate (1e-4), prior/MLP learning rate (1e-3), decay, and gradient clipping. MLP fitting detaches real conditions and targets and follows the denoising branch's fitting-loss coefficient and warm-up schedule. Appendix F.5 specifies these controls.
The architecture-specific fitting objectives remain different: target-feature MSE for the MLP and noise prediction for FedCGP. FedCGP exceeds this MLP by 4.18/6.94 points at 5/12 epochs in Table 3. This supports the complete prior design in this setting, rather than isolating a benefit of stochasticity alone.
We also directly tested Eq. (4) and Eq. (11) on the same frozen cache with equal fitting budgets. Evaluation identities were excluded from upstream training and head fitting. C@1 (%) below is mean ± SD over three head-fitting seeds.
Eq. (4) improves correspondence, but the MLP still achieves higher frozen-feature C@1. We retain this finding. C@1 tests the map at fixed encoders; Table 3 tests its use during representation learning and aggregation. The system result therefore does not imply that the denoiser is a better translator under every metric. Section 5.3, Appendix F.6, and Table A9 present this distinction and the direct diagnostic.
Q3. Classification with and without generated targets
For each of three parent seeds, receiver branches share a round-10 checkpoint and participation through round 30. Paired prior learning, β = 6, and 12 local epochs remain fixed. Branching precedes the first receiver auxiliary update, and full federated optimization continues in all branches.
Correct targets improve global task-head accuracies by 1.14/0.60 percentage points over auxiliary off and outperform both target controls. Section 5.4 and Table 6 report these results. The negative pilot remains in Appendix D.4: its 100-step normalization/head-only updates test a more restricted setting.
Table A13 additionally removes the entire prior from round 1 at the same β = 6: retrieval decreases from 100.70 ± 0.58 to 97.20 ± 0.68. This separates the total prior contribution from receiver-only use.