acceptodds
Under review as a conference paper at ICLR 2027

From Aggregation to Imagination: Latent World Modeling for Multimodal Federated Learning

Abstract

In multimodal federated learning (MMFL) over nonstationary environments, modality availability couples observation geometry with staleness, producing modality-conditioned temporal misalignment: same-round client evidence may describe different semantic times, and scalar staleness correction cannot remove the resulting directional temporal bias. The remedy is not a better weighting scheme but a different estimand. Federated Latent World Modeling (FedLWM) shifts federation from client-parameter aggregation to a shared latent world state and re-times state inference from server-processing rounds to the semantic times to which the evidence pertains. A fixed hyperspherical semantic reference frame establishes common coordinates, a structured state preserves the geometry on which the dynamics act, and a chart-wise structure-preserving transition supports both retrospective assimilation of delayed evidence and prospective latent imagination at each client's use time. Our theoretical analysis establishes an impossibility result for dynamical prediction from aggregate first- and second-moment summaries: a positive non-closure floor independent of sample size and predictor capacity. It further establishes finite-sample identification guarantees for the structured linear transition blocks and explicit conditions under which retrospective assimilation and prospective imagination improve upon persistence under a shared rollout-error budget. Across four benchmarks, FedLWM consistently outperforms strong baselines under modality-conditioned staleness, with gains widening as modality–staleness coupling intensifies.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multimodal Federated Learning, Federated Latent World Modeling.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cMAzOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates multimodal federated learning under nonstationary environments, introducing a factual limitation that representations have modality-conditioned temporal misalignment. The further theoretical analysis proves that federated learning cannot satisfy the two requirements of a common coordinate domain and a dynamically sufficient state representation to solve this mismatch. To this end, the authors propose Federated Latent World Modeling (FedLWM), using a hyperspherical semantic reference frame for cross-client alignment and a structured latent world state with a structure-preserving transition model. The authors provide theoretical analysis and experiments on various multimodal benchmarks.

Critical strengths
  • This work identifying an interesting and impactful problem of nonstantionary modalities might introduce temporal bias in multimodal federated learning.
  • The proposed framework is conceptually coherent and well-motivated.
  • This paper provides substantial theoretical analysis and extensive experiments covering diverse FL and MMFL baselines and multiple datasets scales for verifying FedLWM.
  • The paper writing is easy to follow.
Critical weaknesses
  • The Figure 3 have several missing symbols and is incomplete, making the modeling of FedLWM ambiguous.
  • Stronger comparisons with timestamp-aware smoothing or filtering methods might be helpful for clarifing the contribution of FedLWM.
Questions for authors

Pleass refer to the critical weaknesses.

1 comment

Thanks for the author's detailed rebuttal. My questions and concerns have been addressed.

I decide to maintain my score, as it reflects my overall assessment of this work.

Reviewer_iifyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies multimodal federated learning when modality availability and observation delays are coupled, so that uploads processed in the same round can describe different semantic times. It names this modality-conditioned temporal misalignment and argues that scalar staleness weighting cannot correct the directional bias induced by modality-dependent likelihood curvature.

The proposed FedLWM replaces parameter aggregation with a time-indexed shared latent world state: a Hyperspherical Semantic Reference Frame (HSRF) provides common coordinates, a structured state of class-conditional primitives preserves the geometry on which the dynamics act, and a chart-wise structure-preserving transition supports retrospective assimilation of delayed evidence and prospective rollout to each client's use time (semantic ephemeris). The paper adds an impossibility result for prediction from aggregate moment summaries, finite-sample identification bounds for the structured transition blocks, and experiments on MELD, CrisisMMD, CREMA-D, and a qualitative CooperScene comparison.

Critical strengths

Section 3.2 locates the difficulty in each modality's information geometry rather than in delay magnitude. The score-bias expansion there shows that the leading bias is oriented by the modality-dependent curvature, and Fig. 1(b) indicates that staleness is modality-conditioned rather than scalar. This separation of the direction and the magnitude of staleness is the paper's most convincing conceptual contribution.

The HSRF and the structured state are designed for identifiability rather than capacity: canonical pooling removes the residual gauge, the transition preserves the admissible set, and the chart-wise factorization reduces a dense full-state operator to p^lin parameters with an explicit estimation floor (Section 5). The B-Mom-XL ablation, which increases predictor capacity fourfold, still leaves the four-step NLL 0.18 above FedLWM, supporting the non-closure argument for discarded state structure.

The controlled sweep in Fig. 4(a) is a useful negative control: the margin over C-Age shrinks toward zero as delay-modality coupling is removed, which is what the formulation predicts and what a generic regularization effect would not reproduce. The CREMA-D comparison reports five-seed dispersion and places the gains in the high-staleness regime, consistent with the proposed mechanism.

Critical weaknesses

The temporal stress behind the main comparison is generated by author-designed service traces, so the external validity of the gains remains open. In S1, modality availability follows a Markov process with drop probability 0.30 and persistence 0.80, delays are drawn uniformly from 0 to 8 rounds, and CrisisMMD receives additional affine feature drift (Appendix E.1). The delay-modality coupling ρ_τM that the study identifies as the mechanism is thus a design parameter rather than a measured property of an asynchronous deployment. The conclusions about real-world nonstationarity are correspondingly tentative.

The quantities that connect the theory to the empirical improvements are not assessed on real data. The conditions of the utility analysis involve the true drift, the use-time displacement, and the local coordinate mismatch, none of which is observable without latent ground truth, and the corresponding check in Fig. 4(d) is confined to the CREMA-D-calibrated environment. The identification bound likewise contains the state-estimation error and a residual confounding term as non-vanishing floors, so part of the guarantee is conditional on the accuracy of the smoother from which the transition is fitted. The orthogonal gauge assumption and the representation error term also appear unquantified for the heterogeneous encoders used in the experiments.

The protocol leaves details unspecified that are needed to interpret the margins. Although Appendix E.1 states that the backbone-parameter protocol is fixed across matched comparisons, the paper does not say whether the reported FedLWM configuration includes the parameter channel: Table 4 separates a world-channel variant (0.72 KiB per regular round) from a world-plus-parameter variant (7.63 MiB plus 0.72 KiB). The labels E-P, E-G1, E-G1′, and E-G2 are defined only in captions, and ACI is never expanded. Which configuration underlies the reported comparisons is therefore not fully clear.

The main comparison tables report point estimates without dispersion or significance assessment. Table 1 states that results are averaged over five seeds but gives single numbers, and Table 2 likewise omits dispersion for its accuracy columns. The absence of uncertainty makes it difficult to judge whether margins such as the 1.07-point S1 E-P lead over FedAsync on MELD (35.97 versus 34.90) are reliable. The extent of the advantage over the strongest baselines therefore remains uncertain.

Theorem 1 appears to restate an information-loss argument rather than a property of federated aggregation. Its proof exhibits two admissible states with equal chart-wise mean and raw second moment that diverge after a weight-mixing transition, so the floor follows for any map of the summary. Because the empirical baselines do not predict dynamics from aggregate moment summaries, the result mainly accounts for the B-Mom ablation, and the "relocation is forced" framing reads as stronger than the construction establishes. The statement is correct, but its scope relative to moment-closure arguments is narrower than claimed.

Paper Formatting and Clarity Concerns

E-P, E-G1, E-G1′, and E-G2 are not expanded at first use, and ACI in Table 5 and Appendix E.2 is used without its full form, so readers may need to infer the metrics from context.

The figures are information-dense at page scale: Figure 4 packs four panels with small axis labels, and the insets in Fig. 5 are difficult to inspect at normal zoom.

The caption of Table 6 states that E-G1′ applies dropout to "the S0 shared model", while Table 1 and Table 2 do not specify which model this metric uses, so comparisons of the missing-modality column across tables may be ambiguous.

Questions for authors

Which protocol do the main tables use, the world channel alone or the world plus parameter channel, given that Appendix E.1 only states that the backbone-parameter protocol is fixed across matched comparisons?

Which shared model does E-G1′ use in Table 1 and in the S1 ablation of Table 2? Table 6 mentions the S0 shared model, but the other tables do not. The CrisisMMD result, where FedLWM moves from 27.74 to 27.30 while most baselines lose four to seven points, would be easier to interpret with the exact protocol.

With L_lag = 4 and delays up to 8 rounds, what fraction of uploads falls outside the smoothing window, and how sensitive are the results to the boundary approximation in Eq. 22?

What are the total communication and computation costs once the five-percent anchor rounds are included, and how do they compare with the 0.72 KiB regular-round figure in Table 4?

Could the authors report a quantitative metric for the CooperScene comparison, for example recall or mAP on the matched frames?

Reviewer_QWhxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FedLWM addresses modality-dependent delays in multimodal federated learning. It aligns client representations to fixed semantic references, maintains a structured mixture-distribution state, assimilates delayed evidence at its observation time, and predicts client-use-time priors. Evaluation includes three recognition benchmarks and qualitative cooperative-driving results.

Critical strengths
  • Explicitly distinguishing observation, arrival, and use times gives a coherent treatment of temporal mismatch.
  • Recognition results show consistent gains under temporal stress, with matched experimental settings and component controls.
Critical weaknesses
  • It is unclear how the method separates data evolution from encoder-training drift. Assumptions 1 and 3–4 impose a common latent process and largely orthogonal representation differences, placing substantial weight on restrictive assumptions.
  • The contribution is smaller than the “world modeling” framing suggests. The pipeline combines established semantic alignment, mixture estimation, delayed-observation smoothing, and prediction. The distinct advance over existing state-estimation approaches remains vague.
  • The theory mostly formalizes design intuitions. The non-closure example illustrates information loss from summarization, while the utility condition essentially requires prediction to beat persistence. Their practical guidance is limited.
  • Validation favors the proposed assumptions: theory-facing simulations use its own generative model, and CrisisMMD includes imposed affine drift. CooperScene provides selected visualizations but no aggregate detection metrics.
Questions for authors
  1. How do you distinguish underlying distribution changes from changes caused by local encoder updates?
  2. Which existing results evaluate dynamics outside the assumed transition family? Are quantitative CooperScene results available?

Comments on the paper

2 comments

Thank you for providing the additional data and relevant explanations. Please incorporate these results into the manuscript. I have no further questions, and I will take our discussion into account when making my final assessment.

The authors have provided a very detailed rebuttal. I kindly ask the reviewers to read it as soon as possible and provide any necessary feedback. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.