acceptodds
Under review as a conference paper at ICLR 2027

FEDWM: PERSONALIZED FEDERATED LEARNING FOR HETEROGENEOUS VISUAL WORLD MODELS

Abstract

Federated learning can combine experience across clients to train visual world models without pooling their trajectories. When clients differ in appearance and dynamics, the central design question is which parts of the predictor to share and where to personalize. We introduce FedWM, a federated training framework that places private residual modules at both the interfaces and internal layers of action-conditioned world models. Clients first learn a shared core through parameter averaging, then freeze it and train private modules locally with the backbone's native prediction objective. We instantiate FedWM separately with predictive DINO-WM and diffusion-based NanoWM across PushT, PointMaze, Wall, and Rope. With matched training steps, FedWM lowers mean held-out prediction error relative to interface-only personalization in all eight backbone–environment settings, each evaluated with three training seeds. A near-capacity-matched DINO-WM PushT ablation also favors internal residuals over wider interfaces. In DINO-WM studies, FedWM retains its advantage over interface-only adaptation with eight clients at fixed total data and in most low-data new-client settings. Closed-loop PushT comparisons favor FedWM over FedAvg on validation but not on the frozen test bank, highlighting the need to evaluate prediction and control separately.

Primary area:
learning on time series and dynamical systems
Keywords:
Federated learning, World models, Personalized federated learning, Visual dynamics, Parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jCjAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper is about training action-conditioned visual world models in a federated setting, where different clients see different appearances, dynamics and geometry. The question the authors actually take aim at is fairly narrow and, I think, the right one: given this kind of heterogeneity, which parts of the model should stay shared and which should be personalized. Their answer is a two-stage recipe. In stage one the clients learn a shared prediction core through plain parameter averaging (FedAvg). In stage two that core is frozen and each client trains its own private residual modules, placed both at the model interfaces and at some internal FFN/MLP layers; the residuals are zero-initialized so the model starts out identical to the backbone, and they are trained on the backbone's own prediction objective. The private weights are never sent to the server, so the adaptation stage costs no communication.

They run this on two fairly different world models, a predictive latent transformer (DINO-WM) and a diffusion-based one (NanoWM), over four environments (PushT, PointMaze, Wall, Rope), with three seeds per backbone-environment pair. The main empirical claim is that adding the internal residuals helps: open-loop held-out prediction error drops versus an interface-only personalization in all eight settings. There is a capacity-matched ablation on DINO-WM PushT that tries to show placing capacity inside the network beats just widening the interface adapters. They also check that things hold up with more clients (K=8, total data fixed) and on unseen clients with little data.

On closed-loop control the picture is less clean, and to their credit they don't paper over it. On PushT, FedWM wins against FedAvg on the validation task bank (38/48 vs 32/48) but loses slightly on the frozen test bank (37/52 vs 38/52). So the open-loop prediction gains do not straightforwardly carry over into planning performance.

Critical strengths

First, this is the first personalized-federated setup for action-conditioned visual world models, and the core question — what to share vs. personalize — is well posed. Running it on two different backbones (DINO-WM and NanoWM) keeps the protocol model-agnostic, and the capacity-matched ablation makes a useful point: where private capacity goes can matter more than how much there is.

Second, the reporting is candid. They disclose the frozen-test closed-loop reversal (37/52 vs 38/52), the ~0.92 rank correlation alongside 26/52 misranked cases, and avoid overclaiming FFN as the optimal insertion point. That kind of honesty is welcome.

Critical weaknesses

The only closed-loop result is on PushT, and it is mixed: FedWM wins on validation (38/48 vs 32/48) but loses on the frozen test bank (37/52 vs 38/52). Is there any setting where the prediction gain actually improves planning? If not, please say so clearly in the paper.

The baseline is a FedPer-style I/O-only variant built on FedWM's own frozen-core protocol, which is a fairly friendly comparison. Can you compare against a stronger personalized FL method under its standard training setup? Also, the internal-residuals advantage is shown only on DINO-WM PushT — does it hold on a second environment, and how does performance scale with more clients or rounds?

Questions for authors

Please see Weaknesses

5 comments
AuthorsOpenReview ↗

Prediction and planning

Thank you for your assessment of prediction gains, control results, and experimental transparency. Your Questions refer to the Critical Weaknesses. Below, we address the five specific concerns, including the newly completed Rope training-budget experiment.

1. Is there a setting where improved prediction is accompanied by improved planning?

A1: Yes. Four-client PushT validation provides a specific setting with improvements in both prediction and control.

Both methods use the same planner. Prediction uses the same checkpoints and reference actions. The 48 episodes per method comprise 16 task/client-configuration conditions, each with three planning seeds.

Table 1. Four-client PushT validation results.

MetricFedWMFedAvg
Success count38/4832/48
Mean final joint position error (pixels)15.3821.90
Reference-action H5 prediction MSE0.299720.32785

This positive evidence applies to the validation setting above.

We repeated the same 52 frozen client-task conditions with five paired CEM planning seeds (0, 1, 2, 42, and 43), keeping the trained checkpoints and planning protocol fixed.

Table 2. Four-client PushT frozen test: aggregate results over five planning seeds.

MetricFedWMFedAvg
Success count186/260178/260
Success rate ↑71.54%68.46%

FedWM achieves a higher aggregate success rate on this frozen task bank, with 8 additional successful episodes and a 3.08-percentage-point advantage over FedAvg.

We additionally evaluate server shared models on standard PushT and PointMaze tasks.

PushT uses the existing shared predictor from four-client federated training, the same 50 tasks, and a five-round MPC budget. PointMaze uses matched training and deployment configurations.

Table 3. Standard-task control with server shared models.

TaskModels comparedSuccess count (rate) ↑
PushTFedWM, FedAvg, official DINO-WM46/50 each (92%)
PointMazeThe same three methods50/50 each (100%)

The early standard PointMaze evaluation had a predictor/encoder mismatch. The shared predictor was trained with client-specific action and proprioception encoders but evaluated with official encoders, yielding 6/50. We subsequently initialized from official weights, froze the official encoders, used matching normalization, and repeated four-client shared training. Evaluating the complete matched configuration achieved 50/50.

The evidence therefore has two levels. Personalized FedWM improves both prediction and control on four-client PushT validation. The server shared model matches the official weights' success counts on standard PushT and PointMaze. The five-seed frozen client test further shows a higher aggregate success rate for personalized FedWM. The paper's downstream conclusions will explicitly refer to these settings.

AuthorsOpenReview ↗

Stronger baselines and capacity controls

2. Can stronger personalized federated methods be evaluated using their standard training procedures?

A2: We added FedPerfix-WM. It retains joint federated training of the shared predictor and private K/V prefixes, aggregating only the shared predictor. It is adapted to causal sequence prediction, the native world-model loss, and private observation output. This contrasts with FedWM's procedure of training the shared core first, then freezing it for private residual adaptation.

PointMaze uses the same backbone initialization and data split, with 136,000 steps and 1,088,000 window exposures per seed. NanoWM PushT uses matched budgets.

Table 4. Comparison with FedPerfix-WM.

Setting / metricFedWMFedPerfix-WM
PointMaze observation MSE0.042860.05211
NanoWM PushT LPIPS0.055380.05305
NanoWM private parameters350,4323,551,776

FedWM reduces PointMaze observation MSE by 17.56% on average across paired seeds, improving in all three seeds. FedPerfix-WM has lower NanoWM LPIPS in all three seeds. In the fixed-rendering PointMaze action intervention, FedWM observation MSE is 0.04820 versus 0.05820.

These complete-method comparisons reveal tradeoffs between prediction metrics and private resource costs.

3. Do the gains from internal adaptation at near-matched capacity reproduce in a second environment?

A3: We extended the comparison to DINO-WM PointMaze and NanoWM PushT.

Table 5. Near-capacity-matched internal adaptation.

SettingBranchPrivate parametersMean paired gain over Wide I/O ↑
PointMazeWide I/O209,244—
PointMazeInternal-only208,8790.67%
PointMazeFedWM209,1960.76%
NanoWM PushTWide I/O350,438—
NanoWM PushTInternal-only350,6586.01%
NanoWM PushTFedWM350,4325.99%

Internal-only and FedWM each have lower error in 2/3 PointMaze seeds and 3/3 NanoWM PushT seeds.

The fixed-rendering PointMaze action intervention shows horizon-dependent gains over Wide I/O.

Table 6. Horizon and client-count comparisons.

ComparisonResult
Wide I/O comparison across three PointMaze source cores, one roundFedWM wins at H1/H3/H5: 0/3, 3/3, 3/3
Rope K4 to K8, three source seedsFedWM error increases: 18.40%, 17.39%, 17.89%

FedWM has lower physical action-response error in all three pairs at H3 and H5, indicating reproducible gains at these multi-step horizons. The Rope expansion preserves its relative advantage over both baselines.

AuthorsOpenReview ↗

Client expansion and training rounds

4. How does performance change as the number of clients increases?

A4: Three source seeds use a fixed split of 1,600/200/200 training/validation/test trajectories. Each method receives 8,000 steps and 64,000 window exposures.

Table 7. Rope performance at different client counts.

MethodK4 error ↓K8 error ↓
FedAvg0.304010.64581
I/O-only0.284380.36274
FedWM0.275980.32535

FedWM leads in every seed at both client counts, while its absolute error increases as the data are subdivided.

Table 8. Additional Rope federation conditions.

ConditionFedWMI/O-onlyFedAvg
Eight added stiffness/viewpoint combinations0.324330.360960.64183
320/80 imbalanced split0.330390.365200.43542
4/8 participation per round0.285670.299460.30358

FedWM leads in all three seeds under each condition.

The added stiffness values lie within the original range. Partial participation uses two local passes per active client per round to match total exposure.

These results show that, in the tested Rope conditions with fixed total data and matched training steps and sample exposure, FedWM retains its prediction advantage over FedAvg and I/O-only as client count increases, configurations expand, data become imbalanced, and participation becomes partial.

5. Does the advantage persist with more federated rounds?

A5: We completed two complementary comparisons.

Each method uses 8,000 steps and 64,000 exposures. Increasing aggregation frequency reduces local work per round.

Table 9. Aggregation frequency at a fixed training budget.

AggregationsFedAvgI/O-onlyFedWM
200.357630.284380.27598
400.304000.281960.27434

FedWM leads in all three seeds at both settings.

The added experiment fixes one local training pass per client per round and increases total training.

FedAvg continues from 40 to 60 rounds. The shared phase used by I/O-only and FedWM continues from 20 to 40 rounds; private training remains at 20 passes. The per-method, per-seed budget increases from 8,000 steps and 64,000 exposures to 12,000 steps and 96,000 exposures. The three source seeds, splits, initialization provenance, and checkpoint-selection rule are retained.

Table 10. Rope results with a higher total training budget.

MethodReference error ↓Higher-budget error ↓
FedAvg0.304000.27515
I/O-only0.284380.26260
FedWM0.275980.26135

All three methods improve in all three seeds. Under the new budget, FedWM has lower error than both baselines in 3/3 seeds. Its mean absolute advantage over FedAvg narrows from 0.02803 to 0.01380, and its advantage over I/O-only narrows from 0.00840 to 0.00125.

The relative advantage persists at both tested total-budget levels.

Reviewer_mugyOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper proposes FedWM, a personalized federated learning framework for heterogeneous visual world models. It combines federated training of a shared backbone with client-specific residual adapters at model interfaces and internal layers. Experiments with DINO-WM and NanoWM across different environments show consistent prediction improvements over interface-only personalization.

Critical strengths

(1) Federated training of heterogeneous world models is meaningful when clients have different dynamics and cannot share raw trajectories. Separating shared model learning from client-specific adaptation is a reasonable approach to balancing cross-client knowledge transfer and local specialization.

(2) Experiments cover two world models, four environments, and ablations on adapter capacity and placement. the paper provides a systematic empirical study of personalized federated training for heterogeneous world models.

Critical weaknesses

(1) "In the older DINO-WM PointMaze, Wall and Rope protocols, both personalized methods use twice as many total optimizer steps as FedAvg.", FedWM uses twice the optimizer steps and training data of FedAvg in three DINO-WM environments, it is therefore unclear how much of the improvement comes from personalization rather than additional training.

(2) The paper does not establish a sufficiently distinctive mechanism that exploits the specific task properties of action-conditioned world models, such as heterogeneous transition dynamics or the relationship between multi-step prediction and planning. The main experiments focus on FedAvg and an interface-only comparator, without evaluating relevant methods for internal Transformer personalization or parameter-efficient federated fine-tuning discussed in the related work.

Questions for authors

(1) What aspects of FedWM are specifically designed for action-conditioned world models, beyond applying generic residual adapters?

(2) Would the gains over FedAvg persist under strictly matched training steps and data exposure?

(3) Although Tables 1 and 2 demonstrate improved future-state prediction, Figure 6 shows no substantial improvement in downstream action planning, with FedWM even slightly underperforming FedAvg on the frozen test set. Does this suggest that the improved prediction accuracy offers limited benefits for downstream decision-making?

6 comments
AuthorsOpenReview ↗

1. Strictly matching optimization steps and data exposure (CW1, Q2) and 2. Specific design choices for action-conditioned world models (CW2, Q1)

We have completed additional budget-matched experiments, comparisons with strong personalized baselines, and control analyses using the same checkpoints. Below, we address each Critical Weakness (CW) and Question (Q).

  1. Strictly matching optimization steps and data exposure (CW1, Q2)

The personalized methods in the original DINO-WM PointMaze, Wall, and Rope experiments included additional private training. We extended FedAvg to 40 rounds to match FedWM’s 20 rounds of shared training followed by 20 passes of private training. For each method and seed, the total optimization steps and training-window exposures were 136,000/1,088,000 for PointMaze, 69,120/552,960 for Wall, and 8,000/64,000 for Rope.

The mean held-out errors for FedWM/FedAvg were 0.04286/0.19989 on PointMaze, 0.03177/0.07170 on Wall, and 0.27618/0.30341 on Rope. The mean paired reductions were 78.57%, 55.70%, and 8.98%, respectively, with improvements across all three seeds in each environment. These additional experiments directly control for optimization steps and cumulative data exposure.

We further increased the total budget by 50% from the baseline used in the Rope training-round study, while maintaining one local pass per client per round. FedAvg continued from 40 to 60 rounds, while the shared training stage of the personalized methods increased from 20 to 40 rounds, with private training remaining at 20 passes. All methods used 12,000 steps and 96,000 window exposures. The errors for FedAvg/I/O-only/FedWM decreased from 0.30400/0.28438/0.27598 to 0.27515/0.26260/0.26135, with FedWM outperforming both baselines across all three seeds. Its mean absolute advantage over I/O-only decreased from 0.00840 to 0.00125. The advantage persisted with additional training, while the gap narrowed. Parameter counts, computational costs, and communication costs are accounted for separately.

  1. Specific design choices for action-conditioned world models (CW2, Q1)

FedWM combines shared predictive capabilities with client-specific adaptation at the interfaces and within the predictor, while preserving the native training objectives and causal structure. In DINO-WM, the observation residual preserves the action slots, the internal residual takes the normalized input to the shared FFN, and the corresponding client-specific action and proprioceptive encoders are retained. In NanoWM, zero-initialized residuals are introduced at the input and conditioning interfaces, internal MLPs, and output layer, while preserving the diffusion-conditioning computation. The shared/private parameter partition and residual modules build on existing methods. Our contribution lies in the specific training framework and empirical validation of these design choices for heterogeneous world models.

Three experiments directly examine these choices. Allowing the DINO-WM input residual to modify the action slots increased PointMaze observation MSE by 0.16% on average in paired comparisons. With the internal bottleneck width fixed at 32, removing I/O adaptation increased error by an average of 1.15% on PointMaze and 1.00% on NanoWM, consistently across all three seeds in each setting. When we fixed the target client’s interfaces and encoders and exchanged only the internal PointMaze modules, the squared physical XY response errors were 0.26966/0.33406/0.54251 for matched modules/modules from a different damping configuration/modules from a different rendering configuration. These results show that the internal modules adapt to both dynamics and visual conditions.

PointMaze action interventions with rendering held fixed show that FedWM’s gains over Wide I/O depend on the prediction horizon. In one set of paired private-training runs using three source cores, FedWM outperformed Wide I/O in 0/3, 3/3, and 3/3 pairs at H1, H3, and H5, respectively. FedWM achieved lower physical action-response error in all three pairs at both H3 and H5, indicating that the advantage at these multi-step horizons can be reproduced. In the federation-scaling experiments on Rope, increasing the number of clients from K=4 to K=8 increased FedWM’s error by 18.40%, 17.39%, and 17.89% across the three source seeds, while preserving its relative advantage over both baselines.

AuthorsOpenReview ↗

3. A stronger baseline with personalization inside the Transformer (CW2) and 4. Whether prediction improvements translate into downstream gains (Q3)

  1. A stronger baseline with personalization inside the Transformer (CW2)

The added FedPerfix-WM baseline retains the joint optimization of the shared predictor and private K/V prefixes during federated training, with only the shared predictor aggregated. Its adaptation to world models preserves the causal mask and native prediction objective and uses a private residual at the observation output. FedWM instead performs private training after freezing the shared core.

On PointMaze, we matched the backbone initialization, data splits, and budget of 136,000 steps and 1,088,000 window exposures per seed. FedWM/FedPerfix-WM achieved observation MSEs of 0.04286/0.05211, corresponding to a mean paired reduction of 17.56%, with improvements across all three seeds. On NanoWM-S/2 PushT, the two methods used matched data, initialization, and total optimization steps for each seed, with checkpoint selection based on the validation set for each method. FedPerfix-WM/FedWM achieved LPIPS scores of 0.05305/0.05538, with FedPerfix-WM obtaining lower LPIPS across all three seeds. Their private parameter counts were 3,551,776/350,432.

Under the same PointMaze action interventions, FedWM achieved lower observation MSE, at 0.04820 versus 0.05820, while FedPerfix-WM achieved lower physical response error, at 0.14766 versus 0.26902. The comparison with this stronger baseline therefore reveals trade-offs across backbones, evaluation metrics, and parameter counts.

  1. Whether prediction improvements translate into downstream gains (Q3)

Lower prediction error can accompany improved control, but this translation depends on how accurately the model predicts and ranks the consequences of candidate actions considered by the planner. We added paired prediction–control analyses using the same checkpoints, together with scores computed from actual endpoints for a fixed set of candidate actions, to examine this relationship directly.

On the four-client PushT validation set, FedWM and FedAvg succeeded in 38/48 and 32/48 episodes, respectively, with mean final joint position errors of 15.38 and 21.90 pixels. Using the same checkpoints and identical reference actions, their H5 prediction MSEs were 0.29972 and 0.32785, respectively. These 48 episodes cover 16 task–client configuration conditions, each evaluated with three planning seeds. This setting therefore shows improvements in both prediction and control.

The frozen test set yielded a different result. FedWM achieved an H5 prediction MSE of 0.39494, lower than FedAvg’s 0.47708, but their success counts were 37/52 and 38/52, respectively. We further examined the original 832 shared candidate action sequences. After excluding the known reference actions, the predicted scores from FedWM and FedAvg selected physically successful actions in 26/52 and 33/52 conditions, respectively. Keeping the candidate set and goal cost identical and instead scoring the encoded simulator-generated endpoints increased this count to 38/52. This fixed-candidate diagnostic shows that errors in predicted endpoint scores can affect action ranking. Lower average prediction error on reference actions does not guarantee that the planner can select successful actions more accurately.

We also evaluated the closed-loop capabilities of the server-side shared models on standard tasks. On standard PushT, the FedWM shared predictor, FedAvg, and official DINO-WM weights all achieved 46/50 successes (92%) on the same 50 tasks with a budget of five MPC rounds. On standard PointMaze, the FedWM shared model with matched training and deployment configurations, FedAvg, and the official weights all achieved 50/50 successes (100%).

The configuration audit for standard PointMaze identified an issue in the early evaluation. A shared predictor trained with client-specific action and proprioceptive encoders had been deployed together with the official encoders, yielding 6/50 successes. We subsequently initialized from the official weights, froze the official encoders, used the corresponding normalization, and repeated shared training across the four clients, achieving 50/50 successes. The correction involved initialization, the encoder strategy, normalization, and shared-model retraining.

These results address the planning capabilities of the shared model on standard tasks and the translation of personalized prediction improvements into client-level control gains. The shared model matched the success counts of the official weights on both sets of standard tasks. Personalized prediction yielded direct control gains on the PushT validation set, while the prediction improvement on the frozen test set has not translated into more successful tasks.

Thank you for the detailed response and additional experiments. The new budget-matched evaluations, comparisons with stronger personalized baselines, and analyses of downstream planning performance have addressed most of my concerns and strengthened the experimental evidence. I also encourage the authors to incorporate these additional experiments and analyses into the manuscript to further strengthen its clarity and persuasiveness.

Overall, I will maintain my original positive recommendation and increase my confidence in this assessment.

Reviewer_waLnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FedWM studies personalization in federated action-conditioned visual world models. Clients first learn a shared prediction core through parameter averaging, then freeze it and train private residual modules at the interfaces and internal layers using each backbone's native objective. Separate DINO-WM and NanoWM federations are evaluated across PushT, PointMaze, Wall, and Rope. Prediction error improves over an interface-only comparator in all eight settings. The paper also studies capacity and placement controls, additional clients, low-data new clients, and closed-loop planning. Planning gains on validation do not carry over to the frozen test bank.

Critical strengths

FedWM improves on interface-only personalization in all eight settings and all 24 paired seed comparisons under matched training schedules (Tables 1, 2, and 5). The consistency across DINO-WM and NanoWM is the main strength. The two-stage shared/private design is clearly specified and preserves each backbone's native objective and architecture. The near-capacity-matched Wide I/O and attention-placement controls help distinguish added capacity from the choice of adaptation location. I also appreciate the analysis of the gap between prediction accuracy and frozen-test control.

Critical weaknesses
  1. The world-model-specific contribution and the rationale for the main design choices remain insufficiently established. The principal comparator is a FedPer-style structural interface-only baseline under the same frozen-core protocol, rather than a faithful adaptation of a close personalized federated learning method. The paper cites related methods, but does not establish what FedWM adds beyond existing shared/private decomposition and internal personalization. The design evidence also leaves important choices unresolved: freezing versus jointly updating the core, internal-only versus combined interface/internal adaptation, and bottleneck width are untested (Appendix E). Attention-based adaptation has lower mean error than the chosen FFN branch in the reported placement studies. I appreciate that the authors disclose this and do not claim FFN optimality; the remaining issue is what principle should guide these choices in other world models. Together, the limited comparative positioning and incomplete design rationale leave the contribution less compelling than the consistent prediction results initially suggest.

  2. The prediction gains do not yet isolate internal placement from capacity and training budget broadly enough. Main comparisons increase private capacity alongside adding internal residuals; for NanoWM, private parameters rise from 50,528 to 350,432. The near-capacity-matched control is valuable, but is limited to DINO-WM PushT, where FedWM improves mean error by approximately 0.43% and wins in two of three blocks (Tables 6–7). This is narrow evidence for attributing the broader gains to internal placement. Separately, personalized DINO-WM methods use twice the FedAvg optimizer steps and data exposure on PointMaze, Wall, and Rope, so those FedAvg gains cannot be cleanly attributed to personalization. The interface-only comparisons are step-matched, and NanoWM and the PushT control substantially address the budget issue; my concern is with attribution across the wider results, not the validity of every comparison. Resolving these two confounds more broadly would materially strengthen the central empirical conclusion.

A secondary limitation is that improved prediction does not establish improved downstream control: frozen-test success is 37/52 for FedWM versus 38/52 for FedAvg. I treat this as a minor concern because the paper acknowledges it prominently; the reported uncertainty also does not establish significant inferiority.

Questions for authors
  1. Comparative contribution (clarification). What world-model-specific design insight does FedWM add beyond the cited internal-personalization approaches? How would a faithful FedPerfix/FedBABU-style adaptation under the same backbone, data, and budget isolate that additional benefit? Please distinguish what the existing evidence establishes from what would require a new comparison.

  2. Capacity attribution (additional evidence). Can you provide near-capacity-matched interface-only/internal-only comparisons for NanoWM and another environment with substantial dynamics shift? How much of the prediction gain remains after matching private capacity?

  3. Federation scope (clarification; broader extensions if available). Does the advantage persist when increasing the number of genuinely distinct client profiles, using unbalanced data, or allowing partial participation? Can the new-client comparison be repeated from the same source core and across multiple source seeds? Please distinguish existing evidence from future work.

  4. Uncertainty (clarification and additional repetitions). Please report the paired block differences for the near-capacity comparison and clarify how its uncertainty changes across independent federation repetitions, rather than only private-training repeats nested within fixed cores. How sensitive are the client-count and new-client findings to the source seed?

6 comments
AuthorsOpenReview ↗

World-model design and stronger baselines(CW1, Q1)

We added stronger personalization baselines, capacity and design ablations, federation extensions with multiple source seeds, and control diagnostics. We respond below in the original order of the Critical Weaknesses (CW) and four Questions (Q). Lower error is better throughout. Percentage improvements are the mean of the relative reductions over paired seeds.

1. World-model design and comparison with closely related methods (CW1, Q1)

A1: FedWM studies which computations should be shared or personalized in heterogeneous action-conditioned world models. Its shared/private partition and residual adaptation build on existing personalized methods. The specific design preserves the native prediction or diffusion objective and causal structure, initializes residual outputs to zero, and jointly adapts observation interfaces and internal prediction computations. In DINO-WM, observation-interface residuals preserve action slots, and internal residuals receive the normalized input used by the shared FFN. Client-specific action and proprioception encoders are deployed with their corresponding predictor. NanoWM instantiates the same principles at the input/conditioning interfaces, internal MLPs, and output layer.

PointMaze module exchange provides evidence of adaptation to client conditions.

Keep the target client's interfaces and encoders fixed; exchange only internal modules.

Table 1. PointMaze internal-module exchange.

Internal-module configurationPhysical XY action-response squared error ↓
Matching0.26966
Different damping0.33406
Different rendering0.54251

The internal modules adapt to both dynamics and visual conditions, with a larger effect from rendering exchange.

Our closely related baseline is FedPerfix-WM. It retains joint federated training of the shared predictor and private K/V prefixes, aggregates only the shared predictor, and is adapted to causal sequence prediction and private observation output.

FedWM freezes the core after the shared stage and then trains private residuals.

PointMaze uses identical initialization and data, with 136,000 steps and 1,088,000 window exposures per seed. NanoWM PushT uses matched budgets.

Table 2. Comparison with FedPerfix-WM.

Setting and metricFedWMFedPerfix-WM
PointMaze observation MSE0.042860.05211
NanoWM PushT LPIPS0.055380.05305
NanoWM PushT private parameters350,4323,551,776

FedWM improves PointMaze error by 17.56% on average and has lower error in 3/3 seeds. FedPerfix-WM has lower NanoWM PushT LPIPS in 3/3 seeds.

In the same PointMaze action intervention, FedWM has lower observation MSE, 0.04820 versus 0.05820. These additional results support prediction/resource tradeoffs in specific settings. The contribution to world models is defined by these designs and empirical findings.

AuthorsOpenReview ↗

Freezing, interfaces, width, placement, and capacity(CW1,CW2, Q2)

2. Freezing, interfaces, width, and placement (CW1)

A2: Freezing versus joint updating.

Both use the same PointMaze core and data, with a 68,000-step private-training budget.

Table 3. Freezing versus joint updating on PointMaze.

ConfigurationMSE ↓Trainable parameters per client
Frozen core0.04079209,196
Joint updating0.0363120,331,796

Joint updating is better in 3/3 seeds, with an average improvement of 11.17%.

Freezing reflects a tradeoff between shared deployment and adaptation cost.

Incremental contribution of interfaces.

The internal width is fixed at 32.

Table 4. Contribution of I/O residuals.

Setting and metricWith I/OWithout I/OAverage increase
PointMaze MSE0.040790.041281.15%
NanoWM PushT LPIPS0.055380.055941.00%

Removing I/O increases error in 3/3 seeds in each setting.

Width.

Width 32 follows the initial protocol. I/O width and the private-training budget are fixed.

Table 5. PointMaze internal-width comparison.

Internal widthMSE ↓Private parameters
160.04154131,532
320.04079209,196
640.03995364,524

Width 64 improves over width 32 by approximately 2.06% on average, with approximately 74% more parameters.

Placement.

The PointMaze physical-response comparison fixes rendering and matches capacity.

Table 6. Residual-placement comparisons.

SettingVariantError ↓
DINO-WM PushTAttention0.08126
DINO-WM PushTFFN0.08146
NanoWM PushTMLP0.05538
NanoWM PushTAttention0.05610
PointMaze physical response, initialFFN0.26902
PointMaze physical response, initialAttention0.28979
PointMaze physical response, private seeds 42–44FFN0.26804
PointMaze physical response, same repetitionAttention0.30927

Attention has lower DINO-WM PushT error in 7 of the 9 additional private-training pairs. MLP has lower NanoWM PushT error in 3/3 seeds. Across the three additional PointMaze private-training seeds, FFN has lower mean action-response error averaged over H1/H3/H5 than Attention (3/3).

For a new backbone, insertion sites can be compared on validation data at matched capacity and budget, followed by frozen testing. Placement depends on the backbone and metric.

3. Gains after matching capacity (CW2, Q2)

A3: PointMaze includes damping values 1.0 and 1.35.

Table 7. Capacity-matched comparisons.

SettingBranchPrivate parametersAverage gain over Wide I/O ↑
PointMazeWide I/O209,244—
PointMazeInternal-only208,8790.67%
PointMazeFedWM209,1960.76%
NanoWM PushTWide I/O350,438—
NanoWM PushTInternal-only350,6586.01%
NanoWM PushTFedWM350,4325.99%

Internal-only and FedWM each have lower error than Wide I/O in 2/3 PointMaze seeds and 3/3 NanoWM PushT seeds.

The three branches share the corresponding core, data, private-training budget, and checkpoint-selection rule.

Internal-only here adjusts width to match capacity. The interface ablation in Point 2 fixes the internal width.

AuthorsOpenReview ↗

Matched budgets, federation, and adaptation(CW2,Q3)

4. Matching training budgets (CW2)

A4: FedAvg training is extended to 40 rounds, matching FedWM's 20 shared rounds plus 20 private-training passes. Budgets are per method and seed.

Table 8. Prediction error with matched training budgets.

EnvironmentSteps / window exposuresFedWMFedAvgAverage improvement ↑
PointMaze136,000 / 1,088,0000.042860.1998978.57%
Wall69,120 / 552,9600.031770.0717055.70%
Rope8,000 / 64,0000.276180.303418.98%

All nine pairs improve.

Using the common reference from the Rope round-count study, we further fix one local pass per round and increase the total budget by 50%. Each method receives 12,000 steps and 96,000 exposures. FedAvg/I/O-only/FedWM errors are 0.27515/0.26260/0.26135. FedWM has lower error than both baselines in 3/3 seeds, while its absolute advantage over I/O-only narrows from 0.00840 to 0.00125. Steps and exposure are strictly matched; parameter-dependent computation and communication are reported separately.

5. Federation scope and same-source adaptation (Q3)

A5: Three source seeds use a fixed total split of 1,600/200/200 trajectories. Each method receives 8,000 steps and 64,000 exposures.

Table 9. Rope federation extensions.

ConditionFedWMI/O-onlyFedAvg
Eight stiffness/viewpoint configurations0.324330.360960.64183
320/80 imbalanced split0.330390.365200.43542
4/8 clients participating per round0.285670.299460.30358

FedWM leads in all three seeds under each of the three conditions.

The added stiffness values 0.7 and 0.8 lie between the original values 0.6 and 0.9.

Partial participation uses two local passes per active client per round to match exposure, halving logical communication.

Same-source new-client adaptation comprises three independent source cores, three target clients, and three methods, totaling 27 units. Each group shares the initial predictor, 20 training trajectories, and minibatch order. Each uses 430 steps and 3,400 clip exposures and is tested on 25 target trajectories. FedWM has lower error than I/O-only in 8/9 comparisons and full fine-tuning in 5/9. FedWM beats full fine-tuning in 3/3 comparisons on the extrapolation client, whereas full fine-tuning leads in 3/3 on the interpolation client.

These extensions hold under the stated conditions. In the existing single-seed NanoWM K8 study, FedAvg/FedWM errors are 0.07189/0.07818 on PushT and 0.00896/0.00926 on Wall, showing differences in scaling across backbones.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.