- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper is about training action-conditioned visual world models in a federated setting, where different clients see different appearances, dynamics and geometry. The question the authors actually take aim at is fairly narrow and, I think, the right one: given this kind of heterogeneity, which parts of the model should stay shared and which should be personalized. Their answer is a two-stage recipe. In stage one the clients learn a shared prediction core through plain parameter averaging (FedAvg). In stage two that core is frozen and each client trains its own private residual modules, placed both at the model interfaces and at some internal FFN/MLP layers; the residuals are zero-initialized so the model starts out identical to the backbone, and they are trained on the backbone's own prediction objective. The private weights are never sent to the server, so the adaptation stage costs no communication.
They run this on two fairly different world models, a predictive latent transformer (DINO-WM) and a diffusion-based one (NanoWM), over four environments (PushT, PointMaze, Wall, Rope), with three seeds per backbone-environment pair. The main empirical claim is that adding the internal residuals helps: open-loop held-out prediction error drops versus an interface-only personalization in all eight settings. There is a capacity-matched ablation on DINO-WM PushT that tries to show placing capacity inside the network beats just widening the interface adapters. They also check that things hold up with more clients (K=8, total data fixed) and on unseen clients with little data.
On closed-loop control the picture is less clean, and to their credit they don't paper over it. On PushT, FedWM wins against FedAvg on the validation task bank (38/48 vs 32/48) but loses slightly on the frozen test bank (37/52 vs 38/52). So the open-loop prediction gains do not straightforwardly carry over into planning performance.
Critical strengths
First, this is the first personalized-federated setup for action-conditioned visual world models, and the core question — what to share vs. personalize — is well posed. Running it on two different backbones (DINO-WM and NanoWM) keeps the protocol model-agnostic, and the capacity-matched ablation makes a useful point: where private capacity goes can matter more than how much there is.
Second, the reporting is candid. They disclose the frozen-test closed-loop reversal (37/52 vs 38/52), the ~0.92 rank correlation alongside 26/52 misranked cases, and avoid overclaiming FFN as the optimal insertion point. That kind of honesty is welcome.
Critical weaknesses
The only closed-loop result is on PushT, and it is mixed: FedWM wins on validation (38/48 vs 32/48) but loses on the frozen test bank (37/52 vs 38/52). Is there any setting where the prediction gain actually improves planning? If not, please say so clearly in the paper.
The baseline is a FedPer-style I/O-only variant built on FedWM's own frozen-core protocol, which is a fairly friendly comparison. Can you compare against a stronger personalized FL method under its standard training setup? Also, the internal-residuals advantage is shown only on DINO-WM PushT — does it hold on a second environment, and how does performance scale with more clients or rounds?
Questions for authors
Please see Weaknesses
5 comments
Stronger baselines and capacity controls
2. Can stronger personalized federated methods be evaluated using their standard training procedures?
A2: We added FedPerfix-WM. It retains joint federated training of the shared predictor and private K/V prefixes, aggregating only the shared predictor. It is adapted to causal sequence prediction, the native world-model loss, and private observation output. This contrasts with FedWM's procedure of training the shared core first, then freezing it for private residual adaptation.
PointMaze uses the same backbone initialization and data split, with 136,000 steps and 1,088,000 window exposures per seed. NanoWM PushT uses matched budgets.
Table 4. Comparison with FedPerfix-WM.
| Setting / metric | FedWM | FedPerfix-WM |
|---|---|---|
| PointMaze observation MSE | 0.04286 | 0.05211 |
| NanoWM PushT LPIPS | 0.05538 | 0.05305 |
| NanoWM private parameters | 350,432 | 3,551,776 |
FedWM reduces PointMaze observation MSE by 17.56% on average across paired seeds, improving in all three seeds. FedPerfix-WM has lower NanoWM LPIPS in all three seeds. In the fixed-rendering PointMaze action intervention, FedWM observation MSE is 0.04820 versus 0.05820.
These complete-method comparisons reveal tradeoffs between prediction metrics and private resource costs.
3. Do the gains from internal adaptation at near-matched capacity reproduce in a second environment?
A3: We extended the comparison to DINO-WM PointMaze and NanoWM PushT.
Table 5. Near-capacity-matched internal adaptation.
| Setting | Branch | Private parameters | Mean paired gain over Wide I/O ↑ |
|---|---|---|---|
| PointMaze | Wide I/O | 209,244 | — |
| PointMaze | Internal-only | 208,879 | 0.67% |
| PointMaze | FedWM | 209,196 | 0.76% |
| NanoWM PushT | Wide I/O | 350,438 | — |
| NanoWM PushT | Internal-only | 350,658 | 6.01% |
| NanoWM PushT | FedWM | 350,432 | 5.99% |
Internal-only and FedWM each have lower error in 2/3 PointMaze seeds and 3/3 NanoWM PushT seeds.
The fixed-rendering PointMaze action intervention shows horizon-dependent gains over Wide I/O.
Table 6. Horizon and client-count comparisons.
| Comparison | Result |
|---|---|
| Wide I/O comparison across three PointMaze source cores, one round | FedWM wins at H1/H3/H5: 0/3, 3/3, 3/3 |
| Rope K4 to K8, three source seeds | FedWM error increases: 18.40%, 17.39%, 17.89% |
FedWM has lower physical action-response error in all three pairs at H3 and H5, indicating reproducible gains at these multi-step horizons. The Rope expansion preserves its relative advantage over both baselines.
Client expansion and training rounds
4. How does performance change as the number of clients increases?
A4: Three source seeds use a fixed split of 1,600/200/200 training/validation/test trajectories. Each method receives 8,000 steps and 64,000 window exposures.
Table 7. Rope performance at different client counts.
| Method | K4 error ↓ | K8 error ↓ |
|---|---|---|
| FedAvg | 0.30401 | 0.64581 |
| I/O-only | 0.28438 | 0.36274 |
| FedWM | 0.27598 | 0.32535 |
FedWM leads in every seed at both client counts, while its absolute error increases as the data are subdivided.
Table 8. Additional Rope federation conditions.
| Condition | FedWM | I/O-only | FedAvg |
|---|---|---|---|
| Eight added stiffness/viewpoint combinations | 0.32433 | 0.36096 | 0.64183 |
| 320/80 imbalanced split | 0.33039 | 0.36520 | 0.43542 |
| 4/8 participation per round | 0.28567 | 0.29946 | 0.30358 |
FedWM leads in all three seeds under each condition.
The added stiffness values lie within the original range. Partial participation uses two local passes per active client per round to match total exposure.
These results show that, in the tested Rope conditions with fixed total data and matched training steps and sample exposure, FedWM retains its prediction advantage over FedAvg and I/O-only as client count increases, configurations expand, data become imbalanced, and participation becomes partial.
5. Does the advantage persist with more federated rounds?
A5: We completed two complementary comparisons.
Each method uses 8,000 steps and 64,000 exposures. Increasing aggregation frequency reduces local work per round.
Table 9. Aggregation frequency at a fixed training budget.
| Aggregations | FedAvg | I/O-only | FedWM |
|---|---|---|---|
| 20 | 0.35763 | 0.28438 | 0.27598 |
| 40 | 0.30400 | 0.28196 | 0.27434 |
FedWM leads in all three seeds at both settings.
The added experiment fixes one local training pass per client per round and increases total training.
FedAvg continues from 40 to 60 rounds. The shared phase used by I/O-only and FedWM continues from 20 to 40 rounds; private training remains at 20 passes. The per-method, per-seed budget increases from 8,000 steps and 64,000 exposures to 12,000 steps and 96,000 exposures. The three source seeds, splits, initialization provenance, and checkpoint-selection rule are retained.
Table 10. Rope results with a higher total training budget.
| Method | Reference error ↓ | Higher-budget error ↓ |
|---|---|---|
| FedAvg | 0.30400 | 0.27515 |
| I/O-only | 0.28438 | 0.26260 |
| FedWM | 0.27598 | 0.26135 |
All three methods improve in all three seeds. Under the new budget, FedWM has lower error than both baselines in 3/3 seeds. Its mean absolute advantage over FedAvg narrows from 0.02803 to 0.01380, and its advantage over I/O-only narrows from 0.00840 to 0.00125.
The relative advantage persists at both tested total-budget levels.
Prediction and planning
Thank you for your assessment of prediction gains, control results, and experimental transparency. Your Questions refer to the Critical Weaknesses. Below, we address the five specific concerns, including the newly completed Rope training-budget experiment.
A1: Yes. Four-client PushT validation provides a specific setting with improvements in both prediction and control.
Both methods use the same planner. Prediction uses the same checkpoints and reference actions. The 48 episodes per method comprise 16 task/client-configuration conditions, each with three planning seeds.
Table 1. Four-client PushT validation results.
This positive evidence applies to the validation setting above.
We repeated the same 52 frozen client-task conditions with five paired CEM planning seeds (0, 1, 2, 42, and 43), keeping the trained checkpoints and planning protocol fixed.
Table 2. Four-client PushT frozen test: aggregate results over five planning seeds.
FedWM achieves a higher aggregate success rate on this frozen task bank, with 8 additional successful episodes and a 3.08-percentage-point advantage over FedAvg.
We additionally evaluate server shared models on standard PushT and PointMaze tasks.
PushT uses the existing shared predictor from four-client federated training, the same 50 tasks, and a five-round MPC budget. PointMaze uses matched training and deployment configurations.
Table 3. Standard-task control with server shared models.
The early standard PointMaze evaluation had a predictor/encoder mismatch. The shared predictor was trained with client-specific action and proprioception encoders but evaluated with official encoders, yielding 6/50. We subsequently initialized from official weights, froze the official encoders, used matching normalization, and repeated four-client shared training. Evaluating the complete matched configuration achieved 50/50.
The evidence therefore has two levels. Personalized FedWM improves both prediction and control on four-client PushT validation. The server shared model matches the official weights' success counts on standard PushT and PointMaze. The five-seed frozen client test further shows a higher aggregate success rate for personalized FedWM. The paper's downstream conclusions will explicitly refer to these settings.