- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Policy-Soup, a framework for consolidating heterogeneous diffusion-policy experts into a unified visuomotor policy. It combines frozen visual features and attention outputs from multiple experts while retaining their differently sized FFNs through sparse routing. Experiments across environments, tasks, robot embodiments, and real-world settings show improved performance on source domains and unseen combinations of domain variations.
Critical strengths
-
The paper addresses the practical and underexplored problem of consolidating specialists with different model capacities, and proposes a coherent combination of feature fusion, attention aggregation, and heterogeneous FFN routing.
-
The evaluation is broad, covering RoboTwin 2.0, LIBERO, PushT, multiple tasks and embodiments, and real-robot experiments. The reported improvements are particularly strong under several mixed-domain conditions.
Critical weaknesses
-
The evaluation lacks a standard pooled-data joint-training baseline. Policy-Soup receives demonstrations from all source domains and undergoes substantial additional consolidation training, whereas the individual experts see only one domain. Without a dense policy or standard MoE trained on the same pooled data with matched optimization and capacity, it is unclear whether the gains arise from the proposed consolidation method or simply from multi-domain training and increased model capacity.
-
The resource cost of the “unified” policy is not evaluated. Policy-Soup retains and executes all visual encoders and attention branches, with sparse routing applied only to the FFNs. Its parameter count, memory, FLOPs, and latency may therefore grow approximately linearly with the number of experts, but no comparison with a single expert, ensemble, or jointly trained policy is reported.
Questions for authors
-
Please provide comparisons with a dense policy and a standard homogeneous MoE trained on the same pooled demonstrations, using matched training steps and parameter or compute budgets. This would clarify whether the gains come from the proposed expert-consolidation mechanism rather than pooled multi-domain training or increased capacity.
-
Please report the total and active parameter counts, inference latency, memory usage, and FLOPs as the number of experts increases. Since all visual encoders and attention branches remain active, these measurements are necessary to assess whether Policy-Soup offers a practical advantage over an ensemble or jointly trained policy.
4 comments
Author Response to Reviewer Hx2V (part 2)
Comment 2. Total and active resource costs
Reviewer comment
The resource cost of the “unified” policy is not evaluated. Policy-Soup retains and executes all visual encoders and attention branches, with sparse routing applied only to the FFNs. Its parameter count, memory, FLOPs, and latency may therefore grow approximately linearly with the number of experts, but no comparison with a single expert, ensemble, or jointly trained policy is reported.
Please report the total and active parameter counts, inference latency, memory usage, and FLOPs as the number of experts increases. Since all visual encoders and attention branches remain active, these measurements are necessary to assess whether Policy-Soup offers a practical advantage over an ensemble or jointly trained policy.
Response
Consolidation is not full-parameter fine-tuning. Source visual encoders and attention weights are frozen; expert FFNs, routers, mixing weights, normalization layers and shared embeddings are trainable. Trainable parameters are 21.187M/82.736M (25.6%) on open_laptop and 19.611M/81.160M (24.2%) on shake_bottle. Sequence-level top-2 routing executes only two FFN experts per sample in each decoder layer. Different samples and layers can select different experts, so a batch can involve all experts.
Table 2. Training cost by stage.
| Task | Est. unit | Shared pretrain | Four experts | Consolidation | Full pipeline | Dense 3500 ep. |
|---|---|---|---|---|---|---|
open_laptop | EFLOPs | 1.171 | 4.269 | 14.672 | 20.112 | 15.781 |
open_laptop | GPU-h | |||||
shake_bottle | EFLOPs | 1.171 | 4.966 | 17.048 | 23.184 | 18.346 |
shake_bottle | GPU-h |
For open_laptop, specialist training plus consolidation reconstructs 299.088M window exposures versus 299.369M in the dense 3500-epoch target; the difference is sampler padding/drop-last. Shared visual pretraining adds 23.408M windows from extra tasks. The historical consolidation uses global batch 1024, whereas the scratch pooled reference uses global128. Capacity, exposure, initialization and computation are therefore reported separately. End-to-end cost includes pretraining, all experts and consolidation; only consolidation is marginal cost for an existing library.
Table 3. Measured deployment cost per full action update.
| Model | Total M | Union M | GFLOPs | Peak GiB | p50 ms | p95 ms |
|---|---|---|---|---|---|---|
open_laptop | ||||||
| Light (smallest) | 20.276 | 20.276 | 30.080 | 0.119 | 910.295 | 958.221 |
| Table | 21.326 | 21.326 | 31.758 | 0.124 | 926.263 | 993.375 |
| Background | 20.801 | 20.801 | 30.919 | 0.123 | 916.897 | 992.326 |
| Sensor noise (largest) | 22.377 | 22.377 | 33.436 | 0.128 | 918.817 | 979.163 |
| Four-policy ensemble | 84.780 | 84.779 | 126.192 | 0.367 | 3700.791 | 3985.646 |
| Pooled dense DP | 82.508 | 82.508 | 119.341 | 0.352 | 952.796 | 1005.016 |
| Policy-Soup | 82.736 | 80.667 | 108.150 | 0.359 | 4100.860 | 4369.581 |
shake_bottle | ||||||
| Light (smallest) | 19.225 | 19.225 | 28.402 | 0.115 | 927.462 | 991.487 |
| Table | 21.326 | 21.326 | 31.758 | 0.124 | 926.263 | 975.853 |
| Background | 20.276 | 20.276 | 30.080 | 0.119 | 916.256 | 959.744 |
| Sensor noise (largest) | 22.377 | 22.377 | 33.436 | 0.128 | 925.482 | 969.661 |
| Four-policy ensemble | 83.204 | 83.203 | 123.676 | 0.359 | 3716.687 | 3967.364 |
| Pooled dense DP | 81.046 | 81.046 | 117.937 | 0.355 | 949.037 | 1016.392 |
| Policy-Soup | 81.160 | 79.649 | 106.185 | 0.352 | 4132.447 | 4428.969 |
Union counts unique executed network parameters across the full 100-step inference update, excluding fixed normalizer statistics; it is not the trainable count in Comment 1. Frozen encoders and attention still execute. Top-2 selects FFNs per sample and layer, not two complete policies; selections can change across denoising steps. The four-policy ensemble runs complete policies sequentially on one GPU and averages action chunks after denormalization.
Policy-Soup offers a capability–cost trade-off rather than model compression. Compared with a four-policy ensemble, it reduces inference matrix FLOPs by approximately 14% while maintaining similar peak GPU memory usage, at the cost of approximately 11% higher median latency in the current implementation. Consolidation does not require full-parameter fine-tuning: only 24–26% of the total parameters are trainable, and top-2 routing activates only two FFN experts per sample in each decoder layer. The frozen visual encoders and attention branches nevertheless remain part of the computation. Under the same evaluation protocol, Policy-Soup achieves better compositional generalization than the ensemble, with lower inference matrix FLOPs and a modest increase in latency.
The response partially addresses my concerns, but the new controls remain imperfectly matched: Policy-Soup benefits from separately pretrained specialists and additional visual pretraining, while the pooled baselines use different initialization and training settings. Moreover, Policy-Soup has similar peak memory and 11% higher latency than the four-policy ensemble, despite only a 14% FLOP reduction. Therefore, the benefits of the proposed consolidation mechanism over joint training or an ensemble remain insufficiently established, and I maintain my rating of 2.
Thank you for your follow-up comment. We respectfully disagree that our pooled controls remain unfairly matched or that the advantage over joint training and ensembling is unestablished.
1. Fairness of Initialization, MoE Controls, and Training Budgets
Neither Policy-Soup nor any baseline uses external pretrained weights (e.g., ImageNet). Moreover, Pooled Dense DP loads the exact same shared visual encoder as our specialists ("Shared pretrain", ~44 GPU-h in Table 2); our earlier phrase "scratch pooled reference" simply meant training directly on pooled data without stage-1 specialists (including that shared ~44 GPU-h stage brings Pooled Dense DP to ~1728 GPU-h vs. ~1600 GPU-h for Policy-Soup on open_laptop). To isolate stage-1 specialist initialization itself, Table 5 and Appendix Table 10 (Policy-Soup w/o init) already evaluate the same MoE architecture trained jointly on pooled data without specialist initialization (while Table 4 controls for homogeneous vs. heterogeneous FFN widths), confirming clear gains on unseen compositions (0.434 vs. 0.334 Mixed success on RoboTwin 2.0; 0.699 vs. 0.617 IoU on PushT).
Data exposure is also matched ( pooled-equivalent epochs; 299.1M vs. 299.4M windows), and the remaining optimization settings actually give Pooled Dense DP more capacity to fit—updating 100% of parameters (vs. 24.2–25.6% in Policy-Soup) and using a smaller global batch size (128 across all 3,500 epochs vs. 1024 during the 2,500 consolidation epochs, yielding more gradient updates). Even so, Policy-Soup improves Held-out and Mixed success by +13.0 / +7.5 pp on open_laptop and +8.0 / +4.5 pp on shake_bottle.
| Setting (open_laptop) | 4 Specialists + Policy-Soup | Pooled Dense DP |
|---|---|---|
| External pretrained weights | None | None |
| Initial visual encoder | Same shared encoder | Same shared encoder |
| Equivalent pooled epochs | 3500 | |
| Window exposures | 299.088M | 299.369M |
| Trainable parameters | 21.187M (25.6%) | 82.508M (100%) |
| Global batch size | 128 (stage 1) / 1024 (stage 2) | 128 (all 3,500 ep.) |
| Full pipeline GPU-h | ~1600 | ~1728 (1684 + 44) |
2. Practical Advantage Over the Four-Policy Ensemble
The four-policy ensemble is not a unified model: it runs four separate, uncoordinated policies and heuristically averages their output actions at test time. Without shared representations or routing, out-of-domain policies corrupt the averaged trajectories, causing severe failure on unseen compositions (0.123 Mixed success) and offering no unified solution across tasks or embodiments.
In contrast, Policy-Soup consolidates heterogeneous specialists into one generalizable policy, using fewer parameters and 14% fewer FLOPs while achieving 3.8x higher Mixed-domain success (0.463 vs. 0.123). Because Policy-Soup already requires fewer FLOPs, the 0.4s latency gap over 100 denoising steps (4.10s vs. 3.70s) stems solely from sequential Python-level branch execution in our prototype rather than algorithmic cost. Replacing four separate policies with a single unified policy that lifts Seen, Held-out, and Mixed success by +26.0, +31.0, and +34.0 pp clearly establishes its practical benefit.
| Method (open_laptop) | Unified Model? | Params (M) | GFLOPs | Peak GiB | Latency p50 (s) | Seen | Held-out | Mixed |
|---|---|---|---|---|---|---|---|---|
| Four-Policy Ensemble | No (4 models) | 84.78 | 126.19 | 0.367 | 3.70 | 0.315 | 0.340 | 0.123 |
| Policy-Soup (Ours) | Yes | 82.74 | 108.15 | 0.359 | 4.10 | 0.575 | 0.650 | 0.463 |
Author Response to Reviewer Hx2V (part 1)
Thank you for highlighting specialist consolidation and the broad evaluation. We address pooled-training controls and the training/deployment tradeoff below.
Comment 1. Pooled-data baselines and fair training budgets
Reviewer comment
Response
The decisive controls are a pooled dense DP with complete-policy capacity matched to Soup. The pooled dense DP use one visual encoder and shared attention; the MoE uses token-wise top-2 FFNs. The parameter counts differ from Soup by at most 1.4% across the two tasks. The main comparison separates source factors from unseen compositions.
Table 1. Capacity-matched pooled-policy comparison.
open_laptopopen_laptopshake_bottleshake_bottlePerformance cells combine success rate and Wilson 95% interval; training-seed variation is reported separately. Entries refer to the evaluated model version.
The pooled baselines target 3500 epochs, corresponding to 1000 specialist epochs plus 2500 pooled consolidation epochs per source demonstration. The baseline protocol uses FP32, AdamW (10−4 learning rate), cosine decay, 500-update warm-up, EMA and global batch 128. Fixed epochs or predeclared budgets select checkpoints; test success does not select them. The submitted Table 5 addresses initialization within a particular layout, separately from these pooled dense and standard-MoE controls.