acceptodds
Under review as a conference paper at ICLR 2027

Policy-Soup: Generalizable Manipulation through Heterogeneous Expert Merging

Abstract

Robust visuomotor manipulation requires policies to operate across diverse domains. Domain-specialized experts provide a natural way to handle domain variation, but maintaining separate experts does not provide a unified solution for mixed or previously unseen combinations of domain variations. An important yet underexplored question is therefore how to consolidate their complementary capabilities into a single reusable policy. This becomes particularly challenging when experts differ in model capacity, as conventional parameter merging assumes structural compatibility. Thus, we propose Policy-Soup, a method for consolidating heterogeneous policy experts into a unified policy. Our key insight is that different policy components encode different degrees of transferable and specialized knowledge. Policy-Soup shares transferable information through learned fusion of expert visual features and layer-wise aggregation of attention outputs, while preserving specialization by retaining heterogeneous feed-forward networks at their original widths and selecting among them through sequence-level sparse routing. The unified policy is trained on pooled source-domain demonstrations, without requiring data for every combination of environmental variations. We evaluate Policy-Soup on RoboTwin 2.0, LIBERO, PushT, and real-world manipulation across cross-environment, cross-task, and cross-embodiment settings. Policy-Soup preserves strong source-domain performance, improves robustness to held-out and compositional shifts, and consolidates capabilities across tasks and robot embodiments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Robotic Manipulation, Heterogeneous Model Merging, Domain Generalization, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Hx2VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Policy-Soup, a framework for consolidating heterogeneous diffusion-policy experts into a unified visuomotor policy. It combines frozen visual features and attention outputs from multiple experts while retaining their differently sized FFNs through sparse routing. Experiments across environments, tasks, robot embodiments, and real-world settings show improved performance on source domains and unseen combinations of domain variations.

Critical strengths
  1. The paper addresses the practical and underexplored problem of consolidating specialists with different model capacities, and proposes a coherent combination of feature fusion, attention aggregation, and heterogeneous FFN routing.

  2. The evaluation is broad, covering RoboTwin 2.0, LIBERO, PushT, multiple tasks and embodiments, and real-robot experiments. The reported improvements are particularly strong under several mixed-domain conditions.

Critical weaknesses
  1. The evaluation lacks a standard pooled-data joint-training baseline. Policy-Soup receives demonstrations from all source domains and undergoes substantial additional consolidation training, whereas the individual experts see only one domain. Without a dense policy or standard MoE trained on the same pooled data with matched optimization and capacity, it is unclear whether the gains arise from the proposed consolidation method or simply from multi-domain training and increased model capacity.

  2. The resource cost of the “unified” policy is not evaluated. Policy-Soup retains and executes all visual encoders and attention branches, with sparse routing applied only to the FFNs. Its parameter count, memory, FLOPs, and latency may therefore grow approximately linearly with the number of experts, but no comparison with a single expert, ensemble, or jointly trained policy is reported.

Questions for authors
  1. Please provide comparisons with a dense policy and a standard homogeneous MoE trained on the same pooled demonstrations, using matched training steps and parameter or compute budgets. This would clarify whether the gains come from the proposed expert-consolidation mechanism rather than pooled multi-domain training or increased capacity.

  2. Please report the total and active parameter counts, inference latency, memory usage, and FLOPs as the number of experts increases. Since all visual encoders and attention branches remain active, these measurements are necessary to assess whether Policy-Soup offers a practical advantage over an ensemble or jointly trained policy.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer Hx2V (part 1)

Thank you for highlighting specialist consolidation and the broad evaluation. We address pooled-training controls and the training/deployment tradeoff below.

Comment 1. Pooled-data baselines and fair training budgets

Reviewer comment

The evaluation lacks a standard pooled-data joint-training baseline. Policy-Soup receives demonstrations from all source domains and undergoes substantial additional consolidation training, whereas the individual experts see only one domain. Without a dense policy or standard MoE trained on the same pooled data with matched optimization and capacity, it is unclear whether the gains arise from the proposed consolidation method or simply from multi-domain training and increased model capacity.

Please provide comparisons with a dense policy and a standard homogeneous MoE trained on the same pooled demonstrations, using matched training steps and parameter or compute budgets. This would clarify whether the gains come from the proposed expert-consolidation mechanism rather than pooled multi-domain training or increased capacity.

Response

The decisive controls are a pooled dense DP with complete-policy capacity matched to Soup. The pooled dense DP use one visual encoder and shared attention; the MoE uses token-wise top-2 FFNs. The parameter counts differ from Soup by at most 1.4% across the two tasks. The main comparison separates source factors from unseen compositions.

Table 1. Capacity-matched pooled-policy comparison.

TaskMethodParams MSeenHeld-outMixed
open_laptopPooled dense DP82.5080.5600.5200.388
open_laptopPolicy-Soup82.7360.5750.6500.463
shake_bottlePooled dense DP81.0460.8200.8800.645
shake_bottlePolicy-Soup81.1600.8500.9600.690

Performance cells combine success rate and Wilson 95% interval; training-seed variation is reported separately. Entries refer to the evaluated model version.

The pooled baselines target 3500 epochs, corresponding to 1000 specialist epochs plus 2500 pooled consolidation epochs per source demonstration. The baseline protocol uses FP32, AdamW (10−410^{-4} learning rate), cosine decay, 500-update warm-up, EMA and global batch 128. Fixed epochs or predeclared budgets select checkpoints; test success does not select them. The submitted Table 5 addresses initialization within a particular layout, separately from these pooled dense and standard-MoE controls.

AuthorsOpenReview ↗

Author Response to Reviewer Hx2V (part 2)

Comment 2. Total and active resource costs

Reviewer comment

The resource cost of the “unified” policy is not evaluated. Policy-Soup retains and executes all visual encoders and attention branches, with sparse routing applied only to the FFNs. Its parameter count, memory, FLOPs, and latency may therefore grow approximately linearly with the number of experts, but no comparison with a single expert, ensemble, or jointly trained policy is reported.

Please report the total and active parameter counts, inference latency, memory usage, and FLOPs as the number of experts increases. Since all visual encoders and attention branches remain active, these measurements are necessary to assess whether Policy-Soup offers a practical advantage over an ensemble or jointly trained policy.

Response

Consolidation is not full-parameter fine-tuning. Source visual encoders and attention weights are frozen; expert FFNs, routers, mixing weights, normalization layers and shared embeddings are trainable. Trainable parameters are 21.187M/82.736M (25.6%) on open_laptop and 19.611M/81.160M (24.2%) on shake_bottle. Sequence-level top-2 routing executes only two FFN experts per sample in each decoder layer. Different samples and layers can select different experts, so a batch can involve all experts.

Table 2. Training cost by stage.

TaskEst. unitShared pretrainFour expertsConsolidationFull pipelineDense 3500 ep.
open_laptopEFLOPs1.1714.26914.67220.11215.781
open_laptopGPU-h≈44\approx 44≈156\approx 156≈1399\approx 1399≈1600\approx 1600≈1684\approx 1684
shake_bottleEFLOPs1.1714.96617.04823.18418.346
shake_bottleGPU-h≈44\approx 44≈183\approx 183≈1601\approx 1601≈1828\approx 1828≈1907\approx 1907

For open_laptop, specialist training plus consolidation reconstructs 299.088M window exposures versus 299.369M in the dense 3500-epoch target; the difference is sampler padding/drop-last. Shared visual pretraining adds 23.408M windows from extra tasks. The historical consolidation uses global batch 1024, whereas the scratch pooled reference uses global128. Capacity, exposure, initialization and computation are therefore reported separately. End-to-end cost includes pretraining, all experts and consolidation; only consolidation is marginal cost for an existing library.

Table 3. Measured deployment cost per full action update.

ModelTotal MUnion MGFLOPsPeak GiBp50 msp95 ms
open_laptop
Light (smallest)20.27620.27630.0800.119910.295958.221
Table21.32621.32631.7580.124926.263993.375
Background20.80120.80130.9190.123916.897992.326
Sensor noise (largest)22.37722.37733.4360.128918.817979.163
Four-policy ensemble84.78084.779126.1920.3673700.7913985.646
Pooled dense DP82.50882.508119.3410.352952.7961005.016
Policy-Soup82.73680.667108.1500.3594100.8604369.581
shake_bottle
Light (smallest)19.22519.22528.4020.115927.462991.487
Table21.32621.32631.7580.124926.263975.853
Background20.27620.27630.0800.119916.256959.744
Sensor noise (largest)22.37722.37733.4360.128925.482969.661
Four-policy ensemble83.20483.203123.6760.3593716.6873967.364
Pooled dense DP81.04681.046117.9370.355949.0371016.392
Policy-Soup81.16079.649106.1850.3524132.4474428.969

Union counts unique executed network parameters across the full 100-step inference update, excluding fixed normalizer statistics; it is not the trainable count in Comment 1. Frozen encoders and attention still execute. Top-2 selects FFNs per sample and layer, not two complete policies; selections can change across denoising steps. The four-policy ensemble runs complete policies sequentially on one GPU and averages action chunks after denormalization.

Policy-Soup offers a capability–cost trade-off rather than model compression. Compared with a four-policy ensemble, it reduces inference matrix FLOPs by approximately 14% while maintaining similar peak GPU memory usage, at the cost of approximately 11% higher median latency in the current implementation. Consolidation does not require full-parameter fine-tuning: only 24–26% of the total parameters are trainable, and top-2 routing activates only two FFN experts per sample in each decoder layer. The frozen visual encoders and attention branches nevertheless remain part of the computation. Under the same evaluation protocol, Policy-Soup achieves better compositional generalization than the ensemble, with lower inference matrix FLOPs and a modest increase in latency.

The response partially addresses my concerns, but the new controls remain imperfectly matched: Policy-Soup benefits from separately pretrained specialists and additional visual pretraining, while the pooled baselines use different initialization and training settings. Moreover, Policy-Soup has similar peak memory and 11% higher latency than the four-policy ensemble, despite only a 14% FLOP reduction. Therefore, the benefits of the proposed consolidation mechanism over joint training or an ensemble remain insufficiently established, and I maintain my rating of 2.

Thank you for your follow-up comment. We respectfully disagree that our pooled controls remain unfairly matched or that the advantage over joint training and ensembling is unestablished.

1. Fairness of Initialization, MoE Controls, and Training Budgets

Neither Policy-Soup nor any baseline uses external pretrained weights (e.g., ImageNet). Moreover, Pooled Dense DP loads the exact same shared visual encoder as our specialists ("Shared pretrain", ~44 GPU-h in Table 2); our earlier phrase "scratch pooled reference" simply meant training directly on pooled data without stage-1 specialists (including that shared ~44 GPU-h stage brings Pooled Dense DP to ~1728 GPU-h vs. ~1600 GPU-h for Policy-Soup on open_laptop). To isolate stage-1 specialist initialization itself, Table 5 and Appendix Table 10 (Policy-Soup w/o init) already evaluate the same MoE architecture trained jointly on pooled data without specialist initialization (while Table 4 controls for homogeneous vs. heterogeneous FFN widths), confirming clear gains on unseen compositions (0.434 vs. 0.334 Mixed success on RoboTwin 2.0; 0.699 vs. 0.617 IoU on PushT).

Data exposure is also matched (4×1000×0.25+2500=35004 \times 1000 \times 0.25 + 2500 = 3500 pooled-equivalent epochs; 299.1M vs. 299.4M windows), and the remaining optimization settings actually give Pooled Dense DP more capacity to fit—updating 100% of parameters (vs. 24.2–25.6% in Policy-Soup) and using a smaller global batch size (128 across all 3,500 epochs vs. 1024 during the 2,500 consolidation epochs, yielding more gradient updates). Even so, Policy-Soup improves Held-out and Mixed success by +13.0 / +7.5 pp on open_laptop and +8.0 / +4.5 pp on shake_bottle.

Setting (open_laptop)4 Specialists + Policy-SoupPooled Dense DP
External pretrained weightsNoneNone
Initial visual encoderSame shared encoderSame shared encoder
Equivalent pooled epochs4×(1000×0.25)+2500=35004 \times (1000 \times 0.25) + 2500 = 35003500
Window exposures299.088M299.369M
Trainable parameters21.187M (25.6%)82.508M (100%)
Global batch size128 (stage 1) / 1024 (stage 2)128 (all 3,500 ep.)
Full pipeline GPU-h~1600~1728 (1684 + 44)

2. Practical Advantage Over the Four-Policy Ensemble

The four-policy ensemble is not a unified model: it runs four separate, uncoordinated policies and heuristically averages their output actions at test time. Without shared representations or routing, out-of-domain policies corrupt the averaged trajectories, causing severe failure on unseen compositions (0.123 Mixed success) and offering no unified solution across tasks or embodiments.

In contrast, Policy-Soup consolidates heterogeneous specialists into one generalizable policy, using fewer parameters and 14% fewer FLOPs while achieving 3.8x higher Mixed-domain success (0.463 vs. 0.123). Because Policy-Soup already requires fewer FLOPs, the 0.4s latency gap over 100 denoising steps (4.10s vs. 3.70s) stems solely from sequential Python-level branch execution in our prototype rather than algorithmic cost. Replacing four separate policies with a single unified policy that lifts Seen, Held-out, and Mixed success by +26.0, +31.0, and +34.0 pp clearly establishes its practical benefit.

Method (open_laptop)Unified Model?Params (M)GFLOPsPeak GiBLatency p50 (s)SeenHeld-outMixed
Four-Policy EnsembleNo (4 models)84.78126.190.3673.700.3150.3400.123
Policy-Soup (Ours)Yes82.74108.150.3594.100.5750.6500.463
Reviewer_68zPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Policy-Soup consolidates independently trained diffusion-policy experts using learned visual-feature fusion, attention-output aggregation, and sequence-level routing over FFNs with different widths. Experiments evaluate environmental generalization and consolidation across tasks and embodiments.

Critical strengths
  • Reusing specialized policies to handle unseen combinations of environmental variations is a meaningful problem.

  • The method shows encouraging improvements over source experts on several compositional shifts, with supporting real-world experiments.

  • The fixed-parameter capacity comparison and expert-initialization ablation provide useful initial evidence.

Critical weaknesses
  1. Expert consolidation versus capacity-matched pooled training. Each of the NN source experts is trained on its own domain for 1,000 epochs, followed by 2,500 epochs of consolidation on the pooled source data. Each source demonstration therefore receives 3,500 epochs of exposure across the two stages. A key missing baseline is a single dense DP with approximately the same total parameter count as Policy-Soup -- NN times the size of an individual expert—trained on the same pooled demonstrations for 3,500 epochs. How does this baseline perform, particularly on unseen domain compositions? This comparison would help determine whether separate specialization and consolidation provide benefits beyond pooled data, increased capacity, and training exposure. Please also report training FLOPs or GPU-hours, since matching data exposure and parameter count does not necessarily match compute.

  2. Value of heterogeneous capacity allocation. Table 4 supports heterogeneous allocation under a fixed FFN parameter budget, but does not show whether it is preferable to simply using width 1536 for every expert. This increases FFN parameters by only approximately 14%, with a smaller relative increase in whole-model size. Please include this homogeneous baseline under the same training protocol. If difficulty-guided allocation is a central contribution, shuffled width assignments would additionally test whether matching capacity to domain difficulty matters.

Questions for authors

Please see the two experimental questions in the Weaknesses section; their results could materially affect my assessment.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer 68zP (part 1)

Thank you for recognizing the compositional-generalization setting and for identifying the decisive capacity and exposure controls. We address pooled training, maximum width and width assignment separately.

Empty cells denote results or records not yet available; target settings and architectural calculations are identified separately.

Comment 1. Capacity-matched pooled training and total cost

Reviewer comment

Each of the source experts is trained on its own domain for 1,000 epochs, followed by 2,500 epochs of consolidation on the pooled source data. Each source demonstration therefore receives 3,500 epochs of exposure across the two stages.

A key missing baseline is a single dense DP with approximately the same total parameter count as Policy-Soup … trained on the same pooled demonstrations for 3,500 epochs.

How does this baseline perform, particularly on unseen domain compositions? This comparison would help determine whether separate specialization and consolidation provide benefits beyond pooled data, increased capacity, and training exposure. Please also report training FLOPs or GPU-hours, since matching data exposure and parameter count does not necessarily match compute.

Response

The pooled dense control matches the complete deployed model capacity and targets 3500 pooled epochs. For open_laptop, it has 82.508M parameters versus 82.736M for the profiled Soup checkpoint (−0.276%-0.276\%); for shake_bottle, the counts are 81.046M and 81.160M (−0.141%-0.141\%). The dense networks use a single ResNet-18 and an eight-layer Transformer: d=768d=768, FFN width 2304 for the former and d=736d=736, FFN width 2576 for the latter. Performance is separated into source-factor, held-out-factor and compositional conditions.

Table 1. Capacity-matched pooled-policy comparison.

TaskMethodParams MSeenHeld-outMixed
open_laptopPooled dense DP82.5080.5600.5200.388
open_laptopPolicy-Soup82.7360.5750.6500.463
shake_bottlePooled dense DP81.0460.8200.8800.645
shake_bottlePolicy-Soup81.1600.8500.9600.690

The baseline uses FP32 AdamW at 10−410^{-4}, global batch 128, cosine decay, 500-update warm-up and EMA. Fixed epochs or predeclared budgets select checkpoints, not test success.

The 1000 specialist plus 2500 consolidation epochs reconstruct 299.088M target-task window exposures on open_laptop, versus 299.369M for the dense target (sampler drop-last/padding). Shared pretraining adds 23.408M extra-task windows. Historical consolidation uses global batch 1024 rather than the scratch reference's 128. Matching capacity and exposure therefore does not establish matched training compute. Four experts is the summed training cost of four domain-specific policies, each trained for 1000 epochs on its own source domain. Consolidation is the subsequent 2500-epoch training of the merged Policy-Soup on pooled source demonstrations, not parameter merging alone. Full pipeline includes shared visual pretraining (counted once), all four expert-training runs, and consolidation. Consolidation alone is the marginal training cost when the pretrained expert library is available.

Table 2. Training cost by stage.

TaskEst. unitShared pretrainFour expertsConsolidationFull pipelineDense 3500 ep.
open_laptopEFLOPs1.1714.26914.67220.11215.781
open_laptopGPU-h≈44\approx 44≈156\approx 156≈1399\approx 1399≈1600\approx 1600≈1684\approx 1684
shake_bottleEFLOPs1.1714.96617.04823.18418.346
shake_bottleGPU-h≈44\approx 44≈183\approx 183≈1601\approx 1601≈1828\approx 1828≈1907\approx 1907

RTX 3090 FP32 cached-input batch16 replays (10 warm-up, 50 timed steps). GPU-h are single-GPU-equivalent compute hours, not measured end-to-end GPU-hours: H=Nwindows/(rwindows/s 3600)H=N_{\mathrm{windows}}/(r_{\mathrm{windows/s}}\,3600). Excluded: I/O, DDP, validation/checkpointing, initialization, search and failed runs. Shared pretraining is counted once per task and deduplicated across tasks. Soup replays include freezing/top-2; no further discount is applied.

End-to-end cost includes all three stages; consolidation alone is marginal cost for an existing library. Profiled Soup widths are 1024/1280/1152/1536 and 768/1280/1024/1536, respectively, distinct from the submitted allocation family in Comment 2; costs are not transferred between those model versions.

AuthorsOpenReview ↗

Author Response to Reviewer 68zP (part 2)

Comment 2. Uniform maximum-width experts

Reviewer comment

Table 4 supports heterogeneous allocation under a fixed FFN parameter budget, but does not show whether it is preferable to simply using width 1536 for every expert. This increases FFN parameters by only approximately 14%, with a smaller relative increase in whole-model size. Please include this homogeneous baseline under the same training protocol.

Response

We compare three capacity allocations on RoboTwin 2.0 open_laptop: heterogeneous widths of 1024/1536/1536/1280, equal-budget uniform widths of 1344, and uniform maximum widths of 1536. All three use the same training data, specialist-training and consolidation schedules, and top-2 routing.

The heterogeneous allocation achieves a mixed-condition success rate of 46.3%, compared with 42.8% for the uniform-maximum baseline, an improvement of 3.5 percentage points. Increasing every expert’s width to 1536 raises the FFN parameter count from 22.071M to 25.223M, or 14.28%, without improving success rates on the seen or held-out conditions. This percentage refers specifically to the FFN component; the relative increase in whole-policy parameters is smaller.

Table 3. Capacity allocation on RoboTwin 2.0 open_laptop.

AllocationExpert FFN widthsFFN MSeen SRHeld-out SRMixed SR
Heterogeneous1024/1536/1536/128022.0710.5750.6500.463
Equal-budget uniform1344/1344/1344/134422.0710.5450.6400.390
Uniform maximum1536/1536/1536/153625.2230.5700.6500.428

At the same FFN parameter budget, heterogeneous allocation also improves mixed-condition success by 7.3 percentage points over the all-1344 baseline. Together, these comparisons support the value of heterogeneous capacity allocation in this setting: it achieves better performance under mixed conditions with fewer FFN parameters than the uniform-maximum alternative.

Reviewer_68SeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper provides a method for merging expert policies in a setting where the experts are Diffusion transformers with shared encoder architectures but varying FF widths. The approach merges the experts using a mixture-of-experts approach for FFNs and weight averaging for other parts. The paper shows the approach's benefits in 3 cross-domain settings: cross-embodiment, cross-task, and cross-environment (visual variations).

Critical strengths
  • Three cross-domain settings are considered: cross-env, cross-task, and cross-embodiment. This demonstrates the method's applicability across diverse settings.
  • The task set spans 3 simulation settings (RoboTwin, Libero, PushT) and a real-world setting.
Critical weaknesses
  • the assumptions of the problem setting are quite strong. In particular, the assumptions (diffusion transformer, with experts only varying in the FFN width; availability of expert domain data during merging) can likely imply that experts were trained in a similar fashion, weakening the motivation for having different FFN widths: in such a scenario, using the same (max) of the width seems feasible, instead of introducing additional hyperparameters to tune per expert. And if the expert domain data is already available during merging then directly train the combined model, why assuming intermediate expert policies is needed? The current text lacks justification for the alternate problem settings. The following can help justify the assumptions:
    • a practical setting where the assumptions hold and the method is applicable will be much appreciated, and help justify the assumptions.
    • empirical evidence that the trained from scratch on combined data doesn't work. Table 5 only shows it for a small subset of tasks, and I have noted below a question about the experiments done for Table 5.
  • other variations in the experts: such as depth or number of transformer blocks, are also common scenarios to encounter practically, but the assumptions limit the scope of the method.

Experiments:

  • the details about dataset size used for finetuning, and the visual variations used in training vs evaluation (what does a mixed or unseen configuration look like) are not clear.
  • ideally, the results should provide confidence intervals to better compare the numbers.
  • Table 1: the naive baseline of training a single model with the combined dataset, without training expert policies, can provide a better signal for baseline comparison as it will share the same data, architecture and compute budget, while the oracle action expert policy is not a fair comparison, with respect to either the data, architecture, or compute.
  • Table 6: the other parameter merging techniques shown are mentioned to be training-free; however, the proposed method uses training to optimize the merged network. so, it is not a fair comparison.
  • Experiments seem to use 3-5 experts only. Understanding whether the method can scale to a larger or smaller number of experts is important.

minor:

  • The method description relies the experts to use Diffusion Transformers - but this is only mentioned in the diagram. since the approach is specific to diff transformers, method section should specify the choice explicitly.

  • L192: I disagree with the statement that: "the encoder trained on different source domains carry complementary visual cues. a common well trained visual encoder can work equally well to extract features for different domains. could the authors clarify the statement, or provide empirical evidence to support it?

Questions for authors
  • L160: Di is an atomic domain: what does "atomic" mean here? does it mean only one axis varies? can a domain contain mixture of variations? how is the atomicity assumption used in the method? and if not used, please consider removing it from the problem formulation.

  • the size of the final merged network is a function of the individual experts: more/bigger experts result in larger final merged networks. what tradeoff does merging provide in terms of compute and inference cost, apart from capability to handle mixed/unseed configurations.

  • Table 5 says that the two used the same hyperparameters. Does the compute budget for the "train expert and merge" include the budget needed for training the experts? If not, then the experiment becomes unfair to training from scratch (without initialization).

  • Another important baseline missing is: what if the trained-from-scratch architecture is not a mixture of experts? In particular, mixture-of-experts architectures often benefit from good initializations (to avoid collapse), and the results in Table 5 are expected. Thus, it seems important to understand whether the gains are specifically because mixture of experts was the chosen approach?

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer 68Se (part 1)

Thank you for the detailed comments. We clarify the heterogeneous consolidation framework and its practical motivation, and address the experimental questions below.

Comment 1. Assumptions and practical motivation

Reviewer comment

the assumptions of the problem setting are quite strong. In particular, the assumptions (diffusion transformer, with experts only varying in the FFN width; availability of expert domain data during merging) can likely imply that experts were trained in a similar fashion, weakening the motivation for having different FFN widths: in such a scenario, using the same (max) of the width seems feasible, instead of introducing additional hyperparameters to tune per expert. And if the expert domain data is already available during merging then directly train the combined model, why assuming intermediate expert policies is needed? The current text lacks justification for the alternate problem settings.

a practical setting where the assumptions hold and the method is applicable will be much appreciated, and help justify the assumptions.

Response

This concern reflects two misunderstandings of our framework and its deployment setting.

First, the availability of source demonstrations does not remove the motivation for reusing independently developed specialists. Our intended setting is the consolidation of specialists that have already been developed independently. For example, a robot deployment may accumulate policies for different tasks or environments over time, each trained under its own capacity and optimization budget. Task difficulty and convergence speed can motivate different model widths and training durations. Independent development allows these choices to be made for each specialist and can avoid unnecessary computation on simpler tasks. Related work in language modeling, such as Branch-Train-MiX, also studies independent expert training followed by MoE consolidation.

Access to the source demonstrations makes it possible to train the fusion components and router during consolidation. The expert checkpoints preserve the representations acquired during specialist development. Policy-Soup uses both resources to construct a unified policy, and the pooled-data baselines in Comment 2 directly evaluate the alternative of training a policy on the combined demonstrations.

Second, Policy-Soup supports heterogeneous experts through compatible module interfaces and does not require equal-shaped FFN parameters. Each expert FFN maps the shared token representation back to the same output dimension, even when its internal hidden width differs:

F(h)=∑i∈Sgifi(h),fi:Rd→Rd.F(h)=\sum_{i\in\mathcal S}g_i f_i(h),\qquad f_i:\mathbb R^d\rightarrow\mathbb R^d.

This permits routed output aggregation without equal-shaped FFN parameters. Shape-compatible attention parameters can also be averaged as an alternative merging operation. In the evaluated Policy-Soup implementation, attention is combined through learned aggregation of the retained branches' outputs. The essential requirement is an aligned input/output representation at each aggregation point; the expert FFNs can retain their different capacities. We will clarify these requirements and the deployment motivation in Section 3.

Comment 2. Pooled-training coverage and the oracle reference

Reviewer comment

empirical evidence that the trained from scratch on combined data doesn't work. Table 5 only shows it for a small subset of tasks, and I have noted below a question about the experiments done for Table 5.

Table 1: the naive baseline of training a single model with the combined dataset, without training expert policies, can provide a better signal for baseline comparison as it will share the same data, architecture and compute budget, while the oracle action expert policy is not a fair comparison, with respect to either the data, architecture, or compute.

Response

The source oracle is a condition-wise best-specialist reference. To provide a direct comparison with training on pooled demonstrations, we report the following controls at approximately matched complete-policy parameter counts.

  • Pooled dense DP uses one visual encoder and shared attention and is trained directly on the combined source data. It uses the same shared visual-encoder initialization as the specialists, without the domain-specialist training stage.
  • Homogeneous MoE uses one visual encoder, shared attention, and homogeneous FFNs with token-wise top-2 routing. It is trained directly on pooled data without domain-specialist initialization.
  • Soup-layout scratch retains Policy-Soup's heterogeneous branch layout and trains the policy without pretrained specialist weights. This compares specialist reuse with joint training of the same architecture.
AuthorsOpenReview ↗

Author Response to Reviewer 68Se (part 2)

Comment 2. Pooled-training coverage and the oracle reference (continued)

Table 2. Pooled-policy comparison at approximately matched total capacity.

TaskMethodParams MSeenHeld-outMixed
open_laptopPooled dense DP82.5080.5600.5200.388
open_laptopHomogeneous MoE83.8310.5450.6400.390
open_laptopSoup-layout scratch82.7360.5700.6700.460
open_laptopPolicy-Soup82.7360.5750.6500.463
shake_bottlePooled dense DP81.0460.8200.8800.645
shake_bottleSoup-layout scratch81.1600.8300.9400.710
shake_bottlePolicy-Soup81.1600.8500.9600.690

Compared with the pooled dense baseline, Policy-Soup improves Mixed success by 7.5 percentage points on open_laptop and 4.5 points on shake_bottle, and Held-out success by 13.0 and 8.0 points, respectively. On open_laptop, its Mixed success also exceeds the homogeneous MoE baseline by 7.3 points.

Policy-Soup achieves comparable task performance to Soup-layout scratch while reducing the additional training workload when an existing specialist library is available. Their Mixed success rates are 0.463 versus 0.460 on open_laptop and 0.690 versus 0.710 on shake_bottle. During consolidation, Policy-Soup reuses and freezes the pretrained visual encoders and attention branches, while optimizing the FFNs, routers, fusion weights, and shared components. This reduces the parameter-update workload and avoids relearning all policy components, allowing the existing specialists to be consolidated with comparable task performance.

Comment 3. Other forms of architectural heterogeneity

Reviewer comment

other variations in the experts: such as depth or number of transformer blocks, are also common scenarios to encounter practically, but the assumptions limit the scope of the method.

Response

The output-aggregation principle can combine modules with different internal structures when their input/output representations are aligned at the chosen aggregation points. Our evaluated Diffusion Transformer implementation aligns decoder depth, token dimension, and attention/action interfaces while allowing different FFN hidden widths.

Experts with different depths would require corresponding aggregation points, for example by aligning groups of blocks. The same interface principle offers a way to construct such extensions, including combinations of other module architectures, but these configurations have not been evaluated in the current experiments. We will distinguish the general interface principle from the specific heterogeneous configurations evaluated in this paper.

Comment 4. Dataset sizes and evaluation variations

Reviewer comment

the details about dataset size used for finetuning, and the visual variations used in training vs evaluation (what does a mixed or unseen configuration look like) are not clear.

Response

The two RoboTwin tasks used for the additional pooled-training comparisons have the following data sizes. Trajectory counts are per source domain, while training-window counts aggregate all four source domains.

Table 4. Demonstration counts and trajectory-level splits.

TaskSource domainsTraj./domainTrain traj./domainVal. traj./domainTotal train windows
open_laptopL/T/B/S10099185,534
shake_bottleL/T/B/S10099199,460

The split is by complete trajectory, using IDs 0-98 for training and 99 for validation. Training-window counts in L/T/B/S order are 21,398/21,373/21,144/21,619 for open_laptop and 24,824/24,894/24,970/24,772 for shake_bottle.

The RoboTwin visual variations are defined as follows.

  • L, lighting randomizes directional and point-light RGB intensities.
  • T, table texture randomizes wall and tabletop textures, with separate texture pools for training collection and unseen-texture evaluation.
  • B, background adds up to ten distractor objects to the tabletop while avoiding overlap with task-relevant objects.
  • S, sensor noise applies one of five camera corruptions at severity 3, comprising motion blur, Gaussian blur, zoom blur, fog, and glass blur.
  • O, object texture changes the target object's surface texture using DTD images and is held out from target-task training in these four-expert experiments.

The T and B configurations also include table-height perturbation in [−0.03,0][-0.03,0] m. Thus, each source domain denotes a predefined training distribution and can include more than one varying property. Appendix A.2 provides the detailed randomization parameters.

AuthorsOpenReview ↗

Author Response to Reviewer 68Se (part 3)

Comment 4. Dataset sizes and evaluation variations (Continued)

Training uses the individual L/T/B/S source collections. In the paper's RoboTwin evaluation protocol, Mixed-seen configurations are L+T, L+B, B+S, L+T+B, L+B+S, and L+T+B+S, as listed in Tables 7 and 10. These activate multiple predefined perturbations simultaneously within an episode. The Mixed-held-out configuration L+T+B+S+O additionally changes the target object's texture. For example, an L+T+B+S episode combines changed lighting, wall/table textures, tabletop distractors, and camera corruption. Such combinations are absent from the source training collections.

The RoboTwin protocol evaluates 100 accepted episodes per configuration using the expert-solvable scene filter and the standard task-success criterion, with the same scene-seed procedure across policies. Shared visual-encoder pretraining separately uses click_bell L/T/B data and clean collections of place_burger_fries, lift_pot, and place_object_stand, totaling 450 trajectories and 46,829 training windows. The pooled dense baseline and specialists use the same shared encoder initialization.

Comment 5. Confidence intervals

Reviewer comment

ideally, the results should provide confidence intervals to better compare the numbers.

Response

We have reported results across multiple random seeds for a subset of experiments. Computational constraints have limited additional repetitions during the rebuttal period. In the final version of the paper, we will provide 95% confidence intervals for all reported performance results and specify the number of seeds and evaluation episodes used for each experiment.

Comment 6. Table 6 and training-free merging comparisons

Reviewer comment

Table 6: the other parameter merging techniques shown are mentioned to be training-free; however, the proposed method uses training to optimize the merged network. so, it is not a fair comparison.

Response

The submitted Table 6 evaluates attention-output aggregation against replacing the attention branches in a consolidated policy with merged attention parameters. The merging variants are evaluated without additional training after replacement.

To give the alternative merging methods the same opportunity for adaptation, we will add a baseline with a subsequent training stage. The attention-merge baseline combines compatible attention parameters before adaptation on the pooled source data, averages compatible non-FFN parameters, and trains the resulting policy and router on the pooled data.

Both methods will use the same pretrained source specialists, pooled demonstrations, and consolidation training schedule. We will specify the encoder initialization and trainable modules for each method and compare success rates on seen, held-out, and mixed conditions.

Table 6. Policy performance after pooled adaptation.

TaskMethodSeen SRHeld-out SRMixed SR
open_laptopAttention merge + adaptation0.2950.5300.213
open_laptopPolicy-Soup0.5750.6500.463

Comment 7. Scaling the number of experts

Reviewer comment

Experiments seem to use 3-5 experts only. Understanding whether the method can scale to a larger or smaller number of experts is important.

Response

Our experiments cover six common domain types across the evaluated benchmarks. Each consolidated policy combines three to five specialists, corresponding to the source domains included in that experimental setting. Appendix B.4 further reports five-expert consolidation on grab_roller and turn_switch, with all five RoboTwin domains included as training domains.

These results establish consolidation of up to five experts in the evaluated settings. To evaluate larger libraries, we are extending the experiments to combine more task specialists in a multi-task setting. These experiments are still in progress because of limited computational resources, and we will include their results in the camera-ready version.

Comment 8. Explicit identification of the base architecture

Reviewer comment

The method description relies the experts to use Diffusion Transformers - but this is only mentioned in the diagram. since the approach is specific to diff transformers, method section should specify the choice explicitly.

Response

We will explicitly state in Section 3 that the implementation evaluated in this paper uses a Diffusion Transformer, as detailed in Appendix A.1. We will also state the interface requirements for expert consolidation, distinguishing the evaluated architecture from the broader output-aggregation principle described in Comments 1 and 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.