acceptodds
Under review as a conference paper at ICLR 2027

Shared Spectral Coordinates for Composable Embodied Skills

Abstract

Vision-language-action (VLA) models are increasingly expected to accumulate skills across embodiments with different action spaces, without retraining for each addition. We represent each skill in the spectral coordinates of a frozen VLA backbone, so that independently learned skills can be composed without retraining the backbone, pooling their training data, or aligning them after the fact. For selected backbone projections, we freeze the pretrained singular basis and represent each skill as compact gain vectors that rescale the singular values, so all skills share one coordinate system from the start. Since alignment alone does not guarantee functional compatibility, we regularize the gains toward the pretrained configuration during learning and, after composition, recalibrate only the lightweight embodiment-specific readouts. Across autonomous driving, robot manipulation, and humanoid control, the method supports two- and three-skill composition within a single backbone using only 0.34M backbone adaptation parameters per skill, versus 1.09M for rank-1 and 8.7M for rank-8 LoRA experts, while retaining competitive offline performance and effective closed-loop control. Staying close to the pretrained configuration improves robustness to interference, and readout recalibration repairs most of the mismatch introduced by composition. Because setting all skill coefficients to zero recovers the pretrained backbone exactly, skills can be activated, combined, or removed at deployment time.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, model merging, parameter-efficient fine-tuning, singular value fine-tuning, skill composition, multi-embodiment learning, autonomous driving, robot manipulation, humanoid control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KnZkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to combine independently trained robot skills with different action spaces inside one vision-language backbone. For selected weight matrices of Qwen3-VL-2B, it fixes the singular vectors from an SVD and represents each skill as gain vectors that rescale the singular values (about 0.34M parameters per skill), regularized toward 1 during training. Skills are composed by adding their weighted deviations from 1, after which only each embodiment's action head is retrained. On CARLA driving, LIBERO manipulation and Unitree G1 humanoid control in simulation, two- and three-skill compositions remain functional with fewer parameters than LoRA, but with varying drops in closed-loop success.

Critical strengths
  1. The idea is simple and clear.
  2. The experiments include 3 different embodiments and action spaces, including closed-loop evaluation.
Critical weaknesses
  1. The need for composition is not established. Each embodiment knows its identity at inference time and uses its own action head. Simply loading the matching skill per embodiment (1.4 ms from CPU, as measured in the paper) would give isolated-expert performance (97% / 84% / 89% closed-loop), whereas composition gives 95% / 75% / 75%. The paper concedes that isolated experts are simpler when one embodiment suffices but never presents a scenario where several skills must be active.
  2. Closed-loop results are reported only for the proposed method, not for any baseline (LoRA merging or joint training).
  3. Rank-8 LoRA merges are better on both tasks; the advantage of the method only in parameter size. But the saving in parameter size is quite small compared to the backbone. so the saving matters little in practice. Rank-1 LoRA (1.09M) merges to a comparable result.
  4. The paper does not state which token's 2048-dimensional representation the heads read, or whether language instructions are used.
  5. Active skills degrade language ability.
Questions for authors
  1. Can you do an ablation study on λ (e.g., 0, 0.01 and 0.1) with the same training length?
  2. Can you show closed-loop results for LoRA merging and joint training?
  3. Does the positive transfer on driving come from composition itself?
  4. When a skill with a large deviation is added, or an existing skill is removed, do the existing heads still need no recalibration?
2 comments
AuthorsOpenReview ↗

Response to KnZk (Part 1 of 2)

Response to KnZk W1

We agree that selecting the isolated expert by identity is simpler when only one skill is needed and preserves the single-task performance reported in Table 1 of the manuscript. The 1.4 ms in Table 9 of the manuscript measures the time to load skill gains from the CPU.

In Experiment 2 (details in vVeF W2 and Q2), CARLA and LIBERO run concurrently on one GPU. The server knows both task identities, and each control loop sends its own input and continues running during inference. If a new action is unavailable at a control update, the controller holds its last valid command. We use the same 100 episode pairs in every mode. The same results appear in Table V4 of our response to vVeF W2 and Q2.

Deployment modeResident backbonesResponse latencySuccessful episode pairs
Fixed composition128 ms83/100
Batched switching1105 ms63/100
Sequential switching1145 ms58/100
Two prebuilt dense isolated experts256 ms86/100

With one resident backbone, fixed composition served both control loops with 77 ms lower response latency and 20 more successful pairs than batched switching. The two prebuilt dense experts completed the most pairs and required two resident backbones.

Response to KnZk W2 and Q2

Thank you for suggesting these comparisons. We evaluated LoRA merging and joint training in the two-skill setting. Following Appendix A.3 of the manuscript, we ran 100 CARLA episodes and 1,000 LIBERO episodes per configuration, using the same routes and initial states across methods.

The 0.35M LoRA baseline applies rank-1 adaptation in 9 of 28 blocks and merges the experts with Task Arithmetic (TA). See our response to oKgT W2 and Q2 for the baseline setup. The LoRA results also appear in Table O5 of our response to oKgT W2 and Q2.

Table K07. Closed-loop success with LoRA merging and joint training

MethodBackbone adaptation parameters per skillCARLA successLIBERO success
Ours0.34M95%77%
Rank-1 TA, 9 blocks0.35M94%71%
Rank-1 TA, 28 blocks1.09M95%74%
Rank-8 KnOTS+TIES8.7M96%80%
Joint trainingN/A95%78%

Our method scores 6 percentage points higher on LIBERO than the 0.35M LoRA baseline. It matches joint training on CARLA and scores 1 percentage point lower on LIBERO. We obtain these results by training each expert on its own task data and composing the experts after training.

Response to KnZk W3

We added a 0.35M rank-1 LoRA baseline to the offline comparison in Table 1 of the manuscript. It adapts the same seven projection types in 9 of 28 blocks and merges the experts with Task Arithmetic (TA). All methods use the same coefficient search and head recalibration budgets(details in oKgT W2 and Q2).

Table K08. Offline errors after merging or composition

MethodBackbone adaptation parameters per skillDriving ADE ↓Manipulation MAE ↓
Ours, additive0.34M0.2910.119
Rank-1 LoRA + TA, 9 blocks0.35M0.3020.131
Rank-1 LoRA + TA, 28 blocks1.09M0.2910.126
Rank-8 LoRA + TA8.7M0.2780.108
Ours, Spectral-TIES0.34M0.2850.110
Rank-8 KnOTS+TIES8.7M0.2730.110

At a similar parameter budget, our additive method lowers driving ADE by 3.6% and manipulation MAE by 9.2% relative to the 0.35M baseline. It matches the 28-block rank-1 baseline on driving and lowers manipulation error with about one-third as many adaptation parameters. The rank-8 baselines give the lowest errors. Spectral-TIES matches rank-8 KnOTS+TIES on manipulation at 0.110 with about 4% of its adaptation parameters.

The shared backbone accounts for most of a single model’s size. The savings accumulate when many skills share that backbone. In FP16, the adaptation weights take 0.69 MB per skill for our method and 17.4 MB for rank-8 LoRA. For 100 skills, those weights total about 68.8 MB and 1.74 GB.

AuthorsOpenReview ↗

Response to KnZk (Part 2 of 2)

Response to KnZk W4

Thank you for pointing out these missing details. Each action head reads the backbone’s 2048-dimensional final-layer hidden state at the last token of the input after applying the chat template. For G1, we append the 16-dimensional proprioceptive state to this vector before passing it to the head.

All three tasks use a camera image and a language instruction. LIBERO uses its task instruction, and G1 uses a block-stacking instruction. For CARLA, the driving instruction is followed by the ego state, recent trajectory, nearby vehicles, traffic light state, and navigation command in text form. For each task, all methods use the same input format.

Response to KnZk W5

We agree that active skills reduce general language performance, as shown in Section 4.5 and Table 5 of the manuscript. Our control evaluations use language instructions with the skills active.

We compared two driving experts trained for 20 epochs with zero weight decay on the gains. We measured GSM8K accuracy with the driving skill active.

Table K09. GSM8K accuracy with the driving skill active

Gain regularizationGSM8K accuracy
λ=050%
λ=162%

Stronger regularization improved GSM8K accuracy by 12 percentage points in this test. The choice of λ also affects control accuracy (details in oKgT W1 and Q1). For language-only use, setting all skill coefficients to zero restores the pretrained configuration and its scores on our evaluation subsets, 78.0% on GSM8K and 45.0% on MMLU.

Response to KnZk Q1 and Q3

We trained the driving experts at five values of λ for 20 epochs without early stopping. For each setting, we evaluated the same driving expert alone and with the manipulation skill added at a weight of 0.5. We recalibrated the action heads in both cases (the same sweep is reported in Table V3 of our response to vVeF W1 and Q1).

λIsolated driving ADEComposed driving ADEComposed minus isolated
00.3200.336+0.016
0.010.3220.293−0.029
0.10.3280.290−0.038
10.3360.293−0.043
100.3600.318−0.042

Q1. Gain regularization. With training length and stopping rules held fixed, both λ=0.01 and λ=0.1 lower composed driving ADE relative to λ=0. The lowest composed ADE is 0.290 at λ=0.1.

Q3. Positive transfer. In Table 11 of the manuscript, adding the manipulation skill lowers driving ADE from 0.3256 ± 0.0031 to 0.291 under the same head recalibration budget. Figure 2(b) of the manuscript shows that error increases when the original head is reused, so recalibration matters for realizing this gain. In the new sweep, adding manipulation lowers ADE in four of the five settings and raises it at λ=0. The driving benefit therefore depends on how the expert was trained.

Response to KnZk Q4

Whether a head needs recalibration depends on the skill change. In Table 2 of the manuscript, we reuse the driving and manipulation heads when adding G1 at a weight of 0.25. We also test removing skills and increasing their weights from the three-skill configuration P₀=(1.0,0.5,0.25) for driving, manipulation, and G1. Increasing a weight tests a larger gain change along an existing skill’s direction.

For each active skill, we divide the error with the head calibrated at P₀ by the error after recalibrating that head on the new configuration. Mean gain change is measured relative to P₀.

Table K10. Effect of skill changes on head reuse

Change from P₀Mean gain changeDriving ADE ratioManipulation MAE ratioG1 nMSE ratio
Reference P₀01.00×0.99×0.99×
Remove G10.00891.02×1.00×N/A
Double G1 weight0.00891.06×1.02×1.18×
Quadruple G1 weight0.02671.18×1.10×1.35×
Remove manipulation0.04401.55×N/A1.28×
Double manipulation weight0.04402.70×1.32×3.25×

After removing G1, the reused heads perform about as well as the recalibrated heads. After removing manipulation, both remaining heads benefit from recalibration. Removing manipulation and doubling its weight give the same mean gain change, with driving error ratios of 1.55× and 2.70×, respectively. When adding a new skill with a large deviation, we would check how much each head’s input features change and use its validation error under the new configuration to decide which heads to recalibrate.

Planned revisions

Following your suggestions, we will revise Sections 4.2, 4.3, and 4.5 and Appendices A.2, A.3, A.4, A.6, B.2, C.1, and C.6 of the manuscript.

Reviewer_oKgTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how independently trained embodied skills can be composed within a shared vision-language backbone. The method freezes the pretrained singular vectors and represents each skill through gain vectors that scale the singular values. Skills are combined by adding their deviations from the pretrained configuration, followed by task-local recalibration of embodiment-specific action heads.

Experiments cover autonomous driving in CARLA, manipulation on LIBERO-goal, and humanoid block stacking in Isaac Lab. Each skill uses approximately 0.34M backbone adaptation parameters, compared with 1.09M for rank-1 LoRA and 8.7M for rank-8 LoRA on the same projections. The three-skill configuration achieves closed-loop success rates of 95%, 75%, and 75% on the three embodiments.

The main contribution is an empirical demonstration that compact spectral experts can support composition across heterogeneous control spaces, together with experiments examining the roles of spectral adaptation and readout recalibration.

Critical strengths

S1. The paper addresses a useful problem with a clear approach.

Learning skills independently and composing them afterward avoids joint backbone training whenever a new embodiment is introduced. The shared spectral basis provides consistent coordinates, while readout recalibration addresses the feature changes caused by composition. The paper distinguishes coordinate alignment from functional compatibility and experimentally examines whether skills retain effective control after composition.

S2. The parameter-efficiency claim is supported by relevant comparisons.

In Table 1, additive composition achieves a driving ADE of 0.291 and a manipulation MAE of 0.119, compared with 0.291 and 0.126 for rank-1 LoRA with Task Arithmetic, using about 31% of its adaptation parameters. Spectral-TIES further improves the results to 0.285 and 0.110. Rank-8 KnOTS+TIES achieves better driving accuracy, but requires over 25 times as many adaptation parameters.

These results demonstrate a useful accuracy–parameter trade-off. Rank-1 LoRA provides a direct comparison at a lower parameter budget. Appendix A.4 also specifies matched coefficient-search and readout-calibration budgets across methods.

S3. The readout experiments provide meaningful evidence about why composition works.

Figure 2(b) shows that composition increases normalized driving ADE from 1.00 to 1.78, while recalibrating only the action head reduces it to 0.90. Table 11 shows that giving isolated experts the same recalibration budget does not produce a comparable improvement.

Table 3 further shows that recalibrating a head on the unadapted backbone yields substantially worse performance than the full method. These controls support the contribution of spectral adaptation and distinguish the benefits of composition from the effects of additional head training.

S4. The evaluation includes closed-loop control, incremental composition, and failure cases.

LIBERO closed-loop evaluation covers ten tasks with 100 episodes per task, and CARLA and G1 also receive closed-loop evaluation. Table 2 shows that, when adding G1, recalibrating only the new readout gives nearly the same results as recalibrating all readouts.

The paper also reports performance losses after composition and an unsuccessful skill combination in Appendix D.4. These results help establish the method’s scope.

Critical weaknesses

W1. The regularization comparison does not isolate the effect of the regularizer.

Appendix A.1 states that both driving variants use the same regularization. The over-trained variant runs for 100 epochs without early stopping, while the main configuration uses early stopping within 30 epochs. The comparison therefore does not separate regularization from training duration and early stopping.

The association between smaller spectral deviations and better composition in Section 4.3 is supported, but the stronger claims in Figure 2(a), Section 4.5, and the conclusion require a matched-schedule comparison with and without the regularizer, or revised wording.

W2. The closed-loop comparison with merging baselines is incomplete.

Table 1 compares merging baselines only through offline metrics. The proposed composition retains effective control, but LIBERO success falls from 84% to 75% in the three-skill setting, and G1 success falls from 89% to 75%. Without closed-loop baseline results, it is unclear how this performance cost compares with other merging methods.

Evaluating rank-1 TA and one strong rank-8 merging baseline under the same protocol would clarify whether the favorable offline accuracy–parameter trade-off also holds in closed-loop control.

Questions for authors

Q1. Since both driving configurations use the same regularization, how can the effect of Equation (5) be separated from training duration and early stopping? Can you provide a matched-schedule comparison with and without the regularizer, or revise the associated conclusions?

Q2. How do rank-1 TA and a strong rank-8 merging baseline perform under the same closed-loop protocol? Does the favorable offline accuracy–parameter trade-off also hold for control success?

Q3. Section 4.2 establishes the goal of multi-skill coexistence and reports positive transfer in offline driving performance. When embodiment identity is known, under what deployment conditions is a fixed composed configuration preferable to switching between isolated spectral experts?

3 comments
AuthorsOpenReview ↗

Response to oKgT

Response to oKgT W1 and Q1

Thank you for checking Figure 2(a) against Appendix A.1 in the manuscript and pointing out the difference in training schedules. Both driving experts used λ = 0.01. The main run used early stopping with a limit of 30 epochs. The other ran for 100 epochs without early stopping.

To test the gain penalty in Eq. (5) of the manuscript, we trained all five settings for 20 epochs without early stopping. We fixed the manipulation weight at 0.5 and measured driving ADE after head recalibration for both the isolated and composed models. The same results appear in Table V3 of our response to vVeF W1 and Q1.

λIsolated driving ADEComposed driving ADE
00.3200.336
0.010.3220.293
0.10.3280.290
10.3360.293
100.3600.318

Composed ADE is lowest at λ = 0.1. At λ = 10, both isolated and composed ADE are higher than at λ = 0.1. With training length and stopping rules held fixed, both λ = 0.01 and λ = 0.1 improve driving after composition relative to λ = 0.

Response to oKgT W2 and Q2

Thank you for suggesting a closed-loop comparison with LoRA merging. We tested rank-1 LoRA with Task Arithmetic (TA) and rank-8 LoRA with KnOTS+TIES in the two-skill setting. Following Appendix A.3 of the manuscript, we ran 100 CARLA episodes and 1,000 LIBERO episodes per configuration, with the same routes and initial states across methods.

We also tested a 0.35M rank-1 TA baseline. It adapts the same seven projection types as our method in nine evenly spaced decoder blocks out of 28 and follows the rank-1 LoRA training protocol in the manuscript. All merging and composition methods use the same coefficient search and head recalibration budgets.

Table O5. Closed-loop success in the two-skill setting

MethodBackbone adaptation parameters per skillIsolated success (CARLA / LIBERO)Success after merging or composition (CARLA / LIBERO)
Ours0.34M97% / 84%95% / 77%
Rank-1 TA, 9 blocks0.35M97% / 82%94% / 71%
Rank-1 TA, 28 blocks1.09M97% / 84%95% / 74%
Rank-8 KnOTS+TIES8.7M98% / 88%96% / 80%

At a similar parameter budget, our method achieves 6 percentage points higher LIBERO success. Compared with 28-block rank-1 TA, it matches CARLA success and scores 3 points higher on LIBERO with about one-third as many adaptation parameters. Rank-8 KnOTS+TIES achieves the highest success on both tasks with over 25 times as many adaptation parameters as our method. Relative to the isolated experts, LIBERO success drops by 8 to 11 points after LoRA merging and by 7 points after our composition.

Response to oKgT Q3

When only one skill is needed at a time, selecting the isolated expert by task identity is simpler. Fixed composition is useful when several control loops need fast responses from one resident backbone.

In Experiments 1a and 1b (details in vVeF W2 and Q2), we measure the time to return K skill outputs from the same input. Fixed composition runs the backbone once and shares its features across the requested action heads.

Table O6. Median latency for K skill outputs from the same input (ms)

Deployment modeK = 1K = 2K = 3
Sequential switching (factored weights)71145216
Batched switching (factored weights)N/A105162
Switching with dense weight rebuildsN/AN/A630
Fixed composition (dense weights)282828

The 630 ms result includes about 181 ms per dense weight rebuild.

In Experiment 2 (details in vVeF W2 and Q2), CARLA and LIBERO send separate inputs to one GPU and continue running during inference. Each controller holds its last valid command if a new one is unavailable at an update. With one resident backbone, fixed composition took 28 ms and completed 83 of 100 episode pairs, compared with 105 ms and 63 of 100 for batched switching.

Planned revisions

Following your suggestions, we will revise Sections 4.2 and 4.3, Figure 2(a), and Appendices A.3, A.4, A.6, and C.6 of the manuscript.

Thank you for the additional experiments. The fixed 20-epoch comparison resolves my concern in W1. With the training schedule held constant, moderate regularization lowers composed driving ADE even though the isolated expert's ADE changes little. This supports the revised claim about the effect of the gain penalty in the tested setting.

The new closed-loop comparisons also address my main concern in W2. At a similar parameter budget, the proposed method achieves 77% LIBERO success versus 71% for rank-1 Task Arithmetic, with similar CARLA success. The stronger rank-8 baseline achieves the highest success rates with substantially more adaptation parameters. I also appreciate the concurrent-control experiment, which gives a concrete answer to Q3: fixed composition is useful when two controllers need timely outputs from a single resident backbone.

These results strengthen my confidence in the paper's accuracy–parameter trade-off. The direct closed-loop comparison with merging baselines covers two skills, so the relative performance of the methods when G1 is added remains open. On balance, I maintain my original rating of weak acceptance.

Dear Reviewer oKgT,

Thank you for reviewing our new experiments and for your continued support. We appreciate your recognition that the new results address your main concerns.

Regarding the three-skill comparison, our two-skill results and the stable performance after adding G1 suggest that our method may also perform well against merging baselines in the three-skill setting.

Due to time constraints, we could not complete it during the rebuttal. We plan to include this comparison in the future.

Thank you again for your helpful feedback.

The Authors

Reviewer_vVeFOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the composition of independently learned embodied skills within a shared vision-language backbone. Each skill is represented by spectral gain vectors that rescale pretrained singular values, together with an embodiment-specific action head. The experts are combined through arithmetic on their gains, followed by action-head recalibration. Experiments evaluate two- and three-expert composition across driving, robotic manipulation, and humanoid control in simulation.

Critical strengths
  1. The framework provides a compact representation of embodied experts, requiring approximately 0.34M backbone adaptation parameters per skill, excluding the action heads. Independent skill acquisition and a shared backbone offer a useful modular deployment interface.
  2. The evaluation includes closed-loop experiments across three heterogeneous control domains. Analyses of readout recalibration, coefficient sensitivity, incremental expert addition, and failure cases help clarify the conditions under which composition remains effective.
  3. The focus on unifying heterogeneous embodied tasks is interesting. By considering driving, manipulation, and humanoid control together, the paper broadens the scope beyond unifying different robot embodiments within a single domain, such as manipulation. This setting raises meaningful challenges for sharing representations across distinct task objectives and action spaces.
Critical weaknesses
  1. Limited methodological novelty relative to Transformer². The central ingredients—fixed pretrained singular directions, task-specific gain vectors, independently trained spectral experts, and expert combination using CEM-selected coefficients. The main additions appear to be adaptation to heterogeneous continuous action spaces, gain regularization, and post-composition readout recalibration. These constitute a relevant extension, but the paper does not sufficiently establish a substantial methodological advance over prior spectral-expert frameworks. A direct, controlled comparison with a Transformer²-style baseline adapted to the same control setting would better isolate the contribution.
  2. Insufficient comparison with routing or expert-switching alternatives. Although isolated expert results are reported, the paper lacks a deployment comparison against MoE-style routing or the simpler alternative of selecting an expert using the known embodiment identity. The latter could share the same frozen backbone while storing the same compact gain vectors and action heads, without requiring a jointly trained router. This comparison is particularly relevant because three-expert fusion reduces LIBERO success from 84% to 75% and G1 success from 89% to 75%. Without matched comparisons of success rate, memory, latency, and calibration cost, the practical advantage of fixed fusion over expert switching remains unclear.
  3. No physical-robot validation. All closed-loop experiments are conducted in simulation: CARLA, LIBERO, and Isaac Lab for G1. Consequently, the robustness of composed experts and their recalibrated readouts under real perception noise, actuation errors, and dynamics mismatch remains unverified. Physical-robot experiments would substantially strengthen the evidence for practical cross-embodiment skill accumulation and deployment.
Questions for authors
  1. How much of the improvement comes specifically from gain regularization, readout recalibration, and the composition rule, relative to a Transformer²-style spectral-expert baseline with identical action heads and training budgets? In particular, can the effect of regularization be separated from training duration and early stopping?
  2. What concrete deployment benefit does fixed fusion provide over retaining the same gain vectors and readouts and selecting the isolated expert using the known embodiment identity? Could the authors compare task success, memory usage, and switching latency under matched deployment budgets?
2 comments
AuthorsOpenReview ↗

Response to vVeF W1 and Q1

Thank you for suggesting comparison with Transformer² and for asking us to separate the effect of regularization from training duration.

Transformer² selects or mixes spectral experts for a target task. Our contribution is to combine models learned independently on their own data into one fixed model that serves multiple closed-loop tasks with different continuous action spaces. Our composition rule lets us set each skill’s weight independently while keeping the pretrained reference fixed. At deployment, every request uses the same composed backbone weights, and task identity selects the action head. For example, a warehouse GPU could process separate images and instructions from a mobile robot and a picking arm and return navigation waypoints and manipulation action chunks, respectively. Experiment 2 tests this deployment setting in simulation by running CARLA and LIBERO concurrently on one GPU (details in vVeF W2 and Q2).

Table V1. Effect of readout recalibration

We compare the original and recalibrated heads on the same fixed composition.

Driving checkpointCoefficients (driving, manipulation)Driving ADEManipulation L1
Epoch 10(1.0, 0.5)0.668 → 0.272 (59.3% lower)0.244 → 0.139 (43.0% lower)
Epoch 5(1.0, 1.0)1.324 → 0.473 (64.3% lower)0.192 → 0.127 (33.9% lower)

Updating only the heads reduces error on both tasks by adapting them to the composed features.

Table V2. Comparison with Transformer²-style mixing

All rules use the same experts and starting heads. For each rule, we run 40 CEM evaluations with the heads fixed, then recalibrate across three seeds. The fixed-coefficient test uses a driving weight of 1.0 and manipulation weights from 0.5 to 1.0 before normalization.

RuleCEM coefficients (driving, manipulation)CEM driving ADECEM manipulation L1Fixed-coefficient driving ADE
Transformer²-style, unnormalized(0.95, 0.05)0.2093 ± 0.00230.1531 ± 0.00121.67 to 2.18
Transformer²-style, normalized(0.97, 0.03)0.2082 ± 0.00320.1478 ± 0.00210.250 to 0.480
Our rule(0.90, 0.14)0.2092 ± 0.00300.1523 ± 0.00380.237 to 0.456

After CEM and recalibration, all three rules favor driving and give similar errors. At the tested fixed weights, our rule gives lower driving ADE than unnormalized mixing and similar ADE to normalized mixing. It keeps the pretrained reference fixed when one skill’s coefficient stays at 1 and another changes.When the coefficients sum to one, our rule is algebraically equivalent to normalized gain mixing. Unlike normalized mixing, it also permits independently adjusted skill coefficients while retaining a fixed pretrained reference.

Table V3. Effect of gain regularization at a fixed training length

Both driving experts in manuscript used λ=0.01\lambda=0.01, with different training lengths and stopping rules. We trained all five settings for 20 epochs without early stopping and fixed the manipulation weight at 0.5. We report driving ADE after head recalibration for both models.

λ\lambdaIsolated driving ADEComposed driving ADEComposed minus isolated
00.3200.336+0.016
0.010.3220.293−0.029
0.10.3280.290−0.038
10.3360.293−0.043
100.3600.318−0.042

Moderate gain regularization improves composed driving at the tested manipulation weight with training length and stopping rules held fixed. The original λ=0.01\lambda=0.01 lowers composed ADE from 0.336 to 0.293, and λ=0.1\lambda=0.1 gives the lowest value at 0.290. At λ=10\lambda=10, both isolated and composed errors are higher than at λ=0.1\lambda=0.1.

AuthorsOpenReview ↗

Response to vVeF W2, Q2, and W3 (with Planned Revisions)

Response to vVeF W2 and Q2

Thank you for suggesting expert selection by embodiment identity as a baseline. We compare it with fixed composition using Qwen3-VL-2B in FP16. Switching selects the expert by task identity and uses its original action head.

Deployment modes

  • Sequential switching runs the isolated experts one at a time on one resident backbone.
  • Batched switching uses same backbone and applies the selected expert’s gains to each batch input using precomputed SVD factors.
  • Fixed composition uses one prebuilt dense backbone and its calibrated action heads.
  • Two dense isolated experts keeps both prebuilt expert backbones in Experiment 2.

Experiments 1a and 1b. We measure latency and peak GPU memory for one to three skill outputs from the same input, timing 200 requests per mode. Fixed composition shares features from one backbone pass across the requested heads.

Experiment 2. CARLA and LIBERO run concurrently on one GPU, each with its own image and language instruction. CARLA replans every 0.3 seconds, and LIBERO requests a new action chunk after executing the current one. Both environments advance during inference. If no new action is ready at a control update, the controller holds its last valid command. All modes use the same control schedules and action-hold rule. The two batched modes use the same request scheduler and compute features separately for each input. Each mode runs the same 100 pairs of CARLA routes and LIBERO initial states. A pair succeeds only if both tasks succeed.

Table V4. Deployment results

MeasureSequential switchingBatched switchingFixed compositionTwo dense isolated experts
Resident backbones1112
E2 successful episode pairs58/10063/10083/10086/100
E2 response latency (ms)1451052856
E1/E1b median latency for three outputs from the same input (ms)21616228N/A
E1/E1b peak GPU memory (GB)5.756.508.58N/A

With same backbone, fixed composition responded 77 ms faster and completed 20 more episode pairs than batched switching. The latency gap includes the difference between dense and factored execution.

Switching reuses the original action heads without further calibration. For fixed composition, recalibration, when needed, takes about 7 minutes per head for driving and manipulation and about 10 minutes for G1.

Table 1’s 77% LIBERO success rate comes from 1,000 standalone episodes with the two-skill composition. E2’s 83 successful pairs imply at least 83 successes on each task within its separate 100-pair evaluation.

Response to vVeF W3

We agree that testing on real vehicles, robot arms, and humanoid is important. Our current results come from simulation. In real-world tests, we need to consider sensor noise, errors in control commands, and differences between simulated and real motion. As noted in the paper, these tests are our next step, and we have started preparing for them.

Planned revisions

Following your suggestions, we will revise the Introduction, Sections 4.2 and 4.3, and Appendices A.6 and C.6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.