- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to combine independently trained robot skills with different action spaces inside one vision-language backbone. For selected weight matrices of Qwen3-VL-2B, it fixes the singular vectors from an SVD and represents each skill as gain vectors that rescale the singular values (about 0.34M parameters per skill), regularized toward 1 during training. Skills are composed by adding their weighted deviations from 1, after which only each embodiment's action head is retrained. On CARLA driving, LIBERO manipulation and Unitree G1 humanoid control in simulation, two- and three-skill compositions remain functional with fewer parameters than LoRA, but with varying drops in closed-loop success.
Critical strengths
- The idea is simple and clear.
- The experiments include 3 different embodiments and action spaces, including closed-loop evaluation.
Critical weaknesses
- The need for composition is not established. Each embodiment knows its identity at inference time and uses its own action head. Simply loading the matching skill per embodiment (1.4 ms from CPU, as measured in the paper) would give isolated-expert performance (97% / 84% / 89% closed-loop), whereas composition gives 95% / 75% / 75%. The paper concedes that isolated experts are simpler when one embodiment suffices but never presents a scenario where several skills must be active.
- Closed-loop results are reported only for the proposed method, not for any baseline (LoRA merging or joint training).
- Rank-8 LoRA merges are better on both tasks; the advantage of the method only in parameter size. But the saving in parameter size is quite small compared to the backbone. so the saving matters little in practice. Rank-1 LoRA (1.09M) merges to a comparable result.
- The paper does not state which token's 2048-dimensional representation the heads read, or whether language instructions are used.
- Active skills degrade language ability.
Questions for authors
- Can you do an ablation study on λ (e.g., 0, 0.01 and 0.1) with the same training length?
- Can you show closed-loop results for LoRA merging and joint training?
- Does the positive transfer on driving come from composition itself?
- When a skill with a large deviation is added, or an existing skill is removed, do the existing heads still need no recalibration?
2 comments
Response to KnZk (Part 2 of 2)
Response to KnZk W4
Thank you for pointing out these missing details. Each action head reads the backbone’s 2048-dimensional final-layer hidden state at the last token of the input after applying the chat template. For G1, we append the 16-dimensional proprioceptive state to this vector before passing it to the head.
All three tasks use a camera image and a language instruction. LIBERO uses its task instruction, and G1 uses a block-stacking instruction. For CARLA, the driving instruction is followed by the ego state, recent trajectory, nearby vehicles, traffic light state, and navigation command in text form. For each task, all methods use the same input format.
Response to KnZk W5
We agree that active skills reduce general language performance, as shown in Section 4.5 and Table 5 of the manuscript. Our control evaluations use language instructions with the skills active.
We compared two driving experts trained for 20 epochs with zero weight decay on the gains. We measured GSM8K accuracy with the driving skill active.
Table K09. GSM8K accuracy with the driving skill active
| Gain regularization | GSM8K accuracy |
|---|---|
| λ=0 | 50% |
| λ=1 | 62% |
Stronger regularization improved GSM8K accuracy by 12 percentage points in this test. The choice of λ also affects control accuracy (details in oKgT W1 and Q1). For language-only use, setting all skill coefficients to zero restores the pretrained configuration and its scores on our evaluation subsets, 78.0% on GSM8K and 45.0% on MMLU.
Response to KnZk Q1 and Q3
We trained the driving experts at five values of λ for 20 epochs without early stopping. For each setting, we evaluated the same driving expert alone and with the manipulation skill added at a weight of 0.5. We recalibrated the action heads in both cases (the same sweep is reported in Table V3 of our response to vVeF W1 and Q1).
| λ | Isolated driving ADE | Composed driving ADE | Composed minus isolated |
|---|---|---|---|
| 0 | 0.320 | 0.336 | +0.016 |
| 0.01 | 0.322 | 0.293 | −0.029 |
| 0.1 | 0.328 | 0.290 | −0.038 |
| 1 | 0.336 | 0.293 | −0.043 |
| 10 | 0.360 | 0.318 | −0.042 |
Q1. Gain regularization. With training length and stopping rules held fixed, both λ=0.01 and λ=0.1 lower composed driving ADE relative to λ=0. The lowest composed ADE is 0.290 at λ=0.1.
Q3. Positive transfer. In Table 11 of the manuscript, adding the manipulation skill lowers driving ADE from 0.3256 ± 0.0031 to 0.291 under the same head recalibration budget. Figure 2(b) of the manuscript shows that error increases when the original head is reused, so recalibration matters for realizing this gain. In the new sweep, adding manipulation lowers ADE in four of the five settings and raises it at λ=0. The driving benefit therefore depends on how the expert was trained.
Response to KnZk Q4
Whether a head needs recalibration depends on the skill change. In Table 2 of the manuscript, we reuse the driving and manipulation heads when adding G1 at a weight of 0.25. We also test removing skills and increasing their weights from the three-skill configuration P₀=(1.0,0.5,0.25) for driving, manipulation, and G1. Increasing a weight tests a larger gain change along an existing skill’s direction.
For each active skill, we divide the error with the head calibrated at P₀ by the error after recalibrating that head on the new configuration. Mean gain change is measured relative to P₀.
Table K10. Effect of skill changes on head reuse
| Change from P₀ | Mean gain change | Driving ADE ratio | Manipulation MAE ratio | G1 nMSE ratio |
|---|---|---|---|---|
| Reference P₀ | 0 | 1.00× | 0.99× | 0.99× |
| Remove G1 | 0.0089 | 1.02× | 1.00× | N/A |
| Double G1 weight | 0.0089 | 1.06× | 1.02× | 1.18× |
| Quadruple G1 weight | 0.0267 | 1.18× | 1.10× | 1.35× |
| Remove manipulation | 0.0440 | 1.55× | N/A | 1.28× |
| Double manipulation weight | 0.0440 | 2.70× | 1.32× | 3.25× |
After removing G1, the reused heads perform about as well as the recalibrated heads. After removing manipulation, both remaining heads benefit from recalibration. Removing manipulation and doubling its weight give the same mean gain change, with driving error ratios of 1.55× and 2.70×, respectively. When adding a new skill with a large deviation, we would check how much each head’s input features change and use its validation error under the new configuration to decide which heads to recalibrate.
Planned revisions
Following your suggestions, we will revise Sections 4.2, 4.3, and 4.5 and Appendices A.2, A.3, A.4, A.6, B.2, C.1, and C.6 of the manuscript.
Response to KnZk (Part 1 of 2)
Response to KnZk W1
We agree that selecting the isolated expert by identity is simpler when only one skill is needed and preserves the single-task performance reported in Table 1 of the manuscript. The 1.4 ms in Table 9 of the manuscript measures the time to load skill gains from the CPU.
In Experiment 2 (details in vVeF W2 and Q2), CARLA and LIBERO run concurrently on one GPU. The server knows both task identities, and each control loop sends its own input and continues running during inference. If a new action is unavailable at a control update, the controller holds its last valid command. We use the same 100 episode pairs in every mode. The same results appear in Table V4 of our response to vVeF W2 and Q2.
With one resident backbone, fixed composition served both control loops with 77 ms lower response latency and 20 more successful pairs than batched switching. The two prebuilt dense experts completed the most pairs and required two resident backbones.
Response to KnZk W2 and Q2
Thank you for suggesting these comparisons. We evaluated LoRA merging and joint training in the two-skill setting. Following Appendix A.3 of the manuscript, we ran 100 CARLA episodes and 1,000 LIBERO episodes per configuration, using the same routes and initial states across methods.
The 0.35M LoRA baseline applies rank-1 adaptation in 9 of 28 blocks and merges the experts with Task Arithmetic (TA). See our response to oKgT W2 and Q2 for the baseline setup. The LoRA results also appear in Table O5 of our response to oKgT W2 and Q2.
Table K07. Closed-loop success with LoRA merging and joint training
Our method scores 6 percentage points higher on LIBERO than the 0.35M LoRA baseline. It matches joint training on CARLA and scores 1 percentage point lower on LIBERO. We obtain these results by training each expert on its own task data and composing the experts after training.
Response to KnZk W3
We added a 0.35M rank-1 LoRA baseline to the offline comparison in Table 1 of the manuscript. It adapts the same seven projection types in 9 of 28 blocks and merges the experts with Task Arithmetic (TA). All methods use the same coefficient search and head recalibration budgets(details in oKgT W2 and Q2).
Table K08. Offline errors after merging or composition
At a similar parameter budget, our additive method lowers driving ADE by 3.6% and manipulation MAE by 9.2% relative to the 0.35M baseline. It matches the 28-block rank-1 baseline on driving and lowers manipulation error with about one-third as many adaptation parameters. The rank-8 baselines give the lowest errors. Spectral-TIES matches rank-8 KnOTS+TIES on manipulation at 0.110 with about 4% of its adaptation parameters.
The shared backbone accounts for most of a single model’s size. The savings accumulate when many skills share that backbone. In FP16, the adaptation weights take 0.69 MB per skill for our method and 17.4 MB for rank-8 LoRA. For 100 skills, those weights total about 68.8 MB and 1.74 GB.