acceptodds
Under review as a conference paper at ICLR 2027

Discovering Parameterized Motor Skills for Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models have shown strong potential for general-purpose robotic manipulation, yet their learned behaviors often remain tied to specific task contexts. A key observation is that semantically different manipulation tasks can share recurring short-horizon motor behaviors. Inspired by unsupervised skill discovery in reinforcement learning (RL), we formulate reusable behavior in VLA as discovering, parameterizing, and planning reusable motor skills. We introduce , a framework that discovers recurring local motion patterns from multi-task demonstrations and converts them into detachable parameterized skill modules around a frozen task-adapted VLA. Specifically, a factorized vector-quantized variational autoencoder identifies recurring motion patterns, and each discovered skill is encoded as a low-rank weight update around the frozen supervised fine-tuning (SFT) VLA policy. This parameterization provides compact and detachable skill modules that can be selectively activated while preserving the frozen base policy and its native action interface. During execution, a vision-language router performs skill planning from the current observation and language instruction by choosing either a parameterized skill or the base policy. Experiments on LIBERO and RoboTwin 2.0 show consistent improvements over SFT references, while cross-task skill reuse and real-world evaluation demonstrate that learned skills can transfer across task contexts. Our results point toward a VLA paradigm in which increasingly capable high-level reasoning can leverage reusable motor skills, rather than relearning low-level behaviors for each new task. Code is available at https://anonymous.4open.science/r/0D2F.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision–Language–Action Models, Skill Discovery, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_29BGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses a RL skill discovery-style motivation for VLAs. They use a VQ-VAE with some factorization to create independent skills. Then they train LORA adaptors on top of a VLA to output the actions of those skills. Lastly they train a router to choose the skill which yields the lowest prediction accuracy, with an additional regularizer for the VQ vae predicted skill. This yields a VLA with separate skills to be used in different environments.

Critical strengths

The idea of breaking VLA behaviors into more intrepretable skills is intuitive and I like the connection to RL. Using LORA to parameterize skills instead of the skill variable is an interesting idea. This allows them to preserve pretraining while also specializing towards skills.

Critical weaknesses

I dont find the experiments very convincing. They train on data from the same simulation as they use during evaluation. Even when the skills are from a different task set, they retrain the VLR. A better experiment would evaluate the model on a completely different sim, or sim2real, etc, to test if these learned skills generalize or not. Libero is pretty saturated so these results are not too interesting either.

Questions for authors
  1. How much does the second term in equation 11 matter? It seems unnecessary if the motivation in this section is correct.
  2. How does the adaptation cost compare to other methods? The training pipeline is extensive.
  3. Are you finetuning pi 0.5 on the new tasks since you finetune your model (the router)?
  4. How does your method compare to instead conditioning on the latent skill representation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 29BG (1/3). Response to W1 (Part 1/2): broader-task evaluation, frozen-module transfer, and VLR-only adaptation.

Response to Reviewer 29BG

Thank you for the feedback and for recognizing the connection to skill learning and the use of low-rank parameter updates to preserve the shared policy while specializing local motor behavior.

Overview of our response.

  • Performance and reuse: broader tasks, frozen-module transfer, VLR-only adaptation, and held-out real-robot compositions (Tables 1–3; W1).
  • Motion supervision: success and boundary-action ablations (Table 4; Q1).
  • Adaptation cost: measured preparation time (Q2(a)) and added parameters (Table 5).
  • Baseline training: target-task SFT for π0.5 (Q3).
  • Skill representation: comparison with latent conditioning at matched total adaptation capacity (Q4).

W1. I dont find the experiments very convincing. They train on data from the same simulation as they use during evaluation. Even when the skills are from a different task set, they retrain the VLR. A better experiment would evaluate the model on a completely different sim, or sim2real, etc, to test if these learned skills generalize or not. Libero is pretty saturated so these results are not too interesting either.

Response. Our contribution is to represent recurring local motions as independently activatable parameter skills, separating skill content from selection. Tables 1–3 test performance on harder tasks, frozen-module reuse, robustness under LIBERO-Plus perturbations, and held-out real-robot compositions.

(a) Broader task coverage. Table 1 compares the methods on the remaining 33 RoboTwin2.0 Hard tasks, with three representative examples.

Table 1. SR (%) on 33 additional RoboTwin2.0 Hard tasks. 100 rollouts per task.

MethodBeat Block HammerOpen MicrowavePlace Cans Plasticbox33 tasks Avg.
π0.523417341.45
PlugVLA33597348.08(+6.63)

PlugVLA improves average SR from 41.45% to 48.08% (+6.63 pp). Beat Block Hammer and Open Microwave gain 10 and 18 pp. This extends evaluation beyond the original task groups and the high-success LIBERO setting.

(b) Transfer and VLR adaptation. RoboTwin transfers the bank and router between the 8/9-task groups without updating either module; the base still receives target-task SFT. LIBERO-Plus evaluates robustness on LIBERO tasks across seven perturbation categories: object layout, camera viewpoint, robot initial state, language, lighting, background, and sensor noise. We distinguish direct evaluation of LIBERO-trained models from additional VLR-only training on LIBERO-Plus demonstrations.

Table 2. Transfer and VLR-only adaptation: SR (%).

Table 2a. RoboTwin transfer. PlugVLA's source skill bank and VLR remain frozen; both methods use target-task SFT. 100 rollouts per task.

Target tasks/settingπ0.5PlugVLAGain (pp)
RoboTwin 9 tasks (Easy)82.284.3+2.1
RoboTwin 8 tasks (Easy)56.858.2+1.4

Table 2b. LIBERO-Plus evaluation. All models start from training on original LIBERO data. π0.5 and frozen-VLR PlugVLA use no LIBERO-Plus training data. Adapted-VLR PlugVLA additionally trains only the VLR on LIBERO-Plus demonstrations, keeping the base policy and skill bank frozen.

Perturbationπ0.5PlugVLA (frozen VLR)PlugVLA (adapted VLR)
Object layout84.684.785.9
Camera71.875.179.6
Robot75.075.276.4
Language86.785.787.5
Lighting91.993.994.7
Background94.395.096.5
Sensor87.685.990.4
Avg.84.685.187.3

With both modules frozen, PlugVLA improves RoboTwin transfer by 2.1/1.4 pp and LIBERO-Plus by 0.5 pp on average. On LIBERO-Plus, five of seven categories improve, while language and sensor noise decline by 1.0/1.8 pp.

VLR-only adaptation raises the LIBERO-Plus average to 87.3%, gaining 2.2 pp over the frozen VLR and 2.7 pp over π0.5. All seven categories improve over the frozen VLR, led by camera/sensor (+4.6/+4.5 pp). This supports adapting skill selection to the target distribution while reusing unchanged skill parameters.

AuthorsOpenReview ↗

Response to Reviewer 29BG (2/3). Continuation of our response to W1 (Part 2/2): held-out real-robot compositions and the observed limits of skill reuse and our response to Q1.

(c) Real-robot generalization. Both methods use only A/B demonstrations for SFT, discovery, skill fitting, and router training. C–J are held out from every stage; Table 3 identifies the reused grasp and placement components.

Table 3. Real-robot grasp-and-placement tasks and results. 10 rollouts per task per method; entries are success counts. The arrow denotes grasping followed by placement/stacking, e.g., “Pick up the yellow bag, then place it on the right.” Object goals specify stacking on top of that object. ✓/✗: seen/unseen in A/B; placement denotes the object–goal pair.

IDTask: grasp → place/stackGrasp seenGoal seenPlacement seenπ0.5Ours
AYellow bag → right✓✓✓79
BPurple bag → on green box✓✓✓810
CYellow bag → on green box✓✓✗37
DGreen box → right✗✓✗08
EPurple bag → right✓✓✗48
FYellow bag → on purple bag✓✗✗05
GPurple bag → on yellow bag✓✗✗04
HYellow bag → left✓✗✗00
IPurple bag → left✓✗✗00
JGreen box → left✗✗✗00

PlugVLA raises success on held-out recombinations C/E from 30%/40% to 70%/80% and succeeds on D/F/G where π0.5 records zero successes. These tasks change an object’s role as the manipulated object or placement support, supporting motor-skill reuse under changed task contexts. Both methods fail on the unseen left-goal tasks H–J, delimiting the observed scope of reuse.

Q1. How much does the second term in equation 11 matter? It seems unnecessary if the motivation in this section is correct.

Response. The auxiliary term retains discovered motion structure alongside action-fit supervision. We test its contribution through success and boundary-action ablations, then relate it to motion retention during deployment.

(a) Complementary supervision targets. zz groups demonstrations for skill fitting; k∗k^* selects the fitted update with the lowest action loss in context. The auxiliary term connects context-dependent selection to the discovery grouping; Table 4 tests its practical contribution.

(b) Supervision ablation. Table 4 compares action-fit-only, discovery-label-only, and mixed supervision; mixed uses auxiliary weight 0.1. The base policy and skill bank remain frozen during router training.

Table 4. Router-supervision ablation: SR (%) and boundary action change. 100 rollouts per task; boundary statistics cover all 8 tasks (800 per method). Boundary change is the mean normalized RMS change in arm commands across chunks on RoboTwin8 Hard, excluding grippers; lower indicates smoother transitions.

SupervisionObjectiveSR (%): RoboTwin8 Easy | HardMean boundary action change
FM-onlyLk∗L_{k^*}59.2 | 29.30.033
VQ-onlyLzL_z58.6 | 28.70.025
MixedLk∗+0.1LzL_{k^*}+0.1L_z61.9 | 32.90.027

Mixed supervision achieves the highest SR, gaining 2.7/3.6 pp over FM-only on RoboTwin8 Easy/Hard and reducing boundary change from 0.033 to 0.027. VQ-only yields the smallest change (0.025) but lower SR. Combining the targets therefore improves success while retaining smoother transitions than action-fit supervision alone.

(c) Motion retention during execution. On RoboTwin8 Hard, predicted skills match executed-motion labels in 89.34% of successful rollouts versus 46.78% of failed ones (see our response to Reviewer TYMh, Q2). This associates successful execution with retained motion structure. The ablation tests the auxiliary term’s effect; the agreement analysis supports the relevance of its motion-based target.

The auxiliary term guides skill prediction, rather than directly penalizing adjacent action differences; boundary changes are measured consequences of the resulting policy.

AuthorsOpenReview ↗

Response to Reviewer 29BG (3/3). Q2,3,4.

Q2. How does the adaptation cost compare to other methods? The training pipeline is extensive.

Response. We report preparation time and added parameters, compare the stages with related training pipelines, and explain bank reuse. Candidate scoring is performed offline.

(a) Measured preparation time. On one 80 GB A100, RoboTwin8 skill fitting, scoring, and router training each use 3 epochs and take 4.9/2.6/3.8 h (11.3 h total); base SFT takes 54 h over 6 epochs. These stages fit local specializations, construct action-fit labels, and learn skill prediction.

Candidate scoring is offline. Deployment predicts a skill and executes its update through the shared policy, without candidate-wise loss evaluation.

(b) Parameter overhead. Table 5 includes frozen additions and excludes base weights and routing/retrieval heads, separating parameter storage from the measured preparation time.

Table 5. Additional model parameters.

MethodAdded parameters (M)
KinRT-LoRA1,076.724
BehaviorVLA33.627
PlugVLA47.481

(c) Related training pipelines. Related methods also construct supervision and organize training into stages:

  • KinRT clusters action trajectories and uses cluster IDs to supervise an observation-based expert router.
  • ChatVLA-2 [1] augments reasoning annotations with GPT-4o, co-trains on image-text and robot data, then freezes the VLM to train the action expert.
  • InstructVLA [2] pretrains an action expert, then freezes it for VLA instruction tuning with MoE adaptation, using GPT-4o-assisted annotations.

PlugVLA similarly constructs supervision before learning its predictor, using existing demonstrations to fit parameter skills, score their action fit, and train the VLR. These stages separate skill content from skill selection, enabling reuse of the fitted bank.

(d) Preparation and reuse. The additional preparation builds a reusable skill bank. In cross-task reuse, its parameters remain fixed while the base receives target-task SFT and the router adapts selection.

Q3. Are you finetuning pi 0.5 on the new tasks since you finetune your model (the router)?

Response. Yes, π0.5 receives target-task SFT, and PlugVLA uses the same reference policy.

Q4. How does your method compare to instead conditioning on the latent skill representation?

Response. We compare latent skill embeddings with parameterized skills at approximately matched total adaptation capacity, directly testing the representation choice.

Latent baseline implementation. The baseline uses 16 learned skill embeddings and layer-specific FiLM modulation before the action-expert FFNs, with the SFT backbone frozen. It shares our segment assignments, flow-matching objective, and router architecture, with its own recomputed routing labels; total adaptation capacity and fitting exposure are matched, but active capacity and FLOPs differ.

(a) Matched-capacity results. PlugVLA uses 47.481M adaptation parameters versus 47.493M for latent embeddings, excluding the router. It improves LIBERO from 97.1% to 98.0% (+0.9 pp) and RoboTwin8 Hard from 27.0% to 32.9% (+5.9 pp). Evaluation uses 50 rollouts per LIBERO task and 100 per RoboTwin8 task.

(b) Representation choice. The gains support encoding recurring behavior as detachable parameter updates. Each update specializes gated feature formation and residual modulation while preserving the shared policy’s native action interface; the router predicts its activation from task context.

References

[1] Z. Zhou et al. ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledge. NeurIPS, 2025.

[2] S. Yang et al. Vision-Language-Action Instruction Tuning: From Understanding to Manipulation. ICLR, 2026.

Reviewer_ierxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PLUGVLA, a framework for discovering recurring short-horizon motor patterns from multi-task robot demonstrations, representing each pattern as a detachable low-rank parameter specialization of a frozen task-adapted VLA, and planning when to activate these skills with a vision-language router. Skill discovery uses a factorized VQ-VAE over end-effector position, orientation, and gripper motion, selecting the K most frequent motion patterns. Each selected skill is instantiated as rank-16 LoRA updates to the gate/up projections plus residual-gate increments in the action-expert layers of a frozen π0.5 policy. A vision-language router reads frozen VLM prefix features and predicts a short-horizon sequence over the base policy and the skill bank, executing only the first choice before replanning. The method is evaluated on LIBERO, RoboTwin 2.0, and a RealMan dual-arm real-robot task-composition setting. Ablations compare random routing and joint-space skill discovery against the full model.

Critical strengths

The core formulation is clean and useful: separating skill discovery, parameter-space skill representation, and context-dependent skill planning gives a principled interface for reusable motor behavior in VLA models. Representing each discovered pattern as an independently activatable low-rank update around a frozen SFT policy is a practical and conceptually appealing alternative to latent-conditioned skill policies or task-specific experts.

The empirical story is broad and includes a real-robot task-composition study plus ablations. The cross-task skill-bank reuse protocol—freezing skill parameters and adapting only the router—is a meaningful test of whether the learned parameter specializations remain useful beyond the contexts in which they were fitted. The Random Routing and Joint VQ ablations provide useful initial evidence for context-conditioned selection and EEF-based discovery, respectively.

Critical weaknesses

The quantitative evidence for the central performance claims is not sufficiently transparent in the version I could assess. The main comparison tables (Tables 1–3) are not visible in the provided manuscript text, and the paper does not discuss variance across seeds, confidence intervals, or statistical significance. This matters because some reported transfer gains are small (e.g., 0.9 percentage points), and the main claim that PLUGVLA outperforms representative VLA baselines needs per-task/suite results and uncertainty estimates to be convincing.

The “plug-and-play” and cross-task reuse claims are somewhat overclaimed relative to the protocol. Cross-task skill-bank reuse still requires training the router on target-set demonstrations, so it is not zero-shot transfer; the paper acknowledges this, but the framing should be more explicit. For the real-robot held-out Task C, it is also unclear whether the router was trained on Task C data or only on Tasks A/B. If the router is adapted on Task C, the result supports task-composition after planner adaptation, not reuse under an unseen task without target-context supervision. The evidence is also limited to one platform and a small object/task set.

Questions for authors

In the cross-task skill-bank reuse experiments, exactly which parameters are trained on the target task set? Is the router trained on target demonstrations only, and are the skill parameters completely frozen? For the real-robot held-out Task C, was the router trained on any Task C demonstrations, or only on Tasks A and B?

How sensitive is PLUGVLA to the number of skills KK, the factorized codebook sizes, LoRA rank rr, the auxiliary label weight λsrc\lambda_{src}, and the planning horizon PP? What is the computational overhead of computing candidate-wise flow-matching losses for router supervision?

The Joint VQ ablation changes the discovery input from EEF motion to joint-space trajectories, which confounds the input representation with the factorization. Can you add an ablation isolating the factorized quantization (e.g., a single monolithic VQ codebook over EEF motion) to support the claim that factorization itself matters?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ierx (1/4).

Response to Reviewer ierx

Thank you for the detailed feedback and for recognizing the distinction between skill discovery, parameter-space skill representation, and context-dependent skill planning.

Overview of our response.

  • Performance: evaluation variability (Table 1; W1); Manuscript Tables 1–3 are reproduced at the end.
  • Skill reuse: held-out real-robot tasks, frozen-module transfer, and VLR-only adaptation (Tables 2–3; W2).
  • Training protocol: target-task parameter updates and real-robot training-data exposure (Q1).
  • Design choices: hyperparameter sensitivity (Table 4; Q2) and factorized EEF quantization (Table 5; Q3); offline scoring cost is reported in Q2.

W1. The quantitative evidence for the central performance claims is not sufficiently transparent in the version I could assess. The main comparison tables (Tables 1–3) are not visible in the provided manuscript text, and the paper does not discuss variance across seeds, confidence intervals, or statistical significance. This matters because some reported transfer gains are small (e.g., 0.9 percentage points), and the main claim that PLUGVLA outperforms representative VLA baselines needs per-task/suite results and uncertainty estimates to be convincing.

Response. Manuscript Tables 1–3, reproduced at the end of this response, provide the original task/suite comparisons. Table 1 reports variation across evaluation seeds, including RT9→8 transfer.

(a) Task- and suite-level results. Manuscript Tables 1–3 reproduce the RoboTwin and LIBERO comparisons, including representative baselines.

PlugVLA reaches 98.0% average SR on LIBERO, improving Long from 92.4% to 94.8%. The original RoboTwin Hard results are 70.3% on the 9-task group and 32.9% on the 8-task group. The skill-reuse rows separately evaluate frozen source skill parameters with target-router adaptation.

(b) Evaluation variability. We evaluate fixed checkpoints using seeds 0, 7, and 42. Table 1 reports mean ± sample SD of the seed-level aggregate SR; LIBERO aggregates suites, and RoboTwin aggregates tasks.

Table 1. SR (%) across evaluation seeds, mean ± sample SD. Three seeds; 50 rollouts per LIBERO task and 100 per RoboTwin task per seed. RT9→8 uses a frozen source bank and target-router adaptation. Gain is in pp.

MethodLIBERORT8 HardRT9 HardRT9→8 transfer
π0.596.90±0.1026.43±0.2164.50±0.5626.47 ± 0.42
PlugVLA98.07±0.1233.00±0.5670.20±0.3627.53 ± 0.25
Gain (pp)+1.17+6.57+5.70+1.06

PlugVLA improves over π0.5 on every evaluation seed, with mean gains of 1.17/6.57/5.70 pp on LIBERO/RoboTwin8 Hard/RoboTwin9 Hard. In RT9→8 transfer, the frozen source bank with target-router adaptation achieves 27.53±0.25%, versus 26.47±0.42% for π0.5, a modest 1.06 pp mean gain.

Alongside performance, the frozen-bank experiments test our central contribution: whether learned motor specializations remain useful when reused across task contexts without refitting their parameters.

AuthorsOpenReview ↗

Response to Reviewer ierx (2/4). W2. The “plug-and-play” and cross-task reuse claims are somewhat overclaimed relative to the protocol.

W2. The “plug-and-play” and cross-task reuse claims are somewhat overclaimed relative to the protocol. Cross-task skill-bank reuse still requires training the router on target-set demonstrations, so it is not zero-shot transfer; the paper acknowledges this, but the framing should be more explicit. For the real-robot held-out Task C, it is also unclear whether the router was trained on Task C data or only on Tasks A/B. If the router is adapted on Task C, the result supports task-composition after planner adaptation, not reuse under an unseen task without target-context supervision. The evidence is also limited to one platform and a small object/task set.

Response. We clarify the adaptation protocol and add held-out real-robot evaluation, frozen-module transfer, and VLR-only adaptation.

(a) Original cross-task reuse. The base and router use target demonstrations; the source bank stays frozen. “Plug-and-play” refers to attaching or bypassing fixed skill updates; the protocol includes target adaptation.

(b) Held-out real-robot tasks. No C–J demonstrations enter SFT, discovery, skill fitting, or router training. Both methods train only on A/B. Table 2 quantifies Task C and the additional held-out tasks.

Table 2. Real-robot results. Success counts over 10 rollouts per task per method. Arrows denote grasping then placement/stacking; object goals denote stacking on that object. ✓/✗: seen/unseen in A/B; placement denotes the object–goal pair.

IDTask: grasp → place/stackGrasp seenGoal seenPlacement seenπ0.5Ours
AYellow bag → right✓✓✓79
BPurple bag → on green box✓✓✓810
CYellow bag → on green box✓✓✗37
DGreen box → right✗✓✗08
EPurple bag → right✓✓✗48
FYellow bag → on purple bag✓✗✗05
GPurple bag → on yellow bag✓✗✗04
HYellow bag → left✓✗✗00
IPurple bag → left✓✗✗00
JGreen box → left✗✗✗00

Task C combines A’s grasp object with B’s placement goal; PlugVLA raises success from 30% to 70%. It also improves recombination E and succeeds on object-role changes D/F/G where π0.5 records zero successes. These results quantify reuse under task contexts absent from training.

(c) Transfer and VLR adaptation. Table 3 compares frozen-module transfer with VLR-only adaptation.

Table 3. Transfer and VLR-only adaptation: SR (%).

Table 3a. RoboTwin transfer. PlugVLA's source skill bank and VLR remain frozen; both methods use target-task SFT. 100 rollouts per task.

Target tasks/settingπ0.5PlugVLAGain (pp)
RoboTwin 9 tasks (Easy)82.284.3+2.1
RoboTwin 8 tasks (Easy)56.858.2+1.4

Table 3b. LIBERO-Plus evaluation. π0.5 (LIBERO) and frozen-VLR PlugVLA use no LIBERO-Plus training data. The other three methods use LIBERO-Plus demonstrations: π0.5 (SFT) and ACoT-VLA [1] undergo SFT with their VLM modules frozen; PlugVLA trains only the VLR, keeping its VLM, the rest of the base policy, and the skill bank frozen. Best results are bold.

Perturbationπ0.5 (LIBERO)PlugVLA (frozen VLR)π0.5 (SFT)ACoT-VLA (SFT)PlugVLA (adapted VLR)
Layout84.684.785.184.985.9
Camera71.875.170.591.279.6
Robot75.075.252.662.576.4
Language86.785.781.480.387.5
Light91.993.994.395.194.7
Background94.395.094.791.596.5
Noise87.685.979.388.390.4
Avg.84.685.179.784.887.3

Frozen modules improve both RoboTwin transfer directions and five of seven LIBERO-Plus categories (+0.5 pp on average). VLR-only adaptation reaches 87.3%, exceeding π0.5 (SFT) by 7.6 pp and ACoT-VLA by 2.5 pp. Among the three target-adapted methods, PlugVLA leads in five of seven categories; ACoT-VLA leads on Camera and Light. These results support reusing learned motor specializations through adapted skill selection, without refitting the skill parameters.

[1] Linqing Zhong et al. ACoT-VLA: Action Chain-of-Thought for Vision-Language-Action Models. CVPR, 2026.

AuthorsOpenReview ↗

Response to Reviewer ierx (3/4). Q1,2,3.

Q1. In the cross-task skill-bank reuse experiments, exactly which parameters are trained on the target task set? Is the router trained on target demonstrations only, and are the skill parameters completely frozen? For the real-robot held-out Task C, was the router trained on any Task C demonstrations, or only on Tasks A and B?

Response.

  • Original RoboTwin cross-task reuse: The base policy receives target-task SFT, and the router is trained on target demonstrations only. The source skill bank remains completely frozen.
  • Real-robot evaluation: SFT, skill discovery, skill fitting, and router training use only Tasks A/B. No demonstrations from Task C or the additional held-out tasks D–J enter any training stage.

Q2. How sensitive is PLUGVLA to the number of skills, the factorized codebook sizes, LoRA rank, the auxiliary label weight, and the planning horizon? What is the computational overhead of computing candidate-wise flow-matching losses for router supervision?

Response. The requested factors span discovery (KK, codebooks), representation (LoRA rank), and planning (auxiliary weight, horizon). We report results for all five factors and separately quantify the offline cost of constructing router labels.

(a) Hyperparameter sensitivity. Table 4 evaluates all five requested factors on Place Dual Shoes. PP counts predicted skill positions.

Table 4. Hyperparameter sensitivity on Place Dual Shoes: SR (%). 20 rollouts per configuration.

FactorValuesSR (%)
Number of skills K8/16/3260/70/55
Factorized codebook sizes(4,4,3) / (5,5,4) / (6,6,5)60/70/55
LoRA rank8/16/3265/70/70
Auxiliary label weight λ0/0.1/160/70/65
Planning horizon P1/2/460/70/70

Discovery granularity shows the largest variation in these sweeps. Relative to the middle settings (70% SR), smaller/larger skill counts and codebooks reduce SR by 10/15 pp. Increasing rank from 16 to 32 or the planning horizon from 2 to 4 leaves SR at 70%, whereas rank 8 and horizon 1 yield 65% and 60%.

Moderate auxiliary supervision performs best in the tested range: weight 0.1 achieves 70%, compared with 60% at 0 and 65% at 1. Together, these results support the chosen discovery granularity and auxiliary weight on Place Dual Shoes, with no further gain from increasing rank or planning horizon beyond 16 and 2.

(b) Offline candidate scoring. Evaluating the base policy and skill banks over 3 epochs takes 2.6 h on one 80 GB A100 for RoboTwin8. The policy and bank remain frozen while flow-matching losses define k∗k^*. Labels can be reused while models, data, and scoring rules remain fixed.

Deployment does not repeat candidate scoring. The router predicts a skill, and its update is applied to the shared policy. The measured cost therefore belongs to preparation of supervision.

Q3. The Joint VQ ablation changes the discovery input from EEF motion to joint-space trajectories, which confounds the input representation with the factorization. Can you add an ablation isolating the factorized quantization (e.g., a single monolithic VQ codebook over EEF motion) to support the claim that factorization itself matters?

Response. We hold the EEF motion input fixed and compare monolithic with factorized quantization. This tests factorization separately from the input-representation change in the original Joint VQ ablation.

(a) EEF-input control. We replace factorized quantization with a single monolithic VQ codebook over EEF motion, keeping all other settings identical to PlugVLA.

Table 5. Factorized versus monolithic EEF quantization: SR (%). 50 rollouts per LIBERO task; 100 per RoboTwin8 task per seed.

EEF quantizationLIBERORoboTwin8 tasks Hard
Monolithic VQ97.42±0.1830.26±0.83
Factorized VQ98.07±0.1233.00±0.56

(b) Contribution of factorization. Factorized VQ improves LIBERO from 97.42% to 98.07% (+0.65 pp) and RoboTwin8 Hard from 30.26% to 33.00% (+2.74 pp). With EEF inputs and the remaining settings fixed, these gains support the contribution of factorized motion quantization.

Reviewer_TYMhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PLUGVLA, a framework for discovering and reusing motor skills in vision-language-action models. It discovers recurring motion patterns from demonstrations, represents them through skill-specific LoRA adapters on a frozen, task-adapted π0.5 policy, and trains a vision-language router to select the appropriate adapter during execution. Experiments evaluate task performance and cross-task reuse of the learned skill bank.

Critical strengths
  1. The framework provides a clear modular design that separates skill discovery, skill parameterization, and context-dependent selection. The adapters can be activated independently while preserving the shared base policy.
  2. The skill discovery procedure does not require manually annotated skill or task-phase labels. The use of end-effector motion patterns provides a reasonable basis for identifying local behaviors shared across tasks.
  3. The cross-task experiments provide useful evidence for skill reuse: a frozen skill bank remains beneficial after adapting only the router to target-task demonstrations. This supports the practical value of separating skill parameters from their selection mechanism.
Critical weaknesses
  1. Limited evaluation coverage and quantitative real-world validation. The RoboTwin evaluation covers selected 9-task and 8-task groups. Broader coverage, ideally including the full task set, would better establish the generality of the discovered skills. Although real-robot experiments are included, they primarily provide a qualitative task-composition example without trial counts, success rates, or quantitative baseline comparisons.
  2. Insufficient ablations to isolate the contribution of skill discovery. The method adds a bank of trainable LoRA adapters and a router after the base policy has already undergone SFT. The current ablations examine routing and motion representation, but do not sufficiently distinguish the benefits of the discovered skill structure from those of additional parameters and training. Comparisons with continued standard LoRA fine-tuning and adapter banks trained on random data partitions, under matched parameter and training budgets, would strengthen the attribution of the gains.
  3. Insufficient analysis of routing stability. The selected adapter remains fixed within each action chunk, but successive replanning steps may alternate between adapters under ambiguous observations. The paper presents routing sequences but does not quantify switching frequency, consistency across successive predictions, or action discontinuities at chunk boundaries. Consequently, it remains unclear whether undesirable switching occurs and how it affects execution.
Questions for authors
  1. In the cross-task skill-reuse experiments, was the shared SFT policy trained on demonstrations from the target task group? For the real-robot composition experiment, were Task C demonstrations used during SFT or router training? Please clarify the data exposure at each training stage to establish precisely what is held out.
  2. Since the router’s primary supervision is the lowest-loss adapter rather than the original discovery label, how often do these two labels agree? To what extent do the adapters selected during deployment retain the motion characteristics identified during skill discovery?
8 comments
AuthorsOpenReview ↗

Response to Reviewer TYMh(1/4).

Response to Reviewer TYMh

Thank you for the detailed feedback and for recognizing the modular formulation, annotation-free skill discovery, and reuse of frozen skill parameters.

Overview of our response.

  • Coverage and real-robot reuse: 33 additional tasks and held-out compositions (Tables 1–2; W1).
  • Skill discovery: continued LoRA and random-partition controls (Table 3; W2).
  • Execution: routing statistics and skill replacement (Tables 4–5; W3).
  • Data and motion structure: training exposure (Q1), label agreement and supervision ablation (Table 6; Q2).

W1. Limited evaluation coverage and quantitative real-world validation. The RoboTwin evaluation covers selected 9-task and 8-task groups. Broader coverage, ideally including the full task set, would better establish the generality of the discovered skills. Although real-robot experiments are included, they primarily provide a qualitative task-composition example without trial counts, success rates, or quantitative baseline comparisons.

Response. We add the remaining 33 RoboTwin2.0 tasks and quantitative real-robot comparisons. The former tests broader coverage; the latter tests held-out compositions after training only on A/B.

(a) Broader task coverage. On the remaining 33 RoboTwin2.0 Hard tasks, PlugVLA improves average success from 41.45% to 48.08% (+6.63 pp). Table 1 includes three representative tasks and the full 33-task average.

Table 1. SR (%) on 33 additional RoboTwin2.0 Hard tasks. 100 rollouts per task.

MethodBeat Block HammerOpen MicrowavePlace Cans Plasticbox33 tasks Avg.
π0.523417341.45
PlugVLA33597348.08(+6.63)

Beat Block Hammer and Open Microwave improve by 10 and 18 pp, while Place Cans Plasticbox retains baseline performance. The average gain extends the evidence beyond the original 8/9-task groups.

(b) Real-robot generalization. All training stages use only A/B; C–J are held out. Table 2 identifies the grasp objects, destinations, and object–goal pairs relative to A/B.

Table 2. Real-robot grasp-and-placement tasks and results. 10 rollouts per task per method; entries are success counts. The arrow denotes grasping followed by placement/stacking, e.g., “Pick up the yellow bag, then place it on the right.” Object goals specify stacking on top of that object. ✓/✗: seen/unseen in A/B; placement denotes the object–goal pair.

IDTask: grasp → place/stackGrasp seenGoal seenPlacement seenπ0.5Ours
AYellow bag → right✓✓✓79
BPurple bag → on green box✓✓✓810
CYellow bag → on green box✓✓✗37
DGreen box → right✗✓✗08
EPurple bag → right✓✓✗48
FYellow bag → on purple bag✓✗✗05
GPurple bag → on yellow bag✓✗✗04
HYellow bag → left✓✗✗00
IPurple bag → left✓✗✗00
JGreen box → left✗✗✗00

PlugVLA improves familiar-component recombinations C/E from 30%/40% to 70%/80% and succeeds on D/F/G where π0.5 records zero successes. In D, the green box becomes the manipulated object; in F/G, bags become placement supports. These gains support reuse across new compositions and object roles.

AuthorsOpenReview ↗

Response to Reviewer TYMh(2/4). W2. Insufficient ablations to isolate the contribution of skill discovery.

W2. Insufficient ablations to isolate the contribution of skill discovery. The method adds a bank of trainable LoRA adapters and a router after the base policy has already undergone SFT. The current ablations examine routing and motion representation, but do not sufficiently distinguish the benefits of the discovered skill structure from those of additional parameters and training. Comparisons with continued standard LoRA fine-tuning and adapter banks trained on random data partitions, under matched parameter and training budgets, would strengthen the attribution of the gains.

Response. Our method organizes parameter specialization by recurring local motion. We test this design against continued LoRA and a random-partition bank, which preserves bank architecture and parameter count while changing the grouping of demonstrations (Table 3).

(a) Controlled capacity and training. All three adaptation variants train for 3 epochs at similar adapter capacity. Random and discovered partitions use the same bank architecture and identical parameter counts.

Table 3. Skill-discovery controls: SR (%) and adapter parameters. 50 rollouts per LIBERO task; 100 per RoboTwin8 Easy task.

MethodLIBERORoboTwin8 EasyAdapter parameters (M)
base SFT96.957.3—
+ Continued LoRA97.057.447.693
+ Random-partition bank97.259.147.481
+ Skill-discovery bank (ours)98.061.947.481

(b) Contribution of discovered grouping. PlugVLA gains 1.0/4.5 pp over continued LoRA on LIBERO/RoboTwin8 Easy and 0.8/2.8 pp over the random bank. The random-bank comparison fixes bank capacity and architecture, directly supporting motion-based organization of demonstrations. The continued-LoRA comparison supports the broader skill-based adaptation pipeline.

AuthorsOpenReview ↗

Response to Reviewer TYMh(3/4). W3. Insufficient analysis of routing stability and Q1. In the cross-task skill-reuse experiments, was the shared SFT policy trained on demonstrations from the target task group?

W3. Insufficient analysis of routing stability. The selected adapter remains fixed within each action chunk, but successive replanning steps may alternate between adapters under ambiguous observations. The paper presents routing sequences but does not quantify switching frequency, consistency across successive predictions, or action discontinuities at chunk boundaries. Consequently, it remains unclear whether undesirable switching occurs and how it affects execution.

Response. We measure switching frequency, prediction consistency, and boundary action changes, then replace predicted skills to test their execution effects. These analyses connect planner behavior to both command transitions and task success.

(a) Prediction consistency. On RoboTwin8 Hard, skills change at 63.13% of adjacent replanning steps, while predictions for the same future position agree 76.68% of the time. This measures the consistency of overlapping plans while allowing updates as observations change.

Task-related skill sequences. Appendix Fig. 10 (first row) illustrates switching with task progression in Handover Block: approaching the block, grasping with the left gripper, approaching again, grasping with the right gripper, approaching the target, and lowering the block for placement. Reusing an approach skill at different stages shows how returning to a previous skill can serve task-specific needs, providing context for the switching statistics.

(b) Boundary action changes. We average the normalized RMS difference between adjacent chunks’ last and first executed arm commands, excluding grippers. PlugVLA lowers this mean from 0.032 to 0.027 (Table 4). Replanning permits skill changes as observations evolve; these results describe average command transitions, rather than the effect of each switch.

Table 4. Routing consistency and boundary action changes on RoboTwin8 Hard. 100 rollouts per task across all 8 tasks (800 per method). Lower boundary change indicates smoother command transitions; brackets show 95% confidence intervals of the mean.

SourceSwitching frequency (%)Time-aligned consistency (%)Mean boundary action change
PlugVLA rollouts63.1376.680.027 [0.025, 0.029]
Base Policy rollouts——0.032 [0.030, 0.034]

(c) Execution effects of skill prediction. For five frequent skills, we execute a designated replacement whenever that skill is predicted. Table 5 measures how overriding the prediction affects task success.

Table 5. Effect of replacing predicted skills on Place Dual Shoes: SR (%). 20 rollouts per configuration. Rows: predicted skill; columns: replacement. Diagonal: unmodified policy.

Predicted skillE0E1E11E12E15
E07060654540
E16570604540
E116550705040
E125045507060
E154060605570

Replacing predicted skills reduces success by 5–30 pp; replacing E0 with E15 lowers it from 70% to 40%. The replacement-dependent changes show that the parameterizations have distinct execution effects. Table 4 describes the deployed policy’s transitions, while Table 5 tests the consequences of altering its predictions.

Q1. In the cross-task skill-reuse experiments, was the shared SFT policy trained on demonstrations from the target task group? For the real-robot composition experiment, were Task C demonstrations used during SFT or router training? Please clarify the data exposure at each training stage to establish precisely what is held out.

Response. Yes, RoboTwin SFT uses target-task demonstrations. No Task C data are used in real-robot SFT or router training. The stage-specific data exposure is as follows.

(a) RoboTwin cross-task reuse. The shared policy receives target-task SFT. The source skill bank remains frozen, and the router uses target demonstrations. For 8→9 transfer, SFT/router training uses the 9-task group and the bank comes from the 8-task group; the reverse direction exchanges these roles.

(b) Real-robot held-out compositions. SFT, skill discovery, skill fitting, and router training all use only A/B data. Neither C nor the additional held-out tasks D–J contributes demonstrations to any stage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.