acceptodds
Under review as a conference paper at ICLR 2027

Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAs

Abstract

Vision-language-action (VLA) models often struggle to generalize to skill combinations absent from their fine-tuning demonstrations, even when every constituent skill has been demonstrated. We focus on a vision shortcut as one failure mode: during fine-tuning, visual observations can serve as a proxy for the instruction, so a policy may execute a demonstrated combination associated with similar observations rather than the instructed combination. This motivates counterfactual pairs that hold a demonstration observation fixed while changing the instruction to specify an undemonstrated combination. These pairs lack demonstrated actions for supervision. Crucially, the currently required skill has already been demonstrated, but actions from those executions cannot be transferred directly because the same skill can require different actions across observations. We propose CRAFT, which transfers supervision from demonstrated executions of the required skill to counterfactual pairs through reusable skill representations. Across three VLA models and two simulation benchmarks, CRAFT improves success on undemonstrated combinations while maintaining high success on demonstrated ones. CRAFT also improves compositional generalization on a real robot.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Compositional Generalization, Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wCrsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether fine-tuned VLAs can recombine demonstrated constituent skills into combinations without corresponding demonstrations. It proposes CRAFT, which separates skill and state representations and uses counterfactual instruction–observation pairs to transfer supervision from demonstrated executions of the currently required skill. The method is evaluated across three VLA backbones on two simulation benchmarks and on a real robot.

Critical strengths
  1. The problem and method are well motivated. The distinction between having demonstrated all constituent skills and being able to recombine them is meaningful. CRAFT provides a principled way to construct counterfactual supervision without directly transferring actions across incompatible observations.
  2. The empirical improvements are large and consistent. CRAFT substantially improves undemonstrated combination success across three VLA models and both simulation benchmarks while largely preserving demonstrated task performance.
Critical weaknesses
  1. The checkpoint selection protocol weakens the central generalization claim. The paper selects checkpoints using success on the undemonstrated combinations and then reuses the same rollouts for final reporting. Since these combinations are exactly the paper’s main generalization target, this introduces selection bias. The cited prior VLA works also report the best-performing checkpoint, but they select checkpoints based on in-distribution downstream task performance, rather than on a held-out compositional generalization set.
  2. The compositional-generalization setting is relatively structured. The method assumes a fixed operation sequence, known operation–entity skills, and that every constituent skill has already been demonstrated. In addition, the target undemonstrated combinations are instantiated as counterfactual instructions during training. Thus, the paper studies generalization without action demonstrations for those combinations, rather than generalization to completely unseen task specifications.
Questions for authors
  1. Can the authors select checkpoints without using performance on the undemonstrated evaluation combinations? For example, using a fixed training step or a separate validation criterion?

  2. Can the authors evaluate on target combinations that are not explicitly instantiated as counterfactual instructions during training? This would clarify how much CRAFT generalizes to genuinely unseen combinations versus learning from counterfactual supervision for known target combinations.

8 comments
AuthorsOpenReview ↗

Response to Reviewer wCrs (1/2)

We thank the reviewer for the careful assessment and the concrete suggestions. We respond to each point below, and the corresponding revisions are marked in blue in the revised paper.

W1 / Q1. Checkpoint selection without undemonstrated combinations

The improvement of CRAFT over the baselines remains when checkpoints are selected without using performance on undemonstrated combinations. We have added the supporting results in Appendix D.6.

Fixed training step. For π0.5\pi_{0.5} and GR00T N1.7 on both benchmarks, CRAFT achieves the highest success on undemonstrated combinations at every evaluated checkpoint (Figure D.7). The improvement therefore holds at each of the evaluated training steps.

Selection by success on demonstrated combinations. The same holds when checkpoints are selected by in-distribution performance, i.e., success on demonstrated combinations, as in the prior work the reviewer refers to. In Figure D.7, the success of CRAFT at any evaluated checkpoint is above that of every standard fine-tuning baseline at any evaluated checkpoint, and also above the results of CAG-VA and Skill Composition in Table 1, which are reported at their best checkpoints. The improvement therefore holds whichever checkpoint this criterion selects for each method.

Held-out selection. We also evaluate selection and reporting on disjoint combinations. For every split of the 12 undemonstrated Pick–Place combinations into six for selection and six for reporting, we select the checkpoint on the selection combinations and report the selected checkpoint's success on the remaining six (Table D.2). Under this selection, CRAFT achieves 82.8% and 68.4% for π0.5\pi_{0.5} and GR00T N1.7, respectively, compared with 45.1% and 2.9% for the best standard fine-tuning baseline.

AuthorsOpenReview ↗

Response to Reviewer wCrs (2/2)

W2 / Q2. Structure of the setting and combinations excluded from counterfactual instructions

Following the reviewer's suggestion, we have added an evaluation on combinations that appear in neither the demonstrations nor the counterfactual instructions (Section 5.4, Table 3). On these combinations, CRAFT achieves higher success than the compared baselines.

The setting we study. Our formulation assumes a fixed operation sequence, such as picking followed by placing, and that every constituent skill appears in the demonstrations (Section 3). These conditions isolate the distinction, noted by the reviewer among the strengths, between having demonstrated all constituent skills and being able to recombine them. In the demonstrations, a later skill is executed only from the states produced by the preceding skill of a demonstrated combination. For example, placing on the blue plate is always demonstrated with the blue cube in the gripper. An undemonstrated combination therefore requires a skill to be executed from a state in which that skill was never demonstrated, which is the challenge described in Section 3. Under these conditions, standard fine-tuning achieves substantially lower success on undemonstrated combinations than on demonstrated ones (Table 1).

Counterfactual instructions. We would like to clarify that instantiating a combination as a counterfactual instruction requires no additional demonstrations, as doing so only changes the instruction paired with an observation from an existing demonstration. The resulting pair has no demonstrated action target, and providing supervision for such pairs is the problem CRAFT addresses.

Combinations excluded from counterfactual instructions (Q2). Using π0.5\pi_{0.5} on Pick–Place, we exclude four of the 12 undemonstrated combinations from the counterfactual instructions and select the checkpoint by success on the remaining eight. The four excluded combinations are therefore used neither for training nor for checkpoint selection, and their instructions never appear during training. Table 3 reports the success rates (%) on these four excluded combinations (mean ± std over three evaluation seeds).

MethodSuccess (%)
FT (Full)5.0 ± 0.5
FT (LoRA)43.7 ± 2.0
FT (Frozen)15.0 ± 2.2
CAG-VA62.0 ± 1.3
CRAFT (Excluded)69.5 ± 2.3
CRAFT79.8 ± 1.0

On the excluded combinations, CRAFT (Excluded) achieves higher success than CAG-VA and standard fine-tuning, and lower success than CRAFT trained with these combinations included. These results suggest that the skill representations learned by CRAFT generalize to combinations beyond those instantiated, and that instantiating a combination further contributes to success on that combination.

We have added this result in Section 5.4 and the corresponding details in Appendix B.2.

We hope our responses and revisions address the reviewer's comments. We will respond promptly to any further questions.

Reviewer_wCrsOpenReview ↗

Reply to authors' rebuttal

Thanks for authors' rebuttal. While I am more confident that CRAFT is effective for the problem they studied, I remain concerned that the problem formulation is overly constrained, assuming a fixed scene and operation sequence, and demonstrations covering all constituent skills. I think this narrow scope limits its insights for the broader VLA community, therefore I want to maintain my current rating.

Reviewer_RsTjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies compositional generalization in fine-tuned VLAs: whether a policy can recombine demonstrated skills (e.g., pick yellow cube, place on green plate) into combinations absent from the demonstrations. The authors attribute part of the failure to a "vision shortcut" and propose CRAFT, which adds learnable skill and state query tokens, a contrastive skill-representation loss, and a counterfactual loss. The counterfactual loss pairs a fixed observation with a changed instruction and supervises it through a demonstrated execution of the required skill. They introduce two LIBERO-based benchmarks (Pick-Place, Pick-Place-Press) and evaluate π0, π0.5 and GR00T N1.7, plus a small real-robot study. CRAFT raises undemonstrated-combination success substantially over standard fine-tuning, CAG-VA and a skill-composition baseline, while keeping demonstrated success high.

Critical strengths
  1. The problem is well motivated and the gains are large and consistent. Undemonstrated success rises from roughly 1-8% to 32-84% across three VLA architectures and two benchmarks, and the strong baselines do not close the gap.

  2. The method is conceptually clean and the analysis is reasonably thorough. The ablations show that the skill-changing counterfactual term and the combination of both counterfactual terms matter, and the representation-reuse and fixed-observation action-prediction analyses give mechanistic support for the claims.

Critical weaknesses
  1. Checkpoint selection uses the undemonstrated evaluation rollouts for both selection and reporting (Appendix B.4). This inflates the headline undemonstrated numbers and makes the comparison to baselines (which were also selected this way, but with differing sensitivity to checkpoint) hard to trust. A held-out validation split of undemonstrated combinations is needed.

  2. The setting is narrow and the vision-shortcut explanation is only weakly tested. Tasks have a fixed operation sequence, few entities (colors), and uniformly structured benchmarks with skill labels from a VLM-based segmentation. The shortcut hypothesis is supported only by correlational evidence (Appendix A) and never isolated as far as I can tell, and the real-robot study is small (one model, 5 trials per combination, one baseline) with no variance estimates.

Questions for authors
  1. How do results change if checkpoints are selected on a held-out set of undemonstrated combinations, or if the final checkpoint is reported? Please also report the per-checkpoint trajectory for CRAFT and the baselines.
  2. Could the Skill Composition baseline be made stronger (e.g., learned rather than gripper-rule transitions)? Its low Pick-Place-Press result may reflect implementation limits rather than the approach.
  3. Does CRAFT scale to larger entity sets, more than three operations, or objects/skills that differ in geometry rather than color? Why does performance vary so much by combination (e.g., blue-cube/green-plate), and does it depend on a gap between demonstrated and undemonstrated visual contexts?
  4. How much does CRAFT depend on the accuracy of VLM skill segmentation? Please include a sensitivity analysis or an oracle-segmentation comparison.
  5. Is the assumption that future entity assignments do not affect current actions validated empirically, and what happens when it is violated?
6 comments
AuthorsOpenReview ↗

Response to Reviewer RsTj (1/5)

We thank the reviewer for the detailed and constructive review. We respond to each point below, and the corresponding revisions are marked in blue in the revised paper.

W1 / Q1. Checkpoint selection and effect on the comparison

The comparison between CRAFT and the baselines does not depend on checkpoint selection. Table 1 reports, for each method, the checkpoint with the highest success on undemonstrated combinations, following prior VLA evaluations that report the best-performing checkpoint (Liu et al., 2023, Bjorck et al., 2025, Kim et al., 2025, Li et al., 2025). As requested, we now report results at every evaluated checkpoint, including the final checkpoint, and under held-out checkpoint selection (Appendix D.6).

Results at every evaluated checkpoint (Figure D.7). For π0.5\pi_{0.5} and GR00T N1.7 on both benchmarks, CRAFT achieves the highest success on undemonstrated combinations at every evaluated checkpoint. At the final checkpoint, which the reviewer asks about, CRAFT reaches 81.7% and 62.8% on Pick–Place for π0.5\pi_{0.5} and GR00T N1.7, respectively, compared with at most 39.2% and 1.5% for standard fine-tuning. On Pick–Place–Press, CRAFT reaches 50.3% and 32.3%, respectively, compared with at most 13.3% and 1.0% for standard fine-tuning.

Sensitivity to the checkpoint. The reviewer notes that methods may differ in their sensitivity to the checkpoint. As Figure D.7 shows, in all four configurations the success of CRAFT at any evaluated checkpoint is above the success of every standard fine-tuning baseline at any evaluated checkpoint. CRAFT's improvement over these baselines therefore holds regardless of which evaluated checkpoint is chosen for each method. The success of CRAFT at any evaluated checkpoint is also above the results of CAG-VA and Skill Composition in Table 1, which are reported at their best checkpoints.

Held-out checkpoint selection (Table D.2). Following the reviewer's suggestion, we separate selection from reporting. For every split of the 12 undemonstrated Pick–Place combinations into six for selection and six for reporting (924 splits), we select the checkpoint with the highest success on the selection combinations and report the selected checkpoint's success on the remaining six. Table D.2 reports the mean and standard deviation across splits for the evaluation seed used in checkpoint selection.

ModelMethodSelected on all 12Held-out selection
π0.5\pi_{0.5}CRAFT84.382.8 ± 4.4
π0.5\pi_{0.5}FT (LoRA)46.045.1 ± 4.8
GR00T N1.7CRAFT72.368.4 ± 8.3
GR00T N1.7FT (LoRA)3.02.9 ± 1.1

Under held-out selection, CRAFT maintains higher success than the best standard fine-tuning baseline.

In summary, CRAFT achieves the highest success at every evaluated checkpoint, including the final checkpoint, and maintains higher success than the best standard fine-tuning baseline under held-out selection. We have added these results in Appendix D.6.

AuthorsOpenReview ↗

Response to Reviewer RsTj (2/5)

W2. Setting, evidence for the vision shortcut, and the real-robot evaluation

We respond to the three points in turn.

Setting. Our benchmarks are designed as a controlled setting where every constituent skill appears in the demonstrations and only the combination is new, so that failures on undemonstrated combinations can be attributed to recombination rather than to new objects, skills, or scenes. Entities are distinguished by color for the same reason (Appendix C), and the method has no component specific to color.

Even with a fixed operation sequence, such as picking followed by placing, recombining demonstrated skills remains challenging. In the demonstrations, a later skill is executed only from the states produced by the preceding skill of a demonstrated combination. For example, placing on the blue plate is always demonstrated with the blue cube in the gripper. Placing the red cube on the blue plate therefore requires executing that placing skill with the red cube in the gripper, a state in which the skill was never demonstrated (Section 3). Learning each skill from constituent-skill segments does not resolve this challenge (see Q2), whereas the analysis in Section 5.3 suggests that CRAFT can execute the required skill from states reached during other skill executions.

Even in this setting, standard fine-tuning achieves substantially lower success on undemonstrated combinations than on demonstrated ones for all three VLA models (Table 1). As the reviewer notes among the strengths, the strong baselines do not close the gap, and the gains of CRAFT are consistent across three models and two benchmarks. The setting also reflects how demonstrations scale. The number of constituent skills grows with the sum of the entity counts across operations, whereas the number of combinations grows with their product. In our benchmarks, the demonstrations cover 4 of 16 and 2 of 8 combinations while containing every constituent skill. We address scaling and the VLM annotation under Q3 and Q4.

Evidence for the vision shortcut. We would like to clarify that the evidence in the paper is not limited to correlations. The paper includes experiments that change a single factor.

  • Changing only the instruction (Figure D.3). With the observation held fixed, the action predictions of FT (Full) remain largely unchanged across instructions, whereas those of CRAFT change with the instructed entity.
  • Removing the instruction (Figure A.3). On LIBERO-Goal, the fine-tuned π0.5\pi_{0.5} policy still exhibits demonstrated-task behaviors under an empty instruction.
  • Strengthening language conditioning at inference (Appendix D.5). CAG-VA amplifies the difference between the language-conditioned and vision-only predictions without retraining the policy. For π0.5\pi_{0.5} (LoRA), stronger guidance improves success on undemonstrated combinations on both benchmarks (Table D.1).
  • Changing only the objective (Table 2). With the same architecture, π0.5\pi_{0.5} trained with the flow-matching loss alone achieves 1.9% success on undemonstrated combinations, compared with 84.2% when the auxiliary objectives of CRAFT are added, which apply counterfactual supervision under changed instructions with the observation held fixed.

These experiments are consistent with the vision shortcut, as the action predictions of the fine-tuned policy show limited dependence on the instruction.

In addition, we have added an analysis of rollout outcomes on our benchmarks (Appendix D.7, Table D.3). With standard fine-tuning, most unsuccessful rollouts complete a demonstrated combination instead of the instructed one, which is the behavior expected under a vision shortcut (Section 1). For example, with FT (Full), 86.2% of the rollouts on undemonstrated combinations of Pick–Place and 94.7% on Pick–Place–Press complete a demonstrated combination.

Real-robot evaluation. The main comparison is conducted in simulation across three models and two benchmarks, while the real-robot evaluation examines whether the improvement observed in simulation also holds on a real robot. On undemonstrated combinations, CRAFT succeeds in 43 of 60 trials and FT (Full) in 9 of 60 (with 95% Wilson confidence intervals of 59.2–81.5% and 8.1–26.1%). On demonstrated combinations, both methods succeed in 19 of 20 trials.

In summary, our benchmarks provide a controlled setting that isolates skill recombination, the vision shortcut is supported by experiments that change a single factor and by the analysis of rollout outcomes, and the real-robot evaluation is consistent with the results in simulation. We have added the analysis of rollout outcomes in Appendix D.7 and clarified why the benchmarks are designed this way in Appendix C.

AuthorsOpenReview ↗

Response to Reviewer RsTj (3/5)

Q2. Skill Composition baseline and the transition rule

The reviewer asks whether this baseline could be made stronger, for example with learned rather than gripper-rule transitions, and whether the low success of Skill Composition reflects implementation limits rather than the approach. To examine this, we analyzed the rollout outcomes of this baseline (Appendix D.7) and found that the transition rule does not account for most of the baseline's failures.

Transition rule. Skill Composition fine-tunes the VLA on constituent-skill segments and, at deployment, advances to the next skill with a rule based on the observed gripper state. The rule advances from picking to placing when the gripper closes on the cube, and from placing to pressing when the gripper opens (Appendix B.4). These events occur regardless of which combination is instructed.

Where the failures occur. With π0.5\pi_{0.5}, 62.2% of the Skill Composition rollouts on undemonstrated combinations of Pick–Place and 80.3% on Pick–Place–Press complete a demonstrated combination instead of the instructed one. Completing a demonstrated combination requires every transition between skills to occur, so these failures do not arise from the transition rule failing to advance to the next skill. This is the same pattern as with standard fine-tuning (Table D.3).

Interpretation. As described under W2, in the demonstrations a later skill is executed only from the states produced by the preceding skill of a demonstrated combination. Segmenting the demonstrations by skill does not change this. In Pick–Place, every segment of placing on the blue plate still starts with the blue cube in the gripper. A policy trained on these segments can therefore still predict the action from the observation with limited reliance on the instruction. A learned transition would change when the skills switch, but not this property of the training data.

We have added this analysis in Appendix D.7 and the description of the transition rule in Appendix B.4.

Q3. Scaling, variation across combinations, and visual context

Scaling. Our two benchmarks differ in the number of entities and operations. Pick–Place has four entities per operation and two operations, and Pick–Place–Press has two entities per operation and three operations. CRAFT improves success on undemonstrated combinations on both benchmarks for all three models (Table 1). CRAFT constructs supervision by matching demonstrated executions of the required skill, and the training objectives have no component specific to the number of entities or operations. Distinguishing entities by geometry rather than color changes only the objects and their demonstrations, while this matching procedure, the architecture, and the training objectives remain the same. The benchmarks use up to four entities per operation and up to three operations, which keeps all objects within the robot's reachable workspace (Appendix C). Evaluating CRAFT at larger scales, with more entities and longer operation sequences, is left for future work (Section 6).

Variation across combinations and visual context. The values in Figure D.1 are means over three evaluation seeds, each with 50 trials per combination. Some combinations, such as blue-cube/green-plate noted by the reviewer, show lower success for all three models, whereas others, such as red-cube/green-plate, show success that differs substantially across models.

Regarding the gap in visual context, demonstrated and undemonstrated combinations share the same distribution of initial scenes. The same cubes and plates are present, and only the instruction differs. A difference arises during execution. For example, observations in which the blue cube is carried toward the green plate are absent from the demonstrations, where the blue cube is always carried to the blue plate. The analysis in Section 5.3 suggests that CRAFT can execute the required skill from states reached during other skill executions. How far such states are from the demonstrated ones may differ across combinations and contribute to the variation in success.

We have described the design of the benchmarks in Appendix C and revised the Limitations (Section 6).

Reviewer_S8FfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates compositional generalization in VLA models, focusing on the ability to execute unseen combinations of previously demonstrated skills. The authors identify vision shortcuts as a potential failure mode and propose CRAFT, a counterfactual skill-representation alignment framework.

CRAFT learns reusable skill representations through an action-prediction-based contrastive objective and introduces counterfactual supervision by aligning representations from counterfactual observation–instruction pairs with demonstrated executions of the required skills. Experiments on two compositional manipulation benchmarks across three VLA architectures show substantial improvements over standard fine-tuning and alternative baselines. The paper further provides ablation studies, representation analyses, and real-robot experiments.

Critical strengths
  1. The paper addresses a bottleneck in VLA adaptation: recombining previously demonstrated skills into task combinations absent from the fine-tuning demonstrations. CRAFT provides an interesting solution by transferring supervision through reusable skill representations rather than requiring direct action labels for counterfactual observation–instruction pairs.

  2. Convincing performance. CRAFT demonstrates substantial improvements in success rates on undemonstrated combinations across three VLA architectures and two benchmarks, while maintaining strong performance on demonstrated combinations.

  3. Supportive Ablation studies. The comparisons against different fine-tuning strategies, CAG-VA, and skill composition baselines are useful. The training-objective ablations, instruction-change analyses, and functional skill-representation reuse experiment provide additional support for the proposed mechanism. The real-robot experiments further strengthen the empirical evidence.

Critical weaknesses

1. The counterfactual supervision mechanism relies on an assumption of skill independence. Section 3 assumes that, given a fixed observation and currently required skill, changing the targets of future operations does not affect the action required for the current operation. While this assumption is reasonable for the considered benchmarks, it may not hold in more complex manipulation tasks where current actions depend on downstream goals. For example, the appropriate grasp configuration may depend on the subsequent placement or manipulation objective. The paper explicitly acknowledges this assumption. Nevertheless, it limits the applicability of the proposed counterfactual supervision mechanism to tasks with sufficiently separable skills. Further discussion of its validity and potential extensions to more tightly coupled skills would strengthen the methodological contribution.

2. The contribution of skill contrastive learning is not fully isolated in the ablation study. Table 2 provides useful ablations of the proposed training objectives and demonstrates substantial improvements when combining skill-changing and skill-preserving counterfactual supervision. However, all reported configurations involving counterfactual supervision also include the skill contrastive loss. The configuration combining both counterfactual objectives without the skill contrastive loss is missing. Therefore, it remains unclear how much of the full method's performance gain is attributable to contrastive skill-representation learning, beyond the combined counterfactual supervision itself.

Questions for authors

Q1. Section 3 assumes that changing future operation targets does not affect the action required for the current skill. Could the authors discuss how this assumption affects CRAFT in tasks where current actions depend on downstream goals, such as goal-dependent grasp selection?

Q2. Table 2 provides useful ablations of the auxiliary objectives. Could the authors additionally report a configuration combining the skill-changing and skill-preserving counterfactual losses without the skill contrastive loss?

5 comments
AuthorsOpenReview ↗

Response to Reviewer S8Ff (1/2)

We thank the reviewer for the careful reading and constructive feedback. We respond to each point below, and the corresponding revisions are marked in blue in the revised paper.

W1 / Q1. Assumption on future entity assignments and extension to coupled skills

We would like to clarify that the assumption in Section 3 is tied to defining a skill as an operation–entity pair, rather than to the architecture or training objectives of CRAFT. CRAFT can therefore, in principle, be extended to coupled skills by defining the skill to include the information on which the current action depends. We explain this below.

How CRAFT uses the skill definition. CRAFT is built on the principle that the current action is determined by the current observation and the currently required skill. Specifically, the action expert is conditioned on a state representation and a skill representation, where the skill representation is computed from the image and the full instruction (Section 4.1). How a skill is defined depends on the task, and this definition enters the training objectives by determining which executions share a skill in Lskill\mathcal L_{\mathrm{skill}} (Section 4.2) and whether a counterfactual instruction preserves or changes the currently required skill in Lcf\mathcal L_{\mathrm{cf}} (Section 4.3).

In the setting formulated in Section 3, an operation–entity pair is sufficient to determine the current action, and our benchmarks are constructed to satisfy this setting (Appendix C). For example, how a cube is picked does not depend on the plate on which the cube will be placed. As the reviewer notes, this assumption is reasonable for the considered benchmarks. Even with such separable skills, standard fine-tuning achieves substantially lower success on undemonstrated combinations than on demonstrated ones (Table 1), which is the problem we address.

Extension to coupled skills. In the reviewer's example of goal-dependent grasp selection, executions of the same operation–entity pair contain different grasps, so CRAFT may use a grasp demonstrated for a different placement as the supervision target. In this case, the skill is again sufficient to determine the current action once defined to include the information on which the action depends. For example, suppose that placing on a flat plate requires a top grasp and placing on a shelf requires a side grasp. The picking skill is then defined as (pick, object, grasp type), so that executions with the same label share a grasp. Changing the placement in the instruction during picking falls into one of the two cases that CRAFT already handles. If the new placement requires a different grasp type, the currently required skill changes and the pair is supervised by Lchange\mathcal L_{\mathrm{change}}. If the new placement requires the same grasp type, the currently required skill and the action remain the same and the pair is supervised by Lpreserve\mathcal L_{\mathrm{preserve}}. This requires no change to the architecture because the skill representation is computed from the full instruction, which already specifies the placement. The skill labels are used only for training, so no additional input is required at deployment. The definition of a skill changes, while the architecture and objectives of CRAFT remain the same.

Recombination and generality. Under this extension, recombination remains possible among placements that require the same grasp type, and the demonstrations need to cover each (object, grasp type) pair. The grasp type is one example. More generally, a skill can include how an operation is executed, such as the direction from which an object is approached or the pose in which the object is held, whenever a later operation depends on how the current operation is executed. Such information is increasingly available. Recent VLA models are conditioned on metadata about how a task is performed (Physical Intelligence, 2026), and task-appropriate grasps can be predicted from an image and an instruction (Deshpande et al., 2025). We leave the empirical validation of this extension for future work.

In summary, an operation–entity pair is the skill definition for the setting we study, and CRAFT can, in principle, be extended to coupled skills without changing the model architecture or training objectives. We have revised the Limitations (Section 6) to reflect this discussion.

References

  • Physical Intelligence. "π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities." arXiv:2604.15483, 2026.
  • Deshpande et al. "GraspMolmo: Generalizable Task-Oriented Grasping via Large-Scale Synthetic Data Generation." CoRL 2025.
AuthorsOpenReview ↗

Response to Reviewer S8Ff (2/2)

W2 / Q2. Contribution of Lskill\mathcal L_{\mathrm{skill}} beyond the counterfactual objectives

As requested, we trained the configuration Lfm+Lchange+Lpreserve\mathcal L_{\mathrm{fm}} + \mathcal L_{\mathrm{change}} + \mathcal L_{\mathrm{preserve}} without Lskill\mathcal L_{\mathrm{skill}} for both π0.5\pi_{0.5} and GR00T N1.7 on Pick–Place and added this configuration to Table 2. The new row is shown below together with CRAFT.

Success rate (%) on Pick–Place (mean ± std over three evaluation seeds), in the format of Table 2. Skill, Pres., and Change denote Lskill\mathcal L_{\mathrm{skill}}, Lpreserve\mathcal L_{\mathrm{preserve}}, and Lchange\mathcal L_{\mathrm{change}}. The first row is the new configuration and the second is CRAFT.

SkillPres.Changeπ0.5\pi_{0.5} Demo.π0.5\pi_{0.5} Undem.GR00T N1.7 Demo.GR00T N1.7 Undem.
✓✓100.0 ± 0.00.0 ± 0.099.2 ± 0.30.4 ± 0.1
✓✓✓99.0 ± 0.984.2 ± 0.399.0 ± 0.572.1 ± 0.9

With the two counterfactual terms but without Lskill\mathcal L_{\mathrm{skill}}, success on undemonstrated combinations remains low for both models, while success on demonstrated combinations remains high. The gain of CRAFT is therefore obtained only when the counterfactual terms are combined with Lskill\mathcal L_{\mathrm{skill}}. In Table 2, Lskill\mathcal L_{\mathrm{skill}} without the counterfactual terms also yields low success on undemonstrated combinations. Together, these results suggest that the gain is not attributable to Lskill\mathcal L_{\mathrm{skill}} or to the counterfactual terms alone, and that the three auxiliary objectives are complementary.

This result is consistent with the roles of the objectives (Section 4.4). Lskill\mathcal L_{\mathrm{skill}} jointly learns reusable skill representations and the action expert that interprets them, whereas Lcf\mathcal L_{\mathrm{cf}} does not update the action expert. By design, Lcf\mathcal L_{\mathrm{cf}} extends the reusable skill representations formed by Lskill\mathcal L_{\mathrm{skill}} to undemonstrated combinations.

This row adds the requested configuration to the training-objective ablations, which the reviewer found supportive of the proposed mechanism. We have added the result to Table 2 and this discussion to Section 5.4.

We hope our responses and revisions address the reviewer's comments. We will respond promptly to any further questions.

Dear Reviewer S8Ff,

Thank you again for your time and constructive review. As the discussion period is coming to a close, we would like to kindly check whether our responses and the revised paper have addressed your questions. Please let us know if you have any further questions or if anything remains unclear.

Comments on the paper

2 comments

Dear Reviewers,

Please review the authors' rebuttal and the other reviewers' comments, and participate in the discussion before it closes on November 18 (AoE). Kindly indicate whether your concerns have been adequately addressed and raise any remaining questions or concerns.

Thank you for your time and contributions.

Best regards,

Area Chair

Dear Reviewers,

A quick reminder that the discussion period ends on November 18 (AoE). If you haven't already, please review the authors' rebuttal and provide a brief follow-up indicating whether your concerns have been adequately addressed or if further clarification is needed.

Your timely feedback would be greatly appreciated and will help facilitate the final decision.

Thank you for your time and contributions!

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.