- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether fine-tuned VLAs can recombine demonstrated constituent skills into combinations without corresponding demonstrations. It proposes CRAFT, which separates skill and state representations and uses counterfactual instruction–observation pairs to transfer supervision from demonstrated executions of the currently required skill. The method is evaluated across three VLA backbones on two simulation benchmarks and on a real robot.
Critical strengths
- The problem and method are well motivated. The distinction between having demonstrated all constituent skills and being able to recombine them is meaningful. CRAFT provides a principled way to construct counterfactual supervision without directly transferring actions across incompatible observations.
- The empirical improvements are large and consistent. CRAFT substantially improves undemonstrated combination success across three VLA models and both simulation benchmarks while largely preserving demonstrated task performance.
Critical weaknesses
- The checkpoint selection protocol weakens the central generalization claim. The paper selects checkpoints using success on the undemonstrated combinations and then reuses the same rollouts for final reporting. Since these combinations are exactly the paper’s main generalization target, this introduces selection bias. The cited prior VLA works also report the best-performing checkpoint, but they select checkpoints based on in-distribution downstream task performance, rather than on a held-out compositional generalization set.
- The compositional-generalization setting is relatively structured. The method assumes a fixed operation sequence, known operation–entity skills, and that every constituent skill has already been demonstrated. In addition, the target undemonstrated combinations are instantiated as counterfactual instructions during training. Thus, the paper studies generalization without action demonstrations for those combinations, rather than generalization to completely unseen task specifications.
Questions for authors
-
Can the authors select checkpoints without using performance on the undemonstrated evaluation combinations? For example, using a fixed training step or a separate validation criterion?
-
Can the authors evaluate on target combinations that are not explicitly instantiated as counterfactual instructions during training? This would clarify how much CRAFT generalizes to genuinely unseen combinations versus learning from counterfactual supervision for known target combinations.
8 comments
Response to Reviewer wCrs (2/2)
W2 / Q2. Structure of the setting and combinations excluded from counterfactual instructions
Following the reviewer's suggestion, we have added an evaluation on combinations that appear in neither the demonstrations nor the counterfactual instructions (Section 5.4, Table 3). On these combinations, CRAFT achieves higher success than the compared baselines.
The setting we study. Our formulation assumes a fixed operation sequence, such as picking followed by placing, and that every constituent skill appears in the demonstrations (Section 3). These conditions isolate the distinction, noted by the reviewer among the strengths, between having demonstrated all constituent skills and being able to recombine them. In the demonstrations, a later skill is executed only from the states produced by the preceding skill of a demonstrated combination. For example, placing on the blue plate is always demonstrated with the blue cube in the gripper. An undemonstrated combination therefore requires a skill to be executed from a state in which that skill was never demonstrated, which is the challenge described in Section 3. Under these conditions, standard fine-tuning achieves substantially lower success on undemonstrated combinations than on demonstrated ones (Table 1).
Counterfactual instructions. We would like to clarify that instantiating a combination as a counterfactual instruction requires no additional demonstrations, as doing so only changes the instruction paired with an observation from an existing demonstration. The resulting pair has no demonstrated action target, and providing supervision for such pairs is the problem CRAFT addresses.
Combinations excluded from counterfactual instructions (Q2). Using on Pick–Place, we exclude four of the 12 undemonstrated combinations from the counterfactual instructions and select the checkpoint by success on the remaining eight. The four excluded combinations are therefore used neither for training nor for checkpoint selection, and their instructions never appear during training. Table 3 reports the success rates (%) on these four excluded combinations (mean ± std over three evaluation seeds).
| Method | Success (%) |
|---|---|
| FT (Full) | 5.0 ± 0.5 |
| FT (LoRA) | 43.7 ± 2.0 |
| FT (Frozen) | 15.0 ± 2.2 |
| CAG-VA | 62.0 ± 1.3 |
| CRAFT (Excluded) | 69.5 ± 2.3 |
| CRAFT | 79.8 ± 1.0 |
On the excluded combinations, CRAFT (Excluded) achieves higher success than CAG-VA and standard fine-tuning, and lower success than CRAFT trained with these combinations included. These results suggest that the skill representations learned by CRAFT generalize to combinations beyond those instantiated, and that instantiating a combination further contributes to success on that combination.
We have added this result in Section 5.4 and the corresponding details in Appendix B.2.
We hope our responses and revisions address the reviewer's comments. We will respond promptly to any further questions.
Reply to authors' rebuttal
Thanks for authors' rebuttal. While I am more confident that CRAFT is effective for the problem they studied, I remain concerned that the problem formulation is overly constrained, assuming a fixed scene and operation sequence, and demonstrations covering all constituent skills. I think this narrow scope limits its insights for the broader VLA community, therefore I want to maintain my current rating.
Response to Reviewer wCrs (1/2)
We thank the reviewer for the careful assessment and the concrete suggestions. We respond to each point below, and the corresponding revisions are marked in blue in the revised paper.
W1 / Q1. Checkpoint selection without undemonstrated combinations
The improvement of CRAFT over the baselines remains when checkpoints are selected without using performance on undemonstrated combinations. We have added the supporting results in Appendix D.6.
Fixed training step. For π0.5 and GR00T N1.7 on both benchmarks, CRAFT achieves the highest success on undemonstrated combinations at every evaluated checkpoint (Figure D.7). The improvement therefore holds at each of the evaluated training steps.
Selection by success on demonstrated combinations. The same holds when checkpoints are selected by in-distribution performance, i.e., success on demonstrated combinations, as in the prior work the reviewer refers to. In Figure D.7, the success of CRAFT at any evaluated checkpoint is above that of every standard fine-tuning baseline at any evaluated checkpoint, and also above the results of CAG-VA and Skill Composition in Table 1, which are reported at their best checkpoints. The improvement therefore holds whichever checkpoint this criterion selects for each method.
Held-out selection. We also evaluate selection and reporting on disjoint combinations. For every split of the 12 undemonstrated Pick–Place combinations into six for selection and six for reporting, we select the checkpoint on the selection combinations and report the selected checkpoint's success on the remaining six (Table D.2). Under this selection, CRAFT achieves 82.8% and 68.4% for π0.5 and GR00T N1.7, respectively, compared with 45.1% and 2.9% for the best standard fine-tuning baseline.