- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Segmentation-Guided Semantic Distillation (SGSD) for robot manipulation. A task-specific Attention U-Net is trained on foreground masks, and its intermediate features supervise the final block of a policy’s visual encoder through feature, direction, and magnitude alignment losses. The distillation objective is optimized jointly with the action loss, and the teacher is discarded at inference. Experiments cover five RoboTwin tasks and three real-world manipulation settings, together with representation diagnostics and adaptation ablations.
Critical strengths
-
The approach has practical appeal and reported empirical benefits. SGSD introduces no additional inference-time components. The paper reports improvements over frozen and action-only adaptation, including real-world results across several policy architectures.
-
The paper includes useful controls and informative negative results. The matched action-only baseline, teacher correspondence controls, supervision-placement experiments, and teacher-transfer analysis help characterize the approach. I appreciate the appendix’s acknowledgment that better spatial diagnostics do not consistently imply better manipulation performance.
Critical weaknesses
- The methodological novelty is limited, and the most important alternative explanation is not sufficiently tested.
The approach combines a segmentation-trained teacher, intermediate feature matching, restricted backbone adaptation, and standard alignment objectives. As the related-work section acknowledges, feature distillation for robot learning and auxiliary supervision during policy training are established ideas. The distinguishing choices appear to be the task-specific teacher and direct supervision of the final visual block. Moreover, Appendix B shows that the direction and magnitude losses reweight components already present in feature MSE.
These choices may form a useful adaptation recipe, but the current experiments do not convincingly establish their necessity. In particular, the paper does not compare against directly training an auxiliary segmentation head using the same masks and visual update set. The frozen and action-only baselines do not receive this additional spatial supervision, while the shuffled-teacher controls deliberately remove useful correspondence. These comparisons establish that corresponding supervision helps, but do not establish why the proposed teacher-feature distillation is preferable to a simpler use of the same annotations.
A matched auxiliary-segmentation baseline and a more direct comparison with related distillation approaches are important for assessing the contribution beyond combining existing components.
- The proposed artifact-correction explanation is stronger than the supporting evidence.
The paper motivates SGSD through spatial artifacts in pretrained ViTs and links their correction to improved manipulation. However, several appendix results complicate this interpretation. In Table 16, intermediate-block distillation achieves higher foreground probe IoU than SGSD but substantially lower task success. Appendix E.2 also reports that the transferred teacher has more foreground-concentrated feature norms while producing a weaker policy. The inference interventions in Table 15 do not provide clear evidence that manipulating high-norm responses changes task performance.
These findings do not invalidate the reported improvements, but they leave open whether SGSD primarily corrects the proposed artifacts or supplies useful task-specific supervision through another mechanism. The correspondence controls support the importance of aligned targets; they do not resolve this distinction. Consequently, the paper’s mechanistic claims should be narrowed, or supported by more discriminating evidence.
- The evidence for a broadly applicable VLA adaptation method remains limited.
Most detailed methodological and diagnostic studies use ACT with DINOv2, with additional experiments on Diffusion Policy. Using these policies is reasonable, but improvements in this setting do not by themselves establish a general solution to spatial representation problems in VLAs. Although the real-world results include OpenVLA-OFT, π0, and TinyVLA, the paper provides limited model-specific detail on feature matching, trainable parameter sets, and adaptation settings for these architectures.
The reliance on task-specific teachers also limits the demonstrated scope. In Table 12, a transferred teacher improves dual-bottle success from 43% to 46%, compared with 63% for the task-matched teacher. The randomized-domain evaluation reports only 0 versus 4 successes. These results support usefulness within the evaluated task distributions, but provide limited evidence of transferable spatial correction. Together with the additional annotation and training requirements, they make the broader applicability claims less convincing.
Questions for authors
-
What is the performance of a direct auxiliary-segmentation baseline using the same foreground masks, trainable visual parameters, and a comparable training budget? What evidence establishes the benefit of teacher-feature distillation over this simpler alternative?
-
How do the authors reconcile the artifact-correction explanation with configurations that have stronger spatial diagnostics but weaker task performance? Which results specifically distinguish artifact correction from the general benefit of additional task-specific supervision?
-
Could the authors provide the exact adaptation and feature-alignment configurations for OpenVLA-OFT, π0, and TinyVLA, and clarify the scope of transfer supported by the task-specific teacher and domain-randomization results?
2 comments
Thank you for the additional experiments and clarifications. The same-mask auxiliary-segmentation controls meaningfully address an important concern, and I appreciate the explicit acknowledgment that artifact correction has not been isolated as the mechanism behind the gains.
However, my concern about the central contribution remains. The new controls establish an advantage over the tested segmentation heads, while the claimed benefit of projection-free distillation still lacks a matched closed-loop comparison against learned-projection distillation. This is particularly important given the response’s emphasis on projection-free supervision as the distinguishing contribution.
The narrower mechanistic claims are appropriate, but they leave the contribution primarily as an incremental adaptation recipe. The VLA feature-width clarification is helpful, although the exact supervised features and trainable parameter sets still require clearer documentation.
The rebuttal strengthens the empirical evidence, but the remaining gaps in novelty and validation of the central design choices keep me below the acceptance threshold. I therefore maintain the score.
We thank all reviewers for their careful reading and constructive feedback. We have added the requested controls, clarified the VLA feature interfaces, and narrowed the mechanistic and novelty claims.
We thank the reviewer for the detailed report and for recognizing the useful controls and negative results. Our novelty is not the combination of distillation, segmentation supervision, and alignment losses per se, but the finding that a learnable projection can absorb correction while the backbone remains defective (Fig.~2), motivating a projection-free, final-block, task-trained dense alignment design for VLA adaptation. We agree that the direction and magnitude terms reweight components of feature MSE and do not present them as independent new losses. Revisions appear in the Intro, Related Work, and Limitations; details are in the Novelty Clarification section of the Appendix.
W1: Limited methodological novelty and insufficient testing of the most important alternative. Q1: Matched auxiliary-segmentation baseline. Response: We added a direct mask-prediction control using the same foreground annotations, trainable visual parameters, and a comparable policy-training budget. Five-task mean success was 37.4%, 33.8%, and 32.6% across three auxiliary weights, compared with 60.0% for SGSD (Table A2); matched action-only tuning reached 39.2% (submitted Table 1). The best auxiliary setting also fell below SGSD on real-world ACT, with 29.8 versus 44.7 mean successes per stage (Table A3). We also agree with your reading of Appendix B: the direction and magnitude terms reweight components of squared feature error (submitted Table 5: MSE alone 47/100, +direction 56, +magnitude 58, both 70). We now describe these as objective weighting terms rather than independent new losses.
Table A2: Same-mask comparison on the five RoboTwin tasks.
Table A3: Real-world manipulation success counts.
W2: The artifact-correction explanation is stronger than the evidence. Q2: Reconciling artifact-correction explanation with diagnostics. Response: We agree and have narrowed the mechanistic claim. Foreground readability is a probe property, not a behavioral property. In submitted Table 16, intermediate-block distillation achieved a higher post-normalization probe IoU (0.8281 versus 0.8128) yet succeeded in only 23/100 trials versus 70/100, though that comparison also changes the trainable block. The transferred teacher and high-norm interventions similarly do not establish a clear behavioral effect. The comparisons that support our revised claim are the correspondence controls (70/100 aligned, 46/100 shuffled, 38/100 wrong image) and Tables A2/A3, which distinguish dense feature transfer from direct mask supervision. We explicitly state that artifact correction is not isolated.
W3: Limited evidence for broad VLA applicability. Q3: Exact VLA configurations and scope of transfer. Response: Table A6 specifies the feature interfaces. The teacher's pre-classifier decoder is constructed at the student width (2176, 1152, and 1024), pooled to 16×16, and reshaped to B×256×C. We clarified trainable parameter sets in Section 3.2 and the Appendix; a module-by-module inventory is added as a limitation. On scope, the task-matched teacher outperforms a transferred teacher on dual-bottle (63 vs 46 successes, with 43 for frozen), and randomized-domain is 4 vs 0 successes on 84 shared scenes (p=0.125). These results limit the benefit to evaluated task distributions.
Table A6: Confirmed VLA tensor dimensions.
The method requires foreground annotations. Appendix A reports ~4-5 GPU-hours/task for teacher training. We have added a limitations paragraph stating these points explicitly.