acceptodds
Withdrawn from ICLR 2027

Segmentation-Guided Semantic Distillation For Robot Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained Vision Transformer (ViT) features may contain high-response background tokens and exhibit spatial-semantic misalignment, which can weaken the spatial perception of Vision-Language-Action (VLA) models and thereby impede spatially precise robotic control. Action-supervised adaptation does not reliably remove these artifacts. Therefore, we propose Segmentation-Guided Semantic Distillation (\method), which transfers dense features from a CNN segmentation teacher trained on robotic manipulation data directly into the VLA visual backbone. During policy fine-tuning, three auxiliary losses, namely spatially corresponding feature matching, directional alignment, and magnitude alignment, are jointly optimized with the action prediction loss. On five RoboTwin tasks and three real-world robot-arm tasks, improves success rates over frozen-backbone baselines by 17.5 and 18.7 percentage points, respectively. These results show that task-matched spatial distillation improves both foreground-aligned representations and manipulation success. Code and models are available at: https://anonymous.4open.science/r/SGSD-Segmentation-Guided-Semantic-Distillation-0760.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision Transformers, Vision-Language-Action, Visual Artifacts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_44gkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Segmentation-Guided Semantic Distillation (SGSD) for robot manipulation. A task-specific Attention U-Net is trained on foreground masks, and its intermediate features supervise the final block of a policy’s visual encoder through feature, direction, and magnitude alignment losses. The distillation objective is optimized jointly with the action loss, and the teacher is discarded at inference. Experiments cover five RoboTwin tasks and three real-world manipulation settings, together with representation diagnostics and adaptation ablations.

Critical strengths
  1. The approach has practical appeal and reported empirical benefits. SGSD introduces no additional inference-time components. The paper reports improvements over frozen and action-only adaptation, including real-world results across several policy architectures.

  2. The paper includes useful controls and informative negative results. The matched action-only baseline, teacher correspondence controls, supervision-placement experiments, and teacher-transfer analysis help characterize the approach. I appreciate the appendix’s acknowledgment that better spatial diagnostics do not consistently imply better manipulation performance.

Critical weaknesses
  1. The methodological novelty is limited, and the most important alternative explanation is not sufficiently tested.

The approach combines a segmentation-trained teacher, intermediate feature matching, restricted backbone adaptation, and standard alignment objectives. As the related-work section acknowledges, feature distillation for robot learning and auxiliary supervision during policy training are established ideas. The distinguishing choices appear to be the task-specific teacher and direct supervision of the final visual block. Moreover, Appendix B shows that the direction and magnitude losses reweight components already present in feature MSE.

These choices may form a useful adaptation recipe, but the current experiments do not convincingly establish their necessity. In particular, the paper does not compare against directly training an auxiliary segmentation head using the same masks and visual update set. The frozen and action-only baselines do not receive this additional spatial supervision, while the shuffled-teacher controls deliberately remove useful correspondence. These comparisons establish that corresponding supervision helps, but do not establish why the proposed teacher-feature distillation is preferable to a simpler use of the same annotations.

A matched auxiliary-segmentation baseline and a more direct comparison with related distillation approaches are important for assessing the contribution beyond combining existing components.

  1. The proposed artifact-correction explanation is stronger than the supporting evidence.

The paper motivates SGSD through spatial artifacts in pretrained ViTs and links their correction to improved manipulation. However, several appendix results complicate this interpretation. In Table 16, intermediate-block distillation achieves higher foreground probe IoU than SGSD but substantially lower task success. Appendix E.2 also reports that the transferred teacher has more foreground-concentrated feature norms while producing a weaker policy. The inference interventions in Table 15 do not provide clear evidence that manipulating high-norm responses changes task performance.

These findings do not invalidate the reported improvements, but they leave open whether SGSD primarily corrects the proposed artifacts or supplies useful task-specific supervision through another mechanism. The correspondence controls support the importance of aligned targets; they do not resolve this distinction. Consequently, the paper’s mechanistic claims should be narrowed, or supported by more discriminating evidence.

  1. The evidence for a broadly applicable VLA adaptation method remains limited.

Most detailed methodological and diagnostic studies use ACT with DINOv2, with additional experiments on Diffusion Policy. Using these policies is reasonable, but improvements in this setting do not by themselves establish a general solution to spatial representation problems in VLAs. Although the real-world results include OpenVLA-OFT, π0, and TinyVLA, the paper provides limited model-specific detail on feature matching, trainable parameter sets, and adaptation settings for these architectures.

The reliance on task-specific teachers also limits the demonstrated scope. In Table 12, a transferred teacher improves dual-bottle success from 43% to 46%, compared with 63% for the task-matched teacher. The randomized-domain evaluation reports only 0 versus 4 successes. These results support usefulness within the evaluated task distributions, but provide limited evidence of transferable spatial correction. Together with the additional annotation and training requirements, they make the broader applicability claims less convincing.

Questions for authors
  1. What is the performance of a direct auxiliary-segmentation baseline using the same foreground masks, trainable visual parameters, and a comparable training budget? What evidence establishes the benefit of teacher-feature distillation over this simpler alternative?

  2. How do the authors reconcile the artifact-correction explanation with configurations that have stronger spatial diagnostics but weaker task performance? Which results specifically distinguish artifact correction from the general benefit of additional task-specific supervision?

  3. Could the authors provide the exact adaptation and feature-alignment configurations for OpenVLA-OFT, π0, and TinyVLA, and clarify the scope of transfer supported by the task-specific teacher and domain-randomization results?

2 comments
AuthorsOpenReview ↗

We thank all reviewers for their careful reading and constructive feedback. We have added the requested controls, clarified the VLA feature interfaces, and narrowed the mechanistic and novelty claims.

We thank the reviewer for the detailed report and for recognizing the useful controls and negative results. Our novelty is not the combination of distillation, segmentation supervision, and alignment losses per se, but the finding that a learnable projection can absorb correction while the backbone remains defective (Fig.~2), motivating a projection-free, final-block, task-trained dense alignment design for VLA adaptation. We agree that the direction and magnitude terms reweight components of feature MSE and do not present them as independent new losses. Revisions appear in the Intro, Related Work, and Limitations; details are in the Novelty Clarification section of the Appendix.

W1: Limited methodological novelty and insufficient testing of the most important alternative. Q1: Matched auxiliary-segmentation baseline. Response: We added a direct mask-prediction control using the same foreground annotations, trainable visual parameters, and a comparable policy-training budget. Five-task mean success was 37.4%37.4\%, 33.8%33.8\%, and 32.6%32.6\% across three auxiliary weights, compared with 60.0%60.0\% for SGSD (Table A2); matched action-only tuning reached 39.2%39.2\% (submitted Table 1). The best auxiliary setting also fell below SGSD on real-world ACT, with 29.8 versus 44.7 mean successes per stage (Table A3). We also agree with your reading of Appendix B: the direction and magnitude terms reweight components of squared feature error (submitted Table 5: MSE alone 47/100, +direction 56, +magnitude 58, both 70). We now describe these as objective weighting terms rather than independent new losses.

Table A2: Same-mask comparison on the five RoboTwin tasks.

MethodStandBottlesLaptopCardHammerMean (%)
Frozen backbone104369385442.8
SGSD346383507060.0
Auxiliary Head λseg=0.1\lambda_{seg}=0.183960334737.4
Auxiliary Head λseg=1\lambda_{seg}=173757323633.8
Auxiliary Head λseg=5\lambda_{seg}=563656313432.6

Table A3: Real-world manipulation success counts.

MethodLarge-object (Pick/Place)Small-object (Pick/Place)Cluttered small-object (Pick/Place)Avg.
ACT40 / 3837 / 3326 / 2232.7
ACT+SGSD49 / 4846 / 4243 / 4044.7
Aux. Head λseg=0.1\lambda_{seg}=0.137 / 3533 / 3223 / 1929.8
Aux. Head λseg=1\lambda_{seg}=135 / 3332 / 3120 / 1626.7
Aux. Head λseg=5\lambda_{seg}=533 / 3331 / 3020 / 1524.0

W2: The artifact-correction explanation is stronger than the evidence. Q2: Reconciling artifact-correction explanation with diagnostics. Response: We agree and have narrowed the mechanistic claim. Foreground readability is a probe property, not a behavioral property. In submitted Table 16, intermediate-block distillation achieved a higher post-normalization probe IoU (0.8281 versus 0.8128) yet succeeded in only 23/100 trials versus 70/100, though that comparison also changes the trainable block. The transferred teacher and high-norm interventions similarly do not establish a clear behavioral effect. The comparisons that support our revised claim are the correspondence controls (70/100 aligned, 46/100 shuffled, 38/100 wrong image) and Tables A2/A3, which distinguish dense feature transfer from direct mask supervision. We explicitly state that artifact correction is not isolated.

W3: Limited evidence for broad VLA applicability. Q3: Exact VLA configurations and scope of transfer. Response: Table A6 specifies the feature interfaces. The teacher's pre-classifier decoder is constructed at the student width (2176, 1152, and 1024), pooled to 16×1616 \times 16, and reshaped to B×256×CB \times 256 \times C. We clarified trainable parameter sets in Section 3.2 and the Appendix; a module-by-module inventory is added as a limitation. On scope, the task-matched teacher outperforms a transferred teacher on dual-bottle (63 vs 46 successes, with 43 for frozen), and randomized-domain is 4 vs 0 successes on 84 shared scenes (p=0.125p=0.125). These results limit the benefit to evaluated task distributions.

Table A6: Confirmed VLA tensor dimensions.

PolicyStudent feature dimensionTeacher: dense -> pooled -> tokens
OpenVLA-OFTB×256×2176B\times 256\times 2176B×2176×224×224B\times 2176\times 224\times 224 -> B×2176×16×16B\times 2176\times 16\times 16 -> B×256×2176B\times 256\times 2176
π0\pi_0B×256×1152B\times 256\times 1152B×1152×224×224B\times 1152\times 224\times 224 -> B×1152×16×16B\times 1152\times 16\times 16 -> B×256×1152B\times 256\times 1152
TinyVLAB×256×1024B\times 256\times 1024B×1024×224×224B\times 1024\times 224\times 224 -> B×1024×16×16B\times 1024\times 16\times 16 -> B×256×1024B\times 256\times 1024

The method requires foreground annotations. Appendix A reports ~4-5 GPU-hours/task for teacher training. We have added a limitations paragraph stating these points explicitly.

Thank you for the additional experiments and clarifications. The same-mask auxiliary-segmentation controls meaningfully address an important concern, and I appreciate the explicit acknowledgment that artifact correction has not been isolated as the mechanism behind the gains.

However, my concern about the central contribution remains. The new controls establish an advantage over the tested segmentation heads, while the claimed benefit of projection-free distillation still lacks a matched closed-loop comparison against learned-projection distillation. This is particularly important given the response’s emphasis on projection-free supervision as the distinguishing contribution.

The narrower mechanistic claims are appropriate, but they leave the contribution primarily as an incremental adaptation recipe. The VLA feature-width clarification is helpful, although the exact supervised features and trainable parameter sets still require clearer documentation.

The rebuttal strengthens the empirical evidence, but the remaining gaps in novelty and validation of the central design choices keep me below the acceptance threshold. I therefore maintain the score.

Reviewer_ajnYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Segmentation-Guided Semantic Distillation (SGSD) to improve spatial representations in VLA models. The method uses a CNN segmentation teacher to provide dense feature supervision for the visual backbone during policy training. It combines feature, direction, and magnitude alignment losses with the action prediction loss. The authors evaluate SGSD on five RoboTwin tasks and three real-world manipulation tasks, showing improved success rates across different visual backbones and robot policies. Several ablation studies are also conducted to examine the effects of the distillation losses and teacher features.

Critical strengths

The paper provides a fairly broad experimental evaluation, covering multiple visual backbones, robot policies, and both simulation and real-world tasks. The results show consistent improvements in manipulation success, and the ablation studies help demonstrate the importance of spatially aligned teacher features.

Critical weaknesses

SGSD shows good results, but its novelty over existing segmentation and feature distillation methods seems limited. The paper also lacks sufficient implementation details on how direct feature matching is applied to VLA models with different visual backbones and feature dimensions, making the cross-model results difficult to reproduce.

Questions for authors
  1. What is the main advantage of directly distilling CNN features into the visual backbone compared with existing segmentation-based approaches? Could the authors clarify what makes this design different from prior work?
  2. Section 3.2 states that all student models use 768-dimensional features, but OpenVLA-OFT and π0 have different visual feature dimensions. Could the authors clarify which features are used for distillation and how they are aligned with the teacher features without a learnable projection?
1 comment
AuthorsOpenReview ↗

We thank all reviewers for their careful reading and constructive feedback. We have added the requested controls, clarified the VLA feature interfaces, and narrowed the mechanistic and novelty claims.

We thank the reviewer for the constructive summary and for recognizing the breadth of the experimental evaluation, and we address the critical weakness and the two questions below. We do not claim novelty in distillation or segmentation supervision per se; our contribution is the finding that a learnable projection PϕP_\phi can absorb correction while the backbone remains defective (Fig.~2), motivating a projection-free, final-block, task-trained dense alignment design for VLA adaptation. The auxiliary-head, Transformer-teacher, correspondence, and placement controls isolate this design. These revisions appear in the Intro, Related Work, and Limitations; for details, please see the Novelty Clarification section in the Appendix.

Critical Weakness: limited novelty and insufficient implementation details. Response: We agree that the novelty claim should be more precise. SGSD combines established ingredients, but the design choice we evaluate is the use of a task-trained teacher to provide dense, spatially aligned decoder features directly at the student's final visual block during policy adaptation. We have revised the related-work discussion to position SGSD within dense representation alignment and narrowed the contribution claim accordingly.

Q1: Advantage over existing segmentation-based approaches. Response: The distinction is the form and placement of the supervision. A segmentation head learns a low-dimensional binary decision at each location. SGSD instead matches the teacher's multichannel decoder representation directly at the student's final visual block, providing a dense target in the feature space consumed by the action decoder.

Table A2: Same-mask comparison on the five RoboTwin tasks.

MethodStandBottlesLaptopCardHammerMean (%)
Frozen backbone104369385442.8
SGSD346383507060.0
Auxiliary Head λseg=0.1\lambda_{seg}=0.183960334737.4
Auxiliary Head λseg=1\lambda_{seg}=173757323633.8
Auxiliary Head λseg=5\lambda_{seg}=563656313432.6

Table A3: Real-world manipulation success counts.

MethodLarge-object (Pick/Place)Small-object (Pick/Place)Cluttered small-object (Pick/Place)Avg.
ACT40 / 3837 / 3326 / 2232.7
ACT+SGSD49 / 4846 / 4243 / 4044.7
Aux. Head λseg=0.1\lambda_{seg}=0.137 / 3533 / 3223 / 1929.8
Aux. Head λseg=1\lambda_{seg}=135 / 3332 / 3120 / 1626.7
Aux. Head λseg=5\lambda_{seg}=533 / 3331 / 3020 / 1524.0

As shown, the best auxiliary-head setting gave 37.4%37.4\% versus 60.0%60.0\% for SGSD (Table A2), and 29.8 versus 44.7 on real-world ACT (Table A3). Within the dense-feature approach, feature MSE alone achieved 47/100 on hammer and the complete objective achieved 70/100 (submitted Table 5).

Q2: Feature dimensions in OpenVLA-OFT and π0\pi_0. Response: The 768-dimensional example in Section 3.2 does not describe OpenVLA-OFT or π0\pi_0. OpenVLA-OFT is supervised at B×256×2176B \times 256 \times 2176, and π0\pi_0 at B×256×1152B \times 256 \times 1152. We configure the final decoder of each segmentation teacher to emit the corresponding channel width. We adaptively average-pool the teacher feature map from 224×224224 \times 224 to 16×1616 \times 16, flatten the spatial grid, and transpose to token-major order. The resulting tensor matches the student's visual sequence without a learnable distillation projection, as detailed in Table A6.

Table A6: Confirmed VLA tensor dimensions.

PolicyStudent feature dimensionTeacher: dense -> pooled -> tokens
OpenVLA-OFTB×256×2176B\times 256\times 2176B×2176×224×224B\times 2176\times 224\times 224 -> B×2176×16×16B\times 2176\times 16\times 16 -> B×256×2176B\times 256\times 2176
π0\pi_0B×256×1152B\times 256\times 1152B×1152×224×224B\times 1152\times 224\times 224 -> B×1152×16×16B\times 1152\times 16\times 16 -> B×256×1152B\times 256\times 1152
TinyVLAB×256×1024B\times 256\times 1024B×1024×224×224B\times 1024\times 224\times 224 -> B×1024×16×16B\times 1024\times 16\times 16 -> B×256×1024B\times 256\times 1024

The revised text is in Section 2, Section 3.2, Section 4.4, and Table A6. For full implementation details, please refer to the Appendix.

Reviewer_jRA2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that high-response background patches and spatial-semantic misalignment in the pretrained ViT encoder can cause the precision reduction of the visual features used by VLA models for robot control. They propose Segmentation-Guided Semantic Distillation (SGSD) with an Attention U-Net teacher trained for foreground segmentation on robot demonstration images and masks. To match the frozen teacher features, the last last visual Transformer block’s features are bilinearly upsampled. The authors apply a combination of feature MSE, cosine alignment and magnitude alignment losses along with the policy loss. The method is evaluated across five simulated RoboTwin tasks and three real-world manipulation settings, with several vision backbones and action-policy architectures. Results show improvements over frozen backbone and action-only tuning.

Critical strengths
  • Consistent improvement across five simulated RoboTwin tasks and three real-world tasks, all considered backbones and robot policies.

  • Careful evaluation, including loss ablations, paired-scene significance tests, shuffled position and wrong image controls, showing that image-aligned targets matter.

Critical weaknesses

Insufficient isolation of contribution of the proposed distillation. While the experimental results are convincing, it remains unclear whether the improvement archived specifically from the proposed CNN-to-ViT feature distillation or more generally from additional segmentation supervision. The authors use a segmentation-trained teacher but do not consider a simpler alternative: an auxiliary segmentation head on top of ViT encoder, trained on the same data. Without this, the contribution of distillation is unclear. Additionally, the authors motivate their choice of a CNN teacher through its locality but do not confirm empirically that the CNN-based teacher architecture is essential. A comparison with a Transformer-based teacher would help clarify this point.

Questions for authors
  1. Could the authors compare SGSD against training the visual backbone with an auxiliary segmentation head using the same foreground masks?

  2. How important is the CNN architecture of the teacher? Would a Transformer-based segmentation teacher trained on the same data give similar improvements?

  3. How is SGSD integrated into the different VLA architectures? Section 3.2 states that the student and teacher feature dimensions match, allowing direct feature distillation without an additional learned projection. However, OpenVLA-OFT, π0 and Tiny-VLA use wider visual encoders. Could the authors clarify how feature matching is achieved?

1 comment
AuthorsOpenReview ↗

We thank all reviewers for their careful reading and constructive feedback. We have added the requested controls, clarified the VLA feature interfaces, and narrowed the mechanistic and novelty claims.

We thank the reviewer for recognizing the empirical breadth and controls. We agree isolation is essential: our novelty is not distillation or segmentation supervision per se, but the finding that a learnable projection PϕP_\phi can absorb correction while the backbone remains defective (Fig.~2), motivating our projection-free, final-block, task-trained dense alignment design; the added auxiliary-head, Transformer-teacher, and correspondence controls isolate this design. We have revised the Intro, Related Work, and Limitations accordingly, and address the critical weakness and three questions below. Details are in the Novelty Clarification section of the Appendix.

Critical Weakness: insufficient isolation of the contribution of distillation. Q1: Comparison with an auxiliary segmentation head. Response: We agree these are the key controls. We added an auxiliary segmentation head on the final DINOv2 patch grid, using the same masks and trainable visual parameters as SGSD; the head is removed at inference, and the policy reads the vision backbone features. At auxiliary weights 0.1/1/5, mean simulation success was 37.4%/33.8%/32.6%, vs.\ 60.0% for SGSD and 42.8% for the frozen backbone; on the real robot, the best auxiliary setting averaged 29.8 successes per stage, vs.\ 44.7 for ACT+SGSD and 32.7 for ACT. Thus, low-dimensional binary mask supervision does not reproduce the gains of dense teacher-feature matching; see the appendix for more details.

Table A2: Same-mask comparison on the five RoboTwin tasks.

MethodStandBottlesLaptopCardHammerMean (%)
Frozen backbone104369385442.8
SGSD346383507060.0
Auxiliary Head λseg=0.1\lambda_{seg}=0.183960334737.4
Auxiliary Head λseg=1\lambda_{seg}=173757323633.8
Auxiliary Head λseg=5\lambda_{seg}=563656313432.6

Table A3: Real-world manipulation success counts.

MethodLarge-object (Pick/Place)Small-object (Pick/Place)Cluttered small-object (Pick/Place)Avg.
ACT40 / 3837 / 3326 / 2232.7
ACT+SGSD49 / 4846 / 4243 / 4044.7
Aux. Head λseg=0.1\lambda_{seg}=0.137 / 3533 / 3223 / 1929.8
Aux. Head λseg=1\lambda_{seg}=135 / 3332 / 3120 / 1626.7
Aux. Head λseg=5\lambda_{seg}=533 / 3331 / 3020 / 1524.0

Q2: Importance of the CNN teacher architecture. Response: We have added SegFormer and MaskFormer teachers trained on the same demonstrations and foreground masks, frozen during policy training, and used with the same distillation losses and student update set. Both Transformer teachers improved over the frozen baseline, but were slightly below the Attention U-Net: 55.8%55.8\% (SegFormer) and 55.4%55.4\% (MaskFormer) versus 60.0%60.0\% (U-Net) in simulation (Table A4), with a similar ordering on the real robot (Table A5). We conclude that a CNN is not required, but teacher architecture affects the quality of the dense spatial target.

Table A4: Teacher-architecture comparison on five RoboTwin tasks.

MethodObject standDual bottlesOpen laptopMove cardHammerAvg.
Frozen baseline104369385442.8
CNN teacher (SGSD, U-Net)346383507060.0
SegFormer teacher315978476455.8
MaskFormer teacher326077466255.4

Table A5: Real-world teacher-architecture comparison.

PolicyCNN teacher (SGSD)SegFormer teacherMaskFormer teacher
ACT44.741.040.3
Diffusion Policy48.546.546.2

Q3: Feature matching in wider VLA encoders. Response: The 768-channel description applies only to the reference visual backbones. For OpenVLA-OFT, π0\pi_0, and TinyVLA, the student widths are 2176, 1152, and 1024. We set the teacher's pre-classifier decoder width to the corresponding student width CC. The teacher produces a B×C×224×224B\times C\times 224\times 224 feature map, which we average-pool to 16×1616\times 16 and reshape into B×256×CB\times 256\times C tokens for matching. No learned student projection is required.

Table A6: Confirmed VLA tensor dimensions.

PolicyStudent feature dimensionTeacher: dense -> pooled -> tokens
OpenVLA-OFTB×256×2176B\times 256\times 2176B×2176×224×224B\times 2176\times 224\times 224 -> B×2176×16×16B\times 2176\times 16\times 16 -> B×256×2176B\times 256\times 2176
π0\pi_0B×256×1152B\times 256\times 1152B×1152×224×224B\times 1152\times 224\times 224 -> B×1152×16×16B\times 1152\times 16\times 16 -> B×256×1152B\times 256\times 1152
TinyVLAB×256×1024B\times 256\times 1024B×1024×224×224B\times 1024\times 224\times 224 -> B×1024×16×16B\times 1024\times 16\times 16 -> B×256×1024B\times 256\times 1024
These results are now reported in Section 4.4 and Supplementary Tables A2-A6. Detailed descriptions are provided in the Appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.