- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes a task-based hierarchical distillation method for VLA models. Instead of always learning from fixed teacher layers, the student learns from a weighted combination of teacher layers selected according to the task instruction and action-generation stage. The selection mechanism is removed after training. The eight-layer student achieves 96.72% success on LIBERO, while the six-layer version achieves 2.30× faster inference than the teacher with a small performance drop.
Critical strengths
Strengths: This paper proposes a hierarchical distillation method that transfers knowledge from a teacher model to a more lightweight student model across different tasks.
Critical weaknesses
Weaknesses
- The motivation for hierarchical distillation is not very convincing. The paper argues that the student model should learn from specific teacher layers depending on the task. However, this raises several questions. First, the student is initialized using only a subset of the teacher’s layers, which means its downstream capabilities could differ substantially due to the missing layers. Second, the idea that certain blocks represent particular tasks is not sufficiently justified. Establishing a clear correspondence between individual layers and specific tasks remains a significant challenge in deep learning, and relying on gradient-based evidence alone is insufficient.
- No real-robot demo: The evaluation relies on simulation, with the CALVIN experiment limited to five tasks, each reset independently. Real-robot experiments is missing. It is also recommending to include the the full long-horizon benchmark.
- Comparisons with existing adaptive distillation methods are limited. One could argue that a simple distillation method might achieve similar results under a fair experimental setup. Please specify how a fair experimental setup is ensured.
- The design is not well justified with clear ablation study, and this issue is related to the first weakness.
Questions for authors
-
What evidence shows that specific teacher layers provide better supervision for particular tasks, especially after removing layers changes the student’s representations? Is there supporting evidence beyond gradient compatibility?
-
Can you provide real-robot experiments and evaluate on the full CALVIN long-horizon benchmark?
-
How does the method compare with simpler and existing adaptive distillation methods, and how are fair comparisons ensured?
-
Can you provide further ablations to justify the teacher-layer neighborhood and flow-stage design choices?
11 comments
Response to Reviewer cLXb (2)
[W2 & Q2: Real-Robot Experiments and CALVIN Evaluation Scope]
refers to: "No real-robot demo: The evaluation relies on simulation, with the CALVIN experiment limited to five tasks, each reset independently. Real-robot experiments is missing. It is also recommending to include the the full long-horizon benchmark."
We have added closed-loop experiments on the Cobot Magic platform, an implementation of the ALOHA (A Low-cost Open-source Hardware System for Bimanual Teleoperation) architecture, to assess real-world task completion by dense , Shallow- L8, and TFC-KD L8 under the same evaluation protocol.
Real-robot experimental setup. We select Stack Cups, Wipe Table, and Sort Objects, involving precise object interaction, continuous trajectory execution, and multi-object manipulation, respectively. We collect 50 expert demonstrations per task, totaling 150, and jointly train one policy on all three tasks. Both eight-layer students use the same demonstrations and evaluation rules and are initialized from the same dense teacher trained on these real-robot tasks. Each model undergoes 20 independent trials per task: 60 per model and 180 across all three models. Evaluation randomizes initial object positions, target positions, and relevant layouts. All models share pregenerated scene configurations, run in interleaved order, and use predefined success criteria.
Real-robot results:
| Model | Stack Cups | Wipe Table | Sort Objects | Average Success Rate |
|---|---|---|---|---|
| Dense | 75%(15/20) | 75%(15/20) | 65%(13/20) | 71.67%(43/60) |
| Shallow- L8 | 70%(14/20) | 65%(13/20) | 55%(11/20) | 63.33%(38/60) |
| TFC-KD L8 | 75%(15/20) | 75%(15/20) | 70%(14/20) | 73.33%(44/60) |
TFC-KD outperforms Shallow- at the same depth on all three tasks, increasing average success from 63.33% to 73.33% (+10.00 percentage points), comparable to the dense teacher (71.67%). These results support better preservation of manipulation capabilities on the tested real-robot tasks with a fixed eight-layer deployment architecture and no additional inference-time routing computation, providing further real-world validation of the simulation results.
CALVIN Evaluation Scope and Long-Horizon Evaluation Status. As suggested, we have initiated evaluation on the official CALVIN benchmark of five consecutive language instructions. The full evaluation requires substantial computation and simulation resources and remains incomplete under current resource limits; we therefore report no final metrics at this stage. The original CALVIN-5 experiments use a matched subset of five class-balanced atomic tasks with independent resets. At maximum interaction budgets of 240/160/120 steps, Shallow- L8 achieves 93.47%/86.13%/84.13% success, and TFC-KD L8 achieves 99.87%/97.87%/96.80%. These results provide atomic-task validation in another simulation environment. The ongoing official evaluation will assess consecutive multi-task execution. We will include the real-robot setup and results in the revision and clarify the scope of CALVIN-5 and the status of the official long-horizon evaluation.
Response to Reviewer cLXb (3)
[W3 & Q3: Comparisons with Existing Methods and Experimental Fairness]
refers to: "Comparisons with existing adaptive distillation methods are limited. One could argue that a simple distillation method might achieve similar results under a fair experimental setup. Please specify how a fair experimental setup is ensured."
Thank you for this suggestion. We added the comparisons below. All methods share the frozen dense teacher, L8 student architecture, depth-aligned initialization, and LIBERO 40-task training data. The added SemCKD, C2KD, and TED implementations use 30K student optimizer updates with an effective batch size of 64 and retain the Flow Matching and teacher Velocity KD objectives. All results use training seed 42 and the same evaluation protocol: 50 episodes per task for each of three environment seeds (7, 17, and 27), totaling 6,000 episodes per model.
Distillation mechanisms. Fixed Depth-Aligned All-Layer KD supervises every student layer with the attention map of its fixed, depth-aligned teacher layer, without candidate-layer mixing. Generic Router + Monotonicity uses pooled contextual vision-language features and continuous flow time as inputs to an MLP that predicts mixing weights over candidate teacher attention maps. The mixed maps supervise student attention, using the same temperature schedule and teacher-layer ordering regularizer as TFC-KD. SemCKD uses cross-sample feature relations to learn sample-dependent soft associations between student and teacher layers and applies weighted intermediate-feature matching. C2KD uses task-related feature filters and teacher–student feature similarity to select supervision targets for the last two student layers. TED uses task-aware feature filters followed by fixed depth-aligned feature matching. During student distillation, teacher filters remain frozen, while student filters remain trainable. These implementations preserve distinct intermediate-supervision mechanisms within the shared VLA compression setting.
| Method | Spatial | Object | Goal | Long | Avg. (%) |
|---|---|---|---|---|---|
| Shallow- L8 | 97.47 | 96.53 | 95.53 | 91.07 | 95.15 |
| Fixed Depth-Aligned All-Layer KD | 98.13 | 97.07 | 96.33 | 91.47 | 95.75 |
| Generic Router + Monotonicity | 98.60 | 97.47 | 96.27 | 90.33 | 95.67 |
| SemCKD | 92.40 | 95.13 | 84.53 | 67.00 | 84.77 |
| C2KD | 88.13 | 95.60 | 88.80 | 72.00 | 86.13 |
| TED | 95.07 | 98.13 | 93.60 | 86.33 | 93.28 |
| TFC-KD L8 | 98.73 | 98.00 | 97.27 | 92.93 | 96.73 |
TFC-KD achieves the highest average success rate of 96.73%, exceeding Fixed Depth-Aligned All-Layer KD and Generic Router + Monotonicity by 0.98% and 1.06%, respectively. These comparisons indicate that broader supervision or generic routing with an ordering constraint alone does not explain its advantage. TFC-KD achieves an average success rate of 96.73%, compared with 84.77% for SemCKD, 86.13% for C2KD, and 93.28% for TED. Together, these comparisons support the effectiveness of TFC-KD for flow-based VLA compression. We will include the results, method-specific adaptation details, and training budgets in the revised manuscript.
Response to Reviewer cLXb (1)
Thank you for assessing our motivation, comparison fairness, ablation design, and evaluation scope. We agree that the original evidence does not establish a one-to-one correspondence between individual teacher Transformer layers and specific semantic tasks. Below, additional controlled experiments clarify our claims, with responses organized by W1–W4 and Q1–Q4.
[W1 & Q1: Motivation for Condition-Dependent Teacher-Layer Supervision]
Thank you for raising these questions. We apologize that our wording suggested a fixed correspondence between teacher layers and tasks. Our method does not assume that individual Transformer blocks represent or specialize in particular tasks. Instead, we investigate whether the relative suitability of teacher attention representations for supervising a compressed student varies with the task instruction and flow stage.
Our closest baseline, Shallow-π, uses attention supervision from a fixed intermediate teacher layer. Such a fixed target does not adapt its source to the task or the stage of iterative action generation. To introduce this flexibility, TFC-KD assigns soft weights to attention maps from nearby teacher layers for each retained student layer. These weights depend on the task instruction and flow stage and are learned during distillation. The resulting weighted attention target allows the contributions of candidate teacher layers to vary across conditions without assigning a particular task to a particular layer.
Regarding initialization, the parameters copied from selected teacher layers provide the initial student weights; they do not impose a permanent teacher–student correspondence. We agree that removing layers changes the student representations, and we do not assume that an initialized student layer remains equivalent to its teacher counterpart. During distillation, the student parameters and routing weights are updated, while the frozen teacher continues to provide attention targets. All compared L8 students share the same architecture and initialization from teacher layers
[0, 2, 5, 7, 10, 12, 15, 17], so this starting point is held consistent across methods.Finally, gradient compatibility is an auxiliary post-training diagnostic, not a criterion used to train the router or assign teacher layers to tasks. It measures the cosine similarity between the student gradients induced by attention distillation and the action objective, assessing their local, first-order optimization compatibility. The routing weights are learned through the attention-distillation objective and ordering regularization, without gradient-compatibility labels.