acceptodds
Under review as a conference paper at ICLR 2027

SemAE: Semantic-Aligned Action Expert for Language-Following Robot Control

Abstract

Action experts are crucial components for modeling robot actions, especially for transferring semantic knowledge from vision-language model (VLM) to high-dimensional, complex actions. However, recent studies have revealed that existing action experts can degrade the semantic knowledge contained in pretrained VLMs, leading to VLAs that, despite excelling at complex and dexterous manipulation, often fail to follow language instructions accurately. To address this challenge, we propose SemAE, an encoder-decoder action expert that enables mutual transformation between robot actions and semantic embeddings, allowing VLAs to seamlessly inherit the knowledge in VLMs. Specifically, we introduce motion primitive distillation on the action encoder, which enables it to compress robot actions into a compact set of latent embeddings that are well-aligned with the semantic space of the pretrained VLM. Moreover, we propose a patch-wise action flow decoder, which efficiently decompresses a single semantic embedding into future action sequence. Built on SemAE, SemVLA achieves strong language-following with fast inference via parallel decoding of semantic action embeddings. Extensive experiments across multiple standard benchmarks and real robots demonstrate that SemAE effectively aligns robot actions with semantic knowledge, substantially enhancing the language-following capabilities of Vision-Language-Action models. The source code will be publicly available on our project page

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning: Imitation Learning, Grasping and Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2VwMOpenReview ↗
Rating
1(−1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SemAE, an encoder-decoder action expert, and proposes SemVLA based on it. Motivation is that flow matching action experts like pi0 hurt the VLM semantic knowledge and instruction following. The paper evaluates the proposed method on LIBERO, LIBERO-PRO, SimplerEnv, Polaris and real WidowX, Franka, DROID, and defines three language following metrics as average success over selected tasks, claims ~7 points gain over pi0.5.

Critical strengths
  1. Broad evaluation with strong baselines, and SemAE is at or near the top on most of them. Even without the language story, a continuous latent expert matching pi0.5 is useful.
  2. The VLM only predicts latents once and the small decoder iterates, much faster than latent flow matching with similar performance, beats AR and MAR.
Critical weaknesses
  1. The headline language following numbers don’t match the paper’s own tables. I tried to recompute them with the task mapping in Table 3 and got different numbers for all three metrics. The text also claims 15 points gain for Franka category but the table shows ~5. The ablation tables report yet another set of numbers. This renders the results suspicious.
  2. The metrics are just task success on hand-picked tasks, they don’t measure language following, while the paper itself admits this. Some task assignments also look arbitrary. The main claim is that SemAE preserves VLM knowledge but the backbone is never evaluated on any VLM benchmark. Real world results are 11-22 trials, so the gains are 1-2 trials, and error bars are large
  3. In Table 19, OpenVLA, VQ-VLA and pi0-FAST rows are exactly identical, looks like copy paste error. Again, this makes the results highly suspicious.
  4. Some implementation details are inconsistent (flow steps 10 vs 100, patch size).
Questions for authors

same as above. Most important is to explain the numbers and run VLM benchmarks.

10 comments

Thank you for your careful review. We noticed that you updated the new version of review comments. We address both versions below so that our response covers your original questions as well as your updated concerns. We begin with the calculations behind Figures 1(d) and 5. Table R-18 reports the VLM benchmark results you requested.

W1 & W2. Task mapping, success criteria, and consistency of language-following results

Revised review (W1). The headline language following numbers don’t match the paper’s own tables. I tried to recompute them with the task mapping in Table 3 and got different numbers for all three metrics. The text also claims 15 points gain for Franka category but the table shows ~5. The ablation tables report yet another set of numbers. This renders the results suspicious.

Revised review (W2). The metrics are just task success on hand-picked tasks, they don’t measure language following, while the paper itself admits this.

Initial review (Q1). What exact per-setting numbers and task subsets were used for Figure 5 and the 82.25/88.00/75.50 aggregates? How do they reconcile with Tables 1, 2, 18, 19, 21, and 22 under the Table 3 mapping? Specifically, where does the 15-point Franka Category gain come from?

Initial review (W1). Major concern: the headline language-following numbers do not match the paper’s own tables. Using the task mapping in Table 3 and the corresponding result tables, I get the following SemAE/π0.5 aggregates:

Affordance: 85.3/79.0 (+6.3), vs. the reported 82.25/75.50 (+6.75).

Indirect Reference: 87.8/81.1 (+6.7), vs. the reported 88.00/80.00 (+8.00).

Category: 82.6/78.3 (+4.25), vs. the reported 75.50/68.75 (+6.75).

The platform-level claims in Sec. 4.2 also conflict with the tables:

a. The text claims a 15-point Category gain on Franka, but Kitchen in Table 2 is 100.0 vs. 95.5.

b. The text says Indirect Reference “improves by 7–9 points in every setting,” but the LIBERO-PRO Sem columns give about 1.75 points in simulation.

c. The Simulation Affordance gain is claimed as 3 points, but LIBERO-Object gives 1.3.

On top of that, the ablation tables (5–8) report a default-model profile of 0.76/0.65/0.83, which matches neither of the above. It’s never stated which setting these numbers come from. The AI-use statement says an assistant-written script did the aggregation and updated the summary figure, which makes this more concerning. As written, the main quantitative claim of the paper cannot be verified.

Initial review (W2-a). a. The three metrics are just task success on selected tasks. The paper itself admits in Sec. 4.3 that success cannot separate grounding from execution, and no wrong-object/wrong-target rate is reported.

Author response

1. Language-following evaluation and the data behind Figures 1(d) and 5

Table 3 identifies the tasks used for each metric. The task-success columns in Tables 1, 2, 18, 19, 21, and 22 measure completion of the full manipulation task in the standard scene. For the three language-following metrics (affordance identification, indirect reference handling, category clustering) in Figure 5, we ran separate language-following trials with additional distractor objects, and all three metrics score target selection. Affordance Identification tests whether the robot selects the instructed object among manipulable alternatives using an explicit attribute such as color. Indirect Reference Handling tests whether it resolves a relational or functional description, such as the cloth whose color matches the cube. Category Clustering tests whether it selects an object belonging to the category named in the instruction. A trial counts as successful when the robot shows that it selected the instructed object. Once that choice is clear, the trial still counts even if a subsequent action fails.

Table R-11 reports the three language-following scores plotted in Figure 1(d). Table R-12 gives the success rates for each setting in Figure 5. Table R-13 (simulation), Table R-14 (Franka), Table R-15 (WidowX 250s), and Table R-16 (DROID) give the number of correct target selections and the total number of trials. Within each setting, we average the success rates of the tasks, suites, or task groups that Table 3 assigns to each metric. To obtain each score in Table R-11, we round each setting’s rate to two decimal places and then average the four rates with equal weight.

For example, the seven DROID task rates in Table R-16 give GR00T an Affordance Identification score of (14/25 + 14/25 + 15/25 + 16/25 + 17/25 + 74/100 + 36/75) / 7 ≈ 0.6086, which rounds to 0.61 in Table R-12(a). Averaging GR00T’s four setting-specific rates gives (0.62 + 0.55 + 0.27 + 0.61) / 4 = 0.5125, the GR00T entry in Table R-11.

Table R-11. Original Data plotted in Figure 1(d). Each value is the equally weighted mean of the four setting-specific success rates in the corresponding language-following panel of Figure 5.

ModelAffordance IdentificationIndirect Reference HandlingCategory Clustering
SemAE0.82250.88000.7550
GR00T0.51250.50500.4075
π0.50.75500.80000.6875
π00.60500.76000.5075
π0-FAST0.62250.51500.5500

Table R-12. Original Data plotted in Figure 5.

(a) Affordance Identification

ModelSimulationFrankaWidowX 250sDROID
SemAE0.870.790.820.81
GR00T0.620.550.270.61
π0.50.840.640.770.77
π00.710.300.730.68
π0-FAST0.810.660.270.75

(b) Indirect Reference Handling

ModelSimulationFrankaWidowX 250sDROID
SemAE0.980.910.910.72
GR00T0.690.640.210.48
π0.50.910.820.820.65
π00.980.730.800.53
π0-FAST0.750.460.180.67

(c) Category Clustering

ModelSimulationFrankaWidowX 250sDROID
SemAE0.540.790.720.97
GR00T0.160.550.180.74
π0.50.540.640.720.85
π00.280.300.640.81
π0-FAST0.310.660.360.87

Table R-13. Success counts and evaluation trials for language-following tasks in simulation.

Task or task groupMetricSemAEGR00Tπ0.5π0π0-FAST
LIBERO-Object suiteAffordance Identification435/500310/500420/500355/500405/500
LIBERO-PRO Spatial, SemIndirect Reference Handling480/500340/500460/500500/500365/500
LIBERO-PRO Object, SemIndirect Reference Handling485/500350/500465/500480/500370/500
LIBERO-PRO Goal, SemIndirect Reference Handling495/500355/500445/500485/500380/500
LIBERO-PRO Long, SemIndirect Reference Handling500/500335/500450/500495/500385/500
Food BussingCategory Clustering26/507/5025/5012/5013/50
Organize ToolsCategory Clustering28/509/5029/5016/5018/50

Table R-14. Success counts and evaluation trials for language-following tasks on Franka. Each basic subtask is evaluated over 67 trials. Three Cube contains three subtasks, Kitchen contains two, and Plush Toy contains one.

Task or task groupMetricSemAEGR00Tπ0.5π0π0-FAST
Three CubeAffordance Identification159/201111/201129/20160/201133/201
Plush ToyIndirect Reference Handling61/6743/6755/6749/6731/67
KitchenCategory Clustering106/13474/13486/13440/13488/134

Table R-15. Success counts and evaluation trials for language-following tasks on WidowX 250s. Each basic subtask is evaluated over 67 trials. Put Carrot on Plate contains two subtasks, Put Green Cup on Pink Cloth contains four, and Put Purple Cup on White Plate contains one.

Task or task groupMetricSemAEGR00Tπ0.5π0π0-FAST
Put Carrot on PlateAffordance Identification110/13436/134103/13498/13436/134
Put Green Cup on Pink ClothIndirect Reference Handling244/26856/268220/268214/26848/268
Put Purple Cup on White PlateCategory Clustering48/6712/6748/6743/6724/67

Table R-16. Success counts and evaluation trials for language-following tasks in the DROID setting. Each model is evaluated over 300 trials across the seven tasks. Pick Bread and Arrange Cube contribute to multiple metrics using the same trial counts.

TaskMetricSemAEGR00Tπ0.5π0π0-FAST
Flip BottleAffordance Identification18/2514/2518/2515/2517/25
Add IngredientsAffordance Identification19/2514/2519/2516/2518/25
Cover the PotAffordance Identification20/2515/2519/2517/2519/25
Put SpoonAffordance Identification21/2516/2520/2518/2519/25
Put BowlAffordance Identification21/2517/2521/2519/2520/25
Pick BreadAffordance Identification / Category Clustering97/10074/10085/10081/10087/100
Arrange CubeAffordance Identification / Indirect Reference Handling54/7536/7549/7540/7550/75

2. Calculations for the seven results questioned in the review

Below, we show how we calculated each of the seven sets of results the reviewer questioned.

  • Average success rate for Affordance Identification. The review gives 85.3% and 79.0%. But using the rates in Table R-12(a), SemAE scores (0.87 + 0.79 + 0.82 + 0.81) / 4 = 0.8225 (82.25%), and π0.5 scores (0.84 + 0.64 + 0.77 + 0.77) / 4 = 0.7550 (75.50%). The gain is 6.75 percentage points.

  • Average success rate for Indirect Reference Handling. The review gives 87.8% and 81.1%. The rates in Table R-12(b) give (0.98 + 0.91 + 0.91 + 0.72) / 4 = 0.8800 (88.00%) for SemAE and (0.91 + 0.82 + 0.82 + 0.65) / 4 = 0.8000 (80.00%) for π0.5, a gain of 8.00 percentage points.

  • Average success rate for Category Clustering. The review gives 82.6% and 78.3%. The rates in Table R-12(c) give (0.54 + 0.79 + 0.72 + 0.97) / 4 = 0.7550 (75.50%) for SemAE and (0.54 + 0.64 + 0.72 + 0.85) / 4 = 0.6875 (68.75%) for π0.5, a gain of 6.75 percentage points.

  • Category Clustering gain on Franka. The 100.0% and 95.5% in Table 2 measure Kitchen task completion. In the language-following Kitchen trials (Table R-14), SemAE selected the correct target in 106/134 trials and π0.5 in 86/134. The difference is 100 × (106 − 86) / 134 ≈ 14.93 percentage points. Table R-12(c) gives the plotted rates of 0.79 and 0.64, giving 15 percentage points.

  • Indirect Reference Handling gains across settings. The review’s simulation estimate of about 1.75 points comes from Table 19. In the language-following trials, the Sem shifts in the four LIBERO-PRO suites (Table R-13) give (480 + 485 + 495 + 500) / 2000 = 0.98 for SemAE and (460 + 465 + 445 + 450) / 2000 = 0.91 for π0.5. In the same model order, Franka (Table R-14) gives 61/67 and 55/67 (0.91 and 0.82), WidowX 250s (Table R-15) gives 244/268 and 220/268 (0.91 and 0.82), and DROID (Table R-16) gives 54/75 and 49/75 (0.72 and 0.65). Subtracting the plotted rates in Table R-12(b) gives 7, 9, 9, and 7 percentage points, respectively.

  • Affordance Identification gain in simulation. The 1.3-point LIBERO-Object gain in Table 18 comes from task completion (99.5% − 98.2%). In the language-following trials for that suite (Table R-13), SemAE scores 435/500 and π0.5 scores 420/500. The gain in Table R-12(a) is therefore 100 × (435/500 − 420/500) = 3 percentage points.

  • Default SemAE scores in Tables 5 to 8. The ablations use a subset of the tasks and subtasks in Table 3, with the same SemAE checkpoint and target-selection criterion as Figure 5. Table R-17 identifies the six tasks or subtasks and their 276 distinct trials. We give each selected task or subtask equal weight within its metric.

    • Affordance Identification. On DROID, Put Spoon, Put Bowl, and Arrange Cube (Blue) give (21/25 + 21/25 + 15/25) / 3 = 0.76.
    • Indirect Reference Handling. Put Green Cup on Pink Cloth on WidowX 250s (cup starts at the bottom of the sink) and the same DROID Arrange Cube (Blue) trials give (47/67 + 15/25) / 2 ≈ 0.6507, reported as 0.65.
    • Category Clustering. Kitchen Pot on Franka and Put Purple Cup on White Plate on WidowX 250s give (63/67 + 48/67) / 2 ≈ 0.8284, reported as 0.83.

    Tables 5, 6, and 8 report these three scores. Averaging the three unrounded values gives approximately 0.7464, reported as 0.75 for Language Following in Tables 5 and 7.

Table R-17. Task mapping and success counts for the default SemAE language-following ablations in Tables 5 to 8.

SettingTask or subtaskLanguage-following metric(s)SemAE successes / trials
DROIDPut SpoonAffordance Identification21/25
DROIDPut BowlAffordance Identification21/25
DROIDArrange Cube (Blue)Affordance Identification, Indirect Reference Handling15/25
WidowX 250sPut Green Cup on Pink Cloth (cup starts at the bottom of the sink)Indirect Reference Handling47/67
WidowX 250sPut Purple Cup on White PlateCategory Clustering48/67
FrankaKitchen PotCategory Clustering63/67
Reviewer_sv4gOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SemAE, a two-stage approach to connecting a pretrained VLM with a continuous-action decoder. It first trains an action encoder to align with VLM features while learning to reconstruct actions, then trains the VLM to predict these action representations in parallel for decoding into robot actions. The authors evaluate the resulting SemVLA model on simulation and real-robot tasks, with language-following tests and component ablations.

Critical strengths
  1. The two-stage design offers a concrete way to connect VLM representations to continuous robot actions. The component ablations show that both motion-primitive distillation and the action decoder contribute to task performance, making the method more compelling than an architectural proposal alone.

  2. The paper evaluates the method across simulation and several real-robot settings, including tasks designed to test language following. This breadth makes the reported gains potentially consequential, subject to clarification of the experimental settings.

Critical weaknesses

The paper claims that semantic alignment preserves the VLM’s understanding and improves robot instruction following, but it does not directly measure semantic retention or separate improved understanding from improved action execution. Stage 1 gives the teacher VLM images overlaid with future robot trajectories, yet the paper provides no ablation study measuring the contribution of this design.

The real-robot experiments mix robot-based labels (Franka and WidowX) with a dataset-based label (DROID), making the setups hard to follow. It is unclear which tasks use task-specific fine-tuning and which, if any, are evaluated zero-shot after training on BridgeData V2 or DROID. The paper should also state whether the baselines were trained with comparable data, compute, and training settings.

Questions for authors
  1. Please provide direct evidence that SemAE preserves the VLM’s semantic understanding after robot training, and clarify whether the observed robot gains come from better instruction understanding or better action execution.

  2. Please ablate the future end-effector trajectory overlay used in Stage 1 to measure this design choice’s contribution.

  3. Please clarify the real-robot experiment labels and settings: which Franka, DROID, and WidowX tasks use task-specific fine-tuning, which (if any) are evaluated zero-shot after BridgeData V2 or DROID training, and whether the baselines use comparable data, compute, and training settings.

  4. Please verify the identical per-task results for OpenVLA, VQ-VLA, and π0-FAST in LIBERO-PRO Table 19 against the original evaluation records.

2 comments

Thank you for your careful review and constructive feedback. We hope the clarifications below address the four questions you raised.

W1 & Q1. Direct evidence of semantic preservation. After robot training, SemAE shows no substantial decline on RoboVQA or ERQA compared with its original VLM backbone, Qwen2.5-VL-3B (Table R-9). This provides direct evidence that SemAE preserves the VLM’s semantic understanding on these benchmarks. On both benchmarks, π₀.₅ scores substantially below its original PaliGemma backbone.

Table R-9. Language and grounding performance of the original VLM backbones and their robot-trained models. Bold marks the highest score in each column, including ties.

ModelRoboVQAERQAAverage
PaliGemma45.627.336.5
Qwen2.5-VL-3B55.935.345.6
π₀.₅0.00.00.0
SemAE55.435.345.4

Both benchmarks evaluate embodied understanding relevant to robot instruction following. RoboVQA assesses video-grounded planning, affordance reasoning, and task success recognition in human and robot activities. ERQA evaluates spatial, action, and task reasoning in physical scenes. Because these QA benchmarks assess understanding without requiring physical action execution, SemAE’s advantage over π₀.₅ supports our interpretation that the observed robot gains come mainly from better instruction understanding.

W1 & Q2. Future trajectory overlays. Adding the projected future end-effector trajectory to the teacher’s current image improves success on all six Polaris tasks and raises the average success rate. Table R-10 compares distillation without future trajectory overlays (A) and with future trajectory overlays (B, SemAE) in Stage 1. Both Stage 1 distillation settings use the same backbone, data, initialization, and training budget. They differ only in whether the projected future end-effector trajectory is overlaid on the teacher’s current image.

Table R-10. Ablation of future end-effector trajectory overlays on Polaris.

Polaris taskA.Without overlayB.With overlay (SemAE)
BlockStackKitchen0.450.57
FoodBussing0.440.54
PanClean0.560.57
MoveLatteCup0.340.44
OrganizeTools0.570.67
TapeIntoContainer0.840.85
Average0.5330.607

W2 & Q3. Real-world evaluation settings. We use the dataset-based label “DROID” alongside the robot-based labels “Franka” and “WidowX” to distinguish the two Franka hardware setups.

  • Franka. We use a Franka Emika Panda arm with the standard Franka Hand gripper. All tasks use task-specific fine-tuning.
  • DROID. We use a Franka Research 3 arm with a third-party Robotiq 2F-85 gripper, matching the DROID dataset’s hardware configuration. All tasks are evaluated zero-shot after training on DROID.
  • WidowX. All tasks are evaluated zero-shot after training on BridgeData V2.

Within each evaluation setting, all models use the same data, compute, and training settings.

Q4. Verification of LIBERO-PRO results. The VQ-VLA and π₀-FAST rows in the original Table 19 mistakenly repeated OpenVLA’s evaluation results. We thank the reviewer for pointing this out. The corrected results below show that VQ-VLA and π₀-FAST both have lower average success rates than OpenVLA. The manuscript’s other conclusions related to Table 19 were based on the original evaluation records of VQ-VLA and π₀-FAST and still hold after this correction.

Table 19. Corrected LIBERO-PRO success rates. SemAE has the highest average across all 20 settings.

Suite / shiftOpenVLAVQ-VLAπ₀-FASTπ₀π₀.₅SemAE
Average0.520.34590.43870.440.530.59
Spatial / object0.970.34100.48200.950.970.98
Spatial / position0.000.34100.48200.000.200.11
Spatial / semantic0.970.34100.48200.970.970.98
Spatial / task0.000.34100.48200.000.010.12
Spatial / environment0.890.34100.48200.600.460.92
Object / object0.980.35950.48400.940.980.97
Object / position0.000.35950.48400.000.170.09
Object / semantic0.980.35950.48400.900.960.98
Object / task0.000.35950.48400.000.010.12
Object / environment0.000.35950.48400.290.730.63
Goal / object0.960.37650.44300.940.970.98
Goal / position0.000.37650.44300.000.380.10
Goal / semantic0.980.37650.44300.930.970.98
Goal / task0.000.37650.44300.000.000.11
Goal / environment0.980.37650.44300.390.460.54
Long / object0.810.30650.49000.790.920.93
Long / position0.000.30650.02200.000.080.21
Long / semantic0.960.30650.78000.820.930.96
Long / task0.000.30650.13600.000.010.37
Long / environment0.850.30650.30100.270.460.64
Reviewer_MehJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SemAE trains a continuous action autoencoder using flow reconstruction and layer-wise distillation from a frozen VLM that observes projected end-effector trajectories. A trainable Q-Former extracts the teacher-side targets. SemVLA then predicts the frozen action encoder's embeddings through parallel action slots and decodes continuous action patches. Evaluation spans LIBERO, LIBERO-PRO, SimplerEnv, Polaris, and three physical robot settings, with additional text-action co-training and architectural ablations.

Critical strengths
  1. The action interface avoids discrete action quantization while keeping expensive latent prediction to one backbone pass. Its evaluation across several simulators and robot settings gives the practical proposal useful breadth. The reported Franka and WidowX success rates are 73.7% and 89.4%, respectively, compared with 65.9% and 84.1% for pi0.5. The computational table also distinguishes backbone-only latency from the full 82.6 ms inference path, rather than treating the 19.1 ms language-backbone time as end-to-end latency.
  2. The paper includes direct controls for important components. On Polaris, removing semantic distillation lowers average success from 0.607 to 0.487 while retaining the flow decoder; removing flow or parallel decoding gives 0.540 or 0.562 (Table 21). This is stronger evidence than comparing heterogeneous policies alone. The final-layer versus layer-wise distillation and parameter-matched MLP decoder comparisons further help delimit which parts of the recipe are useful.
Critical weaknesses
  1. The central language-following evaluation is not yet traceable to its stated task mapping. Appendix A.1 maps Franka Category Clustering to Kitchen, but Table 2 gives 100.0% versus 95.5% task success for SemAE and pi0.5, a 4.5-point difference rather than the 15-point gain stated in Section 4.2. It maps Simulation Affordance Identification to LIBERO-Object, where Table 18 gives 99.5% versus 98.2%, a 1.3-point difference rather than the stated 3 points. If Figure 5 uses separate evaluation subsets, their trial counts and mapping are needed; the referenced tables cannot currently reproduce the advertised aggregation. Further, explicit cup-color selection and placing one named food item do not by themselves establish category-level or indirect-reference reasoning. Task completion also mixes selecting the correct target with executing the manipulation.
  2. Useful distillation is not yet equivalent to preserving the pretrained semantic space. The action-only encoder cannot distinguish two scene/instruction contexts containing the same action chunk, while its teacher-side targets depend on those contexts and are transformed by a trainable Q-Former. Low cosine loss therefore need not mean that the original VLM semantics are retained. The no-distillation control establishes an auxiliary-supervision benefit, but does not isolate trajectory-conditioned semantics from generic teacher regularization. Section 3 also explicitly borrows the layer-wise distillation and patch-wise flow design from UniFlow, making the action-specific evidence particularly important. Attention maps are not sufficient, and Appendix A.2 reports different baseline adaptation regimes and 1.8-2.3 times the training cost without a matched-budget semantic-preservation comparison.
Questions for authors
  1. Please provide the exact task subsets, trial counts, and platform-level numerators underlying Figures 1 and 5, and reconcile the two mapping discrepancies above. Can correct-target selection be reported separately from downstream manipulation success?
  2. With the same backbone, data, initialization, and budget, what benefit remains over a reconstruction-only action interface with an alternative teacher target, such as an unoverlaid current image? Please specify how Q-Former targets are constrained and whether held-out language/grounding performance supports semantic preservation rather than only improved control.
  3. Which sampling count generated the reported control results: the ten decoder passes used in Table 9's latency measurement, or the 100 sampling steps listed in Tables 10-17? Please give end-to-end timing for the actual evaluated setting.
4 comments

Thank you for your careful review. We address your questions about the language-following evaluation, the contribution of teacher supervision, and inference timing below.

W1 & Q1. Task mapping, target selection, and the gains in Figures 1(d) and 5.

W1. Appendix A.1 maps Franka Category Clustering to Kitchen, but Table 2 gives 100.0% versus 95.5% task success for SemAE and pi0.5, a 4.5-point difference rather than the 15-point gain stated in Section 4.2. It maps Simulation Affordance Identification to LIBERO-Object, where Table 18 gives 99.5% versus 98.2%, a 1.3-point difference rather than the stated 3 points. … Further, explicit cup-color selection and placing one named food item do not by themselves establish category-level or indirect-reference reasoning.

Q1. Please provide the exact task subsets, trial counts, and platform-level numerators underlying Figures 1 and 5, and reconcile the two mapping discrepancies above. Can correct-target selection be reported separately from downstream manipulation success?

1. Setting for Language-following experiments

For panels (a) to (c) of Figure 5, we ran separate language-following trials using the task groups in Table 3 with additional distractor objects. We counted a trial as correct once the robot’s actions showed that it had chosen the instructed target, even if the subsequent manipulation failed. Tables 2 and 18 report whether the robot completed the manipulation in the standard Kitchen and LIBERO-Object evaluations.

The task assignment depends on how the instruction identifies the target. In Franka Three Cube, color specifies which cube to select (Affordance Identification). In WidowX Purple Cup, several types of purple cup share the color cue. The same instruction applies across these cup types, which is why we score the task under Category Clustering.

The named-food tasks also differ in the choice they require. Franka Kitchen (Banana) requires selecting a banana among oranges, apples, and lemons, so it contributes to Category Clustering (Table R-4). DROID Pick Bread contributes to both Affordance Identification and Category Clustering (Table R-6). On Franka and DROID, the tasks scored under Indirect Reference Handling require a relational choice. Franka Plush Toy specifies the nearer of two toys, and DROID Arrange Cube specifies the cloth whose color matches the cube.

Tables R-3 to R-6 give correct selections out of total trials for every listed task group. Each model was evaluated over 2,600 trials in simulation (Table R-3), 402 on Franka (Table R-4), 469 on WidowX 250s (Table R-5), and 300 on DROID (Table R-6). The DROID total counts each trial once, including trials used for two metrics. Panels (a) to (c) of Table R-2 report the three target-selection rates for each setting.

2. Calculations for the two gains and the averages in Figure 1(d)

  • Franka Category Clustering. In Table 2, Kitchen task completion was 100.0% for SemAE and 95.5% for π0.5 over 22 standard trials per model, a gap of 4.5 percentage points. The language-following evaluation used 67 Kitchen Banana trials and 67 Kitchen Pot trials per model. SemAE selected the correct target in 106/134 trials and π0.5 in 86/134 (Table R-4). The gain is 100 × (106 − 86) / 134 ≈ 14.93 percentage points. Figure 5(c) plots the rates as 0.79 and 0.64, giving the reported 15 percentage points (Table R-2(c)).
  • Simulation Affordance Identification. Table 18 reports LIBERO-Object task completion of 99.5% for SemAE and 98.2% for π0.5, a gap of 1.3 percentage points. In the language-following trials for that suite, SemAE selected the correct target in 435/500 trials and π0.5 in 420/500 (Table R-3). The gain in Figure 5(a) is 100 × (435 − 420) / 500 = 3 percentage points (Table R-2(a)).

When multiple tasks or suites contribute to a metric within one setting, their individual rates have equal weight. For DROID Affordance Identification, SemAE made 250 correct selections in 300 trials (Table R-6). The mean of the seven task rates is (18/25 + 19/25 + 20/25 + 21/25 + 21/25 + 97/100 + 54/75) / 7 ≈ 0.8071, plotted as 0.81 in Table R-2(a).

For Figure 1(d), we rounded each setting’s rate to two decimal places as plotted in Figure 5, then averaged the four rates with equal weight. The calculations below use simulation, Franka, WidowX 250s, and DROID in that order.

  • Affordance Identification. SemAE scores (87% + 79% + 82% + 81%) / 4 = 82.25%, and π0.5 scores (84% + 64% + 77% + 77%) / 4 = 75.50% (Table R-2(a)).
  • Indirect Reference Handling. SemAE scores (98% + 91% + 91% + 72%) / 4 = 88.00%, and π0.5 scores (91% + 82% + 82% + 65%) / 4 = 80.00% (Table R-2(b)).
  • Category Clustering. SemAE scores (54% + 79% + 72% + 97%) / 4 = 75.50%, and π0.5 scores (54% + 64% + 72% + 85%) / 4 = 68.75% (Table R-2(c)).

Table R-1. Language-following scores in Figure 1(d). Each value is the equally weighted mean of the four setting-specific target-selection rates in the corresponding panel of Figure 5.

ModelAffordance IdentificationIndirect Reference HandlingCategory Clustering
SemAE0.82250.88000.7550
GR00T0.51250.50500.4075
π0.50.75500.80000.6875
π00.60500.76000.5075
π0-FAST0.62250.51500.5500

Table R-2. Target-selection rates by setting in Figure 5(a) to (c).

(a) Affordance Identification

ModelSimulationFrankaWidowX 250sDROID
SemAE0.870.790.820.81
GR00T0.620.550.270.61
π0.50.840.640.770.77
π00.710.300.730.68
π0-FAST0.810.660.270.75

(b) Indirect Reference Handling

ModelSimulationFrankaWidowX 250sDROID
SemAE0.980.910.910.72
GR00T0.690.640.210.48
π0.50.910.820.820.65
π00.980.730.800.53
π0-FAST0.750.460.180.67

(c) Category Clustering

ModelSimulationFrankaWidowX 250sDROID
SemAE0.540.790.720.97
GR00T0.160.550.180.74
π0.50.540.640.720.85
π00.280.300.640.81
π0-FAST0.310.660.360.87

Table R-3. Correct target selections out of total trials in simulation.

Task or task groupMetricSemAEGR00Tπ0.5π0π0-FAST
LIBERO-Object suiteAffordance Identification435/500310/500420/500355/500405/500
LIBERO-PRO Spatial, SemIndirect Reference Handling480/500340/500460/500500/500365/500
LIBERO-PRO Object, SemIndirect Reference Handling485/500350/500465/500480/500370/500
LIBERO-PRO Goal, SemIndirect Reference Handling495/500355/500445/500485/500380/500
LIBERO-PRO Long, SemIndirect Reference Handling500/500335/500450/500495/500385/500
Food BussingCategory Clustering26/507/5025/5012/5013/50
Organize ToolsCategory Clustering28/509/5029/5016/5018/50

Table R-4. Correct target selections out of total trials on Franka. Each basic subtask is evaluated over 67 trials. Three Cube contains three subtasks, Kitchen contains two, and Plush Toy contains one.

Task or task groupMetricSemAEGR00Tπ0.5π0π0-FAST
Three CubeAffordance Identification159/201111/201129/20160/201133/201
Plush ToyIndirect Reference Handling61/6743/6755/6749/6731/67
KitchenCategory Clustering106/13474/13486/13440/13488/134

Table R-5. Correct target selections out of total trials on WidowX 250s. Each basic subtask is evaluated over 67 trials. Put Carrot on Plate contains two subtasks, Put Green Cup on Pink Cloth contains four, and Put Purple Cup on White Plate contains one.

Task or task groupMetricSemAEGR00Tπ0.5π0π0-FAST
Put Carrot on PlateAffordance Identification110/13436/134103/13498/13436/134
Put Green Cup on Pink ClothIndirect Reference Handling244/26856/268220/268214/26848/268
Put Purple Cup on White PlateCategory Clustering48/6712/6748/6743/6724/67

Table R-6. Correct target selections out of total trials on DROID. Each model is evaluated over 300 trials across the seven tasks. Pick Bread and Arrange Cube contribute to multiple metrics using the same trial counts.

TaskMetricSemAEGR00Tπ0.5π0π0-FAST
Flip BottleAffordance Identification18/2514/2518/2515/2517/25
Add IngredientsAffordance Identification19/2514/2519/2516/2518/25
Cover the PotAffordance Identification20/2515/2519/2517/2519/25
Put SpoonAffordance Identification21/2516/2520/2518/2519/25
Put BowlAffordance Identification21/2517/2521/2519/2520/25
Pick BreadAffordance Identification / Category Clustering97/10074/10085/10081/10087/100
Arrange CubeAffordance Identification / Indirect Reference Handling54/7536/7549/7540/7550/75

W2 & Q2 (1/3). What constrains the Q-Former targets?

W2. The action-only encoder cannot distinguish two scene/instruction contexts containing the same action chunk, while its teacher-side targets depend on those contexts and are transformed by a trainable Q-Former. Low cosine loss therefore need not mean that the original VLM semantics are retained.

Q2. Please specify how Q-Former targets are constrained …

Additional experiments and technical details demonstrate that SemAE does maintain the original VLM’s semantics through distillation:

  1. We agree with the reviewer’s concern that an action-only encoder cannot distinguish two scene/instruction contexts containing the same action chunk. However, this case is rarely observed in practice: robot datasets rarely contain two identical action chunks. For example, HuggingFaceVLA/libero can be separated into 248,070 action chunks with a chunk size of 16, and we find that all these chunks were distinct. Therefore, the action encoder does not need to distinguish different contexts for identical chunks.
  2. Supervising the student (i.e., the action-only encoder) with teacher targets derived from privileged scene and instruction context is consistent with generalized distillation. The purpose of our distillation process is to make the action encoder learn task-related semantics solely from different action chunks, without additional scene or instruction context as input.
  3. During training, gradients from the cosine action-distillation loss are stopped before reaching the Q-Former. Instead, the Q-Former receives independent supervision through Q-Former's own readout head. This head maps Q-Former outputs back to the VLM hidden space, where a KL loss aligns the readout and frozen teacher feature distributions. This supervision trains the Q-Former to retain teacher semantics while preventing the action-distillation loss from adjusting its targets to match the action encoder.
  4. Table R-8 provides further empirical evidence. After training on robot action data, the VLM backbone shows no substantial performance decline on language and grounding benchmarks compared with the original Qwen2.5-VL-3B checkpoint. These results support the retention of the original VLM’s language and grounding capabilities.

W2 & Q2 (2/3). What does the future trajectory overlay contribute?

W2. The no-distillation control establishes an auxiliary-supervision benefit, but does not isolate trajectory-conditioned semantics from generic teacher regularization. Section 3 also explicitly borrows the layer-wise distillation and patch-wise flow design from UniFlow, making the action-specific evidence particularly important.

Q2. With the same backbone, data, initialization, and budget, what benefit remains over a reconstruction-only action interface with an alternative teacher target, such as an unoverlaid current image?

Table R-7 compares three supervision settings on Polaris: (A) a reconstruction-only action interface, (B) reconstruction with distillation from an unoverlaid current image, and (C) reconstruction with distillation from a trajectory-overlaid image (SemAE). All three settings use the same backbone, data, initialization, and training budget, differing only in their training supervision. Settings A and C are reported in Table 21, while setting B is an additional experiment. Distilling from images overlaid with projected end-effector trajectories achieves an average success rate of 0.607, compared with 0.533 for distillation from unoverlaid images and 0.487 for reconstruction alone. These results demonstrate the additional benefit of trajectory-conditioned supervision beyond generic teacher regularization.

Table R-7. Comparison of teacher supervision on Polaris. Success rates for a reconstruction-only action interface and two distillation settings using either an unoverlaid current image or a trajectory-overlaid image.

SettingBlockStackKitchenFoodBussingPanCleanMoveLatteCupOrganizeToolsTapeIntoContainerAverage
A. Reconstruction-only action interface0.370.380.550.270.510.840.487
B. Distillation from an unoverlaid current image0.450.440.560.340.570.840.533
C. Distillation from a trajectory-overlaid image (SemAE)0.570.540.570.440.670.850.607

W2 & Q2 (3/3). Does the robot-trained VLM retain language and grounding ability?

W2. Useful distillation is not yet equivalent to preserving the pretrained semantic space. … Attention maps are not sufficient, and Appendix A.2 reports different baseline adaptation regimes and 1.8-2.3 times the training cost without a matched-budget semantic-preservation comparison.

Q2. … whether held-out language/grounding performance supports semantic preservation rather than only improved control.

Table R-8 presents the language and grounding performance of pi0.5 and SemAE after training on the same dataset with the same compute budget. Their VLM backbones (PaliGemma and Qwen2.5-VL-3B) are also included for comparison. Under the same compute budget, pi0.5 is initialized from PaliGemma and trained on EO-Data 1.5M, while SemAE is initialized from Qwen2.5-VL-3B and completes its two-stage training on EO-Data 1.5M. The benchmark results show that pi0.5’s language capabilities are severely degraded, while SemAE’s language and grounding performance shows no substantial decline. These results indicate that SemAE effectively preserves the pretrained VLM’s semantic capabilities.

Table R-8. Language and grounding performance of pi0.5 and SemAE compared with their VLM backbones, PaliGemma and Qwen2.5-VL-3B.

ModelRoboVQAERQAAverage
PaliGemma45.627.336.5
Qwen2.5-VL-3B55.935.345.6
pi0.50.00.00.0
SemAE55.435.345.4

Q3. Sampling steps and end-to-end inference time.

Q3. Which sampling count generated the reported control results: the ten decoder passes used in Table 9's latency measurement, or the 100 sampling steps listed in Tables 10-17? Please give end-to-end timing for the actual evaluated setting.

All reported control results use 10 flow decoder forward passes per action chunk, which is also the configuration used for the latency evaluation in Table 9. So Table 9 has already reported the end-to-end timing for the actual evaluated setting. The value 100 in Tables 10-17 refers to the scheduler’s training sampling step (base time-grid size), while 10 is the number of inference sampling steps. This distinction follows an established pattern in Diffusers’ FlowMatchEulerDiscreteScheduler, which separates num_train_timesteps from num_inference_steps. We will revise Tables 10-17 to report these parameters separately as “Scheduler time-grid size = 100” and “Inference sampling steps = 10”.

Comments on the paper

1 comment

We thank the reviewers for their careful assessments of SemAE. All three highlighted the breadth of our evaluation across four simulation benchmarks and three real-robot settings, and sv4g noted the inclusion of tasks designed specifically to test language following. As 2VwM observed, SemAE performs at the top in most comparisons with strong baselines, and a continuous latent action expert competitive with π0.5 is useful in its own right.

Reviewer sv4g highlighted how SemAE’s two-stage design connects the representations of a pretrained vision-language model (VLM) to continuous robot actions. MehJ noted that this interface avoids action quantization. Both MehJ and 2VwM recognized that the VLM predicts all action latents in a single forward pass. 2VwM further noted that only the small flow decoder iterates and that parallel latent prediction is faster than latent flow matching while achieving comparable language-following scores. MehJ and sv4g highlighted the component ablations showing that motion-primitive distillation and the action decoder each improve performance.

Additionally, we thank Reviewer MehJ for highlighting our comparison of layer-wise and final-layer-only distillation (Table 6) and our use of a parameter-matched MLP decoder as a control (Table 7). MehJ also noted that Table 9 reports language-backbone latency separately from end-to-end model latency.

Clarifications and additional results

We briefly summarize the concerns raised by multiple reviewers here. They are all clarified in detail in our replies to each reviewer.

  • Language-following scores. We ran the Figure 5 language-following trials separately from the standard manipulation tests and added distractors to the scenes. We group tasks by the language skill needed to select the target. In language-following experiments, a correct selection counts as a success even if the subsequent grasp fails. The per-task counts in our response to 2VwM (Tables R-13 to R-16) reproduce all three Figure 5 metrics. For each metric, Figure 1(d) reports the equally weighted mean of the four settings’ scores in Figure 5. Table R-17 lists the task subset for the default-model ablations.
  • Semantic preservation after robot training. SemAE’s RoboVQA and ERQA scores remain close to those of its original Qwen2.5-VL-3B backbone after the robot training. With the same data and compute budget, SemAE outperforms π0.5 on both embodied VQA benchmarks (Table R-8 in our response to MehJ).
  • Contribution of future trajectory overlays. With the backbone, data, initialization, and training budget fixed, adding the future end-effector trajectory to the Stage 1 teacher’s input image raised average success on Polaris by 7.4 percentage points over distillation from the image alone (Table R-10 in our response to sv4g).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.