- Rating
- 1(−1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SemAE, an encoder-decoder action expert, and proposes SemVLA based on it. Motivation is that flow matching action experts like pi0 hurt the VLM semantic knowledge and instruction following. The paper evaluates the proposed method on LIBERO, LIBERO-PRO, SimplerEnv, Polaris and real WidowX, Franka, DROID, and defines three language following metrics as average success over selected tasks, claims ~7 points gain over pi0.5.
Critical strengths
- Broad evaluation with strong baselines, and SemAE is at or near the top on most of them. Even without the language story, a continuous latent expert matching pi0.5 is useful.
- The VLM only predicts latents once and the small decoder iterates, much faster than latent flow matching with similar performance, beats AR and MAR.
Critical weaknesses
- The headline language following numbers don’t match the paper’s own tables. I tried to recompute them with the task mapping in Table 3 and got different numbers for all three metrics. The text also claims 15 points gain for Franka category but the table shows ~5. The ablation tables report yet another set of numbers. This renders the results suspicious.
- The metrics are just task success on hand-picked tasks, they don’t measure language following, while the paper itself admits this. Some task assignments also look arbitrary. The main claim is that SemAE preserves VLM knowledge but the backbone is never evaluated on any VLM benchmark. Real world results are 11-22 trials, so the gains are 1-2 trials, and error bars are large
- In Table 19, OpenVLA, VQ-VLA and pi0-FAST rows are exactly identical, looks like copy paste error. Again, this makes the results highly suspicious.
- Some implementation details are inconsistent (flow steps 10 vs 100, patch size).
Questions for authors
same as above. Most important is to explain the numbers and run VLM benchmarks.
10 comments
Table R-11. Original Data plotted in Figure 1(d). Each value is the equally weighted mean of the four setting-specific success rates in the corresponding language-following panel of Figure 5.
| Model | Affordance Identification | Indirect Reference Handling | Category Clustering |
|---|---|---|---|
| SemAE | 0.8225 | 0.8800 | 0.7550 |
| GR00T | 0.5125 | 0.5050 | 0.4075 |
| π0.5 | 0.7550 | 0.8000 | 0.6875 |
| π0 | 0.6050 | 0.7600 | 0.5075 |
| π0-FAST | 0.6225 | 0.5150 | 0.5500 |
Table R-12. Original Data plotted in Figure 5.
(a) Affordance Identification
| Model | Simulation | Franka | WidowX 250s | DROID |
|---|---|---|---|---|
| SemAE | 0.87 | 0.79 | 0.82 | 0.81 |
| GR00T | 0.62 | 0.55 | 0.27 | 0.61 |
| π0.5 | 0.84 | 0.64 | 0.77 | 0.77 |
| π0 | 0.71 | 0.30 | 0.73 | 0.68 |
| π0-FAST | 0.81 | 0.66 | 0.27 | 0.75 |
(b) Indirect Reference Handling
| Model | Simulation | Franka | WidowX 250s | DROID |
|---|---|---|---|---|
| SemAE | 0.98 | 0.91 | 0.91 | 0.72 |
| GR00T | 0.69 | 0.64 | 0.21 | 0.48 |
| π0.5 | 0.91 | 0.82 | 0.82 | 0.65 |
| π0 | 0.98 | 0.73 | 0.80 | 0.53 |
| π0-FAST | 0.75 | 0.46 | 0.18 | 0.67 |
(c) Category Clustering
| Model | Simulation | Franka | WidowX 250s | DROID |
|---|---|---|---|---|
| SemAE | 0.54 | 0.79 | 0.72 | 0.97 |
| GR00T | 0.16 | 0.55 | 0.18 | 0.74 |
| π0.5 | 0.54 | 0.64 | 0.72 | 0.85 |
| π0 | 0.28 | 0.30 | 0.64 | 0.81 |
| π0-FAST | 0.31 | 0.66 | 0.36 | 0.87 |
Table R-13. Success counts and evaluation trials for language-following tasks in simulation.
| Task or task group | Metric | SemAE | GR00T | π0.5 | π0 | π0-FAST |
|---|---|---|---|---|---|---|
| LIBERO-Object suite | Affordance Identification | 435/500 | 310/500 | 420/500 | 355/500 | 405/500 |
| LIBERO-PRO Spatial, Sem | Indirect Reference Handling | 480/500 | 340/500 | 460/500 | 500/500 | 365/500 |
| LIBERO-PRO Object, Sem | Indirect Reference Handling | 485/500 | 350/500 | 465/500 | 480/500 | 370/500 |
| LIBERO-PRO Goal, Sem | Indirect Reference Handling | 495/500 | 355/500 | 445/500 | 485/500 | 380/500 |
| LIBERO-PRO Long, Sem | Indirect Reference Handling | 500/500 | 335/500 | 450/500 | 495/500 | 385/500 |
| Food Bussing | Category Clustering | 26/50 | 7/50 | 25/50 | 12/50 | 13/50 |
| Organize Tools | Category Clustering | 28/50 | 9/50 | 29/50 | 16/50 | 18/50 |
Table R-14. Success counts and evaluation trials for language-following tasks on Franka. Each basic subtask is evaluated over 67 trials. Three Cube contains three subtasks, Kitchen contains two, and Plush Toy contains one.
| Task or task group | Metric | SemAE | GR00T | π0.5 | π0 | π0-FAST |
|---|---|---|---|---|---|---|
| Three Cube | Affordance Identification | 159/201 | 111/201 | 129/201 | 60/201 | 133/201 |
| Plush Toy | Indirect Reference Handling | 61/67 | 43/67 | 55/67 | 49/67 | 31/67 |
| Kitchen | Category Clustering | 106/134 | 74/134 | 86/134 | 40/134 | 88/134 |
Table R-15. Success counts and evaluation trials for language-following tasks on WidowX 250s. Each basic subtask is evaluated over 67 trials. Put Carrot on Plate contains two subtasks, Put Green Cup on Pink Cloth contains four, and Put Purple Cup on White Plate contains one.
| Task or task group | Metric | SemAE | GR00T | π0.5 | π0 | π0-FAST |
|---|---|---|---|---|---|---|
| Put Carrot on Plate | Affordance Identification | 110/134 | 36/134 | 103/134 | 98/134 | 36/134 |
| Put Green Cup on Pink Cloth | Indirect Reference Handling | 244/268 | 56/268 | 220/268 | 214/268 | 48/268 |
| Put Purple Cup on White Plate | Category Clustering | 48/67 | 12/67 | 48/67 | 43/67 | 24/67 |
Table R-16. Success counts and evaluation trials for language-following tasks in the DROID setting. Each model is evaluated over 300 trials across the seven tasks. Pick Bread and Arrange Cube contribute to multiple metrics using the same trial counts.
| Task | Metric | SemAE | GR00T | π0.5 | π0 | π0-FAST |
|---|---|---|---|---|---|---|
| Flip Bottle | Affordance Identification | 18/25 | 14/25 | 18/25 | 15/25 | 17/25 |
| Add Ingredients | Affordance Identification | 19/25 | 14/25 | 19/25 | 16/25 | 18/25 |
| Cover the Pot | Affordance Identification | 20/25 | 15/25 | 19/25 | 17/25 | 19/25 |
| Put Spoon | Affordance Identification | 21/25 | 16/25 | 20/25 | 18/25 | 19/25 |
| Put Bowl | Affordance Identification | 21/25 | 17/25 | 21/25 | 19/25 | 20/25 |
| Pick Bread | Affordance Identification / Category Clustering | 97/100 | 74/100 | 85/100 | 81/100 | 87/100 |
| Arrange Cube | Affordance Identification / Indirect Reference Handling | 54/75 | 36/75 | 49/75 | 40/75 | 50/75 |
2. Calculations for the seven results questioned in the review
Below, we show how we calculated each of the seven sets of results the reviewer questioned.
-
Average success rate for Affordance Identification. The review gives 85.3% and 79.0%. But using the rates in Table R-12(a), SemAE scores (0.87 + 0.79 + 0.82 + 0.81) / 4 = 0.8225 (82.25%), and π0.5 scores (0.84 + 0.64 + 0.77 + 0.77) / 4 = 0.7550 (75.50%). The gain is 6.75 percentage points.
-
Average success rate for Indirect Reference Handling. The review gives 87.8% and 81.1%. The rates in Table R-12(b) give (0.98 + 0.91 + 0.91 + 0.72) / 4 = 0.8800 (88.00%) for SemAE and (0.91 + 0.82 + 0.82 + 0.65) / 4 = 0.8000 (80.00%) for π0.5, a gain of 8.00 percentage points.
-
Average success rate for Category Clustering. The review gives 82.6% and 78.3%. The rates in Table R-12(c) give (0.54 + 0.79 + 0.72 + 0.97) / 4 = 0.7550 (75.50%) for SemAE and (0.54 + 0.64 + 0.72 + 0.85) / 4 = 0.6875 (68.75%) for π0.5, a gain of 6.75 percentage points.
-
Category Clustering gain on Franka. The 100.0% and 95.5% in Table 2 measure Kitchen task completion. In the language-following Kitchen trials (Table R-14), SemAE selected the correct target in 106/134 trials and π0.5 in 86/134. The difference is 100 × (106 − 86) / 134 ≈ 14.93 percentage points. Table R-12(c) gives the plotted rates of 0.79 and 0.64, giving 15 percentage points.
-
Indirect Reference Handling gains across settings. The review’s simulation estimate of about 1.75 points comes from Table 19. In the language-following trials, the Sem shifts in the four LIBERO-PRO suites (Table R-13) give (480 + 485 + 495 + 500) / 2000 = 0.98 for SemAE and (460 + 465 + 445 + 450) / 2000 = 0.91 for π0.5. In the same model order, Franka (Table R-14) gives 61/67 and 55/67 (0.91 and 0.82), WidowX 250s (Table R-15) gives 244/268 and 220/268 (0.91 and 0.82), and DROID (Table R-16) gives 54/75 and 49/75 (0.72 and 0.65). Subtracting the plotted rates in Table R-12(b) gives 7, 9, 9, and 7 percentage points, respectively.
-
Affordance Identification gain in simulation. The 1.3-point LIBERO-Object gain in Table 18 comes from task completion (99.5% − 98.2%). In the language-following trials for that suite (Table R-13), SemAE scores 435/500 and π0.5 scores 420/500. The gain in Table R-12(a) is therefore 100 × (435/500 − 420/500) = 3 percentage points.
-
Default SemAE scores in Tables 5 to 8. The ablations use a subset of the tasks and subtasks in Table 3, with the same SemAE checkpoint and target-selection criterion as Figure 5. Table R-17 identifies the six tasks or subtasks and their 276 distinct trials. We give each selected task or subtask equal weight within its metric.
- Affordance Identification. On DROID, Put Spoon, Put Bowl, and Arrange Cube (Blue) give (21/25 + 21/25 + 15/25) / 3 = 0.76.
- Indirect Reference Handling. Put Green Cup on Pink Cloth on WidowX 250s (cup starts at the bottom of the sink) and the same DROID Arrange Cube (Blue) trials give (47/67 + 15/25) / 2 ≈ 0.6507, reported as 0.65.
- Category Clustering. Kitchen Pot on Franka and Put Purple Cup on White Plate on WidowX 250s give (63/67 + 48/67) / 2 ≈ 0.8284, reported as 0.83.
Tables 5, 6, and 8 report these three scores. Averaging the three unrounded values gives approximately 0.7464, reported as 0.75 for Language Following in Tables 5 and 7.
Table R-17. Task mapping and success counts for the default SemAE language-following ablations in Tables 5 to 8.
| Setting | Task or subtask | Language-following metric(s) | SemAE successes / trials |
|---|---|---|---|
| DROID | Put Spoon | Affordance Identification | 21/25 |
| DROID | Put Bowl | Affordance Identification | 21/25 |
| DROID | Arrange Cube (Blue) | Affordance Identification, Indirect Reference Handling | 15/25 |
| WidowX 250s | Put Green Cup on Pink Cloth (cup starts at the bottom of the sink) | Indirect Reference Handling | 47/67 |
| WidowX 250s | Put Purple Cup on White Plate | Category Clustering | 48/67 |
| Franka | Kitchen Pot | Category Clustering | 63/67 |
Thank you for your careful review. We noticed that you updated the new version of review comments. We address both versions below so that our response covers your original questions as well as your updated concerns. We begin with the calculations behind Figures 1(d) and 5. Table R-18 reports the VLM benchmark results you requested.
W1 & W2. Task mapping, success criteria, and consistency of language-following results
Author response
1. Language-following evaluation and the data behind Figures 1(d) and 5
Table 3 identifies the tasks used for each metric. The task-success columns in Tables 1, 2, 18, 19, 21, and 22 measure completion of the full manipulation task in the standard scene. For the three language-following metrics (affordance identification, indirect reference handling, category clustering) in Figure 5, we ran separate language-following trials with additional distractor objects, and all three metrics score target selection. Affordance Identification tests whether the robot selects the instructed object among manipulable alternatives using an explicit attribute such as color. Indirect Reference Handling tests whether it resolves a relational or functional description, such as the cloth whose color matches the cube. Category Clustering tests whether it selects an object belonging to the category named in the instruction. A trial counts as successful when the robot shows that it selected the instructed object. Once that choice is clear, the trial still counts even if a subsequent action fails.
Table R-11 reports the three language-following scores plotted in Figure 1(d). Table R-12 gives the success rates for each setting in Figure 5. Table R-13 (simulation), Table R-14 (Franka), Table R-15 (WidowX 250s), and Table R-16 (DROID) give the number of correct target selections and the total number of trials. Within each setting, we average the success rates of the tasks, suites, or task groups that Table 3 assigns to each metric. To obtain each score in Table R-11, we round each setting’s rate to two decimal places and then average the four rates with equal weight.
For example, the seven DROID task rates in Table R-16 give GR00T an Affordance Identification score of (14/25 + 14/25 + 15/25 + 16/25 + 17/25 + 74/100 + 36/75) / 7 ≈ 0.6086, which rounds to 0.61 in Table R-12(a). Averaging GR00T’s four setting-specific rates gives (0.62 + 0.55 + 0.27 + 0.61) / 4 = 0.5125, the GR00T entry in Table R-11.