- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PGOT distills discrete-action VLA models using an action-space Wasserstein loss, higher weights at changes in expert behavior, and teacher supervision filtered by agreement with demonstrations. The method compresses 7B/9B teachers into 0.8B students without changing student inference. Experiments on simulated UAV navigation and LIBERO-10 show improved success and three-out-of-three completion, supported by component ablations and action-distribution diagnostics.
Critical strengths
-
Using decoded action distances is a sensible improvement over treating all token errors equally. The one-dimensional transport loss is simple to compute, and the method handles the categorical gripper separately.
-
The gains appear in two different control domains. Table 1 shows that the small students retain teacher-level success while improving repeated completion, which is relevant to practical deployment.
-
The component analysis is unusually detailed. Leave-one-out ablations, paired completion counts, and measurements of distant action errors help explain where the improvements come from.
Critical weaknesses
-
All models use training seed 42. The reported standard deviations cover three decoding rounds, not independent training runs (Section 4.1). This leaves the reliability of the 2.40–2.67-point success gains unclear.
-
Pass3 measures consistency under token sampling at temperature 0.3. Without greedy decoding or a temperature sweep, it is hard to separate better control from reduced sensitivity to decoding randomness.
-
The method is sensitive to the transport weight: on LIBERO-10, success drops from 49.47% at weight 10 to 43.00% or 41.87% at weights 5 or 20 (Appendix H). The selection protocol and comparable tuning budgets for the adapted baselines are not clear enough.
Questions for authors
-
Do the gains over FKL and Shallow-π persist across independent training seeds, with uncertainty computed over both trained models and evaluation configurations?
-
Does PGOT retain its advantage under greedy decoding and across sampling temperatures, when SR and Pass3 are reported together?
-
How were the loss weights selected without using the final evaluation tasks, and what tuning budget was given to each baseline?
5 comments
Response to Reviewer r2sW (2/3)
2. Greedy decoding and temperature sweep (W2, Q2). We evaluated all methods with greedy decoding (T = 0) and at sampling temperatures T ∈ {0.3, 0.5, 0.7} (App. K, Table 13), covering the teacher, the Student SFT reference, and all baselines. Entries are SR / Pass³. "Best over T" gives each method's best value over the four settings, separately for each metric; bold marks the best student value:
| Policy | T = 0 | 0.3 | 0.5 | 0.7 | Avg. | Best over T |
|---|---|---|---|---|---|---|
| Teacher SFT | 84.89 / 74.67 | 87.56 / 78.67 | 82.00 / 70.00 | 81.33 / 64.67 | 83.94 / 72.00 | 87.56 / 78.67 |
| Student SFT | 50.67 / 41.33 | 53.33 / 38.67 | 57.33 / 42.00 | 50.67 / 33.33 | 53.00 / 38.83 | 57.33 / 42.00 |
| FKL | 82.89 / 73.33 | 85.33 / 74.67 | 84.67 / 74.00 | 84.44 / 68.67 | 84.33 / 72.67 | 85.33 / 74.67 |
| RKL | 81.78 / 74.67 | 78.00 / 68.00 | 83.11 / 74.67 | 76.67 / 62.00 | 79.89 / 69.83 | 83.11 / 74.67 |
| ABKD | 80.44 / 70.67 | 80.89 / 67.33 | 79.78 / 66.67 | 80.44 / 65.33 | 80.39 / 67.50 | 80.89 / 70.67 |
| ToDi | 74.89 / 63.33 | 75.56 / 62.00 | 78.22 / 67.33 | 76.67 / 64.67 | 76.33 / 64.33 | 78.22 / 67.33 |
| Shallow-π | 83.11 / 76.00 | 85.33 / 74.00 | 85.78 / 74.67 | 83.78 / 72.00 | 84.50 / 74.17 | 85.78 / 76.00 |
| VLA-AD | 74.00 / 62.00 | 74.89 / 58.00 | 79.33 / 65.33 | 74.22 / 56.67 | 75.61 / 60.50 | 79.33 / 65.33 |
| PGOT | 84.22 / 75.33 | 88.00 / 82.00 | 89.11 / 76.67 | 87.11 / 77.33 | 87.11 / 77.83 | 89.11 / 82.00 |
The reviewer's question is whether PGOT's gain comes only from a more concentrated action distribution that is less affected by sampling, in which case greedy decoding would remove the gap. The sweep shows that this is not the case:
- Sampling is beneficial, so greedy decoding is not a substitute. Every policy in the table reaches its highest SR at T = 0.3 or 0.5, not under greedy decoding (e.g., Teacher SFT 84.89 → 87.56, FKL 82.89 → 85.33, Shallow-π 83.11 → 85.78, PGOT 84.22 → 89.11). Moderate sampling is therefore the better operating regime for these policies. Switching to greedy decoding would not close the gap: it lowers the SR of every method relative to its best setting.
- PGOT leads without sampling. Under greedy decoding, PGOT has the highest student SR (84.22 vs. 83.11 for Shallow-π and 82.89 for FKL), and its Pass³ is on par with the best baseline (75.33 vs. 76.00, one task out of 150). Part of the gain therefore comes from better most-likely actions, not only from robustness to sampling.
- PGOT leads at every sampling temperature and under each method's best setting. It has the best student SR and Pass³ at T = 0.3, 0.5, and 0.7. With the best temperature chosen separately for each method and metric, PGOT leads the best baseline by 3.33 SR and 6.00 Pass³ points, so its advantage cannot be removed by changing the decoding configuration of the baselines.
- PGOT keeps the benefit of sampling while reducing its downside. Sampling raises SR, but it occasionally executes actions far from the most likely one. Geometry reduces the student's probability on such actions (17.26% less mass at 32 or more bins from the expert, §4.4.1). PGOT therefore gains more from sampling than the two strongest baselines: +4.89 SR from greedy decoding to its best temperature, against +2.44 for FKL and +2.67 for Shallow-π. Robustness to sampling is thus a desirable property of the learned action distribution: the policy benefits from sampling without the failures that far-off sampled actions would cause.
Finally, Pass³ remains below SR at T = 0 because the simulator is not fully deterministic: images rendered at the same pose differ slightly between runs, so repeated greedy executions can diverge. We revised the description of Pass³ in §4.1, which previously attributed the gap between SR and Pass³ to decoding noise alone.
(continued in the next comment)
Response to Reviewer r2sW (3/3)
3. Loss-weight selection and tuning budgets (W3, Q3).
- Selection. For PGOT, we searched the weight of its core component, Geometry (w₁), over {5, 10, 20} using a single training seed (42) and decoding temperature (0.3). The same value, w₁ = 10, is best in both domains and is used throughout (App. H).
- Protocol. All methods share data, learning rate, batch size (128), and number of steps (9,785 on CoNav-UAV and 10,000 on LIBERO-10). They are evaluated at their final checkpoints without early stopping or checkpoint selection (§4.1).
- Equal budget. To ensure a fair comparison, we gave the two strongest baselines the same search budget under this protocol: one key coefficient each, searched over three values (new Table 3, App. C):
| Setting | SR | Pass³ | CR | MMBC |
|---|---|---|---|---|
| FKL, λ_FKL = 1 (default) | 85.33 | 74.67 | 11.78 | 5.27 |
| FKL, λ_FKL = 2 / 3 | 73.56 / 75.78 | 54.67 / 58.67 | 26.22 / 23.56 | 3.54 / 3.72 |
| Shallow-π, λ_attn = 1 (default) | 85.33 | 74.00 | 10.89 | 5.09 |
| Shallow-π, λ_attn = 2 / 3 | 70.00 / 73.56 | 53.33 / 59.33 | 29.56 / 25.33 | 3.52 / 3.81 |
| PGOT, w₁ = 10 (default) | 88.00 | 82.00 | 8.00 | 5.72 |
| PGOT, w₁ = 5 / 20 | 82.44 / 83.11 | 71.33 / 75.33 | 11.33 / 11.56 | 5.41 / 5.46 |
- The comparison uses matched tuning budgets. Both baselines perform best at their default settings, which are those reported in the main table. Table 1 therefore compares the best of three settings for PGOT with the best of three for each baseline, and PGOT leads on every metric shown.
- The advantage holds in experiments not used to select w₁. The seed and temperature studies in our responses 1 and 2 above were run afterwards with w₁ = 10 unchanged. PGOT still achieves the highest student SR under seeds 43 and 44 and at T ∈ {0, 0.5, 0.7}, so its advantage is not limited to the setting in which w₁ was selected.
- The sensitivity is not specific to PGOT. Doubling the coefficient reduces SR by 4.89 points for PGOT, against 11.77 for FKL and 15.33 for Shallow-π. The drop observed for w₁ therefore reflects a general property of this setting, in which a compact student must balance expert supervision against teacher supervision, and PGOT is the least sensitive of the three objectives (App. H).
We hope these additional experiments address your concerns about the reliability of our results. If so, we would be grateful if you would consider raising your rating, and we are happy to provide any further details.
Response to Reviewer r2sW (1/3)
We thank the reviewer for the detailed review. We appreciate the recognition that using decoded action distances is a sensible improvement over treating all token errors equally, that the gains appear in two different control domains with relevance to practical deployment, and that our component analysis is unusually detailed. The questions on training seeds, decoding, and hyperparameter selection are important for the reliability of our conclusions, and we address each with new experiments below.
1. Independent training seeds (W1, Q1). We retrained PGOT, FKL, and Shallow-π on CoNav-UAV with seeds 43 and 44 (App. J, Table 12). The aggregate SR uncertainty is computed over all nine runs (3 trained models × 3 evaluation rounds), so it covers both training and evaluation variation.
(continued in the next comment)