- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents MEF-VLA, which introduces motion-to-event foresight as an auxiliary learning objective for VLA policies. A kinematic detector identifies upcoming interaction events in demonstrations, and dense point tracking constructs displacement fields from the current observation to the next event. Features of these remaining-motion fields supervise learned event queries, which also condition the action expert. Event detection, tracking, and target construction are performed offline, with only the event queries retained at inference. Experiments include LIBERO, zero-shot LIBERO-Plus, CALVIN D→D, and three physical task families. The paper reports improvements over a matched vanilla VLA and ablations separating event-aligned horizons from remaining-motion supervision.
Critical strengths
-
The contribution is clearly formulated and supported by meaningful matched ablations. The action-horizon variant retains motion supervision, while the event-frame variant retains event alignment. The comparisons provide direct evidence for both design choices, beyond a comparison with an action-only baseline.
-
CALVIN five-task success improves from 58.1% to 67.1%. On held-out three-stage physical instructions, success increases from 31/90 to 53/90 trials, with improvements in all three task families. The reported inference overhead is approximately 1.1%, supporting practical use of the training objective.
Critical weaknesses
-
The long-horizon contribution is not isolated as thoroughly as the LIBERO contribution. The event-horizon and event-frame ablations are reported on LIBERO and LIBERO-Plus, whereas CALVIN and longer physical compositions are compared mainly against the vanilla VLA. Thus, the longer-composition gains do not establish whether event alignment and remaining-motion supervision are both necessary, rather than predictive auxiliary supervision more generally.
-
The reported training configuration uses a single seed, and several margins over prior methods are small. Event validation uses one annotator and a ±1 s matching tolerance on LIBERO. Detector sensitivity is evaluated through annotation alignment rather than downstream policy success, leaving uncertainty about how false events or interactions without a pronounced pause affect learning.
Questions for authors
-
Can the authors report the action-horizon and event-frame variants on CALVIN and held-out three-stage physical instructions? These results would determine whether the proposed target construction specifically explains the longer-composition gains.
-
Are the matched policy improvements stable across training seeds? Please provide uncertainty that accounts for task or trajectory variation, particularly for the relatively small standard-LIBERO margins.
-
How does downstream success change when event-detector parameters are varied or event references are perturbed? Does event alignment remain accurate on CALVIN and the physical demonstrations, including interactions without clear motion pauses?
5 comments
Response to Reviewer JAGe (2/3)
W2 / Q2: Training seeds and task-level uncertainty
MEF-VLA results across training seeds
| Vanilla VLA | MEF-VLA, seed 42 | Seed 123 | Seed 2026 | MEF-VLA mean ± SD | |
|---|---|---|---|---|---|
| LIBERO Avg. (%) | 96.5 | 98.10 | 98.05 | 97.75 | 98.0 ± 0.2 |
| LIBERO-Plus Total (%) | 69.3 | 78.2 | 77.0 | 75.8 | 77.0 ± 1.2 |
The three full-model runs share the evaluation seed and instances within each benchmark. Each run scores above the seed-42 results for Vanilla VLA, Queries only, A1, and A2 on both benchmarks.
For task-level uncertainty, we use a paired bootstrap with 2,000 resamples (training seed 42): on LIBERO, it resamples tasks within each suite and rollouts within each task; on LIBERO-Plus, it resamples original tasks together with all of their perturbed instances.
| MEF-VLA minus | LIBERO, points [95% CI] | LIBERO-Plus, points [95% CI] |
|---|---|---|
| Queries only | +1.00 [−0.20, 2.20] | +2.52 [0.84, 4.02] |
| A1 (fixed-horizon flow) | +1.25 [−0.05, 2.50] | +4.90 [2.02, 8.08] |
| A2 (event-frame RGB) | +0.75 [−0.45, 2.30] | +4.52 [2.36, 6.75] |
All three LIBERO-Plus intervals exclude zero, and the margins over A1 and A2 also hold across training seeds.
Margins over prior methods. We agree that the margins over prior methods on LIBERO are small, and these cross-paper comparisons also differ in pretraining and training budget. The baselines added for Reviewer CkRP show this dependence: π₀.₅ exceeds MEF-VLA on LIBERO-Plus with its robot pretraining (83.0% vs. 78.2%), but trained with its official configuration and without robot pretraining, it reaches 54.0%. We therefore base our claims on the controlled comparisons above and on applying the full MEF method to FLOWER, which raises its CALVIN D→D Avg. Len. from 4.324 to 4.412.
Response to Reviewer JAGe (3/3)
W2 / Q3: Event references
Downstream sensitivity. We retrained MEF-VLA on LIBERO at training seed 42 using two alternative speed quantiles, and . We evaluated the resulting policies on LIBERO and zero-shot LIBERO-Plus.
| Event references | Missed (of 200) | False events | LIBERO Avg. | LIBERO-Plus Total |
|---|---|---|---|---|
| 36 | 36 | 97.80 | 76.03 | |
| (submitted) | 17 | 53 | 98.10 | 78.22 |
| 14 | 68 | 97.75 | 75.73 | |
| , range over three training seeds | 17 | 53 | 97.75–98.10 | 75.75–78.22 |
Both settings change the numbers of missed and false detections. Relative to , the largest drops are 0.35 percentage points on LIBERO and 2.49 points on LIBERO-Plus, close to the score ranges across the three full-model runs. Both settings still score above A1 and A2 on both benchmarks.
Cross-dataset alignment. On LIBERO, the mean absolute time difference across matched reference–annotation pairs is 0.29 s (Appendix D). Using the same protocol, the primary annotator also labeled 40 CALVIN demonstrations (stratified by subtask type) and 30 real-world demonstrations (10 per task family), marking interactions and flagging those without a pronounced motion pause. Precision, recall, and coverage use a ±1 s matching tolerance, and F1 is reported at both ±0.5 s and ±1 s.
Cross-dataset alignment with primary annotations
| Dataset | Demos | Interactions | Precision | Recall | F1 ±0.5 s | F1 ±1 s | Coverage ±1 s |
|---|---|---|---|---|---|---|---|
| LIBERO | 80 | 200 | 77.5 | 91.5 | 75.2 | 83.9 | 97.5 |
| CALVIN | 40 | 54 | 72.5 | 92.6 | 81.3 | 81.3 | 92.6 |
| Real world | 30 | 98 | 63.4 | 93.9 | 72.4 | 75.7 | 93.9 |
At ±1 s, recall exceeds 91% on all three datasets. The kinematic criterion limits detection of interactions without a pronounced pause, with 4 of 19 recovered, including 1 of 5 on CALVIN and 1 of 4 in the real-world demonstrations. These interactions account for 5.4% of the 352 annotations. Extending event detection to such interactions remains future work (Sec. 5).
Agreement with a second annotator. A second annotator independently relabeled 42 demonstrations (20 LIBERO, 10 CALVIN, and 12 real-world). At ±0.5 s on this subset, F1 is 87.8% between the two annotators and 84.4% between the detector and the second annotator.
Revisions. (i) Sec. 4.4 reports the CALVIN and held-out three-stage target ablations (Tables 4 and 20). The Introduction summarizes these results, and the Abstract updates the ablation takeaway. (ii) Appendix C.4 reports results across three full-model training seeds and paired intervals for the seed-42 comparisons (Tables 13–14). (iii) Sec. 4.2 discusses differences in pretraining and training budget across Tables 1–3. (iv) Appendix D reports cross-dataset detector validation, F1 at ±0.5 s, second-annotator agreement, and downstream policy success after retraining with two alternative detector settings (Table 19).
We thank the reviewer for these questions, which clarified which components carry the long-horizon gains and how detector errors affect learning. We hope the new evidence addresses the reviewer's concerns and would be grateful if the reviewer would consider raising the score.
Response to Reviewer JAGe (1/3)
We thank the reviewer for the thorough assessment and for recognizing the matched ablations and the multi-stage results. In short, removing either event alignment or remaining-motion supervision lowers success on CALVIN and on the held-out real-world instructions; the margins over these ablations hold across training seeds and tasks on LIBERO-Plus; and detector errors in the tested range change success by an amount comparable to the variation across training seeds.
W1 / Q1: A1 and A2 on CALVIN and held-out three-stage instructions
A1 (action horizon: H=8 on CALVIN, H=16 on the robot) and A2 (event frame) use the full model's configuration on each benchmark. On CALVIN, all policies are evaluated on the official 1,000 five-task sequences with the same evaluation seed. On the robot, the queries-only control, A1, and A2 use the same one- and two-stage training demonstrations and real-time chunking (RTC) settings as MEF-VLA and Vanilla VLA. All five policies are evaluated on the same held-out three-stage instructions and initial configurations, with 30 trials per task family.
CALVIN D→D ablations
Held-out three-stage real-world ablations
Both components contribute to the longer-composition gains. Replacing the event-aligned horizon (A1) lowers CALVIN Avg. Len. by 0.081 and three-stage success by 9 of 90 trials; replacing remaining event-flow with event-frame features (A2) lowers them by 0.187 and 10 trials. MEF-VLA leads both variants in every task family. On CALVIN, its margin over A1 is 0.6 points on the first task and 3.4 points on five tasks, and relative to the queries-only control, event-frame prediction adds 0.030 in Avg. Len., fixed-horizon flow adds 0.136, and remaining event-flow adds 0.217.