acceptodds
Under review as a conference paper at ICLR 2027

MEF-VLA: Motion-to-Event Foresight for Vision-Language-Action Models

Abstract

Vision-language-action (VLA) models have advanced robot manipulation, and predictive world modeling further supports action learning through future-state prediction. Despite these advances, policies can still deviate from the instructed sequence or fail to complete individual interactions in multi-stage tasks. To improve the guidance provided by visual foresight, we revisit which future to predict and what to represent. A fixed prediction horizon can end during intermediate motion or extend beyond the next interaction, resulting in inconsistent guidance for task progress. Even with a reference aligned with task progress, complete future-frame targets retain static scene appearance while leaving the spatial change from the current state implicit. We introduce MEF-VLA (Motion-to-Event Foresight), which learns the motion remaining until the next interaction event. Event frames marking upcoming interactions define adaptive prediction horizons. Remaining event-flow represents the motion toward these shared references as state-dependent spatial displacement. Learned event queries predict remaining-motion features and condition the action expert, adding negligible inference overhead. In simulation, MEF-VLA improves success rates over the vanilla VLA by 1.6 percentage points on LIBERO, 8.9 on zero-shot LIBERO-Plus, and 9.0 on five-task sequences in CALVIN DD. Across three real-world task families, MEF-VLA trained on one- and two-stage instructions achieves 58.9% average success on held-out three-stage instructions, compared with 34.4% for the vanilla VLA. Ablations further demonstrate the benefits of both event-aligned horizons and remaining-motion supervision for task success and zero-shot robustness.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Visual Foresight, Multi-Stage Manipulation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_JAGeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MEF-VLA, which introduces motion-to-event foresight as an auxiliary learning objective for VLA policies. A kinematic detector identifies upcoming interaction events in demonstrations, and dense point tracking constructs displacement fields from the current observation to the next event. Features of these remaining-motion fields supervise learned event queries, which also condition the action expert. Event detection, tracking, and target construction are performed offline, with only the event queries retained at inference. Experiments include LIBERO, zero-shot LIBERO-Plus, CALVIN D→D, and three physical task families. The paper reports improvements over a matched vanilla VLA and ablations separating event-aligned horizons from remaining-motion supervision.

Critical strengths
  1. The contribution is clearly formulated and supported by meaningful matched ablations. The action-horizon variant retains motion supervision, while the event-frame variant retains event alignment. The comparisons provide direct evidence for both design choices, beyond a comparison with an action-only baseline.

  2. CALVIN five-task success improves from 58.1% to 67.1%. On held-out three-stage physical instructions, success increases from 31/90 to 53/90 trials, with improvements in all three task families. The reported inference overhead is approximately 1.1%, supporting practical use of the training objective.

Critical weaknesses
  1. The long-horizon contribution is not isolated as thoroughly as the LIBERO contribution. The event-horizon and event-frame ablations are reported on LIBERO and LIBERO-Plus, whereas CALVIN and longer physical compositions are compared mainly against the vanilla VLA. Thus, the longer-composition gains do not establish whether event alignment and remaining-motion supervision are both necessary, rather than predictive auxiliary supervision more generally.

  2. The reported training configuration uses a single seed, and several margins over prior methods are small. Event validation uses one annotator and a ±1 s matching tolerance on LIBERO. Detector sensitivity is evaluated through annotation alignment rather than downstream policy success, leaving uncertainty about how false events or interactions without a pronounced pause affect learning.

Questions for authors
  1. Can the authors report the action-horizon and event-frame variants on CALVIN and held-out three-stage physical instructions? These results would determine whether the proposed target construction specifically explains the longer-composition gains.

  2. Are the matched policy improvements stable across training seeds? Please provide uncertainty that accounts for task or trajectory variation, particularly for the relatively small standard-LIBERO margins.

  3. How does downstream success change when event-detector parameters are varied or event references are perturbed? Does event alignment remain accurate on CALVIN and the physical demonstrations, including interactions without clear motion pauses?

5 comments
AuthorsOpenReview ↗

Response to Reviewer JAGe (1/3)

We thank the reviewer for the thorough assessment and for recognizing the matched ablations and the multi-stage results. In short, removing either event alignment or remaining-motion supervision lowers success on CALVIN and on the held-out real-world instructions; the margins over these ablations hold across training seeds and tasks on LIBERO-Plus; and detector errors in the tested range change success by an amount comparable to the variation across training seeds.

W1 / Q1: A1 and A2 on CALVIN and held-out three-stage instructions

A1 (action horizon: H=8H=8 on CALVIN, H=16H=16 on the robot) and A2 (event frame) use the full model's configuration on each benchmark. On CALVIN, all policies are evaluated on the official 1,000 five-task sequences with the same evaluation seed. On the robot, the queries-only control, A1, and A2 use the same one- and two-stage training demonstrations and real-time chunking (RTC) settings as MEF-VLA and Vanilla VLA. All five policies are evaluated on the same held-out three-stage instructions and initial configurations, with 30 trials per task family.

CALVIN D→D ablations

CALVIN D→D12345Avg. Len.
Vanilla VLA89.879.971.864.958.13.645
Queries only (λMEF=0\lambda_{\mathrm{MEF}}=0)89.881.173.267.761.13.729
A2 (event-frame RGB)88.780.374.569.662.83.759
A1 (fixed-horizon flow)90.983.277.171.663.73.865
MEF-VLA91.584.477.973.767.13.946

Held-out three-stage real-world ablations

Real world, three-stagePill SortingSandwich MakingCup StackingTotal
Vanilla VLA7/307/3017/3031/90
Queries only11/308/3021/3040/90
A1 (fixed-horizon flow)11/3010/3023/3044/90
A2 (event-frame RGB)12/3010/3021/3043/90
MEF-VLA15/3013/3025/3053/90

Both components contribute to the longer-composition gains. Replacing the event-aligned horizon (A1) lowers CALVIN Avg. Len. by 0.081 and three-stage success by 9 of 90 trials; replacing remaining event-flow with event-frame features (A2) lowers them by 0.187 and 10 trials. MEF-VLA leads both variants in every task family. On CALVIN, its margin over A1 is 0.6 points on the first task and 3.4 points on five tasks, and relative to the queries-only control, event-frame prediction adds 0.030 in Avg. Len., fixed-horizon flow adds 0.136, and remaining event-flow adds 0.217.

AuthorsOpenReview ↗

Response to Reviewer JAGe (2/3)

W2 / Q2: Training seeds and task-level uncertainty

MEF-VLA results across training seeds

Vanilla VLAMEF-VLA, seed 42Seed 123Seed 2026MEF-VLA mean ± SD
LIBERO Avg. (%)96.598.1098.0597.7598.0 ± 0.2
LIBERO-Plus Total (%)69.378.277.075.877.0 ± 1.2

The three full-model runs share the evaluation seed and instances within each benchmark. Each run scores above the seed-42 results for Vanilla VLA, Queries only, A1, and A2 on both benchmarks.

For task-level uncertainty, we use a paired bootstrap with 2,000 resamples (training seed 42): on LIBERO, it resamples tasks within each suite and rollouts within each task; on LIBERO-Plus, it resamples original tasks together with all of their perturbed instances.

MEF-VLA minusLIBERO, points [95% CI]LIBERO-Plus, points [95% CI]
Queries only+1.00 [−0.20, 2.20]+2.52 [0.84, 4.02]
A1 (fixed-horizon flow)+1.25 [−0.05, 2.50]+4.90 [2.02, 8.08]
A2 (event-frame RGB)+0.75 [−0.45, 2.30]+4.52 [2.36, 6.75]

All three LIBERO-Plus intervals exclude zero, and the margins over A1 and A2 also hold across training seeds.

Margins over prior methods. We agree that the margins over prior methods on LIBERO are small, and these cross-paper comparisons also differ in pretraining and training budget. The baselines added for Reviewer CkRP show this dependence: π₀.₅ exceeds MEF-VLA on LIBERO-Plus with its robot pretraining (83.0% vs. 78.2%), but trained with its official configuration and without robot pretraining, it reaches 54.0%. We therefore base our claims on the controlled comparisons above and on applying the full MEF method to FLOWER, which raises its CALVIN D→D Avg. Len. from 4.324 to 4.412.

AuthorsOpenReview ↗

Response to Reviewer JAGe (3/3)

W2 / Q3: Event references

Downstream sensitivity. We retrained MEF-VLA on LIBERO at training seed 42 using two alternative speed quantiles, q=0.15q=0.15 and q=0.25q=0.25. We evaluated the resulting policies on LIBERO and zero-shot LIBERO-Plus.

Event referencesMissed (of 200)False eventsLIBERO Avg.LIBERO-Plus Total
q=0.15q=0.15363697.8076.03
q=0.20q=0.20 (submitted)175398.1078.22
q=0.25q=0.25146897.7575.73
q=0.20q=0.20, range over three training seeds175397.75–98.1075.75–78.22

Both settings change the numbers of missed and false detections. Relative to q=0.20q=0.20, the largest drops are 0.35 percentage points on LIBERO and 2.49 points on LIBERO-Plus, close to the score ranges across the three full-model runs. Both settings still score above A1 and A2 on both benchmarks.

Cross-dataset alignment. On LIBERO, the mean absolute time difference across matched reference–annotation pairs is 0.29 s (Appendix D). Using the same protocol, the primary annotator also labeled 40 CALVIN demonstrations (stratified by subtask type) and 30 real-world demonstrations (10 per task family), marking interactions and flagging those without a pronounced motion pause. Precision, recall, and coverage use a ±1 s matching tolerance, and F1 is reported at both ±0.5 s and ±1 s.

Cross-dataset alignment with primary annotations

DatasetDemosInteractionsPrecisionRecallF1 ±0.5 sF1 ±1 sCoverage ±1 s
LIBERO8020077.591.575.283.997.5
CALVIN405472.592.681.381.392.6
Real world309863.493.972.475.793.9

At ±1 s, recall exceeds 91% on all three datasets. The kinematic criterion limits detection of interactions without a pronounced pause, with 4 of 19 recovered, including 1 of 5 on CALVIN and 1 of 4 in the real-world demonstrations. These interactions account for 5.4% of the 352 annotations. Extending event detection to such interactions remains future work (Sec. 5).

Agreement with a second annotator. A second annotator independently relabeled 42 demonstrations (20 LIBERO, 10 CALVIN, and 12 real-world). At ±0.5 s on this subset, F1 is 87.8% between the two annotators and 84.4% between the detector and the second annotator.

Revisions. (i) Sec. 4.4 reports the CALVIN and held-out three-stage target ablations (Tables 4 and 20). The Introduction summarizes these results, and the Abstract updates the ablation takeaway. (ii) Appendix C.4 reports results across three full-model training seeds and paired intervals for the seed-42 comparisons (Tables 13–14). (iii) Sec. 4.2 discusses differences in pretraining and training budget across Tables 1–3. (iv) Appendix D reports cross-dataset detector validation, F1 at ±0.5 s, second-annotator agreement, and downstream policy success after retraining with two alternative detector settings (Table 19).

We thank the reviewer for these questions, which clarified which components carry the long-horizon gains and how detector errors affect learning. We hope the new evidence addresses the reviewer's concerns and would be grateful if the reviewer would consider raising the score.

Reviewer_CkRPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a visual foresight method for VLA models. Alongside action prediction, the model predicts a representation of the remaining motion from the current observation to the next interaction event as additional supervision. The learned event queries also condition action generation. The method is evaluated on three simulation benchmarks and three real-world task families.

Critical strengths
  1. A useful decomposition of visual foresight. The authors decompose visual prediction into “which future to predict” and “what to represent,” namely, which future time point to predict and what information to predict at that time. I find this decomposition reasonable and insightful. It provides a useful perspective for understanding predictive supervision in VLA models and motivates examining the prediction horizon and representation separately.
Critical weaknesses
  1. The choice of the next event as the prediction endpoint is not sufficiently justified. I am not fully convinced that a variable event horizon provides appropriate guidance for a fixed-length action chunk. For example, if the next event occurs after 8 steps within a 16-step chunk, the remaining event-flow does not explicitly describe the motion needed after that event. Conversely, the event may lie beyond the action horizon. Although the full chunk is supervised by the action loss, the paper does not sufficiently explain or evaluate how event-based guidance handles this temporal mismatch.

  2. The baseline comparisons are incomplete. Table 2 includes only three external baselines and omits competitive methods such as FLOWER [1], which reports results on the same CALVIN D→D setting. Table 3 also lacks comparisons with π₀.₅ and Cosmos Policy [2]. These comparisons would help establish how competitive MEF-VLA is on long-horizon execution and robustness, beyond its improvement over the authors’ vanilla VLA.

References

[1] Moritz Reuss et al. FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Flow Models. CoRL, 2025.

[2] Moo Jin Kim et al. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. ICLR, 2026.

Questions for authors
  1. Clarification and additional evidence on the temporal mismatch. Could the authors explain in more detail or provide experimental evidence showing how event-based guidance remains effective when the event horizon and the action-chunk horizon are temporally mismatched? This would help determine whether event-based guidance remains beneficial when the event horizon is shorter or longer than the action-chunk horizon.

  2. Presentation suggestions. In the right panel of Figure 1 (“What Future to Predict?”), please include the corresponding current observation to make the Remaining Event-flow easier to understand. Table 1 should also be self-contained: please explain the meanings of bold and underlined values in its caption, even though these conventions are described in the main text.

5 comments
AuthorsOpenReview ↗

Response to Reviewer CkRP (1/3)

We thank the reviewer for the careful review and for finding the distinction between which future to predict and what to represent useful. In short, as the reviewer notes, the action loss supervises the full chunk; in addition, actions after an in-chunk event are predicted in later training samples with query states trained toward the following event, and an event beyond the chunk supplies the spatial target that the chunk advances toward. Our experiments show that event-based guidance remains beneficial whether the event horizon is shorter or longer than the action horizon, and applying the full MEF method to FLOWER, the strongest CALVIN D→D baseline, also improves it.

W1 / Q1: How event-based guidance handles the temporal mismatch

Event within the chunk (Δt<H\Delta_t<H). In the reviewer's example, the event occurs at step 8 of a 16-step chunk, and Laction\mathcal{L}_{\mathrm{action}} supervises steps 9–16. These post-event actions also appear in the samples that start at or after the event, where the action expert predicts them conditioned on query states trained on the remaining motion to the following event (Eq. (1)); every action is thus predicted under its own upcoming event in some training sample. The steps immediately after the event also involve little motion (App. A.4). Ending this sample's target at the event keeps it informative: the displacement to the chunk endpoint would sum the approach and the post-event motion, which can reverse direction (reaching down to grasp, then lifting), whereas the displacement to the event covers the approach alone.

Event beyond the chunk (Δt>H\Delta_t>H). The auxiliary target is the remaining net displacement to the event, whose direction and magnitude FlowRGB encodes with one dataset-level scale (App. A.4), and the demonstrated actions determine how far the chunk advances toward it. A target at the chunk endpoint would instead lie a fixed number of steps ahead, so where it falls within the current phase varies with the demonstrator's speed. The next event is defined by each trajectory's own kinematics, so observations within a phase share it as their reference, and its horizon adapts to execution speed (Sec. 3.2).

The query states serve the full chunk. Both losses update the event queries and the shared VLM (Sec. 3.4): LMEF\mathcal{L}_{\mathrm{MEF}}, with a small weight (λMEF=0.05\lambda_{\mathrm{MEF}}=0.05 on LIBERO), trains the query states to encode the motion toward the next event, and Laction\mathcal{L}_{\mathrm{action}}, back-propagated through the action expert, trains them on the full chunk. Each action step has its own position embedding and cross-attends to the image and instruction tokens and the query states, so training determines how the reference is used at each chunk position. Related methods also decouple the reference horizon from the chunk: CoT-VLA [4] predicts a subgoal image 1–24 frames ahead (dataset-specific ranges) before a 10-step chunk, HDP [5] generates 16-step action sequences toward the predicted next contact, and PerAct [6] assigns each observation to its next keyframe, found by a velocity heuristic.

[4] Zhao et al. CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models. CVPR 2025.
[5] Wang et al. Hierarchical Diffusion Policy: Manipulation Trajectory Generation via Contact Guidance. IEEE T-RO 2025.
[6] Shridhar et al. Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation. CoRL 2022.

AuthorsOpenReview ↗

Response to Reviewer CkRP (2/3)

Event-based guidance remains beneficial in both cases. Both cases are common in training: among samples with a complete action window, 26.7% and 71.3% of LIBERO samples (H=16H=16) have Δt<H\Delta_t<H and Δt>H\Delta_t>H, and 72.6% of CALVIN samples (H=8H=8) have Δt>H\Delta_t>H. We compare MEF-VLA with the queries-only control, which has no auxiliary target, and, to examine each case separately, retrain MEF-VLA with the chunk endpoint ht=min⁡(t+H,T−1)h_t=\min(t+H,T-1) as the target for one class of samples at a time, keeping HH, the queries, decoder, training settings, training seed, and evaluation instances fixed:

  • Extended to HH, max⁡(τ(t),ht)\max(\tau(t),h_t): samples whose event falls inside the chunk use the chunk endpoint, so their target covers the full chunk, including the post-event motion in the reviewer's example.
  • Capped at HH, min⁡(τ(t),ht)\min(\tau(t),h_t): samples whose event lies beyond the chunk use the chunk endpoint.
  • A1 uses the chunk endpoint for both classes.
Training targetEvent used as target forLIBERO Avg.LIBERO-Plus Total
None (queries only)–97.1075.69
Next event (MEF-VLA)both cases98.1078.22
Extended to HHΔt>H\Delta_t>H only96.9574.56
Capped at HHΔt<H\Delta_t<H only96.9075.68
Chunk endpoint (A1)neither96.8573.3

MEF-VLA exceeds the queries-only control by 1.0 points on LIBERO and 2.5 points on LIBERO-Plus (95% CI over tasks: 0.8 to 4.0). Replacing the event with the chunk endpoint in either case lowers success on both benchmarks: by 1.15 and 1.20 points on LIBERO, below all three training seeds of MEF-VLA (97.75–98.10%), and by 3.7 and 2.5 points on LIBERO-Plus. Event-based guidance therefore remains beneficial whether the event horizon is shorter or longer than the action horizon.

Other chunk lengths. Extended and Capped were run on LIBERO; on CALVIN and the real-world tasks, we compare with the queries-only control and A1:

EvaluationHHQueries onlyChunk endpoint (A1)Next event (MEF-VLA)
CALVIN D→D Avg. Len.83.7293.8653.946
Real world, three-stage successes (of 90)16404453

The full MEF method also improves FLOWER, which uses H=10H=10 (comment 3/3).

AuthorsOpenReview ↗

Response to Reviewer CkRP (3/3)

W2: Additional baselines

π₀.₅ and Cosmos Policy use their official LIBERO checkpoints and inference settings on the same 10,030 LIBERO-Plus instances as MEF-VLA. The models we train for this comparison (π₀.₅-scratch, FLOWER, and FLOWER + MEF) use training seed 42, the seed of the submitted models and of all our controls.

CALVIN D→D comparisons

CALVIN D→D12345Avg. Len.
FLOWER + MEF96.292.688.584.379.64.412
FLOWER [1], published97.492.486.981.374.94.35 ± 0.02
FLOWER, our reproduction98.293.587.680.572.64.324
MEF-VLA91.584.477.973.767.13.946
π₀.₅ [3]92.584.076.671.064.43.885

Zero-shot LIBERO-Plus comparisons

LIBERO-PlusCameraRobotLang.LightBackgr.NoiseLayoutTotal
π₀.₅ [3]70.272.085.294.795.485.485.683.0
Cosmos Policy [2]68.752.188.597.183.185.982.778.8
MEF-VLA61.255.787.495.696.682.479.378.2
π₀.₅-scratch41.430.162.364.661.967.056.054.0

FLOWER exceeds MEF-VLA on CALVIN D→D, and π₀.₅ and Cosmos Policy exceed it on LIBERO-Plus by 4.8 and 0.6 points; MEF-VLA is above π₀.₅ on CALVIN D→D. These three policies build on robot pretraining (FLOWER, π₀.₅) or on a pretrained video model fine-tuned at a global batch size of 1,920 (Cosmos Policy), whereas MEF-VLA starts from an image–language VLM with a randomly initialized action expert. To separate pretraining from the policy, we trained π₀.₅ from scratch on LIBERO using the official OpenPI configuration without its robot-pretrained weights (π₀.₅-scratch, last row). π₀.₅-scratch reaches 54.0% on LIBERO-Plus, 24.2 points below MEF-VLA and below it in every perturbation category, so the advantage of the official π₀.₅ checkpoint depends on robot pretraining.

MEF on the strongest CALVIN policy. We applied the full MEF method to FLOWER. FLOWER and FLOWER + MEF are both trained with FLOWER's official code, configuration, and pretrained initialization (H=10H=10), select the checkpoint by online rollouts, and are evaluated on the official 1,000 sequences with the same evaluation seed. MEF raises FLOWER's Avg. Len. from 4.324 to 4.412, above the published 4.35, and its five-task success from 72.6% to 79.6%. The method therefore transfers to a robot-pretrained policy with a different architecture and chunk length.

Q2: Figure 1 and Table 1. The right panel of Figure 1 now shows the paired current and event frames alongside the resulting remaining event-flow. FlowRGB is defined in current-image coordinates. The caption explains that hue encodes displacement direction and brightness encodes normalized magnitude. The caption of Table 1 in the revised manuscript states that the best and second-best results are bold and underlined, respectively.

Revisions. (i) Sec. 3.2 clarifies how the next-event target relates to the action chunk, and Appendix C.3 reports the training-sample statistics and the Extended and Capped controls (Table 12). (ii) Table 4 reports the CALVIN ablations, and Table 20 includes Queries only and A1 on held-out three-stage instructions. (iii) Table 2 adds FLOWER, Table 3 adds π₀.₅, π₀.₅-scratch, and Cosmos Policy, and Appendix C.5 compares FLOWER and FLOWER + MEF (Table 15). (iv) Sec. 4.2 discusses differences in pretraining and training budget across Tables 1–3. (v) Figure 1 shows the paired current frame, and its caption explains the FlowRGB encoding. The Table 1 caption explains the bold and underlined entries.

We thank the reviewer for these questions, which led to a clearer account of how the event reference guides fixed-length chunks and where MEF stands among current policies. We hope the new evidence addresses the reviewer's concerns and would be grateful if the reviewer would reconsider the rating.

[1] Reuss et al. FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Flow Models. CoRL 2025.
[2] Kim et al. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. ICLR 2026.
[3] Physical Intelligence et al. π₀.₅: a Vision-Language-Action Model with Open-World Generalization. CoRL 2025.

Reviewer_gK7FOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MEF-VLA, which adds a training-time auxiliary prediction objective to a VLA policy (Qwen3-VL-4B backbone with a GR00T-style DiT flow-matching action expert). Instead of predicting the future at a fixed horizon, the model predicts the motion remaining from the current observation to the next interaction event.

  • Events are detected offline as onsets of persistent low-speed intervals in the robot pose trajectory.
  • The remaining motion is the net current-to-event displacement estimated by a frozen dense tracker (AllTracker). It is encoded as a FlowRGB image and embedded with a frozen DINOv2.
  • Sixteen learned event queries appended to the VLM input regress these patch features through a lightweight decoder that is used only during training.
  • The action expert conditions on the full hidden sequence, including the query states.
Critical strengths
  • Clear framing. Decomposing visual foresight into “which future” (the temporal reference) and “what to represent” (appearance vs. displacement) organizes the design space cleanly. The two ablations (A1, A2) map directly onto these two axes.
  • Practical design. The tracker, DINOv2 encoder, and decoder are all training-only; deployment adds only 16 query tokens (Table 8). Table 9 reports 118.29 → 119.55 ms per chunk. The one-time preprocessing cost is also reported (18.76 GPU-hours for LIBERO).
  • Useful supporting analyses. These include validation of the event references against 200 human annotations (F1 83.9%, coverage 97.5%) with parameter sensitivity (App. D), a failure-mode breakdown (App. E.3), and per-cell success counts (App. E.2) that allow independent verification.
Critical weaknesses
  • Attribution of the gains is incomplete. The vanilla baseline has no event queries at all. The full-vs-vanilla comparison therefore bundles three factors: (i) extra learnable tokens, (ii) the presence of any auxiliary dense-prediction objective, and (iii) the specific motion-to-event target. A1 and A2 isolate only (iii). Table 10 suggests that the generic part is substantial. On LIBERO-Plus Total, A1 and A2 already improve over vanilla by about 4 pp on average, and the full model is about 4.5 pp above the better ablation. In the two categories with the largest gains, most of the improvement is already present in the ablations. Noise: +28.6 for the full model vs. +23.7 for the ablation average. Camera: +20.2 vs. +15.4. Two controls would clarify how much the action expert actually relies on the foresight information: a model with event queries but λ_MEF = 0, and test-time masking of the query states.
  • The ablations do not cover the headline long-horizon results. A1 and A2 are evaluated only on LIBERO and LIBERO-Plus. The CALVIN and real-world results carry the paper’s main claim about multi-stage execution, but they are compared only against vanilla. It is therefore not shown that event alignment or remaining flow, rather than auxiliary foresight in general, drives the long-horizon gains.
Questions for authors
  • What are the results for (a) event queries with λ_MEF = 0 and (b) the full model with query states masked at inference?
  • How many evaluation episodes are used per LIBERO and LIBERO-Plus setting? Are results available over multiple seeds?
4 comments
AuthorsOpenReview ↗

Response to Reviewer gK7F (1/2)

We thank the reviewer for the careful analysis of Table 10 in the submitted manuscript and for proposing two controls that test it directly. In short, the reviewer is right that the query interface accounts for much of the LIBERO-Plus gain: with λMEF=0\lambda_{\mathrm{MEF}}=0, the queries alone raise LIBERO-Plus Total from 69.3% to 75.7%. At the matched training seed, remaining event-flow adds 2.5 points on top of this control and is the only tested target that improves over it on all three benchmarks, and masking shows that the action expert relies on the query states.

W1 / Q1(a): Event queries with λMEF=0\lambda_{\mathrm{MEF}}=0

The queries-only control keeps the 16 event queries and the action expert's access to their states, and trains them only through the action loss. Within each benchmark, the query-based rows share the architecture, data, training budget, training seed, and evaluation instances, and differ only in the auxiliary target. Camera, Robot, and Noise are LIBERO-Plus categories.

ConfigurationLIBERO Avg.LIBERO-Plus TotalCameraRobotNoiseCALVIN Avg. Len.
Vanilla VLA (no queries)96.569.341.061.253.83.645
Queries only97.175.755.749.182.03.729
A1 (fixed-horizon flow)96.973.356.242.976.93.865
A2 (event-frame RGB)97.473.756.546.478.03.759
MEF-VLA (remaining event-flow)98.178.261.255.782.43.946

The control separates the three factors in the reviewer's decomposition:

  • (i) Query interface. Without a target, the queries raise LIBERO by 0.6 points, LIBERO-Plus Total by 6.4 points, and CALVIN Avg. Len. by 0.084. The interface follows prior query-based foresight (Mantis, DreamVLA; Sec. 3.4); our claim concerns the target trained on it.
  • (ii) Any auxiliary dense-prediction objective. Relative to the control, fixed-horizon flow and event-frame RGB lower LIBERO-Plus Total by 2.4 and 2.0 points and add 0.136 and 0.030 in CALVIN Avg. Len. The effect of an auxiliary objective therefore depends on its target.
  • (iii) The motion-to-event target. At the matched training seed, remaining event-flow is the only target that improves over the control on all three benchmarks: +1.0 points on LIBERO (1,962 vs. 1,942 of 2,000 rollouts), +2.5 on LIBERO-Plus (95% CI over tasks: 0.8 to 4.0), and +0.217 in CALVIN Avg. Len. On CALVIN, it accounts for 0.217 of the 0.301 gain over Vanilla VLA.

The categories highlighted by the reviewer. The Noise gain comes with the query interface (82.0 for queries only, 82.4 for MEF-VLA), and A1 and A2 reduce it. On Camera, the interface raises success from 41.0 to 55.7; A1 and A2 add at most 0.8 points, whereas remaining event-flow raises it to 61.2. On Robot, the interface lowers success from 61.2 to 49.1; remaining event-flow recovers 6.6 points, whereas A1 and A2 fall further.

W1 / Q1(b): Masking the query states at inference

InferenceSpatialObjectGoalLongLIBERO Avg.LIBERO-Plus Total
MEF-VLA99.498.698.695.898.178.2
Query states masked91.888.484.275.084.963.2

Masking the 16 query positions in the action expert's cross-attention lowers success by 13.2 points on LIBERO and 15.0 points on LIBERO-Plus, with declines in every suite and perturbation category and the largest drop on LIBERO-Long (20.8 points); the masked policy falls below Vanilla VLA. The action expert therefore relies on the query states, which both losses train; the queries-only control isolates what the MEF target adds to them.

AuthorsOpenReview ↗

Response to Reviewer gK7F (2/2)

W2: Target ablations on long-horizon tasks

We trained A1 (H=8H=8 on CALVIN, H=16H=16 on the robot) and A2 with the full model's configuration on each benchmark. On CALVIN, all policies are evaluated on the official 1,000 five-task sequences with the same evaluation seed. On the robot, the queries-only control, A1, and A2 use the same one- and two-stage training demonstrations and real-time chunking (RTC) settings as MEF-VLA and Vanilla VLA. All five policies are evaluated on the same held-out three-stage instructions and initial configurations, with 30 trials per task family.

CALVIN D→D ablations

CALVIN D→D5 tasks (%)Avg. Len.
Vanilla VLA58.13.645
Queries only61.13.729
A2 (event-frame RGB)62.83.759
A1 (fixed-horizon flow)63.73.865
MEF-VLA67.13.946

Held-out three-stage real-world ablations

Real world, three-stage successesPill SortingSandwich MakingCup StackingTotal (of 90)
Vanilla VLA771731
Queries only1182140
A1 (fixed-horizon flow)11102344
A2 (event-frame RGB)12102143
MEF-VLA15132553

Removing either component lowers long-horizon success. Replacing the event-aligned horizon (A1) lowers CALVIN Avg. Len. by 0.081 and three-stage success by 9 of 90 trials; replacing remaining event-flow with event-frame features (A2) lowers them by 0.187 and 10 trials. MEF-VLA leads both variants in every real-world family. On held-out three-stage instructions, MEF-VLA succeeds in 53 of 90 trials, compared with 40 for the queries-only control. On CALVIN, relative to the queries-only control, event-frame prediction adds 0.030 in Avg. Len., fixed-horizon flow adds 0.136, and remaining event-flow adds 0.217: predicting motion accounts for most of the long-horizon gain, and aligning the motion target with events adds to it.

Q2: Evaluation size and training seeds

  • LIBERO: 50 rollouts per task, 500 per suite, and 2,000 per policy.
  • LIBERO-Plus: one rollout per perturbed instance, 10,030 per policy; Total pools all instances, so each category is weighted by its size.
  • CALVIN: the official 1,000 five-task sequences. Real world: 30 trials per task family, instruction length, and method.

The ablations in Q1(a) use training seed 42. To measure training variability, we also trained MEF-VLA with seeds 123 and 2026. On LIBERO and LIBERO-Plus, the three full-model runs use evaluation seed 7 and the same evaluation instances within each benchmark. MEF-VLA averages 98.0 ± 0.2% on LIBERO and 77.0 ± 1.2% on LIBERO-Plus across the three runs (mean ± SD, Table 13 of the revised manuscript). Each run scores above the seed-42 results for Vanilla VLA, Queries only, A1, and A2 on both benchmarks.

Revisions. (i) Sec. 4.4 and Table 4 report the queries-only control, and Table 10 gives its per-perturbation results. (ii) Table 4 reports the CALVIN target ablations, Table 20 reports the held-out three-stage ablations including Queries only, and Appendix C.2 reports inference-time query-state masking (Table 11). (iii) Sec. 4.1 specifies the LIBERO and LIBERO-Plus evaluation counts, and Table 5 lists the simulation evaluation seeds. Appendix C.4 reports results across three full-model training seeds and paired intervals for the seed-42 comparisons (Tables 13–14).

We thank the reviewer again for these controls, which make the attribution of the gains explicit. They show that the action expert depends on the query states, that the target determines whether auxiliary prediction helps, and that both target components contribute to the long-horizon gains. We hope these results address the reviewer's concerns and would be grateful if the reviewer would consider raising the score.

We thank the authors for a thorough and candid response. The queries-only control, the inference-time masking, the CALVIN and real-world target ablations, and the evaluation counts address Q1–Q3. The new decomposition is informative: the query interface accounts for most of the LIBERO-Plus gain (+6.4 of +8.9), while the target accounts for most of the long-horizon gain on CALVIN (+0.217 of +0.301) and in the real world (+13 of +22 three-stage successes). We are raising our score accordingly.

AuthorsOpenReview ↗

Thank you for your feedback

Thank you for considering our response and updating your assessment. We appreciate your time and constructive feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback. Reviewers found the decomposition of visual foresight into which future to predict and what to represent useful (gK7F, CkRP, JAGe), the training-only design practical (gK7F, JAGe), and the matched ablations informative (gK7F, JAGe). All models trained for this response use training seed 42, the seed of the submitted models, except the two additional MEF-VLA seeds (123 and 2026), and all policies on a benchmark share its evaluation seed and instances. The new results address the questions raised across the reviews:

  1. Query interface and target (gK7F). A queries-only control (λMEF=0\lambda_{\mathrm{MEF}}=0) shows that the event-query interface accounts for much of the LIBERO-Plus gain (69.3% → 75.7%). At the matched training seed, remaining event-flow is the only tested target that improves over this control on LIBERO (+1.0 points), LIBERO-Plus (+2.5), and CALVIN D→D (+0.217 Avg. Len.). On held-out three-stage instructions, MEF-VLA succeeds in 53 of 90 trials, compared with 40 for Queries only and 31 for Vanilla VLA. Masking the query states at test time lowers LIBERO-Plus Total from 78.2% to 63.2%, so the action expert relies on them.
  2. Event horizon and action chunk (CkRP). During training, actions after an in-chunk event are predicted in later samples with query states trained toward the following event, and an event beyond the chunk supplies the spatial target that the chunk advances toward. Event-based guidance remains beneficial whether the event horizon is shorter or longer than the action horizon: MEF-VLA exceeds the queries-only control, and using the chunk endpoint instead of the event for samples whose event falls inside the chunk, beyond it, or both (A1) lowers LIBERO-Plus Total by 3.7, 2.5, and 4.9 points.
  3. Long-horizon target ablations (gK7F, JAGe). Replacing the event-aligned horizon (A1) or the remaining-motion representation (A2) lowers CALVIN Avg. Len. from 3.946 to 3.865 and 3.759, and held-out three-stage real-world success from 53/90 to 44/90 and 43/90. On CALVIN, predicting motion accounts for most of the gain over the queries-only control, and event alignment adds to it.
  4. Baselines (CkRP, JAGe). We added FLOWER, π₀.₅, and Cosmos Policy, which build on robot or video pretraining. Trained with its official configuration but without robot pretraining, π₀.₅ reaches 54.0% on LIBERO-Plus, 24.2 points below MEF-VLA, and applying the full MEF method to FLOWER raises its Avg. Len. from 4.324 to 4.412.
  5. Seeds and uncertainty (gK7F, JAGe). Over three training seeds, MEF-VLA reaches 98.0 ± 0.2% on LIBERO and 77.0 ± 1.2% on LIBERO-Plus, and every seed exceeds Vanilla VLA, A1, and A2 on both. On LIBERO-Plus, paired bootstrap intervals over tasks exclude zero for the margins over the queries-only control, A1, and A2.
  6. Event references (JAGe). Two alternative detector settings, each altering both missed and false events, change success by an amount comparable to the seed variation and stay above A1 and A2. On CALVIN and real-world demonstrations, the detector reaches 81.3% and 75.7% F1 at ±1 s, and its agreement with a second annotator (84.4% F1 at ±0.5 s) is close to inter-annotator agreement (87.8%).

Revisions.

We have uploaded the revised manuscript, with changes to the text and tables marked in blue.

  • The Abstract updates the ablation takeaway, and the Introduction summarizes the new CALVIN and real-world ablations.
  • Sec. 3.2 clarifies how the next-event target relates to the action chunk. Appendix C.3 reports the training-sample statistics and the Extended and Capped controls (Table 12).
  • Sec. 4.1 specifies the LIBERO and LIBERO-Plus evaluation counts, and Appendix A.1 lists the simulation evaluation seeds (Table 5).
  • Sec. 4.2 discusses differences in pretraining and training budget across Tables 1–3. Table 2 adds FLOWER, Table 3 adds two π₀.₅ configurations and Cosmos Policy, and Appendix C.5 compares FLOWER and FLOWER + MEF (Table 15).
  • Sec. 4.4 and Table 4 report the queries-only control and the CALVIN target ablations. Appendix E.2 reports the held-out three-stage ablations by task family, including Queries only (Table 20).
  • Appendix C.2 reports per-perturbation results and inference-time query-state masking (Tables 10–11). Appendix C.4 reports full-model seed variation and paired intervals for the seed-42 comparisons (Tables 13–14). Appendix D adds detector validation on CALVIN and real-world demonstrations, F1 at ±0.5 s, second-annotator agreement, and downstream detector sensitivity.
  • Figure 1 shows the paired current frame, and its caption explains the FlowRGB encoding. The Table 1 caption explains the bold and underlined entries.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.