acceptodds
Under review as a conference paper at ICLR 2027

Event-Aligned Visual Action Reasoning for World Action Models

Abstract

World-Action Models (WAMs) utilize future visual prediction as an intermediate reasoning process to guide action generation. However, existing WAMs typically structure visual imagination according to predefined temporal intervals, without explicitly accounting for the different roles of task-critical interactions and connecting transitions. We argue that effective visual foresight should align directly with task-relevant interactions and their corresponding reasoning demands. To this end, we introduce an event-aligned visual action reasoning framework that organizes visual-action prediction around interaction events. Through event-aligned visual-action supervision, WAM learns to generate event-aligned visual context in each imagined rollout, placing greater emphasis on critical state changes that inform action generation. This shapes the visual reasoning granularity according to the underlying interaction dynamics, with detailed reasoning around task-critical events and coarser progression through connecting transitions. Furthermore, we introduce an execution validity head that identifies the valid portion of each predicted action sequence, avoiding redundant actions during chunked inference. Experiments demonstrate a 10.26 percentage point improvement in DOMINO success rate over baseline and competitive performance on RoboTwin 2.0. It also transfers from DOMINO Level 1 to Levels 2 and 3 without target-level adaptation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Visual Action Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TZD8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces an event-aligned visual action reasoning framework for World Action Models (WAMs), robot policies that imagine future frames before generating actions. It addresses 1 main concern from previous WAMs: visual imagination follows fixed time intervals, so long transitions (reaching, carrying) take most of the prediction while the short interactions that actually change the object state (grasp, release, press) get little attention. The main idea is to reorganize training trajectories around interaction events: transitions are compressed into a single chunk, while the moment the interaction happens is kept at full temporal resolution. This way the model learns to focus its imagination on the events that change object state and guide the following motions. The backbone is LingBot-VA, plus a small head that skips the padded actions created by this reorganization. The method improves DOMINO success rate by 10.26 points over LingBot-VA and is on par on RoboTwin 2.0.

Critical strengths
  • Intuitive, well-motivated idea by shifting the imagination budget from long transitions to the short interactions that change object state.
  • Specific, concrete allocation scheme with transitions compressed to one chunk, the moment the state change is visible kept at full resolution, nearby motion compressed into the remaining slots, and no chunk crossing an event boundary. This is a finer-grained design than prior event- or keyframe-based WAMs.
  • Strong controlled ablation (Table 4) that isolate the idea itself rather than the amount of data.
  • Large gain on dynamic manipulation with +10.3 success rate over LingBot-VA on DOMINO, and better transfer to unseen difficulty levels.
Critical weaknesses
  • The novelty relative to event- and keyframe-based world models is not established. WALL-WM already organizes WAM training around semantic events, with variable-length execution chunks. SKIP keeps event frames (approach, contact, grasp, release) and fills the gaps by interpolation. Both are mentioned in a single sentence, with no discussion of how the proposed scheme differs and no experimental comparison. The contribution is effectively the within-event budget allocation and the validity head, but the paper presents event-based organization itself as new. Can the authors clarify?

Compressing transitions discards information that the policy may need, and failure cases are not analyzed. Each transition is resampled into a single chunk, so a long reach or carry is represented by only a few frames, and only the actions paired with those frames are kept as targets (Fig. 2, App. A). The paper treats transitions as carrying no task-relevant information. This assumption is questionable in several common situations:

  1. Moving objects. objects can move, so intercepting them depends on approach timing. Because compressed chunks cover variable amounts of real time, the imagined frames no longer have a fixed time step. The model must therefore infer how much time each predicted chunk covers, which matters most precisely when timing is critical.

  2. Approach geometry and pre-grasp alignment. For hanging, insertion, or placing with a specific orientation, success is largely decided by the trajectory and orientation during the approach, before contact happens.

  3. Choosing the target object. In scenes with several candidate objects, the direction of the approach reveals which object the policy intends to manipulate.

  4. Safety and collisions. Obstacle avoidance and collision-free paths are properties of the transition.

Questions for authors

See above, critical weaknesses section. Also, there are misplaced citations in the paper. For examples, in the introduction, one citation appears as (?).

5 comments

Thank you for the detailed feedback and constructive comments! We are delighted that you highlighted our "intuitive, well-motivated idea" with its "finer-grained design", as well as the "strong controlled ablation" and "large gain on dynamic manipulation". In the following, we clarify each of your concerns in detail.


W1: Novelty of Our Event-Aligned Organization and Distinction from WALL-WM & SKIP

Thank you for raising this point and we have added the discussion of the difference between our method and WALL-WM and SKIP. We want to clarify that our contribution is not simply organizing trajectories by event itself. As stated in our first contribution (Sec. 1, Lines 99–102), we propose "an event-aligned visual action reasoning framework that organizes future prediction with task-critical interactions as focus to reason more carefully over crucial state changes. Instead of inheriting a predefined temporal structure from sampled trajectories, our formulation enables the reasoning process to adapt its granularity to the interaction dynamics." Our event-aligned organization structure trajectories into Interaction vs. Transition events and further into native-resolution Evidence vs. anchored Support phases so that a closed-loop WAM adapts its visual reasoning granularity autonomously. This is fundamentally new and distinct from both WALL-WM and SKIP in problem formulation, temporal structure, and inference mechanism:

  1. Difference from WALL-WM (Uniform event clips with external test-time oracles vs. Adaptive-granularity autonomous reasoning):

    • WALL-WM merely uses semantic events to define where each training clip starts and ends, while still inheriting a fixed, uniform sampling stride across all events. It cannot autonomously organize event-aligned rollouts at test time: its event mode requires both the next-event text description and the event length to be supplied externally by a VLM or other module at every step.
    • Our method reshapes the visual reasoning granularity itself rather than clipping uniform-stride videos. Combined with our execution validity head, event-aligned visual imagination and variable-length execution emerge autonomously at inference from the global task instruction alone, without any external sub-event prompts or duration oracles.
  2. Difference from SKIP (Offline action-conditioned video interpolator vs. Closed-loop WAM policy):

    • SKIP is not a control policy or an action prediction framework. It is an action-conditioned video synthesizer that takes ground-truth/known actions as input to reconstruct dense intermediate frames between keyframes for synthetic data generation.
    • Our method is a closed-loop WAM policy that predicts future visual states at interaction-dependent rates as intermediate spatial reasoning to generate robot actions, never reconstructing skipped transition frames.
  3. Regarding Experimental Comparison:

    • SKIP cannot be evaluated as a manipulation policy as it requires actions as input rather than predicting them.
    • WALL-WM has not released its pretrained weights or the checkpoint of its VLM conditioner, which its inference relies on. Moreover, WALL-WM does not report results under an evaluation setting comparable to ours, so its reported number cannot be compared directly. We will include a direct comparison once its weights and VLM conditioner become available.

W2-1: Does Compressing Transitions Discard Task-Critical Information?

Thank you for raising these practical manipulation scenarios. In this work, our goal is to demonstrate that a World-Action Model can and should imagine future at adaptive temporal rates. Specifically, viewing and predicting the video at variable speeds (e.g., 1×1\times native speed during critical physical interactions vs. 2×2\times or 3×3\times fast-forward during free-space transitions). Since task-critical state changes are predominantly concentrated in interaction phases for general manipulation, resampling each transition into a single chunk serves as a hyperparameter-free instantiation to prove the effectiveness of multi-rate visual imagination.

Before discussing each scenario, we clarify a key design property of our Interaction Taxonomy: Transitions are compressed, not discarded, and pre-contact alignment is preserved in Support Phases. Each transition event is still represented by a full chunk of KK visual-action slots. More importantly, local motions immediately preceding and following contact, such as pre-grasp alignment, gripper closing before contact, and lifting off after contact, are categorized as the Interaction Event's leading and trailing Support Phases. By Rule 3 (Page 5, Lines 256–257), the start and end frames of support phases are strictly anchored, and our Reduced Context ablation in Tab.4 confirms that retaining these surrounding support/transition slots boosts SR from 37.83% to 42.83%.

Below, we address how our framework handles each of the four scenarios with empirical evidence from the paper:

  1. Moving objects and interception timing: All 35 tasks in DOMINO are dynamic manipulation tasks with continuously moving objects. Rather than suffering from variable chunk durations, our method improves DOMINO Level 1 SR from 32.57% to 42.83% (+10.26%) (Tab. 1) and achieves state-of-the-art zero-shot transfer to faster, unseen object dynamics in Level 2 (29.4% vs. 16.9% SR) and Level 3 (9.7% vs. 6.8% SR) (Tab. 2). Two mechanisms enable this: (i) Spatial grounding via visual imagination: the model first predicts the visual state showing where the moving object and gripper will meet at the end of the chunk, which grounds the action generation (when we mask access to imagination under the exact same event sampling, SR drops from 42.83% to 30.69%); and (ii) Boundary-aligned closed-loop correction: since a transition chunk never crosses into an interaction event (Rule 1, Page 5, Line 250-251) and real observations replace predicted states after each chunk, any approach timing drift is corrected by a fresh visual rollout right at the interaction boundary before grasping begins (Page 6, Lines 270–273).
  2. Approach geometry and pre-grasp/placement alignment:
    Since pre-contact approach alignment is preserved across the transition-to-interaction boundary and within the leading Support Phase, our method excels on tasks requiring strict geometric and orientation alignment. As shown in Tab. C2 and Tab. D3, on tasks governed by oriented placement (S: placed_oriented), precise insertion, or small-target alignment, our method substantially outperforms LingBot-VA: such as Place Fan (50% vs. 14%), Place Mouse Pad (64% vs. 19%), Move Stapler Pad (20% vs. 10%), Rotate QRcode (34% vs. 26%), and real-world Tennis Ball Insertion (58.67% vs. 42.67%, Tab. C1).
  3. Choosing the target object in multi-object scenes:
    Even when an approach is compressed into a single initial chunk, the predicted visual frames explicitly resolve target selection. In Fig. 3, two objects are present: at Chunk 0 (Object approach), LingBot-VA diffuses its attention across both objects and fails, whereas our model immediately concentrates both video attention (Fig. 3a) and action attention (Fig. 3c, red box) onto the intended moving target (22% vs. 11% SR, Tab. C2).
  4. Safety and collision avoidance:
    As defined in Sec. 4.1 (Page 7, Lines 355–356), the DOMINO Manipulation Score (MS) explicitly penalizes workspace violations and clutter collisions. Our method improves average MS from 45.55 to 55.92 (+10.37) across all 35 Level 1 tasks (Tab. 1) and from 30.37 to 45.42 (+15.05) on Level 2 transfer (Tab. 2).

W2-2: Failure Case Analysis and Limitations:

Thank you for the suggestion to include a failure case analysis. We have added a detailed comparison of failure episodes against LingBot-VA (across Tab. C2) with qualitative visualizations in Sec. C.7 (page 21), as well as a new subsection on limitations and future work in Sec. 6 (page 10) of the revised manuscript. While our method improves upon or matches the baseline on 26 of the 35 tasks, the primary performance drops occur on Adjust Bottle (48% vs. 81% SR), Move Can Pot (32% vs. 48% SR), and Place Can Basket (24% vs. 37% SR).

  • Analysis of Failure Cases: These tasks require the robot arm to follow a precise trajectory during mid-air transport. Using a single chunk for the entire carry motion may miss important intermediate waypoints. As a result, the arm may move outside the target height range or collide with the basket rim, as visualized in Fig. C2(page 21).

  • Limitation & Future Work: We agree with the reviewer that different task categories carry different information demands across events. We have incorporated this discussion into the revision, noting that extending our framework from a fixed 1-chunk transition rule to task-adaptive transition budgets (e.g., allocating additional chunks when transport displacement or path curvature is large) is an exciting direction for future work.


Q1: Misplaced Citation

Yes — the Fast-WAM citation was broken when reorganizing model names, which is why it renders as (?) . This is fixed in the revision.

Reviewer_B73vOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes event-aligned visual action reasoning for World-Action Models, organizing visual-action supervision around interaction events while preserving detailed evidence of critical state changes. An execution validity head skips repeated actions introduced by fixed-size chunk construction. Built on LingBot-VA, the method improves DOMINO success rate from 32.57% to 42.83%, achieves competitive RoboTwin 2.0 performance, and demonstrates transfer to harder dynamic settings. Ablations examine event alignment, evidence preservation, surrounding context, and execution validity.

Critical strengths
  1. Meaningful improvement on dynamic manipulation. The method improves DOMINO success rate by 10.26 percentage points over LingBot-VA, with additional gains on harder Level 2/3 settings.
  2. Informative sampling ablations. Under the same total slot budget, Event Uniform improves success rate from 29.17% to 39.60%, and explicit evidence preservation further increases it to 42.83%. These comparisons support the value of interaction-aware supervision.
Critical weaknesses
  1. The evidence does not fully isolate improved visual reasoning from changes in action supervision. Event-aligned sampling modifies both visual targets and action targets. The existing ablations establish the effectiveness of the combined design, but do not determine whether the gains specifically depend on event-aligned visual prediction. Attention visualizations are suggestive rather than causal evidence, particularly after the policies follow different trajectories.
  2. Broader gains are modest or uncertain, and some results are inconsistent. RoboTwin performance is competitive rather than leading. Real-world results use only 25 trials per task and include a decrease on drawer manipulation from 64% to 56%. Moreover, Appendix C.2 reports transfer success rates of 26.7%/8.7%, whereas Tables 2 and C3 report 29.4%/9.7%. Repeated training runs and confidence intervals would strengthen the conclusions.
Questions for authors
  1. How are overlapping detector records converted into a complete event partition and evidence/support boundaries? How are real-world demonstrations annotated, and how sensitive is performance to boundary errors?
  2. How does Algorithm 1 handle F=1 when both support phases are non-empty? Please clarify the mapping between source frames, VAE temporal groups, and action-validity labels.
  3. Can you isolate the contribution of visual reasoning using an action-only policy with the same event-aligned supervision, or a controlled ablation of predicted visual context? Can execution-time masking also be evaluated with the trained checkpoint held fixed?
  4. Which transfer results are correct: 26.7%/8.7% in Appendix C.2 or 29.4%/9.7% in Tables 2 and C3? Please provide uncertainty estimates for the main comparisons.
  5. Do the efficiency gains persist when considering all evaluation episodes and actual execution time? The current shared-success comparison supports a narrower claim than overall deployment efficiency.
5 comments

We appreciate your careful evaluation and constructive questions! We are pleased to hear that you recognized the value of our "interaction-aware supervision" and found our "informative sampling ablations" and "meaningful improvement on dynamic manipulation" convincing. Below, we first address the main concerns in W1–W2 (and corresponding Q3–Q4) with new controlled experiments and multi-seed/expanded evaluations, followed by detailed clarifications for Q1, Q2, and Q5. All updates have been incorporated into the revised manuscript.


W1 & Q3-1: Isolating Event-Aligned Visual Reasoning from Action Supervision

We want to clarify a property of World-Action Models (WAM). As described in Sec. 3.1 and Eq. 1, action chunks are conditioned on imagined future visual chunks in the causal latent space. With event-aligned reorganization, chunks can span different real-time durations. The imagined future visuals predict where the robot and objects will be at the end of each chunk, providing spatial grounding for action generation. Without these predictions, variable-duration action targets alone lack spatial grounding.

To separate the effect of event-aligned visual imagination from action target resampling and execution masking, we conducted the control experiment suggested by the reviewer on DOMINO (added to Sec. C.4 and Tab. C5): We trained and evaluated a direct action-prediction variant, Ours(direct action), which retains our event-aligned sampling, resampled action targets, and execution validity head, but adopts the attention masking strategy from Fast-WAM. Specifically, we mask the attention from the current action prediction to the corresponding predicted visual chunk, so that actions are predicted directly from historical observations without conditioning on imagined future visuals.

MethodSR$\uparrow$MS$\uparrow$
LingBot-VA32.5745.55
Ours (Direct Action)30.6945.35
Ours42.8355.92

We draw the following conclusions:

  1. Action-side changes alone are insufficient. Ours (Direct Action) achieves 30.69% SR, even below LingBot-VA (32.57%), suggesting that event-aligned action supervision alone is insufficient.
  2. Visual imagination improves SR by 12.14 points. Compared to Ours (Direct Action), our full method improves SR from 30.69% to 42.83%, directly showing the contribution of visual conditioning under the same action-side settings.
  3. Why visual imagination matters. Event-aligned chunks span different real-time durations. Imagined future visuals predict where the robot and objects will be at the end of each chunk, providing spatial grounding for action generation.

Our existing ablations Tab.4 and Tab.C7 further support this finding. Episode Uniform, Event Uniform, and Ours use the same total action-slot budget (211,664 slots) and Execution Validity Head. In particular, Event Uniform and Ours share the same event boundaries and per-event slot budgets, but Ours preserves more native-resolution visual evidence (54.86% vs. 47.55% evidence-phase ratio), improving SR from 39.60% to 42.83%.


Q3-2: Execution Timing & Evaluating execution-time masking with trained checkpoint

In Tab. 4, we compared models separately trained with and without the execution validity head under identical event-aligned sampling, showing that the head improves DOMINO SR from 39.11% to 42.83%. To further isolate the inference-time effect of validity masking as requested, we took our fixed trained checkpoint (Ours) and evaluated it directly on DOMINO with inference-time validity masking disabled (updated in Tab. 4):

MethodsSR$\uparrow$MS$\uparrow$
Trained w/o Execution Head39.1152.33
Ours (w/o Inference Masking)38.9453.04
Ours42.8355.92

Holding the trained checkpoint strictly fixed, disabling the validity mask at inference time causes SR to drop from 42.83% to 38.94%. Without the mask, executing repeated actions causes the robot to stall at repeated targets, which degrades manipulation timing in dynamic scenes and delays subsequent visual rollouts as discussed in Sec. 3.3.

W2 & Q4-1 & Q4-2: Multi-Seed Uncertainty, Expanded Real-World Evaluation, Baseline Comparisons, and Transfer Results

We address each point raised in W2 and Q4 below:

  1. Multi-seed training uncertainty on DOMINO (Q4-2).

    Following your suggestion, we trained and evaluated both LingBot-VA and our method across three independent random seeds on DOMINO (35 tasks ×\times 100 evaluation episodes per seed) to quantify multi-run training and evaluation uncertainty:

    MethodsSR$\uparrow$MS$\uparrow$
    LingBot-VA32.96±0.5832.96\pm 0.5845.90±0.5045.90\pm 0.50
    Ours43.25±1.2343.25\pm 1.2355.96±1.1255.96\pm 1.12

    Across the three independent training runs, our method achieves a mean improvement of +10.29% in SR (32.96%→43.25%32.96\% \to 43.25\%) and +10.06 in MS (45.90→55.9645.90 \to 55.96). Even the lower bound of Ours at 42.02%42.02\% SR exceeds the upper bound of LingBot-VA at 33.54%33.54\% SR by 8.48 percentage points, confirming that our gains on dynamic manipulation are highly robust to training seed variance.

  2. Expanded real-world evaluation from 25 to 75 rollouts per task (W2).

    To reduce small-sample variance in real-world evaluation, we expanded our real-robot evaluation by 3×3\times, from 25 to 75 independent rollouts per task (updated in Sec. C.1 and Tab. C1):

    MethodTennis Ball InsertionBimanual Cup PlacementDrawer ManipulationAverage
    Lingbot-VA42.67%21.33%60.00%41.33%
    Ours58.67%30.67%57.33%48.89%

    With 75 rollouts per task, our method improves the overall average real-world SR by +7.56 percentage points (41.33%→48.89%41.33\% \to 48.89\%), with substantial gains on Tennis Ball Insertion (+16.00%) and Bimanual Cup Placement (+9.34%). On Drawer Manipulation, ours and LingBot-VA are similar (differ by only two single trials).

  3. RoboTwin 2.0 performance (W2).

    On RoboTwin 2.0 benchmark, top WAM baselines already saturate around 91–93% SR as shown in Tab.1. Our method achieves 93.02% (Clean) / 91.58% (Random) / 92.30% (Avg.), outperforming the re-evaluated LingBot-VA checkpoint (91.21%). More importantly, although ImageWAM succeeds in static RoboTwin scenes, it fails to model intermediate motion trajectories, causing both ImageWAM and Fast-WAM to drop below 20%20\% SR on dynamic manipulation tasks in DOMINO (18.86%18.86\% vs. our 42.83%42.83\%). Our framework is the only method that maintains top-tier performance on static tasks while achieving substantial gains on dynamic manipulation.

  4. Correct transfer results in Appendix C.2 (W2 & Q4-1).

    Thank you for catching this textual inconsistency. 29.4% / 9.7% SR (and 45.42 / 24.23 MS) reported in Tab. 2 and Tab. C3 are the exact, correct results, which match the per-task averages across all 10 transfer tasks in Tab. C3. The numbers 26.7%/8.7% mentioned in the text of Appendix Sec. C.2 were leftover text from an earlier draft checkpoint that we inadvertently missed updating. We have corrected the prose in Sec. C.2 to match Tab. 2 and Tab. C3.


Q1-1: From Overlapping Detector Records to Contiguous Event Partition

Thank you for highlighting these implementation details. We have expanded Sec. D.3 (Page 27-28) in the revised manuscript to formally specify the conversion procedure:

  • Detector records. Each detected state change produces a record containing onset, effect, and confirmation frames as in Eq.5, Tab.D1-D2. These records may overlap, especially in bimanual tasks where two arms perform different actions simultaneously, e.g., one arm releases object and the other receives it in a handover episode.
  • Event boundaries (si,ei)(s_i, e_i). We merge overlapping detector evidence windows by taking their temporal union, keeping the earliest start and latest end. We then use detector effect frames and gripper opening/closing onsets to determine event boundaries, filtering out redundant cuts within the same interaction. The resulting boundaries partition the trajectory into contiguous, non-overlapping events (Eq. 2).
  • Evidence and support phases. Within each interaction event, the merged detector evidence windows define the Evidence Phase [siev,eiev][s_i^{ev},e_i^{ev}]. The remaining frames before and after [siev,eiev][s_i^{ev},e_i^{ev}] within (si,ei)(s_i,e_i) form the Support Phases PP and QQ, respectively, as illustrated in Page 4 and Fig. A1. Events without evidence windows are classified as Transition Events.

Q1-2: Real-World Annotation Process & Sensitivity to Boundary Errors

We have updated Sec. C.1 (Page 18) of the revised manuscript to formally document our real-world annotation pipeline and quality verification protocol:

  1. Real-World annotation. Mirroring Sec. D.1, we use Claude Opus 5 to draft task-specific event specifications, which are manually reviewed and fixed. We then automatically annotate the 179 teleoperated real-world demonstrations (45 / 44 / 90 across the three tasks) using RGB observations and recorded gripper signals, with visual detection rules calibrated on three pilot episodes per task (e.g., a grasp triggers when the object visibly moves with the gripper; a drawer event triggers upon visible displacement relative to the cabinet). Auditing all 179 annotated episodes against the recorded videos using our custom verification interface (Fig. D1, Page 29) confirmed that only 6 out of 179 episodes (3.4%) required manual boundary adjustment, yielding a 96.6% pass rate for automated boundary localization without simulator states. (Using these verified annotations, we also expanded our real-world evaluation to 75 rollouts per task, as reported in W2 above.)

  2. Sensitivity to boundary accuracy. Our method does not rely on frame-exact boundary localization. By Rule 2 in Sec.3.2, the entire evidence window is retained at native temporal resolution and buffered on both sides by support phases. Thus, a shift of a few frames in an event or evidence boundary merely reallocates boundary frames between adjacent phases without discarding core contact observations. Moreover, as shown in Tab. 4, under the same slot budget, coarse event boundaries alone (Event Uniform, which ignores internal evidence/support boundaries) already recover 10.43 of the 13.66-point SR gain over Episode Uniform (29.17%→39.60%29.17\% \to 39.60\%), and explicit evidence boundary placement contributes the remaining +3.23 points (39.60%→42.83%39.60\% \to 42.83\%).


Q2-1: Resolving the F=1 Case in Algorithm 1

When F=1 and both P and Q are non-empty, we follow the similar standalone-support rule used for F=0 (Algorithm1 line8). Specifically, we assign the remaining slot to one support phase and allocate a separate chunk of K slots to the other. When F=0, each non-empty support phase is allocated a separate chunk. This ensures that both interaction boundaries are preserved. We have updated Algorithm 1 and Appendix A in the revised PDF to state this explicitly.


Q2-2: Mapping between Source Frames, VAE Temporal Groups, and Validity Labels

As described in Sec. 3.1, Sec. 3.3 and Sec. B.2, following the pretrained LingBot-VA backbone, each prediction chunk consists of CC causal video VAE latent temporal groups (C∈1,2,3,4C\in{1,2,3,4} during training and C=2C=2 at inference). Each latent temporal group encodes 4 resampled visual frames and aligns with 16 action slots (Page 17, Lines 877–880).

During chunk construction, if a segment is shorter than its allocated slots, source frames (and their aligned action groups) are padded by repetition prior to VAE encoding (Page 6, Lines 277–281). The binary validity target mn,j∈0,1m_{n,j}\in{0,1} is set to 1 for action slots aligned with the first occurrence of a source frame, and 0 for slots resulting from repeated padding frames as shown in Fig. 2a. Since event durations and boundaries are unknown at inference time (Page 6, Line 279), the number of valid slots varies dynamically per chunk and cannot be replaced by a static skip rule.

Reviewer_FP5cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the temporal organization of visual–action prediction in world-action models. It constructs training supervision around task-relevant interaction events, preserving temporal detail within evidence phases while compressing surrounding support and transition motion. A lightweight execution validity head identifies repeated padding positions and determines which predicted actions to execute. Built on LingBot-VA, the method improves DOMINO success rate from 32.57% to 42.83%, achieves competitive results on RoboTwin 2.0, and is evaluated through cross-level transfer and three real-robot tasks.

Critical strengths
  1. The motivation is well grounded and supported by meaningful controlled ablations. Brief, consequential interactions and longer connecting motions carry different information for manipulation. Under a matched slot budget, episode-uniform sampling, event-uniform sampling, and evidence-preserving sampling achieve 29.17%, 39.60%, and 42.83% success, respectively. These comparisons provide substantive support for event-level supervision organization beyond an overall comparison against the original backbone.

  2. The improvement on dynamic manipulation is practically meaningful, and the execution mechanism is lightweight. The method improves DOMINO success by 10.26 percentage points over LingBot-VA and retains an advantage under stronger dynamics. The execution validity head adds only a small projection layer and improves success under the same event-aligned sampling. Measurements on shared successful episodes also show fewer issued action commands and visual-rollout requests.

Critical weaknesses
  1. The experiments do not sufficiently separate improved visual reasoning from changes to action trajectories and execution timing. Reorganizing visual supervision also resamples the corresponding action targets and removes repeated execution positions, thereby changing action spacing and execution behavior. Matching the training slot budget does not control these effects, particularly in dynamic environments where timing matters. The ablations establish that event organization is useful, but do not establish how much of the benefit specifically comes from event-aligned visual imagination. Attention visualizations provide qualitative support, while later visualizations follow different policy trajectories. A direct action-prediction control using the same event sampling and execution rules is missing, limiting attribution of the central contribution.

  2. The essential annotation-to-chunk conversion is not fully reproducible from the description. Appendix D describes simulator states, contacts, success predicates, and detector timestamps in considerable detail. However, Section D.3 explicitly distinguishes potentially overlapping detector records from the final contiguous event partition, without fully specifying their conversion into event boundaries and evidence/support phases. Algorithm 1 and Appendix A also leave a concrete case unresolved: when F=1 and both support phases are nonempty, how are the support-allocation and interaction-boundary-preservation rules satisfied? The production of equivalent annotations for real-robot demonstrations is also unclear. Since these boundaries directly determine the training targets, these omissions affect both reproducibility and the assessment of applicability.

Questions for authors
  1. Targeted additional evidence: Can a direct action-prediction policy be evaluated using the same event sampling, action targets, and execution-mask mechanism, or can another control separate visual organization from action-timing effects? This would materially affect my assessment of the visual-reasoning contribution.

  2. Method clarification: What are the complete rules for converting potentially overlapping detector records into the final contiguous event partition? How are evidence/support boundaries determined, and how is the F=1 case handled when both support phases are present?

  3. Annotation reliability and applicability: How are event annotations obtained for the real-robot demonstrations? Are there annotation-quality checks or boundary-perturbation experiments that characterize sensitivity to boundary accuracy?

4 comments

Thank you for the thorough review and insightful suggestions! We are glad you found our approach "well grounded" with a "lightweight" execution mechanism and the experimental evaluation "practically meaningful" and "supported by meaningful controlled ablations". Below we address each of your questions in detail.


W1 / Q1: Isolating Event-Aligned Visual Reasoning from Action Supervision and Execution Timing

We want to clarify a property of World-Action Models (WAM). As described in Sec. 3.1 and Eq. 1, action chunks are conditioned on imagined future visual chunks in the causal latent space. With event-aligned reorganization, chunks can span different real-time durations. The imagined future visuals predict where the robot and objects will be at the end of each chunk, providing spatial grounding for action generation. Without these predictions, variable-duration action targets alone lack spatial grounding.

To separate the effect of event-aligned visual imagination from action target resampling and execution masking, we conducted the control experiment suggested by the reviewer on DOMINO (added to Sec. C.4 and Tab. C5): We trained and evaluated a direct action-prediction variant, Ours(direct action), which retains our event-aligned sampling, resampled action targets, and execution validity head, but adopts the attention masking strategy from Fast-WAM. Specifically, we mask the attention from the current action prediction to the corresponding predicted visual chunk, so that actions are predicted directly from historical observations without conditioning on imagined future visuals.

MethodSRMS
LingBot-VA32.5745.55
Ours (Direct Action)30.6945.35
Ours42.8355.92

We draw the following conclusions:

  1. Action-side changes alone are insufficient. Ours (Direct Action) achieves 30.69% SR, even below LingBot-VA (32.57%), suggesting that event-aligned action supervision alone is insufficient.
  2. Visual imagination improves SR by 12.14 points. Compared to Ours (Direct Action), our full method improves SR from 30.69% to 42.83%, directly showing the contribution of visual conditioning under the same action-side settings.
  3. Why visual imagination matters. Event-aligned chunks span different real-time durations. Imagined future visuals predict where the robot and objects will be at the end of each chunk, providing spatial grounding for action generation.

Our existing ablations Tab.4 and Tab.C7 further support this finding. Episode Uniform, Event Uniform, and Ours use the same total action-slot budget (211,664 slots) and Execution Validity Head. In particular, Event Uniform and Ours share the same event boundaries and per-event slot budgets, but Ours preserves more native-resolution visual evidence (54.86% vs. 47.55% evidence-phase ratio), improving SR from 39.60% to 42.83%.

W2 / Q2: From Overlapping Detector Records to Contiguous Event Partition & F=1 Case:

Thank you for highlighting these implementation details. We have expanded Sec. D.3 (Page 27-28) in the revised manuscript to formally specify the conversion procedure and the F=1F=1 boundary case. Below, we detail the exact conversion process and clarify how F=1F=1 is handled in Algorithm 1:

  1. From Overlapping Detector Records to Contiguous Event Partition & Evidence Phases:

    As described in Sec.D.2-D.3, we convert detector records into contiguous events in three steps:

    • Detector records. Each detected state change produces a record containing onset, effect, and confirmation frames as in Eq.5, Tab.D1-D2. These records may overlap, especially in bimanual tasks where two arms perform different actions simultaneously, e.g., one arm releases object and the other receives it in a handover episode.
    • Event boundaries (si,ei)(s_i, e_i). We first merge overlapping detector evidence windows by taking their temporal union (keeping the earliest start and latest end) so concurrent bimanual actions form a single evidence interval. Around each merged evidence window, we determine the outer event boundaries (si,ei)(s_i, e_i) using the surrounding gripper opening/closing onsets and terminal detector anchors (teff/tconft_{\text{eff}} / t_{\text{conf}}), while filtering out redundant intermediate release cuts within an active interaction. This guarantees si≤siev<eiev≤eis_i \le s_i^{\text{ev}} < e_i^{\text{ev}} \le e_i and partitions the trajectory into contiguous, non-overlapping events (Eq. 2).
    • Evidence and support phases. Within each interaction event, the merged detector evidence windows define the Evidence Phase [siev,eiev][s_i^{ev},e_i^{ev}]. The remaining frames before and after [siev,eiev][s_i^{ev},e_i^{ev}] within (si,ei)(s_i,e_i) form the Support Phases PP and QQ, respectively, as illustrated in Page 4 and Fig. A1. Events without evidence windows are classified as Transition Events.
  2. Resolving the F=1 Case in Algorithm 1:

    When F=1F=1 and both P and Q are non-empty, we follow the similar standalone-support rule used for F=0F=0 (Algorithm1 line8). Specifically, we assign the remaining slot to one support phase and allocate a separate chunk of K slots to the other, (nP,nQ)=(1,K)(n_P, n_Q)=(1,K). When F=0F=0, each non-empty support phase is allocated a separate chunk. This ensures that both interaction boundaries are preserved. We have updated Algorithm 1 and Appendix A in the revised PDF to state this explicitly.

W2 / Q3: Real-World Annotation Process & Sensitivity to Boundary Errors

Thank you for raising this important point regarding reproducibility and real-world applicability. We have updated Sec. C.1 (Page 18) of the revised manuscript to formally document our real-world annotation pipeline, quality verification protocol, and expanded evaluation results. We address each aspect of your question below:

  1. Real-World annotation. We follow a similar annotation procedure as described in Sec.D.1. Specifically, Claude Opus 5 is used to draft task-specific event specifications, which are manually reviewed and fixed. We then automatically annotate the real-world demonstrations using RGB observations and gripper signals, with visual detection rules calibrated on three pilot episodes per task (e.g., a grasp triggers only when the object visibly moves with the gripper; a drawer event triggers upon visible displacement relative to the cabinet). Since our real-world dataset contains only 179 demonstrations (45/44/90 across three tasks), we checked all 179 annotated episodes to verify that the detected event sequences match the task specifications and that event and evidence boundaries are consistent with the recorded videos (Fig. D1, Page 29). Only 6 out of 179 episodes (3.4%) required manual correction, confirming a 96.6% pass rate for automated boundary localization without simulator states.

  2. Sensitivity to boundary accuracy. Our method does not rely on frame-exact boundary localization. By Rule 2 in Sec.3.2, the entire evidence window is retained at native temporal resolution and buffered on both sides by support phases. Thus, a shift of a few frames in an event or evidence boundary merely reallocates boundary frames between adjacent phases without discarding core contact observations. Moreover, as shown in Tab. 4, under the same slot budget, coarse event boundaries alone (Event Uniform, which ignores internal evidence/support boundaries) already recover 10.43 of the 13.66-point SR gain over Episode Uniform (29.17%→39.60%29.17\% \to 39.60\%), and explicit evidence boundary placement contributes the remaining +3.23 points (39.60%→42.83%39.60\% \to 42.83\%).

  3. Additional real-world evaluation. We extend our real-world evaluation in Sec.C.1, from 25 to 75 rollouts per task. The updated results are shown below.

    MethodTennis Ball InsertionBimanual Cup PlacementDrawer ManipulationAverage
    LingBot-VA42.67%21.33%60.00%41.33%
    Ours58.67%30.67%57.33%48.89%

    These experiments evaluate the effectiveness of event-aligned visual reasoning on real-world manipulation tasks using verified event annotations without simulator states.

Dear Reviewer,

Thank you for your thorough review and insightful suggestions! In our responses, we have addressed each of your concerns:

  • For W1/Q1, we have completed and added the Ours (Direct Action) control experiment on DOMINO to separate the contribution of event-aligned visual imagination from action-side changes. This proves that event-aligned visual imagination contributes a +12.14% SR gain, and action-side changes alone are insufficient (30.69% vs. our 42.83%).
  • For W2/Q2, we have detailed the exact conversion procedure from overlapping detectors to contiguous events and resolved the F=1F=1 boundary case in Algorithm 1.
  • For W2/Q3, we documented the real-world annotation pipeline, confirming a 96.6% automated pass rate across all 179 episodes, and expanded our real-world evaluation to 75 rollouts per task.

We would greatly appreciate it if you could check our responses and let us know whether they have addressed your concerns. If you have any further questions, please let us know and we would be more than happy to discuss them.

Thank you again for your time and valuable feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for Area Chair and Reviewers

We thank the Area Chair and reviewers for their thoughtful feedback. Reviewers highlighted the paper’s clear motivation, strong performance on dynamic manipulation tasks, and well-controlled ablations. Below, we summarize the main results and clarifications from the rebuttal.


1. An event-aligned visual action reasoning framework focused on task-critical interactions

Our paper proposes a novel event-aligned visual action reasoning framework that focuses future prediction on task-critical interactions to reason more carefully over crucial state changes. Rather than inheriting a predefined temporal structure from sampled trajectories, our formulation enables the reasoning process to adapt its granularity to the interaction dynamics. WALL-WM’s event mode with uniform temporal sampling requires externally supplied next-event descriptions for each segment, whereas our framework performs event-aligned visual action reasoning directly from the global task instruction and observations. Unlike SKIP’s action-conditioned video interpolation, our method predicts actions for closed-loop execution.

The original matched-budget ablations provide controlled evidence for the framework (Tab. 4). With the same number of supervised slots, DOMINO success increases from 29.17% with Episode Uniform, to 39.60% with Event Uniform, and to 42.83% with our framework, supporting the importance of aligning prediction with interactions and preserving detailed evidence of crucial state changes.

The rebuttal directly addresses FP5c and B73v’s attribution question. The direct-action control retains the same event-aligned sampling, resampled action targets, and execution-validity head while removing access to predicted future visual context. The full model achieves 42.83% SR versus 30.69% for this control, demonstrating the benefit of imagined visual context to provide spatial ground for action prediction (Sec. C.4, Tab. C5).


2. An execution validity head that supports continuous action execution

Our execution validity head identifies the valid portion of each predicted action sequence under event-aligned reasoning. By avoiding the execution of redundant actions, it supports greater action continuity and smoother execution. The head is lightweight, with only 3,073 parameters and less than 0.1 ms of overhead.

To answer B73v’s question about its effect specifically at execution time, we evaluated inference-time masking with the trained checkpoint held fixed. Disabling masking reduces SR from 42.83% to 38.94% (Tab. 4), directly demonstrating the benefit of using the predicted validity labels during execution.


3. Strong performance on challenging dynamic manipulation, with robust gains and efficient deployment

The original evaluation demonstrates a 10.26 percentage-point improvement over LingBot-VA on DOMINO (42.83% versus 32.57% SR), together with stronger transfer to harder dynamic settings and competitive performance on RoboTwin 2.0.

The rebuttal reinforces this performance advantage across repeated runs and expanded real-robot evaluation, addressing FP5c and B73v’s concerns. Across three seeds, mean DOMINO success is 43.25% versus 32.96%. With the real-robot evaluation expanded to 75 rollouts per task, average success is 48.89% versus 41.33%, a 7.56 percentage-point improvement (Sec. C.1, Tab. C1).

The full framework also provides practical deployment benefits. Beyond the original analysis of 738 episodes solved by both methods, evaluating all successful episodes for each method shows 38.0% fewer action commands (164.23 → 101.80), 28.8% fewer visual-rollout requests (6.15 → 4.38), and a 31.6% reduction in mean wall-clock time per successful episode. These measurements address B73v’s question about whether the efficiency advantage extends beyond the shared-success subset.


4. The reproducibility and applicability questions are addressed in the revision

The event-partition procedure, including the F = 1 case, and the mapping from source frames to VAE temporal groups and validity labels are now fully specified (Sec. D.3, Alg. 1, App. A). For real-robot demonstrations, annotations use RGB observations and gripper signals. Auditing all 179 demonstrations found that 173 required no manual adjustment. The App. C.2 numbers now are corrected to match Tab. 2 and Tab. C3. The revision also includes failure analysis on mid-air transport tasks and discusses task-adaptive transition budgets, addressing TZD8’s questions (Sec. C.7, Sec. 6).


We would appreciate feedback from Reviewers FP5c and B73v on whether these additional results and clarifications address their concerns. We are happy to clarify any remaining points before the discussion closes. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.