acceptodds
Under review as a conference paper at ICLR 2027

AsyncMoT: Interaction-Guided Asynchronous Mixture of Transformers for Generalized and Robust Robotic Manipulation

Abstract

Robotic manipulation involves interaction contexts that persist across control steps but change at irregular, interaction-dependent moments. Yet existing policies typically couple interaction reasoning to control frequency or predefined task structure, creating a mismatch between when robots act and when they need to reason. We introduce AsyncMoT, an interaction-guided asynchronous Mixture of Transformers that makes when to reason an explicit policy decision. Given a high-level task instruction, AsyncMoT maintains a structured interaction state capturing the current stage, target object, and action-relevant region, and selectively refreshes it as the interaction evolves. Between updates, cached Key-Value features preserve interaction guidance across control steps, while the action expert continuously adapts motor commands to current visual and proprioceptive observations. This decoupling further enables interaction supervision to be shared across heterogeneous robot tasks and human videos, without requiring human-to-robot action correspondence. AsyncMoT achieves average success rates of 98.1% on LIBERO and 88.9% on LIBERO-Plus, 66.3% on RoboCasa-24 and 35.9% on RoboCasa365, demonstrating strong performance across long-horizon, distribution-shifted, and compositional manipulation settings.

Primary area:
applications to robotics, autonomy, planning
Keywords:
1.Vision-Language-Action Models 2.Robot Manipulation 3.Hierarchical Policies 4.Learning from Human Videos

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_n57rOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

As I understand it, the main idea is to give the low-level policy some reusable context: what it is doing, which object it is working with, and where to focus, maybe heuristic. A separate learned module decides when to update that context.

The authors test the system in simulation and on a real robot, with ablations on the update strategy and individual components.

But I still have concerns about how reliable the system is.(Especially generalization & latency. important for real excution)

Critical strengths

1.The system is well put together, and the different modules fit together well.

2.The paper is easy to follow, and the figures explain the pipeline well.

3.Update high-level context only when needed while continuously adjusting actions makes sense.

Critical weaknesses
  1. About latency. I’d like to see the timing of both levels, especially the difference between KEEP and REFRESH. Does the action policy wait for a refresh? If not, how long does it keep using stale context? What about the latency of the whole system?

  2. TWO LEVEL SETUP MAKE IT UNCLEAR. It’s hard to tell whether the gains come from better refresh decisions or from the action policy adapting to this particular setup. The real-robot and unseen-object results show overall performance, but don’t isolate the watcher’s reliability in new settings or how much the gains depend on this specific combination.

  3. To me, this feels more like a well-built system than a clear methodological advance. Providing structured context and learning when to update it seems useful, BUT I’m not yet convinced what it adds beyond existing fast–slow, caching, or event-triggered approaches. Dress it up a bit?

  4. The ablations show that removing these components hurts performance, but don’t explain what distinct information each one adds.(SR not enough; Why Three? one more can work?In fact, this part is the most novel in paper, but unclear now.)

Questions for authors

SEE Critical Weaknesses Please;

5 comments

We thank the reviewer for recognizing the coherence of our system, the clarity of the presentation, and the rationale for updating high-level context while continuously adapting actions. Below, we address the concerns about latency, the source of performance gains, methodological novelty, and the distinct roles of the interaction components.

W1. Latency and execution during refresh.

On REFRESH, the action policy waits for the updated interaction context before generating the next action chunk, while the robot continues executing previously generated actions. A pause occurs only if the action buffer is exhausted before the refreshed chunk is ready. On KEEP, the policy reuses the cached interaction context until the next REFRESH decision without recomputing the high-level state.

We measured average latencies on a single NVIDIA A100 GPU. Watcher latency measures the time to make a KEEP/REFRESH decision. High-level refresh includes updating the interaction context and its cache. Action-policy latency measures the generation of an action chunk, excluding refresh and waiting time. End-to-end latency is measured from observation availability to execution of the first action in the resulting chunk.

Average latency (ms)LIBERO-PlusRoboCasa365
Watcher3.23.3
High-level refresh and cache update47.648.8
Action policy (KEEP / REFRESH)29.4 / 29.729.8 / 30.1
End-to-end (KEEP / REFRESH)112.6 / 158.3118.4 / 174.7

Pauses caused by an exhausted action buffer account for 0.8% and 1.6% of total execution time on LIBERO-Plus and RoboCasa365, respectively.

W2. Refresh reliability versus action-policy adaptation.

Thank you for raising this concern. We isolate the contribution of refresh decisions through controlled refresh-schedule ablations: all schedules use the same trained action policy, interaction representations, training data, action objective, and evaluation protocol. The action policy is not retrained; only the test-time KEEP/REFRESH rule is changed.

The learned boundary-driven schedule outperforms fixed-rate and every-step updates on LIBERO-Plus, RoboCasa365, and RoboMME:

Refresh scheduleLIBERO-PlusRoboCasa365RoboMME
1 Hz76.521.817.0
2 Hz75.121.218.8
Every step79.423.620.5
Ours88.235.928.1

The same comparison on real-robot OOD Tidy-Up gives 80.0% for our schedule, compared with 25.0%, 30.0%, and 30.0% for 1 Hz, 2 Hz, and every-step reconstruction, respectively. Since the action policy is fixed across these conditions, the performance gap isolates the contribution of the refresh decisions rather than adaptation of the action policy to a particular two-level schedule.

The watcher’s behavior in new scenes is directly shown in Appendix~F. The traces show REFRESH when the active interaction target changes and KEEP during continued execution within the same interaction. Thus, the watcher responds to changes in interaction-state validity rather than to visual or motor changes alone.

Together, the controlled schedule ablations and the unseen execution traces show that AsyncMoT benefits from learning when the retained interaction context remains valid and when it must be replaced, rather than simply from combining a high-level module with an action expert.

W3. Relation to fast--slow, caching, and event-triggered methods.

AsyncMoT makes the validity interval of interaction context an explicit policy variable. Its methodological advance is to jointly model interaction content and interaction timing: Stage, Object, and Region define the context to be retained, while the Boundary Watcher learns when that context should be refreshed.

The key distinction is the unit at which computation is scheduled. Fast--slow policies typically operate at a task or subtask level and do not explicitly detect interaction changes within that level. A target or action region may change while the nominal operation remains the same, making the previously computed guidance invalid. KV caching addresses the cost of reuse but does not provide a learned invalidation rule. Event-triggered computation reacts to generic input or control changes, which do not directly determine whether the current interaction guidance remains applicable.

AsyncMoT implements this interaction-level scheduling through the K/V cache and the Mixture-of-Transformers action expert. The Boundary Watcher determines when the persistent Stage/Object/Region context should be replaced; between such updates, the action expert combines the retained interaction K/V features with current visual and proprioceptive features to continuously generate actions. Thus, the slow computation persists at the interaction level, while motor control remains responsive at every step.

The refresh-schedule ablations isolate this design: with the model, training data, action objective, and evaluation protocol fixed, changing only the refresh rule produces substantial differences across simulation and real-robot settings. The gain therefore comes from learning the validity interval of interaction guidance, rather than from using a generic fast--slow decomposition, cache, or event trigger.

W4. What does each interaction representation contribute, and why three?

We agree that the previous leave-one-out ablation did not make the role of each representation sufficiently explicit. We therefore added single-branch ablations on LIBERO under the same training and evaluation protocol as the full model. All variants use the same demonstrations, backbone, two-stage training schedule, Boundary Watcher, action objective, action horizon, and evaluation protocol; only the available interaction representation is changed.

ConfigurationSpatialObjectGoalLongAvg.
Stage only83.885.482.676.682.1
Object only74.475.872.662.071.2
Region only76.277.874.864.873.4
Full Stage + Object + Region98.899.298.695.898.1

Each single representation provides useful but incomplete guidance, whereas the full model performs substantially better across all four suites. Their roles are also distinct: Stage represents the current interaction phase, Object identifies the manipulated entity, and Region specifies the action-relevant spatial grounding. The existing leave-one-out results further show that removing any one branch degrades the complete model.

The choice of three is therefore a compact, task-grounded factorization of interaction state along three complementary axes: when the interaction is taking place, what entity is involved, and where the action should be grounded. A single axis cannot specify the complete interaction, while the three together preserve semantic progress, target identity, and spatial grounding. This factorization also leaves a natural interface for adding further task-specific factors, such as contact or affordance, when they provide distinct interaction information.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_jAnxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AsyncMoT treats long-horizon manipulation as two clocks. Visual observations, proprioception, and motor commands update every control step, while a structured interaction state is kept in a key–value cache and refreshed only when a lightweight Boundary Watcher outputs REFRESH rather than KEEP. Training is two-stage: interaction targets first, including human video, then refresh timing and robot control. Superior performance is reported over several simulation and real robot benchmarks.

Critical strengths

A controlled LIBERO schedule comparison is the paper’s strongest positive result. Table 5 Q1 varies the refresh rule. Boundary-driven refresh reaches 98.1% average and 95.8% on Long, against 91.8% / 86.0% for every-step reconstruction, 90.9% / 84.8% at 1 Hz, 90.3% / 84.0% at 2 Hz, and 67.0% / 49.6% for the initial state only. On this benchmark, the learned boundary outperforms a fixed clock, and raising that clock from 1 Hz to 2 Hz does not help. This is consequential because it is the only direct timing comparison in the manuscript, and it is confined to LIBERO, where every-step reconstruction is already at 91.8% and the boundary targets are the kinematic labels in Appendix B. It raises the paper’s floor. The recommendation turns on whether that timing effect is what produces the harder-benchmark and real-robot gains.

Critical weaknesses

The headline gains are not yet attributable to asynchronous refresh. Table 3’s “Ours” row (63.7 / 30.4 / 10.1 / 35.9) is the with-human-video setting in Table 8. Robot-only RoboCasa365 overall is 32.1, tied with RLDX-1. The introduction compares 35.9% with GR00T N1.6 at 21.9% and does not put this data difference next to that comparison. The ablation that tests “when versus how often” is only on LIBERO (Table 5), where every-step reconstruction is already at 91.8%. There is no timing ablation on RoboCasa365, RoboMME, or the real robot. Table 1 reports published baseline scores, AsyncMoT’s size is left as “—”, and only RoboMME has a seed interval. Pretraining uses DROID, RH20T, HOI4D, and Open-AoE (Section 3.3), with no matched-data control.

The interaction state is neither specified tightly nor shown on a new scene. Appendix B builds boundary labels from gripper transitions, low-motion intervals, and changes in end-effector direction, then fills Stage with GPT-5.6, Object with Qwen3-VL plus a frozen FLUX encoder, and Region with SAM. The manuscript gives no label audit, no boundary precision or recall, and no test-time refresh rate. The rule that turns ztBz^B_t into KEEP or REFRESH is not stated, and NN, w+w_+, the weights in Eqs. 4–5, and the chunk horizon HH have no values. Laction\mathcal{L}_{\text{action}} is only “the original continuous-action objective” (Section 3.2). “w/o Stage-2” still scores 86.3% on LIBERO (Table 5). Separately, zero-shot interaction prediction is never shown qualitatively. Figure 4(a) is an execution rollout of Tidy-Up with unseen objects. Predicted Stage text, object grounding, region heatmaps, and KEEP/REFRESH traces do not appear for that rollout, for dynamic target displacement, or for any other held-out scene in the main text or the appendix. Success rates alone leave open the possibility that a stronger low-level policy succeeded while the cached interaction state was stale or wrong. That is the mechanism the paper asks the reader to accept.

Questions for authors
  1. Please show the interaction prediction on new scenes, especially the real robot. For dynamic target displacement and out-of-distribution Tidy-Up, plot the predicted Stage, object localization, region heatmap, and KEEP/REFRESH decision over time beside the execution frames in Figure 4(a). State the threshold on ztBz^B_t and the resulting refresh rate.
  2. Please define the action expert and Laction\mathcal{L}_{\text{action}} (parameterization and chunk horizon HH), give λS,λO,λR,λM,λB\lambda_S, \lambda_O, \lambda_R, \lambda_M, \lambda_B, and state what is removed in “w/o Stage-2”, which still scores 86.3% on LIBERO (Table 5).
7 comments

Thank you for your careful review and for recognizing the controlled refresh-schedule comparison on LIBERO. Your comments highlight the need to better isolate the contribution of asynchronous refresh, validate interaction predictions in unseen scenes, and clarify the implementation details. Below, we address your two main weaknesses point by point and answer the related questions.

W1. Attribution of the gains to asynchronous refresh

Thank you for raising this concern. We agree that the contribution of refresh timing should be evaluated separately from the benefit of additional training data. We address these points below.

Response 1.1 — Separating the data setting from the baseline comparison.

We agree that the original presentation conflated different training settings. The 35.9% RoboCasa365 result is obtained with robot demonstrations plus human-video supervision, whereas robot-only AsyncMoT achieves 32.1%. We now report these two settings separately and identify the data setting alongside the headline result.

For the robot-only comparison, AsyncMoT reaches the same 32.1% overall score as RLDX-1. This comparison is favorable to AsyncMoT in terms of model and training scale: AsyncMoT uses two PaliGemma-3B branches (approximately 6B backbone parameters), whereas RLDX-1 is built on an 8B Qwen3-VL backbone and uses a three-stage training pipeline with large-scale public robot data, in-house robot data, and synthetic data. The RLDX-1 report also describes approximately 195 hours of pre-training and 15 hours of mid-training on 64 NVIDIA H200 GPUs. Thus, the robot-only result shows that AsyncMoT remains competitive despite using a smaller backbone and a substantially lighter training setup.

The increase from 32.1% to 35.9% is therefore the contribution of adding human-video supervision to AsyncMoT. We have revised Table 3 and the introduction to make the robot-only and robot-plus-human-video settings explicit.

Response 1.2 — Refresh timing beyond LIBERO

We agree that the LIBERO ablation alone is insufficient to establish the benefit of asynchronous refresh in more challenging settings. We therefore added ablations on LIBERO-Plus, RoboCasa365, RoboMME, and real-world OOD Tidy-Up. For each benchmark, all schedules use the same model, training data, action objective, and evaluation protocol; only the interaction-refresh schedule is changed. The results are:

Refresh scheduleLIBERO-PlusRoboCasa365RoboMME
1 Hz76.521.817.0
2 Hz75.121.218.8
Every step79.423.620.5
Ours88.235.928.1

The benefit persists across all three benchmarks. In particular, every-step reconstruction achieves 23.6% on RoboCasa365, versus 35.9% for our policy. These results support retaining useful interaction guidance between selective updates rather than reconstructing it at every step.

On real-world OOD Tidy-Up, the gap is larger: our policy achieves 80.0%, compared with 25.0%, 30.0%, and 30.0% for 1 Hz, 2 Hz, and every-step reconstruction, respectively. These additional experiments address the concern that the evidence for asynchronous refresh was limited to LIBERO.

Response 1.3 — Matched-data control.

To isolate the effect of interaction supervision from the additional pretraining data, we trained a data-matched π0.5\pi_{0.5} baseline using the same four sources: DROID, RH20T, HOI4D, and Open-AoE. For HOI4D and Open-AoE, the two human-video sources, we followed the action-alignment approach of EgoVerse [1] and mapped hand trajectories into a shared end-effector representation consisting of position, orientation, and gripper state. We mixed these retargeted trajectories with robot demonstrations at a fixed 2:1 robot-to-human ratio and trained the action expert with the same action objective and downstream training protocol.

With the matched data, π0.5\pi_{0.5} improves only from 85.7% to 85.9% on LIBERO-Plus and from 52.9% to 53.4% on RoboCasa-24, whereas AsyncMoT achieves 88.5% and 66.4%, respectively. This control shows that the gains are not explained by the additional pretraining data alone.

[1] Ryan Punamiya et al. EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World. 2026.

Response 1.4 — Model size and variability.

AsyncMoT uses two PaliGemma-3B branches, corresponding to approximately 6B backbone parameters. The lightweight Boundary Watcher adds negligible overhead. We additionally report four-seed results on LIBERO-Plus and RoboCasa-24: 88.5±0.3%88.5 \pm 0.3\% and 66.4±0.4%66.4 \pm 0.4\%, respectively.

W2. Interaction-state specification and validation.

We appreciate this concern. Task success alone does not establish whether the retained interaction state remains useful during execution. Below, we clarify the supervision and implementation, examine interaction predictions in new scenes, and explain the “w/o Stage-2” ablation.

Response 2.1 — Annotation quality.

We audited the interaction annotations on 900 training trajectories, with 300 trajectories each from LIBERO, RoboCasa-24, and RoboCasa365. For Boundary, Qwen3.8-Max independently annotated the videos without access to our labels. Using one-to-one matching within five frames, precision/recall are 98.34%/97.01%, 97.92%/95.30%, and 96.64%/94.88%, respectively.

On the same trajectories, GPT-6 Astra independently reviewed the candidate Stage, Object, and Region annotations against the task instruction and observed execution. It was given the annotation being evaluated, but had no access to the model predictions or the annotation-generation procedure. It checks whether the Stage description is semantically consistent with the executed interaction interval, whether the Object identifies the manipulated entity, and whether the Region is spatially valid and free of background leakage. The average acceptance rates over the audited interaction annotations are 99.1%, 94.6%, and 93.4% for Stage, Object, and Region, respectively.

The main ambiguity occurs when a cup is placed into a bowl before the loaded bowl is transported: the bowl changes from a destination into the manipulated object, while the annotation must distinguish the intended target from the cup--bowl assembly.

Response 2.2 — Test-time interaction predictions and refresh decisions. (Q1)

We agree that task success alone cannot establish whether the cached interaction state is appropriate. We now provide time-aligned execution frames, interaction predictions, and KEEP/REFRESH traces for real-robot OOD Tidy-Up, dynamic target displacement, and an unseen RoboCasa365 scene. These show the interaction state used by the action policy as the target changes across objects in Tidy-Up, as the target moves during execution, and from the left to the right cabinet door in RoboCasa365. At REFRESH, the predicted state updates the interaction K/V cache; during KEEP, that same cache remains in use. The corresponding visualizations and details are provided in Appendix~F.

We visualize Object by decoding the predicted latent with the frozen FLUX decoder, and Region by overlaying the predicted 24×2424\times24 heatmap. Since Stage is a continuous embedding, we evaluate it against offline Stage descriptions annotated using GPT-6 Astra and encoded with the same Stage encoder used during training, obtaining a cosine similarity of 0.963.

The KEEP/REFRESH decision thresholds the boundary probability σ(ztB)\sigma(z_t^B) at 0.5. The actual refresh rates are 0.17 Hz for Tidy-Up and 0.63 Hz for RoboCasa365. The dynamic-target trace further shows repeated refreshes as the target grounding changes during execution. The paired visualizations and traces show what interaction guidance is retained between updates and when it is replaced.

Response 2.3 — Action expert and implementation details (Q2)

Our action expert is a Transformer that predicts continuous action chunks through conditional flow matching, conditioned on current observations, the task instruction, and cached interaction features. For LIBERO/LIBERO-Plus, RoboCasa-24, and the real robot, each action contains three translation components, three rotation components, and one gripper command. RoboCasa365 uses 12-dimensional actions that additionally include base and control-mode commands. RoboMME predicts seven joint displacements and one gripper command. The chunk horizon HH is 10 for LIBERO/LIBERO-Plus and the real robot, 32 for RoboCasa-24, and 16 for RoboCasa365 and RoboMME.

For a normalized target action chunk AA and Gaussian noise ϵ\epsilon, we use Aτ=(1−τ)A+τϵA^{\tau}=(1-\tau)A+\tau\epsilon and minimize Laction=E[∣vθ(Aτ,τ;ct)−(ϵ−A)∣F2]/(Hd)L_{\mathrm{action}}=\mathbb{E}[\\|v_{\theta}(A^{\tau},\tau;c_t)-(\epsilon-A)\\|_F^2]/(Hd).

Here, ctc_t denotes the conditioning information described above, and dd is the number of valid action dimensions.

The Boundary Watcher uses N=8N=8 recent steps. We set w+=nKEEP/nREFRESHw_{+}=n_{\mathrm{KEEP}}/n_{\mathrm{REFRESH}} using valid training labels to balance the two classes; uncertain labels are excluded.

For joint training, the loss weights are λS=0.20\lambda_S=0.20, λO=0.15\lambda_O=0.15, λR=0.15\lambda_R=0.15, λM=0.40\lambda_M=0.40, and λB=0.20\lambda_B=0.20 across benchmarks. The coefficient of the action loss is 1.

Response 2.4 — Interpretation of “w/o Stage-2” (Q2)

“w/o Stage-2” removes the entire Stage-2 optimization, including all objectives defined in that stage, while retaining the Stage-1 initialization and all interaction modules, followed by the same benchmark-specific fine-tuning as the full model. It therefore does not remove the interaction branch or the action supervision used during subsequent fine-tuning. Training on LIBERO demonstrations still yields 86.3% success, compared with 98.1% for the full model.

We added the same ablation on the more challenging RoboCasa-24 benchmark, where omitting Stage-2 yields only 12.9% success. The performance drop is much larger on RoboCasa-24 than on LIBERO, showing that the missing Stage-2 training is substantially more important for the harder benchmark.

MethodP&PDoorsDrawersKnobsLeversButtonsInsertionAvg.
w/o Stage-134.660.368.550.556.764.325.548.8
w/o Stage-21.122.328.515.523.716.71.512.9
Reviewer_ENoqOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AsyncMoT aims to address a "when to reason" problem that is not fully explored in existing works on long-horizon robotic manipulation.

To this end, AsyncMoT provides a VLA that follows a fast-slow asynchronous architecture with a high-level VLM and a low-level action expert, highlighting the capability to proactively refresh interaction context used to condition the low-level action expert.

Efforts are made in the High-Level VLM side to realize that objective, with the main design being a binary classifier, namely the boundary-watcher, that determines when to refresh the KV cache produced by VLM. Other coherent designs to make this watcher effective include the three query tokens that are used to study interaction representations in multi-task representation learning, through which the watcher can make decisions focusing on interaction stage changes, and the low-level action expert can get compact high-level conditioning to generate actions adaptively.

To demonstrate interaction between objects and embodiments, AsyncMoT models three representations, namely the Stage, Object, and Region branches. General LLM and VLM models such as ChatGPT and Qwen-VL are used to annotate ground truth for stage description, Cropped object image,and actionrelevant spatial heatmap. These representations are then used to supervise the VLM in pretraining stage.

The results evaluated on simulation benchmarks demonstrate promising performance on long-horizon manipulation compared with baselines. The ablation experiments on refreshing policy, interaction representations, training strategy shows positive results.

Critical strengths
  1. The problem of "when to reason" is well-defined, and the insight of proactively refreshing the interaction context is inspiring.
  2. The solution to the problem, namely the design of the boundary-watcher, is simple and elegant, with experiments showing promising performance.
  3. The annotation method for those three interaction representations is easy to follow, and its ablation study strengthens the evidence of their effectiveness.
Critical weaknesses
  1. The annotation pipeline for the ground truth of the boundary-watcher is not well-presented in the main text, and its annotation quality is not assessed directly. In my view, this is an essential part that determines the effectiveness of the architecture and serves as the foundation of the refreshing mechanism.
Questions for authors

My questions are for the annotation pipeline mentioned in Weakness1:

  1. Can you present it in a clearer way in the main text?
  2. Are there any limitations? For example, is the pipeline capable of annotating interaction stage refreshing points effectively on tasks that involve dynamic scenes or high-dimensional embodiments like dexterous hands?
  3. I'm curious about the result of the annotation pipeline. When would the ground truth suggest refreshing the interaction context and why would such refresh moments be helpful from your perspective? Some examples from actual data trajectories would help.
  4. What is the quality of the annotation pipeline? Can you evaluate the quality in a more direct way instead of using downstream task performance as indirect evidence? For example, you could use a general VLM or agents to annotate a subset of the data to get a validation set and compare the annotation results with yours. This would help me get a more comprehensive assessment of the basis of your boundary-watcher design.
3 comments

We thank the reviewer for the thoughtful feedback and positive assessment of AsyncMoT. The main concern is the clarity and reliability of boundary supervision, which we address below.

W1. Boundary Annotation Procedure and Quality.

Our responses to Q1–Q4 explain how the labels are constructed, discuss the pipeline’s limitations, illustrate refresh events with actual trajectories, and address annotation quality through direct evaluation.

Q1. Can you present the annotation pipeline more clearly in the main text?

Thank you for this suggestion. We have clarified the annotation procedure in the main text and provided the detailed protocol in Appendix E.1. For a demonstration of length TT, we read joint positions qtq_t, binary gripper states gtg_t, and the sampling interval Δt\Delta t. Joint velocities are estimated using central differences, with one-sided differences at the endpoints:

\qquad v_t=\|\dot q_t\|_2.$$ We identify candidate timestamps from gripper state transitions, $g_t\neq g_{t-1}$, and local speed minima: $$v_t<\min(v_{t-1},v_{t+1}).$$ Qwen3-VL-235B-A22B-Instruct reviews the task instruction and timestamped video with candidate frames marked. It retains, removes, adjusts, or merges candidates according to observed interaction changes and adds missed boundaries. A boundary is retained when a change in the interaction makes the existing guidance no longer applicable, including changes in the target or action-relevant region even when the semantic stage remains unchanged. Each accepted boundary includes supporting frames and a brief explanation; uncertain cases are excluded from supervision. The boundary frame and the five frames on either side receive REFRESH labels, while the remaining valid frames receive KEEP labels. **Q2. Applicability and limitations.** Our evaluation supports the pipeline’s applicability to both dynamic scenes and dexterous manipulation. We evaluated 30 trajectories from DOMINO, which features moving-object manipulation, and 40 trajectories from Adroit, which involves high-dimensional dexterous-hand control. Qwen3.8-Max reviewed the same trajectories without access to the initial labels. Using the same boundary-matching protocol as in Q4, precision/recall were 98.60%/96.52% on DOMINO and 97.50%/95.24% on Adroit. Further details are provided in Appendix E.2. The main limitations concern ambiguous contact transitions. In dynamic scenes, simultaneous target and robot motion can make contact onset difficult to distinguish from continued approach. For dexterous hands, progressive finger closure can obscure when a stable grasp is established. These ambiguities can cause boundary timing errors, particularly when the contact region is occluded. **Q3. Boundary labels and their interpretation.** The annotated boundaries mark moments when continued execution requires a change in the retained interaction context. These include completing one interaction and beginning another, or changing the target and action region while continuing the same operation. Refreshing matters because the robot may continue the correct type of motion while being guided toward an outdated target or region. For example, when the robot moves from one cabinet to another, the operation remains “closing,” but the target and action-relevant region change. REFRESH redirects subsequent actions to the new cabinet; otherwise, the robot could execute the correct motion toward the previous target. Conversely, during cup transport, the annotation indicates KEEP despite substantial viewpoint changes: the same object and destination still guide the task, while current observations support motor adjustments. We have added four annotated trajectory examples from RoboCasa365 and LIBERO-Plus, with further analysis in Appendix E.3. **Q4. Direct evaluation of annotation quality.** We sampled 300 training trajectories each from LIBERO, RoboCasa-24, and RoboCasa365, covering all training tasks in each benchmark. Qwen3.8-Max annotated the videos using the task instructions, without access to our pipeline’s labels. Using one-to-one boundary matching within 5 frames, precision (the fraction of our boundaries matched by the reference) and recall (the fraction of reference boundaries recovered by our pipeline) were 98.34%/97.01%, 97.92%/95.30%, and 96.64%/94.88%, respectively. The evaluation protocol and results are provided in Appendix E.4. The reference VLM annotated refreshes more frequently than our pipeline. Inspection showed that some additional boundaries marked brief pauses or motion adjustments within an ongoing interaction, while the retained guidance still correctly specified the target and action-relevant region. These events did not require updated interaction context, despite visible changes in execution. Together, the boundary-level results and examination of disagreements support the reliability of the supervision used to train the Boundary Watcher.

I appreciate the authors' timely reply and further clarification regarding the questions and weaknesses raised. I remain positive toward this work and will maintain my rating.

Thank you for your positive assessment and for your time and constructive feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for recognizing the clear formulation of “when to reason” and the simple and elegant Boundary Watcher (ENoq), the controlled refresh-schedule comparison (jAnx), and the coherent integration of persistent context with continuous action adaptation (n57r). We also appreciate the positive assessment of the interaction-representation ablations (ENoq). Our rebuttal addresses four central concerns with additional experiments and execution-level evidence.

1. Reliability of interaction supervision. We audited 900 training trajectories against independent boundary annotations, obtaining 96.64%–98.34% precision and 94.88%–97.01% recall across LIBERO, RoboCasa-24, and RoboCasa365. Separate assessments of Stage, Object, and Region yield acceptance rates of 99.1%, 94.6%, and 93.4%, respectively. The boundary pipeline also achieves 98.60%/96.52% precision/recall on dynamic manipulation trajectories from DOMINO and 97.50%/95.24% on dexterous-hand trajectories from Adroit. These evaluations directly assess supervision quality beyond downstream task success. Appendix E documents the protocols and limitations, including ambiguous contact onset and progressive finger closure.

2. Validity of interaction guidance during execution. Appendix F shows time-aligned execution frames, predicted interaction states, and KEEP/REFRESH traces. In real-robot OOD Tidy-Up, the predicted target changes across manipulated objects, with an average refresh rate of 0.17 Hz. During dynamic target displacement, repeated refreshes update the target grounding as the object moves. In unseen RoboCasa365, the target and action region switch from the left to the right cabinet door while the operation remains “closing”; the refresh rate is 0.63 Hz. These examples show why an interaction update need not coincide with a semantic stage change: the operation can remain valid while its target or spatial guidance becomes outdated. At REFRESH, the updated predictions replace the interaction K/V cache; during KEEP, the retained cache continues to guide execution.

3. Attribution of the performance gains. We extended refresh-schedule comparisons to three further simulation benchmarks and real-robot OOD Tidy-Up, keeping the trained policy fixed and changing only the test-time refresh rule. Compared with every-step reconstruction, boundary-driven refresh raises success from 79.4% to 88.2% on LIBERO-Plus, 23.6% to 35.9% on RoboCasa365, and 20.5% to 28.1% on RoboMME. On real-robot OOD Tidy-Up, success rises from 30.0% to 80.0%. Thus, more frequent reconstruction does not substitute for selecting when to refresh. Separately, matching the additional training data for π0.5\pi_{0.5} yields only +0.2 points on LIBERO-Plus and +0.5 on RoboCasa-24, indicating that additional data alone does not explain AsyncMoT’s gains.

4. Methodological contribution and complementarity. AsyncMoT learns both what interaction guidance to retain and when to replace it: Stage, Object, and Region specify persistent guidance, while the Boundary Watcher determines when it becomes outdated. This separates changes requiring new high-level guidance from motor adjustments handled by the action policy. The controlled refresh comparisons support the learned timing, while the added single-branch ablations support the representation design: on LIBERO, the full model achieves 98.1%, versus 82.1% with Stage only, 71.2% with Object only, and 73.4% with Region only.

We also quantify execution latency. On RoboCasa365, end-to-end KEEP/REFRESH latency is 118.4/174.7 ms on a single A100. The robot executes buffered actions during refresh, with buffer-exhaustion pauses accounting for 1.6% of execution time. We additionally clarify the action objective, implementation parameters, and training retained in the Stage-2 ablation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.