- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
As I understand it, the main idea is to give the low-level policy some reusable context: what it is doing, which object it is working with, and where to focus, maybe heuristic. A separate learned module decides when to update that context.
The authors test the system in simulation and on a real robot, with ablations on the update strategy and individual components.
But I still have concerns about how reliable the system is.(Especially generalization & latency. important for real excution)
Critical strengths
1.The system is well put together, and the different modules fit together well.
2.The paper is easy to follow, and the figures explain the pipeline well.
3.Update high-level context only when needed while continuously adjusting actions makes sense.
Critical weaknesses
-
About latency. I’d like to see the timing of both levels, especially the difference between KEEP and REFRESH. Does the action policy wait for a refresh? If not, how long does it keep using stale context? What about the latency of the whole system?
-
TWO LEVEL SETUP MAKE IT UNCLEAR. It’s hard to tell whether the gains come from better refresh decisions or from the action policy adapting to this particular setup. The real-robot and unseen-object results show overall performance, but don’t isolate the watcher’s reliability in new settings or how much the gains depend on this specific combination.
-
To me, this feels more like a well-built system than a clear methodological advance. Providing structured context and learning when to update it seems useful, BUT I’m not yet convinced what it adds beyond existing fast–slow, caching, or event-triggered approaches. Dress it up a bit?
-
The ablations show that removing these components hurts performance, but don’t explain what distinct information each one adds.(SR not enough; Why Three? one more can work?In fact, this part is the most novel in paper, but unclear now.)
Questions for authors
SEE Critical Weaknesses Please;
5 comments
W3. Relation to fast--slow, caching, and event-triggered methods.
AsyncMoT makes the validity interval of interaction context an explicit policy variable. Its methodological advance is to jointly model interaction content and interaction timing: Stage, Object, and Region define the context to be retained, while the Boundary Watcher learns when that context should be refreshed.
The key distinction is the unit at which computation is scheduled. Fast--slow policies typically operate at a task or subtask level and do not explicitly detect interaction changes within that level. A target or action region may change while the nominal operation remains the same, making the previously computed guidance invalid. KV caching addresses the cost of reuse but does not provide a learned invalidation rule. Event-triggered computation reacts to generic input or control changes, which do not directly determine whether the current interaction guidance remains applicable.
AsyncMoT implements this interaction-level scheduling through the K/V cache and the Mixture-of-Transformers action expert. The Boundary Watcher determines when the persistent Stage/Object/Region context should be replaced; between such updates, the action expert combines the retained interaction K/V features with current visual and proprioceptive features to continuously generate actions. Thus, the slow computation persists at the interaction level, while motor control remains responsive at every step.
The refresh-schedule ablations isolate this design: with the model, training data, action objective, and evaluation protocol fixed, changing only the refresh rule produces substantial differences across simulation and real-robot settings. The gain therefore comes from learning the validity interval of interaction guidance, rather than from using a generic fast--slow decomposition, cache, or event trigger.
W4. What does each interaction representation contribute, and why three?
We agree that the previous leave-one-out ablation did not make the role of each representation sufficiently explicit. We therefore added single-branch ablations on LIBERO under the same training and evaluation protocol as the full model. All variants use the same demonstrations, backbone, two-stage training schedule, Boundary Watcher, action objective, action horizon, and evaluation protocol; only the available interaction representation is changed.
| Configuration | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|
| Stage only | 83.8 | 85.4 | 82.6 | 76.6 | 82.1 |
| Object only | 74.4 | 75.8 | 72.6 | 62.0 | 71.2 |
| Region only | 76.2 | 77.8 | 74.8 | 64.8 | 73.4 |
| Full Stage + Object + Region | 98.8 | 99.2 | 98.6 | 95.8 | 98.1 |
Each single representation provides useful but incomplete guidance, whereas the full model performs substantially better across all four suites. Their roles are also distinct: Stage represents the current interaction phase, Object identifies the manipulated entity, and Region specifies the action-relevant spatial grounding. The existing leave-one-out results further show that removing any one branch degrades the complete model.
The choice of three is therefore a compact, task-grounded factorization of interaction state along three complementary axes: when the interaction is taking place, what entity is involved, and where the action should be grounded. A single axis cannot specify the complete interaction, while the three together preserve semantic progress, target identity, and spatial grounding. This factorization also leaves a natural interface for adding further task-specific factors, such as contact or affordance, when they provide distinct interaction information.
Dear Reviewer,
I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.
We thank the reviewer for recognizing the coherence of our system, the clarity of the presentation, and the rationale for updating high-level context while continuously adapting actions. Below, we address the concerns about latency, the source of performance gains, methodological novelty, and the distinct roles of the interaction components.
W1. Latency and execution during refresh.
On REFRESH, the action policy waits for the updated interaction context before generating the next action chunk, while the robot continues executing previously generated actions. A pause occurs only if the action buffer is exhausted before the refreshed chunk is ready. On KEEP, the policy reuses the cached interaction context until the next REFRESH decision without recomputing the high-level state.
We measured average latencies on a single NVIDIA A100 GPU. Watcher latency measures the time to make a KEEP/REFRESH decision. High-level refresh includes updating the interaction context and its cache. Action-policy latency measures the generation of an action chunk, excluding refresh and waiting time. End-to-end latency is measured from observation availability to execution of the first action in the resulting chunk.
Pauses caused by an exhausted action buffer account for 0.8% and 1.6% of total execution time on LIBERO-Plus and RoboCasa365, respectively.
W2. Refresh reliability versus action-policy adaptation.
Thank you for raising this concern. We isolate the contribution of refresh decisions through controlled refresh-schedule ablations: all schedules use the same trained action policy, interaction representations, training data, action objective, and evaluation protocol. The action policy is not retrained; only the test-time KEEP/REFRESH rule is changed.
The learned boundary-driven schedule outperforms fixed-rate and every-step updates on LIBERO-Plus, RoboCasa365, and RoboMME:
The same comparison on real-robot OOD Tidy-Up gives 80.0% for our schedule, compared with 25.0%, 30.0%, and 30.0% for 1 Hz, 2 Hz, and every-step reconstruction, respectively. Since the action policy is fixed across these conditions, the performance gap isolates the contribution of the refresh decisions rather than adaptation of the action policy to a particular two-level schedule.
The watcher’s behavior in new scenes is directly shown in Appendix~F. The traces show REFRESH when the active interaction target changes and KEEP during continued execution within the same interaction. Thus, the watcher responds to changes in interaction-state validity rather than to visual or motor changes alone.
Together, the controlled schedule ablations and the unseen execution traces show that AsyncMoT benefits from learning when the retained interaction context remains valid and when it must be replaced, rather than simply from combining a high-level module with an action expert.