acceptodds
Under review as a conference paper at ICLR 2027

: Reactive Real-time Flow Policies

Abstract

Generalist manipulation policies increasingly take the form of action-chunking flow policies built on large pretrained backbones. Such chunks run open-loop, so the policy cannot react to sensory input arriving mid-execution, sacrificing reactivity. Replanning more often would restore it, but the perception-to-action pipeline (a large backbone plus multiple denoising steps) is too slow: this latency forbids frequent replanning and leaves committed actions stale, making such policies ill-suited for dynamic, closed-loop control. We present , which makes these policies reactive and real-time while retaining large backbones, expressive multi-modal policies, and multi-action prediction. Built on the per-position noise schedule of diffusion forcing, contributes two ideas. First, it splits conditioning into a fast channel (proprioception, fresh every tick) and an asynchronously updated slow channel (vision-language features), so the policy reacts to proprioception within a chunk while tolerating stale vision. Second, a latency-adaptive flow schedule treats in-flight actions as inpainting conditioning and emits actions in one denoising step per call, letting one trained model adapt to varying hardware latency. Requiring minimal modification to existing architectures, can be finetuned from a pretrained policy: applied to GR00T-N1.7 on a real xArm6+XHand platform, it replans closed-loop roughly faster than the base policy, acting on a fresh observation every ms. Across simulation and real-world manipulation tasks, improves the success rate by up to 25% in simulation and 30% in the real world over the strongest baseline.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reactivity, Real-time Inference, Flow Policies, Robot Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eSo2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes π\piR$^2$, a flow policy that combines asynchronous vision-language processing with frequent proprioceptive updates. A partially denoised action buffer and a delay-dependent schedule allow actions to be emitted with one denoising step per call. Experiments include simulated cube reorientation and four real-world manipulation tasks using GR00T-N1.7, with approximately 25Hz proprioceptive feedback and reported success improvements of up to 30 percentage points.

Critical strengths
  1. Separating slow visual processing from fast proprioceptive feedback addresses a practical bottleneck in large action-chunking policies and allows corrections during execution.
  2. The action buffer combines incremental denoising with conditioning on already committed actions. This provides a useful way to reduce per-call computation while maintaining continuity.
  3. The four real-world tasks demonstrate the method on contact-rich manipulation. The force traces provide concrete examples of how faster feedback can prevent excessive gripping or missed contact.
Critical weaknesses
  1. The hardware comparison does not isolate the algorithmic gain. πR² uses two RTX A5000 GPUs, while the baselines use one. A real-world comparison with RTC using the same asynchronous visual processing and hardware is missing.
  2. Robustness to changing latency is not directly established. The main study sweeps fixed delays, while the deployment algorithm uses a rolling estimate and repeats actions when predictions arrive late. Performance under latency spikes and the frequency of this fallback are not reported.
  3. The simulation latency accounting needs clarification. Section 4.1.2 derives the delay budget from four denoising steps, whereas Appendix A.1 uses 15 steps for Flow and RTC. Since the assigned delays strongly affect the results, the mapping from NFE counts to simulated latency should be explicit.
Questions for authors
  1. How does π\piR$^2$ perform on a single GPU? Does its advantage remain against RTC with the same asynchronous visual processing and hardware allocation?
  2. Can you report results with delay changing within an episode, including sudden spikes and the frequency of repeated-action fallback?
  3. Section 4.1.2 derives the simulated latency budget from four denoising steps, while Appendix A.1 uses 15 steps for Flow/RTC. How are the actual NFE counts mapped to the delays assigned to each method?
6 comments
AuthorsOpenReview ↗

Response to Reviewer eSo2 (1/2)

We thank the reviewer for recognizing the value of the action buffer and the force-trace analysis.

[W1] + [Q1] Two GPUs. The second GPU cannot shorten the baselines' policy latency, which is a serial chain that includes the VLM latency. Train-time RTC with asynchronous VLM processing, as the reviewer asks, is our "w/o staircase" ablation (Leap Cube in the submission, and Kinetix in this rebuttal), where hardware plays no role, and πR2\pi R^2 beats it at every unit delay on both Leap Cube and Kinetix, by a margin that widens with the delay. On the same two GPUs on the robot, RTC with asynchronous VLM still has a policy latency of 94.4 ms vs. 31.5 ms for πR2\pi R^2, and even on a single GPU πR2\pi R^2's policy latency (69.3 ms) is lower than that of every RTC configuration (R3). See [C2] for details.

[W2] + [Q2] Robustness to varying latency. Both the policy latency and the cache age vary during our real-world rollouts, and our reported results were obtained under this variation (R2). Across the twelve evaluation rollouts shown in R2 (3,999 policy calls), the policy call interval fluctuates with spikes of up to 6.0× its nominal value (2.6% of calls deviate by more than ±10%), and the cache age varies continuously by a median of 0.67 ticks between consecutive calls. The hold-last-action fallback fires on 58 of 5,401 control ticks (1.07%), including two Box rollouts that hold 17 times each and still succeed. This is by design: πR2\pi R^2 measures dd at every call and the schedule follows it, so the buffer is not rebuilt; only its per-position noise targets shift, and it settles within a few calls (App. A.4). See [C3] for details.

[W3] + [Q3] Simulated latency accounting. We apologize for the confusion. In simulation, the policy latency and cache age are set by a fixed cost model from GR00T-N1.7 (VLM latency dvlm=0.75d0d_{\mathrm{vlm}} = 0.75d_0 and denoising-step latency ddn=0.25d0d_{\mathrm{dn}} = 0.25d_0, where the unit delay d0d_0 is the cost of GR00T's default 4 denoising steps), not by how many denoising steps the small simulation network runs. Flow and RTC run 15 denoising steps on Leap Cube and up to 5 on Kinetix for quality, but are charged only GR00T's 4 steps, which favors them. R4 lists the policy latency and cache age charged to each method; we will add it to the main draft. A method is excluded at a given d0d_0 when its charged policy latency exceeds H/2H/2 (H=16H = 16 on Leap Cube, H=8H = 8 on Kinetix), as it then no longer allows continuous real-time execution.

AuthorsOpenReview ↗

Response to Reviewer eSo2 (2/2)

R4. Policy latency dd and cache age dvisd_{\mathrm{vis}} charged to each method in simulation, in control ticks, with dvlm=0.75d0d_{\mathrm{vlm}} = 0.75d_0 and ddn=0.25d0d_{\mathrm{dn}} = 0.25d_0. "–" means the charged dd exceeds H/2H/2, so the schedule cannot fit a clean prefix and a noise tail of length dd inside the chunk. For FASTER, n(d)n(d) is the ready-step of the last position it emits under its own schedule (α=0.6\alpha = 0.6, u0=0.9u_0 = 0.9), with K=15K = 15 steps on Leap Cube and K=5K = 5 on Kinetix; because the schedule depends on the delay, dd is the smallest self-consistent solution. πR2\pi R^2's cache age is charged as dvis=⌈dvlm⌉=⌈0.75d0⌉d_{\mathrm{vis}} = \lceil d_{\mathrm{vlm}} \rceil = \lceil 0.75d_0 \rceil, the lower end of its real-world range.

Leap Cube, H=16H = 16, d0∈{1,2,3,5}d_0 \in \lbrace 1, 2, 3, 5 \rbrace

MethodCharged policy latency dddddvisd_{\mathrm{vis}}
Standard Flow, RTC, Train-time RTCdvlm+4ddn=⌈1.75d0⌉d_{\mathrm{vlm}} + 4d_{\mathrm{dn}} = \lceil 1.75d_0 \rceil2 / 4 / 6 / –– (in dd)
FASTER⌈0.75d0+n(d)Kd0⌉\lceil 0.75d_0 + \frac{n(d)}{K}d_0 \rceil1 / 3 / 4 / –– (in dd)
SDP⌈d0⌉\lceil d_0 \rceil, stalls1 / 2 / 3 / 5– (in dd)
πR2\pi R^21ddn=⌈0.25d0⌉1d_{\mathrm{dn}} = \lceil 0.25d_0 \rceil1 / 1 / 1 / 21 / 2 / 3 / 4
πR2\pi R^2 w/o asyncdvlm+1ddn=⌈1.0d0⌉d_{\mathrm{vlm}} + 1d_{\mathrm{dn}} = \lceil 1.0d_0 \rceil1 / 2 / 3 / 5– (in dd)
πR2\pi R^2 w/o staircase4ddn=⌈1.0d0⌉4d_{\mathrm{dn}} = \lceil 1.0d_0 \rceil1 / 2 / 3 / 51 / 2 / 3 / 4
πR2\pi R^2 w/o latency-adaptive⌈0.25d0⌉\lceil 0.25d_0 \rceil, stalls1 / 1 / 1 / 21 / 2 / 3 / 4

Kinetix, H=8H = 8, d0∈{1,2,3,4}d_0 \in \lbrace 1, 2, 3, 4 \rbrace

MethodCharged policy latency dddddvisd_{\mathrm{vis}}
Standard Flow, RTC, Train-time RTCdvlm+4ddn=⌈1.75d0⌉d_{\mathrm{vlm}} + 4d_{\mathrm{dn}} = \lceil 1.75d_0 \rceil2 / 4 / – / –– (in dd)
FASTER⌈0.75d0+n(d)Kd0⌉\lceil 0.75d_0 + \frac{n(d)}{K}d_0 \rceil1 / 4 / – / –– (in dd)
SDP⌈d0⌉\lceil d_0 \rceil, stalls1 / 2 / 3 / 4– (in dd)
πR2\pi R^21ddn=⌈0.25d0⌉1d_{\mathrm{dn}} = \lceil 0.25d_0 \rceil1 / 1 / 1 / 11 / 2 / 3 / 3
πR2\pi R^2 w/o asyncdvlm+1ddn=⌈1.0d0⌉d_{\mathrm{vlm}} + 1d_{\mathrm{dn}} = \lceil 1.0d_0 \rceil1 / 2 / 3 / 4– (in dd)
πR2\pi R^2 w/o staircase4ddn=⌈1.0d0⌉4d_{\mathrm{dn}} = \lceil 1.0d_0 \rceil1 / 2 / 3 / 41 / 2 / 3 / 3
πR2\pi R^2 w/o latency-adaptive⌈0.25d0⌉\lceil 0.25d_0 \rceil, stalls1 / 1 / 1 / 11 / 2 / 3 / 3

Thanks for clarifying how the simulation delays were set. I had missed that Flow and RTC run 15 denoising steps on Leap Cube but are charged the cost of four GR00T steps. That answers my question. Please include R4 and this explanation in the paper.

The rollout numbers also answer my question about the fallback. The robot repeated the last action on 58 of 5,401 control ticks. I was glad to see the per-rollout counts too, since both Box trials with 17 holds still succeeded.

My remaining question is about task success on the robot. With both methods on two GPUs and using asynchronous vision, π\piR$^2$ takes 31.5 ms per call and RTC takes 94.4 ms. That addresses my concern about the latency comparison. Do you also have success rates for those two setups on any of the robot tasks? I’d like to know how much of a difference the shorter call time made in practice.

Reviewer_HPaCOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes πR², a framework for improving the real-time reactivity of action-chunking flow policies built on large pretrained vision-language-action (VLA) models. The authors identify two limitations of existing policies: high inference latency caused by repeated vision-language backbone processing and multi-step flow denoising, and limited closed-loop reactivity caused by executing action chunks open-loop.

πR² addresses these limitations through two main components. First, it separates conditioning into a fast channel containing proprioceptive information, which is refreshed at every control step, and a slow channel containing vision-language features, which is updated asynchronously. Second, it introduces a latency-adaptive, position-dependent noise schedule that maintains a partially denoised action buffer, conditions on in-flight actions, and generates executable actions using one denoising step per policy call.

The method is evaluated on a simulated Leap Cube Reorientation task and four real-world manipulation tasks using a fine-tuned GR00T-N1.7 policy on an xArm6+XHand platform. The authors report improved success rates over several baselines and approximately 4× faster closed-loop replanning than the original policy.

The paper addresses an important problem in deploying large generative policies for robotic manipulation. However, the problem formulation and scope of the claimed reactivity improvements need clarification, and the experimental evidence is not yet sufficient to establish the generality and robustness of the proposed approach. The relationship between the individual technical contributions and existing real-time action-generation methods also requires stronger empirical validation.

Critical strengths
  1. Important and practically relevant problem.

The paper addresses the latency–reactivity trade-off in large pretrained manipulation policies. Enabling high-frequency control without discarding the expressive action distributions and pretrained representations of modern VLAs is an important research direction with clear practical relevance.

  1. A reasonable decomposition of computation.

Separating frequently available proprioceptive information from expensive vision-language features is a sensible architectural choice. Asynchronous feature computation can reduce the latency of the action-generation loop without requiring the large backbone to execute at every control tick.

  1. A coherent integration of latency handling and action generation.

The proposed position-dependent noise schedule explicitly accounts for in-flight actions and varying inference delays. Combining this schedule with asynchronous feature updates provides a plausible mechanism for maintaining continuous execution while generating new actions incrementally.

  1. Evaluation on physical robotic hardware.

The experiments include four real-world manipulation tasks involving contact-rich interactions, object rearrangement, and reactive grasping. The comparison against synchronous inference, naive asynchronous inference with temporal ensembling, and Train-Time RTC provides useful evidence that the proposed design can improve practical execution.

  1. Component ablations and implementation details.

The simulation study includes ablations of asynchronous feature processing, the staircase noise schedule, and latency adaptation. The appendix also provides training hyperparameters, deployment procedures, and pseudocode, which are helpful for assessing the method's implementation and reproducibility.

Critical weaknesses
  1. The problem formulation and definition of reactivity are insufficiently precise.

The paper motivates πR² through the need for real-time, closed-loop manipulation, but several distinct properties are treated somewhat interchangeably: inference latency, control frequency, replanning frequency, feedback freshness, and reactivity to environmental changes. These properties are related but not equivalent.

In particular, the method refreshes proprioception at every policy call while updating visual and language features asynchronously. Therefore, its ability to react to new contact forces or joint states differs fundamentally from its ability to respond to newly observed object motion, changing scene geometry, or task-level visual events. The paper's broad claims about reactivity would benefit from a more explicit formulation of these distinctions.

The paper also does not clearly characterize the conditions under which fresh proprioception is sufficient to compensate for stale visual information. This assumption may hold for local contact corrections, but it is less obviously valid when successful control depends on detecting object displacement, tracking a moving target, or recovering from a substantial deviation from the original plan.

Consequently, the central problem remains underspecified: is the primary objective to reduce action-generation latency, to improve feedback frequency, to maintain continuity under asynchronous execution, or to improve closed-loop task performance under delayed multimodal observations? A precise definition and corresponding evaluation protocol would make the contributions and conclusions substantially clearer.

  1. The experimental evaluation is too limited to support the broad claims.

The simulation evaluation uses only one task, Leap Cube Reorientation, and the real-world evaluation contains four tasks on a single robot platform with a single pretrained VLA backbone, GR00T-N1.7. Although these experiments provide useful initial evidence, they do not sufficiently establish that the proposed approach generalizes across manipulation settings, policy architectures, and latency regimes.

The simulation experiments primarily evaluate state-based policies, with a low-dimensional state subset serving as a proxy for the slow visual channel. This setup does not fully reproduce the challenges of a large VLA with high-dimensional visual observations and language conditioning. The real-world experiments partially address this gap, but the limited task diversity leaves several questions unresolved.

Furthermore, each real-world method–task combination is evaluated over only 20 trials. Such a sample size can make success-rate estimates sensitive to initial conditions and stochastic execution, particularly when the observed differences are relatively small. The paper does not report confidence intervals or statistical significance tests for these real-world results.

More extensive experiments are needed to establish whether the observed improvements are consistent and whether the proposed approach remains effective beyond the particular tasks and hardware configuration considered.

  1. The empirical evidence does not sufficiently isolate the contribution of each proposed component.

The method combines asynchronous vision-language feature processing, proprioception-reactive denoising, and a latency-adaptive noise schedule. The simulation ablations are useful, but the evidence does not fully establish the individual contribution of these components to the reported real-world improvements.

For example, asynchronous feature processing reduces the time spent waiting for the vision-language backbone, while the latency-adaptive schedule reduces the number of denoising steps required to emit executable actions. Both changes can improve effective control frequency. However, the experiments do not fully disentangle improvements caused by increased feedback frequency from those caused by smoother action transitions, in-flight action conditioning, or changes in the effective execution horizon.

The real-world evaluation would be stronger with controlled variants that separately enable asynchronous feature updates, fresh proprioceptive conditioning, and latency adaptation under comparable control and inference budgets. In particular, a latency-matched comparison would help determine whether the proposed method offers benefits beyond simply executing more frequent policy updates.

Questions for authors

Problem definition: How do the authors formally distinguish real-time inference, closed-loop reactivity, control frequency, and observation freshness? Which of these is the primary objective optimized by πR², and how should improvements in each be measured independently?

Proprioceptive versus visual reactivity: Since the fast channel refreshes proprioception but the vision-language features may be stale, under what assumptions can the method react reliably to environmental changes? Can the authors provide quantitative results that isolate reactivity to new proprioceptive feedback from reactivity to new visual observations?

Visual-delay robustness: How does performance change as the visual-feature delay increases beyond the training range of five control ticks? What happens when visual latency fluctuates rapidly or when the cached visual representation becomes inconsistent with the current scene?

Latency-matched baselines: Can the authors provide comparisons against relevant streaming or latency-aware policies under matched wall-clock inference budgets and control frequencies? In particular, can they establish whether πR² retains its advantage when the baselines are tuned for comparable deployment conditions?

3 comments
AuthorsOpenReview ↗

Response to Reviewer HPaC

We thank the reviewer for the careful review and for recognizing the practical relevance of the problem.

[W1] + [Q1] Defining reactivity. We agree the terms were used loosely. We now define them formally in the Common Response (Terminology) and will add these definitions to Sec. 3: VLM latency dvlmd_{\mathrm{vlm}}, cache age dvisd_{\mathrm{vis}}, policy latency dd, proprioception staleness spros_{\mathrm{pro}}, VLM feature staleness sviss_{\mathrm{vis}}, and execution horizon hh. Our objective is task success under delayed observations. πR2\pi R^2 achieves it by reducing proprioception staleness without increasing VLM feature staleness (see Goal in the Common Response; measured on the robot in R3).

[W1b] + [Q2] Proprioceptive vs. visual reactivity. πR2\pi R^2 does not trade visual reactivity for proprioceptive reactivity. A baseline's policy latency is d=dvlm+Kddnd = d_{\mathrm{vlm}} + K d_{\mathrm{dn}} (K=4K = 4), and its execution horizon is at least dd, so spro=svis=1.5ds_{\mathrm{pro}} = s_{\mathrm{vis}} = 1.5d on average. For πR2\pi R^2, the policy latency is ddnd_{\mathrm{dn}} and the cache age dvisd_{\mathrm{vis}} is 1.5dvlm1.5d_{\mathrm{vlm}} on average, so svis=1.5(dvlm+ddn)s_{\mathrm{vis}} = 1.5(d_{\mathrm{vlm}} + d_{\mathrm{dn}}): smaller than the baseline. This is why πR2\pi R^2 works on Catch Book, which needs both: the hand approaches the book as the ball falls (visual reactivity), then stabilizes the grasp from contact forces (proprioceptive reactivity). πR2\pi R^2 reaches 0.55 success vs. 0.25 for the strongest baseline.

[Q3] Visual delay robustness. Both the policy latency dd and the cache age dvisd_{\mathrm{vis}} vary during our real-world rollouts, and our reported results were obtained under this variation (R2). The policy receives the measured dvisd_{\mathrm{vis}} through a delay embedding at every call. Across the twelve evaluation rollouts shown in R2 (3,999 policy calls), dvisd_{\mathrm{vis}} moves by a median of 0.67 ticks between consecutive calls and by up to 6.0 ticks in a single one, so the variation is rapid rather than slow drift; 35 calls carry it past the 5-tick training range, peaking at 13.9 ticks (2.8× the range), with one Catch Book rollout spending 20 calls there. In those excursions the cached features are measurably inconsistent with the scene (mean absolute pixel difference against the live frame rises by up to 1.6×), and all twelve rollouts succeed. On two GPUs, the measured dvisd_{\mathrm{vis}} has a median of 2.36 ticks, well inside the training range; even the VLM feature staleness of executed actions, which adds the policy latency and the position within the chunk, never exceeds 5 ticks (max 5.0) and is lower than that of every RTC configuration (R3). See [C3] for details.

[W2] Lack of experiments. We add 6 Kinetix tasks, the ones whose state includes objects beyond proprioception, and evaluate all baselines and ablations. At small delays (d0=1,2d_0 = 1, 2) πR2\pi R^2 is comparable to the baselines; beyond that, their policy latency no longer fits the chunk, while πR2\pi R^2 keeps 0.769 at d0=3d_0 = 3–4; removing its staircase drops this to 0.642 at d0=4d_0 = 4, and removing the asynchronous slow channel to 0.646 (R1). See [C1] for details. Rollout videos for all real-world evaluations on our website also show the benefit of πR2\pi R^2 intuitively.

[W3] Real-world ablations. Each real-world cell requires 20 physical trials per task, so we run the component ablations in simulation, where latency can be controlled exactly (Leap Cube, and now Kinetix). On the robot, the VLM and policy latency measurements (R3) show how much fresher πR2\pi R^2's observations are, and the force traces (Fig. 6, 9) show the effect directly: πR2\pi R^2 reacts to contact as it happens, while baselines follow a stale plan.

[Q4] Latency-matched baselines. FASTER and Streaming Diffusion Policy are the two streaming instantiations closest to πR2\pi R^2 (FASTER additionally accounts for policy latency, while SDP assumes zero policy latency), and we compare against both on Leap Cube and Kinetix, each charged its own policy latency under the same cost model (R4). We also compare against Train-time RTC with the same asynchronous VLM processing (see [C1], [C2]).

AuthorsOpenReview ↗

Response to Reviewer HPaC (continued)

A real-world ablation ([W3]). We have now evaluated RTC with an asynchronous VLM cache on the robot. On the Insert Box task it achieves 11/20 success and 52/80 progress on two GPUs, against 10/20 and 53/80 for synchronous Train-time RTC and 16/20 and 68/80 for πR2\pi R^2 (Table 4, main draft), so moving the VLM off the action path is not sufficient on hardware either. See [C4] for the staleness measurements behind this and the corresponding simulation comparison.

Dear Reviewer HPaC,

Thank you again for your time and valuable feedback. As the author–reviewer discussion period ends today, we wanted to kindly follow up on our responses. In summary, we gave formal definitions of the latency and staleness terms ([C1], Terminology), added a second simulation benchmark with all baselines and ablations ([C1], R1), reported how both delays vary during the real-world rollouts ([C3], R2), and measured real-world latency and staleness ([C2], R3). Addressing [W3], we have also now run one of our component ablations on the robot rather than in simulation; the result is in [C4]. We hope these address your concerns, and if so, we would be grateful if you would consider increasing your score. If any points remain unclear, we would be happy to clarify them before the deadline.

Best regards,

The Authors

Reviewer_G5jFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to address a common problem in VLA models for use in robotics, where large flow-matching VLAs (e.g., π0, GR00T) predict action chunks at a time and execute them open-loop, ignoring the feedback from the faster execution loop. This has caused the robot to act on stale observations and cannot react mid-chunk. To address this issue, the paper proposes to run the vision-language backbone asynchronously on its own GPU, and feed fresh proprioception (joint state, torque, fingertip force) into the action head (running on the second GPU) on every call. A diffusion-forcing "staircase" noise schedule then releases dd-clean actions with one denoising step per call, and the adapts dd to the measured delay. In result, GR00T-N1.7 on an xArm6+XHand replans at 25 Hz and beats the best baseline by 5–30 points on 4 dexterous tasks (up to 25 in simulation).

Critical strengths
  • The paper presents a practical recipe, where it changes only the action head of a large real model (pairing the head with an asynchronous VLM cache that carries a staleness embedding inside a production VLA), and replanning goes from about 7 to 25 times per second.
  • Empirical evaluation also shows real gains on real hardware with a hand, on hard contact tasks.
Critical weaknesses
  • The novelty is overstated to me. It is a technical integration rather than a conceptual-level advances. Similar ideas (splitting slow vision from fast movement) have been explored in some earlier works, and the paper does not cite or compare to them. To name a few
    • Helix: S2 at 7–9 Hz as an asynchronous background process, S1 at 200 Hz taking the latest observation plus the latest S2 latent, on separate GPUs.
    • HiRT, CoRL 2024
    • Reactive Diffusion Policy: slow latent diffusion plus a fast tactile/force loop, RSS 2025
  • The real-world evidence does not separate "reacts to proprioception" from "lower latency / extra hardware". The proposed method uses two GPUs and has less delay, but the other methods use one GPU. So the gain may come mostly from lower delay, not from proprioception. There is no real-world ablation such as RTC with an asynchronous VLM, or pir2 with frozen proprioception.
  • Simulation could also be extended to other tasks for more convincing results, in addtion to the only one task used. E.g., there are no results on shared benchmarks such as Kinetix, a set of 12 dynamic tasks.
Questions for authors
  • On the robot, can you report training-time RTC and FASTER with the same async VLM cache on two GPUs, and pir2 with proprioception frozen within each sub-chunk? This would separate reactivity from latency.
  • For comparability with published numbers (RTC, training-time RTC, FASTER, etc), can you report pir2 on the 12-task Kinetix benchmark?
  • Why does pir2 drop from 0.93 (d = 0, Fig 3a) to 0.72 at d0 = 1: the 100- vs 800-epoch budget (Tab 1) or staleness?
3 comments
AuthorsOpenReview ↗

Response to Reviewer G5jF

We thank the reviewer for recognizing the practical recipe and the real-world gains.

[W1] Novelty vs. existing slow/fast systems. We thank the reviewer for these references; we will cite all three. Meanwhile, we would like to clarify that none of them fully address our setting: a pretrained flow-matching VLA whose action head needs multiple denoising steps, with actions already in flight under variable policy latency.

  • Helix is a blog post without technical details or experiments. Its fast system is regression-based, not iterative denoising. Its proprioception consists only of joint angles and finger positions, which is not enough for contact detection; πR2\pi R^2 additionally uses joint torques and fingertip forces.
  • HiRT likewise has no iterative denoising. Its fast component is a small action head that takes vision as input, which requires architectural modification. πR2\pi R^2 is plug-and-play on an existing VLA.
  • Reactive Diffusion Policy also requires architectural modification (an asymmetric tokenizer) for closed-loop tactile feedback, so it is not compatible with existing VLAs. It also does not account for the policy latency between chunk predictions, so it lacks real-time continuous rollout.

In our setting the slow/fast split alone is not enough, because the action head still needs several denoising steps per call. Our latency-adaptive staircase schedule makes each call a single denoising step on fresh proprioception while conditioning on the in-flight actions. Without it (RTC + async VLM), success drops as the delay grows (Leap Cube: 0.42 vs. 0.67 at d0=3d_0 = 3; Kinetix: 0.642 vs. 0.769 at d0=4d_0 = 4).

[W2-a] + [Q1] More hardware (two GPUs). The second GPU cannot shorten the baselines' policy latency, which is a serial chain that includes the VLM latency. RTC with asynchronous VLM processing, as the reviewer asks, is our "w/o staircase" ablation (Leap Cube in the submission, and Kinetix in this rebuttal), where hardware plays no role, and πR2\pi R^2 beats it at every unit delay on both Leap Cube and Kinetix, by a margin that widens with the delay. On the same two GPUs on the robot, RTC with asynchronous VLM still has a policy latency of 94.4 ms vs. 31.5 ms for πR2\pi R^2, and even on a single GPU πR2\pi R^2's policy latency (69.3 ms) is lower than that of every RTC configuration (R3). See [C2] for details.

[W2-b] + [Q1] Lower latency vs. reacting to proprioception. Policy latency bounds proprioceptive reactivity: a policy cannot act on new proprioception more often than its policy latency allows. Beyond latency, three results show that reacting to proprioception matters: (i) in simulation, raising VLM feature staleness at fixed proprioception staleness costs little, while raising proprioception staleness costs much more: πR2\pi R^2 keeps d=1d = 1 as d0d_0 grows, so only its VLM feature cache age rises (1 → 3 ticks), and its success stays nearly flat (Leap Cube 0.72 → 0.67 from d0=1d_0 = 1 to 3). In contrast, without the async slow channel the policy latency itself grows with d0d_0, and success drops much more (0.70 → 0.50), even though both variants perform the same at d0=1d_0 = 1, where both have d=1d = 1; (ii) longer execution horizons hh hurt even with zero policy latency (Fig. 3a); and (iii) on the robot, πR2\pi R^2 modulates its finger actions when the hand makes contact with the object, leading to higher success (Fig. 6, 9).

[W3] + [Q2] More simulation results. We add 6 Kinetix tasks, the ones whose state includes objects beyond proprioception (the other 6 contain only proprioception, so the slow/fast split does not apply), and evaluate all baselines and ablations. At small delays (d0=1,2d_0 = 1, 2) πR2\pi R^2 is comparable to the baselines; beyond that, their policy latency no longer fits the chunk, while πR2\pi R^2 keeps 0.769 at d0=3d_0 = 3–4; removing its staircase drops this to 0.642 at d0=4d_0 = 4, and removing the asynchronous slow channel to 0.646 (R1). See [C1] for details.

[Q3] Performance drop from zero latency to d0=1d_0 = 1. The drop comes from staleness: at d0=1d_0 = 1, πR2\pi R^2 runs with policy latency d=1d = 1 and cache age dvis=1d_{\mathrm{vis}} = 1, whereas Fig. 3a has neither. It is not due to the training budget: with zero policy latency, performance plateaus earlier for all methods.

AuthorsOpenReview ↗

Response to Reviewer G5jF (continued)

Real-robot result for RTC with asynchronous VLM processing ([W2-a], [Q1]). We have now evaluated RTC with an asynchronous VLM cache on the robot. On the Insert Box task it achieves 11/20 success and 52/80 progress on two GPUs, against 10/20 and 53/80 for synchronous Train-time RTC and 16/20 and 68/80 for πR2\pi R^2 (Table 4, main draft), so moving the VLM off the action path is not sufficient on hardware either. See [C4] for the staleness measurements behind this and the corresponding simulation comparison.

Dear Reviewer G5jF,

Thank you again for your time and valuable feedback. As the author–reviewer discussion period ends today, we wanted to kindly follow up on our responses. In summary, we added a second simulation benchmark with all baselines and ablations ([C1], R1), real-world latency and staleness measurements on one and two GPUs ([C2], R3), and the delay charged to each method in simulation ([C3], R4). We have also now evaluated RTC with an asynchronous VLM cache on the robot, which is the comparison you asked for in [W2-a] and [Q1]; the result is in [C4]. We hope these address your concerns, and if so, we would be grateful if you would consider increasing your score. If any points remain unclear, we would be happy to clarify them before the deadline.

Best regards,

The Authors

Comments on the paper

8 comments
AuthorsOpenReview ↗

Common Response (1/5)

We thank all reviewers for their constructive feedback. Below we address the shared concerns; all new results (R1–R4) are in the Rebuttal section of our anonymous project website (https://pir2-iclr2027.github.io/). Reviewers found the problem important (HPaC), the recipe practical (G5jF), and the action buffer an effective way to cut per-call computation (eSo2). πR² makes large pretrained VLAs reactive with a single change to the action head, removes both the VLM latency and most of the denoising latency from the action loop, and replans about 4× faster than the base policy.

Terminology and problem formulation (HPaC). To avoid confusion, we use the following terms throughout and will add them to Sec. 3. Latencies are in control ticks (40 ms on the robot) unless given in ms, measured on RTX A5000/A6000 GPUs.

  • VLM latency dvlmd_{\mathrm{vlm}}: the time of one vision-language backbone forward pass (image preprocessing + VLM), about 60 ms for GR00T-N1.7.
  • Cache age dvisd_{\mathrm{vis}}: how old the cached VLM features are when the policy is called (πR² only). The VLM runs asynchronously and refreshes the cache every dvlmd_{\mathrm{vlm}}, so dvisd_{\mathrm{vis}} lies between dvlmd_{\mathrm{vlm}} and 2dvlm2d_{\mathrm{vlm}} (1.5dvlm1.5d_{\mathrm{vlm}} on average); it is given to the policy through a delay embedding. The baselines have no cache: their VLM latency is included in dd.
  • Denoising-step latency ddnd_{\mathrm{dn}}: the time of one denoising step of the action head, about 20 ms.
  • Policy latency dd: the time of one policy call, from reading proprioception to emitting the actions conditioned on it, i.e., the delay of the proprioception the policy conditions on; it also equals the number of actions in flight. For πR², d=ddnd = d_{\mathrm{dn}} (one denoising step, with the VLM running asynchronously). For the baselines, which have no fast/slow split, d=dvlm+Kddnd = d_{\mathrm{vlm}} + K d_{\mathrm{dn}} with K=4K = 4 denoising steps.
  • Proprioception staleness spros_{\mathrm{pro}}: how old the proprioception behind an executed action is, which is what proprioceptive reactivity depends on. It is the policy latency plus the action's offset within the executed chunk, spro=d+h/2s_{\mathrm{pro}} = d + h/2 on average; with h=dh = d this is 1.5d1.5d, ≈ 210 ms for the baselines and 1.5ddn1.5d_{\mathrm{dn}} (about one control tick) for πR².
  • VLM feature staleness sviss_{\mathrm{vis}}: how old the visual features behind an executed action are: svis=dvis+d+h/2s_{\mathrm{vis}} = d_{\mathrm{vis}} + d + h/2 on average. The baselines have no cache: the VLM runs inside every call, so dvlmd_{\mathrm{vlm}} is already included in dd and the visual features are exactly as old as the proprioception, i.e., equal to the proprioception staleness (≈ 210 ms). For πR², dvisd_{\mathrm{vis}} averages 1.5dvlm1.5d_{\mathrm{vlm}}, giving 1.5dvlm+1.5ddn1.5d_{\mathrm{vlm}} + 1.5d_{\mathrm{dn}} (≈ 120 ms). Unlike dd and dvisd_{\mathrm{vis}}, which are delays at conditioning time, spros_{\mathrm{pro}} and sviss_{\mathrm{vis}} are averages over executed actions and also count each action's offset within its chunk.
  • Execution horizon hh: the number of actions executed before the next plan that uses new observations. We set h=dh = d for all methods, i.e., each policy replans as soon as its next chunk is ready, which maximizes its replanning frequency.
  • Unit delay d0d_0 (simulation only): the cost of GR00T-N1.7's default 4 denoising steps, so that dvlm=0.75d0d_{\mathrm{vlm}} = 0.75d_0 and ddn=0.25d0d_{\mathrm{dn}} = 0.25d_0.

Goal. πR2\pi R^2 enables the policy to act on up-to-date proprioception, i.e., it reduces proprioception staleness spros_{\mathrm{pro}} without sacrificing VLM feature staleness sviss_{\mathrm{vis}}. With the costs above, proprioception staleness drops from ≈ 210 ms to about one control tick, and VLM feature staleness also decreases, from ≈ 210 ms to ≈ 120 ms. On the robot we measure 48 ms and 141 ms for πR², vs. 203 ms for both channels of the synchronous baseline (R3).

AuthorsOpenReview ↗

Common Response (2/5)

[C1] More experiments (G5jF, HPaC)

We add Kinetix (R1), a benchmark of dynamic tasks used in prior real-time chunking work (RTC, Train-time RTC) that requires reacting quickly to the environment. This complements Leap Cube with a broader set of tasks. Of the 12 Kinetix levels, 6 include object states beyond proprioception (grasp_easy, catapult, chain_lander, catcher_v3, trampoline, car_launch); we treat these as the slow channel, analogous to the vision-derived cube pose in Leap Cube. The other 6 contain only proprioception, so the slow/fast split does not apply to them. With chunk length H=8H = 8, we evaluate all methods at d0∈{1,2,3,4}d_0 \in \lbrace 1, 2, 3, 4 \rbrace over 3 training seeds and 2,048 rollouts per level, charging each method its own policy latency and cache age under the same cost model as Leap Cube (R4).

Result. With H=8H = 8, the baselines' charged policy latency exceeds H/2H/2 beyond d0=2d_0 = 2, so only πR2\pi R^2, its ablations and SDP (which stalls) can run at d0=3d_0 = 3 and 4. At these larger delays πR2\pi R^2 keeps 0.769, while SDP stays below 0.1. At small delays (d0=1,2d_0 = 1, 2), where multiple denoising steps still fit, πR2\pi R^2 is comparable to the strongest baseline (0.819 vs. 0.852 at d0=1d_0 = 1) and slightly ahead of it at d0=2d_0 = 2 (0.804 vs. 0.794). Both components matter more as the delay grows: without the staircase, success falls to 0.707 at d0=3d_0 = 3 and 0.642 at d0=4d_0 = 4; without the asynchronous slow channel, to 0.744 and 0.646; and removing latency adaptation collapses performance (≈ 0.20 throughout).

R1. Kinetix, the 6 levels whose state includes objects beyond proprioception, H=8H = 8, success rate, mean ± std over 3 training seeds (2,048 rollouts per level). Each method is charged the delay its own deployment incurs (R4); "–" marks settings excluded because the charged policy latency exceeds H/2=4H/2 = 4.

Methodd0=1d_0=1d0=2d_0=2d0=3d_0=3d0=4d_0=4
Standard Flow (async)0.655 ± 0.0000.569 ± 0.001––
RTC0.838 ± 0.0000.690 ± 0.003––
Train-time RTC0.852 ± 0.0170.794 ± 0.006––
FASTER0.826 ± 0.0730.763 ± 0.041––
SDP0.180 ± 0.0100.153 ± 0.0240.088 ± 0.0030.091 ± 0.001
piR2 (ours)0.819 ± 0.0040.804 ± 0.0060.769 ± 0.0080.769 ± 0.008
w/o async slow channel0.822 ± 0.0040.807 ± 0.0100.744 ± 0.0290.646 ± 0.017
w/o staircase0.812 ± 0.0170.776 ± 0.0210.707 ± 0.0270.642 ± 0.059
w/o latency-adaptive0.210 ± 0.0310.190 ± 0.0350.197 ± 0.0300.197 ± 0.030

AuthorsOpenReview ↗

Common Response (3/5)

[C2] Two GPUs (G5jF, eSo2)

  • The second GPU does not benefit the baselines. Without a fast/slow split, their policy latency is a serial chain (obs → backbone → denoise → act), d=dvlm+Kddnd = d_{\mathrm{vlm}} + K d_{\mathrm{dn}}, that extra devices cannot shorten. Extra devices help only when the pipeline is split in parallel, as in our fast/slow split, which removes dvlmd_{\mathrm{vlm}} from the policy latency. Decentralized compute (server & edge) is increasingly standard, and single-GPU operation should be possible via GPU-level scheduling of both channels, which we leave for future work.

  • Baselines with asynchronous slow/fast processing. RTC with asynchronous VLM processing (our "w/o staircase" ablation) is already evaluated in simulation on Leap Cube, and we now add Kinetix results (R1). Because RTC still needs multiple denoising steps per call, its policy latency stays larger than ours, and πR2\pi R^2 outperforms it at every unit delay on both Leap Cube and Kinetix, by a margin that widens with the delay (Kinetix: 0.769 vs. 0.707 at d0=3d_0 = 3 and 0.769 vs. 0.642 at d0=4d_0 = 4). On the real robot, RTC with the same asynchronous VLM on the same two GPUs still has a policy latency of 94.4 ms, 3.0× πR2\pi R^2's 31.5 ms (R3).

  • πR2\pi R^2 with a single GPU. We measure real-world policy latency and VLM latency for πR2\pi R^2 and Train-time RTC on 1 and 2 GPUs, with and without asynchronous VLM processing, and without any dedicated GPU scheduling (R3). Overall, πR2\pi R^2 on a single GPU achieves lower proprioception and VLM feature staleness than RTC with asynchronous VLM processing on 2 GPUs. On one GPU, πR2\pi R^2's policy latency rises from 31.5 to 69.3 ms; this is still lower than every RTC configuration, including RTC with asynchronous VLM on two GPUs (94.4 ms).

R3. Real-world policy latency, VLM latency and staleness of policy calls (1 tick = 40 ms). VLM latency is measured as the interval between VLM cache refreshes; the synchronous baseline has no separate VLM refresh, since its VLM runs inside every policy call. Cache age dvisd_{\mathrm{vis}} is the median delay of the visual features at conditioning time. Staleness columns are medians over executed control ticks and also count each action's offset within its chunk: spro=d+h/2s_{\mathrm{pro}} = d + h/2 and svis=dvis+d+h/2s_{\mathrm{vis}} = d_{\mathrm{vis}} + d + h/2 on average. For example, the synchronous RTC baseline has no cache (its VLM latency is included in dd), yet its VLM feature staleness is 5.08 ticks. The last column is the share of executed control ticks whose VLM feature staleness exceeds 5 ticks.

MethodGPUsVLMPolicy latency ddVLM latency dvlmd_{\mathrm{vlm}}Cache age dvisd_{\mathrm{vis}}Proprioception staleness spros_{\mathrm{pro}}VLM feature staleness sviss_{\mathrm{vis}}sviss_{\mathrm{vis}} > 5 ticks
πR2\pi R^22async31.5 ms (0.79 ticks)60.8 ms (1.52 ticks)94 ms (2.36 ticks)48 ms (1.21 ticks)141 ms (3.52 ticks), max 5.0 ticks0.0%
πR2\pi R^21async69.3 ms (1.73 ticks)69.4 ms (1.73 ticks)88 ms (2.21 ticks)104 ms (2.61 ticks)194 ms (4.85 ticks), max 6.3 ticks40.5%
Train-time RTC (continuous)1sync133.3 ms (3.33 ticks)– (in dd)– (in dd)203 ms (5.08 ticks)203 ms (5.08 ticks), max 8.2 ticks50.8%
Train-time RTC (continuous)2async94.4 ms (2.36 ticks)59.8 ms (1.49 ticks)96 ms (2.40 ticks)142 ms (3.55 ticks)235 ms (5.87 ticks), max 8.3 ticks83.8%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.