acceptodds
Under review as a conference paper at ICLR 2027

Echo: Coupled Action and Task Status Signal Denoising for Vision Language Action Models

31 views

Abstract

Recognizing task completion and failure enables robots to stop upon success and initiate recovery or replanning after failure. Existing methods augment Vision Language Action (VLA) policies with task status signal prediction to support these execution decisions. However, learning and integrating these signals can require fine-grained supervision, external monitoring models, or complex architectural designs. Moreover, the potential of task status signals to directly guide action generation remains underexplored. We present Echo, a framework that combines task status signal learning without frame-level progress annotations with a compact design for status prediction and action generation within a single VLA. Specifically, Echo represents task status with stop and failure signals, added as two dimensions of each action chunk and denoised together with robot actions. Their combination distinguishes ongoing execution, successful completion, and failure. To learn these signals without fine-grained progress annotations, Echo constructs successful, failed, and recovery trajectories with complementary supervision properties. To enable status signal-guided action generation, Echo controls interactions between the signals and actions and denoises the signals faster than the actions. These allow later action denoising steps to condition on cleaner status estimates while preserving accurate signal prediction. Extensive real-world manipulation experiments show that Echo accurately predicts task status and consistently improves policy performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot manipulation, Task status signal, Vision Language Action Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2QKtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Echo, a framework for training VLAs to understand the current status of task progress by jointly predicting task progress signals and actions. It represents task status using a stop signal (to detect whether execution should terminate) and a failure signal (to detect whether execution is in a failure state). To learn these signals, Echo trains VLAs with success, failure, pre-intervention, and recovery demonstrations. Echo also uses this signal during action generation by asynchronously denoising action and status, where it directly predict status signals, while denoising actions conditioned on the predicted signals.

Critical strengths
  1. This paper is well-motivated and addresses a practically important problem.
  2. The proposed method is simple yet effective.
  3. Real-world experiments and component-wise analysis demonstrate the effectiveness of the proposed method clearly.
Critical weaknesses
  1. The proposed method needs manually annotated subtask boundaries, intervention points, and categorized success/failure/recovery trajectories. It should include discussions of the practical cost of obtaining these annotations and whether the annotations can be obtained automatically.
  2. Echo compresses execution status into only two scalar signals, stop and failure. While this is sufficient for the evaluated tasks, more complex long-horizon manipulation may involve qualitatively different failure modes that require different recovery behaviors. It is unclear whether a single failure signal provides enough information to distinguish such cases and guide appropriate corrective actions.
Questions for authors
  1. It would be practical if the paper included analysis of the sensitivity to annotation quality, as manual annotation is required, yet the proposed method avoids dense frame-level progress annotations.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2QKt

Thank you for your constructive comments and questions. We address each concern below.

W1&Q1W1 \& Q1

Annotation Cost, Automation, and Boundary Noise

Our data collection pipeline uses successful demonstrations, autonomous failure trajectories, and intervention-based recovery demonstrations. Subtask boundaries in successful multi-subtask demonstrations are manually annotated, requiring approximately 1–2 seconds per boundary, while single-subtask demonstrations require no manual segmentation. For recovery demonstrations, intervention timestamps are recorded automatically during teleoperation. Terminal failure labels are assigned at the episode level, without frame-level failure annotations.

Automatic subtask segmentation methods, such as NVIDIA FLORA, could potentially be incorporated to further reduce annotation effort. However, developing an automatic segmentation method is beyond the scope of this work.

We further evaluate robustness to boundary annotation noise on LIBERO-Recovery by uniformly perturbing successful subtask boundaries within ±0.2 s. The models are retrained and evaluated under the same protocol. As shown in Table 8, the perturbations have a limited effect on status prediction and task success across both backbones, indicating that Echo is reasonably robust to moderate boundary inaccuracies. The experimental details are included in the revised manuscript's Appendix, “LIBERO-Recovery Simulation and Additional Ablations.”

Table 8. Sensitivity to ±0.2 s boundary perturbations %.

Metricπ₀.₅ Originalπ₀.₅ PerturbedGR00T OriginalGR00T Perturbed
Overall signal accuracy87.8189.5590.2489.88
Ongoing-frame recall88.4190.4793.3292.71
Success-frame recall79.2376.0740.3443.46
Balanced signal accuracy83.8283.2766.8368.09
Task success rate42.4341.8739.4139.08

[W2] Expressiveness of Two Status Signals

The two task status signals are designed to represent whether execution is ongoing, successfully completed, or failed, rather than to encode the specific type of failure or the corrective action required.

Echo generates actions conditioned jointly on visual observations, language instructions, robot state, and predicted task status signals. Consequently, similar status signals can lead to different recovery actions depending on the current observations and task context. The ability to execute these corrective behaviors is learned from recovery demonstrations, while the predicted signals provide additional conditioning information during action generation. Our experiments in Section 4.4.4, “Effect of Failure Data Quantity,” further demonstrate the benefits of incorporating failure and recovery data into policy training.

Thank you again for your valuable feedback. We would be happy to discuss any further questions or concerns.

Thank you again for your thoughtful review. We have provided detailed responses to your concerns, including additional experiments and analyses. As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses. We would also be happy to address any remaining questions or concerns.

Reviewer_27wkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Echo, a VLA framework that jointly predicts robot actions with two task-status signals: a stop signal and a failure signal. These signals distinguish ongoing execution, successful completion, recoverable failure, and terminal failure, and are appended to each action chunk. Echo learns them from successful demonstrations, failed rollouts, pre-intervention failure segments, and recovery demonstrations without explicit frame-level progress labels. A controlled attention pattern lets actions attend to status signals while keeping stop and failure signals separate, and asynchronous denoising makes status estimates available earlier to guide later action updates. Experiments use π0.5 on five real-world manipulation tasks, with status-prediction metrics, task-completion scores, ID/OOD evaluation, component ablations, signal interventions, and failure-data scaling. Echo improves task completion over an action-only baseline and obtains strong frame- and trajectory-level status prediction.

Critical strengths

1.The paper proposes a compact and coherent way to make task-status estimates directly influence action generation, rather than using them only as an external monitor. The controlled attention and asynchronous denoising ablations provide evidence that this coupling contributes to execution performance.

2.The validation is unusually comprehensive for a real-robot VLA paper: five manipulation tasks, out-of-distribution object placements, comparisons with several monitoring methods, component ablations, signal interventions, and analysis of different failure-data quantities. The consistent gains over the matched action-only π0.5 baseline support practical relevance.

Critical weaknesses

1.The supervision and evaluation do not fully establish that Echo learns genuine frame-level task status. Many labels are derived from padding boundaries, manually defined segment boundaries, intervention points, or recovery-segment endpoints; failure onset is often not directly annotated. Moreover, trajectory F1 counts any correct prediction within a 15-frame/3-second window, and Appendix A.8 shows large changes with the threshold and window. Thus the high reported F1 may partly reflect permissive boundary-based evaluation rather than timely failure or completion detection.

2.The execution gains are not fully isolated from the additional data and supervision available to Echo. Echo uses failure and pre-intervention segments for status learning, while the action-only baseline does not process these observations. The ablation shows that coupling matters, but it does not separate the effect of status-guided action conditioning from extra failure-data exposure or multitask regularization. Comparisons to external methods are also limited mainly to Insert Flower and use different supervision and evaluation protocols.

Questions for authors
  1. Which parts of the data construction require manual annotation: subtask boundaries, intervention points, recovery start/end points, and failure labels? How much additional annotation effort is required compared with collecting ordinary demonstrations?

  2. Were the thresholds δs = δf = 0.5 and the 15-frame trajectory window selected using a separate validation set before evaluation? Can the authors clarify whether the reported status and task results remain strong under a fixed protocol chosen independently of each task?

  3. Was there an action-only control trained with the same failure and pre-intervention observations, but with those actions excluded from imitation and without status-conditioned action generation? If so, how does it compare; if not, how do the authors rule out extra data or multitask regularization as the source of the execution gains?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 27wk

Thank you for the detailed feedback. We address your concerns below.

W1&Q1W1 \& Q1

Supervision quality and evaluation metrics

The collection and annotation protocol is described in our response to all reviewers. In addition to the original paper’s class-wise recognition analysis AppendixA7˙Appendix A\.7, we now report false stops, missed outcomes, detection delays, and K-consecutive-query trade-offs on real-robot tasks and LIBERO-Recovery. Please see shared Tables 3a, 3b, and 4 above. The analyses appear in the revised Appendix Detailed Task-Status Prediction Results and LIBERO-Recovery Simulation and Additional Ablations.

[Q2] Threshold and evaluation-window selection

The original appendix already analyzes sensitivity to the signal thresholds. Performance remains relatively stable over a neighborhood of 0.5. This is also consistent with the signal visualizations: near a state transition, the predicted values often change sharply, potentially moving from low to high within only a few frames. The boundary-based construction of the targets contributes to this behavior. Neither the 0.5 thresholds nor the 15-frame (3 s) trajectory evaluation window were selected using a separate validation set. For fair comparison, we consistently use a fixed threshold of 0.5 and 3s across all experiments. In practical deployment, thresholds could be further calibrated using methods such as conformal prediction on a small set of evaluation rollouts. Such deployment-specific calibration is beyond the scope of this work.

W2&Q3W2 \& Q3

Controlling for failure-data exposure and broader baselines

To control for additional data exposure, we conducted a simulation comparison in which the action-only baseline receives the same failure data and sampling proportions as Echo, but failure-only samples do not contribute an action imitation loss. Under this matched data-sampling protocol, Echo still achieves a higher task success rate.

Table 6. Action-only baseline with additional failure-data exposure.

MethodSuccessful episodes / 48,030Success rate
Action-only Baseline + Failure Data18,10437.69%
Action-only Baseline18,20337.90%
Echo18,92739.41%

We also extended comparisons with three existing baseline methods to two additional real-robot tasks, rather than restricting the comparison to Insert Flower. The results appear in the revised Appendix Baseline Implementations and Performance Analysis.

Table 7. Additional real-robot baseline comparisons. Mean ± SE; Frame and Traj. F1 are Macro-F1 %.

Insert Pen

MethodScore / 10Frame F1 ± SETraj. F1 ± SE
π0.56.87 ± 0.84——
CycleVLA5.67 ± 0.9271.33 ± 3.8866.67 ± 8.75
VLA-Corrector7.33 ± 0.82——
Echo 3−class3-class8.60 ± 0.4689.33 ± 1.4596.67 ± 3.33
RoboMeter 2−class2-class—66.34 ± 3.47100.00 ± 0.00
Echo 2−class2-class8.60 ± 0.4689.56 ± 1.4393.33 ± 4.63

Open Drawer

MethodScore / 10Frame F1 ± SETraj. F1 ± SE
π0.55.40 ± 0.77——
CycleVLA7.83 ± 0.7347.99 ± 0.230.00 ± 0.00
VLA-Corrector6.67 ± 0.88——
Echo 3−class3-class7.90 ± 0.2688.01 ± 1.2290.00 ± 5.57
RoboMeter 2−class2-class—61.67 ± 2.1793.33 ± 4.63
Echo 2−class2-class7.90 ± 0.2688.39 ± 1.4893.33 ± 4.63

The revised manuscript presents these comparisons in Appendix Baseline Implementations and Performance Analysis. Echo has the highest mean task score on both tasks and higher frame-level F1 than the corresponding status baselines. Three-class and two-class results are evaluated under their respective protocols.

Finally, the original paper already includes two-class and three-class evaluation settings aligned with the outputs of the relevant baselines. Where a baseline did not prescribe a success/failure threshold, we selected the threshold yielding its best performance in our reproduction, aiming to avoid disadvantaging the baseline. Further details appear in the appendix ∗∗AppendixA.6∗∗**Appendix A.6**.

We appreciate your suggestions again and are happy to clarify any remaining concerns.

Thank you again for your thoughtful review. We have provided detailed responses to your concerns, including additional experiments and analyses. As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses. We would also be happy to address any remaining questions or concerns.

Reviewer_688vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents Echo, a vision-language-action framework that jointly predicts robot actions, stop signals, and failure signals to distinguish ongoing execution, successful completion, and terminal failure, while using status information to guide action generation. It combines successful demonstrations, failure trajectories, and recovery demonstrations, using segment boundaries and recovery progress to supervise and reduce the need for frame-level progress annotations. Controlled attention interactions and asynchronous denoising allow status signals to become clean earlier and condition subsequent action updates. Using π0.5\pi_{0.5}, the authors evaluate status prediction and task execution on five real-world manipulation tasks. Out-of-distribution tests, component ablations, signal interventions, and failure-data scaling analyses provide evidence for the effectiveness of status–action coupling.

Critical strengths
  1. Practical relevance. The paper integrates task completion detection, failure recognition, and action generation, addressing a core requirement for reliable robotic execution.
  2. Novel coupling of status and actions. Status signals directly inform action generation, with ablations and signal interventions providing evidence for the proposed mechanism.
  3. Well-motivated asynchronous denoising. The method exploits the earlier stabilization of status estimates to provide cleaner conditioning information for subsequent action updates.
Critical weaknesses
  1. Uncertain supervision fidelity and scalability. The approach still relies on human interventions, trajectory segmentation, and labels. Boundary-based targets and the linear recovery assumption may not accurately reflect execution status, while annotation costs at scale remain insufficiently analyzed.
  2. Insufficient justification for joint denoising. Although continuous denoising is a feasible way to model discrete status, comparisons with simpler classification heads or sequential status prediction and action generation are missing.
  3. Evaluation protocol complicates interpretation. Task scores incorporate manually defined penalties, and trajectory-level metrics allow for detection delay. The paper does not report key measures such as false alarm and premature termination rates.
  4. Incomplete attribution of recovery gains. The contributions of additional status supervision, recovery demonstrations, and the coupling mechanism are not fully disentangled. Direct recovery metrics, such as recovery success rate, are also missing.
  5. Limited validation and reproducibility. Experiments cover only one flow-based VLA π0.5\pi_{0.5} backbone. Evidence for dynamic environments, status recognition generalization, applicability to other architectures, and performance on publicly reproducible simulation benchmarks remains insufficient.
Questions for authors
  1. Annotation and supervision: What manual annotations are required for each data type, and at what cost? Has the agreement between boundary-based failure labels, recovery-progress targets, and actual execution status been assessed? How sensitive is the method to boundary errors?
  2. Joint denoising mechanism: What advantages justify joint denoising over an independent classification head, auxiliary status prediction, or predicting status before generating actions? Can the evidence distinguish status-guided action generation from action features improving status prediction?
  3. Evaluation and false alarms: Do SS/FS predictions actually trigger stopping or intervention during deployment? Could you report false alarms, premature termination, missed detections, and detection latency, beyond F1 within the prescribed evaluation window?
  4. Recovery and attribution: Under matched initial failure states, how do Echo, the action-only baseline, and the variant without status-to-action attention compare in recovery success rate and recovery time? This would directly test whether status signals improve recovery.
  5. Generalization: Are the evaluated failure types, objects, and scenes represented in the training data? Are there results on unseen failure types or dynamic disturbances that distinguish generalizable status recognition from recognition of familiar training patterns?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 688v (Part 1)

Thank you for your thoughtful questions. We respond to the corresponding weaknesses and questions together.

W1&Q1W1 \& Q1

Annotation, supervision, and boundary sensitivity

To assess sensitivity to boundary annotation errors, we conducted an experiment on LIBERO-Recovery. We randomly perturbed successful subtask boundaries within ±200 ms and retrained and evaluated the model under otherwise identical settings. Status prediction changed little, whereas task success declined to some extent. This suggests that status recognition is relatively insensitive to moderate boundary errors, while shifts in completion boundaries can affect the quality of the actions learned near the end of a task and consequently reduce task success.

Table 1. Boundary perturbation π05˙andGR00Tπ0\.5 and GR00T.

MetricOriginal Echo π05˙π0\.5Boundary ±0.2 sChange pppp
Overall signal accuracy87.81%89.55%+1.74
Ongoing-frame recall88.41%90.47%+2.06
Success-frame recall79.23%76.07%−3.16
Balanced signal accuracy83.82%83.27%−0.55
Task success rate42.43%41.87%−0.56
MetricOriginal Echo GR00TGR00TBoundary ±0.2 sChange pppp
Overall signal accuracy90.24%89.88%−0.36
Ongoing-frame recall93.32%92.71%−0.61
Success-frame recall40.34%43.46%+3.12
Balanced signal accuracy66.83%68.09%+1.26
Task success rate39.41%39.08%−0.33

The collection procedure and annotation costs are described in our response to all reviewers above.

W2&Q2W2 \& Q2

Joint denoising versus classification-based alternatives

We implemented two additional non-diffusion status-prediction baselines on LIBERO-Recovery. The first adds a classification head after the VLM to predict three classes ongoing/success/failureongoing/success/failure. The ground-truth transition is assigned using the last five frames of each episode. The second uses the predicted status from this classifier as an input to the action expert, thereby conditioning action generation on a separately predicted status. Table 2 reports the results. Echo achieves higher task success and frame-level status accuracy, with fewer false terminal predictions during ongoing execution. The corresponding results appear in the revised manuscript, Appendix LIBERO-Recovery Simulation and Additional Ablations.

Table 2. Non-diffusion status baselines on LIBERO-Recovery %.

MethodTask SRFrame accuracyFalse terminal rate during ongoing execution
Echo42.4387.8111.59
Classifier26.5547.4352.26
Sequential27.1948.1651.48

W3&Q3W3 \& Q3

False alarms, missed detections, and latency

We have added false-stop, missed-detection, and detection-latency analyses for all five real-robot tasks, along with the effect of requiring K consecutive status predictions. We also report analogous results and scene-clustered confidence intervals on LIBERO-Recovery. The full results appear in shared Tables 3a, 3b, and 4 above, rather than being repeated here. The corresponding analyses appear in the revised Appendix Detailed Task-Status Prediction Results and LIBERO-Recovery Simulation and Additional Ablations.

AuthorsOpenReview ↗

Response to Reviewer 688v (Part 2)

W4&Q4W4 \& Q4

Recovery attribution

We performed a recovery-focused ablation on LIBERO-Recovery. Each evaluation episode begins from a perturbed state requiring recovery. The evaluated methods use one-to-one matched initial failure states, enabling paired comparisons under identical initial conditions. We report both recovery success and recovery time in the revised manuscript Appendix∗∗LIBERO−RecoverySimulationandAdditionalAblations∗∗Appendix **LIBERO-Recovery Simulation and Additional Ablations**.

Table 5. Recovery success and timing ablation in simulation.

MethodRecovery successMean duration (all episodes)Mean duration (successful episodes)
Echo GR00TGR00T18,927/48,030 394˙139\.41%19.274 s8.933 s
Two-class action-only baseline18,203/48,030 379˙037\.90%19.430 s8.665 s
w/o S→AS \to A18,073/48,030 376˙337\.63%19.494 s8.710 s

The success-rate trend is consistent with our existing real-robot ablation. Recovery time, however, does not exhibit a clear monotonic pattern: it depends on the durations and characteristics of the recovery and nominal demonstration trajectories. Accordingly, elapsed time alone should not be treated as a definitive measure of recovery quality.

W5&Q5W5 \& Q5

Reproducibility, backbone transfer, and generalization

To improve reproducibility, we have added anonymous links to the LIBERO-Recovery implementation and data, together with additional simulation experiments in the revised paper. We evaluated Echo on two VLA backbones, π0.5 and GR00T. For each backbone, Echo and its baseline start from the same pretrained checkpoint and train for one epoch under matched configurations. Echo improves task success relative to the corresponding baselines in both settings π05˙:424˙3π0\.5: 42\.43% vs\. 41\.94%; GR00T: 39\.41% vs\. 37\.90% revisedSection∗∗SimulationEvaluation∗∗andAppendix∗∗LIBERO−RecoverySimulationandAdditionalAblations∗∗revised Section **Simulation Evaluation** and Appendix **LIBERO-Recovery Simulation and Additional Ablations**.

Regarding generalization, the real-robot evaluation primarily tests changes in object positions. The simulation evaluation also varies object placements, spatial relationships, colors, and scene attributes across approximately 4,800 scenarios. Echo retains an advantage over the baselines across these variations, providing evidence of generalization beyond exact training configurations.

We distinguish between unseen failure appearances and fundamentally unseen recovery skills. When a failure arises from a novel configuration but can be corrected using recovery behaviors represented in training, Echo can generalize to the new configuration. By contrast, supervised fine-tuning alone should not be expected to synthesize an entirely unseen class of recovery actions reliably. This limitation is shared by imitation-based VLA policies more generally. A scalable path forward is to collect more deployment failures, including failure-only trajectories, together with corrective demonstrations where needed, and use them to expand both failure recognition and recovery competence. This interpretation is consistent with our failure-data scaling experiment in Section 4.4.4, “Effect of Failure Data Quantity.”

Thank you again for your constructive feedback. We welcome further questions and will update the paper as needed.

Thank you again for your thoughtful review. We have provided detailed responses to your concerns, including additional experiments and analyses. As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses. We would also be happy to address any remaining questions or concerns.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to all the reviewers and chairs

We sincerely thank the reviewers and the Chairs for their careful assessment and constructive feedback. We first address several shared concerns and clarify the abbreviations used below: W1, W2, etc. refer to the numbered weaknesses, while Q1, Q2, etc. refer to the numbered questions in each review.

All three reviewers raised questions about our data collection and annotation protocol. Section 3.2.2, “Data Construction and Supervision,” briefly describes this protocol. We provide additional details here and have also included them in the appendix of the revised manuscript.

First, our four data types arise naturally from three collection modes: ii successful demonstrations obtained by teleoperation; iiii recovery trajectories consisting of teleoperated actions during takeover-mode collection; iiiiii pre-intervention, not-yet-terminal failure trajectories consisting of policy-generated actions during takeover-mode collection; and iviv terminal failure trajectories obtained from autonomous policy rollouts.

After training an initial policy on successful demonstrations, we can collect takeover-mode trajectories and autonomous rollouts in parallel and use the resulting data to further improve the policy. The four data types are recorded as separate episodes, so their episode boundaries are explicit and do not require manual segmentation. Intervention times are also logged automatically by the collection software. The only boundaries that may require manual annotation are subtask transitions within successful demonstrations; Single-subtask demonstrations require no manual post hoc segmentation. Annotating one subtask boundary takes approximately 1–2 seconds. This provides a relatively efficient collection and low-cost annotation pipeline.

Second, the linear recovery target is a practical approximation, consistent with the use of progress-like targets in related work such as π0.6/RECAP and VLAC. Our collection protocol also aims to avoid recovery trajectories with substantial backward progress or extended stagnation. More importantly, Echo does not require a perfectly calibrated, frame-accurate recovery-progress estimate: the signals are designed to distinguish ongoing execution, success, and failure, and to provide a conditioning cue that helps steer actions toward recovery. The favorable results on LIBERO-Recovery provide additional empirical support for this design.

Third, we have expanded the experiments, including large-scale simulation on LIBERO-Recovery and additional real-robot baseline comparisons. Anonymous links to the simulation code and data are provided in the manuscript. Our simulation evaluation covers approximately 4,800 scenarios, with ten trials per scenario. The results support the task-success advantage of Echo over the baselines and provide additional evidence about the usefulness of the status signals. We discuss the new real-world and simulation results in the reviewer-specific responses below.

AuthorsOpenReview ↗

Shared additional experiments: detection reliability and latency (Part 1)

Response to Reviewer 688v

W3&Q3W3 \& Q3

and 27wk

W1&Q1W1 \& Q1

We additionally computed false stops, missed detections, and detection latency on all five real-robot tasks. These results also address the related questions raised by Reviewers 688v and 27wk. Real-robot status queries are nominally issued every 0.2 s.

Table 3a. Additional real-robot status metrics (K=1).

TaskPremature stop signalsSuccess outcome correct (15 queries)Success signal detected within 3 sSuccess missed in a full 3-s windowMedian success delayFailure missed in a full 3-s window
Capsule Toy0/3030/3030/300/211.000 s–
Open Drawer10/3027/3030/300/291.300 s–
Insert Flower1/3024/2525/250/130.800 s0/5
Insert Pen0/3027/2828/280/150.600 s0/2
Make Tanghulu1/3015/1616/160/50.600 s1/14
Total12/150123/129129/1290/830.800 s1/21

A status prediction can occasionally exhibit a one-frame spike. Requiring the predicted state to exceed its threshold for K consecutive queries before confirming a transition provides a practical trade-off between false alarms and detection latency. During deployment, K can be selected using a small number of calibration rollouts. Real-robot status queries are nominally issued every 0.2 s.

Table 3b. Consecutive-query confirmation trade-off.

Consecutive queries KFalse stop signalsFirst reported outcome correctSuccess outcome correct (15 queries)Success signal detected within 3 sSuccess missed in a full 3-s windowMedian success delay
112/150126/150123/129129/1290/830.800 s
21/150139/150126/129128/1291/831.033 s
30/150143/150124/129125/1293/831.267 s

Across 150 real-robot trials, we observe 12 premature stop reports. With (K=1), the outcome is correctly identified within the 15-query evaluation window in 123 of 129 successful episodes, while a success signal is detected in all 129 episodes. Success signal detection records whether a success prediction occurs within the window, whereas outcome correctness prioritizes failure predictions when both outcomes are detected. Consequently, six successful episodes contain both success and failure predictions and are classified as failures.

The full-window miss rate is evaluated on the 83 successful episodes with an available 3-second observation window. No success is missed in this subset at (K=1). Among 21 failure outcomes, one is missed within a full 3-second window.

AuthorsOpenReview ↗

Shared additional experiments: detection reliability and latency (Part 2)

We also conducted an similar analysis in the LIBERO-Recovery simulation benchmark.

Table 4. Additional simulation status metrics and confidence intervals.

ModelTask successSignal frame accuracy ≤1≤1Success-frame recallPremature report / final successTimeout-related false success
π0.5 Echo20,381/48,030 ∗∗424˙3**42\.43%**87.81%79.23%12,969/20,38113,977/27,649
π0.5 action-only20,145/48,030 ∗∗419˙4**41\.94%**N/AN/AN/AN/A
GR00T Echo18,927/48,030 ∗∗394˙1**39\.41%**90.24%40.34%2,038/18,92711,651/29,103
GR00T action-only18,203/48,030 ∗∗379˙0**37\.90%**N/AN/AN/AN/A
ModelK valuesPremature reports K=1/2/3K=1/2/3Success detected K=1/2/3K=1/2/3Missed within full 3-s window K=1/2/3K=1/2/3Median detection delay K=1/2/3K=1/2/3
π0.5 Echo1 / 2 / 312,969 / 4,969 / 1,43319,749 / 18,998 / 18,3721,238 / 2,258 / 3,1330.25 / 0.35 / 0.75 s
GR00T Echo1 / 2 / 32,038 / 349 / 13917,919 / 16,821 / 15,1502,571 / 6,770 / 11,5130.75 / 1.85 / 3.20 s
Scene levelπ0.5 Echo − baseline (percentage points)GR00T Echo − baseline (percentage points)
All scenes+0.491
9595% CI: +0\.085, +0\.889

| +1.507 [95% CI: +1.056, +1.972] |

In simulation, a premature terminal report denotes a terminal-status prediction before task completion in a rollout that eventually succeeds. These reports are counted per rollout rather than per frame; the denominator is the number of eventually successful rollouts. Such predictions may be transient: for π0.5 Echo, requiring three consecutive terminal predictions reduces premature reports from 12,969 to 1,433, while median detection delay increases from 0.25 to 0.75 s. For GR00T Echo, the corresponding count decreases from 2,038 to 139, with median delay increasing from 0.75 to 3.20 s. These results characterize the trade-off between terminal-report reliability and detection latency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.