- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DreamAvoid, a critical-phase test-time dreaming framework that enables VLA policies to anticipate and avoid failures in fine-grained manipulation. The key idea is to keep the base VLA policy frozen during routine execution, and to invoke extra computation only when a lightweight Dream Trigger detects an imminent critical phase. The framework consists of three modules: a Dream Trigger that predicts critical-phase onset, an Action Proposer that samples diverse candidate action chunks via stochastic flow integration, and a Dream Evaluator that uses a world model to “dream” short-horizon futures for each candidate and selects the action with the highest predicted progress value.
The main contributions are: (1) a selective test-time dreaming framework that concentrates computation on consequential moments; (2) an autonomous boundary learning paradigm that collects success, failure, and boundary data during deployment to give the world and value models failure awareness; and (3) a pilot study showing that explicit spatiotemporal dreaming improves failure prediction over direct semantic evaluation of numerical action sequences.
Validation includes real-world manipulation on four tasks (Cup Sleeving, Charger Plugging, Cap Opening, Screw Insertion), where DreamAvoid with autonomous boundary learning reaches 72.5% average success, versus 48.8% for the base policy and 54.4% for GPC-RANK. Ablations show that both SDE-based candidate generation and the learned Dream Evaluator are important. Simulation results report gains on RoboTwin 2.0 (66.3% to 70.8%) and SimplerEnv (Bridge 59.9% to 63.6%; Fractal 76.4% to 80.7%). The paper also reports trigger sensitivity, failure audits, latency analysis, and statistical tests.
Critical strengths
The paper correctly identifies that VLA failures in fine-grained manipulation are concentrated in brief, low-tolerance critical phases rather than spread uniformly across the trajectory. The proposed Dream Trigger allows extra computation to be invoked only when such a phase is detected, keeping the base policy frozen during routine execution. This is consequential because always-on test-time compute is expensive and can interfere with smooth execution. The real-world results support the design: DreamAvoid reaches 72.5% average success versus 48.8% for the base policy and 54.4% for GPC-RANK, and the ablations show that both SDE-based candidate generation and the learned Dream Evaluator are necessary for this gain.
Base VLA policies are trained mainly on successful demonstrations, so their world models tend to produce optimistic futures and cannot reliably distinguish subtle success–failure boundaries. The proposed autonomous boundary learning paradigm collects success, failure, and boundary data during deployment and uses it to fine-tune the world and value models. This is consequential because it tackles a core limitation of existing VLA failure-avoidance methods rather than only adding inference-time search. The improvement from DA-Vanilla to DA-ABL (66.9% to 72.5% real-world average) provides direct evidence that boundary and failure data materially improve preventive action selection.
Critical weaknesses
DreamAvoid relies on DreamDojo for world modeling, Gemini 3.6 Flash for offline critical-phase onset annotation and pilot evaluation, and Robometer for progress labels. The Dream Trigger is trained on tens of thousands of frames annotated by Gemini, and the value model depends on Robometer-generated pseudo-labels. This is consequential because the reported gains may not transfer if these external models are unavailable, updated, or replaced, and the annotation process itself may introduce systematic bias. The paper does not report sensitivity to annotation quality or alternative world models, nor does it provide a fully self-contained training recipe. For a method that claims practical failure avoidance, this dependency raises serious concerns about reproducibility and broad adoption.
The proposed paradigm requires deploying an initial version of DreamAvoid, whose Dream Evaluator is known to produce optimistic hallucinations, in order to collect failure and boundary data. During this data-collection phase, the system may select locally plausible but globally dangerous actions, causing real irreversible failures. The paper reports that this process naturally exposes “trickier boundary cases,” but it does not provide safety guarantees, fallback mechanisms, or human-in-the-loop constraints during collection. This is consequential because the method is intended for safety-critical manipulation, where deploying an unreliable evaluator to learn from its own failures can be unacceptable. Without a safe exploration protocol or a clear argument that the initial system is already safe enough, the autonomous boundary learning claim remains difficult to justify for real-world deployment.
Questions for authors
Safety of autonomous boundary learning. The paper proposes deploying an initial DreamAvoid to collect failure and boundary data, but the initial Dream Evaluator is described as optimistic and may select dangerous actions. Could you clarify what safety mechanisms, human supervision, or fallback constraints were used during this data collection? How many real-world failures were caused during collection, and were any irreversible?
Real-time feasibility of test-time dreaming. The critical-phase onset is annotated within 1–2 s before decisive contact, but a single Dream Evaluator intervention is reported to take about 2.133 s, including 2.0 s for video generation. Could you clarify whether the dreaming completes before the intervention window closes? How often does the system miss the window, and how does latency affect success?
Statistical significance of boundary learning. In the real-world comparison, DA-ABL versus DA-Vanilla gives p = 0.251, which does not reach significance. Could you provide additional trials or task-level analyses to establish whether autonomous boundary learning provides a reliable gain over DA-Vanilla? This directly affects the strength of the central contribution.
Reliability of Gemini annotations. Critical-phase onsets for trigger training are annotated offline by Gemini 3.6 Flash. Could you report inter-annotator agreement, human verification rates, or sensitivity to annotation noise? If these annotations are biased or inconsistent, the Dream Trigger and the evaluation of intervention timing may be affected.
Trigger generalization and sensitivity. Tables 3 and 4 report threshold and soft-label sensitivity, but Table 4 appears limited to Charger Plugging. Could you clarify how γ, β, and trigger performance vary across all four real-world tasks and the simulation benchmarks? Is a separate trigger head and threshold needed for each task or stage?
Dependency on external models. The pipeline relies on DreamDojo, Gemini 3.6 Flash, and Robometer. Could you clarify how performance changes if these are replaced by open-source or self-contained alternatives? Is the method reproducible without access to these specific models, and were any ablations run with different world or reward models?
False negatives in the failure audit. The failure audit reports 6/42 = 14.3% false negatives, meaning hazardous actions were scored above threshold and executed. Could you discuss whether this rate is acceptable for safety-critical manipulation and what safety layer would prevent these failures in deployment?
Compute-matched comparison with GPC-RANK. GPC-RANK is always-on, while DreamAvoid intervenes selectively. Could you provide a compute-matched comparison in wall-clock time or FLOPs, and clarify whether the reported latency advantage accounts for the full pipeline, including trigger polling, video generation, and value scoring?
2 comments
Dear Reviewer UmJD,
As the discussion period ends in just over one day (Nov 18, AoE), we would appreciate it if you could review our rebuttal and let us know whether your concerns have been addressed. To summarize the new experiments and analyses added in our revision:
-
Annotation Noise Sensitivity (Appendix C.1.1): Retraining all four real-world triggers with uniform onset jitter of and frames maintains 92.5% and 93.8% phase recall (vs. 95.0% without jitter), confirming robustness to annotation noise.
-
Statistical Significance of ABL (Appendix F): Across all 16 real-world and SimplerEnv task strata, DA-ABL outperforms DA-Vanilla in 15, ties in 1, and loses in 0 (sign test ), and both significantly outperform the base policy and GPC-RANK ().
-
Per-Task Trigger Results & Compute Comparison (Table 3 & Table 8): We reported per-task trigger recall (85%–100%) and precision (90.5%–100%) across all four tasks, and confirmed that total rollout overhead including asynchronous polling is at most 3.2–3.9 s per rollout (vs. 8.33–13.88 s for GPC-RANK).
-
Collection Safety Protocol (Section 4.4 & Appendix D.4): We documented our two-operator supervision, hardware e-stop/limits, and confirmed zero hardware damage or safety incidents across all 400 collection rollouts.
We are happy to provide further clarification if needed. Thank you for your time and detailed review.
Best regards,
The Authors
Thank you for recognizing critical-phase dreaming and boundary learning. The revision documents the collection safety protocol (W2 & Q1) and adds significance tests, onset-noise and per-task trigger results (Q3–Q5).
W1 & Q6. Gemini 3.6 Flash (critical-phase onsets for trigger training) and Robometer (frame-level progress for value targets) are used only offline. Their updates or unavailability do not affect a trained system. DreamDojo-2B only initializes the world model, which is then post-trained on our robot data and distilled. DreamAvoid's recipe takes onsets, progress labels and an action-conditioned video model as inputs and is not tied to these specific models.
Without any world model, the no-video critic (LGCu W1) still ranks success above failure chunks in 67% of pairs (83% with dreamed video). Under ±5- or ±10-frame onset jitter, the retrained trigger still detects 92.5–93.8% of critical phases (95.0% without jitter; Q4). Robometer is open-weight (Appendix C.3.2), and the revised Appendix C lists the shared and task-specific settings.
W2 & Q1. Collection used DA-Vanilla, trained on teleoperation data only: π0.5 stays frozen, candidates are sampled around its actions, and DA-Vanilla already fails less often than π0.5 alone (66.9% vs 48.8% success). Collection and evaluation used two supervising operators, a hardware emergency stop and AgileX-specified limits/protections. Operators aborted a rollout when the gripper risked hitting the table hard enough to cut power; aborted evaluation trials count as failures.
Of the 400 collection rollouts (100 per task), 147 were task failures, from which ABL learns; “irreversible” means an unrecoverable task state. Objects were a paper cup, charger/unpowered socket, spray cap and screw. No hardware was damaged and the robot touched no one; the emergency stop was triggered twice during policy rollouts. Section 4.4, Appendix D.4 and the Ethics Statement now document this protocol.
Q2. Yes, when the intervention starts before contact: dreaming pauses the robot in place (Appendix D.3), so the 2.133 s lengthens the episode without consuming the pre-contact margin. The trigger fires inside the annotated window for 76/80 phases (Table 3), and only 2 of 160 DA-ABL trials failed without a timely intervention (Appendix C.3.3). Reported success includes these pauses.
Q3. Across 4 real and 12 SimplerEnv task strata, DA-ABL is higher than DA-Vanilla in 15, tied in 1 and lower in none (sign test, p = 6.1×10⁻⁵). The real-world gain is 5.6 points (p = 0.251); 80% power would need ≈1,050 trials per variant, so we added to Appendix F a cross-strata consistency test and DA-Vanilla's CMH tests: both variants beat the base policy and GPC-RANK (real-world p ≤ 0.018). DA-ABL's evaluator also recalls more failures than DA-Vanilla's (69% vs 55%) at equal precision and ranks executed chunks better (76% vs 66%; added to Appendix C.3.4).
Q4. Operators judge success from the final physical state, not with Gemini or Robometer (added to Appendix D.1), so annotation errors do not enter measured success. Table 4's Charger sweep (β = 2–10 frames) examines soft-label sensitivity: phase recall stays at 90–100%. To test sensitivity to onset noise, we jittered every training onset uniformly by up to ±5 or ±10 frames, retrained the triggers of all four tasks and repeated Table 3's evaluation (added to Appendix C.1.1). Phase recall is 92.5% and 93.8%, respectively (95.0% without jitter), and offline detector events per trajectory rise only from 1.03 to 1.06 and 1.11.
Q5. All four real tasks (one critical phase each) share γ = 0.5 and β = 5 frames. Each task has its own small trigger (frozen DINOv2 + MLP, one output per stage), trained on its 100 teleoperation demonstrations. By task, Table 3's phase recall at γ = 0.5 is 20/20 (Cup), 17/20 (Charger), 20/20 (Cap) and 19/20 (Screw), and event precision is 20/21, 17/20, 20/20 and 19/21 (added to Appendix C.1.1). Simulation trigger parameters are selected on a validation split before testing (Section 6).
Q6. Addressed in W1 & Q6 above.
Q7. The safety layer is the human supervision and hardware safeguards of W2 & Q1; DreamAvoid adds failure avoidance on top of them (Appendix A). The 6/42 (14.3%) misses led to task failures already counted in the reported 72.5% success (vs 48.8% for π0.5); none caused hardware damage, and the robot touched no one.
Q8. Table 8 includes proposal, video generation and value scoring (0.093 + 2.000 + 0.040 = 2.133 s), vs 0.694 s per call for GPC-RANK. DreamAvoid intervenes at most once per critical phase, i.e., at most once per rollout on our real tasks, vs 12–20 GPC-RANK replans, with 72.5% vs 54.4% success. Asynchronous trigger polling runs off the critical path, so Table 8 excludes it; counting it anyway (≈0.5–0.8 s) gives at most 3.2–3.9 s per rollout (added to the Table 8 note) vs 8.33–13.88 s for GPC-RANK.