acceptodds
Under review as a conference paper at ICLR 2027

DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policies

Abstract

Vision-Language-Action (VLA) models are often brittle in fine-grained manipulation, where minor action errors during the critical phases can rapidly escalate into irrecoverable failures. Since existing VLA models rely predominantly on successful demonstrations for training, they lack an explicit awareness of failure during these critical phases. To address this, we propose DreamAvoid, a critical-phase test-time dreaming framework that enables VLA models to anticipate and avoid failures. We also introduce an autonomous boundary learning paradigm to refine the system's understanding of the subtle boundary between success and failure. Specifically, we (1) utilize a Dream Trigger to determine whether the execution has entered a critical phase, (2) sample multiple candidate action chunks from the VLA via an Action Proposer, and (3) employ a Dream Evaluator, jointly trained on mixed data (success, failure, and boundary cases), to “dream” the short-horizon futures corresponding to the candidate actions, evaluate their values, and select the optimal action. We conduct extensive evaluations on real-world manipulation tasks and simulation benchmarks. The results demonstrate that DreamAvoid can effectively avoid failures, thereby improving the overall task success rate. Across four real-world tasks, DreamAvoid achieves 72.5% success, compared with 48.8% for the base policy and 54.4% for GPC-RANK. Our code is available at https://anonymous.4open.science/r/DreamAvoid.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLA, test-time, world model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UmJDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DreamAvoid, a critical-phase test-time dreaming framework that enables VLA policies to anticipate and avoid failures in fine-grained manipulation. The key idea is to keep the base VLA policy frozen during routine execution, and to invoke extra computation only when a lightweight Dream Trigger detects an imminent critical phase. The framework consists of three modules: a Dream Trigger that predicts critical-phase onset, an Action Proposer that samples diverse candidate action chunks via stochastic flow integration, and a Dream Evaluator that uses a world model to “dream” short-horizon futures for each candidate and selects the action with the highest predicted progress value.

The main contributions are: (1) a selective test-time dreaming framework that concentrates computation on consequential moments; (2) an autonomous boundary learning paradigm that collects success, failure, and boundary data during deployment to give the world and value models failure awareness; and (3) a pilot study showing that explicit spatiotemporal dreaming improves failure prediction over direct semantic evaluation of numerical action sequences.

Validation includes real-world manipulation on four tasks (Cup Sleeving, Charger Plugging, Cap Opening, Screw Insertion), where DreamAvoid with autonomous boundary learning reaches 72.5% average success, versus 48.8% for the base policy and 54.4% for GPC-RANK. Ablations show that both SDE-based candidate generation and the learned Dream Evaluator are important. Simulation results report gains on RoboTwin 2.0 (66.3% to 70.8%) and SimplerEnv (Bridge 59.9% to 63.6%; Fractal 76.4% to 80.7%). The paper also reports trigger sensitivity, failure audits, latency analysis, and statistical tests.

Critical strengths

The paper correctly identifies that VLA failures in fine-grained manipulation are concentrated in brief, low-tolerance critical phases rather than spread uniformly across the trajectory. The proposed Dream Trigger allows extra computation to be invoked only when such a phase is detected, keeping the base policy frozen during routine execution. This is consequential because always-on test-time compute is expensive and can interfere with smooth execution. The real-world results support the design: DreamAvoid reaches 72.5% average success versus 48.8% for the base policy and 54.4% for GPC-RANK, and the ablations show that both SDE-based candidate generation and the learned Dream Evaluator are necessary for this gain.

Base VLA policies are trained mainly on successful demonstrations, so their world models tend to produce optimistic futures and cannot reliably distinguish subtle success–failure boundaries. The proposed autonomous boundary learning paradigm collects success, failure, and boundary data during deployment and uses it to fine-tune the world and value models. This is consequential because it tackles a core limitation of existing VLA failure-avoidance methods rather than only adding inference-time search. The improvement from DA-Vanilla to DA-ABL (66.9% to 72.5% real-world average) provides direct evidence that boundary and failure data materially improve preventive action selection.

Critical weaknesses

DreamAvoid relies on DreamDojo for world modeling, Gemini 3.6 Flash for offline critical-phase onset annotation and pilot evaluation, and Robometer for progress labels. The Dream Trigger is trained on tens of thousands of frames annotated by Gemini, and the value model depends on Robometer-generated pseudo-labels. This is consequential because the reported gains may not transfer if these external models are unavailable, updated, or replaced, and the annotation process itself may introduce systematic bias. The paper does not report sensitivity to annotation quality or alternative world models, nor does it provide a fully self-contained training recipe. For a method that claims practical failure avoidance, this dependency raises serious concerns about reproducibility and broad adoption.

The proposed paradigm requires deploying an initial version of DreamAvoid, whose Dream Evaluator is known to produce optimistic hallucinations, in order to collect failure and boundary data. During this data-collection phase, the system may select locally plausible but globally dangerous actions, causing real irreversible failures. The paper reports that this process naturally exposes “trickier boundary cases,” but it does not provide safety guarantees, fallback mechanisms, or human-in-the-loop constraints during collection. This is consequential because the method is intended for safety-critical manipulation, where deploying an unreliable evaluator to learn from its own failures can be unacceptable. Without a safe exploration protocol or a clear argument that the initial system is already safe enough, the autonomous boundary learning claim remains difficult to justify for real-world deployment.

Questions for authors

Safety of autonomous boundary learning. The paper proposes deploying an initial DreamAvoid to collect failure and boundary data, but the initial Dream Evaluator is described as optimistic and may select dangerous actions. Could you clarify what safety mechanisms, human supervision, or fallback constraints were used during this data collection? How many real-world failures were caused during collection, and were any irreversible?

Real-time feasibility of test-time dreaming. The critical-phase onset is annotated within 1–2 s before decisive contact, but a single Dream Evaluator intervention is reported to take about 2.133 s, including 2.0 s for video generation. Could you clarify whether the dreaming completes before the intervention window closes? How often does the system miss the window, and how does latency affect success?

Statistical significance of boundary learning. In the real-world comparison, DA-ABL versus DA-Vanilla gives p = 0.251, which does not reach significance. Could you provide additional trials or task-level analyses to establish whether autonomous boundary learning provides a reliable gain over DA-Vanilla? This directly affects the strength of the central contribution.

Reliability of Gemini annotations. Critical-phase onsets for trigger training are annotated offline by Gemini 3.6 Flash. Could you report inter-annotator agreement, human verification rates, or sensitivity to annotation noise? If these annotations are biased or inconsistent, the Dream Trigger and the evaluation of intervention timing may be affected.

Trigger generalization and sensitivity. Tables 3 and 4 report threshold and soft-label sensitivity, but Table 4 appears limited to Charger Plugging. Could you clarify how γ, β, and trigger performance vary across all four real-world tasks and the simulation benchmarks? Is a separate trigger head and threshold needed for each task or stage?

Dependency on external models. The pipeline relies on DreamDojo, Gemini 3.6 Flash, and Robometer. Could you clarify how performance changes if these are replaced by open-source or self-contained alternatives? Is the method reproducible without access to these specific models, and were any ablations run with different world or reward models?

False negatives in the failure audit. The failure audit reports 6/42 = 14.3% false negatives, meaning hazardous actions were scored above threshold and executed. Could you discuss whether this rate is acceptable for safety-critical manipulation and what safety layer would prevent these failures in deployment?

Compute-matched comparison with GPC-RANK. GPC-RANK is always-on, while DreamAvoid intervenes selectively. Could you provide a compute-matched comparison in wall-clock time or FLOPs, and clarify whether the reported latency advantage accounts for the full pipeline, including trigger polling, video generation, and value scoring?

2 comments

Thank you for recognizing critical-phase dreaming and boundary learning. The revision documents the collection safety protocol (W2 & Q1) and adds significance tests, onset-noise and per-task trigger results (Q3–Q5).

W1 & Q6. Gemini 3.6 Flash (critical-phase onsets for trigger training) and Robometer (frame-level progress for value targets) are used only offline. Their updates or unavailability do not affect a trained system. DreamDojo-2B only initializes the world model, which is then post-trained on our robot data and distilled. DreamAvoid's recipe takes onsets, progress labels and an action-conditioned video model as inputs and is not tied to these specific models.

Without any world model, the no-video critic (LGCu W1) still ranks success above failure chunks in 67% of pairs (83% with dreamed video). Under ±5- or ±10-frame onset jitter, the retrained trigger still detects 92.5–93.8% of critical phases (95.0% without jitter; Q4). Robometer is open-weight (Appendix C.3.2), and the revised Appendix C lists the shared and task-specific settings.

W2 & Q1. Collection used DA-Vanilla, trained on teleoperation data only: π0.5 stays frozen, candidates are sampled around its actions, and DA-Vanilla already fails less often than π0.5 alone (66.9% vs 48.8% success). Collection and evaluation used two supervising operators, a hardware emergency stop and AgileX-specified limits/protections. Operators aborted a rollout when the gripper risked hitting the table hard enough to cut power; aborted evaluation trials count as failures.

Of the 400 collection rollouts (100 per task), 147 were task failures, from which ABL learns; “irreversible” means an unrecoverable task state. Objects were a paper cup, charger/unpowered socket, spray cap and screw. No hardware was damaged and the robot touched no one; the emergency stop was triggered twice during policy rollouts. Section 4.4, Appendix D.4 and the Ethics Statement now document this protocol.

Q2. Yes, when the intervention starts before contact: dreaming pauses the robot in place (Appendix D.3), so the 2.133 s lengthens the episode without consuming the pre-contact margin. The trigger fires inside the annotated window for 76/80 phases (Table 3), and only 2 of 160 DA-ABL trials failed without a timely intervention (Appendix C.3.3). Reported success includes these pauses.

Q3. Across 4 real and 12 SimplerEnv task strata, DA-ABL is higher than DA-Vanilla in 15, tied in 1 and lower in none (sign test, p = 6.1×10⁻⁵). The real-world gain is 5.6 points (p = 0.251); 80% power would need ≈1,050 trials per variant, so we added to Appendix F a cross-strata consistency test and DA-Vanilla's CMH tests: both variants beat the base policy and GPC-RANK (real-world p ≤ 0.018). DA-ABL's evaluator also recalls more failures than DA-Vanilla's (69% vs 55%) at equal precision and ranks executed chunks better (76% vs 66%; added to Appendix C.3.4).

Q4. Operators judge success from the final physical state, not with Gemini or Robometer (added to Appendix D.1), so annotation errors do not enter measured success. Table 4's Charger sweep (β = 2–10 frames) examines soft-label sensitivity: phase recall stays at 90–100%. To test sensitivity to onset noise, we jittered every training onset uniformly by up to ±5 or ±10 frames, retrained the triggers of all four tasks and repeated Table 3's evaluation (added to Appendix C.1.1). Phase recall is 92.5% and 93.8%, respectively (95.0% without jitter), and offline detector events per trajectory rise only from 1.03 to 1.06 and 1.11.

Q5. All four real tasks (one critical phase each) share γ = 0.5 and β = 5 frames. Each task has its own small trigger (frozen DINOv2 + MLP, one output per stage), trained on its 100 teleoperation demonstrations. By task, Table 3's phase recall at γ = 0.5 is 20/20 (Cup), 17/20 (Charger), 20/20 (Cap) and 19/20 (Screw), and event precision is 20/21, 17/20, 20/20 and 19/21 (added to Appendix C.1.1). Simulation trigger parameters are selected on a validation split before testing (Section 6).

Q6. Addressed in W1 & Q6 above.

Q7. The safety layer is the human supervision and hardware safeguards of W2 & Q1; DreamAvoid adds failure avoidance on top of them (Appendix A). The 6/42 (14.3%) misses led to task failures already counted in the reported 72.5% success (vs 48.8% for π0.5); none caused hardware damage, and the robot touched no one.

Q8. Table 8 includes proposal, video generation and value scoring (0.093 + 2.000 + 0.040 = 2.133 s), vs 0.694 s per call for GPC-RANK. DreamAvoid intervenes at most once per critical phase, i.e., at most once per rollout on our real tasks, vs 12–20 GPC-RANK replans, with 72.5% vs 54.4% success. Asynchronous trigger polling runs off the critical path, so Table 8 excludes it; counting it anyway (≈0.5–0.8 s) gives at most 3.2–3.9 s per rollout (added to the Table 8 note) vs 8.33–13.88 s for GPC-RANK.

Dear Reviewer UmJD,

As the discussion period ends in just over one day (Nov 18, AoE), we would appreciate it if you could review our rebuttal and let us know whether your concerns have been addressed. To summarize the new experiments and analyses added in our revision:

  1. Annotation Noise Sensitivity (Appendix C.1.1): Retraining all four real-world triggers with uniform onset jitter of ±5\pm 5 and ±10\pm 10 frames maintains 92.5% and 93.8% phase recall (vs. 95.0% without jitter), confirming robustness to annotation noise.

  2. Statistical Significance of ABL (Appendix F): Across all 16 real-world and SimplerEnv task strata, DA-ABL outperforms DA-Vanilla in 15, ties in 1, and loses in 0 (sign test p=6.1×10−5p = 6.1 \times 10^{-5}), and both significantly outperform the base policy and GPC-RANK (p≤0.018p \le 0.018).

  3. Per-Task Trigger Results & Compute Comparison (Table 3 & Table 8): We reported per-task trigger recall (85%–100%) and precision (90.5%–100%) across all four tasks, and confirmed that total rollout overhead including asynchronous polling is at most 3.2–3.9 s per rollout (vs. 8.33–13.88 s for GPC-RANK).

  4. Collection Safety Protocol (Section 4.4 & Appendix D.4): We documented our two-operator supervision, hardware e-stop/limits, and confirmed zero hardware damage or safety incidents across all 400 collection rollouts.

We are happy to provide further clarification if needed. Thank you for your time and detailed review.

Best regards,

The Authors

Reviewer_kQtZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes DreamAvoid, a framework that augments a frozen vision-language-action policy with selective test-time computation during critical manipulation phases. A Dream Trigger detects when intervention is needed, an Action Proposer samples diverse candidate action chunks, and a Dream Evaluator uses a world model to predict their short-horizon visual consequences and a learned value model to rank them. The paper also introduces Autonomous Boundary Learning (ABL), which uses deployment-collected success, failure, and boundary trajectories to adapt the world and value models. The approach is evaluated on four real-world manipulation tasks and the RoboTwin 2.0 and SimplerEnv benchmarks.

Critical strengths

A well-motivated integration of selective test-time computation with substantial empirical benefits in real-world manipulation.

Concentrating additional computation on critical interaction phases is a sensible design for contact-sensitive manipulation, where a brief local error can determine the outcome of an otherwise reasonable trajectory. DreamAvoid combines this scheduling strategy with diverse action proposals and predicted-consequence evaluation while keeping the base VLA policy frozen. The complete system achieves 72.5% success across four real-world tasks, compared with 48.8% for the base policy and 54.4% for GPC-RANK, with 40 trials per task per method. The statistical comparisons in Appendix F support these improvements over the two baselines. Aggregate improvements on RoboTwin 2.0 and SimplerEnv provide additional evidence of the framework’s utility. These results make the paper a useful empirical contribution to improving VLA execution through selective test-time planning.

Critical weaknesses

The deployed Dream Evaluator lacks a sufficiently direct and systematic evaluation of its prediction and ranking quality.

The method relies on the evaluator assigning higher values to actions with better actual consequences, particularly near success–failure boundaries. However, the reported evaluations only partially establish this capability. Table 1 reports 80% accuracy for DreamDojo-generated videos assessed by Gemini, which uses a different downstream evaluator from the learned value model deployed in DreamAvoid. The failure audit in Appendix C.3.3 reports 85.7% hazard recall over 42 evaluated failure-causing action chunks. This is useful evidence about missed hazards, but it does not characterize false alarms on successful actions or the correctness of rankings across candidate actions.

Consequently, it remains unclear how reliably the deployed evaluator distinguishes locally plausible but failing actions from genuinely useful alternatives, and whether ABL improves this distinction. A direct evaluation against actual execution outcomes, covering successful, failed, and boundary cases, would address this gap. In particular, comparing DA-Vanilla and DA-ABL on the same held-out evaluation set would help establish whether boundary learning improves consequence evaluation. This matters because the encouraging task-level success rates do not, by themselves, establish the proposed mechanism of improved failure awareness.

Questions for authors

Clarification of the existing failure audit: What diagnostic threshold was used in Appendix C.3.3, how was it selected, and was it fixed independently of the test outcomes?

Additional validation of the central mechanism: Could the authors evaluate the deployed Dream Evaluator on a common held-out set containing successful, failed, and boundary outcomes, and report failure precision/recall together with candidate-ranking quality for both DA-Vanilla and DA-ABL? Comparing predicted rankings with rankings derived from actual execution outcomes would be particularly informative. This would materially strengthen the evidence that boundary learning improves the evaluator’s ability to select better actions.

4 comments

Thank you for recognizing the critical-phase design and the real-world gains. Below we add the direct test of the Dream Evaluator you requested (W1 & Q2): at the same 72% failure precision, boundary learning raises the evaluator's failure recall from 55% to 69% and its within-stage pairwise ranking accuracy from 66% to 76%.

W1 & Q2. As you suggested, we directly test the deployed evaluator's prediction and ranking quality by comparing DA-Vanilla and DA-ABL on a common held-out set of real executed chunks with known outcomes (added to Section 5 and Appendix C.3.4). Unlike Table 1's Gemini judge, this test uses the deployed world and value models.

Two existing comparisons bear on ranking across candidate actions, since each changes only how candidates are selected. With the same SDE Action Proposer, the evaluator raises success from 45.0% under random selection to 72.5% (Table 2; Appendix F, CMH p=1.9×10−7p=1.9\times10^{-7}): its top-ranked candidates succeed far more often than random ones. DA-Vanilla and DA-ABL share the frozen base policy, candidate settings and teleoperation-trained trigger; ABL fine-tunes only the Dream Evaluator, on deployment rollouts with successful, boundary and failed trajectories (Section 4.4). DA-ABL is higher than DA-Vanilla on 3 real tasks and tied on 1, and higher on all 12 SimplerEnv tasks (Tables 9, 12; sign test added to Appendix F). Since ABL changes only the evaluator, this consistent gain comes from its selections; the test below measures the underlying consequence evaluation directly.

The set contains critical-phase chunks from the Table 2 evaluation trials, none used for training: 160 base-policy trials (78 successes) and 160 Random Select trials (72 successes). Both trial groups are independent of either evaluator's selections; Random Select executes SDE candidates sampled as in deployment. Labels are fixed before evaluator scoring: F is the first chunk Robometer flags as an irreversible failure in a failed trial (z=−1z=-1, Section 4.3; including the single-misstep failures called boundary cases in Section 4.4); B (boundary) is a negative-progress chunk in a successful trial; S is every other critical-phase chunk of a successful trial. The set has 138 F, 74 B and 168 S chunks. Each evaluator scores its own world model's dreams, as deployed.

Failures are flagged with the same sign threshold (v^<0\hat v<0, Appendix C.3.3), fixed independently of held-out outcomes. We report failure precision (over flagged F and S chunks), recall, false-alarm rate on S, and, for ranking quality, within-stage pairwise ranking accuracy: how often the evaluator's predicted order of two chunks of the same stage matches the order derived from their actual execution outcomes (S > B > F).

MethodFailure precisionFailure recallFalse-alarm rate on SWithin-stage pairwise ranking accuracy
DA-Vanilla72%55%18%66%
DA-ABL72%69%22%76%

Boundary learning thus raises the deployed evaluator's failure recall and ranking accuracy at unchanged precision. This is direct evidence for the proposed mechanism of improved failure awareness, beyond task-level success rates.

Q1. The threshold is 0: a chunk is flagged as a failure risk when its value score v^\hat v (predicted progress delta) is negative. It is diagnostic only; deployment always executes the top-ranked candidate, without this threshold (Algorithm 1, lines 13–14). This sign cutoff follows from the definition of v^\hat v, so it uses no data, involves no tuning and was fixed independently of the test outcomes (added to Appendix C.3.3).

Q2. Answered jointly with W1 above (W1 & Q2), which reports failure precision, recall and ranking accuracy for DA-Vanilla and DA-ABL on a common held-out set of successful, boundary and failed chunks.

Dear Reviewer kQtZ,

As the discussion period ends in just over one day (Nov 18, AoE), we would appreciate it if you could review our rebuttal and let us know whether your concerns regarding the direct evaluation of the deployed Dream Evaluator have been addressed.

In summary, we added the requested direct evaluation on a common held-out set of 380 real-world executed chunks (138 failure, 74 boundary, 168 success) using our deployed world and value models (added to Section 5 and Appendix C.3.4):

  1. Improved Failure Recall & Ranking via ABL: At the exact same 72% failure precision, Autonomous Boundary Learning (DA-ABL) raises the deployed evaluator's failure recall from 55% to 69% and its within-stage pairwise ranking accuracy from 66% to 76% over DA-Vanilla.

  2. Threshold Clarification: We also clarified in Appendix C.3.3 that the diagnostic cutoff (ΔV<0\Delta V < 0) is fixed directly from the value definition without any tuning on test outcomes, while deployment always executes the top-ranked candidate.

We are happy to provide any further clarification if needed. Thank you very much for your time and helpful suggestions.

Best regards,

The Authors

Thank you for the detailed response and manuscript revisions.

First, the clarification in Appendix C.3.3 confirms that the zero sign threshold on predicted progress delta follows directly from its definition, involves no tuning, and was fixed independently of test outcomes.

Second, the new evaluation on held-out executed chunks in Section 5 and Appendix C.3.4, alongside the no-video critic ablation, directly validates the deployed Dream Evaluator. Showing that DA-ABL improves failure recall and within-stage pairwise ranking accuracy over DA-Vanilla at matched precision resolves my main concern about consequence evaluation.

With my concerns resolved, I will maintain my positive rating of 3 (Weak Accept).

Thank you for your prompt reply and continued support! We are glad that our revisions addressed your concerns, and we deeply appreciate your constructive feedback in helping improve the paper.

Reviewer_LGCuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DreamAvoid, a test-time framework that helps VLA policies avoid failures in fine-grained manipulation. The authors observe that such policies often behave reasonably over most of a trajectory but fail during brief, low-tolerance critical phases, such as the 1-2 s before decisive contact. In these phases a small action error can escalate into an irrecoverable failure. They attribute this partly to training on successful demonstrations only, which gives the policy little sense of what failure looks like. DreamAvoid therefore keeps the base policy frozen and runs it unchanged during routine execution, spending extra computation only when a critical phase is detected.

Critical strengths
  • Compelling problem setting. The observation that failures concentrate in brief critical phases is valid, and the design of spending test-time compute only during these phases is practical.
  • Consistent gain. The proposed method outperforms the base policy and GPC-RANK across all four tasks. The gains are also significant under task-stratified CMH tests.
  • Clear writing. The paper is written intuitively and is easy to understand.
Critical weaknesses
  • Lack of ablation for the core claim. The value model Vω(ot,ut,y^t)V_\omega(o_t, u_t, \hat y_t) already takes utu_t as input, yet there is no comparison with Vω(ot,ut)V_\omega(o_t, u_t) trained on the same data, that is, a critic without video input. This raises the question of whether dreaming is truly necessary.
  • Lack of ablation on value model design components. Separate ablations are needed for real-dream joint training, oversampling, and the ranking loss.
  • Concern about limited generality. Under the proposed method, each task requires its own stage definitions, trigger head, value model, σ\sigma, world model post-training, and ABL data collection. This raises concerns about extending the method to generalize across tasks.
  • Missing related work. The proposed method samples candidate action chunks from a generative policy, predicts the corresponding future observations with an action-conditioned world model, and then scores the candidates to select one. Such approaches have been discussed in prior work [1, 2], and the authors should cite and compare with these works. In particular, given this prior work, the claim in Appendix A that existing verifier or MPC works predominantly rely on current-state scoring or pixel-level goal alignment is inaccurate.

[1] From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment (Wu et al., RSS 2025)

[2] Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance (Nakamoto et al., CoRL 2024)

Questions for authors

See above.

4 comments

Thank you for recognizing the critical-phase setting, the consistent gains over the base policy and GPC-RANK, and the clear writing. The revision compares our critic with a no-video critic (W1), ablates each value-model component separately (W2), lists the shared and task-specific settings (W3) and discusses both works you suggested (W4).

W1. Dreamed video improves consequence evaluation for both a VLM judge (Table 1) and our deployed critic (below). On the same 40 episodes, a prompted VLM given current and past frames achieves 57.5%/67.5% accuracy with joint/EEF action text and 80.0% with dreamed video. The dream adds physical priors from large-scale video pretraining (Section 4.3) and makes small action differences visible, such as the plug hitting the socket edge (Section 3).

As you suggested, we trained a no-video critic Vω(ot,ut)V_\omega(o_t,u_t) that drops only the dreamed-video input y^t\hat{y}_t and keeps the training data, targets, oversampling and loss fixed. Both critics score the critical-phase chunks of the 320 base-policy and Random Select trials in Table 2, none of which were used for training. We compare the two by within-stage pairwise ranking accuracy: how often a critic orders two executed chunks of the same stage as their outcomes do (success > boundary > failure, where a boundary chunk is a setback in a trial that still succeeds). Dreamed video improves accuracy on every pair type (added to Section 5 and Appendix C.3.4):

CriticSuccess–failure pairsBoundary–failure pairsSuccess–boundary pairs
Ours (dreamed video)83%77%61%
No-video critic67%63%53%

The gains are largest on pairs with a failure, where the dream reveals consequences that the current state and action alone do not.

W2. We ablated each of the three components separately, comparing the full model with real-only and dream-only variants (real-dream joint training), a uniform-sampling variant (priority oversampling) and a λrank=0\lambda_{rank}=0 variant (ranking loss). All else, including data, world model and training setup, is unchanged. On the same chunks as in W1, we report each model's pairwise ranking accuracy, with dreamed video as input (added to Section 5 and Appendix C.3.5):

ModelSuccess–failureBoundary–failure
Full model83%77%
Real-only73%65%
Dream-only79%74%
Uniform sampling75%68%
λrank=0\lambda_{rank}=080%72%

Every ablation lowers both accuracies. Training without dreams costs the most: the real-only critic reaches only 65% on boundary–failure pairs, close to the no-video critic in W1 (63%). Uniform sampling costs 8–9 points, and dropping real futures or the ranking loss costs 3–5 points.

W3. Task-specific adaptation reuses the base policy's demonstrations (π0.5\pi_{0.5}: 100 per task), with model-generated onset and progress labels. Real tasks share the trigger design, β=5\beta=5, γ=0.5\gamma=0.5, Nc=8N_c=8, H=50H=50, the world-model post-training recipe, and value targets, loss and oversampling. As with the base policy's own per-task post-training, stage onsets, trigger/value weights and σ=0.05\sigma=0.05–0.10.1 are set per task.

ABL adds 100 autonomous rollouts per task without new demonstrations. DA-Vanilla, trained on demonstrations alone, already reaches 66.9%, versus 48.8% for the base policy and 54.4% for GPC-RANK.

SimplerEnv supports the recipe's broader applicability: with GR00T N1.6 and two robots, all 12 tasks share σ=0.3\sigma=0.3, Nc=4N_c=4 and H=16H=16, and DA-ABL beats the base policy on every task (Bridge: 59.9% → 63.6%; Fractal: 76.4% → 80.7%; CMH p=0.012/0.010p=0.012/0.010). The revised Appendix C lists the shared and task-specific settings.

W4. Thank you for these references. The revision cites and discusses both in Section 2 and Appendix A, and revises the summary you flagged: verifier and MPC methods include current state–action scoring (RoboMonkey, V-GPS) and predicted-consequence scoring (Visual MPC, GPC, FOREWARN). FOREWARN uses a VLM to reason over latent world-model predictions.

Our main baseline, GPC-RANK, is a sample–predict–select method, and DreamAvoid outperforms it on all four real tasks (mean 72.5% vs 54.4%, CMH p=5.3×10−4p=5.3\times10^{-4}). Relative to V-GPS and FOREWARN, DreamAvoid combines (i) a learned critical-phase trigger that invokes dreaming at most once per critical phase; (ii) pixel-space dreams from a pretrained video model and a critic trained on real and dreamed futures; and (iii) ABL adaptation of both models using boundary data from the system's own critical-phase candidates.

Table 1 compares both kinds of scoring with the same VLM judge: 57.5%/67.5% accuracy with action text vs 80.0% with dreamed futures, in the spirit of FOREWARN. W1 makes the same comparison for our critic: against a no-video critic that, like V-GPS, scores only the current state and action, dreamed video raises success–failure ranking accuracy from 67% to 83%.

Dear authors,

I have read the authors' response and the comments from the other reviewers. I appreciate the authors' efforts and have no further questions on W2 and W4. However, I still remain uncertain whether the necessity of dreaming has been shown beyond offline ranking of chunks from different trials. The reported per-pair numbers imply a pooled ranking accuracy of roughly 62–63% for no-video critic, only a few points below that of DA-Vanilla’s evaluator (66%), and DA-Vanilla already reaches 66.9% task success in closed loop. What closed-loop success rate does the same pipeline achieve with no-video critic, as in the other ablations of Table 2? Also, does the video critic select better candidates from the same state?

Thank you for the follow-up.

Closed loop. With the no-video critic, the same pipeline reaches 62.5% average success (100/160), 10 percentage points below DA-ABL's 72.5%. As in the other Table 2 ablations, one component changes: the critic that picks among the candidates is the no-video critic Vω(ot,ut)V_\omega(o_t,u_t) from our earlier reply, which scores candidates without dreamed video and is trained on DA-ABL's data. The trigger, the SDE sampling of eight candidates per intervention and the protocol (40 trials per task) are DA-ABL's (added to Table 2, Section 5 and Appendix F). Success rates (%), with DA-Vanilla from Table 12:

MethodCupChargerCapScrewAvg.
π0.5\pi_{0.5}62.547.555.030.048.8
Repeated ODE67.555.070.037.557.5
Random Select65.035.055.025.045.0
No-video critic80.055.075.040.062.5
DA-Vanilla82.560.080.045.066.9
DA-ABL90.067.580.052.572.5

Same state. Yes. When only the video critic ranks the executed candidate in its top four, the trial succeeds 62.2% of the time (23/37), versus 32.6% (14/43) when only the no-video critic does. We further compare the two critics' candidate preferences using the 160 Random Select trials in Table 2, none of which were used to train either critic. Random Select uses DA-ABL's trigger and SDE sampler and executes one of the eight candidates uniformly at random. For trials with complete candidate records, we use the first intervention and rank the same candidate pool offline with both critics; the video critic scores dreams generated by DA-ABL's world model.

The table directly compares two groups: trials where only the video critic ranks the executed candidate in its top four, and trials where only the no-video critic does. Because each critic selects four candidates, their two exclusive sets have equal size. Uniform random execution therefore makes the two groups equally likely in every state.

Executed candidate's groupTrialsSuccess rate
In the video critic's top four only3762.2% (23/37)
In the no-video critic's top four only4332.6% (14/43)

We hope the closed-loop and same-state results resolve your remaining concern about the necessity of dreaming, and we would be grateful if you would consider raising your score.

Dear Reviewer LGCu,

As the discussion period ends in just over one day (Nov 18, AoE), we would sincerely appreciate it if you could review our follow-up response and let us know whether your remaining concerns regarding the necessity of dreaming have been addressed.

In brief, following your follow-up questions, we added both closed-loop and same-state candidate selection experiments comparing our video-based Dream Evaluator against the no-video critic (added to Table 2, Section 5, and Appendix F):

  1. Closed-Loop Real-World Success: Using the exact same pipeline and training data, the no-video critic achieves 62.5% average success (100/160) across the four tasks, which is 10.0 percentage points lower than DA-ABL (72.5%) and also below DA-Vanilla (66.9%).

  2. Same-State Candidate Selection: On the same candidate pools sampled from identical states (160 Random Select trials), when only the video critic ranks the executed action in its top four, the trial succeeds 62.2% (23/37) of the time, compared to 32.6% (14/43) when only the no-video critic does.

  3. Component & Offline Ablations: We also added full offline ranking ablations isolating the no-video critic (67% vs. 83% success–failure ranking accuracy), real-only/dream-only training, uniform sampling, and ranking loss, alongside discussions of V-GPS and FOREWARN.

We hope these new closed-loop and same-state results resolve your concerns, and we would be very grateful if you would consider raising your score. We remain happy to provide any further clarification if needed.

Thank you for your time and constructive feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.