acceptodds
Withdrawn from ICLR 2027

When to Warn, Not What to Do: Calibrated Language Warnings for Frozen World-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World-action models (WAMs) combine future visual prediction and action generation for language-conditioned robot control. Collision safety requires anticipating unsafe contact and guiding responses that preserve task progress, yet collision records do not specify corrective actions. Learning recovery or language-feedback policies requires additional supervision or interaction. We propose risk-triggered language steering, which learns when to warn and guides a frozen WAM with a fixed prompt without corrective-action supervision. A classifier trained on collision-derived -step warning labels reads risk from predicted-video and action hidden states. Unsafe-class split conformal calibration sets a threshold targeting unsafe recall under the unmodified policy. Alerts trigger regeneration of the pending chunk with the warning appended to the task instruction. Experiments with LingBot-VA on seen tasks across four SafeLIBERO simulation suites show 89.3–99.0% AUROC and mean unsafe recall meeting nominal targets. Online, mean collision avoidance rate rises from 9.4% to 18.8%, versus 3.8% with the same warning always active. Mean task success is 30.0%, versus 29.4% without intervention.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World-action model, Robot Safety, Conformal Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_k7TSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work focuses on the problem of collision risk mitigation in world-action models. The authors propose a method for monitoring the risk level in the world-action model and then steering the language instruction with a fixed warning text if the risk is above a threshold. The paper presents empirical evaluation on the LingBot-VA and the SafeLIBERO dataset suite, demonstrating reduced collision rates.

Critical strengths

This paper considers an important problem of preventing world-action models from executing unsafe actions. The authors propose an interesting method for steering the language inputs to include a fixed warning prompt to help reduce the collision rate.

Critical weaknesses
  • As the authors write in the limitations, “[e]valuation uses LingBot-VA on seen SafeLIBERO tasks in simulation with one hand-written warning.”
    • This work does not explore the language steering (with appended warnings) in sufficient detail. This makes the empirical findings quite weak and insufficient.
    • It is unclear how well the method generalizes to unseen tasks.
  • The paper only focuses on a single mode of safety, namely collisions.
    • There are many other forms of unsafe behavior that are not captured in the empirical study. The warning, as well, only focuses on collisions with obstacles, thus further limiting the impact of this work.
  • Some figures/tables are not very helpful or clear.
    • In Figure 3, can you explain what the table exactly means? I understand that there is an alpha chosen for each split, but I’m not sure what value the table brings other than showing that each unsafe recall matches the target (1-alpha), which was by design?
  • Some baselines are not properly explained:
    • What exactly is the rotation variance baseline?
    • What exactly is the time index baseline? How is it implemented? Is it just a single scalar dimension?
    • How was the N = 4 chosen for Best-of-N? Why not higher N?
Questions for authors
  • How does the method perform on other world-action models (such as Cosmos policy) and on datasets beyond SafeLIBERO (such as the vanilla LIBERO, LIBERO-plus, LIBERO-pro, etc)?
  • Can the method generalize to other modes/risks of safety concerns?
  • Can you show that the risk head has lower risk prediction with the steered language than without? This would isolate the effects of steering in appropriately reducing collision rate.
  • Can you address the concerns about the clarity of figures/tables above?
  • Can you provide answers to the following on the baselines? (Copied from the Weaknesses)
    • What exactly is the rotation variance baseline?
    • What exactly is the time index baseline? How is it implemented? Is it just a single scalar dimension?
    • How was the N = 4 chosen for Best-of-N? Why not higher N?
1 comment

Thank you for the detailed review.

Backbones and safety scope. Obstacle collisions are a central safety concern in robot manipulation and the specific hazard we study. The revision adds experiments with a second backbone alongside LingBot-VA.

Recall and false alarms. We clarify that 1-alpha sets the nominal recall target. Table 2 reports the recall measured on held-out unsafe chunks. Figure 3 shows how often the same thresholds flag safe chunks. An always-alarm rule, for example, achieves full recall and flags every safe chunk.

Baseline definitions. Rotation variance sums the sample variances of rotation coordinates over the chunk horizon across four draws. The time-index baseline uses the scalar t/T, the chunk index divided by the episode horizon. Best-of-N selects the lowest-risk candidate under a fixed budget of four.

Reviewer_KSh8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes risk-triggered language steering for frozen world-action models. A lightweight classifier predicts imminent collisions from video and action hidden states. An unsafe-class conformal threshold triggers a fixed warning and regeneration of the pending action chunk, without corrective-action demonstrations or backbone fine-tuning. On four SafeLIBERO suites, the method achieves 89.3%–99.0% AUROC and increases mean collision avoidance from 9.38% to 18.75%, with similar task success point estimates (29.38% versus 30.00%).

Critical strengths
  1. Practical problem formulation. Learning when to intervene from collision annotations, while relying on a frozen model to adjust its actions, avoids collecting corrective demonstrations or fine-tuning the backbone.
  2. Evaluation of both detection and closed-loop behavior. Feature comparisons support the presence of collision-risk information in the hidden states. The mean collision-avoidance gain is 9.4 percentage points, with a paired-bootstrap 95% confidence interval of [4.4, 14.4]. The poor performance of persistent warnings is also a useful finding.
Critical weaknesses
  1. Limited novelty. The method primarily combines existing ideas in risk detection, conformal calibration, and prompt-based intervention. Although the combination has practical value, its distinct technical contribution beyond prior work is not sufficiently clear.
  2. Limited validation across backbones. The risk-triggering and language-steering mechanism does not appear inherently tied to LingBot-VA, yet the full method is evaluated online only with this backbone. The appendix compares offline risk detection on OpenVLA and π0.5, but this does not establish that the complete intervention procedure works across architectures.
Questions for authors
  1. What is the main technical contribution beyond combining existing risk readouts, conformal calibration, and prompt-based recovery?
  2. Which components depend on LingBot-VA’s architecture or feature interfaces? Could the authors evaluate the complete intervention procedure on another compatible backbone to establish its broader applicability?
1 comment

Thank you for the careful review.

Technical contribution. We respectfully disagree with the novelty assessment. Our contribution is to use collision timestamps to learn when a pending action chunk should be regenerated, before any of its actions execute. The risk readout uses joint video and action representations, while the frozen backbone generates the response to a fixed warning. This setup requires neither corrective demonstrations nor backbone fine-tuning.

Cross-backbone validation. The revision adds evaluation with another backbone across all four SafeLIBERO suites.

Reviewer_VGxgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a lightweight collision-risk classifier for predicted future states from WAMs, using a threshold to determine when a generated action chunk is dangerous and requires regeneration with a cautionary prompt. They utilize heuristics for regulated multiple interventions, and across four SafeLIBER suites, the method increases collision-free episode rates. The central contribution is a potentially useful separation between learning when intervention is needed and relying on the frozen policy to generate the corrective behavior.

Critical strengths
  1. A well-scoped supervision/problem setting. Collision timestamps are substantially cheaper to collect than corrective demonstrations. So leveraging these to predict collisions and activate an existing policy mode is a motivated idea.

  2. Offline strong results. The offline performance for collision detection offers strong evidence for the predictability potential for collision detection.

Critical weaknesses
  1. The results are entirely in simulation and additionally do not isolate the claimed benefits of warning timing. The method involves several modifications across predicting risky moments, regenerating plans (with underlying frozen WAM), and modifying the instructions. Additional baselines/ablations are required to isolate the contribution of each of these system decisions. The authors acknowledge these missing controls in the limitations; however, that is not a valid limitation and must be addressed to fairly evaluate the legitimacy of their proposal.

  2. The reported metrics are not accurate of the downstream behavior we care about. The harmonic mean of CAR and TSR is not the proportion of episodes that are both successful and collision-free. It is possible to improve CAR by making unsuccessful episodes safer while leaving successful executions involve collisions. I think it is better to report just the intersection of task success and safe episode.

  3. The safety improvement is motivated, however despite the method the collision rate remains very high. 81.25% of episodes still collide under the proposed method. For a paper that is intended to address this, this is a apparent gap that must be either carefully characterized or improved.

Questions for authors
  1. Does warning-conditioned regeneration outperform risk-triggered resampling without a warning?
  2. Does risk-aligned warning timing outperform matched random warning windows?
  3. Does the method improve successful, collision-free completion?
  4. How are temporally correlated samples handled during calibration and repartitioning?
  5. Where do the remaining collisions occur within the controller’s monitoring, warning, and cooldown phases?
1 comment

Thank you for the thoughtful review.

Safe task completion. The revision directly reports safe success rate (SSR) for both backbones, measured as the fraction of episodes that complete the task without a collision.

Residual collisions. The role of the warning may have been misunderstood. In the warning-only setting, even a correctly timed alert still requires the frozen backbone to generate a feasible corrective action. The high residual collision rate is a real limitation of the full system. Interpreting it requires distinguishing missed alerts from unsuccessful recovery after an alert.

Reviewer_fTQfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LingBot-VA generates video before actions, so both representations are available before the pending chunk is committed. The paper uses this to learn only when to intervene: a lightweight classifier over the pooled video and action hidden states, supervised by collision timestamps and thresholded by unsafe-class split conformal calibration; on alert, the pending chunk is regenerated with a single fixed caution sentence appended to the instruction. The warning content is fixed, and only its timing is learned. The most informative result is the ablation: the same warning left active throughout an episode performs worse than no warning at all.

Critical strengths
  1. Deployment safety for frozen world-action models is an important problem, and the observation that collision records can supervise risk identification without specifying a corrective action is a real and well-identified gap. "Learn when, let the frozen model decide what" is a clean division of labour, and it is the right way to avoid learning recovery for which no supervision exists.
  2. Exploiting the video-before-action architecture is elegant. Exploiting the fact that both representations are available before the pending chunk is committed — rather than after — matches the architecture and costs no extra rollout. The ablation showing that video-only and action-only features are both worse than the joint representation supports the design.
  3. The presentation is complete and the limitations section is honest. Compact and well organized, with algorithm pseudocode, metric definitions, and the pipeline figure all in place; the limitations section is honest about both the post-intervention gap and the low absolute collision-avoidance rate.
Critical weaknesses
  1. The paper never places itself against published levels on the same benchmark, so the value of the problem cannot be judged. Published plug-in safety layers on this same benchmark reach collision-avoidance rates several times higher than what is reported here, while remaining training-free and cheap enough for high-frequency control. One of them is cited in the related work but neither compared against nor quantified. Without that context the reader cannot tell whether this is a complementary niche — a purely linguistic method that works where no filter can be inserted — or a strictly weaker approach to the same problem.
  2. The conformal guarantee does not cover the deployment regime, and its key assumption is conceded as unmet. he theorem bounds a quantity for a fresh inference from the unmodified policy, but once a sticky window and cooldown are applied, rollouts are no longer generated by that policy: exchangeability fails and the guarantee no longer applies — the standard covariate-shift failure of conformal prediction under intervention. The conclusion concedes the outcome half but not the coverage half: after the first alert the recall guarantee is effectively void, yet the same threshold continues to be used. A remark also states outright that the paper's own procedure does not establish the theorem's hypothesis, with no remedy.
  3. The closest existing work is cited but not run; the comparison set is only internal variants. A recent corrector — detecting deviation from observation history, discarding the pending chunk, and requerying the frozen policy with staged recovery prompts — is structurally almost identical to this intervention, differing only in trigger and prompt content; another uses calibrated-predictor gating, which is precisely this paper's calibration-as-gate. Neither is run, and all baselines that are run are internal variants, so the comparison set can only support "beats things constructed for comparison."
Questions for authors
  1. Where does this method sit relative to published plug-in safety layers that already report far higher collision-avoidance rates on the same benchmark? Will the paper state that comparison and position itself against it?
  2. The online gain rests on a small number of episodes. Do the authors agree that the claim should be scoped to the aggregate rather than made per suite?
  3. Does the always-on versus triggered contrast isolate when the warning is issued, or only the fact that it was triggered? Would you state it as a contrast rather than as a mechanism about timing?
  4. Given that rollouts after the first intervention are no longer generated by the unmodified policy, does the coverage guarantee extend past it? Would the authors restate the theorem to its actual scope?
1 comment

Thank you for the careful review. We will add AEGIS as a published plug-in safety baseline and focus our online claims on aggregate results.

Conformal scope. Our claim may have been misunderstood as extending to post-intervention rollouts. The bound applies only to unsafe inferences under the unmodified policy, assuming exchangeability of risk scores computed from the frozen backbone’s latent representations.

Warning timing. The triggered-versus-always-on result compares two intervention policies. Isolating the effect of timing requires a matched warning budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.