acceptodds
Withdrawn from ICLR 2027

Vision-Language Signals in Constrained RL: Safety Gains Without Anticipation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safe reinforcement learning seeks policies that maximise task performance while satisfying safety constraints. In driving benchmarks, however, collision costs typically appear only at the time of collision, providing no advance warning of an approaching hazard. Frozen vision–language models can provide dense semantic feedback, yet it remains unclear whether their scores anticipate collisions and which component drives an observed safety improvement. Episodic cost can also favour policies that make little task progress. To address these gaps, we propose VLM- Safe-RL, a framework that integrates frozen CLIP signals into PPO-Lagrangian through reward shaping and an augmented multiplier update. On MetaDrive Hard, which combines the densest traffic with the largest map, the catastrophe rate falls from 31.6% to 19.4%. FormulaOne-L2 analysis finds no evidence that the CLIP signals anticipate collisions and shows that the VLM term has a negligible effect on the Lagrange multiplier. These findings show a conditional reduction in observed catastrophe rate without evidence of collision anticipation.

Primary area:
reinforcement learning
Keywords:
safe reinforcement learning, constrained MDPs, Lagrangian methods, vision–language models, reward shaping, autonomous driving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1YosOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether frozen vision–language model signals can provide useful safety feedback in constrained reinforcement learning, and whether those signals anticipate collisions. It proposes VLM-Safe-RL, which integrates frozen CLIP scores into PPO-Lagrangian. Positive and negative prompt groups are kept separate: a confidence-gated positive score shapes the task reward, while the epoch-mean negative score is added to the Lagrange multiplier update. The environment cost itself is unchanged.

The main contributions are: (1) a framework that combines frozen CLIP signals with PPO-Lagrangian through reward shaping and an augmented multiplier update; (2) a separated positive/negative CLIP design in which reward shaping and dual updates use different signals; and (3) a diagnostic study that tests whether the VLM signal anticipates collisions and which component drives any safety change.

Validation spans MetaDrive Easy/Medium/Hard and Bullet Safety-Gym, with FormulaOne-L2 used for component and lead-time analysis. On MetaDrive Hard, gated CLIP shaping reduces catastrophe rate from 31.6% to 19.4%, but performance worsens on Easy and shows no detectable change on Medium or Bullet. On FormulaOne-L2, no CLIP score rises before collision, the VLM term has negligible influence on the Lagrange multiplier, and lower cost largely tracks less distance travelled; no policy completes the route. The paper concludes that gated CLIP shaping gives a conditional reduction in observed catastrophe rate without evidence of collision anticipation.

Critical strengths

Unusually rigorous and transparent diagnostic analysis of whether the VLM signal anticipates collisions. The paper does not stop at reporting a safety improvement. It directly tests lead time on FormulaOne-L2 and finds that no CLIP score rises before contact, that the VLM term has a negligible effect on the Lagrange multiplier, and that lower cost largely reflects less distance travelled on an unsolved route. This is consequential because it prevents overclaiming and clearly separates a conditional reduction in observed catastrophe rate from genuine collision anticipation.

A clean and interpretable separation of positive and negative CLIP signals into different optimization paths. The positive score shapes the task reward, while the epoch-mean negative score enters the multiplier update. This design is simple, modular, and easy to analyze, and the paper’s component diagnostics provide a useful template for evaluating VLM-based safety signals in constrained RL. Even though the reported gains are mixed, this separation is a meaningful methodological contribution that supports clearer future work on learned safety feedback.

Critical weaknesses

The central safety claim is conditional, non-general, and mechanistically unresolved. The paper reports a large catastrophe-rate reduction on MetaDrive Hard (31.6% to 19.4%), but performance worsens on Easy (14.0% to 32.7%), shows no detectable change on Medium or Bullet, and the authors themselves state that the Hard experiment does not isolate confidence gating from reward shaping. A less active policy may explain part of the reduction, and the multiplier term has negligible measured influence. This is consequential because the paper’s primary contribution is a safety improvement, yet the evidence supports only a narrow, setting-specific correlation without identifying which component causes it. A method whose benefit appears on one difficulty setting, reverses on another, and cannot be attributed to its proposed mechanism does not provide a reliable basis for acceptance.

The core premise—that frozen VLM signals can provide anticipatory safety feedback—is contradicted by the paper’s own diagnostics. The FormulaOne-L2 lead-time analysis finds no CLIP score that rises before collision; all pooled AUCs lie below chance and below the step-index control. The VLM term never changes the sign of the multiplier update in 8,572 logged epochs, and lower cost on the unsolved route largely tracks reduced distance travelled. This is consequential because the paper’s motivation and method are built on the idea that CLIP can supply dense, advance warning of hazards. Without evidence of anticipation, the augmented multiplier path is effectively inert, and the observed Hard reduction cannot be attributed to the proposed semantic safety signal. The paper therefore does not establish its central scientific claim.

Questions for authors

Is the MetaDrive Hard improvement caused by the proposed mechanism or by reduced task activity? The paper reports that 28.6% of the VLM arm’s episodes are cost-free with return below 20, compared with 5.2% for PPOLag, and that lower cost may partly reflect less active policies. Could you provide a matched-activity or matched-progress comparison? Does the Hard reduction persist when controlling for path length, net progress, or episode return?

Can you isolate confidence gating from reward shaping on MetaDrive Hard? The paper states that the Hard comparison does not separate confidence gating from reward shaping. Could you report matched runs for ungated shaped reward, gating without shaping, and the full method? Without this, the source of the Hard reduction remains unresolved.

What explains the adverse Easy result? Catastrophe rate increases from 14.0% to 32.7% on Easy, and all three seeds favour PPOLag. Can you identify whether this comes from reward shaping, gate behaviour, prompt selection, or optimization variance? Would more seeds or a controlled density study change the interpretation?

Does the VLM signal anticipate collisions on MetaDrive, where the improvement is observed? FormulaOne-L2 shows no CLIP score rising before contact. Could you run the same lead-time analysis on MetaDrive Hard? If no anticipation appears there either, what mechanism remains for the observed improvement?

Does the method reduce to reward shaping only? The VLM term never reverses the sign of the multiplier update and changes the multiplier by at most 3.6×10⁻⁴. Could you report results with η₂=0 while keeping the same reward shaping? If the multiplier path is inert, the dual augmentation is not contributing to the reported effect.

What is the role of the confidence gate when it is nearly non-selective? On MetaDrive Hard, median gate values are reported as 0.90–0.98. Could you report the gate distribution on Hard and its relationship to collision risk? What changes if the gate is disabled while keeping the same shaped reward?

Are the comparisons matched enough to support the safety claim? The matched PPOLag control uses only two seeds, while the unmatched baseline differs in update iterations, initial multiplier, and seeds. Could you provide a fully matched control with the same seeds, update schedule, λ₀, and evaluation episodes? This directly affects whether the Hard result is attributable to the method.

Do the results generalize beyond in-distribution MetaDrive scenarios? MetaDrive evaluation seeds map into the training pool, and Bullet episodes cannot be seed-matched. Could you report held-out scenario evaluation and additional seeds? This would materially affect the strength of the generalization claim.

Reviewer_7inwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the practical utility of signals from frozen vision-language models (VLMs) in constrained reinforcement learning. We propose VLM-Safe-RL, which integrates CLIP signals into the PPO-Lagrangian framework: similarity scores from positive prompts are used for reward shaping via a confidence-gated mechanism, while the average similarity from negative prompts informs the update of the Lagrangian multiplier—all while keeping the environment’s safety cost definition unchanged. Beyond evaluating final safety metrics, the study analyzes whether the visual signals provide collision warnings, whether the auxiliary terms effectively influence the Lagrangian multiplier, and whether lower safety costs are merely a byproduct of reduced agent movement. On the MetaDrive Hard benchmark, our method reduced the catastrophe rate from 31.6% to 19.4%; however, performance deteriorated on the Easy setting, and no clear benefits were observed on the Medium or Bullet settings. Diagnostic analysis on the FormulaOne-L2 environment reveals that the CLIP signals tested failed to provide reliable, proactive collision warnings, and the negative signals had a negligible impact on the Lagrangian multiplier; furthermore, none of the policies successfully completed the designated routes. Consequently, the primary contribution of this work lies in uncovering conditional performance gains and fundamental mechanistic limitations, rather than proposing a universally effective solution for safety-constrained control.

Critical strengths

The paper clearly distinguishes between the information content of a signal, the pathway through which it enters the learning algorithm, and the resulting strategic behavior—thereby avoiding the pitfall of interpreting a decline in collision rates as direct evidence that the model possesses an inherent capacity for risk anticipation.

The authors report performance gains in "Hard" scenarios, performance degradation in "Easy" scenarios, and mixed results in other environments. They also disclose limitations—such as unmatched baselines, incomplete training, and constraints on scenario distribution—which help define the boundaries of their conclusions.

Through an analysis of effective budgets, Adam update replays, and ablation studies, the authors demonstrate that the negative VLM (Vision-Language Model) term has virtually no direct impact on the multipliers. This distinction is crucial for differentiating between the mere inclusion of a "safety signal" and the signal actually exerting a functional effect.

The "FormulaOne" analysis reveals that low costs may stem from reduced movement or less exposure to risk, rather than superior driving proficiency. Furthermore, the paper distinguishes between training returns (which incorporate shaped rewards) and raw environmental returns, ensuring that comparisons remain fair.

The appendix provides detailed data, including results for individual random seeds, variations across training runs, confidence intervals, permutation tests, prompts, and specific update parameters. Notably, the "Hard" setting utilizes ten distinct training seeds, a methodological rigor that is more compelling than what is typically found in smaller-scale ablation studies.

Critical weaknesses

In the MetaDrive "Hard" scenario, comparisons involve simultaneous adjustments to reward shaping, confidence gating, and multiplier terms, while the component-wise analysis is primarily conducted in the (unsolved) FormulaOne environment. Consequently, the paper does not clarify whether the performance gains observed in the "Hard" setting stem from semantic information, generic reward bias, or a shift toward more conservative driving behavior.

In the "Hard" setting, 28.6% of episodes under the VLM-based approach result in zero cost but also yield returns below 20, compared to only 5.2% for the baseline. This suggests that the observed reduction in "disaster" rates may, in part, simply reflect a decrease in driving activity. While environmental returns have improved, further validation—using metrics such as route completion rate, distance traveled, and risk per unit distance—is required to confirm genuine safety gains.

The current VLM-based term is significantly smaller than the difference between the environmental cost and the budget, meaning its impact is predictably near zero. While this indicates that the current implementation has limited effect, it does not preclude the possibility that VLM-based safety feedback—if properly scaled, independently constrained, or temporally modeled—could be effective.

The conclusions regarding collision warnings have a limited scope. The analysis of these warnings focuses specifically on the FormulaOne-L2 environment, particular prompt formulations, the CLIP model, and policy distributions from incomplete tasks; no such analysis was performed for the MetaDrive "Hard" scenario. Furthermore, the appendix notes correlations even when "reverse" prompt scoring is used; therefore, it is essential to distinguish between "warnings that fail to reach a threshold in the expected direction" and "signals that contain no information whatsoever about future risks."

The gating mechanism relies on the absolute difference between the similarity scores of positive and negative prompts; consequently, even when the model is strongly biased toward a negative prompt, a significant positive reward component may still be retained. This mechanism measures the degree of separation between similarity scores, but it does not guarantee that the direction of the reward aligns with safety. Moreover, in the "Hard" setting, the gate is active almost continuously, rendering its necessity unclear.

The MetaDrive test scenarios are drawn from the same pool as the training data, offering no insight into performance on unseen scenarios. Additionally, because map scale and traffic density vary simultaneously, it is impossible to isolate the specific conditions under which the performance gains emerge. The variance observed across independent training runs is of a similar magnitude to the reported gains, and most component-wise comparisons rely on only three to five random seeds, which limits the precision of the conclusions.

Questions for authors

Could you provide an ablation study on the components using the "Hard" setting in MetaDrive? Specifically, please compare the original PPOLag, a version without gated semantic rewards, one with gated semantic rewards, and a version with the multiplier term removed. Additionally, include control experiments—such as using a constant reward (matched to the mean) or scrambling the image signals—for comparison.

Does the reduction in risk on the "Hard" setting come at the cost of task progress? Could you report metrics such as route completion rate, the proportion of time spent stopped, total distance traveled, and collision rate per unit distance? Furthermore, please compare risk levels while controlling for similar task progress.

Does the multiplier term remain ineffective even after its scale is properly matched? Is it possible—while maintaining stable optimization—to ensure that the VLM-based term and the environmental cost differential are of comparable magnitude? Please also disentangle the individual effects of an overly small scale, Adam normalization, and temporal aggregation.

Could the early-warning analysis be extended to environments where the primary objective is reward maximization? Please evaluate the "time-to-collision" (TTC) or similar early-warning metrics on both the "Hard" and "Easy" MetaDrive settings. Additionally, calibrate the inverse scores using separate training and test sets, while controlling for confounding factors such as vehicle speed, temporal positioning, and road geometry.

Does the absolute-value gating mechanism inadvertently reward states that are clearly dangerous? Could you compare the current gating approach against alternatives—such as one-sided safety gating or a signed difference in similarity—and report the distribution of rewards across safe, dangerous, and ambiguous states?

Reviewer_vuXkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VLM-Safe-RL studies whether frozen vision-language models (VLMs) can supply useful safety signals for constrained RL. It augments PPO-Lagrangian in two ways: (1) a confidence-gated positive-prompt CLIP score is added to the environment reward; (2) the epoch-mean negative-prompt CLIP score is added to the Lagrange-multiplier update. Evaluated on MetaDrive, Bullet Safety-Gym, and Safety-Gymnasium FormulaOne, the headline result is a catastrophe-rate drop on MetaDrive Hard, but the gain does not transfer to Medium/Bullet, Easy deteriorates, and FormulaOne shows CLIP signals do not anticipate collisions and the VLM term negligibly affects the multiplier.

Critical strengths

1/ Clear, well-motivated research problem: whether frozen VLM signals anticipate collisions and retain timing within a constrained learner. 2/ Sound theory presentation. 3/ The replay analysis provides convincing evidence that the implemented multiplier behaves as described.

Critical weaknesses

1/ In Section 3.1, Page 3 ​, it shows “M = (S, A, P, r, c, d, γ)” ​, the CMDP tuple symbols are never defined, making the rest of the formulation hard to follow.

2/ In “Eq. (3), Section 3.3, Page 4” ​, it shows “κt = |2σ(s(mt − c)) − 1|” ​, σ is not defined (is it a hyperparameter?) and has no effective specification.

3/ In “Eq. (3), Section 3.3, Page 4” ​, it shows “the confidence gate κ_t = |2σ(s(m_t − c)) − 1|” ​, the absolute value makes the gate symmetric around c — both strongly positive and strongly negative margins yield κ_t≈1, so a danger-frame can still trigger a positive reward bonus. It is a nondirectional magnitude, not a safety-monotone gate.

4/ In “Table 1 and Section 5, Page 6” ​, it shows “Hard 31.6%→19.4%, Medium 26.0%→28.0%, Bullet 13.2%→12.7%, Easy 14.0%→32.7%” ​, the method helps in only one of four settings and hurts Easy; since traffic density and map size vary together, the driving environmental factor cannot be identified.

Questions for authors

see the above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.