- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigate whether CLIP can detect hazards in simulated environments and find that its safety score mainly reflects scene similarity, which is not particularly surprising given CLIPs known limitations.
Critical strengths
I do not consider this paper has any strengths to compensate for the limited novelty, and methodological concerns discussed bellow.
Critical weaknesses
My main concern is that most findings are already well known. It is not surprising that CLIP is sensitive to prompts, has problems with negation etc. These limitations have been studied before, and the authors even cite several papers. Repeating known limitations in a safe-RL environment is not, by itself, a sufficient contribution for ICLR.
Besides, the Results section is extremely complicated and hard to read and follow which makes the claimed contribution impossible to evaluate. There are many different configurations, statistical comparisons, and ablations, but the actual message is just unclear. I bet an LLM reviewer would like it, but for a human it is, in this form, really difficult to understand which observations are fundamental and which are specific to the experimental setup.
Following on that, I also have some serious doubts about the methodology. The main results rely on an old CLIP model (released in 2021) evaluated on synthetic data. CLIP was not specifically trained on such data, and its limitations under domain shift are already well studied. Therefore, I am not sure why the authors find its behavior so surprising. Without properly investigating the effect of domain shift, how can we know whether the observed results are caused by the proposed scene-template mechanism or simply by using a model that is not well suited for this domain? The limited experiments with other models do not really answer this question.
Last but not least, I find it quite disrespectful to the whole community that, while we are on one hand discouraged from using LLMs to assist with reviews for example, the authors submit a manuscript that reads like it was written almost entirely by Claude. The authors acknowledge AI-assisted drafting, but it is not clear how far this assistance actually went. Was AI also used to implement, run, and analyze the experiments? If so, were the results independently verified? Given how difficult the Results section is to follow, I find this particularly concerning. I have nothing against using LLMs as assistants, but ... reviewers should not have to spend hours trying to understand and verify AI slop.
Questions for authors
- Q1. What is the main novel contribution beyond already known CLIP limitations?
- Q2. Why do the main experiments rely on CLIP ViT-B/32 from 2021?
- Q3. How do you distinguish the proposed scene-template mechanism from domain shift caused by synthetic data?
- Q4. Did you verify that CLIP can reliably recognize relevant objects and hazards in these environments?
- Q5. Why are the main mechanism experiments not repeated with modern generative VLMs?
- Q6. How representative are the results for current VLM-based safety methods?
- Q7. Why do the results differ so much across camera views and environments?
- Q8. Why was the closed-loop evaluation not repeated on genuinely held-out scenarios?
- Q9. What conclusions can be drawn from the constant-confidence experiment when the difference from the deployed model is statistically unresolved?
- Q10. What is the practical significance of these findings for developing or evaluating modern VLM-based safety systems?