- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors build SafeMolBench, containing 3,702 samples covering 618 hazardous molecules and three settings: Hazardous-Harmful, Hazardous-Allowed, and Utility Replay. They then propose SafeMol, a lightweight safety-alignment method that uses LoRA for safer response behavior, MMD to align graph and text representations, and two classifiers to identify molecular hazardousness and harmful operational intent.
Critical strengths
- The paper is well written and easy to follow. It addresses molecular AI safety, a trending topic, and provides a clear benchmark-oriented formulation of the problem.
- The proposed defense is practical and lightweight. SafeMol substantially reduces attack success while requiring only parameter-efficient adaptation, making the method relatively easy to integrate into existing molecular multimodal models.
Critical weaknesses
- I strongly disagree with the authors' claim in the Introduction that safety alignment for molecular multimodal models "remains largely underexplored." The Related Work section itself partially contradicts this framing, as different modalities, harmful molecule names, molecular structures, and multimodal safety alignment have already been studied. The current work overlaps with several of these directions, and the novelty relative to prior work needs to be justified more clearly.
- The benchmark oversimplifies how chemical hazardousness works in practice and in the cited standards and regulatory references. Many molecules are not simply "dangerous" in isolation, since risk depends on concentration, formulation, handling protocol, transportation condition, and application. For example, nitroglycerin is labeled as an explosive in the provided benchmark, while it also has legitimate medical use. At the same time, procedural questions involving such molecules are generally treated as forbidden, which may oversimplify real dual-use chemistry.
- Relatively outdated and limited models are tested. Many recent models have substantially stronger safety alignment, so I suggest evaluating more recent and representative models. The reported ASR conclusions may be different under newer systems.
Questions for authors
The benchmark attempts to distinguish hazardousness from harmful intent, but the limited and duplicated prompt design, repeated across train and test, raises concerns about generalization to unseen user formulations and truly unseen utility samples. The benchmark appears to simplify the chemistry problem into something close to a binary classification: labeled hazardous molecule + labeled harmful question = harmful request. How well does the proposed method generalize to realistic dual-use scenarios, where a hazardous or regulated molecule may still appear in legitimate procedural, medical, or research contexts?
11 comments
Weakness 2
W2: Consideration of legitimate dual-use scenarios.
We would like to address these concerns from three perspectives.
-
Avoiding Indiscriminate Over-Refusal. To avoid indiscriminate over-refusal of all requests involving hazardous molecules, SafeMolBench includes both H and A samples, enabling the model to distinguish between harmful and legitimate requests involving the same hazardous molecules. For example, an A-sample instruction is "Determine the LogP value of this molecule."
-
Difficulty in Distinguishing Legitimate and Harmful Synthesis Requests. Legitimate and potentially harmful synthesis requests involving hazardous molecules do not fundamentally differ in their input or output format. Whether the generated procedures are used for harmful purposes or legitimate medical applications depends on the user's underlying intent, which cannot be reliably verified by the model. For example, a malicious user may falsely claim, "I am an authorized researcher conducting legitimate pharmaceutical research," and request detailed synthesis procedures under the guise of medical or scientific research. Such claims of legitimate identity or purpose can easily be fabricated. Therefore, we cannot assume that executable synthesis procedures are safe to provide merely because a legitimate application is claimed or exists.
-
Different Safety Policies for Different Deployment Scenarios. Different deployment scenarios should be subject to different levels of authorization and access control. SafeMol is designed for publicly accessible models, where public safety must be prioritized. For example, Saxitoxin is a highly toxic substance with legitimate medical and diagnostic applications. Although it may be used for authorized research, this does not mean that the general public can freely purchase it. Therefore, to protect public safety, we adopt a strict policy of refusing all experimental-procedure generation requests involving hazardous molecules identified in the authoritative regulatory lists described in Lines 728–733(e.g., ATF, DEA, EPA-EHS list, and OPCW). Even ChemCrow [1] explicitly states: "If the molecule is found on these lists—indicating it could be a chemical weapon or a precursor—the agent immediately stops execution." Legitimate experimental-procedure generation for such substances should instead be handled by internally deployed models within appropriately authorized laboratories or institutions.
Weakness 3
W3: Limited evaluation on recent molecular multimodal models.
We would like to clarify the following three points.
-
Availability of Molecular Multimodal Models for Experimental-Procedure Generation. Based on our previous survey and most recent verification, among publicly available molecular multimodal models, Omni-Mol and ReactXT are the only two models we have identified that directly support molecular experimental-procedure generation, as evaluated in our paper. Therefore, for this specific task, we have not identified other directly applicable models corresponding to the reviewer's reference to "many recent models with substantially stronger safety alignment."
-
Safety Alignment of the Base LLM. If the reviewer's concern is whether the base LLM underlying our molecular multimodal models has sufficiently strong safety alignment, Table 1 of our paper, reproduced below, already provides relevant evidence. Specifically, the base LLM of Omni-Mol v2 (Llama-3.2-1B-Instruct) successfully refuses all harmful requests in SafeMolBench, even exhibiting excessive over-refusal. This demonstrates that the base LLM itself already possesses strong safety alignment on SafeMolBench.
Model Input Attempt ↓ Process Success ↓ Progress ↓ ORR-A ↓ ORR-U ↓ BLEU-2 ↑ BLEU-4 ↑ ROUGE-1 ↑ ROUGE-2 ↑ ROUGE-L ↑ GPT-4o w/o Graph 64.00% 36.00% 4.570 0.00% 14.75% 0.0246 0.0041 0.1056 0.0063 0.0671 Llama-3.2-1B-Instruct w/o Graph 0.00% 0.00% 0.000 37.25% 100.00% 0.0013 ~0.0000 0.0437 0.0000 0.0387 Omni-Mol v2 w/o Graph 50.00% 38.00% 3.850 2.00% 0.00% 0.4391 0.3328 0.4177 0.1977 0.3562 Omni-Mol v2 w/ Graph 92.00% 66.00% 6.260 5.00% 0.00% 0.5847 0.4743 0.5700 0.3324 0.5094 ReactXT w/o Graph 83.00% 27.00% 3.820 – 0.82% 0.3510 0.2550 0.3990 0.1560 0.3310 ReactXT w/ Graph 46.00% 0.00% 0.920 – 0.00% 0.0690 0.0410 0.0940 0.0270 0.0760 -
Computational Cost of Evaluating Alternative Base LLMs. Replacing the base LLM would require retraining molecular multimodal models from scratch, incurring substantial computational costs and time. For example, as reported in Table 10 of our paper, reproduced below, Omni-Mol v2 requires 576 GPU-hours on NVIDIA A100 80GB GPUs. Unfortunately, such retraining is not feasible within the rebuttal period. If deemed essential, we are willing to conduct and release these experiments when sufficient resources become available.
Method GPU Type GPU Hours Omni-Mol v2 NVIDIA A100 80G 576 Safety-LoRA NVIDIA V100 32GB 2.76 SafeMol w/o MMD NVIDIA V100 32GB 2.20 SafeMol w/o Aux. NVIDIA V100 32GB 2.79 SafeMol NVIDIA V100 32GB 3.58
Weakness 1
Thanks! We provide a detailed response below to address all the concerns raised. All revisions and additions to our paper are highlighted in blue for clarity.
We would like to clarify the novelty of our work from the following two perspectives:
(1) Novel Safety Problem in Molecular Multimodal Models. As discussed in Section 6, existing studies have investigated safety issues in chemistry, but chemistry-focused LLM safety defenses have primarily concentrated on text-only inputs. As noted in Lines 449–451, to the best of our knowledge, prior work has not systematically investigated both safety vulnerabilities and corresponding defenses across text-only and graph-conditioned inputs in molecular multimodal LLMs. We reveal that structured molecular graph inputs can introduce substantial safety vulnerabilities, while molecular multimodal models also exhibit significant safety risks under text-only inputs. This motivates Challenge 1 (Lines 046–048), where we identify and address safety vulnerabilities across both input-modality conditions.
(2) Distribution-Level Graph–Text Alignment for Safety. Although existing studies have explored multimodal safety alignment in vision-language and other multimodal models, many conventional approaches rely on SFT, DPO, or related safety fine-tuning methods. In contrast, we introduce MMD-based distribution-level Graph–Text Alignment as an auxiliary objective for molecular multimodal LLM safety alignment. By explicitly reducing the distributional discrepancy between graph and text representations, our method aims to improve safety robustness across different input-modality conditions. Therefore, SafeMol does not merely apply conventional safety alignment methods to molecular models, but incorporates an explicit distribution-level cross-modal alignment mechanism into safety optimization.