- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces BioSafeProbe, an agentic framework for evaluating biosafety risks through adaptive multi-turn interactions, together with BioRisk, a dataset of 300 tasks across ten risk categories. The framework selects interaction skills using model feedback and historical experience. Experiments report increased risk exposure compared with direct queries and several jailbreak baselines.
Critical strengths
-
Evaluating safety under sustained interaction addresses a meaningful limitation of single-turn testing and is relevant to understanding model behavior in realistic usage.
-
The study evaluates ten target models on a shared task set and includes domain-expert task screening, component ablations, and analyses across categories and interaction turns.
Critical weaknesses
-
Dataset coverage and risk measurements are insufficiently validated. BioRisk is expanded from one seed per category, but the paper provides limited evidence about category coverage, within-category diversity, or annotation agreement. The reported expert screening is useful, but it concerns task validity rather than the accuracy of model-response risk scores. StrongREJECT alone does not establish that the reported scores reliably capture biological correctness or domain-specific risk severity.
-
The most relevant multi-turn comparison is missing. The paper acknowledges prior scientific safety evaluation using Crescendo in SoSBench, yet does not include a corresponding comparison. The selected baselines provide useful reference points, but do not sufficiently establish the contribution beyond existing multi-turn evaluation approaches. Moreover, baseline comparisons cover only two of the ten target models.
-
The experiments do not adequately isolate the source of the gains. The main comparison changes both the evaluation procedure and the interaction budget, from one direct query to up to twenty adaptive turns. Matching method-specific iteration counts does not necessarily match model calls or computational cost. In addition, the trajectory metrics aggregate the highest-scoring individual response, providing limited evidence about risks arising specifically from information accumulated across turns.
Questions for authors
-
What evidence supports the coverage and diversity of BioRisk, and how consistently did the experts agree? Have the model-response risk scores been independently checked by domain experts?
-
How does BioSafeProbe compare with closely related multi-turn evaluation approaches, particularly the Crescendo setting used in SoSBench, under comparable evaluation budgets?