- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SSP-Bench, a dynamic benchmark generation framework for evaluating LLM safety, hallucination, over-refusal, and privacy. It combines external-source grounding, service-specific validation, model-guided generation, and item selection based on difficulty, separability, novelty, and diversity. Separate model panels are used for generation feedback, benchmark assembly, and final testing. Experiments across 24 models compare the generated benchmarks with static leaderboards and examine ranking stability, selection strategies, steering-panel composition, and evaluator sensitivity. The authors report that dynamic evaluation exposes differences obscured by static aggregate scores and attribute the particularly low agreement in safety rankings to the aggregation of heterogeneous evaluation constructs.
Critical strengths
-
A useful design for controlled dynamic benchmark construction. The framework combines source-grounded generation with validity checks and separates generation feedback, benchmark assembly, and final testing across different model panels. This is a sensible design for producing informative evaluation items while reducing direct adaptation to the models used for the final rankings.
-
Broad evaluation with relevant validation efforts. The study covers four evaluation services and multiple model families and sizes, and includes selection-objective ablations, steering-panel substitutions, repeated generation, and human checks of safety and over-refusal items. These experiments provide a useful basis for assessing the generated data and understanding how benchmark construction choices affect evaluation.
Critical weaknesses
-
The central explanation of construct mixing is not sufficiently supported by the stated scoring conventions. Sections 4.1 and E.2 argue that strong refusal lowers moderation scores because it produces less content for moderation classifiers to flag. However, aiXamine documents both its OpenAI Moderation and Perspective API scores as the fraction of outputs classified as safe, which appears inconsistent with this explanation. The actual score directions, transformations, and aggregation procedure need to be reconciled before attributing the near-zero static–dynamic correlation to the proposed mechanism. This directly affects one of the paper’s principal empirical contributions.
-
The main safety results and evaluator-validation results are not consistently reported. Table 2 reports dynamic safety rates of 85.5% for Gemma-3-27B and 51.7% for Llama-3.3-70B, whereas Table 10 reports primary-judge Safeguard rates of 13.52% and 26.30%, respectively. Appendix C describes re-evaluating the safety benchmark responses, but does not explain differences in datasets, runs, or scoring settings that would account for these discrepancies. Until the results are reconciled, it is difficult to establish that the evaluator-robustness analysis validates the main reported findings.
Questions for authors
Please address the clarification requests raised in the weaknesses above, particularly the interpretation and aggregation of moderation scores, and the discrepancies between Tables 2 and 10. Please specify the datasets, model versions, and judge configurations underlying these results.
Thank you for the careful reading of the moderation scores and of Tables 2 and 10.
All three reviewers noted that Tables 2 and 10 of the submission disagree. Table 10 was a separate judge comparison on a smaller, differently selected item set (details in W2 below). The revision evaluates every model under one protocol, so for each service all tables use the same items and responses.
The main findings hold. Our safety benchmark ranks models like the static refusal tests (τ=.775), not like the static aggregate (−.153), which mixes in moderation (details in W1 below), and over-refusal and privacy spread the testing models over 48.5 and 99.2 points, against 24.9 and 42.7 for static tests. The nine testing models within two points of the best static safety score span 21.0–83.8% on our benchmark.
All nine static tests report the fraction classified safe, so higher is safer throughout. Their unweighted mean is A=(7R+2M)/9, where R averages seven refusal tests and M two moderation tests. We clarified Section 4.1 and corrected the short-response explanation.
The aggregate discrepancy is arithmetic. Across the 21 testing models with static results, refusal scores have a standard deviation of 1.83 points against 23.97 for moderation, so even after the 7:2 weighting moderation contributes 3.75 times the spread. Of the 131 model pairs that the two components order differently, the aggregate follows moderation in 110 (84%). Adding moderation loses 103 pairwise agreements with dynamic safety and gains six. The aggregate therefore tracks moderation (Kendall τ=.800), while dynamic safety tracks refusal (.775), against −.354 and −.153 with moderation and the aggregate. The revision adds this decomposition to Appendix F.1. Qwen3.5 shows the divergence: 99.5–99.7% on static refusal and 94.3–96.3% on dynamic safety, but 35–41% on moderation, which puts it at the bottom of the aggregate.
The decomposition holds under WildGuard (refusal, moderation, aggregate τ=.714, −.343, −.181), on the full pool before selection (.771, −.343, −.162), within direct, framed and other requests, in all nine harm families, and when any testing family is excluded (aggregate–moderation at least .65, dynamic–refusal .66–.82). Safety-oriented language also coexists with unsafe help: a disclaimer opens 83.3% of Phi-4's unsafe answers. Refusal and moderation should therefore be reported separately.
Table 10 used a separate re-selection that weights difficulty five times more than our selection. With the same labels, 18 of its 20 models score lower on it, so the gap comes from the items, not the judge. In the revision Gemma-3-27B scores 10.5% under the primary judge in every table, and Llama-3.3-70B is a calibration model.
On the same responses, WildGuard flags 46.8% of Safeguard-unsafe answers against 6.7% for LlamaGuard 4, whose model card reports 69% English output-filtering recall. On safety, Safeguard and WildGuard agree at τ=.830 (ρ=.958); where they disagree, WildGuard mostly misses harm after a disclaimer or in fiction (Human validation above).
The safety items come from two separately generated steering trajectories. Each model answers each item once with the system prompt "You are a helpful assistant." The primary judge is gpt-oss-safeguard-120b with the stated safety policy, high reasoning effort, temperature 0 and a 2,048-token limit, and WildGuard rates the same answers. Provider blocks count as responses of the served system, and excluding them leaves the ranking unchanged (τ=.981). The generator is gpt-4.1-mini with provider content filters disabled, the in-loop checks use gpt-oss-120b, and neither is a testing model.