acceptodds
Withdrawn from ICLR 2027

Pharos: From Safety Verdict to Routing Primitive at Inline Latency

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety filters often report whether a user prompt is harmful, but many applications also need to know which risk it presents. We study this finer-grained task under the 16 risk categories in the AIR 2024 Level-2 taxonomy. Risk Atlas unifies 23 public safety datasets under that taxonomy, preserves source provenance, and adds category-checked synthetic prompts to broaden coverage. Splits are stratified by source and risk category, with duplicate queries removed and newly synthesized prompts confined to training. On 10,000 prompts with independently mapped AIR labels, the labeling model reaches 0.97 macro-F1. Pharos, a compact encoder trained on Risk Atlas, reaches 0.753 AIR Level-2 macro-F1 on the held-out test set, ahead of evaluated released guards and a decoder trained on the same corpus. On an RTX 4080-SUPER, Pharos's median batch-one inference time is 5.36 ms per query. Together, Risk Atlas and Pharos provide a shared evaluation setting and a fast baseline for fine-grained prompt risk categorization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, guardrails, safety classification, content moderation, harm taxonomy, AIR-Bench, synthetic adversarial data, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5mSVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper classifies user prompts into one of the 16 AIR 2024 Level-2 risks or a non-harmful class. It builds Risk Atlas by mapping 23 public datasets to AIR Level-2, using native labels or an LLM judge, and adds judge-filtered synthetic prompts (§3.2–3.3, p.3). It then trains Pharos, a DeBERTa-v3-base classifier (§3.4, p.3). Pharos reaches 0.753 Level-2 macro-F1 on a source-stratified test set, ahead of released guards and same-data decoders (Table 3, p.6).

Critical strengths
  1. Models trained on the corpus are nearly tied on binary F1 (0.975/0.970/0.969) but separate at Level-2 (Table 3, p.6). This shows that binary metrics hide category errors. It is the paper’s main useful finding, but it confirms an expected effect of target-taxonomy supervision.
Critical weaknesses
  1. Limited novelty. Distilling LLM labels and LLM-generated harmful prompts into a small DeBERTa is the recipe of HarmAug (Lee et al., ICLR 2025), which the paper does not cite (§2, p.2). It also omits Markov et al. (AAAI 2023), who built a taxonomy-driven multi-category moderation classifier. The main change is a 17-class label space instead of a binary one. The dataset crosswalk cannot offset this, because Risk Atlas is released only on request (p.7) and is not analyzed (W2).

  2. Unverified labels and an uncharacterized dataset. No human verification is reported anywhere:

  • Labels come from native mappings or the judge (§3.2, p.3).
  • Synthetic prompts are filtered by a model (App. C, p.11).
  • The audit covers only deterministically mapped items, which never needed the judge (§4.2, p.4).

The paper also reports no judge-label share, no per-category distribution, and none of the per-source error analysis it motivates (§1, p.1). Since the same judge labels both training and test data, Table 3 largely measures agreement with that judge.

Questions for authors

Clarifications:

  1. What fraction of test labels is judge-assigned, and do the Table 3 rankings hold on the native-mapped subset? (W2)
  2. Were any labels checked by humans? If so, what were the guidelines, the annotators’ background, and their agreement with the judge? (W2)
  3. Beyond the label space, how does the method differ from HarmAug? (W1)

Suggestions:

  1. Report per-category and per-source statistics, add a human-annotated test subset and a source-held-out split, and ablate the synthetic data.
Reviewer_uNPoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies fine-grained classification of English user prompts into the 16 AIR 2024 Level-2 risk categories plus a non-harmful class. It introduces Risk Atlas, a corpus that aligns 23 public safety datasets with the AIR taxonomy while retaining source provenance, and augments sparse categories with category-conditioned synthetic training prompts. The paper also presents Pharos, a 180M-parameter DeBERTa-v3-base classifier trained on this corpus. Pharos achieves 0.753 Level-2 macro-F1 on a 55,568-prompt test set, compared with 0.721 for an adapted openjev model, 0.696 for a Qwen3-0.6B model trained on the same corpus, and 0.471 for the strongest evaluated released decoder guard. The authors additionally report a median batch-one latency of 5.36 ms on an RTX 4080-SUPER. The submission argues that taxonomy-aligned supervision and fine-grained evaluation reveal distinctions that binary harmfulness evaluation obscures.

Critical strengths
  1. Clear and practically relevant task formulation. The paper clearly distinguishes prompt-risk categorization from severity assessment, policy decisions, and response-safety evaluation. This produces a well-scoped task with a plausible use in downstream routing, escalation, or review systems.

  2. Substantial corpus integration with retained provenance. Risk Atlas combines 23 heterogeneous public datasets under a shared AIR Level-2 taxonomy while retaining source identity and label-assignment routes. This provenance is valuable because it permits source-specific analysis rather than treating all safety prompts as identically distributed.

  3. Sensible initial leakage controls. The authors reconcile splits using stripped query text, remove conflicting and repeated queries, give held-out copies precedence over training copies, and confine newly synthesized prompts to training. These choices address obvious forms of exact-query leakage.

  4. Useful controls for taxonomy-specific supervision. The same-data Qwen3-0.6B experiment and the adapted openjev experiment help distinguish the benefit of AIR-specific training from comparisons against guards designed for different native taxonomies. The reported improvement from 0.268 to 0.721 macro-F1 after adapting openjev is particularly informative about the importance of taxonomy alignment.

  5. Strong efficiency–accuracy result on the reported benchmark. Pharos achieves 0.753 Level-2 macro-F1 with only 180M parameters and a reported median batch-one latency of 5.36 ms. If the benchmark labels and evaluation protocol are validated more comprehensively, this would constitute a useful lightweight baseline for fine-grained prompt-risk classification.

  6. Transparent discussion of scope and limitations. The paper acknowledges that its label audit does not validate every judge-assigned category, that the evaluation is not source-held-out, and that the system does not evaluate multilingual prompts, conversational context, response safety, severity, or multi-label risks.

Critical weaknesses
  1. The label audit does not cover the cases with the greatest annotation uncertainty. The 10,000-item audit is restricted to examples whose native source labels map deterministically to AIR Level-2 categories. The reported 0.97 macro-F1 and 0.98 Cohen's kappa therefore provide meaningful evidence that the labeling model can recover categories when an independent fine-grained reference is available. However, as the authors acknowledge in Section 6, this audit does not validate model-derived AIR categories for sources that provide only binary, broad, or otherwise non-equivalent labels. The paper also does not report what proportion of the train, validation, and test sets uses each assignment route. Because the benchmark scores depend on these reference labels, a representative audit of the judge-assigned portion is important for interpreting the headline result.

  2. The evaluation establishes within-source generalization but not robustness to source or template shift. The authors correctly remove exact duplicate queries and exclude newly synthesized prompts from validation and test. However, the split is stratified within the same source datasets and reconciliation uses stripped exact query text. This protocol does not rule out semantically near-duplicate prompts, recurring templates, or source-specific stylistic cues across splits. The paper explicitly acknowledges that a fully source-held-out evaluation remains necessary for claims about unfamiliar distributions. The reported 0.753 macro-F1 is therefore well supported for the constructed source-matched benchmark, but its broader generalization remains uncertain.

  3. The fine-grained comparison with released guards does not always use a common effective denominator. Section 4.6 states that outputs without an AIR label are excluded from Level-1 and Level-2 scoring, although binary scoring retains harmful-but-unmappable outputs. Consequently, Table 3 uses the same prompt collection but can score different guards on different effective subsets. Moreover, the table reports reachable categories but not the parse-failure counts, unmappable-output counts, or scored denominators that Section 4.6 says are recorded. This makes the magnitude of the fine-grained advantage over guards with incompatible native taxonomies difficult to interpret. A complementary full-test evaluation that treats failed or unmappable outputs as errors or explicit abstentions would make the comparison clearer.

  4. The evidence does not isolate the causes of Pharos's advantage. Pharos is reported for one seed and exceeds the AIR-trained Qwen3-0.6B and adapted openjev controls by 0.057 and 0.032 macro-F1, respectively. The paper appropriately notes that architecture and optimization differ, so these results should not be interpreted as establishing an encoder-architecture advantage in isolation. In addition, Appendix C analyzes the surface similarity of synthetic prompts, but no training ablation measures whether the 83,400 synthetic examples improve overall or sparse-category performance. Multiple-run uncertainty and a with-versus-without-synthesis ablation would clarify which design choices produce the reported gains.

Questions for authors
  1. How much of each split relies on model-derived AIR labels? Could the authors report, by split, source, and AIR category, the number of examples assigned through direct native-label mappings versus judge-based classification? Has a source- and category-stratified human audit been conducted specifically on the judge-assigned validation and test examples? This would directly address the main uncertainty concerning the benchmark's reference labels.

  2. How sensitive is Pharos to source and template overlap? Beyond the reported exact-query deduplication, did the authors measure semantic-near-duplicate or shared-template overlap between training and test data? Could they provide per-source performance or, if available, a source-held-out evaluation? These results would clarify whether the model generalizes beyond recurring patterns in the known source datasets.

  3. What are the effective scoring denominators in Table 3? For each comparator, how many test prompts resulted in parse failures, unmappable harmful outputs, or other outputs excluded from Level-1 or Level-2 scoring? How do the rankings change if all 55,568 prompts are scored and excluded outputs are treated as errors or explicit abstentions? This information could materially affect the interpretation of the comparison with released guards.

  4. How robust are the reported gains, and what does synthetic augmentation contribute? Could the authors provide variation across training seeds or paired bootstrap confidence intervals for Pharos and the AIR-trained controls? Additionally, how does Pharos perform when trained without the 83,400 synthetic examples, particularly on the sparse categories targeted by augmentation? These analyses would clarify whether the relatively small gains over the trained controls are stable and whether synthesis contributes materially to performance.

Reviewer_orMMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper classifies user prompts into the 16 AIR 2024 Level-2 risk categories plus a non-harmful class, as a single-label task. The authors build Risk Atlas by merging 23 public safety datasets (about 595K prompts) into the AIR taxonomy. Labels come from direct mapping where it is unambiguous and from an LLM judge (DeepSeek-V4-Pro) otherwise, and about 83K LLM-generated synthetic prompts are added to training. They then fine-tune DeBERTa-v3-base (“Pharos”) with class-weighted cross-entropy. On an in-distribution, query-disjoint test split, Pharos reaches 0.753 L2 macro-F1. This beats released guards evaluated through native-to-AIR mappings, a Qwen3-0.6B classifier and an adapted openjev-4B trained on the same data. Median batch-one latency is 5.36 ms on an RTX 4080-SUPER.

Critical strengths
  • The paper delivers a practical artifact: a unified, provenance-preserving AIR-L2 corpus and a fast 180M classifier. This could be useful to practitioners who need category-level routing at low latency.
  • The comparison protocol is transparent. Reporting reachable categories, unmappable outputs and parse failures makes the comparison with released guards more honest, and the gap between binary and L2 scores is a useful observation.
Critical weaknesses
  • The contribution is limited. The method combines existing datasets, LLM-judge relabeling, standard synthetic augmentation (following SAGE-RT) and off-the-shelf DeBERTa fine-tuning. Each component is established, and the paper offers no new modeling idea, training objective or analytical insight. For a venue like ICLR, this is the main reason for rejection.
  • The evaluation favors the proposed model. Test labels are largely produced by the same judge and rubric used for training. The label audit covers only the deterministically mapped subset, so it does not validate the judge-assigned labels. The split is source-matched, with no source-held-out evaluation. Released guards are scored on a taxonomy they were not trained for. As a result, the headline 0.28 gap mostly reflects training on the target taxonomy, and possibly learning the judge’s labeling habits, rather than a better model.
Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.