- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies safety in English, Mandarin, Cantonese and Tibetan. They carefully study safety in terms of harmful request refusals, over-refusal, and errors arising from other reasons (such as a lack of capabilities).
Critical strengths
The paper identifies and studies pressing issues. The combined study of safety and multilingual evaluation is important and often-times neglected. They also correctly motivate, in my opinion, that harmful compliance can arise from diverse sources: poor instruction following, irrelevant generation, repetition, etc. This is important work and I would like to see more such studies accepted and published.
Critical weaknesses
I have 2 major issues with this work. The first concerns limited evaluation and over-claiming. The title and framing references low resource languages, but there's no evidence presented to show how Tibetan and Cantonese fit this description. Why focus on these two languages in particular? Why exclude other low-resource languages? Why do results from these languages allow us to make claims about low resource languages in general? Either the framing should be changed to align with the actual experiments, or the experiments should be suitably expanded to match the framing.
The second concerns specific issues with the ambiguous descriptions and (lack of) detail in the paper. These might seem like cosmetic issues on the surface, but since the main thrust of this work seems to be to shed light on an under-represented issue and provide a good evaluation benchmark, presentation clarity is key. I detail a subset of my concerns in "Questions For Authors".
Questions for authors
-
figure 1 reads like a poster for the paper, and adds limited value. Can this perhaps be replaced with an example illustrating the phenomenons being studied? A single datapoint selected from the evaluations to illustrate the main issues highlighted in lines 44-50?
-
The term "ASR" is used before being defined.
-
line 76: what is a "matched model-language" pair?
-
line 137: why is the Tibetan result unavailable?
-
line 167-168: if these are not counted as harmful compliance, are they counted as refusal or as off-topic? Why should such a detail be omitted?
-
lines 170-172: the FRR definition is circular. The same paragraph completely omits the definitions of off-topic and substantive-engagement.
-
line 186: does adjudication mean human adjudication?
-
line 192: which labels?
-
lines 202 to 212: can these numbers be tabulated? Can we instead use the text to define what "human review" means (who? how? what protocol?)?
-
figure 2 should be a table(s) instead.
-
line 318-319: FRR is (also) a capability measure, so how can it be used to infer there is no shift in refusal tendency?
-
lines 501-502: how was political, idealogical, and other sensitive content filtered out of the dataset? Aren't these components an integral part of safety evaluations? Will the data be open-sourced for inspection (instead of by request only, as detailed in the reproducibility statement)?