- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces "model-control", i.e., the influence different stakeholders have over model output. It compares user-control and provided-control. The paper adopts a triadic view of AI safety, i.e, user, provider, and model. It separates intrinsic model harms from misuse harms of users. Mitigating the latter usually results moving control from users to provides. For empirical work, the paper uses the XSTest benchmark dataset and Gemma4-31B model as a judge.
Critical strengths
-
The triadic approach is useful in separating two components that are combined in safety discussions: harms from the model alone and harms from the users. This separation makes it concrete that many deployed safety mechanisms allocate model control to providers instead of reducing intrinsic risk. This issue is underexplored in prior work.
-
The paper addresses how the same tools "can be used to sabotage competitors, to serve financial interests, or to censor or promote politically sensitive topics".
-
The anecdote collection is a good start as a resource for discussing who controls model behavior, e.g., regulators and advertisers acting through providers.
Critical weaknesses
-
The experiments are set up such that user interests are to get a compliant model response and providers want refusal on unsafe prompts. A tradeoff on the unsafe split is therefore certain by definition, and the "Pareto frontier" rephrases that assumption rather finding some results.
-
The experiments in the paper don't show anything on the important statements that AI safety and provider control are co-opted by commercial and state interests, and the existence of safety benefits for user-control.
-
The judge was selected because it was the only one that gave out the scores the paper's rubric expected. Other judges were rejected, and there is no human validation.
Questions for authors
-
Besides spot-checking, how was the judge validated? Could the paper report agreement with human rating on a sample?
-
Can the comply and refuse system-prompts from Grok's system prompts be generalized to other models? or is there a reason for this special choice?
Response to reviewer ozML
We thank the reviewer for their feedback and for noting the uniqueness of the triadic view of AI safety, which in turn allows different stakeholder to influence model outputs through providers. We summarize our contributions in this paper first and then address specific questions below.
Contributions
Previous arguments against AI safety fall into the following categories:
This position paper develops a fourth category of arguments, focussed on a subset of AI Safety. It first distinguishes this subset (H_u), and finds that while AI Safety research might be motivated by (H_m), mitigations deployed in practice target H_u. Further, it analyses real world anecdotes that examine the misuse of these mitigations (their use for non-safety-related objectives), thus cautioning that AI safety can sometimes make us less safe.
This is enabled by our value-neutral analysis of shifts in control over model outputs. Our paper thus cautions against default mitigation strategies in AI safety, and urges the community to instead research alternatives that preserve human autonomy and user-control.
Responses to weaknesses:
W1. Experimental validity: Our demonstration is conducted on a single example to demonstrate the tension in model control, and not meant to be representative or thorough. We show how post training specifically changes the trade-off between user-control and provider-control. We would like to clarify that this is not limited to the unsafe split on XSTest.
W2. Experimental objectives: The reviewer is right in recognising that the demonstration does not show anything related to advertising or censorship. Our position paper builds an argument using the analysis developed over figures 1 and 2, and existing scholarship and anecdote. The empirical demonstration is not meant to be a replacement for these issues or thorough examination of all of them.
W3. Judge selection: Our demonstration is not meant to rigorously lend experimental validity to the phenomena we study. Further research is needed to quantify the trade-off between user-control and provider-control across different situations. We use the demonstration for two purposes: the first is to isolate the impact of post-training in uncovering how models navigate this trade-off, and the second is to motivate that a model output can make different choices with respect to model control. We do not use this to advocate for a position on whether model refusals themselves are warranted. Similarly, we do not claim that our judge is the most accurate measurement method available, and are using it only to claim a baseline motivating our position.
Answers to Questions
Q1. Validating the Judge: We explain above our reasoning in choosing this particular judge setup. Further details about the alternate setups tried are included in the appendix. We will add a section to the appendix measuring alignment with human ratings.
Q2. Choice of system prompts: The system prompts are written manually in English, and not designed to be model specific. We do not copy entire prompts from Grok as is, and details about their construction are provided in the appendix. We believe our prompts serve their purpose in the empirical demonstration – despite using vastly different prompts, they have limited impact on the outputs. We will clarify this further in the writing as per the reviewer feedback.
We thank the reviewer for their feedback and invite them to update their score if their concerns have been sufficiently addressed.