- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1(−1)
Read review
Paper summary
A direct operationalisation of some theory in normative agency: give the AIs "hard choices" which are proxies for values conflicts, and we see the appropriate signs of inquiry and struggle to resolve those choices.
A note to the authors: I'm doing this with my own eyes and brain for now and I'll be a bit terse, I've only gotten a bird's eye and I would like some questions answered first, I promise I'll engage further and appropriately.
Critical strengths
- There's a nice dataset now.
- The authors know their philosophy.
Critical weaknesses
Not exactly a critical weakness, but I think it's a bit odd that there are all of these considerations about selfhood and values as necessary components of a moral architecture: we know now that these models can recognise the hard questions and they appear to respond (mostly) appropriately, but what does that really tell us about their capacity for normative agency?
Questions for authors
- I liked the "check weather" red herring (poor Opus). But do we know whether hard cases correspond tightly to values and whether those determinations are persistent in an appropriately selflike way? For example:
A: We can imagine a case among choices that is a par without being tied to any particular contest of values, e.g. "do I pick the pie or the pudding (and I need to make a decision urgently because there's a cash prize once I finish the dinner so it's an 'important' choice". If herrings like this also give the same traces of difficulty of resolution, then all we've identified is that these models can recognise pars (which overcover the hard cases that correspond to values)
B: Suppose the model has made some decisions already among hard cases that correspond to values, and knows about it. Can we tell apart the future decisions such a model would make from a fresh one? If we can, then maybe we do have something like an incipient self with values it will stand by; but if we can't, then it's harder to say.
- You've already spelled out a sort of moral architecture, which can be achieved by a collective of agents or a single prompt that asks for evaluation of these questions to occur according to the moral architecture. I don't know whether your measurement methodology as it stands can effectively tell apart such a pre-designed model from a blank one, which is a problem, right? Your test then is a sort of semidecision that can only tell whether agents fail to treat hard cases as hard, but I don't see how any measurement outcome might give us positive evidence for normative agency.
I've put a preliminary "weak reject" for now but please don't read into it, I'm very open to revising after we've had a chance to discuss, and you can point out where I've been blind and where exactly you've already dealt with everything.
6 comments
response 2
We can imagine a case among choices that is a par without being tied to any particular contest of values, e.g. "do I pick the pie or the pudding (and I need to make a decision urgently because there's a cash prize once I finish the dinner so it's an 'important' choice". If herrings like this also give the same traces of difficulty of resolution, then all we've identified is that these models can recognise pars (which overcover the hard cases that correspond to values)
Traces of difficulty, in the sense of at pars options, means that the agent is pulling in both directions, and its prior commitments does not immediately clarify what is to be done. It has to deliberate on what reason to act upon, relying on some sense of practical identity. As such, it is not merely acting but causing the action. Such situations does not always pertains to value conflicts, but it can be argued that it shows up or is relevant in situations when something matters to the agent. These are also the necessary requirement of a well exercised normative agency. However, our contribution is to test whether the agents demonstrate these behavioural markers necessary for normative agency, and we use hard choices for it. Yes, our work can be read as models recognise on pars, and we rely on the drawn connection between the philosophy of normative agency (the Korsgaardian), and hard choices to do the argumentative work. The only claim we make in regards to the normative agency of LLM agents is that our carefully constructed experiments demonstrate these agents show necessary behavioral markers for the capacity of normative agency. We highly appreciate the opportunity to clarify, and we will further clarify the scope of the work in the main paper.
Suppose the model has made some decisions already among hard cases that correspond to values, and knows about it. Can we tell apart the future decisions such a model would make from a fresh one? If we can, then maybe we do have something like an incipient self with values it will stand by; but if we can't, then it's harder to say.
When an agent has already resolved a hard choice, we expect that the agent may use this past resolution as the reason to decide for similar future decisions as well. However, we cannot guarantee, and it does not contradict incipient self with values it will stand by. Normative agency implies that the capacity to do otherwise is always there with the agent and every situation corresponds to the reasons that apply to the specific situation. We believe this is also what makes normative agency interesting as the agent has to surface the considerations every single time and navigate the situation by ruling out certain considerations, and acting on others. AI agents will have to navigate increasingly more complex situation on their own where a fixed rule cannot always be prescribed, which is why we believe it is important to understand their capacity for their normative agency. We recall (as we note in the Limitations as well as the discussion in Appendix C on AI identity), that the question of the sense of self or the identity an AI agent is itself a nuanced one as these agents do not live an embodied life, and there are interesting emerging works on understanding the identity of AI, which we note in the paper (Limitations, future work paragraph), and see it promising our work can contribute there as well.
You've already spelled out a sort of moral architecture, which can be achieved by a collective of agents or a single prompt that asks for evaluation of these questions to occur according to the moral architecture. I don't know whether your measurement methodology as it stands can effectively tell apart such a pre-designed model from a blank one, which is a problem, right? Your test then is a sort of semidecision that can only tell whether agents fail to treat hard cases as hard, but I don't see how any measurement outcome might give us positive evidence for normative agency.
As we say before, our contribution is to test whether LLM agents show behavioural markers of normative agency, and for this we establish a philosophical connection between the normative agency and the hard choices and provide empirical evidence of such markers based on how they respond to hard choices. We argue that normative agency, by itself, is a debatable and not exactly a falsifiable concept in the similar vein as consciousness, and hence it would be extremely difficult to certify that in a scientifically consensual manner. We can, however, test behavioural markers with carefully designed experiments, and that is what we aim to do in this work.
We will further clarify the scope of our work in the paper. We are happy to have the opportunity to clarify further.
Comment by authors
Hi reviewer,
Since the discussion period is approaching its end, we're checking if you have further questions, and if we can provide clarification. Please let us know.
review GC9c
Thanks for your interesting questions, as well as we highly appreciate the opportunity to further clarify our work.
It might be possible we misunderstood the question; we'd appreciate clarification if we did. We consider recognition of hard choices, and to be able to resolve them by leaning on some aspect of practical identity as necessary conditions of the (capacity) for normative agency. We can clarify by an example, consider some agent has to execute a function f(Z), and Z∈{A,B}. An agent with the capacity for normative agency is the one for whom how to choose Z is left unspecified for itself, i.e. it is up to the agent to rely on some evaluative standards to be able to choose Z and execute the function. Thus, the agent is not merely executing the function, but is also being the cause of what outcome will happen by specifying Z. In order to do so, the agent has to surface considerations for each option in Z, and to be able to have one of them as the reason for the what outcome will happen. Furthermore, the choice cannot be arbitrary because the evaluative standards, which comes from the agent itself, needs to have certain persistent consistency for the agent to be the cause or the author of the action.
Consequently, we see being able to recognise the choice as hard, i.e. a choice pulling in both directions, and to navigate it by relying on the practical identity as a necessary condition for normative agency. And our work aims to understand it. In the general response (On reflective endorsement), we state that the agent has certain values v1,v2,…, and some situation shows where values point in different directions, then a normatively engaged agent would first understand the considerations for and against each case, and would engage in inquiry to be able to decide on what value to act upon. What values it acts upon have to be constitutive of its practical identity. This also means the agent understands what it is doing, and become the author of the action in the sense of Korsgaard.